Skip to content

Support num_simulations=0 in all policies. - #123

Open
carlosgmartin wants to merge 1 commit into
google-deepmind:mainfrom
carlosgmartin:num_simulations_zero
Open

carlosgmartin wants to merge 1 commit into
google-deepmind:mainfrom
carlosgmartin:num_simulations_zero

Conversation

@carlosgmartin

Copy link
Copy Markdown
Contributor

With no simulations, a policy should act based on the prior. Currently, it doesn't:

  • gumbel_muzero_policy fails with IndexError: index is out of bounds for axis 1 with size 0.
  • muzero_policy and stochastic_muzero_policy act uniformly at random over all actions, including invalid ones.

The cause and fix for each is described below.

gumbel_muzero_policy

search traces the simulation loop body even when it runs zero times. Tracing reaches gumbel_muzero_root_action_selection, which indexes a table with one column per simulation (action_selection.py:147), so it fails with:

IndexError: index is out of bounds for axis 1 with size 0

Fix: skip the loop in search when num_simulations == 0. The existing post-search code then acts on the prior: the action is argmax(gumbel + logits) over valid actions, and action_weights is the prior restricted to valid actions.

The behavior for num_simulations >= 1 is unchanged.

muzero_policy and stochastic_muzero_policy

action_weights comes from Tree.summary().visit_probs, which falls back to 1 / num_actions when the root has no visits (tree.py:110). This ignores both the prior and invalid_actions.

Fix: when num_simulations == 0, use the softmax of the root's prior logits, which already include Dirichlet noise and exclude invalid actions.

The behavior for num_simulations >= 1 is unchanged.

Tests

Added test_gumbel_muzero_policy_without_simulations and test_muzero_policies_without_simulations. Both fail without this change and pass with it. All existing tests pass.

With no simulations, the search loop is skipped, and the MuZero and Stochastic MuZero policies act on the root prior, which excludes invalid actions, instead of the uniform visit probabilities of an unvisited root.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant