Self-play

Search can teach the model in two ways: its games become training decisions, and policy rollouts provide measured winrates for the judge.

~10k

searched games across a 24-deck matrix

37.8M+

positions labeled for judge training

Direct gameplay training

We generated roughly 10,000 searched games on an NVIDIA RTX 3090, repacked them into decisions, and fine-tuned the existing model. This turns stronger play into a training signal for the next generation.

v22 vs v37 fine-tune, policy-only. Each cell shows the gain in percentage points for one deck pair.
Generate self-play games with exploration on moves sampled, temperature 0.7 Pack decisions become training targets value and policy per decision Fine-tune a candidate from the previous best on the latest days of play Gate candidate vs. gate opponent majority of a set, not one game a promoted candidate becomes the next generator generate → pack → fine-tune → gate, forever
The flywheel. A candidate is accepted only if it holds its win rate against the gate opponent and beats the previous best, and the comparison is a set of games decided by majority, not a single match.

Early rules pruned weak lines and suggested alternatives without overriding evaluation. As the model learned those behaviors, we removed the rules with no loss in performance.

Winrate gains were modest over the two rounds we could afford. They suggest a process that may compound, but do not yet establish a long-run scaling trend.

Winrate approximation training

The judge also has a direct target: measured outcomes from a position. This method labels policy-only playouts rather than mining searched games.

  1. Select positions from policy-only self-play.
  2. Fork each selected position 1,000 times with different random seeds.
  3. Play each fork to completion with the policy and record its winrate.
  4. Train the judge toward that measured winrate.

GPU batching makes this far cheaper than mining searched games. We labeled over 37.8 million positions, reducing the judge’s mean prediction error by roughly 2 percentage points.

Judge prediction error fell from 11.87 to 9.59–9.97 percentage points (16–19% less error than BASE). B1000 gameplay showed no clear strength gain; all paired 95% confidence intervals include 50%.

We observed no clear strength gain in searched gameplay. We hypothesize this is due to insufficient scale. Even with 37.8 million positions played out, they originated from only 37,800 root positions. We believe this is not enough data to avoid overfitting, and therefore plan to continue this experiment.

With sufficient compute, we believe this method can scale up significantly. Labeling millions or billions more positions can improve judge accuracy more, which improves search recommendations. Whether or not this method is more efficient than mining search gameplay remains to be seen.

See Search for how the judge guides planning, and Performance for the post-deadline model comparison.