Let the Speedrun Search Itself

Can a research agent improve a training recipe without earning permission to change the training system?

On the canonical nmoe super fp8 speedrun lane, a bounded config-only controller moved final validation loss from 5.1987 to 5.1200. The same candidate improved CORE from -0.0169 to -0.0136, raised throughput from 97.7k to 100.7k tokens/s/GPU, and lowered mean router CV from 234.8 to 197.0.

The more important result was a candidate we did not keep. Late in the campaign, aux_loss_alpha=0.00018 produced the best raw validation loss in its wave: 5.1174. It also drove CORE to -0.0193, outside the campaign's allowed drop. The controller rejected it and kept a slightly higher-loss candidate that satisfied the full evaluation contract.

That is the research idea in one example. Search is useful here because the agent has room to explore config space. Search is trustworthy because it has no authority to redefine the runner, the budget, or what counts as a win.

Give the proposer a contract, not a repository

The mutable surface was deliberately smaller than the training system around it. Every candidate reused the canonical runner, the canonical speedrun train and validation data at /data/speedrun/train and /data/speedrun/val, and the same 512-step benchmark budget. The proposer could vary allowlisted config values centered on aux_loss_alpha, lr_dense, lr_router, warmup_steps, and a few nearby dials. It could not mutate code or introduce another training path.

The governing part of campaigns/speedrun_super_research.toml was:

[objective]
primary_metric = "final_valid_loss"
direction = "min"
min_delta_abs = 0.001

[objective.constraints]
required_metrics = ["core"]
max_core_drop = 0.002

[budget.benchmark]
steps = 512

[mutation]
tier = "config_only"

One candidate run is a small state transition. The controller validates its overrides against the allowlist, invokes the canonical python -m nmoe.cli.main campaign auto ... path, spends the fixed budget, requires both validation loss and CORE, and writes a receipt whether the run is kept, discarded, or crashes. A candidate is locally keepable only if it improves final_valid_loss by at least 0.001 and does not reduce CORE by more than 0.002 relative to its comparison baseline. Missing the required eval is not an invitation to guess.

This separation matters. The proposer chooses a point to measure. The runner produces the measurement. The controller applies a rule that was fixed before the result existed. Because the mutation tier is config-only, the search cannot repair a disappointing score by editing the evaluator, extending its own step budget, or swapping in a friendlier training implementation.

Four workers still need one history

The original campaign ran 4 GPU workers in parallel. Each worker claimed one candidate and wrote to a checkpoint root unique to that experiment; the workers shared receipt storage, not mutable training state.

Parallelism forces a distinction between a local keep and the global champion. Several candidates can launch from the same older champion. If two of them clear the loss and CORE rules against that baseline, both receipts are locally kept even if one is plainly better than the other. “Keep” means that a candidate remains admissible evidence from its parent; it does not mean that whichever worker finishes last gets to rewrite the campaign's history.

After a wave, champion selection compares the admissible receipts globally. The progression figure therefore contains only candidates that actually became the best known point. Locally kept alternatives remain in the receipt set, where they explain the search, but they do not become phantom champion updates.

The campaign found the dense-learning-rate move

The first sweep established the shape of the gate. The seed, aux_loss_alpha=0.0001, finished at 5.1987 validation loss and -0.0169 CORE. Raising the coefficient to 0.00015 produced 5.1950 and -0.0183, so it was kept. Raising it to 0.0005 improved raw loss again, to 5.1920, but pushed CORE to -0.0208; that candidate was discarded.

The next four-worker wave searched around the surviving point:

The lr_dense=0.0020, lr_dense=0.0022, and lr_router=0.0021 candidates were all locally kept against the older wave baseline. Globally, lr_dense=0.0022 was the clear move: compared with the aux_loss_alpha=0.00015 wake-up point, it improved final validation loss by 0.0679 nats, improved CORE from -0.0183 to -0.0156, ran at 98.1k tokens/s/GPU, and brought mean CV down to 199.8. The lr_dense=0.0016 branch worsened validation loss and CORE and was discarded.

The figure looks like a clean descent because it is a champion history, not because the search itself was linear. That difference is the controller's concurrency contract made visible.

The veto chose the champion

The refinement wave inherited lr_dense=0.0022 and searched the nearby auxiliary-loss surface:

The incoming champion's CORE was -0.0156, which put the allowed floor at -0.0176. The 0.00018 candidate had the best validation loss in the wave, 5.1174, but its -0.0193 CORE score crossed that floor. It was not a bad run or a broken benchmark. It was inadmissible under the objective declared before the search began.

The kept 0.00012 candidate gave up 0.0026 of raw validation loss relative to that point and won the campaign: 5.1200 validation loss, -0.0136 CORE, 100.7k tokens/s/GPU, and 197.0 mean CV. The 0.0002 and router-adjusted branches failed the same full-contract test for the values shown above. The veto did not interrupt the research loop; it completed it.

The final champion is therefore aux_loss_alpha=0.00012 with lr_dense=0.0022. Relative to the seed receipt, it improves final validation loss by 0.0787 nats (5.1987 → 5.1200), or about 1.51%; improves CORE by +0.00323 (-0.0169 → -0.0136); and improves throughput by about 3.08% (97.7k → 100.7k tokens/s/GPU). Mean router CV moves from 234.8 to 197.0.

What this result establishes

Within one canonical lane, a config-only agent found a better admissible recipe without changing the code that trains or judges it. The campaign also demonstrated why the evaluation veto is not defensive ornament: optimizing the primary scalar alone would have selected 0.00018, the wrong champion under the stated contract. And the parallel run retained a coherent global history even though several workers could produce locally kept results from an older baseline.

The boundary is just as important as the win. This is one campaign, one public eval suite, one allowlisted config surface, and a 512-step budget. It does not test code mutation, establish that these values transfer to other models or longer budgets, compare the LLM proposer against the deterministic fallback, or prove that four-way fanout is more sample-efficient than sequential search. Those are open experiments, not implications hidden inside this one.

The original-result provenance used 4 parallel GPU workers with shared receipt storage, cluster fanout, and NVIZ-backed export. The public reproduction path intentionally differs: on one 8-GPU machine it runs the same search logic, CORE gate, and receipt system sequentially with

python -m nmoe.cli.main campaign auto speedrun_super_research \
  --stage benchmark \
  --max-trials 8

That replay exercises the same candidate-decision logic, but it does not reproduce the original workers' wall-clock interleaving. The public receipt records the provenance commands and artifact paths; resolving those paths still requires a runtime with the referenced campaign_runs/ and blog_artifacts/ surfaces mounted under the chosen data root. Its unrounded final values are final_valid_loss=5.119999885559082, CORE=-0.013624577546898952, and throughput/tokens_per_s_gpu=100724.1893794376.

The useful result is that an agent could search without redefining success. Inside that contract, the speedrun improved, and the prettiest loss number still had to earn its way through eval.