Fewer Tokens, More Time

I wanted a research loop fast enough to be wrong in public before I had time to become attached to the mistake.

Keller Jordan's modded-nanogpt speedruns were the obvious provocation: a small dense model, a fixed target, public logs, and enough pressure on the recipe that an idea could die in hours instead of surviving for weeks on good manners. I wanted that tempo inside nmoe. The extra complication was sparsity. A dense speedrun can ask, with only a little hand-waving, how quickly the model learned. An MoE speedrun has to finish the sentence: quickly in tokens, quickly in wall time, or quickly after paying the routing bill?

The first complete matrix answered by refusing to pick one. In BF16, MoE-256 reached the validation-loss threshold after 2.55B logged tokens; dense needed 4.36B. Dense, however, spent 21.8 minutes in the training loop and MoE-256 spent 59.4, with setup, validation, and CORE excluded. I had built one race and accidentally installed two clocks.

I needed a fixed annoyance

Before I let the sparse models compete, I wanted our dense lane to miss something public and immovable.

The anchor was Keller's June 2024 AdamW record. Its public log reports tel = 3.275959 at step 9536; the accompanying code commit pins the recipe neighborhood. My first nmoe calibration kept the broad silhouette—9536 steps, 256 × 2048 tokens per step, GPT-2 tokenization, AdamW at 1.8e-3, and a WSD schedule—without pretending to be an exact reproduction. nmoe had untied embeddings, SwiGLU instead of GELU, μP-style initialization, and a different warmup/warmdown decomposition. Those are not footnotes if the claim is parity, so the claim was not parity.

At the shared step 9536, our lane landed at 3.480672: about +0.205 nats behind the public log.

The miss was useful precisely because it was annoying. A private benchmark can drift until every run looks heroic relative to the run immediately before it; a public number is less cooperative. I rounded the anchor to 3.28 and used it as the closure matrix's stop threshold. The newer dense lane first crossed it at the validation on step 8320.

There was another piece of embarrassment to preserve. The January lanes that first shaped this post carried eval_tasks=core in their configs while eval_enabled=false. They were loss-only runs wearing an evaluation-shaped hat. The 3 × 3 closure was the first pass that actually returned end-of-run CORE for every cell.

The swap was not as clean as the table wanted

The closure matrix crossed dense, MoE-64, and MoE-256 with three expert-precision profiles: BF16, FP8, and NVFP4. That wording matters. The precision switch selects block-scaled kernels inside MoE expert layers; dense MLPs, attention, embeddings, and the LM head remain BF16. The three dense cells are repeated BF16 controls under different profile labels, not dense FP8 and NVFP4 experiments in disguise.

Each run had at most 9536 steps at 524,288 tokens per step, AdamW at 1.8e-3, the same WSD schedule, SDPA attention, a 3.28 loss threshold, and end-of-run CORE. The routed expert matrices used AdamW betas (0.9, 0.99); dense and shared FFNs and the other non-routed groups used (0.9, 0.95). That optimizer-group choice travels with the MoE architecture here. I can call this a dense–sparse comparison, but not a routing-only ablation.

The architectural swap held active FFN width per token near 3072:

The arithmetic in the middle column is the contract I chose: one token sees about 3072 active FFN channels in every family. It aligns a nominal matrix-width axis and nothing more. It does not equalize FLOPs, communication, routing work, kernel efficiency, or the amount of data seen by each expert; the total-parameter column makes the scale of that mismatch explicit.

Then the clocks disagreed

Six lanes reached the target. These are their first observed crossings:

In BF16, MoE-256 used 41.5% fewer logged tokens than dense to cross 3.28, then took about 2.7× as much training-loop time. MoE-64 made the same bargain less dramatically. Saying “the sparse model was faster” would therefore be wonderfully efficient: one short sentence, two incompatible meanings.

CORE refused to rescue me with a universal ranking. At the observed BF16 crossing points, dense had the highest score. Under the FP8 expert profile, MoE-256 had both the highest observed CORE and the fewest tokens to threshold. These are one-run task aggregates at nearby rather than identical losses; they describe this matrix, not a law about precision.

NVFP4 was a different regime again. Neither sparse lane reached 3.28 by step 9536, and the repeated BF16 dense control under that profile label missed too. Final losses were 3.3047 for dense, 3.5950 for MoE-64, and 3.4573 for MoE-256; CORE was 0.0494, 0.0309, and 0.0491, respectively. The three precision profiles do not collapse to one generic sparse verdict.

The threshold was not an equal-loss experiment

Validation ran every 128 steps. A crossing in this table means the first observed validation with loss ≤ 3.28, not the exact token at which two curves became equal. The matrix reports logged tokens, training-loop time, CORE, and global parameters around those stopping points. It does not give equal-token or equal-time outcomes, and a memory-matched comparison would require different architectures.

The schedule adds another wrinkle. WSD decay begins at step 7488, so the dense crossings happen during decay while all four sparse crossings happen during sustain. CORE at the stopping point therefore mixes architecture, routed-expert optimizer settings, token exposure, and schedule phase. That does not erase the measurements; it prevents a clean causal reading of them.

There is one run per cell, and CORE carries task and item noise. I would want repeated seeds, common-token checkpoints, and larger models before treating the architecture-by-precision ordering as general. Active FFN width is only one possible swap contract, too: at fixed global tokens, nominal routed exposure per expert falls roughly as K/E.

Public anchor: June 2024 run record · recipe commit b6b0a0d. Run identities, exact values, and figure mappings are recorded in nmoe/repro/0003.receipts.json.

I wanted the loop to tell me which idea deserved a larger run. It gave me a more useful answer than a winner: in BF16, MoE-256 first crossed 3.28 with 41.5% fewer logged training tokens and 2.7× more training-loop time. The next experiment has to improve the second clock without giving back the first, and I have to stop pretending that “faster” names both.