I Built Super-4096 to Fail

I built a model with 4096 routed experts because I wanted to watch it fail.

This was not neutral science. Super-4096 was a provocation: keep the active expert width in familiar territory, spread the work across an absurdly large catalog, and see whether each expert became too starved of data to learn. The experiment was supposed to give me a boundary. More importantly (and less admirably), I thought I already knew what the boundary would mean.

At validation step 512, loss was 4.7179 and falling. By step 2048 it was 3.6494. If you watched only the curve, as I had trained myself to do, you would call the run healthy. In the paired router snapshot at step 500, though, one layer's distribution over 4096 experts had the entropy of a uniform distribution over 13.7. Layer 11 was sending 14.26% of all routed assignments to one expert. With top-seven routing, 14.29% is what you get when the same expert is selected for essentially every token.

Why build something this unreasonable?

Super-4096 had 4096 routed experts in every layer, selected K=7 of them for each token, and kept one shared expert always on. The eight active expert widths—the seven routed experts plus the shared one—summed to 3072, like the dense FFN we had been using, but that did not make the whole experiment compute-matched. The router was wider, the stored capacity was enormous, and the traffic reaching any one routed expert was thin. This was a stress test, not a fair fight.

The planned run was 12,000 steps of 524,288 tokens, or 6,291,456,000 tokens in total. If the assignments were perfectly balanced, each routed expert would receive

token assignments. Under that same planned token budget, MoE-64 with K=6 would give each routed expert 589,824,000 assignments. That is 54.9× as much exposure per expert, which is a very elaborate way of saying that Super-4096 was built to be hungry.

There is an important correction here because the original version of this post committed a common research-blog sin: it turned a configured horizon into an observed result. The config said 12,000, so the prose spoke as if all 12,000 steps had happened. They had not. The corrected receipt set supports the baseline scan through step 3600 and the matched comparisons through step 2048. The 6.291B-token arithmetic explains the experiment I designed; it is not a result I get to claim.

The number was fourteen

Router entropy is one of those measurements that becomes less informative the moment you report it without translation. The worst layer at the router snapshot at step 500 had assignment entropy H=2.6173 nats. Exponentiating it gives an entropy-equivalent support:

This does not mean exactly fourteen experts received traffic, that the remaining 4082 were permanently dead, or that the same fourteen experts dominated the whole dataset. The router metrics count hard top-K assignments in the latest global batch, all-reduce them across ranks, and normalize them independently for each layer. E_eff=13.7 means that this one batch distribution over 4096 choices was as concentrated as a uniform distribution over roughly fourteen choices. It is a photograph, not a death certificate.

The maximum share makes the picture harder to rationalize. A top-K router can select a particular expert at most once for each token. If one expert appears in nearly every token's top-seven set, it receives roughly 1/7 of all assignment counts: 14.2857%. Layer 11 was at 14.26% at router step 500. This is not “some imbalance.” It is the signature of one expert becoming almost universal.

And the concentration got worse while the model got better. Between the router snapshots at steps 500 and 2000, mean CV across layers rose from 1544.99% to 1652.59%, and the mean of each layer's maximum assignment share rose from 11.63% to 13.42%. Over the paired validation points, loss fell from 4.7179 to 3.6494.

CV here is the coefficient of variation of expert assignment shares in one global batch, reported as a percentage and averaged across layers. It is useful for watching one configuration become more or less concentrated. It is not a universal router-health score: change E, the batch occupancy, or the routing geometry and the sampling scale changes underneath it. (Metrics are very accommodating this way. They will happily become incomparable without changing their names.)

The corrected scan also killed a story I liked. The old draft said layer 0 collapsed first and the failure propagated upward through the network. It was a clean causal picture and it was wrong. Using 0.9/K = 12.857% as the saturation marker, layers 6 and 11 crossed by step 100, layer 5 by 400, layer 0 by 700, and layer 3 only by 3600. The router did not collapse like a building falling from the ground floor up. Different layers built local monopolies on different clocks.

I had a beautiful explanation, which should have made me suspicious

My explanation was starvation. Each expert saw too little data, so the router found a few early winners, sent them more traffic, and allowed the rest to wither. It is a plausible mechanism and may still be part of the answer. The problem is that I wrote it as though the run had already isolated it.

The obvious control was to reduce the number of experts. Through step 2048, nominal assignments per routed expert rise from 1.835M at E=4096, to 3.670M at E=2048, to 7.340M at E=1024. The comparison asks whether that fourfold increase was enough to dislodge the near-universal expert within the window we observed.

Giving each expert two or four times as much nominal traffic moved effective support from 13.7 to 13.9 and 15.2; layer 11 remained pinned near the same 1/K ceiling. The later loss ordering was not monotonic either. At step 2048, E=2048 reached 3.5964, but E=1024 reached 3.7272, worse than the E=4096 baseline at 3.6494.

I cannot turn this into “exposure does not matter.” Changing E changes stored parameter count, router geometry, occupancy statistics, and the extreme-value behavior of the expert distribution along with assignments per expert. Raw CV also changes mechanically with the size of the catalog. These are not clean “tokens only” controls; they are different models, and even 7.340M assignments per expert may sit below whatever exposure threshold matters. The honest sentence is annoyingly short: within this seed and window, a fourfold increase in nominal exposure did not remove the near-1/K layer. These controls do not identify why.

The control I most want to name is the one I cannot name

The last row changed the regime much more than reducing E. It turned on the Switch-style auxiliary balancing loss with α=10⁻⁴. In compact form the term is $\alpha E \sum_e f_e q_e$, where f_e is the fraction of hard assignments sent to expert e and q_e is its differentiable normalized share of the selected top-K gate weights. The loss penalizes the router when hard assignment traffic and selected gate weight concentrate on the same experts.

At validation step 512, this run had loss 4.6906; its router snapshot at step 500 had worst-layer effective support 47.2 and layer 11's maximum share was 4.03%. At validation step 2048, loss was 3.5728; at router step 2000, mean CV was 955.37% and the mean per-layer maximum was 4.40%. The corresponding baseline numbers were 3.6494, 1652.59%, and 13.42%.

I would very much like to call this “the aux-loss result.” I cannot. The published baseline had aux_loss_alpha=0 and a separate aux-free router-bias updater running at 10⁻⁴. The treatment turned the bias updater off and the auxiliary loss on. There is no identified bias=0, aux=0 control in the receipt, so the effect belongs to the pair: adding the auxiliary objective, removing the bias update, or an interaction between them. Calling it aux-only would turn a promising result into a false one for the sake of a cleaner sentence.

Nor did the combined intervention make routing healthy. Effective support rose from 13.7 to 47.2, which sounds wonderful until you remember that 47.2 is 1.15% of 4096. A 4.03% maximum share is about 165× the uniform share of 1/4096. The intervention made the snapshot substantially less concentrated and got a lower loss in this run; its batch distribution was still nowhere near uniform across 4096 experts.

There is also one seed per condition and no uncertainty estimate. The DB and parquet paths cited by the receipt are external artifacts rather than files shipped with this site. This is enough evidence to describe what happened in these runs and not enough to promote the pattern into a law (a distinction research blogs are structurally designed to tempt you to forget).

Loss was answering a smaller question

The mistake was not trusting the loss curve. The curve was correct. Validation cross-entropy measures predictive fit on the fixed validation stream under the model's actual computation; it neither attributes the improvement to a component nor measures the breadth of the routing distribution. This run put those two facts beside each other: loss fell while the batch-level assignment metrics remained severely concentrated. The main language-modeling objective does not directly reward using the nominal expert catalog evenly.

By step 2048, one corrected Super-4096 seed had done two things at once: validation loss improved and routed assignments remained violently concentrated. Lowering E did not remove the near-1/K layer. A combined bias-off-plus-aux-on intervention changed the routing distribution far more and reached a lower loss, but the available controls do not tell us which half of that intervention deserves the credit.

Receipts: 0006. The legacy Super-4096 figures on the site encode an older run and are deliberately excluded here.

I built Super-4096 to learn how little data an expert could survive on. What I learned instead was that one router layer could present a batch distribution as narrow as fourteen effective choices while loss kept falling. The curve was never lying. I was asking it the wrong question.