The NVFP4 Fix Was Fighting Itself

I had already written the clever version of this post.

I wanted the NVFP4 gap to BF16 to be about rounding stiction. The expert weights could keep moving in BF16 while their cached four-bit codes stopped crossing quantization boundaries; dither and router sensitivity made the mechanism feel satisfyingly physical. I had an explanation before I had the control that could earn it.

Then we found a flaw in the blockscaled backward replay. The live forward used blockscaled expert weights, but the custom backward could reconstruct a clean BF16 expert while computing the gradient. The runs I had been explaining were differentiating a nearby model, not the one that had produced their forward activations.

After fixing the replay, we ran the experiment that should have come first: split the two gains in the old NVFP4 rescue. At step 384, the embedding gain improved validation loss, the low-logits gain worsened it, and the old pair finished worse than using neither. I had been treating one configuration recipe as one idea. The model had received two interventions at opposite ends of the network.

The backward replay belonged to a cleaner model

The expert path keeps BF16 master weights but executes through cached blockscaled FP8 or NVFP4 weights. Its backward pass has to rematerialize the operands needed for the expert gradient. Before the fix, the default replay could use the BF16 masters even though the forward had used the quantized cache. Forward and backward therefore described different functions. The difference may have been small, but it landed exactly where this post wanted to make a precision claim.

The corrected replay follows the same precision profile as the live forward. Only those corrected runs appear in the gain comparison below.

The gain names are also a little misleading. They are gated to the NVFP4 recipe, but neither one lives inside the expert quantizer:

fp4_embed_gain multiplies the BF16 embedding output before the first transformer block. It changes the scale of the residual stream from the first layer onward, along with the gradient returning to the embedding. fp4_logits_gain scales the final BF16 logits used by both training and validation, changing softmax temperature and backward magnitude. Calling these “FP4 gains” tells you when they are enabled; it does not tell you how local they are.

Four runs, two signs

The corrected test was a 2 × 2 factorial on MoE-64: 384 steps, one fixed-seed run per arm. The default NVFP4 ExpertAdamW/cache-emission path and the dither setting stayed fixed while the two gains moved independently.

The conditional effects keep the same signs on both backgrounds. Raising the embedding gain lowers step-384 loss by 0.1812 with unit logits and by 0.1051 with low logits. Lowering the logits gain raises it by 0.1542 with unit embeddings and by 0.2303 with the larger embedding gain.

They do not combine additively. Summing the two effects measured against the unit arm predicts a net change of −0.0270. The old pair instead lands at +0.0491, leaving a positive interaction of +0.0761. Within this quartet, the low-logits setting is more damaging when the embedding gain is large.

There is a trap in calling that last knob harmful without qualification: it touches the scoreboard. Validation cross-entropy is computed from the scaled logits, so fp4_logits_gain=0.125 directly changes the logits being scored as well as the training dynamics that produced them. These runs establish the observed loss ordering; they do not prove that low logits damaged an otherwise identical learned representation. With one run per arm and only 384 steps, they establish local conditional signs, not a universal calibration law.

Then I bundled the long run again

Embed-only was the clean winner in the quartet, so we carried fp4_embed_gain=10.667 with unit logits to step 9536. Except the long run also set expert_opt=adamw. I had separated the gains and then changed the cache path underneath the follow-up.

That override uses standard torch.optim.AdamW for the BF16 expert master weights and rebuilds the quantized cache after the optimizer step. The default NVFP4 path uses fused ExpertAdamW, which updates the masters and emits the cache together. Those paths have different cache-emission semantics, including how the configured dither enters. The long run is therefore a combined recipe, not a long-horizon replication of the embed-only arm and not an optimizer ablation.

At step 9536, the BF16 baseline had validation loss 3.126422. The combined NVFP4 run had 3.173490, a gap of +0.047068. Across the five recorded checkpoints from step 7808 through 9536, the gap ranged from +0.0451 to +0.0485, with mean +0.04678.

The narrow late range makes the residual easy to describe and no easier to explain. The receipt has no full-length unit-gain NVFP4 control, no full-length embed-only run on fused ExpertAdamW, and no matched unit-gain arm on standard AdamW. It cannot separate embedding scale from optimizer/cache-path behavior or their interaction.

Nor does the corrected quartet settle my original stiction story. Dither was held fixed, and this evidence bundle contains neither a no-dither arm nor a code-flip artifact. Stiction may still be part of the NVFP4 dynamics; this experiment neither tests it nor establishes that the gains matter more.

The remaining +0.047 could reflect a defect, a scale mismatch, an optimizer/cache interaction, straight-through-estimator behavior, or an ordinary cost of this four-bit path. Calling it either a bug or an irreducible precision tax would get ahead of the controls. The next useful experiment is a long-horizon gain-by-cache-path factorial with repeated seeds, followed separately by the dither ablation. Activation RMS, logit RMS, gradient norms, and code-flip telemetry would let the mechanisms compete rather than asking the final loss to adjudicate all of them.

Receipt: 0005. The quartet and long-run measurements point to external DB and parquet artifacts; the historical source revision, resolved configs, and exported artifacts are not bundled with the site.

I had packaged two global gains as one NVFP4 rescue and then tried to explain the package with a local quantization story. The corrected quartet gave me a smaller result and a better rule: split the fix before explaining it.