The Rack Is the Computer

The machine changed shape while our software was looking the other way.

An NVLink domain is no longer a handful of GPUs with a fast side channel. On GB300 NVL72 it is an entire rack: 72 accelerators behind a scale-up fabric, built and sold as one computational object. Training stacks still tend to greet that object by drawing a rank mesh over it and then cutting the model across tensor, pipeline, data, context, and expert axes. Every cut buys capacity by adding a synchronization schedule. Enough cuts and the machine begins to look less like a computer than a peace treaty among collectives.

RDEP starts from a harsher design rule: the active dense program should fit on each accelerator in the domain. Sparse capacity should be owned once across the domain.

Attention, embeddings, routers, normalization, shared computation, and the dense optimizer boundary execute replica-locally. Routed tokens travel to the experts that own them. The NVLink domain is the training computer; the rank is a place where part of that computer happens to live.

This is not “expert parallelism when convenient,” and it is not principally a faster all-to-all. It is a model-system architecture. If the active dense core requires permanent tensor or pipeline partitioning inside the target domain, the model has left RDEP's design center. The first question becomes whether capacity belongs in the dense core at all, not which new mesh axis can be made to carry it.

That position would be architecture astronautics if it did not pay rent. On the same eight B200 GPUs and the largest jointly demonstrated stable shared workload, RDEP reached 24,871 tokens/s against 12,447 for the closest public tensor-parallel plus NCCL baseline, while using 58.92 rather than 171.10 GiB of peak reserved HBM per logged GPU: 2.00× the throughput at 2.90× lower peak memory. RDEP also completed twice the baseline's largest demonstrated stable token budget per step.

The interesting part is why.

The full argument and receipts are in RDEP: The NVLink Domain as a Sparse Training Computer.

The sparse replication tax

Suppose a domain has width $W$. Each rank processes $T$ source tokens, each token contributes exactly $K$ accepted routes—the no-drop case—and an expert instance contains $P_{\mathrm{expert}}$ parameters. Let $\rho$ be the number of physical replicas of each unique expert, $C$ the unique sparse capacity in the domain, and $\bar m$ the mean routed rows reaching a resident expert instance.

The bookkeeping collapses to an exact identity:

This is the sparse replication tax. If every data-parallel rank carries its own copy of the expert bank, then $\rho=W$. The domain owns no more unique sparse capacity than one rank, and each physical expert sees only its local routes. If each expert is owned once across the domain, $\rho=1$. The same hardware recovers an exact factor of $W$ in the product of unique capacity and rows per resident expert.

When capacity drops routes, $WTK$ in the identity becomes the actual accepted-row count. The tax is an ownership identity, not permission to count work the operator declined to execute.

That dividend does not force one scaling policy. Let the number of unique experts grow as

At widths where ownership is integral, $\alpha=0$ keeps capacity fixed and grows mean rows per expert by $W$: the heating endpoint. At $\alpha=1$, capacity grows by $W$ while rows per expert stay fixed: the capacity endpoint. Every interior choice gets both: $W^\alpha$ more unique capacity and $W^{1-\alpha}$ more rows per expert. Resident expert parameters per accelerator fall by $W^{1-\alpha}$ relative to replicated ownership.

This is the conceptual center of RDEP. Pooling routes is not an incidental batching trick. It is what happens when the system stops cloning sparse state and lets every dense replica feed one domain-wide expert bank.

One domain, two kinds of work

Every rank in RDEP plays two roles. It is a front end for its local token stream, and it owns a shard of the domain's experts. The dense path is local. The sparse path follows the realized route graph.

A selected token-expert pair becomes a route row: activation payload, expert identity, gate weight, source identity, and enough position information to return the result to the exact token and slot that produced it. Sources publish owner counts. Owners derive disjoint receive spans. Sources write rows directly into peer-visible memory. Owners group the accepted rows by local expert, execute the expert computation, and return outputs along the reverse graph. Backward traverses the same accepted set.

That last sentence is where many fast prototypes go to die. Moving the correct number of bytes is not enough. Payload and metadata must share the same split tables; dropped rows and gate renormalization must agree in forward and backward; the return must preserve token-slot identity; every owner must consume precisely the rows the router admitted. RDEP's protocol is an execution of the sparse operator, not a courier service standing beside it.

The route graph also tells us which synchronization is necessary now. In the current two-tray GB300 trainer, deriving a backward sender mask from the current send set avoids waiting on owners with no current offset for a source. Against the repaired plain path, the promoted default moved mean step time over recorded steps 3–6 from 1517.38 to 1401.92 ms and reported throughput from 358.27 to 388.31 TFLOPS/GPU. In a separate rank-0 profile, the affected metadata wall moved from 127.74 to 97.46 ms.

The whole-step result is a four-step engineering A/B, not a population estimate, and earlier attempts to remove broader waits failed the BF16 gradient gate. The separate profile is directionally consistent with it. Once the route graph is authoritative, both computation and phase dependencies can follow it. Attractive timings without exact gradients are merely a fast way to train another model.

Heating the experts without adding work

The clean mechanism experiment holds local work fixed. It uses 64 experts, top-6 routing, hidden size 2048, expert width 1408, and $T=4096$ local tokens while widening the participating domain from $W=1$ to $W=8$.

This is weak scaling. Global source tokens rise with $W$, but mean routed work per owner remains exactly

What changes is the shape of those rows. At $W=1$, each owner holds 64 experts and sees 384 rows per expert on average. At $W=8$, each owner holds eight experts and sees 3,072 rows per expert. The owner does not perform more routed work; it performs the same work in fewer, taller groups.

The physical result follows the algebra. In the matched direct-grouped receipts, throughput rose from 905 to 1,203 TFLOPS/GPU, useful MFU rose from 40.2% to 53.5%, and padding fell from 1.33× to 1.04×; the intermediate useful-MFU points were 48.5% and 52.4% at $W=2$ and $W=4$. In the controlled full-path harness, max-rank CUDA-event forward-plus-backward p50 fell from 5.044 to 3.958 ms. Useful MFU here uses NVIDIA's 2.25-PFLOPS/GPU dense B200 BF16 peak.

None of this is free fixed-global-work scaling. It is a per-GPU efficiency gain at fixed local owner work, purchased by removing replicated expert instances and concentrating the same rows into more useful matrix shapes. That distinction matters because otherwise every weak-scaling plot eventually becomes a perpetual-motion machine in a blazer.

Routing skew changes the ending, not the arithmetic. In the $W=8$ stress sweep, the busiest owner's grouped path remained between roughly 1.20 and 1.33 PFLOPS as Zipf skew increased. Layer throughput still fell because owners finished at different times and the slowest one set the wall. RDEP turns a cold, fragmented expert-kernel problem into a placement, routing, and makespan problem. This is progress because the latter can be attacked directly. It is not absolution.

The competitive result, with the labels left on

The closest public baseline we could run on the same eight B200s was TorchTitan's DeepSeek-V3 16B recipe using tensor parallelism plus NCCL. The comparison matches hardware, depth, hidden size, routed expert width, number of routed and shared experts, top-$k$, precision, sequence length, and the token workload at the shared point. It is not an exact semantic or optimization match: vocabulary, dense-FFN width, tokenizer, optimizer and data path, runtime/software identity, and reporting windows differ. The RDEP compile and activation-checkpoint settings and the exact source and container digests for both stacks are absent from the publication archive. The figures are finite-run descriptive means over RDEP steps 10–29 ($n=20$) and TorchTitan steps 5–9 ($n=5$). This is the closest public mesh-first system comparison we could close, not a claim that we searched every possible baseline configuration on Earth.

At 131,072 token presentations per step:

The shared point gives the 2.00× throughput and 2.90× memory result.

The feasibility frontier is a second, narrower statement. RDEP completed 200 steps at 262,144 token presentations per step. The comparator's largest demonstrated stable point was 131,072; its next tested point, 196,608, ran out of memory in the routed MoE output buffer. We did not exhaustively search either system's absolute maximum. We demonstrated these points, and the gap between them is already useful.

Rack width means rack width

Eight GPUs are a development lane. The target is the domain.

On all 72 GPUs of a GB300 NVL72, the protocol passed fabric smoke, conservation and ownership invariants, and a real routed BF16 forward-plus-backward step. Computed from mean max-rank wall, the communication-focused top-6 forward-plus-backward benchmark at $T=4096$ reached 25.86 million source tokens/s, or 155.2 million route rows/s. Separately, its observed max-rank p50/p99 were 8.48/51.12 ms. A routed BF16 compute step at $T=1024$, hidden size 2048, and expert width 1408 produced finite outputs and gradients on every rank at 18.19 ms max-rank latency.

Dispatch p99 at the hottest forward-plus-backward point reached 44.37 ms. A single delayed peer becomes very visible at rack width. We therefore claim full-domain protocol execution, 25.86 million source tokens/s from mean max-rank wall, and 8.48 ms median max-rank latency—not a solved tail.

The protocol has also accumulated millions of GPU-hours in GB300 NVL72-class production training with zero recorded protocol-level failures: no row loss or duplication, route-identity or ownership violation, transport deadlock, or incorrect forward/backward reconstruction attributed to RDEP. That category is intentionally smaller than “nothing ever failed.” Models, data, schedulers, nodes, and the rest of distributed systems retained their traditional ability to ruin an afternoon.

Two frontier receipts, not one heroic anecdote

RDEP's scale evidence comes from two different lanes, and combining them would make both less informative.

The first is a verified 64-GPU configuration and launch for a 1.651-trillion-total, 57.49-billion-active base model. Its domain-wide routed bank is about 1.635 trillion parameters, with ten experts per owner. In the full graph with four auxiliary heads (about 1.654 trillion total and 59.96 billion active), the expert bank, router, and FFN normalization are frozen, leaving 13.71 billion parameters trainable; the base trunk accounts for 11.24 billion of them. The model graph and launch are retained. The referenced runtime, accelerator SKU, and physical-topology artifacts are not. This establishes the configured capacity envelope, not a receipt-closed training trajectory.

The second is a separate, receipt-closed full-parameter FP8 run: 672.056 billion total parameters, 38.582 billion active, 64 GB300 GPUs, and 2,000 completed steps. All dense, router, and expert parameters were optimizer-bearing. Across 524,288,000 token presentations, median logged performance was 76,172.5 global tokens/s and 338.310 model-estimated TFLOPS/GPU.

That run repeatedly wrapped one 25-million-token shard and has no held-out trajectory. It demonstrates sustained full-parameter systems execution at 672B, not model-quality convergence. The distinction is not a footnote. A systems paper earns the right to say what ran; it does not inherit the right to say what the run learned.

Designing for the machine we have

RDEP is not a promise that tensor or pipeline parallelism will never be needed again. Dense replication has a memory envelope, cross-domain scale-out still exists, route skew creates makespan, and rack-width tails remain unfinished work. Legacy models whose dense cores do not fit need compatibility paths.

The claim is about the default. On a scale-up machine, begin with the machine boundary. Keep the active dense program local to every accelerator. Own sparse state once. Let the realized route graph determine movement and synchronization. Spend the reclaimed replication factor on some combination of more capacity, hotter compute, and less resident state.

For years we designed distributed models as if communication links were the scarce miracle and duplicate parameters were harmless. The hardware has now handed us a rack with a common memory fabric and asked what kind of program belongs on it.

RDEP's answer is simple enough to be impolite: stop treating the rack as 72 computers.

Paper: RDEP: The NVLink Domain as a Sparse Training Computer. Public artifact receipt: 0009.