Mixture of experts / Report summary

MoE and the Network Cliff

Selective expert computation can save work, but token dispatch, return traffic and synchronization determine whether those savings survive at scale.

June 2026 · 7 pages · 1.6 MB · Tolaga Research

Download full report (PDF) ↓

Executive summary

Mixture-of-Experts architectures activate a subset of expert networks for each token. This lets model capacity grow without activating every parameter on every step. The report examines the accompanying routing cost: tokens must reach their selected experts and their results must return before execution can continue.

As experts spread across more devices, traffic can cross from fast local interconnects into slower inter-node fabric. Combined with other parallel communication and uneven expert loading, that transition can erode the apparent compute advantage. The report calls this configuration-dependent transition the network cliff.

Key findings

  • Expert parallelism changes both the distribution of compute and the physical boundaries crossed by traffic.
  • The study uses 512 H100-class GPUs across 64 eight-GPU nodes, with 64 experts and two active experts per token.
  • At EP16, modeled inter-node traffic approaches 400 GB per step and all-to-all latency is 1.47 times expert compute latency.
  • The EP32 stress case exceeds 800 GB per step, with an all-to-all-to-compute latency ratio of 3.36.

What creates the cliff

The study varies expert parallelism from EP1 to EP32. With eight GPUs per node, smaller expert groups can largely remain within a fast intra-node domain under the study's placement assumptions. Wider groups increasingly use inter-node connections. Tensor-parallel traffic contributes separately to intra-node demand, while other training communication can compete for shared resources.

Four mechanisms interact: dispatch and combine traffic, expansion across network tiers, competition with tensor/data/pipeline communication, and uneven expert loading. The busiest expert or constrained communication stage can delay progress even when average utilization appears acceptable. Aggregate active-parameter counts do not capture those effects.

Infrastructure implications

Expert placement should be evaluated together with topology and the other parallelism degrees. Increasing expert parallelism may improve fit or distribute computation while simultaneously increasing exposed communication. The useful range depends on local-domain size, bandwidth, latency and the competing traffic mix.

Routing, batching and load balancing are also part of the infrastructure design. A larger or faster fabric cannot be assessed solely through advertised bandwidth; the question is how effectively the actual traffic can use it. Conversely, a result observed at an eight-GPU boundary should not be generalized to every scale-up architecture.

Assumptions and limitations

The figures are modeled training results for the specified topology and placement assumptions, not production measurements or universal EP thresholds. EP32 is explicitly a theoretical stress-test regime. The reported ratios compare all-to-all latency with expert compute latency, not total end-to-end speedup.

Source: MoE and the Network Cliff, PDF pages 1–6. This summary describes the published report; consult the PDF for figures and references.