Mixture of experts: how sparse routing scales language models.

Most large language models whose architecture has been publicly disclosed in the past two years use some form of mixture of experts. Mixtral, DeepSeek-V3, Kimi K3, and their peers all use sparse expert routing to decouple model capacity from inference cost. The total parameter count can run into the trillions while the parameters active for any given token stay in the tens or low hundreds of billions.

The concept predates deep learning in its current form. Jacobs, Jordan, Nowlan, and Hinton introduced it in 1991 as a way to partition a problem across specialized sub-networks. Making it work inside transformers at scale required solving a cluster of engineering problems around routing stability, load balancing, and cross-device communication. This post covers what mixture of experts is, how routing works, what makes training tricky, and where the architecture sits today.

The core idea

A mixture of experts (MoE) layer replaces a single feed-forward network with N parallel expert networks and a gating function, the router, that decides which experts process each input. In a dense transformer, every parameter participates in every forward pass. In a MoE transformer, the router selects a small subset of experts per token, so most expert parameters sit idle for any given input.

This creates two independent knobs. Total parameters control model capacity: how much the model can learn and store. Active parameters control inference cost: how much compute is spent per token. A model with eight experts and top-2 routing activates two expert networks per token, so you can build four times the expert capacity at roughly similar per-token cost.

Only the feed-forward layers are replicated into experts. Attention, embeddings, and normalization are shared and always active. Since feed-forward layers account for roughly two-thirds of parameters in a standard transformer, replacing them with routed experts multiplies the total model size without a proportional increase in forward-pass compute. Mixtral 8x7B (Jiang et al., 2024), for example, has 46.7 billion total parameters but only about 12.9 billion active per token.

From soft mixtures to sparse gates

The original mixture of experts (Jacobs et al., 1991) used a gating network to weight the outputs of several small expert networks, with every expert contributing to every input. The final output was a weighted sum across all experts. This "soft" mixture works when you have a handful of experts, but it scales poorly: adding more experts linearly increases the compute per input.

Sparse mixtures fix this by activating only a few experts per input. The landmark paper by Shazeer et al. (2017) demonstrated this at language-model scale, inserting sparsely-gated MoE layers into an LSTM model and reaching 137 billion parameters while activating only a small fraction per step. The gating function used a learned linear projection plus noise, followed by top-k selection. This was the first demonstration that sparse expert models could match dense models at substantially lower training compute.

The move to transformers followed with GShard (Lepikhin et al., 2021), which scaled MoE to 600 billion parameters across thousands of TPU chips using expert parallelism, where each expert lives on a different device. Switch Transformers (Fedus et al., 2022) simplified routing further, to top-1 (one expert per token), and showed this was sufficient when paired with a well-tuned load balancing loss. Top-1 routing halves the expert compute compared to top-2 and reduces the communication overhead of moving tokens between devices.

How routing works

The router is typically a single linear layer. It takes a token's hidden representation and projects it to a vector of scores, one per expert. The top-k experts by score are selected, their scores are renormalized via softmax over the selected set, and each selected expert processes the token independently. The final output is a weighted sum of the expert outputs, using the normalized scores as combination weights.

Top-1 routing sends each token to exactly one expert. This minimizes expert computation but gives each token only a single expert's transformation. Top-2 routing, used in Mixtral and many recent open-weight models, sends each token to two experts and combines the results. The quality improvement over top-1 is consistent across published comparisons, at the cost of roughly doubling the expert compute per token. Both choices preserve differentiability through the routing decision, because the combination weights are continuous even though the expert selection is discrete.

DeepSeekMoE (Dai et al., 2024) introduced a variation with finer-grained experts: more experts, each smaller, paired with shared experts that process every token. The shared experts handle patterns common across all inputs, while the routed experts specialize. DeepSeek-V3 (DeepSeek-AI, 2024) continued this design with 256 routed experts (8 selected per token) plus one shared expert that is always active. The shared pathway guarantees a baseline of processing for every token regardless of how routing decisions fall.

The load balancing problem

Left unconstrained, the router tends to collapse. A few experts attract most of the traffic; the rest starve. The mechanism is straightforward: a popular expert receives more gradient signal, which makes it better, which draws more tokens to it. Unused experts get no learning signal and stagnate. Without intervention, the model converges to using a small fraction of its total expert capacity.

The standard remedy is an auxiliary loss that penalizes uneven expert utilization. Switch Transformers (Fedus et al., 2022) formalized this as the dot product of two vectors: the fraction of tokens dispatched to each expert and the average router probability assigned to each expert. Summing this product across experts gives a differentiable balancing signal, scaled by a tunable coefficient. Setting the coefficient too low allows collapse; setting it too high forces uniform routing and prevents experts from specializing. ST-MoE (Zoph et al., 2022) studied this tradeoff across model sizes and found that settings tuned at small scale did not reliably transfer to larger models.

A complementary mechanism is expert capacity: each expert is assigned a fixed buffer per batch, and tokens routed to a full expert are either dropped or sent through a residual path. This caps the worst-case imbalance but introduces dropped tokens, inputs that receive no expert processing at all. The drop rate becomes a training metric worth tracking.

More recent work has moved toward removing the auxiliary loss entirely. DeepSeek-V3 (DeepSeek-AI, 2024) adds a bias term to each expert's routing score during selection but excludes it from the output combination weights. The bias is adjusted dynamically based on observed load, nudging underused experts' scores upward without distorting the final weighted output. This sidesteps the conflict between the main training objective and a separate balancing loss competing for gradient bandwidth.

The compute-parameter tradeoff

The central claim of MoE is more model quality per FLOP. The evidence is strong and consistent. GLaM (Du et al., 2022), a 1.2 trillion parameter model with 64 experts per MoE layer, matched GPT-3's quality across several benchmarks while using roughly one-third the training compute. Mixtral 8x7B, with 46.7B total and about 12.9B active parameters, matched or exceeded Llama 2 70B on most benchmarks at the inference cost of a much smaller dense model.

Clark et al. (2022) derived scaling laws specifically for routed language models and found two regularities. First, increasing the number of experts (and therefore total parameters) improves loss at a diminishing rate: going from 1 to 64 experts yields a large gain, but going from 64 to 512 yields a smaller one. Second, there is a consistent quality gap they call a "routing tax": for the same total parameter count, a dense model edges out a sparse one, because imperfect routing means some tokens land on a suboptimal expert. The tradeoff remains favorable in aggregate because the compute savings from sparsity more than compensate for this penalty.

These scaling properties explain why MoE dominates at the frontier. When compute budgets are large enough that the next step up in dense model size becomes prohibitively expensive to train and serve, MoE lets you keep adding capacity by adding experts at a manageable marginal cost.

Expert parallelism and serving

Training and serving a MoE model requires distributing experts across accelerators. In expert parallelism, each device holds a subset of experts. During the forward pass, tokens are dispatched to whichever device holds their selected expert via an all-to-all communication step, and results are gathered back after expert computation. The cost of this communication depends on how many tokens cross device boundaries and on interconnect bandwidth.

At training time, expert parallelism is combined with data parallelism and often tensor or pipeline parallelism on top. Keeping all of these balanced while maintaining even expert load is as much a systems problem as a modeling one. The Kimi K3 technical report, for example, spends much of its length on kernel co-design, memory management, and expert-parallel balance rather than on model architecture. The plumbing around MoE is where much of the engineering effort goes.

At inference time, MoE models have a distinctive cost profile. Per-token compute is determined by the active parameters, which is moderate. But the full set of expert weights must reside in memory (or be loaded fast enough from storage), and the memory footprint is set by the total parameter count. A model with 100B active and one trillion total parameters needs memory for the full trillion, even though each forward pass touches only a tenth of it. This makes MoE models memory-bound for many serving configurations, and it is the reason that quantization and weight offloading strategies matter more for MoE than for dense models of comparable inference speed.

What MoE does and does not buy you

MoE reliably delivers better quality per FLOP. Given a fixed compute budget for training or serving, a MoE model will encode more knowledge and produce better outputs than a dense model at the same per-token cost. On knowledge-intensive tasks and multilingual benchmarks, the advantage is especially clear, because more expert capacity means more room to store facts and linguistic patterns.

What MoE does not change is the reasoning depth available at a given active parameter count. A 13B-active MoE model knows more than a 13B dense model, but the computation applied to each token is still roughly 13 billion parameters' worth. Tasks that demand deep sequential reasoning over a single input, rather than broad knowledge recall, see less benefit from the extra capacity that idle experts provide.

MoE also introduces engineering complexity at every stage: training instability from load imbalance, the need for expert-parallel infrastructure, memory-bound serving, and hyperparameters (capacity factors, balancing loss coefficients, expert granularity) that do not exist in dense models. For small-scale work or single-GPU deployment, a dense model is often the more practical choice. The architecture pays for its complexity at scale, which is why it has become the standard at the frontier rather than across the board.

Sources and further reading

Foundational work

• Shazeer et al. (2017). Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. ICLR 2017. First demonstration of sparse MoE at large scale in language modeling.

MoE in transformers

• Lepikhin et al. (2021). GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding. ICLR 2021. Scaled MoE to 600 billion parameters across thousands of TPU chips.
• Fedus et al. (2022). Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity. JMLR 2022. Introduced top-1 routing and the standard auxiliary load-balancing loss formulation.
• Zoph et al. (2022). ST-MoE: Designing Stable and Transferable Sparse Expert Models. Systematic study of training stability and hyperparameter transfer across MoE model sizes.

Scaling and analysis

• Du et al. (2022). GLaM: Efficient Scaling of Language Models with Mixture-of-Experts. ICML 2022. Matched GPT-3 quality at roughly one-third the training compute using a 1.2T parameter MoE.
• Clark et al. (2022). Unified Scaling Laws for Routed Language Models. Derived scaling laws for sparse expert models, including the diminishing returns of adding experts and the routing tax.

Recent MoE architectures

• Jiang et al. (2024). Mixtral of Experts. Open-weight MoE release that popularized the architecture outside large labs.
• Dai et al. (2024). DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models. Introduced fine-grained experts and shared experts for improved specialization.
• DeepSeek-AI (2024). DeepSeek-V3 Technical Report. Auxiliary-loss-free load balancing via dynamic bias terms added to routing scores.

All posts