LLM AtlasObserver · 0 XP
Pricing
Mixture of Experts

Layer 1 · Intuition

Mixture of Experts

How to get a model with a trillion parameters that costs about as much to run as one with a tenth of that — by only ever using a slice of it at a time.

6 min read40 XP

Listen to this lesson

Podcast-style walkthrough · ~6 min

No recording yet · generated on first play

A router picks a small subset of 'expert' sub-networks per token; the rest sit idle for that token.

A dense transformer uses every single parameter for every single token. That's simple and it works, but it means parameter count and compute cost are welded together — want a bigger, smarter model, pay for it on every token, forever, at inference too. Mixture of Experts (MoE) breaks that link: it replaces one big feed-forward block with many smaller ones ('experts'), and a lightweight router decides, per token, which handful of experts actually get to process it.

The headline number: total vs. active parameters

  • Total parameters — everything stored on disk and in GPU memory across all experts. This is what determines VRAM footprint and how much 'knowledge' the model can hold.
  • Active parameters — what actually does compute for any given token, i.e. the router plus the small number of chosen experts. This is what determines FLOPs per token, and roughly, inference latency.

Mixtral 8x7B, for instance, has roughly 47B total parameters but only about 13B active per token (two of eight experts, plus shared attention layers). It runs at close to the speed of a 13B dense model while drawing on the capacity of something much larger — this active/total split is the entire economic case for MoE.

Dense model

  • Every token touches every parameter
  • Compute cost ∝ total parameter count
  • Simple to train, simple to serve
  • Scaling capacity means scaling cost 1:1

Sparse MoE

  • Each token touches a small subset of parameters
  • Compute cost ∝ active parameters, not total
  • Routing adds real training and serving complexity
  • Capacity and cost can scale independently
MoE trades engineering complexity for a much better capacity-per-FLOP ratio.

The catch: memory doesn't shrink

Cheaper compute per token does not mean cheaper to *host*. All experts, used or not, must sit in GPU memory ready to be routed to at any moment — you don't know in advance which expert a given token will need. A sparse model with 47B total parameters needs roughly the VRAM of a 47B dense model, even though it computes like a 13B one. This tension — 'compute of a small model, memory footprint of a big one' — is the single biggest practical headache in deploying MoE.

Check your understanding

4 questions · answer all to submit

  1. 1.In an MoE layer, what determines the FLOPs spent processing a single token?

  2. 2.Why does a sparse MoE model still require roughly as much GPU memory as its total parameter count implies, despite using far fewer active parameters per token?

  3. 3.What is the best characterization of the relationship between a dense model's parameter count and its inference compute cost, compared to an MoE model?

  4. 4.Mixtral 8x7B has roughly 47B total parameters but about 13B active per token. What does this ratio primarily explain?

Go deeper: L2 Mechanics