The Model Landscape · L2 Mechanics

How dense vs. MoE architectures, training compute, and release strategy actually shape a model's behavior and cost.

Dense model

Every token uses all parameters
Simple to serve
Compute cost = full parameter count per token

Mixture-of-Experts (MoE)

Router picks a few experts per token
Total params can be huge, active params much smaller
Cheaper inference per token, more total capacity

The dense/MoE split explains a lot of the 'why is this huge model somehow cheap to run' confusion.

A dense 70B model uses all 70B parameters for every single token it processes. An MoE model like Mixtral-8x7B has a much larger *total* parameter count spread across multiple 'expert' feed-forward blocks, but a learned router activates only a couple of experts per token — so the *active* parameter count per token (and thus inference cost) is much smaller than the total. This lets MoE models pack more total capacity into the same inference budget, at the cost of more total memory needed to hold all experts.