Mixture of Experts (MoE) Explained: Why Huge Models Run Fast but Need So Much Memory

Open the model card of a recent open-weight release and you will often see two parameter counts instead of one: something like "30B total, 3B active." Those two numbers are the signature of a mixture-of-experts (MoE) model, and they explain a puzzle that confuses almost everyone the first time: how can a model be enormous, yet generate text faster than a much smaller one, while still demanding a huge amount of memory?

This guide explains how MoE works, what total and active parameters really mean, and what the design implies for speed, VRAM and the hardware you choose.

Dense models versus sparse models

A traditional ("dense") transformer uses every parameter for every token. If a model has 70 billion parameters, each generated token passes through all 70 billion of them. Compute and memory traffic grow in lockstep with size, which is why bigger dense models are slower and costlier to run.

A mixture-of-experts model is sparse. It still has a very large number of parameters, but for each token it only uses a small fraction of them. The model decides, token by token, which parts to activate. The idea is old, but it has become the dominant way to build very large language models because it breaks the link between how much a model can store and how much work it does per token.

What is inside an MoE layer

A transformer layer has two main blocks: self-attention, which mixes information between tokens (see Stage 7, ), and a feed-forward network (FFN), which processes each token individually. Most of a model's parameters live in the FFNs.

An MoE model replaces the single FFN in a layer with several parallel FFNs, called experts, plus a small router (also called a gating network). The process for each token looks like this:

1. The router looks at the token's current representation and scores every expert.
2. It selects the top few, commonly one or two in early designs and more in recent ones.
3. Only those experts run on the token. Their outputs are combined, weighted by the router's scores.
4. All the other experts sit idle for that token.

Attention layers are normally shared and always active. Some newer designs also include one or more "shared" experts that every token passes through, alongside the routed ones.

Total parameters versus active parameters

This is where the two numbers come from.

• Total parameters: everything stored in the model, all experts included.
• Active parameters: what a single token actually uses: the attention layers, the router, and the selected experts.

A few well-known examples (approximate figures):

Notice that Mixtral 8x7B is not 56 billion parameters, even though 8 × 7 = 56. The experts are separate, but the attention layers and embeddings are shared across them, so the total is smaller than the naive multiplication suggests. Likewise, "A3B" in a model name tells you the active count directly.

Why labs build MoE models

The appeal is capacity per unit of compute. A model with more total parameters can store more knowledge and patterns. With MoE, you can grow total parameters a lot without growing the per-token cost at the same rate. For a fixed budget of training compute or serving compute, MoE models tend to deliver better quality than a dense model with the same active size.

A folk rule of thumb says an MoE model behaves roughly like a dense model somewhere between its active size and its total size. It is only a heuristic, since results depend on the architecture and training data, but it captures the right intuition: you do not get the quality of a dense model as large as the total count, and you get more than the active count alone suggests.

The catch: all experts must live in memory

Here is the part that surprises newcomers. Sparse computation does not mean sparse storage. The router can pick any expert for the next token, so for fast inference every expert has to be ready, which means the whole model must be loaded.

So an MoE model is cheap in compute per token and expensive in memory. Take a 30B-total model with 3B active, quantized to 4-bit at roughly 0.6 bytes per parameter:

• Weights in memory: about 30 × 0.6 = 18 GB.
• Weights read per token: about 3 × 0.6 = 1.8 GB.

A dense 30B model at the same precision also needs about 18 GB of memory, but it reads all 18 GB for every token. That is the whole trade: same memory bill, roughly a tenth of the per-token reading work.

If you want to see how the numbers play out for your own setup, the has a section on mixture-of-experts models, and covers the basics of weights, quantization and overhead.

Why MoE models are fast

Generating text is mostly limited by memory bandwidth, because each new token requires reading the active weights from memory. (We cover this in .) A rough ceiling is:

tokens per second ≈ memory bandwidth ÷ bytes read per token

On a GPU with about 900 GB/s of bandwidth, a dense 30B model at 4-bit has a theoretical ceiling near 50 tokens per second (900 ÷ 18). The 3B-active MoE has a ceiling near 500 (900 ÷ 1.8). Real numbers come out lower for both, because attention, the KV cache and overhead also take time, but the ratio is the point. MoE models feel much faster than dense models of the same total size.

A subtlety: batching erodes the advantage

The speed advantage is strongest when you serve one request at a time, as on a personal machine. In a busy server that batches many requests together, different tokens in the same batch are routed to different experts. With a large enough batch, almost every expert gets used in each step, so the server ends up reading most of the model's weights anyway. MoE still saves compute, but the memory-bandwidth advantage shrinks, and serving MoE models efficiently takes extra engineering: experts are often spread across several GPUs, which adds communication costs between them.

Running MoE models on your own hardware

This design creates an opportunity for home setups. Since only a small subset of experts is used per token, a runtime can keep the attention layers and shared parts on the GPU, and leave the bulk of the expert weights in system RAM. You trade speed for capacity.

Using the earlier example, reading 1.8 GB per token from system RAM at about 80 GB/s gives a ceiling of roughly 44 tokens per second, which is usable for interactive chat. Try the same with a dense 30B model and you are reading 18 GB per token from RAM, a ceiling near 4 tokens per second. That is why a big MoE model with plenty of RAM can be more practical on a mid-range PC than a dense model of similar quality. Apple's unified-memory machines are attractive for the same reason; see .

Remember that the KV cache is separate: it depends on the layers, attention heads and context length, not on how many experts exist. Our guide shows how to estimate it.

The training challenge: keeping experts balanced

MoE training has a notorious failure mode. If the router sends most tokens to a few favorite experts, those experts get better, which makes the router favor them even more, while the rest never learn. This is called routing collapse, and it wastes most of the model's capacity.

Designers fight it in several ways. Common techniques include an auxiliary load-balancing loss that penalizes uneven expert usage, limits on how many tokens an expert may take in a batch, and a bit of noise in the routing decision so that experts keep getting explored. Newer approaches adjust balancing without adding a loss term that can interfere with the main objective. The details differ between labs, but the goal is the same: use all the experts roughly evenly.

Myths worth dropping

"Each expert specializes in a topic, like a math expert or a history expert." It is a tempting picture, but studies of trained routers generally find that experts specialize in patterns that are much more granular and less human-readable, such as certain token types or syntactic roles. Do not expect to find a tidy "coding expert."

"MoE means running several separate models." It does not. Experts are components inside one network that is trained jointly, and the router is trained together with them.

"Active parameters tell you the memory needed." They tell you the compute and bandwidth per token. Memory is governed by total parameters.

"MoE models are always better." They win on quality per unit of compute, but they are harder to fine-tune, harder to serve efficiently at scale, and need more memory than their speed would suggest. For small models and tight memory budgets, a dense model can be the better choice.

Fine-tuning MoE models

Fine-tuning an MoE model is trickier than a dense one. The router can become unstable if you push it too hard, and you have to decide whether to adapt only attention layers, the experts, or both. Memory is also larger, because you are holding all the experts even though few are used per token. Our helps estimate what your hardware can handle, and Stage 9, , explains the adapter approach most people use.

How to choose between dense and MoE

• You have plenty of RAM or VRAM and want speed: MoE is often the better option.
• You are tight on memory: a dense model of modest size can fit more comfortably than a large MoE.
• You serve many users at once: MoE saves compute, but check whether your serving stack supports it well.
• You plan to fine-tune heavily: dense models are simpler to adapt.
• You compare models by size: always compare both numbers. A "30B" label on an MoE tells you almost nothing about speed unless you know the active count.

Key takeaways

• An MoE layer replaces one feed-forward network with many experts and a router that picks a few per token.
• Total parameters set the memory you need. Active parameters set the work per token.
• Speed behaves like a small model; memory behaves like a large one.
• Big batches reduce the speed advantage, because many experts get used at once.
• Experts are not neat topic specialists, and keeping them evenly used is a core training problem.

Next time a model card says "30B total, 3B active," you will know exactly what that buys you and what it costs.