On a Mac, the memory you need is roughly the model file size plus the KV-cache plus a working margin for the OS and the app. For a quantized 7–8B model at short-to-moderate context, 16GB is workable but tight, 24GB is comfortable, and 32GB or more gives you headroom for longer contexts and larger models. Long contexts, not parameter count alone, are what usually blow past a memory budget.
Why Unified Memory Matters on Apple Silicon
On most PCs, the CPU and GPU have separate pools of RAM, and a model running on the GPU must fit entirely in VRAM (video memory), often 8–24GB and not upgradable. Apple Silicon changes this: the CPU, GPU, and Neural Engine share one physical pool called unified memory. There is no separate VRAM copy, so the GPU can address nearly the whole pool, which is why a Mac with 64GB can hold models that would demand an expensive discrete GPU with comparable VRAM.
The catch is that this pool is shared with everything else. macOS itself, your browser, and the app running the model all draw from the same memory. Apple also does not let the GPU use the full amount: the driver caps GPU-accessible memory below the total (the exact ceiling varies by chip and macOS version), and the system needs a reserve to stay responsive. Plan on roughly 60–75% of total memory being realistically available for model weights and KV-cache, and treat that as an estimate rather than a spec.
Bandwidth matters as much as capacity. Memory bandwidth (how fast data moves between memory and compute) largely determines tokens per second, and it scales with chip tier: base chips sit in the low hundreds of GB/s, Pro chips higher, Max chips higher still, and Ultra chips highest. Capacity decides what you can load; bandwidth decides how fast it runs.
Sizing Models by Parameter Count and Quantization
Model size starts with parameter count: a "7B" model has 7 billion learned weights, and "70B" has 70 billion. Weights are stored at some precision. In FP16 (16 bits, 2 bytes per weight), a 7B model needs about 14GB just for weights. Quantization stores weights at lower precision: 8-bit is about 1 byte per weight, and 4-bit about 0.5 bytes, with small overhead for scales and metadata. The rough formula:
weights_GB ≈ params_billions × bytes_per_weight
4-bit: 7B ≈ 3.5–4.5GB 13B ≈ 7–8GB 30B ≈ 16–18GB 70B ≈ 38–42GB
8-bit: 7B ≈ 7–8GB 13B ≈ 13–14GB 30B ≈ 30–32GB 70B ≈ 70GB+
Quantization cuts memory sharply, and 4-bit is the usual sweet spot: quality loss is modest for most tasks, though smaller models degrade more noticeably than large ones. Below 4-bit (3-bit and lower), quality drops faster and gains shrink, so it is rarely worth it unless you are desperate to fit a model. Note that these are weight-only figures; you still add the KV-cache and runtime overhead, covered next. Also budget for the app and OS: 2–4GB is a reasonable working margin on a 16GB machine, more if you keep many browser tabs open.
The KV-Cache: The Hidden Memory Cost of Long Contexts
Weights are the easy part. The KV-cache — the stored key and value tensors for every token the model has already seen — is the cost that surprises people. Transformer attention lets each new token look back at all previous tokens; to avoid recomputing them, the model caches their internal representations. That cache grows linearly with context length and batch size, and it lives in the same unified memory as everything else.
Size it with:
KV bytes ≈ 2 (K and V) × layers × kv_heads × head_dim × bytes_per_value × tokens
kv_heads is the number of key/value heads. Many modern models use grouped-query attention (GQA), where several query heads share one KV head, which shrinks the cache substantially. A typical 7B–8B model might have 32 layers, 8 KV heads, a head dimension of 128, and store values in fp16 (2 bytes).
Worked example, using those assumptions:
2 × 32 × 8 × 128 × 2 = 131,072 bytes per token ≈ 0.125 MiB/token
8,192 tokens → ~1 GiB
32,768 tokens → ~4 GiB
So a “small” 8B model at 32K context can quietly add about 4 GiB on top of its weights. Multiply by concurrent sequences if you batch requests. Long-context work is memory work.
Practical Recommendations: 16GB, 24GB, 36GB, 64GB, and Beyond
Numbers below are estimates from stated assumptions: weights ≈ parameters × bits ÷ 8, plus roughly 2–4 GiB for macOS and overhead, plus KV-cache from the formula above.
- 16GB: Comfortable for 7B–8B at 4-bit (about 4–5 GiB of weights) with moderate context. 13B–14B at 4-bit fits tightly and leaves little headroom.
- 24GB: 13B–14B at 4-bit with room for longer contexts; 30B-class models at 4-bit are possible but cramped.
- 36GB: A sweet spot for 30B–34B at 4-bit with useful context, or 70B at very aggressive quantization (slow and tight).
- 64GB: 70B at 4-bit (roughly 35–40 GiB of weights) plus a working context budget; multiple smaller models or heavier batching.
- 96GB–128GB+: 70B at higher precision, 100B-class models at low bit-widths, and serious long-context or multi-user workloads.
Sizing checklist:
1. Pick model size and quantization → estimate weights (params × bits ÷ 8)
2. Add OS + app overhead: budget 2–4 GiB
3. Compute KV-cache at your target context (formula above)
4. Add 10–20% safety margin
5. Compare total to unified memory; if over, lower quantization,
shorten context, choose a GQA model, or step up a memory tier
Rule of thumb: choose the tier where your weights, KV-cache, and overhead fit with at least 10% to spare — headroom is what keeps generation smooth instead of swapping.
Frequently Asked Questions
How much unified memory do I need to run a 7B model on a Mac?
For a 7B model at 4-bit quantization, the weights alone take roughly 4–4.5GB, so a 16GB Mac can run it with room for the KV-cache and macOS overhead, while 24GB gives comfortable headroom for longer contexts. At 8-bit you should budget around 8GB for weights, making 16GB tight and 24GB the realistic minimum. These are estimates from the rule of thumb that memory scales with parameter count times bits per weight, divided by 8.
Does quantization reduce memory usage enough to run larger models?
Yes—quantization lowers the bits used per weight, so 4-bit cuts weight memory to roughly a quarter of the 16-bit baseline, letting a 16GB machine host a 13B model that would otherwise need around 26GB. The trade-off is some quality loss, which is usually modest at 4-bit for everyday tasks but grows at more aggressive settings. Note that quantization shrinks weights, not the KV-cache, so long contexts still cost the same.
How does context length affect memory through the KV-cache?
The KV-cache stores the key and value tensors for every token the model has seen, so its size grows linearly with context length and batch size. A rough formula is:
KV bytes ≈ 2 × layers × kv_heads × head_dim × tokens × bytes_per_value
With grouped-query attention, kv_heads is far smaller than the attention head count, which keeps this manageable—often a few GB at 8K–32K tokens for a 7B–13B model. Long contexts can therefore rival the weights themselves in memory use.
Can a 16GB Mac run local LLMs comfortably?
Yes, for models up to about 7B–8B at 4-bit with moderate context, 16GB is workable, though macOS and other apps share the same pool, so keep background usage low. "Comfortably" is the caveat: 13B at 4-bit fits but leaves little slack, and anything larger or with very long contexts will spill into slowdowns or fail to load. Treat 16GB as an entry point, not a ceiling.
Is unified memory shared between the CPU and GPU on Apple Silicon?
Yes—on Apple Silicon the CPU, GPU, and Neural Engine draw from a single unified memory pool, so there is no separate VRAM to copy weights into. This is why a Mac can run LLMs at all without a discrete GPU, but it also means the model competes with the entire system for the same RAM. The practical consequence: your usable budget is total memory minus what macOS and your apps need.