Running LLMs locally means you download the model weights (the billions of learned parameters) and do the math on your own machine instead of sending prompts to a company's server. Your main constraint is memory, not raw compute: the model must fit in VRAM (GPU memory), unified memory, or system RAM, and quantization (storing weights at lower precision) is how you shrink that footprint. In practice, a 7B-parameter model at 4-bit quantization needs roughly 4–5 GB for weights plus space for context, so 8 GB of VRAM is a realistic entry point and 12–24 GB opens up 13B–34B models.
Why Run LLMs Locally
The first reason is privacy and control: prompts, documents, and outputs never leave your machine, which matters if you work with anything confidential. The second is cost predictability: after hardware, there is no per-token billing, so heavy experimental use has no marginal price. The third is availability and stability: a local model does not change behavior or disappear when a provider updates or retires an API model, and it works offline.
There are real trade-offs. Frontier-quality models are too large for consumer hardware, so you are choosing among smaller open-weight models that are good but not identical to the biggest hosted ones. You also own the operations: downloading weights (often several GB), installing a runtime, managing drivers, and tuning settings. Throughput on a single consumer GPU is fine for interactive chat and light automation but not for serving many users at once.
For learners, the practical argument is that local inference makes the concepts tangible. You can watch memory usage change with context length, compare quantized versus full-precision outputs, and see why a model that "fits" in theory still runs out of memory in practice. That hands-on loop teaches more than reading about inference ever will.
Start with a clear goal: chat and drafting on one machine, coding assistance, batch processing of documents, or experimentation with many models. The goal decides whether you should prioritize memory capacity, memory bandwidth, or raw compute.
GPU, CPU and Unified Memory: What Matters
LLM inference is memory-bandwidth-bound, not compute-bound. Generating each token requires reading the model's weights (or the active portion of them) from memory, so tokens per second scale roughly with how fast memory can feed the processor, not with how many math units it has. This is why a modest GPU with fast VRAM often beats a powerful CPU with slow RAM, and why a top-end GPU can still feel slow on a model that spills out of VRAM.
Three hardware layouts matter. Discrete GPUs have dedicated VRAM connected by high-bandwidth links; a model that fits entirely in VRAM runs fast, and one that partially spills to system RAM slows dramatically because data crosses a much narrower bus. CPUs use system RAM, which is plentiful and cheap but typically an order of magnitude slower in bandwidth, so CPU-only inference works for smaller models and low-concurrency use but not for snappy large-model chat. Unified memory (Apple Silicon, some integrated and server designs) shares one pool between CPU and GPU, so capacity is large and there is no copy bottleneck, but bandwidth sits between discrete GPUs and CPU RAM depending on the chip.
VRAM capacity decides whether a model runs comfortably; memory bandwidth decides how fast. A useful mental model:
tokens/sec ~ memory_bandwidth / bytes_read_per_token
bytes_read_per_token ~ active_parameters * bytes_per_parameter
With quantization, bytes per parameter drop from 2 (16-bit) to about 0.5 (4-bit), which is why 4-bit models run several times faster when bandwidth-bound and also fit in far less memory. Estimate rather than trust exact figures: take the chip's published bandwidth, divide by the model's on-disk size in bytes, and treat the result as an optimistic ceiling you will not fully reach.
CPU-only setups are still useful: small models (roughly 1B–8B at 4-bit) on a modern desktop CPU with 16–32 GB of RAM can produce readable output at a few tokens per second, enough for batch jobs and experimentation. Hybrid setups (some layers on GPU, the rest on CPU) let you run a larger model than VRAM allows, at the cost of speed. Prefer a GPU with more VRAM over a faster GPU with less when choosing hardware for this purpose.
VRAM Math: Model Weights, KV-Cache and Overhead
VRAM (video memory) holds three things during inference: the model weights, the KV-cache, and runtime overhead. If any one is underestimated, you get an out-of-memory error or silent spillover to system RAM that tanks speed.
Weights are the easy part. Parameters × bytes per parameter:
- FP16: 2 bytes/param → a 7B model needs ~14 GB.
- 8-bit: 1 byte/param → ~7 GB.
- 4-bit: ~0.5–0.6 bytes/param → ~3.5–4.5 GB (quantization metadata adds a little).
KV-cache (key/value cache) stores attention states for every token in context. It scales with layers × hidden size × context length × bytes per value, and it grows linearly as the conversation lengthens. As a working estimate, budget roughly 0.5–1 GB per 8K tokens for a 7B model in FP16, doubling for 13B-class models. Crucially, the cache must fit in addition to the weights.
Overhead covers the compute graph, CUDA/ROCm context, and fragmentation: plan for 1–2 GB of headroom.
Worked example. Assume a 7B model at 4-bit weights, 8K context, FP16 KV-cache, and 1.5 GB overhead:
- Weights: 7B × 0.55 bytes ≈ 3.9 GB
- KV-cache: ~0.7 GB (estimated at ~0.09 GB per 1K tokens for this size)
- Overhead: 1.5 GB
- Total ≈ 6.1 GB
So an 8 GB GPU fits this comfortably; a 6 GB card is borderline once you raise context. Double-check by watching actual usage in your runtime's monitoring tool rather than trusting the estimate.
Quick formula:
total_GB ≈ params_B × bytes_per_param
+ (context_tokens / 1000) × kv_GB_per_1k
+ overhead_GB
Checklist before loading a model:
- Weights fit with ≥1 GB spare
- KV-cache budgeted for your maximum context, not the default
- Overhead allowance included
- Offloading plan if it doesn't fit (fewer GPU layers, smaller quant, shorter context)
Quantization Formats Explained: GGUF, GPTQ and AWQ
Quantization reduces the precision of weights (and sometimes activations) to shrink memory and speed up inference, trading a small amount of quality for large savings.
GGUF is a file format used mainly by llama.cpp and its ecosystem. It stores quantized weights in blocks with per-block scales and supports mixed precision (e.g., most layers at 4-bit, sensitive layers kept higher). It runs on CPU, GPU, and Apple Silicon, and supports partial GPU offload — you choose how many layers go to the GPU. Common variants: Q4_K_M (a good quality/size balance) and Q5_K_M (slightly larger, closer to full precision).
GPTQ is a post-training quantization method producing 4-bit (typically) weights for GPU inference, usually served via ExLlama-style backends. It quantizes layer by layer using calibration data to minimize error. It is GPU-focused and generally fast, but less flexible for CPU or mixed offload.
AWQ (Activation-aware Weight Quantization) also targets 4-bit GPU inference. It protects the most salient weight channels based on activation magnitudes, which often preserves quality slightly better than naive round-to-nearest at the same bit width. Like GPTQ, it assumes a GPU backend.
Practical guidance: choose GGUF if you want portability, CPU or Apple Silicon support, or partial GPU offload. Choose GPTQ or AWQ if you have a capable GPU and want maximum throughput with a fixed 4-bit footprint. All three at 4-bit land in a similar memory range; differences show up in speed, backend support, and small quality variations that are hard to generalize without benchmarking on your own tasks — so measure perplexity or task accuracy on a sample before committing.
Frequently Asked Questions
How much VRAM do I need to run a 7B model locally?
It depends almost entirely on the quantization level. As a rough estimate, a 7B model needs about 4–5 GB of VRAM at 4-bit quantization, 7–8 GB at 8-bit, and roughly 14 GB at 16-bit (FP16), before adding KV-cache and runtime overhead. Add 1–2 GB of headroom for the KV-cache at moderate context lengths and for the inference framework itself, so a 4-bit 7B model fits comfortably on an 8 GB GPU, while 16-bit needs a 16–24 GB card.
What is quantization and does it hurt model quality?
Quantization stores the model's weights in lower-precision number formats (for example 4-bit or 8-bit integers instead of 16-bit floats), which shrinks memory use and speeds up inference. Quality loss is usually small at 8-bit and modest at 4-bit for most chat and coding tasks, though very aggressive quantization (2–3 bit) or small models degrade noticeably. In practice, a larger model at 4-bit often beats a smaller model at full precision within the same VRAM budget.
Can I run LLMs on CPU or Apple Silicon without a GPU?
Yes. CPU inference works but is slow, typically a few tokens per second for 7B-class models on a modern desktop CPU, so it suits experimentation rather than interactive use. Apple Silicon is different: its unified memory lets the GPU use the same RAM as the CPU, so a Mac with 16–64 GB of memory can run models far larger than a typical consumer GPU's VRAM would allow, at respectable speeds via Metal-based runtimes.
What is the difference between GGUF, GPTQ and AWQ?
GGUF is a file format used mainly by llama.cpp-style runtimes that supports flexible CPU/GPU offloading and many quantization levels, making it the most portable choice. GPTQ and AWQ are GPU-oriented quantization methods: GPTQ quantizes weights post-training with a calibration dataset, while AWQ protects the most salient weights during quantization, often preserving quality slightly better at 4-bit. GPTQ and AWQ typically require a CUDA-capable GPU and a runtime such as ExLlama or vLLM, whereas GGUF runs almost anywhere.
How does context length affect VRAM usage?
Every token in the context window needs a stored key and value vector for each layer and attention head, collectively called the KV-cache, and this grows linearly with context length. The extra memory is modest at short contexts but can reach several gigabytes at long ones; as an estimate, doubling the context roughly doubles KV-cache size. Reducing context length, using grouped-query attention models, or quantizing the KV-cache are the standard ways to keep it in check.