Pick GGUF when you want to run a model on CPU or mixed CPU/GPU hardware, especially through llama.cpp or tools built on it. Pick GPTQ or AWQ when you are serving on GPUs and want fast, memory-efficient inference with formats supported by GPU-focused runtimes. The right choice is mostly determined by where the model will run and which inference engine you use, not by the formats themselves.
Why Quantization Formats Matter at Inference Time
Quantization means storing a model's weights in a lower-precision numeric format than the 16-bit floats (FP16) or 8-bit integers (INT8) used during training or standard deployment. A weight is just a number; reducing its precision shrinks the file size and, on the right hardware, reduces memory traffic and speeds up the matrix multiplications that dominate inference. The catch is that different formats do this reduction differently, and those differences determine which hardware and software can run them efficiently.
This is why "format" and "runtime" are inseparable. A quantization format is not just a file layout; it encodes assumptions about how kernels (the low-level compute routines) will load and multiply those numbers. GGUF is designed around llama.cpp's CPU and mixed-device execution, while GPTQ and AWQ assume GPU kernels, typically CUDA, that unpack weights on the fly. A model in the "wrong" format may technically load but run slowly or not at all.
Precision also interacts with memory beyond the weights. The KV-cache β the stored keys and values for previously processed tokens that let a model attend to context without recomputation β grows with context length and batch size, and quantization formats generally do not shrink it automatically. Format choice sets your weight memory floor; runtime settings and cache quantization handle the rest. Treat sizing as an estimate: a rough rule is bytes-per-weight times parameter count, plus overhead for the cache, so a 7B model at 4-bit is roughly 3.5β4 GB of weights before cache and runtime overhead.
GGUF: CPU-Friendly and llama.cpp-Native
GGUF (GPT-Generated Unified Format) is the file format used by llama.cpp and the ecosystem around it: local chat apps, CPU-capable servers, and tools that run on laptops or Apple Silicon. It packages weights, tokenizer data, and metadata in one file and supports many quantization levels, from aggressive 2-bit variants up to 8-bit and beyond, often with mixed precision where sensitive layers keep more bits. The format is built so that a single file can be memory-mapped and executed across CPU and GPU (via backends like Metal, CUDA, or Vulkan), splitting layers between devices when the model does not fit in VRAM.
Its main strength is portability. You can run a GGUF model on a machine with no discrete GPU, or offload part of it to whatever GPU is present, which makes it the default choice for local and edge deployment. The tradeoff is that throughput on high-end GPUs is generally lower than purpose-built GPU serving stacks, because llama.cpp prioritizes broad hardware support over peak batched throughput.
GPTQ and AWQ: GPU-Optimized Weight Quantization
GPTQ and AWQ are post-training quantization methods aimed at GPU inference, usually at 4-bit. GPTQ (GPT Quantization) quantizes weights layer by layer, using a small calibration dataset to minimize the error each weight change introduces in that layer's output. AWQ (Activation-aware Weight Quantization) takes a different angle: it identifies the small fraction of weights that matter most based on activation patterns and protects them, quantizing the rest more aggressively. In practice both produce 4-bit models that load into GPU runtimes such as vLLM, TensorRT-LLM, or Text Generation Inference, where custom kernels keep memory bandwidth low and batch throughput high.
Choose between them by what your serving stack supports and by measured quality on your task. Both are mature enough that the difference often comes down to specific model and runtime combinations; test on your own prompts rather than assuming one wins. Neither is a good fit for pure-CPU execution, and neither is natively loadable by llama.cpp without conversion.
Choosing by Hardware, Workload, and Quality Budget
Start with hardware. No GPU, or a laptop with unified memory: GGUF. One or more datacenter GPUs serving many concurrent requests: GPTQ or AWQ through a GPU runtime. A single consumer GPU where the model barely fits: GGUF with partial offload can be more flexible, while a 4-bit GPTQ/AWQ model may leave more VRAM for the KV-cache and larger batches.
Then consider workload. Interactive single-user chat tolerates lower throughput and favors easy setup, which points to GGUF. High-concurrency API serving needs batching efficiency and favors GPU-native formats. Finally, set a quality budget: 8-bit is close to lossless in practice, 4-bit is the usual sweet spot, and 2β3-bit variants trade noticeable quality for size. Validate with your own evaluations rather than trusting labels alone.
FAQ
What is the difference between GGUF, GPTQ, and AWQ? GGUF is a file format and runtime ecosystem centered on llama.cpp, supporting CPU and mixed-device inference with many quantization levels. GPTQ and AWQ are GPU-oriented 4-bit quantization methods whose outputs run in GPU serving stacks; they differ in how they decide which weights to protect during quantization.
Which quantization format is best for running models on CPU? GGUF. It is the native format for llama.cpp and related CPU-capable tools, and it is designed for memory-mapped, mixed CPU/GPU execution. GPTQ and AWQ assume GPU kernels and are not the practical choice for CPU-only machines.
Does quantization reduce model quality? Yes, but how much depends on the bit width and method. 8-bit is typically near-lossless for most tasks; 4-bit is usually a good balance; 2β3-bit can degrade reasoning and long-context behavior noticeably. Always evaluate on your own prompts, since damage varies by model and task.
Can I use GPTQ or AWQ models with llama.cpp? Not directly. llama.cpp loads GGUF files, so GPTQ or AWQ weights must be converted, and conversion may lose the method's specific calibration benefits or not be supported for every architecture. If you need llama.cpp, source a GGUF build of the model.
How does quantization affect KV-cache memory during inference?
Weight quantization does not automatically shrink the KV-cache; cache size depends on layers, context length, batch size, and head dimensions. Estimate it as roughly 2 * layers * heads * head_dim * bytes_per_value * tokens, and note that many runtimes offer separate cache quantization (for example 8-bit or 4-bit caches) that reduces this independently of the weight format.
GPTQ and AWQ: GPU-Optimized Weight Quantization
Both formats quantize model weights β the trained parameters β to low-bit integers so they fit in GPU memory and move through the memory bus faster. Inference frameworks (vLLM, TensorRT-LLM, ExLlama, text-generation-webui) load them directly onto the GPU.
GPTQ (Generative Pre-trained Transformer Quantization) is post-training, layer-by-layer quantization. Each layer is quantized while the rest of the model stays in higher precision, and the algorithm minimizes the squared error between the original and quantized layer outputs using a small calibration dataset (a few hundred text samples). Weights are typically stored as 4-bit integers with per-group scales and zero-points that map integers back to real values. The result: roughly 4x smaller than 16-bit weights, with a modest quality loss that is usually small at 4 bits and noticeable at 3 bits or below. GPTQ kernels are mature and widely supported, including for older GPUs without new-format acceleration.
AWQ (Activation-aware Weight Quantization) keeps the same 4-bit weight storage but chooses scales differently: it observes activations (the values flowing through the network at runtime) on calibration data and protects the small fraction of weight channels that, multiplied by large activations, matter most. The claim is better quality at the same bit width, especially for instruction-tuned models. AWQ kernels are fast, but framework support is somewhat narrower than GPTQ's.
Neither format speeds up prefill alone; the benefit is memory footprint and decode-time bandwidth. Note that the KV cache β memory for past tokens' keys and values β is not compressed by GPTQ or AWQ; that needs separate KV-cache quantization.
Worked example (assumptions stated): a 7B-parameter model at FP16 = 7e9 x 2 bytes = 14 GB of weights. At 4-bit with group size 128, add 2 bytes of scale/zero per 128 weights: 4 + 16/128 = 4.125 bits per weight, i.e. 7e9 x 4.125 / 8 = ~3.6 GB β about 26% of FP16. If the GPU has 24 GB, FP16 leaves ~10 GB minus KV cache and activations; 4-bit leaves ~20 GB, enough for long contexts or larger batches.
Choosing by Hardware, Workload, and Quality Budget
Checklist before downloading a quantized model:
- Hardware: CPU-only or Apple Silicon unified memory β GGUF. NVIDIA GPU with enough VRAM β GPTQ or AWQ.
- Runtime: llama.cpp/Ollama β GGUF only. vLLM/TensorRT-LLM/ExLlama β GPTQ or AWQ (verify the specific model is supported).
- Quality budget: if you can afford it, prefer 8-bit; 4-bit is the usual sweet spot; treat 3-bit and below as lossy experiments.
- Task sensitivity: code, math, and long-context reasoning degrade first β test on your own prompts, not just perplexity.
- Context length: budget KV-cache memory separately; it grows linearly with tokens and batch size.
- Fallback plan: keep the original FP16/BF16 checkpoint so you can re-quantize if quality disappoints.
Short sanity check idea in pseudocode:
bits_per_weight = weight_bits + (scale_bits + zero_bits) / group_size
weights_gb = params * bits_per_weight / 8 / 1e9
kv_gb = 2 * layers * kv_heads * head_dim * tokens * batch * bytes_per_kv
fits = weights_gb + kv_gb + activations_gb < vram_gb
If it fits with headroom, choose the higher bit width; if not, step down to 4-bit GPTQ/AWQ on GPU or GGUF on CPU before shrinking further.
Frequently Asked Questions
What is the difference between GGUF, GPTQ, and AWQ?
GGUF is a file format built for llama.cpp that stores quantized weights (and often the tokenizer and metadata) optimized for CPU and mixed CPU/GPU inference. GPTQ and AWQ are weight-quantization methods designed for GPU inference: GPTQ quantizes weights layer by layer using calibration data to minimize error, while AWQ (Activation-aware Weight Quantization) protects the small fraction of weights that matter most to activations. In short: GGUF is a container aimed at flexible, often CPU-side deployment, while GPTQ and AWQ are GPU-oriented quantization schemes typically served by engines like vLLM, TensorRT-LLM, or ExLlama.
Which quantization format is best for running models on CPU?
GGUF, essentially by default: it is the native format of llama.cpp, which is built around CPU execution and supports many quantization levels (commonly from roughly 2-bit up to 8-bit) plus optional GPU offload of some layers. GPTQ and AWQ assume GPU kernels and are not practical for pure-CPU inference. If your machine has no discrete GPU, pick a GGUF build whose bit width fits your RAM with headroom for the KV cache (the per-token attention memory).
Does quantization reduce model quality?
Yes, in general β lower precision means some loss of fidelity, and the drop grows as bit width shrinks. At 8-bit the difference is usually negligible; at 4-bit it is often small but measurable; below 4-bit, degradation becomes common and task-dependent. Actual impact varies by model, quantization method, and calibration data, so treat any specific quality claim as something to verify on your own workload rather than a universal rule.
Can I use GPTQ or AWQ models with llama.cpp?
No. llama.cpp loads GGUF files, not GPTQ or AWQ checkpoints, so a GPTQ/AWQ model must be converted to GGUF first β and conversion may not preserve the exact quantized weights. Some tools can repackage or requantize, but the straightforward path is: use GGUF with llama.cpp, and use GPTQ/AWQ with a GPU server such as vLLM or ExLlama. Choose the runtime first, then the format follows.
How does quantization affect KV-cache memory during inference?
Weight quantization shrinks the model's parameters, but the KV cache β stored keys and values for every token in context β is separate and usually kept at higher precision (often FP16). So 4-bit weights can cut model memory roughly 4x versus FP16, yet long contexts can still consume large KV-cache memory. Some engines support quantized KV caches to reduce this further, at some quality cost; budget for it explicitly:
KV bytes β 2 (K and V) Γ layers Γ kv_heads Γ head_dim Γ seq_len Γ bytes_per_value