LLM Quantization Benchmark: GGUF vs AWQ vs GPTQ vs EXL2 Architecture & Selection Guide
Comprehensive technical comparison of modern post-training quantization. Perplexity loss curves, GPU VRAM requirements, CPU offloading, and runtime performance matrices.
βοΈ Why is model weight quantization necessary?
A large language model of 70B (70 billion parameters) under uncompressed standard half precision (FP16/BF16, each parameter occupies 2 bytes) requires 140 GB of video memory only for the weights themselves; coupled with the huge KV Cache during inference runtime, 2 A100s (80G) must be equipped to barely start.
Through Model quantization, 16-bit floating-point number mapping is compressed into 8-bit, 4-bit or even 2-bit integer representation. The graphics memory overhead of the 70B model can be reduced to 35~40 GB, and only one RTX 3090/4090 or mainstream consumer-grade dual cards can run smoothly!
π Horizontal panoramic comparison of four mainstream mass formats
| Format standards | Represent ecology | memory reduction ratio | Precision Loss (PPL) | Hardware affinity | Recommended applicable scenarios |
|---|---|---|---|---|---|
| GGUF | llama.cpp / Ollama / LM Studio | About 70% (Q4_K_M) | Very low (<0.1 PPL) | CPU + GPU Hybrid / Apple M Series | Local PC, Mac, laptop without discrete graphics |
| AWQ | vLLM / TGI / SGLang | About 70% (4-bit) | Extremely low (protecting the key 1% weight) | NVIDIA Tensor Core (Ada/Ampere) | Enterprise high-concurrency API services and production clusters |
| GPTQ | AutoGPTQ / ExLlama / vLLM | About 70% (4-bit) | Low | General purpose NVIDIA GPU | Highly mature and compatible with older GPUs (such as Turing) |
| EXL2 | ExLlamaV2 / TabbyAPI | Freely adjustable (2.2~8.0 bpw) | Excellent | Single card extreme reasoning | Single-user extremely fast generation and accurate filling of video memory |
π¬ Core Mathematical Principles and Technical Differences
π 1. Why is AWQ (Activation-aware Weight Quantization) the first choice for servers?
AWQ found that not all model weights are equally important: less than 1% of salient weights carry the core output of the vast majority of attention mechanisms. AWQ does not blindly degrade the entire matrix uniformly, but observes significant weights based on forward activation and compresses the remaining 99% parameters into 4-bit while protecting this 1% floating point accuracy. This enables AWQ to achieve both extremely high throughput and almost lossless perplexity in vLLM high-concurrency scenarios.
π 2. Why can GGUF take the personal desktop by storm?
GGUF is a single-file, fully-encapsulated binary specification designed by Georgi Gerganov. Unlike Hugging Face's traditional dozens of safetensors, GGUF solidifies tensor metadata, hyperparameters, and word segmentation vocabularies into one .gguf file. It natively supports the mmap memory mapping of the operating system, can complete model mounting in milliseconds, and can perform fine-grained layer offload between memory (RAM) and video memory (VRAM).
π― Engineer Decision Tree: Which format should I pull?
Β Β β Pull the GGUF format (Q4_K_M or Q5_K_M is preferred) and use it with Ollama.
Β Β β Pull the AWQ format and use vLLM to enable PagedAttention and continuous batch processing.
Β Β β Pull EXL2 format (about 4.0~5.0 bpw) and cooperate with ExLlamaV2 engine.