Quantization makes model weights cheaper to store by representing them with fewer bits. It can turn an impossible local experiment into a practical one. The mistake is treating the downloaded file size as the complete memory requirement, or treating “4-bit” as a promise of identical quality across methods.
Start with weight arithmetic
The simplest estimate is parameter count × bits per parameter ÷ 8. An 8-billion-parameter model at 16 bits uses about 16 billion bytes for the raw weights. At 4 bits, the arithmetic is about 4 billion bytes. Those are decimal gigabytes; in binary units they are roughly 14.9 GiB and 3.73 GiB.
| Raw precision | Raw bytes for 8B parameters | Approximate binary size |
|---|---|---|
| 16 bits | 16,000,000,000 | 14.9 GiB |
| 8 bits | 8,000,000,000 | 7.45 GiB |
| 4 bits | 4,000,000,000 | 3.73 GiB |
This is arithmetic, not a measured memory benchmark. Real formats carry scales, metadata and sometimes higher-precision tensors. The runtime also needs buffers. Leave headroom rather than selecting a model that fits only on paper.
Context uses memory too
During autoregressive generation, the runtime commonly keeps attention keys and values for previous tokens in a cache. Its size depends on the architecture, context length, cache precision and batch size. Two models with the same parameter count can have quite different cache requirements.
Longer conversations and parallel requests can push an otherwise comfortable model beyond the memory limit. Test the context you intend to use, not only a one-sentence prompt immediately after loading. Our context-window guide explains why filling the advertised window can also hurt usefulness.
On unified-memory machines, model allocations compete with the operating system and your other applications. On a discrete GPU, host RAM and VRAM are separate budgets. Offloading can make a model run, but crossing a slower interface can change its latency significantly.
A bit count is not a quality score
Quantization methods choose different compromises: which tensors to quantize, how to group values, how to calculate scales and whether to use calibration data. The Hugging Face quantization overview describes the broad memory/accuracy tradeoff. The method and checkpoint documentation tell you what a particular artifact actually contains.
Compare the original and quantized versions on a fixed task set. Include tasks with tight formatting requirements, exact answers and longer context. A response can sound fluent while losing the detail your application depends on. Avoid accepting a model solely because its first chat response looks normal.
Fewer bits do not guarantee speed
An efficient low-bit kernel can reduce memory traffic and accelerate inference. An unsupported format, dequantization overhead or partial CPU offload can erase that benefit. Prompt processing and token generation may also respond differently to the same change.
Measure both on your target hardware. If you use llama.cpp, record the runtime revision, model file, context settings, acceleration backend and offload configuration. “Runs on my laptop” is useful as a personal note; it is not yet a reproducible performance result.
Choose with a margin
First estimate weights. Add the expected cache and runtime allowance, then leave space for the rest of the machine. Load the model and test your longest representative request while observing memory. Finally, compare useful output quality against a larger-precision reference or another deployment route.
If the model repeatedly forces you to trade away the quality or context you need, use a smaller model or a hosted endpoint. The API-versus-local budget guide helps decide whether the hardware compromise is worth making.
Questions, answered.
How much memory do 8 billion parameters need at 4 bits?
The raw weight arithmetic is about 4 billion bytes, or 3.73 GiB. Quantization metadata, unquantized components, runtime buffers and the KV cache increase the actual requirement.
Does a 4-bit model always run faster?
No. Speed depends on the hardware, kernels, memory bandwidth and workload. Lower weight precision does not guarantee lower latency.
Published 4 November 2026. Source-check date: 4 November 2026. How we report.
