Q4 vs Q8: How Much VRAM Do You Really Need?
Quantization is the single biggest lever for fitting a model on your GPU. In plain English: it shrinks the model so it uses less VRAM, with a small hit to quality. Q4 is the sweet spot for most people.
What quantization actually is
A model stores billions of numbers (weights). Quantization stores each number with less precision — think of it like saving a photo at slightly lower quality to make the file smaller. Q4 = 4-bit, Q8 = 8-bit. Fewer bits = smaller file = less VRAM.
How much VRAM each quant needs
Roughly, for a 13B model:
VRAM FOR A 13B MODEL
Add ~1–2 GB for context. Numbers scale with model size.
Which one should you use?
| Quant | Quality | Use it when |
|---|---|---|
| Q4_K_M | ~98% of full | Default. Best size-to-quality. Start here. |
| Q5_K_M | ~99% | You have VRAM to spare and want a touch more. |
| Q8_0 | ~99.5% | Quality-critical work and plenty of VRAM. |
| Q3 / Q2 | Noticeably worse | Only to squeeze a bigger model onto a small card. |
Bottom line: pick
Q4_K_M unless you have a specific reason not to. The quality loss vs Q8 is tiny; the VRAM savings are huge.Want the exact quant that fits your GPU + model? The calculator does the math.
Check your GPU free →
New to this? Start with How to Run an LLM Locally, or see what a 12GB GPU can run.