Everyone tells you "use Q4," but nobody shows you what you give up. So this guide takes the 3B size, using two published models, and walks through their Q4, Q5 and Q8 GGUF builds on the four things that actually matter on a phone or laptop: file size on disk, RAM while running, generation speed, and output quality. The sizes come from the published GGUF files, RAM and speed follow from them, and quality is what published quantization comparisons consistently find. No new benchmark was run for this guide. The short version is that Q4 wins for most people, but the why is worth understanding, because it tells you exactly when to step up to Q5 or Q8. If you want the format basics first, read our GGUF guide; this post is the deep-dive on what each quant trades away.
Want the short version? Jump to the summary table. PocketLLM chooses a free default from total iPhone RAM. The choice does not guarantee a successful load. Join the launch list.
Q4 cuts about 40% off the file size of Q8 and a quarter to a third off its RAM while losing very little quality on everyday tasks — it's the right default for phones and 8 GB laptops. Q5 is a sensible middle ground if you have headroom. Q8 is near-original fidelity but worth its larger footprint only when memory is plentiful. Across published comparisons Q4 lands close to Q8 on drafting and summarizing, with the gap showing mainly on hard reasoning. Pick the highest quant that fits your RAM with room to spare.
PocketLLM is launching soon. Private, on-device AI, starting on iPhone, with more platforms planned. No account required, no in-app tracking, and chat processing stays on your device. Join the launch list and be first in.
How to compare quants
- One size, three quants. The comparison below holds the model size fixed and varies only its Q4, Q5 and Q8 GGUF builds, so the only variable is precision. The figures are approximations for a 3B model, not measurements taken for this article.
- File size: taken from the published GGUF builds, rounded. It varies a little by quant variant (Q4_K_M versus Q4_0, for example). Sizes are from the published Llama 3.2 3B and Qwen 2.5 3B GGUF builds (Q4_K_M, Q5_K_M, Q8_0).
- RAM needed: estimated as the file size plus 1 to 2 GB for the context cache and runtime overhead. Longer context means more.
- Speed: generation speed tracks how many bytes move per token, so it follows the file size; a lower quant is usually faster on the same device, though runtime and hardware matter too.
- Quality: what published quantization comparisons report for everyday drafting and summarising versus hard reasoning. This guide did not run its own grading.
A note on honesty: these are approximations for a 3B model, not measurements from this guide. Exact figures vary by model, quant variant and device, but the relationships, Q4 smallest and usually fastest, Q8 largest and closest to the original, hold in general.
What each quant traded
Q4 — smallest, fastest, slight quality cost
At Q4 a 3B model is about 2 GB on disk and wants roughly 3 to 4 GB of RAM in practice, and it is usually the fastest of the three because generation speed tracks how many bytes have to move per token. Quality sits slightly below Q8, which is invisible on everyday drafting and summarizing and faintly noticeable on the hardest reasoning prompts. For phones and 8 GB Macs this is the clear default: you get nearly all the quality for about 60% of the disk footprint.
Q5 — the middle ground
Q5 adds roughly 15% to the file (about 2.3 GB) and a little RAM (around 3.5 to 4.5 GB), runs slightly slower than Q4, and in published comparisons recovers most of the small quality gap to Q8. It's the pick when you have headroom over Q4 but not enough to justify Q8, or when you want a touch more reliability on reasoning without doubling the size.
Q8 — near-original, heaviest
Q8 is about 3.4 GB on disk, needs around 4.5 to 5.5 GB of RAM, and runs a little slower again, while staying closest to the original model's output, which is where published comparisons put it on hard reasoning. The catch is the footprint: on an 8 GB device that's tight once the OS takes its share. Q8 makes sense on 16 GB machines where you want the most a given model can give.
The summary table
| Quant | File size (3B) | RAM needed (estimate) |
|---|---|---|
| Q4 | ~2.0 GB | ~3 to 4 GB |
| Q5 | ~2.3 GB | ~3.5 to 4.5 GB |
| Q8 | ~3.4 GB | ~4.5 to 5.5 GB |
How to read the table
The headline is the shape of the curve, not the exact points. Going Q4 → Q8 adds about 70% to the size and roughly 1.4 GB of RAM, while quality improves only marginally. That's diminishing returns: most of the model's ability survives aggressive quantization, so the cheap quant captures almost all the value. The exception is hard reasoning, where every point counts — if that's your workload and you have the memory, the higher quant earns its keep. For everything else, the size and speed savings of Q4 win.
How to pick a quant for your device
Use one rule: choose the highest quant that fits your RAM with comfortable headroom for the system. On a phone or 8 GB Mac, Q4 of a 3B is the safe default; Q5 or Q8 can fit when context is short and little else is running, but they crowd memory. On a 16 GB Mac, you can run Q8 of a small model or, often better, spend that memory on a larger model at Q4, since a bigger model at Q4 usually beats a smaller one at Q8. When you're choosing the model itself, our best small language models roundup pairs naturally with this quant guidance.
For phone use, read the checks for quantized LLMs on iPhone.
Why this matters for on-device apps
On a phone, quant choice is the difference between a model that runs smoothly and one that swaps and stutters. PocketLLM chooses a free default from the iPhone's total RAM. Before each load, a pre-flight check measures free process memory. If the model does not fit, the app refuses the load and tells you to choose a smaller model. The choice does not guarantee a successful load. If you want the broader app landscape, see our best on-device LLM apps for iPhone roundup.
Want the right quant picked for you and run privately on iPhone? PocketLLM handles quant selection on-device with zero telemetry. Join the launch list.
Frequently asked questions
What is the difference between Q4, Q5, and Q8 quantization?
They differ in how many bits represent each weight: the labels are nominal: Q4 stores weights at roughly 4 bits, Q5 at roughly 5, and Q8 at 8, with some tensors and metadata kept at higher precision, so files come out a little larger than the label implies. Fewer bits means a smaller file and less RAM but slightly lower output quality; more bits means a larger file and more RAM but quality closer to the original. In both size and quality the order is Q4 < Q5 < Q8. The quality gap between Q4 and Q8 is generally small for everyday tasks and grows on the hardest reasoning, which is the consistent finding across published quantization comparisons.
Which quantization level is best?
For most people on phones and 8 GB laptops, Q4 is best because it cuts file size by about 40% versus Q8 and RAM with it while losing very little quality on everyday tasks. Q5 is a reasonable middle ground if you have headroom. Q8 is worth it only when memory is plentiful and you want the highest fidelity from a given model. The best quant is the highest one that fits your RAM comfortably while leaving room for the system.
Does Q4 quantization hurt quality a lot?
No, not for most uses. Published comparisons consistently show Q4 landing close to Q8 on everyday drafting, summarizing and chat, at about 40% less disk and roughly a quarter to a third less RAM. The gap widens on the hardest multi-step reasoning, where higher precision helps. For the typical on-device workload, Q4 is the right trade-off: large size savings for a small, usually invisible quality cost.
How much RAM does each quant need?
As a rule of thumb, RAM is the file size plus 1 to 2 GB for context and runtime. For a 3B model, Q4 is about 2 GB on disk and needs around 3 to 4 GB of RAM, Q5 is about 2.3 GB needing roughly 3.5 to 4.5 GB, and Q8 is about 3.4 GB needing around 4.5 to 5.5 GB. The higher the quant, the more memory the model occupies. Always leave headroom for the operating system, so on an 8 GB device a 3B at Q4 is comfortable while a 7B at Q8 is not.
Is a higher quant always slower?
Often slightly, because a higher quant means more data to move through memory per token, so Q8 typically generates a bit slower than Q4 on the same model and device. The difference is usually modest on Apple Silicon. More important than the small speed change is whether the larger Q8 file fits in RAM at all — if it forces the system to swap, speed collapses, which is why matching the quant to your memory matters more than chasing raw bits.