TurboQuant compresses the KV cache at runtime — the same model, the same quality, a fraction of the RAM. Here's exactly what changed and why it matters.
When a language model generates text, it stores every token's key-value pairs in memory. This KV cache grows linearly with context length — and on a phone with 4-6 GB of shared RAM, it's often the bottleneck that decides how much context you can use.
Same model file. Same inference quality. Dramatically less memory pressure.
Measured for Gemma-3 1B (18 layers, 8 heads, head_dim=128). The savings scale linearly with any model.
Memory in MB. Lower is better.
Where compression really shines — long context.
The short answer: almost none. Q8 is virtually lossless. Even Q4 maintains >98% inner product correlation — meaning the model's attention scores are nearly identical to FP16.
Reconstruction error per value. FP16 = 0 (perfect). Lower = better.
TurboQuant library achieves TQ4 MSE = 0.009 and TQ3 MSE = 0.034, matching the ICLR 2026 paper exactly.
Attention scores are computed via dot products. This shows how accurately they're preserved.
Above 0.98 correlation means the model "pays attention" to the same things. Q8 is effectively perfect.
| Mode | Bits / Value | Compression | Memory Used | MSE | Quality | Recommendation |
|---|---|---|---|---|---|---|
| Off (FP16) | 16 bit | 1.0x | 100% | 0.000 | Best | When RAM is abundant |
| Q8_0 | 8 bit | 2.0x | 50% | < 0.001 | Near-Perfect | Default for most devices Recommended |
| Q5_0 | 5 bit | 3.2x | 31% | ~0.005 | Great | Good balance of speed and quality |
| Q4_0 | 4 bit | 4.0x | 25% | ~0.009 | Good | Tight RAM or long context needs |
| Q4 Asymmetric | 4 bit (mixed) | 3.5x | 28% | ~0.007 | Good+ | Better accuracy than plain Q4 |
When you compress the KV cache, the freed memory can be used for more context tokens. This means the model can "remember" more of your conversation.
If you have 36 MB available for KV cache (what FP16 uses for 512 tokens), here's how many tokens each mode can fit:
Q4_0 fits 4x more context in the same memory. That's the difference between remembering 2 paragraphs and remembering 2 pages.
Before: 256 token context, frequent "forgetting" mid-conversation. After with Q4:
1024 tokens — full conversations retainedBefore: 512 tokens, could handle short chats. After with Q8:
1024 tokens — with zero quality lossBefore: 2048 tokens max. After with Q4:
8192 tokens — entire documents in contextBased on the ICLR 2026 paper by Google Research. The key insight: KV vectors are roughly Gaussian, so optimal scalar quantization can be precomputed — zero calibration needed.
Multiply each KV vector by a random orthogonal matrix (seeded, reproducible). This spreads outlier values uniformly across all coordinates, which is critical for the next step. Uses fast Givens rotations — O(n log n) instead of O(n²).
After rotation, each coordinate is independently quantized using precomputed codebooks optimal for the Gaussian distribution. For 4-bit: 16 reconstruction levels. For 3-bit: 8 levels. The codebook centroids and decision boundaries are compile-time constants — no runtime training.
The quantization error (residual) is projected through a random Gaussian matrix, and only the sign bits are kept. This makes inner product estimation unbiased via the Johnson-Lindenstrauss lemma. Adds ~1 effective bit of precision at minimal cost.
Attention scores (Q·K⊃T) are computed directly from compressed blocks using codebook lookup + scaled dot product. No full dequantization needed for the hot path. NEON intrinsics on ARM64 process 4 values per instruction.
The standalone turboquant.h/cpp library (full algorithm + NEON) is compiled in parallel — ready for deeper integration when llama.cpp adds a pluggable KV cache API.
Every edge case is handled. The model always loads.
If a quantized KV type fails during context creation (some older model architectures), Lucy automatically retries with FP16 and logs a warning. The user never sees an error.
Zero crash riskOn armeabi-v7a (32-bit ARM), aggressive quantization modes are automatically clamped. Q5_0 falls back to Q8_0 at compile time. The user can't accidentally pick a mode that's too slow.
Compile-time safetyWhen KV compression is set to "Off", the code path is exactly the same as before this update. No new allocations, no new branches, no performance impact. Zero regression.
100% backward compatibleThis radar chart shows how each mode balances five key metrics.
Q8 is the sweet spot — almost no quality loss, significant memory savings. Q4 trades a small amount of quality for maximum compression.