Google TurboQuant

Long context is a memory problem. The KV cache, the running store of key and value vectors that attention reads back on every token, balloons with sequence length and quietly eats your VRAM. Google Research's TurboQuant attacks exactly that: it squeezes the KV cache to 2.5 to 4 bits with no calibration data and near-optimal error. The community already wired it into llama.cpp and vLLM, so you can try it this afternoon. The headline numbers, though, need an asterisk or three.

What TurboQuant actually is

First, the thing people keep getting wrong: TurboQuant quantizes the KV cache, not the model weights. It does not replace GPTQ, AWQ, or your GGUF Q4 weights. It stacks on top of them. Weight quantization shrinks the model on disk and in memory; TurboQuant shrinks the per-token cache that grows as you generate. Different problem, complementary fix.

It comes from a Google Research paper, "TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate" (Zandieh, Daliri, Hadian, Mirrokni), posted to arXiv in April 2025 and accepted at ICLR 2026. Google blogged about it in March 2026. Two properties make it interesting for builders: it is online (compresses vectors as they are written, no separate pass) and it is data-oblivious (no calibration set, no tuning run). You point it at a model and go.

How it works

The trick is a random rotation. Quantization hates outliers, because a few large coordinates force a coarse scale on everything else. TurboQuant multiplies each key or value vector by a random orthogonal matrix first, which spreads outlier energy evenly across dimensions and pushes the per-coordinate distribution toward a predictable shape. After that rotation, each coordinate is close to independent, so a per-coordinate Lloyd-Max quantizer (the analytically optimal scalar quantizer for a known distribution) does the rest. Because the distribution is known ahead of time, there are no per-block scales or zero-points to store, which is where the bit savings come from.

That is the MSE variant. There is a second variant tuned for attention. Attention does not care about reconstructing vectors, it cares about dot products. So TurboQuant runs the MSE quantizer at one bit lower, then applies a 1-bit Quantized Johnson-Lindenstrauss transform to the residual error. At read time, a special estimator combines both parts to recover an unbiased inner product. The paper argues the whole scheme lands within a small constant factor of the information-theoretic distortion floor.

The numbers, and the asterisks

Google's reported results are genuinely good. On Llama-3.1-8B-Instruct, long-context quality barely moves at 3.5 bits per channel.

Pair that with the memory math (roughly 6x smaller KV cache at 3-bit, and up to 8x faster attention at 4-bit on an H100) and it sounds like a free lunch. It is not, and the people who run inference for a living said so.

The vLLM team published the first large independent study in May 2026, testing Llama-3.3-70B, Qwen3-30B-A3B, and others on H100s. Their verdict: plain FP8 KV cache is still the best default. Here is the shape of it.

Read that bottom row twice. At aggressive 3-bit settings on a 256K context, long-context retrieval drops by about 30 percent and reasoning accuracy falls roughly 20 points. The compression is real, but so is the tax, and FP8 gets you 2x for free with no measurable loss. TurboQuant earns its keep when you are memory-bound and FP8's 2x is not enough, not when you are chasing raw throughput.

How to run it today

There is no official Google package. Every usable build is a community fork. Two are worth your time.

llama.cpp (local, Apple Silicon or a single GPU)

Community forks add tq3_0 (3-bit, about 4.9x KV compression) and tq4_0 (4-bit, about 3.8x) cache types. Build for your backend, then pass the cache type at runtime:

# Build (pick your backend)
cmake -B build -DGGML_METAL=ON && cmake --build build -j   # Apple Silicon
# cmake -B build -DGGML_CUDA=ON  && cmake --build build -j  # NVIDIA

# Run with a 4-bit TurboQuant KV cache
./build/bin/llama-cli -m model.gguf \
  --cache-type-k tq4_0 --cache-type-v tq4_0 \
  -c 32768 -p "Summarize this long document..."

vLLM (server, NVIDIA)

The community vLLM fork exposes TurboQuant as a kv-cache dtype. Build from source, then serve:

vllm serve /models/your-model \
  --tensor-parallel-size 4 \
  --attention-backend TRITON_ATTN \
  --kv-cache-dtype turboquant35 \
  --enable-turboquant

The under-10-minute move: take a model that currently runs out of memory at your target context, switch the KV cache to tq4_0 (or turboquant35 on vLLM), and see whether the longer context now fits. If it does and your eval holds, you win memory for almost nothing. If quality dips, step up the bit-width or fall back to FP8.

The caveats nobody puts on the slide

Three things temper the hype.

No official code. Google published a paper and a blog post, not a library. TechCrunch called it a lab breakthrough, and that is accurate. Anything labeled "Google's TurboQuant library" is community code, maintained by individuals, not Google.

The RaBitQ dispute. The core move, random rotation before quantization, is the same idea behind RaBitQ (Gao and Long, 2024). Critics on OpenReview and elsewhere argue the paper mischaracterized RaBitQ's method and guarantees, and that the speed comparison pitted TurboQuant on an A100 GPU against RaBitQ on a single CPU core with multithreading switched off. One author acknowledged multiprocessing was disabled. Treat the head-to-head speed claims against RaBitQ as unsettled.

The famous numbers are narrow. The 8x is attention-logit computation only, not end-to-end inference. The "zero accuracy loss" applies to the 3.5-bit setting on long-context benchmarks; the paper never ran MMLU, GSM8K, or HumanEval, and the vLLM study found real drops once you push to 3-bit or stress reasoning.

Who should use it, and who should wait

Try it if you run long contexts on memory-bound hardware: a consumer GPU, a Mac, an edge box that hits out-of-memory before it hits a speed wall. A 4-bit KV cache can be the difference between a 32K context fitting or not. Models like Gemma 4, Qwen3.5, and DeepSeek-V4 are all popular community targets, and MiniMax's long-context models showed up in the vLLM test set too.

Wait, or just use FP8, if you are on datacenter GPUs optimizing for throughput, or running aggressive low-bit configs on reasoning-heavy workloads. In those cases FP8 KV cache gives you 2x with no throughput penalty and no accuracy hit, and that is hard to beat.

Sources and further reading

• TurboQuant paper (arXiv 2504.19874)
• Google Research announcement
• vLLM: first comprehensive TurboQuant study
• llama.cpp community discussion and benchmarks
• The RaBitQ attribution dispute
• ICLR 2026 OpenReview thread

Tested on: not independently tested. The figures here are paper- or community-reported via the sources above, with the vLLM study and the RaBitQ dispute flagged as such.
Date checked: 2026-06-09