How Cloudflare Fits Giant AI Models Into GPUs — and Keeps Them Fast
KV cache quantization, weight compression, and cache integrity checks let Cloudflare serve Kimi K2.6 and GLM 5.2 at scale with no accuracy loss.
The 60-second version
Cloudflare's Workers AI team published three techniques for running large MoE models at scale: FP8 KV cache quantization, INT4 weight compression, and KV cache integrity checking — all with no meaningful accuracy loss.
Key points
- FP8 KV cache doubles context capacity (686K → 1.37M tokens) and delivers 41% higher throughput at 64 concurrent requests on Kimi K2.6
- INT4 weight compression cuts GLM 5.2 from 705GB to 421GB, boosting decode speed up to 55% at low concurrency
- KV cache integrity checks cost under 1% on throughput and latency, preventing data cross-contamination at scale
- Disaggregated prefill/decode lets Cloudflare apply each optimization where it helps most: INT4 for decode, FP8 for prefill
Verdict. Real, measurable efficiency gains with no accuracy trade-off. The disaggregated architecture is the key enabler — it turns what would be compromises into choices.
The problemBig models, bigger memory demands
Running a large Mixture-of-Experts model on a GPU is a memory puzzle. The model weights take up space, and the KV cache — the scratchpad that stores every token's attention keys and values — takes up even more. For long-context models like Moonshot's Kimi K2.6 and Z.ai's GLM 5.2, the KV cache fills up GPU memory first, limiting how many requests can share a single GPU.
Cloudflare's Workers AI team published a deep dive on three techniques they layer on top of their existing disaggregated prefill-decode architecture to fit these models into memory and keep them fast. All experiments run on SGLang, with patches upstreamed to open source.
Technique 1KV cache quantization (FP8)
By default, the KV cache is stored in 16-bit BF16 precision. Cloudflare stores it in 8-bit FP8 (e4m3) instead, halving its size. On Kimi K2.6, that raises the context capacity from ~686,000 tokens to ~1.37 million — twice as much.
The trade-off? FP8 attention kernels have to convert values as they read them, so each individual request is a few percent slower. But the real win is capacity. At 64 concurrent requests, FP8 reaches 2,192 tok/s — 41% higher than BF16's peak of 1,558 — for roughly 30% less cost per token.
| Concurrent requests | BF16 KV (tok/s) |
|---|---|
| 1 | 137 |
| 8 | 731 |
| 16 | 1,106 |
| 32 | 1,558 |
| 64 | Out of memory |
Because Cloudflare runs disaggregated prefill and decode, they apply FP8 only where it helps: decode is memory-bound, so FP8 wins on capacity. Prefill is compute-bound, so they leave it in BF16 for higher throughput.
| Benchmark | BF16 KV |
|---|---|
| GSM8K | 94.24 |
| ARC-Easy | 89.06 |
| ARC-Challenge | 66.72 |
| MMLU | 89.11 |
| MMLU-Pro | 80.29 |
| Tool-call validity | 92.2% |
Across the evaluation suite, FP8 and BF16 caches are indistinguishable in accuracy.
Technique 2Weight compression (INT4)
For GLM 5.2, Cloudflare compresses model weights from 8-bit FP8 down to 4-bit INT4. The checkpoint shrinks from 705 GB to 421 GB (40% reduction), and per-GPU memory across an 8-way tensor-parallel deployment drops from ~88 GB to ~52 GB, leaving room for ~1.18 million tokens of KV cache.
Smaller weights make decode faster because generating each token means streaming the model's weights out of GPU memory. Less data, sooner tokens. At low concurrency, the gain is dramatic:
| Concurrent requests | GLM FP8 (tok/s) |
|---|---|
| 1 | 60 |
| 8 | 425 |
| 16 | 683 |
| 32 | 994 |
| 64 | 1,672 |
Prefill behaves differently — it's compute-bound, and INT4 weights need to be expanded back out before the model can multiply with them. So prefill is slower: ~8,660 tok/s in INT4 versus ~10,160 in FP8. But with disaggregated architecture, Cloudflare runs INT4 for decode and FP8 for prefill — the best of both worlds.
Accuracy stays within 0.8 points of the FP8 model across every benchmark.
Technique 3KV cache integrity protection
Both techniques above let many more requests share one GPU's memory. That efficiency is the whole point, but it also means hundreds of requests are reading and writing pages of the same physical KV cache. At Cloudflare's request volumes, even a one-in-a-billion mistake would show up regularly.
Their solution: every physical cache page gets a tag that changes when the page is reallocated. Before decode operations read from the cache, the server checks that the tags match. If anything doesn't match, the affected request is aborted rather than returning data from the wrong page.
| Concurrency | Throughput change |
|---|---|
| 1 | −0.53% |
| 2 | −0.38% |
| 4 | −0.79% |
| 8 | −0.43% |
The cost is under 1% on both throughput and tail latency. The check runs as a separate batch operation rather than being fused into the attention kernel, avoiding a race condition between GPU thread groups.
What's nextExpanding coverage
Cloudflare is expanding FP8 KV caches across more of the fleet, validating NVFP4 weights on Blackwell (NVIDIA's next-gen GPU architecture), and working toward making integrity checks the default everywhere at negligible cost.
These optimizations let Cloudflare support more customers at lower cost with no change in model accuracy — and they're already in production.
Primary sourcesCloudflare Blog