Cloudflare cuts Workers AI inference costs with FP8 and INT4 compression

Cloudflare said on August 3 it deployed three memory-optimization techniques on Workers AI, its serverless inference platform, to cut costs and expand context for the open-weight Kimi K2.6 and GLM 5.2 models.

The headline change is context length. Cloudflare said FP8 KV-cache quantization (storing the model’s key-value cache in 8-bit floating-point numbers) halves that cache’s size and doubles usable context on Moonshot’s Kimi K2.6 from 686,000 to 1.37 million tokens. The company said the technique also lifts peak throughput 41% at maximum concurrency and lowers cost per token about 30%, with no change in benchmark accuracy.

For Z.ai’s GLM 5.2, Cloudflare turned to INT4 weight compression, which stores model weights as 4-bit integers. That shrank the checkpoint 40%, from 705GB to 421GB, and cut per-GPU memory use to 52GB, the company said, while raising decode speed as much as 55% at low concurrency. Cloudflare put the resulting accuracy loss at under 0.8 points.

Cloudflare also added an integrity check on the shared KV cache to guard against data corruption, which it said adds less than 1% overhead to throughput and latency, per Cloudflare’s engineering blog.

All of the performance and accuracy figures are Cloudflare’s own and have not been independently verified. Quantization can degrade output quality in ways aggregate benchmarks miss, so the practical effect on production workloads will depend on the task.