KV Cache

Scaling KV Cache Beyond Memory with Mooncake SSD Offloading

Mooncake extends KV cache beyond expensive memory by pooling local NVMe SSDs into a distributed, persistent cache tier that preserves long-context reuse and reduces TTFT for agentic workloads.

Jul 15, 2026

How Much KV Cache Budget Do We Need for LLM Serving?

Estimate the KV Cache capacity for LLM inference workloads by analyzing hit rate and prefill speedup under different cache budgets, with the help of KV Cache Hit Rate Simulator.

Jun 26, 2026