KV Cache | KVCache.AI

Scaling KV Cache Beyond Memory with Mooncake SSD Offloading

Wed, 15 Jul 2026 00:00:00 +0000

SSD Offloading Becomes Necessary, Practical and Profitable

The rise of agentic workloads fundamentally changes the role of KV cache. Coding assistants, multi-turn reasoning, and tool-using agents repeatedly reuse long context prefixes while appending only a small number of new tokens at each interaction. Tool invocations can easily span tens or even hundreds of rounds, pushing context lengths to hundreds of thousands of tokens. This creates both a tremendous opportunity and a serious challenge: when KV cache is effectively retained, cache hit rates can exceed 95%; but once a cache miss occurs, the system may have to recompute hundreds of thousands of tokens, incurring substantial prefill overhead.

However, the lifetimes of KV caches vary dramatically; some will be reused hours or even longer later. Retaining all KV caches in expensive HBM or DRAM is therefore both inefficient and economically unsustainable. To quantify this, we analyzed production traces from Qwen-Bailian. This dataset consists of a two-hour sampled and anonymized KV cache trace from a single Qwen model-serving cluster on Aliyun Bailian. For our analysis, we select two representative traces: reasoning-intensive chat (thinking) and code generation (coder).

As shown in the figure, KV block reuse exhibits pronounced temporal-locality polarization, while block lifetimes span orders of magnitude. In coding scenarios, 69.2% of reused blocks are re-accessed within ten minutes, whereas cold reuse after more than 30 minutes accounts for only 10.3%. In reasoning scenarios, this skew is even more extreme: 83.0% of reused blocks exhibit short-cycle reuse, while only 5.0% fall into the cold-reuse category. These distributions show that KV cache access does not follow a simple monotonic decay. Instead, it forms a bimodal pattern where hot clusters coexist with cold tails. As a result, LRU or fixed-TTL policies may prematurely evict cold yet still valuable KV blocks, leading to catastrophic cache misses and massive prefill recomputation later.

This naturally raises a question: can we use cheaper, more cost-effective SSDs to store these long-tail KV blocks? In our previous blog, How Much KV Cache Budget Do We Need for LLM Serving?, we answered this question through a quantitative analysis of agentic workloads. We showed that the marginal benefit of provisioning additional DRAM capacity quickly diminishes, making a tiered storage hierarchy increasingly attractive. More importantly, because the vast majority of KV cache hits can already be served by a relatively small DRAM cache, only a tiny fraction of requests need to access the SSD tier. In other words, SSDs can dramatically expand KV cache capacity while imposing very little pressure on the read path.

Even more appealingly, KV cache hit rate translates nonlinearly into prefill speedup: if the hit rate is $r$ , only $1-r$ of prefill computation needs to be recomputed, yielding an ideal speedup of $\frac{1}{1-r}$ . As agentic workloads often operate in a high-hit-rate regime, even a small improvement in long-tail KV cache hits can translate into a disproportionately large gain in prefill performance.

Mooncake Solution: A Distributed SSD Pool for KV Cache

Motivated by this observation, we built SSD offloading support directly into Mooncake Store. As shown in the figure below, Mooncake organizes local SSDs across serving nodes into a distributed KV cache pool, just as it already does for DRAM. DRAM remains the low-latency serving tier for hot data, while the larger, lower-cost SSD tier extends KV cache lifetime. A master coordinates the global metadata of all replicas across both tiers, allowing any compute node to transparently discover and reuse KV blocks stored on another node’s local SSD.

Data Flow

Mooncake keeps SSD access off the application’s critical write path and moves data asynchronously between the DRAM and SSD tiers. The complete flow consists of four operations: offloading KV blocks to SSD, loading them on demand, promoting frequently reused blocks back to DRAM, and recovering SSD metadata after restarts.

Offloading: SSD offloading is completely asynchronous and never blocks the application write path. Newly generated KV cache is first stored in DRAM and immediately becomes available for serving. Background heartbeat messages periodically synchronize with the master, which decides which objects should be offloaded based on the configured policy. The local node then batches the selected KV blocks and flushes them to its local SSD. Mooncake currently supports two offloading policies: eager offloading, which continuously flushes newly created KV cache in the background, and watermark-based eviction, which only starts offloading when DRAM utilization exceeds a configurable threshold.

Loading and Promotion: When a requested KV block is no longer available in DRAM, the requester first queries the master for the location of its SSD replica. The target node reads the data from its local SSD into a temporary DRAM buffer, after which the requester pulls the data directly through RDMA or TCP. To avoid repeatedly paying SSD access latency, the master continuously tracks access patterns of SSD-resident objects. Frequently accessed KV blocks are proactively promoted back into DRAM, allowing subsequent requests to be served entirely from the memory tier.

Metadata Recovery: SSD-resident KV blocks persist across storage-service and master restarts, but their locations must be restored to the master before the blocks can be reused. Mooncake delegates this recovery process to individual storage nodes. When a node starts, it scans its local on-disk metadata and re-registers every valid object with the master, making the recovered SSD replicas globally discoverable once again.

Optimizing the SSD Data Path

Simply adding an SSD tier is not enough. The storage stack itself must sustain high throughput under the highly concurrent access patterns of LLM serving. Mooncake therefore optimizes the entire SSD data path to minimize software overhead and keep modern NVMe devices fully utilized.

When MOONCAKE_OFFLOAD_USE_URING is enabled, Mooncake switches from POSIX I/O to an io_uring-based implementation. Instead of sharing a global ring, each thread owns a private io_uring instance, eliminating lock contention and allowing I/O to scale with concurrency.

This per-thread design also enables batching multiple read requests into a single submission, increasing queue depth and better utilizing NVMe parallelism. In addition, the staging buffer is registered as a fixed buffer with io_uring, avoiding repeated buffer registration on every I/O.

Within the io_uring path, O_DIRECT can optionally be enabled to bypass the page cache. Mooncake transparently handles unaligned requests with temporary aligned buffers, so users do not need to manage 4 KB alignment themselves.

Pluggable Storage Backends

A storage backend defines how objects are organized on disk, including their layout, indexing, allocation, and eviction strategy. Since different workloads prioritize different trade-offs, such as throughput and eviction granularity, Mooncake Store provides a pluggable backend interface with three built-in implementations:

BucketStorageBackend (default) groups objects into buckets before flushing them to disk. Each bucket consists of a data file and a metadata index, written once the bucket reaches a size or key threshold. Fixed-size slots ensure sequential reads and writes, delivering higher SSD throughput. Bucket-level FIFO and LRU eviction policies are currently supported.
OffsetAllocatorStorageBackend stores all objects in a single pre-allocated file. Like Mooncake’s DRAM storage, it uses an OffsetAllocator to manage space, enabling object-level allocation and eviction.
FilePerKeyBackend (demo) stores each object as an individual file under a two-level hash-sharded directory tree. While not designed for performance, it is useful for debugging and inspecting on-disk data.

Benchmark Results

We evaluate end-to-end performance on a single DGX node equipped with 8 × A100-SXM4-40GB GPUs and dual RDMA NICs. The evaluation compares four configurations:

GPU only: KV cache is stored only in GPU memory.
HiCache L1 + L2: KV cache is stored across both device and host memory.
HiCache L1 + L2 + Mooncake: KV cache is stored across device memory, host memory, and Mooncake DRAM.
HiCache L1 + L2 + Mooncake with SSD: KV cache is stored across device memory, host memory, and Mooncake DRAM plus SSD.

As shown in the figure above, SSD offloading significantly reduces TTFT while improving prefill throughput.

To understand where these gains come from, we break down TTFT and KV cache hit rate by conversation round, with the output length fixed to one token to isolate prefill latency. During the first six rounds, the memory pool is sufficient for both configurations, and they achieve nearly identical hit rates above 80%. The difference emerges once memory is exhausted. Starting from round 7, Mooncake without SSD must evict KV cache, causing the hit rate to collapse from 83% to 36% and TTFT to increase from 6 s to 16 s. With SSD offloading, evicted KV blocks remain available on NVMe, keeping the hit rate above 84% through round 8 and limiting TTFT to 9.4 s.

Ongoing Work

Mooncake’s SSD offloading support is only the first step toward a fully tiered KV cache architecture. We are actively exploring three major extensions:

GPU Direct Storage (GDS). Currently, SSD offloading still stages data through DRAM when moving KV cache between SSDs and GPU memory. GDS removes this intermediate hop, enabling KV cache to be transferred directly between NVMe SSDs and GPU HBM. This reduces both data movement overhead and end-to-end latency.

Distributed Storage Backends. While local NVMe pooling is easy to deploy and operate, many users already have mature distributed storage systems they prefer to reuse. To support both models, Mooncake is introducing a pluggable backend interface for distributed storage systems, with 3FS planned as the first integration.

NVMe-oF Integration. While the current SSD tier is built on local NVMe devices, we are also exploring NVMe over Fabrics (NVMe-oF) to extend the same abstraction across remote storage. By integrating NVMe-oF with Mooncake’s Transfer Engine, remote NVMe devices can be accessed with the same high-performance data path as local SSDs, enabling storage resources to be pooled across machines without sacrificing efficiency.

Acknowledgment

We would like to thank everyone in the community who has contributed to or supported this work. In particular, we are grateful to:

Approaching.AI: Jiahao Lu, Zuoyuan Zhang, Chizheng Fang, Ke Yang
Ant Group: Xinjie Zhu, Yongke Zhao, Yue Wan, Tingwei Huang
NVIDIA: Mahesh Bapatu
Alibaba Cloud: Xinpeng Zhao, Bo Gao, Xuchun Shang, Teng Ma

Mooncake project: https://github.com/kvcache-ai/Mooncake
Documentation: https://kvcache-ai.github.io/Mooncake/design/ssd-offload.html
Usage: https://kvcache-ai.github.io/Mooncake/deployment/ssd-offload.html

How Much KV Cache Budget Do We Need for LLM Serving?

Fri, 26 Jun 2026 00:00:00 +0000

In LLM inference clusters, the storage budget for the KV Cache directly affects the KV Cache hit rate and prefill throughput. If the cache is configured too small, reusable KV entries will be evicted too early. If it is configured too large, valuable storage resources will be wasted. Therefore, the key question in KV Cache capacity planning is: given a specific workload and model configuration, how much cache space should be allocated to achieve a sufficiently high hit rate and reasonable marginal returns?

To address this, we have released the KVCache Hit Rate Simulator, a tool for simulating KV Cache hit rates. Based on request traces and model parameters, it can calculate the KV Cache hit rate under different KV Cache capacities and further estimate the prefill performance improvement brought by KV Cache reuse. In this article, we will:

Introduce how to use this tool to properly configure and optimize KV Cache storage capacity in online inference clusters.
Share some observations and insights from our analysis.

Tool: https://kvcache.ai/tools/kv-cache-hit-rate-simulator/

Step 1: Parameter Configuration

The KV Cache hit rate is highly dependent on reuse patterns across requests. Therefore, the first step is to prepare a trace that can represent your online workload. You can use your own local trace or directly select one of the built-in traces provided by the tool.

We have prepared several types of traces in advance, such as Tool Agent, Coding Agent, and RAG. We recommend choosing the trace type that most closely matches your online business scenario.

For local traces, data security is usually the primary concern. KVCache Hit Rate Simulator does not upload user data. Trace parsing and hit-rate simulation are both performed locally in the browser. Since the tool itself is open source, users can also deploy it directly in their own environment to further satisfy internal data security and compliance requirements.

In addition, different models have different KV Cache sizes. Therefore, when running the simulation, you also need to select the corresponding model and precision.

Step 2: Data Analysis

After the parameters are selected, the tool automatically calculates the relevant metrics. The first thing to examine is the figure showing how the KV Cache hit rate changes as the KV Cache capacity increases.

First, look at the upper bound of the KV Cache hit rate. The horizontal line of “ceiling” in the figure represents the theoretically maximum hit rate that the workload can achieve under the assumption of unlimited KV Cache capacity. In other words, it is the theoretical upper bound of the hit rate.

Next, focus on the actual hit rates under different cache budgets. Under the same cache capacity, different cache eviction algorithms may have a significant impact on the hit rate. Since most inference systems currently use the LRU algorithm, we can mainly focus on the line corresponding to LRU.

Note: For LRU algorithms based on prefix reuse, eviction naturally starts from the leaf nodes of the radix tree.

In general, we want the hit rate to be as close as possible to the theoretical upper bound. However, when the hit rate approaches this upper bound, the marginal benefit of further increasing KV Cache capacity drops rapidly. Therefore, we need to make a trade-off between storage capacity and compute efficiency.

When making this trade-off, the hit-rate curve alone can be misleading. The same absolute increase in hit rate can translate into very different performance gains depending on the hit-rate range. This is particularly important for today’s agentic workloads, which often have very high KV Cache hit-rate ceilings, typically above 90%.

In the high-hit-rate range, each additional percentage point becomes increasingly valuable. For example, increasing the hit rate from 50% to 55% yields roughly a 10% prefill speedup, while increasing it from 90% to 95% can nearly double prefill performance.

Therefore, when deciding whether to further increase KV Cache capacity, we should look beyond the percentage-point improvement in hit rate. The key question is whether the resulting prefill speedup is still growing meaningfully. This is where the second figure becomes useful: it shows how the ideal prefill speedup changes as the KV Cache capacity increases.

The prefill speedup is directly calculated from the KV Cache hit rate. Intuitively, if the hit rate is $r$ , then only $1-r$ of the prefill computation needs to be recomputed. Therefore, the ideal prefill speedup is: $\frac{1}{1-r}$ .

That is, in the ideal case, prefill can be accelerated to $\frac{1}{1-r}$ times the speed of full recomputation.

In the figure above, the trace is Weka Claude Code and the model is GLM-5.1. When the KV Cache budget is 512 GB, 1 TB, 2 TB, and 4 TB, the ideal speedup is 6.3x, 7.9x, 8.7x, and 9.0x, respectively. Therefore, it is reasonable to consider configuring the KV Cache capacity at 2 TB. At this point, the system has already achieved performance gains close to the maximum, while avoiding the low marginal returns caused by further capacity expansion. Overall, this offers the best cost-effectiveness.

Some Insights

The cost-effectiveness of KV Cache storage capacity, namely the prefill speedup contributed by each GB of storage, typically decreases gradually as capacity increases. Once the marginal benefit of additional storage falls below a certain level, it may be worth introducing cheaper storage media and building a tiered cache.

Using the figure above as an example, when the KV Cache budget is 512 GB and 2 TB, the ideal speedup is 6.3x and 8.7x, respectively, corresponding to hit rates of 84.2% and 88.5%. In this case, a two-tier cache strategy with 512 GB of DRAM and 2 TB of SSD can be considered. Assuming that each tier independently uses the LRU algorithm, the two tiers follow a write-through policy, and Get operations first check for hits in DRAM, then the 512 GB DRAM tier can provide a 6.3x performance improvement, while the 2 TB SSD tier contributes the remaining 2.4x speedup.

More interestingly, the hit rates above show that among the 88.5% of KV cache hit, about 95.1% can be satisfied directly by DRAM, while only about 4.9% need to fall back to the SSD tier. In other words, although SSD access is slower, the SSD tier carries very little read-path pressure. This makes SSD an attractive option for cost-effectively expanding KV Cache capacity in such scenarios.

It is also important to note that the analysis above assumes each KV Cache entry is stored only once globally. If duplicate copies exist across the system, the required KV Cache capacity should be increased accordingly.

This is also where our open-source project Mooncake becomes relevant. Mooncake is a distributed caching system designed around KV Cache. With Mooncake Store, each KV Cache entry only needs to be stored once globally, which can significantly reduce redundancy. As a result, the capacity analysis above, which assumes a single global copy of each KV Cache entry, better reflects what can be achieved in a real deployment.

Mooncake Store also supports two-tier caching with DRAM and SSD. In addition, the local SSD on each node can be shared across the entire inference cluster, allowing KV Cache capacity to be expanded at a lower cost while enabling large-scale KV Cache reuse for high-hit-rate agentic workloads.

Use the simulator online: https://kvcache.ai/tools/kv-cache-hit-rate-simulator/

Deploy the simulator locally: https://github.com/kvcache-ai/kvcache-blog

Try Mooncake: https://github.com/kvcache-ai/Mooncake