OasisKV: Scaling In-Decode KV Cache Beyond HBM with Lookahead Sparse Prefetching

Review date: 2026-08-12 Author: Zhongzhu Zhou Paper reviewed: OasisKV: Scaling In-Decode KV Cache Beyond HBM with Lookahead Sparse Prefetching Paper authors: Can Xiao, Sukmin Cho, Junbong We, Zhixiong Niu, Jianyi Cheng, Yiren Zhao, Youngjin Kwon, Yongqiang Xiong, Rui Ma, Junyi Liu (Imperial College London, KAIST, Microsoft Research, University of Edinburgh) arXiv: 2608.08097 Venue/Status: Preprint (cs.DC), August 2026

1. The problem: HBM capacity, not compute, caps how many requests you can serve

If you have spent any time reading LLM-serving papers, you have seen some version of this story before: the KV cache grows linearly with context length and batch size, GPU high-bandwidth memory (HBM) is finite and expensive, and once the KV cache fills that memory, you cannot admit more requests — no matter how much idle compute you have sitting around. OasisKV is a systems paper about exactly this bottleneck, but it earns its place in the “efficient ML” canon by doing something conceptually clean: it notices that speculative decoding, a technique originally invented to make LLM inference faster, also happens to produce, completely for free, the single piece of information a KV-cache prefetching system needs most — an accurate look one step into the future.

Before getting into how OasisKV exploits this, it’s worth spending real time on why this problem is hard in the first place, because the paper’s design only makes sense once you understand the three separate constraints it’s threading simultaneously: prediction accuracy, decode-step timing, and network bandwidth.

Prerequisites: what you need to know before diving in

Autoregressive decoding and the KV cache. An LLM answers a prompt by first running a prefill pass over the whole prompt (computing hidden states for every prompt token in parallel), and then entering a decode loop that produces one new token per forward pass. Each decode step must let every new query attend to the keys and values of every previous token — so the model needs to keep all of those keys and values (“KV cache”) resident in memory for the entire lifetime of the request. For a transformer with $N_\text{layer}$ layers, $N_\text{kv}$ KV heads, head dimension $d_h$, and $s$ bytes per KV element, a batch of $B$ requests at context length $L$ needs

$$M_\text{KV}(B, L) = B \cdot L \cdot N_\text{layer} \cdot 2 N_\text{kv} d_h \cdot s$$

bytes of KV memory (the factor of 2 is for keys and values). This is Eq. (1) in the paper, and it is the single equation that explains almost everything else: KV memory grows linearly in both context length and batch size, with no way around it for a dense (non-sparse) model. Given a fixed HBM KV-cache budget $M_\text{HBM}$, the maximum batch size the GPU can serve is

$$B_\text{max} = \left\lfloor \frac{M_\text{HBM}}{L \cdot N_\text{layer} \cdot 2 N_\text{kv} d_h \cdot s} \right\rfloor.$$

Plug in real numbers and the constraint bites hard: for a 32B-class GQA model at BF16 with a representative 32.7K-token context (the paper cites production coding-agent traces averaging this length), one request needs about 8.6 GB of KV cache. Reserve the entire 80 GB of an H100’s HBM purely for KV cache — an unrealistic best case, since weights, activations, and CUDA graph workspace also need room — and you can still only fit nine concurrent requests. This is “the memory wall”: long-context, agentic workloads are pushing LLM serving from a compute-bound regime into a capacity-bound one.

Sparse attention and why it doesn’t remove the capacity problem. Attention scores are heavily skewed in practice — for a given query, only a small subset of historical tokens meaningfully contribute to the output. Sparse-attention methods (Quest, MInference, NSA, and others) exploit this by selecting a top-$K$ subset of historical tokens/blocks to attend to at each step, instead of scanning the whole KV history. This genuinely reduces computation — attention over $K$ tokens instead of $L$ tokens is cheaper — but by itself does nothing for capacity: a naive implementation still keeps the entire KV cache resident in GPU HBM, because you don’t know in advance which subset will be needed next. Sparse attention alone solves the latency half of the problem, not the memory half.

KV retrieval: solving capacity, at the cost of latency. The natural next idea is to keep only the active subset of KV cache in GPU HBM and store the rest in cheaper, larger host memory (CPU DRAM) or even remote memory, fetching the needed blocks on demand each step. Systems like ArkVale, RetroInfer, and ShadowKV do this. It solves the capacity problem — GPU memory now only holds a bounded working set — but it puts the fetch itself on the decode critical path: the attention computation for the current step cannot proceed until the needed KV blocks have finished crossing PCIe (or the network). Fig. 2 in the paper makes this concrete with a roofline-based latency breakdown: for Qwen3-32B with context length 32K on a single H100, fetching just 10% of the attended KV cache from CPU DRAM inflates per-token decode latency (TPOT) by 14–78%, with the overhead growing with batch size and active KV-cache size. Once transfer time exceeds the compute time of the decode step itself, simply overlapping fetch with compute stops being sufficient — you’re now bandwidth-bound, not compute-bound.

KV prefetching: solving both, if you can predict the future accurately enough. The obvious fix for “fetch is on the critical path” is to predict which blocks the next step will need, and start fetching them before they’re required, so the transfer happens in the background while the current step’s compute is still running. This is KV prefetching, and it’s exactly where OasisKV positions itself. But prefetching only helps if the prediction is (a) accurate — a miss means either dropping context (hurting accuracy) or falling back to an on-demand fetch (right back on the critical path) — and (b) cheap to produce, ideally without training a dedicated predictor model, and (c) fast enough that predicting, selecting, and transferring all complete within the tiny window one decode step affords.

Speculative decoding, briefly. Speculative decoding (Leviathan et al. 2023) speeds up decoding by having a cheap “draft” model or module propose several candidate future tokens, then having the expensive target model verify all of them in one parallel forward pass — since verifying $k$ tokens costs barely more than verifying 1 in the memory-bandwidth-bound decode regime, you get $k$ tokens’ worth of progress for close to the price of one. Modern draft mechanisms like EAGLE-3 use lightweight multi-token-prediction (MTP) heads attached to the target model itself, avoiding the need for a fully separate model.

The paper’s central observation is that these draft tokens — which many production serving stacks are already computing for speculative decoding — are themselves an accurate, free, one-step-ahead signal for exactly the prediction problem KV prefetching needs to solve. That’s the idea OasisKV is built around.

2. Three challenges that make “just use draft tokens” harder than it sounds

Fig. 3 in the paper lays out four decode pipeline variants side by side — dense attention, sparse attention, sparse with KV retrieval, sparse with KV prefetch — and the pattern is a two-by-two matrix: dense attention has neither latency nor capacity fixed, sparse attention fixes latency (of the attention computation) but not capacity, KV retrieval fixes capacity but not latency (because fetch sits on the critical path), and only KV prefetch fixes both simultaneously, provided the prediction and pipeline machinery actually works.

Turning “use draft tokens as the lookahead signal” into a working, production-grade system raises three genuinely separate engineering problems, and the paper is refreshingly explicit about measuring each one before proposing a fix.

Challenge 1 — accurate lookahead at low overhead. A prediction is worthless if it’s wrong often, and worthless in a different way if it’s expensive to compute (e.g., requiring a second full forward pass, or a full scan of the context). Existing training-free predictors typically reuse the current step’s query as a proxy for the next step’s query — Fig. 4 in the paper shows this proxy recovers the true top-20 blocks with only 83.9% average accuracy, and the accuracy varies substantially across layers. A system built on this proxy either misses important blocks regularly or must widen its retrieval budget defensively to compensate, eating into the capacity gains sparsity was supposed to deliver.

Challenge 2 — fitting prediction, selection, and transfer inside one decode step. Even a perfect prediction is useless if you can’t act on it fast enough. Production serving engines like vLLM run heavily optimized decode kernels, which means the “overlap window” — the time available to hide a prefetch behind foreground compute — is short. The paper derives a hard per-step byte budget:

$$C_\text{token} = B_\text{link} \cdot T_\text{decode}$$

where $B_\text{link}$ is the effective inter-tier bandwidth and $T_\text{decode}$ is the compute time available before the prefetched blocks are needed (this is Eq. 3 in the paper). Concretely, for Qwen3-8B on a single H100 with sparse attention and a 2K active context, one decode step takes about 17 ms, and a typical $\approx$64 GB/s PCIe link can move roughly 118 tokens’ worth of newly-active KV per request per step before the transfer stops being hideable and starts stalling decode. This is a small number — smaller than most naive prefetching designs assume — and it directly trades off against accuracy: fetch too conservatively (a small working set) and you protect the bandwidth budget but risk missing context; keep a larger resident set and you lower future traffic but burn the very HBM capacity the whole design exists to conserve.

Challenge 3 — staging KV cache across the network without blowing up host memory. Under prefill-decode (PD) disaggregation — a common production pattern where dedicated “prefill” GPUs handle the compute-heavy prompt-processing phase and separate “decode” GPUs handle the memory-bound generation phase — the decode node must somehow obtain each request’s KV cache from wherever it was computed. The naive approach transfers and stages the entire KV cache into decode-node host DRAM before decoding starts. This creates two separate problems: it puts a large, capacity-scaling transfer directly on the admission critical path (inflating time-to-first-token, TTFT, especially badly when a high fraction of the prompt is already cache-hit and only a small uncached suffix actually needs prefill compute — yet the entire cached KV still has to cross the network), and it makes decode-node DRAM capacity — not GPU HBM — the new bottleneck on batch size. The paper’s worked example: a Qwen3-235B-A22B model on an 8-GPU decode node with 1 TB of CPU DRAM, at 100K average context, has a batch-size ceiling of roughly 26 requests from DRAM staging alone, before accounting for runtime buffers — even though sparse attention would let GPU HBM support a much larger batch.

Figure 1 (paper Fig.1): A roofline view of the design space. The x-axis is “Token-KV intensity” — tokens generated per decode pass per byte of KV cache moved over HBM or off-GPU links — and the y-axis is decode throughput. Dense vLLM (black dot) sits well below the dense ceiling. Sparse attention (teal triangle) moves right along the roofline by raising token-KV intensity, but is still capped by the dense ceiling because it doesn’t shrink HBM residency. Existing KV prefetch and KV retrieval methods (red x, orange diamond) are pinned near the CPU-GPU IO bandwidth roof — far from either ceiling. OasisKV (gold star) is the only point that reaches the sparse ceiling, by simultaneously raising token-KV intensity and breaking the HBM-residency requirement — the green arrow traces this joint move.

These three requirements — accurate training-free lookahead, a pipeline that respects the per-step byte budget, and sparse network staging under disaggregation — define exactly the three technical sections of OasisKV’s design, covered next.

3. Look-ahead attention: turning a draft token into a free top-K predictor

Figure 2 (paper Fig.5): OasisKV’s compute plane (GPU) runs a foreground forward pass alongside a background prefetch pipeline (Top-K prediction, Select KV, KV Transfer). The memory plane spans GPU sparse/compressed/draft KV pools, CPU local KV pool, and — for disaggregated serving — a remote KV pool. The control plane’s pool manager and scheduler coordinate block-table bookkeeping and per-step request dispatch across all three memory tiers.

3.1 The core mechanism: propagate the draft token through the current sparse working set

Here is the key design insight, and it’s worth being precise about exactly what problem it solves. Naively, you might think: “just run the draft token through the model and see what top-K blocks it attends to.” But this has a chicken-and-egg problem. The draft token at layer $l$ needs some KV working set to attend to, and if the GPU only holds a bounded, sparse working set (which is the entire point — that’s what makes this scalable), then the draft token’s query is itself computed using an incomplete view of history. Two specific failure modes follow: (1) the draft query’s hidden state, computed through preceding layers using the incomplete working set, may be distorted by the missing context; and (2) even if the draft query itself is fine, the resident working set contains zero information about the blocks that are not resident — so there’s no way to rank them for potential admission.

OasisKV’s fix leans on an empirical regularity that prior KV-retrieval work has also observed: block importance has strong temporal locality across adjacent decoding steps. In other words, the normal token’s currently-resident sparse working set is very likely to already contain most of what the next step’s top-K set will need. So instead of trying to solve the chicken-and-egg problem by giving the draft token a privileged view of the full context, OasisKV does something simpler: propagate the draft token through the same resident sparse KV that the normal token uses. Fig. 6 in the paper validates that this works surprisingly well — the top-K set predicted by this propagated draft query agrees with the true next-token query’s top-K set at 98.2% or higher in every one of Qwen3-8B’s 36 layers, averaging 98.74%.

Figure 3 (paper Fig.7): Left — the full KV cache lives in CPU memory and is compressed two ways for the GPU: min/max pooling produces small per-block “compressed key” summaries, while block-wise sparsification produces the actual resident sparse KV working set. Right — at each layer, the attention kernel stacks the normal query $q_l^t$ and the draft query $\tilde q_l^{t+1}$ and runs them together over the same resident sparse KV, producing both outputs in one kernel call. The draft query is then reused a second time: it scans the compressed-key summaries (not the actual sparse KV) to predict which blocks the next step will need, and those predicted blocks are prefetched from the CPU-resident full KV cache.

3.2 Compressed-key summaries: ranking blocks without paying for the full KV cache

A subtlety in the mechanism above: to rank non-resident blocks (i.e., the ones not currently in the sparse GPU working set), the draft query needs some signal about what’s in those blocks, without actually paying the cost of reading the full keys — that would defeat the purpose of sparsity. OasisKV’s answer, following the Quest sparse-attention design, is to maintain a per-block compressed-key summary: for each logical block, store just the coordinate-wise minimum and maximum of the keys inside that block (Fig. 7, left). Because a summary is only two vectors per block, rather than $B$ full key vectors (where $B$ is the block size, e.g. 16), this signal costs a small, fixed fraction of the full KV cache — the paper’s default configuration reserves two summary rows per block, occupying 1/16 of full KV size — and can be scanned by every KV head independently and cheaply to produce a global top-K ranking over the entire context, all without restoring a single actual key.

The combined foreground cost is therefore small: the QKV projection produces the normal and draft query together (one extra projection, not a second full forward pass), the attention kernel processes both stacked queries over one resident working set in a single call (no extra attention computation beyond doubling the query rows, which is nearly free in the memory-bound decode regime), and the summary scan for top-K prediction reads only two vectors per block rather than restoring keys. This is why the paper can claim “low-overhead foreground” — the added foreground work is genuinely marginal, not a second parallel model execution.

3.3 The fully asynchronous background pipeline

Once the draft query has produced a top-K prediction for the next step, three more stages need to happen before that prediction becomes useful: KV selection (compare the predicted set against the currently resident set, identify which blocks are missing, and build a transfer plan), and KV transfer (actually move the missing blocks from CPU to GPU). For a model with $N_\text{layer}$ layers, the pipeline needs to sustain one full prediction-selection-transfer cycle roughly every $\Delta_\text{prefetch} = T_\text{step} / N_\text{layer}$ — otherwise the backlog of pending prefetch tasks grows across decoding steps and the whole point of “one step ahead” prefetching collapses.

The key structural fact OasisKV exploits is that the three stages, across different layers, are independent — because each layer maintains its own compressed keys, its own resident mapping, and its own KV storage. This means top-K prediction for layer $l{+}1$, KV selection for layer $l$, and KV transfer for layer $l{-}1$ can all run concurrently, on different background CUDA streams, while the foreground (attention + FFN) runs on the default stream.

Figure 4 (paper Fig.8): The asynchronous prefetch pipeline traced across two decoding steps. Red arrows follow one layer’s dependency chain: the draft query at step $t$ drives top-K prediction, which drives KV selection, which drives KV transfer — all completing before that same layer’s attention runs at step $t+1$. Different layers’ stages (prediction for layer $l+1$, selection for layer $l$, transfer for layer $l-1$) execute concurrently on separate background CUDA streams.

Algorithm 1 — Layer-local asynchronous look-ahead prefetch (reconstructed from §4.2.2/4.2.3 description).

Input: layer index l, decode step t
1  At layer l of step t:
2      (q_normal, q_draft, k, v) <- QKVProjection(hidden_state_l)     # one shared projection
3      out_normal, out_draft <- AttentionKernel(stack(q_normal, q_draft), ResidentSparseKV[l])
4      submit_async(TopKPrediction, worker=l, input=q_draft, keys=CompressedKeySummary[l])
5  # Background worker for TopKPrediction (layer l), runs on stream A:
6      topK_pred[l] <- ScanCompressedKeys(q_draft, CompressedKeySummary[l])   # ranks ALL logical blocks
7      signal_event(topK_ready[l])
8      submit_async(KVSelection, worker=l, wait_on=topK_ready[l])
9  # Background worker for KVSelection (layer l), runs on stream B:
10     resident[l] <- CurrentResidentBlockSet(l)
11     missing[l] <- topK_pred[l] \ resident[l]        # predicted but not resident
12     evict_candidates[l] <- resident[l] \ topK_pred[l], sorted by LRU (last-selected step)
13     transfer_plan[l] <- CappedPairing(missing[l], evict_candidates[l], cap=C)  # see Algorithm 2
14     signal_event(selection_ready[l])
15     submit_async(KVTransfer, worker=l, wait_on=selection_ready[l])
16 # Background worker for KVTransfer (layer l), runs on stream C:
17     for (src_cpu_block, dst_gpu_page) in transfer_plan[l]:
18         UVA_gather_copy(src_cpu_block -> dst_gpu_page)     # PCIe transfer
19     UpdateHeadWiseMapping(l, transfer_plan[l])
20     signal_event(transfer_done[l, t])
21 # Before running layer l's attention at step t+1:
22     wait_on(transfer_done[l, t])       # ONLY this layer's transfer, not all layers
23     proceed with foreground attention for layer l, step t+1

The critical implementation detail in lines 21-22 is what the paper calls layer-local synchronization: before running layer $l$‘s attention at step $t{+}1$, the foreground only waits for that specific layer’s KV transfer issued at step $t$ — not for every layer’s transfer to complete at the start of the step. This lets early layers begin their step-$(t{+}1)$ computation as soon as their own blocks are ready, while background workers are still busy preparing later layers’ transfers. The pipeline’s steady-state throughput is governed by whichever of the three stages is slowest:

$$T_\text{pipe} = \max\{T_\text{pred}^l, T_\text{select}^l, T_\text{transfer}^l\} \le \Delta_\text{prefetch} \tag{4}$$

And as the motivation analysis already established, KV transfer over PCIe is almost always the bottleneck stage — which is exactly why the next design piece (delta selection and capped eviction) targets transfer volume specifically, not prediction or selection cost.

3.4 Delta selection and capped eviction: bounding transfer volume without an accuracy cliff

If every predicted top-K set were transferred wholesale, per-step PCIe traffic could spike arbitrarily whenever the selection drifts sharply between steps — exactly the failure mode Challenge 2 warned about. OasisKV’s fix is a two-part policy applied at the KV-selection stage:

1. Delta selection. First intersect the predicted set with the currently resident set. Blocks that are already resident and already predicted generate zero PCIe traffic — they just stay where they are. Only the set-difference (predicted-but-not-resident) needs to be fetched at all.
2. Capped eviction. Among the resident blocks that are not in the new predicted set, rank them by recency of last selection (least-recently-selected first) and treat these as eviction candidates. Pair each admitted (missing) block with one eviction candidate, and cap the number of admitted pairs per KV head per step at a fixed budget $C$ — regardless of how large the raw predicted-missing set is.

Algorithm 2 — Capped eviction pairing (per KV head, per layer, per step).

Input: predicted top-K set P, resident set R, cap C
1  intersection <- P ∩ R                     # stays resident, zero transfer
2  missing <- P \ R                          # needs to be fetched
3  eviction_pool <- R \ P                     # candidates to evict, sorted by LRU
4  n_admit <- min(|missing|, C)               # HARD CAP regardless of |missing|
5  admitted <- missing[0 : n_admit]           # e.g., by prediction rank
6  evicted  <- eviction_pool[0 : n_admit]     # least-recently-selected first
7  transfer_plan <- zip(admitted, evicted)    # each pair = one CPU-source/GPU-dest copy
8  # Blocks in missing beyond n_admit are simply NOT fetched this step —
9  #   served implicitly by whatever remains resident; re-attempted next step
10 return transfer_plan

The resulting per-layer transfer is bounded at $C$ blocks per KV head, or $C \cdot H$ block-head entries across $H$ heads, regardless of how many positions changed in the predicted top-K set. This is the mechanism’s central trade-off, and the paper quantifies it directly (§5.5, Table 2): sweeping the fetch cap from a ratio of 0.01 to “fetch everything,” per-step traffic on Qwen3-8B grows from 0.30 GB to 5.05 GB and decode throughput collapses from 2,178 to 824 tok/s — a 2.6x drop — while accuracy rises only modestly (74.90 to a peak of 77.40 avg@32 on AIME24). Once the PCIe link saturates (around a fetch ratio of 0.10, where effective bandwidth plateaus at 30-34 GB/s), moving more bytes no longer buys any accuracy either — it purely stalls the step. The paper’s chosen default operating point, a fetch ratio of 0.05, holds accuracy within 0.1 point of dense full attention while achieving 2.5x the throughput of the “fetch everything” extreme.

Why this design, and what’s the alternative? The obvious alternative would be an adaptive cap that grows or shrinks based on how much the selection has drifted, rather than a fixed constant. The paper doesn’t explore this; a fixed cap is simple to reason about and bounds worst-case behavior deterministically, but it means the system can’t distinguish between “selection is stable, no need to fetch much” and “selection just drifted a lot due to a genuine context shift, and we’re now silently serving several steps of stale/wrong context until the capped fetches catch up.” This is a real boundary case: a fixed cap protects bandwidth uniformly, but doesn’t protect accuracy uniformly across different workload dynamics.

3.5 Head-wise KV mapping: fitting sparse decoding into PagedAttention’s block model

PagedAttention (the memory manager underlying vLLM) represents each request’s KV cache as a growing sequence of logical blocks, each mapped to a physical GPU page, with every page storing the same contiguous token range for every KV head. This works cleanly for dense decoding, where the context only ever grows. It breaks for sparse decoding for two separate reasons: (1) a resident GPU block’s content must be able to change — which historical token range it represents — as the top-K selection shifts across decoding steps, rather than monotonically growing; and (2) different KV heads select different blocks, so the assumption that all heads within one physical page share the same logical identity no longer holds.

Figure 5 (paper Fig.9): The GPU and CPU each keep an unmodified physical KV pool and page table (top and bottom). A new intermediate “Head-wise Mapping” table (middle) records, per KV head, which CPU logical block currently backs each GPU logical block — e.g., GPU logical block 0’s H0 head currently mirrors CPU logical block 3, while its H1 head mirrors CPU logical block 1. Updating the sparse selection only rewrites entries in this middle table plus the corresponding GPU page contents; the original page tables above and below are never touched.

OasisKV’s fix adds an intermediate head-wise logical-to-logical mapping layer above the existing (unchanged) GPU and CPU page tables. For each KV head independently, this mapping records which CPU logical block is currently mirrored into which GPU logical block. Updating the top-K selection then only requires modifying the affected head-wise mapping entries and copying the corresponding GPU contents — the original page tables are never touched. This has a nice side effect for multi-GPU deployments under tensor parallelism: because each GPU only owns a subset of KV heads, each GPU can update its local head-wise mappings completely independently, with no cross-GPU synchronization required.

Why this design, and what’s the alternative? The obvious alternative is to replace PagedAttention’s page tables outright with a sparse-native memory manager, or maintain an entirely separate memory pool for sparse requests. Both would complicate the runtime considerably — replacing page tables risks breaking compatibility with everything else vLLM already does with them (prefix caching, CUDA-graph capture, etc.), and a separate pool would need its own admission/eviction logic and would prevent dense and sparse requests from being batched together in one call. The head-wise mapping layer is a comparatively surgical fix: it changes only what’s needed (per-head block identity) while leaving the surrounding infrastructure alone — at the cost of one extra indirection on every KV access.

4. Remote partial fetching: extending the same signal across the network

Under PD disaggregation, the naive approach — stage every request’s complete KV cache into decode-node host DRAM before decode starts — has two costs, as established in Section 2: it inflates TTFT (because the full transfer sits on the admission critical path) and it makes decode-node DRAM capacity the binding constraint on batch size, defeating the point of sparsity. OasisKV’s fix, remote partial fetching (RPF), splits the transfer into two pieces that mirror the lookahead-prefetch idea at the network level.

4.1 Partial transfer at admission

Instead of transferring the full KV cache, OasisKV transfers only three things at handoff: (1) the KV blocks selected for the first decoding step, (2) the compressed-key cache, and (3) the draft token’s KV state. There’s a subtlety in how the first-step selection is even obtained: selecting which blocks the first decode step needs requires the query of the first generated token — but prefill alone only produces hidden states for prompt tokens, not a decode query. So the prefill node runs one additional decoding step purely to obtain this query, uses it to compute the top-K set per KV head, and unions these per-head sets at block granularity (a block is included if any head selects it) before sending the union’s data and indices across the network.

Crucially, the prefill GPU is released as soon as the full KV cache is staged in its own local host DRAM — it does not wait for the (much smaller) partial transfer to the decode node to complete. The decode worker, on handoff, transfers compressed keys and draft KV state GPU-to-GPU directly, and the selected first-step KV blocks host-DRAM-to-host-DRAM, then uses the transmitted block indices to populate its own initial sparse working set.

4.2 Network prefetch during decode: treating drift as a cache miss, not an approximation

As decoding proceeds, the top-K selection naturally drifts — a block chosen at some later step may simply not be among the (much smaller) partial set that crossed the network at admission. OasisKV’s design choice here is philosophically important: rather than approximating (silently attending to whatever happens to be resident and accepting the accuracy loss), it treats a selection drift as a genuine cache miss and fetches the specific missing block from the prefill node’s staging pool — using the same background-pipeline machinery as local PCIe prefetching (§4.2.2), just redirected to a remote read instead of a local UVA copy. Because this selection executes one decoding step ahead of when the block is actually used, the remote fetch overlaps with foreground attention exactly as the local case does. Misses across different requests in the same batch are aggregated per layer and transferred together, amortizing per-transfer overhead.

The key invariant this design achieves: each block crosses the network at most once, ever — once fetched due to a miss, it becomes resident (subject to the same capped-eviction policy as local prefetching) and never needs to be re-fetched — and blocks that are never selected are never transferred at all. This has an important consequence for how network traffic is distributed over time: rather than one large burst at admission followed by pure local compute, transfer is spread across the decoding steps, avoiding a single administrative bottleneck at handoff.

Quantifying the savings (§5.5.2, Fig. 14). At admission, RPF transfers a predicted union averaging 0.52 GiB and 0.46 GiB per request at 24K and 32K context, respectively — versus 3.37 GiB and 4.50 GiB for full transfer, a 6.5x and 9.7x reduction. Because this union is sized by the fixed top-K budget rather than context length, it doesn’t grow with longer contexts the way full transfer does. Decode-time misses (‘drift’) add back 2.03-2.53 GiB over a 2,048-token generation, bringing the total end-to-end savings down to a more modest 1.33-1.50x versus full transfer — but the same per-step fetch cap used for local prefetching (§3.4) bounds this drift too: at a fetch ratio of 0.05, drift falls from 2.53 to 1.61 GiB, restoring the total savings to 2.2x. RPF also smooths network utilization over time — full transfer leaves the link idle 60% of the time punctuated by 5.3 GB/s bursts at handoff, while RPF with a fetch cap spreads traffic to a steadier 0.89-1.02 GB/s median with lower 2.1-2.5 GB/s peaks.

Why this design, and what’s the alternative? The alternative the paper doesn’t take is prefix-caching-aware transfer — reusing KV cache across requests that share a prompt prefix, which is common in multi-turn/agentic workloads. The paper explicitly notes its prototype does not yet support prefix caching, and instead offers only an analytic model (Fig. 15) estimating that RPF would reduce TTFT by 2.0-2.2x at a 90% prefix-cache hit rate over a 100 Gbps link. This is a real gap: prefix caching is exactly the workload pattern (repeated agentic turns, shared system prompts) that motivated the paper’s introduction, and the lack of an empirical (not just analytic) measurement here is a limitation worth flagging up front, before we get to the full limitations discussion in Section 6.

5. Experimental results: does the mechanism actually deliver?

5.1 Setup in brief

All experiments run on a server with eight H100 GPUs (80 GB HBM3 each), two Xeon Platinum 8480C CPUs (112 cores across two NUMA nodes), 2 TB host memory, NVLink/NVSwitch intra-node, and PCIe Gen5 x16 per GPU-to-CPU link; PD-disaggregation experiments span two such nodes over 400-Gbps RoCE-capable NICs. Models evaluated: Qwen3-8B (dense), Qwen3-235B-A22B (MoE, under TP8), and Llama-3.1-8B-Instruct, each paired with a public EAGLE-3 draft head. OasisKV is implemented on top of vLLM v0.12.0’s V1 engine, with compressed-key updates, head-wise top-K prediction, sparse page mapping, and KV transfer implemented in C++/CUDA, and NIXL/UCX used for cross-node KV transfer under disaggregation. Default sparsity configuration: block size $B=16$, $K=128$ blocks (2,048 tokens) selected per KV head, no sink tokens, no dense layers, giving each head 129 resident blocks (2,064 tokens) including its active local block.

Baselines: dense vLLM with FlashAttention-3 (the “no-compression” reference), and three KV-prefetch/retrieval frameworks — ShadowKV, InfiniGen, and FreeKV — each run in its own native framework (none of these three support cross-GPU experiments, which is itself a telling gap in prior art). For accuracy, comparisons are additionally made against Quest (sparse attention) and FreeKV, using implementations from FreeKV’s own repository.

5.2 Single- and multi-GPU throughput: the sparsity-to-throughput conversion actually happens

Figure 6 (paper Fig.11): Decode throughput (top row), running batch size (middle row), and per-token latency/TPOT (bottom row) swept over max concurrency, at 16K and 32K context, for Qwen3-8B (single H100) and Qwen3-235B (TP8, eight H100s). OasisKV (blue) keeps scaling with concurrency well past the point where dense vLLM (orange) flatlines, and comfortably outperforms ShadowKV/InfiniGen/FreeKV (which either plateau early or become unreachable due to OOM at higher concurrency — the gaps in their curves are exactly these unreachable points).

The headline numbers: on Qwen3-8B at 16K context, dense vLLM saturates by max concurrency 32 (about 676 tok/s) because HBM fills up, while OasisKV keeps scaling all the way to 2.1x dense throughput (1,398 vs. 676 tok/s) at concurrency 128. The mechanism behind this gain is genuinely two separate effects layered together, and it’s worth being precise about which is which: (1) at small-to-moderate batch sizes, OasisKV’s lower per-step TPOT (17.7 vs. 23.5 ms at concurrency 16) lets it process more tokens at the same concurrency — this comes from the bounded 2,048-token attention working set reducing compute, combined with the asynchronous pipeline hiding most of the prefetch overhead; (2) at larger batch sizes, OasisKV’s TPOT actually rises above dense attention’s, but this stops mattering because the 2,048-token KV bound now supports 90-95 concurrent requests where dense supports only 22 — the capacity gain outweighs a per-step latency regression. This is an honest and important nuance: OasisKV does not uniformly win on latency; it wins on the product of concurrency and (bounded) latency, and it’s transparent about the crossover.

On the 235B MoE model under TP8, the story shifts because KV cache is now a much smaller share of total GPU memory (weights dominate for a 235B model spread across 8 GPUs), so sparsity saves relatively less HBM, while OasisKV’s added per-step overhead (draft query projection, background pipeline bookkeeping) remains roughly fixed — the result is that OasisKV’s TPOT is actually worse than dense at low batch. It only overtakes dense once batch size clears 32 (16K) or 16 (32K), topping out at 1.9x throughput. This is a legitimate and useful negative result: the paper doesn’t hide that sparsity’s payoff is workload- and model-dependent, and is smaller when KV cache isn’t the dominant memory consumer to begin with.

5.3 PD disaggregation: RPF actually changes the achievable operating point, not just the constant factor

Under PD disaggregation with Qwen3-8B, dense vLLM saturates early because it retains every active request’s full KV cache in HBM at the decode node — its achieved rate never exceeds 0.27 req/s at 24K context or 0.19 req/s at 32K, capping throughput at 550-554 and 384-386 tok/s respectively, with 20-33 preemptions per 32K run (preemption here meaning the scheduler had to evict/pause requests to fit memory — a direct symptom of the capacity wall). Both OasisKV configurations (with and without RPF) keep scaling well past this point, reaching 1,204-1,210 tok/s at 24K and 884-888 tok/s at 32K — 2.1-2.3x dense throughput. RPF’s specific contribution is on the memory side: full transfer holds 3.38-4.52 GiB of decode-node host memory per request for its entire lifetime (aggregate occupancy reaching 161-209 GiB at high offer rates), while RPF lowers this to 1.54-1.73 GiB per request (2.2-2.6x less), with aggregate occupancy of only 46-76 GiB — and this saving grows with context length, since the RPF-transferred union is sized by the fixed top-K budget while full transfer grows with prompt length.

5.4 Real reasoning workload: the number that matters most for a serving team

Figure 7 (paper Fig.13): Decode throughput bars (left axis) and AIME24 accuracy (avg@32, right axis, dashed line = dense accuracy) as $K$ shrinks from dense to $K=64$. On Qwen3-8B, shrinking $K$ from 192 to 64 raises the speedup from 1.39x to 1.89x at a 4.4-point accuracy cost (76.77 -> 72.40); on Qwen3-235B the same sweep goes from 1.08x to 1.27x at a steeper 6.9-point cost. Notably, accuracy stays essentially flat for $K \geq 128$ and even edges past full attention at $K=192$ on both models (within run-to-run noise) — the real cliff only appears at $K=64$.

On the real AIME24 workload end-to-end (not just synthetic sweeps), Qwen3-8B reaches 2,083 tok/s versus 1,235 tok/s for dense (1.69x speedup) at nearly lossless accuracy (−0.1 points), using the 0.05 fetch-ratio cap established in §5.5. Qwen3-235B under TP8 reaches 1,546 vs. 1,283 tok/s (1.20x) at accuracy 83.85 vs. 84.69. The paper is candid that Qwen3-8B is the “more fetch-bound” case — its KV occupies a larger fraction of GPU memory relative to weights, making KV fetch the dominant bottleneck, which is exactly why the fetch cap matters more there.

5.5 Accuracy versus prior retrieval methods

Table 1 (paper Table 1): Accuracy comparison against Quest and FreeKV, each read against the full-attention anchor of its own software stack (HuggingFace Transformers for Quest/FreeKV, vLLM for OasisKV) — the paper is careful to note these numbers should never be compared across stacks, only each method’s $\Delta$ from its own anchor. Across long-input (LongBench v2), long-output (AIME24/25, GPQA-Diamond) regimes, OasisKV’s $\Delta$ from its own full-attention anchor is consistently the smallest or tied-smallest magnitude: on Qwen3-8B LongBench v2 overall, OasisKV is $-0.40$ versus Quest’s $-0.60$ and FreeKV’s $-1.20$; on the long-output reasoning average, OasisKV loses only 0.35 avg@$k$ points versus Quest’s 2.83 and FreeKV’s 2.63.

The honest caveat here (which the paper states plainly, and which is worth restating because it’s easy to skim past): Quest and FreeKV were evaluated in a different serving stack (HuggingFace Transformers) than OasisKV’s own full-attention anchor (vLLM), which uses different sampling and kernel implementations. So while the relative degradation-from-own-anchor comparison is methodologically sound, it does not establish that OasisKV’s absolute accuracy is better than Quest/FreeKV’s absolute accuracy — only that each system’s own sparsity-induced accuracy loss, measured against its own reference point, is smaller for OasisKV. This is a genuinely careful thing for the authors to have flagged, and it also happens to be exactly the kind of nuance a critical reader should double-check before citing the paper’s accuracy numbers as a blanket “OasisKV is more accurate than Quest/FreeKV.”

6. Limitations

The paper is unusually explicit about several of its own limitations, and it’s worth cataloguing them alongside a few the paper doesn’t foreground as strongly.

Explicitly acknowledged by the authors:

1. Draft tokens are used only as a lookahead signal, not for actual speculative verification. The paper states plainly that it “currently uses the draft token solely as a lookahead signal for KV prefetching and forces its rejection” — meaning the accepted-token throughput benefit that speculative decoding normally provides (verifying multiple draft tokens per forward pass) is not being captured here at all. The draft model’s compute cost is being paid, but its main traditional payoff is deliberately thrown away. The paper frames enabling joint speculative verification and prefetching as future work.
2. The prototype does not support prefix caching under disaggregation. As discussed in §4.2, the TTFT benefit of RPF under realistic prefix-cache hit rates is only estimated analytically (Fig. 15), not measured empirically on the actual implementation.
3. MoE models see a smaller and sometimes negative benefit at low batch. On Qwen3-235B-A22B, OasisKV’s TPOT is worse than dense at low concurrency, because the fixed per-step prefetch-pipeline overhead is amortized over a KV cache that’s already a small fraction of total memory. The paper reports this transparently rather than cherry-picking only the favorable regime.

Limitations the paper understates or leaves implicit:

1. No sensitivity analysis for the temporal-locality assumption itself. The entire look-ahead mechanism (§4.2.1) rests on the empirical observation that adjacent decoding steps have highly correlated top-K sets (Fig. 6’s 98.74% agreement). This was measured on GSM8K with Qwen3-8B/EAGLE-3. There is no experiment probing what happens on workloads with genuinely abrupt attention-pattern shifts — e.g., a sudden topic change in a long multi-turn conversation, or retrieval-augmented generation where a newly-injected document chunk becomes suddenly relevant. The capped-eviction mechanism (§3.4) is specifically designed to survive some drift, but the paper never stress-tests the boundary between “the temporal-locality assumption holds well enough for capped eviction to compensate” and “the assumption is violated badly enough that several steps of decoding proceed on genuinely stale context.”
2. EAGLE-3 draft-head dependency is a real deployment constraint, softly stated. OasisKV’s accuracy and prefetch quality depend on having a well-trained EAGLE-3 (or similar MTP) draft head for the specific target model. The paper notes drafts “are increasingly released as reusable artifacts,” which is true for the three models tested, but is not true for the long tail of fine-tuned or lesser-known open models a production deployer might actually run. For models without a public draft head, deploying OasisKV would first require training one — a nontrivial cost the paper’s “training-free” framing (footnote 1, careful as it is) slightly undersells for that scenario.
3. The fixed fetch-cap $C$ is a single global hyperparameter tuned per-experiment, not adapted online. As discussed in §3.4, the paper never explores adaptive capping, and doesn’t report how sensitive the 0.05 default is across workloads beyond the ones tested — a serving team deploying this in production would need to re-tune this constant for their own traffic mix, with no guidance on how to do so besides re-running the same kind of sweep shown in Table 2.
4. No memory-pressure interaction study with other GPU consumers. All experiments appear to run OasisKV in isolation on dedicated hardware. Real production clusters run multiple models, CUDA-graph-heavy batching, and sometimes co-located non-inference workloads; the paper doesn’t discuss how the background CUDA streams for prediction/selection/transfer interact with, or get starved by, other GPU work contending for SM/copy-engine resources.

7. Critical analysis

Weaknesses and flaws specific to this paper. The single biggest internal tension in the paper is between its title’s promise (“lookahead sparse prefetching”) and §5.3.1’s own admission that on the MoE model, OasisKV’s per-step latency is worse than dense attention at low concurrency — the throughput win there is entirely a capacity story (more concurrent requests fit), not a latency story. This is not dishonest — the paper reports it clearly — but it means the paper’s abstract-level framing (“turn sparsity into throughput gain”) is doing some work to paper over a genuinely two-sided trade-off that a reader skimming only the abstract and Figure 13 could easily miss. A more careful framing would state upfront that OasisKV’s benefit decomposes into a latency term (positive at low batch, negative for MoE) and a capacity term (always positive when KV cache is a meaningful fraction of memory), and that the net benefit depends on which term dominates for a given model/workload.

Second, the ablation in Table 2 (fetch-cap sweep) is run only on Qwen3-8B with a single workload (AIME24). Given how central the fetch-cap hyperparameter is to the entire mechanism’s throughput-accuracy trade-off, a single-model, single-workload ablation is thin evidence for generalization. It would have strengthened the paper considerably to show the same sweep on Qwen3-235B and on a non-reasoning long-context workload, to see whether the “0.05 is the sweet spot” finding is model-specific or a more general property of the PCIe-bandwidth-vs-decode-step-time ratio.

Third, the paper compares against ShadowKV, InfiniGen, and FreeKV, but explicitly notes these baselines “are unreachable in their frameworks” at higher concurrency due to OOM or unsupported configurations, and “none of them run their framework with cross-GPU experiments.” This makes the multi-GPU (Qwen3-235B, TP8) comparison in Fig. 11 somewhat one-sided — OasisKV is being compared against baselines operating well outside the regime they were designed or tested for by their own original authors, rather than against a genuinely competitive multi-GPU-aware retrieval system (none currently exists, which is itself the paper’s point, but it does mean the multi-GPU results are more a demonstration of “OasisKV works where nothing else does” than a head-to-head win against a comparably-engineered competitor).

Limitations the authors understate (beyond §6’s list). The accuracy-preservation claims (“within 0.7 points of full attention”) are drawn from a specific and fairly narrow benchmark suite — AIME24/25, GPQA-Diamond, and LongBench v2. These are respectable but heavily reasoning/QA-flavored benchmarks. There’s no evaluation on tasks that might stress long-range copying or verbatim retrieval (e.g., needle-in-a-haystack-style exact-recall tasks, or code generation requiring precise cross-reference to a specific earlier function definition), which are exactly the tasks where losing access to a specific non-top-K historical token could cause a sharp, discontinuous failure rather than a smooth accuracy degradation. Sparse-attention methods in general are known to behave unevenly across task types, and this paper’s benchmark selection, while reasonable, doesn’t probe the failure modes most likely to be sensitive to exactly the kind of approximation OasisKV makes.

Concrete, specific improvement suggestions.

1. Report a task-type-stratified accuracy breakdown that specifically includes exact-recall/needle-in-a-haystack benchmarks, not just reasoning and general long-context QA, to characterize failure modes under top-K selection more precisely than aggregate LongBench v2 scores can.
2. Run the fetch-cap ablation (Table 2) on at least one more model and one more workload family (e.g., Qwen3-235B on a coding-agent trace rather than AIME24) to establish whether the 0.05 default generalizes or needs to be about the ratio of PCIe bandwidth to decode-step time — the paper hints this ratio is the real driver but never validates it across configurations.
3. Add an adaptive (rather than fixed) fetch cap that responds to a measured drift-rate signal, and report whether this closes any of the accuracy gap observed at aggressive $K=64$ settings without sacrificing the throughput gains — directly testing the alternative the current fixed-cap design forgoes.
4. Empirically measure RPF’s TTFT benefit under prefix caching, rather than relying solely on the analytic model in Fig. 15 — this is precisely the workload pattern (multi-turn agentic traffic) that motivates the paper’s opening paragraphs, and an empirical number would considerably strengthen the disaggregated-serving contribution.

8. Reproducibility notes

• Model checkpoints: Qwen3-8B, Qwen3-235B-A22B, Llama-3.1-8B-Instruct are all public. EAGLE-3 draft heads used are also named explicitly and public: Tengyunw/qwen3_8b_eagle3, nvidia/Qwen3-235B-A22B-Eagle3, TanBaby/EAGLE3-LLaMA3.1-Instruct-8B-YARN-64K.
• Serving stack: vLLM v0.12.0 V1 engine, extended with a custom sparse-attention backend, GPU model runner, KV-cache manager, and scheduler support. Core mechanisms (compressed-key updates, head-wise top-K prediction, sparse page mapping, KV transfer) are implemented in C++/CUDA with persistent background workers on separate CUDA streams. Cross-node transfer for PD disaggregation uses NIXL 1.3.0 over UCX 1.21.0.
• Default sparsity config: block size $B=16$, $K=128$ blocks (2,048 tokens) per KV head, compressed-key cache at 1/16 of full KV size, no sink tokens, no dense layers, default fetch ratio 0.05.
• Hardware: 8x H100 80GB HBM3, dual Xeon Platinum 8480C (112 cores, 2 NUMA nodes), 2 TB host DRAM, NVLink/NVSwitch intra-node, PCIe Gen5 x16 GPU-CPU, ConnectX-7 NICs at 400 Gbps RoCE for disaggregation experiments.
• What’s not yet released/described: the paper does not mention a public code release URL as of this preprint; the exact CUDA-stream synchronization implementation (beyond the algorithmic description in §4.2.2) and the head-wise mapping data structure’s exact memory layout would need to be inferred or reimplemented from the paper’s description alone.
• Caveat for reproducers: because ShadowKV/InfiniGen/FreeKV comparisons run in a different framework (HuggingFace Transformers) than OasisKV’s own vLLM-based implementation, any reproduction attempt comparing absolute numbers across these systems needs to carefully control for the serving-stack difference, exactly as the paper itself cautions in Table 1’s own caption.

9. Where this fits in the broader KV-cache-efficiency landscape

OasisKV sits at the intersection of three lines of work this blog has covered before: KV retrieval systems (ArkVale, RetroInfer, ShadowKV — solve capacity, pay a latency tax on the critical path), KV prefetching systems (SpeCache, InfiniGen, FreeKV — solve latency by predicting ahead, but with training-free predictors that sacrifice accuracy, per Fig. 4’s 83.9% proxy-accuracy finding), and speculative decoding infrastructure (EAGLE-3 and its production integration into modern serving stacks) that OasisKV repurposes rather than extends. Its genuine contribution is recognizing that these three lines of work compose: the machinery serving teams are already deploying for speculative decoding throughput gains is, almost incidentally, exactly the machinery a KV-prefetcher needs for an accurate, training-free, one-step-ahead signal. Whether this composition becomes a standard pattern in future serving systems will likely depend on how broadly EAGLE-style draft heads get adopted as a default component of production LLM serving — if draft heads become as ubiquitous as PagedAttention itself, OasisKV’s core trick becomes close to free to deploy; if draft-head coverage stays uneven across the long tail of models, the training-free framing (footnote 1) becomes a real deployment cost for exactly the models that lack one.

10. Conclusion

OasisKV’s core contribution is a genuinely elegant piece of systems co-design: recognizing that speculative-decoding draft tokens are, essentially for free, an accurate one-step-ahead signal for KV-cache prefetching, and then doing the considerable engineering work — a low-overhead foreground propagation trick, a fully asynchronous cross-layer background pipeline, a capped-eviction policy that bounds worst-case PCIe traffic, a head-wise memory-mapping layer that fits sparse decoding into PagedAttention’s existing block model, and a remote-partial-fetching extension for disaggregated serving — needed to make that one insight actually deliver production-grade throughput. The headline numbers (1.69x on a real reasoning workload, up to 2.1x on multi-GPU long-context serving, 2.1-2.3x under PD disaggregation, all within roughly 0.7 accuracy points of full attention) are earned through careful measurement of exactly where the design’s constraints bite, and the paper is commendably candid about the cases where the benefit is smaller or one-sided (the MoE low-batch latency regression, the missing prefix-caching evaluation, the single-workload fetch-cap ablation). For anyone building or evaluating long-context LLM serving infrastructure, OasisKV is a strong argument that speculative decoding and KV-cache management should not be designed as two separate subsystems — the signal one produces is exactly what the other needs.