Move the KV, Not the Work: P2P Cache Sharing in llm-d
A request arrives and its 48K-token prefix is already sitting in a KV cache, but the pod holding it is busy. Route the request to the cache owner and it queues behind that pod's backlog; route it to an idle pod and that pod spends seconds recomputing work the cluster already did. Waiting and recomputing are both wrong answers. P2P (peer-to-peer) KV cache sharing adds the third option: send the request to the best pod for load, and move the finished KV to it.
A busy owner is only one way locality breaks. Load balancing may select another worker, or the required KV may have been generated by a different serving role. In every such case the llm-d router already knows how much of a request's prefix each candidate holds. With P2P, that knowledge becomes a transfer instruction: route the request to the best pod overall, then tell it where to fetch the missing prefix.
Replaying a real recorded coding-agent session that forks 43 parallel subagents over a shared 40K-token prefix, adding P2P removed every straggler branch a copy could exist for: p90 branch-start TTFT dropped from 11.2 s to 1.6 s (-86%), and each verified pull replaced ~13 s of prefix recomputation with a sub-second transfer.
This is not a universal speedup. When routing already produces a local hit, P2P correctly does nothing. When locality conflicts with load balance or serving topology, it can replace seconds of repeated computation with a peer transfer. The operating principle is simple: local hits first; portable reuse when locality breaks.
The rest of this post answers four questions: how expensive is a pull; whether it improves load-balanced serving; whether it preserves session history across prefill/decode (P/D) roles; and when it should stay inactive.
The Gap: Prefix Caches Are Per-Pod
Shared system prompts, common documents, agentic loops, and multi-turn conversations all repeat long prefixes. Reusing their KV skips a large portion of prefill work, reduces time to first token (TTFT), and leaves more GPU cycles for decode. For more background, see KV-Cache Wins You Can See.
llm-d already improves reuse in two ways. Prefix-aware routing sends requests to pods that hold their prefixes. KV offloading keeps copies in a larger CPU tier, and a filesystem backend can extend that tier to local or shared storage for persistence and larger capacity.
Routing alone cannot make every good placement local, while shared storage adds an infrastructure dependency and a storage data path. Often the required blocks are already in another pod's CPU tier, one network hop away. P2P cache sharing uses that middle path.
How P2P Works
P2P pulling is a small, opt-in scheduling step. It is off by default because the crossover and available CPU-tier and fabric capacity are deployment-specific; without calibration, a short-prefix pull can cost more than recomputation. Every participating vLLM instance can serve two roles, selected per request:
- Consumer. Pulls matching KV blocks from a peer instead of computing them locally.
- Producer. Serves blocks from its CPU offload tier when another peer requests them.
The consumer sends the producer the hashes of the blocks it needs. The producer reports which blocks remain available, then writes the matching blocks over NIXL, the NVIDIA Inference Xfer Library. The scheduler describes the operation as a pull because the consumer requests it; after the lookup handshake, the producer performs the data-path write. Neither GPU performs the peer-to-peer copy, so serving a peer costs the producer CPU memory bandwidth and network capacity, not GPU compute; in the fork experiments below, a verified pull was served by a decode worker while it carried its own load. A normal miss falls back to computation.
The EPP selects the destination and source; the sidecar passes that decision to the engine; NIXL moves matching blocks CPU-to-CPU.
The router's Endpoint Picker (EPP) supplies the source decision from its prefix index. It compares the peer with the most cached tokens against the pod selected to run the request. If the peer's lead is large enough to beat the measured transfer crossover, the EPP names that peer as the KV source; a tie or self-match stays local. Among near-tied holders, the EPP samples the source by load, weighting peers inversely to their waiting-queue depth, so concurrent pulls spread across producers instead of converging on one. The measurements in this post use the precise, KV-event-fed index so the source decision reflects the blocks the engines actually report.
P2P also composes with P/D disaggregation. A prefill worker can pull history generated by a decoder, compute only the new portion, and continue the normal P/D flow. No application change is required.
| Situation | P2P behavior |
|---|---|
| Placement finds a local prefix | Stays idle; moving zero bytes is optimal |
| Load balancing selects another pod | Pulls the prefix instead of recomputing it |
| A later P/D turn needs decode-generated history | Moves that history to the prefill worker |
What We Measured
We evaluated four models, from an 8B dense model to a 753B wide-EP MoE, across aggregated and P/D-disaggregated deployments. The experiments make two kinds of claims. In an isolated A/B, P2P is the only policy difference. In a system-policy comparison, placement or the CPU offload stack changes with it, so the result belongs to the complete serving policy.
The three anchor results below establish the transfer economics, the clean load-spill payoff, and the P/D session-continuity payoff. The document Q&A and smaller-model experiments then show where the same mechanism appears under different serving policies.
1. Price the Transfer Before Using It
Pulling is useful only when it costs less than recomputing. The crossover depends on the model, KV representation, hardware, and network, so it must be measured rather than assumed. A fresh prefix in these experiments means a newly salted token sequence that is seeded on the source and absent from the consumer before the timed request.
Setup: a single pod pair, KV source injected directly, no router in the path
The driver seeds a fresh prefix on the source pod and requests it on the consumer, injecting the KV source into the request rather than letting the router decide. This measures the transfer itself rather than any placement policy.
On openai/gpt-oss-120b with H200 GPUs, the pull won at every measured prefix
length. Its latency grew much more slowly than recompute:
| Prefix tokens | Recompute | P2P pull | Delta |
|---|---|---|---|
| 2,048 | 78 ms | 35 ms | -56% |
| 8,192 | 250 ms | 57 ms | -77% |
| 16,384 | 510 ms | 86 ms | -83% |
| 32,768 | 1,173 ms | 165 ms | -86% |
| 49,152 | 1,988 ms | 235 ms | -88% |
Recompute grows far faster than the peer transfer; the advantage reaches 88% at 48K tokens.
The crossover moved on the 753B GLM testbed, whose KV footprint is about 93 KB per token. Pull and recompute were roughly tied near 8K tokens; at 12K the pull was 27% faster, and at 24K it was 61% faster. A 24K prefix is about 2.2 GiB of KV, while a 70K prefix is about 6.2 GiB. Break-even depends on the ratio of prefill cost to KV-transfer cost, not model size alone. On the GLM rig, short-prefix recompute stayed below the measured 1.2-1.3 second pull floor, so P2P began winning only around 8.7K tokens.
This is why the EPP uses a per-deployment minimum cached-token advantage rather than pulling every remote hit. The production threshold should sit above the measured crossover and include margin for fabric contention and producer load.
2. When Load Breaks Locality
The cleanest end-to-end result comes from a real recorded workload: a
Claude Code session from the public semianalysisai/cc-traces-weka-062126
corpus that forks 43 parallel subagents, every one inheriting the same
40K-token context. The burst arrives faster than any single worker can absorb,
so placement must send siblings to workers that have never seen the prefix.
Whether those workers recompute or pull is the only policy difference.
Setup: GLM-5.2-FP8 on 32x H200 P/D-disaggregated; token-load placement with and without the source producer; AIPerf trace replay
753B MoE, two prefill and two decode instances, each 8-way data/expert
parallel. Both arms run the same load-modeled prefix-affinity placement
(prefix-cache-affinity-filter over the precise index with
token-load-scorer); the P2P arm adds only the p2p-source-producer at
minCachedTokenDelta: 12288. The workload is replayed with AIPerf
(weka_trace, fixed schedule, ignore_eos), which reconstructs the recorded
token counts, KV-block sharing, and subagent spawn timing exactly. Two fork
groups of matched size (43 and 44 children, prefixes within 0.8%) serve as
twins so each can act as the other's control, with cold caches enforced
between arms; the comparison was then repeated with the windows swapped.
Every pull was verified at three layers: the router directive, the source
engine's transfer session, and the destination engine's
external_prefix_cache_hits counter.
Every branch start in the same fork under both arms. P2P leaves only the burst head - the siblings that arrived before any copy of the prefix existed.
Without the pull, the six siblings that scattered onto cold workers each recomputed the prefix - and recomputing simultaneously, they slowed each other from 6.6 s to 11-12.5 s apiece, putting p90 branch-start TTFT at 11.2 s. With the pull, the same placements fetched the prefix in about a second, and p90 fell to 1.6 s. The median holds at ~1.2 s in both arms because most siblings land warm either way; what P2P removes is the straggler band. The only slow starts left are the burst head - siblings that arrive before any copy of the prefix exists anywhere, which no mechanism can serve. The reversed-window replicate reproduced the shape (p90 -66% with the wider burst head), and a separately measured pull on a 75K-token fork start replaced a 13.9-second cold prefill with a 790 ms transfer, within 6% of a value predicted in advance from the recompute rate measured in earlier runs.
A smaller Llama-3.1-8B shared-prefix pool showed the same causal shape. At 8
requests per second, P2P reduced median request latency by 43%. Near
saturation, it raised the fleet ceiling by 22% and peak token throughput by
32%. The gain grew with load because the no-pull control consumed capacity
recomputing cross-pod misses:
| Offered rate | Without P2P | With P2P |
|---|---|---|
| 4 req/s | 1.12 s p50 | 0.93 s p50 |
| 8 req/s | 2.49 s p50 | 1.41 s p50 (-43%) |
| 12 req/s | 12.2 s p50, 9.9 req/s achieved | 2.1 s p50, 11.6 req/s achieved |
| 16 req/s | 21.3 s p50, 10.3 req/s achieved | 7.8 s p50, 12.6 req/s achieved (+22%) |
Setup: 4 aggregated H200 pods; identical load-balanced placement on both sides
A pool of 64 shared 16K prefixes, larger than any single pod's cache. Both
runs use the same load-balanced placement and differ only by the
p2p-source-producer. These campaign configurations are archived with the
measurement record rather than shipped with the guide.
Document Q&A: A Complete Policy Comparison
The document Q&A experiment asks a related but different question: can load-aware placement plus P2P outperform precise affinity when many active sessions queue behind their document owners? It compares complete serving policies, not P2P as a single toggle.
Setup: gpt-oss-120b on 16 aggregated H200 pods; epp-affinity.yaml versus epp-load-p2p.yaml
The baseline is the guide's precise prefix-affinity configuration,
epp-affinity.yaml. The candidate is load-aware placement plus the pull,
epp-load-p2p.yaml, at minCachedTokenDelta 2048. Both run the precise
index at the fleet-matched podCacheSize: 32.
The workload used 192 distinct 48K-token documents, each queried through six short turns, with 128 conversations active at once. The corpus fit in aggregate GPU KV capacity, so owner contention rather than capacity scarcity drove the result. Warm precise affinity kept median TTFT at 0.3 seconds, but its p99 reached 25.2 seconds as sessions queued on their owners. Load-aware placement plus P2P paid a 0.6-second median to move displaced prefixes, then reduced p99 to 16.6 seconds and increased throughput by 35%. On a freshly rolled fleet with empty cache tiers, it completed every request while affinity encountered 48 client timeouts.
The result belongs to the combined policy: load balancing removes the owner queues, and P2P makes that placement affordable by turning remote misses into transfers. It also shows why median latency alone is insufficient. Affinity wins the cheapest local hit, while load-aware placement wins the tail and the fleet throughput when ownership becomes a queueing constraint.
3. Preserve Session History Across P/D Roles
Multi-turn P/D sessions create a locality break that placement cannot eliminate. The decoder generates the newest KV history, but a prefill worker handles the next turn. Without a peer transfer, the prefiller rebuilds history that already exists on another serving role.
The mechanism was first isolated on Llama-3.1-8B. The prefiller received
decode-generated history by peer pull, moving 477K tokens at one topology and
1.65M tokens under higher contention. Per-turn TTFT stayed near parity because
recomputing a short answer on an 8B model was already cheap; the benefit was
reclaimed prefill capacity rather than a visible latency reduction.
Pulling a generated turn requires the next request to reproduce the same token IDs: Llama re-renders assistant turns verbatim, while models whose templates drop reasoning segments can pull only input context and re-prefilled history.
The user-visible payoff appeared on Qwen3-30B-A3B-Thinking under the
agentic-serving workload: the conversation-replay profile from the
agentic-serving guide,
which models coding sessions over large reused contexts with tool-call pauses
between turns. Its context, turn-count and tool-gap ranges are scaled to this
six-GPU testbed - 24 conversations, dynamic system prompts of 10K to 100K
tokens, 4 to 40 turns each, and tool-call gaps of 1 to 20 seconds - while the
per-turn shapes (about 1,500 input and 425 output tokens) match the guide's
profile. Those gaps give session KV time to leave GPU memory, so a returning
turn must either rebuild the accumulated history or pull its reusable blocks
from a peer. Prompts averaged 61.9K tokens.
Setup: 2 prefill and 4 decode pods; identical placement, the P2P side adds the offload tier and the pull
One H200 per pod. Both runs carry NixlConnector for the P/D handoff and use
identical placement plugins and weights. The P2P run adds the CPU offload
tier, enables the pull on the decode routing sidecar, and adds the
p2p-source-producer at minCachedTokenDelta 1024.
On the reproduced run, the P2P stack reduced median TTFT from 6.83 to 1.09 seconds and raised throughput from 0.82 to 1.24 requests per second: 6.3x lower median TTFT and 50% more throughput.
The original run measured 4.8x lower median TTFT and 33% more throughput. A later reproduction reached 6.3x and 50%; both show the same session-history payoff.
A separate run directly observed 1.23 million tokens of session history moved between peers, confirming the mechanism. The extreme tail did not improve. The worst case on both sides was the first prefill of a cold 100K-token context, and P2P can reuse only KV that someone has already computed. It removes repeated work; it does not remove the first computation.
Where P2P Should Stay Inactive
The negative controls are part of the result, not footnotes. They define where the feature should remain quiet:
- A local hit is already optimal. Under precise affinity on the wide-EP testbed, no remote peer had a cached-token advantage. No pull fired and the P2P configuration behaved like the control.
- Local CPU restores are not peer transfers. Against the
P/D disaggregation guide
as shipped with plain
NixlConnector, adding the offload stack cut median TTFT from 11.94 to 1.16 seconds (10.3x) - but under that guide's prefix-affine placement the peer pull stayed idle, so the gain belongs to the local CPU tier. - A restarted EPP has no source map. Peers may still hold KV, but the new index cannot name them until it learns new state. P2P is not restart recovery.
- A first-seen prefix must still be computed. The first request creates reusable KV; later requests can move it.
These controls are why an enabled configuration is not enough evidence. A valid benchmark must show that placement created a useful remote source and, when causal attribution matters, that matching blocks actually moved.
The Operational Rule
Calibrate before enabling pulls broadly. Measure recompute versus transfer on a warmed peer pair, choose a threshold above the crossover, and verify transferred bytes rather than inferring success from timing alone. The first connection between two peers may pay a one-time session-establishment cost that steady-state pulls do not, so a single cold probe prices the transient rather than the data path.
Every peer must use identical block-size and hash-seed settings. A mismatch produces different block hashes and silently degrades P2P to zero matching transfers.
The P2P KV Cache Sharing guide carries the deployment manifests, CPU-tier sizing, compatibility rules, verification gates, calibration workload, and complete benchmark reports.
What's Next
The results above establish the basic economics and two production-shaped payoff cases. The next step is to measure P2P under fleet changes and traffic patterns that create locality breaks dynamically:
- Hot-prefix skew. Drive a non-uniform prefix distribution that concentrates work on a few cache owners, then measure per-worker prefill balance and p99 TTFT with load-aware placement and P2P.
- Scale-out warmup. Add a cold replica under steady shared-prefix traffic and compare how quickly it reaches useful TTFT and cache-hit levels with and without peer pulls.
- Restart and preemption recovery. Restart a prefill fleet under live multi-turn sessions, where every conversation must recover its context at once. Without pulls this is a synchronized recompute storm; with them, the decode tier serves the history back. Simultaneous cold recomputes already measurably slow each other in the fork experiment, so this is where the per-pull saving should compound into the mean.
- Prefetch ahead of arrival. Trigger the pull from local or remote CPU when a session's next request is predictable, so the transfer overlaps idle time instead of the request's critical path.
Local Hits First, Portable Reuse When Locality Breaks
The transfer does not create compute or network capacity. It decouples placement from cache locality, allowing the scheduler to optimize for load or topology without automatically paying the full recompute cost. The crossover measurement prices each transfer; the fleet experiments show how that price compounds into the tail latency and throughput users feel.
Keep the request with the cache when that is the best placement. When load or topology requires a different worker, move the KV instead of recomputing it.