P2P KV Cache Sharing#

Overview#

In a multi-node deployment, every node runs its own LMCache server process, and each one caches the KV of the requests it has served in its local memory. Without sharing, a prefix that was computed on one node has to be recomputed from scratch when the same prefix arrives on a different node.

Peer-to-peer (P2P) KV cache sharing turns the per-node caches into one logical cache. When a node looks up a prefix that is not in its own memory, it can read that prefix’s KV directly from the memory of the peer node that holds it, over the datacenter network using RDMA. The result is a much higher effective cache hit rate across the fleet, without any central storage tier in the hot path.

The transfer is a one-sided RDMA read from the requesting node into its own L1 buffer; the node that owns the data is not interrupted to serve it. On an RDMA-capable network (InfiniBand / RoCE) this is dramatically faster than recomputing the prefix or round-tripping through a shared object store.

How it works#

P2P involves three pieces:

  • Coordinator — a small HTTP service (one per deployment) that tracks which LMCache servers are alive. Each server registers with it and heartbeats; the coordinator answers “who are my live peers?” queries. It only manages membership — it never sees KV data or participates in lookups. See Multi-Server Coordination.

  • LMCache server — each server runs a P2P controller that periodically asks the coordinator for the current peer list and, for every live peer, opens a connection used to look up and read that peer’s KV.

  • Transfer channel — the RDMA layer that performs the actual remote memory reads. Each server registers its L1 buffer once at startup so peers can read from it.

On a cache miss, a node asks the peer that owns the prefix to lock and locate it, receives the remote addresses, RDMA-reads the KV into its own L1, and serves the request from there. Peers are discovered and connected automatically as they join, and disconnected automatically as they leave — no static peer lists.

Requirements#

  • A coordinator. P2P needs the coordinator for peer discovery. A server started with --p2p-advertise-url but no --coordinator-url refuses to start.

  • An RDMA-capable network (InfiniBand / RoCE) is strongly recommended for production performance. By default P2P uses the nixl transfer engine, which ships as the optional lmcache[nixl] extra — install it with uv pip install lmcache[nixl] (or pip install lmcache[nixl]) before enabling P2P.

  • A single, contiguous L1 region. The transfer channel registers the whole L1 buffer for RDMA, so P2P is incompatible with the GDS L1 tier (--gds-l1-path) and the Device-DAX L1 tier (--l1-devdax-path); the server refuses to start in those configurations.

Configuration#

P2P is enabled per server by the --p2p-advertise-url flag. The relevant lmcache server flags are:

Flag

Description

--p2p-advertise-url HOST:PORT

Transfer-channel endpoint this server advertises to peers. Setting it enables P2P. Must be an address other nodes can reach.

--p2p-listen-url HOST:PORT

Address the transfer-channel server binds to. Defaults to --p2p-advertise-url; set it to bind 0.0.0.0 while advertising a routable IP.

--p2p-lookup-timeout SECONDS

Deadline for a peer lookup before it counts as a miss (default 30).

--p2p-load-timeout SECONDS

Deadline for a peer KV read before it counts as a failure (default 30).

--p2p-transfer-engine ENGINE

Transfer-channel implementation (default nixl).

P2P also reuses the coordinator connection flags (--coordinator-url, --coordinator-advertise-ip, --coordinator-heartbeat-interval); the heartbeat interval doubles as the peer-discovery poll interval. See lmcache server and lmcache coordinator for the full flag lists.

Tip

Increase the L1 buffer alignment to at least 64 KB (--l1-align-bytes 65536) on servers that participate in P2P. A larger alignment lets the transfer channel issue bigger, better-aligned RDMA reads and noticeably improves transfer performance. The default (4 KB) is fine for non-P2P deployments.

Transfer engine backends#

The transfer engine is the component that performs the remote memory reads, selected per server with --p2p-transfer-engine.

Engine

Description

nixl (default)

RDMA-based transport, shipped as the optional lmcache[nixl] extra (uv pip install lmcache[nixl]). Runs over InfiniBand / RoCE fabrics.

mooncake_te

Uses Mooncake Transfer Engine for P2P transfers. It uses TCP by default and can use RDMA when configured through environment variables.

Mooncake Transfer Engine#

Select the Mooncake Transfer Engine backend by passing --p2p-transfer-engine mooncake_te to lmcache server. This backend uses TCP unless an RDMA protocol and device are specified. To use RDMA, set MC_TE_PROTOCOL to rdma and MC_TE_DEVICE to the RDMA device name when starting each server. For example:

MC_TE_PROTOCOL=rdma MC_TE_DEVICE=mlx5_0 \
    lmcache server \
    <argument-list> \
    --p2p-transfer-engine mooncake_te

Replace mlx5_0 with the RDMA device available on the node. Both variables must be set for RDMA; when they are omitted, Mooncake Transfer Engine uses TCP.

Running a multi-node deployment#

The example below brings up a two-node fleet. Adding more nodes is just more copies of the per-node step, all pointing at the same coordinator.

Step 1 — start the coordinator (on a host all nodes can reach, here 10.0.0.1):

lmcache coordinator --host 0.0.0.0 --port 9300

Step 2 — on each node, start the LMCache server with P2P enabled. Replace <NODE_IP> with that node’s routable address (e.g. 10.0.0.2, 10.0.0.3, …):

lmcache server \
    --host 0.0.0.0 --port 5555 \
    --http-port 8080 \
    --l1-size-gb 100 --eviction-policy LRU \
    --l1-align-bytes 65536 \
    --coordinator-url http://10.0.0.1:9300 \
    --coordinator-advertise-ip <NODE_IP> \
    --p2p-advertise-url <NODE_IP>:8500

--coordinator-advertise-ip is the address peers use to reach this node’s control plane, and --p2p-advertise-url is the RDMA transfer endpoint. Here the server binds 0.0.0.0 (all interfaces) and advertises <NODE_IP>.

Step 3 — on each node, start vLLM pointed at the local LMCache server via the connector. vLLM never talks to peers directly — the LMCache server it connects to does P2P on its behalf:

vllm serve <model> \
    --port 8000 \
    --kv-transfer-config '{"kv_connector":"LMCacheMPConnector","kv_role":"kv_both","kv_load_failure_policy":"recompute","kv_connector_extra_config":{"lmcache.mp.port":5555}}'

Once both nodes have registered, a prefix first served on node 2 will be served from node 2’s cache when the same prefix later arrives on node 3 — read over RDMA instead of recomputed.

Note

kv_load_failure_policy is a vLLM connector setting that LMCache passes through unchanged; LMCache’s own L2 adapters do not currently read or enforce it. A genuine L2 load failure (a peer unreachable, an object backend erroring out, etc.) and an ordinary cache miss are not distinguished internally — both fall through to recompute, regardless of what kv_load_failure_policy is set to. If your deployment needs load failures to surface as an error rather than silently recompute, that distinction is not yet implemented at the connector boundary.

Running on a single node (testing & debugging)#

You can exercise the entire P2P path on a single multi-GPU machine by running two LMCache servers and two vLLM servers over localhost, plus one coordinator. This is the recommended way to develop and debug P2P.

# Terminal 1 — coordinator
lmcache coordinator --host 0.0.0.0 --port 9300

# Terminal 2 — node "A": LMCache server
lmcache server \
    --host 127.0.0.1 --port 6555 --http-port 7555 \
    --l1-size-gb 50 --eviction-policy LRU \
    --l1-align-bytes 65536 \
    --instance-id node-a \
    --coordinator-url http://127.0.0.1:9300 \
    --coordinator-advertise-ip 127.0.0.1 \
    --p2p-advertise-url 127.0.0.1:8555

# Terminal 3 — node "A": vLLM on GPU 0
CUDA_VISIBLE_DEVICES=0 vllm serve <model> --port 8000 \
    --kv-transfer-config '{"kv_connector":"LMCacheMPConnector","kv_role":"kv_both","kv_load_failure_policy":"recompute","kv_connector_extra_config":{"lmcache.mp.port":6555}}'

# Terminal 4 — node "B": LMCache server
lmcache server \
    --host 127.0.0.1 --port 6556 --http-port 7556 \
    --l1-size-gb 50 --eviction-policy LRU \
    --l1-align-bytes 65536 \
    --instance-id node-b \
    --coordinator-url http://127.0.0.1:9300 \
    --coordinator-advertise-ip 127.0.0.1 \
    --p2p-advertise-url 127.0.0.1:8556

# Terminal 5 — node "B": vLLM on GPU 1
CUDA_VISIBLE_DEVICES=1 vllm serve <model> --port 8001 \
    --kv-transfer-config '{"kv_connector":"LMCacheMPConnector","kv_role":"kv_both","kv_load_failure_policy":"recompute","kv_connector_extra_config":{"lmcache.mp.port":6556}}'

The two servers must differ in every port: request transport (--port), HTTP (--http-port), and the P2P transfer endpoint (--p2p-advertise-url). Give each a distinct --instance-id so they are easy to tell apart.

To test the path, send a long prompt to vLLM A (port 8000) and then the same prompt to vLLM B (port 8001). B has never seen the prompt and its own cache is empty, so any LMCache hit on B must have been read from A over P2P.

Note

On a single host, localhost traffic typically uses the loopback/TCP path rather than RDMA, so latencies are not representative of a real RDMA fabric. Single-node mode is for functional testing and debugging; benchmark performance on a real multi-node RDMA deployment.

Verifying P2P is working#

Query a server’s status endpoint to see its P2P state and discovered peers:

curl -s http://127.0.0.1:7555/status | python3 -m json.tool

Look for p2p_state (registered once it has joined the coordinator) and p2p_peer_count (the number of connected peers; 1 in the two-node example). You can also list the fleet from the coordinator:

curl -s http://127.0.0.1:9300/instances | python3 -m json.tool

With LMCACHE_LOG_LEVEL=DEBUG, each server logs Added P2P adapter ... for peer <instance-id> when it connects to a peer, and Removed P2P adapter ... when a peer leaves.

Using the agent skills#

The repository ships three agent skills under .claude/skills/ that package the recipes on this page as runnable scripts, so a coding agent (Claude Code, Codex, Cursor, …) or a person can bring up, test and diagnose a P2P deployment without improvising:

Skill

What it does

lmcache-p2p-launch

Probes each node (GPUs, RDMA devices and their NUMA placement, Python stack, free ports, model cache), optionally creates a venv, writes one env.sh per LMCache instance, starts the coordinator, the servers and vLLM, and waits until every node reports p2p_state=registered with the expected peer count. Its references explain the UCX multi-rail settings (UCX_TLS must include cuda_copy with recent nixl; UCX_MAX_RMA_RAILS counts paths, two per LAG bond) and the single-node variant.

lmcache-p2p-test

Cross-node correctness (basic_test.py: a prompt served cold on one node and from the peer on the other, with num_lmcache_cached_tokens and identical outputs as the pass criteria), a stress ladder (concurrency, long documents, sustained mixed load), the lmcache bench engine long-doc-qa workload across nodes, Prometheus scraping of both servers with a live per-node monitor, and a Markdown summary of all results.

lmcache-p2p-diagnose

A symptom-to-fix decision tree (peers not registered, server exiting with NIXL_ERR_BACKEND, zero cached tokens, l1_oom / l1_contended fallbacks, single-NIC or TCP transfers), plus scripts that check RDMA readiness, probe the NIXL/UCX backend under a candidate environment, show which NICs carried a transfer, parse UCX traces, and collect a debug bundle for an issue.

With an agent, describe the task and the matching skill loads on its own, for example “bring up LMCache P2P between host A and host B with Qwen/Qwen3-8B”, “stress P2P at concurrency 32”, or “only one NIC carries P2P traffic”. Without an agent, the scripts run directly; every one of them reads the env.sh written by the launch skill:

S=.claude/skills
# on every node: describe the deployment (the coordinator runs on 10.0.0.1)
$S/lmcache-p2p-launch/scripts/gen_env.sh --coordinator-ip 10.0.0.1 \
    --peer-ip 10.0.0.3 --peer-ssh node3 --model Qwen/Qwen3-8B --gpu 0 --l1-gb 80
# coordinator node, then the others
$S/lmcache-p2p-launch/scripts/start_node.sh --with-coordinator
$S/lmcache-p2p-launch/scripts/wait_ready.sh --peers 1
# prove a transfer, then stress it with the monitor running
python $S/lmcache-p2p-test/scripts/basic_test.py
$S/lmcache-p2p-test/scripts/start_prometheus.sh
python $S/lmcache-p2p-test/scripts/stress_p2p.py --concurrency 32 --tag conc32
# if something is off: which NICs moved bytes, and a bundle to attach to an issue
$S/lmcache-p2p-diagnose/scripts/rail_check.sh --peer-ssh node3
$S/lmcache-p2p-diagnose/scripts/collect_bundle.sh

Each SKILL.md records the LMCache, vLLM and nixl versions it was last verified with.

Limitations#

  • Read-only. A node reads KV from its peers; it never writes into a peer’s memory. This keeps each node the sole owner of its L1.

  • One hop. A node reads directly from the peer that holds the prefix; reads are not chained across multiple peers.