P2P KV Cache Sharing#

Overview#

In a multi-node deployment, every node runs its own LMCache server process, and each one caches the KV of the requests it has served in its local memory. Without sharing, a prefix that was computed on one node has to be recomputed from scratch when the same prefix arrives on a different node.

Peer-to-peer (P2P) KV cache sharing turns the per-node caches into one logical cache. When a node looks up a prefix that is not in its own memory, it can read that prefix’s KV directly from the memory of the peer node that holds it, over the datacenter network using RDMA. The result is a much higher effective cache hit rate across the fleet, without any central storage tier in the hot path.

The transfer is a one-sided RDMA read from the requesting node into its own L1 buffer; the node that owns the data is not interrupted to serve it. On an RDMA-capable network (InfiniBand / RoCE) this is dramatically faster than recomputing the prefix or round-tripping through a shared object store.

How it works#

P2P involves three pieces:

  • Coordinator — a small HTTP service (one per deployment) that tracks which LMCache servers are alive. Each server registers with it and heartbeats; the coordinator answers “who are my live peers?” queries. It only manages membership — it never sees KV data or participates in lookups. See Multi-Server Coordination.

  • LMCache server — each server runs a P2P controller that periodically asks the coordinator for the current peer list and, for every live peer, opens a connection used to look up and read that peer’s KV.

  • Transfer channel — the RDMA layer that performs the actual remote memory reads. Each server registers its L1 buffer once at startup so peers can read from it.

On a cache miss, a node asks the peer that owns the prefix to lock and locate it, receives the remote addresses, RDMA-reads the KV into its own L1, and serves the request from there. Peers are discovered and connected automatically as they join, and disconnected automatically as they leave — no static peer lists.

Requirements#

  • A coordinator. P2P needs the coordinator for peer discovery. A server started with --p2p-advertise-url but no --coordinator-url refuses to start.

  • An RDMA-capable network (InfiniBand / RoCE) is strongly recommended for production performance. By default P2P uses the nixl transfer engine, which is shipped with LMCache, so there is nothing extra to install.

  • A single, contiguous L1 region. The transfer channel registers the whole L1 buffer for RDMA, so P2P is incompatible with the GDS L1 tier (--gds-l1-path) and the Device-DAX L1 tier (--l1-devdax-path); the server refuses to start in those configurations.

Configuration#

P2P is enabled per server by the --p2p-advertise-url flag. The relevant lmcache server flags are:

Flag

Description

--p2p-advertise-url HOST:PORT

Transfer-channel endpoint this server advertises to peers. Setting it enables P2P. Must be an address other nodes can reach.

--p2p-listen-url HOST:PORT

Address the transfer-channel server binds to. Defaults to --p2p-advertise-url; set it to bind 0.0.0.0 while advertising a routable IP.

--p2p-lookup-timeout SECONDS

Deadline for a peer lookup before it counts as a miss (default 30).

--p2p-load-timeout SECONDS

Deadline for a peer KV read before it counts as a failure (default 30).

--p2p-transfer-engine ENGINE

Transfer-channel implementation (default nixl).

P2P also reuses the coordinator connection flags (--coordinator-url, --coordinator-advertise-ip, --coordinator-heartbeat-interval); the heartbeat interval doubles as the peer-discovery poll interval. See lmcache server and lmcache coordinator for the full flag lists.

Tip

Increase the L1 buffer alignment to at least 64 KB (--l1-align-bytes 65536) on servers that participate in P2P. A larger alignment lets the transfer channel issue bigger, better-aligned RDMA reads and noticeably improves transfer performance. The default (4 KB) is fine for non-P2P deployments.

Transfer engine backends#

The transfer engine is the component that performs the remote memory reads, selected per server with --p2p-transfer-engine.

Engine

Description

nixl (default)

RDMA-based transport, shipped with LMCache. Runs over InfiniBand / RoCE fabrics.

nixl is the only backend available today. The transfer engine is a pluggable abstraction, so additional backends can be added in the future without changing the rest of the P2P stack.

Running a multi-node deployment#

The example below brings up a two-node fleet. Adding more nodes is just more copies of the per-node step, all pointing at the same coordinator.

Step 1 — start the coordinator (on a host all nodes can reach, here 10.0.0.1):

lmcache coordinator --host 0.0.0.0 --port 9300

Step 2 — on each node, start the LMCache server with P2P enabled. Replace <NODE_IP> with that node’s routable address (e.g. 10.0.0.2, 10.0.0.3, …):

lmcache server \
    --host 0.0.0.0 --port 5555 \
    --http-port 8080 \
    --l1-size-gb 100 --eviction-policy LRU \
    --l1-align-bytes 65536 \
    --coordinator-url http://10.0.0.1:9300 \
    --coordinator-advertise-ip <NODE_IP> \
    --p2p-advertise-url <NODE_IP>:8500

--coordinator-advertise-ip is the address peers use to reach this node’s control plane, and --p2p-advertise-url is the RDMA transfer endpoint. Here the server binds 0.0.0.0 (all interfaces) and advertises <NODE_IP>.

Step 3 — on each node, start vLLM pointed at the local LMCache server via the connector. vLLM never talks to peers directly — the LMCache server it connects to does P2P on its behalf:

vllm serve <model> \
    --port 8000 \
    --kv-transfer-config '{"kv_connector":"LMCacheMPConnector","kv_role":"kv_both","kv_load_failure_policy":"recompute","kv_connector_extra_config":{"lmcache.mp.port":5555}}'

Once both nodes have registered, a prefix first served on node 2 will be served from node 2’s cache when the same prefix later arrives on node 3 — read over RDMA instead of recomputed.

Running on a single node (testing & debugging)#

You can exercise the entire P2P path on a single multi-GPU machine by running two LMCache servers and two vLLM servers over localhost, plus one coordinator. This is the recommended way to develop and debug P2P.

# Terminal 1 — coordinator
lmcache coordinator --host 0.0.0.0 --port 9300

# Terminal 2 — node "A": LMCache server
lmcache server \
    --host 127.0.0.1 --port 6555 --http-port 7555 \
    --l1-size-gb 50 --eviction-policy LRU \
    --l1-align-bytes 65536 \
    --instance-id node-a \
    --coordinator-url http://127.0.0.1:9300 \
    --coordinator-advertise-ip 127.0.0.1 \
    --p2p-advertise-url 127.0.0.1:8555

# Terminal 3 — node "A": vLLM on GPU 0
CUDA_VISIBLE_DEVICES=0 vllm serve <model> --port 8000 \
    --kv-transfer-config '{"kv_connector":"LMCacheMPConnector","kv_role":"kv_both","kv_load_failure_policy":"recompute","kv_connector_extra_config":{"lmcache.mp.port":6555}}'

# Terminal 4 — node "B": LMCache server
lmcache server \
    --host 127.0.0.1 --port 6556 --http-port 7556 \
    --l1-size-gb 50 --eviction-policy LRU \
    --l1-align-bytes 65536 \
    --instance-id node-b \
    --coordinator-url http://127.0.0.1:9300 \
    --coordinator-advertise-ip 127.0.0.1 \
    --p2p-advertise-url 127.0.0.1:8556

# Terminal 5 — node "B": vLLM on GPU 1
CUDA_VISIBLE_DEVICES=1 vllm serve <model> --port 8001 \
    --kv-transfer-config '{"kv_connector":"LMCacheMPConnector","kv_role":"kv_both","kv_load_failure_policy":"recompute","kv_connector_extra_config":{"lmcache.mp.port":6556}}'

The two servers must differ in every port: ZMQ (--port), HTTP (--http-port), and the P2P transfer endpoint (--p2p-advertise-url). Give each a distinct --instance-id so they are easy to tell apart.

To test the path, send a long prompt to vLLM A (port 8000) and then the same prompt to vLLM B (port 8001). B has never seen the prompt and its own cache is empty, so any LMCache hit on B must have been read from A over P2P.

Note

On a single host, localhost traffic typically uses the loopback/TCP path rather than RDMA, so latencies are not representative of a real RDMA fabric. Single-node mode is for functional testing and debugging; benchmark performance on a real multi-node RDMA deployment.

Verifying P2P is working#

Query a server’s status endpoint to see its P2P state and discovered peers:

curl -s http://127.0.0.1:7555/status | python3 -m json.tool

Look for p2p_state (registered once it has joined the coordinator) and p2p_peer_count (the number of connected peers; 1 in the two-node example). You can also list the fleet from the coordinator:

curl -s http://127.0.0.1:9300/instances | python3 -m json.tool

With LMCACHE_LOG_LEVEL=DEBUG, each server logs Added P2P adapter ... for peer <instance-id> when it connects to a peer, and Removed P2P adapter ... when a peer leaves.

Limitations#

  • Read-only. A node reads KV from its peers; it never writes into a peer’s memory. This keeps each node the sole owner of its L1.

  • One hop. A node reads directly from the peer that holds the prefix; reads are not chained across multiple peers.