Deployment Guide#

This page covers deploying LMCache multiprocess mode in Docker and Kubernetes environments, along with production best practices.

Docker#

LMCache container:

docker run --runtime nvidia --gpus all \
    --network host \
    --ipc host \
    lmcache/standalone:nightly \
    /opt/venv/bin/lmcache server \
    --l1-size-gb 60 --eviction-policy LRU --max-workers 4 --port 6555

vLLM container:

docker run --runtime nvidia --gpus all \
    --network host \
    --ipc host \
    lmcache/vllm-openai:latest-nightly \
    Qwen/Qwen3-14B \
    --kv-transfer-config \
    '{"kv_connector":"LMCacheMPConnector", "kv_role":"kv_both", "kv_connector_extra_config": {"lmcache.mp.port": 6555}}'

Required Docker flags:

  • --network host – Allows the vLLM container to reach LMCache on localhost.

  • --ipc host – Required for CUDA IPC shared memory transfers between containers (see Isolated IPC below for the plan to remove this requirement).

  • --runtime nvidia --gpus all – GPU access via the NVIDIA container runtime.

HTTP server variant:

For health-check and cache management API support (useful with container orchestrators), use the HTTP server entry point:

docker run --runtime nvidia --gpus all \
    --network host \
    --ipc host \
    lmcache/standalone:nightly \
    /opt/venv/bin/lmcache server \
    --l1-size-gb 60 --eviction-policy LRU --max-workers 4 --port 6555

Isolated IPC (running without --ipc host)#

CUDA IPC in MP mode has two legs, and by default both depend on a shared /dev/shm tmpfs – which is what --ipc host / hostIPC: true really provides:

  • KV-cache memory sharing: the default registration path uses PyTorch storage IPC, which keeps a reference-counter file in /dev/shm.

  • Event ordering (the per-STORE/RETRIEVE device events): CUDA interprocess event handles only resolve when both containers share a /dev/shm tmpfs.

The isolated IPC setting removes both: KV-cache registration switches to raw CUDA IPC memory handles (docs/design/v1/platform/cuda/ipc_wrapper.md) and event ordering to timeline-semaphore events carried over CUDA IPC memory handles (docs/design/v1/platform/cuda/timeline_semaphore_event_ipc.md). Both rendezvous in the kernel driver, so they work across containers that share nothing – no host IPC namespace, no common /dev/shm, no --ipc host. It must be enabled on both sides of a deployment:

# LMCache server
lmcache server --isolated-ipc --l1-size-gb 60 --eviction-policy LRU

# vLLM
vllm serve Qwen/Qwen3-14B --kv-transfer-config \
    '{"kv_connector":"LMCacheMPConnector", "kv_role":"kv_both",
      "kv_connector_extra_config": {"lmcache.mp.port": 6555,
                                    "lmcache.mp.isolated_ipc": true}}'

The two event mechanisms exchange incompatible handles, so a half-enabled deployment fails loudly at event import (the vLLM side crashes immediately; the server side logs the error and the worker times out after lmcache.mp.mq_timeout).

Note

With isolated IPC enabled on both sides, the MP data path has no /dev/shm dependency left: --ipc host (Docker) and hostIPC: true / the shared /dev/shm mount (Kubernetes) can be dropped. The two containers only need access to the same GPUs and distinct PID values (any regular container setup provides both).

Current limitations:

  • Supported by the vLLM MP connector only. SGLang, TensorRT-LLM, CacheBlend, and qstore still create raw CUDA interprocess events and require isolated IPC to stay off (the default).

  • KV-cache tensors must live in cudaMalloc-style memory. CUDA VMM allocations have no IPC memory handle, so PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True and vLLM’s sleep mode (CuMemAllocator) are incompatible with isolated IPC; registration fails with an explanatory error.

  • Requires the cuda-python package on both sides (included in the CUDA requirement files).

Kubernetes#

LMCache is designed for a DaemonSet + Deployment pattern: one LMCache server per node (DaemonSet) shared by multiple vLLM pods (Deployment).

Example YAML files are provided in examples/multi_process/.

Prerequisites#

  • Kubernetes cluster with GPU support (NVIDIA GPU Operator installed)

  • At least 4 GPUs per node

  • kubectl configured to access your cluster

Step-by-Step#

Step 1: Create namespace

kubectl create namespace multi-process

Step 2: Deploy LMCache DaemonSet

kubectl apply -f examples/multi_process/lmcache-daemonset.yaml

Step 3: Deploy vLLM

kubectl apply -f examples/multi_process/vllm-deployment.yaml

Note

The default model is Qwen/Qwen3-14B. For gated models (e.g., Llama), create a Secret with your Hugging Face token:

kubectl create secret generic vllm-secrets \
  --from-literal=hf_token=your_hf_token_here \
  -n multi-process

Then add the HF_TOKEN environment variable to the vLLM container spec.

Step 4: Monitor deployment

# DaemonSet status
kubectl get daemonset -n multi-process
kubectl get pods -n multi-process -l app=lmcache-server

# vLLM status
kubectl get pods -n multi-process -l app=vllm-deployment -w

# LMCache logs (for a specific node)
VLLM_NODE=$(kubectl get pod -n multi-process -l app=vllm-deployment \
    -o jsonpath='{.items[0].spec.nodeName}')
LMCACHE_POD=$(kubectl get pod -n multi-process -l app=lmcache-server \
    --field-selector spec.nodeName=$VLLM_NODE \
    -o jsonpath='{.items[0].metadata.name}')
kubectl logs -n multi-process $LMCACHE_POD -f

Step 5: Send test requests

kubectl port-forward -n multi-process deployment/vllm-deployment 8000:8000

curl -X POST http://localhost:8000/v1/completions \
    -H "Content-Type: application/json" \
    -d "{
        \"model\": \"Qwen/Qwen3-14B\",
        \"prompt\": \"$(printf 'Explain the significance of KV cache in language models.%.0s' {1..100})\",
        \"max_tokens\": 10
    }"

Architecture Notes#

  • DaemonSet uses ``hostNetwork: true`` so vLLM pods discover the LMCache server via status.hostIP.

  • Both containers mount ``/dev/shm`` from the host to enable CUDA IPC memory sharing (see Isolated IPC above for the plan to remove this requirement).

  • GPUs are NOT requested in the DaemonSet – this allows GPUs to remain exclusively allocated to vLLM pods. The NVIDIA container runtime automatically provides GPU access for IPC-based memory transfers.

  • Multiple vLLM pods on the same node automatically connect to the same LMCache DaemonSet instance.

Note

LMCache pods on nodes without GPUs will crash with CUDA initialization errors. This is expected – LMCache only needs to run on GPU nodes where vLLM pods are scheduled.

Health Checking (HTTP Server)#

For Kubernetes liveness/readiness probes, deploy the HTTP server variant instead. Use the /healthcheck endpoint:

livenessProbe:
  httpGet:
    path: /healthcheck
    port: 8080
  initialDelaySeconds: 10
  periodSeconds: 30
readinessProbe:
  httpGet:
    path: /healthcheck
    port: 8080
  initialDelaySeconds: 5
  periodSeconds: 10

Monitoring Integration#

Prometheus metrics are enabled by default on port 9090. Add a ServiceMonitor or Prometheus scrape annotation to collect metrics from the LMCache DaemonSet pods. See Observability for metric details.

Cleanup#

kubectl delete -f examples/multi_process/vllm-deployment.yaml
kubectl delete -f examples/multi_process/lmcache-daemonset.yaml
kubectl delete namespace multi-process

Production Best Practices#

Worker count (``–max-workers``, ``–max-gpu-workers``, ``–max-cpu-workers``): --max-workers sets both the GPU affinity pool and CPU normal pool sizes (default 1). Use --max-gpu-workers to override the GPU pool independently — set it to at least the number of vLLM instances sharing the cache server so each instance gets its own dedicated thread. Use --max-cpu-workers to override the CPU pool for lookup and other non-GPU operations.

L1 memory sizing (``–l1-size-gb``): Allocate as much CPU memory as available after accounting for the OS and vLLM. A larger L1 cache means fewer L2 round-trips.

Eviction tuning:

  • --eviction-trigger-watermark 0.8 (default) triggers eviction when L1 is 80% full.

  • --eviction-ratio 0.2 (default) frees 20% of allocated memory per eviction cycle.

  • Lower the watermark or increase the ratio if you observe frequent evictions under steady load.

Logging: Use LMCACHE_LOG_LEVEL=DEBUG during initial setup to verify L2 store/load activity. Switch to INFO (default) for production to reduce log volume.

Transfer Mode (--supported-transfer-mode, --shm-name)#

LMCache supports two worker → server transfer paths: an lmcache-driven path (server pulls/pushes via CUDA IPC or CPU SHM, used for STORE/RETRIEVE) and an engine-driven path (PREPARE/COMMIT, used by CPU-only or non-CUDA accelerator workers). The server picks which paths to load via --supported-transfer-mode:

  • lmcache_driven (default) – load only the server-driven transfer path. Supports CUDA devices (IPC) and CPU devices (SHM), and skips allocating the engine-driven prepare/commit resources (pickle codec).

  • engine_driven – load only the engine-driven path. Use when serving CPU-only or non-CUDA accelerator workers.

  • auto – load both paths. Workers of either device type can connect without manual configuration; the server has no upfront knowledge of the connecting worker’s device.

When the engine-driven path is loaded (auto or engine_driven), KV transfers between the server and vLLM workers use a pickle-based path by default. Pass --shm-name with a segment name to create a shared-memory (SHM) pool instead:

Value

Effect

"" (empty string, default)

No SHM pool; KV transfer uses the pickle-based path. Works when /dev/shm is unavailable or when running without --ipc host in Docker.

"my_pool" (any non-empty name)

Create a SHM pool with that exact segment name and use it for KV transfers. The deterministic, human-readable name also helps with monitoring and debugging.

Examples:

# Engine-driven path with pickle transfer (no SHM) -- the default:
lmcache server --l1-size-gb 60 --eviction-policy LRU \
    --supported-transfer-mode engine_driven

# Engine-driven path with a named SHM segment:
lmcache server --l1-size-gb 60 --eviction-policy LRU \
    --supported-transfer-mode engine_driven --shm-name "lmcache_pool"