部署指南#

本页介绍如何在 Docker 和 Kubernetes 环境中部署 LMCache 多进程模式，并提供生产环境的最佳实践建议。

Docker#

LMCache 容器：

docker run --runtime nvidia --gpus all \
    --network host \
    --ipc host \
    lmcache/standalone:nightly \
    /opt/venv/bin/lmcache server \
    --l1-size-gb 60 --eviction-policy LRU --max-workers 4 --port 6555

vLLM 容器：

docker run --runtime nvidia --gpus all \
    --network host \
    --ipc host \
    lmcache/vllm-openai:latest-nightly \
    Qwen/Qwen3-14B \
    --kv-transfer-config \
    '{"kv_connector":"LMCacheMPConnector", "kv_role":"kv_both", "kv_connector_extra_config": {"lmcache.mp.port": 6555}}'

所需的 Docker 标志：

  • --network host -- 允许 vLLM 容器访问本地主机上的 LMCache。

  • --ipc host -- Required for CUDA IPC shared memory transfers between containers in the default (legacy) mode; see Isolated IPC below for running without it.

  • --runtime nvidia --gpus all -- 通过 NVIDIA 容器运行时访问 GPU。

HTTP 服务器变体：

对于健康检查和缓存管理 API 支持（在容器编排器中非常有用），请使用 HTTP 服务器入口点：

docker run --runtime nvidia --gpus all \
    --network host \
    --ipc host \
    lmcache/standalone:nightly \
    /opt/venv/bin/lmcache server \
    --l1-size-gb 60 --eviction-policy LRU --max-workers 4 --port 6555

Isolated IPC (running without --ipc host)#

CUDA IPC in MP mode has two legs, and by default both depend on a shared /dev/shm tmpfs -- which is what --ipc host / hostIPC: true really provides:

  • KV-cache memory sharing: the default registration path uses PyTorch storage IPC, which keeps a reference-counter file in /dev/shm.

  • Event ordering (the per-STORE/RETRIEVE device events): CUDA interprocess event handles only resolve when both containers share a /dev/shm tmpfs.

The isolated IPC setting removes both: KV-cache registration switches to raw CUDA IPC memory handles (docs/design/v1/platform/devices/cuda/ipc_wrapper.md) and event ordering to timeline-semaphore events carried over CUDA IPC memory handles (docs/design/v1/platform/devices/cuda/timeline_semaphore_event_ipc.md). Both rendezvous in the kernel driver, so they work across containers that share nothing -- no host IPC namespace, no common /dev/shm, no --ipc host. It must be enabled on both sides of a deployment:

# LMCache server
lmcache server --isolated-ipc --l1-size-gb 60 --eviction-policy LRU

# vLLM
vllm serve Qwen/Qwen3-14B --kv-transfer-config \
    '{"kv_connector":"LMCacheMPConnector", "kv_role":"kv_both",
      "kv_connector_extra_config": {"lmcache.mp.port": 6555,
                                    "lmcache.mp.isolated_ipc": true}}'

The two event mechanisms exchange incompatible handles, so a half-enabled deployment fails loudly at event import (the vLLM side crashes immediately; the server side logs the error and the worker times out after lmcache.mp.mq_timeout).

备注

With isolated IPC enabled on both sides, the MP data path has no /dev/shm dependency left: --ipc host (Docker) and hostIPC: true / the shared /dev/shm mount (Kubernetes) can be dropped. The two containers only need access to the same GPUs and distinct PID values (any regular container setup provides both). The Kubernetes Kubernetes Operator wires isolated IPC by default on NVIDIA (spec.isolatedIPC).

Current limitations:

  • Supported by the vLLM MP connector only. SGLang, TensorRT-LLM, CacheBlend, and qstore still create raw CUDA interprocess events and require isolated IPC to stay off (the default).

  • KV-cache tensors must live in cudaMalloc-style memory. CUDA VMM allocations have no IPC memory handle, so PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True and vLLM's sleep mode (CuMemAllocator) are incompatible with isolated IPC; registration fails with an explanatory error.

  • Requires the cuda-python package on both sides (included in the CUDA requirement files).

Kubernetes#

LMCache 设计为 DaemonSet + Deployment 模式：每个节点一个 LMCache 服务器（DaemonSet），由多个 vLLM pod（Deployment）共享。

在 examples/multi_process/ 中提供了示例 YAML 文件。

先决条件#

  • 支持 GPU 的 Kubernetes 集群（安装了 NVIDIA GPU Operator）

  • 每个节点至少需要 4 个 GPU

  • kubectl 配置为访问您的集群

Classic GPU Operator installs register RuntimeClass nvidia. CDI+NRI installs often have no RuntimeClass objects. On those clusters omit runtimeClassName on the LMCache DaemonSet and request a management CDI device (see GPU Operator NRI/CDI below). The operator path is GPU Operator NRI/CDI.

逐步指南#

步骤 1：创建命名空间

kubectl create namespace multi-process

步骤 2：部署 LMCache DaemonSet

kubectl apply -f examples/multi_process/lmcache-daemonset.yaml

步骤 3：部署 vLLM

kubectl apply -f examples/multi_process/vllm-deployment.yaml

备注

默认模型是 Qwen/Qwen3-14B。对于受限模型（例如 Llama），请使用您的 Hugging Face Token 创建一个 Secret：

kubectl create secret generic vllm-secrets \
  --from-literal=hf_token=your_hf_token_here \
  -n multi-process

然后将 HF_TOKEN 环境变量添加到 vLLM 容器规格中。

步骤 4：监控部署

# DaemonSet status
kubectl get daemonset -n multi-process
kubectl get pods -n multi-process -l app=lmcache-server

# vLLM status
kubectl get pods -n multi-process -l app=vllm-deployment -w

# LMCache logs (for a specific node)
VLLM_NODE=$(kubectl get pod -n multi-process -l app=vllm-deployment \
    -o jsonpath='{.items[0].spec.nodeName}')
LMCACHE_POD=$(kubectl get pod -n multi-process -l app=lmcache-server \
    --field-selector spec.nodeName=$VLLM_NODE \
    -o jsonpath='{.items[0].metadata.name}')
kubectl logs -n multi-process $LMCACHE_POD -f

步骤 5：发送测试请求

kubectl port-forward -n multi-process deployment/vllm-deployment 8000:8000

curl -X POST http://localhost:8000/v1/completions \
    -H "Content-Type: application/json" \
    -d "{
        \"model\": \"Qwen/Qwen3-14B\",
        \"prompt\": \"$(printf 'Explain the significance of KV cache in language models.%.0s' {1..100})\",
        \"max_tokens\": 10
    }"

架构说明#

  • DaemonSet 使用 ``hostNetwork: true`` 以便 vLLM pod 通过 status.hostIP 发现 LMCache 服务器。

  • Both containers mount ``/dev/shm`` from the host to enable CUDA IPC memory sharing in the default (legacy) mode; see Isolated IPC above for running without it.

  • GPUs are NOT requested in the DaemonSet -- this allows GPUs to remain exclusively allocated to vLLM pods. On classic GPU Operator installs the NVIDIA container runtime (RuntimeClass nvidia) provides GPU access for IPC-based memory transfers. On CDI+NRI installs there is no nvidia RuntimeClass; omit it and request a management CDI device instead (see below).

  • 多个 vLLM pod 在同一节点上会自动连接到同一个 LMCache DaemonSet 实例。

GPU Operator NRI/CDI#

If kubectl get runtimeclass is empty, do not set runtimeClassName on the DaemonSet. Request the management CDI device on the pod (the container name must match the annotation key):

metadata:
  annotations:
    nvidia.cdi.k8s.io/container.lmcache-server: management.nvidia.com/gpu=all

If the DaemonSet namespace is not the GPU Operator install namespace, NVIDIA Container Toolkit ≥ v1.20.0 takes extra namespaces from NRI_MANAGEMENT_CDI_DEVICE_NAMESPACES (Helm toolkit.env, or ClusterPolicy spec.toolkit.env). (NVIDIA: Requesting a Management CDI Device) Below v1.20.0 that env is not available, so the DaemonSet can see GPUs only in the toolkit install namespace (typically gpu-operator).

For reference, toolkit v1.20.0 became the GPU Operator Helm default in v26.7.0. The default allowed namespace is the toolkit install namespace; from 1.20, additional namespaces can be listed in NRI_MANAGEMENT_CDI_DEVICE_NAMESPACES, and the toolkit namespace remains allowed.

备注

没有 GPU 的节点上的 LMCache pod 会因 CUDA 初始化错误而崩溃。这是预期的——LMCache 只需要在调度有 vLLM pod 的 GPU 节点上运行。

健康检查 (HTTP 服务器)#

若要配置 Kubernetes 存活探针/就绪探针，请部署 HTTP 服务器变体，并使用 /healthcheck 端点：

livenessProbe:
  httpGet:
    path: /healthcheck
    port: 8080
  initialDelaySeconds: 10
  periodSeconds: 30
readinessProbe:
  httpGet:
    path: /healthcheck
    port: 8080
  initialDelaySeconds: 5
  periodSeconds: 10

监控集成#

默认情况下，Prometheus 指标在 9090 端口启用。添加 ServiceMonitor 或 Prometheus 抓取注释以收集 LMCache DaemonSet Pod 的指标。有关指标详细信息，请参见 可观察性。

清理#

kubectl delete -f examples/multi_process/vllm-deployment.yaml
kubectl delete -f examples/multi_process/lmcache-daemonset.yaml
kubectl delete namespace multi-process

生产最佳实践#

工作线程数 (``--max-workers``, ``--max-gpu-workers``, ``--max-cpu-workers``): --max-workers 同时设置 GPU 亲和性池和 CPU 普通池的大小（默认值为 1）。使用 --max-gpu-workers 可单独覆盖 GPU 池大小——建议将其设置为至少等于共享同一缓存服务器的 vLLM 实例数量，以便每个实例拥有专属的工作线程。使用 --max-cpu-workers 可覆盖用于查找及其他非 GPU 操作的 CPU 池大小。

L1 内存大小 (``--l1-size-gb``): 在考虑操作系统和 vLLM 后，分配尽可能多的 CPU 内存。更大的 L1 缓存意味着更少的 L2 往返。

逐出调优:

  • --eviction-trigger-watermark 0.8（默认值）在 L1 达到 80% 满时触发逐出。

  • --eviction-ratio 0.2（默认值）在每个逐出周期释放 20% 的分配内存。

  • 如果在稳定负载下观察到频繁的逐出，请降低水位线或增加比率。

日志记录: 在初始设置期间使用 LMCACHE_LOG_LEVEL=DEBUG 来验证 L2 存储/加载活动。 在生产中切换到 INFO（默认）以减少日志量。

传输模式 (--supported-transfer-mode, --shm-name)#

LMCache 支持两条工作节点 → 服务器传输路径：一种是 lmcache 驱动 路径（服务器通过 CUDA IPC 或 CPU SHM 拉取/推送，用于 STORE/RETRIEVE），另一种是 引擎驱动 路径（PREPARE/COMMIT，仅由 CPU 或非 CUDA 加速器工作节点使用）。服务器通过 --supported-transfer-mode 选择加载哪些路径：

  • lmcache_driven (default) -- load only the server-driven transfer path. Supports CUDA devices (IPC) and CPU devices (SHM), and skips allocating the engine-driven prepare/commit resources (pickle codec).

  • engine_driven -- 仅加载引擎驱动路径。适用于仅使用 CPU 或非 CUDA 加速器的工作节点场景。

  • auto -- load both paths. Workers of either device type can connect without manual configuration; the server has no upfront knowledge of the connecting worker's device.

When the engine-driven path is loaded (auto or engine_driven), KV transfers between the server and vLLM workers use a pickle-based path by default. Pass --shm-name with a segment name to create a shared-memory (SHM) pool instead:

值

效果

"" (empty string, default)

No SHM pool; KV transfer uses the pickle-based path. Works when /dev/shm is unavailable or when running without --ipc host in Docker.

"my_pool"（任何非空名称）

Create a SHM pool with that exact segment name and use it for KV transfers. The deterministic, human-readable name also helps with monitoring and debugging.

示例：

# Engine-driven path with pickle transfer (no SHM) -- the default:
lmcache server --l1-size-gb 60 --eviction-policy LRU \
    --supported-transfer-mode engine_driven

# Engine-driven path with a named SHM segment:
lmcache server --l1-size-gb 60 --eviction-policy LRU \
    --supported-transfer-mode engine_driven --shm-name "lmcache_pool"