llm-d Integration#
llm-d is an open-source, Kubernetes-native stack for distributed LLM inference. Its router selects a vLLM pod for each request. LMCache adds a KV cache layer behind those pods. Each vLLM pod stores KV cache in the LMCache server on its node. For a repeated prefix, the pod reads the KV cache from that server.
No change to llm-d is necessary. The integration uses the released llm-d router chart, router image and P/D sidecar.
Architecture#
The llm-d Router (the Endpoint Picker, EPP) gives a score to each model-server pod for each request and selects one pod. In standalone mode, an Envoy sidecar in the EPP pod receives the requests. A Gateway API provider is not necessary.
The InferencePool selects the model-server pods by label. It also gives the port that the router sends requests to.
The model servers are vLLM Deployments. LMCache connects here. Each pod uses the
lmcache/vllm-openaiimage. Each pod gives the LMCache connector config to vLLM with--kv-transfer-config.One LMCache server runs on each GPU node. All vLLM pods on that node use it.
client -> Envoy (8081) -> EPP selects a pod -> vLLM pod (8000)
|
| LMCacheMPConnector (CUDA IPC)
v
LMCache server on the same node
Tested versions#
The tests for this page used one GPU node and these versions:
Component |
Version |
|---|---|
llm-d router chart |
|
llm-d P/D sidecar |
|
Gateway API Inference Extension (InferencePool CRD) |
|
LMCache server and vLLM image |
|
Prerequisites#
A Kubernetes cluster with one GPU node or more, and the NVIDIA GPU Operator. Two GPUs on one node are necessary for P/D disaggregation.
kubectland Helm 3.8 or later.A Hugging Face token, if the model is gated.
Qwen/Qwen3-0.6Bis not gated.
Prefix KV reuse#
Create the namespace#
The LMCache server, the router and the model servers use one namespace. A pod can read a ConfigMap only from its own namespace.
kubectl create namespace llm-d
Deploy the LMCache server#
lmcache_engine.yaml has three objects:
A DaemonSet that runs
lmcache serverwith--isolated-ipcon each GPU node.The
lmcache-engineService withinternalTrafficPolicy: Local. Each vLLM pod connects to the server on its own node, because CUDA IPC operates only on one node.The
lmcache-engine-connectionConfigMap. Itskv-transfer-config.jsonkey contains theLMCacheMPConnectorsettings. The two other keys are for P/D disaggregation.
# LMCache server for llm-d model servers, without the LMCache Operator.
# The operator creates the same three objects for an LMCacheEngine with
# isolated IPC: a DaemonSet, a node-local Service and the connection ConfigMap.
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: lmcache-engine
namespace: llm-d
labels:
app.kubernetes.io/name: lmcache
app.kubernetes.io/instance: lmcache-engine
spec:
selector:
matchLabels:
app.kubernetes.io/name: lmcache
app.kubernetes.io/instance: lmcache-engine
template:
metadata:
labels:
app.kubernetes.io/name: lmcache
app.kubernetes.io/instance: lmcache-engine
spec:
runtimeClassName: nvidia
nodeSelector:
nvidia.com/gpu.present: "true" # GPU nodes only
containers:
- name: lmcache
image: lmcache/vllm-openai:v0.5.6 # the same tag as the vLLM pods
command: ["/opt/venv/bin/lmcache", "server"]
args:
- --host=0.0.0.0
- --port=5555
- --http-port=8080
- --l1-size-gb=20
- --chunk-size=256
- --max-workers=1
- --hash-algorithm=blake3
- --isolated-ipc # KV over driver-level CUDA IPC; no hostIPC
- --eviction-policy=LRU
- --eviction-trigger-watermark=0.80
- --eviction-ratio=0.20
- --prometheus-port=9090
env:
# The server sees every GPU of the node and requests none.
- name: NVIDIA_VISIBLE_DEVICES
value: "all"
- name: NVIDIA_DRIVER_CAPABILITIES
value: "all"
ports:
- name: server
containerPort: 5555
- name: http
containerPort: 8080
# Remove the comment marks if `nvidia-smi -L` fails in this pod.
# securityContext:
# privileged: true
startupProbe:
tcpSocket:
port: 5555
periodSeconds: 10
failureThreshold: 30
resources:
requests:
cpu: "2"
memory: 25Gi # L1 size + 5 Gi
limits:
memory: 38Gi
---
apiVersion: v1
kind: Service
metadata:
name: lmcache-engine
namespace: llm-d
spec:
selector:
app.kubernetes.io/name: lmcache
app.kubernetes.io/instance: lmcache-engine
internalTrafficPolicy: Local # each vLLM pod connects to the server on its own node
ports:
- name: server
port: 5555
- name: http
port: 8080
---
apiVersion: v1
kind: ConfigMap
metadata:
name: lmcache-engine-connection
namespace: llm-d
data:
kv-transfer-config.json: |
{"kv_connector":"LMCacheMPConnector","kv_connector_module_path":"lmcache.integration.vllm.lmcache_mp_connector","kv_role":"kv_both","kv_connector_extra_config":{"lmcache.mp.host":"tcp://lmcache-engine.llm-d.svc.cluster.local","lmcache.mp.port":"5555","lmcache.mp.isolated_ipc":true}}
# The two keys below are for P/D disaggregation only.
kv-transfer-config-prefiller.json: |
{"kv_connector":"MultiConnector","kv_role":"kv_producer","kv_connector_extra_config":{"connectors":[{"kv_connector":"NixlConnector","kv_role":"kv_producer","kv_load_failure_policy":"fail"},{"kv_connector":"LMCacheMPConnector","kv_connector_module_path":"lmcache.integration.vllm.lmcache_mp_connector","kv_role":"kv_both","kv_connector_extra_config":{"lmcache.mp.host":"tcp://lmcache-engine.llm-d.svc.cluster.local","lmcache.mp.port":"5555","lmcache.mp.isolated_ipc":true}}]}}
kv-transfer-config-decoder.json: |
{"kv_connector":"MultiConnector","kv_role":"kv_consumer","kv_connector_extra_config":{"connectors":[{"kv_connector":"NixlConnector","kv_role":"kv_consumer","kv_load_failure_policy":"fail"},{"kv_connector":"LMCacheMPConnector","kv_connector_module_path":"lmcache.integration.vllm.lmcache_mp_connector","kv_role":"kv_both","kv_connector_extra_config":{"lmcache.mp.host":"tcp://lmcache-engine.llm-d.svc.cluster.local","lmcache.mp.port":"5555","lmcache.mp.isolated_ipc":true}}]}}
Apply the manifest from the root of the LMCache repository. Then wait for the server:
kubectl apply -f examples/llm_d_integration/kubernetes/lmcache_engine.yaml
kubectl -n llm-d rollout status ds/lmcache-engine
kubectl -n llm-d logs ds/lmcache-engine | grep "cache server is running"
# LMCache zmq cache server is running on 0.0.0.0:5555
Make sure that each server pod sees the GPUs of its node. The server opens the KV cache of vLLM through CUDA IPC. Thus it must see every GPU. It requests no GPU:
for p in $(kubectl -n llm-d get pod \
-l app.kubernetes.io/instance=lmcache-engine -o name); do
kubectl -n llm-d exec "$p" -- nvidia-smi -L | wc -l
done
Each pod prints the GPU count of its node. If a pod prints an NVML error,
remove the comment marks from securityContext.privileged in the manifest.
Then apply the manifest again.
Note
The lmcache.mp.isolated_ipc value in the ConfigMap must agree with the
--isolated-ipc flag of the server. Refer to the Isolated IPC section
of the Deployment Guide guide. For a cluster without the nvidia
RuntimeClass, refer to GPU Operator NRI/CDI in the same guide.
Install the InferencePool CRD#
The llm-d Router creates an InferencePool. The CRD is in the Gateway API
Inference Extension release that llm-d v1.0.0 uses:
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/releases/download/v1.6.2/v1-manifests.yaml
kubectl wait --for=condition=Established \
crd/inferencepools.inference.networking.k8s.io --timeout=60s
Install the llm-d Router#
The router selects the model-server pods by label. It sends requests to the vLLM port:
# Values for the llm-d-router-standalone chart (v1.0.0).
router:
modelServers:
matchLabels:
llm-d.ai/guide: lmcache # the label of the decode pods in decode.yaml
targetPorts:
- number: 8000 # the vLLM port
epp:
replicas: 1
helm install llm-d oci://ghcr.io/llm-d/charts/llm-d-router-standalone \
--version v1.0.0 --namespace llm-d \
-f examples/llm_d_integration/kubernetes/router_values.yaml --wait
The chart gives the release name to its objects. llm-d-epp is the router
Deployment and Service. llm-d is the InferencePool. Port 8081 of the
Service is the Envoy listener for clients.
Note
The chart requests 8 CPUs for the EPP container and 4 CPUs for the Envoy
container. It requests 8 Gi of memory for each container. For a small
cluster, set router.epp.resources and router.proxy.resources to
lower values.
With router.inferencePool.create: false, the chart creates no
InferencePool. The EPP then selects the pods by label, and the CRD is not
necessary.
Deploy the decode model server#
The llm-d guides deploy their model servers from kustomize overlays.
decode.yaml has the same structure as their decode base. It adds the
LMCache settings:
# vLLM decode model server for the llm-d Router. It connects to the LMCache
# server of lmcache_engine.yaml. The structure is the same as the llm-d
# `decode` base: container `modelserver`, port 8000, the `llm-d.ai/role` label.
apiVersion: apps/v1
kind: Deployment
metadata:
name: decode
namespace: llm-d
labels:
llm-d.ai/role: decode
llm-d.ai/guide: lmcache
spec:
replicas: 1
selector:
matchLabels:
llm-d.ai/role: decode
llm-d.ai/guide: lmcache
template:
metadata:
labels:
llm-d.ai/role: decode
llm-d.ai/guide: lmcache # the router selects this label
spec:
terminationGracePeriodSeconds: 120 # vLLM unregisters its KV cache before it stops
containers:
- name: modelserver
image: lmcache/vllm-openai:v0.5.6
command: ["vllm", "serve"]
args:
- Qwen/Qwen3-0.6B
- --port=8000
- --no-enable-prefix-caching # a repeated prompt goes to LMCache; remove in production
- --kv-transfer-config
- $(LMCACHE_KV_CONFIG)
env:
- name: PYTHONHASHSEED
value: "0"
- name: LMCACHE_KV_CONFIG
valueFrom:
configMapKeyRef:
name: lmcache-engine-connection
key: kv-transfer-config.json
ports:
- name: modelserver
containerPort: 8000
readinessProbe:
httpGet:
path: /health
port: modelserver
initialDelaySeconds: 60
periodSeconds: 15
failureThreshold: 60
resources:
requests:
cpu: "4"
memory: 16Gi
nvidia.com/gpu: "1"
limits:
memory: 32Gi
nvidia.com/gpu: "1"
volumeMounts:
- name: shm
mountPath: /dev/shm
volumes:
- name: shm
emptyDir:
medium: Memory
sizeLimit: 8Gi
kubectl apply -f examples/llm_d_integration/kubernetes/decode.yaml
kubectl -n llm-d rollout status deploy/decode --timeout=15m
Setting |
Reason |
|---|---|
|
The llm-d overlays use |
|
The token hashes must be the same in all processes. |
|
Kubernetes reads the connector JSON from the ConfigMap when the pod starts. It puts the JSON into the vLLM arguments. |
|
|
|
With the prefix cache of vLLM on, vLLM serves a repeated prompt from its own cache. The request then does not go to LMCache. Keep the flag for the test. Remove it in production to use the two cache layers. |
Memory-backed |
The 64 Mi default of the container is too small for the vLLM workers. |
Do a test#
Send requests to the router, not to the model pod:
kubectl -n llm-d port-forward svc/llm-d-epp 8081:8081 &
Send one long prompt two times. The prompt must be longer than one LMCache chunk (256 tokens by default):
PROMPT="$(printf 'Tell me a long story about a brave knight who lived in a faraway kingdom. %.0s' $(seq 1 90))"
for i in 1 2; do
curl -s localhost:8081/v1/completions -H 'Content-Type: application/json' \
-d "{\"model\":\"Qwen/Qwen3-0.6B\",\"prompt\":\"$PROMPT\",\"max_tokens\":16}" \
| jq -r '.choices[0].text'
done
The LMCache server log shows the store operation of the first request. It also shows the retrieve operation of the second request:
kubectl -n llm-d logs -l app.kubernetes.io/instance=lmcache-engine --tail=200 \
| grep -E "Stored|Retrieved"
# Stored 1280 tokens in 0.003 seconds
# Retrieved 1280 tokens in 0.002 seconds
A Registered KV cache for GPU ID line in the same log shows that the
CUDA IPC handshake with vLLM is complete.
P/D disaggregation#
The llm-d Router can send the prefill and the decode of each request to different pods. One LMCache server is the KV cache layer of the two pods:
The router selects a decode pod and a prefill pod for each request. It selects them by the
llm-d.ai/rolelabel.A routing sidecar in the decode pod sends the prompt to the prefill pod with
max_tokens=1.The prefill pod stores the prefix in LMCache. For a repeated prompt, it reads the prefix from LMCache.
The sidecar sends the request to the vLLM of the decode pod. That vLLM gets the KV cache from the prefill pod through NIXL. Then it generates the output.
Refer to Disaggregated Prefill for the connector design. This section uses the namespace and the LMCache server of Prefix KV reuse. Each vLLM pod requests one GPU.
Remove the first router and its decode pod#
Each release of the router chart makes a ConfigMap with the name envoy.
Thus one namespace can contain only one release. The decode pod of
Prefix KV reuse also uses one GPU. If you did that procedure, remove its
router and its decode pod. Keep the LMCache server:
helm -n llm-d uninstall llm-d
kubectl -n llm-d delete -f examples/llm_d_integration/kubernetes/decode.yaml
Install the router with the P/D plugins#
These values agree with the llm-d v1.0.0 pd-disaggregation guide. This
release creates no InferencePool. Its EPP selects the pods by label.
# Values for the llm-d-router-standalone chart (v1.0.0), P/D disaggregation.
# The plugins agree with the llm-d v1.0.0 `pd-disaggregation` guide. The
# router disaggregates every request. The `prefill` and `decode` profiles
# select pods by the `llm-d.ai/role` label.
router:
inferencePool:
create: false # the EPP selects the pods by the label below
modelServers:
matchLabels:
llm-d.ai/guide: lmcache-pd
targetPorts:
- number: 8000 # the prefill vLLM, and the routing sidecar of the decode pod
epp:
replicas: 1
pluginsConfigFile: pd-plugins.yaml
pluginsConfig:
apiVersion: llm-d.ai/v1
kind: EndpointPickerConfig
plugins:
- type: always-disagg-pd-decider
- type: disagg-profile-handler
parameters:
deciders:
prefill: always-disagg-pd-decider
- type: prefill-filter
- type: decode-filter
- type: queue-scorer
- type: max-score-picker
- type: metrics-data-source
parameters:
scheme: http
path: /metrics
insecureSkipVerify: true
- type: core-metrics-extractor
schedulingProfiles:
- name: prefill
plugins:
- pluginRef: prefill-filter
- pluginRef: queue-scorer
- pluginRef: max-score-picker
- name: decode
plugins:
- pluginRef: decode-filter
- pluginRef: queue-scorer
- pluginRef: max-score-picker
helm upgrade --install llm-d-pd oci://ghcr.io/llm-d/charts/llm-d-router-standalone \
--version v1.0.0 --namespace llm-d \
-f examples/llm_d_integration/kubernetes/pd_router_values.yaml --wait
Deploy the prefill and decode pods#
Each pod reads the config of its role from the connection ConfigMap. Each
config is a MultiConnector with NixlConnector first and
LMCacheMPConnector second. The decode pod also runs the llm-d routing
sidecar. The sidecar listens on port 8000 and sends requests to vLLM on port
8200.
# Prefill and decode model servers for llm-d P/D disaggregation. Each pod
# reads the connector config of its role from the ConfigMap of
# lmcache_engine.yaml.
apiVersion: apps/v1
kind: Deployment
metadata:
name: prefill
namespace: llm-d
labels:
llm-d.ai/role: prefill
llm-d.ai/guide: lmcache-pd
spec:
replicas: 1
selector:
matchLabels:
llm-d.ai/role: prefill
llm-d.ai/guide: lmcache-pd
template:
metadata:
labels:
llm-d.ai/role: prefill
llm-d.ai/guide: lmcache-pd
spec:
terminationGracePeriodSeconds: 120
containers:
- name: modelserver
image: lmcache/vllm-openai:v0.5.6
command: ["vllm", "serve"]
args:
- Qwen/Qwen3-0.6B
- --port=8000
- --no-enable-prefix-caching # a repeated prompt goes to LMCache
- --disable-hybrid-kv-cache-manager
- --kv-transfer-config
- $(LMCACHE_KV_CONFIG)
env:
- name: PYTHONHASHSEED
value: "0"
- name: LMCACHE_KV_CONFIG
valueFrom:
configMapKeyRef:
name: lmcache-engine-connection
key: kv-transfer-config-prefiller.json
- name: VLLM_NIXL_SIDE_CHANNEL_HOST
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: VLLM_HTTP_TIMEOUT_KEEP_ALIVE
value: "120"
ports:
- name: modelserver
containerPort: 8000
- name: nixl
containerPort: 5600
readinessProbe:
httpGet:
path: /health
port: 8000
initialDelaySeconds: 60
periodSeconds: 15
failureThreshold: 60
resources:
requests:
cpu: "4"
memory: 16Gi
nvidia.com/gpu: "1"
limits:
memory: 32Gi
nvidia.com/gpu: "1"
volumeMounts:
- name: shm
mountPath: /dev/shm
volumes:
- name: shm
emptyDir:
medium: Memory
sizeLimit: 8Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: decode-pd
namespace: llm-d
labels:
llm-d.ai/role: decode
llm-d.ai/guide: lmcache-pd
spec:
replicas: 1
selector:
matchLabels:
llm-d.ai/role: decode
llm-d.ai/guide: lmcache-pd
template:
metadata:
labels:
llm-d.ai/role: decode
llm-d.ai/guide: lmcache-pd
spec:
terminationGracePeriodSeconds: 120
initContainers:
- name: routing-proxy # native sidecar: runs for the life of the pod
image: ghcr.io/llm-d/llm-d-router-disagg-sidecar:v1.0.0
restartPolicy: Always
args:
- --port=8000
- --model-server-port=8200
- --kv-connector=nixlv2
- --secure-serving=false
ports:
- name: sidecar
containerPort: 8000
containers:
- name: modelserver
image: lmcache/vllm-openai:v0.5.6
command: ["vllm", "serve"]
args:
- Qwen/Qwen3-0.6B
- --port=8200
- --enable-prompt-tokens-details # the response shows cached_tokens
- --disable-hybrid-kv-cache-manager
- --kv-transfer-config
- $(LMCACHE_KV_CONFIG)
env:
- name: PYTHONHASHSEED
value: "0"
- name: LMCACHE_KV_CONFIG
valueFrom:
configMapKeyRef:
name: lmcache-engine-connection
key: kv-transfer-config-decoder.json
- name: VLLM_NIXL_SIDE_CHANNEL_HOST
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: VLLM_HTTP_TIMEOUT_KEEP_ALIVE
value: "120"
ports:
- name: vllm
containerPort: 8200
- name: nixl
containerPort: 5600
readinessProbe:
httpGet:
path: /health
port: 8200
initialDelaySeconds: 60
periodSeconds: 15
failureThreshold: 60
resources:
requests:
cpu: "4"
memory: 16Gi
nvidia.com/gpu: "1"
limits:
memory: 32Gi
nvidia.com/gpu: "1"
volumeMounts:
- name: shm
mountPath: /dev/shm
volumes:
- name: shm
emptyDir:
medium: Memory
sizeLimit: 8Gi
kubectl apply -f examples/llm_d_integration/kubernetes/pd_pods.yaml
kubectl -n llm-d rollout status deploy/prefill --timeout=20m
kubectl -n llm-d rollout status deploy/decode-pd --timeout=20m
Setting |
Reason |
|---|---|
|
It is necessary for |
|
The NIXL side channel must listen on the pod IP, not on localhost. |
|
A repeated prompt goes to LMCache. Thus the server log shows the retrieve operation. Remove the flag in production to use the two cache layers. |
|
The sidecar keeps idle connections for 90 s. A shorter keep-alive time of vLLM causes connection resets. |
|
The response then shows |
Do a P/D test#
Send one long prompt two times through the P/D router:
kubectl -n llm-d port-forward svc/llm-d-pd-epp 8082:8081 &
PROMPT="$(printf 'Tell me a long story about a brave knight who lived in a faraway kingdom. %.0s' $(seq 1 90))"
for i in 1 2; do
curl -s localhost:8082/v1/completions -H 'Content-Type: application/json' \
-d "{\"model\":\"Qwen/Qwen3-0.6B\",\"prompt\":\"$PROMPT\",\"max_tokens\":16}" \
| jq '.usage.prompt_tokens_details.cached_tokens'
done
# 0 first request
# 1280 second request (the number changes with the prompt length)
Then examine each part of the path:
# The decode pod got the KV cache through NIXL.
kubectl -n llm-d logs deploy/decode-pd -c modelserver \
| grep "NIXL compatibility check passed"
# The prefill pod stored the prefix. Then it read the prefix for the second request.
kubectl -n llm-d logs -l app.kubernetes.io/instance=lmcache-engine --tail=200 \
| grep -E "Stored|Retrieved"
# Stored 1280 tokens in 0.004 seconds
# Retrieved 1280 tokens in 0.001 seconds
# Each pod served the two requests (vLLM request counters).
for r in prefill:8000 decode:8200; do
p=$(kubectl -n llm-d get pod \
-l llm-d.ai/guide=lmcache-pd,llm-d.ai/role=${r%%:*} \
-o jsonpath='{.items[0].metadata.name}')
echo "${r%%:*}: $(kubectl get --raw \
/api/v1/namespaces/llm-d/pods/$p:${r##*:}/proxy/metrics \
| awk '/^vllm:request_success_total/ {s+=$2} END {print s}') requests"
done
# prefill: 2 requests
# decode: 2 requests
Prefix-aware routing with two or more pods#
The prefix-cache-scorer of the router compares token blocks. Thus the EPP
must tokenize each prompt. The default plugins of the chart have no tokenizer.
Without a tokenizer, the scorer gives 0 to every pod. The router then does not
send a repeated prefix to the pod that has it.
Add the tokenizer sidecar of the chart and the token-producer plugin to
router_values.yaml. Use the model name of the decode pods. The router and
vLLM then tokenize in the same way:
router:
tokenizer:
enabled: true
flavor: python
modelName: Qwen/Qwen3-0.6B
port: 8000 # loopback port in the EPP pod
env: [] # put HF_TOKEN here for a gated model
resources:
requests: { cpu: "1", memory: 2Gi }
limits: { memory: 4Gi }
epp:
pluginsConfigFile: lmcache-plugins.yaml
pluginsConfig: # the default plugins of the chart and the token-producer
apiVersion: llm-d.ai/v1
kind: EndpointPickerConfig
plugins:
- type: token-producer
parameters:
modelName: Qwen/Qwen3-0.6B
vllm: { url: http://localhost:8000, timeout: 10s }
- type: queue-scorer
- type: kv-cache-utilization-scorer
- type: prefix-cache-scorer
- type: metrics-data-source
parameters: { scheme: http, path: /metrics, insecureSkipVerify: true }
- type: core-metrics-extractor
schedulingProfiles:
- name: default
plugins:
- { pluginRef: queue-scorer, weight: 2 }
- { pluginRef: kv-cache-utilization-scorer, weight: 2 }
- { pluginRef: prefix-cache-scorer, weight: 3 }
A test with two decode pods on one node used these values. Five repeats of one
prompt went to one pod. The scorer gave 1 to that pod and 0 to the other pod.
The other pod then got the same prompt directly. It read the prefix from the
LMCache server (Retrieved in the server log), because the two pods use one
server.
The scorer is approximate. It records the pod that received each prefix. It has no data about the KV cache in the LMCache server.
Use the LMCache Operator#
The LMCache Operator creates the same DaemonSet, Service
and ConfigMap from an LMCacheEngine resource. The ConfigMap name is
<engine-name>-connection. With spec.pd, the ConfigMap also has the
prefiller and decoder keys. Use that ConfigMap name in decode.yaml and
pd_pods.yaml.
The injection webhook of the operator does not change a container that sets
command. The llm-d model-server overlays set
command: ["vllm", "serve"]. Thus this page sets the connector config in
the Deployment. The tests of Prefix KV reuse and P/D disaggregation used
an LMCache server as in lmcache_engine.yaml.
Troubleshooting#
Symptom |
Cause |
Fix |
|---|---|---|
|
No model-server pod is Ready. |
Wait until the decode Deployment is Ready. |
503 continues, and the EPP log shows no endpoints. |
|
Use the same |
|
The InferencePool CRD is not installed. |
Install the CRD, or set |
|
A different release of the router chart is in the namespace. |
Uninstall that release (refer to Remove the first router and its decode pod). |
vLLM stops at |
The LMCache server pod does not see the GPUs of the node. |
Set |
The LMCache server log shows no |
The prompt is shorter than one chunk, or the request went to a server on a different node. |
Use a longer prompt. Read the log of the server on the node of the vLLM pod. |
Cleanup#
Delete the model servers and the router release that are in the namespace. Then delete the LMCache server and the namespace:
kubectl -n llm-d delete --ignore-not-found \
-f examples/llm_d_integration/kubernetes/pd_pods.yaml \
-f examples/llm_d_integration/kubernetes/decode.yaml
for r in llm-d-pd llm-d; do
if helm -n llm-d status "$r" >/dev/null 2>&1; then
helm -n llm-d uninstall "$r"
fi
done
kubectl delete -f examples/llm_d_integration/kubernetes/lmcache_engine.yaml
kubectl delete namespace llm-d