Docker deployment

Warning

This page documents the behavior of LMCache’s in-process mode (deprecated). Please consider using LMCache MP mode for better feature support and performance. For the MP mode equivalent of this page, see Deployment Guide.

Prerequisites: Docker Engine 27.0+

See Installation for pulling images.

Running the container

IMAGE=<IMAGE_NAME>:<TAG>
mkdir -p "$HOME/.cache/huggingface"
docker run --runtime nvidia --gpus all \
    --env "HF_TOKEN=<REPLACE_WITH_YOUR_HF_TOKEN>" \
    --env "LMCACHE_CHUNK_SIZE=256" \
    --env "LMCACHE_LOCAL_CPU=True" \
    --env "LMCACHE_MAX_LOCAL_CPU_SIZE=5" \
    --volume "$HOME/.cache/huggingface:/home/lmcache/.cache/huggingface" \
    --network host \
    $IMAGE \
    meta-llama/Llama-3.1-8B-Instruct --kv-transfer-config \
    '{"kv_connector":"LMCacheConnectorV1","kv_role":"kv_both"}'

See the docker run example for more details.

The official CUDA vLLM and standalone images run as lmcache (uid 1000). The bind mount above reuses the host’s existing Hugging Face cache. Its directory and contents must be readable and writable by container uid 1000; configure host permissions or ACLs if the cache has different ownership.

For a separate Docker-managed cache, replace the mount with --volume hf-cache:/home/lmcache/.cache/huggingface. A new empty named volume inherits the image’s pre-created cache directory ownership; existing volumes retain their permissions. It does not reuse models in the host’s cache. To reuse an old /root cache, use --user 0 and retain its original mount path.

ROCm (AMD)

The AMD Infinity hub for vLLM offers a prebuilt, optimized image for the AMD Instinct™ MI300X. See LLM inference performance validation on AMD Instinct MI300X for full instructions.

Validated environment: rocm/vllm-dev:nightly_0624_rc2_0624_rc2_20250620, MI300X, vLLM V1.

docker run -it \
    --network=host \
    --group-add=video \
    --ipc=host \
    --cap-add=SYS_PTRACE \
    --security-opt seccomp=unconfined \
    --device /dev/kfd \
    --device /dev/dri \
    -v <path_to_your_models>:/app/model \
    -e HF_HOME="/app/model" \
    --name lmcache_rocm \
    rocm/vllm-dev:nightly_0624_rc2_0624_rc2_20250620 \
    bash

XPU (Intel)

The Intel vLLM XPU hub offers a prebuilt, optimized docker image designed for validating inference performance on the Intel GPU accelerators like ARC770, B60/B70, and future products.

Validated environment: intel/vllm:0.17.0-xpu, Intel B60 GPU, vLLM V1.

docker run --privileged \
    -it --rm --name vllm-xpu \
    -u root \
    --ipc=host --net=host \
    --cap-add=ALL \
    --device /dev/dri:/dev/dri \
    -v /dev/dri/by-path:/dev/dri/by-path \
    --entrypoint /bin/bash intel/vllm:0.17.0-xpu