Quickstart#

This guide helps you get LMCache running end-to-end in a couple of minutes. Use the tabs below to switch the engine. Steps are the same; only the libraries and launch commands change.

vLLM

Install LMCache

uv venv --python 3.12
source .venv/bin/activate
uv pip install lmcache vllm

LMCache supports two deployment modes with vLLM:

Multiprocess (MP) mode – recommended. LMCache runs as a standalone service and vLLM attaches via LMCacheMPConnector. Scales better, exposes management/observability endpoints, and supports sharing one cache across multiple engine instances.
In-process mode – LMCache runs inside the vLLM process via LMCacheConnectorV1. Single command, convenient for quick single-node experiments.

MP mode (recommended)

Start the LMCache server. --host / --port set the request address vLLM connects to; --transport selects ZMQ (the default) or gRPC. The defaults are spelled out here so the two commands line up:

# chunk-size 16 is an illustrative demo value so a short
# prompt produces visible cache traffic; use the default
# (256) in production.
lmcache server \
    --host localhost --port 5555 \
    --l1-size-gb 20 --eviction-policy LRU --chunk-size 16

The request port (--port, default 5555) accepts connections from vLLM; the HTTP frontend (default 8080) serves the management and metrics endpoints. See Configuration Reference for the full list of lmcache server and connector options.

Start vLLM with the MP connector in a separate terminal. Point the connector at the server above via lmcache.mp.host / lmcache.mp.port in kv_connector_extra_config. The host may include a transport prefix: tcp:// for ZMQ or grpc:// for gRPC. A bare host/host:port is also accepted and is normalized to tcp:// automatically. See Request Transport for supported endpoint schemes and transport-selection behavior:

vllm serve Qwen/Qwen3-8B \
    --port 8000 --kv-transfer-config \
    '{"kv_connector":"LMCacheMPConnector", "kv_role":"kv_both", "kv_connector_extra_config": {"lmcache.mp.host": "localhost", "lmcache.mp.port": 5555}}'

Note

Where does LMCacheMPConnector resolve to? This depends on your vLLM version:

vLLM < 0.20.0 – "kv_connector":"LMCacheMPConnector" always resolves to vLLM’s built-in vllm.distributed.kv_transfer.kv_connector.v1.LMCacheMPConnector; there is no way to redirect it to the LMCache-shipped implementation.
vLLM >= 0.20.0 – "kv_connector":"LMCacheMPConnector" still defaults to vLLM’s built-in connector, but you can opt in to the LMCache-shipped implementation (lmcache.integration.vllm.lmcache_mp_connector) by adding kv_connector_module_path:
```
vllm serve Qwen/Qwen3-8B \
    --port 8000 --kv-transfer-config \
    '{"kv_connector":"LMCacheMPConnector", "kv_connector_module_path":"lmcache.integration.vllm.lmcache_mp_connector", "kv_role":"kv_both"}'
```
The LMCache-shipped connector tracks the latest LMCache server protocol and ships fixes/features ahead of the version vendored into vLLM, so prefer it whenever you are on vLLM 0.20.0 or newer.

Test – open a new terminal and send two requests whose prompts share a prefix:

First request

curl http://localhost:8000/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3-8B",
    "prompt": "Qwen3 is the latest generation of large language models in Qwen series, offering a comprehensive suite of dense and mixture-of-experts",
    "max_tokens": 100,
    "temperature": 0.7
  }'

Second request

curl http://localhost:8000/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3-8B",
    "prompt": "Qwen3 is the latest generation of large language models in Qwen series, offering a comprehensive suite of dense and mixture-of-experts (MoE) models",
    "max_tokens": 100,
    "temperature": 0.7
  }'

You should see LMCache logs like this – in MP mode the store/retrieve logs come from the standalone lmcache server process, one entry per chunk.

First request – cache is empty, so every aligned chunk is offloaded:

[2026-04-22 19:49:56,316] LMCache INFO: Stored 16 tokens in 0.023 seconds (server.py:390:lmcache.v1.multiprocess.server)
[2026-04-22 19:49:56,555] LMCache INFO: Stored 16 tokens in 0.005 seconds (server.py:390:lmcache.v1.multiprocess.server)
[2026-04-22 19:49:56,691] LMCache INFO: Stored 16 tokens in 0.005 seconds (server.py:390:lmcache.v1.multiprocess.server)
...

Second request – the shared prefix is retrieved from CPU RAM; only the new tail is stored:

[2026-04-22 19:50:04,686] LMCache INFO: Retrieved 16 tokens in 0.003 seconds (server.py:573:lmcache.v1.multiprocess.server)
[2026-04-22 19:50:04,832] LMCache INFO: Stored 16 tokens in 0.005 seconds (server.py:390:lmcache.v1.multiprocess.server)
[2026-04-22 19:50:04,968] LMCache INFO: Stored 16 tokens in 0.005 seconds (server.py:390:lmcache.v1.multiprocess.server)
...

For request-level statistics (hit ratio, bytes transferred) see Observability.

In-process mode

Start vLLM with LMCache embedded in the engine process:

# The chunk size here is only for illustration purpose, use default one (256) later
LMCACHE_CHUNK_SIZE=8 \
vllm serve Qwen/Qwen3-8B \
    --port 8000 --kv-transfer-config \
    '{"kv_connector":"LMCacheConnectorV1", "kv_role":"kv_both"}'

Note

To customize further, create a config file. See Configuring LMCache for all options.

Alternative simpler command:

vllm serve <MODEL NAME> \
    --kv-offloading-backend lmcache \
    --kv-offloading-size <SIZE IN GB> \
    --disable-hybrid-kv-cache-manager

The --disable-hybrid-kv-cache-manager flag is mandatory. All configuration options from the Configuring LMCache page still apply.

Test – open a new terminal and send two requests whose prompts share a prefix:

First request

curl http://localhost:8000/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3-8B",
    "prompt": "Qwen3 is the latest generation of large language models in Qwen series, offering a comprehensive suite of dense and mixture-of-experts",
    "max_tokens": 100,
    "temperature": 0.7
  }'

Second request

curl http://localhost:8000/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3-8B",
    "prompt": "Qwen3 is the latest generation of large language models in Qwen series, offering a comprehensive suite of dense and mixture-of-experts (MoE) models",
    "max_tokens": 100,
    "temperature": 0.7
  }'

You should see LMCache logs like this – in-process mode emits the logs inline with the vLLM engine core.

First request – prompt is offloaded to LMCache:

(EngineCore_DP0 pid=458469) [2025-09-30 00:08:43,982] LMCache INFO: Stored 31 out of total 31 tokens. size: 0.0040 gb, cost 1.95 ms, throughput: 1.98 GB/s; offload_time: 1.88 ms, put_time: 0.07 ms

Second request – hits the cache and stores the new tail:

Reqid: cmpl-6709d8795d3c4464b01999c9f3fffede-0, Total tokens 32, LMCache hit tokens: 24, need to load: 8
(EngineCore_DP0 pid=494270) [2025-09-30 01:12:36,502] LMCache INFO: Retrieved 8 out of 24 required tokens (from 32 total tokens). size: 0.0011 gb, cost 0.55 ms, throughput: 1.98 GB/s;
(EngineCore_DP0 pid=494270) [2025-09-30 01:12:36,509] LMCache INFO: Storing KV cache for 8 out of 32 tokens (skip_leading_tokens=24)
(EngineCore_DP0 pid=494270) [2025-09-30 01:12:36,510] LMCache INFO: Stored 8 out of total 8 tokens. size: 0.0011 gb, cost 0.43 ms, throughput: 2.57 GB/s; offload_time: 0.40 ms, put_time: 0.03 ms

Total tokens 32: The new prompt has 32 tokens after tokenization.
LMCache hit tokens: 24: 24 tokens (full 8-token chunks) were found in the cache from the first request that stored 31 tokens.
Need to load: 8: vLLM auto prefix caching uses block size 16; 16 tokens already sit in GPU RAM, so LMCache only loads 24-16=8.
Why 24 hit tokens instead of 31? LMCache hashes every 8 tokens (8, 16, 24, 31). It matches page-aligned chunks, so it uses the 24-token hash.
Stored another 8 tokens: The new 8 tokens form a full chunk and are stored for future reuse.

SGLang

Important

The SGLang integration currently supports only MP (multi-process) mode. LMCache must run as a standalone server; in-process mode is not supported. Use an SGLang version that contains SGLang PR #38652 and an LMCache version that contains LMCache PR #4828.

Install SGLang

uv venv --python 3.12
source .venv/bin/activate
uv pip install --prerelease=allow lmcache "sglang"

Configure the LMCache connection

cat > lmcache_config.yaml <<'EOF'
mp_host: 127.0.0.1
mp_port: 5555
EOF

mp_host and mp_port identify the standalone LMCache request server. A bare host uses the default ZMQ transport.

Start the LMCache server

# A small chunk size makes cache reuse visible with the short demo
# prompts below. Use the default (256) for production workloads.
lmcache server \
  --host 127.0.0.1 --port 5555 \
  --l1-size-gb 10 --eviction-policy LRU --chunk-size 16

To use gRPC instead of ZMQ, include the grpc:// scheme in the connector configuration:

cat > lmcache_config.yaml <<'EOF'
mp_host: grpc://127.0.0.1
mp_port: 5555
EOF

Then start the LMCache server with the matching transport:

lmcache server \
  --host 127.0.0.1 --port 5555 --transport grpc \
  --l1-size-gb 10 --eviction-policy LRU --chunk-size 16

Warning

When serving a hybrid model, add --separate-object-groups to the LMCache server command. This option is required so full-attention, sliding-window, and recurrent-state cache groups retain their distinct storage semantics. Omitting it can restore incompatible KV or state data and cause accuracy problems.

Start SGLang in a separate terminal

python -m sglang.launch_server \
  --model-path Qwen/Qwen3-8B \
  --host 0.0.0.0 \
  --port 30000 \
  --enable-lmcache \
  --lmcache-config-file "$PWD/lmcache_config.yaml"

The SGLang process owns the GPU-resident radix cache and connects to the LMCache server for external lookup, retrieve, and store operations. The server independently manages its CPU and remote cache tiers.

Test – open a new terminal and send two requests whose prompts share a prefix. Clear SGLang’s local radix cache between them so the second request must retrieve the shared prefix from LMCache.

First request

curl http://localhost:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3-8B",
    "messages": [{"role": "user", "content": "Qwen3 is the latest generation of large language models in Qwen series, offering a comprehensive suite of dense and mixture-of-experts"}],
    "max_tokens": 100,
    "temperature": 0.7
  }'

Clear the local SGLang radix cache

sleep 2  # allow the asynchronous LMCache store to finish
curl -X POST "http://localhost:30000/flush_cache?timeout=60"

Second request

curl http://localhost:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3-8B",
    "messages": [{"role": "user", "content": "Qwen3 is the latest generation of large language models in Qwen series, offering a comprehensive suite of dense and mixture-of-experts (MoE) models"}],
    "max_tokens": 100,
    "temperature": 0.7
  }'

The first request populates LMCache. After the local radix cache is cleared, the second request retrieves its shared, chunk-aligned prefix from the standalone LMCache server. You can confirm the retrieve through the server metrics endpoint:

curl -s http://localhost:8080/metrics \
  | grep lmcache_mp_l1_read_chunks_total

TensorRT-LLM

Note

This integration depends on the connector preset registry from NVIDIA/TensorRT-LLM PR #12626 and the matching LMCache adapter, neither of which has shipped in a stable release yet. Until they do, install both from source:

uv venv --python 3.12
source .venv/bin/activate

# LMCache from source (dev branch)
uv pip install git+https://github.com/LMCache/LMCache.git@dev

# TensorRT-LLM from source — see NVIDIA's build guide:
# https://nvidia.github.io/TensorRT-LLM/installation/build-from-source-linux.html

Once both ship in a stable release, the install command will be:

uv pip install lmcache "tensorrt_llm>=<version>" \
    --extra-index-url https://pypi.nvidia.com

LMCache integrates with TensorRT-LLM via TRT-LLM’s KV Cache Connector API and supports two deployment modes:

In-process mode (connector: lmcache) – LMCache runs as a singleton inside the TRT-LLM process. Simplest setup; no extra service to manage.
MP mode (connector: lmcache-mp) – LMCache runs as a standalone server. Multiple TRT-LLM workers on the same node can share the cache, and the cache survives a TRT-LLM crash.

In-process mode

Configure LMCache via env vars:

export PYTHONHASHSEED=0  # required — chunk hashing depends on stable hash()
export LMCACHE_CHUNK_SIZE=256
export LMCACHE_LOCAL_CPU=True
export LMCACHE_MAX_LOCAL_CPU_SIZE=2.0  # GiB

Build the TRT-LLM LLM with connector: lmcache:

from tensorrt_llm import LLM, SamplingParams
from tensorrt_llm.llmapi.llm_args import (
    KvCacheConfig, KvCacheConnectorConfig,
)

llm = LLM(
    model="Qwen/Qwen2-1.5B-Instruct",
    backend="pytorch",
    kv_cache_config=KvCacheConfig(enable_block_reuse=True),
    kv_connector_config=KvCacheConnectorConfig(connector="lmcache"),
)

out = llm.generate(["Your prompt here"], SamplingParams(max_tokens=64))
print(out[0].outputs[0].text)

MP mode

PYTHONHASHSEED=0 must be set in both terminals – chunk hashing depends on a stable hash(), and the server and client must agree on the seed.

Start the LMCache server:

export PYTHONHASHSEED=0
lmcache server \
    --l1-size-gb 10 --eviction-policy LRU --chunk-size 256

In a separate terminal, point TRT-LLM at the server via server_url:

export PYTHONHASHSEED=0
python run_trtllm.py

where run_trtllm.py contains:

from tensorrt_llm import LLM, SamplingParams
from tensorrt_llm.llmapi.llm_args import (
    KvCacheConfig, KvCacheConnectorConfig,
)

llm = LLM(
    model="Qwen/Qwen2-1.5B-Instruct",
    backend="pytorch",
    kv_cache_config=KvCacheConfig(enable_block_reuse=True),
    kv_connector_config=KvCacheConnectorConfig(
        connector="lmcache-mp",
        server_url="tcp://localhost:5555",
    ),
)

out = llm.generate(["Your prompt here"], SamplingParams(max_tokens=64))
print(out[0].outputs[0].text)

Note

The TRT-LLM adapter reads LMCacheEngineConfig the same way the vLLM adapter does: LMCACHE_CONFIG_FILE for a YAML file, otherwise individual LMCACHE_* environment variables. See Configuring LMCache for all options.

🎉 You now have LMCache caching and reusing KV caches across all three engines.

More MP server options#

The vLLM MP example above runs lmcache server locally on the default ports. Common variations:

Custom port, remote host, or gRPC – by default the connector talks to localhost:5555 over ZMQ. To use a different port, a server on another host, or gRPC, pass lmcache.mp.host / lmcache.mp.port in kv_connector_extra_config and select the matching server transport:

vllm serve Qwen/Qwen3-8B --kv-transfer-config \
  '{"kv_connector":"LMCacheMPConnector", "kv_role":"kv_both", "kv_connector_extra_config": {"lmcache.mp.host": "tcp://10.0.0.1", "lmcache.mp.port": 6555}}'

For gRPC, start the server with --transport grpc and use a grpc://10.0.0.1 connector host.

CPU-only (no GPU) – the server runs with a StubCPUDevice and shares KV tensors with vLLM over POSIX shared memory. Start lmcache server normally, then set lmcache.mp.mp_transfer_mode=lmcache_driven on the vLLM side to enable the zero-copy SHM handle path (the default auto routing maps non-CUDA devices to engine_driven, a worker-side gather/scatter copy path that the server only loads when started with --supported-transfer-mode engine_driven or auto).

Docker – see Docker deployment.

HTTP management endpoints (health, clear-cache, status) – see HTTP API.

Next Steps#

Performance Testing: Try the Benchmarking section to experience LMCache’s performance benefits with more comprehensive examples
Production: Deploy LMCache with Docker or Kubernetes, plus observability and tuning – see Deployment Guide