Configuration Reference#

This page documents every CLI argument accepted by the LMCache multiprocess server. Arguments are grouped by the config module that defines them.

Per-request LMCache configuration#

vLLM clients can attach request-scoped LMCache metadata through the top-level kv_transfer_params field. When vLLM uses LMCacheMPConnector, entries whose keys start with lmcache. are forwarded with the request across the MP scheduler and worker IPC paths. Other kv_transfer_params entries are reserved for the transfer layer and are not forwarded to LMCache.

For example:

curl -X POST http://localhost:8000/v1/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "Qwen/Qwen3-14B",
        "prompt": "Explain KV cache reuse.",
        "max_tokens": 32,
        "kv_transfer_params": {
            "lmcache.tag.tenant": "example-tenant",
            "lmcache.ttl": 60
        }
    }'

The connector carries these values on lookup, prefetch, store, retrieve, and lookup-lock cleanup operations so server-side features can inspect the same request metadata throughout the request lifecycle.

Important

Forwarding a value does not by itself make the MP server act on it or make it part of cache identity. The current MP server treats request configs as metadata. Do not rely on lmcache.tag.*, lmcache.ttl, lmcache.skip_save, or another request config for isolation, expiration, or cache-control behavior in MP mode unless the selected server-side feature explicitly documents support for it. The in-process LMCacheConnectorV1 may interpret these values differently.

The MP server’s default --kv-load-policy acts on lmcache.skip_load: a request that sets it to true gets a lookup miss and is recomputed by the engine. Load policies receive the request configs as sent. The vLLM LMCacheMPConnector, the unified SGLang MP connector, and the TensorRT-LLM MP adapter add lmcache.hint.engine_computed_tokens, the tokens the engine already holds in its own prefix cache, to their lookups. vLLM omits it on eager-prefetch lookups, which run before vLLM checks its prefix cache.

MP Server#

Source: lmcache/v1/multiprocess/config.py

Argument

Default

Description

--transport

zmq

Request transport exposed by the server. Choices: zmq or grpc.

--instance-id

(unset, default UUID v4)

Stable identity of this MP server. Used as the coordinator membership key and projected onto the OTel service.instance.id resource attribute on every metric and span (so telemetry and coordinator membership share one id). When the flag is not passed, defaults to a random UUID v4 minted at startup.

--host

localhost

Host address to bind the selected request server.

--port

5555

Port to bind the selected request server.

--chunk-size

256

Minimum chunk size for KV cache operations (in tokens). The vLLM MP connector may negotiate it upward to satisfy the model’s KV geometry; see Chunk-Size Negotiation.

--null-block-id

0

Engine block ID that denotes absent KV data. Keep the default for vLLM-compatible layouts. Engines where block 0 is valid, such as ATOM native PAGE/STATE transfer, can use -1.

--max-workers

1

Base number of worker threads. Sets the default for both the GPU (affinity) pool and the CPU (normal) pool. Can be overridden per-pool with --max-gpu-workers and --max-cpu-workers.

--max-gpu-workers

(inherits --max-workers)

Worker threads for the GPU affinity pool (STORE/RETRIEVE). Requests from the same vLLM instance are always dispatched to the same thread, eliminating GPU transfer lock contention.

--max-cpu-workers

(inherits --max-workers)

Worker threads for the normal CPU pool (LOOKUP, etc.).

--grpc-server-workers

32

gRPC request-dispatch threads. Used only with --transport grpc.

--grpc-server-mode

sync

gRPC server implementation: sync or aio. aio receives requests on one event loop and hands only the handlers to worker threads. Used only with --transport grpc.

--hash-algorithm

blake3

Hash algorithm for token-based operations. Choices: builtin, sha256_cbor, blake3.

--engine-type

default

Cache engine backend type. default uses standard prefix caching; blend composes the CacheBlend BlendModule into the engine for non-prefix KV reuse and requires --supported-transfer-mode to be lmcache_driven or auto. Choices: default, blend.

--supported-transfer-mode

lmcache_driven

Which worker → server transfer paths the server loads. lmcache_driven (default) enables only the server-driven transfer path (STORE/RETRIEVE, supports both CUDA IPC and CPU SHM); engine_driven enables only the non-GPU (PREPARE/COMMIT) transfer path; auto loads both so workers of either device type can connect without manual configuration. Choices: lmcache_driven, engine_driven, auto.

--isolated-ipc / --no-isolated-ipc

false

Assume engine workers and this server run in containers that share no host IPC namespace (hostIPC) and no common /dev/shm, and use IPC mechanisms that work there: on CUDA, raw CUDA IPC memory handles for KV-cache registration (instead of PyTorch storage IPC, which needs a shared /dev/shm) and timeline-semaphore events (instead of CUDA interprocess event handles). Must match the workers’ lmcache.mp.isolated_ipc setting – the two mechanisms exchange incompatible event handles, and a mismatch fails at event import on whichever side receives the foreign handle. Currently supported by the vLLM MP connector only; the default stays false until the integrations that still create raw CUDA interprocess events (SGLang, TensorRT-LLM, CacheBlend, qstore) migrate.

--runtime-plugin-locations

[]

Zero or more paths to runtime plugin scripts or directories to launch alongside the server. Plugins are spawned by MPRuntimePluginLauncher and receive the full server config via the LMCACHE_RUNTIME_PLUGIN_CONFIG environment variable.

--runtime-plugin-config

"{}"

JSON string of extra key-value config forwarded to runtime plugins via LMCACHE_RUNTIME_PLUGIN_EXTRA_CONFIG. Example: '{"plugin.frontend.heartbeat_url": "http://localhost:5000/heartbeat"}'.

--script-allowed-imports

[]

Space-separated list of Python module names that scripts posted to the HTTP /run_script endpoint are allowed to import. Example: --script-allowed-imports numpy pandas.

--run-script-api-enabled

false

Enable the POST /run_script HTTP endpoint, which executes caller-supplied Python in-process. The restricted builtins are not a security boundary — treat this as full remote code execution and only enable it on a trusted network.

--shm-name

""

SHM segment name for non-GPU KV transfer (only used when the non-GPU path is loaded, i.e. --supported-transfer-mode is auto or engine_driven). "" (empty string, default): SHM disabled; KV transfer uses the typed inline path. Any other value: create a SHM pool and use that exact name for its segment.

--kv-event-log-size

32768

CPU store-completion and eviction records retained for subscriptions and reconnects. Workers receive pushes over ZMQ or gRPC and republish them as KV events. 0 disables the channel. The channel requires the observability event bus, which --disable-observability turns off. When vLLM KV events are enabled, workers subscribe to the server automatically. When the channel is disabled, the server answers the subscription with enabled=False. The worker then reports only its own completed stores.

--worker-reap-timeout-seconds

120.0

Silence budget (seconds) after which a worker that has sent at least one heartbeat PING but then gone quiet has its KV cache registration reaped, freeing the leaked GPU context and CUDA IPC handles. 0 disables reaping. Keep this at least 3x the engine adapter’s lmcache.mp.heartbeat_interval (default 10s) so a few missed pings never reap a live worker; the adapter warns at startup if its interval is raised without raising this.

--worker-registration-grace-seconds

3600.0

Silence budget (seconds) for a worker that registered but has never sent a PING (still warming up, or died before its first request). Must be >= --worker-reap-timeout-seconds. Generous by default so slow model warmup is never mistaken for a dead worker.

--kv-load-policy

DEFAULT

Per-request choice between serving a lookup and letting the engine recompute the prompt. The policy runs before the prefetch, so a declined lookup fetches nothing from L2 and takes no locks; the engine sees a miss. DEFAULT serves every lookup unless the request sets "lmcache.skip_load": true in kv_transfer_params. A package.module:ClassName value loads a custom lmcache.v1.multiprocess.kv_load_policy.KVLoadPolicy subclass from the server’s environment, constructed with --runtime-plugin-config.

--enable-segmented-prefix

False

CacheBlend (--engine-type blend) only: on a mid-prefix L2 retrieve failure, retain the gapped prefix so the post-gap chunks stay L1-resident and only the dropped gap is recomputed, instead of truncating the prefix at the gap. No effect for other engines. See Fault Inject for a way to exercise it.

--enable-dedup-content

False

--engine-type blend only: skip fingerprint registration for a chunk whose content is already indexed, so the same text stored behind two prefixes is indexed once. No effect for other engines.

--separate-object-groups / --no-separate-object-groups

False

Split a hybrid model’s kernel groups into one object group per cross-chunk attention window (full attention, each sliding-window size, mamba/GDN) at KV-cache registration. Off by default; pass --separate-object-groups to enable it. Required for Mamba / linear-attention hybrids (it lets their recurrent state be cached independently, and is what allows --max-num-batched-tokens to exceed twice the block size). For a non-hybrid model it makes no difference — every layer resolves to one object group. See Hybrid Attention Models.

Lookup Hash Logging#

Source: lmcache/v1/mp_observability/subscribers/logging/lookup_hash.py

When enabled, the server publishes chunk hashes computed during lookup() as MP_LOOKUP events on the EventBus. The LookupHashLoggingSubscriber writes these to rotating JSONL files for offline analysis. Disabled by default. These arguments are part of the Observability group.

Argument

Default

Description

--lookup-hash-log-dir

"" (disabled)

Directory to write lookup hash JSONL files. An empty string disables logging.

--lookup-hash-log-rotation-interval

21600 (6 h)

Time interval in seconds before rotating to a new log file.

--lookup-hash-log-rotation-max-size

104857600 (100 MB)

Max file size in bytes before rotating even if the time interval has not elapsed.

--lookup-hash-log-max-files

100

Max number of log files to keep. Oldest files are deleted when this limit is exceeded.

HTTP Frontend#

Source: lmcache/v1/multiprocess/config.py

The HTTP frontend is included when running lmcache server.

Argument

Default

Description

--http-host

127.0.0.1

Host to bind the HTTP (FastAPI/uvicorn) server.

--http-port

8080

Port to bind the HTTP server.

--http-api-key-file

unset

Read a bearer API key from a file. Repeat the option to accept multiple keys during rotation. LMCACHE_HTTP_API_KEY_FILE and LMCACHE_HTTP_API_KEY provide environment alternatives. All sources are cumulative. When none are set, authentication is disabled. / and /healthcheck remain public when authentication is enabled.

P2P#

Source: lmcache/v1/multiprocess/config.py

These flags configure peer-to-peer KV cache sharing between MP servers (see P2P KV Cache Sharing). They are registered by add_p2p_args() on the lmcache server parser. P2P is enabled when --p2p-advertise-url is set, which additionally requires a coordinator URL via --coordinator-url (or LMCACHE_COORDINATOR_URL).

Argument

Default

Description

--p2p-advertise-url

"" (P2P disabled)

Transfer-channel server host:port this instance advertises to peers. Setting it enables P2P (also requires --coordinator-url).

--p2p-listen-url

""

Transfer-channel server host:port to bind. Defaults to --p2p-advertise-url.

--p2p-lookup-timeout

30.0

Seconds before a peer lookup result counts as a miss.

--p2p-load-timeout

30.0

Seconds before a peer load counts as a failure.

--p2p-transfer-engine

nixl

Transfer-channel implementation to use.

L1 Memory Manager#

Source: lmcache/v1/distributed/config.py

Use repeatable --l1-manager '<JSON>' for tagged DRAM, Device-DAX, and GDS managers. See Multiple L1 Managers for the schema, placement order, fixed L2 affinity, and per-manager reporting. The flags below remain the legacy single-L1 interface.

Argument

Default

Description

--l1-size-gb

required without --l1-manager

Size of the L1 tier in GB. Sizes the pinned-DRAM L1 by default, or the GDS slab file when --gds-l1-path is set (see GDS L1 Tier below).

--l1-use-lazy / --no-l1-use-lazy

True

Enable or disable lazy allocation for L1 memory. Pass --l1-use-lazy to enable (default) or --no-l1-use-lazy to explicitly disable.

--l1-init-size-gb

20

Initial allocation size (GB) when using lazy allocation.

--l1-align-bytes

4096

Alignment size in bytes (default 4 KB).

--l1-use-hugepages / --no-l1-use-hugepages

False

Allocate the L1 pool from the 2 MiB hugepage pool instead of regular pinned memory. It requires pre-allocated hugepages (sysctl vm.nr_hugepages). Mutually exclusive with --shm-name and --l1-use-lazy (enabling it auto-disables lazy).

--l1-devdax-path

(not set)

Optional /dev/dax* device or mmap-able file to use as the L1 backing arena. When set, disable lazy allocation with --no-l1-use-lazy and leave --shm-name at its default "" (SHM transfer disabled) because the L1 bytes live in the DAX mapping. If a DAX L2 adapter with the same device_path is registered, that adapter’s max_dax_size_gb is used as the L1 Device-DAX overflow size.

--l1-mlock

False

ROCm only. mlock the DRAM L1 so memory compaction cannot migrate it. See Locked L1 on ROCm below. --l1-manager key: mlock.

Locked L1 on ROCm#

On ROCm, hipHostRegister does not page-lock memory: when the kernel migrates a registered L1 page (memory compaction), the driver evicts all GPU queues of the server until the mapping is restored, which stalls transfers for seconds. --l1-mlock keeps the L1 as ordinary anonymous memory (O_DIRECT, RDMA registration and memory-cgroup accounting are unchanged) and locks it. Requirements, checked at startup:

  • the node sets sysctl -w vm.compact_unevictable_allowed=0; otherwise compaction still moves locked pages;

  • the server may lock the whole L1: docker run --ulimit memlock=-1 or CAP_IPC_LOCK.

Not covered: explicit page migration (move_pages, migrate_pages, NUMA balancing), khugepaged when transparent huge pages are always (use madvise or never), and host memory registered by other processes such as vLLM. Device-DAX L1 is rejected.

GDS L1 Tier#

Source: lmcache/v1/distributed/config.py

Opt-in. Setting --gds-l1-path switches the L1 medium from pinned DRAM to an NVMe slab file accessed via GPUDirect Storage DMA. The CPU pinned-DRAM tier is then disabled, and --l1-size-gb sizes the slab. Disable byte-array L2 adapters when this is on (the GDS tier exposes no L1 memory buffer for them to register).

The DMA path is selected automatically by platform: cuFile (libcufile.so) on NVIDIA and hipFile (libhipfile.so, ROCm/hipFile) on AMD ROCm. The same flags apply to both; no configuration change is needed to switch vendors.

muFile (libmufile.so) is selected automatically on MUSA, or explicitly with --gds-l1-backend mufile. It uses the SmartIO muFile stream-ordered API and a filesystem slab, so the SmartIO runtime and its MUSA-compatible libmufile.so must be installed on every worker.

uGDS (libugds.so) is a third, opt-in backend selected with --gds-l1-backend ugds. It is a user-space GPUDirect Storage library that builds NVMe commands and rings doorbells from user space, so its IO path issues no syscall. LMCache can use uGDS on either NVIDIA CUDA or AMD ROCm. Each deployment must use a libugds.so built for its active platform. Unlike cuFile and hipFile, uGDS does not use a filesystem: the slab is mapped directly onto a raw character device, and --gds-l1-path must name that device (for example /dev/ugds_drv0) rather than a directory. The first --l1-size-gb bytes of the device are the slab, so the device must be at least that large and must not hold anything else.

Phoenix (libphoenix.so) is a fourth opt-in backend selected with --gds-l1-backend phx. Phoenix (phxfs) provides a kernel-mediated user-space NVMe-to-GPU DMA path with a very low software-stack overhead. Like cuFile and hipFile it uses a filesystem slab: --gds-l1-path names an NVMe directory, --gds-l1-use-direct-io applies, and the slab file can share the disk with other data. Each GPU staging buffer is registered with phxfs (phxfs_regmem, 64 KiB-aligned) and the slab is read and written with stream-ordered submissions (phxfs_read_stream / phxfs_write_stream) that keep the DMA ordered with the other work on the stream. A libphoenix build without the stream-ordered API fails to load. Follow the Phoenix installation guide to build libphoenix.so, load the phoenixfs kernel module, and verify the installation.

Note

AMD hipFile requires ROCm >= 7.2.0. The zero-copy GPUDirect fast path additionally needs a kernel built with CONFIG_PCI_P2PDMA, amdgpu-dkms >= 30.20.1, and the slab on a local NVMe ext4/xfs filesystem; where those are unavailable hipFile transparently falls back to a host-bounce compatibility path (correct, but not zero-copy).

Note

uGDS requires its kernel module loaded and the NVMe device bound to it, and a platform-matching libugds.so reachable through the loader (LD_LIBRARY_PATH or ldconfig). Because the device is claimed by ugds_drv rather than the kernel NVMe driver, it carries no filesystem and cannot be shared with any other consumer while in use. Follow the uGDS installation guide to build and load the kernel module, bind the NVMe device, build libugds.so, and verify the installation.

At startup LMCache queries the namespace capacity through uGDSGetDeviceCapacity and rejects an aligned --l1-size-gb value larger than the device. The installed libugds.so must provide this API; LMCache fails closed with an upgrade message when an older library cannot report capacity.

Warning

uGDS requires an entire dedicated SSD whose contents may be destroyed. Ensure the SSD is not used for any other purpose and that its contents are not critical.

Note

Phoenix requires the phoenixfs kernel module loaded and a platform-matching libphoenix.so reachable through the loader (ldconfig or LD_LIBRARY_PATH). It has been validated on NVIDIA GPUs; other platforms require a matching libphoenix build and are not yet tested.

Argument

Default

Description

--gds-l1-path

Not set

NVMe directory for the GDS L1 slab, or the raw device path when --gds-l1-backend ugds is used. Setting this enables the GDS L1 tier; with cuFile, hipFile, or phx one shared slab per process lives at <path>/lmcache_gds_slab.bin.

--gds-l1-backend

auto

GDS implementation: auto, cufile, hipfile, mufile, ugds, or phx. auto selects cuFile on CUDA, hipFile on ROCm, and muFile on MUSA.

--gds-l1-use-direct-io / --no-gds-l1-use-direct-io

True

Open the slab with O_DIRECT (required for the GDS DMA fast path on ext4). Ignored by ugds, whose IO bypasses the kernel entirely.

L1 Manager TTLs#

Source: lmcache/v1/distributed/config.py

Argument

Default

Description

--l1-write-ttl-seconds

600

Time-to-live for each object’s write lock (seconds).

--l1-read-ttl-seconds

300

Time-to-live for each object’s read lock (seconds).

Eviction Policy#

Source: lmcache/v1/distributed/config.py

Argument

Default

Description

--eviction-policy

required

Eviction policy. Choices: LRU, ARC, IsolatedLRU, noop. ARC adaptively balances recently created keys and frequently accessed keys. It keeps key-only ghost history for completed policy evictions; no KV data is retained in the ghost lists. Use noop for buffer-only mode where L1 acts as a pure write buffer (data is deleted from L1 after L2 store). IsolatedLRU maintains one LRU list per cache_salt and requires per-cache_salt quotas to be configured at runtime via the /quota HTTP endpoints (see Quota Management); a cache_salt with no registered quota has an effective limit of 0 bytes, so its data is evicted at the next eviction cycle (allowlist semantics).

--eviction-trigger-watermark

0.8

Memory usage ratio (0.0–1.0) that triggers eviction.

--eviction-ratio

0.2

Fraction of allocated memory to evict when triggered (0.0–1.0).

L2 Policies#

Source: lmcache/v1/distributed/config.py

Argument

Default

Description

--l2-store-policy

default

L2 store policy. Determines which adapters receive each key and whether keys are deleted from L1 after L2 store. The default policy stores all keys to all adapters and keeps L1. The skip_l1 policy stores all keys to all adapters and then deletes them from L1 (buffer-only mode). Choices: default, skip_l1.

--l2-prefetch-policy

default

L2 prefetch policy. Determines which adapter loads each key when multiple adapters have it. The default policy picks the first adapter (lowest index). Prefetched keys are temporary (deleted after the reader finishes). The retain policy uses the same load plan but keeps prefetched keys permanently in L1. Choices: default, retain.

--l2-prefetch-max-in-flight

8

Maximum number of concurrent prefetch (L2 load) requests. Limits how many in-flight loads the PrefetchController may issue at once, preventing excessive L1 memory pressure.

--periodic-notifier-interval-ms

5

Interval in milliseconds for the periodic event notifier heartbeat. A native C++ background thread writes to all registered file descriptors at this interval, waking controller poll loops for L2 adapters that lack native async completion callbacks.

L2 Adapters#

Source: lmcache/v1/distributed/l2_adapters/config.py

L2 adapters are configured via repeatable --l2-adapter <JSON> arguments. Each JSON object must include a "type" field that selects the adapter type. The optional "affinity_tag" field (default "_default") names the host-backed L1 used for both stores and reloads; see Multiple L1 Managers. The order of --l2-adapter arguments determines the adapter order (cascade).

Registered adapter types: nixl_store, nixl_store_dynamic, fs, fs_native, mock, mooncake_store, aerospike, bigtable, sagemaker-hyperpod, s3, hfbucket, resp, valkey, plugin, native_plugin, raw_block, dax, fault_inject. (A p2p type is also registered, but it is wired in dynamically by the P2P subsystem rather than configured via --l2-adapter.)

Each adapter type’s required and optional fields, plus per-backend examples, are documented on its own page under Secondary KV Storage – including the adapters not detailed inline here (fs_native, raw_block, dax, mooncake_store, aerospike, bigtable, sagemaker-hyperpod, hfbucket, resp, valkey).

Multiple adapters (cascade)#

Pass --l2-adapter multiple times. Adapters are used in the order given:

--l2-adapter '{"type": "nixl_store", "backend": "POSIX", "backend_params": {"file_path": "/data/ssd/l2", "use_direct_io": "false"}, "pool_size": 64}' \
--l2-adapter '{"type": "nixl_store", "backend": "GDS", "backend_params": {"file_path": "/data/nvme/l2", "use_direct_io": "true"}, "pool_size": 128}'

Observability#

Source: lmcache/v1/mp_observability/config.py

See Observability for full details on the three modes (metrics, logging, tracing).

Argument

Default

Description

--disable-observability

off

Master switch: disable the EventBus entirely.

--disable-metrics

off

Skip metrics subscribers (no Prometheus endpoint).

--disable-grpc-metrics

off

Skip gRPC Python runtime metrics while keeping LMCache metrics enabled. By default, the MP server enables these metrics only for --transport grpc.

--disable-logging

off

Skip logging subscribers.

--enable-tracing

off

Register tracing subscribers. Requires --otlp-endpoint.

--event-bus-queue-size

10000

Max events in the EventBus queue before tail-drop.

--otlp-endpoint

(none)

OTLP gRPC endpoint for exporting metrics and traces.

--prometheus-port

9090

Port for the Prometheus /metrics endpoint.

--metrics-sample-rate

0.01

Fraction of chunks/blocks in (0, 1.0] to track for lifecycle histograms. Counters always count every event regardless of this setting.

--trace-level

(none)

Enable trace recording at the given level. storage records StorageManager public-API calls for offline replay via lmcache trace. events records the cache-event stream this server emits for the MP coordinator, with or without one configured. See Tracing and Debugging.

--trace-output

(none)

Path to write the trace file. If omitted while --trace-level is set, a timestamped file under $TMPDIR (lmcache-trace-<pid>-<UTC>.lct) is minted and its path is logged at INFO.

--enable-extra-logging

off

Periodic INFO logs: per-GPU L0<->L1 transfer stats and L1 memory usage. See Logging.

--extra-logging-interval

10.0

Seconds between extra-logging emissions.

vLLM Client Configuration#

On the vLLM side, specify the LMCache server host and port via the kv_connector_extra_config parameter. The URL scheme selects the request transport: use tcp:// for ZMQ or grpc:// for gRPC, matching the server’s --transport setting. A bare host is accepted and normalized to tcp:// for backward compatibility:

vllm serve Qwen/Qwen3-14B \
    --kv-transfer-config \
    '{"kv_connector":"LMCacheMPConnector", "kv_role":"kv_both", "kv_connector_extra_config": {"lmcache.mp.host": "127.0.0.1", "lmcache.mp.port": 6000}}'

To target multiple LMCache servers from a single vLLM deployment, pass a list (or comma-separated string) of server URLs via lmcache.mp.server_urls. When set, server_urls takes precedence over the single-server host / port keys; vLLM’s world size must be divisible by the number of servers, and each worker connects only to its locally-assigned server (global ranks are sliced into contiguous blocks, one block per server). Multi-server mode currently supports tensor parallelism only – pipeline parallelism (pp_size > 1) and data parallelism (dp_size > 1) are rejected with a clear error.

vllm serve Qwen/Qwen3-14B \
    --tensor-parallel-size 4 \
    --kv-transfer-config \
    '{"kv_connector":"LMCacheMPConnector", "kv_role":"kv_both", "kv_connector_extra_config": {"lmcache.mp.server_urls": "tcp://host1:6667,tcp://host2:6667"}}'

Decode context parallelism (DCP)#

--decode-context-parallel-size is supported. Under DCP, vLLM shards the attention KV cache across ranks along the token axis, so each rank holds only a strided 1/dcp slice and one block ID spans block_size * dcp tokens. LMCache stores each rank’s opaque page as its own object and a chunk counts as a hit only when every rank’s slice is present. Non-trivial --cp-kv-cache-interleave-size values are supported when they evenly divide every resolved attention cache block size. Because interleave changes the token-to-slot byte layout, the connector automatically adds the DCP size and interleave value to its internal cache namespace. This prevents pages written by one interleave layout from being loaded by another. It does not change the model name served by vLLM, but the decorated cache model name is visible in MP metric labels so operators can distinguish incompatible cache layouts.

The vLLM MP connector automatically negotiates a compatible chunk size. For a single attention group, the required alignment is block_size * decode_context_parallel_size. For a hybrid model, it is the least common multiple of every attention group’s DCP-scaled block span and every recurrent-state group’s unscaled physical block span. The server treats --chunk-size as a minimum and rounds it up to the smallest compatible multiple; see Chunk-Size Negotiation.

The example model also needs a vLLM build that can run it under DCP: Kimi-Linear DCP support landed after v0.27.1 (vLLM commit 63ac04a61e, PR #50484). On stock v0.27.1 the command below fails at startup with Kimi-K3 MultiHeadLatentAttention does not support context parallelism.

# block_size 1024 x dcp 2 -> a configured minimum of 256 negotiates to 2048
lmcache server --host localhost --port 6000 --chunk-size 256 \
    --l1-size-gb 20 --eviction-policy LRU

vllm serve moonshotai/Kimi-Linear-48B-A3B-Instruct \
    --trust-remote-code \
    --tensor-parallel-size 2 \
    --decode-context-parallel-size 2 \
    --kv-transfer-config \
    '{"kv_connector":"LMCacheMPConnector", "kv_role":"kv_both", "kv_connector_extra_config": {"lmcache.mp.host": "127.0.0.1", "lmcache.mp.port": 6000}}'

Pipeline parallelism and multiple LMCache servers may both be combined with DCP. These combinations are rejected at startup:

Rejected with DCP

Reason

--prefill-context-parallel-size > 1

Adds a second KV shard axis this connector does not map.

Fewer than dcp_size ranks per LMCache server

No server holds a complete set of shards, and lookup takes the minimum hit count across servers, so it reports no hits.

An interleave value that is non-positive, larger than a resolved attention cache block, or does not evenly divide every resolved attention block is rejected at connector startup.

decode_context_parallel_size > tensor_parallel_size is rejected by vLLM itself, so this connector does not re-check it.

LMCacheMPConnector reads the following keys from kv_connector_extra_config:

Connector extra_config Keys#

All connector-level options are passed through kv_connector_extra_config and use the lmcache.mp. prefix.

By default, MP caches only prompt tokens, avoiding new cache entries from sampled output when fixed prompts are replayed.

Set "lmcache.mp.save_decode_cache": true in kv_connector_extra_config for resumable or streaming sessions, where generated tokens become part of a growing prompt across turns within the same request. This setting is separate from the in-process connector’s save_decode_cache YAML/environment setting.

Key

Default

Description

lmcache.mp.save_decode_cache

false

Cache generated tokens in addition to prompt tokens.

lmcache.mp.server_urls

(unset)

Multi-server deployment: list (or comma-separated string) of <transport>://<host>:<port> URLs, e.g. "tcp://host1:6667,tcp://host2:6667". The transport prefix may be omitted – bare host:port entries such as "host1:6667,host2:6667" are normalized to tcp:// by the connector. When set, takes precedence over lmcache.mp.host / lmcache.mp.port; the vLLM world size must be divisible by the number of servers, and each worker connects to its locally-assigned server.

lmcache.mp.host

tcp://localhost

Single-server deployment: request transport and host of the LMCache MP server. Use tcp:// for ZMQ or grpc:// for gRPC. A bare localhost / 127.0.0.1 is normalized to tcp:// by the connector. Ignored when lmcache.mp.server_urls is set.

lmcache.mp.port

5555

Single-server deployment: port of the LMCache MP server. Must match the server’s --port. Ignored when lmcache.mp.server_urls is set.

lmcache.mp.mq_timeout

300.0

Timeout (seconds) for blocking message-queue requests, including the initial chunk-size query and KV cache registration/unregistration. If the server does not respond within this window, the connector raises ConnectionError on startup.

lmcache.mp.heartbeat_interval

10.0

Interval (seconds) between periodic heartbeat pings sent from the connector to the server.

lmcache.mp.nonblocking_lookup_status

true

Poll lookup-status replies without blocking the scheduler by default. Set to false to wait for each status RPC reply in the current callback, for example when long prefill steps delay observation of an already-ready reply. LOOKUP acknowledgement polling remains asynchronous. Available with the current LMCacheMPConnector.

lmcache.mp.eager_prefetch

false

Submit the LMCache lookup when a request enters vLLM’s waiting queue, allowing L2-to-L1 KV staging to overlap with scheduler queue wait. Resumable requests are skipped because their token IDs may be incomplete at enqueue time.

lmcache.mp.autostart

false

Whether vLLM worker 0 should start a local lmcache server process before workers connect to it. Other local workers wait for the server to become reachable. Only localhost and 127.0.0.1 are supported. IPv6 endpoints, including ::1, raise ValueError before startup because the MP ZMQ transport does not enable IPv6 sockets. Auto-start supports exactly one server endpoint; configuring multiple lmcache.mp.server_urls raises ValueError during connector initialization.

lmcache.mp.autostart.wait_timeout

90.0

Timeout (seconds) to wait for the auto-started server to respond to ZMQ PING requests. Must be positive and finite.

lmcache.mp.autostart.server_args

""

Extra command-line arguments passed to the auto-started MP HTTP server process. Required server settings such as --l1-size-gb and --eviction-policy must be supplied here. For example, pass --l1-size-gb 20 --eviction-policy LRU. Endpoint flags such as --host, --port, and --http-host are rejected because the auto-started ZMQ and HTTP listeners are bound to the local connector endpoint. If multiple auto-started MP servers run on the same host, pass distinct --http-port values here to avoid HTTP frontend port conflicts.

lmcache.mp.mp_transfer_mode

auto

Routing mode for the worker -> server transfer context. One of auto (CUDA -> lmcache_driven, others -> engine_driven), lmcache_driven (force the IPC / SHM zero-copy handle path — LMCache server pulls data via device handles), or engine_driven (force the worker-side gather/scatter copy path). Overrides the LMCACHE_MP_TRANSFER_MODE env var when set.

lmcache.mp.isolated_ipc

false

Assume the vLLM workers and the LMCache server run in containers that share no host IPC namespace (hostIPC) and no common /dev/shm, and use IPC mechanisms that work there: on CUDA, raw CUDA IPC memory handles for KV-cache registration and timeline-semaphore events instead of CUDA interprocess event handles. Set it together with the server’s --isolated-ipc flag – a mismatch fails at event import on whichever side receives the foreign handle.

lmcache.mp.use_vmm_api

false

Set when the engine allocates its KV cache through the CUDA VMM API (vLLM’s --enable-cumem-allocator): such memory has no legacy CUDA IPC handle, so KV-cache registration exports it via cuMemExportToShareableHandle instead (a fabric handle when the allocation is fabric-exportable – requires an IMEX channel device, e.g. NVIDIA_IMEX_CHANNELS=0 – or a POSIX fd otherwise). Composes with lmcache.mp.isolated_ipc for fabric-exportable pools; a POSIX-fd-only pool under isolated IPC is rejected at registration.

lmcache.mp.lazy_offload

false

Buffer stores on the scheduler and submit them according to the selected lazy-offload policy. Requires vLLM prefix caching. See Lazy KV Cache Offload for behavior, limitations, and tuning guidance.

lmcache.mp.lazy_offload_policy

EVICTION_AWARE

Lazy drain policy. EVICTION_AWARE drains blocks near the GPU free queue’s eviction head. Set FIFO explicitly to keep the count-triggered behavior.

lmcache.mp.lazy_offload_horizon_steps

2.5

EVICTION_AWARE only: estimated scheduler steps of block consumption treated as imminent eviction. Must be greater than zero. Larger values store earlier and reduce eviction losses, but may store GPU-resident hot content and increase lower-tier eviction pressure.

lmcache.mp.lazy_offload_max_drain_per_step

64

EVICTION_AWARE only: maximum store operations emitted per scheduler step. A value below the concurrent prefill admission rate can lose buffered operations to eviction.

lmcache.mp.lazy_offload_max_deferral_seconds

0.0

EVICTION_AWARE only: how long a buffered operation may wait before it is emitted regardless of eviction pressure. Not a hard bound: no drain runs on a step that schedules no tokens, a request whose store is already in flight is skipped, and due operations that do not fit in max_drain_per_step wait for a later step. Zero leaves emission entirely to the danger window. Set it below the reuse interval the workload has to beat.

lmcache.mp.lazy_offload_threshold

100

FIFO only: number of finished buffered requests that triggers a drain.

lmcache.mp.lazy_offload_select_count

10

FIFO only: maximum finished requests emitted by one drain.

To let vLLM worker 0 start a local MP server automatically:

vllm serve Qwen/Qwen3-14B \
    --kv-transfer-config \
    '{"kv_connector":"LMCacheMPConnector", "kv_role":"kv_both", "kv_connector_extra_config": {"lmcache.mp.autostart": true, "lmcache.mp.autostart.server_args": "--l1-size-gb 20 --eviction-policy LRU"}}'

Auto-start is a convenience for single-node, single-server deployments. The MP server is a child of vLLM worker 0, not an independently managed service.

Note

LMCache’s adapter shutdown does not explicitly terminate this child, but vLLM’s process-tree cleanup may terminate it. Its lifetime depends on the vLLM version and exit path; neither survival nor automatic cleanup is guaranteed. Stop any remaining auto-started server when it is no longer needed.

For servers that must survive vLLM restarts or be shared across vLLM instances, and for multi-node TP/PP deployments, start and manage the server separately. For example, run the server in a separate terminal or service manager and leave auto-start disabled in vLLM:

# Terminal 1: independently managed MP server
lmcache server --host 127.0.0.1 --port 5555 \
    --http-host 127.0.0.1 --l1-size-gb 20 --eviction-policy LRU

# Terminal 2: connect-only vLLM instance
vllm serve Qwen/Qwen3-14B \
    --kv-transfer-config '{
        "kv_connector": "LMCacheMPConnector",
        "kv_connector_module_path":
            "lmcache.integration.vllm.lmcache_mp_connector",
        "kv_role": "kv_both",
        "kv_connector_extra_config": {
            "lmcache.mp.host": "127.0.0.1",
            "lmcache.mp.port": 5555,
            "lmcache.mp.autostart": false
        }
    }'

Environment Variables#

Variable

Description

LMCACHE_LOG_LEVEL

Log level for LMCache (DEBUG, INFO, WARNING, ERROR). Set to DEBUG to see L2 store activity, prefetch results, etc.

PYTHONHASHSEED

Set to a fixed value for reproducible hashing across processes (relevant when using --hash-algorithm builtin).

LMCACHE_TRACK_USAGE

Set to false to disable anonymous usage statistics (see below).

DO_NOT_TRACK

Set to 1 to disable anonymous usage statistics (cross-tool convention).

LMCACHE_USAGE_TRACK_INTERVAL

Seconds between continuous usage-telemetry flushes (default 600). See Usage Stats Collection.

Full Example#

lmcache server \
    --host 0.0.0.0 \
    --port 6555 \
    --chunk-size 512 \
    --max-workers 4 \
    --max-gpu-workers 2 \
    --hash-algorithm blake3 \
    --engine-type default \
    --lookup-hash-log-dir /data/lmcache/lookup_hashes \
    --lookup-hash-log-rotation-interval 21600 \
    --lookup-hash-log-rotation-max-size 104857600 \
    --lookup-hash-log-max-files 100 \
    --l1-size-gb 100 \
    --l1-use-lazy \
    --l1-init-size-gb 20 \
    --l1-align-bytes 4096 \
    --l1-write-ttl-seconds 600 \
    --l1-read-ttl-seconds 300 \
    --eviction-policy noop \
    --l2-store-policy skip_l1 \
    --eviction-trigger-watermark 0.9 \
    --eviction-ratio 0.1 \
    --l2-prefetch-policy default \
    --l2-prefetch-max-in-flight 8 \
    --periodic-notifier-interval-ms 5 \
    --l2-adapter '{"type": "nixl_store", "backend": "POSIX", "backend_params": {"file_path": "/data/lmcache/l2", "use_direct_io": "false"}, "pool_size": 64}' \
    --prometheus-port 9090 \
    --metrics-sample-rate 0.01 \
    --enable-tracing \
    --otlp-endpoint http://localhost:4317

Anonymous Usage Statistics#

The MP server reports anonymous usage statistics: a one-time environment/configuration snapshot at startup and interval counters (tokens retrieved/stored, bytes stored, uptime) every LMCACHE_USAGE_TRACK_INTERVAL seconds. No prompts, keys, KV-cache data, model names, or --instance-id are ever sent, and reporting can never affect serving. Opt out with LMCACHE_TRACK_USAGE=false or DO_NOT_TRACK=1; see Usage Stats Collection for details.