Kimi-Linear

A hybrid architecture from Moonshot AI that interleaves Kimi Delta Attention (KDA) linear-attention layers with Multi-head Latent Attention (MLA) full-attention layers. Like other Mamba / linear-attention hybrids, the KDA layers keep a recurrent state cache (a convolution + delta-net state) instead of a paged key/value cache; LMCache reinterprets that state as an opaque page at registration time, so prefix caching and KV reuse work end to end. See Hybrid Attention Models for the general handling of Mamba / linear-attention models.

Validated models

• moonshotai/Kimi-Linear-48B-A3B-Instruct (48B-parameter MoE, 3B active; 2 GPUs)

vLLM

Engine documentation: Kimi-Linear in vLLM supported models (architecture KimiLinearForCausalLM).

Status: Validated with LMCache.

As a Mamba / linear-attention hybrid, Kimi-Linear needs the same three settings as the other GDN hybrids: the align Mamba cache mode, prefix caching, and a chunk size matched to vLLM’s unified block size N. That block size is model- and parallelism-specific — vLLM logs Setting attention block size to N tokens at startup. Because the KDA state is sharded across tensor-parallel ranks (while the MLA cache is not), N depends on --tensor-parallel-size; read it from the log for your own configuration:

Model	Unified block size N	GPUs (TP)
moonshotai/Kimi-Linear-48B-A3B-Instruct	944	2

Set the LMCache server’s --chunk-size to that N (or a multiple of it) and enable --separate-object-groups, then set vLLM’s --max-num-batched-tokens to at least N. Keeping it below 2N (e.g. 2N-1) snapshots the KDA state at every block boundary for the finest cache reuse; larger values raise prefill throughput at the cost of coarser reuse (see Why these settings).

Note

N scales inversely with the tensor-parallel size for this model, so changing --tensor-parallel-size changes both derived flags. The MLA cache is replicated across TP ranks (its per-rank size is fixed), while the KDA state is sharded (its per-rank size is divided by the TP degree). vLLM picks N so an attention page is at least as large as a KDA state page, so a smaller per-rank KDA state yields a smaller N — and vice versa. Concretely, going from TP=2 → TP=1 doubles the per-rank KDA state, so N doubles from 944 to 1888; --chunk-size (= N) and the recommended --max-num-batched-tokens (2N-1, i.e. 3775) both double to match. Always re-read N from the startup log after changing the TP degree — do not scale it by hand.

Start the LMCache MP server (N = 944):

lmcache server --chunk-size 944 --separate-object-groups \
    --l1-size-gb 100 --eviction-policy LRU

Start vLLM with the LMCache MP connector (2 GPUs, N = 944 → 2N-1 = 1887):

vllm serve moonshotai/Kimi-Linear-48B-A3B-Instruct \
    --tensor-parallel-size 2 \
    --trust-remote-code \
    --enable-prefix-caching \
    --mamba-cache-mode align \
    --max-num-batched-tokens 1887 \
    --kv-transfer-config \
    '{"kv_connector":"LMCacheMPConnector", "kv_role":"kv_both"}'

Why these settings:

• --mamba-cache-mode align and --enable-prefix-caching are required. align is the only Mamba cache mode the KDA backend supports, and prefix caching must be on for LMCache to store and reuse the recurrent state.

• --separate-object-groups (server) is required for hybrid Mamba / linear-attention models: it gives the KDA layers their own cache objects so their recurrent state is stored and loaded independently of the MLA layers. It is what allows --max-num-batched-tokens to exceed 2N; without it the budget must stay in [N, 2N).

• --chunk-size (server) must be a multiple of the unified block size N — --chunk-size N is the simplest choice and gives the finest cache granularity. LMCache raises at engine startup if it is not.

• --max-num-batched-tokens must be at least N: a scheduler step must advance at least one whole block, since align snapshots the KDA state only at the end of each step, on a block boundary, and the scheduler splits prefills into whole N-token blocks. Two regimes:

  • ``[N, 2N)`` (e.g. ``2N-1``) — every step advances exactly one block, so LMCache snapshots the state at every block boundary: the finest partial-prefix reuse. 2N-1 also leaves N-1 spare budget so decodes co-schedule with a prefill block instead of serializing behind it (exactly N blocks new prefills once any request is decoding).

  • ``≥ 2N`` (requires --separate-object-groups) — larger prefill steps raise throughput, but a step now spans several blocks and only its last block gets a snapshot, so reuse is coarser: cached prefixes align to step boundaries rather than to every block.

• --trust-remote-code loads Kimi-Linear’s custom modeling code.

• --tensor-parallel-size 2 shards the weights across two GPUs. Adjust it to your hardware — but note it changes N and the two derived flags (see the note above).

No attention-backend or --no-disable-hybrid-kv-cache-manager flag is needed; LMCacheMPConnector advertises hybrid support and vLLM auto-selects the KDA and MLA backends. For the generic LMCache + vLLM wiring (ports, remote hosts), see Quickstart.

SGLang

Status: Not validated with LMCache.

TRT-LLM

Status: Not validated with LMCache.

CacheBlend support

Not supported: the hybrid groups’ cached pages are byte-opaque (see Caveats).

Compression support

Method	Status	Notes
CacheGen	Not supported	Hybrid groups’ cached pages are byte-opaque.

Caveats

• Generation is not guaranteed bit-exact between a cached and a fresh run under concurrent load: KDA / GDN linear-attention backends do not support vLLM’s batch-invariant mode, so kernel results can vary with batch composition. Validate with a score-level comparison, not a token-level diff.

• Cached pages for the KDA and MLA groups are byte-opaque views, so content-aware processing (CacheGen, CacheBlend) does not apply, and cache entries must not be shared across engines with different attention backends or kernel block sizes.

• vLLM’s Mamba prefix caching in align mode is experimental upstream.