Kimi K3#

Moonshot AI's flagship hybrid Mixture-of-Experts model. Like Kimi-Linear, it interleaves Kimi Delta Attention (KDA) linear-attention layers with Multi-head Latent Attention (MLA) full-attention layers. The KDA layers keep a recurrent state cache (a convolution + delta-net state) instead of a paged key/value cache; LMCache reinterprets that state as an opaque page at registration time, so prefix caching and KV reuse work end to end. See Hybrid Attention 模型 for the general handling of Mamba / linear-attention models.

Validated models#

moonshotai/Kimi-K3 (8× NVIDIA B300)
moonshotai/Kimi-K3 MXFP4 (8× AMD MI355X, ROCm 10, TP 8 / DCP 8, fp8 KV cache)

vLLM

Engine documentation: Kimi K3 in vLLM supported models.

Status: Validated with LMCache. Verified on NVIDIA B300 (8 GPUs).

备注

Version requirements. Kimi K3 support has not landed in a stable vLLM release yet — use the upstream pre-release Docker image for now (a stable vLLM release with K3 support is expected to follow, likely 0.26.1 or 0.27):

docker pull vllm/vllm-openai:kimi-k3

On the LMCache side, use a nightly build from 2026-07-27 or newer (see 安装); stable LMCache releases include K3 support starting from 0.5.3.

As a Mamba / linear-attention hybrid, Kimi K3 needs the same three settings as the other KDA / GDN hybrids: the align Mamba cache mode, prefix caching, and a chunk size negotiated from vLLM's unified block size N (see 步骤 1 — 找到模型的统一块大小 N for how N is determined). For Kimi K3 at --tensor-parallel-size 8 the unified block size is N = 768:

Model	Unified block size `N`	GPUs (TP)
`moonshotai/Kimi-K3`	768	8

N depends on the tensor-parallel size (the KDA state is sharded across TP ranks while the MLA cache is not), so after changing --tensor-parallel-size always re-read N from vLLM's Setting attention block size to N tokens startup log line and re-derive --max-num-batched-tokens from it.

Start the LMCache MP server with an explicitly compatible minimum (a lower minimum would negotiate to N = 768):

lmcache server \
    --port 6555 \
    --chunk-size 768 \
    --separate-object-groups \
    --max-workers 4 \
    --l1-size-gb 100 \
    --eviction-policy LRU

Start vLLM with the LMCache MP connector (8 GPUs):

export VLLM_ENABLE_K3_LATENT_MOE_TAIL_FUSION=1

vllm serve moonshotai/Kimi-K3 \
    --trust-remote-code \
    --load-format dummy \
    --moe-backend auto \
    --gpu-memory-utilization 0.95 \
    --tensor-parallel-size 8 \
    --no-enable-flashinfer-autotune \
    --enable-auto-tool-choice \
    --tool-call-parser kimi_k3 \
    --reasoning-parser kimi_k3 \
    --enable-prefix-caching \
    --mamba-cache-mode align \
    --max-num-batched-tokens 1500 \
    --kv-transfer-config \
    '{"kv_connector":"LMCacheMPConnector","kv_role":"kv_both","kv_connector_extra_config":{"lmcache.mp.port":6555}}'

Why these settings — LMCache-side (required):

--mamba-cache-mode align and --enable-prefix-caching are required. align is the only Mamba cache mode the KDA backend supports, and prefix caching must be on for LMCache to store and reuse the recurrent state.
--separate-object-groups (server) is required for hybrid Mamba / linear-attention models: it gives the KDA layers their own cache objects and is what lets --max-num-batched-tokens exceed 2N.
--chunk-size (server) is a minimum. The connector negotiates the smallest multiple of N that is at least that value. See Chunk-Size Negotiation.
--max-num-batched-tokens must be at least N (= 768); the validated value is 1500. align snapshots the KDA state only on a block boundary at the end of a scheduler step. Within [N, 2N) every step advances exactly one block, snapshotting every boundary (finest reuse); with --separate-object-groups values ≥ 2N are allowed too, trading coarser reuse for higher prefill throughput. See Hybrid Attention 模型 for the full rationale.
The server's --port 6555 must match lmcache.mp.port in the connector config.

Model-side (Kimi K3 serving requirements):

VLLM_ENABLE_K3_LATENT_MOE_TAIL_FUSION=1 enables the fused latent-MoE tail path for Kimi K3.
--tool-call-parser kimi_k3, --reasoning-parser kimi_k3, and --enable-auto-tool-choice wire up K3's tool-calling and reasoning output formats.
--trust-remote-code loads Kimi K3's custom modeling code; --moe-backend auto lets vLLM pick the MoE kernel backend; --no-enable-flashinfer-autotune skips FlashInfer autotuning at startup.
--load-format dummy initializes random weights so the serving stack can be validated without downloading the full checkpoint — drop it to serve the real weights.
--tensor-parallel-size 8 shards the weights across eight GPUs. Adjust it to your hardware — but note it changes N and the two derived flags (see above).

No attention-backend or --no-disable-hybrid-kv-cache-manager flag is needed; LMCacheMPConnector advertises hybrid support and vLLM auto-selects the KDA and MLA backends. For the generic LMCache + vLLM wiring (ports, remote hosts), see 快速入门.

AMD MI355X / ROCm 10 (TP 8, decode context parallel 8). Validated under a 70-concurrency agentic trace with a 1.8 TB L1:

The +rocm7.2 LMCache wheel loads on ROCm 10 (its native extensions link libamdhip64.so.7, which ROCm 10 still ships).
With DCP 8 and fp8 KV the MLA group registers 12288 tokens per block; --chunk-size 12288 works, and a lower minimum negotiates to 12288.
Set --l1-init-size-gb equal to --l1-size-gb for a terabyte-scale L1. With the default lazy expansion the server keeps pinning host memory while vLLM registers its KV caches, and the GPU mappings can stall in the driver for minutes (heartbeats time out and the connector drops into degraded mode). Eager pinning finishes before vLLM starts.
Under deep queues a request can wait many minutes between its lookup and its retrieve. Raise --l1-read-ttl-seconds and --session-ttl-seconds above that delay (7200 here); with the defaults (300 / 600) the prefetch read locks expire before the retrieve, which fails with KEY_NOT_READABLE and is reported to vLLM as invalid_block_ids -- fatal for a hybrid model with several KV cache groups.

If there are any issues with vLLM setup, please refer to the vLLM Recipes for more details.

SGLang

Status: Not validated with LMCache.

TRT-LLM

Status: Not validated with LMCache.

CacheBlend support#

Not supported: the hybrid groups' cached pages are byte-opaque (see Caveats).

Compression support#

Method	Status	Notes
CacheGen	Not supported	Hybrid groups' cached pages are byte-opaque.

Caveats#

Pre-release engine support. Until vLLM ships K3 in a stable release, pin both sides: the vllm/vllm-openai:kimi-k3 Docker image on the vLLM side and an LMCache nightly from 2026-07-27 or newer (stable from 0.5.3).
Generation is not guaranteed bit-exact between a cached and a fresh run under concurrent load: KDA / GDN linear-attention backends do not support vLLM's batch-invariant mode, so kernel results can vary with batch composition. Validate with a score-level comparison, not a token-level diff.
Cached pages for the KDA and MLA groups are byte-opaque views, so content-aware processing (CacheGen, CacheBlend) does not apply, and cache entries must not be shared across engines with different attention backends or kernel block sizes.
vLLM's Mamba prefix caching in align mode is experimental upstream.