Hybrid Attention 模型#

一些模型在其层中交错使用多种注意力类型——最常见的是在大多数层上使用 sliding-window attention，而在少数层上使用 full attention。vLLM 通过其混合 KV 缓存管理器来服务这些模型，该管理器将模型的层分成多个 KV 缓存组（每种注意力行为一个）。

LMCache 多进程连接器 (LMCacheMPConnector) 支持这些混合模型：它为每个组存储和检索 KV Cache，因此前缀缓存和 KV 重用的工作方式与普通模型相同。

验证过的混合模型#

经过验证的 hybrid attention 架构的配方页面：

模型

注意力布局

配方

Gemma 3

sliding window + full

Gemma 3

Gemma 4

sliding window + full

Gemma 4

gpt-oss

sliding window + full

gpt-oss

Qwen3.5 / Qwen3.6 / Qwen3.8 series

Mamba / GDN + full

Qwen3.5 / Qwen3.6 / Qwen3.8 series

Qwen3.8-Flash-Next

GDN linear-attention + QSA sparse full (scratch ring)

Qwen3.8-Flash-Next

Kimi-Linear

KDA linear-attention + MLA full

Kimi-Linear

Kimi K3

KDA linear-attention + MLA full

Kimi K3

DeepSeek-V4-Flash

稀疏-MLA（多个 KV 组）

DeepSeek-V4-Flash

DeepSeek-V4.1-Flash

Sparse-MLA + per-layer sliding window (9 KV groups, scratch ring)

DeepSeek-V4.1-Flash

GLM 5.1/5.2

动态稀疏注意力（多个 KV 组）

GLM 5.1/5.2

GLM-5.3

Dynamic Sparse Attention (single KV group)

GLM-5.3

MiniMax-M3

稀疏注意力 + 闪电索引器（一个组中的混合 KV 格式）

MiniMax M3

已支持的内容#

所有层均使用 standard paged attention 的模型——包括混合 sliding-window 和 full attention 的混合模型——在没有特殊配置的情况下得到支持。示例：

模型系列

注意力布局

状态

Gemma 2 / Gemma 3

交错 sliding-window + full

支持

gpt-oss

交错 sliding-window + full

支持

Qwen3.5（以及其他 Gated-DeltaNet 混合模型）

交错 Mamba/GDN + full

支持（见下文）

Llama, Qwen2/Qwen3 (dense), Mistral, …

单一注意力类型

支持

只需像往常一样将 vLLM 指向 LMCache 服务器（请参见 快速入门）；LMCache 在注册时会自动检测模型的 KV 缓存组。

Per-request scratch groups are excluded from caching, including on older vLLM forks that mark them with participates_in_prefix_caching=False instead of prefix_cacheable=False.

备注

因为 LMCacheMPConnector 向 vLLM 声明了混合支持，vLLM 为这些模型保持其混合 KV 缓存管理器 启用（它不会回退到单一统一组）。您不需要 --no-disable-hybrid-kv-cache-manager 或任何相关标志。

对象组分离#

At KV-cache registration LMCache can bucket a hybrid model's layers into object groups — the unit it stores and retrieves as one object. With --separate-object-groups each distinct cross-chunk attention window becomes its own object group: full-attention layers form one group, and each sliding-window size (mamba / GDN included) forms another. The default is off — every layer shares a single full-attention object group.

# default: a single full-attention object group for all layers
lmcache server --chunk-size 256 --l1-size-gb 100

# one object group per attention window (required for mamba / GDN hybrids)
lmcache server --chunk-size 256 --l1-size-gb 100 --separate-object-groups

For a non-hybrid model (a single attention behavior) the setting makes no difference — every layer resolves to one object group either way. For a Mamba / linear-attention hybrid it is required (see below): it gives the linear-attention layers their own cache objects so their recurrent state is stored and loaded independently of the full-attention layers.

Mamba / Linear-Attention 混合模型#

Models that interleave Mamba / Gated-DeltaNet (GDN) linear-attention layers with full attention — the Qwen3.5, Qwen3.6 and Qwen3.8 series (Qwen/Qwen3.5-0.8B, Qwen/Qwen3.6-27B, Qwen/Qwen3.8-27B, …), Qwen3-Next, Kimi-Linear (moonshotai/Kimi-Linear-48B-A3B-Instruct), Kimi K3 (moonshotai/Kimi-K3), and other GDN hybrids — are supported. Unlike a paged key/value cache, their linear-attention layers keep a recurrent state cache (a convolution + SSM state). LMCache reinterprets that state as an opaque page at registration time, so prefix caching and KV reuse work end to end without any model-specific transfer code.

This section is the general procedure for any such model; the settings are identical across models.

Unified block size and chunk size#

For a Mamba / GDN hybrid, vLLM forces one block size N across all KV cache groups, chosen large enough that an attention page is at least as big as a Mamba state page. It depends on the model's head dimensions and GDN state size (and, when the recurrent state is sharded, on the tensor-parallel size), so it is model-specific. vLLM prints it once at startup:

INFO ... interface.py:670] Setting attention block size to 784 tokens to
ensure that attention page size is >= mamba page size.

Nothing has to be derived from N by hand: the LMCache server's --chunk-size is a minimum, and the connector negotiates the smallest multiple of N that is at least that value (N = 784 resolves the default minimum of 256 to 784, while N = 544 resolves it to 544). See Chunk-Size Negotiation for the full lifecycle and mixed-model constraints. For reference, values grow with model size:

模型

统一块大小 N

GPU

Qwen/Qwen3.6-27B

784

1

Qwen/Qwen3.8-27B

784

1

Qwen/Qwen3.5-0.8B

544

1

moonshotai/Kimi-Linear-48B-A3B-Instruct

944

2

moonshotai/Kimi-K3

768

8

Required flags#

  1. LMCache server --separate-object-groups — gives the linear-attention layers their own cache objects so their recurrent state is stored and loaded independently of the full-attention layers. --chunk-size can stay at its default; the connector negotiates it up to a multiple of N:

    lmcache server --chunk-size 256 --separate-object-groups \
    --l1-size-gb 100 --eviction-policy LRU

  2. vLLM --mamba-cache-mode align --enable-prefix-caching — align 是强制性的（GDN 后端不支持 all 模式）：:

    vllm serve <model> \
    --enable-prefix-caching --mamba-cache-mode align \
    --kv-transfer-config \
    '{"kv_connector":"LMCacheMPConnector", "kv_role":"kv_both"}'

不需要 --no-disable-hybrid-kv-cache-manager 或 attention-backend 标志；LMCacheMPConnector 声明支持混合模型，vLLM 会自动选择 GDN 后端。

注意事项#

  • 生成在缓存和新运行之间不是位精确的：GDN 后端不支持 vLLM 的批量不变模式。请使用分数级比较进行验证（参见 验证正确性），而不是 token 级差异。

  • 缓存页面是 字节不透明 的，因此内容感知特性（CacheGen 压缩、CacheBlend）不适用，并且缓存条目不得在具有不同注意力后端或内核块大小的引擎之间共享。

  • 这些模型中的几个是 视觉-语言 模型（它们加载一个视觉塔）。经过验证的支持路径是 文本 KV 缓存；图像/视频 KV 缓存尚未经过验证。

  • vLLM 的 Mamba 前缀缓存在 align 模式下被标记为实验性。

See the Qwen3.5 / Qwen3.6 / Qwen3.8 recipe for the validated end-to-end commands and the per-model block sizes.

尚未支持的内容#

At registration the connector rejects only the group specs the transfer path cannot serve correctly (validate_kv_cache_groups in lmcache/integration/vllm/kv_cache_group_edits.py):

  • Encoder-decoder cross-attention caches (CrossAttentionSpec).

  • Mamba groups in none mode (mamba_cache_mode other than "align" or "all"): they keep no reusable per-block state snapshots.

DeepSeek-V4-style compressed / indexer caches are supported — specs that declare slot compression (compress_ratio > 1 / tq_slot_size > 0) are served by the compression path in lmcache.v1.kv_layer_groups and are not rejected. See the DeepSeek-V4-Flash and GLM 5.2 recipes for validated end-to-end commands.

验证正确性#

为了确保混合模型的 KV 被正确缓存和重用，您可以将冷启动与从 LMCache 提供的运行进行比较：

  1. 对 vLLM + LMCache 运行评估（例如在 gsm8k 上使用 lm_eval）。这会计算 KV Cache 并 存储 在 LMCache 中。

  2. 仅重置 vLLM 的本地前缀缓存，保持 LMCache 管理的缓存不变（需要以 VLLM_SERVER_DEV_MODE=1 启动 vLLM）:

    curl -X POST http://localhost:8000/reset_prefix_cache

    省略 reset_external=true 查询参数，以便保留 LMCache 缓存。

  3. 重新运行相同的评估。vLLM 现在在其本地缓存中未命中，因此前缀 KV 从 LMCache 中 检索。得分应与第一次运行匹配。

该项目以 hma_lm_eval 持续集成测试的形式提供此验证（参见 .buildkite/k3_tests/multiprocess）。

另请参阅#

  • 快速入门 — 启动 LMCache 服务器和 vLLM 客户端。

  • 关于如何检测和处理组的设计说明：docs/design/integration/vllm/hybrid-kv-cache-groups.md 在源代码树中。