Qwen3.5 / Qwen3.6 / Qwen3.8 series#

A hybrid architecture interleaving Mamba / Gated-DeltaNet (GDN) linear-attention layers with full-attention layers, shared by the Qwen3.5, Qwen3.6 and Qwen3.8 series (all use the Qwen3_5ForConditionalGeneration architecture). LMCache reinterprets the recurrent state caches as opaque pages at registration time; see Hybrid Attention 模型 for the general handling of Mamba / linear-attention models.

验证过的模型#

引擎文档： vLLM 支持的模型中的 Qwen3.5（架构 Qwen3_5ForConditionalGeneration）。

状态: 已通过 LMCache 验证。

该系列中的每个模型都需要相同的三项设置：align Mamba 缓存模式、前缀缓存，以及与 vLLM 的统一块大小匹配的分块大小。该块大小因模型而异——vLLM 在启动时会输出 Setting attention block size to N tokens：

模型

统一块大小 N

GPU

Qwen/Qwen3.8-27B

784

1

Qwen/Qwen3.6-27B

784

1

Qwen/Qwen3.5-0.8B

544

1

Set the LMCache server's --chunk-size to that N (or a multiple of it) and enable --separate-object-groups, then set vLLM's --max-num-batched-tokens to at least N. Keeping it below 2N (e.g. 2N-1) snapshots the Mamba state at every block boundary for the finest cache reuse; larger values raise prefill throughput at coarser reuse — see the note below.

Qwen3.6-27B (1 GPU, N = 784 → 2N-1 = 1567):

lmcache server --chunk-size 784 --separate-object-groups \
    --l1-size-gb 100 --eviction-policy LRU

vllm serve Qwen/Qwen3.6-27B \
    --enable-prefix-caching \
    --mamba-cache-mode align \
    --max-num-batched-tokens 1567 \
    --kv-transfer-config \
    '{"kv_connector":"LMCacheMPConnector", "kv_role":"kv_both"}'

Qwen3.8-27B (1 GPU, N = 784 → 2N-1 = 1567): same block size as Qwen3.6-27B, so the commands above apply unchanged apart from the model id.


Qwen3.5-0.8B (1 GPU, N = 544 → 2N-1 = 1087): 与上述相同，使用 --chunk-size 544 和 --max-num-batched-tokens 1087。

--mamba-cache-mode align is required (GDN does not support the all mode). --separate-object-groups (server) is required for hybrid models so the Mamba layers get their own cache objects; it is also what lets --max-num-batched-tokens exceed 2N. --max-num-batched-tokens must be at least N: align snapshots the Mamba state at scheduler-step ends on a block boundary, and the scheduler splits prefills into whole N-token blocks. Within [N, 2N) every step advances exactly one block, so LMCache snapshots every block boundary (finest reuse); prefer ``2N-1``, whose spare N-1 budget lets decodes co-schedule with a prefill block. Setting it to exactly N makes the per-step budget one block, so once any request is decoding (consuming ≥1 token of the budget) no new request can start prefill — execution serializes to one request at a time. (Benchmarked on Qwen3.6-27B: at N a cold / low-hit run ran ~7× slower with GPU batch stuck at 1; 2N-1 restored full batching. With a warm LMCache cache (~97 % hit) the gap is small since little prefill remains, but 2N-1 is the safe default.) Values ≥ 2N raise prefill throughput with larger steps but snapshot only the last block of each step, so cached prefixes align to step boundaries rather than every block. If vLLM reports "max_num_seqs exceeds available Mamba cache blocks", lower --max-num-seqs to ≤ that count (each decode sequence needs one Mamba block) or raise --gpu-memory-utilization.

有关通用 LMCache + vLLM 连接（端口、远程主机），请参见 快速入门。

状态： 未通过 LMCache 验证。

状态： 支持。TRT-LLM + LMCache 的配置方式，请参阅 快速入门。

CacheBlend 支持#

不支持：混合组的缓存页面是字节不透明的（请参见注意事项）。

压缩支持#

方法

状态

备注

CacheGen

不支持

混合组的缓存页面是字节不透明的。

注意事项#

  • 缓存运行与全新运行的生成结果不保证逐位一致：GDN 后端不支持 vLLM 的批不变模式。预期结果在得分层面等价，而非 token 层面严格一致（CI 验证门为 hma_lm_eval_qwen3_5 gsm8k 存储与检索对比）。

  • Mamba 和 full-attention 组的缓存页面是字节不透明视图，因此不适用内容感知处理，缓存条目不得在具有不同注意力后端或内核块大小的引擎之间共享。

  • vLLM 的 Mamba 前缀缓存在 align 模式下属于实验性功能。

  • Qwen/Qwen3.6-27B and Qwen/Qwen3.8-27B are vision-language models (they load a vision tower); the LMCache validation covers text generation (the hma_lm_eval_qwen3_5 gsm8k store-vs-retrieve gate runs on Qwen3.5 / Qwen3.6). Caching of image/video KV is not validated.