Gemma 4#

验证过的模型#

引擎文档: vLLM 支持模型中的 Gemma 4（架构 Gemma4ForConditionalGeneration 适用于 31B/E4B，Gemma4UnifiedForConditionalGeneration 适用于 12B）。

状态: 已通过 LMCache 验证。

启动 LMCache MP 服务器：

lmcache server --l1-size-gb 100 --eviction-policy LRU

使用 LMCache MP 连接器启动 vLLM：

vllm serve google/gemma-4-31B-it \
    --tensor-parallel-size 2 \
    --kv-transfer-config \
    '{"kv_connector":"LMCacheMPConnector", "kv_role":"kv_both"}'

较小的 google/gemma-4-12B-it 和 google/gemma-4-E4B-it 在单个 GPU 上运行：

vllm serve google/gemma-4-12B-it \
    --kv-transfer-config \
    '{"kv_connector":"LMCacheMPConnector", "kv_role":"kv_both"}'

调整 --tensor-parallel-size 以匹配您的硬件。有关通用 LMCache + vLLM 连接（端口、远程主机），请参见 快速入门。

如果在 vLLM 设置中遇到任何问题，请参考 vLLM Recipes 以获取更多详细信息。

状态： 未通过 LMCache 验证。

状态： 支持。TRT-LLM + LMCache 的配置方式，请参阅 快速入门。

CacheBlend 支持#

压缩支持#

方法

状态

备注

CacheGen

未验证

MTP (speculative decoding) support#

Gemma 4 supports MTP speculative decoding through separate assistant checkpoints (google/gemma-4-<size>-it-assistant), which vLLM loads as the draft model. The draft layers carry their own KV cache; LMCache detects them from vLLM's speculative_config and stores/retrieves the draft-layer KV together with the target model's -- no extra LMCache flags are required.

Status: Validated with LMCache (vLLM MP connector):

  • google/gemma-4-31B-it + google/gemma-4-31B-it-assistant (2 GPUs)

  • google/gemma-4-12B-it + google/gemma-4-12B-it-assistant (1 GPU, needs --enforce-eager; see below)

  • google/gemma-4-E4B-it + google/gemma-4-E4B-it-assistant (1 GPU)

Add to the vllm serve command shown above:

--speculative-config \
    '{"method":"mtp","model":"google/gemma-4-31B-it-assistant","num_speculative_tokens":1}' \
--attention-backend TRITON_ATTN

警告

--attention-backend TRITON_ATTN is required when MTP is enabled. The target model alone auto-selects the Triton backend (FlashAttention does not support Gemma 4's 512-dim global-attention heads), but the assistant draft model's backend selection picks FlashAttention and the engine crashes at startup with "FlashAttention forward only supports head dimension at most 256". Pinning the backend explicitly covers both models.

Validation evidence (gsm8k store-vs-retrieve, 100 samples, exact score match required under VLLM_BATCH_INVARIANT=1): 31B scored 0.78 in both the computed and the LMCache-retrieved run with the MTP acceptance rate unchanged (0.911 vs 0.910); 12B scored 0.28 in both runs (acceptance 0.854 vs 0.853); E4B scored 0.69 in both runs with acceptance rate identical (0.807). Cold-vs-warm TTFT improved 3.7x (31B) / 1.7x (12B) / 1.4x (E4B) with MTP enabled throughout.

备注

The 12B Unified assistant requires --enforce-eager on the vLLM nightly tested: its draft head applies a token-suppression list held in an unpinned CPU tensor (gemma4_mtp.py, compute_logits), which aborts CUDA graph capture with "Cannot copy between CPU and CUDA tensors during CUDA graph capture". This is an upstream vLLM issue, unrelated to LMCache; 31B and E4B assistants are unaffected.

注意事项#

  • 混合 KV Cache，具有异构块大小。 Gemma 4 交错了 sliding window 和 full attention 层，两者的头维度不同（sliding window 层为 256，full attention 层为 512），因此 vLLM 通过为这两种注意力类型分配不同的 block_size\ s（例如 google/gemma-4-E4B-it：sliding window 层为 32，full attention 层为 16）来统一物理页面大小。LMCache 按各自的块大小存储和检索每个 KV Cache 组，无需额外配置。

  • 跨层 KV 共享。 google/gemma-4-E4B-it 在层与层之间重用一些层的 KV 缓存。LMCache 仅存储拥有缓存的层；共享层的 KV 存储在相同的块中，并会自动恢复。