Welcome to LMCache!

Redis for LLMs.

Note

We are currently in the process of upgrading our documentation to provide better guidance and examples. Some sections may be under construction. Thank you for your patience!

Star Watch Fork

LMCache lets LLMs prefill each text only once. By storing the KV caches of all reusable texts, LMCache can reuse the KV caches of any reused text (not necessarily prefix) in any serving engine instance. It thus reduces prefill delay, i.e., time to first token (TTFT), as well as saves the precious GPU cycles and memory. By combining LMCache with vLLM, LMCaches achieves 3-10x delay savings and GPU cycle reduction in many LLM use cases, including multi-round QA and RAG.

For more information, check out the following:

• LMCache blogs

• Join LMCache slack workspace

• Our papers:

  • CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving

  • CacheBlend: Fast Large Language Model Serving with Cached Knowledge Fusion

  • Do Large Language Models Need a Content Delivery Network?

Documentation

Getting Started

• Installation
  • Prerequisites
  • Setup using Python
  • Setup using Docker
• Quickstart Examples
  • KV Cache Offloading
  • KV Cache Sharing
  • Disaggregated Prefill
  • Detailed Examples
• TroubleShoot
• FAQ

KV Cache offloading and sharing

• CPU RAM
• Local storage
• Redis
• InfiniStore
• MoonCake
• ValKey

Disaggregated prefill

• Using NIXL
  • 1p1d
  • XpYd
• Using shared storage

KV Cache management

• LMCache Controller
• Lookup the KV cache
• Persist the KV cache
• Clear the KV cache

KV Cache Optimizations

• Compression
• Blending

Use LMCache in production

• Docker deployment
• Kubernetes deployment

Developer Guide

• Contributing guide
• Dockerfile

API Reference

• LMCache configurations
• Adding new storage backends

Community

• Community meetings
• Blogs