Compression

KV cache compression can greatly reduces the size of the cache, which can be beneficial for both storage/memory usage and loading speed. Currently, we support the following compression algorithms:

• CacheGen: CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving

• CacheGen