Compression#

KV cache compression can greatly reduces the size of the cache, which can be beneficial for both storage/memory usage and loading speed. Currently, we support the following compression algorithms: