LMCache Backend#
- class lmcache.storage_backend.abstract_backend.LMCBackendInterface[source]#
- batched_get(keys: Iterable[CacheEngineKey]) Iterable[Tensor | None] [source]#
Retrieve the kv cache chunks by the given keys in a batched manner
- Parameters:
keys – the iterator of keys of the token chunks, including prefix hash and format
- Returns:
the iterator of kv cache of the token chunks, in the format of a big tensor and None if the key is not found
- batched_put(keys_and_chunks: Iterable[Tuple[CacheEngineKey, Tensor]], blocking=True) int [source]#
Store the multiple keys and KV cache chunks into the cache engine in a batched manner.
- Parameters:
keys – the iterable of keys of the token chunks, in the format of CacheEngineKey
kv_chunks – the iterable of kv cache of the token chunks, in the format of a big tensor
blocking – whether to block the call before the operation is completed
- Returns:
the number of chunks are stored
- abstract close()[source]#
Do the cleanup things Children classes should override this method if necessary
- abstract contains(key: CacheEngineKey) bool [source]#
Query if a key is in the cache or not
- abstract get(key: CacheEngineKey) Tensor | None [source]#
Retrieve the KV cache chunk by the given key
- Parameters:
key – the key of the token chunk, including prefix hash and format
- Returns:
the kv cache of the token chunk, in the format of a big tensor and None if the key is not found
- abstract put(key: CacheEngineKey, kv_chunk: Tensor, blocking=True) None [source]#
Store the KV cache of the tokens into the cache engine.
- Parameters:
key – the key of the token chunk, in the format of CacheEngineKey
kv_chunk – the kv cache of the token chunk, as a big tensor.
blocking – to block the call before the operation is completed.
- Returns:
None
Note
The KV cache should NOT have the “batch” dimension.