LMCache Backend Interface#

class StorageBackendInterface(dst_device: str = 'cuda')[source]#
abstract close() → None[source]#

Close the storage backend.

abstract contains(key: CacheEngineKey) → bool[source]#

Check whether key is in the storage backend.

abstract exists_in_put_tasks(key: CacheEngineKey) → bool[source]#

Check whether key is in the ongoing put tasks.

abstract get_blocking(key: CacheEngineKey) → MemoryObj | None[source]#

A blcocking function to get the kv cache from the storage backend.

Parameters:

key (CacheEngineKey) – The key of the MemoryObj.

Returns:

MemoryObj. None if the key does not exist.

abstract submit_prefetch_task(key: CacheEngineKey) → Future | None[source]#

An async function to get the MemoryObj from the storage backend.

Parameters:

key (CacheEngineKey) – The key of the MemoryObj.

Returns:

a future object. None if the key does not exist.

abstract submit_put_task(key: CacheEngineKey, obj: MemoryObj) → Future | None[source]#

An async function to put the MemoryObj into the storage backend.

Parameters:

  • key (CacheEngineKey) – The key of the MemoryObj.

  • obj (MemoryObj) – The MemoryObj to be stored.

Returns:

a future object

class StorageManager(config: LMCacheEngineConfig, metadata: LMCacheEngineMetadata, allocator: MemoryAllocatorInterface, lookup_server: LookupServerInterface | None = None)[source]#

The StorageManager is responsible for managing the storage backends.

allocate(shape: Size, dtype: dtype, eviction=True) → MemoryObj | None[source]#

Allocate memory object with memory allocator. Use LRU evictor if eviction is enabled.

close()[source]#
contains(key: CacheEngineKey, search_range: List[str] | None = None) → bool[source]#

Check whether the key exists in the storage backend.

Parameters:

  • key (CacheEngineKey) – The key to check.

  • search_range (Optional[List[str]]) – The range of storage backends

to search in. Should be a subset of [“Hot”, “LocalDiskBackend”] for now. If None, search in all backends.

return: True if the key exists in the specified storage backends.

get(key: CacheEngineKey) → MemoryObj | None[source]#

Blocking function to get the memory object from the storages.

prefetch(key: CacheEngineKey) → None[source]#

Launch a prefetch request in the storage backend. Non-blocking

prefetch_callback(future, key)[source]#

Update metadata after prefetch.

put(key: CacheEngineKey, memory_obj: MemoryObj) → None[source]#

Non-blocking function to put the memory object into the storages. Do not store if the same object is being stored (handled here by storage manager) or has been stored (handled by storage backend).