lmcache.experimental.storage_backend package

Subpackages

• lmcache.experimental.storage_backend.evictor package
  • Submodules
  • lmcache.experimental.storage_backend.evictor.base_evictor module
    • BaseEvictor
      • BaseEvictor.update_on_hit()
      • BaseEvictor.update_on_put()
    • PutStatus
      • PutStatus.ILLEGAL
      • PutStatus.LEGAL
  • lmcache.experimental.storage_backend.evictor.lru_evictor module
    • LRUEvictor
      • LRUEvictor.update_on_hit()
      • LRUEvictor.update_on_put()
  • Module contents
    • LRUEvictor
      • LRUEvictor.update_on_hit()
      • LRUEvictor.update_on_put()
    • PutStatus
      • PutStatus.ILLEGAL
      • PutStatus.LEGAL

Submodules

lmcache.experimental.storage_backend.abstract_backend module

class lmcache.experimental.storage_backend.abstract_backend.StorageBackendInterface(dst_device: str = 'cuda')

abstract close() → None

Close the storage backend.

abstract contains(key: CacheEngineKey) → bool

Check whether key is in the storage backend.

abstract exists_in_put_tasks(key: CacheEngineKey) → bool

Check whether key is in the ongoing put tasks.

abstract get_blocking(key: CacheEngineKey) → MemoryObj | None

A blcocking function to get the kv cache from the storage backend.

Parameters:

key (CacheEngineKey) – The key of the MemoryObj.

Returns:

MemoryObj. None if the key does not exist.

abstract submit_prefetch_task(key: CacheEngineKey) → Future | None

An async function to get the MemoryObj from the storage backend.

Parameters:

key (CacheEngineKey) – The key of the MemoryObj.

Returns:

a future object. None if the key does not exist.

abstract submit_put_task(key: CacheEngineKey, obj: MemoryObj) → Future | None

An async function to put the MemoryObj into the storage backend.

Parameters:

• key (CacheEngineKey) – The key of the MemoryObj.

• obj (MemoryObj) – The MemoryObj to be stored.

Returns:

a future object

lmcache.experimental.storage_backend.local_disk_backend module

class lmcache.experimental.storage_backend.local_disk_backend.LocalDiskBackend(config: LMCacheEngineConfig, loop: AbstractEventLoop, memory_allocator: MemoryAllocatorInterface, dst_device: str = 'cuda', lookup_server: LookupServerInterface | None = None)

Bases: StorageBackendInterface

async async_load_bytes_from_disk(path: str, dtype: dtype, shape: Size) → MemoryObj | None

Async load bytearray from disk.

async_save_bytes_to_disk(key: CacheEngineKey, memory_obj: MemoryObj) → None

Convert KV to bytes and async store bytes to disk.

close() → None

Close the storage backend.

contains(key: CacheEngineKey) → bool

Check whether key is in the storage backend.

exists_in_put_tasks(key: CacheEngineKey) → bool

Check whether key is in the ongoing put tasks.

get_blocking(key: CacheEngineKey) → MemoryObj | None

Blocking get function.

insert_key(key: CacheEngineKey, memory_obj: MemoryObj) → None

load_bytes_from_disk(path: str, dtype: dtype, shape: Size) → MemoryObj | None

Load bytearray from disk.

load_disk(path: str, backend: str = 'bytes', dtype: dtype | None = None, shape: Size | None = None) → MemoryObj | None

Load KV from disk.

remove(key: CacheEngineKey) → None

submit_prefetch_task(key: CacheEngineKey) → Future | None

An async function to get the MemoryObj from the storage backend.

Parameters:

key (CacheEngineKey) – The key of the MemoryObj.

Returns:

a future object. None if the key does not exist.

submit_put_task(key: CacheEngineKey, memory_obj: MemoryObj) → Future | None

An async function to put the MemoryObj into the storage backend.

Parameters:

• key (CacheEngineKey) – The key of the MemoryObj.

• obj (MemoryObj) – The MemoryObj to be stored.

Returns:

a future object

lmcache.experimental.storage_backend.storage_manager module

class lmcache.experimental.storage_backend.storage_manager.StorageManager(config: LMCacheEngineConfig, metadata: LMCacheEngineMetadata, allocator: MemoryAllocatorInterface, lookup_server: LookupServerInterface | None = None)

The StorageManager is responsible for managing the storage backends.

allocate(shape: Size, dtype: dtype, eviction=True) → MemoryObj | None

Allocate memory object with memory allocator. Use LRU evictor if eviction is enabled.

close()

contains(key: CacheEngineKey, search_range: List[str] | None = None) → bool

Check whether the key exists in the storage backend.

Parameters:

• key (CacheEngineKey) – The key to check.

• search_range (Optional[List[str]]) – The range of storage backends

to search in. Should be a subset of [“Hot”, “LocalDiskBackend”] for now. If None, search in all backends.

return: True if the key exists in the specified storage backends.

get(key: CacheEngineKey) → MemoryObj | None

Blocking function to get the memory object from the storages.

prefetch(key: CacheEngineKey) → None

Launch a prefetch request in the storage backend. Non-blocking

prefetch_callback(future, key)

Update metadata after prefetch.

put(key: CacheEngineKey, memory_obj: MemoryObj) → None

Non-blocking function to put the memory object into the storages. Do not store if the same object is being stored (handled here by storage manager) or has been stored (handled by storage backend).

Module contents

lmcache.experimental.storage_backend.CreateStorageBackends(config: LMCacheEngineConfig, metadata: LMCacheEngineMetadata, loop: AbstractEventLoop, memory_allocator: MemoryAllocatorInterface, dst_device: str = 'cuda', lookup_server: LookupServerInterface | None = None) → OrderedDict[str, StorageBackendInterface]