lmcache.experimental.server.storage_backend package#

Submodules#

lmcache.experimental.server.storage_backend.abstract_backend module#

class lmcache.experimental.server.storage_backend.abstract_backend.LMSBackendInterface[source]#
abstract close()[source]#

Do the cleanup things Children classes should override this method if necessary

abstract contains(key: CacheEngineKey) → bool[source]#

Query if a key is in the cache or not

abstract get(key: CacheEngineKey) → LMSMemoryObj | None[source]#

Retrieve LMSMemoryObj by the given key

Input:

key: the CacheEngineKey

Output:

An LMSMemoryObj object that contains the KV cache bytearray with the some metadata

abstract list_keys() → List[CacheEngineKey][source]#

List all keys in the cache server

Output:

All keys in the cache server

abstract put(client_meta: ClientMetaMessage, kv_chunk_bytes: bytearray) → None[source]#

Store the KV cache of the tokens into the cache server.

Parameters:

  • key – the key of the token chunk, in the format of CacheEngineKey

  • client_meta – metadata sent by the client

  • kv_chunk_bytes – the kv cache (bytearray) of the token chunk

Returns:

None

lmcache.experimental.server.storage_backend.local_backend module#

class lmcache.experimental.server.storage_backend.local_backend.LMSLocalBackend[source]#

Bases: LMSBackendInterface

close()[source]#

Do the cleanup things Children classes should override this method if necessary

contains(key: CacheEngineKey) → bool[source]#

Query if a key is in the cache or not

get(key: CacheEngineKey) → LMSMemoryObj | None[source]#

Retrieve LMSMemoryObj by the given key

Input:

key: the CacheEngineKey

Output:

An LMSMemoryObj object that contains the KV cache bytearray with the some metadata

list_keys() → List[CacheEngineKey][source]#

List all keys in the cache server

Output:

All keys in the cache server

put(client_meta: ClientMetaMessage, kv_chunk_bytes: bytearray) → None[source]#

Store the KV cache of the tokens into the cache server.

Parameters:

  • key – the key of the token chunk, in the format of CacheEngineKey

  • client_meta – metadata sent by the client

  • kv_chunk_bytes – the kv cache (bytearray) of the token chunk

Returns:

None

remove(key: CacheEngineKey) → None[source]#

Module contents#

lmcache.experimental.server.storage_backend.CreateStorageBackend(device: str) → LMSBackendInterface[source]#