lmcache.storage_backend.mem_pool package

Submodules

lmcache.storage_backend.mem_pool.base_pool module

class lmcache.storage_backend.mem_pool.base_pool.BasePool

Interface for mem pool

abstract allocate(kv_chunk: Tensor) → KVObj | None

Allocate a buffer memory pointer from the memory pool.

Input:

kv_chunk: the kv tensor to be stored

Returns:

KVObj with a memory pointer (torch tensor view). None if memory is full.

Note

This does not perform the actual memory movement.

abstract free(kv_obj: KVObj)

Free the corresponding memory chunk

Input:

the KVObj to be freed

class lmcache.storage_backend.mem_pool.base_pool.KVObj(chunk_idx: int, size: int, data: torch.Tensor)

chunk_idx: int

data: Tensor

size: int

lmcache.storage_backend.mem_pool.local_pool module

class lmcache.storage_backend.mem_pool.local_pool.LocalCPUBufferPool(metadata: LMCacheMemPoolMetadata)

Bases: LocalCPUPool

allocate(kv_chunk: Tensor) → KVObj | None

Allocate a buffer memory pointer from the memory pool.

Input:

kv_chunk: the kv tensor to be stored

Returns:

KVObj with a memory pointer (torch tensor view). None if memory is full.

Note

This does not perform the actual memory movement.

class lmcache.storage_backend.mem_pool.local_pool.LocalCPUPool(metadata: LMCacheMemPoolMetadata)

Bases: LocalPool

class lmcache.storage_backend.mem_pool.local_pool.LocalGPUPool(metadata: LMCacheMemPoolMetadata)

Bases: LocalPool

only for unit testing, might not be useful in production

class lmcache.storage_backend.mem_pool.local_pool.LocalPool(metadata: LMCacheMemPoolMetadata)

Bases: BasePool

allocate(kv_chunk: Tensor) → KVObj | None

Allocate a buffer memory pointer from the memory pool.

Input:

kv_chunk: the kv tensor to be stored

Returns:

KVObj with a memory pointer (torch tensor view). None if memory is full.

Note

This does not perform the actual memory movement.

free(kv_obj: KVObj) → None

Free the corresponding memory chunk

Input:

the KVObj to be freed

init_max_chunk_num(metadata: LMCacheMemPoolMetadata) → int

Initialize the maximum number of chunks in the memory pool.

Module contents

class lmcache.storage_backend.mem_pool.KVObj(chunk_idx: int, size: int, data: torch.Tensor)

chunk_idx: int

data: Tensor

size: int

class lmcache.storage_backend.mem_pool.LocalCPUBufferPool(metadata: LMCacheMemPoolMetadata)

Bases: LocalCPUPool

allocate(kv_chunk: Tensor) → KVObj | None

Allocate a buffer memory pointer from the memory pool.

Input:

kv_chunk: the kv tensor to be stored

Returns:

KVObj with a memory pointer (torch tensor view). None if memory is full.

Note

This does not perform the actual memory movement.

class lmcache.storage_backend.mem_pool.LocalCPUPool(metadata: LMCacheMemPoolMetadata)

Bases: LocalPool

class lmcache.storage_backend.mem_pool.LocalGPUPool(metadata: LMCacheMemPoolMetadata)

Bases: LocalPool

only for unit testing, might not be useful in production

class lmcache.storage_backend.mem_pool.LocalPool(metadata: LMCacheMemPoolMetadata)

Bases: BasePool

allocate(kv_chunk: Tensor) → KVObj | None

Allocate a buffer memory pointer from the memory pool.

Input:

kv_chunk: the kv tensor to be stored

Returns:

KVObj with a memory pointer (torch tensor view). None if memory is full.

Note

This does not perform the actual memory movement.

free(kv_obj: KVObj) → None

Free the corresponding memory chunk

Input:

the KVObj to be freed

init_max_chunk_num(metadata: LMCacheMemPoolMetadata) → int

Initialize the maximum number of chunks in the memory pool.