Platform

LMCache is device-neutral and supports mainstream accelerator hardware. The platform layer detects the active device backend and provides the device-specific operations used by LMCache without tying the rest of LMCache to a single hardware vendor.

Device backends are provided in two ways:

• Built-in backends are maintained in the LMCache repository and released with LMCache.

• Device plugins are vendor-maintained Python packages installed alongside LMCache. They register a backend through the lmcache.device_plugins entry-point group and can follow an independent release cycle.

Built-in devices

LMCache currently includes the following built-in device backends:

• CPU

• CUDA

• HPU

• MUSA

• Neuron

• NPU

• RBLN

• ROCm

• SUPA

• XPU

Using a device plugin

Install the vendor’s device-plugin wheel in the same Python environment as LMCache and the serving engine:

python -m pip install lmcache-foo-device

Replace lmcache-foo-device with the package name published by the device vendor.

Restart the LMCache and serving-engine processes after installation. LMCache discovers installed lmcache.device_plugins entry points automatically and selects the available backend. No LMCache source change or manual registration is required.

To verify which device plugins are installed:

python - <<'PY'
from importlib.metadata import entry_points

for plugin in entry_points(group="lmcache.device_plugins"):
    print(plugin.name, "=", plugin.value)
PY

If more than one available backend uses the same device type, select the intended plugin by its backend name before starting LMCache and the serving engine:

export LMCACHE_DEVICE_BACKEND=foo

Replace foo with the plugin’s registered backend name.

Only install plugins from trusted sources because entry-point packages execute Python code when the platform registry initializes. Device vendors and backend developers can follow Adding a New Device Backend to implement and package a plugin.