LMCache Plugin Framework

The LMCache plugin system allows developers to extend functionality by running custom scripts alongside LMCache processes. Plugins can be written in Python and Bash for now, and are managed by the PluginLauncher class.

Key Use Cases

• Start metric reporters for centralized monitoring

• Implement log reporters for log collection systems

• Report process-level metrics to alerting systems

• Implement health checks and service discovery

• Custom cache management operations

Configuration

Plugins are configured through environment variables and configuration files:

Environment Variables: - LMCACHE_PLUGIN_ROLE: Process role (e.g., SCHEDULER, WORKER) - LMCACHE_PLUGIN_CONFIG: JSON string containing plugin configuration - LMCACHE_PLUGIN_WORKER_ID: Current worker ID - LMCACHE_PLUGIN_WORKER_COUNT: Total worker count in cluster

Configuration File (lmcache.yaml):

plugin_locations: ["/path/to/plugins"]
extra_config:
  custom_setting: value

Plugin Naming Convention

Plugin filenames determine execution targets:

Role-Specific Plugins: - Format: <ROLE>[_<WORKER_ID>][_<DESCRIPTION>].<EXTENSION> - Examples:

• scheduler_foo_plugin.py: Runs only on SCHEDULER

• worker_0_test.sh: Runs only on worker ID 0

• all_plugin.sh: Runs on all workers

Notes: - Role names are case-insensitive - Worker ID must be numeric when specified - To target a specific worker ID, the filename must have at least three parts separated by underscores (e.g., worker_<ID>_<DESCRIPTION>.ext). A file named worker_<DESCRIPTION>.ext will run on all workers.

Execution Model

1. Interpreter Detection: - Uses shebang line (e.g., #!/opt/venv/bin/python) - Fallback interpreters:

   • .py → python

   • .sh → bash

2. Output Handling: - Stdout/stderr captured continuously - Logged with plugin name prefix

3. Process Management: - Launched as subprocesses - Terminated when parent process exits

Example Plugins

Python Plugin (scheduler_foo_plugin.py):

#!/opt/venv/bin/python
# SPDX-License-Identifier: Apache-2.0
"""Example plugin for LMCache system
This plugin runs continuously and exits when parent process terminates"""

# Standard
import json
import os
import signal
import time

# First Party
from lmcache.integration.vllm.utils import lmcache_get_config
from lmcache.v1.config import LMCacheEngineConfig


# Graceful exit handler
def handle_exit(signum, frame):
    print("Received termination signal, exiting...")
    exit(0)


signal.signal(signal.SIGTERM, handle_exit)

role = os.getenv("LMCACHE_PLUGIN_ROLE")
worker_id = os.getenv("LMCACHE_PLUGIN_WORKER_ID")
worker_count = os.getenv("LMCACHE_PLUGIN_WORKER_COUNT")
config_str = os.getenv("LMCACHE_PLUGIN_CONFIG")
try:
    config = LMCacheEngineConfig.from_json(config_str)
except json.JSONDecodeError as e:
    print(f"Error parsing LMCACHE_PLUGIN_CONFIG: {e}")
    config = lmcache_get_config()

print(
    f"Python plugin running with role: {role}, worker_id: {worker_id}, "
    f"worker_count: {worker_count}"
)
print(f"Config: {config}")

# Main loop
loop_count = 0
while True:
    print(f"Scheduler plugin is running... (loop_count: {loop_count})")
    loop_count += 1
    time.sleep(10)

Bash Plugin (all_plugin.sh):

#!/bin/bash
# Example plugin for LMCache system
# This plugin runs continuously and exits when parent process terminates

# Handle termination signal
trap "echo 'Received termination signal, exiting...'; exit 0" SIGTERM

role="$LMCACHE_PLUGIN_ROLE"
worker_id="$LMCACHE_PLUGIN_WORKER_ID"
worker_count="$LMCACHE_PLUGIN_WORKER_COUNT"
config="$LMCACHE_PLUGIN_CONFIG"

echo "All plugin started for role: $role, worker ID: $worker_id, worker count: $worker_count"
echo "All plugin accept LMCache Config: $config"

loop_count=0
while true; do
    echo "All plugin is running for ${role} ${worker_id}...(loop_count: ${loop_count})"
    loop_count=$((loop_count + 1))
    sleep 10
done

Best Practices

1. Keep plugins lightweight and efficient

2. Use descriptive naming conventions

3. Implement graceful error handling

4. Include shebang for portability

5. Validate configuration inputs

6. Add timeout mechanisms for long operations