lmcache query#

The lmcache query command runs a single, read-only query and reports the result as a metrics report. It has three targets:

lmcache query {engine,coordinator,kvcache} [options]

engine — send one OpenAI-compatible inference request to a serving engine's HTTP API and report token and latency metrics.
coordinator — read one of the MP coordinator's read-only HTTP APIs.
kvcache — 查询 KV Cache 端点（尚未实现）。

查询引擎#

query engine 子命令向引擎 API 发送一个请求并报告指标。 --prompt 支持占位符： {lmcache} 加载 lmcache/cli/documents/lmcache.txt，并且可以通过 --documents NAME=PATH 传递自定义文档。提示的令牌计数直接来自引擎报告的使用数据 (stream_options: {include_usage: true})。

lmcache query engine --url http://localhost:8000/v1 \
  --prompt "{lmcache} Summarize LMCache usage." \
  --format terminal \
  --max-tokens 128

================= Query Engine =================
Model:                         facebook/opt-125m
Input tokens:                                618
--------------- Latency Metrics ----------------
Output tokens:                                 9
TTFT (ms):                                 26.88
TPOT (ms/token):                            0.91
Total latency (ms):                        35.05
Throughput (tokens/s):                   1100.64
================================================

选项#

标志	必需的	描述
`--url URL`	是	服务引擎基础 URL（例如 `http://localhost:8000/v1`）。
`--prompt TEXT`	是	带有可选 `{name}` 占位符的提示文本。 `{lmcache}` 展开为捆绑的示例文档。
`--model ID`	不	服务引擎的模型 ID。如果省略，将根据引擎报告的使用情况自动检测。
`--max-tokens N`	不	最大完成令牌数（默认：128）。
`--timeout SECS`	不	HTTP 超时（单位：秒，默认值：30）。
`--documents NAME=PATH`	不	在 `--prompt` 中加载 `{NAME}` 的文件文本。接受一个或多个 `NAME=PATH` 值。
`--completions`	不	仅使用 `POST /v1/completions`。
`--chat-first`	不	首先尝试 `/v1/chat/completions`，然后回退到 `/v1/completions`。
`--format`	不	输出格式：`terminal`（默认）或`json`。
`--output PATH`	不	将指标保存到文件中（格式遵循 `--format`）。
`-q` / `--quiet`	不	抑制标准输出。仅返回退出代码。

query coordinator#

The query coordinator subcommand reads one of the MP coordinator's read-only HTTP APIs and renders the reply as a metrics report — the same information curl returns, but aligned into columns and with byte counts and ratios already formatted. Pick the API with --api; everything else is optional.

lmcache query coordinator --api NAME [--url URL] [options]

Only reads are exposed. The coordinator's mutating routes are either server-to-coordinator plumbing (POST /events, POST /instances, heartbeats) or belong to a command that owns the action — quotas are written with lmcache quota. See 多服务器协调 for the HTTP surface itself and the meaning of each field.

The default --url is http://127.0.0.1:9300, matching the coordinator's default port. A URL without a scheme is assumed to be http://, and a trailing slash is ignored, so coordinator:9300 and http://coordinator:9300/ are both accepted.

APIs#

`--api`	Reads	Extra flags	Reports
`usage`	`/instances/usage`, or `/instances/{id}/usage` with `--instance`	`--instance` (optional)	Per-compartment occupancy against declared capacity, busiest first.
`instances`	`/instances`	—	Registered MP servers with their addresses and P2P URLs.
`health`	`/healthz`	—	Coordinator liveness.
`directory`	`/directory/stats`	—	Key-directory size and blend-index counts.
`keys`	`/directory/keys`	`--limit` (default: 20)	A page of directory keys and where each one is placed.
`quota`	`/quota`, or `/quota/{salt}` with `--cache-salt`	`--cache-salt` (optional)	Per-salt L2 usage against quota.
`quota-config`	`/quota/config`	—	The default limit applied to salts with no explicit quota.
`prefetch`	`/cache/prefetches/{instance}/{request_id}`	`--instance` and `--request-id` (required)	One warm-prefetch request's progress.
`metrics`	`/metrics`	—	Prometheus text, passed through verbatim.

Fleet memory#

--api usage is the everyday view: one row per memory compartment (tier/backend) per server, sorted so the fullest compartment is first.

$ lmcache query coordinator --api usage

============== Coordinator: usage ==============
instance        compartment      used  capacity    ratio
--------------------------------------------------------
mp-gpu7         l1/dram      48.00 GB  64.00 GB    75.0%
mp-gpu8         l1/dram       2.00 GB  64.00 GB     3.1%
mp-gpu7         l2/fs        12.00 GB        --  unknown
(fleet-shared)  l2/s3         7.00 GB        --  unknown
================================================

Reading the table:

A capacity of -- and a ratio of unknown mean the server never declared a capacity for that compartment — an unmeasured tier, not an empty one. A 0 would be misleading, so it is never printed there.
A compartment shared by the whole fleet (e.g. one S3 bucket behind every server) is attributed to (fleet-shared) rather than to any one server.
Numeric columns are right-aligned so sizes line up on the decimal point.

Add --instance to narrow the report to one server. The instance id must be known to the coordinator — registered, holding bytes, or having declared capacity — otherwise the coordinator answers 404 and the command exits 1:

$ lmcache query coordinator --api usage --instance mp-gpu7

============== Coordinator: usage ==============
instance  compartment      used  capacity    ratio
--------------------------------------------------
mp-gpu7   l1/dram      48.00 GB  64.00 GB    75.0%
mp-gpu7   l2/fs        12.00 GB        --  unknown
================================================

Fleet membership#

$ lmcache query coordinator --api instances

============ Coordinator: instances ============
instance  address        mq port  p2p url
-----------------------------------------------------
mp-gpu7   10.0.0.7:8101     8201  tcp://10.0.0.7:8301
mp-gpu8   10.0.0.8:8101       --  --
================================================

-- marks a value the server did not advertise (no MQ port, no P2P URL), kept distinct from 0 and from an empty string.

Quota and usage#

With no --cache-salt, the fleet-wide listing:

$ lmcache query coordinator --api quota

============== Coordinator: quota ==============
Total usage (GiB):                         19.00
cache salt  usage GiB  quota GiB  quota set
-------------------------------------------
tenant-a        12.50      20.00  yes
(default)        6.50       0.00  no
================================================

(default) is the un-salted (empty-string) tenant. quota set is no when no explicit quota exists for that salt — such a salt is governed by --api quota-config instead, so a 0.00 quota column there does not mean "zero bytes allowed".

With --cache-salt, one tenant:

$ lmcache query coordinator --api quota --cache-salt tenant-a

============== Coordinator: quota ==============
Cache salt:                             tenant-a
Quota (GiB):                               20.00
Quota set:                                  True
Usage (GiB):                               12.50
================================================

备注

To address the un-salted tenant, pass the sentinel the HTTP API uses: --cache-salt _default. An empty --cache-salt '' builds the path /quota/, which does not address the empty salt.

The default limit for salts with no explicit quota:

$ lmcache query coordinator --api quota-config

========== Coordinator: quota-config ===========
Default limit (GiB):               none (exempt)
================================================

none (exempt) means the default is unset, so unquota'd salts are exempt from eviction. That is distinct from a default of 0.0, which makes every byte under an unquota'd salt evictable. See 多服务器协调 for why a freshly restarted coordinator starts out exempt.

Key directory#

Directory size, and how much of it is fragment-matchable by CacheBlend:

$ lmcache query coordinator --api directory

============ Coordinator: directory ============
Keys:                                      18422
Placements:                                20117
----------------- Blend index ------------------
Contents:                                     91
Chunks:                                     1740
Claims:                                     1802
Namespaces:                                    3
Table size:                                 1740
================================================

Claims counts (chunk, namespace) pairs and Namespaces the distinct (model, salt, world size) triples holding indexed content. A match is only offered to a requester whose namespace claims the chunk, so Claims above Chunks means tenants are sharing content — the signal Chunks alone cannot give.

A page of keys and their placements. --limit sets the page size (default 20; the endpoint accepts 1–10000, and a value outside that range is rejected by the coordinator and exits 1):

$ lmcache query coordinator --api keys --limit 2

============== Coordinator: keys ===============
Matching keys:                             18422
chunk         model                    rank  salt       placements
---------------------------------------------------------------------------------------
abababababab  meta-llama/Llama-3.1-8B     0  tenant-a   mp-gpu7:l1/dram, (shared):l2/s3
cdcdcdcdcdcd  meta-llama/Llama-3.1-8B     0  (default)  mp-gpu8:l1/dram
================================================

Matching keys is the total in the directory, not the number of rows shown. The chunk hash is truncated to 12 hex characters — enough to correlate with a log line, without pushing the placements column off the terminal. A placement with no owning instance (a fleet-shared backend) reads as (shared). The directory is a live structure, so successive pages may skip or repeat keys.

Prefetch progress#

--api prefetch polls a warm prefetch submitted earlier via POST /cache/prefetches. Both --instance and --request-id are required; omitting either exits 2 without issuing a request.

$ lmcache query coordinator --api prefetch --instance mp-gpu7 --request-id abc123

============ Coordinator: prefetch =============
Status:                                completed
Found keys:                                   12
Total keys:                                   12
================================================

While the load is still running, only Status: pending is reported. The labels come from the reply itself, so a server that adds fields will show them without a CLI change.

警告

The first poll that observes completion drops the job on the MP server. Polling the same --request-id again returns 404 and exits 1.

Prometheus metrics#

--api metrics is the one API that is not a metrics report: the coordinator's /metrics body is written to stdout unchanged, so it can be piped to other tools. --format, --output, and --quiet do not apply to it.

lmcache query coordinator --api metrics | promtool check metrics
lmcache query coordinator --api metrics | grep lmcache_coordinator

选项#

标志	必需的	描述
`--api NAME`	是	Which API to read. One of `usage`, `instances`, `health`, `directory`, `keys`, `quota`, `quota-config`, `prefetch`, `metrics`.
`--url URL`	不	Coordinator base URL (default: `http://127.0.0.1:9300`). The scheme may be omitted.
`--instance ID`	不	Instance id. Narrows `--api usage` to one server; required by `--api prefetch`.
`--cache-salt SALT`	不	Narrows `--api quota` to one tenant. Use `_default` for un-salted traffic.
`--request-id ID`	不	Prefetch request id. Required by `--api prefetch`.
`--limit N`	不	Rows to request for `--api keys` (default: 20).
`--format`	不	Output format: `terminal` (default) or `json`. Ignored by `--api metrics`.
`--output PATH`	不	Save the report to a file (format follows `--format`). Ignored by `--api metrics`.
`-q` / `--quiet`	不	Suppress stdout output. Exit code only. Ignored by `--api metrics`.

JSON output#

--format json renders the same report as a JSON object. Table APIs become a list of row objects under the table's key:

$ lmcache query coordinator --api usage --format json
{
  "title": "Coordinator: usage",
  "metrics": {
    "usage": [
      {
        "instance": "mp-gpu7",
        "compartment": "l1/dram",
        "used": "48.00 GB",
        "capacity": "64.00 GB",
        "ratio": "75.0%"
      },
      {
        "instance": "mp-gpu8",
        "compartment": "l1/dram",
        "used": "2.00 GB",
        "capacity": "64.00 GB",
        "ratio": "3.1%"
      },
      {
        "instance": "mp-gpu7",
        "compartment": "l2/fs",
        "used": "12.00 GB",
        "capacity": "--",
        "ratio": "unknown"
      },
      {
        "instance": "(fleet-shared)",
        "compartment": "l2/s3",
        "used": "7.00 GB",
        "capacity": "--",
        "ratio": "unknown"
      }
    ]
  }
}

警告

Table rows carry display strings, not raw numbers: "48.00 GB", "75.0%", "12.50", "yes", and the placeholders "--" / "unknown". They are meant for reading and for grepping, not for arithmetic. For machine-readable values, curl the endpoint directly:

curl -s http://127.0.0.1:9300/instances/usage | jq \
  '.instances[].modules[] | {tier, backend, used_bytes, usage_ratio}'

Exit codes#

Code	Meaning
`0`	Success.
`1`	The coordinator was unreachable, returned a non-2xx status (e.g. an unknown instance id or prefetch request), or replied with something that was not JSON.
`2`	Usage error: a required flag for the chosen `--api` is missing (e.g. `--api prefetch` without `--request-id`). No request is sent.

This makes the command usable as a readiness check:

until lmcache query coordinator --api health -q; do sleep 1; done

Common patterns#

Find the fullest server before pinning work to it:

lmcache query coordinator --api usage | head -5

Watch a fleet fill up:

watch -n 5 'lmcache query coordinator --api usage'

Check one tenant against its budget:

lmcache query coordinator --api quota --cache-salt tenant-a
lmcache query coordinator --api quota-config   # what un-quota'd salts get

Point at a coordinator in Kubernetes:

lmcache query coordinator --api instances \
  --url http://coordinator.default.svc:9300

Limitations#

Reports are a formatted subset of each reply, not the whole thing. --api usage does not surface the registered flag, so a server that deregistered while its L2 bytes survive appears in the table like any other; cross-check with --api instances. --api directory omits the per-instance L1 key counts, and --api keys omits each key's token count.
--api keys requests a page size only. The endpoint's tier, instance_id, backend, and offset filters are not exposed — curl /directory/keys for those.
--api quota reports the L2 tier, which is the tier quotas are enforced on. The endpoint's tier parameter is not exposed.