原始块（Rust）#

一个内置的 L2 适配器，通过 Rust 原始设备 I/O 接口将 KV 对象存储在原始块设备或预设大小文件的固定大小槽位中。它复用现有的原始块元数据检查点模型，并在预取期间直接写入调用方提供的加载缓冲区。

必填字段：

  • device_path: 原始设备路径或预设大小的文件路径。

  • slot_bytes: 固定的槽大小（以字节为单位）。必须与 block_align 对齐。

可选字段：

  • capacity_bytes：可用设备字节的可选上限。默认值 0 表示使用完整的设备/文件大小。

  • use_odirect: true 或 false（默认 true）。

  • block_align: Device alignment in bytes (default 4096). Must be a power of two.

  • header_bytes: 每个槽位的头部预留大小（默认 4096）。

  • meta_total_bytes: 保留的元数据检查点区域（默认 256MiB）。

  • meta_magic / meta_version: 元数据检查点的标识符与版本号调节参数。

  • meta_checkpoint_interval_sec / meta_idle_quiet_ms / meta_enable_periodic / meta_verify_on_load: 从旧版原始块后端沿用的检查点与恢复控制参数。

  • load_checkpoint_on_init: 在启动时加载现有的设备元数据检查点（默认值为 true）。设置为 false 以从空的内存索引开始。

  • enable_zero_copy: 尽可能尝试对齐的直接缓冲区 I/O。

  • io_engine: Rust 原始块 I/O 引擎。有效值为 "posix"（默认同步 pread/pwrite 路径），"io_uring"（直接 Rust io_uring 系统调用路径）。

  • use_uring_cmd: 通过 io_uring 命令接口启用 NVMe 直通以直接访问设备。需要 io_engine="io_uring" 和 NVMe 字符设备节点（例如，/dev/ng0n1）。

  • iouring_queue_depth: io_engine="io_uring" 的队列深度。

  • max_data_transfer_size: use_uring_cmd=true 的最大数据传输大小。大数据传输会被拆分成适合设备限制的小块。

  • fdp_enabled: Enables NVMe Flexible Data Placement (FDP) discovery and non-zero placement identifier registration. cache_salt values with ":" bucket prefixes use FDP placement by default. Requires io_engine="io_uring" and use_uring_cmd=true.

  • fdp_placement_ids: Optional non-zero placement identifier list for KV data placement. If omitted, the adapter uses all device-reported non-zero identifiers except meta_checkpoint_placement_id.

  • fdp_data_placement_policy: KV data placement policy. "none" omits FDP directives for KV data writes. "cache_salt_prefix" assigns case-insensitive cache_salt prefixes to FDP placement identifiers. "cache_salt_rank" keeps that bucket isolation and further separates rank-local write streams within each bucket when identifiers are available. The default is "cache_salt_prefix" when fdp_enabled=true.

  • fdp_slot_reuse_policy: Free-slot reuse policy. "pid_affinity" prefers a free slot last assigned to the same FDP placement identifier before falling back to any free slot. "none" disables affinity-based reuse. The default is "pid_affinity" when fdp_enabled=true and "none" otherwise.

  • meta_checkpoint_placement_id: Optional non-zero placement identifier for metadata checkpoint payload/header writes. Omit it to keep checkpoint writes on default NVMe placement.

  • num_store_workers / num_lookup_workers / num_load_workers: 每种操作类型的工作线程数量。

注意：

  • raw_block 是一个由服务端管理的 MP 适配器。它 不 支持 MP 模式下按 TP 粒度配置设备路径映射。

  • raw_block 在所有支持的引擎中保持 "type": "raw_block"。

  • raw_block 负责设备上的槽分配、检查点和通过 RawBlockCore 的恢复。槽回收由共享/全局 L2 逐出控制器或显式的 delete() 调用驱动。

  • slot_bytes, header_bytes, and meta_total_bytes must be multiples of block_align.

  • 如果启用了 use_odirect，则服务器的 --l1-align-bytes 应至少为 block_align。

  • With O_DIRECT, raw-block I/O rejects offsets and total I/O lengths that are not multiples of block_align. Misaligned write buffers use an aligned bounce buffer.

  • persist_enabled 对此适配器必须保持为 true。

  • 对于 use_uring_cmd=true，device_path 必须使用 NVMe 字符设备节点（例如，/dev/ng0n1），而不是块设备节点（/dev/nvme0n1）。字符设备提供直接的 NVMe 命令直通。

  • For use_uring_cmd=true, block_align must be a multiple of the NVMe namespace LBA size. An incompatible value is rejected when the device opens.

  • use_uring_cmd 需要设置 io_engine="io_uring"。

  • When use_uring_cmd=true, use_odirect is ignored for NVMe namespace character devices. FDP examples set use_odirect=false because io_uring_cmd uses NVMe passthrough rather than the POSIX write path.

  • FDP registers only non-zero placement identifiers. fdp_placement_ids is the KV data placement pool: if omitted, all discovered non-zero identifiers except meta_checkpoint_placement_id are used; if provided, every identifier must be reported by the device and must not contain 0.

  • meta_checkpoint_placement_id must not overlap with fdp_placement_ids. Keeping metadata checkpoints and KV data on separate placement identifiers avoids mixing long-lived raw-block metadata with cache data buckets.

  • With fdp_data_placement_policy="cache_salt_prefix", cache_salt values containing ":" are bucketed by the prefix before ":". The bucket name is case-insensitive, so RAG:app1, rag:app2, and rag: share the same bucket. Values without ":" and values with an empty prefix use no FDP directive. Buckets are assigned exclusive FDP placement identifiers in first-seen order while identifiers are available.

  • If the number of discovered cache-salt buckets exceeds the number of registered FDP placement identifiers, additional buckets continue to store with no FDP directive and the adapter emits one warning. report_status() exposes a fallback count and a bounded bucket sample instead of retaining the complete fallback bucket set. Empty cache_salt values also use default NVMe placement with no directive.

  • With fdp_data_placement_policy="cache_salt_rank", placement is keyed by the case-insensitive cache_salt bucket prefix and the local rank encoded in kv_rank. The suffix after ":" remains part of the object key but does not affect FDP placement. For example, batch:app-a and batch:app-b share the batch bucket: local rank 0 from both applications shares one placement identifier, local rank 1 shares another, and so on. Each bucket can use at most the detected visible GPU count worth of placement identifiers. Extra ranks, or buckets that cannot obtain an identifier, store with no FDP directive.

  • Sharing placement identifiers within a bucket can be intentional because the available FDP placement-identifier pool is finite. Operators can assign the same prefix to applications with similar cache lifetimes, or use a distinct prefix for each application when placement isolation is required. For example, app01:<suffix> through app16:<suffix> can provide 16 application buckets with eight rank streams each, consuming up to 128 placement identifiers when at least 128 identifiers are available and the adapter detects at least eight visible GPUs. Applications using the same prefix share the placement identifiers for equal local ranks instead of receiving separate sets.

  • Bucket-to-placement assignments are process-local. Restart recovery does not need them for correctness because cache_salt is part of the object key, but first-seen FDP placement assignments may change after adapter restart.

  • Slot affinity is process-local and is not stored in metadata checkpoints. After recovery, free slots have no recorded affinity until they are reused.

  • Store and retrieve requests must use the same cache_salt to address the same object. This is the normal LMCache key identity rule; FDP placement is a write directive and is not used to locate data on reads.

  • Metadata checkpoint writes use meta_checkpoint_placement_id when configured, otherwise they use default NVMe placement with no directive.

配置示例：

# Basic raw_block with posix I/O
--l2-adapter '{"type": "raw_block", "device_path": "/dev/nvme0n1", "slot_bytes": 1048576, "block_align": 4096, "header_bytes": 4096, "meta_total_bytes": 268435456, "use_odirect": true, "num_store_workers": 2, "num_lookup_workers": 1, "num_load_workers": 4}'

# With io_uring
--l2-adapter '{"type": "raw_block", "device_path": "/dev/nvme0n1", "slot_bytes": 1048576, "io_engine": "io_uring", "iouring_queue_depth": 256, "use_odirect": true}'

# With io_uring_cmd (NVMe passthrough)
--l2-adapter '{"type": "raw_block", "device_path": "/dev/ng0n1", "slot_bytes": 1048576, "io_engine": "io_uring", "use_uring_cmd": true, "iouring_queue_depth": 256, "max_data_transfer_size": 131072, "use_odirect": false}'

# With FDP discovery and cache_salt prefix placement enabled
--l2-adapter '{"type": "raw_block", "device_path": "/dev/ng0n1", "slot_bytes": 1048576, "io_engine": "io_uring", "use_uring_cmd": true, "fdp_enabled": true, "use_odirect": false}'

# With FDP discovery and cache_salt/rank placement enabled
--l2-adapter '{"type": "raw_block", "device_path": "/dev/ng0n1", "slot_bytes": 1048576, "io_engine": "io_uring", "use_uring_cmd": true, "fdp_enabled": true, "fdp_data_placement_policy": "cache_salt_rank", "use_odirect": false}'

# With FDP discovery only, keeping KV data writes on default NVMe placement
--l2-adapter '{"type": "raw_block", "device_path": "/dev/ng0n1", "slot_bytes": 1048576, "io_engine": "io_uring", "use_uring_cmd": true, "fdp_enabled": true, "fdp_data_placement_policy": "none", "use_odirect": false}'

# With eviction
--l2-adapter '{"type": "raw_block", "device_path": "/dev/nvme0n1", "slot_bytes": 1048576, "load_checkpoint_on_init": false, "eviction": {"eviction_policy": "LRU", "trigger_watermark": 0.9, "eviction_ratio": 0.1}}'

Hardware-gated FDP status validation:

FDP live-device validation is opt-in because it requires an FDP-capable NVMe namespace character device. The status probe opens the character device through the Rust raw-block binding and calls fetch_fdp_status() with a read-only file descriptor. It does not issue writes, initialize the MP adapter layout, write KV data, or verify the MP adapter's cache-salt placement policy.

LMCACHE_TEST_FDP_CHAR_DEVICE=/dev/ng0n1 \
  pytest -q tests/v1/storage_backend/test_raw_block_fdp_status_probe.py

When the variable is not set, the test skips. If the configured device, kernel, or controller does not support the FDP status query, the test skips with the underlying capability error. A passing status probe only confirms that the live device can answer the FDP status query; full adapter initialization and KV write placement on hardware are separate validations.