SageMaker HyperPod#

一个由 Amazon SageMaker HyperPod 的 ai-toolkit 守护进程支持的 L2 适配器，提供 LMCache 的共享内存访问。

必填字段：

  • url: ai-toolkit 守护进程 URL (sagemaker-hyperpod://host:port)。

可选字段：

  • bucket (字符串，默认 "lmcache")：ai-toolkit 缓存命名空间。

  • shared_memory_name (string, default \"shared_memory\"): 共享内存段的名称。

  • max_concurrent_requests (int, default 100): 最大并发 HTTP 请求数。

  • max_connections (int, default 256): 总的 HTTP 连接池限制。

  • max_connections_per_host (int, default 128): 每个主机的连接限制。

  • timeout_ms (int, default 5000): HTTP 传输超时。

  • lease_wait_timeout_ms (int, default 1000): 守护进程在回答之前可以持有租约请求的时间。

  • lease_ttl_ms (int, default 30000): 服务器端租约生命周期。

  • put_stream_chunk_bytes (int, default 65536): HTTP PUT 块大小。

  • max_lease_size_mb (float, optional): 接受的租约有效负载的上限。

  • use_https (bool, default false): 启用 HTTPS 而不是 HTTP。

配置示例：

# Basic (node IP via the Kubernetes downward API)
--l2-adapter '{"type": "sagemaker-hyperpod", "url": "sagemaker-hyperpod://'"${NODE_IP}"':9200"}'

# Custom namespace and faster miss detection
--l2-adapter '{"type": "sagemaker-hyperpod", "url": "sagemaker-hyperpod://'"${NODE_IP}"':9200", "bucket": "team-a", "lease_wait_timeout_ms": 250}'

Kubernetes 部署要求#

ai-toolkit 守护进程是节点本地的（hostNetwork，端口 9200），因此 LMCache 服务器必须在与守护进程相同的节点上运行。

节点 IP 的环境变量#

添加 NODE_IP 环境变量以解析本地节点的 IP 地址：

env:
  - name: NODE_IP
    valueFrom:
      fieldRef:
        fieldPath: status.hostIP

/dev/shm 卷配置#

SageMaker Hyperpod requires /dev/shm for high-performance shared memory operations. The right configuration depends on whether the LMCache server runs in the same pod as the inference engine or in a separate pod.

Separate pods (Recommended) — LMCache server in its own pod

GPU IPC between the inference engine and the LMCache server requires a shared IPC namespace; across pods this is only possible through the host. Set hostIPC: true on both pods:

spec:
  hostIPC: true

With hostIPC: true the pod's /dev/shm is the host's, so the daemon's segment is already visible — do not add the subPath mount in this case (the host IPC namespace shadows it).

Same pod (LMCache server and inference engine as containers in one pod)

The pod-shared IPC namespace already covers the GPU IPC between the engine and the LMCache server, so only the daemon's shared-memory segment needs to be exposed. Mount it from the host with subPath:

volumeMounts:
  - name: dshm
    mountPath: /dev/shm/shared_memory
    subPath: shared_memory

volumes:
  - name: dshm
    hostPath:
      path: /dev/shm