liangmenghuangandClaude Fable 5.1
7a80b79698
Signed-off-by: liangmengh <liangmengh@nvidia.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
116 lines
4.1 KiB
Markdown
116 lines
4.1 KiB
Markdown
# OpenPI Robot Policy WebSocket API
|
|
|
|
Use `WS /v1/realtime/robot/openpi` for low-latency robot-policy inference with
|
|
OpenPI-compatible clients. A persistent WebSocket carries observations to the
|
|
loaded policy model and returns action arrays.
|
|
|
|
Despite the `/realtime` prefix, this endpoint does not use the OpenAI Realtime
|
|
event schema. It uses binary MessagePack frames with NumPy extensions.
|
|
|
|
## Availability
|
|
|
|
The endpoint is available only when the loaded diffusion policy configuration
|
|
contains `policy_server_config`. The server sends that model-specific
|
|
configuration to the client as the first binary frame.
|
|
|
|
For example, DreamZero declares:
|
|
|
|
```yaml
|
|
policy_server_config:
|
|
image_resolution: [180, 320]
|
|
n_external_cameras: 2
|
|
needs_wrist_camera: true
|
|
needs_stereo_camera: false
|
|
needs_session_id: true
|
|
action_space: joint_position
|
|
```
|
|
|
|
Without this configuration, the WebSocket returns `Robot policy not available`
|
|
and closes.
|
|
|
|
## DreamZero Quick Start
|
|
|
|
Start the bundled DreamZero deployment:
|
|
|
|
```bash
|
|
vllm serve GEAR-Dreams/DreamZero-DROID --omni --port 8091 \
|
|
--served-model-name dreamzero-droid \
|
|
--deploy-config vllm_omni/deploy/dreamzero_tp1_cfg2.yaml \
|
|
--enforce-eager --disable-log-stats
|
|
```
|
|
|
|
Install the optional client dependencies, download the sample camera inputs,
|
|
and run the client:
|
|
|
|
```bash
|
|
pip install openpi-client websockets opencv-python
|
|
|
|
hf download YangshenDeng/vllm-omni-dreamzero-assets \
|
|
--repo-type dataset \
|
|
--local-dir outputs/dreamzero/assets
|
|
|
|
python examples/online_serving/dreamzero/openpi_client.py \
|
|
--host 127.0.0.1 \
|
|
--port 8091 \
|
|
--video-dir outputs/dreamzero/assets
|
|
```
|
|
|
|
## Protocol
|
|
|
|
The connection is request-response after an initial server handshake:
|
|
|
|
```text
|
|
connect
|
|
<- msgpack(policy_server_config)
|
|
|
|
infer
|
|
-> msgpack({"endpoint": "infer", "session_id": "...", "seed": <optional int>, ...observation})
|
|
<- msgpack(ndarray | dict[str, ndarray])
|
|
|
|
reset
|
|
-> msgpack({"endpoint": "reset"})
|
|
<- msgpack({"status": "reset successful"})
|
|
```
|
|
|
|
If `endpoint` is omitted, the server treats the message as `infer`. Each
|
|
inference message produces one action response after the engine request
|
|
completes; action tokens or intermediate tensors are not streamed.
|
|
|
|
NumPy arrays use the marker format implemented by `openpi-client`. The server
|
|
also accepts the legacy vLLM NumPy marker representation on input. JSON text
|
|
frames are not observation messages.
|
|
|
|
## Sessions and Reset
|
|
|
|
- The API layer's current-session and first-call counters are scoped to each
|
|
WebSocket connection.
|
|
- `session_id` identifies model-side state across observations on that
|
|
connection. If omitted, it defaults to `default`.
|
|
- The first inference for a session is sent to the model with `reset=true`.
|
|
- Changing `session_id`, or sending the `reset` command, causes the next
|
|
inference to start with `reset=true`.
|
|
- The policy pipeline owns observation transforms and persistent model state,
|
|
normally keyed by `session_id`; the API layer forwards the raw observation
|
|
dictionary.
|
|
- An optional integer `seed` in an inference message becomes the engine
|
|
request's `sampling_params.seed`. A policy whose sampling consumes the
|
|
request generator (GR00T-N1.7) returns the same action chunk for the same
|
|
observation and seed. If omitted, the engine assigns a random per-request
|
|
seed, so unseeded inferences are independent of each other. The API layer
|
|
consumes `seed`; it is not forwarded inside the observation.
|
|
|
|
## Limits and Errors
|
|
|
|
- Maximum inbound payload size is 64 MiB.
|
|
- The server closes an idle connection after 30 seconds.
|
|
- Invalid binary input returns a MessagePack
|
|
`{"type":"error","message":"Invalid request payload"}` response.
|
|
- Inference failures return a generic `Internal inference error` without
|
|
exposing an internal traceback.
|
|
- Unsupported NumPy object, structured, and complex dtypes are rejected.
|
|
|
|
Observation keys, camera layout, state tensors, and action shapes are defined
|
|
by the loaded policy rather than by this transport. Read the handshake before
|
|
constructing observations. See the [DreamZero example](https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/dreamzero)
|
|
for a complete OpenPI client and DROID simulation loop.
|