bdfc77147d
Signed-off-by: Meng <meng_chen99@163.com> Signed-off-by: Yangshen Deng <yangshen.d@outlook.com> Signed-off-by: Suli Wang <50828270+wsl2000@users.noreply.github.com> Co-authored-by: Meng <meng_chen99@163.com> Co-authored-by: Suli Wang <50828270+wsl2000@users.noreply.github.com> Co-authored-by: amy-why-3459 <wuhaiyan17@huawei.com>
133 lines
4.0 KiB
Markdown
133 lines
4.0 KiB
Markdown
# DreamZero
|
|
|
|
DreamZero is the robot-policy serving example for the OpenPI-compatible `/v1/realtime/robot/openpi` endpoint.
|
|
|
|
## Supported checkpoints
|
|
|
|
| Model | HuggingFace repo |
|
|
|---|---|
|
|
| DreamZero-DROID | `GEAR-Dreams/DreamZero-DROID` |
|
|
|
|
## Quick start
|
|
|
|
### Start the server
|
|
|
|
```bash
|
|
bash examples/online_serving/dreamzero/run_server.sh
|
|
```
|
|
|
|
By default this launches:
|
|
|
|
```bash
|
|
vllm serve GEAR-Dreams/DreamZero-DROID --omni --port 8000 \
|
|
--served-model-name dreamzero-droid \
|
|
--deploy-config vllm_omni/deploy/dreamzero_tp1_cfg2.yaml \
|
|
--enforce-eager --disable-log-stats
|
|
```
|
|
|
|
Override `MODEL`, `PORT`, `HOST`, `DEPLOY_CONFIG`, or `SERVED_MODEL_NAME` through the script environment if needed.
|
|
|
|
### Install example dependencies
|
|
|
|
The core `pip install -e .` setup does not include the extra packages used by the DreamZero example scripts.
|
|
|
|
- `openpi_client.py`:
|
|
`openpi-client`, `websockets`, `opencv-python`
|
|
- `droid_sim_eval_client.py`:
|
|
`mediapy`, `websockets`, `openpi-client`
|
|
|
|
The DROID sim-eval script also needs an Isaac Lab environment that provides `isaaclab`, `isaaclab_tasks`, `sim_evals`, and `gymnasium`.
|
|
|
|
If you run the DROID client on Python < 3.12, also install `typing-extensions`.
|
|
|
|
### Configure TP and CFG parallelism
|
|
|
|
The bundled DreamZero configs intentionally keep only:
|
|
|
|
| Config | Purpose |
|
|
|---|---|
|
|
| `vllm_omni/deploy/dreamzero.yaml` | Default TP=1, CFG parallel disabled |
|
|
| `vllm_omni/deploy/dreamzero_tp1_cfg2.yaml` | TP=1, CFG parallel size=2 |
|
|
|
|
For other topologies, use CLI parallelism flags and update stage 0 `devices` with `--stage-overrides`. The number of listed devices must match `tensor_parallel_size * cfg_parallel_size`.
|
|
|
|
TP=2 with CFG parallel disabled:
|
|
|
|
```bash
|
|
vllm serve GEAR-Dreams/DreamZero-DROID --omni --port 8000 \
|
|
--served-model-name dreamzero-droid \
|
|
--deploy-config vllm_omni/deploy/dreamzero.yaml \
|
|
--tensor-parallel-size 2 \
|
|
--stage-overrides '{"0": {"devices": "0,1"}}' \
|
|
--enforce-eager --disable-log-stats
|
|
```
|
|
|
|
TP=2 with CFG parallel size=2:
|
|
|
|
```bash
|
|
vllm serve GEAR-Dreams/DreamZero-DROID --omni --port 8000 \
|
|
--served-model-name dreamzero-droid \
|
|
--deploy-config vllm_omni/deploy/dreamzero.yaml \
|
|
--tensor-parallel-size 2 \
|
|
--cfg-parallel-size 2 \
|
|
--stage-overrides '{"0": {"devices": "0,1,2,3"}}' \
|
|
--enforce-eager --disable-log-stats
|
|
```
|
|
|
|
### Download example assets
|
|
|
|
The OpenPI client and DROID sim-eval example expect the three camera MP4 files in `outputs/dreamzero/assets`.
|
|
|
|
```bash
|
|
hf download YangshenDeng/vllm-omni-dreamzero-assets --repo-type dataset --local-dir outputs/dreamzero/assets
|
|
```
|
|
|
|
### Run the OpenPI client
|
|
|
|
```bash
|
|
python examples/online_serving/dreamzero/openpi_client.py \
|
|
--host 127.0.0.1 \
|
|
--port 8000 \
|
|
--video-dir outputs/dreamzero/assets
|
|
```
|
|
|
|
This client uses downloaded example videos and talks to the OpenPI websocket server.
|
|
|
|
### Run DROID sim eval
|
|
|
|
```bash
|
|
${ISAACLAB_LAUNCHER} -p examples/online_serving/dreamzero/droid_sim_eval_client.py \
|
|
--host 127.0.0.1 \
|
|
--port 8000 \
|
|
--scene 1 \
|
|
--episodes 1 \
|
|
--headless \
|
|
--device cuda:0
|
|
```
|
|
|
|
Set `ISAACLAB_LAUNCHER=path/to/isaaclab.sh` from the vLLM-Omni repository root before running the command.
|
|
This launches Isaac Lab / sim-evals and runs the DROID benchmark loop against the same websocket endpoint.
|
|
|
|
### Export comparison videos offline
|
|
|
|
```bash
|
|
python examples/offline_inference/dreamzero/export_prediction_video.py \
|
|
--deploy-config vllm_omni/deploy/dreamzero_tp1_cfg2.yaml \
|
|
--save-input-video \
|
|
--save-gif
|
|
```
|
|
|
|
The export script uses local `Omni` inference, not the websocket server. It writes the input rollout video and the predicted output video artifacts for side-by-side inspection.
|
|
|
|
### MolmoSpace demo
|
|
|
|
```bash
|
|
python examples/online_serving/dreamzero/molmospace_dreamzero_eval_demo.py \
|
|
--host 127.0.0.1 \
|
|
--port 8000 \
|
|
--benchmark_dir /path/to/benchmark \
|
|
--output_dir /path/to/output
|
|
```
|
|
|
|
This demo adapts DreamZero to the MolmoSpace-style remote policy eval loop.
|