[Docs] CLI Docs updates (#2978)

Signed-off-by: wuhang <wuhang6@huawei.com>
This commit is contained in:
wuhang authored and GitHub committed 2026-04-24 11:25:22 +08:00
1 parent 19980bb37c
commit 2e067bbc24
8 files changed
+312 -49

No files matched your search

+16 -14
View File
@@ -19,7 +19,12 @@ Or use the convenience script:
```bash
cd /workspace/vllm-omni/examples/online_serving/bagel
# Initialize all stages within a single unified session (legacy operational sequence)
bash run_server.sh
# Initialize each stage in a discrete isolated process terminal
bash run_server_stage_cli.sh --stage 0
bash run_server_stage_cli.sh --stage 1
```
```bash
@@ -112,12 +117,13 @@ mooncake_master \
**2. Launch Stage 0 (Thinker / Orchestrator)** on the orchestrator node:
```bash
# API server port for client requests: 8000
vllm serve ByteDance-Seed/BAGEL-7B-MoT --omni \
--port 8000 \ # API server port for client requests
--port 8000 \
--stage-configs-path vllm_omni/model_executor/stage_configs/bagel_multiconnector.yaml \
--stage-id 0 \
-oma <ORCHESTRATOR_IP> \
-omp 8091
--omni-master-address <ORCHESTRATOR_IP> \
--omni-master-port 8091
```
**3. Launch Stage 1 (DiT)** on the remote node in headless mode:
@@ -127,8 +133,8 @@ vllm serve ByteDance-Seed/BAGEL-7B-MoT --omni \
--stage-configs-path vllm_omni/model_executor/stage_configs/bagel_multiconnector.yaml \
--stage-id 1 \
--headless \
-oma <ORCHESTRATOR_IP> \
-omp 8091
--omni-master-address <ORCHESTRATOR_IP> \
--omni-master-port 8091
```
**Mooncake Master arguments:**
@@ -145,14 +151,10 @@ vllm serve ByteDance-Seed/BAGEL-7B-MoT --omni \
| Argument | Description |
| :------- | :---------- |
| `--stage-id` | Which stage this process runs (0 = Thinker, 1 = DiT) |
| `--headless` | Run without the API server (worker-only mode) |
| `-oma` | Orchestrator master address |
| `-omp` | Orchestrator master port for Stage 1 to connect to Stage 0 for task coordination |
> [!IMPORTANT]
> **Startup Order**: Stage 0 (orchestrator) must be launched **before** Stage 1 (headless).
> Stage 0 will appear to hang on startup until Stage 1 (worker) connects — this is expected behavior.
| `--stage-id` | Designates the pipeline stage assigned to the process (e.g., 0 = Thinker, 1 = DiT) |
| `--headless` | Executes the worker stage autonomously without initializing an API server |
| `--omni-master-address` | Specifies the IP address binding the Orchestrator master node |
| `--omni-master-port` | Specifies the targeted port establishing task coordination between Stage 1 and Stage 0 |
**Network Requirements**
@@ -162,7 +164,7 @@ All nodes must have network connectivity to each other. Ensure the following por
| :--- | :------- | :------ | :-------- |
| 50051 | TCP | Mooncake Master RPC | Worker → Orchestrator |
| 8080 | TCP | Mooncake HTTP Metadata Server | Worker → Orchestrator |
| 8091 | TCP | Orchestrator Master (`-omp`) | Worker → Orchestrator |
| 8091 | TCP | Orchestrator Master (`--omni-master-port`) | Worker → Orchestrator |
| 8000 | TCP | API Server (`--port`) | Client → Orchestrator |
| 9003 | TCP | Metrics (optional) | Monitoring → Orchestrator |
@@ -116,8 +116,8 @@ run_stage_0() {
--port "$PORT" \
--stage-configs-path "$STAGE_CONFIGS_PATH" \
--stage-id 0 \
-oma "$MASTER_ADDRESS" \
-omp "$MASTER_PORT" \
--omni-master-address "$MASTER_ADDRESS" \
--omni-master-port "$MASTER_PORT" \
"${EXTRA_ARGS[@]}"
}
@@ -127,8 +127,8 @@ run_stage_1() {
--stage-configs-path "$STAGE_CONFIGS_PATH" \
--stage-id 1 \
--headless \
-oma "$MASTER_ADDRESS" \
-omp "$MASTER_PORT" \
--omni-master-address "$MASTER_ADDRESS" \
--omni-master-port "$MASTER_PORT" \
"${EXTRA_ARGS[@]}"
}
+70 -8
View File
@@ -12,21 +12,80 @@ Please refer to [README.md](../../../README.md)
vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091
```
The default deploy config at `vllm_omni/deploy/qwen3_omni_moe.yaml` is loaded
automatically by the model registry — no `--deploy-config` flag needed for the
common case. Async-chunk streaming is **enabled by default** in the bundled config.
NPU / ROCm / XPU per-platform deltas are merged in automatically from the
`platforms:` section of the same YAML.
The default deployment configuration, situated at `vllm_omni/deploy/qwen3_omni_moe.yaml`, is resolved and loaded
automatically via the model registry, obviating the `--deploy-config` flag in standard deployment topologies.
Asynchronous chunk streaming operates as **enabled by default** within this bundled configuration.
Additionally, NPU, ROCm, and XPU per-platform configuration deltas are deterministically merged from the
`platforms`: section of the corresponding YAML.
**Note:** The OpenAI-style **`/v1/realtime`** WebSocket (streaming PCM audio in, audio + transcription out) is **not supported** when `async_chunk` is enabled. Use the default omni layout or a stage config with `async_chunk: false` for realtime sessions.
If you have a custom deploy YAML, point at it explicitly:
**Note:** The OpenAI-style **`/v1/realtime`** WebSocket interface (facilitating streaming PCM audio input alongside audio and transcription output)
is currently **unsupported** while the `async_chunk` configuration attribute is enabled.
It is requisite to instantiate the default omni architecture or utilize a deployment configuration specifying `async_chunk: false` to facilitate real-time streaming sessions.
To explicitly utilize a custom deployment YAML, mandate the configuration path accordingly:
```bash
vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091 \
--deploy-config /path/to/your_deploy_config.yaml
```
### Launch individual stages (stage-based CLI)
Use the stage-based CLI when you want to run one stage per process.
**1. Stage 0 (Thinker + API server)**
```bash
vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \
--port 8091 \
--stage-id 0 \
--omni-master-address 127.0.0.1 \
--omni-master-port 26000
```
**2. Stage 1 (Talker)**
```bash
vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \
--stage-id 1 \
--headless \
--omni-master-address 127.0.0.1 \
--omni-master-port 26000
```
**3. Stage 2 (Code2Wav)**
```bash
vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \
--stage-id 2 \
--headless \
--omni-master-address 127.0.0.1 \
--omni-master-port 26000
```
Append `--deploy-config /path/to/your_deploy_config.yaml` to each node invocation if it is necessary
to explicitly override the bundled deployment YAML schema.
For standard **unified-process** launcher, stage-specific CLI configuration tuning is conventionally implemented
via the `--stage-overrides` directive, as demonstrated below:
```bash
vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091 \
--stage-overrides '{"1": {"gpu_memory_utilization": 0.5}}'
```
Conversely, within the stage-based CLI paradigm, `--stage-overrides` modifiers are typically **unnecessary**
for this category of optimization. Given that each instantiation strictly initiates a single functional stage,
parameter flags can be systematically assigned directly onto that specific stage's command sequence:
```bash
vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \
--stage-id 1 \
--headless \
--gpu-memory-utilization 0.5 \
--omni-master-address 127.0.0.1 \
--omni-master-port 26000
```
### Tuning deployment parameters
Most engine knobs (`max_num_batched_tokens`, `max_model_len`, `enforce_eager`,
@@ -93,6 +152,9 @@ vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091 \
Per-stage values are always treated as explicit and beat YAML defaults for
the named stage. Other stages keep their YAML values.
If you switch to the stage-based CLI, the same per-stage tuning can usually be
passed directly on that stage's command instead of using `--stage-overrides`.
#### 3. Custom deploy YAML
When per-stage overrides get long, write a small overlay YAML that inherits