[Docs] CLI Docs updates (#2978)
Signed-off-by: wuhang <wuhang6@huawei.com>
This commit is contained in:
1 parent
19980bb37c
commit
2e067bbc24
8 files changed
+312
-49
No files matched your search
@@ -19,7 +19,12 @@ Or use the convenience script:
|
||||
|
||||
```bash
|
||||
cd /workspace/vllm-omni/examples/online_serving/bagel
|
||||
# Initialize all stages within a single unified session (legacy operational sequence)
|
||||
bash run_server.sh
|
||||
|
||||
# Initialize each stage in a discrete isolated process terminal
|
||||
bash run_server_stage_cli.sh --stage 0
|
||||
bash run_server_stage_cli.sh --stage 1
|
||||
```
|
||||
|
||||
```bash
|
||||
@@ -112,12 +117,13 @@ mooncake_master \
|
||||
**2. Launch Stage 0 (Thinker / Orchestrator)** on the orchestrator node:
|
||||
|
||||
```bash
|
||||
# API server port for client requests: 8000
|
||||
vllm serve ByteDance-Seed/BAGEL-7B-MoT --omni \
|
||||
--port 8000 \ # API server port for client requests
|
||||
--port 8000 \
|
||||
--stage-configs-path vllm_omni/model_executor/stage_configs/bagel_multiconnector.yaml \
|
||||
--stage-id 0 \
|
||||
-oma <ORCHESTRATOR_IP> \
|
||||
-omp 8091
|
||||
--omni-master-address <ORCHESTRATOR_IP> \
|
||||
--omni-master-port 8091
|
||||
```
|
||||
|
||||
**3. Launch Stage 1 (DiT)** on the remote node in headless mode:
|
||||
@@ -127,8 +133,8 @@ vllm serve ByteDance-Seed/BAGEL-7B-MoT --omni \
|
||||
--stage-configs-path vllm_omni/model_executor/stage_configs/bagel_multiconnector.yaml \
|
||||
--stage-id 1 \
|
||||
--headless \
|
||||
-oma <ORCHESTRATOR_IP> \
|
||||
-omp 8091
|
||||
--omni-master-address <ORCHESTRATOR_IP> \
|
||||
--omni-master-port 8091
|
||||
```
|
||||
|
||||
**Mooncake Master arguments:**
|
||||
@@ -145,14 +151,10 @@ vllm serve ByteDance-Seed/BAGEL-7B-MoT --omni \
|
||||
|
||||
| Argument | Description |
|
||||
| :------- | :---------- |
|
||||
| `--stage-id` | Which stage this process runs (0 = Thinker, 1 = DiT) |
|
||||
| `--headless` | Run without the API server (worker-only mode) |
|
||||
| `-oma` | Orchestrator master address |
|
||||
| `-omp` | Orchestrator master port for Stage 1 to connect to Stage 0 for task coordination |
|
||||
|
||||
> [!IMPORTANT]
|
||||
> **Startup Order**: Stage 0 (orchestrator) must be launched **before** Stage 1 (headless).
|
||||
> Stage 0 will appear to hang on startup until Stage 1 (worker) connects — this is expected behavior.
|
||||
| `--stage-id` | Designates the pipeline stage assigned to the process (e.g., 0 = Thinker, 1 = DiT) |
|
||||
| `--headless` | Executes the worker stage autonomously without initializing an API server |
|
||||
| `--omni-master-address` | Specifies the IP address binding the Orchestrator master node |
|
||||
| `--omni-master-port` | Specifies the targeted port establishing task coordination between Stage 1 and Stage 0 |
|
||||
|
||||
**Network Requirements**
|
||||
|
||||
@@ -162,7 +164,7 @@ All nodes must have network connectivity to each other. Ensure the following por
|
||||
| :--- | :------- | :------ | :-------- |
|
||||
| 50051 | TCP | Mooncake Master RPC | Worker → Orchestrator |
|
||||
| 8080 | TCP | Mooncake HTTP Metadata Server | Worker → Orchestrator |
|
||||
| 8091 | TCP | Orchestrator Master (`-omp`) | Worker → Orchestrator |
|
||||
| 8091 | TCP | Orchestrator Master (`--omni-master-port`) | Worker → Orchestrator |
|
||||
| 8000 | TCP | API Server (`--port`) | Client → Orchestrator |
|
||||
| 9003 | TCP | Metrics (optional) | Monitoring → Orchestrator |
|
||||
|
||||
|
||||
@@ -116,8 +116,8 @@ run_stage_0() {
|
||||
--port "$PORT" \
|
||||
--stage-configs-path "$STAGE_CONFIGS_PATH" \
|
||||
--stage-id 0 \
|
||||
-oma "$MASTER_ADDRESS" \
|
||||
-omp "$MASTER_PORT" \
|
||||
--omni-master-address "$MASTER_ADDRESS" \
|
||||
--omni-master-port "$MASTER_PORT" \
|
||||
"${EXTRA_ARGS[@]}"
|
||||
}
|
||||
|
||||
@@ -127,8 +127,8 @@ run_stage_1() {
|
||||
--stage-configs-path "$STAGE_CONFIGS_PATH" \
|
||||
--stage-id 1 \
|
||||
--headless \
|
||||
-oma "$MASTER_ADDRESS" \
|
||||
-omp "$MASTER_PORT" \
|
||||
--omni-master-address "$MASTER_ADDRESS" \
|
||||
--omni-master-port "$MASTER_PORT" \
|
||||
"${EXTRA_ARGS[@]}"
|
||||
}
|
||||
|
||||
|
||||
@@ -12,21 +12,80 @@ Please refer to [README.md](../../../README.md)
|
||||
vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091
|
||||
```
|
||||
|
||||
The default deploy config at `vllm_omni/deploy/qwen3_omni_moe.yaml` is loaded
|
||||
automatically by the model registry — no `--deploy-config` flag needed for the
|
||||
common case. Async-chunk streaming is **enabled by default** in the bundled config.
|
||||
NPU / ROCm / XPU per-platform deltas are merged in automatically from the
|
||||
`platforms:` section of the same YAML.
|
||||
The default deployment configuration, situated at `vllm_omni/deploy/qwen3_omni_moe.yaml`, is resolved and loaded
|
||||
automatically via the model registry, obviating the `--deploy-config` flag in standard deployment topologies.
|
||||
Asynchronous chunk streaming operates as **enabled by default** within this bundled configuration.
|
||||
Additionally, NPU, ROCm, and XPU per-platform configuration deltas are deterministically merged from the
|
||||
`platforms`: section of the corresponding YAML.
|
||||
|
||||
**Note:** The OpenAI-style **`/v1/realtime`** WebSocket (streaming PCM audio in, audio + transcription out) is **not supported** when `async_chunk` is enabled. Use the default omni layout or a stage config with `async_chunk: false` for realtime sessions.
|
||||
|
||||
If you have a custom deploy YAML, point at it explicitly:
|
||||
**Note:** The OpenAI-style **`/v1/realtime`** WebSocket interface (facilitating streaming PCM audio input alongside audio and transcription output)
|
||||
is currently **unsupported** while the `async_chunk` configuration attribute is enabled.
|
||||
It is requisite to instantiate the default omni architecture or utilize a deployment configuration specifying `async_chunk: false` to facilitate real-time streaming sessions.
|
||||
|
||||
To explicitly utilize a custom deployment YAML, mandate the configuration path accordingly:
|
||||
```bash
|
||||
vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091 \
|
||||
--deploy-config /path/to/your_deploy_config.yaml
|
||||
```
|
||||
|
||||
### Launch individual stages (stage-based CLI)
|
||||
|
||||
Use the stage-based CLI when you want to run one stage per process.
|
||||
|
||||
**1. Stage 0 (Thinker + API server)**
|
||||
|
||||
```bash
|
||||
vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \
|
||||
--port 8091 \
|
||||
--stage-id 0 \
|
||||
--omni-master-address 127.0.0.1 \
|
||||
--omni-master-port 26000
|
||||
```
|
||||
|
||||
**2. Stage 1 (Talker)**
|
||||
|
||||
```bash
|
||||
vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \
|
||||
--stage-id 1 \
|
||||
--headless \
|
||||
--omni-master-address 127.0.0.1 \
|
||||
--omni-master-port 26000
|
||||
```
|
||||
|
||||
**3. Stage 2 (Code2Wav)**
|
||||
|
||||
```bash
|
||||
vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \
|
||||
--stage-id 2 \
|
||||
--headless \
|
||||
--omni-master-address 127.0.0.1 \
|
||||
--omni-master-port 26000
|
||||
```
|
||||
|
||||
Append `--deploy-config /path/to/your_deploy_config.yaml` to each node invocation if it is necessary
|
||||
to explicitly override the bundled deployment YAML schema.
|
||||
|
||||
For standard **unified-process** launcher, stage-specific CLI configuration tuning is conventionally implemented
|
||||
via the `--stage-overrides` directive, as demonstrated below:
|
||||
|
||||
```bash
|
||||
vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091 \
|
||||
--stage-overrides '{"1": {"gpu_memory_utilization": 0.5}}'
|
||||
```
|
||||
|
||||
Conversely, within the stage-based CLI paradigm, `--stage-overrides` modifiers are typically **unnecessary**
|
||||
for this category of optimization. Given that each instantiation strictly initiates a single functional stage,
|
||||
parameter flags can be systematically assigned directly onto that specific stage's command sequence:
|
||||
|
||||
```bash
|
||||
vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \
|
||||
--stage-id 1 \
|
||||
--headless \
|
||||
--gpu-memory-utilization 0.5 \
|
||||
--omni-master-address 127.0.0.1 \
|
||||
--omni-master-port 26000
|
||||
```
|
||||
|
||||
### Tuning deployment parameters
|
||||
|
||||
Most engine knobs (`max_num_batched_tokens`, `max_model_len`, `enforce_eager`,
|
||||
@@ -93,6 +152,9 @@ vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091 \
|
||||
Per-stage values are always treated as explicit and beat YAML defaults for
|
||||
the named stage. Other stages keep their YAML values.
|
||||
|
||||
If you switch to the stage-based CLI, the same per-stage tuning can usually be
|
||||
passed directly on that stage's command instead of using `--stage-overrides`.
|
||||
|
||||
#### 3. Custom deploy YAML
|
||||
|
||||
When per-stage overrides get long, write a small overlay YAML that inherits
|
||||
|
||||
Reference in new issue
Block a user