[Docs] CLI Docs updates (#2978)
Signed-off-by: wuhang <wuhang6@huawei.com>
This commit is contained in:
1 parent
19980bb37c
commit
2e067bbc24
8 files changed
+312
-49
No files matched your search
@@ -22,9 +22,16 @@ Or use the convenience script:
|
||||
|
||||
```bash
|
||||
cd /workspace/vllm-omni/examples/online_serving/bagel
|
||||
# Launch both stages in one session (legacy convenience flow)
|
||||
bash run_server.sh
|
||||
|
||||
# Launch a single stage per terminal
|
||||
bash run_server_stage_cli.sh --stage 0
|
||||
bash run_server_stage_cli.sh --stage 1
|
||||
```
|
||||
|
||||
If you have a custom stage configs file, launch the server with the command below:
|
||||
|
||||
```bash
|
||||
vllm serve ByteDance-Seed/BAGEL-7B-MoT --omni --port 8091 --stage-configs-path /path/to/stage_configs_file
|
||||
```
|
||||
@@ -115,12 +122,13 @@ mooncake_master \
|
||||
**2. Launch Stage 0 (Thinker / Orchestrator)** on the orchestrator node:
|
||||
|
||||
```bash
|
||||
# API server port for client requests: 8000
|
||||
vllm serve ByteDance-Seed/BAGEL-7B-MoT --omni \
|
||||
--port 8000 \ # API server port for client requests
|
||||
--port 8000 \
|
||||
--stage-configs-path vllm_omni/model_executor/stage_configs/bagel_multiconnector.yaml \
|
||||
--stage-id 0 \
|
||||
-oma <ORCHESTRATOR_IP> \
|
||||
-omp 8091
|
||||
--omni-master-address <ORCHESTRATOR_IP> \
|
||||
--omni-master-port 8091
|
||||
```
|
||||
|
||||
**3. Launch Stage 1 (DiT)** on the remote node in headless mode:
|
||||
@@ -130,8 +138,8 @@ vllm serve ByteDance-Seed/BAGEL-7B-MoT --omni \
|
||||
--stage-configs-path vllm_omni/model_executor/stage_configs/bagel_multiconnector.yaml \
|
||||
--stage-id 1 \
|
||||
--headless \
|
||||
-oma <ORCHESTRATOR_IP> \
|
||||
-omp 8091
|
||||
--omni-master-address <ORCHESTRATOR_IP> \
|
||||
--omni-master-port 8091
|
||||
```
|
||||
|
||||
**Mooncake Master arguments:**
|
||||
@@ -150,8 +158,8 @@ vllm serve ByteDance-Seed/BAGEL-7B-MoT --omni \
|
||||
| :------- | :---------- |
|
||||
| `--stage-id` | Which stage this process runs (0 = Thinker, 1 = DiT) |
|
||||
| `--headless` | Run without the API server (worker-only mode) |
|
||||
| `-oma` | Orchestrator master address |
|
||||
| `-omp` | Orchestrator master port for Stage 1 to connect to Stage 0 for task coordination |
|
||||
| `--omni-master-address` | Orchestrator master address |
|
||||
| `--omni-master-port` | Orchestrator master port for Stage 1 to connect to Stage 0 for task coordination |
|
||||
|
||||
> [!IMPORTANT]
|
||||
> **Startup Order**: Stage 0 (orchestrator) must be launched **before** Stage 1 (headless).
|
||||
@@ -165,7 +173,7 @@ All nodes must have network connectivity to each other. Ensure the following por
|
||||
| :--- | :------- | :------ | :-------- |
|
||||
| 50051 | TCP | Mooncake Master RPC | Worker → Orchestrator |
|
||||
| 8080 | TCP | Mooncake HTTP Metadata Server | Worker → Orchestrator |
|
||||
| 8091 | TCP | Orchestrator Master (`-omp`) | Worker → Orchestrator |
|
||||
| 8091 | TCP | Orchestrator Master (`--omni-master-port`) | Worker → Orchestrator |
|
||||
| 8000 | TCP | API Server (`--port`) | Client → Orchestrator |
|
||||
| 9003 | TCP | Metrics (optional) | Monitoring → Orchestrator |
|
||||
|
||||
|
||||
@@ -15,15 +15,72 @@ Please refer to [README.md](https://github.com/vllm-project/vllm-omni/tree/main/
|
||||
vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091
|
||||
```
|
||||
|
||||
If you want to open async chunking for qwen3-omni, launch the server with command below
|
||||
The default deployment configuration situated at `vllm_omni/deploy/qwen3_omni_moe.yaml` is resolved and loaded
|
||||
automatically via the model registry, obviating the necessity for the `--deploy-config` flag in standard deployment topologies.
|
||||
Asynchronous chunk streaming is **enabled by default** within the bundled configuration.
|
||||
|
||||
To explicitly utilize a custom deployment YAML, specify the configuration path:
|
||||
```bash
|
||||
vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091 --deploy-config /vllm_omni/deploy/qwen3_omni_moe.yaml
|
||||
vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091 \
|
||||
--deploy-config /path/to/deploy_config_file
|
||||
```
|
||||
|
||||
If you have custom stage configs file, launch the server with command below
|
||||
### Launch individual stages (stage-based CLI)
|
||||
|
||||
Adopt the stage-based CLI architecture to independently instantiate execution processes per functional stage.
|
||||
|
||||
**1. Stage 0 (Thinker + API server)**
|
||||
|
||||
```bash
|
||||
vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091 --deploy-config /path/to/deploy_config_file
|
||||
vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \
|
||||
--port 8091 \
|
||||
--stage-id 0 \
|
||||
--omni-master-address 127.0.0.1 \
|
||||
--omni-master-port 26000
|
||||
```
|
||||
|
||||
**2. Stage 1 (Talker)**
|
||||
|
||||
```bash
|
||||
vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \
|
||||
--stage-id 1 \
|
||||
--headless \
|
||||
--omni-master-address 127.0.0.1 \
|
||||
--omni-master-port 26000
|
||||
```
|
||||
|
||||
**3. Stage 2 (Code2Wav)**
|
||||
|
||||
```bash
|
||||
vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \
|
||||
--stage-id 2 \
|
||||
--headless \
|
||||
--omni-master-address 127.0.0.1 \
|
||||
--omni-master-port 26000
|
||||
```
|
||||
|
||||
Add `--deploy-config /path/to/deploy_config_file` to every command if you want
|
||||
to override the bundled deploy YAML.
|
||||
|
||||
For the regular one-process launch, stage-specific CLI tuning is usually done
|
||||
with `--stage-overrides`, for example:
|
||||
|
||||
```bash
|
||||
vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091 \
|
||||
--stage-overrides '{"1": {"gpu_memory_utilization": 0.5}}'
|
||||
```
|
||||
|
||||
For the stage-based CLI, you usually do **not** need `--stage-overrides` for
|
||||
that kind of change. Since each command launches one stage, just pass the knob
|
||||
directly on that stage command:
|
||||
|
||||
```bash
|
||||
vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \
|
||||
--stage-id 1 \
|
||||
--headless \
|
||||
--gpu-memory-utilization 0.5 \
|
||||
--omni-master-address 127.0.0.1 \
|
||||
--omni-master-port 26000
|
||||
```
|
||||
|
||||
### Send Multi-modal Request
|
||||
|
||||
Reference in new issue
Block a user