From 0470e967716059685df48ff6f78d37457100cd33 Mon Sep 17 00:00:00 2001 From: amy-why-3459 Date: Mon, 27 Apr 2026 21:41:13 +0800 Subject: [PATCH] [Docs] Modify Qwen3-Omni's recipe (#3109) Signed-off-by: amy-why-3459 --- docs/cli/serve.md | 8 +- docs/configuration/stage_configs.md | 10 +- .../examples/online_serving/qwen3_omni.md | 10 +- examples/online_serving/qwen3_omni/README.md | 10 +- recipes/Qwen/Qwen3-Omni.md | 228 ++++++++++++++---- 5 files changed, 210 insertions(+), 56 deletions(-) diff --git a/docs/cli/serve.md b/docs/cli/serve.md index 035fa0567..9d1747e0b 100644 --- a/docs/cli/serve.md +++ b/docs/cli/serve.md @@ -12,9 +12,11 @@ The stage-based CLI is designed for deployments that require launching each pipe `vllm_omni/model_executor/stage_configs/`, the `--stage-configs-path` parameter remains mandatory. Example: Initializing Stage 0 (Orchestrator and API Server): +The commands below show a common device mapping where Stage 0 uses GPU 0 and +worker stages use GPU 1 via `CUDA_VISIBLE_DEVICES`. ```bash -vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \ +CUDA_VISIBLE_DEVICES=0 vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \ --port 8091 \ --stage-id 0 \ --omni-master-address 127.0.0.1 \ @@ -24,7 +26,7 @@ vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \ Example: Initializing a Headless Worker Stage (Stage 1): ```bash -vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \ +CUDA_VISIBLE_DEVICES=1 vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \ --stage-id 1 \ --headless \ --omni-master-address 127.0.0.1 \ @@ -46,7 +48,7 @@ vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091 \ the stage-based CLI permits the direct initialization of Stage 1 with explicit parameters: ```bash -vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \ +CUDA_VISIBLE_DEVICES=1 vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \ --stage-id 1 \ --headless \ --gpu-memory-utilization 0.5 \ diff --git a/docs/configuration/stage_configs.md b/docs/configuration/stage_configs.md index 45bacfb78..3a17f1c40 100644 --- a/docs/configuration/stage_configs.md +++ b/docs/configuration/stage_configs.md @@ -102,15 +102,17 @@ The stage-based CLI paradigm facilitates the execution of discrete pipeline stag For migrated architectures, the system automatically resolves and loads the bundled deployment YAML. Consequently, the primary execution path does **not** necessitate the explicit definition of `--deploy-config`: +the example below uses `CUDA_VISIBLE_DEVICES=0` for Stage 0 and +`CUDA_VISIBLE_DEVICES=1` for Stage 1. ```bash -vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \ +CUDA_VISIBLE_DEVICES=0 vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \ --port 8091 \ --stage-id 0 \ --omni-master-address 127.0.0.1 \ --omni-master-port 26000 -vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \ +CUDA_VISIBLE_DEVICES=1 vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \ --stage-id 1 \ --headless \ --omni-master-address 127.0.0.1 \ @@ -132,7 +134,7 @@ Conversely, in the context of the **stage-based CLI** paradigm, given that each can be defined uniformly via explicit CLI flags on the corresponding instantiation command, rendering composite `--stage-overrides` JSON strings unnecessary: ```bash -vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \ +CUDA_VISIBLE_DEVICES=1 vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \ --stage-id 1 \ --headless \ --gpu-memory-utilization 0.5 \ @@ -189,7 +191,7 @@ Within the stage-based CLI paradigm, equivalent configuration parameters can inh as command-line arguments to the designated single-stage process instantiation: ```bash -vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \ +CUDA_VISIBLE_DEVICES=0 vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \ --stage-id 0 \ --max-num-seqs 8 \ --omni-master-address 127.0.0.1 \ diff --git a/docs/user_guide/examples/online_serving/qwen3_omni.md b/docs/user_guide/examples/online_serving/qwen3_omni.md index 22d89ee80..413c80b7e 100644 --- a/docs/user_guide/examples/online_serving/qwen3_omni.md +++ b/docs/user_guide/examples/online_serving/qwen3_omni.md @@ -28,11 +28,13 @@ vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091 \ ### Launch individual stages (stage-based CLI) Adopt the stage-based CLI architecture to independently instantiate execution processes per functional stage. +The example below pins Stage 0 to GPU 0 and Stage 1/2 to GPU 1 via +`CUDA_VISIBLE_DEVICES`. **1. Stage 0 (Thinker + API server)** ```bash -vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \ +CUDA_VISIBLE_DEVICES=0 vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \ --port 8091 \ --stage-id 0 \ --omni-master-address 127.0.0.1 \ @@ -42,7 +44,7 @@ vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \ **2. Stage 1 (Talker)** ```bash -vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \ +CUDA_VISIBLE_DEVICES=1 vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \ --stage-id 1 \ --headless \ --omni-master-address 127.0.0.1 \ @@ -52,7 +54,7 @@ vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \ **3. Stage 2 (Code2Wav)** ```bash -vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \ +CUDA_VISIBLE_DEVICES=1 vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \ --stage-id 2 \ --headless \ --omni-master-address 127.0.0.1 \ @@ -75,7 +77,7 @@ that kind of change. Since each command launches one stage, just pass the knob directly on that stage command: ```bash -vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \ +CUDA_VISIBLE_DEVICES=1 vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \ --stage-id 1 \ --headless \ --gpu-memory-utilization 0.5 \ diff --git a/examples/online_serving/qwen3_omni/README.md b/examples/online_serving/qwen3_omni/README.md index c85970555..2f40b8347 100644 --- a/examples/online_serving/qwen3_omni/README.md +++ b/examples/online_serving/qwen3_omni/README.md @@ -31,11 +31,13 @@ vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091 \ ### Launch individual stages (stage-based CLI) Use the stage-based CLI when you want to run one stage per process. +The example below pins Stage 0 to GPU 0 and Stage 1/2 to GPU 1 via +`CUDA_VISIBLE_DEVICES`. **1. Stage 0 (Thinker + API server)** ```bash -vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \ +CUDA_VISIBLE_DEVICES=0 vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \ --port 8091 \ --stage-id 0 \ --omni-master-address 127.0.0.1 \ @@ -45,7 +47,7 @@ vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \ **2. Stage 1 (Talker)** ```bash -vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \ +CUDA_VISIBLE_DEVICES=1 vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \ --stage-id 1 \ --headless \ --omni-master-address 127.0.0.1 \ @@ -55,7 +57,7 @@ vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \ **3. Stage 2 (Code2Wav)** ```bash -vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \ +CUDA_VISIBLE_DEVICES=1 vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \ --stage-id 2 \ --headless \ --omni-master-address 127.0.0.1 \ @@ -78,7 +80,7 @@ for this category of optimization. Given that each instantiation strictly initia parameter flags can be systematically assigned directly onto that specific stage's command sequence: ```bash -vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \ +CUDA_VISIBLE_DEVICES=1 vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \ --stage-id 1 \ --headless \ --gpu-memory-utilization 0.5 \ diff --git a/recipes/Qwen/Qwen3-Omni.md b/recipes/Qwen/Qwen3-Omni.md index f78e4dda2..49552d871 100644 --- a/recipes/Qwen/Qwen3-Omni.md +++ b/recipes/Qwen/Qwen3-Omni.md @@ -1,76 +1,141 @@ -# Qwen3-Omni for multimodal chat on 1x A100 80GB +# Qwen3-Omni ## Summary - Vendor: Qwen - Model: `Qwen/Qwen3-Omni-30B-A3B-Instruct` -- Task: Multimodal chat with text, image, audio, or video input +- Task: Multimodal chat with text/image/audio/video input - Mode: Online serving with the OpenAI-compatible API - Maintainer: Community ## When to use this recipe -Use this recipe when you want a known-good starting point for serving -`Qwen/Qwen3-Omni-30B-A3B-Instruct` with vLLM-Omni on a single 80 GB A100 and -validate the deployment with the existing multimodal client examples in this -repository. +Use this recipe as a practical baseline for running +`Qwen/Qwen3-Omni-30B-A3B-Instruct` with the same serving paths already +exercised by repository examples and tests. ## References -- Upstream or canonical docs: +- User guide: [`docs/user_guide/examples/online_serving/qwen3_omni.md`](../../docs/user_guide/examples/online_serving/qwen3_omni.md) -- Related example under `examples/`: +- Example guide: [`examples/online_serving/qwen3_omni/README.md`](../../examples/online_serving/qwen3_omni/README.md) -- Related issue or discussion: - [RFC: add recipes folder](https://github.com/vllm-project/vllm-omni/issues/2645) -## Hardware Support - -This recipe currently documents one tested-style reference configuration for -CUDA GPU serving. Add more sections for other hardware as community validation -lands. - -## GPU - -### 1x A100 80GB - -#### Environment +## Environment - OS: Linux - Python: 3.10+ -- Driver / runtime: NVIDIA CUDA environment with an A100 80 GB GPU -- vLLM version: Match the repository requirements for your checkout -- vLLM-Omni version or commit: Use the commit you are deploying from +- vLLM / vLLM-Omni: use versions from your current checkout, >=0.18.0 -#### Command +## Start server (single command) -Start the server from the repository root: +From repository root: ```bash vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091 ``` -Async chunking is enabled by default in the bundled deployment config. For -common runtime tuning, prefer CLI overrides instead of editing or passing a -custom YAML file: +Notes: -```bash -# Disable async chunking for /v1/realtime sessions -vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091 \ - --no-async-chunk -``` +- `--omni` is required. +- The default deploy config `vllm_omni/deploy/qwen3_omni_moe.yaml` is loaded + automatically by model registry. +- `async_chunk` is enabled by default in this deploy config. +- Platform deltas under `platforms:` (NPU/ROCm/XPU) are merged automatically on + matching runtimes. -Use a custom deploy config only for advanced cases such as custom topology, -connector wiring, or a larger overlay of stage defaults: +For advanced customization, pass an overlay YAML: ```bash vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091 \ --deploy-config /path/to/your_qwen3_omni_overrides.yaml ``` +#### Runtime tuning + +Prefer CLI overrides for day-to-day tuning: + +```bash +# Disable async chunking when using /v1/realtime +vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091 \ + --no-async-chunk + +# Example per-stage tuning in unified launch +vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091 \ + --stage-overrides '{"1": {"gpu_memory_utilization": 0.5}}' + +# Tune max_num_seqs per stage (single process launch) +vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091 \ + --stage-overrides '{ + "0": {"max_num_seqs": 8}, + "1": {"max_num_seqs": 4}, + "2": {"max_num_seqs": 4} + }' +``` + +#### Stage-based launch (one stage per process) + +Use three terminals (one per stage). Start with the default commands below, then +add `--max-num-seqs` only if you need explicit per-stage concurrency control. + +Default stage-based commands: + +```bash +# Stage 0: Thinker + API server +CUDA_VISIBLE_DEVICES=0 vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \ + --port 8091 \ + --stage-id 0 \ + --omni-master-address 127.0.0.1 \ + --omni-master-port 26000 & + +# Stage 1: Talker +CUDA_VISIBLE_DEVICES=1 vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \ + --stage-id 1 \ + --headless \ + --omni-master-address 127.0.0.1 \ + --omni-master-port 26000 & + +# Stage 2: Code2Wav +CUDA_VISIBLE_DEVICES=1 vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \ + --stage-id 2 \ + --headless \ + --omni-master-address 127.0.0.1 \ + --omni-master-port 26000 & +``` + +Optional: explicit per-stage `max_num_seqs` tuning: + +```bash +# Stage 0 +CUDA_VISIBLE_DEVICES=0 vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \ + --port 8091 \ + --stage-id 0 \ + --max-num-seqs 8 \ + --omni-master-address 127.0.0.1 \ + --omni-master-port 26000 & + +# Stage 1 +CUDA_VISIBLE_DEVICES=1 vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \ + --stage-id 1 \ + --headless \ + --max-num-seqs 4 \ + --omni-master-address 127.0.0.1 \ + --omni-master-port 26000 & + +# Stage 2 +CUDA_VISIBLE_DEVICES=1 vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \ + --stage-id 2 \ + --headless \ + --max-num-seqs 4 \ + --omni-master-address 127.0.0.1 \ + --omni-master-port 26000 & +``` + +If you use custom deploy YAML, add `--deploy-config` to each stage command. + #### Verification -Run one of the existing example clients after the server is ready: +After server startup, run a multimodal example client: ```bash python examples/online_serving/openai_chat_completion_client_for_multimodal_generation.py \ @@ -80,7 +145,7 @@ python examples/online_serving/openai_chat_completion_client_for_multimodal_gene --host localhost ``` -For a quick API smoke test, request text-only output: +Quick API smoke test (text-only output): ```bash curl http://localhost:8091/v1/chat/completions \ @@ -92,8 +157,89 @@ curl http://localhost:8091/v1/chat/completions \ }' ``` +`{"modalities":["text","audio"]}` means the model returns both text and audio in +the same response. Use it when you want transcription/content text and TTS audio +together. + +Quick API smoke test (text + audio output): + +```bash +curl http://localhost:8091/v1/chat/completions \ + -H "Content-Type: application/json" \ + -d '{ + "model": "Qwen/Qwen3-Omni-30B-A3B-Instruct", + "messages": [{"role": "user", "content": "Describe vLLM in brief."}], + "modalities": ["text", "audio"] + }' +``` + +Realtime WebSocket check (`/v1/realtime`) requires async chunk disabled: + +```bash +python examples/online_serving/qwen3_omni/openai_realtime_client.py \ + --url ws://localhost:8091/v1/realtime \ + --model Qwen/Qwen3-Omni-30B-A3B-Instruct \ + --input-wav /path/to/input_16k_mono.wav \ + --output-wav realtime_output.wav +``` + +#### Benchmark with `vllm bench` + +After the server is up, you can run online serving benchmarks with +`vllm bench serve --omni`. + +Text-focused random workload: + +```bash +vllm bench serve \ + --omni \ + --host localhost \ + --port 8091 \ + --model Qwen/Qwen3-Omni-30B-A3B-Instruct \ + --endpoint /v1/chat/completions \ + --backend openai-chat-omni \ + --dataset-name random \ + --num-prompts 40 \ + --max-concurrency 4 \ + --random-input-len 2500 \ + --random-output-len 900 \ + --ignore-eos \ + --extra-body '{"modalities":["text"]}' \ + --percentile-metrics ttft,tpot,itl,e2el +``` + +If you want benchmark requests to return both text and audio, switch +`--extra-body` to: + +```bash +--extra-body '{"modalities":["text","audio"]}' +``` + +Synthetic multimodal workload (`random-mm`): + +```bash +vllm bench serve \ + --omni \ + --host localhost \ + --port 8091 \ + --model Qwen/Qwen3-Omni-30B-A3B-Instruct \ + --endpoint /v1/chat/completions \ + --backend openai-chat-omni \ + --dataset-name random-mm \ + --num-prompts 40 \ + --max-concurrency 4 \ + --random-input-len 2500 \ + --random-output-len 900 \ + --random-mm-base-items-per-request 2 \ + --random-mm-limit-mm-per-prompt '{"image":1,"video":1,"audio":1}' \ + --random-mm-bucket-config '{"(32, 32, 1)": 0.5, "(0, 1, 1)": 0.5}' \ + --ignore-eos \ + --extra-body '{"modalities":["text"]}' \ + --percentile-metrics ttft,tpot,itl,e2el +``` + #### Notes -- Memory usage: Size depends on runtime options and output modalities; leave headroom for multimodal workloads. Prefer CLI overrides such as `--gpu-memory-utilization` for routine tuning. -- Key flags: `--omni` is required; async chunking is enabled by default; use `--no-async-chunk` for realtime sessions and `--deploy-config` only for advanced custom deployments. -- Known limitations: The `/v1/realtime` WebSocket flow is currently unsupported while async chunking is enabled. This starter recipe is intentionally narrow and focuses on the single-GPU online-serving path already documented in the repo examples. +- `/v1/realtime` is unsupported while `async_chunk` is enabled. +- The default deploy uses `SharedMemoryConnector`; this is for single-host + stage wiring.