[Perf][Qwen3-TTS] Enable event-driven orchestration by default (#7088)

Signed-off-by: Sy03 <1370724210@qq.com>
This commit is contained in:
Sy03 authored and GitHub committed 2026-09-21 12:09:19 -07:00
1 parent 60a933b9ab
commit 473da2cf82
13 files changed
+231 -27

No files matched your search

+1 -1
View File
@@ -85,7 +85,7 @@ depends on the installed kernels and model path.
| `SPEAKER_MAX_UPLOADED` | Integer; default `1000` | Speech server; read when speaker storage initializes | Environment-only setting. A non-integer logs a warning and uses `1000`; range is not otherwise validated. | Stable |
| `VLLM_OMNI_ABORT_TIMEOUT` | Float seconds; default `2` | Engine abort wait for Videos DELETE and `generate()` cancel/error cleanup; read when `async_omni` imports | Environment-only setting. A non-float raises `ValueError` during import. | Experimental |
| `VLLM_OMNI_ASYNC_OUTPUT_TIMEOUT` | Float seconds; default `600` | Diffusion engine async-output wait in `step_streaming`; resolved per call on the request path, not at import | Environment-only setting. A non-float or `<=0` value warns once and uses the default. | Experimental |
| `VLLM_OMNI_EVENT_DRIVEN_ORCH` | `1`, `true`, `yes` or `on` enables; default `0` (off) | Orchestration loop and the serving-side final-output drain; read once when the `Orchestrator` is constructed | Environment-only setting. Values are stripped and case-normalized; any unrecognized value leaves the legacy poll loop selected. | Experimental |
| `VLLM_OMNI_EVENT_DRIVEN_ORCH` | `1`, `true`, `yes` or `on` enables; defaults to on for Qwen3-TTS and off for other pipelines | Pipeline default computed from `pipeline_config.model_type` at engine initialization; env override resolved at `Orchestrator` construction and separately when the serving-side final-output drain starts | An explicit env value wins; otherwise both consumers use the engine's pipeline default. Values are stripped and case-normalized; any unrecognized value selects the legacy poll loop. Set before server startup. | Experimental |
| `VLLM_OMNI_INPUT_WAIT_TIMEOUT_S` | Float seconds; default `600`; `<=0` disables | Full-payload input coordinator, not async-chunk transfer; read when the scheduler module imports in each worker | Environment-only setting. A non-float logs a warning and uses `600`. | Stable operational control |
| `VLLM_OMNI_ORCH_MONITOR_PATH` | Filesystem path; default `<current-working-directory>/vllm_omni_orch_monitor_<timestamp>.json` | Orchestrator monitor enabled by `--enable-orch-monitor`; read when the monitor is created | Environment-only path override. Parent directories are created; write errors are logged. | Diagnostic |
| `VLLM_OMNI_VIDEO_SYNC_TIMEOUT` | Float seconds; default `600` | Synchronous Videos API; read when the API server module imports | Environment-only setting. A non-float raises `ValueError` during import. | Experimental |
+55 -3
View File
@@ -22,7 +22,7 @@ profiler_config:
Supported fields:
| Field | Description |
|---|---|
| --- | --- |
| `profiler` | Profiler backend. Supported values: `torch`, `cuda`. Use `torch` for `trace.json`, Excel operator tables, and optional memory snapshots. Use `cuda` for Nsight Systems only. |
| `torch_profiler_dir` | Output directory for torch-profiler artifacts. Required when `profiler: torch`. |
| `torch_profiler_use_gzip` | Compress `trace_rank*.json` into `trace_rank*.json.gz`. |
@@ -287,6 +287,52 @@ Recommended viewers:
For upstream background on the underlying vLLM profiling infrastructure, see the [vLLM profiling guide](https://docs.vllm.ai/en/stable/contributing/profiling/).
## 6. Profiling Omni AR/TTS stages
For Omni AR and TTS stages, put `profiler_config` under each stage that should
be captured. The serving API then controls a narrow window with
`POST /start_profile` and `POST /stop_profile`; model initialization and
warmup stay outside the trace.
Example stage override for a host-stack torch trace:
```bash
PROFILE='{"profiler":"torch","torch_profiler_dir":"/tmp/qwen3_tts_profile",'\
'"torch_profiler_with_stack":true,"torch_profiler_record_shapes":false,'\
'"torch_profiler_use_gzip":true,"torch_profiler_dump_cuda_time_total":true}'
vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice --omni \
--deploy-config vllm_omni/deploy/qwen3_tts_high_concurrency.yaml \
--stage-overrides "{\"0\":{\"profiler_config\":$PROFILE},\"1\":{\"profiler_config\":$PROFILE}}"
```
After one or more warmup requests, capture one short measured request:
```bash
curl -sS -X POST http://127.0.0.1:8000/start_profile \
-H 'Content-Type: application/json' -d '{"stages":[0,1]}'
# send one fixed /v1/audio/speech or /v1/chat/completions request here
curl -sS -X POST http://127.0.0.1:8000/stop_profile \
-H 'Content-Type: application/json' -d '{"stages":[0,1]}'
```
For an Nsight Systems CUDA timeline, use the same stage override with
`{"profiler":"cuda"}` and wrap the server:
```bash
nsys profile --trace=cuda,nvtx,osrt --sample=process-tree \
--cpuctxsw=process-tree --trace-fork-before-exec=true \
--capture-range=cudaProfilerApi --capture-range-end=stop-shutdown \
--cuda-graph-trace=node -o /tmp/qwen3_tts_nsys \
vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice --omni ...
```
For Omni AR workers, `profiler="cuda"` requires the native CUDA profiler
wrapper and uses `cudaProfilerStart/Stop`; the report is only valid if the
`/start_profile` and `/stop_profile` calls both succeed. Use `nsys stats` for
CUDA API, GPU kernel, OS runtime, and NVTX summaries. Treat profiler latency as
diagnostic only; use a separate no-profiler run for performance numbers.
## 7. Orchestrator Monitor
> **Warning:** Diagnostic only. Adds lightweight counters on the orchestrator poll loop and per-replica queue depth. Disable in production unless you are actively debugging orchestrator saturation.
@@ -314,9 +360,11 @@ If unset, the monitor writes `./vllm_omni_orch_monitor_<MMDDHHMM>.json` at shutd
Each 1-second window records:
| Series | Description |
|---|---|
| --- | --- |
| `windows.duration_s` | Wall time covered by the window |
| `windows.loop_idle` / `windows.loop_active` | Orchestrator poll-loop iterations with no work vs. active forwarding |
| `windows.dispatch_queue_size` | Ready-queue size sampled at each monitor window boundary |
| `windows.dispatch_queue_high_water` | Maximum Event Driven ready-queue size observed in each window |
| `replicas.<stage,replica>.outputs_queue_size` | MP client `outputs_queue` backlog for that replica |
| `replicas.<stage,replica>.inflight` | Requests currently bound/routed to the replica |
@@ -331,12 +379,16 @@ orchestrator wakes only once per 0.5 s reconcile timeout, while a busy one still
records one iteration per routed output. Window counts and the `loop_active_pct`
summary are therefore not comparable across the two modes.
For Event Driven runs, use `dispatch_queue_high_water` rather than only the
window-boundary `dispatch_queue_size`: a serial dispatcher can drain a burst
before the window closes, hiding a transient ready-queue backlog in the latter.
### Relationship to other diagnostics
This monitor is intentionally separate from the existing profiling tools:
| Tool | Scope | Output |
|---|---|---|
| --- | --- | --- |
| `--enable-diffusion-pipeline-profiler` | Diffusion stage functions (`vae.decode`, `diffuse`, …) | Per-request `stage_durations` / logs |
| `--enable-ar-profiler` | AR stage generation time | Per-request `stage_durations` |
| `profiler_config` (`torch` / `cuda`) | GPU/CPU kernels inside a stage worker | `trace.json`, Nsight, operator tables |
+1 -1
View File
@@ -79,7 +79,7 @@ the boundary affected by the in-flight stage client/process refactor in
responsibilities proposed only by that PR are not current contracts.
The orchestration loop also has an opt-in event-driven mode
(`VLLM_OMNI_EVENT_DRIVEN_ORCH=1`, default off) proposed in
(`VLLM_OMNI_EVENT_DRIVEN_ORCH=1`, default off except for Qwen3-TTS) proposed in
[#5221](https://github.com/vllm-project/vllm-omni/pull/5221). It changes poll
cadence only: the routing, ordering, and terminal-state contracts below hold
identically on both loops.
+12 -9
View File
@@ -957,22 +957,24 @@ Use `/v1/audio/voices` to list available voices for the loaded model.
## Orchestration Loop (experimental)
Multi-stage omni deployments route stage outputs through a single orchestrator
loop. By default that loop polls every stage replica on a 1 ms cadence. An
opt-in event-driven mode replaces the poll with one reader task per live stage
loop. The legacy loop polls every stage replica on a 1 ms cadence. The
event-driven mode replaces the poll with one reader task per live stage
replica awaiting its client directly, and switches the serving-side
final-output drain to a condition-variable wakeup at the same time.
final-output drain to a condition-variable wakeup at the same time. Qwen3-TTS
uses this mode by default; other pipelines keep the legacy poll unless enabled.
**Configuration (environment variables):**
| Variable | Default | Description |
| --- | --- | --- |
| `VLLM_OMNI_EVENT_DRIVEN_ORCH` | `0` (off) | Switches the orchestration loop and the final-output drain from the legacy 1 ms poll to event-driven wakeups. Enabled by `1`, `true`, `yes`, or `on`, matched case-insensitively after surrounding whitespace is stripped; any other value leaves it off. |
| `VLLM_OMNI_EVENT_DRIVEN_ORCH` | On for Qwen3-TTS; off for other pipelines | Switches the orchestration loop and the final-output drain from the legacy 1 ms poll to event-driven wakeups. An explicit value wins; otherwise the pipeline default computed at engine initialization is used. The override is resolved at orchestrator construction and separately when the final-output drain starts. `1`, `true`, `yes`, or `on` enables it, ignoring case and surrounding whitespace; other values select the legacy poll loop. |
Set it on the process that runs the orchestrator (stage 0 of an omni
deployment) before starting the server:
Set it on the process that runs the orchestrator (stage 0 of an omni deployment)
before starting the server when overriding the pipeline default. For example,
explicitly disable event-driven orchestration for Qwen3-TTS:
```bash
export VLLM_OMNI_EVENT_DRIVEN_ORCH=1
VLLM_OMNI_EVENT_DRIVEN_ORCH=0 \
vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-Base \
--omni \
--port 8091
@@ -982,8 +984,9 @@ The server logs the selected loop mode and its reader/poller counts once at
startup, so you can confirm which loop is live.
Routing, output ordering, and terminal-state behavior are identical on both
loops; only the poll cadence changes. Leaving the variable unset keeps the
legacy poll loop, which is the supported default.
loops; only the poll cadence changes. Leaving the variable unset selects the
pipeline default: event-driven for Qwen3-TTS, and legacy polling for other
pipelines, including Qwen3-Omni.
**Known limitations:**