Merge remote-tracking branch 'origin/main' into qwen-image-2.1
# Conflicts: # tests/entrypoints/openai_api/test_image_server.py # vllm_omni/entrypoints/openai/api_server.py
This commit is contained in:
commit
3e274a0024
220 files changed
+22342
-1079
No files matched your search
Binary file not shown.
|
Before Width: | Height: | Size: 267 KiB After Width: | Height: | Size: 244 KiB |
@@ -85,7 +85,7 @@ depends on the installed kernels and model path.
|
||||
| `SPEAKER_MAX_UPLOADED` | Integer; default `1000` | Speech server; read when speaker storage initializes | Environment-only setting. A non-integer logs a warning and uses `1000`; range is not otherwise validated. | Stable |
|
||||
| `VLLM_OMNI_ABORT_TIMEOUT` | Float seconds; default `2` | Engine abort wait for Videos DELETE and `generate()` cancel/error cleanup; read when `async_omni` imports | Environment-only setting. A non-float raises `ValueError` during import. | Experimental |
|
||||
| `VLLM_OMNI_ASYNC_OUTPUT_TIMEOUT` | Float seconds; default `600` | Diffusion engine async-output wait in `step_streaming`; resolved per call on the request path, not at import | Environment-only setting. A non-float or `<=0` value warns once and uses the default. | Experimental |
|
||||
| `VLLM_OMNI_EVENT_DRIVEN_ORCH` | `1`, `true`, `yes` or `on` enables; default `0` (off) | Orchestration loop and the serving-side final-output drain; read once when the `Orchestrator` is constructed | Environment-only setting. Values are stripped and case-normalized; any unrecognized value leaves the legacy poll loop selected. | Experimental |
|
||||
| `VLLM_OMNI_EVENT_DRIVEN_ORCH` | `1`, `true`, `yes` or `on` enables; defaults to on for Qwen3-TTS and off for other pipelines | Pipeline default computed from `pipeline_config.model_type` at engine initialization; env override resolved at `Orchestrator` construction and separately when the serving-side final-output drain starts | An explicit env value wins; otherwise both consumers use the engine's pipeline default. Values are stripped and case-normalized; any unrecognized value selects the legacy poll loop. Set before server startup. | Experimental |
|
||||
| `VLLM_OMNI_INPUT_WAIT_TIMEOUT_S` | Float seconds; default `600`; `<=0` disables | Full-payload input coordinator, not async-chunk transfer; read when the scheduler module imports in each worker | Environment-only setting. A non-float logs a warning and uses `600`. | Stable operational control |
|
||||
| `VLLM_OMNI_ORCH_MONITOR_PATH` | Filesystem path; default `<current-working-directory>/vllm_omni_orch_monitor_<timestamp>.json` | Orchestrator monitor enabled by `--enable-orch-monitor`; read when the monitor is created | Environment-only path override. Parent directories are created; write errors are logged. | Diagnostic |
|
||||
| `VLLM_OMNI_VIDEO_SYNC_TIMEOUT` | Float seconds; default `600` | Synchronous Videos API; read when the API server module imports | Environment-only setting. A non-float raises `ValueError` during import. | Experimental |
|
||||
|
||||
@@ -293,19 +293,25 @@ vllm serve Qwen/Qwen2.5-Omni-7B --omni --port 8091 --deploy-config /path/to/depl
|
||||
|
||||
## Qwen3-TTS with Model Runner V2
|
||||
|
||||
Qwen3-TTS can opt into the native CUDA Model Runner V2 pipeline on vLLM
|
||||
0.29.0. Select one of these deployment profiles:
|
||||
Qwen3-TTS runs the native CUDA Model Runner V2 pipeline by default on vLLM
|
||||
0.29.0. MRV2 is an experimental feature for this model: the bundled default
|
||||
profile selects it on CUDA only, and its scheduler and delivery paths are
|
||||
still being qualified. Set `model_runner: v1` in a copy of the deploy config
|
||||
to opt out; the `platforms:` sections of `qwen3_tts.yaml` keep V1 on NPU, XPU,
|
||||
ROCm and MUSA. Select one of these deployment profiles:
|
||||
|
||||
| Profile | Runner | Code2Wav graph batches | Intended use |
|
||||
| --- | --- | --- | --- |
|
||||
| `qwen3_tts.yaml` | V1 | Existing defaults | Existing deployment / regression control |
|
||||
| `qwen3_tts_mrv2.yaml` | V2 | B1 | Native runner validation |
|
||||
| `qwen3_tts.yaml` | V2 (default) | Existing defaults | Shipped default; experimental |
|
||||
| `qwen3_tts_mrv2.yaml` | V2 | B1 | Explicit MRV2 profile (same runner selection as the default) |
|
||||
| `qwen3_tts_high_concurrency_mrv2.yaml` | V2 | B1, B2 | Opt-in throughput tuning |
|
||||
| `qwen3_tts_high_concurrency_mrv2_b4.yaml` | V2 | B1, B2, B3, B4 | Experimental throughput / buffered playback |
|
||||
| `qwen3_tts_high_concurrency.yaml` | V1 | Existing defaults | V1 high-concurrency control |
|
||||
|
||||
```bash
|
||||
# MRV2 is the default; pass a copy with `model_runner: v1` to force V1.
|
||||
vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-Base --omni \
|
||||
--deploy-config vllm_omni/deploy/qwen3_tts_mrv2.yaml
|
||||
--deploy-config /path/to/qwen3_tts_v1.yaml
|
||||
```
|
||||
|
||||
The V2 profiles bound Talker prefill to 512 tokens per step and select the
|
||||
|
||||
@@ -177,10 +177,10 @@ Failed jobs: 1/2
|
||||
cd tests
|
||||
|
||||
# Stability: Qwen3-Omni
|
||||
pytest -s -v dfx/stability/scripts/test_stability_qwen3_omni.py -m slow
|
||||
pytest -s -v dfx/stability/scripts/run_stability_qwen3_omni.py -m slow
|
||||
|
||||
# Stability: Wan2.2 (v1/videos diffusion benchmark loop)
|
||||
pytest -s -v dfx/stability/scripts/test_stability_wan22.py -m slow
|
||||
pytest -s -v dfx/stability/scripts/run_stability_wan22.py -m slow
|
||||
|
||||
# Reliability: Qwen3-Omni (H100 × 2)
|
||||
pytest -s -v dfx/reliability/test_reliability_qwen3_omni.py -m slow
|
||||
|
||||
@@ -555,7 +555,7 @@ L5 level testing focuses on the performance of model services under ***long-runn
|
||||
|
||||
- ***Trigger Timing***: **`Weekly`** (weekly) or **`Days before Release`** (several days before a major release). Due to long execution times, the frequency is lower.
|
||||
- ***Run Command***:
|
||||
- ***Stability***: `pytest -s -v tests/dfx/stability/scripts/test_stability_qwen3_omni.py` or `pytest -s -v tests/dfx/stability/scripts/test_stability_wan22.py` (or add `test_stability_<model>.py` alongside a matching JSON config)
|
||||
- ***Stability***: `pytest -s -v tests/dfx/stability/scripts/run_stability_qwen3_omni.py` or `pytest -s -v tests/dfx/stability/scripts/run_stability_wan22.py` (or add `run_stability_<model>.py` alongside a matching JSON config)
|
||||
- ***Reliability***: `pytest -s -v tests/dfx/reliability/test_reliability_<model>.py -m slow` (current suites: `qwen3_omni`, `wan22`, `hunyuan_image`).
|
||||
- ***Script Example***:
|
||||
|
||||
@@ -564,7 +564,7 @@ L5 level testing focuses on the performance of model services under ***long-runn
|
||||
|
||||
##### Stability
|
||||
|
||||
When you want to add L5-level stability test cases, add or extend the appropriate JSON file under `tests/dfx/stability/tests/` (for example `test_qwen3_omni.json` for Omni bench traffic, or `test_wan22.json` for diffusion `/v1/videos` workloads). Pair the JSON with `tests/dfx/stability/scripts/test_stability_<model>.py`. The following illustrates the Qwen3-Omni shape:
|
||||
When you want to add L5-level stability test cases, add or extend the appropriate JSON file under `tests/dfx/stability/tests/` (for example `test_qwen3_omni.json` for Omni bench traffic, or `test_wan22.json` for diffusion `/v1/videos` workloads). Pair the JSON with `tests/dfx/stability/scripts/run_stability_<model>.py`. The following illustrates the Qwen3-Omni shape:
|
||||
|
||||
```json
|
||||
{
|
||||
|
||||
@@ -22,7 +22,7 @@ profiler_config:
|
||||
Supported fields:
|
||||
|
||||
| Field | Description |
|
||||
|---|---|
|
||||
| --- | --- |
|
||||
| `profiler` | Profiler backend. Supported values: `torch`, `cuda`. Use `torch` for `trace.json`, Excel operator tables, and optional memory snapshots. Use `cuda` for Nsight Systems only. |
|
||||
| `torch_profiler_dir` | Output directory for torch-profiler artifacts. Required when `profiler: torch`. |
|
||||
| `torch_profiler_use_gzip` | Compress `trace_rank*.json` into `trace_rank*.json.gz`. |
|
||||
@@ -287,6 +287,52 @@ Recommended viewers:
|
||||
|
||||
For upstream background on the underlying vLLM profiling infrastructure, see the [vLLM profiling guide](https://docs.vllm.ai/en/stable/contributing/profiling/).
|
||||
|
||||
## 6. Profiling Omni AR/TTS stages
|
||||
|
||||
For Omni AR and TTS stages, put `profiler_config` under each stage that should
|
||||
be captured. The serving API then controls a narrow window with
|
||||
`POST /start_profile` and `POST /stop_profile`; model initialization and
|
||||
warmup stay outside the trace.
|
||||
|
||||
Example stage override for a host-stack torch trace:
|
||||
|
||||
```bash
|
||||
PROFILE='{"profiler":"torch","torch_profiler_dir":"/tmp/qwen3_tts_profile",'\
|
||||
'"torch_profiler_with_stack":true,"torch_profiler_record_shapes":false,'\
|
||||
'"torch_profiler_use_gzip":true,"torch_profiler_dump_cuda_time_total":true}'
|
||||
|
||||
vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice --omni \
|
||||
--deploy-config vllm_omni/deploy/qwen3_tts_high_concurrency.yaml \
|
||||
--stage-overrides "{\"0\":{\"profiler_config\":$PROFILE},\"1\":{\"profiler_config\":$PROFILE}}"
|
||||
```
|
||||
|
||||
After one or more warmup requests, capture one short measured request:
|
||||
|
||||
```bash
|
||||
curl -sS -X POST http://127.0.0.1:8000/start_profile \
|
||||
-H 'Content-Type: application/json' -d '{"stages":[0,1]}'
|
||||
# send one fixed /v1/audio/speech or /v1/chat/completions request here
|
||||
curl -sS -X POST http://127.0.0.1:8000/stop_profile \
|
||||
-H 'Content-Type: application/json' -d '{"stages":[0,1]}'
|
||||
```
|
||||
|
||||
For an Nsight Systems CUDA timeline, use the same stage override with
|
||||
`{"profiler":"cuda"}` and wrap the server:
|
||||
|
||||
```bash
|
||||
nsys profile --trace=cuda,nvtx,osrt --sample=process-tree \
|
||||
--cpuctxsw=process-tree --trace-fork-before-exec=true \
|
||||
--capture-range=cudaProfilerApi --capture-range-end=stop-shutdown \
|
||||
--cuda-graph-trace=node -o /tmp/qwen3_tts_nsys \
|
||||
vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice --omni ...
|
||||
```
|
||||
|
||||
For Omni AR workers, `profiler="cuda"` requires the native CUDA profiler
|
||||
wrapper and uses `cudaProfilerStart/Stop`; the report is only valid if the
|
||||
`/start_profile` and `/stop_profile` calls both succeed. Use `nsys stats` for
|
||||
CUDA API, GPU kernel, OS runtime, and NVTX summaries. Treat profiler latency as
|
||||
diagnostic only; use a separate no-profiler run for performance numbers.
|
||||
|
||||
## 7. Orchestrator Monitor
|
||||
|
||||
> **Warning:** Diagnostic only. Adds lightweight counters on the orchestrator poll loop and per-replica queue depth. Disable in production unless you are actively debugging orchestrator saturation.
|
||||
@@ -314,9 +360,11 @@ If unset, the monitor writes `./vllm_omni_orch_monitor_<MMDDHHMM>.json` at shutd
|
||||
Each 1-second window records:
|
||||
|
||||
| Series | Description |
|
||||
|---|---|
|
||||
| --- | --- |
|
||||
| `windows.duration_s` | Wall time covered by the window |
|
||||
| `windows.loop_idle` / `windows.loop_active` | Orchestrator poll-loop iterations with no work vs. active forwarding |
|
||||
| `windows.dispatch_queue_size` | Ready-queue size sampled at each monitor window boundary |
|
||||
| `windows.dispatch_queue_high_water` | Maximum Event Driven ready-queue size observed in each window |
|
||||
| `replicas.<stage,replica>.outputs_queue_size` | MP client `outputs_queue` backlog for that replica |
|
||||
| `replicas.<stage,replica>.inflight` | Requests currently bound/routed to the replica |
|
||||
|
||||
@@ -331,12 +379,16 @@ orchestrator wakes only once per 0.5 s reconcile timeout, while a busy one still
|
||||
records one iteration per routed output. Window counts and the `loop_active_pct`
|
||||
summary are therefore not comparable across the two modes.
|
||||
|
||||
For Event Driven runs, use `dispatch_queue_high_water` rather than only the
|
||||
window-boundary `dispatch_queue_size`: a serial dispatcher can drain a burst
|
||||
before the window closes, hiding a transient ready-queue backlog in the latter.
|
||||
|
||||
### Relationship to other diagnostics
|
||||
|
||||
This monitor is intentionally separate from the existing profiling tools:
|
||||
|
||||
| Tool | Scope | Output |
|
||||
|---|---|---|
|
||||
| --- | --- | --- |
|
||||
| `--enable-diffusion-pipeline-profiler` | Diffusion stage functions (`vae.decode`, `diffuse`, …) | Per-request `stage_durations` / logs |
|
||||
| `--enable-ar-profiler` | AR stage generation time | Per-request `stage_durations` |
|
||||
| `profiler_config` (`torch` / `cuda`) | GPU/CPU kernels inside a stage worker | `trace.json`, Nsight, operator tables |
|
||||
|
||||
@@ -79,7 +79,7 @@ the boundary affected by the in-flight stage client/process refactor in
|
||||
responsibilities proposed only by that PR are not current contracts.
|
||||
|
||||
The orchestration loop also has an opt-in event-driven mode
|
||||
(`VLLM_OMNI_EVENT_DRIVEN_ORCH=1`, default off) proposed in
|
||||
(`VLLM_OMNI_EVENT_DRIVEN_ORCH=1`, default off except for Qwen3-TTS) proposed in
|
||||
[#5221](https://github.com/vllm-project/vllm-omni/pull/5221). It changes poll
|
||||
cadence only: the routing, ordering, and terminal-state contracts below hold
|
||||
identically on both loops.
|
||||
|
||||
+63
-17
@@ -9,9 +9,11 @@ This feature is inherited from [vLLM’s Sleep Mode](https://blog.vllm.ai/2025/1
|
||||
## 1. Feature Documentation
|
||||
|
||||
### Overview
|
||||
|
||||
Omni Sleep Mode provides a mechanism to "sleep" specific model stages. When a stage enters sleep, its physical VRAM is reclaimed by the system, while the process state is preserved for rapid "wake-up" without full re-initialization.
|
||||
|
||||
### Sleep Levels
|
||||
|
||||
We support two levels of hibernation to balance recovery speed and memory efficiency:
|
||||
|
||||
| Level | Name | Mechanism | Recovery Speed | Memory Freed |
|
||||
@@ -29,15 +31,67 @@ Omni Sleep Mode is optimized for high-performance computing backends:
|
||||
* **Huawei NPU**: Supported via Ascend memory scavenging.
|
||||
|
||||
### Hardware Requirements
|
||||
|
||||
* **Memory Considerations**: System RAM must be sufficient to hold offloaded weights during sleep.
|
||||
* **TP Support**: Tensor Parallel groups synchronize sleep/wake transitions across all workers.
|
||||
|
||||
---
|
||||
### Quiescing a busy diffusion stage: `pause_generation(mode="keep")`
|
||||
|
||||
`sleep()` on a diffusion stage is a worker-level RPC: it offloads memory but does not stop the
|
||||
stage's scheduler, so a stage that still has queued requests must be made quiet first. Aborting
|
||||
discards the batch that is running. `pause_generation(mode="keep")` keeps it instead:
|
||||
|
||||
* The batch that is already running finishes on the weights it started with and its outputs are
|
||||
delivered normally.
|
||||
* Requests that have not started stay queued and run after `resume_generation()`, on whatever
|
||||
weights are loaded by then. A request never continues denoising across a weight change.
|
||||
* The call returns only after every replica has acknowledged: the batch that was running has
|
||||
finished executing, its asynchronous output copies are done, and the device is synchronized; its
|
||||
outputs are then delivered on the normal path. On a non-preemptible backend that acknowledgement
|
||||
arrives at the next batch boundary, so expect it to take as long as the remaining batch.
|
||||
* Control RPCs (`list_loras`, `sleep`, `wake_up`, `resume_generation`) keep working while paused.
|
||||
|
||||
Recommended order for a weight update on a diffusion stage:
|
||||
|
||||
```python
|
||||
await engine.pause_generation(mode="keep", clear_cache=False)
|
||||
await engine.sleep(level=1) # optional: free memory for the trainer
|
||||
await engine.wake_up() # admission stays closed until resume
|
||||
# install new weights here
|
||||
await engine.resume_generation()
|
||||
```
|
||||
|
||||
Support matrix and limits:
|
||||
|
||||
| Configuration | `mode="keep"` |
|
||||
| :--- | :--- |
|
||||
| Request-level execution (`step_execution=False`) with the in-process (`uni`) executor | Supported |
|
||||
| Request-level execution with the multi-process (`mp`) executor, including asynchronous output | Supported; the acknowledgement waits for the output copies |
|
||||
| Request-level execution with asynchronous KV prefetch | Supported; the acknowledgement joins any prefetch still in flight |
|
||||
| Step-level execution (`step_execution=True` / streaming output) | Not supported: the call raises `NotImplementedError` |
|
||||
| `mode="abort"` / `mode="wait"` / `wait_for_inflight_requests=True` on a diffusion stage | Unchanged: frontend admission is paused only |
|
||||
|
||||
Behavior notes:
|
||||
|
||||
* Callers that already passed `mode="keep"` for a diffusion stage used to get an immediate return
|
||||
that only closed frontend admission. They now also stop the backend and wait for the ACK.
|
||||
* A timeout or error from `pause_generation` means the pause did **not** complete; the scheduler
|
||||
stays closed and the caller must retry the pause or call `resume_generation()` explicitly. Do not
|
||||
sleep or mutate weights after a failed pause.
|
||||
* `resume_generation(stage_ids=...)` that leaves another paused stage closed (an AR stage, or a
|
||||
keep-paused diffusion stage) keeps frontend admission closed as well, so new requests cannot
|
||||
queue on a stage that will not schedule them. Admission reopens once every stage paused by
|
||||
`pause_generation` has been resumed.
|
||||
* `clear_cache` has no diffusion-specific effect.
|
||||
* Without a pause call nothing changes: no barrier runs and batching, abort and output ordering are
|
||||
as before.
|
||||
|
||||
---
|
||||
|
||||
## 2. Usage Examples
|
||||
|
||||
### Python API Example
|
||||
|
||||
You can programmatically control the lifecycle of stages using the `AsyncOmni` engine.
|
||||
|
||||
```python
|
||||
@@ -65,6 +119,7 @@ if __name__ == "__main__":
|
||||
```
|
||||
|
||||
### server command Example
|
||||
|
||||
Start the server with sleep mode enabled:
|
||||
|
||||
The first method
|
||||
@@ -92,9 +147,6 @@ python3 -m vllm_omni.entrypoints.openai.api_server \
|
||||
|
||||
```
|
||||
|
||||
|
||||
|
||||
|
||||
### Test Scenarios & Commands
|
||||
|
||||
#### Scenario 1: LLM Engine Sleep
|
||||
@@ -113,9 +165,8 @@ curl -X POST http://localhost:8000/v1/omni/sleep \
|
||||
|
||||
Tip: Open a new terminal and run rocm-smi or nvidia-smi or to observe the immediate drop in VRAM usage.
|
||||
|
||||
|
||||
|
||||
#### Scenario 2: Diffusion Sleep
|
||||
|
||||
Objective: Verify VRAM reclamation for Stage 1 (Diffusion).
|
||||
|
||||
Trigger sleep (Level 1 or Level 2) via client:
|
||||
@@ -128,9 +179,8 @@ curl -X POST http://localhost:8000/v1/omni/sleep \
|
||||
|
||||
```
|
||||
|
||||
|
||||
|
||||
#### Scenario 3: Multi-Stage Coordinated Stress Test
|
||||
|
||||
Objective: Test concurrent sleep and rapid wake-up across multiple stages.
|
||||
|
||||
Concurrent Sleep (Stage 0 & 1):
|
||||
@@ -143,7 +193,6 @@ curl -X POST http://localhost:8000/v1/omni/sleep \
|
||||
|
||||
```
|
||||
|
||||
|
||||
Rapid Wake-up:
|
||||
|
||||
```
|
||||
@@ -154,8 +203,8 @@ curl -X POST http://localhost:8000/v1/omni/wakeup \
|
||||
|
||||
```
|
||||
|
||||
|
||||
#### Scenario 4: Full Lifecycle Memory Audit & Functional Integrity
|
||||
|
||||
Objective: Audit the complete flow from Sleep to Wake-up followed by an Inference validation.
|
||||
|
||||
Check Initial State: Observe baseline VRAM usage.
|
||||
@@ -196,20 +245,16 @@ curl -X POST http://localhost:8000/v1/images/generations \
|
||||
|
||||
```
|
||||
|
||||
|
||||
|
||||
|
||||
## 3. API Reference
|
||||
|
||||
|
||||
### Methods
|
||||
|
||||
| Method | Arguments | Return Type | Description |
|
||||
| :--- | :--- | :--- | :--- |
|
||||
| **sleep** | `stage_ids: List[int], level: int` | `List[OmniACK]` | Triggers hibernation for specified stages. |
|
||||
| **wake_up** | `stage_ids: List[int]` | `List[OmniACK]` | Reloads weights and re-maps memory. |
|
||||
|
||||
|
||||
| **pause_generation** | `mode: str, stage_ids: List[int]` | `None` | Stops admission; with `mode="keep"` also stops diffusion schedulers and returns after their ACK. |
|
||||
| **resume_generation** | `stage_ids: List[int]` | `None` | Reopens paused schedulers, then admission. |
|
||||
|
||||
### OmniACK Dataclass Fields
|
||||
|
||||
@@ -233,7 +278,8 @@ The metadata field is a dynamic dictionary containing hardware-specific telemetr
|
||||
}
|
||||
```
|
||||
|
||||
#### Core Utility:
|
||||
#### Core Utility
|
||||
|
||||
**VRAM Reclamation Audit (total_freed_gib)**: Converts raw freed_bytes into human-readable GiB. It serves as the primary metric to verify that Level 2 sleep has successfully purged model weights from VRAM.
|
||||
|
||||
**Residual & Fragmentation Monitoring (rank_residual_gib)**: Reports the remaining VRAM footprint after memory de-mapping. A low residual value (e.g., 2.07 GiB) confirms a successful "clean" state, ensuring the device is ready for high-memory co-located tasks like training or diffusion pipelines.
|
||||
|
||||
@@ -957,22 +957,24 @@ Use `/v1/audio/voices` to list available voices for the loaded model.
|
||||
## Orchestration Loop (experimental)
|
||||
|
||||
Multi-stage omni deployments route stage outputs through a single orchestrator
|
||||
loop. By default that loop polls every stage replica on a 1 ms cadence. An
|
||||
opt-in event-driven mode replaces the poll with one reader task per live stage
|
||||
loop. The legacy loop polls every stage replica on a 1 ms cadence. The
|
||||
event-driven mode replaces the poll with one reader task per live stage
|
||||
replica awaiting its client directly, and switches the serving-side
|
||||
final-output drain to a condition-variable wakeup at the same time.
|
||||
final-output drain to a condition-variable wakeup at the same time. Qwen3-TTS
|
||||
uses this mode by default; other pipelines keep the legacy poll unless enabled.
|
||||
|
||||
**Configuration (environment variables):**
|
||||
|
||||
| Variable | Default | Description |
|
||||
| --- | --- | --- |
|
||||
| `VLLM_OMNI_EVENT_DRIVEN_ORCH` | `0` (off) | Switches the orchestration loop and the final-output drain from the legacy 1 ms poll to event-driven wakeups. Enabled by `1`, `true`, `yes`, or `on`, matched case-insensitively after surrounding whitespace is stripped; any other value leaves it off. |
|
||||
| `VLLM_OMNI_EVENT_DRIVEN_ORCH` | On for Qwen3-TTS; off for other pipelines | Switches the orchestration loop and the final-output drain from the legacy 1 ms poll to event-driven wakeups. An explicit value wins; otherwise the pipeline default computed at engine initialization is used. The override is resolved at orchestrator construction and separately when the final-output drain starts. `1`, `true`, `yes`, or `on` enables it, ignoring case and surrounding whitespace; other values select the legacy poll loop. |
|
||||
|
||||
Set it on the process that runs the orchestrator (stage 0 of an omni
|
||||
deployment) before starting the server:
|
||||
Set it on the process that runs the orchestrator (stage 0 of an omni deployment)
|
||||
before starting the server when overriding the pipeline default. For example,
|
||||
explicitly disable event-driven orchestration for Qwen3-TTS:
|
||||
|
||||
```bash
|
||||
export VLLM_OMNI_EVENT_DRIVEN_ORCH=1
|
||||
VLLM_OMNI_EVENT_DRIVEN_ORCH=0 \
|
||||
vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-Base \
|
||||
--omni \
|
||||
--port 8091
|
||||
@@ -982,8 +984,9 @@ The server logs the selected loop mode and its reader/poller counts once at
|
||||
startup, so you can confirm which loop is live.
|
||||
|
||||
Routing, output ordering, and terminal-state behavior are identical on both
|
||||
loops; only the poll cadence changes. Leaving the variable unset keeps the
|
||||
legacy poll loop, which is the supported default.
|
||||
loops; only the poll cadence changes. Leaving the variable unset selects the
|
||||
pipeline default: event-driven for Qwen3-TTS, and legacy polling for other
|
||||
pipelines, including Qwen3-Omni.
|
||||
|
||||
**Known limitations:**
|
||||
|
||||
|
||||
@@ -6,29 +6,29 @@
|
||||
ASR -> AURA -> Qwen3-TTS Talker -> Code2Wav
|
||||
```
|
||||
|
||||
Qwen3-TTS remains two engine stages so the pipeline reuses the existing native
|
||||
Talker and Code2Wav implementation.
|
||||
**Supported online path:** Realtime duplex at `/v1/realtime?duplex=1`.
|
||||
|
||||
The default deploy profile (`vllm_omni/deploy/aura_omni.yaml`) sets
|
||||
`session_mode: duplex`. The `aura_omni` pipeline declares `duplex_plugin`, and
|
||||
`DuplexOmniEngine` requires that mode (same pattern as MiniCPM-o duplex
|
||||
deploys). Turn-based `chat/completions` is not a supported AURA online mode
|
||||
against this profile.
|
||||
|
||||
`async_chunk` defaults to `false` in that file (safer if a turn-based
|
||||
`OmniOrchestrator` ever loads it). Duplex Realtime Stage2→3 still needs
|
||||
async chunks — use the smoke deploy below, or set `async_chunk: true`
|
||||
yourself.
|
||||
|
||||
```bash
|
||||
vllm serve aurateam/AURA \
|
||||
--omni \
|
||||
--deploy-config vllm_omni/deploy/aura_omni.yaml \
|
||||
--served-model-name aurateam/AURA \
|
||||
--trust-remote-code
|
||||
bash examples/online_serving/aura_omni/run_duplex_smoke_serve.sh
|
||||
python examples/online_serving/aura_omni/smoke_duplex_realtime_client.py
|
||||
```
|
||||
|
||||
Configure local checkpoints by editing per-stage `model` values in
|
||||
`vllm_omni/deploy/aura_omni.yaml`. The deploy file sets
|
||||
`pipeline: aura_omni`, so the four-stage topology is used even if the
|
||||
command-line `--model` points at one of the component checkpoints.
|
||||
Configure local checkpoints by editing per-stage `model` values in the deploy
|
||||
or smoke YAML. The file sets `pipeline: aura_omni`, so the four-stage topology
|
||||
is used even if the command-line `--model` points at one component checkpoint.
|
||||
|
||||
Send requests with `"model": "aurateam/AURA"`. The ASR, AURA, and Qwen3-TTS
|
||||
checkpoint paths are internal stage models from the deploy YAML, not the
|
||||
OpenAI-facing served model name.
|
||||
|
||||
The AURA stage can emit `<|silent|>`. Silent outputs are treated as a gate:
|
||||
they produce no Qwen3-TTS Talker input, so no audio is synthesized for that
|
||||
turn.
|
||||
Silent Stage1 outputs (`<|silent|>` / id `151669`) skip TTS for that turn.
|
||||
|
||||
## GPU Utilization Recommendation
|
||||
|
||||
@@ -40,26 +40,21 @@ VRAM each stage can reserve. Start with this split for a single GPU:
|
||||
- Stage 2 (Qwen3-TTS Talker): `0.20`
|
||||
- Stage 3 (Qwen3-TTS Code2Wav): `0.20`
|
||||
|
||||
## TTS Modes
|
||||
## TTS modes (stage extras)
|
||||
|
||||
`aura_omni` can pass AURA text to Qwen3-TTS in two task modes:
|
||||
When the duplex session supplies TTS extras, AURA text can feed Qwen3-TTS as:
|
||||
|
||||
- `Base`: voice clone from `tts_ref_audio` with ICL enabled in the AURA
|
||||
pipeline. Provide both `tts_ref_audio` and `tts_ref_text`. Set
|
||||
`tts_x_vector_only_mode=true` to disable ICL and use speaker embedding only.
|
||||
- `CustomVoice`: predefined speaker mode. Use a Qwen3-TTS CustomVoice
|
||||
checkpoint for stages 2 and 3 in `aura_omni.yaml`, then pass
|
||||
`tts_task_type=CustomVoice` and `tts_speaker`.
|
||||
- `Base`: voice clone from `tts_ref_audio` (optional x-vector-only mode)
|
||||
- `CustomVoice`: predefined speaker (`tts_speaker`) with a CustomVoice checkpoint
|
||||
on stages 2 and 3
|
||||
|
||||
By default, AURA responses are passed to Qwen3-TTS as text. Set
|
||||
`additional_information.tts_pass_token_ids=true` to pass AURA-generated
|
||||
assistant token ids directly instead. Even when token passthrough is disabled,
|
||||
the stage processor uses AURA token ids when available to estimate the Talker
|
||||
prefill length, so scheduling does not rely on raw character length.
|
||||
Optional `tts_pass_token_ids` passes AURA assistant token ids into Talker
|
||||
instead of detokenized text.
|
||||
|
||||
The example client exposes this as:
|
||||
## Old chat / Gradio / curl scripts (unsupported)
|
||||
|
||||
```bash
|
||||
python examples/online_serving/aura_omni/openai_chat_completion_client.py \
|
||||
--tts-pass-token-ids
|
||||
```
|
||||
`examples/online_serving/aura_omni/` still contains chat-completions, curl, and
|
||||
Gradio helpers from an older turn-based path. They do **not** work against the
|
||||
shipped duplex profile. That path was an incomplete stand-in before streaming
|
||||
I/O and session history were available; it is not a supported AURA mode. Use
|
||||
the Realtime duplex smoke client for online verification.
|
||||
@@ -247,12 +247,18 @@ curl http://localhost:8091/v1/chat/completions \
|
||||
-d @payload.json
|
||||
```
|
||||
|
||||
BAGEL img2img derives its generated image size from the input image. It
|
||||
preserves the input aspect ratio, aligns dimensions to the latent stride, and
|
||||
applies the checkpoint size limit; explicit `height` and `width` values do not
|
||||
override this model-specific resize policy. In step mode, the effective size is
|
||||
resolved before scheduler admission so requests with different resulting
|
||||
shapes are not combined.
|
||||
When no output size is requested (`size=auto` on `/v1/images/edits`, or no
|
||||
`height`/`width` in a chat request), BAGEL img2img derives the canvas from the
|
||||
input image: it preserves the input aspect ratio, aligns dimensions to the
|
||||
latent stride (16 px), and applies the checkpoint size limit. An explicit
|
||||
`size`, `height`, or `width` is honored instead, in both the single-stage and
|
||||
the two-stage deployment; the input image is still resized to a stride-aligned
|
||||
size for the VAE/ViT prefill. The latent grid needs multiples of 16, so a
|
||||
requested side that is not one is floored to the next multiple (1000x700
|
||||
generates 992x688) and the server logs a warning; the response reports the
|
||||
generated size. A side above the checkpoint limit is rejected with HTTP 400.
|
||||
In step mode, the effective size is resolved before scheduler admission so
|
||||
requests with different resulting shapes are not combined.
|
||||
|
||||
### Image to Text (img2text)
|
||||
|
||||
@@ -318,8 +324,8 @@ curl http://localhost:8091/v1/chat/completions \
|
||||
| `--server` / `-s` | `http://localhost:8091` | Server URL |
|
||||
| `--image-url` / `-i` | `None` | Input image URL or local path (img2img/img2text) |
|
||||
| `--modality` / `-m` | `text2img` | `text2img`, `img2img`, `img2text`, `text2text` |
|
||||
| `--height` | `512` | Text-to-image output height; BAGEL img2img derives its size from the input image |
|
||||
| `--width` | `512` | Text-to-image output width; BAGEL img2img derives its size from the input image |
|
||||
| `--height` | `512` | Output height, a multiple of 16; omit both `--height` and `--width` to let img2img derive the size from the input image |
|
||||
| `--width` | `512` | Output width, a multiple of 16; omit both `--height` and `--width` to let img2img derive the size from the input image |
|
||||
| `--steps` | `25` | Number of inference steps |
|
||||
| `--seed` | `42` | Random seed |
|
||||
| `--negative` | `None` | Negative prompt for CFG |
|
||||
|
||||
Reference in new issue
Block a user