Merge remote-tracking branch 'origin/main' into qwen-image-2.1

# Conflicts:
#	tests/entrypoints/openai_api/test_image_server.py
#	vllm_omni/entrypoints/openai/api_server.py
This commit is contained in:
Isotr0py committed 2026-09-22 01:23:45 +00:00
commit 3e274a0024
220 files changed
+22342 -1079

No files matched your search

Binary file not shown.

Before

Width:  |  Height:  |  Size: 267 KiB

After

Width:  |  Height:  |  Size: 244 KiB

+1 -1
View File
@@ -85,7 +85,7 @@ depends on the installed kernels and model path.
| `SPEAKER_MAX_UPLOADED` | Integer; default `1000` | Speech server; read when speaker storage initializes | Environment-only setting. A non-integer logs a warning and uses `1000`; range is not otherwise validated. | Stable |
| `VLLM_OMNI_ABORT_TIMEOUT` | Float seconds; default `2` | Engine abort wait for Videos DELETE and `generate()` cancel/error cleanup; read when `async_omni` imports | Environment-only setting. A non-float raises `ValueError` during import. | Experimental |
| `VLLM_OMNI_ASYNC_OUTPUT_TIMEOUT` | Float seconds; default `600` | Diffusion engine async-output wait in `step_streaming`; resolved per call on the request path, not at import | Environment-only setting. A non-float or `<=0` value warns once and uses the default. | Experimental |
| `VLLM_OMNI_EVENT_DRIVEN_ORCH` | `1`, `true`, `yes` or `on` enables; default `0` (off) | Orchestration loop and the serving-side final-output drain; read once when the `Orchestrator` is constructed | Environment-only setting. Values are stripped and case-normalized; any unrecognized value leaves the legacy poll loop selected. | Experimental |
| `VLLM_OMNI_EVENT_DRIVEN_ORCH` | `1`, `true`, `yes` or `on` enables; defaults to on for Qwen3-TTS and off for other pipelines | Pipeline default computed from `pipeline_config.model_type` at engine initialization; env override resolved at `Orchestrator` construction and separately when the serving-side final-output drain starts | An explicit env value wins; otherwise both consumers use the engine's pipeline default. Values are stripped and case-normalized; any unrecognized value selects the legacy poll loop. Set before server startup. | Experimental |
| `VLLM_OMNI_INPUT_WAIT_TIMEOUT_S` | Float seconds; default `600`; `<=0` disables | Full-payload input coordinator, not async-chunk transfer; read when the scheduler module imports in each worker | Environment-only setting. A non-float logs a warning and uses `600`. | Stable operational control |
| `VLLM_OMNI_ORCH_MONITOR_PATH` | Filesystem path; default `<current-working-directory>/vllm_omni_orch_monitor_<timestamp>.json` | Orchestrator monitor enabled by `--enable-orch-monitor`; read when the monitor is created | Environment-only path override. Parent directories are created; write errors are logged. | Diagnostic |
| `VLLM_OMNI_VIDEO_SYNC_TIMEOUT` | Float seconds; default `600` | Synchronous Videos API; read when the API server module imports | Environment-only setting. A non-float raises `ValueError` during import. | Experimental |
+11 -5
View File
@@ -293,19 +293,25 @@ vllm serve Qwen/Qwen2.5-Omni-7B --omni --port 8091 --deploy-config /path/to/depl
## Qwen3-TTS with Model Runner V2
Qwen3-TTS can opt into the native CUDA Model Runner V2 pipeline on vLLM
0.29.0. Select one of these deployment profiles:
Qwen3-TTS runs the native CUDA Model Runner V2 pipeline by default on vLLM
0.29.0. MRV2 is an experimental feature for this model: the bundled default
profile selects it on CUDA only, and its scheduler and delivery paths are
still being qualified. Set `model_runner: v1` in a copy of the deploy config
to opt out; the `platforms:` sections of `qwen3_tts.yaml` keep V1 on NPU, XPU,
ROCm and MUSA. Select one of these deployment profiles:
| Profile | Runner | Code2Wav graph batches | Intended use |
| --- | --- | --- | --- |
| `qwen3_tts.yaml` | V1 | Existing defaults | Existing deployment / regression control |
| `qwen3_tts_mrv2.yaml` | V2 | B1 | Native runner validation |
| `qwen3_tts.yaml` | V2 (default) | Existing defaults | Shipped default; experimental |
| `qwen3_tts_mrv2.yaml` | V2 | B1 | Explicit MRV2 profile (same runner selection as the default) |
| `qwen3_tts_high_concurrency_mrv2.yaml` | V2 | B1, B2 | Opt-in throughput tuning |
| `qwen3_tts_high_concurrency_mrv2_b4.yaml` | V2 | B1, B2, B3, B4 | Experimental throughput / buffered playback |
| `qwen3_tts_high_concurrency.yaml` | V1 | Existing defaults | V1 high-concurrency control |
```bash
# MRV2 is the default; pass a copy with `model_runner: v1` to force V1.
vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-Base --omni \
--deploy-config vllm_omni/deploy/qwen3_tts_mrv2.yaml
--deploy-config /path/to/qwen3_tts_v1.yaml
```
The V2 profiles bound Talker prefill to 512 tokens per step and select the
+2 -2
View File
@@ -177,10 +177,10 @@ Failed jobs: 1/2
cd tests
# Stability: Qwen3-Omni
pytest -s -v dfx/stability/scripts/test_stability_qwen3_omni.py -m slow
pytest -s -v dfx/stability/scripts/run_stability_qwen3_omni.py -m slow
# Stability: Wan2.2 (v1/videos diffusion benchmark loop)
pytest -s -v dfx/stability/scripts/test_stability_wan22.py -m slow
pytest -s -v dfx/stability/scripts/run_stability_wan22.py -m slow
# Reliability: Qwen3-Omni (H100 × 2)
pytest -s -v dfx/reliability/test_reliability_qwen3_omni.py -m slow
+2 -2
View File
@@ -555,7 +555,7 @@ L5 level testing focuses on the performance of model services under ***long-runn
- ***Trigger Timing***: **`Weekly`** (weekly) or **`Days before Release`** (several days before a major release). Due to long execution times, the frequency is lower.
- ***Run Command***:
- ***Stability***: `pytest -s -v tests/dfx/stability/scripts/test_stability_qwen3_omni.py` or `pytest -s -v tests/dfx/stability/scripts/test_stability_wan22.py` (or add `test_stability_<model>.py` alongside a matching JSON config)
- ***Stability***: `pytest -s -v tests/dfx/stability/scripts/run_stability_qwen3_omni.py` or `pytest -s -v tests/dfx/stability/scripts/run_stability_wan22.py` (or add `run_stability_<model>.py` alongside a matching JSON config)
- ***Reliability***: `pytest -s -v tests/dfx/reliability/test_reliability_<model>.py -m slow` (current suites: `qwen3_omni`, `wan22`, `hunyuan_image`).
- ***Script Example***:
@@ -564,7 +564,7 @@ L5 level testing focuses on the performance of model services under ***long-runn
##### Stability
When you want to add L5-level stability test cases, add or extend the appropriate JSON file under `tests/dfx/stability/tests/` (for example `test_qwen3_omni.json` for Omni bench traffic, or `test_wan22.json` for diffusion `/v1/videos` workloads). Pair the JSON with `tests/dfx/stability/scripts/test_stability_<model>.py`. The following illustrates the Qwen3-Omni shape:
When you want to add L5-level stability test cases, add or extend the appropriate JSON file under `tests/dfx/stability/tests/` (for example `test_qwen3_omni.json` for Omni bench traffic, or `test_wan22.json` for diffusion `/v1/videos` workloads). Pair the JSON with `tests/dfx/stability/scripts/run_stability_<model>.py`. The following illustrates the Qwen3-Omni shape:
```json
{
+55 -3
View File
@@ -22,7 +22,7 @@ profiler_config:
Supported fields:
| Field | Description |
|---|---|
| --- | --- |
| `profiler` | Profiler backend. Supported values: `torch`, `cuda`. Use `torch` for `trace.json`, Excel operator tables, and optional memory snapshots. Use `cuda` for Nsight Systems only. |
| `torch_profiler_dir` | Output directory for torch-profiler artifacts. Required when `profiler: torch`. |
| `torch_profiler_use_gzip` | Compress `trace_rank*.json` into `trace_rank*.json.gz`. |
@@ -287,6 +287,52 @@ Recommended viewers:
For upstream background on the underlying vLLM profiling infrastructure, see the [vLLM profiling guide](https://docs.vllm.ai/en/stable/contributing/profiling/).
## 6. Profiling Omni AR/TTS stages
For Omni AR and TTS stages, put `profiler_config` under each stage that should
be captured. The serving API then controls a narrow window with
`POST /start_profile` and `POST /stop_profile`; model initialization and
warmup stay outside the trace.
Example stage override for a host-stack torch trace:
```bash
PROFILE='{"profiler":"torch","torch_profiler_dir":"/tmp/qwen3_tts_profile",'\
'"torch_profiler_with_stack":true,"torch_profiler_record_shapes":false,'\
'"torch_profiler_use_gzip":true,"torch_profiler_dump_cuda_time_total":true}'
vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice --omni \
--deploy-config vllm_omni/deploy/qwen3_tts_high_concurrency.yaml \
--stage-overrides "{\"0\":{\"profiler_config\":$PROFILE},\"1\":{\"profiler_config\":$PROFILE}}"
```
After one or more warmup requests, capture one short measured request:
```bash
curl -sS -X POST http://127.0.0.1:8000/start_profile \
-H 'Content-Type: application/json' -d '{"stages":[0,1]}'
# send one fixed /v1/audio/speech or /v1/chat/completions request here
curl -sS -X POST http://127.0.0.1:8000/stop_profile \
-H 'Content-Type: application/json' -d '{"stages":[0,1]}'
```
For an Nsight Systems CUDA timeline, use the same stage override with
`{"profiler":"cuda"}` and wrap the server:
```bash
nsys profile --trace=cuda,nvtx,osrt --sample=process-tree \
--cpuctxsw=process-tree --trace-fork-before-exec=true \
--capture-range=cudaProfilerApi --capture-range-end=stop-shutdown \
--cuda-graph-trace=node -o /tmp/qwen3_tts_nsys \
vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice --omni ...
```
For Omni AR workers, `profiler="cuda"` requires the native CUDA profiler
wrapper and uses `cudaProfilerStart/Stop`; the report is only valid if the
`/start_profile` and `/stop_profile` calls both succeed. Use `nsys stats` for
CUDA API, GPU kernel, OS runtime, and NVTX summaries. Treat profiler latency as
diagnostic only; use a separate no-profiler run for performance numbers.
## 7. Orchestrator Monitor
> **Warning:** Diagnostic only. Adds lightweight counters on the orchestrator poll loop and per-replica queue depth. Disable in production unless you are actively debugging orchestrator saturation.
@@ -314,9 +360,11 @@ If unset, the monitor writes `./vllm_omni_orch_monitor_<MMDDHHMM>.json` at shutd
Each 1-second window records:
| Series | Description |
|---|---|
| --- | --- |
| `windows.duration_s` | Wall time covered by the window |
| `windows.loop_idle` / `windows.loop_active` | Orchestrator poll-loop iterations with no work vs. active forwarding |
| `windows.dispatch_queue_size` | Ready-queue size sampled at each monitor window boundary |
| `windows.dispatch_queue_high_water` | Maximum Event Driven ready-queue size observed in each window |
| `replicas.<stage,replica>.outputs_queue_size` | MP client `outputs_queue` backlog for that replica |
| `replicas.<stage,replica>.inflight` | Requests currently bound/routed to the replica |
@@ -331,12 +379,16 @@ orchestrator wakes only once per 0.5 s reconcile timeout, while a busy one still
records one iteration per routed output. Window counts and the `loop_active_pct`
summary are therefore not comparable across the two modes.
For Event Driven runs, use `dispatch_queue_high_water` rather than only the
window-boundary `dispatch_queue_size`: a serial dispatcher can drain a burst
before the window closes, hiding a transient ready-queue backlog in the latter.
### Relationship to other diagnostics
This monitor is intentionally separate from the existing profiling tools:
| Tool | Scope | Output |
|---|---|---|
| --- | --- | --- |
| `--enable-diffusion-pipeline-profiler` | Diffusion stage functions (`vae.decode`, `diffuse`, …) | Per-request `stage_durations` / logs |
| `--enable-ar-profiler` | AR stage generation time | Per-request `stage_durations` |
| `profiler_config` (`torch` / `cuda`) | GPU/CPU kernels inside a stage worker | `trace.json`, Nsight, operator tables |
+1 -1
View File
@@ -79,7 +79,7 @@ the boundary affected by the in-flight stage client/process refactor in
responsibilities proposed only by that PR are not current contracts.
The orchestration loop also has an opt-in event-driven mode
(`VLLM_OMNI_EVENT_DRIVEN_ORCH=1`, default off) proposed in
(`VLLM_OMNI_EVENT_DRIVEN_ORCH=1`, default off except for Qwen3-TTS) proposed in
[#5221](https://github.com/vllm-project/vllm-omni/pull/5221). It changes poll
cadence only: the routing, ordering, and terminal-state contracts below hold
identically on both loops.
+63 -17
View File
@@ -9,9 +9,11 @@ This feature is inherited from [vLLM’s Sleep Mode](https://blog.vllm.ai/2025/1
## 1. Feature Documentation
### Overview
Omni Sleep Mode provides a mechanism to "sleep" specific model stages. When a stage enters sleep, its physical VRAM is reclaimed by the system, while the process state is preserved for rapid "wake-up" without full re-initialization.
### Sleep Levels
We support two levels of hibernation to balance recovery speed and memory efficiency:
| Level | Name | Mechanism | Recovery Speed | Memory Freed |
@@ -29,15 +31,67 @@ Omni Sleep Mode is optimized for high-performance computing backends:
* **Huawei NPU**: Supported via Ascend memory scavenging.
### Hardware Requirements
* **Memory Considerations**: System RAM must be sufficient to hold offloaded weights during sleep.
* **TP Support**: Tensor Parallel groups synchronize sleep/wake transitions across all workers.
---
### Quiescing a busy diffusion stage: `pause_generation(mode="keep")`
`sleep()` on a diffusion stage is a worker-level RPC: it offloads memory but does not stop the
stage's scheduler, so a stage that still has queued requests must be made quiet first. Aborting
discards the batch that is running. `pause_generation(mode="keep")` keeps it instead:
* The batch that is already running finishes on the weights it started with and its outputs are
delivered normally.
* Requests that have not started stay queued and run after `resume_generation()`, on whatever
weights are loaded by then. A request never continues denoising across a weight change.
* The call returns only after every replica has acknowledged: the batch that was running has
finished executing, its asynchronous output copies are done, and the device is synchronized; its
outputs are then delivered on the normal path. On a non-preemptible backend that acknowledgement
arrives at the next batch boundary, so expect it to take as long as the remaining batch.
* Control RPCs (`list_loras`, `sleep`, `wake_up`, `resume_generation`) keep working while paused.
Recommended order for a weight update on a diffusion stage:
```python
await engine.pause_generation(mode="keep", clear_cache=False)
await engine.sleep(level=1) # optional: free memory for the trainer
await engine.wake_up() # admission stays closed until resume
# install new weights here
await engine.resume_generation()
```
Support matrix and limits:
| Configuration | `mode="keep"` |
| :--- | :--- |
| Request-level execution (`step_execution=False`) with the in-process (`uni`) executor | Supported |
| Request-level execution with the multi-process (`mp`) executor, including asynchronous output | Supported; the acknowledgement waits for the output copies |
| Request-level execution with asynchronous KV prefetch | Supported; the acknowledgement joins any prefetch still in flight |
| Step-level execution (`step_execution=True` / streaming output) | Not supported: the call raises `NotImplementedError` |
| `mode="abort"` / `mode="wait"` / `wait_for_inflight_requests=True` on a diffusion stage | Unchanged: frontend admission is paused only |
Behavior notes:
* Callers that already passed `mode="keep"` for a diffusion stage used to get an immediate return
that only closed frontend admission. They now also stop the backend and wait for the ACK.
* A timeout or error from `pause_generation` means the pause did **not** complete; the scheduler
stays closed and the caller must retry the pause or call `resume_generation()` explicitly. Do not
sleep or mutate weights after a failed pause.
* `resume_generation(stage_ids=...)` that leaves another paused stage closed (an AR stage, or a
keep-paused diffusion stage) keeps frontend admission closed as well, so new requests cannot
queue on a stage that will not schedule them. Admission reopens once every stage paused by
`pause_generation` has been resumed.
* `clear_cache` has no diffusion-specific effect.
* Without a pause call nothing changes: no barrier runs and batching, abort and output ordering are
as before.
---
## 2. Usage Examples
### Python API Example
You can programmatically control the lifecycle of stages using the `AsyncOmni` engine.
```python
@@ -65,6 +119,7 @@ if __name__ == "__main__":
```
### server command Example
Start the server with sleep mode enabled:
The first method
@@ -92,9 +147,6 @@ python3 -m vllm_omni.entrypoints.openai.api_server \
```
### Test Scenarios & Commands
#### Scenario 1: LLM Engine Sleep
@@ -113,9 +165,8 @@ curl -X POST http://localhost:8000/v1/omni/sleep \
Tip: Open a new terminal and run rocm-smi or nvidia-smi or to observe the immediate drop in VRAM usage.
#### Scenario 2: Diffusion Sleep
Objective: Verify VRAM reclamation for Stage 1 (Diffusion).
Trigger sleep (Level 1 or Level 2) via client:
@@ -128,9 +179,8 @@ curl -X POST http://localhost:8000/v1/omni/sleep \
```
#### Scenario 3: Multi-Stage Coordinated Stress Test
Objective: Test concurrent sleep and rapid wake-up across multiple stages.
Concurrent Sleep (Stage 0 & 1):
@@ -143,7 +193,6 @@ curl -X POST http://localhost:8000/v1/omni/sleep \
```
Rapid Wake-up:
```
@@ -154,8 +203,8 @@ curl -X POST http://localhost:8000/v1/omni/wakeup \
```
#### Scenario 4: Full Lifecycle Memory Audit & Functional Integrity
Objective: Audit the complete flow from Sleep to Wake-up followed by an Inference validation.
Check Initial State: Observe baseline VRAM usage.
@@ -196,20 +245,16 @@ curl -X POST http://localhost:8000/v1/images/generations \
```
## 3. API Reference
### Methods
| Method | Arguments | Return Type | Description |
| :--- | :--- | :--- | :--- |
| **sleep** | `stage_ids: List[int], level: int` | `List[OmniACK]` | Triggers hibernation for specified stages. |
| **wake_up** | `stage_ids: List[int]` | `List[OmniACK]` | Reloads weights and re-maps memory. |
| **pause_generation** | `mode: str, stage_ids: List[int]` | `None` | Stops admission; with `mode="keep"` also stops diffusion schedulers and returns after their ACK. |
| **resume_generation** | `stage_ids: List[int]` | `None` | Reopens paused schedulers, then admission. |
### OmniACK Dataclass Fields
@@ -233,7 +278,8 @@ The metadata field is a dynamic dictionary containing hardware-specific telemetr
}
```
#### Core Utility:
#### Core Utility
**VRAM Reclamation Audit (total_freed_gib)**: Converts raw freed_bytes into human-readable GiB. It serves as the primary metric to verify that Level 2 sleep has successfully purged model weights from VRAM.
**Residual & Fragmentation Monitoring (rank_residual_gib)**: Reports the remaining VRAM footprint after memory de-mapping. A low residual value (e.g., 2.07 GiB) confirms a successful "clean" state, ensuring the device is ready for high-memory co-located tasks like training or diffusion pipelines.
+12 -9
View File
@@ -957,22 +957,24 @@ Use `/v1/audio/voices` to list available voices for the loaded model.
## Orchestration Loop (experimental)
Multi-stage omni deployments route stage outputs through a single orchestrator
loop. By default that loop polls every stage replica on a 1 ms cadence. An
opt-in event-driven mode replaces the poll with one reader task per live stage
loop. The legacy loop polls every stage replica on a 1 ms cadence. The
event-driven mode replaces the poll with one reader task per live stage
replica awaiting its client directly, and switches the serving-side
final-output drain to a condition-variable wakeup at the same time.
final-output drain to a condition-variable wakeup at the same time. Qwen3-TTS
uses this mode by default; other pipelines keep the legacy poll unless enabled.
**Configuration (environment variables):**
| Variable | Default | Description |
| --- | --- | --- |
| `VLLM_OMNI_EVENT_DRIVEN_ORCH` | `0` (off) | Switches the orchestration loop and the final-output drain from the legacy 1 ms poll to event-driven wakeups. Enabled by `1`, `true`, `yes`, or `on`, matched case-insensitively after surrounding whitespace is stripped; any other value leaves it off. |
| `VLLM_OMNI_EVENT_DRIVEN_ORCH` | On for Qwen3-TTS; off for other pipelines | Switches the orchestration loop and the final-output drain from the legacy 1 ms poll to event-driven wakeups. An explicit value wins; otherwise the pipeline default computed at engine initialization is used. The override is resolved at orchestrator construction and separately when the final-output drain starts. `1`, `true`, `yes`, or `on` enables it, ignoring case and surrounding whitespace; other values select the legacy poll loop. |
Set it on the process that runs the orchestrator (stage 0 of an omni
deployment) before starting the server:
Set it on the process that runs the orchestrator (stage 0 of an omni deployment)
before starting the server when overriding the pipeline default. For example,
explicitly disable event-driven orchestration for Qwen3-TTS:
```bash
export VLLM_OMNI_EVENT_DRIVEN_ORCH=1
VLLM_OMNI_EVENT_DRIVEN_ORCH=0 \
vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-Base \
--omni \
--port 8091
@@ -982,8 +984,9 @@ The server logs the selected loop mode and its reader/poller counts once at
startup, so you can confirm which loop is live.
Routing, output ordering, and terminal-state behavior are identical on both
loops; only the poll cadence changes. Leaving the variable unset keeps the
legacy poll loop, which is the supported default.
loops; only the poll cadence changes. Leaving the variable unset selects the
pipeline default: event-driven for Qwen3-TTS, and legacy polling for other
pipelines, including Qwen3-Omni.
**Known limitations:**
@@ -6,29 +6,29 @@
ASR -> AURA -> Qwen3-TTS Talker -> Code2Wav
```
Qwen3-TTS remains two engine stages so the pipeline reuses the existing native
Talker and Code2Wav implementation.
**Supported online path:** Realtime duplex at `/v1/realtime?duplex=1`.
The default deploy profile (`vllm_omni/deploy/aura_omni.yaml`) sets
`session_mode: duplex`. The `aura_omni` pipeline declares `duplex_plugin`, and
`DuplexOmniEngine` requires that mode (same pattern as MiniCPM-o duplex
deploys). Turn-based `chat/completions` is not a supported AURA online mode
against this profile.
`async_chunk` defaults to `false` in that file (safer if a turn-based
`OmniOrchestrator` ever loads it). Duplex Realtime Stage2→3 still needs
async chunks — use the smoke deploy below, or set `async_chunk: true`
yourself.
```bash
vllm serve aurateam/AURA \
--omni \
--deploy-config vllm_omni/deploy/aura_omni.yaml \
--served-model-name aurateam/AURA \
--trust-remote-code
bash examples/online_serving/aura_omni/run_duplex_smoke_serve.sh
python examples/online_serving/aura_omni/smoke_duplex_realtime_client.py
```
Configure local checkpoints by editing per-stage `model` values in
`vllm_omni/deploy/aura_omni.yaml`. The deploy file sets
`pipeline: aura_omni`, so the four-stage topology is used even if the
command-line `--model` points at one of the component checkpoints.
Configure local checkpoints by editing per-stage `model` values in the deploy
or smoke YAML. The file sets `pipeline: aura_omni`, so the four-stage topology
is used even if the command-line `--model` points at one component checkpoint.
Send requests with `"model": "aurateam/AURA"`. The ASR, AURA, and Qwen3-TTS
checkpoint paths are internal stage models from the deploy YAML, not the
OpenAI-facing served model name.
The AURA stage can emit `<|silent|>`. Silent outputs are treated as a gate:
they produce no Qwen3-TTS Talker input, so no audio is synthesized for that
turn.
Silent Stage1 outputs (`<|silent|>` / id `151669`) skip TTS for that turn.
## GPU Utilization Recommendation
@@ -40,26 +40,21 @@ VRAM each stage can reserve. Start with this split for a single GPU:
- Stage 2 (Qwen3-TTS Talker): `0.20`
- Stage 3 (Qwen3-TTS Code2Wav): `0.20`
## TTS Modes
## TTS modes (stage extras)
`aura_omni` can pass AURA text to Qwen3-TTS in two task modes:
When the duplex session supplies TTS extras, AURA text can feed Qwen3-TTS as:
- `Base`: voice clone from `tts_ref_audio` with ICL enabled in the AURA
pipeline. Provide both `tts_ref_audio` and `tts_ref_text`. Set
`tts_x_vector_only_mode=true` to disable ICL and use speaker embedding only.
- `CustomVoice`: predefined speaker mode. Use a Qwen3-TTS CustomVoice
checkpoint for stages 2 and 3 in `aura_omni.yaml`, then pass
`tts_task_type=CustomVoice` and `tts_speaker`.
- `Base`: voice clone from `tts_ref_audio` (optional x-vector-only mode)
- `CustomVoice`: predefined speaker (`tts_speaker`) with a CustomVoice checkpoint
on stages 2 and 3
By default, AURA responses are passed to Qwen3-TTS as text. Set
`additional_information.tts_pass_token_ids=true` to pass AURA-generated
assistant token ids directly instead. Even when token passthrough is disabled,
the stage processor uses AURA token ids when available to estimate the Talker
prefill length, so scheduling does not rely on raw character length.
Optional `tts_pass_token_ids` passes AURA assistant token ids into Talker
instead of detokenized text.
The example client exposes this as:
## Old chat / Gradio / curl scripts (unsupported)
```bash
python examples/online_serving/aura_omni/openai_chat_completion_client.py \
--tts-pass-token-ids
```
`examples/online_serving/aura_omni/` still contains chat-completions, curl, and
Gradio helpers from an older turn-based path. They do **not** work against the
shipped duplex profile. That path was an incomplete stand-in before streaming
I/O and session history were available; it is not a supported AURA mode. Use
the Realtime duplex smoke client for online verification.
@@ -247,12 +247,18 @@ curl http://localhost:8091/v1/chat/completions \
-d @payload.json
```
BAGEL img2img derives its generated image size from the input image. It
preserves the input aspect ratio, aligns dimensions to the latent stride, and
applies the checkpoint size limit; explicit `height` and `width` values do not
override this model-specific resize policy. In step mode, the effective size is
resolved before scheduler admission so requests with different resulting
shapes are not combined.
When no output size is requested (`size=auto` on `/v1/images/edits`, or no
`height`/`width` in a chat request), BAGEL img2img derives the canvas from the
input image: it preserves the input aspect ratio, aligns dimensions to the
latent stride (16 px), and applies the checkpoint size limit. An explicit
`size`, `height`, or `width` is honored instead, in both the single-stage and
the two-stage deployment; the input image is still resized to a stride-aligned
size for the VAE/ViT prefill. The latent grid needs multiples of 16, so a
requested side that is not one is floored to the next multiple (1000x700
generates 992x688) and the server logs a warning; the response reports the
generated size. A side above the checkpoint limit is rejected with HTTP 400.
In step mode, the effective size is resolved before scheduler admission so
requests with different resulting shapes are not combined.
### Image to Text (img2text)
@@ -318,8 +324,8 @@ curl http://localhost:8091/v1/chat/completions \
| `--server` / `-s` | `http://localhost:8091` | Server URL |
| `--image-url` / `-i` | `None` | Input image URL or local path (img2img/img2text) |
| `--modality` / `-m` | `text2img` | `text2img`, `img2img`, `img2text`, `text2text` |
| `--height` | `512` | Text-to-image output height; BAGEL img2img derives its size from the input image |
| `--width` | `512` | Text-to-image output width; BAGEL img2img derives its size from the input image |
| `--height` | `512` | Output height, a multiple of 16; omit both `--height` and `--width` to let img2img derive the size from the input image |
| `--width` | `512` | Output width, a multiple of 16; omit both `--height` and `--width` to let img2img derive the size from the input image |
| `--steps` | `25` | Number of inference steps |
| `--seed` | `42` | Random seed |
| `--negative` | `None` | Negative prompt for CFG |