anodyine/vllm-omni
1
0 Fork 0
Code Issues 4 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

[BugFix] Fix leftovers of the legacy OpenAI realtime API event names (#7426)

Browse Source 
Signed-off-by: Nick Cao <ncao@redhat.com>
Co-authored-by: Codex <noreply@openai.com>
This commit is contained in:
Nick CaoandCodex authored and GitHub committed 2026-09-11 22:28:55 +08:00
1 parent 78e1a0ae5c
commit d3493384cf
10 files changed
+103 -48
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

examples/online_serving/qwen3_omni/README.md
+22 -14
View File
@@ -1,5 +1,7 @@
# Qwen3-Omni: Online serving
<!-- markdownlint-disable MD028 -->
## 🛠️ Installation
Please refer to [README.md](../../../README.md)
@@ -19,6 +21,7 @@ Additionally, NPU, ROCm, and XPU per-platform configuration deltas are determini
`platforms`: section of the corresponding YAML.
To explicitly utilize a custom deployment YAML, mandate the configuration path accordingly:
```bash
vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091 \
--deploy-config /path/to/your_deploy_config.yaml
@@ -53,7 +56,7 @@ Use the stage-based CLI when you want to run one stage per process.
The example below pins Stage 0 to GPU 0 and Stage 1/2 to GPU 1 via
`CUDA_VISIBLE_DEVICES`.
**1. Stage 0 (Thinker + API server)**
#### 1. Stage 0 (Thinker + API server)
```bash
CUDA_VISIBLE_DEVICES=0 vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \
@@ -63,7 +66,7 @@ CUDA_VISIBLE_DEVICES=0 vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \
--omni-master-port 26000
```
**2. Stage 1 (Talker)**
#### 2. Stage 1 (Talker)
```bash
CUDA_VISIBLE_DEVICES=1 vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \
@@ -73,7 +76,7 @@ CUDA_VISIBLE_DEVICES=1 vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \
--omni-master-port 26000
```
**3. Stage 2 (Code2Wav)**
#### 3. Stage 2 (Code2Wav)
```bash
CUDA_VISIBLE_DEVICES=1 vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \
@@ -252,11 +255,12 @@ pattern works for Qwen2.5-Omni — replace `base_config:` with the path to
### Send Multi-modal Request
Get into the example folder
```bash
cd examples/online_serving/qwen3_omni
```
#### Send request via python
#### Send request via python
```bash
python examples/online_serving/openai_chat_completion_client_for_multimodal_generation.py --model Qwen/Qwen3-Omni-30B-A3B-Instruct --query-type use_image --port 8091 --host "localhost"
@@ -264,7 +268,7 @@ python examples/online_serving/openai_chat_completion_client_for_multimodal_gene
#### Realtime WebSocket client (`openai_realtime_client.py`)
[`openai_realtime_client.py`](./openai_realtime_client.py) connects to **`ws://<host>:<port>/v1/realtime`**, streams a local WAV as **PCM16 mono @ 16 kHz** in fixed-size chunks, and receives **`response.audio.*`**, **`response.output_text.*`**, and transcript events. By default it uses explicit `input_audio_buffer.commit`; pass **`--server-vad`** to let the server detect speech endpoints and commit the turn automatically. The client concatenates audio deltas and writes **`--output-wav`** (model output is typically **24 kHz**). Optional **`--delta-dump-dir`** saves each delta as `delta_000001.wav`, … for debugging.
[`openai_realtime_client.py`](./openai_realtime_client.py) connects to **`ws://<host>:<port>/v1/realtime`**, streams a local WAV as **PCM16 mono @ 16 kHz** in fixed-size chunks, and receives **`response.output_audio.*`**, **`response.output_text.*`**, and transcript events. By default it uses explicit `input_audio_buffer.commit`; pass **`--server-vad`** to let the server detect speech endpoints and commit the turn automatically. The client concatenates audio deltas and writes **`--output-wav`** (model output is typically **24 kHz**). Optional **`--delta-dump-dir`** saves each delta as `delta_000001.wav`, … for debugging.
Streaming input works well for translation-style use cases; if the Thinker runs while input is still incomplete, consider limiting **`max_tokens`** in your session / server defaults to avoid over-generation.
@@ -289,7 +293,7 @@ python openai_realtime_client.py \
**Arguments:**
| Flag | Default | Description |
|------|---------|-------------|
| ------ | --------- | ------------- |
| `--url` | `ws://localhost:8091/v1/realtime` | Full WebSocket URL including path |
| `--model` | `Qwen/Qwen3-Omni-30B-A3B-Instruct` | Must match the served model (sent in `session.update`) |
| `--input-wav` | *(required)* | Input WAV: mono, 16-bit PCM, **16 kHz** |
@@ -297,7 +301,7 @@ python openai_realtime_client.py \
| `--output-text` | *(optional)* | If set, write final transcription text to this path |
| `--chunk-ms` | `200` | Size of each uploaded audio chunk (milliseconds of audio) |
| `--send-delay-ms` | `0` | Delay between chunk sends (simulate realtime upload) |
| `--delta-dump-dir` | *(optional)* | Directory to write per-`response.audio.delta` WAV files |
| `--delta-dump-dir` | *(optional)* | Directory to write per-`response.output_audio.delta` WAV files |
| `--server-vad` | disabled | Enable server-side Silero VAD instead of sending explicit commits |
| `--num-requests` | `1` | Number of sequential sessions (see `--concurrency`) |
| `--concurrency` | `1` | Max concurrent WebSocket sessions when `--num-requests` > 1 |
@@ -334,7 +338,6 @@ The Python client supports the following command-line arguments:
- `--prompt` (or `-p`): Custom text prompt/question. If not provided, uses default prompt for the selected query type. Example: `--prompt "What are the main activities shown in this video?"`
- `--speaker`: TTS speaker/voice for audio output when requesting audio (e.g. `ethan`, `chelsie`, `aiden`). Omit to use the model default. Example: `--speaker "chelsie"`
For example, to use a local video file with custom prompt:
```bash
@@ -345,22 +348,22 @@ python examples/online_serving/openai_chat_completion_client_for_multimodal_gene
--prompt "What are the main activities shown in this video?"
```
#### Send request via curl
#### Send request via curl
```bash
bash run_curl_multimodal_generation.sh use_image
```
### FAQ
## Modality control
You can control output modalities to specify which types of output the model should generate. This is useful when you only need text output and want to skip audio generation stages for better performance.
### Supported modalities
| Modalities | Output |
|------------|--------|
| ------------ | -------- |
| `["text"]` | Text only |
| `["audio"]` | Audio only |
| `["text", "audio"]` | Text + Audio |
@@ -496,7 +499,9 @@ print(response.choices[0].message.audio)
Supported speaker names depend on the model (e.g. `Ethan`, `Chelsie`, `Aiden`). Omit `speaker` to use the default.
## Streaming Output
If you want to enable streaming output, please set the argument as below. The final output will be obtained just after generated by corresponding stage. We support both text streaming output and audio streaming output. Other modalities can output normally.
```bash
python examples/online_serving/openai_chat_completion_client_for_multimodal_generation.py \
--query-type use_image \
@@ -521,12 +526,14 @@ The convenience script launches both the vLLM server and Gradio demo together:
```
This script will:
1. Start the vLLM server in the background
2. Wait for the server to be ready
3. Launch the Gradio demo
4. Handle cleanup when you press Ctrl+C
The script supports the following arguments:
- `--model`: Model name/path (default: Qwen/Qwen3-Omni-30B-A3B-Instruct)
- `--server-port`: Port for vLLM server (default: 8091)
- `--gradio-port`: Port for Gradio demo (default: 7861)
@@ -537,18 +544,19 @@ The script supports the following arguments:
#### Option 2: Manual Launch (Two-Step Process)
**Step 1: Launch the vLLM API server**
##### Step 1: Launch the vLLM API server
```bash
vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091
```
If you have a custom deploy config file:
```bash
vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091 --deploy-config /path/to/deploy_config_file
```
**Step 2: Run the Gradio demo**
##### Step 2: Run the Gradio demo
In a separate terminal:
@@ -561,7 +569,7 @@ Then open `http://localhost:7861/` on your local browser to interact with the we
The gradio script supports the following arguments:
- `--model`: Model name/path (should match the server model)
- `--api-base`: Base URL for the vLLM API server (default: http://localhost:8091/v1)
- `--api-base`: Base URL for the vLLM API server (default: <http://localhost:8091/v1>)
- `--ip`: Host/IP for Gradio server (default: 127.0.0.1)
- `--port`: Port for Gradio server (default: 7861)
- `--share`: Share the Gradio demo publicly (creates a public link)
examples/online_serving/qwen3_omni/openai_realtime_client.py
+7 -7
View File
@@ -6,14 +6,14 @@
This client:
1) Reads a local WAV file (must be mono, 16-bit PCM, 16kHz),
2) Streams PCM16 chunks to /v1/realtime with OpenAI-style events,
3) Receives response.audio.*, response.output_text.*, and transcript events,
3) Receives response.output_audio.*, response.output_text.*, and transcript events,
4) Saves synthesized audio to an output WAV file and optional text file.
By default each ``response.audio.delta`` is treated as an **incremental PCM**
By default each ``response.output_audio.delta`` is treated as an **incremental PCM**
chunk and all chunks are concatenated into the final ``--output-wav``.
Optional debugging: pass ``--delta-dump-dir DIR`` to write every
``response.audio.delta`` payload as ``delta_000001.wav``, ``delta_000002.wav``, …
``response.output_audio.delta`` payload as ``delta_000001.wav``, ``delta_000002.wav``, …
Usage:
python openai_realtime_client.py \
@@ -187,7 +187,7 @@ async def run_client(
if event_type == "session.created":
continue
if event_type == "response.audio.delta":
if event_type == "response.output_audio.delta":
sr = event.get("sample_rate_hz")
if isinstance(sr, int) and sr > 0:
output_sample_rate = sr
@@ -223,11 +223,11 @@ async def run_client(
print(f"{log_prefix}text usage: {usage}")
continue
if event_type == "response.audio_transcript.done":
if event_type == "response.output_audio_transcript.done":
final_audio_transcript = event.get("transcript", "")
continue
if event_type == "response.audio.done":
if event_type == "response.output_audio.done":
if not server_vad:
break
continue
@@ -339,7 +339,7 @@ def main() -> None:
"--delta-dump-dir",
type=Path,
default=None,
help="If set, each response.audio.delta is saved as delta_NNNNNN.wav under this directory",
help="If set, each response.output_audio.delta is saved as delta_NNNNNN.wav under this directory",
)
parser.add_argument(
"--server-vad",