vllm-omni/docs/user_guide/examples/online_serving/qwen3_omni.md

Qwen3-Omni: Online serving

Source https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/qwen3_omni.

🛠️ Installation

Please refer to README.md

Run examples (Qwen3-Omni)

Launch the Server

vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091

For a local deployment with multiple API frontend processes sharing one set of stage engines, add --api-server-count:

vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \
    --api-server-count 2 --port 8091

This mode supports local EngineCore stages with one local process group per replica, including stages configured with multiple num_replicas. Headless, remote, diffusion, intra-replica data-parallel, and Ray stage deployments remain single-frontend.

The default deployment configuration situated at vllm_omni/deploy/qwen3_omni_moe.yaml is resolved and loaded automatically via the model registry, obviating the necessity for the --deploy-config flag in standard deployment topologies. The bundled Qwen3-Omni setup explicitly sets moe_backend: triton for the Thinker and Talker stages. This keeps both stages on vLLM's Triton unquantized MoE path and avoids the unstable FlashInfer CUTLASS unquantized MoE path on some GPUs. Asynchronous chunk streaming is enabled by default within the bundled configuration.

For Thinker pipeline parallelism, configure pipeline_parallel_size and devices on the Thinker stage. Keep Talker and Code2Wav at pipeline_parallel_size: 1. The default asynchronous scheduling and audio chunk streaming can remain enabled.

To explicitly utilize a custom deployment YAML, specify the configuration path:

vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091 \
    --deploy-config /path/to/deploy_config_file

To serve thinker-only (text output, no talker / code2wav loaded) with Instruct weights, pass the bundled thinker-only deploy YAML. The pipeline: key qwen3_omni_moe_thinker_only overrides the HF enable_audio_output resolver:

vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091 \
    --deploy-config vllm_omni/deploy/qwen3_omni_moe_thinking.yaml

Captioner / Thinking checkpoints (enable_audio_output=false) still auto-select the same single-stage pipeline without --deploy-config.

Launch individual stages (stage-based CLI)

Adopt the stage-based CLI architecture to independently instantiate execution processes per functional stage. The example below pins Stage 0 to GPU 0 and Stage 1/2 to GPU 1 via CUDA_VISIBLE_DEVICES.

1. Stage 0 (Thinker + API server)

CUDA_VISIBLE_DEVICES=0 vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \
    --port 8091 \
    --stage-id 0 \
    --omni-master-address 127.0.0.1 \
    --omni-master-port 26000

2. Stage 1 (Talker)

CUDA_VISIBLE_DEVICES=1 vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \
    --stage-id 1 \
    --headless \
    --omni-master-address 127.0.0.1 \
    --omni-master-port 26000

3. Stage 2 (Code2Wav)

CUDA_VISIBLE_DEVICES=1 vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \
    --stage-id 2 \
    --headless \
    --omni-master-address 127.0.0.1 \
    --omni-master-port 26000

Add --deploy-config /path/to/deploy_config_file to every command if you want to override the bundled deploy YAML.

For the regular one-process launch, stage-specific CLI tuning is usually done with --stage-overrides, for example:

vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091 \
    --stage-overrides '{"1": {"gpu_memory_utilization": 0.5}}'

To experiment with another supported MoE backend, set moe_backend in the deployment YAML for both the Thinker and Talker stages. For the stage-based CLI, pass --moe-backend directly on each stage command.

For other stage settings, pass the option on that stage command. For example, to change its GPU memory utilization:

CUDA_VISIBLE_DEVICES=1 vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \
    --stage-id 1 \
    --headless \
    --gpu-memory-utilization 0.5 \
    --omni-master-address 127.0.0.1 \
    --omni-master-port 26000

Send Multi-modal Request

Get into the example folder

cd examples/online_serving/qwen3_omni

Send request via python

python openai_chat_completion_client_for_multimodal_generation.py --query-type use_image --port 8091 --host "localhost"

The Python client supports the following command-line arguments:

• --query-type (or -q): Query type (default: use_video). Options: text, use_audio, use_image, use_video
• --model (or -m): Model name/path (default: Qwen/Qwen3-Omni-30B-A3B-Instruct)
• --video-path (or -v): Path to local video file or URL. If not provided and query-type is use_video, uses default video URL. Supports local file paths (automatically encoded to base64) or HTTP/HTTPS URLs. Example: --video-path /path/to/video.mp4 or --video-path https://example.com/video.mp4
• --image-path (or -i): Path to local image file or URL. If not provided and query-type is use_image, uses default image URL. Supports local file paths (automatically encoded to base64) or HTTP/HTTPS URLs and common image formats: JPEG, PNG, GIF, WebP. Example: --image-path /path/to/image.jpg or --image-path https://example.com/image.png
• --audio-path (or -a): Path to local audio file or URL. If not provided and query-type is use_audio, uses default audio URL. Supports local file paths (automatically encoded to base64) or HTTP/HTTPS URLs and common audio formats: MP3, WAV, OGG, FLAC, M4A. Example: --audio-path /path/to/audio.wav or --audio-path https://example.com/audio.mp3
• --prompt (or -p): Custom text prompt/question. If not provided, uses default prompt for the selected query type. Example: --prompt "What are the main activities shown in this video?"

For example, to use a local video file with custom prompt:

python openai_chat_completion_client_for_multimodal_generation.py \
    --query-type use_video \
    --video-path /path/to/your/video.mp4 \
    --prompt "What are the main activities shown in this video?"

Send request via curl

bash run_curl_multimodal_generation.sh use_image

Modality control

You can control output modalities to specify which types of output the model should generate. This is useful when you only need text output and want to skip audio generation stages for better performance.

Supported modalities

Modalities	Output
["text"]	Text only
["audio"]	Audio only
["text", "audio"]	Text + Audio
Not specified	Text + Audio (default)

Using curl

Text only

curl http://localhost:8091/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3-Omni-30B-A3B-Instruct",
    "messages": [{"role": "user", "content": "Describe vLLM in brief."}],
    "modalities": ["text"]
  }'

Text + Audio

response=$(curl -s http://localhost:8091/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3-Omni-30B-A3B-Instruct",
    "messages": [{"role": "user", "content": "Describe vLLM in brief."}],
    "modalities": ["text", "audio"]
  }')

echo "$response" | jq -r '.choices[0].message.content'
echo "$response" | jq -r '.choices[1].message.audio.data' | base64 -d > output.wav

Using Python client

python openai_chat_completion_client_for_multimodal_generation.py \
    --query-type use_image \
    --modalities text

Using OpenAI Python SDK

Text only

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8091/v1", api_key="EMPTY")

response = client.chat.completions.create(
    model="Qwen/Qwen3-Omni-30B-A3B-Instruct",
    messages=[{"role": "user", "content": "Describe vLLM in brief."}],
    modalities=["text"]
)
print(response.choices[0].message.content)

Text + Audio

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8091/v1", api_key="EMPTY")

response = client.chat.completions.create(
    model="Qwen/Qwen3-Omni-30B-A3B-Instruct",
    messages=[{"role": "user", "content": "Describe vLLM in brief."}],
    modalities=["text", "audio"]
)
# Response contains two choices: one with text, one with audio
print(response.choices[0].message.content)  # Text response
print(response.choices[1].message.audio)    # Audio response

Streaming Output

If you want to enable streaming output, please set the argument as below. The final output will be obtained just after generated by corresponding stage. We support both text streaming output and audio streaming output. Other modalities can output normally.

python openai_chat_completion_client_for_multimodal_generation.py \
    --query-type use_image \
    --stream

Run Local Web UI Demo

!!! note "Gradio is an optional dependency" The Gradio demo requires the [demo] extras. Install them first:

```bash
pip install 'vllm-omni[demo]'
```

Or, if installing from source: `pip install -e '.[demo]'`

This Web UI demo allows users to interact with the model through a web browser.

Running Gradio Demo

The Gradio demo connects to a vLLM API server. You have two options:

Option 1: One-step Launch Script (Recommended)

The convenience script launches both the vLLM server and Gradio demo together:

./run_gradio_demo.sh --model Qwen/Qwen3-Omni-30B-A3B-Instruct --server-port 8091 --gradio-port 7861

This script will:

1. Start the vLLM server in the background
2. Wait for the server to be ready
3. Launch the Gradio demo
4. Handle cleanup when you press Ctrl+C

The script supports the following arguments:

• --model: Model name/path (default: Qwen/Qwen3-Omni-30B-A3B-Instruct)
• --server-port: Port for vLLM server (default: 8091)
• --gradio-port: Port for Gradio demo (default: 7861)
• --deploy-config: Path to custom deploy config YAML file (optional)
• --server-host: Host for vLLM server (default: 0.0.0.0)
• --gradio-ip: IP for Gradio demo (default: 127.0.0.1)
• --share: Share Gradio demo publicly (creates a public link)

Option 2: Manual Launch (Two-Step Process)

Step 1: Launch the vLLM API server

vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091

If you have a custom deploy config file:

vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091 --deploy-config /path/to/deploy_config_file

Step 2: Run the Gradio demo

In a separate terminal:

python gradio_demo.py --model Qwen/Qwen3-Omni-30B-A3B-Instruct --api-base http://localhost:8091/v1 --port 7861

Then open http://localhost:7861/ on your local browser to interact with the web UI.

The gradio script supports the following arguments:

• --model: Model name/path (should match the server model)
• --api-base: Base URL for the vLLM API server (default: http://localhost:8091/v1)
• --ip: Host/IP for Gradio server (default: 127.0.0.1)
• --port: Port for Gradio server (default: 7861)
• --share: Share the Gradio demo publicly (creates a public link)

Example materials

??? abstract "gradio_demo.py" py --8<-- "examples/online_serving/qwen3_omni/gradio_demo.py" ??? abstract "openai_chat_completion_client_for_multimodal_generation.py" py --8<-- "examples/online_serving/qwen3_omni/openai_chat_completion_client_for_multimodal_generation.py" ??? abstract "qwen3_omni_moe_thinking.yaml" yaml --8<-- "vllm_omni/deploy/qwen3_omni_moe_thinking.yaml" ??? abstract "run_curl_multimodal_generation.sh" sh --8<-- "examples/online_serving/qwen3_omni/run_curl_multimodal_generation.sh" ??? abstract "run_gradio_demo.sh" sh --8<-- "examples/online_serving/qwen3_omni/run_gradio_demo.sh"