a3903810fe
Signed-off-by: lishunyang <lishunyang12@163.com> Signed-off-by: reidliu41 <reid201711@gmail.com> Signed-off-by: Alex Brooks <albrooks@redhat.com> Co-authored-by: reidliu41 <reid201711@gmail.com> Co-authored-by: xiaohajiayou <75477391+xiaohajiayou@users.noreply.github.com> Co-authored-by: Alex Brooks <albrooks@redhat.com> Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
111 lines
3.4 KiB
Markdown
111 lines
3.4 KiB
Markdown
# Qwen3-Omni
|
|
|
|
## Setup
|
|
Please refer to the [stage configuration documentation](https://docs.vllm.ai/projects/vllm-omni/en/latest/configuration/stage_configs/) to configure memory allocation appropriately for your hardware setup.
|
|
|
|
## Run examples
|
|
|
|
### Multiple Prompts
|
|
Get into the example folder
|
|
```bash
|
|
cd examples/offline_inference/qwen3_omni
|
|
```
|
|
Then run the command below. Note: for processing large volume data, it uses py_generator mode, which will return a python generator from Omni class.
|
|
```bash
|
|
bash run_multiple_prompts.sh
|
|
```
|
|
### Single Prompt
|
|
Get into the example folder
|
|
```bash
|
|
cd examples/offline_inference/qwen3_omni
|
|
```
|
|
Then run the command below.
|
|
```bash
|
|
bash run_single_prompt.sh
|
|
```
|
|
If you have not enough memory, you can set thinker with tensor parallel. Just run the command below.
|
|
```bash
|
|
bash run_single_prompt_tp.sh
|
|
```
|
|
|
|
### Modality control
|
|
If you want to control output modalities, e.g. only output text, you can run the command below:
|
|
```bash
|
|
python end2end.py --output-wav output_audio \
|
|
--query-type use_audio \
|
|
--modalities text
|
|
```
|
|
|
|
#### Using Local Media Files
|
|
The `end2end.py` script supports local media files (audio, video, image) via command-line arguments:
|
|
|
|
```bash
|
|
# Use local video file
|
|
python end2end.py --query-type use_video --video-path /path/to/video.mp4
|
|
|
|
# Use local image file
|
|
python end2end.py --query-type use_image --image-path /path/to/image.jpg
|
|
|
|
# Use local audio file
|
|
python end2end.py --query-type use_audio --audio-path /path/to/audio.wav
|
|
|
|
# Combine multiple local media files
|
|
python end2end.py --query-type mixed_modalities \
|
|
--video-path /path/to/video.mp4 \
|
|
--image-path /path/to/image.jpg \
|
|
--audio-path /path/to/audio.wav
|
|
```
|
|
|
|
If media file paths are not provided, the script will use default assets. Supported query types:
|
|
- `use_video`: Video input
|
|
- `use_image`: Image input
|
|
- `use_audio`: Audio input
|
|
- `text`: Text-only query
|
|
- `multi_audios`: Multiple audio inputs
|
|
- `mixed_modalities`: Combination of video, image, and audio inputs
|
|
|
|
### Async-chunk (offline)
|
|
|
|
For true stage-level concurrency -- where downstream stages (Talker, Code2Wav)
|
|
start **before** the upstream stage (Thinker) finishes -- use the async_chunk
|
|
example. This requires:
|
|
|
|
1. A deploy config YAML with ``async_chunk: true`` (e.g.
|
|
``qwen3_omni_moe.yaml``).
|
|
2. Hardware that matches the config (e.g. 2x H100 for the default 3-stage
|
|
config).
|
|
|
|
The async_chunk example uses ``AsyncOmni`` instead of the synchronous ``Omni``
|
|
class, which enables the async orchestrator to receive stage-0 intermediate
|
|
outputs and trigger downstream stages early. Chunk data flows directly between
|
|
stage workers via the in-worker ``OmniChunkTransferAdapter`` / connector,
|
|
**not** through the orchestrator.
|
|
|
|
#### Single prompt
|
|
```bash
|
|
cd examples/offline_inference/qwen3_omni
|
|
bash run_single_prompt_async_chunk.sh
|
|
```
|
|
|
|
#### Multiple prompts with concurrency control
|
|
```bash
|
|
bash run_multiple_prompts_async_chunk.sh --max-in-flight 4
|
|
```
|
|
|
|
#### Text-only output (skip audio generation)
|
|
```bash
|
|
python end2end_async_chunk.py --query-type text --modalities text
|
|
```
|
|
|
|
#### Custom stage config
|
|
```bash
|
|
python end2end_async_chunk.py \
|
|
--query-type use_audio \
|
|
--deploy-config /path/to/your_deploy_config.yaml
|
|
```
|
|
|
|
> **Note**: The synchronous ``end2end.py`` (using ``Omni``) is still the
|
|
> recommended entry point for non-async-chunk workflows. Only use the
|
|
> async_chunk example when you need the stage-level concurrency semantics
|
|
> described in PR #962 / #1151.
|