[Config Refactor] Migrate Ming and Ming-TTS deploy/pipline configs (#3154)

Signed-off-by: Yuanheng Zhao <jonathan.zhaoyh@gmail.com>
Signed-off-by: yuanheng <jonathan.zhaoyh@gmail.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
This commit is contained in:
Yuanheng ZhaoandHongsheng Liu authored and GitHub committed 2026-05-03 14:43:49 +12:00
1 parent 6a837849dd
commit d49356e4ca
25 files changed
+518 -398

No files matched your search

@@ -4,10 +4,10 @@
vLLM-Omni supports two deployment modes:
| Mode | Stage config | Output |
|------|-------------|--------|
| Thinker only (multimodal understanding) | `ming_flash_omni_thinker.yaml` (default `--omni`) | Text |
| Thinker + Talker (omni-speech) | `ming_flash_omni.yaml` | Text + Audio |
| Mode | Deploy config | Output |
|------|--------------|--------|
| Thinker + Talker (omni-speech, default) | `vllm_omni/deploy/ming_flash_omni.yaml` | Text + Audio |
| Thinker only (multimodal understanding) | `vllm_omni/deploy/ming_flash_omni_thinker_only.yaml` | Text |
For standalone TTS (talker only), see [`examples/offline_inference/ming_flash_omni_tts/`](../ming_flash_omni_tts/).
@@ -15,30 +15,30 @@ For standalone TTS (talker only), see [`examples/offline_inference/ming_flash_om
Please refer to the [stage configuration documentation](https://docs.vllm.ai/projects/vllm-omni/en/latest/configuration/stage_configs/) to configure memory allocation appropriately for your hardware setup.
The default `--omni` flag runs thinker only. For omni-speech, pass the two-stage config explicitly:
When no `--deploy-config` is passed, the model registry auto-loads the full thinker+talker `vllm_omni/deploy/ming_flash_omni.yaml` (See [Omni-Speech](#omni-speech-thinker--talker)).
For text-only output without spinning up the talker, pass:
```bash
--stage-configs-path vllm_omni/model_executor/stage_configs/ming_flash_omni.yaml
--deploy-config vllm_omni/deploy/ming_flash_omni_thinker_only.yaml
```
## Run examples
The end-to-end script defaults to built-in assets; pass `--image-path`,
`--audio-path`, or `--video-path` to override.
The end-to-end script defaults to built-in assets; pass `--image-path`, `--audio-path`, or `--video-path` to override.
### Multi-Modality Understanding (Standalone Thinker)
Here we pass thinker-only deploy yaml:
```bash
# Text-only
python examples/offline_inference/ming_flash_omni/end2end.py --query-type text
# Image / audio / video / mixed understanding
python examples/offline_inference/ming_flash_omni/end2end.py --query-type use_image
python examples/offline_inference/ming_flash_omni/end2end.py --query-type use_audio
python examples/offline_inference/ming_flash_omni/end2end.py --query-type use_video --num-frames 16
python examples/offline_inference/ming_flash_omni/end2end.py --query-type use_mixed_modalities \
--image-path /path/to/image.jpg --audio-path /path/to/audio.wav
python examples/offline_inference/ming_flash_omni/end2end.py --deploy-config vllm_omni/deploy/ming_flash_omni_thinker_only.yaml --query-type text
python examples/offline_inference/ming_flash_omni/end2end.py --deploy-config vllm_omni/deploy/ming_flash_omni_thinker_only.yaml --query-type use_image
python examples/offline_inference/ming_flash_omni/end2end.py --deploy-config vllm_omni/deploy/ming_flash_omni_thinker_only.yaml --query-type use_audio
python examples/offline_inference/ming_flash_omni/end2end.py --deploy-config vllm_omni/deploy/ming_flash_omni_thinker_only.yaml --query-type use_video --num-frames 16
```
#### Reasoning (Thinking Mode)
### Reasoning (Thinking Mode)
Reasoning ("detailed thinking on") is applied by the script when
`--query-type reasoning` is set. The default prompt matches Ming's cookbook
@@ -46,19 +46,21 @@ and expects the reference figure from the upstream repo — see
`get_reasoning_query` in `end2end.py`.
```bash
python examples/offline_inference/ming_flash_omni/end2end.py -q reasoning --image-path ./3_0.png
python examples/offline_inference/ming_flash_omni/end2end.py \
--deploy-config vllm_omni/deploy/ming_flash_omni_thinker_only.yaml \
--query-type reasoning \
--image-path ./3_0.png
```
### Omni-speech (thinker + talker)
### Omni-Speech (Thinker + Talker)
To enable spoken output, use the two-stage config and request `audio` (or `text,audio`) modalities.
The default deploy YAML already runs thinker+talker, so spoken output only requires requesting `audio` (or `text,audio`) modalities.
The thinker processes your multimodal input, generates text, then the talker synthesises the response as speech.
**Audio-only output** (speech response, no text):
```bash
python examples/offline_inference/ming_flash_omni/end2end.py \
--query-type text \
--stage-configs-path vllm_omni/model_executor/stage_configs/ming_flash_omni.yaml \
--modalities audio \
--output-dir output_ming_omni_speech
```
@@ -67,14 +69,13 @@ python examples/offline_inference/ming_flash_omni/end2end.py \
```bash
python examples/offline_inference/ming_flash_omni/end2end.py \
--query-type use_audio \
--stage-configs-path vllm_omni/model_executor/stage_configs/ming_flash_omni.yaml \
--modalities text,audio \
--output-dir output_ming_omni_speech
```
Generated `.wav` files are saved to `--output-dir` (default `output_ming`), one per request.
The stage config allocates thinker on GPUs 0–3 and talker on GPU 3 by default. Adjust `devices` in the YAML to match your hardware.
The default deploy YAML allocates thinker on GPUs 0–3 and talker on GPU 3 for a common device topology (4 rather than 5 devices that talker on its own device). Adjust `devices` in a copied YAML and pass it via `--deploy-config` to match your hardware or requirements.
### Modality control
@@ -84,8 +85,7 @@ The stage config allocates thinker on GPUs 0–3 and talker on GPU 3 by default.
| `audio` | Text (internal) | Runs | `<id>.wav` |
| `text,audio` | Text | Runs | `<id>.txt` + `<id>.wav` |
Pass `--stage-configs-path /path/to/your_config.yaml` to any of the commands
above to override the stage config.
Pass `--deploy-config /path/to/your_deploy.yaml` to any of the commands above to override the bundled deploy config.
## Online serving
@@ -20,6 +20,7 @@ from vllm.multimodal.media.audio import load_audio
from vllm.utils.argparse_utils import FlexibleArgumentParser
import vllm_omni
from vllm_omni.engine.arg_utils import nullify_stage_engine_defaults
from vllm_omni.entrypoints.omni import Omni
# Imports the processor also registers itself
@@ -302,14 +303,7 @@ def main(args):
else:
query_result = query_func(processor)
# Initialize Omni (with thinker-only stage config)
omni = Omni(
model=MODEL_NAME,
stage_configs_path=args.stage_configs_path,
log_stats=args.log_stats,
init_timeout=args.init_timeout,
stage_init_timeout=args.stage_init_timeout,
)
omni = Omni.from_cli_args(args, model=MODEL_NAME)
# Thinker sampling params
thinker_sampling_params = SamplingParams(
@@ -410,10 +404,10 @@ def parse_args():
help="Query type.",
)
parser.add_argument(
"--stage-configs-path",
"--deploy-config",
type=str,
default=None,
help="Path to a stage configs YAML file.",
help="Path to a deploy YAML; leave unset to auto-load full thinker+talker. Pass custom for text-only",
)
parser.add_argument(
"--log-stats",
@@ -499,6 +493,7 @@ def parse_args():
help="Output directory for results.",
)
nullify_stage_engine_defaults(parser)
return parser.parse_args()
@@ -3,7 +3,7 @@
This example runs **Ming-flash-omni-2.0 talker-only** offline inference with:
- `model`: `Jonathan1909/Ming-flash-omni-2.0`
- `stage config`: `vllm_omni/model_executor/stage_configs/ming_flash_omni_tts.yaml`
- `deploy config`: `vllm_omni/deploy/ming_flash_omni_tts.yaml`
It follows the Ming cookbook parameter style:
@@ -37,7 +37,7 @@ python examples/offline_inference/ming_flash_omni_tts/end2end.py --case basic
- `--text`: override default text in the selected case
- `--output`: custom output wav path
- `--model`: local model path or HF repo id
- `--stage-configs-path`: custom talker stage config path
- `--deploy-config`: custom talker deploy YAML path
- `--log-stats`: enable runtime stats logs
## Notes
@@ -9,6 +9,7 @@ from vllm.utils.argparse_utils import FlexibleArgumentParser
os.environ["VLLM_WORKER_MULTIPROC_METHOD"] = "spawn"
from vllm_omni.engine.arg_utils import nullify_stage_engine_defaults
from vllm_omni.entrypoints.omni import Omni
from vllm_omni.inputs.data import OmniTokensPrompt
from vllm_omni.model_executor.models.ming_flash_omni.prompt_utils import (
@@ -17,7 +18,6 @@ from vllm_omni.model_executor.models.ming_flash_omni.prompt_utils import (
)
MODEL_NAME = "Jonathan1909/Ming-flash-omni-2.0"
DEFAULT_STAGE_CONFIG = "vllm_omni/model_executor/stage_configs/ming_flash_omni_tts.yaml"
def get_messages(case: str, text_override: str | None) -> dict[str, Any]:
@@ -71,10 +71,10 @@ def parse_args():
parser = FlexibleArgumentParser(description="Ming-flash-omni standalone talker offline e2e example")
parser.add_argument("--model", type=str, default=MODEL_NAME, help="Model name or local path.")
parser.add_argument(
"--stage-configs-path",
"--deploy-config",
type=str,
default=DEFAULT_STAGE_CONFIG,
help="Path to stage configs yaml for standalone talker deployment.",
default="vllm_omni/deploy/ming_flash_omni_tts.yaml",
help="Path to a custom deploy YAML for the TTS deployment. ",
)
parser.add_argument(
"--case",
@@ -88,19 +88,15 @@ def parse_args():
parser.add_argument("--log-stats", action="store_true", default=False, help="Enable stats logging.")
parser.add_argument("--init-timeout", type=int, default=600, help="Engine init timeout in seconds.")
parser.add_argument("--stage-init-timeout", type=int, default=300, help="Single stage init timeout in seconds.")
nullify_stage_engine_defaults(parser)
return parser.parse_args()
def main():
args = parse_args()
omni = Omni(
model=args.model,
stage_configs_path=args.stage_configs_path,
log_stats=args.log_stats,
init_timeout=args.init_timeout,
stage_init_timeout=args.stage_init_timeout,
)
omni = Omni.from_cli_args(args, model=args.model)
messages = get_messages(args.case, args.text)
decode_args = {