diff --git a/examples/offline_inference/ming_flash_omni/README.md b/examples/offline_inference/ming_flash_omni/README.md index be90b408d..179925bb6 100644 --- a/examples/offline_inference/ming_flash_omni/README.md +++ b/examples/offline_inference/ming_flash_omni/README.md @@ -4,10 +4,10 @@ vLLM-Omni supports two deployment modes: -| Mode | Stage config | Output | -|------|-------------|--------| -| Thinker only (multimodal understanding) | `ming_flash_omni_thinker.yaml` (default `--omni`) | Text | -| Thinker + Talker (omni-speech) | `ming_flash_omni.yaml` | Text + Audio | +| Mode | Deploy config | Output | +|------|--------------|--------| +| Thinker + Talker (omni-speech, default) | `vllm_omni/deploy/ming_flash_omni.yaml` | Text + Audio | +| Thinker only (multimodal understanding) | `vllm_omni/deploy/ming_flash_omni_thinker_only.yaml` | Text | For standalone TTS (talker only), see [`examples/offline_inference/ming_flash_omni_tts/`](../ming_flash_omni_tts/). @@ -15,30 +15,30 @@ For standalone TTS (talker only), see [`examples/offline_inference/ming_flash_om Please refer to the [stage configuration documentation](https://docs.vllm.ai/projects/vllm-omni/en/latest/configuration/stage_configs/) to configure memory allocation appropriately for your hardware setup. -The default `--omni` flag runs thinker only. For omni-speech, pass the two-stage config explicitly: +When no `--deploy-config` is passed, the model registry auto-loads the full thinker+talker `vllm_omni/deploy/ming_flash_omni.yaml` (See [Omni-Speech](#omni-speech-thinker--talker)). + +For text-only output without spinning up the talker, pass: ```bash ---stage-configs-path vllm_omni/model_executor/stage_configs/ming_flash_omni.yaml +--deploy-config vllm_omni/deploy/ming_flash_omni_thinker_only.yaml ``` ## Run examples -The end-to-end script defaults to built-in assets; pass `--image-path`, -`--audio-path`, or `--video-path` to override. +The end-to-end script defaults to built-in assets; pass `--image-path`, `--audio-path`, or `--video-path` to override. + +### Multi-Modality Understanding (Standalone Thinker) + +Here we pass thinker-only deploy yaml: ```bash -# Text-only -python examples/offline_inference/ming_flash_omni/end2end.py --query-type text - -# Image / audio / video / mixed understanding -python examples/offline_inference/ming_flash_omni/end2end.py --query-type use_image -python examples/offline_inference/ming_flash_omni/end2end.py --query-type use_audio -python examples/offline_inference/ming_flash_omni/end2end.py --query-type use_video --num-frames 16 -python examples/offline_inference/ming_flash_omni/end2end.py --query-type use_mixed_modalities \ - --image-path /path/to/image.jpg --audio-path /path/to/audio.wav +python examples/offline_inference/ming_flash_omni/end2end.py --deploy-config vllm_omni/deploy/ming_flash_omni_thinker_only.yaml --query-type text +python examples/offline_inference/ming_flash_omni/end2end.py --deploy-config vllm_omni/deploy/ming_flash_omni_thinker_only.yaml --query-type use_image +python examples/offline_inference/ming_flash_omni/end2end.py --deploy-config vllm_omni/deploy/ming_flash_omni_thinker_only.yaml --query-type use_audio +python examples/offline_inference/ming_flash_omni/end2end.py --deploy-config vllm_omni/deploy/ming_flash_omni_thinker_only.yaml --query-type use_video --num-frames 16 ``` -#### Reasoning (Thinking Mode) +### Reasoning (Thinking Mode) Reasoning ("detailed thinking on") is applied by the script when `--query-type reasoning` is set. The default prompt matches Ming's cookbook @@ -46,19 +46,21 @@ and expects the reference figure from the upstream repo — see `get_reasoning_query` in `end2end.py`. ```bash -python examples/offline_inference/ming_flash_omni/end2end.py -q reasoning --image-path ./3_0.png +python examples/offline_inference/ming_flash_omni/end2end.py \ + --deploy-config vllm_omni/deploy/ming_flash_omni_thinker_only.yaml \ + --query-type reasoning \ + --image-path ./3_0.png ``` -### Omni-speech (thinker + talker) +### Omni-Speech (Thinker + Talker) -To enable spoken output, use the two-stage config and request `audio` (or `text,audio`) modalities. +The default deploy YAML already runs thinker+talker, so spoken output only requires requesting `audio` (or `text,audio`) modalities. The thinker processes your multimodal input, generates text, then the talker synthesises the response as speech. **Audio-only output** (speech response, no text): ```bash python examples/offline_inference/ming_flash_omni/end2end.py \ --query-type text \ - --stage-configs-path vllm_omni/model_executor/stage_configs/ming_flash_omni.yaml \ --modalities audio \ --output-dir output_ming_omni_speech ``` @@ -67,14 +69,13 @@ python examples/offline_inference/ming_flash_omni/end2end.py \ ```bash python examples/offline_inference/ming_flash_omni/end2end.py \ --query-type use_audio \ - --stage-configs-path vllm_omni/model_executor/stage_configs/ming_flash_omni.yaml \ --modalities text,audio \ --output-dir output_ming_omni_speech ``` Generated `.wav` files are saved to `--output-dir` (default `output_ming`), one per request. -The stage config allocates thinker on GPUs 0–3 and talker on GPU 3 by default. Adjust `devices` in the YAML to match your hardware. +The default deploy YAML allocates thinker on GPUs 0–3 and talker on GPU 3 for a common device topology (4 rather than 5 devices that talker on its own device). Adjust `devices` in a copied YAML and pass it via `--deploy-config` to match your hardware or requirements. ### Modality control @@ -84,8 +85,7 @@ The stage config allocates thinker on GPUs 0–3 and talker on GPU 3 by default. | `audio` | Text (internal) | Runs | `.wav` | | `text,audio` | Text | Runs | `.txt` + `.wav` | -Pass `--stage-configs-path /path/to/your_config.yaml` to any of the commands -above to override the stage config. +Pass `--deploy-config /path/to/your_deploy.yaml` to any of the commands above to override the bundled deploy config. ## Online serving diff --git a/examples/offline_inference/ming_flash_omni/end2end.py b/examples/offline_inference/ming_flash_omni/end2end.py index e00dcea7b..6c8751839 100644 --- a/examples/offline_inference/ming_flash_omni/end2end.py +++ b/examples/offline_inference/ming_flash_omni/end2end.py @@ -20,6 +20,7 @@ from vllm.multimodal.media.audio import load_audio from vllm.utils.argparse_utils import FlexibleArgumentParser import vllm_omni +from vllm_omni.engine.arg_utils import nullify_stage_engine_defaults from vllm_omni.entrypoints.omni import Omni # Imports the processor also registers itself @@ -302,14 +303,7 @@ def main(args): else: query_result = query_func(processor) - # Initialize Omni (with thinker-only stage config) - omni = Omni( - model=MODEL_NAME, - stage_configs_path=args.stage_configs_path, - log_stats=args.log_stats, - init_timeout=args.init_timeout, - stage_init_timeout=args.stage_init_timeout, - ) + omni = Omni.from_cli_args(args, model=MODEL_NAME) # Thinker sampling params thinker_sampling_params = SamplingParams( @@ -410,10 +404,10 @@ def parse_args(): help="Query type.", ) parser.add_argument( - "--stage-configs-path", + "--deploy-config", type=str, default=None, - help="Path to a stage configs YAML file.", + help="Path to a deploy YAML; leave unset to auto-load full thinker+talker. Pass custom for text-only", ) parser.add_argument( "--log-stats", @@ -499,6 +493,7 @@ def parse_args(): help="Output directory for results.", ) + nullify_stage_engine_defaults(parser) return parser.parse_args() diff --git a/examples/offline_inference/ming_flash_omni_tts/README.md b/examples/offline_inference/ming_flash_omni_tts/README.md index 15b84041d..d0ad9b30d 100644 --- a/examples/offline_inference/ming_flash_omni_tts/README.md +++ b/examples/offline_inference/ming_flash_omni_tts/README.md @@ -3,7 +3,7 @@ This example runs **Ming-flash-omni-2.0 talker-only** offline inference with: - `model`: `Jonathan1909/Ming-flash-omni-2.0` -- `stage config`: `vllm_omni/model_executor/stage_configs/ming_flash_omni_tts.yaml` +- `deploy config`: `vllm_omni/deploy/ming_flash_omni_tts.yaml` It follows the Ming cookbook parameter style: @@ -37,7 +37,7 @@ python examples/offline_inference/ming_flash_omni_tts/end2end.py --case basic - `--text`: override default text in the selected case - `--output`: custom output wav path - `--model`: local model path or HF repo id -- `--stage-configs-path`: custom talker stage config path +- `--deploy-config`: custom talker deploy YAML path - `--log-stats`: enable runtime stats logs ## Notes diff --git a/examples/offline_inference/ming_flash_omni_tts/end2end.py b/examples/offline_inference/ming_flash_omni_tts/end2end.py index 928994510..8a26d1b33 100644 --- a/examples/offline_inference/ming_flash_omni_tts/end2end.py +++ b/examples/offline_inference/ming_flash_omni_tts/end2end.py @@ -9,6 +9,7 @@ from vllm.utils.argparse_utils import FlexibleArgumentParser os.environ["VLLM_WORKER_MULTIPROC_METHOD"] = "spawn" +from vllm_omni.engine.arg_utils import nullify_stage_engine_defaults from vllm_omni.entrypoints.omni import Omni from vllm_omni.inputs.data import OmniTokensPrompt from vllm_omni.model_executor.models.ming_flash_omni.prompt_utils import ( @@ -17,7 +18,6 @@ from vllm_omni.model_executor.models.ming_flash_omni.prompt_utils import ( ) MODEL_NAME = "Jonathan1909/Ming-flash-omni-2.0" -DEFAULT_STAGE_CONFIG = "vllm_omni/model_executor/stage_configs/ming_flash_omni_tts.yaml" def get_messages(case: str, text_override: str | None) -> dict[str, Any]: @@ -71,10 +71,10 @@ def parse_args(): parser = FlexibleArgumentParser(description="Ming-flash-omni standalone talker offline e2e example") parser.add_argument("--model", type=str, default=MODEL_NAME, help="Model name or local path.") parser.add_argument( - "--stage-configs-path", + "--deploy-config", type=str, - default=DEFAULT_STAGE_CONFIG, - help="Path to stage configs yaml for standalone talker deployment.", + default="vllm_omni/deploy/ming_flash_omni_tts.yaml", + help="Path to a custom deploy YAML for the TTS deployment. ", ) parser.add_argument( "--case", @@ -88,19 +88,15 @@ def parse_args(): parser.add_argument("--log-stats", action="store_true", default=False, help="Enable stats logging.") parser.add_argument("--init-timeout", type=int, default=600, help="Engine init timeout in seconds.") parser.add_argument("--stage-init-timeout", type=int, default=300, help="Single stage init timeout in seconds.") + + nullify_stage_engine_defaults(parser) return parser.parse_args() def main(): args = parse_args() - omni = Omni( - model=args.model, - stage_configs_path=args.stage_configs_path, - log_stats=args.log_stats, - init_timeout=args.init_timeout, - stage_init_timeout=args.stage_init_timeout, - ) + omni = Omni.from_cli_args(args, model=args.model) messages = get_messages(args.case, args.text) decode_args = { diff --git a/examples/online_serving/ming_flash_omni/README.md b/examples/online_serving/ming_flash_omni/README.md index 8b7d03e21..dd8c8aa81 100644 --- a/examples/online_serving/ming_flash_omni/README.md +++ b/examples/online_serving/ming_flash_omni/README.md @@ -8,8 +8,8 @@ Please refer to [README.md](../../../README.md) | Mode | Launch command | Output | |------|---------------|--------| -| Thinker only (multimodal understanding) | `vllm serve ... --omni` | Text | -| Thinker + Talker (omni-speech) | `vllm serve ... --omni --stage-configs-path ming_flash_omni.yaml` | Text + Audio | +| Thinker + Talker (omni-speech, default) | `vllm serve ... --omni` | Text + Audio | +| Thinker only (multimodal understanding) | `vllm serve ... --omni --deploy-config vllm_omni/deploy/ming_flash_omni_thinker_only.yaml` | Text | For standalone TTS (talker only), see [`examples/online_serving/ming_flash_omni_tts/`](../ming_flash_omni_tts/). @@ -17,18 +17,20 @@ For standalone TTS (talker only), see [`examples/online_serving/ming_flash_omni_ ### Launch the Server -**Thinker only (text output):** +**Thinker + Talker (omni-speech, text + audio output):** ```bash vllm serve Jonathan1909/Ming-flash-omni-2.0 --omni --port 8091 ``` -**Thinker + Talker (omni-speech, text + audio output):** +The model registry auto-loads corresponding deploy yaml. + +**Thinker-only (text output):** ```bash vllm serve Jonathan1909/Ming-flash-omni-2.0 --omni --port 8091 \ - --stage-configs-path vllm_omni/model_executor/stage_configs/ming_flash_omni.yaml + --deploy-config vllm_omni/deploy/ming_flash_omni_thinker_only.yaml ``` -Pass `--stage-configs-path /path/to/your_config.yaml` to use a custom stage +Pass `--deploy-config /path/to/your_deploy.yaml` to use a custom deploy config. ### Send Multi-modal Request diff --git a/examples/online_serving/ming_flash_omni/run_curl_multimodal_generation.sh b/examples/online_serving/ming_flash_omni/run_curl_multimodal_generation.sh index 768a424e4..715f23edd 100755 --- a/examples/online_serving/ming_flash_omni/run_curl_multimodal_generation.sh +++ b/examples/online_serving/ming_flash_omni/run_curl_multimodal_generation.sh @@ -18,17 +18,6 @@ if [[ ! "$QUERY_TYPE" =~ ^(text|use_audio|use_image|use_video|use_mixed_modaliti exit 1 fi -thinker_sampling_params='{ - "temperature": 0.4, - "top_p": 0.9, - "top_k": -1, - "max_tokens": 16384, - "seed": 42, - "detokenize": true, - "repetition_penalty": 1.05 -}' -# Above is optional, it has a default setting in stage_configs of the corresponding model. - # Define URLs for assets MARY_HAD_LAMB_AUDIO_URL="https://vllm-public-assets.s3.us-west-2.amazonaws.com/multimodal_asset/mary_had_lamb.ogg" CHERRY_BLOSSOM_IMAGE_URL="https://vllm-public-assets.s3.us-west-2.amazonaws.com/vision_model_images/cherry_blossom.jpg" @@ -114,9 +103,6 @@ echo "" request_body=$(cat < str: ) +_CI_DEPLOY_THINKER_ONLY = get_deploy_config_path("ci/ming_flash_omni_thinker_only.yaml") +_CI_DEPLOY = get_deploy_config_path("ci/ming_flash_omni.yaml") + + +def get_eager_config_thinker(): + return modify_stage_config(_CI_DEPLOY_THINKER_ONLY, updates={"stages": {0: {"enforce_eager": True}}}) + + def get_eager_config(): - path = modify_stage_config( - str(Path(__file__).parent.parent / "stage_configs" / "bailingmm_moe_v2_lite_thinker_only_ci.yaml"), - updates={ - "stage_args": { - 0: { - "engine_args.enforce_eager": "true", - }, - }, - }, - ) - return path - - -def get_eager_tts_config(): """Thinker+talker CI config with enforce_eager on the thinker stage.""" - path = modify_stage_config( - str(Path(__file__).parent.parent / "stage_configs" / "bailingmm_moe_v2_lite_ci.yaml"), - updates={ - "stage_args": { - 0: { - "engine_args.enforce_eager": "true", - }, - }, - }, - ) - return path + return modify_stage_config(_CI_DEPLOY, updates={"stages": {0: {"enforce_eager": True}}}) -# Thinker-only config — used by text-output tests. -stage_configs = [get_eager_config()] -test_params = [(model, stage_config) for model in models for stage_config in stage_configs] - -# Thinker+talker config — used by audio-output tests. -stage_configs_tts = [get_eager_tts_config()] -test_params_tts = [(model, stage_config) for model in models for stage_config in stage_configs_tts] +test_params_thinker = [(m, c) for m in models for c in [get_eager_config_thinker()]] +test_params = [(m, c) for m in models for c in [get_eager_config()]] @pytest.mark.core_model @pytest.mark.omni @hardware_test(res={"cuda": "H100"}, num_cards=4) -@pytest.mark.parametrize("omni_runner", test_params, indirect=True) +@pytest.mark.parametrize("omni_runner", test_params_thinker, indirect=True) def test_text_to_text(omni_runner, omni_runner_handler) -> None: """ Test text-only input processing and text output generation. @@ -92,7 +69,7 @@ def test_text_to_text(omni_runner, omni_runner_handler) -> None: @pytest.mark.core_model @pytest.mark.omni @hardware_test(res={"cuda": "H100"}, num_cards=4) -@pytest.mark.parametrize("omni_runner", test_params, indirect=True) +@pytest.mark.parametrize("omni_runner", test_params_thinker, indirect=True) def test_image_to_text(omni_runner, omni_runner_handler) -> None: """ Test image understanding with text output. @@ -109,7 +86,7 @@ def test_image_to_text(omni_runner, omni_runner_handler) -> None: @pytest.mark.core_model @pytest.mark.omni @hardware_test(res={"cuda": "H100"}, num_cards=4) -@pytest.mark.parametrize("omni_runner", test_params, indirect=True) +@pytest.mark.parametrize("omni_runner", test_params_thinker, indirect=True) def test_audio_to_text(omni_runner, omni_runner_handler) -> None: """ Test audio understanding with text output. @@ -128,7 +105,7 @@ def test_audio_to_text(omni_runner, omni_runner_handler) -> None: @pytest.mark.core_model @pytest.mark.omni @hardware_test(res={"cuda": "H100"}, num_cards=4) -@pytest.mark.parametrize("omni_runner", test_params, indirect=True) +@pytest.mark.parametrize("omni_runner", test_params_thinker, indirect=True) def test_video_to_text(omni_runner, omni_runner_handler) -> None: """ Test video understanding with text output. @@ -145,7 +122,7 @@ def test_video_to_text(omni_runner, omni_runner_handler) -> None: @pytest.mark.core_model @pytest.mark.omni @hardware_test(res={"cuda": "H100"}, num_cards=4) -@pytest.mark.parametrize("omni_runner", test_params, indirect=True) +@pytest.mark.parametrize("omni_runner", test_params_thinker, indirect=True) def test_mixed_to_text(omni_runner, omni_runner_handler) -> None: """ Test mixed modality input (image + audio) with text output. @@ -165,7 +142,7 @@ def test_mixed_to_text(omni_runner, omni_runner_handler) -> None: @pytest.mark.advanced_model @pytest.mark.omni @hardware_test(res={"cuda": "H100"}, num_cards=4) -@pytest.mark.parametrize("omni_runner", test_params_tts, indirect=True) +@pytest.mark.parametrize("omni_runner", test_params, indirect=True) def test_text_to_audio(omni_runner, omni_runner_handler) -> None: """ Test text input with audio output via the thinker+talker pipeline. @@ -181,7 +158,7 @@ def test_text_to_audio(omni_runner, omni_runner_handler) -> None: @pytest.mark.advanced_model @pytest.mark.omni @hardware_test(res={"cuda": "H100"}, num_cards=4) -@pytest.mark.parametrize("omni_runner", test_params_tts, indirect=True) +@pytest.mark.parametrize("omni_runner", test_params, indirect=True) def test_image_to_audio(omni_runner, omni_runner_handler) -> None: """ Test image + text input with audio output via the thinker+talker pipeline. diff --git a/tests/e2e/online_serving/test_ming_flash_omni.py b/tests/e2e/online_serving/test_ming_flash_omni.py index 8161c4389..2a67761e2 100644 --- a/tests/e2e/online_serving/test_ming_flash_omni.py +++ b/tests/e2e/online_serving/test_ming_flash_omni.py @@ -6,7 +6,6 @@ Tests multimodal understanding via OpenAI-compatible API. """ import os -from pathlib import Path import pytest @@ -17,29 +16,17 @@ from tests.helpers.media import ( generate_synthetic_video, ) from tests.helpers.runtime import OmniServerParams, dummy_messages_from_mix_data -from tests.helpers.stage_config import modify_stage_config +from tests.helpers.stage_config import get_deploy_config_path os.environ["VLLM_WORKER_MULTIPROC_METHOD"] = "spawn" os.environ["VLLM_TEST_CLEAN_GPU_MEMORY"] = "0" models = ["Jonathan1909/Ming-flash-omni-2.0"] +# Use thinker-only topology to test text outputs +_CI_DEPLOY = get_deploy_config_path("ci/ming_flash_omni_thinker_only.yaml") -def get_eager_config(): - path = modify_stage_config( - str(Path(__file__).parent.parent / "stage_configs" / "bailingmm_moe_v2_lite_ci.yaml"), - updates={ - "stage_args": { - 0: { - "engine_args.enforce_eager": "true", - }, - }, - }, - ) - return path - - -stage_configs = [get_eager_config()] +stage_configs = [_CI_DEPLOY] # Create parameter combinations for model and stage config test_params = [ diff --git a/tests/e2e/stage_configs/bailingmm_moe_v2_lite_ci.yaml b/tests/e2e/stage_configs/bailingmm_moe_v2_lite_ci.yaml deleted file mode 100644 index 53cc73ce0..000000000 --- a/tests/e2e/stage_configs/bailingmm_moe_v2_lite_ci.yaml +++ /dev/null @@ -1,75 +0,0 @@ -# CI stage config for Ming-flash-omni-2.0 thinker+talker pipeline. -# Stage 0: Thinker (multimodal understanding -> text generation) -# Stage 1: Talker (text -> audio waveform via CFM + AudioVAE) -# -# The following config has been verified on 4x H100-80G GPUs -stage_args: - - stage_id: 0 - stage_type: llm - runtime: - devices: "0,1,2,3" - max_batch_size: 1 - engine_args: - model_stage: thinker - model_arch: MingFlashOmniForConditionalGeneration - worker_type: ar - scheduler_cls: vllm_omni.core.sched.omni_ar_scheduler.OmniARScheduler - gpu_memory_utilization: 0.74 - enforce_eager: false - trust_remote_code: true - # Ming Thinker -> talker bridge reads detokenised text from - # source_output.outputs[0].text (not hidden states). - engine_output_type: text - distributed_executor_backend: "mp" - enable_prefix_caching: false - max_num_batched_tokens: 32768 - max_model_len: 32768 - tensor_parallel_size: 4 - hf_config_name: llm_config - load_format: dummy - mm_processor_cache_gb: 0 - compilation_config: - pass_config: - # Disable fused all-reduce to avoid a vllm/flashinfer version mismatch. - fuse_allreduce_rms: false - final_output: true - final_output_type: text - is_comprehension: true - default_sampling_params: - temperature: 0.0 - top_p: 0.9 - max_tokens: 100 - repetition_penalty: 1.05 - seed: 42 - detokenize: true - ignore_eos: false - - - stage_id: 1 - stage_type: llm - runtime: - devices: "3" - max_batch_size: 1 - engine_args: - model_stage: ming_tts - model_arch: MingFlashOmniTalkerForConditionalGeneration - worker_cls: vllm_omni.worker.gpu_generation_worker.GPUGenerationWorker - scheduler_cls: vllm_omni.core.sched.omni_generation_scheduler.OmniGenerationScheduler - gpu_memory_utilization: 0.18 - enforce_eager: true - trust_remote_code: true - engine_output_type: audio - enable_prefix_caching: false - max_num_batched_tokens: 1000000 - tokenizer_subdir: talker/llm - # The HF repo ships BailingMM2Config (thinker-only) at root; - # OmniModelConfig treats that as "stage does not share outer mrope". - hf_config_name: talker_config - load_format: dummy - engine_input_source: [0] - custom_process_input_func: vllm_omni.model_executor.stage_input_processors.ming_flash_omni.thinker2talker - final_output: true - final_output_type: audio - default_sampling_params: - temperature: 0.0 - max_tokens: 1 - seed: 42 diff --git a/tests/e2e/stage_configs/bailingmm_moe_v2_lite_thinker_only_ci.yaml b/tests/e2e/stage_configs/bailingmm_moe_v2_lite_thinker_only_ci.yaml deleted file mode 100644 index fb0c72cc5..000000000 --- a/tests/e2e/stage_configs/bailingmm_moe_v2_lite_thinker_only_ci.yaml +++ /dev/null @@ -1,35 +0,0 @@ -# Thinker stage only -stage_args: - - stage_id: 0 - stage_type: llm - runtime: - devices: "0,1,2,3" - max_batch_size: 1 - engine_args: - model_stage: thinker - model_arch: MingFlashOmniForConditionalGeneration - worker_type: ar - scheduler_cls: vllm_omni.core.sched.omni_ar_scheduler.OmniARScheduler - gpu_memory_utilization: 0.9 - enforce_eager: false - trust_remote_code: true - engine_output_type: latent - distributed_executor_backend: "mp" - enable_prefix_caching: false - max_num_batched_tokens: 32768 - max_model_len: 32768 - tensor_parallel_size: 4 - hf_config_name: llm_config - load_format: dummy - mm_processor_cache_gb: 0 - final_output: true - final_output_type: text - is_comprehension: true - default_sampling_params: - temperature: 0.4 - top_p: 0.9 - max_tokens: 100 - repetition_penalty: 1.05 - seed: 42 - detokenize: true - ignore_eos: false diff --git a/tests/helpers/stage_config.py b/tests/helpers/stage_config.py index 2bb017b81..66a58378d 100644 --- a/tests/helpers/stage_config.py +++ b/tests/helpers/stage_config.py @@ -484,6 +484,52 @@ _CI_OVERLAYS: dict[str, dict[str, Any]] = { }, }, }, + "ming_flash_omni": { + "base_config": "ming_flash_omni.yaml", + "stages": [ + { + "stage_id": 0, + "max_num_seqs": 1, + "gpu_memory_utilization": 0.74, + "max_model_len": 16384, + "max_num_batched_tokens": 16384, + "mm_processor_cache_gb": 0, + "skip_mm_profiling": True, + "enable_flashinfer_autotune": False, + "load_format": "dummy", + "default_sampling_params": { + "temperature": 0.0, + "max_tokens": 100, + }, + }, + { + "stage_id": 1, + "max_num_seqs": 1, + "gpu_memory_utilization": 0.18, + "load_format": "dummy", + }, + ], + }, + "ming_flash_omni_thinker_only": { + "base_config": "ming_flash_omni_thinker_only.yaml", + "stages": [ + { + "stage_id": 0, + "max_num_seqs": 1, + "gpu_memory_utilization": 0.9, + "max_model_len": 16384, + "max_num_batched_tokens": 16384, + "mm_processor_cache_gb": 0, + "skip_mm_profiling": True, + "enable_flashinfer_autotune": False, + "load_format": "dummy", + "default_sampling_params": { + "temperature": 0.4, + "max_tokens": 100, + }, + }, + ], + }, # Single-stage thinker-only topology for the abort test. "qwen2_5_omni_thinker_only": { "async_chunk": False, diff --git a/tests/test_config_factory.py b/tests/test_config_factory.py index 16d49034f..22dcf26d0 100644 --- a/tests/test_config_factory.py +++ b/tests/test_config_factory.py @@ -997,6 +997,162 @@ class TestQwen3TTSPipeline: } +class TestMingFlashOmniPipeline: + def test_registered(self): + import vllm_omni.model_executor.models.ming_flash_omni.pipeline # noqa: F401 + + p = _PIPELINE_REGISTRY.get("ming_flash_omni") + assert p is not None + assert p.model_arch == "MingFlashOmniForConditionalGeneration" + assert len(p.stages) == 2 + assert p.validate() == [] + + def test_thinker_stage(self): + import vllm_omni.model_executor.models.ming_flash_omni.pipeline # noqa: F401 + + s = _PIPELINE_REGISTRY["ming_flash_omni"].get_stage(0) + assert s.model_stage == "thinker" + assert s.execution_type == StageExecutionType.LLM_AR + assert s.owns_tokenizer is True + assert s.requires_multimodal_data is True + assert s.engine_output_type == "text" + assert s.hf_config_name == "llm_config" + assert s.sampling_constraints["detokenize"] is True + + def test_talker_stage(self): + import vllm_omni.model_executor.models.ming_flash_omni.pipeline # noqa: F401 + + s = _PIPELINE_REGISTRY["ming_flash_omni"].get_stage(1) + assert s.model_stage == "ming_tts" + assert s.execution_type == StageExecutionType.LLM_GENERATION + assert s.input_sources == (0,) + assert s.final_output_type == "audio" + assert s.engine_output_type == "audio" + assert s.hf_config_name == "talker_config" + # Per-stage model_arch override (Ming talker has its own self-contained LLM) + assert s.model_arch == "MingFlashOmniTalkerForConditionalGeneration" + assert s.tokenizer_subdir == "talker/llm" + assert s.custom_process_input_func is not None + + def test_talker_stage_processor_wiring_resolves(self): + """The custom_process_input_func string must point to a real callable. + + Lazy string references only fail at first inference otherwise — this + catches typos in the pipeline declaration at import / registration time. + """ + import importlib + + import vllm_omni.model_executor.models.ming_flash_omni.pipeline # noqa: F401 + + s = _PIPELINE_REGISTRY["ming_flash_omni"].get_stage(1) + module_path, _, attr = s.custom_process_input_func.rpartition(".") + module = importlib.import_module(module_path) + assert callable(getattr(module, attr)) + + def test_tts_pipeline_registered(self): + import vllm_omni.model_executor.models.ming_flash_omni.pipeline # noqa: F401 + + p = _PIPELINE_REGISTRY.get("ming_flash_omni_tts") + assert p is not None + assert p.model_arch == "MingFlashOmniTalkerForConditionalGeneration" + assert len(p.stages) == 1 + assert p.validate() == [] + + def test_tts_stage(self): + import vllm_omni.model_executor.models.ming_flash_omni.pipeline # noqa: F401 + + s = _PIPELINE_REGISTRY["ming_flash_omni_tts"].get_stage(0) + assert s.model_stage == "ming_tts" + assert s.execution_type == StageExecutionType.LLM_GENERATION + assert s.input_sources == () + assert s.owns_tokenizer is True + assert s.final_output_type == "audio" + assert s.engine_output_type == "audio" + assert s.hf_config_name == "talker_config" + assert s.tokenizer_subdir == "talker/llm" + + def test_full_yaml_loads_and_merges(self): + """deploy/ming_flash_omni.yaml parses and merges with the registered pipeline.""" + import vllm_omni.model_executor.models.ming_flash_omni.pipeline # noqa: F401 + from vllm_omni.config.stage_config import load_deploy_config, merge_pipeline_deploy + + deploy_path = Path(__file__).parent.parent / "vllm_omni" / "deploy" / "ming_flash_omni.yaml" + if not deploy_path.exists(): + pytest.skip("ming_flash_omni deploy yaml not found") + + deploy = load_deploy_config(deploy_path) + assert len(deploy.stages) == 2 + assert deploy.async_chunk is False + assert deploy.pipeline == "ming_flash_omni" + # We won't test stage 0/1 colocation contract here, + # as there could exist more variant of custom device setup + + pipeline = _PIPELINE_REGISTRY["ming_flash_omni"] + stages = merge_pipeline_deploy(pipeline, deploy) + assert len(stages) == 2 + assert stages[0].yaml_engine_args["model_arch"] == "MingFlashOmniForConditionalGeneration" + assert stages[1].yaml_engine_args["model_arch"] == "MingFlashOmniTalkerForConditionalGeneration" + + def test_tts_yaml_loads_and_merges(self): + """deploy/ming_flash_omni_tts.yaml parses and routes to the TTS-only pipeline.""" + import vllm_omni.model_executor.models.ming_flash_omni.pipeline # noqa: F401 + from vllm_omni.config.stage_config import load_deploy_config, merge_pipeline_deploy + + deploy_path = Path(__file__).parent.parent / "vllm_omni" / "deploy" / "ming_flash_omni_tts.yaml" + if not deploy_path.exists(): + pytest.skip("ming_flash_omni_tts deploy yaml not found") + + deploy = load_deploy_config(deploy_path) + assert len(deploy.stages) == 1 + assert deploy.pipeline == "ming_flash_omni_tts" + + pipeline = _PIPELINE_REGISTRY["ming_flash_omni_tts"] + stages = merge_pipeline_deploy(pipeline, deploy) + assert len(stages) == 1 + assert stages[0].yaml_engine_args["model_arch"] == "MingFlashOmniTalkerForConditionalGeneration" + + def test_thinker_only_pipeline_registered(self): + import vllm_omni.model_executor.models.ming_flash_omni.pipeline # noqa: F401 + + p = _PIPELINE_REGISTRY.get("ming_flash_omni_thinker_only") + assert p is not None + assert p.model_arch == "MingFlashOmniForConditionalGeneration" + assert len(p.stages) == 1 + assert p.validate() == [] + + def test_thinker_only_stage(self): + import vllm_omni.model_executor.models.ming_flash_omni.pipeline # noqa: F401 + + s = _PIPELINE_REGISTRY["ming_flash_omni_thinker_only"].get_stage(0) + assert s.model_stage == "thinker" + assert s.execution_type == StageExecutionType.LLM_AR + assert s.input_sources == () + assert s.owns_tokenizer is True + assert s.requires_multimodal_data is True + assert s.final_output_type == "text" + assert s.engine_output_type == "text" + assert s.hf_config_name == "llm_config" + assert s.sampling_constraints["detokenize"] is True + + def test_thinker_only_yaml_loads_and_merges(self): + """deploy/ming_flash_omni_thinker_only.yaml parses and routes to the thinker-only pipeline.""" + import vllm_omni.model_executor.models.ming_flash_omni.pipeline # noqa: F401 + from vllm_omni.config.stage_config import load_deploy_config, merge_pipeline_deploy + + deploy_path = Path(__file__).parent.parent / "vllm_omni" / "deploy" / "ming_flash_omni_thinker_only.yaml" + if not deploy_path.exists(): + pytest.skip("ming_flash_omni_thinker_only deploy yaml not found") + + deploy = load_deploy_config(deploy_path) + assert len(deploy.stages) == 1 + assert deploy.pipeline == "ming_flash_omni_thinker_only" + + pipeline = _PIPELINE_REGISTRY["ming_flash_omni_thinker_only"] + stages = merge_pipeline_deploy(pipeline, deploy) + assert len(stages) == 1 + assert stages[0].yaml_engine_args["model_arch"] == "MingFlashOmniForConditionalGeneration" + + class TestBaseConfigInheritance: """Test deploy YAML base_config inheritance.""" diff --git a/vllm_omni/config/pipeline_registry.py b/vllm_omni/config/pipeline_registry.py index 6f5c072a3..1a129cad8 100644 --- a/vllm_omni/config/pipeline_registry.py +++ b/vllm_omni/config/pipeline_registry.py @@ -85,6 +85,18 @@ _OMNI_PIPELINES: dict[str, tuple[str, str]] = { "vllm_omni.model_executor.models.fish_speech.pipeline", "FISH_SPEECH_PIPELINE", ), + "ming_flash_omni": ( + "vllm_omni.model_executor.models.ming_flash_omni.pipeline", + "MING_FLASH_OMNI_PIPELINE", + ), + "ming_flash_omni_tts": ( + "vllm_omni.model_executor.models.ming_flash_omni.pipeline", + "MING_FLASH_OMNI_TTS_PIPELINE", + ), + "ming_flash_omni_thinker_only": ( + "vllm_omni.model_executor.models.ming_flash_omni.pipeline", + "MING_FLASH_OMNI_THINKER_ONLY_PIPELINE", + ), "moss_tts_nano": ( "vllm_omni.model_executor.models.moss_tts_nano.pipeline", "MOSS_TTS_NANO_PIPELINE", diff --git a/vllm_omni/deploy/ming_flash_omni.yaml b/vllm_omni/deploy/ming_flash_omni.yaml new file mode 100644 index 000000000..f1cd7b6ad --- /dev/null +++ b/vllm_omni/deploy/ming_flash_omni.yaml @@ -0,0 +1,56 @@ +# Ming-flash-omni-2.0 deploy: thinker + talker. +# +# Stage 0: Thinker: multimodal understanding -> text generation. +# TP=4 (or PP=4) across GPUs 0-3. +# Stage 1: Talker: text -> audio waveform via CFM + AudioVAE. +# Generation worker, runs eager, colocates on the last GPU. +# +# Fields omitted from a stage fall back to StageDeployConfig dataclass +# defaults (see vllm_omni/config/stage_config.py). +# * Stage 0 (thinker) owns GPUs 0-3 with gpu_memory_utilization=0.74, +# as we tend to fit talker in one of the devices. (For now, +# gpu mem util on different devices are not supported to be set +# independently for a single stage.) +# * Stage 1 (talker) colocates on GPU 3 with gpu_memory_utilization=0.18. +# This custom settings is verified and required on devices with 80GB memory. +pipeline: ming_flash_omni +async_chunk: false + +trust_remote_code: true +distributed_executor_backend: mp +enable_prefix_caching: false + +stages: + - stage_id: 0 + max_num_seqs: 1 + gpu_memory_utilization: 0.74 + enforce_eager: false + tensor_parallel_size: 4 + max_num_batched_tokens: 32768 + devices: "0,1,2,3" + # disable fused allreduce+rms for flashinfer version mismatch for now + compilation_config: + pass_config: + fuse_allreduce_rms: false + default_sampling_params: + temperature: 0.4 + top_p: 0.9 + max_tokens: 2048 + repetition_penalty: 1.05 + detokenize: true + ignore_eos: false + + - stage_id: 1 + max_num_seqs: 1 + gpu_memory_utilization: 0.18 + enforce_eager: true + max_num_batched_tokens: 1000000 + devices: "3" + default_sampling_params: + temperature: 0.0 + max_tokens: 1 + +# Ming-flash-omni-2.0 has only been verified on CUDA devices, +# left empty intentionally. Add per-platform overrides here +# when other backends (ROCm/XPU/NPU) are verified end-to-end. +platforms: {} diff --git a/vllm_omni/deploy/ming_flash_omni_thinker_only.yaml b/vllm_omni/deploy/ming_flash_omni_thinker_only.yaml new file mode 100644 index 000000000..88a0202d3 --- /dev/null +++ b/vllm_omni/deploy/ming_flash_omni_thinker_only.yaml @@ -0,0 +1,36 @@ +# Ming-flash-omni-2.0 thinker-only deploy: multimodal understanding -> text. +# +# Single-stage configuration that drops the talker; useful when only text +# output is required (chat / understanding / reasoning) +# +# Stage 0: Thinker — TP=4 across GPUs 0-3 with full memory budget since no +# talker shares the last device. +pipeline: ming_flash_omni_thinker_only +async_chunk: false + +trust_remote_code: true +distributed_executor_backend: mp +enable_prefix_caching: false + +stages: + - stage_id: 0 + max_num_seqs: 1 + gpu_memory_utilization: 0.9 + enforce_eager: false + tensor_parallel_size: 4 + max_num_batched_tokens: 32768 + devices: "0,1,2,3" + # disable fused allreduce+rms for flashinfer version mismatch for now + compilation_config: + pass_config: + fuse_allreduce_rms: false + default_sampling_params: + temperature: 0.4 + top_p: 0.9 + max_tokens: 2048 + repetition_penalty: 1.05 + detokenize: true + ignore_eos: false + +# Ming-flash-omni-2.0 has only been verified on CUDA devices +platforms: {} diff --git a/vllm_omni/deploy/ming_flash_omni_tts.yaml b/vllm_omni/deploy/ming_flash_omni_tts.yaml new file mode 100644 index 000000000..691d37969 --- /dev/null +++ b/vllm_omni/deploy/ming_flash_omni_tts.yaml @@ -0,0 +1,24 @@ +# Ming-flash-omni-2.0 standalone TTS deploy: talker only. +# +# Single-stage configuration for direct text -> audio synthesis +# The talker runs as a generation worker on a single GPU +# and owns the tokenizer. +pipeline: ming_flash_omni_tts +async_chunk: false + +trust_remote_code: true +enable_prefix_caching: false + +stages: + - stage_id: 0 + max_num_seqs: 1 + gpu_memory_utilization: 0.8 + enforce_eager: true + max_num_batched_tokens: 100000 + devices: "0" + default_sampling_params: + temperature: 0.0 + max_tokens: 1 + +# Ming-flash-omni-2.0 has only been verified on CUDA devices +platforms: {} diff --git a/vllm_omni/model_executor/models/ming_flash_omni/ming_flash_omni.py b/vllm_omni/model_executor/models/ming_flash_omni/ming_flash_omni.py index 462c20438..0111f8069 100644 --- a/vllm_omni/model_executor/models/ming_flash_omni/ming_flash_omni.py +++ b/vllm_omni/model_executor/models/ming_flash_omni/ming_flash_omni.py @@ -104,8 +104,8 @@ class MingFlashOmniForConditionalGeneration( "model_arch='MingFlashOmniTalkerForConditionalGeneration' " "directly — the talker has a self-contained LLM that " "tokenises text itself and does not need the multimodal " - "preprocess path. See stage_configs/ming_flash_omni.yaml " - "stage 1 and stage_configs/ming_flash_omni_tts.yaml." + "preprocess path. See vllm_omni/deploy/ming_flash_omni.yaml " + "stage 1 and vllm_omni/deploy/ming_flash_omni_tts.yaml." ) if self.model_stage == "thinker": diff --git a/vllm_omni/model_executor/models/ming_flash_omni/pipeline.py b/vllm_omni/model_executor/models/ming_flash_omni/pipeline.py new file mode 100644 index 000000000..a9d66fbc2 --- /dev/null +++ b/vllm_omni/model_executor/models/ming_flash_omni/pipeline.py @@ -0,0 +1,106 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Ming-flash-omni-2.0 pipeline topology (frozen). + +Stage 0: Thinker — multimodal understanding + text generation. +Stage 1: Talker — text -> audio waveform via CFM + AudioVAE. + +The thinker -> talker bridge passes the detokenized text rather than +hidden states through `ming_flash_omni.thinker2talker`; the talker has a +self-contained Qwen2 LLM that retokenizes the string itself. +""" + +from vllm_omni.config.stage_config import ( + PipelineConfig, + StageExecutionType, + StagePipelineConfig, +) + +_PROC = "vllm_omni.model_executor.stage_input_processors.ming_flash_omni" + + +MING_FLASH_OMNI_PIPELINE = PipelineConfig( + model_type="ming_flash_omni", + model_arch="MingFlashOmniForConditionalGeneration", + # Upstream HF config applies model_type="bailingmm_moe_v2_lite" + # (the thinker sub-config name) rather than "ming_flash_omni". + # Declare the architectures here explicitly for routing + hf_architectures=( + "MingFlashOmniForConditionalGeneration", + "BailingMM2NativeForConditionalGeneration", + ), + stages=( + StagePipelineConfig( + stage_id=0, + model_stage="thinker", + execution_type=StageExecutionType.LLM_AR, + input_sources=(), + final_output=True, + final_output_type="text", + owns_tokenizer=True, + requires_multimodal_data=True, + # Thinker reads the LLM sub-config of BailingMM2Config + hf_config_name="llm_config", + engine_output_type="text", + sampling_constraints={"detokenize": True}, + ), + StagePipelineConfig( + stage_id=1, + model_stage="ming_tts", + # Talker is a self-contained Qwen2 LLM + CFM + AudioVAE; + # it does not share the thinker backbone, so it gets its own arch. + model_arch="MingFlashOmniTalkerForConditionalGeneration", + execution_type=StageExecutionType.LLM_GENERATION, + input_sources=(0,), + final_output=True, + final_output_type="audio", + hf_config_name="talker_config", + engine_output_type="audio", + tokenizer_subdir="talker/llm", + custom_process_input_func=f"{_PROC}.thinker2talker", + ), + ), +) + + +# Standalone TTS variant: talker only. +MING_FLASH_OMNI_TTS_PIPELINE = PipelineConfig( + model_type="ming_flash_omni_tts", + model_arch="MingFlashOmniTalkerForConditionalGeneration", + stages=( + StagePipelineConfig( + stage_id=0, + model_stage="ming_tts", + execution_type=StageExecutionType.LLM_GENERATION, + input_sources=(), + final_output=True, + final_output_type="audio", + owns_tokenizer=True, + hf_config_name="talker_config", + engine_output_type="audio", + tokenizer_subdir="talker/llm", + ), + ), +) + + +# Thinker-only variant: multimodal understanding with text output +MING_FLASH_OMNI_THINKER_ONLY_PIPELINE = PipelineConfig( + model_type="ming_flash_omni_thinker_only", + model_arch="MingFlashOmniForConditionalGeneration", + stages=( + StagePipelineConfig( + stage_id=0, + model_stage="thinker", + execution_type=StageExecutionType.LLM_AR, + input_sources=(), + final_output=True, + final_output_type="text", + owns_tokenizer=True, + requires_multimodal_data=True, + hf_config_name="llm_config", + engine_output_type="text", + sampling_constraints={"detokenize": True}, + ), + ), +) diff --git a/vllm_omni/model_executor/models/ming_flash_omni/talker_module.py b/vllm_omni/model_executor/models/ming_flash_omni/talker_module.py index ee2b6b2a1..360a4f25a 100644 --- a/vllm_omni/model_executor/models/ming_flash_omni/talker_module.py +++ b/vllm_omni/model_executor/models/ming_flash_omni/talker_module.py @@ -963,7 +963,7 @@ class MingAudioGenerator: ) if step > min_new_token and stop_prob > 0.5: - logger.info("Stopping at step %d with stop_prob=%.4f", step, stop_prob) + logger.debug("Stopping at step %d with stop_prob=%.4f", step, stop_prob) break return all_latents diff --git a/vllm_omni/model_executor/stage_configs/bailingmm_moe_v2_lite.yaml b/vllm_omni/model_executor/stage_configs/bailingmm_moe_v2_lite.yaml deleted file mode 100644 index b7d0aeeb7..000000000 --- a/vllm_omni/model_executor/stage_configs/bailingmm_moe_v2_lite.yaml +++ /dev/null @@ -1,46 +0,0 @@ -# Stage config for Ming-flash-omni-2.0 -# Stage 0: Thinker (Multimodal understanding + text generation) -# Stage 1a: Image Generator (Text embeddings -> PIL image) -# Stage 1b: Talker (Text embeddings -> audio waveform) - -async_chunk: false -stage_args: - - stage_id: 0 - stage_type: llm - runtime: - devices: "0,1,2,3" - max_batch_size: 1 - engine_args: - model_stage: thinker - model_arch: MingFlashOmniForConditionalGeneration - # tokenizer_subdir: talker/llm - worker_type: ar - scheduler_cls: vllm_omni.core.sched.omni_ar_scheduler.OmniARScheduler - gpu_memory_utilization: 0.9 - enforce_eager: false - trust_remote_code: true - engine_output_type: latent - distributed_executor_backend: "mp" - enable_prefix_caching: false - max_num_batched_tokens: 32768 - tensor_parallel_size: 4 # Use 4 GPUs for MoE model - # pipeline_parallel_size: 4 - hf_config_name: llm_config - compilation_config: - pass_config: - # there's a version mismatch regarding vllm and flashinfer - # disable fuse allreduce for now - fuse_allreduce_rms: false - final_output: true # Can output text directly - final_output_type: text - is_comprehension: true - default_sampling_params: - temperature: 0.4 - top_p: 0.9 - max_tokens: 2048 - repetition_penalty: 1.05 - seed: 42 - detokenize: true - - # Future Stage 1a: Image Generator (Optional - not yet implemented) - # Future Stage 1b: Talker/TTS (Optional - not yet implemented) diff --git a/vllm_omni/model_executor/stage_configs/ming_flash_omni.yaml b/vllm_omni/model_executor/stage_configs/ming_flash_omni.yaml deleted file mode 100644 index 1e07d16a0..000000000 --- a/vllm_omni/model_executor/stage_configs/ming_flash_omni.yaml +++ /dev/null @@ -1,70 +0,0 @@ -# Multi-stage config for Ming-flash-omni-2.0: Thinker + Talker -# Stage 0: Thinker (multimodal understanding → text generation) -# Stage 1: Talker (text → audio waveform via CFM + AudioVAE) - -stage_args: - - stage_id: 0 - stage_type: llm - runtime: - devices: "0,1,2,3" - max_batch_size: 1 - engine_args: - model_stage: thinker - model_arch: MingFlashOmniForConditionalGeneration - worker_type: ar - scheduler_cls: vllm_omni.core.sched.omni_ar_scheduler.OmniARScheduler - gpu_memory_utilization: 0.74 - enforce_eager: false - trust_remote_code: true - # Ming Thinker -> talker bridging reads the detokenized text via - # source_output.outputs[0].text rather than hidden states - engine_output_type: text - distributed_executor_backend: "mp" - enable_prefix_caching: false - max_num_batched_tokens: 32768 - tensor_parallel_size: 4 - hf_config_name: llm_config - compilation_config: - pass_config: - # there's a version mismatch regarding vllm and flashinfer - # disable fuse allreduce for now - fuse_allreduce_rms: false - final_output: true - final_output_type: text - is_comprehension: true - default_sampling_params: - temperature: 0.4 - top_p: 0.9 - max_tokens: 2048 - repetition_penalty: 1.05 - detokenize: true - ignore_eos: false - - - stage_id: 1 - stage_type: llm - runtime: - devices: "3" - max_batch_size: 1 - engine_args: - # Use the standalone talker class - model_stage: ming_tts - model_arch: MingFlashOmniTalkerForConditionalGeneration - worker_cls: vllm_omni.worker.gpu_generation_worker.GPUGenerationWorker - scheduler_cls: vllm_omni.core.sched.omni_generation_scheduler.OmniGenerationScheduler - gpu_memory_utilization: 0.18 - enforce_eager: true - trust_remote_code: true - engine_output_type: audio - enable_prefix_caching: false - max_num_batched_tokens: 1000000 - tokenizer_subdir: talker/llm - # The HF repo ships BailingMM2Config (thinker-only) at root, - # OmniModelConfig treats that as "stage does not share outer mrope" - hf_config_name: talker_config - engine_input_source: [0] - custom_process_input_func: vllm_omni.model_executor.stage_input_processors.ming_flash_omni.thinker2talker - final_output: true - final_output_type: audio - default_sampling_params: - temperature: 0.0 - max_tokens: 1 diff --git a/vllm_omni/model_executor/stage_configs/ming_flash_omni_tts.yaml b/vllm_omni/model_executor/stage_configs/ming_flash_omni_tts.yaml deleted file mode 100644 index 311f79ec5..000000000 --- a/vllm_omni/model_executor/stage_configs/ming_flash_omni_tts.yaml +++ /dev/null @@ -1,32 +0,0 @@ -# Single-stage TTS config for Ming-flash-omni-2.0 Talker -# Stage 0: Talker only (text → audio waveform via CFM + AudioVAE) -# Use this config for standalone TTS deployment without thinker. - -stage_args: - - stage_id: 0 - stage_type: llm - is_comprehension: true - runtime: - devices: "0" - max_batch_size: 1 - engine_args: - model_stage: ming_tts - model_arch: MingFlashOmniTalkerForConditionalGeneration - worker_cls: vllm_omni.worker.gpu_generation_worker.GPUGenerationWorker - scheduler_cls: vllm_omni.core.sched.omni_generation_scheduler.OmniGenerationScheduler - gpu_memory_utilization: 0.8 - enforce_eager: true - trust_remote_code: true - engine_output_type: audio - enable_prefix_caching: false - max_num_batched_tokens: 100000 - tokenizer_subdir: talker/llm - # NOTE: `hf_config_name` for Ming talker acts as a placeholder - # The HF repo ships BailingMM2Config (thinker-only) at root, - # OmniModelConfig treats that as "stage does not share outer mrope" - hf_config_name: talker_config - final_output: true - final_output_type: audio - default_sampling_params: - temperature: 0.0 - max_tokens: 1