[Config Refactor] Migrate Ming and Ming-TTS deploy/pipline configs (#3154)
Signed-off-by: Yuanheng Zhao <jonathan.zhaoyh@gmail.com> Signed-off-by: yuanheng <jonathan.zhaoyh@gmail.com> Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
This commit is contained in:
Yuanheng ZhaoandHongsheng Liu authored and GitHub committed
1 parent
6a837849dd
commit
d49356e4ca
25 files changed
+518
-398
No files matched your search
@@ -4,10 +4,10 @@
|
||||
|
||||
vLLM-Omni supports two deployment modes:
|
||||
|
||||
| Mode | Stage config | Output |
|
||||
|------|-------------|--------|
|
||||
| Thinker only (multimodal understanding) | `ming_flash_omni_thinker.yaml` (default `--omni`) | Text |
|
||||
| Thinker + Talker (omni-speech) | `ming_flash_omni.yaml` | Text + Audio |
|
||||
| Mode | Deploy config | Output |
|
||||
|------|--------------|--------|
|
||||
| Thinker + Talker (omni-speech, default) | `vllm_omni/deploy/ming_flash_omni.yaml` | Text + Audio |
|
||||
| Thinker only (multimodal understanding) | `vllm_omni/deploy/ming_flash_omni_thinker_only.yaml` | Text |
|
||||
|
||||
For standalone TTS (talker only), see [`examples/offline_inference/ming_flash_omni_tts/`](../ming_flash_omni_tts/).
|
||||
|
||||
@@ -15,30 +15,30 @@ For standalone TTS (talker only), see [`examples/offline_inference/ming_flash_om
|
||||
|
||||
Please refer to the [stage configuration documentation](https://docs.vllm.ai/projects/vllm-omni/en/latest/configuration/stage_configs/) to configure memory allocation appropriately for your hardware setup.
|
||||
|
||||
The default `--omni` flag runs thinker only. For omni-speech, pass the two-stage config explicitly:
|
||||
When no `--deploy-config` is passed, the model registry auto-loads the full thinker+talker `vllm_omni/deploy/ming_flash_omni.yaml` (See [Omni-Speech](#omni-speech-thinker--talker)).
|
||||
|
||||
For text-only output without spinning up the talker, pass:
|
||||
|
||||
```bash
|
||||
--stage-configs-path vllm_omni/model_executor/stage_configs/ming_flash_omni.yaml
|
||||
--deploy-config vllm_omni/deploy/ming_flash_omni_thinker_only.yaml
|
||||
```
|
||||
|
||||
## Run examples
|
||||
|
||||
The end-to-end script defaults to built-in assets; pass `--image-path`,
|
||||
`--audio-path`, or `--video-path` to override.
|
||||
The end-to-end script defaults to built-in assets; pass `--image-path`, `--audio-path`, or `--video-path` to override.
|
||||
|
||||
### Multi-Modality Understanding (Standalone Thinker)
|
||||
|
||||
Here we pass thinker-only deploy yaml:
|
||||
|
||||
```bash
|
||||
# Text-only
|
||||
python examples/offline_inference/ming_flash_omni/end2end.py --query-type text
|
||||
|
||||
# Image / audio / video / mixed understanding
|
||||
python examples/offline_inference/ming_flash_omni/end2end.py --query-type use_image
|
||||
python examples/offline_inference/ming_flash_omni/end2end.py --query-type use_audio
|
||||
python examples/offline_inference/ming_flash_omni/end2end.py --query-type use_video --num-frames 16
|
||||
python examples/offline_inference/ming_flash_omni/end2end.py --query-type use_mixed_modalities \
|
||||
--image-path /path/to/image.jpg --audio-path /path/to/audio.wav
|
||||
python examples/offline_inference/ming_flash_omni/end2end.py --deploy-config vllm_omni/deploy/ming_flash_omni_thinker_only.yaml --query-type text
|
||||
python examples/offline_inference/ming_flash_omni/end2end.py --deploy-config vllm_omni/deploy/ming_flash_omni_thinker_only.yaml --query-type use_image
|
||||
python examples/offline_inference/ming_flash_omni/end2end.py --deploy-config vllm_omni/deploy/ming_flash_omni_thinker_only.yaml --query-type use_audio
|
||||
python examples/offline_inference/ming_flash_omni/end2end.py --deploy-config vllm_omni/deploy/ming_flash_omni_thinker_only.yaml --query-type use_video --num-frames 16
|
||||
```
|
||||
|
||||
#### Reasoning (Thinking Mode)
|
||||
### Reasoning (Thinking Mode)
|
||||
|
||||
Reasoning ("detailed thinking on") is applied by the script when
|
||||
`--query-type reasoning` is set. The default prompt matches Ming's cookbook
|
||||
@@ -46,19 +46,21 @@ and expects the reference figure from the upstream repo — see
|
||||
`get_reasoning_query` in `end2end.py`.
|
||||
|
||||
```bash
|
||||
python examples/offline_inference/ming_flash_omni/end2end.py -q reasoning --image-path ./3_0.png
|
||||
python examples/offline_inference/ming_flash_omni/end2end.py \
|
||||
--deploy-config vllm_omni/deploy/ming_flash_omni_thinker_only.yaml \
|
||||
--query-type reasoning \
|
||||
--image-path ./3_0.png
|
||||
```
|
||||
|
||||
### Omni-speech (thinker + talker)
|
||||
### Omni-Speech (Thinker + Talker)
|
||||
|
||||
To enable spoken output, use the two-stage config and request `audio` (or `text,audio`) modalities.
|
||||
The default deploy YAML already runs thinker+talker, so spoken output only requires requesting `audio` (or `text,audio`) modalities.
|
||||
The thinker processes your multimodal input, generates text, then the talker synthesises the response as speech.
|
||||
|
||||
**Audio-only output** (speech response, no text):
|
||||
```bash
|
||||
python examples/offline_inference/ming_flash_omni/end2end.py \
|
||||
--query-type text \
|
||||
--stage-configs-path vllm_omni/model_executor/stage_configs/ming_flash_omni.yaml \
|
||||
--modalities audio \
|
||||
--output-dir output_ming_omni_speech
|
||||
```
|
||||
@@ -67,14 +69,13 @@ python examples/offline_inference/ming_flash_omni/end2end.py \
|
||||
```bash
|
||||
python examples/offline_inference/ming_flash_omni/end2end.py \
|
||||
--query-type use_audio \
|
||||
--stage-configs-path vllm_omni/model_executor/stage_configs/ming_flash_omni.yaml \
|
||||
--modalities text,audio \
|
||||
--output-dir output_ming_omni_speech
|
||||
```
|
||||
|
||||
Generated `.wav` files are saved to `--output-dir` (default `output_ming`), one per request.
|
||||
|
||||
The stage config allocates thinker on GPUs 0–3 and talker on GPU 3 by default. Adjust `devices` in the YAML to match your hardware.
|
||||
The default deploy YAML allocates thinker on GPUs 0–3 and talker on GPU 3 for a common device topology (4 rather than 5 devices that talker on its own device). Adjust `devices` in a copied YAML and pass it via `--deploy-config` to match your hardware or requirements.
|
||||
|
||||
### Modality control
|
||||
|
||||
@@ -84,8 +85,7 @@ The stage config allocates thinker on GPUs 0–3 and talker on GPU 3 by default.
|
||||
| `audio` | Text (internal) | Runs | `<id>.wav` |
|
||||
| `text,audio` | Text | Runs | `<id>.txt` + `<id>.wav` |
|
||||
|
||||
Pass `--stage-configs-path /path/to/your_config.yaml` to any of the commands
|
||||
above to override the stage config.
|
||||
Pass `--deploy-config /path/to/your_deploy.yaml` to any of the commands above to override the bundled deploy config.
|
||||
|
||||
## Online serving
|
||||
|
||||
|
||||
@@ -20,6 +20,7 @@ from vllm.multimodal.media.audio import load_audio
|
||||
from vllm.utils.argparse_utils import FlexibleArgumentParser
|
||||
|
||||
import vllm_omni
|
||||
from vllm_omni.engine.arg_utils import nullify_stage_engine_defaults
|
||||
from vllm_omni.entrypoints.omni import Omni
|
||||
|
||||
# Imports the processor also registers itself
|
||||
@@ -302,14 +303,7 @@ def main(args):
|
||||
else:
|
||||
query_result = query_func(processor)
|
||||
|
||||
# Initialize Omni (with thinker-only stage config)
|
||||
omni = Omni(
|
||||
model=MODEL_NAME,
|
||||
stage_configs_path=args.stage_configs_path,
|
||||
log_stats=args.log_stats,
|
||||
init_timeout=args.init_timeout,
|
||||
stage_init_timeout=args.stage_init_timeout,
|
||||
)
|
||||
omni = Omni.from_cli_args(args, model=MODEL_NAME)
|
||||
|
||||
# Thinker sampling params
|
||||
thinker_sampling_params = SamplingParams(
|
||||
@@ -410,10 +404,10 @@ def parse_args():
|
||||
help="Query type.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--stage-configs-path",
|
||||
"--deploy-config",
|
||||
type=str,
|
||||
default=None,
|
||||
help="Path to a stage configs YAML file.",
|
||||
help="Path to a deploy YAML; leave unset to auto-load full thinker+talker. Pass custom for text-only",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--log-stats",
|
||||
@@ -499,6 +493,7 @@ def parse_args():
|
||||
help="Output directory for results.",
|
||||
)
|
||||
|
||||
nullify_stage_engine_defaults(parser)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
|
||||
@@ -3,7 +3,7 @@
|
||||
This example runs **Ming-flash-omni-2.0 talker-only** offline inference with:
|
||||
|
||||
- `model`: `Jonathan1909/Ming-flash-omni-2.0`
|
||||
- `stage config`: `vllm_omni/model_executor/stage_configs/ming_flash_omni_tts.yaml`
|
||||
- `deploy config`: `vllm_omni/deploy/ming_flash_omni_tts.yaml`
|
||||
|
||||
It follows the Ming cookbook parameter style:
|
||||
|
||||
@@ -37,7 +37,7 @@ python examples/offline_inference/ming_flash_omni_tts/end2end.py --case basic
|
||||
- `--text`: override default text in the selected case
|
||||
- `--output`: custom output wav path
|
||||
- `--model`: local model path or HF repo id
|
||||
- `--stage-configs-path`: custom talker stage config path
|
||||
- `--deploy-config`: custom talker deploy YAML path
|
||||
- `--log-stats`: enable runtime stats logs
|
||||
|
||||
## Notes
|
||||
|
||||
@@ -9,6 +9,7 @@ from vllm.utils.argparse_utils import FlexibleArgumentParser
|
||||
|
||||
os.environ["VLLM_WORKER_MULTIPROC_METHOD"] = "spawn"
|
||||
|
||||
from vllm_omni.engine.arg_utils import nullify_stage_engine_defaults
|
||||
from vllm_omni.entrypoints.omni import Omni
|
||||
from vllm_omni.inputs.data import OmniTokensPrompt
|
||||
from vllm_omni.model_executor.models.ming_flash_omni.prompt_utils import (
|
||||
@@ -17,7 +18,6 @@ from vllm_omni.model_executor.models.ming_flash_omni.prompt_utils import (
|
||||
)
|
||||
|
||||
MODEL_NAME = "Jonathan1909/Ming-flash-omni-2.0"
|
||||
DEFAULT_STAGE_CONFIG = "vllm_omni/model_executor/stage_configs/ming_flash_omni_tts.yaml"
|
||||
|
||||
|
||||
def get_messages(case: str, text_override: str | None) -> dict[str, Any]:
|
||||
@@ -71,10 +71,10 @@ def parse_args():
|
||||
parser = FlexibleArgumentParser(description="Ming-flash-omni standalone talker offline e2e example")
|
||||
parser.add_argument("--model", type=str, default=MODEL_NAME, help="Model name or local path.")
|
||||
parser.add_argument(
|
||||
"--stage-configs-path",
|
||||
"--deploy-config",
|
||||
type=str,
|
||||
default=DEFAULT_STAGE_CONFIG,
|
||||
help="Path to stage configs yaml for standalone talker deployment.",
|
||||
default="vllm_omni/deploy/ming_flash_omni_tts.yaml",
|
||||
help="Path to a custom deploy YAML for the TTS deployment. ",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--case",
|
||||
@@ -88,19 +88,15 @@ def parse_args():
|
||||
parser.add_argument("--log-stats", action="store_true", default=False, help="Enable stats logging.")
|
||||
parser.add_argument("--init-timeout", type=int, default=600, help="Engine init timeout in seconds.")
|
||||
parser.add_argument("--stage-init-timeout", type=int, default=300, help="Single stage init timeout in seconds.")
|
||||
|
||||
nullify_stage_engine_defaults(parser)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def main():
|
||||
args = parse_args()
|
||||
|
||||
omni = Omni(
|
||||
model=args.model,
|
||||
stage_configs_path=args.stage_configs_path,
|
||||
log_stats=args.log_stats,
|
||||
init_timeout=args.init_timeout,
|
||||
stage_init_timeout=args.stage_init_timeout,
|
||||
)
|
||||
omni = Omni.from_cli_args(args, model=args.model)
|
||||
|
||||
messages = get_messages(args.case, args.text)
|
||||
decode_args = {
|
||||
|
||||
@@ -8,8 +8,8 @@ Please refer to [README.md](../../../README.md)
|
||||
|
||||
| Mode | Launch command | Output |
|
||||
|------|---------------|--------|
|
||||
| Thinker only (multimodal understanding) | `vllm serve ... --omni` | Text |
|
||||
| Thinker + Talker (omni-speech) | `vllm serve ... --omni --stage-configs-path ming_flash_omni.yaml` | Text + Audio |
|
||||
| Thinker + Talker (omni-speech, default) | `vllm serve ... --omni` | Text + Audio |
|
||||
| Thinker only (multimodal understanding) | `vllm serve ... --omni --deploy-config vllm_omni/deploy/ming_flash_omni_thinker_only.yaml` | Text |
|
||||
|
||||
For standalone TTS (talker only), see [`examples/online_serving/ming_flash_omni_tts/`](../ming_flash_omni_tts/).
|
||||
|
||||
@@ -17,18 +17,20 @@ For standalone TTS (talker only), see [`examples/online_serving/ming_flash_omni_
|
||||
|
||||
### Launch the Server
|
||||
|
||||
**Thinker only (text output):**
|
||||
**Thinker + Talker (omni-speech, text + audio output):**
|
||||
```bash
|
||||
vllm serve Jonathan1909/Ming-flash-omni-2.0 --omni --port 8091
|
||||
```
|
||||
|
||||
**Thinker + Talker (omni-speech, text + audio output):**
|
||||
The model registry auto-loads corresponding deploy yaml.
|
||||
|
||||
**Thinker-only (text output):**
|
||||
```bash
|
||||
vllm serve Jonathan1909/Ming-flash-omni-2.0 --omni --port 8091 \
|
||||
--stage-configs-path vllm_omni/model_executor/stage_configs/ming_flash_omni.yaml
|
||||
--deploy-config vllm_omni/deploy/ming_flash_omni_thinker_only.yaml
|
||||
```
|
||||
|
||||
Pass `--stage-configs-path /path/to/your_config.yaml` to use a custom stage
|
||||
Pass `--deploy-config /path/to/your_deploy.yaml` to use a custom deploy
|
||||
config.
|
||||
|
||||
### Send Multi-modal Request
|
||||
|
||||
@@ -18,17 +18,6 @@ if [[ ! "$QUERY_TYPE" =~ ^(text|use_audio|use_image|use_video|use_mixed_modaliti
|
||||
exit 1
|
||||
fi
|
||||
|
||||
thinker_sampling_params='{
|
||||
"temperature": 0.4,
|
||||
"top_p": 0.9,
|
||||
"top_k": -1,
|
||||
"max_tokens": 16384,
|
||||
"seed": 42,
|
||||
"detokenize": true,
|
||||
"repetition_penalty": 1.05
|
||||
}'
|
||||
# Above is optional, it has a default setting in stage_configs of the corresponding model.
|
||||
|
||||
# Define URLs for assets
|
||||
MARY_HAD_LAMB_AUDIO_URL="https://vllm-public-assets.s3.us-west-2.amazonaws.com/multimodal_asset/mary_had_lamb.ogg"
|
||||
CHERRY_BLOSSOM_IMAGE_URL="https://vllm-public-assets.s3.us-west-2.amazonaws.com/vision_model_images/cherry_blossom.jpg"
|
||||
@@ -114,9 +103,6 @@ echo ""
|
||||
request_body=$(cat <<EOF
|
||||
{
|
||||
"model": "Jonathan1909/Ming-flash-omni-2.0",
|
||||
"sampling_params_list": [
|
||||
$thinker_sampling_params
|
||||
],
|
||||
"modalities": ["text"],
|
||||
"messages": [
|
||||
{
|
||||
|
||||
@@ -5,7 +5,7 @@ This directory contains online e2e examples for **Ming-flash-omni-2.0 standalone
|
||||
Server uses:
|
||||
|
||||
- `model`: `Jonathan1909/Ming-flash-omni-2.0`
|
||||
- `stage config`: `vllm_omni/model_executor/stage_configs/ming_flash_omni_tts.yaml`
|
||||
- `deploy config`: `vllm_omni/deploy/ming_flash_omni_tts.yaml`
|
||||
|
||||
## Launch the Server
|
||||
|
||||
@@ -18,7 +18,7 @@ Equivalent manual command:
|
||||
|
||||
```bash
|
||||
vllm serve Jonathan1909/Ming-flash-omni-2.0 \
|
||||
--stage-configs-path vllm_omni/model_executor/stage_configs/ming_flash_omni_tts.yaml \
|
||||
--deploy-config vllm_omni/deploy/ming_flash_omni_tts.yaml \
|
||||
--host 0.0.0.0 \
|
||||
--port 8091 \
|
||||
--trust-remote-code \
|
||||
|
||||
@@ -12,13 +12,13 @@ set -e
|
||||
MODEL="${MODEL:-Jonathan1909/Ming-flash-omni-2.0}"
|
||||
HOST="${HOST:-0.0.0.0}"
|
||||
PORT="${PORT:-8091}"
|
||||
STAGE_CONFIG="${STAGE_CONFIG:-vllm_omni/model_executor/stage_configs/ming_flash_omni_tts.yaml}"
|
||||
DEPLOY_CONFIG="${DEPLOY_CONFIG:-vllm_omni/deploy/ming_flash_omni_tts.yaml}"
|
||||
|
||||
echo "Starting Ming standalone TTS server with model: $MODEL"
|
||||
echo "Stage config: $STAGE_CONFIG"
|
||||
echo "Deploy config: $DEPLOY_CONFIG"
|
||||
|
||||
vllm serve "$MODEL" \
|
||||
--stage-configs-path "$STAGE_CONFIG" \
|
||||
--deploy-config "$DEPLOY_CONFIG" \
|
||||
--host "$HOST" \
|
||||
--port "$PORT" \
|
||||
--trust-remote-code \
|
||||
|
||||
@@ -46,7 +46,7 @@ Adjust `devices` in the YAML to match your hardware.
|
||||
- OS: Linux
|
||||
- Python: 3.10+
|
||||
- CUDA Driver Version: 590.48.01
|
||||
- CUDA 12.5
|
||||
- CUDA 13.0
|
||||
- vLLM version: 0.19.0
|
||||
- vLLM-Omni version or commit: 0.19.0rc1
|
||||
|
||||
@@ -64,7 +64,6 @@ Thinker + talker (text and/or audio output):
|
||||
vllm serve Jonathan1909/Ming-flash-omni-2.0 \
|
||||
--omni \
|
||||
--port 8091 \
|
||||
--stage-configs-path vllm_omni/model_executor/stage_configs/ming_flash_omni.yaml \
|
||||
--log-stats
|
||||
```
|
||||
|
||||
@@ -162,7 +161,7 @@ The bundled `ming_flash_omni_tts.yaml` runs the talker on a single GPU and expos
|
||||
- OS: Linux
|
||||
- Python: 3.10+
|
||||
- CUDA Driver Version: 590.48.01
|
||||
- CUDA 12.5
|
||||
- CUDA 13.0
|
||||
- vLLM version: 0.19.0
|
||||
- vLLM-Omni version or commit: 0.19.0rc1
|
||||
|
||||
@@ -171,7 +170,7 @@ The bundled `ming_flash_omni_tts.yaml` runs the talker on a single GPU and expos
|
||||
```bash
|
||||
vllm serve Jonathan1909/Ming-flash-omni-2.0 \
|
||||
--omni \
|
||||
--stage-configs-path vllm_omni/model_executor/stage_configs/ming_flash_omni_tts.yaml \
|
||||
--deploy-config vllm_omni/deploy/ming_flash_omni_tts.yaml \
|
||||
--port 8091 \
|
||||
--log-stats
|
||||
```
|
||||
|
||||
@@ -6,8 +6,6 @@ import os
|
||||
os.environ["VLLM_WORKER_MULTIPROC_METHOD"] = "spawn"
|
||||
os.environ["VLLM_TEST_CLEAN_GPU_MEMORY"] = "0"
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from tests.helpers.mark import hardware_test
|
||||
@@ -16,7 +14,7 @@ from tests.helpers.media import (
|
||||
generate_synthetic_image,
|
||||
generate_synthetic_video,
|
||||
)
|
||||
from tests.helpers.stage_config import modify_stage_config
|
||||
from tests.helpers.stage_config import get_deploy_config_path, modify_stage_config
|
||||
|
||||
models = ["Jonathan1909/Ming-flash-omni-2.0"]
|
||||
|
||||
@@ -35,48 +33,27 @@ def build_prompt(user_text: str) -> str:
|
||||
)
|
||||
|
||||
|
||||
_CI_DEPLOY_THINKER_ONLY = get_deploy_config_path("ci/ming_flash_omni_thinker_only.yaml")
|
||||
_CI_DEPLOY = get_deploy_config_path("ci/ming_flash_omni.yaml")
|
||||
|
||||
|
||||
def get_eager_config_thinker():
|
||||
return modify_stage_config(_CI_DEPLOY_THINKER_ONLY, updates={"stages": {0: {"enforce_eager": True}}})
|
||||
|
||||
|
||||
def get_eager_config():
|
||||
path = modify_stage_config(
|
||||
str(Path(__file__).parent.parent / "stage_configs" / "bailingmm_moe_v2_lite_thinker_only_ci.yaml"),
|
||||
updates={
|
||||
"stage_args": {
|
||||
0: {
|
||||
"engine_args.enforce_eager": "true",
|
||||
},
|
||||
},
|
||||
},
|
||||
)
|
||||
return path
|
||||
|
||||
|
||||
def get_eager_tts_config():
|
||||
"""Thinker+talker CI config with enforce_eager on the thinker stage."""
|
||||
path = modify_stage_config(
|
||||
str(Path(__file__).parent.parent / "stage_configs" / "bailingmm_moe_v2_lite_ci.yaml"),
|
||||
updates={
|
||||
"stage_args": {
|
||||
0: {
|
||||
"engine_args.enforce_eager": "true",
|
||||
},
|
||||
},
|
||||
},
|
||||
)
|
||||
return path
|
||||
return modify_stage_config(_CI_DEPLOY, updates={"stages": {0: {"enforce_eager": True}}})
|
||||
|
||||
|
||||
# Thinker-only config — used by text-output tests.
|
||||
stage_configs = [get_eager_config()]
|
||||
test_params = [(model, stage_config) for model in models for stage_config in stage_configs]
|
||||
|
||||
# Thinker+talker config — used by audio-output tests.
|
||||
stage_configs_tts = [get_eager_tts_config()]
|
||||
test_params_tts = [(model, stage_config) for model in models for stage_config in stage_configs_tts]
|
||||
test_params_thinker = [(m, c) for m in models for c in [get_eager_config_thinker()]]
|
||||
test_params = [(m, c) for m in models for c in [get_eager_config()]]
|
||||
|
||||
|
||||
@pytest.mark.core_model
|
||||
@pytest.mark.omni
|
||||
@hardware_test(res={"cuda": "H100"}, num_cards=4)
|
||||
@pytest.mark.parametrize("omni_runner", test_params, indirect=True)
|
||||
@pytest.mark.parametrize("omni_runner", test_params_thinker, indirect=True)
|
||||
def test_text_to_text(omni_runner, omni_runner_handler) -> None:
|
||||
"""
|
||||
Test text-only input processing and text output generation.
|
||||
@@ -92,7 +69,7 @@ def test_text_to_text(omni_runner, omni_runner_handler) -> None:
|
||||
@pytest.mark.core_model
|
||||
@pytest.mark.omni
|
||||
@hardware_test(res={"cuda": "H100"}, num_cards=4)
|
||||
@pytest.mark.parametrize("omni_runner", test_params, indirect=True)
|
||||
@pytest.mark.parametrize("omni_runner", test_params_thinker, indirect=True)
|
||||
def test_image_to_text(omni_runner, omni_runner_handler) -> None:
|
||||
"""
|
||||
Test image understanding with text output.
|
||||
@@ -109,7 +86,7 @@ def test_image_to_text(omni_runner, omni_runner_handler) -> None:
|
||||
@pytest.mark.core_model
|
||||
@pytest.mark.omni
|
||||
@hardware_test(res={"cuda": "H100"}, num_cards=4)
|
||||
@pytest.mark.parametrize("omni_runner", test_params, indirect=True)
|
||||
@pytest.mark.parametrize("omni_runner", test_params_thinker, indirect=True)
|
||||
def test_audio_to_text(omni_runner, omni_runner_handler) -> None:
|
||||
"""
|
||||
Test audio understanding with text output.
|
||||
@@ -128,7 +105,7 @@ def test_audio_to_text(omni_runner, omni_runner_handler) -> None:
|
||||
@pytest.mark.core_model
|
||||
@pytest.mark.omni
|
||||
@hardware_test(res={"cuda": "H100"}, num_cards=4)
|
||||
@pytest.mark.parametrize("omni_runner", test_params, indirect=True)
|
||||
@pytest.mark.parametrize("omni_runner", test_params_thinker, indirect=True)
|
||||
def test_video_to_text(omni_runner, omni_runner_handler) -> None:
|
||||
"""
|
||||
Test video understanding with text output.
|
||||
@@ -145,7 +122,7 @@ def test_video_to_text(omni_runner, omni_runner_handler) -> None:
|
||||
@pytest.mark.core_model
|
||||
@pytest.mark.omni
|
||||
@hardware_test(res={"cuda": "H100"}, num_cards=4)
|
||||
@pytest.mark.parametrize("omni_runner", test_params, indirect=True)
|
||||
@pytest.mark.parametrize("omni_runner", test_params_thinker, indirect=True)
|
||||
def test_mixed_to_text(omni_runner, omni_runner_handler) -> None:
|
||||
"""
|
||||
Test mixed modality input (image + audio) with text output.
|
||||
@@ -165,7 +142,7 @@ def test_mixed_to_text(omni_runner, omni_runner_handler) -> None:
|
||||
@pytest.mark.advanced_model
|
||||
@pytest.mark.omni
|
||||
@hardware_test(res={"cuda": "H100"}, num_cards=4)
|
||||
@pytest.mark.parametrize("omni_runner", test_params_tts, indirect=True)
|
||||
@pytest.mark.parametrize("omni_runner", test_params, indirect=True)
|
||||
def test_text_to_audio(omni_runner, omni_runner_handler) -> None:
|
||||
"""
|
||||
Test text input with audio output via the thinker+talker pipeline.
|
||||
@@ -181,7 +158,7 @@ def test_text_to_audio(omni_runner, omni_runner_handler) -> None:
|
||||
@pytest.mark.advanced_model
|
||||
@pytest.mark.omni
|
||||
@hardware_test(res={"cuda": "H100"}, num_cards=4)
|
||||
@pytest.mark.parametrize("omni_runner", test_params_tts, indirect=True)
|
||||
@pytest.mark.parametrize("omni_runner", test_params, indirect=True)
|
||||
def test_image_to_audio(omni_runner, omni_runner_handler) -> None:
|
||||
"""
|
||||
Test image + text input with audio output via the thinker+talker pipeline.
|
||||
|
||||
@@ -6,7 +6,6 @@ Tests multimodal understanding via OpenAI-compatible API.
|
||||
"""
|
||||
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
@@ -17,29 +16,17 @@ from tests.helpers.media import (
|
||||
generate_synthetic_video,
|
||||
)
|
||||
from tests.helpers.runtime import OmniServerParams, dummy_messages_from_mix_data
|
||||
from tests.helpers.stage_config import modify_stage_config
|
||||
from tests.helpers.stage_config import get_deploy_config_path
|
||||
|
||||
os.environ["VLLM_WORKER_MULTIPROC_METHOD"] = "spawn"
|
||||
os.environ["VLLM_TEST_CLEAN_GPU_MEMORY"] = "0"
|
||||
|
||||
models = ["Jonathan1909/Ming-flash-omni-2.0"]
|
||||
|
||||
# Use thinker-only topology to test text outputs
|
||||
_CI_DEPLOY = get_deploy_config_path("ci/ming_flash_omni_thinker_only.yaml")
|
||||
|
||||
def get_eager_config():
|
||||
path = modify_stage_config(
|
||||
str(Path(__file__).parent.parent / "stage_configs" / "bailingmm_moe_v2_lite_ci.yaml"),
|
||||
updates={
|
||||
"stage_args": {
|
||||
0: {
|
||||
"engine_args.enforce_eager": "true",
|
||||
},
|
||||
},
|
||||
},
|
||||
)
|
||||
return path
|
||||
|
||||
|
||||
stage_configs = [get_eager_config()]
|
||||
stage_configs = [_CI_DEPLOY]
|
||||
|
||||
# Create parameter combinations for model and stage config
|
||||
test_params = [
|
||||
|
||||
@@ -1,75 +0,0 @@
|
||||
# CI stage config for Ming-flash-omni-2.0 thinker+talker pipeline.
|
||||
# Stage 0: Thinker (multimodal understanding -> text generation)
|
||||
# Stage 1: Talker (text -> audio waveform via CFM + AudioVAE)
|
||||
#
|
||||
# The following config has been verified on 4x H100-80G GPUs
|
||||
stage_args:
|
||||
- stage_id: 0
|
||||
stage_type: llm
|
||||
runtime:
|
||||
devices: "0,1,2,3"
|
||||
max_batch_size: 1
|
||||
engine_args:
|
||||
model_stage: thinker
|
||||
model_arch: MingFlashOmniForConditionalGeneration
|
||||
worker_type: ar
|
||||
scheduler_cls: vllm_omni.core.sched.omni_ar_scheduler.OmniARScheduler
|
||||
gpu_memory_utilization: 0.74
|
||||
enforce_eager: false
|
||||
trust_remote_code: true
|
||||
# Ming Thinker -> talker bridge reads detokenised text from
|
||||
# source_output.outputs[0].text (not hidden states).
|
||||
engine_output_type: text
|
||||
distributed_executor_backend: "mp"
|
||||
enable_prefix_caching: false
|
||||
max_num_batched_tokens: 32768
|
||||
max_model_len: 32768
|
||||
tensor_parallel_size: 4
|
||||
hf_config_name: llm_config
|
||||
load_format: dummy
|
||||
mm_processor_cache_gb: 0
|
||||
compilation_config:
|
||||
pass_config:
|
||||
# Disable fused all-reduce to avoid a vllm/flashinfer version mismatch.
|
||||
fuse_allreduce_rms: false
|
||||
final_output: true
|
||||
final_output_type: text
|
||||
is_comprehension: true
|
||||
default_sampling_params:
|
||||
temperature: 0.0
|
||||
top_p: 0.9
|
||||
max_tokens: 100
|
||||
repetition_penalty: 1.05
|
||||
seed: 42
|
||||
detokenize: true
|
||||
ignore_eos: false
|
||||
|
||||
- stage_id: 1
|
||||
stage_type: llm
|
||||
runtime:
|
||||
devices: "3"
|
||||
max_batch_size: 1
|
||||
engine_args:
|
||||
model_stage: ming_tts
|
||||
model_arch: MingFlashOmniTalkerForConditionalGeneration
|
||||
worker_cls: vllm_omni.worker.gpu_generation_worker.GPUGenerationWorker
|
||||
scheduler_cls: vllm_omni.core.sched.omni_generation_scheduler.OmniGenerationScheduler
|
||||
gpu_memory_utilization: 0.18
|
||||
enforce_eager: true
|
||||
trust_remote_code: true
|
||||
engine_output_type: audio
|
||||
enable_prefix_caching: false
|
||||
max_num_batched_tokens: 1000000
|
||||
tokenizer_subdir: talker/llm
|
||||
# The HF repo ships BailingMM2Config (thinker-only) at root;
|
||||
# OmniModelConfig treats that as "stage does not share outer mrope".
|
||||
hf_config_name: talker_config
|
||||
load_format: dummy
|
||||
engine_input_source: [0]
|
||||
custom_process_input_func: vllm_omni.model_executor.stage_input_processors.ming_flash_omni.thinker2talker
|
||||
final_output: true
|
||||
final_output_type: audio
|
||||
default_sampling_params:
|
||||
temperature: 0.0
|
||||
max_tokens: 1
|
||||
seed: 42
|
||||
@@ -1,35 +0,0 @@
|
||||
# Thinker stage only
|
||||
stage_args:
|
||||
- stage_id: 0
|
||||
stage_type: llm
|
||||
runtime:
|
||||
devices: "0,1,2,3"
|
||||
max_batch_size: 1
|
||||
engine_args:
|
||||
model_stage: thinker
|
||||
model_arch: MingFlashOmniForConditionalGeneration
|
||||
worker_type: ar
|
||||
scheduler_cls: vllm_omni.core.sched.omni_ar_scheduler.OmniARScheduler
|
||||
gpu_memory_utilization: 0.9
|
||||
enforce_eager: false
|
||||
trust_remote_code: true
|
||||
engine_output_type: latent
|
||||
distributed_executor_backend: "mp"
|
||||
enable_prefix_caching: false
|
||||
max_num_batched_tokens: 32768
|
||||
max_model_len: 32768
|
||||
tensor_parallel_size: 4
|
||||
hf_config_name: llm_config
|
||||
load_format: dummy
|
||||
mm_processor_cache_gb: 0
|
||||
final_output: true
|
||||
final_output_type: text
|
||||
is_comprehension: true
|
||||
default_sampling_params:
|
||||
temperature: 0.4
|
||||
top_p: 0.9
|
||||
max_tokens: 100
|
||||
repetition_penalty: 1.05
|
||||
seed: 42
|
||||
detokenize: true
|
||||
ignore_eos: false
|
||||
@@ -484,6 +484,52 @@ _CI_OVERLAYS: dict[str, dict[str, Any]] = {
|
||||
},
|
||||
},
|
||||
},
|
||||
"ming_flash_omni": {
|
||||
"base_config": "ming_flash_omni.yaml",
|
||||
"stages": [
|
||||
{
|
||||
"stage_id": 0,
|
||||
"max_num_seqs": 1,
|
||||
"gpu_memory_utilization": 0.74,
|
||||
"max_model_len": 16384,
|
||||
"max_num_batched_tokens": 16384,
|
||||
"mm_processor_cache_gb": 0,
|
||||
"skip_mm_profiling": True,
|
||||
"enable_flashinfer_autotune": False,
|
||||
"load_format": "dummy",
|
||||
"default_sampling_params": {
|
||||
"temperature": 0.0,
|
||||
"max_tokens": 100,
|
||||
},
|
||||
},
|
||||
{
|
||||
"stage_id": 1,
|
||||
"max_num_seqs": 1,
|
||||
"gpu_memory_utilization": 0.18,
|
||||
"load_format": "dummy",
|
||||
},
|
||||
],
|
||||
},
|
||||
"ming_flash_omni_thinker_only": {
|
||||
"base_config": "ming_flash_omni_thinker_only.yaml",
|
||||
"stages": [
|
||||
{
|
||||
"stage_id": 0,
|
||||
"max_num_seqs": 1,
|
||||
"gpu_memory_utilization": 0.9,
|
||||
"max_model_len": 16384,
|
||||
"max_num_batched_tokens": 16384,
|
||||
"mm_processor_cache_gb": 0,
|
||||
"skip_mm_profiling": True,
|
||||
"enable_flashinfer_autotune": False,
|
||||
"load_format": "dummy",
|
||||
"default_sampling_params": {
|
||||
"temperature": 0.4,
|
||||
"max_tokens": 100,
|
||||
},
|
||||
},
|
||||
],
|
||||
},
|
||||
# Single-stage thinker-only topology for the abort test.
|
||||
"qwen2_5_omni_thinker_only": {
|
||||
"async_chunk": False,
|
||||
|
||||
@@ -997,6 +997,162 @@ class TestQwen3TTSPipeline:
|
||||
}
|
||||
|
||||
|
||||
class TestMingFlashOmniPipeline:
|
||||
def test_registered(self):
|
||||
import vllm_omni.model_executor.models.ming_flash_omni.pipeline # noqa: F401
|
||||
|
||||
p = _PIPELINE_REGISTRY.get("ming_flash_omni")
|
||||
assert p is not None
|
||||
assert p.model_arch == "MingFlashOmniForConditionalGeneration"
|
||||
assert len(p.stages) == 2
|
||||
assert p.validate() == []
|
||||
|
||||
def test_thinker_stage(self):
|
||||
import vllm_omni.model_executor.models.ming_flash_omni.pipeline # noqa: F401
|
||||
|
||||
s = _PIPELINE_REGISTRY["ming_flash_omni"].get_stage(0)
|
||||
assert s.model_stage == "thinker"
|
||||
assert s.execution_type == StageExecutionType.LLM_AR
|
||||
assert s.owns_tokenizer is True
|
||||
assert s.requires_multimodal_data is True
|
||||
assert s.engine_output_type == "text"
|
||||
assert s.hf_config_name == "llm_config"
|
||||
assert s.sampling_constraints["detokenize"] is True
|
||||
|
||||
def test_talker_stage(self):
|
||||
import vllm_omni.model_executor.models.ming_flash_omni.pipeline # noqa: F401
|
||||
|
||||
s = _PIPELINE_REGISTRY["ming_flash_omni"].get_stage(1)
|
||||
assert s.model_stage == "ming_tts"
|
||||
assert s.execution_type == StageExecutionType.LLM_GENERATION
|
||||
assert s.input_sources == (0,)
|
||||
assert s.final_output_type == "audio"
|
||||
assert s.engine_output_type == "audio"
|
||||
assert s.hf_config_name == "talker_config"
|
||||
# Per-stage model_arch override (Ming talker has its own self-contained LLM)
|
||||
assert s.model_arch == "MingFlashOmniTalkerForConditionalGeneration"
|
||||
assert s.tokenizer_subdir == "talker/llm"
|
||||
assert s.custom_process_input_func is not None
|
||||
|
||||
def test_talker_stage_processor_wiring_resolves(self):
|
||||
"""The custom_process_input_func string must point to a real callable.
|
||||
|
||||
Lazy string references only fail at first inference otherwise — this
|
||||
catches typos in the pipeline declaration at import / registration time.
|
||||
"""
|
||||
import importlib
|
||||
|
||||
import vllm_omni.model_executor.models.ming_flash_omni.pipeline # noqa: F401
|
||||
|
||||
s = _PIPELINE_REGISTRY["ming_flash_omni"].get_stage(1)
|
||||
module_path, _, attr = s.custom_process_input_func.rpartition(".")
|
||||
module = importlib.import_module(module_path)
|
||||
assert callable(getattr(module, attr))
|
||||
|
||||
def test_tts_pipeline_registered(self):
|
||||
import vllm_omni.model_executor.models.ming_flash_omni.pipeline # noqa: F401
|
||||
|
||||
p = _PIPELINE_REGISTRY.get("ming_flash_omni_tts")
|
||||
assert p is not None
|
||||
assert p.model_arch == "MingFlashOmniTalkerForConditionalGeneration"
|
||||
assert len(p.stages) == 1
|
||||
assert p.validate() == []
|
||||
|
||||
def test_tts_stage(self):
|
||||
import vllm_omni.model_executor.models.ming_flash_omni.pipeline # noqa: F401
|
||||
|
||||
s = _PIPELINE_REGISTRY["ming_flash_omni_tts"].get_stage(0)
|
||||
assert s.model_stage == "ming_tts"
|
||||
assert s.execution_type == StageExecutionType.LLM_GENERATION
|
||||
assert s.input_sources == ()
|
||||
assert s.owns_tokenizer is True
|
||||
assert s.final_output_type == "audio"
|
||||
assert s.engine_output_type == "audio"
|
||||
assert s.hf_config_name == "talker_config"
|
||||
assert s.tokenizer_subdir == "talker/llm"
|
||||
|
||||
def test_full_yaml_loads_and_merges(self):
|
||||
"""deploy/ming_flash_omni.yaml parses and merges with the registered pipeline."""
|
||||
import vllm_omni.model_executor.models.ming_flash_omni.pipeline # noqa: F401
|
||||
from vllm_omni.config.stage_config import load_deploy_config, merge_pipeline_deploy
|
||||
|
||||
deploy_path = Path(__file__).parent.parent / "vllm_omni" / "deploy" / "ming_flash_omni.yaml"
|
||||
if not deploy_path.exists():
|
||||
pytest.skip("ming_flash_omni deploy yaml not found")
|
||||
|
||||
deploy = load_deploy_config(deploy_path)
|
||||
assert len(deploy.stages) == 2
|
||||
assert deploy.async_chunk is False
|
||||
assert deploy.pipeline == "ming_flash_omni"
|
||||
# We won't test stage 0/1 colocation contract here,
|
||||
# as there could exist more variant of custom device setup
|
||||
|
||||
pipeline = _PIPELINE_REGISTRY["ming_flash_omni"]
|
||||
stages = merge_pipeline_deploy(pipeline, deploy)
|
||||
assert len(stages) == 2
|
||||
assert stages[0].yaml_engine_args["model_arch"] == "MingFlashOmniForConditionalGeneration"
|
||||
assert stages[1].yaml_engine_args["model_arch"] == "MingFlashOmniTalkerForConditionalGeneration"
|
||||
|
||||
def test_tts_yaml_loads_and_merges(self):
|
||||
"""deploy/ming_flash_omni_tts.yaml parses and routes to the TTS-only pipeline."""
|
||||
import vllm_omni.model_executor.models.ming_flash_omni.pipeline # noqa: F401
|
||||
from vllm_omni.config.stage_config import load_deploy_config, merge_pipeline_deploy
|
||||
|
||||
deploy_path = Path(__file__).parent.parent / "vllm_omni" / "deploy" / "ming_flash_omni_tts.yaml"
|
||||
if not deploy_path.exists():
|
||||
pytest.skip("ming_flash_omni_tts deploy yaml not found")
|
||||
|
||||
deploy = load_deploy_config(deploy_path)
|
||||
assert len(deploy.stages) == 1
|
||||
assert deploy.pipeline == "ming_flash_omni_tts"
|
||||
|
||||
pipeline = _PIPELINE_REGISTRY["ming_flash_omni_tts"]
|
||||
stages = merge_pipeline_deploy(pipeline, deploy)
|
||||
assert len(stages) == 1
|
||||
assert stages[0].yaml_engine_args["model_arch"] == "MingFlashOmniTalkerForConditionalGeneration"
|
||||
|
||||
def test_thinker_only_pipeline_registered(self):
|
||||
import vllm_omni.model_executor.models.ming_flash_omni.pipeline # noqa: F401
|
||||
|
||||
p = _PIPELINE_REGISTRY.get("ming_flash_omni_thinker_only")
|
||||
assert p is not None
|
||||
assert p.model_arch == "MingFlashOmniForConditionalGeneration"
|
||||
assert len(p.stages) == 1
|
||||
assert p.validate() == []
|
||||
|
||||
def test_thinker_only_stage(self):
|
||||
import vllm_omni.model_executor.models.ming_flash_omni.pipeline # noqa: F401
|
||||
|
||||
s = _PIPELINE_REGISTRY["ming_flash_omni_thinker_only"].get_stage(0)
|
||||
assert s.model_stage == "thinker"
|
||||
assert s.execution_type == StageExecutionType.LLM_AR
|
||||
assert s.input_sources == ()
|
||||
assert s.owns_tokenizer is True
|
||||
assert s.requires_multimodal_data is True
|
||||
assert s.final_output_type == "text"
|
||||
assert s.engine_output_type == "text"
|
||||
assert s.hf_config_name == "llm_config"
|
||||
assert s.sampling_constraints["detokenize"] is True
|
||||
|
||||
def test_thinker_only_yaml_loads_and_merges(self):
|
||||
"""deploy/ming_flash_omni_thinker_only.yaml parses and routes to the thinker-only pipeline."""
|
||||
import vllm_omni.model_executor.models.ming_flash_omni.pipeline # noqa: F401
|
||||
from vllm_omni.config.stage_config import load_deploy_config, merge_pipeline_deploy
|
||||
|
||||
deploy_path = Path(__file__).parent.parent / "vllm_omni" / "deploy" / "ming_flash_omni_thinker_only.yaml"
|
||||
if not deploy_path.exists():
|
||||
pytest.skip("ming_flash_omni_thinker_only deploy yaml not found")
|
||||
|
||||
deploy = load_deploy_config(deploy_path)
|
||||
assert len(deploy.stages) == 1
|
||||
assert deploy.pipeline == "ming_flash_omni_thinker_only"
|
||||
|
||||
pipeline = _PIPELINE_REGISTRY["ming_flash_omni_thinker_only"]
|
||||
stages = merge_pipeline_deploy(pipeline, deploy)
|
||||
assert len(stages) == 1
|
||||
assert stages[0].yaml_engine_args["model_arch"] == "MingFlashOmniForConditionalGeneration"
|
||||
|
||||
|
||||
class TestBaseConfigInheritance:
|
||||
"""Test deploy YAML base_config inheritance."""
|
||||
|
||||
|
||||
@@ -85,6 +85,18 @@ _OMNI_PIPELINES: dict[str, tuple[str, str]] = {
|
||||
"vllm_omni.model_executor.models.fish_speech.pipeline",
|
||||
"FISH_SPEECH_PIPELINE",
|
||||
),
|
||||
"ming_flash_omni": (
|
||||
"vllm_omni.model_executor.models.ming_flash_omni.pipeline",
|
||||
"MING_FLASH_OMNI_PIPELINE",
|
||||
),
|
||||
"ming_flash_omni_tts": (
|
||||
"vllm_omni.model_executor.models.ming_flash_omni.pipeline",
|
||||
"MING_FLASH_OMNI_TTS_PIPELINE",
|
||||
),
|
||||
"ming_flash_omni_thinker_only": (
|
||||
"vllm_omni.model_executor.models.ming_flash_omni.pipeline",
|
||||
"MING_FLASH_OMNI_THINKER_ONLY_PIPELINE",
|
||||
),
|
||||
"moss_tts_nano": (
|
||||
"vllm_omni.model_executor.models.moss_tts_nano.pipeline",
|
||||
"MOSS_TTS_NANO_PIPELINE",
|
||||
|
||||
@@ -0,0 +1,56 @@
|
||||
# Ming-flash-omni-2.0 deploy: thinker + talker.
|
||||
#
|
||||
# Stage 0: Thinker: multimodal understanding -> text generation.
|
||||
# TP=4 (or PP=4) across GPUs 0-3.
|
||||
# Stage 1: Talker: text -> audio waveform via CFM + AudioVAE.
|
||||
# Generation worker, runs eager, colocates on the last GPU.
|
||||
#
|
||||
# Fields omitted from a stage fall back to StageDeployConfig dataclass
|
||||
# defaults (see vllm_omni/config/stage_config.py).
|
||||
# * Stage 0 (thinker) owns GPUs 0-3 with gpu_memory_utilization=0.74,
|
||||
# as we tend to fit talker in one of the devices. (For now,
|
||||
# gpu mem util on different devices are not supported to be set
|
||||
# independently for a single stage.)
|
||||
# * Stage 1 (talker) colocates on GPU 3 with gpu_memory_utilization=0.18.
|
||||
# This custom settings is verified and required on devices with 80GB memory.
|
||||
pipeline: ming_flash_omni
|
||||
async_chunk: false
|
||||
|
||||
trust_remote_code: true
|
||||
distributed_executor_backend: mp
|
||||
enable_prefix_caching: false
|
||||
|
||||
stages:
|
||||
- stage_id: 0
|
||||
max_num_seqs: 1
|
||||
gpu_memory_utilization: 0.74
|
||||
enforce_eager: false
|
||||
tensor_parallel_size: 4
|
||||
max_num_batched_tokens: 32768
|
||||
devices: "0,1,2,3"
|
||||
# disable fused allreduce+rms for flashinfer version mismatch for now
|
||||
compilation_config:
|
||||
pass_config:
|
||||
fuse_allreduce_rms: false
|
||||
default_sampling_params:
|
||||
temperature: 0.4
|
||||
top_p: 0.9
|
||||
max_tokens: 2048
|
||||
repetition_penalty: 1.05
|
||||
detokenize: true
|
||||
ignore_eos: false
|
||||
|
||||
- stage_id: 1
|
||||
max_num_seqs: 1
|
||||
gpu_memory_utilization: 0.18
|
||||
enforce_eager: true
|
||||
max_num_batched_tokens: 1000000
|
||||
devices: "3"
|
||||
default_sampling_params:
|
||||
temperature: 0.0
|
||||
max_tokens: 1
|
||||
|
||||
# Ming-flash-omni-2.0 has only been verified on CUDA devices,
|
||||
# left empty intentionally. Add per-platform overrides here
|
||||
# when other backends (ROCm/XPU/NPU) are verified end-to-end.
|
||||
platforms: {}
|
||||
@@ -0,0 +1,36 @@
|
||||
# Ming-flash-omni-2.0 thinker-only deploy: multimodal understanding -> text.
|
||||
#
|
||||
# Single-stage configuration that drops the talker; useful when only text
|
||||
# output is required (chat / understanding / reasoning)
|
||||
#
|
||||
# Stage 0: Thinker — TP=4 across GPUs 0-3 with full memory budget since no
|
||||
# talker shares the last device.
|
||||
pipeline: ming_flash_omni_thinker_only
|
||||
async_chunk: false
|
||||
|
||||
trust_remote_code: true
|
||||
distributed_executor_backend: mp
|
||||
enable_prefix_caching: false
|
||||
|
||||
stages:
|
||||
- stage_id: 0
|
||||
max_num_seqs: 1
|
||||
gpu_memory_utilization: 0.9
|
||||
enforce_eager: false
|
||||
tensor_parallel_size: 4
|
||||
max_num_batched_tokens: 32768
|
||||
devices: "0,1,2,3"
|
||||
# disable fused allreduce+rms for flashinfer version mismatch for now
|
||||
compilation_config:
|
||||
pass_config:
|
||||
fuse_allreduce_rms: false
|
||||
default_sampling_params:
|
||||
temperature: 0.4
|
||||
top_p: 0.9
|
||||
max_tokens: 2048
|
||||
repetition_penalty: 1.05
|
||||
detokenize: true
|
||||
ignore_eos: false
|
||||
|
||||
# Ming-flash-omni-2.0 has only been verified on CUDA devices
|
||||
platforms: {}
|
||||
@@ -0,0 +1,24 @@
|
||||
# Ming-flash-omni-2.0 standalone TTS deploy: talker only.
|
||||
#
|
||||
# Single-stage configuration for direct text -> audio synthesis
|
||||
# The talker runs as a generation worker on a single GPU
|
||||
# and owns the tokenizer.
|
||||
pipeline: ming_flash_omni_tts
|
||||
async_chunk: false
|
||||
|
||||
trust_remote_code: true
|
||||
enable_prefix_caching: false
|
||||
|
||||
stages:
|
||||
- stage_id: 0
|
||||
max_num_seqs: 1
|
||||
gpu_memory_utilization: 0.8
|
||||
enforce_eager: true
|
||||
max_num_batched_tokens: 100000
|
||||
devices: "0"
|
||||
default_sampling_params:
|
||||
temperature: 0.0
|
||||
max_tokens: 1
|
||||
|
||||
# Ming-flash-omni-2.0 has only been verified on CUDA devices
|
||||
platforms: {}
|
||||
@@ -104,8 +104,8 @@ class MingFlashOmniForConditionalGeneration(
|
||||
"model_arch='MingFlashOmniTalkerForConditionalGeneration' "
|
||||
"directly — the talker has a self-contained LLM that "
|
||||
"tokenises text itself and does not need the multimodal "
|
||||
"preprocess path. See stage_configs/ming_flash_omni.yaml "
|
||||
"stage 1 and stage_configs/ming_flash_omni_tts.yaml."
|
||||
"preprocess path. See vllm_omni/deploy/ming_flash_omni.yaml "
|
||||
"stage 1 and vllm_omni/deploy/ming_flash_omni_tts.yaml."
|
||||
)
|
||||
|
||||
if self.model_stage == "thinker":
|
||||
|
||||
@@ -0,0 +1,106 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||
"""Ming-flash-omni-2.0 pipeline topology (frozen).
|
||||
|
||||
Stage 0: Thinker — multimodal understanding + text generation.
|
||||
Stage 1: Talker — text -> audio waveform via CFM + AudioVAE.
|
||||
|
||||
The thinker -> talker bridge passes the detokenized text rather than
|
||||
hidden states through `ming_flash_omni.thinker2talker`; the talker has a
|
||||
self-contained Qwen2 LLM that retokenizes the string itself.
|
||||
"""
|
||||
|
||||
from vllm_omni.config.stage_config import (
|
||||
PipelineConfig,
|
||||
StageExecutionType,
|
||||
StagePipelineConfig,
|
||||
)
|
||||
|
||||
_PROC = "vllm_omni.model_executor.stage_input_processors.ming_flash_omni"
|
||||
|
||||
|
||||
MING_FLASH_OMNI_PIPELINE = PipelineConfig(
|
||||
model_type="ming_flash_omni",
|
||||
model_arch="MingFlashOmniForConditionalGeneration",
|
||||
# Upstream HF config applies model_type="bailingmm_moe_v2_lite"
|
||||
# (the thinker sub-config name) rather than "ming_flash_omni".
|
||||
# Declare the architectures here explicitly for routing
|
||||
hf_architectures=(
|
||||
"MingFlashOmniForConditionalGeneration",
|
||||
"BailingMM2NativeForConditionalGeneration",
|
||||
),
|
||||
stages=(
|
||||
StagePipelineConfig(
|
||||
stage_id=0,
|
||||
model_stage="thinker",
|
||||
execution_type=StageExecutionType.LLM_AR,
|
||||
input_sources=(),
|
||||
final_output=True,
|
||||
final_output_type="text",
|
||||
owns_tokenizer=True,
|
||||
requires_multimodal_data=True,
|
||||
# Thinker reads the LLM sub-config of BailingMM2Config
|
||||
hf_config_name="llm_config",
|
||||
engine_output_type="text",
|
||||
sampling_constraints={"detokenize": True},
|
||||
),
|
||||
StagePipelineConfig(
|
||||
stage_id=1,
|
||||
model_stage="ming_tts",
|
||||
# Talker is a self-contained Qwen2 LLM + CFM + AudioVAE;
|
||||
# it does not share the thinker backbone, so it gets its own arch.
|
||||
model_arch="MingFlashOmniTalkerForConditionalGeneration",
|
||||
execution_type=StageExecutionType.LLM_GENERATION,
|
||||
input_sources=(0,),
|
||||
final_output=True,
|
||||
final_output_type="audio",
|
||||
hf_config_name="talker_config",
|
||||
engine_output_type="audio",
|
||||
tokenizer_subdir="talker/llm",
|
||||
custom_process_input_func=f"{_PROC}.thinker2talker",
|
||||
),
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
# Standalone TTS variant: talker only.
|
||||
MING_FLASH_OMNI_TTS_PIPELINE = PipelineConfig(
|
||||
model_type="ming_flash_omni_tts",
|
||||
model_arch="MingFlashOmniTalkerForConditionalGeneration",
|
||||
stages=(
|
||||
StagePipelineConfig(
|
||||
stage_id=0,
|
||||
model_stage="ming_tts",
|
||||
execution_type=StageExecutionType.LLM_GENERATION,
|
||||
input_sources=(),
|
||||
final_output=True,
|
||||
final_output_type="audio",
|
||||
owns_tokenizer=True,
|
||||
hf_config_name="talker_config",
|
||||
engine_output_type="audio",
|
||||
tokenizer_subdir="talker/llm",
|
||||
),
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
# Thinker-only variant: multimodal understanding with text output
|
||||
MING_FLASH_OMNI_THINKER_ONLY_PIPELINE = PipelineConfig(
|
||||
model_type="ming_flash_omni_thinker_only",
|
||||
model_arch="MingFlashOmniForConditionalGeneration",
|
||||
stages=(
|
||||
StagePipelineConfig(
|
||||
stage_id=0,
|
||||
model_stage="thinker",
|
||||
execution_type=StageExecutionType.LLM_AR,
|
||||
input_sources=(),
|
||||
final_output=True,
|
||||
final_output_type="text",
|
||||
owns_tokenizer=True,
|
||||
requires_multimodal_data=True,
|
||||
hf_config_name="llm_config",
|
||||
engine_output_type="text",
|
||||
sampling_constraints={"detokenize": True},
|
||||
),
|
||||
),
|
||||
)
|
||||
@@ -963,7 +963,7 @@ class MingAudioGenerator:
|
||||
)
|
||||
|
||||
if step > min_new_token and stop_prob > 0.5:
|
||||
logger.info("Stopping at step %d with stop_prob=%.4f", step, stop_prob)
|
||||
logger.debug("Stopping at step %d with stop_prob=%.4f", step, stop_prob)
|
||||
break
|
||||
|
||||
return all_latents
|
||||
|
||||
@@ -1,46 +0,0 @@
|
||||
# Stage config for Ming-flash-omni-2.0
|
||||
# Stage 0: Thinker (Multimodal understanding + text generation)
|
||||
# Stage 1a: Image Generator (Text embeddings -> PIL image)
|
||||
# Stage 1b: Talker (Text embeddings -> audio waveform)
|
||||
|
||||
async_chunk: false
|
||||
stage_args:
|
||||
- stage_id: 0
|
||||
stage_type: llm
|
||||
runtime:
|
||||
devices: "0,1,2,3"
|
||||
max_batch_size: 1
|
||||
engine_args:
|
||||
model_stage: thinker
|
||||
model_arch: MingFlashOmniForConditionalGeneration
|
||||
# tokenizer_subdir: talker/llm
|
||||
worker_type: ar
|
||||
scheduler_cls: vllm_omni.core.sched.omni_ar_scheduler.OmniARScheduler
|
||||
gpu_memory_utilization: 0.9
|
||||
enforce_eager: false
|
||||
trust_remote_code: true
|
||||
engine_output_type: latent
|
||||
distributed_executor_backend: "mp"
|
||||
enable_prefix_caching: false
|
||||
max_num_batched_tokens: 32768
|
||||
tensor_parallel_size: 4 # Use 4 GPUs for MoE model
|
||||
# pipeline_parallel_size: 4
|
||||
hf_config_name: llm_config
|
||||
compilation_config:
|
||||
pass_config:
|
||||
# there's a version mismatch regarding vllm and flashinfer
|
||||
# disable fuse allreduce for now
|
||||
fuse_allreduce_rms: false
|
||||
final_output: true # Can output text directly
|
||||
final_output_type: text
|
||||
is_comprehension: true
|
||||
default_sampling_params:
|
||||
temperature: 0.4
|
||||
top_p: 0.9
|
||||
max_tokens: 2048
|
||||
repetition_penalty: 1.05
|
||||
seed: 42
|
||||
detokenize: true
|
||||
|
||||
# Future Stage 1a: Image Generator (Optional - not yet implemented)
|
||||
# Future Stage 1b: Talker/TTS (Optional - not yet implemented)
|
||||
@@ -1,70 +0,0 @@
|
||||
# Multi-stage config for Ming-flash-omni-2.0: Thinker + Talker
|
||||
# Stage 0: Thinker (multimodal understanding → text generation)
|
||||
# Stage 1: Talker (text → audio waveform via CFM + AudioVAE)
|
||||
|
||||
stage_args:
|
||||
- stage_id: 0
|
||||
stage_type: llm
|
||||
runtime:
|
||||
devices: "0,1,2,3"
|
||||
max_batch_size: 1
|
||||
engine_args:
|
||||
model_stage: thinker
|
||||
model_arch: MingFlashOmniForConditionalGeneration
|
||||
worker_type: ar
|
||||
scheduler_cls: vllm_omni.core.sched.omni_ar_scheduler.OmniARScheduler
|
||||
gpu_memory_utilization: 0.74
|
||||
enforce_eager: false
|
||||
trust_remote_code: true
|
||||
# Ming Thinker -> talker bridging reads the detokenized text via
|
||||
# source_output.outputs[0].text rather than hidden states
|
||||
engine_output_type: text
|
||||
distributed_executor_backend: "mp"
|
||||
enable_prefix_caching: false
|
||||
max_num_batched_tokens: 32768
|
||||
tensor_parallel_size: 4
|
||||
hf_config_name: llm_config
|
||||
compilation_config:
|
||||
pass_config:
|
||||
# there's a version mismatch regarding vllm and flashinfer
|
||||
# disable fuse allreduce for now
|
||||
fuse_allreduce_rms: false
|
||||
final_output: true
|
||||
final_output_type: text
|
||||
is_comprehension: true
|
||||
default_sampling_params:
|
||||
temperature: 0.4
|
||||
top_p: 0.9
|
||||
max_tokens: 2048
|
||||
repetition_penalty: 1.05
|
||||
detokenize: true
|
||||
ignore_eos: false
|
||||
|
||||
- stage_id: 1
|
||||
stage_type: llm
|
||||
runtime:
|
||||
devices: "3"
|
||||
max_batch_size: 1
|
||||
engine_args:
|
||||
# Use the standalone talker class
|
||||
model_stage: ming_tts
|
||||
model_arch: MingFlashOmniTalkerForConditionalGeneration
|
||||
worker_cls: vllm_omni.worker.gpu_generation_worker.GPUGenerationWorker
|
||||
scheduler_cls: vllm_omni.core.sched.omni_generation_scheduler.OmniGenerationScheduler
|
||||
gpu_memory_utilization: 0.18
|
||||
enforce_eager: true
|
||||
trust_remote_code: true
|
||||
engine_output_type: audio
|
||||
enable_prefix_caching: false
|
||||
max_num_batched_tokens: 1000000
|
||||
tokenizer_subdir: talker/llm
|
||||
# The HF repo ships BailingMM2Config (thinker-only) at root,
|
||||
# OmniModelConfig treats that as "stage does not share outer mrope"
|
||||
hf_config_name: talker_config
|
||||
engine_input_source: [0]
|
||||
custom_process_input_func: vllm_omni.model_executor.stage_input_processors.ming_flash_omni.thinker2talker
|
||||
final_output: true
|
||||
final_output_type: audio
|
||||
default_sampling_params:
|
||||
temperature: 0.0
|
||||
max_tokens: 1
|
||||
@@ -1,32 +0,0 @@
|
||||
# Single-stage TTS config for Ming-flash-omni-2.0 Talker
|
||||
# Stage 0: Talker only (text → audio waveform via CFM + AudioVAE)
|
||||
# Use this config for standalone TTS deployment without thinker.
|
||||
|
||||
stage_args:
|
||||
- stage_id: 0
|
||||
stage_type: llm
|
||||
is_comprehension: true
|
||||
runtime:
|
||||
devices: "0"
|
||||
max_batch_size: 1
|
||||
engine_args:
|
||||
model_stage: ming_tts
|
||||
model_arch: MingFlashOmniTalkerForConditionalGeneration
|
||||
worker_cls: vllm_omni.worker.gpu_generation_worker.GPUGenerationWorker
|
||||
scheduler_cls: vllm_omni.core.sched.omni_generation_scheduler.OmniGenerationScheduler
|
||||
gpu_memory_utilization: 0.8
|
||||
enforce_eager: true
|
||||
trust_remote_code: true
|
||||
engine_output_type: audio
|
||||
enable_prefix_caching: false
|
||||
max_num_batched_tokens: 100000
|
||||
tokenizer_subdir: talker/llm
|
||||
# NOTE: `hf_config_name` for Ming talker acts as a placeholder
|
||||
# The HF repo ships BailingMM2Config (thinker-only) at root,
|
||||
# OmniModelConfig treats that as "stage does not share outer mrope"
|
||||
hf_config_name: talker_config
|
||||
final_output: true
|
||||
final_output_type: audio
|
||||
default_sampling_params:
|
||||
temperature: 0.0
|
||||
max_tokens: 1
|
||||
Reference in new issue
Block a user