[Config Refactor] Migrate Ming and Ming-TTS deploy/pipline configs (#3154)
Signed-off-by: Yuanheng Zhao <jonathan.zhaoyh@gmail.com> Signed-off-by: yuanheng <jonathan.zhaoyh@gmail.com> Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
This commit is contained in:
Yuanheng ZhaoandHongsheng Liu authored and GitHub committed
1 parent
6a837849dd
commit
d49356e4ca
25 files changed
+518
-398
No files matched your search
@@ -4,10 +4,10 @@
|
||||
|
||||
vLLM-Omni supports two deployment modes:
|
||||
|
||||
| Mode | Stage config | Output |
|
||||
|------|-------------|--------|
|
||||
| Thinker only (multimodal understanding) | `ming_flash_omni_thinker.yaml` (default `--omni`) | Text |
|
||||
| Thinker + Talker (omni-speech) | `ming_flash_omni.yaml` | Text + Audio |
|
||||
| Mode | Deploy config | Output |
|
||||
|------|--------------|--------|
|
||||
| Thinker + Talker (omni-speech, default) | `vllm_omni/deploy/ming_flash_omni.yaml` | Text + Audio |
|
||||
| Thinker only (multimodal understanding) | `vllm_omni/deploy/ming_flash_omni_thinker_only.yaml` | Text |
|
||||
|
||||
For standalone TTS (talker only), see [`examples/offline_inference/ming_flash_omni_tts/`](../ming_flash_omni_tts/).
|
||||
|
||||
@@ -15,30 +15,30 @@ For standalone TTS (talker only), see [`examples/offline_inference/ming_flash_om
|
||||
|
||||
Please refer to the [stage configuration documentation](https://docs.vllm.ai/projects/vllm-omni/en/latest/configuration/stage_configs/) to configure memory allocation appropriately for your hardware setup.
|
||||
|
||||
The default `--omni` flag runs thinker only. For omni-speech, pass the two-stage config explicitly:
|
||||
When no `--deploy-config` is passed, the model registry auto-loads the full thinker+talker `vllm_omni/deploy/ming_flash_omni.yaml` (See [Omni-Speech](#omni-speech-thinker--talker)).
|
||||
|
||||
For text-only output without spinning up the talker, pass:
|
||||
|
||||
```bash
|
||||
--stage-configs-path vllm_omni/model_executor/stage_configs/ming_flash_omni.yaml
|
||||
--deploy-config vllm_omni/deploy/ming_flash_omni_thinker_only.yaml
|
||||
```
|
||||
|
||||
## Run examples
|
||||
|
||||
The end-to-end script defaults to built-in assets; pass `--image-path`,
|
||||
`--audio-path`, or `--video-path` to override.
|
||||
The end-to-end script defaults to built-in assets; pass `--image-path`, `--audio-path`, or `--video-path` to override.
|
||||
|
||||
### Multi-Modality Understanding (Standalone Thinker)
|
||||
|
||||
Here we pass thinker-only deploy yaml:
|
||||
|
||||
```bash
|
||||
# Text-only
|
||||
python examples/offline_inference/ming_flash_omni/end2end.py --query-type text
|
||||
|
||||
# Image / audio / video / mixed understanding
|
||||
python examples/offline_inference/ming_flash_omni/end2end.py --query-type use_image
|
||||
python examples/offline_inference/ming_flash_omni/end2end.py --query-type use_audio
|
||||
python examples/offline_inference/ming_flash_omni/end2end.py --query-type use_video --num-frames 16
|
||||
python examples/offline_inference/ming_flash_omni/end2end.py --query-type use_mixed_modalities \
|
||||
--image-path /path/to/image.jpg --audio-path /path/to/audio.wav
|
||||
python examples/offline_inference/ming_flash_omni/end2end.py --deploy-config vllm_omni/deploy/ming_flash_omni_thinker_only.yaml --query-type text
|
||||
python examples/offline_inference/ming_flash_omni/end2end.py --deploy-config vllm_omni/deploy/ming_flash_omni_thinker_only.yaml --query-type use_image
|
||||
python examples/offline_inference/ming_flash_omni/end2end.py --deploy-config vllm_omni/deploy/ming_flash_omni_thinker_only.yaml --query-type use_audio
|
||||
python examples/offline_inference/ming_flash_omni/end2end.py --deploy-config vllm_omni/deploy/ming_flash_omni_thinker_only.yaml --query-type use_video --num-frames 16
|
||||
```
|
||||
|
||||
#### Reasoning (Thinking Mode)
|
||||
### Reasoning (Thinking Mode)
|
||||
|
||||
Reasoning ("detailed thinking on") is applied by the script when
|
||||
`--query-type reasoning` is set. The default prompt matches Ming's cookbook
|
||||
@@ -46,19 +46,21 @@ and expects the reference figure from the upstream repo — see
|
||||
`get_reasoning_query` in `end2end.py`.
|
||||
|
||||
```bash
|
||||
python examples/offline_inference/ming_flash_omni/end2end.py -q reasoning --image-path ./3_0.png
|
||||
python examples/offline_inference/ming_flash_omni/end2end.py \
|
||||
--deploy-config vllm_omni/deploy/ming_flash_omni_thinker_only.yaml \
|
||||
--query-type reasoning \
|
||||
--image-path ./3_0.png
|
||||
```
|
||||
|
||||
### Omni-speech (thinker + talker)
|
||||
### Omni-Speech (Thinker + Talker)
|
||||
|
||||
To enable spoken output, use the two-stage config and request `audio` (or `text,audio`) modalities.
|
||||
The default deploy YAML already runs thinker+talker, so spoken output only requires requesting `audio` (or `text,audio`) modalities.
|
||||
The thinker processes your multimodal input, generates text, then the talker synthesises the response as speech.
|
||||
|
||||
**Audio-only output** (speech response, no text):
|
||||
```bash
|
||||
python examples/offline_inference/ming_flash_omni/end2end.py \
|
||||
--query-type text \
|
||||
--stage-configs-path vllm_omni/model_executor/stage_configs/ming_flash_omni.yaml \
|
||||
--modalities audio \
|
||||
--output-dir output_ming_omni_speech
|
||||
```
|
||||
@@ -67,14 +69,13 @@ python examples/offline_inference/ming_flash_omni/end2end.py \
|
||||
```bash
|
||||
python examples/offline_inference/ming_flash_omni/end2end.py \
|
||||
--query-type use_audio \
|
||||
--stage-configs-path vllm_omni/model_executor/stage_configs/ming_flash_omni.yaml \
|
||||
--modalities text,audio \
|
||||
--output-dir output_ming_omni_speech
|
||||
```
|
||||
|
||||
Generated `.wav` files are saved to `--output-dir` (default `output_ming`), one per request.
|
||||
|
||||
The stage config allocates thinker on GPUs 0–3 and talker on GPU 3 by default. Adjust `devices` in the YAML to match your hardware.
|
||||
The default deploy YAML allocates thinker on GPUs 0–3 and talker on GPU 3 for a common device topology (4 rather than 5 devices that talker on its own device). Adjust `devices` in a copied YAML and pass it via `--deploy-config` to match your hardware or requirements.
|
||||
|
||||
### Modality control
|
||||
|
||||
@@ -84,8 +85,7 @@ The stage config allocates thinker on GPUs 0–3 and talker on GPU 3 by default.
|
||||
| `audio` | Text (internal) | Runs | `<id>.wav` |
|
||||
| `text,audio` | Text | Runs | `<id>.txt` + `<id>.wav` |
|
||||
|
||||
Pass `--stage-configs-path /path/to/your_config.yaml` to any of the commands
|
||||
above to override the stage config.
|
||||
Pass `--deploy-config /path/to/your_deploy.yaml` to any of the commands above to override the bundled deploy config.
|
||||
|
||||
## Online serving
|
||||
|
||||
|
||||
@@ -20,6 +20,7 @@ from vllm.multimodal.media.audio import load_audio
|
||||
from vllm.utils.argparse_utils import FlexibleArgumentParser
|
||||
|
||||
import vllm_omni
|
||||
from vllm_omni.engine.arg_utils import nullify_stage_engine_defaults
|
||||
from vllm_omni.entrypoints.omni import Omni
|
||||
|
||||
# Imports the processor also registers itself
|
||||
@@ -302,14 +303,7 @@ def main(args):
|
||||
else:
|
||||
query_result = query_func(processor)
|
||||
|
||||
# Initialize Omni (with thinker-only stage config)
|
||||
omni = Omni(
|
||||
model=MODEL_NAME,
|
||||
stage_configs_path=args.stage_configs_path,
|
||||
log_stats=args.log_stats,
|
||||
init_timeout=args.init_timeout,
|
||||
stage_init_timeout=args.stage_init_timeout,
|
||||
)
|
||||
omni = Omni.from_cli_args(args, model=MODEL_NAME)
|
||||
|
||||
# Thinker sampling params
|
||||
thinker_sampling_params = SamplingParams(
|
||||
@@ -410,10 +404,10 @@ def parse_args():
|
||||
help="Query type.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--stage-configs-path",
|
||||
"--deploy-config",
|
||||
type=str,
|
||||
default=None,
|
||||
help="Path to a stage configs YAML file.",
|
||||
help="Path to a deploy YAML; leave unset to auto-load full thinker+talker. Pass custom for text-only",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--log-stats",
|
||||
@@ -499,6 +493,7 @@ def parse_args():
|
||||
help="Output directory for results.",
|
||||
)
|
||||
|
||||
nullify_stage_engine_defaults(parser)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
|
||||
@@ -3,7 +3,7 @@
|
||||
This example runs **Ming-flash-omni-2.0 talker-only** offline inference with:
|
||||
|
||||
- `model`: `Jonathan1909/Ming-flash-omni-2.0`
|
||||
- `stage config`: `vllm_omni/model_executor/stage_configs/ming_flash_omni_tts.yaml`
|
||||
- `deploy config`: `vllm_omni/deploy/ming_flash_omni_tts.yaml`
|
||||
|
||||
It follows the Ming cookbook parameter style:
|
||||
|
||||
@@ -37,7 +37,7 @@ python examples/offline_inference/ming_flash_omni_tts/end2end.py --case basic
|
||||
- `--text`: override default text in the selected case
|
||||
- `--output`: custom output wav path
|
||||
- `--model`: local model path or HF repo id
|
||||
- `--stage-configs-path`: custom talker stage config path
|
||||
- `--deploy-config`: custom talker deploy YAML path
|
||||
- `--log-stats`: enable runtime stats logs
|
||||
|
||||
## Notes
|
||||
|
||||
@@ -9,6 +9,7 @@ from vllm.utils.argparse_utils import FlexibleArgumentParser
|
||||
|
||||
os.environ["VLLM_WORKER_MULTIPROC_METHOD"] = "spawn"
|
||||
|
||||
from vllm_omni.engine.arg_utils import nullify_stage_engine_defaults
|
||||
from vllm_omni.entrypoints.omni import Omni
|
||||
from vllm_omni.inputs.data import OmniTokensPrompt
|
||||
from vllm_omni.model_executor.models.ming_flash_omni.prompt_utils import (
|
||||
@@ -17,7 +18,6 @@ from vllm_omni.model_executor.models.ming_flash_omni.prompt_utils import (
|
||||
)
|
||||
|
||||
MODEL_NAME = "Jonathan1909/Ming-flash-omni-2.0"
|
||||
DEFAULT_STAGE_CONFIG = "vllm_omni/model_executor/stage_configs/ming_flash_omni_tts.yaml"
|
||||
|
||||
|
||||
def get_messages(case: str, text_override: str | None) -> dict[str, Any]:
|
||||
@@ -71,10 +71,10 @@ def parse_args():
|
||||
parser = FlexibleArgumentParser(description="Ming-flash-omni standalone talker offline e2e example")
|
||||
parser.add_argument("--model", type=str, default=MODEL_NAME, help="Model name or local path.")
|
||||
parser.add_argument(
|
||||
"--stage-configs-path",
|
||||
"--deploy-config",
|
||||
type=str,
|
||||
default=DEFAULT_STAGE_CONFIG,
|
||||
help="Path to stage configs yaml for standalone talker deployment.",
|
||||
default="vllm_omni/deploy/ming_flash_omni_tts.yaml",
|
||||
help="Path to a custom deploy YAML for the TTS deployment. ",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--case",
|
||||
@@ -88,19 +88,15 @@ def parse_args():
|
||||
parser.add_argument("--log-stats", action="store_true", default=False, help="Enable stats logging.")
|
||||
parser.add_argument("--init-timeout", type=int, default=600, help="Engine init timeout in seconds.")
|
||||
parser.add_argument("--stage-init-timeout", type=int, default=300, help="Single stage init timeout in seconds.")
|
||||
|
||||
nullify_stage_engine_defaults(parser)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def main():
|
||||
args = parse_args()
|
||||
|
||||
omni = Omni(
|
||||
model=args.model,
|
||||
stage_configs_path=args.stage_configs_path,
|
||||
log_stats=args.log_stats,
|
||||
init_timeout=args.init_timeout,
|
||||
stage_init_timeout=args.stage_init_timeout,
|
||||
)
|
||||
omni = Omni.from_cli_args(args, model=args.model)
|
||||
|
||||
messages = get_messages(args.case, args.text)
|
||||
decode_args = {
|
||||
|
||||
@@ -8,8 +8,8 @@ Please refer to [README.md](../../../README.md)
|
||||
|
||||
| Mode | Launch command | Output |
|
||||
|------|---------------|--------|
|
||||
| Thinker only (multimodal understanding) | `vllm serve ... --omni` | Text |
|
||||
| Thinker + Talker (omni-speech) | `vllm serve ... --omni --stage-configs-path ming_flash_omni.yaml` | Text + Audio |
|
||||
| Thinker + Talker (omni-speech, default) | `vllm serve ... --omni` | Text + Audio |
|
||||
| Thinker only (multimodal understanding) | `vllm serve ... --omni --deploy-config vllm_omni/deploy/ming_flash_omni_thinker_only.yaml` | Text |
|
||||
|
||||
For standalone TTS (talker only), see [`examples/online_serving/ming_flash_omni_tts/`](../ming_flash_omni_tts/).
|
||||
|
||||
@@ -17,18 +17,20 @@ For standalone TTS (talker only), see [`examples/online_serving/ming_flash_omni_
|
||||
|
||||
### Launch the Server
|
||||
|
||||
**Thinker only (text output):**
|
||||
**Thinker + Talker (omni-speech, text + audio output):**
|
||||
```bash
|
||||
vllm serve Jonathan1909/Ming-flash-omni-2.0 --omni --port 8091
|
||||
```
|
||||
|
||||
**Thinker + Talker (omni-speech, text + audio output):**
|
||||
The model registry auto-loads corresponding deploy yaml.
|
||||
|
||||
**Thinker-only (text output):**
|
||||
```bash
|
||||
vllm serve Jonathan1909/Ming-flash-omni-2.0 --omni --port 8091 \
|
||||
--stage-configs-path vllm_omni/model_executor/stage_configs/ming_flash_omni.yaml
|
||||
--deploy-config vllm_omni/deploy/ming_flash_omni_thinker_only.yaml
|
||||
```
|
||||
|
||||
Pass `--stage-configs-path /path/to/your_config.yaml` to use a custom stage
|
||||
Pass `--deploy-config /path/to/your_deploy.yaml` to use a custom deploy
|
||||
config.
|
||||
|
||||
### Send Multi-modal Request
|
||||
|
||||
@@ -18,17 +18,6 @@ if [[ ! "$QUERY_TYPE" =~ ^(text|use_audio|use_image|use_video|use_mixed_modaliti
|
||||
exit 1
|
||||
fi
|
||||
|
||||
thinker_sampling_params='{
|
||||
"temperature": 0.4,
|
||||
"top_p": 0.9,
|
||||
"top_k": -1,
|
||||
"max_tokens": 16384,
|
||||
"seed": 42,
|
||||
"detokenize": true,
|
||||
"repetition_penalty": 1.05
|
||||
}'
|
||||
# Above is optional, it has a default setting in stage_configs of the corresponding model.
|
||||
|
||||
# Define URLs for assets
|
||||
MARY_HAD_LAMB_AUDIO_URL="https://vllm-public-assets.s3.us-west-2.amazonaws.com/multimodal_asset/mary_had_lamb.ogg"
|
||||
CHERRY_BLOSSOM_IMAGE_URL="https://vllm-public-assets.s3.us-west-2.amazonaws.com/vision_model_images/cherry_blossom.jpg"
|
||||
@@ -114,9 +103,6 @@ echo ""
|
||||
request_body=$(cat <<EOF
|
||||
{
|
||||
"model": "Jonathan1909/Ming-flash-omni-2.0",
|
||||
"sampling_params_list": [
|
||||
$thinker_sampling_params
|
||||
],
|
||||
"modalities": ["text"],
|
||||
"messages": [
|
||||
{
|
||||
|
||||
@@ -5,7 +5,7 @@ This directory contains online e2e examples for **Ming-flash-omni-2.0 standalone
|
||||
Server uses:
|
||||
|
||||
- `model`: `Jonathan1909/Ming-flash-omni-2.0`
|
||||
- `stage config`: `vllm_omni/model_executor/stage_configs/ming_flash_omni_tts.yaml`
|
||||
- `deploy config`: `vllm_omni/deploy/ming_flash_omni_tts.yaml`
|
||||
|
||||
## Launch the Server
|
||||
|
||||
@@ -18,7 +18,7 @@ Equivalent manual command:
|
||||
|
||||
```bash
|
||||
vllm serve Jonathan1909/Ming-flash-omni-2.0 \
|
||||
--stage-configs-path vllm_omni/model_executor/stage_configs/ming_flash_omni_tts.yaml \
|
||||
--deploy-config vllm_omni/deploy/ming_flash_omni_tts.yaml \
|
||||
--host 0.0.0.0 \
|
||||
--port 8091 \
|
||||
--trust-remote-code \
|
||||
|
||||
@@ -12,13 +12,13 @@ set -e
|
||||
MODEL="${MODEL:-Jonathan1909/Ming-flash-omni-2.0}"
|
||||
HOST="${HOST:-0.0.0.0}"
|
||||
PORT="${PORT:-8091}"
|
||||
STAGE_CONFIG="${STAGE_CONFIG:-vllm_omni/model_executor/stage_configs/ming_flash_omni_tts.yaml}"
|
||||
DEPLOY_CONFIG="${DEPLOY_CONFIG:-vllm_omni/deploy/ming_flash_omni_tts.yaml}"
|
||||
|
||||
echo "Starting Ming standalone TTS server with model: $MODEL"
|
||||
echo "Stage config: $STAGE_CONFIG"
|
||||
echo "Deploy config: $DEPLOY_CONFIG"
|
||||
|
||||
vllm serve "$MODEL" \
|
||||
--stage-configs-path "$STAGE_CONFIG" \
|
||||
--deploy-config "$DEPLOY_CONFIG" \
|
||||
--host "$HOST" \
|
||||
--port "$PORT" \
|
||||
--trust-remote-code \
|
||||
|
||||
Reference in new issue
Block a user