anodyine/vllm-omni
1
0 Fork 0
Code Issues 4 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

[Model] Add Ming-flash-omni-2.0 Image Generation (Diffusion) Stage (#2875)

Browse Source 
Signed-off-by: ZhengWG <zwg0606@gmail.com>
Signed-off-by: Zheng Wengang <zwg0606@gmail.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Yuanheng Zhao <54058983+yuanheng-zhao@users.noreply.github.com>
This commit is contained in:
Zheng WengangClaude Opus 4.6Yuanheng Zhao
 authored and GitHub committed 2026-05-27 23:21:20 +08:00
1 parent 36dc32ff81
commit a698269e47
25 files changed
+2614 -42
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

examples/offline_inference/ming_flash_omni/README.md
+13 -1
View File
@@ -2,12 +2,13 @@
[Ming-flash-omni-2.0](https://github.com/inclusionAI/Ming) is an omni-modal model supporting text, image, video, and audio understanding, with text and speech outputs.
vLLM-Omni supports two deployment modes:
vLLM-Omni supports three deployment modes:
| Mode | Deploy config | Output |
|------|--------------|--------|
| Thinker + Talker (omni-speech, default) | `vllm_omni/deploy/ming_flash_omni.yaml` | Text + Audio |
| Thinker only (multimodal understanding) | `vllm_omni/deploy/ming_flash_omni_thinker_only.yaml` | Text |
| Thinker + Imagegen (text-to-image / img2img) | `vllm_omni/model_executor/stage_configs/ming_flash_omni_dual.yaml` | Image (online-serving only at the moment) |
For standalone TTS (talker only), see the [Ming-flash-omni-TTS section in the Text-To-Speech hub](../text_to_speech/README.md#ming-flash-omni-tts).
@@ -87,6 +88,17 @@ The default deploy YAML allocates thinker on GPUs 0–3 and talker on GPU 3 for
Pass `--deploy-config /path/to/your_deploy.yaml` to any of the commands above to override the bundled deploy config.
### Image generation (text-to-image / img2img)
The diffusion-stage image-generation path is currently only wired through
the online OpenAI-compatible chat endpoint (`/v1/chat/completions` with
`"modalities": ["image"]`). For payloads, optional knobs
(`steps`/`cfg`/`seed`/`byte5_text`/`negative_prompt`), and the img2img
reference-image flow, see the
[image-gen section in the online-serving README](../../online_serving/ming_flash_omni/README.md#image-generation-text-to-image).
`end2end.py` does not yet exercise the imagegen stage; that is tracked as
follow-up work.
## Online serving
For online serving via the OpenAI-compatible API, see [examples/online_serving/ming_flash_omni/README.md](../../online_serving/ming_flash_omni/README.md).
examples/online_serving/ming_flash_omni/README.md
+92
View File
@@ -10,6 +10,7 @@ Please refer to [README.md](../../../README.md)
|------|---------------|--------|
| Thinker + Talker (omni-speech, default) | `vllm serve ... --omni` | Text + Audio |
| Thinker only (multimodal understanding) | `vllm serve ... --omni --deploy-config vllm_omni/deploy/ming_flash_omni_thinker_only.yaml` | Text |
| Thinker + Imagegen (text-to-image / img2img) | `vllm serve ... --omni --stage-configs-path vllm_omni/model_executor/stage_configs/ming_flash_omni_dual.yaml` | Image |
For standalone TTS (talker only), see the [Ming-flash-omni-TTS section in the Text-To-Speech hub](../text_to_speech/README.md#ming-flash-omni-tts).
@@ -56,8 +57,98 @@ bash run_curl_multimodal_generation.sh use_image
bash run_curl_multimodal_generation.sh use_audio
bash run_curl_multimodal_generation.sh use_video
bash run_curl_multimodal_generation.sh use_mixed_modalities
bash run_curl_multimodal_generation.sh use_image_gen
```
## Image generation (text-to-image)
Ming-flash-omni-2.0 also exposes an image-generation (diffusion) stage.
Launch with the dual-stage YAML, which adds an image-gen stage behind the
thinker:
```bash
vllm serve Jonathan1909/Ming-flash-omni-2.0 --omni \
--stage-configs-path vllm_omni/model_executor/stage_configs/ming_flash_omni_dual.yaml \
--stage-init-timeout 1800 \
--port 8188
```
Then request image output by passing `"modalities": ["image"]`:
```bash
curl http://127.0.0.1:8188/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "Jonathan1909/Ming-flash-omni-2.0",
"messages": [{"role": "user", "content": "Draw a beautiful girl with short black hair and red dress."}],
"modalities": ["image"]
}' -o /tmp/ming_response.json
python - <<'PY'
import base64, json
r = json.load(open('/tmp/ming_response.json'))
url = r['choices'][0]['message']['content'][0]['image_url']['url']
open('/tmp/ming_gen.png', 'wb').write(base64.b64decode(url.split(',', 1)[1]))
print('saved /tmp/ming_gen.png')
PY
```
### Optional knobs
Pass image-gen overrides via the diffusion-stage `sampling_params_list[1].extra_args.image_gen`:
| Key | Default | Description |
| --- | --- | --- |
| `height` / `width` | from config (1024) | Output resolution (multiples of `vae_scale_factor * 2`, currently 16). |
| `steps` | 30 | Number of FlowMatchEuler denoise steps. |
| `cfg` | 2.0 | Classifier-free guidance scale. |
| `seed` | 42 | Per-request RNG seed (deterministic when ByT5 is also seed-stable). |
| `byte5_text` | (auto) | Override the glyph text for ByT5 enhancement; raw strings are auto-wrapped to Ming's `Text "...". ` format. |
| `negative_prompt` | empty | Real CFG negative conditioning (must be set on **stage-0 thinker** `sampling_params` so `expand_cfg_prompts` spawns the companion). |
Example with all the knobs:
```bash
curl http://127.0.0.1:8188/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "Jonathan1909/Ming-flash-omni-2.0",
"modalities": ["image"],
"sampling_params_list": [
{"temperature":0.4,"top_p":0.9,"top_k":1,"max_tokens":1,"seed":42,
"extra_args":{"image_gen":{"negative_prompt":"ugly, blurry, distorted"}}},
{"seed":42,"extra_args":{"image_gen":{
"steps":6,"cfg":1.5,"height":512,"width":512,"seed":123,
"byte5_text":["理解与生成统一"]}}}
],
"messages": [{"role": "user", "content": "Draw a poster."}]
}' -o /tmp/ming_response.json
```
### img2img (reference image)
Add an `image_url` content part to the user message; the parser routes it
into the diffusion stage as `extra[reference_image]`:
```jsonc
"messages": [{
"role": "user",
"content": [
{"type": "text", "text": "Change the background to a sandy beach at sunset."},
{"type": "image_url", "image_url": {"url": "data:image/png;base64,<base64>"}}
]
}]
```
### GPU layout
The shipped `ming_flash_omni_dual.yaml` allocates the thinker on GPUs 0–3
(TP=4) and the diffusion stage on GPU 4 (TP=1). Copy the YAML and edit
`runtime.devices` per stage to relocate; with fewer GPUs available, drop
the thinker TP to 2 and run the diffusion stage on a free card. Image-gen
warmup takes roughly an extra 30–60 s on top of the thinker — set
`--stage-init-timeout 1800` if the default 300 s is too tight.
## Modality control
| `modalities` | Server config | Output |
@@ -65,6 +156,7 @@ bash run_curl_multimodal_generation.sh use_mixed_modalities
| `["text"]` or omitted | Thinker only | Text |
| `["audio"]` | Thinker + Talker | Audio (speech) |
| `["text", "audio"]` | Thinker + Talker | Text + Audio |
| `["image"]` | Thinker + Imagegen (dual YAML) | Image (PNG, base64 in `choices[0].message.content`) |
For ready-to-copy curl examples (text / audio / multimodal input, SSE
streaming, reasoning mode), see the recipe at
examples/online_serving/ming_flash_omni/run_curl_multimodal_generation.sh
+40 -2
View File
@@ -7,14 +7,15 @@ PORT="${PORT:-8091}"
QUERY_TYPE="${1:-text}"
# Validate query type
if [[ ! "$QUERY_TYPE" =~ ^(text|use_audio|use_image|use_video|use_mixed_modalities)$ ]]; then
if [[ ! "$QUERY_TYPE" =~ ^(text|use_audio|use_image|use_video|use_mixed_modalities|use_image_gen)$ ]]; then
echo "Error: Invalid query type '$QUERY_TYPE'"
echo "Usage: $0 [text|use_audio|use_image|use_video|use_mixed_modalities]"
echo "Usage: $0 [text|use_audio|use_image|use_video|use_mixed_modalities|use_image_gen]"
echo " text: Text-only query"
echo " use_audio: Audio + Text query"
echo " use_image: Image + Text query"
echo " use_video: Video + Text query"
echo " use_mixed_modalities: Audio + Image + Video + Text query"
echo " use_image_gen: Text-to-image (diffusion stage) — saves the generated PNG locally"
exit 1
fi
@@ -100,6 +101,43 @@ esac
echo "Running query type: $QUERY_TYPE"
echo ""
if [[ "$QUERY_TYPE" == "use_image_gen" ]]; then
# Image-gen branch: dual-stage diffusion endpoint, modalities=["image"].
# The shipped ming_flash_omni_dual.yaml runs the thinker on cards 0-3
# (TP=4) and the diffusion stage on card 4 (TP=1). See the image-gen
# section in README.md for knobs and img2img.
out_path="${OUT_PATH:-/tmp/ming_imagegen.png}"
request_body=$(cat <<'EOF'
{
"model": "Jonathan1909/Ming-flash-omni-2.0",
"modalities": ["image"],
"messages": [
{"role": "user", "content": "Draw a beautiful girl with short black hair and red dress."}
]
}
EOF
)
output=$(curl -sS --retry 3 --retry-delay 3 --retry-connrefused \
-X POST "http://localhost:${PORT}/v1/chat/completions" \
-H "Content-Type: application/json" \
-d "$request_body")
# Server returns base64-encoded PNG either as a data URL in
# ``choices[0].message.content[0].image_url.url`` or as a raw string.
b64=$(echo "$output" | jq -r '.choices[0].message.content
| if type == "array"
then (map(select(.image_url? != null)) | .[0].image_url.url // "")
elif type == "string" then . else "" end')
if [[ -z "$b64" ]]; then
echo "Error: no image returned. Raw response:"
echo "$output" | jq '.'
exit 1
fi
if [[ "$b64" == data:image* ]]; then b64="${b64#*,}"; fi
printf '%s' "$b64" | base64 --decode > "$out_path"
echo "Saved generated image: $out_path ($(stat -c%s "$out_path" 2>/dev/null || stat -f%z "$out_path") bytes)"
exit 0
fi
request_body=$(cat <<EOF
{
"model": "Jonathan1909/Ming-flash-omni-2.0",