vllm-omni/examples/online_serving/ming_flash_omni/README.md

# Ming-flash-omni 2.0

## Installation

Please refer to [README.md](../../../README.md)

## Deployment modes

| Mode | Launch command | Output |
|------|---------------|--------|
| Thinker + Talker (omni-speech, default) | `vllm serve ... --omni` | Text + Audio |
| Thinker only (multimodal understanding) | `vllm serve ... --omni --deploy-config vllm_omni/deploy/ming_flash_omni_thinker_only.yaml` | Text |
| Thinker + Imagegen (text-to-image / img2img) | `vllm serve ... --omni --stage-configs-path vllm_omni/model_executor/stage_configs/ming_flash_omni_dual.yaml` | Image |

For standalone TTS (talker only), see the [Ming-flash-omni-TTS section in the Text-To-Speech hub](../text_to_speech/README.md#ming-flash-omni-tts).

## Run examples (Ming-flash-omni 2.0)

### Launch the Server

**Thinker + Talker (omni-speech, text + audio output):**
```bash
vllm serve Jonathan1909/Ming-flash-omni-2.0 --omni --port 8091
```

The model registry auto-loads corresponding deploy yaml.

**Thinker-only (text output):**
```bash
vllm serve Jonathan1909/Ming-flash-omni-2.0 --omni --port 8091 \
    --deploy-config vllm_omni/deploy/ming_flash_omni_thinker_only.yaml
```

Pass `--deploy-config /path/to/your_deploy.yaml` to use a custom deploy
config.

### Send Multi-modal Request

Shared Python client (supports `text | use_image | use_audio | use_video |
use_mixed_modalities`; pass `--image-path` / `--audio-path` / `--video-path`
for local files or URLs, `--modalities text` for output, `--help` for the
full flag list):

```bash
python examples/online_serving/openai_chat_completion_client_for_multimodal_generation.py \
    --model Jonathan1909/Ming-flash-omni-2.0 \
    --query-type use_mixed_modalities \
    --port 8091 --host localhost \
    --modalities text
```

Parameterized curl wrapper in this directory:

```bash
bash run_curl_multimodal_generation.sh text
bash run_curl_multimodal_generation.sh use_image
bash run_curl_multimodal_generation.sh use_audio
bash run_curl_multimodal_generation.sh use_video
bash run_curl_multimodal_generation.sh use_mixed_modalities
bash run_curl_multimodal_generation.sh use_image_gen
```

## Image generation (text-to-image)

Ming-flash-omni-2.0 also exposes an image-generation (diffusion) stage.
Launch with the dual-stage YAML, which adds an image-gen stage behind the
thinker:

```bash
vllm serve Jonathan1909/Ming-flash-omni-2.0 --omni \
    --stage-configs-path vllm_omni/model_executor/stage_configs/ming_flash_omni_dual.yaml \
    --stage-init-timeout 1800 \
    --port 8188
```

Then request image output by passing `"modalities": ["image"]`:

```bash
curl http://127.0.0.1:8188/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
      "model": "Jonathan1909/Ming-flash-omni-2.0",
      "messages": [{"role": "user", "content": "Draw a beautiful girl with short black hair and red dress."}],
      "modalities": ["image"]
    }' -o /tmp/ming_response.json

python - <<'PY'
import base64, json
r = json.load(open('/tmp/ming_response.json'))
url = r['choices'][0]['message']['content'][0]['image_url']['url']
open('/tmp/ming_gen.png', 'wb').write(base64.b64decode(url.split(',', 1)[1]))
print('saved /tmp/ming_gen.png')
PY
```

### Optional knobs

Pass image-gen overrides via the diffusion-stage `sampling_params_list[1].extra_args.image_gen`:

| Key | Default | Description |
| --- | --- | --- |
| `height` / `width` | from config (1024) | Output resolution (multiples of `vae_scale_factor * 2`, currently 16). |
| `steps` | 30 | Number of FlowMatchEuler denoise steps. |
| `cfg` | 2.0 | Classifier-free guidance scale. |
| `seed` | 42 | Per-request RNG seed (deterministic when ByT5 is also seed-stable). |
| `byte5_text` | (auto) | Override the glyph text for ByT5 enhancement; raw strings are auto-wrapped to Ming's `Text "...". ` format. |
| `negative_prompt` | empty | Real CFG negative conditioning (must be set on **stage-0 thinker** `sampling_params` so `expand_cfg_prompts` spawns the companion). |

Example with all the knobs:

```bash
curl http://127.0.0.1:8188/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
      "model": "Jonathan1909/Ming-flash-omni-2.0",
      "modalities": ["image"],
      "sampling_params_list": [
        {"temperature":0.4,"top_p":0.9,"top_k":1,"max_tokens":1,"seed":42,
         "extra_args":{"image_gen":{"negative_prompt":"ugly, blurry, distorted"}}},
        {"seed":42,"extra_args":{"image_gen":{
            "steps":6,"cfg":1.5,"height":512,"width":512,"seed":123,
            "byte5_text":["理解与生成统一"]}}}
      ],
      "messages": [{"role": "user", "content": "Draw a poster."}]
    }' -o /tmp/ming_response.json
```

### img2img (reference image)

Add an `image_url` content part to the user message; the parser routes it
into the diffusion stage as `extra[reference_image]`:

```jsonc
"messages": [{
  "role": "user",
  "content": [
    {"type": "text", "text": "Change the background to a sandy beach at sunset."},
    {"type": "image_url", "image_url": {"url": "data:image/png;base64,<base64>"}}
  ]
}]
```

### GPU layout

The shipped `ming_flash_omni_dual.yaml` allocates the thinker on GPUs 0–3
(TP=4) and the diffusion stage on GPU 4 (TP=1). Copy the YAML and edit
`runtime.devices` per stage to relocate; with fewer GPUs available, drop
the thinker TP to 2 and run the diffusion stage on a free card. Image-gen
warmup takes roughly an extra 30–60 s on top of the thinker — set
`--stage-init-timeout 1800` if the default 300 s is too tight.

## Modality control

| `modalities` | Server config | Output |
|-------------|--------------|--------|
| `["text"]` or omitted | Thinker only | Text |
| `["audio"]` | Thinker + Talker | Audio (speech) |
| `["text", "audio"]` | Thinker + Talker | Text + Audio |
| `["image"]` | Thinker + Imagegen (dual YAML) | Image (PNG, base64 in `choices[0].message.content`) |

For ready-to-copy curl examples (text / audio / multimodal input, SSE
streaming, reasoning mode), see the recipe at
[`recipes/inclusionAI/Ming-flash-omni-2.0.md`](../../../recipes/inclusionAI/Ming-flash-omni-2.0.md).

## OpenAI Python SDK — streaming

```python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8091/v1", api_key="EMPTY")

response = client.chat.completions.create(
    model="Jonathan1909/Ming-flash-omni-2.0",
    messages=[
        {"role": "system", "content": [{"type": "text", "text": "你是一个友好的AI助手。\n\ndetailed thinking off"}]},
        {"role": "user", "content": "请详细介绍鹦鹉的生活习性。"},
    ],
    modalities=["text"],
    stream=True,
)
for chunk in response:
    for choice in chunk.choices:
        if hasattr(choice, "delta") and choice.delta.content:
            print(choice.delta.content, end="", flush=True)
print()
```

The `--stream` flag on the Python client script above shows the same pattern
driven by the shared multimodal client.