vllm-omni/examples/online_serving/ming_flash_omni/README.md

Ming-flash-omni 2.0

Installation

Please refer to README.md

Deployment modes

Mode	Launch command	Output
Thinker + Talker (omni-speech, default)	vllm serve ... --omni	Text + Audio
Thinker only (multimodal understanding)	vllm serve ... --omni --deploy-config vllm_omni/deploy/ming_flash_omni_thinker_only.yaml	Text
Thinker + Imagegen (text-to-image / img2img)	vllm serve ... --omni --deploy-config vllm_omni/deploy/ming_flash_omni_image.yaml	Image

For standalone TTS (talker only), see the Ming-flash-omni-TTS section in the Text-To-Speech hub.

Run examples (Ming-flash-omni 2.0)

Launch the Server

Thinker + Talker (omni-speech, text + audio output):

vllm serve Jonathan1909/Ming-flash-omni-2.0 --omni --port 8091

The model registry auto-loads corresponding deploy yaml.

Thinker-only (text output):

vllm serve Jonathan1909/Ming-flash-omni-2.0 --omni --port 8091 \
    --deploy-config vllm_omni/deploy/ming_flash_omni_thinker_only.yaml

Pass --deploy-config /path/to/your_deploy.yaml to use a custom deploy config.

Send Multi-modal Request

Shared Python client (supports text | use_image | use_audio | use_video | use_mixed_modalities; pass --image-path / --audio-path / --video-path for local files or URLs, --modalities text for output, --help for the full flag list):

python examples/online_serving/openai_chat_completion_client_for_multimodal_generation.py \
    --model Jonathan1909/Ming-flash-omni-2.0 \
    --query-type use_mixed_modalities \
    --port 8091 --host localhost \
    --modalities text

Parameterized curl wrapper in this directory:

bash run_curl_multimodal_generation.sh text
bash run_curl_multimodal_generation.sh use_image
bash run_curl_multimodal_generation.sh use_audio
bash run_curl_multimodal_generation.sh use_video
bash run_curl_multimodal_generation.sh use_mixed_modalities
bash run_curl_multimodal_generation.sh use_image_gen

Image generation (text-to-image)

Ming-flash-omni-2.0 also exposes an image-generation (diffusion) stage. Launch with the image deploy YAML, which adds an image-gen stage behind the thinker:

vllm serve Jonathan1909/Ming-flash-omni-2.0 --omni \
    --deploy-config vllm_omni/deploy/ming_flash_omni_image.yaml \
    --stage-init-timeout 1800 \
    --init-timeout 1800 \
    --port 8091

Then request image output by passing "modalities": ["image"]:

curl -s http://127.0.0.1:8091/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
      "model": "Jonathan1909/Ming-flash-omni-2.0",
      "messages": [{"role": "user", "content": "Please draw a cute cat."}],
      "modalities": ["image"]
    }' | jq -r '.choices[0].message.content[0].image_url.url | split(",")[1]' | base64 -d > ming_imagegen.png

Optional knobs

Pass image-gen overrides via the diffusion-stage sampling_params_list[1].extra_args.image_gen:

Key	Default	Description
height / width	from config (1024)	Output resolution (multiples of vae_scale_factor * 2, currently 16).
steps	30	Number of FlowMatchEuler denoise steps.
cfg	2.0	Classifier-free guidance scale.
seed	42	Per-request RNG seed (deterministic when ByT5 is also seed-stable).
byte5_text	(auto)	Override the glyph text for ByT5 enhancement; raw strings are auto-wrapped to Ming's Text "...". format.
negative_prompt	empty	Real CFG negative conditioning (must be set on stage-0 thinker sampling_params so expand_cfg_prompts spawns the companion).

Example with all the knobs:

curl http://127.0.0.1:8091/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
      "model": "Jonathan1909/Ming-flash-omni-2.0",
      "modalities": ["image"],
      "sampling_params_list": [
        {"temperature":0.4,"top_p":0.9,"top_k":1,"max_tokens":1,"seed":42,
         "extra_args":{"image_gen":{"negative_prompt":"ugly, blurry, distorted"}}},
        {"seed":42,"extra_args":{"image_gen":{
            "steps":6,"cfg":1.5,"height":512,"width":512,"seed":123,
            "byte5_text":["理解与生成统一"]}}}
      ],
      "messages": [{"role": "user", "content": "Draw a poster."}]
    }' | jq -r '.choices[0].message.content[0].image_url.url | split(",")[1]' | base64 -d > ming_imagegen_knobs.png

img2img (reference image)

Add an image_url content part to the user message; the parser routes it into the diffusion stage as extra[reference_image]:

"messages": [{
  "role": "user",
  "content": [
    {"type": "text", "text": "Change the background to a sandy beach at sunset."},
    {"type": "image_url", "image_url": {"url": "data:image/png;base64,<base64>"}}
  ]
}]

GPU layout

The shipped ming_flash_omni_image.yaml allocates the thinker on GPUs 0–3 (TP=4) and the diffusion stage on GPU 4 (TP=1). Copy the YAML and edit devices per stage to relocate; with fewer GPUs available, drop the thinker TP to 2 and run the diffusion stage on a free card. Image-gen warmup takes roughly an extra 30–60 s on top of the thinker — set --stage-init-timeout 1800 if the default 300 s is too tight.

Modality control

modalities	Server config	Output
["text"] or omitted	Thinker only	Text
["audio"]	Thinker + Talker	Audio (speech)
["text", "audio"]	Thinker + Talker	Text + Audio
["image"]	Thinker + Imagegen (image deploy YAML)	Image (PNG, base64 in choices[0].message.content)

For ready-to-copy curl examples (text / audio / multimodal input, SSE streaming, reasoning mode), see the recipe at recipes/inclusionAI/Ming-flash-omni-2.0.md.

OpenAI Python SDK — streaming

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8091/v1", api_key="EMPTY")

response = client.chat.completions.create(
    model="Jonathan1909/Ming-flash-omni-2.0",
    messages=[
        {"role": "system", "content": [{"type": "text", "text": "你是一个友好的AI助手。\n\ndetailed thinking off"}]},
        {"role": "user", "content": "请详细介绍鹦鹉的生活习性。"},
    ],
    modalities=["text"],
    stream=True,
)
for chunk in response:
    for choice in chunk.choices:
        if hasattr(choice, "delta") and choice.delta.content:
            print(choice.delta.content, end="", flush=True)
print()

The --stream flag on the Python client script above shows the same pattern driven by the shared multimodal client.