vllm-omni/examples/offline_inference/text_to_speech/ming_tts/README.md

Ming-omni-tts Offline Inference

end2end.py runs Ming dense 0.5B end to end with vLLM-Omni. It uses the in-repo Ming prompt assembly helper directly, so the example request shape matches the real integration instead of a simplified wrapper.

Files

File	Purpose
end2end.py	Driver: CLI, case loading, prompt construction, orchestration (~150 lines)
cases.yaml	All 11 built-in case definitions (prompt, text, instruction, ref-audio flags, flow controls)
runner.py	Engine management and audio output (streaming + blocking paths)

Model Overview

Ming dense 0.5B is exposed here as a two-stage offline pipeline:

• Stage 0: Qwen2-based AR generation with Ming prompt formatting and inline flow controls
• Stage 1: audio VAE decode to mono 44.1 kHz waveform

config_ming_tts.py adapts the checkpoint's HuggingFace config fields (LLM, DiT, aggregator, AudioVAE, token ids). vllm_omni/deploy/ming_tts.yaml selects the vLLM-Omni pipeline and stage runtime topology, including connectors, async chunking, memory limits, and sampling defaults.

The example supports both:

• Blocking eager via vllm_omni/deploy/ming_tts.yaml
• Async chunk eager via vllm_omni/deploy/ming_tts.yaml (default async_chunk: true)

Setup

Install vLLM-Omni with the platform requirements for your accelerator:

uv pip install -e .

The Ming offline example does not require a separate upstream Ming package. Reference-audio cases use the repo dependencies for audio loading, resampling, and CampPlus speaker extraction, including soundfile, torchaudio, and onnxruntime-rocm in the documented ROCm environment.

The tested ROCm environment is summarized in the Ming recipe.

Supported Cases

These cases cover the upstream dense 0.5B cookbook surface that maps cleanly onto the current vLLM-Omni example:

• style: zero-speaker style-conditioned speech
• ip: zero-speaker IP voice generation
• bgm: music-only generation
• tta: text-to-audio event generation with FlowLoss controls
• emotion: reference-audio speech with emotion control
• basic: reference-audio speech with speed / pitch / volume control
• dialect: reference-audio speech with dialect control
• zero_shot: reference-audio cloning with explicit transcript
• podcast: multi-reference dialogue generation with automatic speaker embedding extraction
• speech_bgm: speech with background music conditioning
• speech_sound: speech with environmental sound conditioning

Quick Start

Run the zero-speaker style example:

python examples/offline_inference/text_to_speech/ming_tts/end2end.py \
    --case style \
    --deploy-config vllm_omni/deploy/ming_tts.yaml \
    --enforce-eager

Run zero-shot cloning with a transcript:

python examples/offline_inference/text_to_speech/ming_tts/end2end.py \
    --case zero_shot \
    --ref-audio /path/to/10002287-00000094.wav \
    --ref-text "在此奉劝大家别乱打美白针。" \
    --deploy-config vllm_omni/deploy/ming_tts.yaml \
    --enforce-eager

Run emotion-controlled speech:

python examples/offline_inference/text_to_speech/ming_tts/end2end.py \
    --case emotion \
    --ref-audio /path/to/emotion_prompt.wav \
    --deploy-config vllm_omni/deploy/ming_tts.yaml \
    --enforce-eager

Run podcast generation with two reference clips:

python examples/offline_inference/text_to_speech/ming_tts/end2end.py \
    --case podcast \
    --ref-audio-paths /path/to/CTS-CN-F2F-2019-11-11-423-012-A.wav /path/to/CTS-CN-F2F-2019-11-11-423-012-B.wav \
    --deploy-config vllm_omni/deploy/ming_tts.yaml \
    --enforce-eager

The script automatically extracts one 192-d speaker embedding per reference WAV using the Ming model's campplus.onnx.

If you already have precomputed multi-speaker embeddings, you can override extraction with:

--speaker-embedding /path/to/podcast_speaker_embeddings.json

where the JSON is a list of speaker embeddings, one 192-d vector per speaker.

Run text-to-audio event generation:

python examples/offline_inference/text_to_speech/ming_tts/end2end.py \
    --case tta \
    --deploy-config vllm_omni/deploy/ming_tts.yaml \
    --enforce-eager

Use async_chunk streaming:

python examples/offline_inference/text_to_speech/ming_tts/end2end.py \
    --case basic \
    --ref-audio /path/to/10002287-00000095.wav \
    --streaming \
    --deploy-config vllm_omni/deploy/ming_tts.yaml \
    --enforce-eager

--streaming uses AsyncOmni and the async chunk deploy config. It currently supports one prompt per process invocation; use blocking mode for --num-prompts > 1.

Collect runtime stats and a manifest:

python examples/offline_inference/text_to_speech/ming_tts/end2end.py \
    --case style \
    --deploy-config vllm_omni/deploy/ming_tts.yaml \
    --enforce-eager \
    --enable-stats \
    --stats-log-file output_audio/ming_style_pipeline.log \
    --metadata-json output_audio/ming_style_manifest.json

Reference Fixtures

The upstream Ming cookbook uses these public audio fixtures from inclusionAI/Ming-omni-tts/data/wavs:

• 10002287-00000094.wav for zero-shot cloning
• 10002287-00000095.wav for basic
• emotion_prompt.wav for emotion
• yue_prompt.wav for dialect
• 00000309-00000300.wav for speech_bgm and speech_sound
• CTS-CN-F2F-2019-11-11-423-012-A.wav and CTS-CN-F2F-2019-11-11-423-012-B.wav for podcast

Validation Matrix

The repo-facing example is intended to cover the same dense TTS workflows used by the local Ming validation script:

Case	Blocking deploy/ming_tts.yaml	Async chunk deploy/ming_tts.yaml	Extra inputs
style	Yes	Optional smoke test	none
ip	Yes	Optional smoke test	none
bgm	Yes	Optional smoke test	none
tta	Yes	Optional smoke test	none
emotion	Yes	Yes	--ref-audio emotion_prompt.wav
basic	Yes	Yes	--ref-audio 10002287-00000095.wav
dialect	Yes	Yes	--ref-audio yue_prompt.wav
zero_shot	Yes	Yes	--ref-audio 10002287-00000094.wav --ref-text ...
podcast	Yes	Yes	two --ref-audio-paths
speech_bgm	Yes	Yes	--ref-audio 00000309-00000300.wav
speech_sound	Yes	Yes	--ref-audio 00000309-00000300.wav

Validated Outputs

The following measurements are retained from an earlier L4 CUDA validation; they are not ROCm benchmark results. Default async_chunk matched blocking output frame counts and Stage-1 patch counts for every case:

Case	Blocking frames / patches / sec	Async chunk frames / patches / sec
style	409248 / 29 / 9.28	409248 / 29 / 9.28
ip	183456 / 13 / 4.16	183456 / 13 / 4.16
bgm	1326528 / 94 / 30.08	1326528 / 94 / 30.08
tta	465696 / 33 / 10.56	465696 / 33 / 10.56
emotion	324576 / 23 / 7.36	324576 / 23 / 7.36
basic	211680 / 15 / 4.80	211680 / 15 / 4.80
dialect	239904 / 17 / 5.44	239904 / 17 / 5.44
zero_shot	409248 / 29 / 9.28	409248 / 29 / 9.28
podcast	437472 / 31 / 9.92	437472 / 31 / 9.92
speech_bgm	296352 / 21 / 6.72	296352 / 21 / 6.72
speech_sound	352800 / 25 / 8.00	352800 / 25 / 8.00

Key Arguments

Argument	Description
--model	Hugging Face repo or local Ming checkpoint path
--deploy-config	Deploy config YAML. Use vllm_omni/deploy/ming_tts.yaml
--case	Built-in demo case
--ref-audio	Single reference wav path for cloning-style cases
--ref-audio-paths	Multiple reference wav paths, used by podcast
--ref-text	Reference transcript. Required for zero_shot
--instructions	Free-form Ming instruction string
--instruction-json	Structured Ming instruction JSON
--speaker-embedding	JSON file containing a 192-d speaker embedding
--extract-speaker-embeddings	Force CampPlus speaker extraction from the provided reference audio paths
--max-decode-steps	Override ming_max_decode_steps
--num-prompts	Repeat the same case N times. Outputs are indexed when N > 1
--streaming	Use AsyncOmni and async_chunk transport
--enforce-eager	Recommended for Ming dense; non-eager is out of scope
--enable-stats / --log-stats	Enable vLLM-Omni per-request stats logging
--stats-log-file	Optional path for the stats log
--metadata-json	Optional path for the run manifest JSON
--stage-init-timeout	Per-stage initialization timeout in seconds
--init-timeout	Total initialization timeout in seconds
--batch-timeout	Batch timeout in seconds
--worker-backend	multi_process or ray
--ray-address	Ray cluster address when using --worker-backend ray

Output

• The script writes one mono 44.1 kHz WAV file per run
• Default output directory: output_audio/
• Default filename: ming_<case>.wav
• When --num-prompts > 1, outputs are indexed as ming_<case>_00000.wav, ..._00001.wav, etc.
• When stats are enabled, the script can also write:
  • a stats log file such as ming_style_pipeline.log
  • a manifest JSON with per-output metadata, stage durations, peak memory info, and streaming client latency metrics when --streaming is used

Notes

• style and ip are zero-speaker paths and do not require a reference clip
• emotion, basic, dialect, speech_bgm, and speech_sound require one reference clip
• zero_shot requires both --ref-audio and --ref-text
• podcast requires at least two reference clips via --ref-audio-paths
• podcast automatically extracts one speaker embedding per reference clip
• --speaker-embedding may contain either one 192-d vector or a list of 192-d vectors
• --enforce-eager was used for the validated runs
• The earlier L4 validation used SDPA for the Ming audio VAE instead of FlashAttention2, which is the preferred default when available.