vllm-omni/recipes/OpenMOSS/MOSS-TTS.md

MOSS-TTS

Summary

• Vendor: OpenMOSS
• Models: OpenMOSS-Team/MOSS-TTS (8B), OpenMOSS-Team/MOSS-TTS-v1.5 (8B), OpenMOSS-Team/MOSS-TTS-Realtime (1.7B), OpenMOSS-Team/MOSS-TTSD-v1.0 (8B), OpenMOSS-Team/MOSS-SoundEffect (8B), OpenMOSS-Team/MOSS-VoiceGenerator (1.7B)
• Task: Text-to-speech synthesis, sound effect generation, zero-shot voice design
• Mode: Online serving via the OpenAI-compatible /v1/audio/speech API; offline inference
• Maintainer: Community

When to use this recipe

Use this recipe for 24 kHz multilingual TTS with voice cloning (20 languages including Chinese and English). Choose a variant based on your latency and quality requirements:

Model	Params	Use case
MOSS-TTS	8B	General TTS, highest quality
MOSS-TTS-v1.5	8B	General TTS upgrade of 1.0: 31 languages, steadier cloning, [pause Xs] markers (set language for best results); same MossTTSDelay API
MOSS-TTS-Realtime	1.7B	Lowest latency (TTFB ~180 ms), streaming-first
MOSS-TTSD-v1.0	8B	Multi-turn dialogue TTS
MOSS-SoundEffect	8B	Sound effect synthesis from text description
MOSS-VoiceGenerator	1.7B	Zero-shot voice design

The variants above share the same codec (OpenMOSS-Team/MOSS-Audio-Tokenizer, ~7 GB) and output 24 kHz mono audio.

MOSS-TTS-Local-Transformer-v1.5 uses MOSS-Audio-Tokenizer-v2 and outputs 48 kHz stereo audio. For Local voice cloning through /v1/audio/speech, provide an accurate ref_text transcript alongside ref_audio. The adapter uses the reference transcript and audio as a continuation prefix, then generates only the requested input speech. Without a nonblank ref_text, Local uses audio-reference generation. Reference transcripts must match the reference audio; they are not style instructions.

For the CUDA MRV2 runner and platform fallbacks, see the Local 1.5 deployment profile. Local 1.5 defaults to MRV2 on CUDA and retains V1 on other platforms.

References

• Offline inference example: examples/offline_inference/text_to_speech/moss_tts/
• Deploy configs: vllm_omni/deploy/moss_tts.yaml and variants
• HuggingFace org: https://huggingface.co/OpenMOSS-Team

Hardware Support

GPU

1x H100 80GB — MOSS-TTS (8B)

Environment

• OS: Linux
• Python: 3.11+
• CUDA 12.8
• vLLM-Omni version: see vllm_omni/__version__.py

Command

# The codec is loaded automatically from OpenMOSS-Team/MOSS-Audio-Tokenizer.
# Override the path with MOSS_TTS_CODEC_PATH if you have a local copy.
vllm serve OpenMOSS-Team/MOSS-TTS --omni --port 8091

Verification

Voice cloning (provide a reference audio clip):

curl -X POST http://localhost:8091/v1/audio/speech \
    -H "Content-Type: application/json" \
    -d '{
        "model": "OpenMOSS-Team/MOSS-TTS",
        "input": "Hello, this is a voice cloning test.",
        "voice": "default",
        "ref_audio": "https://raw.githubusercontent.com/OpenMOSS/MOSS-TTS/main/assets/audio/zh_1.wav",
        "response_format": "wav"
    }' --output output.wav

Notes

• Peak GPU memory: ~18 GB for the talker (8B) + ~8 GB for the codec decoder on the same device. Use gpu_memory_utilization: 0.85 in moss_tts.yaml (default).
• Output: 24 kHz mono WAV.
• The MOSS_TTS_CODEC_PATH environment variable overrides the codec checkpoint location.

---

1x A10G 24GB — MOSS-TTS-Realtime (1.7B)

Environment

• OS: Linux
• Python: 3.11+
• CUDA 12.8

Command

vllm serve OpenMOSS-Team/MOSS-TTS-Realtime --omni --port 8091

Verification

curl -X POST http://localhost:8091/v1/audio/speech \
    -H "Content-Type: application/json" \
    -d '{
        "model": "OpenMOSS-Team/MOSS-TTS-Realtime",
        "input": "This is a low-latency streaming TTS test.",
        "voice": "default",
        "ref_audio": "https://raw.githubusercontent.com/OpenMOSS/MOSS-TTS/main/assets/audio/zh_1.wav",
        "response_format": "wav",
        "stream": true,
        "stream_format": "audio"
    }' --output output.wav

Notes

• Peak GPU memory: ~6 GB for the talker (1.7B) + ~8 GB for the codec decoder.
• First-audio latency (TTFB): ~180 ms on A10G.
• codec_chunk_frames: 15 in moss_tts_realtime.yaml for lower TTFA than the 8B variant.

---

1x A10G 24GB — MOSS-SoundEffect (8B, sound effect synthesis)

Command

vllm serve OpenMOSS-Team/MOSS-SoundEffect --omni --port 8091

Verification

Sound effect synthesis takes a text description instead of reference audio:

curl -X POST http://localhost:8091/v1/audio/speech \
    -H "Content-Type: application/json" \
    -d '{
        "model": "OpenMOSS-Team/MOSS-SoundEffect",
        "input": "Thunder rumbling, rain pattering on a tin roof.",
        "response_format": "wav"
    }' --output thunder.wav

Notes

• No ref_audio required or accepted for MOSS-SoundEffect.
• Input field maps to the ambient_sound parameter in the upstream processor.
• Rate: ~12.5 tokens per second; longer descriptions produce longer audio.

Local 1.5 MRV2 and slot attention

MOSS-TTS-Local-Transformer-v1.5 defaults to the native MRV2 pipeline on CUDA, with the original Local projection and sampler. When nvidia-cuda-mps-control is on PATH, it starts private full-quota MPS. GPUs with at least 140 GiB total memory use the C128 system profile with prefix caching, Triton backbone attention, bounded codec first-chunk decode, native Torch sampler compilation and dense codec graph buckets through 128. Its Talker KV budget is 32 GiB; smaller GPUs or a failed memory query use C64 with utilization-based memory budgets. When the MPS executable is unavailable, the automatic default is C64 MRV2 without MPS. NPU, XPU, ROCm and MUSA retain V1. Explicit deploy configs override automatic selection.

CUDA_VISIBLE_DEVICES=0 OMP_NUM_THREADS=1 \
vllm serve OpenMOSS-Team/MOSS-TTS-Local-Transformer-v1.5 --omni \
  --stage-init-timeout 1200 --init-timeout 1500

Use --deploy-config vllm_omni/deploy/moss_tts_local_v1.yaml to select the previous V1 profile explicitly. C128's batch-prefill and direct-token switches remain confined to the throughput profile; they are not enabled in C64.

The same moss_tts_local_mrv2_optimized.yaml system profile can be selected explicitly. It retains the original Local depth projection and sampling algorithm. The codec admits fast first chunks below 32 active streams; crowded streams use the regular decoder with a dispatch target of 16 and a maximum wait of 6 ms. This preserves the low-load first-audio path while coalescing high-load work. local_compile_audio_sampler: true in the Talker HF overrides compiles the native Torch sampler; explicit request generators use the original helper. Set the override to false in a deployment file to disable that compilation. The setting is specific to this C128 system profile. Both stages share one GPU, with a 32 GiB Talker KV budget; this profile requires H200-class memory and nvidia-cuda-mps-control on PATH.

CUDA_VISIBLE_DEVICES=0 OMP_NUM_THREADS=1 \
vllm serve OpenMOSS-Team/MOSS-TTS-Local-Transformer-v1.5 --omni \
  --deploy-config vllm_omni/deploy/moss_tts_local_mrv2_optimized.yaml \
  --stage-init-timeout 1200 --init-timeout 1500

Evaluate WER, speaker similarity and streaming latency for your workload; throughput gains do not establish quality equivalence. Do not apply partial MPS SM quotas: BF16 GEMM outputs were observed incomplete in that configuration on the validation environment. MPS uses an owned control socket or an explicitly supplied operator socket; only the owned daemon is stopped at shutdown.

The explicit C64 MRV2 and capped-C128 throughput profiles remain available. Both profiles below preserve 1-frame initial and 15-frame steady codec chunks and the model's sampling defaults.

vllm serve OpenMOSS-Team/MOSS-TTS-Local-Transformer-v1.5 --omni \
  --deploy-config vllm_omni/deploy/moss_tts_local_mrv2.yaml

This C64 profile bounds Talker prefill to 512 tokens and retains codec-owned CUDA graphs without compiling the codec with Inductor.

For sustained high concurrency on a large-memory CUDA GPU, use:

CUDA_VISIBLE_DEVICES=0 OMP_NUM_THREADS=4 \
vllm serve OpenMOSS-Team/MOSS-TTS-Local-Transformer-v1.5 --omni \
  --deploy-config vllm_omni/deploy/moss_tts_local_mrv2_high_concurrency.yaml \
  --stage-init-timeout 1200 --init-timeout 1500

The high-concurrency profile places both stages on one GPU, sets each stage's capacity to 128 and fixes the Talker KV budget at 32 GiB. The Talker uses mixed FULL CUDA graphs with both its token budget and capture limit set to 512, so new prompt chunks remain within graph coverage. It also enables hf_overrides.mrv2_gpu_slot_state: Local hidden states, audio codes and continuation control stay in GPU request slots, gathered through MRV2's batch mapping. Prefill conditioning and explicit per-request sampling seeds retain their original behavior. Published audio codes own their storage, so slot reuse cannot overwrite an in-flight output. Set the override to false to compare with the generic Omni model state. The profile selects codec_attention_backend: triton_slot, Inductor mode 3 with combo kernels disabled, and codec CUDA graph buckets through 128. The configuration was validated on one H200. The earlier capacity-256 variant used about 110 GiB of sampled peak GPU memory, including loading and warmup. Reduce the stage capacities, graph buckets and coalescing target together when adapting it to smaller GPUs. Cold codec compilation can take several minutes; subsequent starts can reuse the AOT cache. Graph capture is still performed at startup.

The high-concurrency profile also coalesces ready codec streams with connectors.shm.extra.generation_min_batch_size: 32 and generation_max_wait_ms: 12. The target controls when to stop waiting; it neither caps execution at 32 nor guarantees every codec group has 32 rows. First and final chunks share the bounded window at high concurrency. When fewer than 32 requests remain, dispatch is immediate. Pending outputs are retired before waiting, with the original deadline preserved, so waiting does not prevent the previous batch from releasing its in-flight state. Cancellation and input notifications keep their ordinary scheduler bookkeeping. This option requires a stateful native MRV2 generation stage with TP1/PP1. Set the wait to 0 to disable coalescing; other profiles keep the immediate-dispatch default.

The high-concurrency profile enables two additional Local model-state optimizations under the CUDA stage-0 hf_overrides, alongside mrv2_gpu_slot_state: true:

• mrv2_batch_prefill: true reuses the batch's text embeddings and combines CPU reference-code slices and their destination positions into one pinned upload. Host staging is reused only after the upload event completes; GPU-resident references keep the original path.
• mrv2_direct_tokens: true returns the already determined text token after the Local audio/stop decision. It avoids constructing and sampling the full text vocabulary. Audio-code and binary-stop sampling are unchanged. Requests requiring distribution metadata or token constraints, and unsupported execution modes, retain the normal sampler.

Other profiles leave both flags disabled. Set both flags explicitly when comparing their combined effect. Deployment inheritance replaces a stage's hf_overrides mapping, so an overlay must also retain mrv2_gpu_slot_state: true. Measure first-packet latency as well as throughput: faster prompt preparation can change competition between the Talker and codec sharing the GPU.

For the single-H200 capacity-128 throughput baseline, keep both switches on. In complete Seed-TTS EN1088 runs at client concurrency 128, four combined-switch rounds pooled 344.11 audio-s/s versus 335.78 for batch prefill alone; direct tokens alone had no reliable gain. The combined path had 685 ms mean first audio versus 679 ms across six control rounds, with lower mean completion time. A fixed-seed 128-row audio sample showed mean Whisper WER of 3.68% versus 3.78% for the control. This sample does not establish speaker similarity or exclude small quality changes. At concurrency 256, the combined path had raised mean first-audio latency by about 0.33 s, so other profiles leave the switches off until their latency and quality are checked.

The optional connector setting generation_coalescing_policy: idle_wait waits for an inbox notification when no stream is runnable, all receivers are parked and registered, and no output needs retirement. The first ready arrival starts a fresh coalescing window, preserving the batch budget. Control messages, including cancellations, can wait up to two windows instead of one (24 ms at the 12 ms setting). The default policy remains fixed; neither policy changes the global orchestration default.

The codec backends differ in state access:

• sdpa uses PyTorch attention after gathering and updating the ring cache.
• triton replaces the attention calculation, retaining the ring-cache gather/copy and explicit mask construction.
• triton_slot writes surviving K/V directly into request slots, attends directly to the ring, and advances the active slot offsets in three ordered kernels. Graph-padding rows do not advance persistent state.

The slot path preserves the existing chunk-complete ring semantics, including retaining the final cache-capacity tokens when a chunk exceeds the ring. It does not introduce another cache owner or change request-slot lifetime. Zero-length padding rows skip the attention computation and emit zeros.

The high-concurrency profile includes intermediate codec batch buckets 6, 12 and 24. To compare with power-of-two buckets, change the stage-1 cudagraph_capture_sizes to [1, 2, 4, 8, 16, 32, 64, 128]. Keep the maximum bucket equal to the state capacity to retain terminal-tail coalescing. Smaller buckets reduce padding work but require more graphs; measure complete serving runs before selecting them for a deployment.

For a codec output-path comparison, set connectors.shm.extra.codec_gpu_stream_output: 0 to select the synchronous codec output path; 1 selects GPU output snapshots on a private codec stream. Both configurations use MRV2 and keep streaming audio responses. The option changes input ownership, metadata staging and snapshot handling as well as transfer scheduling, so its timing difference is not just D2H copy time.

Those chunk-complete semantics truncate the causal window: every ring holds exactly context entries and a whole chunk is written before attention runs, so token i of a T-token chunk sees capacity - T + i + 1 keys. With the tokenizer-v2 decoder's per-layer contexts (10/10/8/4/2/1 s, i.e. 125/250/400/ 400/400/400 tokens) a 15-frame chunk is 480 tokens at the last transformer, so its first 80 tokens attend to nothing and no chunk sees the previous one there. Padding a terminal tail to 15 frames therefore decodes it differently from an exact-length decode; the difference is deterministic (identical in fp32 and fp64), not bf16 noise. The connector option codec_ring_headroom: 1 sizes each ring as context + max_chunk_frames * tokens_per_frame (including ramp shapes); chunked streaming then reproduces whole-sequence decoding bit-for-bit in fp64 and padded tails equal exact tails. It increases state/graph memory and capture time; historical C256 experiments observed about a 2% throughput cost. It is off by default and its perceptual quality benefit has not been established. Terminal tails are padded into the regular 15-frame graph bucket in either mode. The slot kernel is specific to the CUDA tokenizer-v2 decoder; other paths retain their existing attention implementation. Event-driven orchestration remains independently selectable and its default is unchanged.

Optional progressive chunks

Set codec_chunk_ramp under connectors.shm.extra to insert smaller chunks before steady decoding, for example:

connectors:
  shm:
    extra:
      codec_chunk_ramp: [1, 4, 15]

The ramp's first entry overrides initial_codec_chunk_frames. After the last entry, chunks use codec_chunk_frames. Each request advances independently; the final partial chunk is flushed and an empty terminal carries no codec tokens. The codec captures every ramp length, sizes the maximum execution step for the largest entry, and uses the ramp's first entry for its dedicated first-chunk graph. Additional shapes increase startup and graph memory.

Ramps are opt-in. They can reduce early playback underrun while increasing codec calls, first-audio latency or total generation time. The measured C128 throughput presets retain 1→15; their published figures do not establish an additional benefit from enabling ramps. The processor and graph-shape tests run without GPU or model weights:

CUDA_VISIBLE_DEVICES= PYTHONPATH=. python -m pytest -q \
  tests/model_executor/stage_input_processors/test_moss_tts_async_chunk.py \
  -m 'core_model and cpu' --run-level core_model

Low-latency and reference-encoding options

moss_tts_local_mrv2_low_latency.yaml keeps both stage capacities at 128 and uses prefill 2048, GPU slot state, batch prefill, direct tokens and eager MTP. Its codec first-chunk fast path uses dedicated graphs and a stream handoff before the regular decoder reuses the request slot. A gate limits contention with regular codec work. Slot waits are bounded at 30 seconds. A failed or stalled decode raises an error rather than replaying a partially advanced slot; a failed output enqueue retains owned PCM for regular delivery. Closing the decoder rejects new jobs. Its regular-batch limit of 16 is internal dispatch policy; it does not change the client concurrency or stage capacities.

Reference encoding runs in the API layer, independently of MRV2. Enable reference graphs explicitly with VLLM_OMNI_MOSS_REF_GRAPHS=1; the default eager encoder is retained when the option is absent. The graph path uses the loaded tokenizer's encoder/quantizer, length buckets and optional compilation (VLLM_OMNI_MOSS_REF_COMPILE=0 disables compilation). It also uses windowed attention; VLLM_OMNI_MOSS_REF_ATTN=sdpa retains the original attention. Compilation and attention changes need not produce bit-identical codes.

For multiple API processes, VLLM_OMNI_MOSS_REF_CODES_SHARED_DIR enables shared reference-code storage and, by default, a single encoder host with four workers. Use a dedicated directory per service, checkpoint and encoding configuration. Workers have their own graph resources; all graph captures complete before serving begins. VLLM_OMNI_MOSS_REF_SHARED_ENCODER=0 retains separate encoders while sharing codes. More workers/graphs consume memory; they do not imply more GPUs. Example after selecting an available GPU and configuring any operator-managed MPS daemon:

VLLM_WORKER_MULTIPROC_METHOD=spawn \
VLLM_OMNI_EVENT_DRIVEN_ORCH=1 \
VLLM_OMNI_CONNECTOR_RECV_POLL_MS=1 \
VLLM_OMNI_MOSS_REF_GRAPHS=1 \
VLLM_OMNI_MOSS_REF_ENCODER_WORKERS=4 \
VLLM_OMNI_MOSS_REF_BATCH_WINDOW_MS=0 \
VLLM_OMNI_MOSS_REF_INFLIGHT=4 \
VLLM_OMNI_MOSS_REF_HOST_WINDOW_MS=2 \
VLLM_OMNI_MOSS_REF_CODES_SHARED_DIR=/dev/shm/moss-local-service \
vllm serve OpenMOSS-Team/MOSS-TTS-Local-Transformer-v1.5 --omni \
  --api-server-count 4 \
  --deploy-config vllm_omni/deploy/moss_tts_local_mrv2_low_latency.yaml \
  --stage-init-timeout 1200 --init-timeout 1500

This example is a configurable serving profile, not the complete launch configuration of a historical benchmark. In particular, the low-latency YAML uses synchronous stage-1 scheduling; the retained source46 experiment used asynchronous stage-1 scheduling, FP8 backbone, MPS and additional encoder environment settings. Do not infer a throughput result from the YAML alone.

Cold versus hot reference measurements

A cold reference misses the encoded-reference cache: the request pays for reference parsing/preparation and GPU encoding before speech generation. A hot reference reuses reference codes but still synthesizes new speech; generated audio is not cached. Pre-registering a reference moves encoding into registration and can make subsequent synthesis hot, but registration latency is still part of the first-use cost.

Model startup/compilation is separate from reference coldness. EN1088 contains repeated references, so its first pass is not uniformly cold. For an all-unique cold pass, use its 666 first-occurrence references, keep order, warm execution with disjoint references, then repeat the same 666 requests. A second pass through independent API-local caches is not guaranteed hot; verify cache coverage before comparing it with a shared-cache result. Report requests/s alongside audio-s/s and retain failed, empty, long and near-silent outputs.

Reproduce the serving benchmark

Use the complete Seed-TTS English test set, including reference audio and text. Set SEED_TTS_DATA to the directory containing en/meta.lst and its 1088 entries. Run the native benchmark after the server is ready:

export VLLM_OMNI_BENCH_AUDIO_SAMPLE_RATE=48000
export VLLM_OMNI_BENCH_AUDIO_CHANNELS=2
export SEED_TTS_WER_EVAL=0

for concurrency in 128; do
  for phase in warm r1 r2; do
    vllm bench serve --omni \
      --model OpenMOSS-Team/MOSS-TTS-Local-Transformer-v1.5 \
      --backend openai-audio-speech --endpoint /v1/audio/speech \
      --dataset-name seed-tts --dataset-path "$SEED_TTS_DATA" \
      --seed-tts-locale en --disable-shuffle --num-prompts 1088 \
      --num-warmups 0 --ready-check-timeout-sec 0 \
      --output-len 256 --max-concurrency "$concurrency" \
      --request-rate inf --seed 42 \
      --extra-body '{"task_type":"Base","max_new_tokens":256}' \
      --save-result --save-detailed --result-dir results/moss-local \
      --result-filename "c${concurrency}-${phase}.json"
  done
done

Discard the complete warm pass and combine measured runs as total generated audio seconds divided by total benchmark duration. Require 1088 successes, zero failures and 1088 nonempty-audio metric samples in each run. The output cap matches the benchmark protocol; success alone does not establish speech quality or that every sentence ended before the cap. Dataset seed 42 does not fix an independent sampling seed for every request.

For an attention-only comparison, copy the high-concurrency YAML beside the original to preserve relative base_config resolution. Change its codec compilation_config.mode to 0 while retaining cudagraph_mode: FULL, and compare codec_attention_backend: triton against triton_slot. Keep all other settings, warmup and client concurrency identical. Restart the server between configurations and use separate result directories. Comparing the compiled slot profile to uncompiled Triton includes both changes.

Kernel and MHA regression tests require CUDA, compatible vLLM/Triton packages, and no model weights:

python -m pytest -q tests/model_executor/models/moss_tts/test_slot_attention.py \
  tests/model_executor/models/moss_tts/test_streaming_attention.py \
  -m 'core_model and cuda' --run-level=core_model