forward() only popped _stream_gens on normal completion, so cancelled,
timed-out, or preempted requests leaked their generator and skipped the
finally block that unlinks the temp WAV files. Implement
on_requests_finished to close each finished generator, which raises
GeneratorExit inside it and triggers the existing cleanup.
Signed-off-by: Yueqian Lin <linyueqian@outlook.com>
Follow-up to #2958. MOSS-TTS-Nano was authored before the schema refactor
and still shipped as a legacy stage_configs yaml; this aligns it with the
same layout the 5 migrated TTS models now use.
- vllm_omni/model_executor/models/moss_tts_nano/pipeline.py declares
MOSS_TTS_NANO_PIPELINE (single LLM_AR stage, owns_tokenizer, audio
output, stop_token_ids=[2] as a hard EOS backstop).
- vllm_omni/deploy/moss_tts_nano.yaml holds runtime knobs (max_num_seqs,
gpu_memory_utilization, enforce_eager, default_sampling_params,
skip_mm_profiling); trust_remote_code stays at deploy top-level.
- vllm_omni/config/pipeline_registry.py registers the entry so the lazy
registry can resolve it.
- moss_tts_nano/__init__.py exports MossTTSNanoForGeneration (VoxCPM2 pattern).
- Removed vllm_omni/model_executor/stage_configs/moss_tts_nano.yaml.
Examples, shell scripts, READMEs, and buildkite-invoked tests are
updated to use `vllm serve <model> --omni` / `--deploy-config`
(auto-load kicks in; no --stage-configs-path or --trust-remote-code).
Signed-off-by: Yueqian Lin <linyueqian@outlook.com>
- Wire ref_audio through /v1/audio/speech: _build_moss_tts_params is now
async, resolves ref_audio via MediaConnector and passes the waveform as
prompt_audio_array so the model can materialise a temp WAV for upstream
inference_stream (previously voice cloning via the REST endpoint was a
no-op). Serving now also requires ref_text when ref_audio is provided.
- Fix per-request EOS in batched decode: replace the shared
_ar_emit_stop_token bool with a _ar_last_chunk_flags list so
compute_logits emits EOS per row; finished requests no longer wait for
the slowest peer in a max_num_seqs=4 batch.
- Snapshot and restore CPU+CUDA RNG state around torch.manual_seed to
limit global-state bleed; add comment noting deterministic output
under concurrent batching is best-effort (upstream inference_stream
uses the global RNG).
- Align _DEFAULT_MODE with the offline example and tests ("voice_clone").
- Clean up outdated "buffer first" comment in _create_stream_gen; document
single-threaded AR-worker assumption for _stream_gens; add one-liner
explaining the load_weights iterator drain.
Signed-off-by: Yueqian Lin <linyueqian@outlook.com>
The upstream inference_stream() API has no voice/spk parameter;
voice preset selection is not yet wired into the call. Remove the
dead assignment to silence ruff F841.
Signed-off-by: Yueqian Lin <linyueqian@outlook.com>
Switch from GPUGenerationWorker to AR worker with OmniARScheduler,
using the VoxCPM-style generator pattern for streaming:
- inference_stream() stored per-request in _stream_gens dict
- Each forward() call yields one audio chunk via next(generator)
- compute_logits() emits EOS only when last chunk is yielded
- AR scheduler loops until EOS, enabling progressive audio output
TTFP reduced from ~3.1s to ~0.11s (30x improvement).
Signed-off-by: Yueqian Lin <linyueqian@outlook.com>
Integrates OpenMOSS-Team/MOSS-TTS-Nano (0.1B) as a single
GPUGenerationWorker stage. Both the AR LM and MOSS-Audio-Tokenizer-Nano
codec run inside MossTTSNanoForGeneration.forward(), removing the need for
an inter-stage connector.
Key design choices:
- Weights loaded in load_weights() not __init__ (avoids pre-CUDA alloc)
- trust_remote_code delegates to upstream HF model classes
- codec path read from config.audio_tokenizer_pretrained_name_or_path
- inference_stream() collects progressive audio chunks for low latency
- 48 kHz stereo output; voice clone + continuation modes supported
Signed-off-by: Yueqian Lin <linyueqian@outlook.com>
Yueqian Lin