Merge main and preserve multi-API engine and duplex guards

Signed-off-by: Sy03 <1370724210@qq.com>

examples/offline_inference/text_to_speech/README.md

@@ -19,7 +19,7 @@ list of supported architectures across all modalities, see
 | Breeze-TTS-2 | `BreezeBlue/Breeze-TTS-2` | 2 (talker + codec) | ✓ | ✓ (async chunk) | voice design / voice direction (`--instruction`) | 24 kHz |
 | CosyVoice3 | `FunAudioLLM/Fun-CosyVoice3-0.5B-2512` | 2 (talker + code2wav) | ✓ | ✓ | — | 24 kHz |
 | Fish Speech S2 Pro | `fishaudio/s2-pro` | dual-AR | ✓ | ✓ | — | 44.1 kHz |
-| Gepard-1.0 | `nineninesix/gepard-1.0` | single (native AR + NanoCodec) | — (zero-shot; cloning WIP) | — (serving WIP) | zero-shot | 22.05 kHz |
+| Gepard-1.0 | `nineninesix/gepard-1.0` | single (native AR + NanoCodec) | — (zero-shot; cloning later) | ✓ (online) | zero-shot | 22.05 kHz |
 | GLM-TTS | `zai-org/GLM-TTS` | 2 (AR + DiT) | ✓ (required) | ✓ | — | 24 kHz |
 | Ming-omni-tts | `inclusionAI/Ming-omni-tts-0.5B` | 2 (AR + audio VAE) | ✓ | ✓ | style / IP / dialect / TTA / podcast | 44.1 kHz |
 | Ming-flash-omni-TTS | `Jonathan1909/Ming-flash-omni-2.0` | single (talker only) | — (caption-controlled) | — | style / IP / basic captions | 44.1 kHz |
@@ -167,7 +167,7 @@ python examples/offline_inference/text_to_speech/glm_tts/end2end.py \
 ```
 
 ### Architecture
-```
+```text
 Text → [Stage 0: AR] → Speech Tokens → [Stage 1: DiT + HiFT] → Audio (24 kHz)
         (Llama-based)    (32k vocab)      (Flow Matching)
 ```
@@ -491,8 +491,8 @@ Not yet — PR1 is zero-shot only (the learned `null_prefix` default voice). Ref
 ### Notes
 - Output: 22.05 kHz mono WAV.
 - First run downloads two checkpoints, not one: the model itself, and the NanoCodec decoder it names in `codec_id` (`nvidia/nemo-nano-codec-22khz-1.89kbps-21.5fps` by default). Both are fetched automatically; expect the first startup to be correspondingly slower.
-- Offline only for now; the `/v1/audio/speech` adapter is a follow-up PR.
-- Deploy config: `vllm_omni/deploy/gepard.yaml` (the example's default; copy it and pass `--deploy-config` to change it).
+- Offline and online: `examples/offline_inference/text_to_speech/gepard/end2end.py` and `/v1/audio/speech` (see the [online Gepard section](../../online_serving/text_to_speech/README.md#gepard-10)).
+- Deploy config: `vllm_omni/deploy/gepard.yaml` (the example's default; copy it and pass `--deploy-config` to change it). Online serving **must** pass `--deploy-config vllm_omni/deploy/gepard.yaml` because the checkpoint self-identifies as `qwen3_5_text`.
 - Generation length and reproducibility are stage settings, not script flags. One output token is one audio frame, so `max_tokens` in the YAML is the frame budget; `seed` makes the in-model 32-head sampling reproducible. The script deliberately passes no `SamplingParams`: one supplied by a caller replaces the stage defaults wholesale rather than merging, which would drop the pipeline's stop token and run every request to `max_tokens`.
 - Text length: short texts are repeated internally to match the training layout (the checkpoint's `text_repetition` block); the upper bound is the stage's `max_model_len`, enforced by the engine. Empty text is rejected rather than voiced.
 - `VLLM_GEPARD_GREEDY=1` swaps the 32-head Gumbel-max sampling for argmax, for reproducible comparisons that do not depend on a seed.

examples/online_serving/barge_in_client.py

@@ -14,7 +14,10 @@ Scenario (see ``barge_in_client_flow.md`` for the diagram)
 
 The demo is model-neutral: it drives any duplex model through the public
 :class:`vllm_omni.clients.duplex.DuplexClient` over
-``/v1/realtime?duplex=1``. Model-specific session shape comes from the
+``/v1/realtime?duplex=1``, or — with ``--inline`` — through
+:class:`vllm_omni.clients.inline_duplex.InlineDuplexClient` on an in-process
+:class:`vllm_omni.entrypoints.duplex_omni.DuplexOmni` (no server needed; the
+model loads in this process). Model-specific session shape comes from the
 per-model presets under ``vllm_omni.clients.<model>`` selected with
 ``--preset`` (default ``minicpmo_4_5``, whose model decides listen/speak on
 its own — no VAD).
@@ -31,6 +34,15 @@ then:
         --interrupt-wav follow_up_16k.wav \
         --output-dir ./duplex_out
 
+or, without a server, in-process (``--deploy-config`` is optional and defaults
+to the model's bundled duplex profile):
+
+    python barge_in_client.py --inline \
+        --model /path/to/MiniCPM-o-4_5 \
+        --ref-audio /path/to/reference_voice.wav \
+        --question-wav question_16k.wav \
+        --interrupt-wav follow_up_16k.wav
+
 Input WAVs must be mono 16 kHz PCM16; when a preset's session takes a
 different capture format (the PersonaPlex preset streams 24 kHz float32),
 the audio is converted before streaming. ``--ref-audio`` is required by the
@@ -55,6 +67,7 @@ if str(REPO_ROOT) not in sys.path:
 from vllm_omni.clients.duplex import (  # noqa: E402
     AudioFormat,
     DuplexClient,
+    DuplexClientBase,
     EventCollector,
     SessionConfig,
     acknowledge_collected_playback,
@@ -139,6 +152,39 @@ def _fold_responses(collector: EventCollector) -> dict[str, dict[str, object]]:
     return out
 
 
+def _make_client(args: argparse.Namespace, config: SessionConfig) -> DuplexClientBase:
+    """Websocket client by default; ``--inline`` drives an in-process DuplexOmni."""
+    if not args.inline:
+        return DuplexClient(
+            args.url,
+            model=args.model,
+            config=config,
+            reconnect=None,
+            heartbeat_interval_s=None,
+            handshake_timeout_s=args.timeout_s,
+        )
+    # Imported lazily: the in-process path loads the model (and its engine
+    # dependencies) in this process, which the websocket path never needs.
+    from vllm_omni.clients.inline_duplex import InlineDuplexClient
+    from vllm_omni.entrypoints.duplex_omni import DuplexOmni
+
+    # Models shipped with custom HF code (MiniCPM-o) need trust_remote_code to
+    # resolve their pipeline; harmless for the others.
+    omni_kwargs: dict[str, object] = {"model": args.model, "trust_remote_code": True}
+    if args.deploy_config:
+        omni_kwargs["deploy_config"] = args.deploy_config
+    omni = DuplexOmni(**omni_kwargs)
+    client = InlineDuplexClient(
+        omni,
+        model=args.model,
+        config=config,
+        handshake_timeout_s=args.timeout_s,
+    )
+    # The client owns no engine; stop the in-process one when the run ends.
+    client.owned_omni = omni  # type: ignore[attr-defined]
+    return client
+
+
 async def run(args: argparse.Namespace) -> int:
     config = _session_config(args.preset, ref_audio_path=args.ref_audio)
     question = _convert_to_session_format(read_pcm16_wav(Path(args.question_wav)), config.input_audio)
@@ -148,105 +194,117 @@ async def run(args: argparse.Namespace) -> int:
     output_dir.mkdir(parents=True, exist_ok=True)
 
     collector = EventCollector()
-    client = DuplexClient(
-        args.url,
-        model=args.model,
-        config=config,
-        reconnect=None,
-        heartbeat_interval_s=None,
-        handshake_timeout_s=args.timeout_s,
-    )
+    client = _make_client(args, config)
     # GREET: open the voice session (handshake happens on enter).
-    async with client:
-        consume_task = asyncio.create_task(collector.consume(client))
+    try:
+        async with client:
+            consume_task = asyncio.create_task(collector.consume(client))
 
-        # ASK: stream the question at realtime pace, then commit the turn.
-        # The model decides on its own to answer (no VAD).
-        print("[1] streaming question ...", file=sys.stderr)
-        await client.stream_pcm(question, chunk_ms=chunk_ms)
-        await client.commit(create_response=False)
-        await wait_for_condition(
-            lambda: collector.count("response.created") > 0,
-            timeout_s=args.timeout_s,
-            label="first response.created",
-        )
-        print("[1] assistant is answering", file=sys.stderr)
+            # ASK: stream the question at realtime pace, then commit the turn.
+            # The model decides on its own to answer (no VAD).
+            print("[1] streaming question ...", file=sys.stderr)
+            await client.stream_pcm(question, chunk_ms=chunk_ms)
+            await client.commit(create_response=False)
+            await wait_for_condition(
+                lambda: collector.count("response.created") > 0,
+                timeout_s=args.timeout_s,
+                label="first response.created",
+            )
+            print("[1] assistant is answering", file=sys.stderr)
 
-        # Let the answer play for a moment, as a listening user would.
-        await asyncio.sleep(args.listen_s)
+            # Let the answer play for a moment, as a listening user would.
+            await asyncio.sleep(args.listen_s)
 
-        # INTERRUPT: talk over the assistant while its audio still streams.
-        # The serving overlap policy decides: enough overlapped speech barges
-        # in (the first response is cancelled) or a short remark is deferred
-        # until the model yields the turn.
-        print("[2] interrupting mid-answer ...", file=sys.stderr)
-        ids_before_interrupt = set(collector.response_ids)
-        await client.stream_pcm(interrupt, chunk_ms=chunk_ms)
-        await client.commit(create_response=False)
+            # INTERRUPT: talk over the assistant while its audio still streams.
+            # The serving overlap policy decides: enough overlapped speech barges
+            # in (the first response is cancelled) or a short remark is deferred
+            # until the model yields the turn.
+            print("[2] interrupting mid-answer ...", file=sys.stderr)
+            ids_before_interrupt = set(collector.response_ids)
+            await client.stream_pcm(interrupt, chunk_ms=chunk_ms)
+            await client.commit(create_response=False)
 
-        # ANSWER AGAIN: wait for a response created *after* the interruption
-        # to finish. The first answer may still be streaming here; matching on
-        # any response.done would let its completion satisfy the wait and cut
-        # off the actual follow-up answer at close.
-        def follow_up_answer_done() -> bool:
-            return any(
-                event.get("type") == "response.done"
-                and collector.response_id(event) is not None
-                and collector.response_id(event) not in ids_before_interrupt
-                for event in collector.events
+            # ANSWER AGAIN: wait for a response created *after* the interruption
+            # to finish. The first answer may still be streaming here; matching on
+            # any response.done would let its completion satisfy the wait and cut
+            # off the actual follow-up answer at close.
+            def follow_up_answer_done() -> bool:
+                return any(
+                    event.get("type") == "response.done"
+                    and collector.response_id(event) is not None
+                    and collector.response_id(event) not in ids_before_interrupt
+                    for event in collector.events
+                )
+
+            await wait_for_condition(
+                follow_up_answer_done,
+                timeout_s=args.timeout_s,
+                label="response.done for the response answering the interruption",
             )
 
-        await wait_for_condition(
-            follow_up_answer_done,
-            timeout_s=args.timeout_s,
-            label="response.done for the response answering the interruption",
-        )
+            # HANG UP: report playback, close the session cleanly.
+            await acknowledge_collected_playback(client, collector)
+            await client.close(timeout_s=args.timeout_s)
+            try:
+                await asyncio.wait_for(consume_task, timeout=5.0)
+            except (asyncio.TimeoutError, asyncio.CancelledError):
+                consume_task.cancel()
 
-        # HANG UP: report playback, close the session cleanly.
-        await acknowledge_collected_playback(client, collector)
-        await client.close(timeout_s=args.timeout_s)
-        try:
-            await asyncio.wait_for(consume_task, timeout=5.0)
-        except (asyncio.TimeoutError, asyncio.CancelledError):
-            consume_task.cancel()
-
-    # Report: one WAV per response; statuses show what the overlap did.
-    responses = _fold_responses(collector)
-    summary: list[dict[str, object]] = []
-    for index, (response_id, item) in enumerate(responses.items(), start=1):
-        audio = item["audio"]
-        assert isinstance(audio, bytes)
-        sample_rate_hz = int(item["sample_rate_hz"])  # type: ignore[arg-type]
-        wav_path = output_dir / f"response_{index}_{item['status']}.wav"
-        if audio:
-            write_pcm16_wav(wav_path, audio, sample_rate_hz=sample_rate_hz)
-        summary.append(
-            {
-                "response_id": response_id,
-                "status": item["status"],
-                "audio_s": round(len(audio) / (sample_rate_hz * 2), 2),
-                "text": item["text"],
-                "wav": wav_path.name if audio else None,
-            }
+        # Report: one WAV per response; statuses show what the overlap did.
+        responses = _fold_responses(collector)
+        summary: list[dict[str, object]] = []
+        for index, (response_id, item) in enumerate(responses.items(), start=1):
+            audio = item["audio"]
+            assert isinstance(audio, bytes)
+            sample_rate_hz = int(item["sample_rate_hz"])  # type: ignore[arg-type]
+            wav_path = output_dir / f"response_{index}_{item['status']}.wav"
+            if audio:
+                write_pcm16_wav(wav_path, audio, sample_rate_hz=sample_rate_hz)
+            summary.append(
+                {
+                    "response_id": response_id,
+                    "status": item["status"],
+                    "audio_s": round(len(audio) / (sample_rate_hz * 2), 2),
+                    "text": item["text"],
+                    "wav": wav_path.name if audio else None,
+                }
+            )
+        (output_dir / "summary.json").write_text(json.dumps(summary, indent=2, ensure_ascii=False))
+        (output_dir / "events.jsonl").write_text(
+            "".join(json.dumps(event, ensure_ascii=False) + "\n" for event in collector.events),
+            encoding="utf-8",
         )
-    (output_dir / "summary.json").write_text(json.dumps(summary, indent=2, ensure_ascii=False))
-    (output_dir / "events.jsonl").write_text(
-        "".join(json.dumps(event, ensure_ascii=False) + "\n" for event in collector.events),
-        encoding="utf-8",
-    )
-    print(json.dumps(summary, indent=2, ensure_ascii=False))
-    cancelled = sum(1 for item in summary if item["status"] == "cancelled")
-    print(
-        f"\n{len(summary)} responses ({cancelled} interrupted); outputs in {output_dir}",
-        file=sys.stderr,
-    )
+        print(json.dumps(summary, indent=2, ensure_ascii=False))
+        cancelled = sum(1 for item in summary if item["status"] == "cancelled")
+        print(
+            f"\n{len(summary)} responses ({cancelled} interrupted); outputs in {output_dir}",
+            file=sys.stderr,
+        )
+    finally:
+        owned_omni = getattr(client, "owned_omni", None)
+        if owned_omni is not None:
+            owned_omni.shutdown()
     return 0
 
 
 def main() -> int:
     parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
     parser.add_argument("--url", default="ws://127.0.0.1:8099/v1/realtime")
-    parser.add_argument("--model", default="openbmb/MiniCPM-o-4_5")
+    parser.add_argument(
+        "--model",
+        default="openbmb/MiniCPM-o-4_5",
+        help="served model name, or the model path with --inline",
+    )
+    parser.add_argument(
+        "--inline",
+        action="store_true",
+        help="run the model in this process through DuplexOmni + InlineDuplexClient instead of connecting to --url",
+    )
+    parser.add_argument(
+        "--deploy-config",
+        default=None,
+        help="deploy YAML for --inline (default: the model's bundled profile)",
+    )
     parser.add_argument(
         "--preset",
         choices=["minicpmo_4_5", "personaplex", "none"],

examples/online_serving/barge_in_client_flow.md

@@ -1,8 +1,10 @@
 # `barge_in_client.py` — what happens, at a glance
 
-The demo plays one everyday conversation pattern against the full-duplex
-server: ask something, get talked-over mid-answer, and see the assistant
-cope. High-level flow below — for the exact wire events, read the
+The demo plays one everyday conversation pattern against a full-duplex
+model: ask something, get talked-over mid-answer, and see the assistant
+cope. It runs either against a server over `/v1/realtime?duplex=1` or, with
+`--inline`, against an in-process `DuplexOmni` — the conversation below is the
+same in both modes. High-level flow — for the exact wire events, read the
 docstring in `barge_in_client.py`.
 
 ```mermaid
@@ -48,7 +50,8 @@ answer is saved as `response_1_cancelled.wav`, a completed one as
 duration per answer.
 
 Companion: the runtime architecture lives in
-[`docs/design/fullduplex.md`](../../docs/design/fullduplex.md); the client
-API is `vllm_omni.clients.duplex.DuplexClient`, documented with the wire
-protocol in
+[`docs/design/fullduplex.md`](../../docs/design/fullduplex.md); the client API is
+`vllm_omni.clients.duplex.DuplexClient` over the websocket and
+`vllm_omni.clients.inline_duplex.InlineDuplexClient` in-process — both are
+documented with the wire protocol in
 [`docs/serving/realtime_duplex_api.md`](../../docs/serving/realtime_duplex_api.md).

examples/online_serving/minicpmo/barge_in_serve.sh

@@ -5,10 +5,10 @@
 # Server side for barge_in_client.py: start MiniCPM-o 4.5 in native
 # full-duplex mode.
 #
-# The duplex deploy profile registers the WS /v1/duplex endpoint (with
-# /v1/realtime?duplex=1 projecting onto the same handler), enables the
-# engine duplex control plane, and bounds the Thinker/Talker/Code2Wav
-# stages to four live duplex sessions on one GPU. See
+# MiniCPM-o 4.5 declares a duplex plugin, so vllm-omni serve runs it through
+# DuplexOmni: the server is duplex-only (WS /v1/realtime?duplex=1, alias
+# /v1/duplex, plus /v1/models and /health) and bounds the
+# Thinker/Talker/Code2Wav stages to four live duplex sessions on one GPU. See
 # vllm_omni/deploy/minicpmo_4_5.yaml (session_mode: duplex) for the session
 # limits; every other duplex_session knob keeps its runtime default
 # (idle TTL 300s, disconnect grace 30s, 16MiB pending input per session).
@@ -22,4 +22,4 @@ exec vllm-omni serve "$MODEL" \
     --deploy-config vllm_omni/deploy/minicpmo_4_5.yaml \
     --trust-remote-code \
     --host 0.0.0.0 --port "$PORT"
-# duplex endpoints: ws://<host>:$PORT/v1/duplex and /v1/realtime?duplex=1
+# duplex endpoint: ws://<host>:$PORT/v1/realtime?duplex=1 (alias /v1/duplex)

examples/online_serving/minicpmo/realtime_duplex_demo.py

@@ -445,7 +445,6 @@ async def run_demo(args: argparse.Namespace) -> dict[str, object]:
             ref_audio=_ref_audio_data_url(args.ref_audio),
             temperature=args.temperature,
         ),
-        session_id=args.session_id,
         reconnect=None,
         heartbeat_interval_s=None,
         handshake_timeout_s=args.timeout_s,
@@ -638,7 +637,6 @@ def parse_args() -> argparse.Namespace:
     parser = argparse.ArgumentParser()
     parser.add_argument("--url", default="ws://localhost:8099/v1/realtime?duplex=1")
     parser.add_argument("--model", default="openbmb/MiniCPM-o-4_5")
-    parser.add_argument("--session-id")
     parser.add_argument(
         "--input-wav",
         help="Mono 16 kHz PCM16 clip to stream. Optional when --input-video carries the audio track.",

examples/online_serving/minicpmo/realtime_web/app/static/app.js

@@ -86,7 +86,6 @@
     url.protocol = window.location.protocol === 'https:' ? 'wss:' : 'ws:';
     url.searchParams.set('duplex', '1');
     url.searchParams.set('model', config.model || 'openbmb/MiniCPM-o-4_5');
-    url.searchParams.set('native_duplex', '1');
     url.searchParams.set('autostart', '0');
     return url.toString();
   }
@@ -458,7 +457,6 @@
         settled = true;
         const extraBody = {
           auto_response: true,
-          native_duplex: true,
         };
         const session = {
           modalities: ['audio', 'text'],

examples/online_serving/text_to_speech/README.md

@@ -17,6 +17,7 @@ For the full list of supported architectures across all modalities, see
 | Breeze-TTS-2 | `BreezeBlue/Breeze-TTS-2` | ✓ (`ref_audio`+`ref_text`) | ✓ (PCM stream) | speaker tags (`S0`..`S9`, default `S0`) | — |
 | Audio8 TTS Preview | `Audio8/Audio8-TTS-Preview-0.6b` | ✓ (`ref_audio`+`ref_text`) | ✓ (PCM stream) | uploaded audio voice only; no presets | ✓ |
 | Fish Speech S2 Pro | `fishaudio/s2-pro` | ✓ (`ref_audio`+`ref_text`) | ✓ (PCM stream) | — | ✓ |
+| Gepard-1.0 | `nineninesix/gepard-1.0` | — (zero-shot default voice) | ✓ (PCM / WAV stream) | `default` only | — |
 | GLM-TTS | `zai-org/GLM-TTS` | ✓ (`ref_audio`+`ref_text`, required) | ✓ (PCM stream) | — | ✓ |
 | IndexTTS-2 | `IndexTeam/IndexTTS-2` | ✓ (`ref_audio` or uploaded `voice`) | `stream=true` response, non-chunk | uploaded audio voice only; no presets | — |
 | IndexTTS-2.5 | native `checkpoints/` bundle | ✓ (`ref_audio` or uploaded `voice`) | `stream=true` response, non-chunk | uploaded audio voice only; no presets | — |
@@ -96,12 +97,52 @@ curl -X POST http://localhost:8091/v1/audio/speech \
     }' --no-buffer | play -t raw -r 24000 -e signed -b 16 -c 1 -
 ```
 
-Adjust the player's sample rate to match the model (44.1 kHz for Fish Speech, 48 kHz for VoxCPM2, 22.05 kHz for IndexTTS-2, and 24 kHz for many others).
+Adjust the player's sample rate to match the model (44.1 kHz for Fish Speech, 48 kHz for VoxCPM2, 22.05 kHz for Gepard and IndexTTS-2, and 24 kHz for many others).
 
 For full request-shape documentation (all parameters, response formats, error codes), see the [Speech API reference](../../../docs/serving/speech_api.md).
 
 ---
 
+## Gepard-1.0
+
+Single-stage native AR TTS at 22.05 kHz mono. Zero-shot only: omit `voice` or pass `"default"`. Voice cloning from reference audio is not available yet.
+
+### Prerequisites
+
+Same NeMo NanoCodec install as the [offline Gepard section](../../offline_inference/text_to_speech/README.md#gepard-10). On a host whose CUDA toolkit cannot build kernels — no `nvcc`/`ninja`, or a consumer Blackwell (`sm_120`) card — also `export VLLM_USE_FLASHINFER_SAMPLER=0` before launch (same note as offline). That env is not a deploy-YAML field.
+
+### Launch
+
+```bash
+vllm-omni serve nineninesix/gepard-1.0 --omni --port 8091 --trust-remote-code \
+    --stage-init-timeout 900 \
+    --deploy-config vllm_omni/deploy/gepard.yaml
+# or:
+./gepard/run_server.sh
+```
+
+`--stage-init-timeout 900` matches the online e2e fixture and the offline example. Serve defaults to 300s, which is often too short for a cold download of the talker plus NanoCodec.
+
+The packaged `vllm_omni/deploy/gepard.yaml` must be passed with `--deploy-config`. The checkpoint self-identifies as `qwen3_5_text`, so omitting the YAML launches a diffusion fallback instead of the Gepard pipeline. The YAML sets `async_chunk: false`, `max_num_seqs: 4`, and currently pins `seed: 42`, so serving is deterministic by default until that YAML seed is removed. Pass an explicit per-request `seed` in tests and clients rather than depending on either default.
+
+### Sending requests
+
+```bash
+python examples/online_serving/text_to_speech/gepard/speech_client.py \
+    --text "Hello, this is Gepard speaking."
+
+python examples/online_serving/text_to_speech/gepard/speech_client.py \
+    --text "Hello, this is Gepard speaking." --seed 7 --stream --output output.pcm
+```
+
+### Notes
+
+- Output: 22.05 kHz mono. `max_new_tokens` is a **frame** budget (1 token = 1 frame = 1024 samples ≈ 46.4 ms at 21.5 fps; adapter bounds 1..4096).
+- Supported request fields: `input` (required), `voice` (`default` only), `response_format` (`wav` default; `wav/pcm/flac/mp3` non-streaming; `opus` 400 because 22.05 kHz is not an Opus sample rate; streaming `pcm/wav` only), `stream` / `stream_format`, `max_new_tokens`, `seed`.
+- Unsupported: `speed`, `extra_params` (including `temperature`/`top_p`/`top_k`), `ref_audio`, `ref_text`, `speaker_embedding`, `task_type`, `instructions`, `language`, and `word_timestamps`.
+- Concurrent requests at `max_num_seqs: 4` are supported. Native-AR recompute preemption is a known limitation of this architecture (a request that is preempted mid-generation can resume incorrectly); keep concurrency at or below `max_num_seqs` and treat preemption as out of scope until the platform fix lands.
+- Optional comparison against the upstream Gepard reference server needs Blackwell/Hopper + CUDA 13 + Postgres and is not part of CI.
+
 ## dots.tts
 
 Single-stage, text-only TTS at 48 kHz. The current integration supports

examples/online_serving/text_to_speech/gepard/run_server.sh

@@ -0,0 +1,33 @@
+#!/bin/bash
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
+
+# Launch vLLM-Omni server for Gepard-1.0 TTS (zero-shot default voice).
+#
+# Usage:
+#   ./run_server.sh
+#   CUDA_VISIBLE_DEVICES=0 ./run_server.sh
+#
+# Packaged deploy config: vllm_omni/deploy/gepard.yaml (async_chunk=false,
+# 22.05 kHz mono, default seed 42 until the YAML seed is removed).
+# Cold start downloads the talker and NeMo NanoCodec; the serve default of
+# --stage-init-timeout 300 is too tight (offline end2end.py uses 900).
+#
+# On a host whose CUDA toolkit cannot JIT FlashInfer (no nvcc/ninja, or a
+# consumer Blackwell sm_120 card), also:
+#   export VLLM_USE_FLASHINFER_SAMPLER=0
+
+set -e
+
+MODEL="${MODEL:-nineninesix/gepard-1.0}"
+PORT="${PORT:-8091}"
+
+echo "Starting Gepard-1.0 server with model: $MODEL"
+
+vllm-omni serve "$MODEL" \
+    --host 0.0.0.0 \
+    --port "$PORT" \
+    --trust-remote-code \
+    --omni \
+    --stage-init-timeout 900 \
+    --deploy-config vllm_omni/deploy/gepard.yaml

examples/online_serving/text_to_speech/gepard/speech_client.py

@@ -0,0 +1,116 @@
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
+
+"""Client for Gepard-1.0 TTS via /v1/audio/speech.
+
+Gepard is zero-shot: omit ``voice`` or pass ``"default"``. Output is 22.05 kHz
+mono. ``seed`` is optional; the packaged deploy YAML currently pins ``seed: 42``
+until that default is removed, so requests without an explicit seed are still
+deterministic.
+
+Examples:
+    python speech_client.py --text "Hello, this is Gepard speaking."
+    python speech_client.py --text "Hello, this is Gepard speaking." --seed 7
+    python speech_client.py --text "Hello, this is Gepard speaking." --stream --output output.pcm
+"""
+
+from __future__ import annotations
+
+import argparse
+
+import httpx
+
+DEFAULT_API_BASE = "http://localhost:8091"
+DEFAULT_API_KEY = "EMPTY"
+DEFAULT_MODEL = "nineninesix/gepard-1.0"
+
+
+def run_tts(args) -> None:
+    payload = {
+        "model": args.model,
+        "input": args.text,
+        "voice": args.voice,
+        "response_format": args.response_format,
+    }
+    if args.seed is not None:
+        payload["seed"] = args.seed
+    if args.max_new_tokens is not None:
+        payload["max_new_tokens"] = args.max_new_tokens
+    if args.stream:
+        payload["stream"] = True
+        payload["stream_format"] = "audio"
+        if args.response_format not in ("pcm", "wav"):
+            print(f"Note: streaming requires pcm/wav; overriding --response-format {args.response_format} to 'pcm'")
+            payload["response_format"] = "pcm"
+
+    print(f"Model: {args.model}")
+    print(f"Text: {args.text}")
+    print(f"Voice: {args.voice}")
+    if args.seed is not None:
+        print(f"Seed: {args.seed}")
+    print("Generating audio...")
+
+    api_url = f"{args.api_base}/v1/audio/speech"
+    headers = {
+        "Content-Type": "application/json",
+        "Authorization": f"Bearer {args.api_key}",
+    }
+
+    if args.stream:
+        output_path = args.output or ("output.wav" if payload["response_format"] == "wav" else "output.pcm")
+        with httpx.Client(timeout=300.0) as client:
+            with client.stream("POST", api_url, json=payload, headers=headers) as resp:
+                if resp.status_code != 200:
+                    print(f"Error: {resp.status_code}")
+                    print(resp.read().decode())
+                    return
+                total_bytes = 0
+                with open(output_path, "wb") as f:
+                    for chunk in resp.iter_bytes():
+                        f.write(chunk)
+                        total_bytes += len(chunk)
+                print(f"Streamed {total_bytes} bytes to: {output_path}")
+        return
+
+    with httpx.Client(timeout=300.0) as client:
+        response = client.post(api_url, json=payload, headers=headers)
+
+    if response.status_code != 200:
+        print(f"Error: {response.status_code}")
+        print(response.text)
+        return
+
+    output_path = args.output or "output.wav"
+    with open(output_path, "wb") as f:
+        f.write(response.content)
+    print(f"Audio saved to: {output_path}")
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser(description="Gepard-1.0 TTS client")
+    parser.add_argument("--api-base", default=DEFAULT_API_BASE, help="API base URL")
+    parser.add_argument("--api-key", default=DEFAULT_API_KEY, help="API key")
+    parser.add_argument("--model", "-m", default=DEFAULT_MODEL, help="Model name")
+    parser.add_argument("--text", required=True, help="Text to synthesize")
+    parser.add_argument("--voice", default="default", help="Voice name (only 'default' is supported)")
+    parser.add_argument("--seed", type=int, default=None, help="Sampling seed")
+    parser.add_argument(
+        "--max-new-tokens",
+        type=int,
+        default=None,
+        help="Frame budget (1 token = 1 frame ≈ 46.4 ms at 21.5 fps)",
+    )
+    parser.add_argument("--stream", action="store_true", help="Enable streaming (PCM output)")
+    parser.add_argument(
+        "--response-format",
+        default="wav",
+        choices=["wav", "mp3", "flac", "pcm"],
+        help="Audio format (default: wav). Streaming is pcm/wav only. Opus is not supported at 22.05 kHz.",
+    )
+    parser.add_argument("--output", "-o", default=None, help="Output file path")
+    args = parser.parse_args()
+    run_tts(args)
+
+
+if __name__ == "__main__":
+    main()

examples/online_serving/text_to_speech/qwen3_tts/batch_speech_client.py

@@ -1,3 +1,6 @@
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
+
 """Batch speech client for Qwen3-TTS via /v1/audio/speech/batch endpoint.
 
 This script demonstrates how to synthesize multiple texts in a single request.
@@ -7,7 +10,7 @@ the reference for each item.
 
 Start the server (with batch-optimized stage settings for best throughput):
 
-    vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice \
+    vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-Base \
         --omni \
         --trust-remote-code \
         --stage-overrides '{"0":{"max_num_seqs":4,"gpu_memory_utilization":0.2},

examples/online_serving/text_to_speech/qwen3_tts/gradio_demo.py

@@ -1,3 +1,6 @@
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
+
 """Gradio demo for Qwen3-TTS with gapless streaming audio playback.
 
 Uses a custom AudioWorklet-based player for gap-free streaming,
@@ -351,7 +354,7 @@ def generate_speech(
         raise gr.Error(f"Failed to decode audio: {e}")
 
 
-def create_app(api_base: str):
+def create_app(api_base: str, task_type: str = "CustomVoice"):
     """Create the FastAPI app with streaming proxy + Gradio UI."""
     fastapi_app = FastAPI()
 
@@ -463,10 +466,11 @@ def create_app(api_base: str):
                     lines=4,
                 )
                 with gr.Row():
-                    task_type = gr.Radio(
+                    task_type_input = gr.Radio(
                         choices=TASK_TYPES,
-                        value="CustomVoice",
+                        value=task_type,
                         label="Task Type",
+                        interactive=False,
                         scale=2,
                     )
                     language = gr.Dropdown(
@@ -479,17 +483,21 @@ def create_app(api_base: str):
                     choices=voices,
                     value=voices[0] if voices else None,
                     label="speaker",
-                    visible=True,
+                    visible=task_type == "CustomVoice",
                     allow_custom_value=True,
                 )
                 instructions = gr.Textbox(
                     label="Instructions",
                     placeholder="e.g., Speak with excitement / A warm, friendly female voice",
                     lines=2,
-                    visible=True,
-                    info="Optional style/emotion instructions",
+                    visible=task_type != "Base",
+                    info=(
+                        "Required: describe the voice style"
+                        if task_type == "VoiceDesign"
+                        else "Optional style/emotion instructions"
+                    ),
                 )
-                with gr.Column(visible=False) as ref_group:
+                with gr.Column(visible=task_type == "Base") as ref_group:
                     ref_audio = gr.Audio(
                         label="Reference Audio (upload for voice cloning)",
                         type="numpy",
@@ -563,7 +571,7 @@ def create_app(api_base: str):
                     autoplay=True,
                     visible=False,
                 )
-                with gr.Column(visible=True) as examples_cv:
+                with gr.Column(visible=task_type == "CustomVoice") as examples_cv:
                     gr.Examples(
                         examples=[
                             [
@@ -588,7 +596,7 @@ def create_app(api_base: str):
                         inputs=[text_input, voice, language, instructions],
                         label="examples",
                     )
-                with gr.Column(visible=False) as examples_vd:
+                with gr.Column(visible=task_type == "VoiceDesign") as examples_vd:
                     gr.Examples(
                         examples=[
                             [
@@ -605,7 +613,7 @@ def create_app(api_base: str):
                         inputs=[text_input, language, instructions],
                         label="examples",
                     )
-                with gr.Column(visible=False) as examples_base:
+                with gr.Column(visible=task_type == "Base") as examples_base:
                     gr.Examples(
                         examples=[
                             [
@@ -653,9 +661,9 @@ def create_app(api_base: str):
                 gr.update(visible=is_base),  # examples_base
             )
 
-        task_type.change(
+        task_type_input.change(
             fn=on_task_change,
-            inputs=[task_type],
+            inputs=[task_type_input],
             outputs=[
                 voice,
                 instructions,
@@ -703,7 +711,7 @@ def create_app(api_base: str):
 
         all_inputs = [
             text_input,
-            task_type,
+            task_type_input,
             voice,
             language,
             instructions,
@@ -788,6 +796,9 @@ def parse_args():
         description="Gradio demo for Qwen3-TTS with gapless AudioWorklet streaming.",
     )
     add_common_args(parser)
+    parser.add_argument(
+        "--task-type", choices=TASK_TYPES, default="CustomVoice", help="Task matching the served checkpoint"
+    )
     return parser.parse_args()
 
 
@@ -796,7 +807,7 @@ def main():
     logging.basicConfig(level=logging.INFO)
     print(f"Connecting to vLLM server at: {args.api_base}")
 
-    app = create_app(args.api_base)
+    app = create_app(args.api_base, task_type=args.task_type)
 
     import uvicorn
 

examples/online_serving/text_to_speech/qwen3_tts/openai_speech_client.py

@@ -8,25 +8,28 @@ to generate audio from text using Qwen3-TTS models.
 
 Examples:
     # CustomVoice task (predefined speaker)
-    python openai_speech_client.py --text "Hello, how are you?" --voice vivian
+    python openai_speech_client.py --text "Hello, how are you?" --speaker vivian
 
     # CustomVoice with emotion instruction
-    python openai_speech_client.py --text "I'm so happy!" --voice vivian \
+    python openai_speech_client.py --text "I'm so happy!" --speaker vivian \
         --instructions "Speak with excitement"
 
     # VoiceDesign task (voice from description)
     python openai_speech_client.py --text "Hello world" \
+        --model Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign \
         --task-type VoiceDesign \
         --instructions "A warm, friendly female voice"
 
     # Base task (voice cloning)
     python openai_speech_client.py --text "Hello world" \
+        --model Qwen/Qwen3-TTS-12Hz-1.7B-Base \
         --task-type Base \
         --ref-audio "https://example.com/reference.wav" \
         --ref-text "This is the reference transcript"
 
     # Base task with pre-computed speaker embedding
     python openai_speech_client.py --text "Hello world" \
+        --model Qwen/Qwen3-TTS-12Hz-1.7B-Base \
         --task-type Base \
         --speaker-embedding embedding.json
 """

examples/online_serving/text_to_speech/qwen3_tts/run_gradio_demo.sh

@@ -1,4 +1,7 @@
 #!/bin/bash
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
+
 # Launch both vLLM server and Gradio demo for Qwen3-TTS
 #
 # Usage:
@@ -179,7 +182,7 @@ fi
 echo ""
 echo "Starting Gradio demo..."
 cd "$SCRIPT_DIR"
-GRADIO_CMD=("python" "gradio_demo.py" "--api-base" "$API_BASE" "--host" "$GRADIO_IP" "--port" "$GRADIO_PORT")
+GRADIO_CMD=("python" "gradio_demo.py" "--api-base" "$API_BASE" "--host" "$GRADIO_IP" "--port" "$GRADIO_PORT" "--task-type" "$TASK_TYPE")
 if [ "$GRADIO_SHARE" = true ]; then
     GRADIO_CMD+=("--share")
 fi