[Model] Add Gepard-1.0 native-AR FSQ/NanoCodec TTS (offline inference) (#5666)

Signed-off-by: Mengjie Zhao <zmj0129@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Yueqian Lin <70319226+linyueqian@users.noreply.github.com>

.buildkite/cuda/test-weekly.yml

@@ -108,6 +108,22 @@ steps:
             --run-level "full_model"
         mirror_hardwares: h100_1
 
+      # Own step: the image has no NanoCodec, so this file import-skips out of
+      # the mark-selected TTS steps above. The overrides keep nemo_toolkit[tts]
+      # from downgrading transformers; see
+      # examples/offline_inference/text_to_speech/README.md.
+      - label: "TTS · Gepard-1.0 · L4"
+        timeout_in_minutes: 60
+        commands:
+          - export VLLM_ALLOW_LONG_MAX_MODEL_LEN="1"
+          - printf '%s\n' "transformers>=5.5.3" "huggingface-hub>=1.0" > /tmp/gepard-overrides.txt
+          - uv pip install --python "$(python3 -c 'import sys; print(sys.executable)')" "nemo_toolkit[tts]==2.7.3" --overrides /tmp/gepard-overrides.txt
+          - >-
+            pytest -s -v tests/e2e/offline_inference/test_gepard_tts.py
+            -m "tts and cuda"
+            --run-level "full_model"
+        mirror_hardwares: l4_1
+
       - label: "Diffusion · L4"
         timeout_in_minutes: 60
         commands:

docs/models/supported_models.md

@@ -98,6 +98,7 @@ th {
 |`DyninOmniForConditionalGeneration` | Dynin-Omni | `snu-aidas/Dynin-Omni` | ✅︎ | | | | — |
 | `MiniCPMO45OmniForConditionalGeneration` | MiniCPM-o 4.5 | `openbmb/MiniCPM-o-4_5` | ✅︎ | | ✅︎ | | — |
 | `ErnieImagePipeline` | ERNIE-Image | `baidu/ERNIE-Image`, `baidu/ERNIE-Image-Turbo` | ✅︎ | ✅︎ | ✅︎ | ✅︎ | — |
+| `GepardTalkerForConditionalGeneration` | Gepard-1.0 | `nineninesix/gepard-1.0` | ✅︎ | | | | — |
 |`HiDreamImagePipeline` | HiDream-I1-Full | `HiDream-ai/HiDream-I1-Full` | ✅︎ | ✅︎ | | | — |
 
 The `Recipe` column links the corresponding deployment recipe. `Published` is

examples/offline_inference/text_to_speech/README.md

@@ -16,6 +16,7 @@ list of supported architectures across all modalities, see
 |---|---|---|---|---|---|---|
 | CosyVoice3 | `FunAudioLLM/Fun-CosyVoice3-0.5B-2512` | 2 (talker + code2wav) | ✓ | ✓ | — | 24 kHz |
 | Fish Speech S2 Pro | `fishaudio/s2-pro` | dual-AR | ✓ | ✓ | — | 44.1 kHz |
+| Gepard-1.0 | `nineninesix/gepard-1.0` | single (native AR + NanoCodec) | — (zero-shot; cloning WIP) | — (serving WIP) | zero-shot | 22.05 kHz |
 | GLM-TTS | `zai-org/GLM-TTS` | 2 (AR + DiT) | ✓ (required) | ✓ | — | 24 kHz |
 | Ming-omni-tts | `inclusionAI/Ming-omni-tts-0.5B` | 2 (AR + audio VAE) | ✓ | ✓ | style / IP / dialect / TTA / podcast | 44.1 kHz |
 | Ming-flash-omni-TTS | `Jonathan1909/Ming-flash-omni-2.0` | single (talker only) | — (caption-controlled) | — | style / IP / basic captions | 44.1 kHz |
@@ -395,6 +396,58 @@ python examples/offline_inference/text_to_speech/qwen3_tts/end2end.py \
 
 ---
 
+## Gepard-1.0
+
+Single-stage native AR TTS at 22.05 kHz. Pipeline: `Qwen3.5 backbone → 32 FSQ codebook heads (one frame/step) → NeMo NanoCodec`. Backbone runs under vLLM paged attention; the 32-head sampling + learned embedding feedback ride the native-AR runner hooks. `enforce_eager` (CUDA graph is a perf follow-up).
+
+### Prerequisites
+The NanoCodec decoder needs NeMo (NVIDIA Open Model License), installed separately:
+
+```bash
+uv pip install "nemo_toolkit[tts]==2.7.3" \
+    --overrides <(printf '%s\n' "transformers>=5.5.3" "huggingface-hub>=1.0")
+```
+
+The overrides are what make that command work. `nemo_toolkit[tts]` declares
+`transformers~=4.57.0`, so a plain install downgrades the version
+`requirements/common.txt` pins and takes `huggingface-hub` back to 0.x with it — after
+which `import vllm` fails. Both packages are named here rather than left to resolve on
+their own: transformers 5.x happens to require `huggingface-hub` 1.x, but relying on
+that would make the second repair an accident of the first. `--constraint` cannot be
+used instead, since it narrows a version range rather than overriding a dependency a
+package declares, and the two sets never intersect. NeMo 2.7.3 decodes correctly
+against transformers 5.x, so nothing is lost by holding it there.
+
+(The same clash is why this is documented here rather than declared in
+`pyproject.toml`: an extra resolves together with the base dependencies, so
+`vllm-omni[gepard]` would be unsatisfiable rather than opt-in. SoulX-Singer English
+SVS documents `nemo_toolkit[asr]` the same way.)
+
+On a host whose CUDA toolkit cannot build kernels for the GPU — no `nvcc`/`ninja`, or a
+consumer Blackwell (`sm_120`) card — also set `VLLM_USE_FLASHINFER_SAMPLER=0`. vLLM's
+sampler JIT-compiles through FlashInfer on first use, and where that build fails the engine
+dies inside `profile_run` long after every import has succeeded, which reads as a model
+failure rather than an environment one. The PyTorch sampler it falls back to costs nothing
+here.
+
+### Quick start (zero-shot, default voice)
+```bash
+python examples/offline_inference/text_to_speech/gepard/end2end.py \
+    --text "Hello, this is a Gepard demo."
+```
+
+### Voice cloning
+Not yet — PR1 is zero-shot only (the learned `null_prefix` default voice). Reference-audio cloning (`ref_compressor` speaker prefix) is a follow-up PR.
+
+### Notes
+- Output: 22.05 kHz mono WAV.
+- First run downloads two checkpoints, not one: the model itself, and the NanoCodec decoder it names in `codec_id` (`nvidia/nemo-nano-codec-22khz-1.89kbps-21.5fps` by default). Both are fetched automatically; expect the first startup to be correspondingly slower.
+- Offline only for now; the `/v1/audio/speech` adapter is a follow-up PR.
+- Deploy config: `vllm_omni/deploy/gepard.yaml` (the example's default; copy it and pass `--deploy-config` to change it).
+- Generation length and reproducibility are stage settings, not script flags. One output token is one audio frame, so `max_tokens` in the YAML is the frame budget; `seed` makes the in-model 32-head sampling reproducible. The script deliberately passes no `SamplingParams`: one supplied by a caller replaces the stage defaults wholesale rather than merging, which would drop the pipeline's stop token and run every request to `max_tokens`.
+- Text length: short texts are repeated internally to match the training layout (the checkpoint's `text_repetition` block); the upper bound is the stage's `max_model_len`, enforced by the engine. Empty text is rejected rather than voiced.
+- `VLLM_GEPARD_GREEDY=1` swaps the 32-head Gumbel-max sampling for argmax, for reproducible comparisons that do not depend on a seed.
+
 ## VoxCPM2
 
 Single-stage native AR TTS at 48 kHz. Pipeline: `feat_encoder → MiniCPM4 → FSQ → residual_lm → LocDiT → AudioVAE`.

examples/offline_inference/text_to_speech/gepard/end2end.py

@@ -0,0 +1,139 @@
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
+"""Offline inference example for Gepard-1.0 TTS via vLLM-Omni.
+
+Single-stage native-AR pipeline: a Qwen3.5 backbone samples one 32-code FSQ
+frame per step; the NeMo NanoCodec decodes the frames to a 22.05 kHz mono
+waveform. Zero-shot: the default learned voice, no reference audio.
+
+Generation length and seed are stage settings rather than flags here. One
+output token is one audio frame, so ``max_tokens`` in the deploy YAML is the
+frame budget; pass a copy of the YAML via ``--deploy-config`` to change it.
+
+Usage:
+  python end2end.py --text "Hello, this is Gepard speaking."
+"""
+
+from __future__ import annotations
+
+import os
+from pathlib import Path
+
+import soundfile as sf
+import torch
+
+import vllm_omni
+from vllm_omni.utils.tracking_parser import TrackingArgumentParser
+
+os.environ.setdefault("VLLM_WORKER_MULTIPROC_METHOD", "spawn")
+
+from vllm_omni import Omni  # noqa: E402
+
+MODEL = "nineninesix/gepard-1.0"
+SAMPLE_RATE = 22050
+# Absolute: the loader hands the path straight to OmegaConf, which resolves a
+# bare name against the current working directory, not the package.
+DEFAULT_DEPLOY_CONFIG = str(Path(vllm_omni.__file__).resolve().parent / "deploy" / "gepard.yaml")
+
+
+def build_request(text: str) -> dict:
+    """Build an Omni request payload for Gepard (zero-shot, no ref audio).
+
+    The model only consumes the [speaker slots, SOT, text, EOT, SOS] layout, so
+    a bare text prompt would have no speaker slots and no SOS.
+    """
+    from transformers import AutoTokenizer
+
+    from vllm_omni.model_executor.models.gepard.configuration_gepard import GepardConfig
+    from vllm_omni.model_executor.models.gepard.prompt import build_gepard_prompt_ids
+
+    # From the checkpoint's own sidecar, which is what the worker builds its
+    # config from too. A layout that disagrees with the trained one does not
+    # fail, it just degrades the speech.
+    cfg = GepardConfig.from_checkpoint(MODEL)
+    tokenizer = AutoTokenizer.from_pretrained(MODEL, trust_remote_code=True)
+    prompt_token_ids = build_gepard_prompt_ids(
+        tokenizer(text, add_special_tokens=False)["input_ids"],
+        config=cfg,
+    )
+    return {
+        "prompt_token_ids": prompt_token_ids,
+        "additional_information": {"text": [text]},
+    }
+
+
+def save_audio(waveform: torch.Tensor, path: str, sample_rate: int = SAMPLE_RATE) -> None:
+    sf.write(path, waveform.float().numpy(), sample_rate)
+    seconds = waveform.numel() / sample_rate
+    print(
+        f"  Saved {path} ({tuple(waveform.shape)}, {sample_rate} Hz, "
+        f"{seconds:.2f}s, peak {float(waveform.abs().max()):.3f})"
+    )
+
+
+def main(args) -> None:
+    omni = Omni(
+        model=MODEL,
+        deploy_config=args.deploy_config,
+        stage_init_timeout=args.stage_init_timeout,
+        init_timeout=args.init_timeout,
+    )
+
+    # No explicit SamplingParams: one supplied here would replace the stage
+    # defaults rather than merge over them, dropping the pipeline's stop token.
+    output_dir = Path(args.output_dir)
+    output_dir.mkdir(parents=True, exist_ok=True)
+
+    print(f"Synthesizing: {args.text!r}")
+    inputs = build_request(text=args.text)
+
+    for i, stage_outputs in enumerate(omni.generate(inputs)):
+        for j, out in enumerate(stage_outputs.outputs):
+            mm = out.multimodal_output
+            if mm is None:
+                print(f"  [req {i}] No audio output.")
+                continue
+            # The consolidation path renames "model_outputs" to "audio".
+            audio = mm.get("audio")
+            if audio is None:
+                audio = mm.get("model_outputs")
+            if isinstance(audio, list):
+                audio = audio[0] if len(audio) else None
+            if audio is None:
+                print(f"  [req {i}] No waveform in multimodal_output (keys: {list(mm)}).")
+                continue
+            sr_tensor = mm.get("sr")
+            sr = int(sr_tensor.item()) if hasattr(sr_tensor, "item") else SAMPLE_RATE
+            save_audio(audio.cpu(), str(output_dir / f"output_{i}_{j}.wav"), sr)
+
+    print("Done.")
+
+
+def parse_args():
+    parser = TrackingArgumentParser(description="Gepard-1.0 offline TTS inference (zero-shot)")
+    parser.add_argument("--text", default="Hello, this is Gepard speaking.", help="Text to synthesize.")
+    parser.add_argument(
+        "--output-dir",
+        default=os.path.join(
+            os.environ.get("XDG_CACHE_HOME", os.path.join(os.path.expanduser("~"), ".cache")),
+            "gepard_output",
+        ),
+        help="Directory for WAV outputs (default: ~/.cache/gepard_output).",
+    )
+    parser.add_argument(
+        "--deploy-config",
+        default=DEFAULT_DEPLOY_CONFIG,
+        help=(
+            "Path to the deploy YAML (default: the packaged vllm_omni/deploy/gepard.yaml). This must "
+            "stay set: without the YAML's `pipeline: gepard` pin the architectures fallback routes "
+            "Qwen3_5ForCausalLM to the diffusion registry. Copy it to change max_tokens or seed."
+        ),
+    )
+    # The upstream defaults are too tight for a cold start.
+    parser.add_argument("--stage-init-timeout", type=int, default=900)
+    parser.add_argument("--init-timeout", type=int, default=1800)
+    return parser.parse_args()
+
+
+if __name__ == "__main__":
+    main(parse_args())

tests/e2e/offline_inference/test_gepard_tts.py

@@ -0,0 +1,345 @@
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
+"""E2E offline inference tests for the Gepard-1.0 single-stage native-AR TTS.
+
+Zero-shot only (the model's default learned voice). These tests need a GPU and
+the NeMo NanoCodec, so the whole file runs in the weekly tier -- see the comment
+above ``pytestmark``. The window arithmetic behind the streaming decode is
+covered on CPU in ``tests/model_executor/models/test_gepard_window.py``.
+"""
+
+from __future__ import annotations
+
+import functools
+import tempfile
+from pathlib import Path
+from typing import NamedTuple
+
+import pytest
+import soundfile as sf
+import torch
+from transformers import AutoTokenizer
+
+from tests.helpers.mark import hardware_test
+from tests.helpers.media import convert_audio_file_to_text
+from tests.helpers.runtime import OmniRunner  # noqa: F401  (indirect fixture)
+from tests.helpers.stage_config import get_deploy_config_path
+from vllm_omni import Omni
+from vllm_omni.model_executor.models.gepard.configuration_gepard import GepardConfig
+from vllm_omni.model_executor.models.gepard.prompt import build_gepard_prompt_ids
+
+# The codec decoder is an optional dependency; skip rather than fail the engine
+# inside the worker process when it is absent. The CI image does not carry it,
+# so the weekly step installs it first -- see .buildkite/cuda/test-weekly.yml.
+# Install recipe: examples/offline_inference/text_to_speech/README.md.
+pytest.importorskip("nemo.collections.tts.models")
+
+MODEL_NAME = "nineninesix/gepard-1.0"
+STAGE_CONFIG = get_deploy_config_path("gepard.yaml")
+SAMPLE_RATE = 22050
+# NanoCodec upsamples one FSQ frame to exactly this many samples.
+SAMPLES_PER_FRAME = 1024
+
+
+@functools.cache
+def _config() -> GepardConfig:
+    """The checkpoint's own layout, not the class defaults.
+
+    The worker builds its config from the ``gepard_config.json`` sidecar, so a
+    prompt assembled from the defaults would silently use a different layout
+    the day the sidecar moves — and a mismatched layout degrades the speech
+    rather than failing, which is the hardest way for this to be found.
+    """
+    return GepardConfig.from_checkpoint(MODEL_NAME)
+
+
+# Prompt -> a word that appears in no other prompt, so a clip can be traced
+# back to the request that asked for it.
+#
+# These are not interchangeable. Batching perturbs every request's sampling
+# trajectory, and on some trajectories this checkpoint's stop head never fires;
+# this set was screened over 10 batched runs with no runaway. Re-screen the
+# whole set when changing one, not just the row changed.
+_DISTINGUISHABLE_PROMPTS = {
+    "He drinks coffee every morning.": "coffee",
+    "Machine learning is interesting.": "learning",
+    "Please close the window before leaving.": "window",
+    "My favorite color is purple.": "purple",
+}
+
+# Above text_repetition.apply_below, so the layout is voiced once instead of
+# repeated. Every other prompt here takes the repeated branch.
+_LONG_TEXT = (
+    "The morning train was late again, so I walked the last two miles along the "
+    "river and reached the office just before nine."
+)
+
+_OMNI_RUNNER_PARAM = (
+    MODEL_NAME,
+    STAGE_CONFIG,
+    {"trust_remote_code": True},
+)
+
+# Only a common word is checked, which needs less ASR than the shared helper's
+# ``small`` default; a miss escalates to it.
+_ASR_MODEL = "base"
+_ASR_ESCALATION_MODEL = "small"
+
+# Transcription dominates this file's runtime -- Whisper runs on CPU on a
+# single-GPU runner -- so only two of the four concurrent clips are read as
+# speech. The pairwise waveform comparison covers all four.
+_TRANSCRIBED_CLIPS = 2
+
+# No level marker at module scope: each test carries the one matching how often
+# its claim needs re-checking, though the weekly step collects the file whole.
+# There is deliberately no core_model row -- under dummy weights the stop head is
+# random, so every request would run to gepard.yaml's max_tokens.
+pytestmark = [
+    pytest.mark.tts,
+    pytest.mark.parametrize("omni_runner", [_OMNI_RUNNER_PARAM], indirect=True),
+]
+
+
+def _build_request(text: str) -> dict:
+    """Assemble the [speaker slots, SOT, text, EOT, SOS] prompt."""
+    tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME, trust_remote_code=True)
+    prompt_token_ids = build_gepard_prompt_ids(
+        tokenizer(text, add_special_tokens=False)["input_ids"],
+        config=_config(),
+    )
+    return {
+        "prompt_token_ids": prompt_token_ids,
+        "additional_information": {"text": [text]},
+    }
+
+
+def _extract_audio(mm) -> torch.Tensor | None:
+    # The consolidation path renames "model_outputs" to "audio".
+    audio = mm.get("audio")
+    if audio is None:
+        audio = mm.get("model_outputs")
+    if isinstance(audio, list):
+        audio = audio[0] if len(audio) else None
+    return audio
+
+
+class _Clip(NamedTuple):
+    wav: torch.Tensor
+    # Counts the frames the codec was handed using only what the engine
+    # returned, never the model's internals.
+    committed_frames: int
+
+
+def _synthesize_all(omni: Omni, texts: list[str]) -> list[_Clip]:
+    """Submit every text in one call; return one clip per request, in
+    submission order.
+
+    ``Omni.generate`` yields in completion order and never sorts, so the pairing
+    is restored from the request id prefix it assigns (``f"{i}_{uuid4()}"``).
+    Without that, the only testable claim is that the batch as a whole looks
+    plausible, which a stable swap between two requests satisfies.
+
+    Nothing here authors a SamplingParams: a fresh one would replace the stage
+    defaults rather than merge over them, dropping the pipeline's stop token.
+    """
+    outputs = omni.generate([_build_request(t) for t in texts])
+    assert len(outputs) == len(texts), f"expected {len(texts)} outputs, got {len(outputs)}"
+
+    by_index: dict[int, _Clip] = {}
+    for stage_outputs in outputs:
+        index = int(str(stage_outputs.request_id).split("_", 1)[0])
+        waveform = None
+        frames = 0
+        for out in stage_outputs.outputs:
+            # Explicit None check: `x or []` would ask an array for a truth value.
+            token_ids = out.token_ids if out.token_ids is not None else []
+            # STOP sits just past head0's range, so a token equal to it is
+            # never a committed frame.
+            frames += sum(1 for t in token_ids if int(t) != _config().stop_token)
+            mm = out.multimodal_output
+            if mm is None:
+                continue
+            audio = _extract_audio(mm)
+            if audio is None:
+                continue
+            sr_t = mm.get("sr")
+            if hasattr(sr_t, "item"):
+                assert int(sr_t.item()) == SAMPLE_RATE
+            waveform = audio.cpu().float()
+        assert waveform is not None, "no audio output produced"
+        by_index[index] = _Clip(wav=waveform, committed_frames=frames)
+
+    assert sorted(by_index) == list(range(len(texts))), (
+        f"request ids did not cover every submitted prompt exactly once: got {sorted(by_index)}"
+    )
+    return [by_index[i] for i in range(len(texts))]
+
+
+def _synthesize(omni: Omni, text: str) -> _Clip:
+    return _synthesize_all(omni, [text])[0]
+
+
+def _assert_clip_is_sane(clip: _Clip) -> None:
+    """The checks every clip has to pass, whatever produced it."""
+    wav = clip.wav
+    assert wav.dim() == 1, f"expected mono 1-D, got shape {tuple(wav.shape)}"
+    assert wav.numel() > 0, "empty waveform"
+    assert torch.isfinite(wav).all(), "non-finite samples"
+
+    # Frame conservation against the engine's own token count, not just
+    # divisibility. This is what catches a payload carrying non-audio data or a
+    # dropped tail; both pass every other check here.
+    assert wav.numel() == clip.committed_frames * SAMPLES_PER_FRAME, (
+        f"{wav.numel()} samples for {clip.committed_frames} committed frames — "
+        f"expected exactly {clip.committed_frames * SAMPLES_PER_FRAME}"
+    )
+
+    # Anything far outside the waveform range is not codec output.
+    assert float(wav.abs().max()) <= 1.5, f"peak {float(wav.abs().max()):.2f} is out of range"
+
+
+def _assert_stopped_on_its_own(clip: _Clip) -> None:
+    """The request must stop on its own rather than run to ``max_tokens``.
+
+    Separate from the structural checks because it reads the stop head's
+    judgement, not the payload's shape: under ``load_format: dummy`` that head
+    is random and never fires, so every clip runs to the token cap.
+    """
+    seconds = clip.wav.numel() / SAMPLE_RATE
+    assert 0.5 < seconds < 30.0, f"implausible duration {seconds:.2f}s — did STOP fire?"
+
+
+def _transcribe(wav: torch.Tensor, tmp_dir: str, name: str, model_size: str = _ASR_MODEL) -> str:
+    path = str(Path(tmp_dir) / f"{name}.wav")
+    sf.write(path, wav.numpy(), SAMPLE_RATE)
+    return convert_audio_file_to_text(path, model_size=model_size, language="en").lower()
+
+
+def _transcribe_for(wav: torch.Tensor, tmp_dir: str, name: str, accepts) -> str:
+    """Transcribe with the fast model, retrying once with the stronger one.
+
+    ``accepts`` decides whether the fast transcript is usable. Escalating only
+    on a miss keeps the average clip at ``_ASR_MODEL`` while a genuine model
+    artifact still fails the caller's assertion, since the stronger ASR
+    mistranscribes it too.
+    """
+    transcript = _transcribe(wav, tmp_dir, name)
+    if accepts(transcript):
+        return transcript
+    return _transcribe(wav, tmp_dir, name, model_size=_ASR_ESCALATION_MODEL)
+
+
+@hardware_test(res={"cuda": "L4"}, num_cards=1)
+@pytest.mark.parametrize(
+    ("text", "keyword"),
+    [
+        # The canonical path: one request, one transcript, the cheapest check
+        # that reads the audio as speech.
+        pytest.param(
+            "Hello, this is Gepard speaking.",
+            "hello",
+            marks=pytest.mark.advanced_model,
+            id="canonical",
+        ),
+        # No keyword: this row carries the structural and stopping checks over a
+        # second text and leaves the transcript to the row above.
+        pytest.param(
+            "The quick brown fox jumps over the lazy dog.",
+            None,
+            marks=pytest.mark.full_model,
+            id="second_text",
+        ),
+    ],
+)
+def test_gepard_offline_zero_shot(omni_runner, run_level: str, text: str, keyword: str | None) -> None:
+    """Zero-shot synthesis produces a finite mono waveform at 22.05 kHz.
+
+    The transcript check is the one assertion here that reads the audio as
+    speech rather than as a tensor. It stands in for a parity comparison, which
+    needs a reference implementation this PR cannot run — noise, a repeated
+    loop or the wrong clip all pass every numeric check above it.
+    """
+    clip = _synthesize(omni_runner, text)
+    _assert_clip_is_sane(clip)
+
+    if run_level not in {"advanced_model", "full_model"}:
+        return  # dummy weights: the audio is structural only, so do not read it
+
+    _assert_stopped_on_its_own(clip)
+    if keyword is None:
+        return
+
+    with tempfile.TemporaryDirectory() as tmp_dir:
+        transcript = _transcribe_for(clip.wav, tmp_dir, "zero_shot", lambda t: keyword in t)
+    assert keyword in transcript, f"expected {keyword!r} in transcript, got {transcript!r}"
+
+
+@pytest.mark.full_model
+@hardware_test(res={"cuda": "L4"}, num_cards=1)
+def test_gepard_offline_long_text_skips_repetition(omni_runner, run_level: str) -> None:
+    """The other branch of the prompt layout, end to end.
+
+    A text at or above ``apply_below`` tokens is voiced once; shorter ones are
+    repeated to carry enough mass against the speaker prefix. The layout
+    assertion keeps this row honest — if a tokenizer change pushed this text
+    back under the threshold it would silently retest the repeated path.
+    """
+    prompt_ids = _build_request(_LONG_TEXT)["prompt_token_ids"]
+    assert prompt_ids.count(_config().start_of_text) == 1, "_LONG_TEXT is no longer above apply_below"
+
+    clip = _synthesize(omni_runner, _LONG_TEXT)
+    _assert_clip_is_sane(clip)
+
+    if run_level not in {"advanced_model", "full_model"}:
+        return
+
+    _assert_stopped_on_its_own(clip)
+    with tempfile.TemporaryDirectory() as tmp_dir:
+        transcript = _transcribe_for(clip.wav, tmp_dir, "long_text", lambda t: "river" in t)
+    assert "river" in transcript, f"expected 'river' in transcript, got {transcript!r}"
+
+
+@pytest.mark.slow
+@hardware_test(res={"cuda": "L4"}, num_cards=1)
+def test_gepard_offline_concurrent_requests_stay_isolated(omni_runner, run_level: str) -> None:
+    """Four requests in one batch, each holding its own generation state.
+
+    The prompts are deliberately unlike each other: cross-talk between the
+    per-request frame history, the codec window or the output routing shows up
+    as a transcript matching the wrong prompt, which similar prompts would
+    hide. ``max_num_seqs`` is 4, so these really do run together.
+    """
+    texts = list(_DISTINGUISHABLE_PROMPTS)
+    clips = _synthesize_all(omni_runner, texts)
+
+    for clip in clips:
+        _assert_clip_is_sane(clip)
+
+    if run_level not in {"advanced_model", "full_model"}:
+        return  # dummy weights: the audio is structural only, so do not read it
+
+    for clip in clips:
+        _assert_stopped_on_its_own(clip)
+
+    # All four clips, not just the transcribed ones: delivering one request's
+    # audio for another leaves two byte-identical waveforms.
+    for i in range(len(clips)):
+        for j in range(i + 1, len(clips)):
+            assert not torch.equal(clips[i].wav, clips[j].wav), (
+                f"clips {i} and {j} are the same waveform — one request's audio was delivered for another"
+            )
+
+    # Each clip must say what its own prompt asked for: accepting any of the
+    # four would pass a stable permutation, which is what a mix-up produces.
+    keywords = set(_DISTINGUISHABLE_PROMPTS.values())
+    with tempfile.TemporaryDirectory() as tmp_dir:
+        for i, clip in enumerate(clips[:_TRANSCRIBED_CLIPS]):
+            expected = _DISTINGUISHABLE_PROMPTS[texts[i]]
+            # Escalate on the expected word being absent, not on "no keyword
+            # at all" -- a neighbour's word is the failure under test.
+            transcript = _transcribe_for(clip.wav, tmp_dir, f"concurrent_{i}", lambda t: expected in t)
+            present = sorted(k for k in keywords if k in transcript)
+            assert present == [expected], (
+                f"clip for {texts[i]!r} transcribed as {transcript!r}: expected {expected!r} and "
+                f"no other prompt's keyword, found {present} — requests bled into each other"
+            )

tests/model_executor/models/test_gepard_window.py

@@ -0,0 +1,363 @@
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
+"""CPU coverage for Gepard's prompt layout, streaming-decode window and config.
+
+The e2e test needs a GPU, NeMo and the checkpoint, so it only runs nightly.
+These pin the parts that are pure logic and break silently: which frames each
+decode covers, which samples survive the lookback trim, the prompt layout, and
+that the packaged deploy config sits where the example looks for it.
+
+The codec is replaced by a shift-invariant stand-in whose output depends only
+on the codes, which makes "chunked emissions concatenate to one whole-history
+decode" an exact assertion — the contract lookback exists to provide.
+"""
+
+from __future__ import annotations
+
+from pathlib import Path
+
+import pytest
+import torch
+import torch.nn as nn
+
+import vllm_omni
+from vllm_omni.config.stage_config import load_deploy_config
+from vllm_omni.model_executor.models.gepard.configuration_gepard import GepardConfig
+from vllm_omni.model_executor.models.gepard.gepard_talker import (
+    CHUNK_FRAMES,
+    FIRST_CHUNK_FRAMES,
+    GepardTalkerForConditionalGeneration,
+    _GepardState,
+)
+from vllm_omni.model_executor.models.gepard.nanocodec import (
+    LOOKBACK_FRAMES,
+    NanoCodec,
+    apply_end_of_speech_tail,
+)
+from vllm_omni.model_executor.models.gepard.prompt import build_gepard_prompt_ids
+
+pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
+
+SAMPLES_PER_FRAME = 1024
+NUM_HEADS = 32
+SAMPLE_RATE = 22050
+# Text-token count at or above which the layout is left unrepeated, read from
+# the checkpoint defaults so the boundary cases follow the config.
+APPLY_BELOW = GepardConfig().text_repetition_apply_below
+
+
+class _ShiftInvariantCodec:
+    """Frame tagged ``v`` -> samples ``v*SPF .. v*SPF+SPF-1``, wherever it sits."""
+
+    def __init__(self) -> None:
+        self.windows: list[int] = []  # frame count of every decode call
+
+    def decode_from_codes(self, codes: torch.Tensor, codes_len: torch.Tensor):
+        assert codes.shape[0] == 1, "streaming decode is per-request"
+        assert codes.shape[1] == NUM_HEADS, f"expected {NUM_HEADS} code channels"
+        tags = codes[0, 0, :].to(torch.int64)  # (T,)
+        self.windows.append(int(tags.shape[0]))
+        offsets = torch.arange(SAMPLES_PER_FRAME, dtype=torch.int64)
+        audio = (tags.unsqueeze(1) * SAMPLES_PER_FRAME + offsets).reshape(1, -1)
+        return audio.to(torch.float32), codes_len * SAMPLES_PER_FRAME
+
+
+def _make_codec() -> tuple[NanoCodec, _ShiftInvariantCodec]:
+    codec = NanoCodec(codec_id="test/fake-codec", sample_rate=SAMPLE_RATE)
+    fake = _ShiftInvariantCodec()
+    codec._codec = fake  # bypasses load(); NeMo is never imported
+    assert codec.is_loaded
+    return codec, fake
+
+
+def _make_talker(codec: NanoCodec) -> GepardTalkerForConditionalGeneration:
+    """A real talker with only the audio-path attributes populated.
+
+    ``__init__`` needs a VllmConfig and builds the backbone; the methods under
+    test touch only these four attributes.
+    """
+    talker = GepardTalkerForConditionalGeneration.__new__(GepardTalkerForConditionalGeneration)
+    nn.Module.__init__(talker)  # sets up _parameters/_buffers/_modules only
+    talker._codec = codec
+    talker._audio_queue = []
+    talker._active_states = {}
+    talker._deferred_cleanup_ids = set()
+    return talker
+
+
+def _frame(tag: int) -> torch.Tensor:
+    """One committed frame: 32 per-dimension codes, all carrying the tag."""
+    return torch.full((NUM_HEADS,), tag, dtype=torch.long)
+
+
+def _expected_samples(num_frames: int) -> torch.Tensor:
+    offsets = torch.arange(SAMPLES_PER_FRAME, dtype=torch.int64)
+    tags = torch.arange(num_frames, dtype=torch.int64).unsqueeze(1)
+    return (tags * SAMPLES_PER_FRAME + offsets).reshape(-1).to(torch.float32)
+
+
+def _drain(talker) -> torch.Tensor:
+    """Concatenate and clear the queued deltas, as make_omni_output does."""
+    if not talker._audio_queue:
+        return torch.empty(0)
+    out = torch.cat([a.reshape(-1) for _, a in talker._audio_queue])
+    talker._audio_queue.clear()
+    return out
+
+
+def _generate(talker, state, num_frames: int, *, stop_at_end: bool) -> None:
+    """Drive the commit loop the way forward() does, one frame per step."""
+    for tag in range(num_frames):
+        state.frames.append(_frame(tag))
+        state.frame_count += 1
+        state.past_first_step = True
+        talker._emit_audio(state, is_final=stop_at_end and tag == num_frames - 1)
+
+
+# Counts straddling both cadence thresholds, where an off-by-one hides.
+@pytest.mark.parametrize(
+    "num_frames",
+    [
+        1,
+        2,
+        FIRST_CHUNK_FRAMES - 1,
+        FIRST_CHUNK_FRAMES,
+        FIRST_CHUNK_FRAMES + 1,
+        CHUNK_FRAMES - 1,
+        CHUNK_FRAMES,
+        CHUNK_FRAMES + 1,
+        FIRST_CHUNK_FRAMES + CHUNK_FRAMES,
+        79,
+        200,
+    ],
+)
+def test_chunked_emissions_equal_a_whole_history_decode(num_frames: int) -> None:
+    codec, _fake = _make_codec()
+    talker = _make_talker(codec)
+    state = _GepardState(request_id="req-0")
+
+    _generate(talker, state, num_frames, stop_at_end=True)
+
+    torch.testing.assert_close(_drain(talker), _expected_samples(num_frames))
+    assert state.emitted_frames == num_frames
+
+
+def test_every_frame_is_emitted_exactly_once() -> None:
+    """The trim must not drop or duplicate samples at a chunk boundary."""
+    codec, _fake = _make_codec()
+    talker = _make_talker(codec)
+    state = _GepardState(request_id="req-0")
+    num_frames = FIRST_CHUNK_FRAMES + 2 * CHUNK_FRAMES + 3
+
+    _generate(talker, state, num_frames, stop_at_end=True)
+    audio = _drain(talker)
+
+    assert audio.numel() == num_frames * SAMPLES_PER_FRAME
+    # Tags are globally unique, so a duplicated or dropped frame shows up here.
+    assert len(torch.unique(audio)) == audio.numel()
+
+
+def test_decode_windows_carry_lookback_context() -> None:
+    """Every decode after the first must re-decode LOOKBACK_FRAMES of context."""
+    codec, fake = _make_codec()
+    talker = _make_talker(codec)
+    state = _GepardState(request_id="req-0")
+    num_frames = FIRST_CHUNK_FRAMES + 2 * CHUNK_FRAMES
+
+    _generate(talker, state, num_frames, stop_at_end=True)
+
+    assert len(fake.windows) >= 3, "expected a first chunk plus steady-state chunks"
+    assert fake.windows[0] == FIRST_CHUNK_FRAMES, "the first chunk has no left context"
+    for window in fake.windows[1:]:
+        assert window == CHUNK_FRAMES + LOOKBACK_FRAMES
+
+
+def test_codec_runs_once_per_chunk_not_once_per_frame() -> None:
+    codec, fake = _make_codec()
+    talker = _make_talker(codec)
+    state = _GepardState(request_id="req-0")
+    num_frames = FIRST_CHUNK_FRAMES + 3 * CHUNK_FRAMES
+
+    _generate(talker, state, num_frames, stop_at_end=True)
+
+    assert len(fake.windows) == 4
+    assert sum(fake.windows) < num_frames * 2  # far below one decode per frame
+
+
+def test_final_emit_drains_a_partial_chunk() -> None:
+    """A request can end mid-chunk (max_tokens, STOP); the frames committed
+    since the last cadence boundary must still reach the queue.
+
+    Arithmetic only: this drives ``_emit_audio`` directly, so it says nothing
+    about *whether* the engine reaches it on the request's last step. That
+    wiring lives in ``test_gepard_wiring.py`` — a bug there leaves this green.
+    """
+    codec, _fake = _make_codec()
+    talker = _make_talker(codec)
+    state = _GepardState(request_id="req-0")
+    num_frames = FIRST_CHUNK_FRAMES + CHUNK_FRAMES + 5
+
+    _generate(talker, state, num_frames, stop_at_end=False)
+    assert state.emitted_frames < num_frames, "a partial chunk should still be pending"
+
+    talker._emit_audio(state, is_final=True)
+
+    torch.testing.assert_close(_drain(talker), _expected_samples(num_frames))
+    assert state.emitted_frames == num_frames
+
+
+def test_emitting_with_nothing_pending_is_a_no_op() -> None:
+    codec, fake = _make_codec()
+    talker = _make_talker(codec)
+    state = _GepardState(request_id="req-0")
+
+    _generate(talker, state, 4, stop_at_end=True)
+    calls_after_flush = len(fake.windows)
+    talker._audio_queue.clear()
+
+    talker._emit_audio(state, is_final=True)
+
+    assert len(fake.windows) == calls_after_flush, "codec ran with no new frames"
+    assert talker._audio_queue == []
+
+
+def test_concurrent_requests_keep_separate_windows() -> None:
+    codec, _fake = _make_codec()
+    talker = _make_talker(codec)
+    states = [_GepardState(request_id=f"req-{i}") for i in range(2)]
+
+    for state in states:
+        _generate(talker, state, FIRST_CHUNK_FRAMES, stop_at_end=True)
+
+    by_req: dict[str, list[torch.Tensor]] = {}
+    for req_id, audio in talker._audio_queue:
+        by_req.setdefault(req_id, []).append(audio.reshape(-1))
+    assert set(by_req) == {"req-0", "req-1"}
+    for chunks in by_req.values():
+        torch.testing.assert_close(torch.cat(chunks), _expected_samples(FIRST_CHUNK_FRAMES))
+
+
+def test_end_of_speech_tail_fades_and_pads() -> None:
+    audio = torch.ones(SAMPLE_RATE)  # 1 s of full-scale signal
+    out = apply_end_of_speech_tail(audio, SAMPLE_RATE, fade_ms=10.0, silence_ms=20.0)
+
+    fade = int(SAMPLE_RATE * 10.0 / 1000.0)
+    pad = int(SAMPLE_RATE * 20.0 / 1000.0)
+    assert out.numel() == SAMPLE_RATE + pad
+    assert torch.all(out[-pad:] == 0.0), "trailing pad must be silent"
+    assert out[SAMPLE_RATE - fade - 1] == 1.0, "the fade must not reach back past fade_ms"
+    # Monotonically ramped to zero — the pad alone would leave the click.
+    ramp = out[SAMPLE_RATE - fade : SAMPLE_RATE]
+    assert torch.all(ramp[:-1] >= ramp[1:])
+    assert ramp[-1] == 0.0
+
+
+def test_end_of_speech_tail_is_a_no_op_by_default() -> None:
+    audio = torch.ones(128)
+    out = apply_end_of_speech_tail(audio, SAMPLE_RATE, fade_ms=0.0, silence_ms=0.0)
+    assert out is audio
+
+
+def _repeat_count(ids: list[int], cfg: GepardConfig) -> int:
+    """How many ``[SOT ... EOT]`` copies the layout carries."""
+    return ids.count(cfg.start_of_text)
+
+
+def test_gepard_prompt_layout() -> None:
+    """The prompt carries the speaker slots and exactly one SOS, at the end."""
+    cfg = GepardConfig()
+    ids = build_gepard_prompt_ids([1, 2, 3], config=cfg)
+
+    slots = [cfg.speaker_token_base + i for i in range(cfg.num_speaker_prefix)]
+    assert ids[: cfg.num_speaker_prefix] == slots
+    assert ids.count(cfg.start_of_speech) == 1, "only the canonical copy may carry SOS"
+    assert ids[-1] == cfg.start_of_speech, "the first frame is sampled from the SOS position"
+
+
+def test_short_text_repeats_to_reach_the_target_token_mass() -> None:
+    """A short text is repeated until its region holds ~target_text_tokens."""
+    cfg = GepardConfig()
+    text = [1, 2, 3, 4]  # target 16 / 4 -> 4 copies
+    ids = build_gepard_prompt_ids(text, config=cfg)
+
+    assert _repeat_count(ids, cfg) == 4
+    assert ids.count(cfg.start_of_speech) == 1, "only the last copy opens the speech region"
+    assert ids[-1] == cfg.start_of_speech
+
+
+@pytest.mark.parametrize(
+    ("num_text_tokens", "repeated"),
+    [
+        (APPLY_BELOW - 1, True),
+        (APPLY_BELOW, False),
+        (APPLY_BELOW + 1, False),
+    ],
+)
+def test_apply_below_is_the_repetition_boundary(num_text_tokens: int, repeated: bool) -> None:
+    """Both sides of the threshold: at apply_below the text is left alone.
+
+    The layout has to match training here or WER collapses, and the boundary
+    is where an off-by-one would be invisible in a single happy-path clip.
+    """
+    cfg = GepardConfig()
+    ids = build_gepard_prompt_ids(list(range(1, num_text_tokens + 1)), config=cfg)
+
+    assert (_repeat_count(ids, cfg) > 1) is repeated
+
+
+def test_max_repeats_caps_a_one_token_text() -> None:
+    """target/1 would ask for 16 copies; the cap keeps the prefill bounded."""
+    cfg = GepardConfig()
+    ids = build_gepard_prompt_ids([1], config=cfg)
+
+    assert _repeat_count(ids, cfg) == cfg.text_repetition_max_repeats
+
+
+def test_repetition_thresholds_are_read_from_the_config() -> None:
+    """The numbers come from the checkpoint sidecar, not from literals.
+
+    A checkpoint whose text_repetition block differs must change the layout;
+    duplicating the values in the builder would silently ignore it.
+    """
+    cfg = GepardConfig(text_repetition={"enabled": True, "target_text_tokens": 9, "apply_below": 8, "max_repeats": 2})
+    assert _repeat_count(build_gepard_prompt_ids([1, 2, 3], config=cfg), cfg) == 2  # ceil(9/3)=3, capped at 2
+    assert _repeat_count(build_gepard_prompt_ids(list(range(8)), config=cfg), cfg) == 1  # at apply_below
+
+
+def test_repetition_can_be_disabled() -> None:
+    cfg = GepardConfig(text_repetition={"enabled": False})
+    ids = build_gepard_prompt_ids([1], config=cfg)
+
+    assert _repeat_count(ids, cfg) == 1
+    assert ids[-1] == cfg.start_of_speech
+
+
+def test_empty_text_is_rejected() -> None:
+    """An empty text would still assemble a valid layout and be voiced.
+
+    ``[slots, SOT, EOT, SOS]`` parses, prefills and generates: the failure is
+    audible, not raised, which is the worst shape for a caller to debug.
+    """
+    with pytest.raises(ValueError, match="at least one text token"):
+        build_gepard_prompt_ids([], config=GepardConfig())
+
+
+def test_packaged_deploy_config_is_where_the_example_looks_for_it() -> None:
+    """The example resolves its default off ``vllm_omni.__file__``; tests reach
+    the same YAML through ``tests.helpers.stage_config``, so nothing covered the
+    packaged location until this.
+
+    ``pipeline: gepard`` is the load-bearing key: the checkpoint self-identifies
+    as qwen3_5_text, so without the pin the architectures fallback routes it to
+    the diffusion registry.
+
+    This asserts the location, not the example's constant — importing the
+    example would pull ``Omni`` and NeMo into a CPU test. Running the quick
+    start verbatim stays a release check.
+    """
+    path = Path(vllm_omni.__file__).resolve().parent / "deploy" / "gepard.yaml"
+    assert path.is_file(), f"packaged deploy config missing at {path}"
+
+    deploy = load_deploy_config(path)
+    assert deploy.pipeline == "gepard"
+    assert deploy.stages, "deploy config declares no stages"

tests/model_executor/models/test_gepard_wiring.py

@@ -0,0 +1,731 @@
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
+"""CPU coverage for the seam between Gepard's talker and the AR runner.
+
+``test_gepard_window.py`` pins the decode arithmetic *below* this layer, and
+the e2e test exercises the layer *above* it but only along the path where the
+stop head fires. Neither covers the wiring itself: whether anything calls the
+final emit on a request's last step, and whether the payload it produces
+survives the runner's sparse-audio routing.
+
+These drive the real ``preprocess`` -> ``forward`` -> ``make_omni_output``
+sequence through a stand-in for ``GPUARModelRunner.execute_model``, and use the
+runner's own ``_resolve_sparse_mm_routing`` to decide what actually reaches the
+caller. Only the LM backbone and the frame sampler are stubbed out; every
+method under test is the real one, so a test here goes red when the engine
+path breaks — which a test that calls the flush by hand cannot do.
+"""
+
+from __future__ import annotations
+
+import logging
+
+import pytest
+import torch
+import torch.nn as nn
+
+from vllm_omni.model_executor.models.gepard.configuration_gepard import GepardConfig
+from vllm_omni.model_executor.models.gepard.gepard_talker import (
+    CHUNK_FRAMES,
+    FIRST_CHUNK_FRAMES,
+    GepardTalkerForConditionalGeneration,
+    _ReqInfo,
+)
+from vllm_omni.model_executor.models.gepard.nanocodec import NanoCodec
+from vllm_omni.model_executor.models.gepard.prompt import build_gepard_prompt_ids
+from vllm_omni.worker.gpu_ar_model_runner import GPUARModelRunner
+
+pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
+
+SAMPLES_PER_FRAME = 1024
+NUM_HEADS = 32
+SAMPLE_RATE = 22050
+LOGGER_NAME = "vllm_omni.model_executor.models.gepard.gepard_talker"
+
+# Each request tags its frames from a distinct base, so a sample that reaches
+# the wrong request is visible in the value, not just the length.
+_TAG_STRIDE = 100_000
+
+
+class _TagCodec:
+    """Frame carrying tag ``v`` -> samples ``v*SPF .. v*SPF+SPF-1``, wherever it sits.
+
+    Shift-invariant, so "what the caller received" can be compared against one
+    whole-history decode exactly. The tag rides channel 1: channel 0 is head0,
+    which the talker keeps inside its real (small) vocab.
+    """
+
+    def __init__(self) -> None:
+        self.decode_calls = 0
+
+    def decode_from_codes(self, codes: torch.Tensor, codes_len: torch.Tensor):
+        assert codes.shape[0] == 1, "streaming decode is per-request"
+        assert codes.shape[1] == NUM_HEADS, f"expected {NUM_HEADS} code channels"
+        self.decode_calls += 1
+        tags = codes[0, 1, :].to(torch.int64)  # (T,)
+        offsets = torch.arange(SAMPLES_PER_FRAME, dtype=torch.int64)
+        audio = (tags.unsqueeze(1) * SAMPLES_PER_FRAME + offsets).reshape(1, -1)
+        return audio.to(torch.float32), codes_len * SAMPLES_PER_FRAME
+
+
+class _StubBackbone(nn.Module):
+    """Stands in for the Qwen3.5 backbone; only shapes matter to this seam."""
+
+    def __init__(self, hidden: int) -> None:
+        super().__init__()
+        self.hidden = hidden
+
+    def embed_input_ids(self, input_ids: torch.Tensor) -> torch.Tensor:
+        return torch.zeros(int(input_ids.shape[0]), self.hidden)
+
+    def forward(
+        self,
+        input_ids: torch.Tensor | None = None,
+        positions: torch.Tensor | None = None,
+        intermediate_tensors=None,
+        inputs_embeds: torch.Tensor | None = None,
+    ) -> torch.Tensor:
+        src = inputs_embeds if inputs_embeds is not None else input_ids
+        return torch.zeros(int(src.shape[0]), self.hidden)
+
+
+def _expected_samples(tag_base: int, num_frames: int) -> torch.Tensor:
+    offsets = torch.arange(SAMPLES_PER_FRAME, dtype=torch.int64)
+    tags = (torch.arange(num_frames, dtype=torch.int64) + tag_base).unsqueeze(1)
+    return (tags * SAMPLES_PER_FRAME + offsets).reshape(-1).to(torch.float32)
+
+
+class _MiniEngine:
+    """Drives the talker the way ``GPUARModelRunner.execute_model`` does.
+
+    Reproduces the three orderings the dropped-tail bug lived in:
+
+    * ``on_requests_finished`` fires at the START of the step AFTER the one
+      that finished the request (it reads ``scheduler_output.finished_req_ids``);
+    * a finished request is not scheduled again, so it is absent from
+      ``req_ids_output_copy`` and from that step's routing;
+    * with nothing left to schedule there is no step at all, so no further
+      ``forward`` runs;
+    * a stop-token finish gets one more scheduled step, a budget finish does
+      not -- under async scheduling only the latter is guarded against it.
+    """
+
+    def __init__(self, talker: GepardTalkerForConditionalGeneration, *, max_model_len: int) -> None:
+        self.talker = talker
+        self.max_model_len = max_model_len
+        self.reqs: dict[str, dict] = {}
+        self.order: list[str] = []
+        self._finished_last_step: list[str] = []
+        self.delivered: dict[str, list[torch.Tensor]] = {}
+        self.committed_frames: dict[str, int] = {}
+        self.steps = 0
+        self._tags_handed_out = 0
+
+    def add_request(
+        self,
+        req_id: str,
+        *,
+        max_tokens: int,
+        stop_after: int | None = None,
+        text_len: int = 3,
+        seed: int | None = None,
+    ) -> None:
+        """Submit a request. An id already seen is a resubmit of that id.
+
+        Tags come from a counter, so a resubmitted id gets frames the previous
+        request could not have produced.
+        """
+        prompt_ids = build_gepard_prompt_ids(list(range(10, 10 + text_len)), config=self.talker.config)
+        self._tags_handed_out += 1
+        self.reqs[req_id] = {
+            "prompt_ids": torch.tensor(prompt_ids, dtype=torch.long),
+            "prompt_len": len(prompt_ids),
+            "max_tokens": max_tokens,
+            "stop_after": stop_after,
+            "stop_fired": False,
+            "seed": seed,
+            "computed": 0,
+            "generated": 0,
+            "finished": False,
+            "ghost_step_pending": False,
+            "next_token": None,
+            "tag_base": _TAG_STRIDE * self._tags_handed_out,
+        }
+        if req_id not in self.order:
+            self.order.append(req_id)
+        self.delivered[req_id] = []
+        self.committed_frames[req_id] = 0
+
+    def abort(self, req_id: str) -> None:
+        """Kill a request the way an aborted client does: no STOP, no budget."""
+        self.reqs[req_id]["finished"] = True
+        self._finished_last_step.append(req_id)
+
+    def _live(self) -> list[str]:
+        return [r for r in self.order if not self.reqs[r]["finished"]]
+
+    def step(self) -> bool:
+        """Run one engine step. False when there was nothing to schedule."""
+        live = self._live()
+        if not live:
+            return False
+        talker = self.talker
+        self.steps += 1
+
+        # --- runner: notify the model of requests finished since last step ---
+        if self._finished_last_step:
+            talker.on_requests_finished(set(self._finished_last_step))
+            self._finished_last_step = []
+
+        # --- runner: per-request preprocess, in input_batch.req_ids order ---
+        embeds_parts, ids_parts = [], []
+        for req_id in live:
+            r = self.reqs[req_id]
+            is_prefill = r["computed"] < r["prompt_len"]
+            if is_prefill:
+                ids = r["prompt_ids"]
+            else:
+                ids = torch.tensor([r["next_token"] or 0], dtype=torch.long)
+            span = int(ids.shape[0])
+            out_ids, out_embeds, _ = talker.preprocess(
+                input_ids=ids,
+                input_embeds=None,
+                request_id=req_id,
+                _omni_prompt_len=r["prompt_len"],
+                _omni_num_computed_tokens=r["computed"],
+                _omni_is_prefill=is_prefill,
+                _omni_max_tokens=r["max_tokens"],
+                _omni_seed=r["seed"],
+            )
+            ids_parts.append(out_ids)
+            embeds_parts.append(out_embeds)
+            r["span"] = span
+
+        inputs_embeds = torch.cat(embeds_parts, dim=0)
+        input_ids = torch.cat(ids_parts, dim=0)
+        positions = torch.arange(int(input_ids.shape[0]), dtype=torch.long)
+
+        # --- runner: forward, then build the multimodal payload ---
+        hidden = talker.forward(input_ids, positions, inputs_embeds=inputs_embeds)
+        results = list(talker._results_queue)
+        omni_out = talker.make_omni_output(hidden)
+        mm = omni_out.multimodal_outputs
+        # compute_logits drains this; it does not touch the audio path.
+        talker._results_queue.clear()
+
+        for req_id in live:
+            state = talker._active_states.get(req_id)
+            if state is not None:
+                self.committed_frames[req_id] = state.frame_count
+
+        # --- runner: sparse-audio routing decides what reaches the caller ---
+        downstream, sparse_index, is_sparse = GPUARModelRunner._resolve_sparse_mm_routing(
+            engine_output_type="audio",
+            req_ids_output_copy=list(live),
+            downstream_req_ids=list(live),
+            multimodal_outputs=mm,
+        )
+        assert is_sparse, "gepard always marks meta.sparse_audio"
+        for rid in downstream:
+            self.delivered[rid].append(mm["model_outputs"][sparse_index[rid]].reshape(-1))
+
+        # --- runner/scheduler bookkeeping: advance and detect finishes ---
+        stopped = {rid for rid, _head0, do_stop in results if do_stop}
+        for req_id in live:
+            r = self.reqs[req_id]
+            r["computed"] += r["span"]
+            r["generated"] = r["computed"] - r["prompt_len"] + 1
+            r["next_token"] = 0
+            length_cap = min(r["max_tokens"], self.max_model_len - r["prompt_len"])
+            if r["ghost_step_pending"]:
+                # This step WAS the extra one. The engine finished the request
+                # on the stop token last step and discards whatever came back
+                # here, whether or not the model chose to stop again.
+                r["finished"] = True
+                self._finished_last_step.append(req_id)
+            elif req_id in stopped:
+                # Ends on the stop token: one more step is already scheduled.
+                r["ghost_step_pending"] = True
+            elif r["generated"] >= length_cap:
+                r["finished"] = True
+                self._finished_last_step.append(req_id)
+        return True
+
+    def run(self, max_steps: int = 4000) -> None:
+        for _ in range(max_steps):
+            if not self.step():
+                return
+        raise AssertionError("engine did not drain")
+
+    def audio_for(self, req_id: str) -> torch.Tensor:
+        chunks = self.delivered[req_id]
+        return torch.cat(chunks) if chunks else torch.empty(0)
+
+
+def _make_talker(*, max_model_len: int = 4096) -> tuple[GepardTalkerForConditionalGeneration, _TagCodec]:
+    """A real talker with only the audio-path collaborators populated.
+
+    ``__init__`` needs a VllmConfig and builds the Qwen3.5 backbone; the seam
+    under test needs none of that, so the backbone and the frame sampler are
+    stubbed and everything else is the class's own code.
+    """
+    cfg = GepardConfig()
+    # Fixed, not cfg.get_text_config().hidden_size: nothing in this seam reads
+    # the real width, and a small one keeps the test off the backbone config.
+    hidden = 64
+    codec = NanoCodec(codec_id="test/fake-codec", sample_rate=SAMPLE_RATE)
+    fake = _TagCodec()
+    codec._codec = fake  # bypasses load(); NeMo is never imported
+    assert codec.is_loaded
+
+    talker = GepardTalkerForConditionalGeneration.__new__(GepardTalkerForConditionalGeneration)
+    nn.Module.__init__(talker)
+    talker.config = cfg
+    talker.model = _StubBackbone(hidden)
+    # requires_grad=False: preprocess writes the prefix into inputs_embeds
+    # in place, and production runs that under inference_mode.
+    talker.null_prefix = nn.Parameter(torch.zeros(cfg.num_speaker_prefix, hidden), requires_grad=False)
+    talker._codec = codec
+    talker._sample_rate = SAMPLE_RATE
+    talker._max_model_len = max_model_len
+    talker._active_states = {}
+    talker._pending_requests = []
+    talker._results_queue = []
+    talker._audio_queue = []
+    talker._deferred_cleanup_ids = set()
+    return talker, fake
+
+
+def _install_scripted_sampling(talker: GepardTalkerForConditionalGeneration, engine: _MiniEngine) -> None:
+    """Replace the 32-head sampler with a scripted one.
+
+    Frames carry a per-request, per-step tag so the caller's audio is
+    self-describing. STOP fires once, on the request's configured frame: on the
+    extra step the engine then runs, re-firing is likely but not guaranteed,
+    and the model may not assume it.
+    """
+    head0_vocab = talker.config.head0_vocab_size
+
+    # Signature mirrors the real ``_sample_frame`` minus ``self``; the scripted
+    # sampler ignores ``generators`` because seeding is decided in
+    # ``_build_with_real_sampler``, which keeps the model's own sampler.
+    def _sample_frame(rows: torch.Tensor, generators: list[torch.Generator | None] | None = None):
+        n = int(rows.shape[0])
+        head0 = torch.zeros(n, dtype=torch.long)
+        heads = torch.zeros(n, NUM_HEADS - 1, dtype=torch.long)
+        stop = torch.zeros(n, dtype=torch.bool)
+        for i, (req_id, _span, _samples_frame, *_rest) in enumerate(talker._pending_requests):
+            state = talker._active_states.get(req_id)
+            frame_index = state.frame_count if state is not None else 0
+            tag = engine.reqs[req_id]["tag_base"] + frame_index
+            head0[i] = tag % head0_vocab
+            heads[i] = tag
+            r = engine.reqs[req_id]
+            if r["stop_after"] is not None and frame_index >= r["stop_after"] and not r["stop_fired"]:
+                r["stop_fired"] = True
+                stop[i] = True
+        return head0, heads, stop
+
+    talker._sample_frame = _sample_frame
+    talker._audio_frame_embed = lambda h0, h: torch.zeros(int(h0.shape[0]), talker.model.hidden)
+
+
+def _build(*, max_model_len: int = 4096) -> tuple[GepardTalkerForConditionalGeneration, _TagCodec, _MiniEngine]:
+    talker, codec = _make_talker(max_model_len=max_model_len)
+    engine = _MiniEngine(talker, max_model_len=max_model_len)
+    _install_scripted_sampling(talker, engine)
+    return talker, codec, engine
+
+
+def _build_with_real_sampler(*, max_model_len: int = 4096) -> tuple[GepardTalkerForConditionalGeneration, _MiniEngine]:
+    """Same seam, but ``_sample_frame`` is the model's own 32-head sampler.
+
+    The stub backbone hands every row the same hidden state, so the sampled
+    codes — and therefore the audio the codec tags them into — vary with
+    nothing but the Gumbel draws. That makes this the one place where "did the
+    caller's seed reach the audio" is a decidable question.
+    """
+    talker, _codec = _make_talker(max_model_len=max_model_len)
+    engine = _MiniEngine(talker, max_model_len=max_model_len)
+    cfg = talker.config
+    talker.vocab_sizes = list(cfg.audio_head_levels)
+    talker.num_heads = cfg.num_audio_heads
+    talker.head0_vocab = cfg.head0_vocab_size
+    talker.stop_token = cfg.stop_token
+    talker.stop_threshold = cfg.stop_threshold
+    talker.temperature = cfg.temperature
+    talker._greedy = False
+    gather_idx, mask = GepardTalkerForConditionalGeneration._build_gather_mask(talker.vocab_sizes)
+    talker._cb_gather_idx = gather_idx
+    talker._cb_mask = mask
+    # Fixed weights: two engine runs must differ only by their RNG.
+    torch.manual_seed(0)
+    talker.fused_codebook_head = nn.Linear(talker.model.hidden, sum(talker.vocab_sizes))
+    talker.stop_head = nn.Linear(talker.model.hidden, 1)
+    # Never stop: these requests end on their token budget, so the frame count
+    # is fixed and only the sampled codes can differ.
+    torch.nn.init.constant_(talker.stop_head.bias, -20.0)
+    talker._audio_frame_embed = lambda h0, h: torch.zeros(int(h0.shape[0]), talker.model.hidden)
+    return talker, engine
+
+
+# --------------------------------------------------------------------------
+# The tail must reach the caller, whichever way the request ends.
+# --------------------------------------------------------------------------
+
+
+def test_last_in_flight_request_delivers_every_committed_frame() -> None:
+    """The only request runs out of budget: there is no later step to flush in.
+
+    This is the case a deferred flush cannot serve at all — nothing runs after
+    the final forward.
+    """
+    _talker, _codec, engine = _build()
+    engine.add_request("req-0", max_tokens=FIRST_CHUNK_FRAMES + CHUNK_FRAMES + 5)
+    engine.run()
+
+    frames = engine.committed_frames["req-0"]
+    assert frames > 0
+    torch.testing.assert_close(
+        engine.audio_for("req-0"),
+        _expected_samples(engine.reqs["req-0"]["tag_base"], frames),
+    )
+
+
+@pytest.mark.parametrize(
+    "max_tokens",
+    [
+        1,
+        FIRST_CHUNK_FRAMES,
+        FIRST_CHUNK_FRAMES + 1,
+        30,
+        FIRST_CHUNK_FRAMES + CHUNK_FRAMES,
+        100,
+        150,
+    ],
+)
+def test_budget_truncated_request_delivers_every_committed_frame(max_tokens: int) -> None:
+    """Lowering max_tokens must shorten the clip, never silently cut it back
+    to the last cadence boundary."""
+    _talker, _codec, engine = _build()
+    engine.add_request("req-0", max_tokens=max_tokens)
+    engine.run()
+
+    frames = engine.committed_frames["req-0"]
+    assert frames == max_tokens, "every sampled step commits one frame when STOP never fires"
+    torch.testing.assert_close(
+        engine.audio_for("req-0"),
+        _expected_samples(engine.reqs["req-0"]["tag_base"], frames),
+    )
+
+
+def test_stop_head_path_still_delivers_every_committed_frame() -> None:
+    """Regression guard: the path that already worked must keep working."""
+    _talker, _codec, engine = _build()
+    engine.add_request("req-0", max_tokens=4000, stop_after=FIRST_CHUNK_FRAMES + 7)
+    engine.run()
+
+    frames = engine.committed_frames["req-0"]
+    assert frames == FIRST_CHUNK_FRAMES + 7, "the STOP frame itself is not committed"
+    torch.testing.assert_close(
+        engine.audio_for("req-0"),
+        _expected_samples(engine.reqs["req-0"]["tag_base"], frames),
+    )
+
+
+def test_the_step_after_a_stop_token_commits_nothing(caplog: pytest.LogCaptureFixture) -> None:
+    """Async scheduling runs one more step on a request that already stopped.
+
+    Committing its frame would strand it — nothing ships after the stop step —
+    and would make the truncation warning fire on a request that delivered
+    everything it produced, which is the one thing that warning must not do.
+    """
+    talker, _codec, engine = _build()
+    stop_after = FIRST_CHUNK_FRAMES + 6
+    engine.add_request("req-0", max_tokens=4000, stop_after=stop_after)
+
+    target = logging.getLogger(LOGGER_NAME)
+    target.addHandler(caplog.handler)
+    prev = target.level
+    target.setLevel(logging.WARNING)
+    try:
+        engine.run()
+        # The cleanup that would warn runs on the step after the request
+        # leaves, and here there is none. Drive it the way the runner would.
+        talker.on_requests_finished({"req-0"})
+        talker._flush_deferred_cleanup()
+    finally:
+        target.removeHandler(caplog.handler)
+        target.setLevel(prev)
+
+    assert engine.reqs["req-0"]["ghost_step_pending"], "the harness did not run the extra step"
+    frames = engine.committed_frames["req-0"]
+    assert frames == stop_after, f"the extra step committed a frame: {frames} > {stop_after}"
+    torch.testing.assert_close(
+        engine.audio_for("req-0"),
+        _expected_samples(engine.reqs["req-0"]["tag_base"], frames),
+    )
+    assert not [m for m in caplog.messages if "undelivered frame" in m], (
+        f"a fully delivered request was reported as truncated; saw {caplog.messages}"
+    )
+
+
+def test_a_resubmitted_request_id_keeps_its_new_state(caplog: pytest.LogCaptureFixture) -> None:
+    """An aborted id can come back before its cleanup has run.
+
+    The runner treats the two as distinct requests, so the pending free must
+    not take the second one's state: without that, the state is popped at the
+    end of the very forward that created it and every later decode falls back
+    to the zero-embedding path — the request keeps producing frames, so only
+    the audio is wrong.
+    """
+    _talker, _codec, engine = _build()
+    engine.add_request("req-0", max_tokens=4000)
+    for _ in range(FIRST_CHUNK_FRAMES + 2):
+        engine.step()
+    engine.abort("req-0")
+
+    # Same id, resubmitted before the model has been told about the abort.
+    engine.add_request("req-0", max_tokens=FIRST_CHUNK_FRAMES + CHUNK_FRAMES + 5)
+
+    target = logging.getLogger(LOGGER_NAME)
+    target.addHandler(caplog.handler)
+    prev = target.level
+    target.setLevel(logging.WARNING)
+    try:
+        engine.run()
+    finally:
+        target.removeHandler(caplog.handler)
+        target.setLevel(prev)
+
+    frames = engine.committed_frames["req-0"]
+    assert frames == engine.reqs["req-0"]["max_tokens"], "the second request lost frames"
+    torch.testing.assert_close(
+        engine.audio_for("req-0"),
+        _expected_samples(engine.reqs["req-0"]["tag_base"], frames),
+        msg="the second request did not receive its own audio",
+    )
+    # The abort still gets reported: the cleanup that would have said so is the
+    # one this cancelled.
+    assert any("resubmitted" in m for m in caplog.messages), (
+        f"the aborted request's truncation went unreported; saw {caplog.messages}"
+    )
+
+
+def test_a_preempted_request_is_reported_rather_than_resumed(caplog: pytest.LogCaptureFixture) -> None:
+    """Recompute preemption re-prefills a request that was mid-generation.
+
+    This model cannot survive that, and the point of the test is that it says
+    so. The recomputed ids are the prompt plus the head0 codes sampled so far,
+    which go through the text embedding table, and each frame's other 31 codes
+    are never in the token stream at all — there is nothing to stitch the two
+    halves back together with, so generation restarts from frame 0.
+
+    The harness re-prefills with the bare prompt, which understates the damage.
+    What is under test is the report and the reset, not the corruption, which
+    needs a real backbone to observe.
+    """
+    _talker, _codec, engine = _build()
+    engine.add_request("req-0", max_tokens=4000)
+    for _ in range(FIRST_CHUNK_FRAMES + 2):
+        engine.step()
+    assert engine.committed_frames["req-0"] > 0
+
+    # Preemption the way the scheduler does it: computed tokens back to 0, so
+    # the next step re-prefills, and no id reaches on_requests_finished.
+    engine.reqs["req-0"]["computed"] = 0
+
+    target = logging.getLogger(LOGGER_NAME)
+    target.addHandler(caplog.handler)
+    prev = target.level
+    target.setLevel(logging.WARNING)
+    try:
+        engine.step()
+    finally:
+        target.removeHandler(caplog.handler)
+        target.setLevel(prev)
+
+    assert any("preempted" in m for m in caplog.messages), (
+        f"a preempted request restarted its audio silently; saw {caplog.messages}"
+    )
+    assert not any("resubmitted" in m for m in caplog.messages), (
+        "a preemption was reported as an id resubmit; the two need different words"
+    )
+    assert engine.committed_frames["req-0"] == 1, "the re-prefill must start the frame history over"
+
+
+def test_concurrent_requests_each_receive_their_own_full_audio() -> None:
+    """One request finishes while another stays live.
+
+    The earlier finisher is the case where a deferred flush *does* run but the
+    payload is filtered out again, because its id has left the output batch.
+    """
+    _talker, _codec, engine = _build()
+    engine.add_request("req-short", max_tokens=FIRST_CHUNK_FRAMES + 3)
+    engine.add_request("req-long", max_tokens=FIRST_CHUNK_FRAMES + CHUNK_FRAMES + 11)
+    engine.run()
+
+    for req_id in ("req-short", "req-long"):
+        frames = engine.committed_frames[req_id]
+        assert frames == engine.reqs[req_id]["max_tokens"]
+        torch.testing.assert_close(
+            engine.audio_for(req_id),
+            _expected_samples(engine.reqs[req_id]["tag_base"], frames),
+            msg=f"{req_id} lost or gained samples",
+        )
+
+
+def test_codec_still_runs_once_per_chunk_not_once_per_frame() -> None:
+    """Emitting on the final step must not degrade into per-frame decoding."""
+    _talker, codec, engine = _build()
+    num_frames = FIRST_CHUNK_FRAMES + 3 * CHUNK_FRAMES
+    engine.add_request("req-0", max_tokens=num_frames)
+    engine.run()
+
+    assert codec.decode_calls == 4, f"expected one decode per chunk, got {codec.decode_calls}"
+
+
+# --------------------------------------------------------------------------
+# Deferred cleanup is cleanup: it must not try to produce output.
+# --------------------------------------------------------------------------
+
+
+def test_deferred_cleanup_never_queues_audio() -> None:
+    """Anything queued from the cleanup hook is unroutable by construction.
+
+    By the time it runs the id has left ``req_ids_output_copy``, so a payload
+    under it is dropped; queueing there only risks leaking one request's audio
+    into the next step's payload for a different request.
+    """
+    talker, _codec, engine = _build()
+    engine.add_request("req-0", max_tokens=FIRST_CHUNK_FRAMES + 4)
+    engine.add_request("req-1", max_tokens=FIRST_CHUNK_FRAMES + CHUNK_FRAMES)
+    while engine.step():
+        if engine.reqs["req-0"]["finished"] and not engine.reqs["req-1"]["finished"]:
+            break
+
+    talker._audio_queue.clear()
+    talker.on_requests_finished({"req-0"})
+    talker._flush_deferred_cleanup()
+
+    assert talker._audio_queue == [], "cleanup emitted audio that routing cannot deliver"
+    assert "req-0" not in talker._active_states, "cleanup must still free the state"
+
+
+def test_cleanup_warns_when_frames_are_left_undelivered(caplog: pytest.LogCaptureFixture) -> None:
+    """An abort is the one case with no last step to emit on. Say so."""
+    _talker, _codec, engine = _build()
+    engine.add_request("req-0", max_tokens=4000)
+    engine.add_request("req-1", max_tokens=4000)
+    for _ in range(FIRST_CHUNK_FRAMES + 3):
+        engine.step()
+    engine.abort("req-0")
+
+    target = logging.getLogger(LOGGER_NAME)
+    target.addHandler(caplog.handler)
+    prev = target.level
+    target.setLevel(logging.WARNING)
+    try:
+        engine.step()  # cleanup for req-0 runs at the start of this step
+    finally:
+        target.removeHandler(caplog.handler)
+        target.setLevel(prev)
+
+    assert any("undelivered frame" in m for m in caplog.messages), (
+        f"aborted request dropped audio without a warning; saw {caplog.messages}"
+    )
+
+
+# --------------------------------------------------------------------------
+# A request's seed must reach the in-model sampler.
+# --------------------------------------------------------------------------
+
+_SEEDED_FRAMES = FIRST_CHUNK_FRAMES + 5
+
+
+def _run_seeded(seed: int | None, *, neighbour: bool = False, neighbour_seed: int | None = None) -> torch.Tensor:
+    """Audio delivered to ``req-a``, optionally sharing the batch with another."""
+    _talker, engine = _build_with_real_sampler()
+    engine.add_request("req-a", max_tokens=_SEEDED_FRAMES, seed=seed)
+    if neighbour:
+        engine.add_request("req-b", max_tokens=_SEEDED_FRAMES, seed=neighbour_seed)
+    engine.run()
+    return engine.audio_for("req-a")
+
+
+def test_a_seeded_request_is_reproducible_within_one_process() -> None:
+    """The bug this pins: the 32 heads are sampled in ``forward``, so a seed
+    that only reaches vLLM's sampler leaves the audio at the mercy of the
+    global RNG — two identical requests in one process then diverge."""
+    torch.testing.assert_close(_run_seeded(1234), _run_seeded(1234))
+
+
+def test_different_seeds_produce_different_audio() -> None:
+    """Guards the opposite failure: a seed that is threaded but never used."""
+    assert not torch.equal(_run_seeded(1234), _run_seeded(4321))
+
+
+def test_a_seeded_request_is_unaffected_by_its_batch() -> None:
+    """Each seeded request draws its own noise, so the noise it sees cannot
+    depend on who else was scheduled alongside it.
+
+    The stub backbone is what makes that checkable. A real one is a bf16
+    batched matmul whose rows are not reproducible across batch shapes, so
+    end-to-end batch invariance is not attainable there; this pins the half
+    the model actually controls.
+    """
+    torch.testing.assert_close(_run_seeded(1234), _run_seeded(1234, neighbour=True, neighbour_seed=999))
+
+
+def test_an_unseeded_neighbour_does_not_disturb_a_seeded_request() -> None:
+    """The mixed case: one row draws from the global RNG, the other must not."""
+    torch.testing.assert_close(_run_seeded(1234), _run_seeded(1234, neighbour=True))
+
+
+# --------------------------------------------------------------------------
+# Budget arithmetic, straight through the helper.
+# --------------------------------------------------------------------------
+
+
+def _info(**kwargs) -> _ReqInfo:
+    base = {
+        "request_id": "req-0",
+        "is_prefill": False,
+        "prompt_len": 20,
+        "num_computed_tokens": 20,
+        "max_tokens": 10,
+        "seed": None,
+    }
+    base.update(kwargs)
+    return _ReqInfo(**base)
+
+
+def test_prefill_step_is_the_first_output_token() -> None:
+    talker, _codec, _engine = _build()
+    first = _info(is_prefill=True, num_computed_tokens=0, max_tokens=1)
+    assert talker._is_last_output_token(first, span=20, samples_frame=True) is True
+    more = _info(is_prefill=True, num_computed_tokens=0, max_tokens=2)
+    assert talker._is_last_output_token(more, span=20, samples_frame=True) is False
+
+
+def test_partial_prefill_chunk_does_not_consume_budget() -> None:
+    """vLLM discards that row's token, so it must not count against max_tokens."""
+    talker, _codec, _engine = _build()
+    info = _info(is_prefill=True, num_computed_tokens=0, prompt_len=40, max_tokens=1)
+    assert talker._is_last_output_token(info, span=10, samples_frame=False) is False
+
+
+def test_budget_is_capped_by_max_model_len() -> None:
+    """A max_tokens larger than the context window never gets reached."""
+    talker, _codec, _engine = _build(max_model_len=30)
+    # prompt 20 + 10 generated fills the window, whatever max_tokens says.
+    at_cap = _info(num_computed_tokens=28, max_tokens=1000)
+    assert talker._is_last_output_token(at_cap, span=1, samples_frame=True) is True
+    below_cap = _info(num_computed_tokens=27, max_tokens=1000)
+    assert talker._is_last_output_token(below_cap, span=1, samples_frame=True) is False
+
+
+def test_missing_budget_is_not_treated_as_exhausted() -> None:
+    """A runner that does not thread max_tokens must not trigger a fade."""
+    talker, _codec, _engine = _build(max_model_len=0)
+    assert talker._is_last_output_token(_info(max_tokens=None), span=1, samples_frame=True) is False

vllm_omni/config/pipeline_registry.py

@@ -51,6 +51,7 @@ from vllm_omni.model_executor.models.dots_tts.pipeline import DOTS_TTS_PIPELINE
 from vllm_omni.model_executor.models.dreamzero.pipeline import DREAMZERO_PIPELINE
 from vllm_omni.model_executor.models.dynin_omni.pipeline import DYNIN_OMNI_PIPELINE
 from vllm_omni.model_executor.models.fish_speech.pipeline import FISH_SPEECH_PIPELINE
+from vllm_omni.model_executor.models.gepard.pipeline import GEPARD_PIPELINE
 from vllm_omni.model_executor.models.glm_image.pipeline import GLM_IMAGE_PIPELINE
 from vllm_omni.model_executor.models.glm_tts.pipeline import GLM_TTS_PIPELINE
 from vllm_omni.model_executor.models.gr00t.pipeline import GR00T_N1D7_PIPELINE
@@ -138,6 +139,7 @@ OMNI_PIPELINES: dict[str, PipelineConfig | PipelineResolverFunc] = {
     "lance": LANCE_PIPELINE,
     "dreamzero": DREAMZERO_PIPELINE,
     "Gr00tN1d7": GR00T_N1D7_PIPELINE,
+    "gepard": GEPARD_PIPELINE,
     "glm_image": GLM_IMAGE_PIPELINE,
     "hunyuan_image_3_moe": HUNYUAN_IMAGE3_PIPELINE,
     "hunyuan_image3_ar": HUNYUAN_IMAGE3_AR_PIPELINE,

vllm_omni/deploy/gepard.yaml

@@ -0,0 +1,40 @@
+# Gepard-1.0: single-stage native-AR FSQ/NanoCodec TTS (~0.5B).
+# Target 1x L4 24GB. Zero-shot only; enforce_eager until the perf PR.
+#
+# The pipeline pin is required, not cosmetic: the checkpoint's config.json
+# self-identifies as qwen3_5_text, so model_type inference cannot reach the
+# gepard registry entry, and the architectures fallback would match
+# Qwen3_5ForCausalLM against the diffusion pipeline.
+pipeline: gepard
+async_chunk: false
+dtype: bfloat16
+trust_remote_code: true
+
+stages:
+  - stage_id: 0
+    max_num_seqs: 4
+    gpu_memory_utilization: 0.3      # AR LM + NanoCodec fit in ~2 GiB
+    enforce_eager: true
+    enable_prefix_caching: false
+    async_scheduling: true
+    max_num_batched_tokens: 4096
+    max_model_len: 4096
+    devices: "0"
+    # Override architectures so vLLM resolves our talker rather than the native
+    # Qwen3.5 text model. Do NOT override model_type: the backbone's own
+    # model_type check needs it to stay qwen3_5_text.
+    hf_overrides:
+      architectures: ["GepardTalkerForConditionalGeneration"]
+    default_sampling_params:
+      # All 32 heads are sampled in-model, so vLLM's sampler is pinned
+      # deterministic and just argmaxes what forward chose.
+      temperature: 0.0
+      top_p: 1.0
+      top_k: -1
+      # One output token is one frame, so this is the frame budget: 1000 frames
+      # is ~46 s at 21.5 fps. A backstop, not a target -- normal utterances are
+      # 40-420 frames -- but a load-bearing one, because this checkpoint's stop
+      # head does not fire on every batched trajectory.
+      max_tokens: 1000
+      seed: 42
+      detokenize: false

vllm_omni/model_executor/models/gepard/__init__.py

@@ -0,0 +1,8 @@
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
+
+# NOTE: Do not import model classes in this file. Importing any
+# submodule in this package triggers __init__.py execution, and
+# both the model registry and pipeline registry import submodules
+# directly — heavy imports here would be loaded as a side effect
+# even though nothing depends on these re-exports.

vllm_omni/model_executor/models/gepard/configuration_gepard.py

@@ -0,0 +1,215 @@
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
+"""Config for Gepard-1.0, a single-stage autoregressive TTS.
+
+Text tokens -> one 32-code FSQ audio frame per step -> NeMo NanoCodec ->
+waveform. The backbone is a vLLM-native ``Qwen3_5ForCausalLM``; the codebook
+heads, binary stop head and voice-clone ref_compressor are Gepard additions.
+
+Parses the model's ``gepard_config.json`` sidecar, which nests the LM
+parameters under ``backbone_config`` and carries audio-head cardinalities,
+special tokens, codec settings and the short-text repetition layout.
+"""
+
+from __future__ import annotations
+
+from transformers import AutoConfig
+from transformers.configuration_utils import PretrainedConfig
+from transformers.utils import logging
+
+logger = logging.get_logger(__name__)
+
+_BACKBONE_MODEL_TYPE = "qwen3_5_text"
+
+# Stripped from the backbone config: the model uses standard 1D RoPE, and these
+# would flip vLLM's uses_mrope() to True.
+_MROPE_KEYS = ("mrope_section", "mrope_interleaved")
+
+
+class GepardConfig(PretrainedConfig):
+    """Configuration for the Gepard-1.0 native-AR TTS model.
+
+    Args mirror ``gepard_config.json``.  Defaults match the trained
+    ``nineninesix/gepard-1.0`` checkpoint so an instance built with no
+    arguments (e.g. dummy/profiling loads) is still self-consistent.
+    """
+
+    model_type = "gepard"
+    keys_to_ignore_at_inference = ["past_key_values"]
+
+    def __init__(
+        self,
+        backbone_config: dict | None = None,
+        audio_heads: dict | None = None,
+        audio_embed_dim: int = 32,
+        special_tokens: dict | None = None,
+        text_repetition: dict | None = None,
+        codec: dict | None = None,
+        voice_cloning: dict | None = None,
+        stop_threshold: float = 0.5,
+        stop_loss_weight: float = 2.0,
+        stop_pos_weight: float = 25.0,
+        temperature: float = 0.3,
+        **kwargs,
+    ):
+        # Everything is assigned before super().__init__() at the bottom:
+        # transformers 5.x runs validators there that call get_text_config().
+        self.backbone_config = self._normalize_backbone(backbone_config)
+
+        self.audio_head_levels = self._parse_audio_heads(audio_heads)
+        self.num_audio_heads = len(self.audio_head_levels)
+        self.audio_embed_dim = audio_embed_dim
+        # head0 is the token vLLM samples; STOP is a synthetic sentinel one past
+        # its valid range.
+        self.head0_vocab_size = self.audio_head_levels[0] if self.audio_head_levels else 8
+        self.stop_token = self.head0_vocab_size
+
+        st = special_tokens or {}
+        self.start_of_text = st.get("start_of_text", 248073)
+        self.end_of_text = st.get("end_of_text", 248074)
+        self.start_of_speech = st.get("start_of_speech", 248070)
+        self.end_of_speech = st.get("end_of_speech", 248071)
+        self.tts_pad = st.get("tts_pad", 248076)
+        # The speaker placeholder slots begin at tokeniser_length.
+        self.tokeniser_length = st.get("tokeniser_length", 248077)
+        self.speaker_token_base = self.tokeniser_length
+
+        # Prompt layout, read by prompt.py. These must match the training
+        # layout, so they come from the checkpoint rather than a literal.
+        tr = text_repetition or {}
+        self.text_repetition_enabled = tr.get("enabled", True)
+        self.text_repetition_target_tokens = tr.get("target_text_tokens", 16)
+        self.text_repetition_apply_below = tr.get("apply_below", 13)
+        self.text_repetition_max_repeats = tr.get("max_repeats", 8)
+
+        # NeMo NanoCodec, which runs outside vLLM.
+        cc = codec or {}
+        self.codec_id = cc.get("codec_id", "nvidia/nemo-nano-codec-22khz-1.89kbps-21.5fps")
+        self.codec_sample_rate = cc.get("sample_rate", 22050)
+        self.codec_frame_rate_hz = cc.get("frame_rate_hz", 21.5)
+        # Per-group FSQ levels; num_codec_groups * len(fsq_levels) == 32.
+        self.fsq_levels = cc.get("fsq_levels", [8, 7, 6, 6])
+        self.num_codec_groups = cc.get("num_layers", 8)
+        self.codec_do_unfold = cc.get("do_unfold", True)
+
+        self.stop_threshold = stop_threshold
+        self.stop_loss_weight = stop_loss_weight
+        self.stop_pos_weight = stop_pos_weight
+        # head0 goes through vLLM's SamplingParams; the other 31 heads use this
+        # in-model. There is no top_p: Gumbel-max has no nucleus step.
+        self.temperature = temperature
+
+        # Carried for the cloning follow-up; zero-shot uses null_prefix only.
+        vc = voice_cloning or {}
+        self.voice_cloning_enabled = vc.get("enabled", True)
+        comp = vc.get("compressor", {}) or {}
+        self.num_speaker_prefix = comp.get("num_queries", 8)
+        self.ref_compressor_num_blocks = comp.get("num_layers", 2)
+        self.ref_compressor_num_heads = comp.get("num_heads", 8)
+        self.ref_compressor_d_model = comp.get("d_model", 1024)
+        self.ref_compressor_ffn_mult = comp.get("ffn_hidden_size_multiplier", 4)
+
+        # Last on purpose — triggers transformers-5.x validators that call
+        # get_text_config(); every attribute they touch must already exist.
+        super().__init__(**kwargs)
+
+    @classmethod
+    def from_checkpoint(
+        cls,
+        model: str,
+        backbone_config: dict | None = None,
+        revision: str | None = None,
+    ) -> GepardConfig:
+        """Build the full config for a checkpoint that self-identifies as the
+        bare backbone: audio fields from the sidecar, backbone fields from the
+        loaded config.
+
+        ``revision`` must be the one the weights came from — a revision that
+        moves the audio-head cardinalities or the special tokens moves the
+        prompt layout and the STOP sentinel with them.
+        """
+        sidecar: dict = {}
+        try:
+            from vllm.transformers_utils.config import get_hf_file_to_dict
+
+            sidecar = dict(get_hf_file_to_dict("gepard_config.json", model, revision=revision) or {})
+        except (OSError, ValueError) as e:
+            # A missing or malformed sidecar is expected; the defaults match the
+            # trained checkpoint. Anything else must surface.
+            logger.warning(
+                "GepardConfig: could not read gepard_config.json from %s (%s: %s); "
+                "audio fields fall back to trained-checkpoint defaults.",
+                model,
+                type(e).__name__,
+                e,
+            )
+        if backbone_config:
+            bb = dict(backbone_config)
+            # The loaded config carries the hf_overrides-patched identity; the
+            # backbone must stay qwen3_5_text or get_text_config() recurses.
+            bb["model_type"] = _BACKBONE_MODEL_TYPE
+            bb.pop("architectures", None)
+            sidecar["backbone_config"] = bb
+        return cls(**sidecar)
+
+    @staticmethod
+    def _parse_audio_heads(audio_heads: dict | None) -> list[int]:
+        """``{level_audio_0: 8, level_audio_1: 7, ...}`` -> ``[8, 7, ...]``.
+
+        Sorted by the numeric suffix, so channel order does not depend on dict
+        insertion order.
+        """
+        if not audio_heads:
+            return [8, 7, 6, 6] * 8
+        return [audio_heads[k] for k in sorted(audio_heads, key=lambda s: int(str(s).rsplit("_", 1)[-1]))]
+
+    @classmethod
+    def _normalize_backbone(cls, backbone_config: dict | None) -> dict:
+        """Return the backbone dict with vestigial mRoPE fields removed.
+
+        They are template leftovers that the trained checkpoint's own
+        ``config.json`` does not carry. Left in, they flip vLLM's
+        ``uses_mrope()`` to True and wire ``positions`` wrongly.
+        """
+        bb = dict(backbone_config or {})
+        rope = bb.get("rope_parameters")
+        if isinstance(rope, dict) and any(k in rope for k in _MROPE_KEYS):
+            rope = {k: v for k, v in rope.items() if k not in _MROPE_KEYS}
+            bb["rope_parameters"] = rope
+            logger.info(
+                "GepardConfig: stripped vestigial mRoPE keys from backbone rope_parameters "
+                "(model uses standard 1D default RoPE)."
+            )
+        return bb
+
+    def get_text_config(self, **kwargs) -> PretrainedConfig:
+        """Return the Qwen3.5 backbone config, which vLLM runs the backbone on."""
+        cached = getattr(self, "_text_config", None)
+        if cached is not None:
+            return cached
+
+        bb = dict(self.backbone_config)
+        # pop, not setdefault: for_model()'s first positional is named
+        # model_type, so leaving the key in **bb passes it twice.
+        model_type = bb.pop("model_type", _BACKBONE_MODEL_TYPE)
+        try:
+            text_config = AutoConfig.for_model(model_type, **bb)
+        except (KeyError, ValueError) as e:
+            # Only an unknown model_type is a legitimate fallback. A TypeError
+            # means the kwargs are wrong and must not degrade silently.
+            text_config = PretrainedConfig(**bb)
+            logger.warning(
+                "GepardConfig: AutoConfig.for_model(%s) failed (%s: %s); using a plain "
+                "PretrainedConfig for the backbone. Verify vLLM still resolves "
+                "the Qwen3.5 model.",
+                model_type,
+                type(e).__name__,
+                e,
+            )
+        self._text_config = text_config
+        return text_config
+
+
+AutoConfig.register("gepard", GepardConfig)
+
+__all__ = ["GepardConfig"]

vllm_omni/model_executor/models/gepard/gepard_talker.py

@@ -0,0 +1,526 @@
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
+"""Gepard-1.0 native-AR talker.
+
+Single-stage AR TTS on a vLLM-native Qwen3.5 backbone. Each step samples one
+32-channel FSQ frame; head0 is the vLLM-facing token and the other 31 are
+side-channel. Generation ends on a learned binary stop head, not an EOS token.
+The NeMo NanoCodec decodes committed frames to a waveform outside vLLM.
+
+Zero-shot only, and ``enforce_eager``; CUDA graph is a perf follow-up.
+"""
+
+from __future__ import annotations
+
+import dataclasses
+import os
+from collections.abc import Iterable
+from typing import Any
+
+import torch
+import torch.nn as nn
+from vllm.config import VllmConfig
+from vllm.logger import init_logger
+from vllm.model_executor.models.qwen3_5 import Qwen3_5ForCausalLM
+from vllm.model_executor.models.utils import AutoWeightsLoader, maybe_prefix
+from vllm.sequence import IntermediateTensors
+
+from vllm_omni.model_executor.models.gepard.configuration_gepard import GepardConfig
+from vllm_omni.model_executor.models.gepard.nanocodec import NanoCodec
+from vllm_omni.model_executor.models.output_templates import OmniOutput
+from vllm_omni.platforms import current_omni_platform
+
+logger = init_logger(__name__)
+
+# Decode cadence: a short first chunk for early audio, then larger ones to
+# amortize the codec call. Does not affect the samples produced.
+FIRST_CHUNK_FRAMES = int(os.environ.get("VLLM_GEPARD_FIRST_CHUNK_FRAMES", "10"))
+CHUNK_FRAMES = int(os.environ.get("VLLM_GEPARD_CHUNK_FRAMES", "86"))
+
+
+@dataclasses.dataclass(frozen=True)
+class _ReqInfo:
+    """Scheduler state the runner threads into ``preprocess``.
+
+    The hook key strings are read only here, so a rename fails in one place.
+    """
+
+    request_id: str
+    is_prefill: bool
+    prompt_len: int
+    num_computed_tokens: int
+    max_tokens: int | None
+    seed: int | None
+
+    @classmethod
+    def from_hook_kwargs(cls, kwargs: dict[str, Any], *, span: int) -> _ReqInfo:
+        # Defaults cover direct calls, which thread no scheduler state.
+        max_tokens = kwargs.get("_omni_max_tokens")
+        seed = kwargs.get("_omni_seed")
+        return cls(
+            request_id=str(kwargs.get("request_id", "default")),
+            is_prefill=bool(kwargs.get("_omni_is_prefill", span > 1)),
+            prompt_len=int(kwargs.get("_omni_prompt_len") or 0),
+            num_computed_tokens=int(kwargs.get("_omni_num_computed_tokens") or 0),
+            max_tokens=None if max_tokens is None else int(max_tokens),
+            seed=None if seed is None else int(seed),
+        )
+
+
+@dataclasses.dataclass
+class _GepardState:
+    """Per-request generation state, keyed by request_id."""
+
+    request_id: str
+    # Embedding of the last frame's 32 codes; the next step's input.
+    curr_embed_for_next: torch.Tensor | None = None
+    # Frame history (T, 32) handed to the codec.
+    frames: list[torch.Tensor] = dataclasses.field(default_factory=list)
+    emitted_frames: int = 0
+    # None falls back to null_prefix; the cloning follow-up sets it.
+    speaker_prefix: torch.Tensor | None = None
+    # None means the request carried no seed and draws from the global RNG.
+    generator: torch.Generator | None = None
+    frame_count: int = 0
+    past_first_step: bool = False
+    is_stopping: bool = False
+
+
+class GepardTalkerForConditionalGeneration(nn.Module):
+    """Gepard native-AR TTS talker (Qwen3.5 backbone + 32 FSQ heads + stop)."""
+
+    have_multimodal_outputs = True
+    has_preprocess = True
+    has_postprocess = False
+
+    def __init__(self, *, vllm_config: VllmConfig, prefix: str = "") -> None:
+        super().__init__()
+        self.vllm_config = vllm_config
+        cfg = vllm_config.model_config.hf_config
+        if not isinstance(cfg, GepardConfig):
+            # hf_overrides patches the loaded config instance rather than
+            # re-classing it, so what arrives here is the backbone config.
+            cfg = GepardConfig.from_checkpoint(
+                vllm_config.model_config.model,
+                backbone_config=cfg.to_dict(),
+                revision=vllm_config.model_config.revision,
+            )
+        self.config = cfg
+        self._device = current_omni_platform.get_torch_device()
+        # Deterministic argmax sampling, for reproducible comparisons.
+        self._greedy = os.environ.get("VLLM_GEPARD_GREEDY", "0") == "1"
+
+        self.vocab_sizes: list[int] = list(cfg.audio_head_levels)  # [8,7,6,6]*8
+        self.num_heads: int = cfg.num_audio_heads
+        self.head0_vocab: int = cfg.head0_vocab_size
+        self.stop_token: int = cfg.stop_token  # sentinel, one past head0's range
+        self.total_vocab: int = sum(self.vocab_sizes)
+        self.stop_threshold: float = cfg.stop_threshold
+        self.temperature: float = cfg.temperature
+        hidden = self.config.get_text_config().hidden_size
+
+        self.fused_codebook_head = nn.Linear(hidden, self.total_vocab, bias=True)
+        self.stop_head = nn.Linear(hidden, 1, bias=True)
+        self.audio_embeddings = nn.ModuleList([nn.Embedding(v, cfg.audio_embed_dim) for v in self.vocab_sizes])
+        self.audio_embed_proj = nn.Sequential(
+            nn.Linear(self.num_heads * cfg.audio_embed_dim, hidden),
+            nn.GELU(),
+            nn.Linear(hidden, hidden),
+            nn.LayerNorm(hidden, elementwise_affine=False),
+        )
+        self.register_buffer("audio_embed_scale", torch.tensor(1.0))
+        # Learned default-voice fallback.
+        self.null_prefix = nn.Parameter(torch.zeros(cfg.num_speaker_prefix, hidden))
+
+        gather_idx, mask = self._build_gather_mask(self.vocab_sizes)
+        self.register_buffer("_cb_gather_idx", gather_idx, persistent=False)
+        self.register_buffer("_cb_mask", mask, persistent=False)
+
+        self.model = Qwen3_5ForCausalLM(vllm_config=vllm_config, prefix=maybe_prefix(prefix, "model"))
+        # Cheap and NeMo-free to construct; load() does the heavy import.
+        self._codec = NanoCodec(codec_id=cfg.codec_id, sample_rate=cfg.codec_sample_rate)
+
+        self._max_model_len: int = int(vllm_config.model_config.max_model_len or 0)
+
+        self._active_states: dict[str, _GepardState] = {}
+        # These queues stay index-aligned across preprocess -> forward ->
+        # compute_logits: preprocess appends in input_batch.req_ids order.
+        # (req_id, span, samples_frame, is_last_token)
+        self._pending_requests: list[tuple[str, int, bool, bool]] = []
+        # (req_id, head0, do_stop); head0 < 0 marks a discarded placeholder row.
+        self._results_queue: list[tuple[str, int, bool]] = []
+        self._audio_queue: list[tuple[str, torch.Tensor | None]] = []
+        self._deferred_cleanup_ids: set[str] = set()
+        self._sample_rate: int = cfg.codec_sample_rate
+
+    @staticmethod
+    def _build_gather_mask(vocab_sizes: list[int]) -> tuple[torch.Tensor, torch.Tensor]:
+        """Gather indices + -inf pad mask so one argmax samples all 31 heads."""
+        num_heads = len(vocab_sizes)
+        max_cb_vocab = max(vocab_sizes[1:])  # == 8 for Gepard; do NOT hardcode 7
+        gather_rows, mask_rows = [], []
+        cum = vocab_sizes[0]
+        for i in range(1, num_heads):
+            v = vocab_sizes[i]
+            gather_rows.append(list(range(cum, cum + v)) + [cum] * (max_cb_vocab - v))
+            mask_rows.append([0.0] * v + [float("-inf")] * (max_cb_vocab - v))
+            cum += v
+        return (
+            torch.tensor(gather_rows, dtype=torch.long),
+            torch.tensor(mask_rows, dtype=torch.float32),
+        )
+
+    def load_weights(self, weights: Iterable[tuple[str, torch.Tensor]]) -> set[str]:
+        """Fuse codebook_heads.{0..31} into fused_codebook_head; delegate the rest."""
+        cb_w: dict[str, torch.Tensor] = {}
+        rest: list[tuple[str, torch.Tensor]] = []
+        for name, tensor in weights:
+            if name.startswith(("aligner.", "supcon_head.")):
+                continue  # training-only
+            if name.startswith("codebook_heads."):
+                cb_w[name] = tensor
+                continue
+            if name.startswith(("model.", "lm_head.")):
+                # We nest the backbone one level deeper than the checkpoint
+                # keys assume, so prepend the extra hop.
+                name = "model." + name
+            rest.append((name, tensor))
+
+        w_parts, b_parts = [], []
+        for i in range(self.num_heads):
+            if (wk := f"codebook_heads.{i}.weight") in cb_w:
+                w_parts.append(cb_w[wk])
+            if (bk := f"codebook_heads.{i}.bias") in cb_w:
+                b_parts.append(cb_w[bk])
+        if w_parts:
+            rest.append(("fused_codebook_head.weight", torch.cat(w_parts, dim=0)))
+        if b_parts:
+            rest.append(("fused_codebook_head.bias", torch.cat(b_parts, dim=0)))
+
+        # NanoCodec loads lazily on first decode, so load_format=dummy and
+        # NeMo-less environments never pay for it.
+        loader = AutoWeightsLoader(self, skip_prefixes=["mtp.", "ref_compressor."])
+        return loader.load_weights(iter(rest))
+
+    def _get_or_create_state(self, request_id: str) -> _GepardState:
+        st = self._active_states.get(request_id)
+        if st is None:
+            st = _GepardState(request_id=request_id)
+            self._active_states[request_id] = st
+        return st
+
+    def on_requests_finished(self, finished_req_ids: set[str] | list[str]) -> None:
+        # Fires BEFORE forward -> defer the actual free until after forward runs,
+        # else we'd drop state the in-flight forward still reads.
+        self._deferred_cleanup_ids.update(str(r) for r in finished_req_ids)
+
+    def _flush_deferred_cleanup(self) -> None:
+        # Cleanup only. Audio queued here is never delivered — the id has left
+        # the output batch by now — so the tail is emitted in forward instead,
+        # and anything still pending here is genuinely lost. Say so.
+        for req_id in self._deferred_cleanup_ids:
+            state = self._active_states.pop(req_id, None)
+            if state is not None and state.frame_count > state.emitted_frames:
+                logger.warning(
+                    "Gepard request %s ended with %d undelivered frame(s): it finished "
+                    "without the stop head firing and without reaching its token budget "
+                    "(aborted or preempted). Its audio is truncated.",
+                    req_id,
+                    state.frame_count - state.emitted_frames,
+                )
+        self._deferred_cleanup_ids.clear()
+
+    def _is_last_output_token(self, info: _ReqInfo, *, span: int, samples_frame: bool) -> bool:
+        """Whether the token sampled this step is the request's budget-final one.
+
+        That step is the last one on which the request is still in the output
+        batch, so it is also the last chance to ship its audio. Counted off the
+        engine's cursor rather than a local tally, so a re-prefill after
+        preemption cannot desync it.
+        """
+        if not samples_frame:
+            # A partial prefill chunk's token is discarded, so it costs no budget.
+            return False
+        budget = info.max_tokens
+        if self._max_model_len:
+            len_cap = self._max_model_len - info.prompt_len
+            budget = len_cap if budget is None else min(budget, len_cap)
+        if budget is None or budget <= 0:
+            return False
+        produced = info.num_computed_tokens + span - info.prompt_len + 1
+        return produced >= budget
+
+    def _emit_audio(self, state: _GepardState, *, is_final: bool) -> None:
+        """Decode this request's un-emitted frames and queue the new samples.
+
+        Queues deltas: each call contributes only samples not queued before.
+        """
+        pending = state.frame_count - state.emitted_frames
+        if pending <= 0:
+            return
+        if not is_final:
+            threshold = FIRST_CHUNK_FRAMES if state.emitted_frames == 0 else CHUNK_FRAMES
+            if pending < threshold:
+                return
+
+        frames = torch.stack(state.frames)  # (T, 32)
+        if not self._codec.is_loaded:
+            # The first decode pays the codec load; warmup is a perf follow-up.
+            self._codec.load(frames.device)
+        audio = self._codec.decode_stream(frames, start_idx=state.emitted_frames, is_final=is_final)
+        if audio is None:
+            return
+        self._audio_queue.append((state.request_id, audio))
+        state.emitted_frames = state.frame_count
+
+    def embed_input_ids(self, input_ids: torch.Tensor, **_: Any) -> torch.Tensor:
+        return self.model.embed_input_ids(input_ids)
+
+    @staticmethod
+    def _gumbel_noise(ref: torch.Tensor, generators: list[torch.Generator | None] | None) -> torch.Tensor:
+        """Gumbel noise shaped like ``ref``; row i uses ``generators[i]`` if set.
+
+        A seeded row draws its own noise of a fixed shape, so the noise it sees
+        does not depend on who shared its batch. Only the noise: the logits come
+        from a bf16 matmul that is not row-wise reproducible, so a batched
+        request can still sample differently from a solo one.
+        """
+        if not generators or all(g is None for g in generators):
+            u = torch.rand_like(ref)
+        else:
+            u = torch.stack(
+                [
+                    torch.rand(ref.shape[1:], generator=g, device=ref.device, dtype=ref.dtype)
+                    if g is not None
+                    else torch.rand(ref.shape[1:], device=ref.device, dtype=ref.dtype)
+                    for g in generators
+                ]
+            )
+        return -torch.log(-torch.log(u.clamp_min_(1e-20)))
+
+    def _sample_frame(
+        self,
+        hidden: torch.Tensor,
+        generators: list[torch.Generator | None] | None = None,
+    ) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
+        """(N, hidden) -> head0 (N,), heads_1_31 (N, 31), do_stop (N,).
+
+        ``generators`` is row-aligned with ``hidden``. The 32 heads are sampled
+        here rather than by vLLM's sampler, so a request's seed only reaches
+        the audio through this.
+        """
+        fused = self.fused_codebook_head(hidden)
+
+        gathered = fused[:, self._cb_gather_idx].float() + self._cb_mask  # (N,31,8)
+        h0_logits = fused[:, : self.head0_vocab].float()
+        if self._greedy:
+            heads = gathered.argmax(dim=-1)
+            head0 = h0_logits.argmax(dim=-1)
+        else:
+            heads = (gathered / self.temperature + self._gumbel_noise(gathered, generators)).argmax(dim=-1)
+            head0 = (h0_logits / self.temperature + self._gumbel_noise(h0_logits, generators)).argmax(dim=-1)
+
+        p_stop = torch.sigmoid(self.stop_head(hidden)).squeeze(-1)
+        do_stop = p_stop > self.stop_threshold
+        return head0, heads, do_stop
+
+    def _audio_frame_embed(self, head0: torch.Tensor, heads_1_31: torch.Tensor) -> torch.Tensor:
+        """Rebuild the next step's input embedding from the 32 sampled codes."""
+        codes = torch.cat([head0.clamp(max=self.head0_vocab - 1).unsqueeze(-1), heads_1_31], dim=-1)
+        parts = [self.audio_embeddings[i](codes[:, i]) for i in range(self.num_heads)]
+        return self.audio_embed_proj(torch.cat(parts, dim=-1)) * self.audio_embed_scale
+
+    def preprocess(self, input_ids: torch.Tensor, input_embeds: torch.Tensor | None = None, **req_infos: Any):
+        # Prefill injects the speaker prefix at the placeholder slots. The
+        # prompt ends on SOS, so forward samples the first frame there.
+        dev = input_ids.device
+        span = int(input_ids.shape[0])
+        info = _ReqInfo.from_hook_kwargs(req_infos, span=span)
+        req_id = info.request_id
+        is_prefill = info.is_prefill
+        prompt_len = info.prompt_len
+        computed = info.num_computed_tokens
+        # A prefill chunk that does not finish the prompt commits no frame:
+        # vLLM discards its sampled token.
+        samples_frame = (not is_prefill) or prompt_len <= 0 or computed + span >= prompt_len
+
+        if is_prefill:
+            state = self._get_or_create_state(req_id)
+            # An aborted id can be resubmitted, and the runner treats the two as
+            # distinct requests. Drop the pending free, or the cleanup at the
+            # end of this forward takes the new request's state with it.
+            resubmitted = req_id in self._deferred_cleanup_ids
+            self._deferred_cleanup_ids.discard(req_id)
+            if resubmitted:
+                # Stand in for the cleanup just cancelled.
+                undelivered = state.frame_count - state.emitted_frames
+                if undelivered > 0:
+                    logger.warning(
+                        "Gepard request %s ended with %d undelivered frame(s) and its id has been "
+                        "resubmitted; the first request's audio is truncated.",
+                        req_id,
+                        undelivered,
+                    )
+            elif state.frame_count:
+                # Not a resubmit, so this is a preemption recompute, and it
+                # cannot be recovered: a frame's 31 side-channel codes are not
+                # in the token stream, so the replayed prompt cannot rebuild the
+                # KV that produced the audio already delivered. Generation
+                # restarts from frame 0 and the caller gets two unrelated halves.
+                logger.warning(
+                    "Gepard request %s was preempted after %d frame(s) and is being recomputed; "
+                    "its audio will not continue from where it stopped. Raise the stage's "
+                    "gpu_memory_utilization or lower max_num_seqs to keep requests from being "
+                    "preempted.",
+                    req_id,
+                    state.frame_count,
+                )
+            state.curr_embed_for_next = None
+            state.frames.clear()
+            state.emitted_frames = 0
+            state.frame_count = 0
+            state.past_first_step = False
+            state.is_stopping = False
+            # SamplingParams.seed only reaches vLLM's sampler, so it never
+            # touches the audio unless it is threaded into a generator here.
+            state.generator = None
+            if info.seed is not None:
+                state.generator = torch.Generator(device=dev)
+                state.generator.manual_seed(info.seed)
+
+            embeds = self.model.embed_input_ids(input_ids)
+            prefix = state.speaker_prefix if state.speaker_prefix is not None else self.null_prefix
+            base = self.config.speaker_token_base
+            spk = (input_ids >= base) & (input_ids < base + self.config.num_speaker_prefix)
+            if bool(spk.any()):
+                embeds[spk] = prefix[input_ids[spk] - base].to(dtype=embeds.dtype)
+        else:
+            state = self._active_states.get(req_id)
+            curr = state.curr_embed_for_next if state is not None else None
+            if curr is not None:
+                embeds = curr.to(device=dev).reshape(1, -1)
+            else:
+                # Defensive only: a decode step always follows a forward that
+                # stashed curr_embed_for_next.
+                hidden = self.config.get_text_config().hidden_size
+                embeds = torch.zeros(1, hidden, device=dev, dtype=self.null_prefix.dtype)
+
+        is_last_token = self._is_last_output_token(info, span=span, samples_frame=samples_frame)
+        self._pending_requests.append((req_id, span, samples_frame, is_last_token))
+        return input_ids, embeds, {}
+
+    def forward(
+        self,
+        input_ids: torch.Tensor,
+        positions: torch.Tensor,
+        intermediate_tensors: IntermediateTensors | None = None,
+        inputs_embeds: torch.Tensor | None = None,
+        **kwargs: Any,
+    ) -> torch.Tensor | OmniOutput:
+        # positions is plain 1D: Gepard uses default RoPE, not mRoPE. input_ids
+        # passes through only when no embeds were spliced (the profile run).
+        hidden = self.model(
+            input_ids=input_ids if inputs_embeds is None else None,
+            positions=positions,
+            intermediate_tensors=intermediate_tensors,
+            inputs_embeds=inputs_embeds,
+        )
+        # Empty _pending_requests means the engine-init profile run: skip.
+        if self._pending_requests:
+            # One sampling row per request: the last scheduled position of its
+            # span. Same row set compute_logits sees at logits_indices.
+            ends, generators, offset = [], [], 0
+            for req_id, span, _samples_frame, _is_last in self._pending_requests:
+                offset += span
+                ends.append(offset - 1)
+                state = self._active_states.get(req_id)
+                generators.append(state.generator if state is not None else None)
+            rows = hidden[torch.tensor(ends, device=hidden.device)]
+            head0, heads_1_31, want_stop = self._sample_frame(rows, generators)
+            frame_embeds = self._audio_frame_embed(head0, heads_1_31)
+            head0_l = head0.tolist()
+            stop_l = want_stop.tolist()
+            for i, (req_id, _span, samples_frame, is_last_token) in enumerate(self._pending_requests):
+                if not samples_frame:
+                    # Partial prefill chunk: vLLM discards this row's token.
+                    self._results_queue.append((req_id, -1, False))
+                    continue
+                state = self._get_or_create_state(req_id)
+                if state.is_stopping:
+                    # Async scheduling runs one more step after a stop token —
+                    # only the max_tokens finish is guarded against it. This
+                    # step's output is discarded, so a frame committed here
+                    # would be stranded and reported as lost audio.
+                    self._results_queue.append((req_id, int(head0_l[i]), True))
+                    continue
+                # Stop may only fire past the first audio frame, so the
+                # prefill-sampled frame and the one after it always commit.
+                do_stop = bool(stop_l[i]) and state.past_first_step
+                if do_stop:
+                    state.is_stopping = True
+                else:
+                    state.frames.append(torch.cat([head0[i : i + 1], heads_1_31[i]]))
+                    state.frame_count += 1
+                    state.curr_embed_for_next = frame_embeds[i]
+                    state.past_first_step = True
+                self._results_queue.append((req_id, int(head0_l[i]), do_stop))
+                # Both ways a request ends have to flush here: this is the last
+                # step on which it is still in the output batch, and a payload
+                # queued after it is dropped by routing.
+                self._emit_audio(state, is_final=do_stop or is_last_token)
+            self._pending_requests.clear()
+        self._flush_deferred_cleanup()
+        return hidden
+
+    def compute_logits(self, hidden_states: torch.Tensor | OmniOutput, sampling_metadata: Any = None):
+        # Vocab-wide logits carrying head0, or STOP for a stopped row. Requires
+        # the pipeline's sampling_constraints, without which vLLM never acts on
+        # the sentinel.
+        if isinstance(hidden_states, OmniOutput):
+            hidden_states = hidden_states.text_hidden_states
+        if hidden_states is None:
+            return None
+        bsz = hidden_states.shape[0]
+        vocab = self.config.get_text_config().vocab_size
+        logits = torch.full((bsz, vocab), float("-inf"), device=hidden_states.device, dtype=hidden_states.dtype)
+        if not self._results_queue:
+            # Engine-init dummy sampler run: neutral one-hot logits so KV-cache
+            # profiling can complete.
+            logits[:, 0] = 0.0
+            return logits
+        # Row i belongs to _results_queue[i]. The pipeline pins temperature=0.0,
+        # so vLLM's sampler argmaxes the single finite entry.
+        for i, (_req_id, head0, do_stop) in enumerate(self._results_queue):
+            if i >= bsz:
+                break
+            if head0 < 0:
+                logits[i, 0] = 0.0  # partial-prefill placeholder; vLLM discards it
+                continue
+            logits[i, self.stop_token if do_stop else head0] = 0.0
+        self._results_queue.clear()
+        return logits
+
+    def make_omni_output(self, model_outputs: torch.Tensor | OmniOutput, **kwargs: Any) -> OmniOutput:
+        if isinstance(model_outputs, OmniOutput):
+            return model_outputs
+        by_req: dict[str, torch.Tensor] = {}
+        for req_id, audio in self._audio_queue:
+            if audio is None:
+                continue
+            by_req[req_id] = (
+                torch.cat([by_req[req_id].reshape(-1), audio.reshape(-1)], dim=0) if req_id in by_req else audio
+            )
+        self._audio_queue.clear()
+
+        # The sparse marker is always carried, even on a step that decoded
+        # nothing: without it the runner ships the pooler payload instead, whose
+        # hidden states land under this stage's "audio" key.
+        ready_req_ids = list(by_req)
+        sr = torch.tensor(self._sample_rate, dtype=torch.int32)
+        mm: dict[str, Any] = {
+            "model_outputs": [by_req[r].reshape(-1) for r in ready_req_ids],
+            "sr": [sr for _ in ready_req_ids],
+            "meta": {"req_id": ready_req_ids, "sparse_audio": ["1"]},
+        }
+        # Deltas, not the whole history: callers concatenate across steps.
+        return OmniOutput(text_hidden_states=model_outputs, multimodal_outputs=mm)

vllm_omni/model_executor/models/gepard/nanocodec.py

@@ -0,0 +1,172 @@
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
+"""NeMo NanoCodec wrapper for Gepard — per-dimension FSQ decode.
+
+Gepard's 32 FSQ heads emit per-dimension codes (8 groups x 4 dims), while stock
+NeMo decode wants composed per-group mixed-radix indices. ``decode_from_codes``
+dequantizes per-dim instead and hands the result to stock ``decode_audio`` — a
+few lines on public NeMo API, no fork.
+
+NeMo is a heavy optional dependency, imported lazily inside ``load``. Decode
+only; ``encode_to_dequantized`` belongs to the voice-cloning follow-up.
+"""
+
+from __future__ import annotations
+
+import os
+
+import torch
+import torch.nn as nn
+from vllm.logger import init_logger
+
+logger = init_logger(__name__)
+
+# Left-context frames prepended to every streaming decode window. NanoCodec is
+# convolutional, so decoding a bare chunk boundary yields a discontinuity; the
+# extra frames are decoded for receptive-field context and trimmed afterwards.
+# Mirrors the reference server's TTS_LOOKBACK_FRAMES.
+LOOKBACK_FRAMES = int(os.environ.get("VLLM_GEPARD_LOOKBACK_FRAMES", "8"))
+
+# End-of-clip shaping. The codec leaves the waveform on a non-zero sample and
+# the hard step to silence is an audible click; a linear fade ramps it out.
+# Both default to 0.0 (off), matching the reference server's defaults.
+END_OF_SPEECH_FADE_MS = float(os.environ.get("VLLM_GEPARD_END_FADE_MS", "0.0"))
+END_OF_SPEECH_SILENCE_MS = float(os.environ.get("VLLM_GEPARD_END_SILENCE_MS", "0.0"))
+
+
+def apply_end_of_speech_tail(
+    audio: torch.Tensor,
+    sample_rate: int,
+    fade_ms: float = END_OF_SPEECH_FADE_MS,
+    silence_ms: float = END_OF_SPEECH_SILENCE_MS,
+) -> torch.Tensor:
+    """Fade the final ``fade_ms`` to zero, then append ``silence_ms`` of pad.
+
+    The fade is what removes the end-of-clip click; padding alone does not.
+    """
+    if audio.numel() == 0 or (fade_ms <= 0 and silence_ms <= 0):
+        return audio
+    out = audio
+    if fade_ms > 0:
+        n = min(out.shape[0], int(sample_rate * fade_ms / 1000.0))
+        if n > 0:
+            # Copy: the caller's tensor is still referenced by the audio queue.
+            out = out.clone()
+            out[-n:] *= torch.linspace(1.0, 0.0, n, dtype=out.dtype, device=out.device)
+    if silence_ms > 0:
+        pad = torch.zeros(int(sample_rate * silence_ms / 1000.0), dtype=out.dtype, device=out.device)
+        out = torch.cat([out, pad], dim=0)
+    return out
+
+
+def _make_unfolded_codec_cls():
+    """Build the ``UnfoldedCodecModel`` subclass. Imports NeMo (optional dep).
+
+    Defined lazily so importing this module never requires NeMo.  The subclass
+    is OUR code layered on stock ``AudioCodecModel`` — not a NeMo fork.
+    """
+    try:
+        from nemo.collections.tts.models import AudioCodecModel  # noqa: PLC0415
+        from omegaconf import open_dict  # noqa: PLC0415
+    except ImportError as e:  # pragma: no cover - env-dependent
+        raise ImportError(
+            "Gepard TTS needs NeMo for the NanoCodec decoder. Install it with "
+            "`pip install nemo_toolkit[tts]` (heavy; NVIDIA Open Model License, "
+            "commercial use OK), or gate the model behind that extra."
+        ) from e
+
+    class UnfoldedCodecModel(AudioCodecModel):
+        """AudioCodecModel + direct decode from per-dimension FSQ codes."""
+
+        def __init__(self, cfg, trainer=None):
+            # SLMDiscriminator pulls microsoft/wavlm-base-plus (~360 MB) and is
+            # training-only — strip it from the config before init.
+            with open_dict(cfg):
+                disc = cfg.get("discriminator", None)
+                if disc is not None and "discriminators" in disc:
+                    disc.discriminators = [d for d in disc.discriminators if "SLM" not in d._target_]
+            super().__init__(cfg, trainer)
+
+        def decode_from_codes(self, codes: torch.Tensor, codes_len: torch.Tensor):
+            """Decode per-dimension discrete FSQ codes to a waveform.
+
+            codes: (B, D, T) per-dim discrete values, D = num_groups * dims.
+            codes_len: (B,) valid frame count per batch element.
+            Returns (audio (B, T_audio), audio_len (B,)).
+            """
+            num_levels = self.vector_quantizer.fsqs[0].num_levels.squeeze()
+            scale = (num_levels // 2).float().to(codes.device)  # e.g. [4,3,3,3]
+            groups = codes.chunk(self.vector_quantizer.num_groups, dim=1)
+            dequantized = torch.cat(
+                [(g - scale[None, :, None]) / scale[None, :, None] for g in groups],
+                dim=1,
+            )
+            return self.decode_audio(inputs=dequantized, input_len=codes_len)
+
+    return UnfoldedCodecModel
+
+
+class NanoCodec(nn.Module):
+    """Thin wrapper owning the (optional) NeMo NanoCodec decoder.
+
+    Constructing this is cheap and NeMo-free; ``load`` does the heavy import +
+    weight load so ``load_format=dummy`` and NeMo-less environments don't pay
+    for it at model construction.
+    """
+
+    def __init__(self, codec_id: str, sample_rate: int) -> None:
+        super().__init__()
+        self.codec_id = codec_id
+        self.sample_rate = sample_rate
+        self._codec: nn.Module | None = None
+
+    @property
+    def is_loaded(self) -> bool:
+        return self._codec is not None
+
+    def load(self, device: torch.device) -> None:
+        """Import NeMo, build the UnfoldedCodecModel, move to device, eval()."""
+        cls = _make_unfolded_codec_cls()
+        codec = cls.from_pretrained(self.codec_id)
+        codec.to(device=device)
+        codec.eval()
+        self._codec = codec
+        logger.info("Gepard NanoCodec loaded from %s on %s", self.codec_id, device)
+
+    @torch.inference_mode()
+    def decode_from_codes(self, codes: torch.Tensor, codes_len: torch.Tensor) -> torch.Tensor:
+        """(B, 32, T) per-dim codes -> waveform (B, T_audio). Returns audio only."""
+        if self._codec is None:
+            raise RuntimeError("NanoCodec.decode_from_codes called before load()")
+        audio, _audio_len = self._codec.decode_from_codes(codes, codes_len)
+        return audio
+
+    @torch.inference_mode()
+    def decode_stream(
+        self,
+        frames: torch.Tensor,
+        *,
+        start_idx: int,
+        lookback: int = LOOKBACK_FRAMES,
+        is_final: bool = False,
+    ) -> torch.Tensor | None:
+        """Decode the tail of a ``(T, 32)`` code history past ``start_idx``.
+
+        The window starts ``lookback`` frames earlier for receptive-field
+        context and those samples are trimmed after, so successive calls
+        concatenate to a single whole-history decode. None if nothing is new.
+        """
+        count = int(frames.shape[0])
+        if count <= start_idx:
+            return None
+        decode_start = max(0, start_idx - lookback)
+        window = frames[decode_start:count]  # (T_win, 32)
+        codes = window.t().unsqueeze(0).contiguous().long()  # (1, 32, T_win)
+        codes_len = torch.full((1,), codes.shape[-1], dtype=torch.long, device=codes.device)
+        audio = self.decode_from_codes(codes, codes_len).reshape(-1)
+        # Derive samples_per_frame from this decode rather than assuming it.
+        samples_per_frame = audio.shape[0] // (count - decode_start)
+        audio = audio[(start_idx - decode_start) * samples_per_frame :]
+        if is_final:
+            audio = apply_end_of_speech_tail(audio, self.sample_rate)
+        return audio

vllm_omni/model_executor/models/gepard/pipeline.py

@@ -0,0 +1,45 @@
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
+"""Gepard-1.0 pipeline topology.
+
+Single-stage AR TTS. The Qwen3.5 backbone runs under paged attention, the 32
+codebook heads and the binary stop head are sampled inside the talker, and the
+NeMo NanoCodec decodes the accumulated frames outside vLLM.
+
+``compute_logits`` scatters head0 into vLLM's vocab-wide logits; head0's valid
+range is 0..7, so the synthetic STOP sentinel is 8. ``stop_token_ids=[8]``
+couples it to vLLM's stop machinery, with ``detokenize=False`` because head0 is
+an audio code. Same idiom as VoxCPM2/MOSS.
+"""
+
+from vllm_omni.config.stage_config import (
+    PipelineConfig,
+    StageExecutionType,
+    StagePipelineConfig,
+)
+
+# head0 valid range is 0..7 (its FSQ cardinality is 8); STOP is the sentinel
+# one past that range. Mirrors the reference config.py STOP_TOKEN = 8.
+_GEPARD_STOP_TOKEN_ID = 8
+
+GEPARD_PIPELINE = PipelineConfig(
+    model_type="gepard",
+    default_deploy_config_name="gepard.yaml",
+    model_arch="GepardTalkerForConditionalGeneration",
+    stages=(
+        StagePipelineConfig(
+            stage_id=0,
+            model_stage="gepard",
+            execution_type=StageExecutionType.LLM_AR,
+            input_sources=(),
+            final_output=True,
+            final_output_type="audio",
+            owns_tokenizer=True,
+            engine_output_type="audio",
+            sampling_constraints={
+                "detokenize": False,
+                "stop_token_ids": [_GEPARD_STOP_TOKEN_ID],
+            },
+        ),
+    ),
+)

vllm_omni/model_executor/models/gepard/prompt.py

@@ -0,0 +1,55 @@
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
+"""Gepard-1.0 prompt layout.
+
+``preprocess`` only consumes this layout — it injects the speaker prefix at the
+placeholder slots and samples the first audio frame from the SOS position. The
+layout itself is built here, by whichever entry point is assembling a request.
+"""
+
+from __future__ import annotations
+
+import math
+from typing import TYPE_CHECKING
+
+if TYPE_CHECKING:
+    from vllm_omni.model_executor.models.gepard.configuration_gepard import GepardConfig
+
+
+def build_gepard_prompt_ids(text_token_ids: list[int], *, config: GepardConfig) -> list[int]:
+    """Assemble ``[speaker_slots] + [SOT, *text, EOT]*(R-1) + [SOT, *text, EOT, SOS]``.
+
+    Only the final copy carries SOS, the learned "audio starts now" gate, so
+    the earlier copies are read as context and never voiced. Short texts repeat
+    because a couple of text tokens carry too little mass against the speaker
+    prefix. This must match the training layout or WER collapses, which is why
+    the thresholds come from the checkpoint's ``text_repetition`` block rather
+    than from literals here.
+
+    Raises ``ValueError`` on empty text: the layout would still be structurally
+    valid and the model would voice it as arbitrary audio, so an empty request
+    has to fail loudly. This is the only entry point in the offline path — there
+    is no adapter above it to check first.
+    """
+    n = len(text_token_ids)
+    if n == 0:
+        raise ValueError("text_token_ids is empty; Gepard needs at least one text token to voice")
+    if (
+        not config.text_repetition_enabled
+        or n >= config.text_repetition_apply_below
+        or n >= config.text_repetition_target_tokens
+    ):
+        repeats = 1
+    else:
+        repeats = max(
+            1,
+            min(
+                math.ceil(config.text_repetition_target_tokens / n),
+                config.text_repetition_max_repeats,
+            ),
+        )
+
+    block = [config.start_of_text, *text_token_ids, config.end_of_text]
+    text_region = block * (repeats - 1) + [*block, config.start_of_speech]
+    speaker_slots = [config.speaker_token_base + i for i in range(config.num_speaker_prefix)]
+    return speaker_slots + text_region

vllm_omni/model_executor/models/registry.py

@@ -295,6 +295,12 @@ _OMNI_MODELS = {
         "fish_speech_dac_decoder",
         "FishSpeechDACDecoder",
     ),
+    ## Gepard-1.0 (nineninesix) — single-stage native-AR FSQ/NanoCodec TTS
+    "GepardTalkerForConditionalGeneration": (
+        "gepard",
+        "gepard_talker",
+        "GepardTalkerForConditionalGeneration",
+    ),
     ## VoxCPM2
     "VoxCPM2TalkerForConditionalGeneration": (
         "voxcpm2",

vllm_omni/worker/gpu_model_runner.py

@@ -1772,6 +1772,16 @@ class OmniGPUModelRunner(GPUModelRunner):
                 req_infos["_omni_prompt_len"] = prompt_len
                 req_infos["_omni_num_computed_tokens"] = num_computed_tokens
                 req_infos["_omni_is_prefill"] = is_prefill
+                # Output-token cap, so a model that must ship a payload on the
+                # request's final step can tell which step that is. A finished
+                # request drops out of req_ids_output_copy, and downstream
+                # routing keys off that list, so a payload produced after its
+                # last step never reaches the caller.
+                sampling_params = getattr(req_state, "sampling_params", None)
+                req_infos["_omni_max_tokens"] = getattr(sampling_params, "max_tokens", None)
+                # Seed, so a model that samples inside forward() can be
+                # reproducible: vLLM's own sampler seeding does not reach it.
+                req_infos["_omni_seed"] = getattr(sampling_params, "seed", None)
                 if callable(batch_decode_preprocess) and self.has_talker_mtp and span_len == 1 and not is_prefill:
                     decode_batch_items.append((req_id, s, req_infos))
                     continue