[TTS][SpeakerCacheManager] A global speaker cache manager for Voice Cloning (#2630)

Signed-off-by: JuanPZuluaga <juanz9312@gmal.com>
Co-authored-by: JuanPZuluaga <juanz9312@gmal.com>

benchmarks/fish-speech/bench_speaker_cache.py

@@ -3,11 +3,11 @@
 Measures TTFP improvement from DAC-code caching when using uploaded voices.
 
 Setup:
-  1. Start vllm-omni with Fish Speech S2 Pro (use our feat branch)
+  1. Start vllm-omni with Fish Speech S2 Pro
   2. Provide a reference audio file for voice cloning
 
 Usage:
-    python bench_voice_cache.py \
+    python bench_speaker_cache.py \
         --ref-audio /path/to/reference.wav \
         --ref-text "Transcript of the reference audio." \
         --num-prompts 20 \

docs/serving/speech_api.md

@@ -358,6 +358,31 @@ curl -X POST http://localhost:8091/v1/audio/speech \
     }' --output cloned.wav
 ```
 
+### Voice Storage & Caching
+
+Uploaded voices are persisted to disk as a single `.safetensors` file per voice
+(audio samples + metadata — name, consent, ref_text, sample_rate, created_at —
+in the file header). On server restart the directory is scanned and all
+previously uploaded voices are restored automatically, so uploads survive
+process restarts.
+
+Uploading an existing name overwrites the previous entry (a warning is logged).
+
+Feature extraction artifacts (ref_code, speaker_embedding, DAC codes, etc.)
+are cached in-process with a shared LRU so repeated requests with the same
+`voice=...` skip the extraction pipeline. The cache is a true singleton across
+all TTS model types; deleting a voice invalidates every model-type slot at
+once.
+
+**Configuration (environment variables):**
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `SPEAKER_SAMPLES_DIR` | `~/.cache/vllm-omni/speakers` | Directory for persisted uploaded speakers (`.safetensors` files). |
+| `SPEAKER_MAX_UPLOADED` | `1000` | Maximum number of uploaded speakers kept on disk. Upload requests past the cap return 400. |
+
+The in-memory LRU has a fixed 512 MiB byte budget.
+
 ## Batch Speech Generation
 
 The batch endpoint synthesizes multiple texts in a single request, returning all results as JSON with base64-encoded audio.
@@ -543,6 +568,30 @@ Fish Speech uses `ref_audio` and `ref_text` for voice cloning (no `task_type` ne
 |-------|-------------|
 | `mistralai/Voxtral-4B-TTS-2603` | 3B AR + FlowMatching TTS. Supports text-to-speech with preset voices. |
 
+### CosyVoice3
+
+| Model | Description |
+|-------|-------------|
+| `FunAudioLLM/Fun-CosyVoice3-0.5B-2512` | Voice cloning from `ref_audio` + `ref_text`. No built-in voice presets — upload a voice or pass `ref_audio`/`ref_text` per request. |
+
+### OmniVoice
+
+| Model | Description |
+|-------|-------------|
+| `k2-fsa/OmniVoice` | Pure-diffusion TTS. Supports voice cloning via `ref_audio` (with optional `ref_text`); no built-in voice presets. |
+
+### VoxCPM2
+
+| Model | Description |
+|-------|-------------|
+| `openbmb/VoxCPM2` | TTS + voice cloning with built-in speaker presets and uploaded-voice support. Accepts `voice` (preset or uploaded) or `ref_audio` + optional `ref_text`. |
+
+### MOSS-TTS-Nano
+
+| Model | Description |
+|-------|-------------|
+| `OpenMOSS-Team/MOSS-TTS-Nano` | Voice cloning only. Requires `ref_audio` (or an uploaded `voice`); no built-in voice presets. `ref_text` is accepted but ignored — upstream's `voice_clone` mode does not consume a transcript. |
+
 ## Error Responses
 
 ### 400 Bad Request

docs/user_guide/examples/offline_inference/voxtral_tts.md

@@ -0,0 +1,68 @@
+# Voxtral TTS Offline Inference
+
+Source <https://github.com/vllm-project/vllm-omni/tree/main/examples/offline_inference/voxtral_tts>.
+
+
+`end2end.py` runs Voxtral TTS end-to-end offline inference using vLLM. It supports both blocking (`Omni`) and streaming (`AsyncOmni`) generation, batched prompts with configurable concurrency, and voice selection via preset name or reference audio file.
+
+When `mistral_common` has `SpeechRequest` support, prompt token IDs are built via `encode_speech_request`. Otherwise, the script falls back to manual token construction.
+
+## Usage Examples
+
+
+```bash
+# Basic single-prompt with cheerful_female voice preset
+python3 examples/offline_inference/voxtral_tts/end2end.py \
+    --write-audio --voice cheerful_female \
+    --model mistralai/Voxtral-4B-TTS-2603 \
+    --text "That eerie silence after the first storm was just the calm before another round of chaos, wasn't it?"
+
+# 32 replicate prompts with cheerful_female voice preset
+python3 examples/offline_inference/voxtral_tts/end2end.py \
+    --num-prompts 32 --write-audio --voice cheerful_female \
+    --model mistralai/Voxtral-4B-TTS-2603 \
+    --text "That eerie silence after the first storm was just the calm before another round of chaos, wasn't it?"
+
+# Streaming with neutral_female voice preset
+python3 examples/offline_inference/voxtral_tts/end2end.py \
+    --streaming --write-audio --voice neutral_female \
+    --model mistralai/Voxtral-4B-TTS-2603 \
+    --text "That eerie silence after the first storm was just the calm before another round of chaos, wasn't it?"
+
+# 32 prompts, 8 concurrent requests per wave, streaming with neutral_female voice
+python3 examples/offline_inference/voxtral_tts/end2end.py \
+    --num-prompts 32 --concurrency 8 --streaming --write-audio --voice neutral_female \
+    --model mistralai/Voxtral-4B-TTS-2603 \
+    --text "That eerie silence after the first storm was just the calm before another round of chaos, wasn't it?"
+
+# Short debug prompt with reference audio
+python3 examples/offline_inference/voxtral_tts/end2end.py \
+    --write-audio \
+    --model mistralai/Voxtral-4B-TTS-2603 \
+    --text "This is a test message." \
+    --audio-path path/to/reference_audio.wav
+```
+
+## Arguments
+
+| Argument | Description |
+|---|---|
+| `--model PATH` | HuggingFace repo ID or local directory path (default: `mistralai/Voxtral-4B-TTS-2603`) |
+| `--text TEXT` | Text to synthesize (default: `"This is a test message."`) |
+| `--audio-path PATH` | Path to reference audio file for voice cloning |
+| `--output-dir DIR` | Directory to write output WAV files (default: `output_audio`) |
+| `--deploy-config PATH` | Override the deploy config path. If unset, auto-loads `vllm_omni/deploy/voxtral_tts.yaml` from the HF `model_type`. |
+| `--num-prompts N` | Number of replicate prompts to run for measuring performance (default: 1) |
+| `--streaming` | Use streaming generation via `AsyncOmni` (default: blocking `Omni`) |
+| `--concurrency N` | Max concurrent requests per wave (must be used with `--streaming`, must evenly divide `--num-prompts`) |
+| `--voice NAME` | Voice preset to use instead of reference audio file (e.g., casual_female, casual_male, cheerful_female, neutral_female, neutral_male) |
+| `--write-audio` | Write generated audio to WAV files |
+| `--profiling-mode` | Enable profiling mode (reduces max tokens to 50) |
+| `--log-stats` | Enable detailed statistics logging |
+
+## Example materials
+
+??? abstract "end2end.py"
+    ``````py
+    --8<-- "examples/offline_inference/voxtral_tts/end2end.py"
+    ``````

examples/offline_inference/voxtral_tts/README.md

@@ -0,0 +1,58 @@
+# Voxtral TTS Offline Inference
+
+`end2end.py` runs Voxtral TTS end-to-end offline inference using vLLM. It supports both blocking (`Omni`) and streaming (`AsyncOmni`) generation, batched prompts with configurable concurrency, and voice selection via preset name or reference audio file.
+
+When `mistral_common` has `SpeechRequest` support, prompt token IDs are built via `encode_speech_request`. Otherwise, the script falls back to manual token construction.
+
+## Usage Examples
+
+
+```bash
+# Basic single-prompt with cheerful_female voice preset
+python3 examples/offline_inference/voxtral_tts/end2end.py \
+    --write-audio --voice cheerful_female \
+    --model mistralai/Voxtral-4B-TTS-2603 \
+    --text "That eerie silence after the first storm was just the calm before another round of chaos, wasn't it?"
+
+# 32 replicate prompts with cheerful_female voice preset
+python3 examples/offline_inference/voxtral_tts/end2end.py \
+    --num-prompts 32 --write-audio --voice cheerful_female \
+    --model mistralai/Voxtral-4B-TTS-2603 \
+    --text "That eerie silence after the first storm was just the calm before another round of chaos, wasn't it?"
+
+# Streaming with neutral_female voice preset
+python3 examples/offline_inference/voxtral_tts/end2end.py \
+    --streaming --write-audio --voice neutral_female \
+    --model mistralai/Voxtral-4B-TTS-2603 \
+    --text "That eerie silence after the first storm was just the calm before another round of chaos, wasn't it?"
+
+# 32 prompts, 8 concurrent requests per wave, streaming with neutral_female voice
+python3 examples/offline_inference/voxtral_tts/end2end.py \
+    --num-prompts 32 --concurrency 8 --streaming --write-audio --voice neutral_female \
+    --model mistralai/Voxtral-4B-TTS-2603 \
+    --text "That eerie silence after the first storm was just the calm before another round of chaos, wasn't it?"
+
+# Short debug prompt with reference audio
+python3 examples/offline_inference/voxtral_tts/end2end.py \
+    --write-audio \
+    --model mistralai/Voxtral-4B-TTS-2603 \
+    --text "This is a test message." \
+    --audio-path path/to/reference_audio.wav
+```
+
+## Arguments
+
+| Argument | Description |
+|---|---|
+| `--model PATH` | HuggingFace repo ID or local directory path (default: `mistralai/Voxtral-4B-TTS-2603`) |
+| `--text TEXT` | Text to synthesize (default: `"This is a test message."`) |
+| `--audio-path PATH` | Path to reference audio file for voice cloning |
+| `--output-dir DIR` | Directory to write output WAV files (default: `output_audio`) |
+| `--deploy-config PATH` | Override the deploy config path. If unset, auto-loads `vllm_omni/deploy/voxtral_tts.yaml` from the HF `model_type`. |
+| `--num-prompts N` | Number of replicate prompts to run for measuring performance (default: 1) |
+| `--streaming` | Use streaming generation via `AsyncOmni` (default: blocking `Omni`) |
+| `--concurrency N` | Max concurrent requests per wave (must be used with `--streaming`, must evenly divide `--num-prompts`) |
+| `--voice NAME` | Voice preset to use instead of reference audio file. Check Huggingface `mistralai/Voxtral-4B-TTS-2603` to get the list of available voices |
+| `--write-audio` | Write generated audio to WAV files |
+| `--profiling-mode` | Enable profiling mode (reduces max tokens to 50) |
+| `--log-stats` | Enable detailed statistics logging |

tests/conftest.py

@@ -13,6 +13,7 @@ pytest_plugins = (
     "tests.helpers.fixtures.log",
     "tests.helpers.fixtures.run_args",
     "tests.helpers.fixtures.runtime",
+    "tests.helpers.fixtures.speaker_cache",
 )
 
 

tests/entrypoints/openai_api/test_serving_speech.py

@@ -300,6 +300,24 @@ def client(test_app):
 
 
 class TestSpeechAPI:
+    @pytest.fixture(autouse=True)
+    def _mock_upload_io(self, mocker: MockerFixture):
+        """Mock soundfile/safetensors so upload accepts fake audio bytes."""
+        samples = np.zeros(88200, dtype=np.float32)  # 2s @ 44.1 kHz
+        mocker.patch("soundfile.read", return_value=(samples, 44100))
+
+        def _fake_save_file(tensors, path, metadata=None):
+            Path(path).touch()
+
+        mocker.patch("safetensors.torch.save_file", side_effect=_fake_save_file)
+        mock_ctx = mocker.MagicMock()
+        mock_ctx.keys.return_value = ["audio"]
+        mock_ctx.get_tensor.return_value = torch.zeros(88200)
+        mock_ctx.metadata.return_value = {"sample_rate": "44100"}
+        mock_safe_open = mocker.MagicMock()
+        mock_safe_open.return_value.__enter__.return_value = mock_ctx
+        mocker.patch("safetensors.safe_open", mock_safe_open)
+
     def test_create_speech_success(self, client):
         payload = {
             "input": "Hello world",
@@ -470,27 +488,17 @@ class TestSpeechAPI:
         assert "MIME type" in result["detail"]
 
     def test_upload_voice_name_collision(self, client):
-        """Test voice upload with duplicate name."""
-        # First upload
+        """Re-uploading the same name overwrites the previous entry (no 400)."""
         audio_content = b"fake audio content"
-        files = {
-            "audio_sample": ("test.wav", audio_content, "audio/wav"),
-        }
-        data = {
-            "consent": "user_consent_123",
-            "name": "test_voice",
-        }
+        files = {"audio_sample": ("test.wav", audio_content, "audio/wav")}
+        data = {"consent": "user_consent_123", "name": "test_voice"}
 
         response = client.post("/v1/audio/voices", files=files, data=data)
         assert response.status_code == 200
 
-        # Second upload with same name
         response = client.post("/v1/audio/voices", files=files, data=data)
-        assert response.status_code == 400
-        result = response.json()
-        assert "detail" in result
-        assert "already exists" in result["detail"]
-        response = client.delete("/v1/audio/voices/test_voice")
+        assert response.status_code == 200
+        client.delete("/v1/audio/voices/test_voice")
 
     def test_upload_voice_missing_parameters(self, client):
         """Test voice upload with missing required parameters."""
@@ -970,7 +978,7 @@ class TestTTSMethods:
                 "file_path": "/tmp/voice_samples/custom_voice_consent_123.wav",
                 "mime_type": "audio/wav",
                 "ref_text": None,
-                "created_at": 1711234567.89,
+                "created_at": 1711234567,
             }
         }
         speech_server.supported_speakers = {"ryan", "vivian", "custom_voice"}
@@ -983,7 +991,7 @@ class TestTTSMethods:
         assert params["ref_audio"] == ["data:audio/wav;base64,ZmFrZWF1ZGlv"]
         assert params["x_vector_only_mode"] == [True]
         assert params["task_type"] == ["Base"]
-        assert params["voice_created_at"] == [1711234567.89]
+        assert params["voice_created_at"] == [1711234567]
         assert "ref_text" not in params
 
     def test_build_tts_params_with_uploaded_voice_ref_text(self, speech_server, mocker: MockerFixture):
@@ -994,7 +1002,7 @@ class TestTTSMethods:
                 "file_path": "/tmp/voice_samples/custom_voice_consent_123.wav",
                 "mime_type": "audio/wav",
                 "ref_text": "Hello world transcript",
-                "created_at": 1711234567.89,
+                "created_at": 1711234567,
             }
         }
         speech_server.supported_speakers = {"ryan", "vivian", "custom_voice"}
@@ -1008,7 +1016,7 @@ class TestTTSMethods:
         assert params["x_vector_only_mode"] == [False]
         assert params["task_type"] == ["Base"]
         assert params["ref_text"] == ["Hello world transcript"]
-        assert params["voice_created_at"] == [1711234567.89]
+        assert params["voice_created_at"] == [1711234567]
 
     def test_build_tts_params_without_uploaded_voice(self, speech_server):
         """Test _build_tts_params does not auto-set ref_audio for non-uploaded voices."""
@@ -1051,28 +1059,29 @@ class TestTTSMethods:
         assert "x_vector_only_mode" not in params
 
     def test_get_uploaded_audio_data(self, speech_server, mocker: MockerFixture):
-        """Test _get_uploaded_audio_data function."""
-        # Mock file operations
-        mock_open = mocker.patch("builtins.open", create=True)
-        mock_b64encode = mocker.patch("base64.b64encode")
-        mock_exists = mocker.patch("pathlib.Path.exists")
-        mock_exists.return_value = True
-        mock_b64encode.return_value = b"ZmFrZWF1ZGlv"
+        """Returns a data URL by loading audio via safetensors + re-encoding WAV."""
+        mocker.patch("pathlib.Path.exists", return_value=True)
+        mocker.patch("soundfile.write")
+        mocker.patch("base64.b64encode", return_value=b"ZmFrZWF1ZGlv")
+        mock_ctx = mocker.MagicMock()
+        mock_ctx.keys.return_value = ["audio"]
+        mock_ctx.get_tensor.return_value = torch.zeros(88200)
+        mock_ctx.metadata.return_value = {"sample_rate": "44100"}
+        mock_safe_open = mocker.MagicMock()
+        mock_safe_open.return_value.__enter__.return_value = mock_ctx
+        mocker.patch("safetensors.safe_open", mock_safe_open)
 
-        # Setup mock file
-        mock_file = mocker.MagicMock()
-        mock_file.read.return_value = b"fakeaudio"
-        mock_open.return_value.__enter__.return_value = mock_file
-
-        # Setup uploaded speaker
         speech_server.uploaded_speakers = {
-            "test_voice": {"name": "test_voice", "file_path": "/tmp/test.wav", "mime_type": "audio/wav"}
+            "test_voice": {
+                "name": "test_voice",
+                "file_path": "/tmp/test.safetensors",
+                "mime_type": "audio/wav",
+                "embedding_source": "audio",
+                "sample_rate": 44100,
+            }
         }
         result = speech_server._get_uploaded_audio_data("test_voice")
-
         assert result == "data:audio/wav;base64,ZmFrZWF1ZGlv"
-        mock_open.assert_called_once_with(Path("/tmp/test.wav"), "rb")
-        mock_b64encode.assert_called_once_with(b"fakeaudio")
 
     def test_get_uploaded_audio_data_missing_file(self, speech_server, mocker: MockerFixture):
         """Test _get_uploaded_audio_data when file is missing."""
@@ -1230,24 +1239,6 @@ class TestTTSMethods:
         # Must NOT have ref_audio — that would fail for safetensors files
         assert "ref_audio" not in params
 
-    def test_validate_rejects_embedding_voice_with_pending_cache(self, speech_server, mocker: MockerFixture):
-        """Validation should reject embedding voices whose cache is not yet ready."""
-        speech_server.uploaded_speakers = {
-            "myvoice": {
-                "name": "myvoice",
-                "file_path": "/tmp/myvoice.safetensors",
-                "mime_type": "application/x-safetensors",
-                "embedding_source": "direct",
-                "cache_status": "pending",
-                "cache_file": None,
-            }
-        }
-        req = OpenAICreateSpeechRequest.model_validate({"input": "Hello", "speaker": "myvoice", "task_type": "Base"})
-        mocker.patch("pathlib.Path.exists", return_value=True)
-        err = speech_server._validate_qwen_tts_request(req)
-        assert err is not None
-        assert "not yet ready" in err
-
     def test_x_vector_only_mode_not_overwritten_for_uploaded_embedding(self, speech_server, mocker: MockerFixture):
         """x_vector_only_mode set by uploaded embedding must not be overwritten by request field."""
         speech_server.uploaded_speakers = {
@@ -2294,6 +2285,7 @@ class TestCosyVoice3Serving:
                 "mm_processor_kwargs": {"prompt_text": "ref text", "sample_rate": 24000},
             }
         )
+        cosyvoice3_server._apply_cosyvoice3_dynamic_tokens = mocker.MagicMock(side_effect=lambda spl, req: spl)
 
         request = OpenAICreateSpeechRequest(
             input="Hello",

tests/helpers/fixtures/speaker_cache.py

@@ -0,0 +1,21 @@
+"""Fixtures for the process-wide speaker cache singleton."""
+
+from __future__ import annotations
+
+import pytest
+
+
+@pytest.fixture
+def fresh_speaker_cache():
+    """Reset the process-wide speaker cache singleton before and after the test."""
+    import vllm_omni.utils.speaker_cache as _sc
+
+    def _reset():
+        with _sc._SINGLETON_LOCK:
+            if _sc._SINGLETON is not None:
+                _sc._SINGLETON.clear()
+            _sc._SINGLETON = None
+
+    _reset()
+    yield
+    _reset()

tests/model_executor/models/test_fish_speech_voice_cache.py

@@ -1,218 +0,0 @@
-"""Tests for Fish Speech DAC-code caching via VoiceEmbeddingCache.
-
-Covers:
-  - Cache miss → DAC encode → store
-  - Cache hit → skip DAC encode, reuse cached ref_codes_fq
-  - Inline ref_audio (no voice name) → no caching, full encode path
-  - Stale-cache protection via created_at
-  - Temp file cleanup on cache hit
-"""
-
-import os
-import tempfile
-
-import numpy as np
-import pytest
-import torch
-from pytest_mock import MockerFixture
-
-pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
-
-
-def _make_info_dict(
-    *,
-    text: str = "Hello world",
-    ref_text: str = "Reference transcript",
-    ref_audio_sr: int = 44100,
-    voice_name: str | None = None,
-    voice_created_at: float | None = None,
-    ref_audio_path: str | None = None,
-) -> dict:
-    """Build a minimal info_dict for _build_structured_voice_clone_prefill_embeds."""
-    d: dict = {
-        "text": text,
-        "ref_text": ref_text,
-        "ref_audio_sr": ref_audio_sr,
-        "fish_structured_voice_clone": True,
-    }
-    if ref_audio_path is not None:
-        d["ref_audio_path"] = ref_audio_path
-    if voice_name is not None:
-        d["voice_name"] = voice_name
-    if voice_created_at is not None:
-        d["voice_created_at"] = voice_created_at
-    return d
-
-
-def _write_temp_npy(wav: np.ndarray | None = None) -> str:
-    """Write a temporary .npy file with dummy audio and return its path."""
-    if wav is None:
-        wav = np.random.randn(44100).astype(np.float32)  # 1 second @ 44.1kHz
-    with tempfile.NamedTemporaryFile(prefix="fish_test_", suffix=".npy", delete=False) as f:
-        np.save(f, wav)
-        return f.name
-
-
-# Fake ref_codes_fq: [frames, codebooks]
-_FAKE_REF_CODES = torch.randint(0, 1024, (10, 10), dtype=torch.long)
-
-
-class TestFishSpeechVoiceCacheIntegration:
-    """Test the cache-hit / cache-miss / no-cache paths in the model."""
-
-    @pytest.fixture
-    def mock_model(self, mocker: MockerFixture):
-        """Create a mock FishSpeechSlowARForConditionalGeneration with cache."""
-        from vllm_omni.utils.voice_cache import VoiceEmbeddingCache
-
-        model = mocker.MagicMock()
-        model._voice_cache = VoiceEmbeddingCache(max_entries=4)
-        model._semantic_begin_id = 151678
-        model._num_codebooks = 10
-        model._codebook_size = 4096
-        model.model_path = "/fake/model"
-        model.codebook_embeddings = mocker.MagicMock()
-        model.codebook_embeddings.weight = mocker.MagicMock()
-        model.codebook_embeddings.weight.device = torch.device("cpu")
-        return model
-
-    def test_cache_miss_stores_codes(self, mock_model):
-        """First request with a named voice should encode and store in cache."""
-        cache = mock_model._voice_cache
-        voice_name = "alice"
-        created_at = 1712345678.0
-
-        # Verify cache starts empty.
-        key = cache.make_cache_key(voice_name, xvec_only=False, created_at=created_at)
-        assert cache.get(key) is None
-
-        # Simulate a cache store (what the model does on miss).
-        cache.put(key, {"ref_codes_fq": _FAKE_REF_CODES.detach().cpu()})
-
-        # Verify it's now cached.
-        cached = cache.get(key)
-        assert cached is not None
-        assert torch.equal(cached["ref_codes_fq"], _FAKE_REF_CODES)
-
-    def test_cache_hit_returns_cached_codes(self, mock_model):
-        """Second request with same voice should hit cache."""
-        cache = mock_model._voice_cache
-        voice_name = "alice"
-        created_at = 1712345678.0
-
-        key = cache.make_cache_key(voice_name, xvec_only=False, created_at=created_at)
-        cache.put(key, {"ref_codes_fq": _FAKE_REF_CODES.detach().cpu()})
-
-        # Hit.
-        cached = cache.get(key)
-        assert cached is not None
-        ref_codes = cached["ref_codes_fq"].to(device=torch.device("cpu"), dtype=torch.long)
-        assert torch.equal(ref_codes, _FAKE_REF_CODES)
-        assert cache.stats()["hits"] >= 1
-
-    def test_no_voice_name_skips_cache(self, mock_model):
-        """Inline ref_audio without voice_name should not use cache."""
-        cache = mock_model._voice_cache
-
-        # Without voice_name, the model should not interact with cache at all.
-        info = _make_info_dict(voice_name=None, ref_audio_path=_write_temp_npy())
-        assert info.get("voice_name") is None
-        # Cache should remain untouched.
-        assert cache.stats()["hits"] == 0
-        assert cache.stats()["misses"] == 0
-
-    def test_stale_cache_on_reupload(self, mock_model):
-        """Re-uploading a voice (new created_at) should not hit old cache."""
-        cache = mock_model._voice_cache
-        voice_name = "alice"
-
-        key_old = cache.make_cache_key(voice_name, xvec_only=False, created_at=1000.0)
-        cache.put(key_old, {"ref_codes_fq": _FAKE_REF_CODES})
-
-        # Re-upload produces a different created_at.
-        key_new = cache.make_cache_key(voice_name, xvec_only=False, created_at=2000.0)
-        assert cache.get(key_new) is None  # miss
-        assert cache.get(key_old) is not None  # old still there
-
-    def test_temp_file_cleaned_on_cache_hit(self):
-        """On cache hit, the temp .npy file written by the entrypoint should be deleted."""
-        tmp_path = _write_temp_npy()
-        assert os.path.exists(tmp_path)
-
-        # Simulate what the model does on cache hit: remove the temp file.
-        try:
-            os.remove(tmp_path)
-        except OSError:
-            pass
-        assert not os.path.exists(tmp_path)
-
-    def test_created_at_zero_disables_cache(self, mock_model):
-        """created_at=0 should not create a cache key (caching disabled)."""
-        cache = mock_model._voice_cache
-
-        info = _make_info_dict(
-            voice_name="bob",
-            voice_created_at=0.0,
-            ref_audio_path=_write_temp_npy(),
-        )
-        # The model checks: if _created_at > 0 → enable cache.
-        # With 0.0, no cache interaction should happen.
-        _created_at = float(info.get("voice_created_at", 0))
-        assert _created_at <= 0
-        assert cache.stats()["hits"] == 0
-        assert cache.stats()["misses"] == 0
-
-
-class TestFishSpeechValidatorUploadedVoice:
-    """Test _validate_fish_tts_request uploaded voice resolution."""
-
-    def test_uploaded_voice_resolves_ref_audio(self, mocker: MockerFixture):
-        """When voice matches an uploaded speaker, ref_audio should be auto-set."""
-        request = mocker.MagicMock()
-        request.input = "Hello"
-        request.voice = "alice"
-        request.ref_audio = None
-        request.ref_text = None
-        request.max_new_tokens = None
-
-        # Uploaded speaker with ref_text.
-        uploaded_speakers = {
-            "alice": {
-                "file_path": "/tmp/fake_audio.wav",
-                "ref_text": "Hi this is Alice",
-                "created_at": 1712345678,
-            },
-        }
-
-        # Simulate: voice in uploaded_speakers, file exists, get_audio returns data URL.
-        mocker.patch("pathlib.Path.exists", return_value=True)
-        voice_lower = request.voice.lower()
-        assert voice_lower in uploaded_speakers
-
-        speaker_info = uploaded_speakers[voice_lower]
-        ref_text_from_upload = speaker_info.get("ref_text")
-        assert ref_text_from_upload == "Hi this is Alice"
-
-    def test_uploaded_voice_without_ref_text_uses_request_ref_text(self, mocker: MockerFixture):
-        """If upload has no ref_text but request provides it, use request's."""
-        request = mocker.MagicMock()
-        request.input = "Hello"
-        request.voice = "bob"
-        request.ref_audio = None
-        request.ref_text = "Request-level transcript"
-        request.max_new_tokens = None
-
-        uploaded_speakers = {
-            "bob": {
-                "file_path": "/tmp/fake_audio.wav",
-                "ref_text": None,
-                "created_at": 1712345678,
-            },
-        }
-
-        voice_lower = request.voice.lower()
-        speaker_info = uploaded_speakers[voice_lower]
-        upload_ref_text = speaker_info.get("ref_text")
-        # Upload has no ref_text, so request.ref_text should remain.
-        assert upload_ref_text is None
-        assert request.ref_text == "Request-level transcript"

tests/test_fish_speech_voice_cache.py

@@ -1,227 +0,0 @@
-"""Tests for Fish Speech DAC-code caching via VoiceEmbeddingCache.
-
-Covers:
-  - Cache miss → DAC encode → store
-  - Cache hit → skip DAC encode, reuse cached ref_codes_fq
-  - Inline ref_audio (no voice name) → no caching, full encode path
-  - Stale-cache protection via created_at
-  - Temp file cleanup on cache hit
-"""
-
-import os
-import tempfile
-from pathlib import Path
-
-import numpy as np
-import pytest
-import torch
-from pytest_mock import MockerFixture
-
-pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
-
-
-def _make_info_dict(
-    *,
-    text: str = "Hello world",
-    ref_text: str = "Reference transcript",
-    ref_audio_sr: int = 44100,
-    voice_name: str | None = None,
-    voice_created_at: float | None = None,
-    ref_audio_path: str | None = None,
-) -> dict:
-    """Build a minimal info_dict for _build_structured_voice_clone_prefill_embeds."""
-    d: dict = {
-        "text": text,
-        "ref_text": ref_text,
-        "ref_audio_sr": ref_audio_sr,
-        "fish_structured_voice_clone": True,
-    }
-    if ref_audio_path is not None:
-        d["ref_audio_path"] = ref_audio_path
-    if voice_name is not None:
-        d["voice_name"] = voice_name
-    if voice_created_at is not None:
-        d["voice_created_at"] = voice_created_at
-    return d
-
-
-def _write_temp_npy(wav: np.ndarray | None = None) -> str:
-    """Write a temporary .npy file with dummy audio and return its path."""
-    if wav is None:
-        wav = np.random.randn(44100).astype(np.float32)  # 1 second @ 44.1kHz
-    with tempfile.NamedTemporaryFile(prefix="fish_test_", suffix=".npy", delete=False) as f:
-        np.save(f, wav)
-        return f.name
-
-
-# Fake ref_codes_fq: [frames, codebooks]
-_FAKE_REF_CODES = torch.randint(0, 1024, (10, 10), dtype=torch.long)
-
-
-class TestFishSpeechVoiceCacheIntegration:
-    """Test the cache-hit / cache-miss / no-cache paths in the model."""
-
-    @pytest.fixture
-    def mock_model(self, mocker: MockerFixture):
-        """Create a mock FishSpeechSlowARForConditionalGeneration with cache."""
-        from vllm_omni.utils.voice_cache import VoiceEmbeddingCache
-
-        model = mocker.MagicMock()
-        model._voice_cache = VoiceEmbeddingCache(max_entries=4)
-        model._semantic_begin_id = 151678
-        model._num_codebooks = 10
-        model._codebook_size = 4096
-        model.model_path = "/fake/model"
-        model.codebook_embeddings = mocker.MagicMock()
-        model.codebook_embeddings.weight = mocker.MagicMock()
-        model.codebook_embeddings.weight.device = torch.device("cpu")
-        return model
-
-    def test_cache_miss_stores_codes(self, mock_model):
-        """First request with a named voice should encode and store in cache."""
-        cache = mock_model._voice_cache
-        voice_name = "alice"
-        created_at = 1712345678.0
-
-        # Verify cache starts empty.
-        key = cache.make_cache_key(voice_name, xvec_only=False, created_at=created_at)
-        assert cache.get(key) is None
-
-        # Simulate a cache store (what the model does on miss).
-        cache.put(key, {"ref_codes_fq": _FAKE_REF_CODES.detach().cpu()})
-
-        # Verify it's now cached.
-        cached = cache.get(key)
-        assert cached is not None
-        assert torch.equal(cached["ref_codes_fq"], _FAKE_REF_CODES)
-
-    def test_cache_hit_returns_cached_codes(self, mock_model):
-        """Second request with same voice should hit cache."""
-        cache = mock_model._voice_cache
-        voice_name = "alice"
-        created_at = 1712345678.0
-
-        key = cache.make_cache_key(voice_name, xvec_only=False, created_at=created_at)
-        cache.put(key, {"ref_codes_fq": _FAKE_REF_CODES.detach().cpu()})
-
-        # Hit.
-        cached = cache.get(key)
-        assert cached is not None
-        ref_codes = cached["ref_codes_fq"].to(device=torch.device("cpu"), dtype=torch.long)
-        assert torch.equal(ref_codes, _FAKE_REF_CODES)
-        assert cache.stats()["hits"] >= 1
-
-    def test_no_voice_name_skips_cache(self, mock_model):
-        """Inline ref_audio without voice_name should not use cache."""
-        cache = mock_model._voice_cache
-
-        # Without voice_name, the model should not interact with cache at all.
-        info = _make_info_dict(voice_name=None, ref_audio_path=_write_temp_npy())
-        assert info.get("voice_name") is None
-        # Cache should remain untouched.
-        assert cache.stats()["hits"] == 0
-        assert cache.stats()["misses"] == 0
-
-    def test_stale_cache_on_reupload(self, mock_model):
-        """Re-uploading a voice (new created_at) should not hit old cache."""
-        cache = mock_model._voice_cache
-        voice_name = "alice"
-
-        key_old = cache.make_cache_key(voice_name, xvec_only=False, created_at=1000.0)
-        cache.put(key_old, {"ref_codes_fq": _FAKE_REF_CODES})
-
-        # Re-upload produces a different created_at.
-        key_new = cache.make_cache_key(voice_name, xvec_only=False, created_at=2000.0)
-        assert cache.get(key_new) is None  # miss
-        assert cache.get(key_old) is not None  # old still there
-
-    def test_temp_file_cleaned_on_cache_hit(self):
-        """On cache hit, the temp .npy file written by the entrypoint should be deleted."""
-        tmp_path = _write_temp_npy()
-        assert os.path.exists(tmp_path)
-
-        # Simulate what the model does on cache hit: remove the temp file.
-        try:
-            os.remove(tmp_path)
-        except OSError:
-            pass
-        assert not os.path.exists(tmp_path)
-
-    def test_created_at_zero_disables_cache(self, mock_model):
-        """created_at=0 should not create a cache key (caching disabled)."""
-        cache = mock_model._voice_cache
-
-        info = _make_info_dict(
-            voice_name="bob",
-            voice_created_at=0.0,
-            ref_audio_path=_write_temp_npy(),
-        )
-        # The model checks: if _created_at > 0 → enable cache.
-        # With 0.0, no cache interaction should happen.
-        _created_at = float(info.get("voice_created_at", 0))
-        assert _created_at <= 0
-        assert cache.stats()["hits"] == 0
-        assert cache.stats()["misses"] == 0
-
-
-class TestFishSpeechValidatorUploadedVoice:
-    """Test _validate_fish_tts_request uploaded voice resolution."""
-
-    def test_uploaded_voice_resolves_ref_audio(
-        self,
-        monkeypatch: pytest.MonkeyPatch,
-        mocker: MockerFixture,
-    ):
-        """When voice matches an uploaded speaker, ref_audio should be auto-set."""
-        request = mocker.MagicMock()
-        request.input = "Hello"
-        request.voice = "alice"
-        request.ref_audio = None
-        request.ref_text = None
-        request.max_new_tokens = None
-
-        # Uploaded speaker with ref_text.
-        uploaded_speakers = {
-            "alice": {
-                "file_path": "/tmp/fake_audio.wav",
-                "ref_text": "Hi this is Alice",
-                "created_at": 1712345678,
-            },
-        }
-
-        # Simulate: voice in uploaded_speakers, file exists, get_audio returns data URL.
-        monkeypatch.setattr(Path, "exists", lambda self: True)
-
-        voice_lower = request.voice.lower()
-        assert voice_lower in uploaded_speakers
-
-        speaker_info = uploaded_speakers[voice_lower]
-        ref_text_from_upload = speaker_info.get("ref_text")
-        assert ref_text_from_upload == "Hi this is Alice"
-
-    def test_uploaded_voice_without_ref_text_uses_request_ref_text(
-        self,
-        mocker: MockerFixture,
-    ):
-        """If upload has no ref_text but request provides it, use request's."""
-        request = mocker.MagicMock()
-        request.input = "Hello"
-        request.voice = "bob"
-        request.ref_audio = None
-        request.ref_text = "Request-level transcript"
-        request.max_new_tokens = None
-
-        uploaded_speakers = {
-            "bob": {
-                "file_path": "/tmp/fake_audio.wav",
-                "ref_text": None,
-                "created_at": 1712345678,
-            },
-        }
-
-        voice_lower = request.voice.lower()
-        speaker_info = uploaded_speakers[voice_lower]
-        upload_ref_text = speaker_info.get("ref_text")
-        # Upload has no ref_text, so request.ref_text should remain.
-        assert upload_ref_text is None
-        assert request.ref_text == "Request-level transcript"

tests/test_speaker_cache.py

@@ -0,0 +1,150 @@
+import threading
+
+import pytest
+import torch
+
+from vllm_omni.utils.speaker_cache import SpeakerEmbeddingCache, get_speaker_cache
+
+pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
+
+
+@pytest.fixture
+def cache():
+    return SpeakerEmbeddingCache(max_bytes=10 * 1024**2)
+
+
+def _k(model: str, name: str, created_at: int = 0) -> tuple[str, str, int]:
+    return (model, name, created_at)
+
+
+class TestSpeakerEmbeddingCacheBehavior:
+    def test_miss_returns_none(self, cache):
+        assert cache.get(_k("voxcpm2", "nonexistent")) is None
+
+    def test_put_and_hit(self, cache):
+        cache.put(_k("voxcpm2", "alice"), {"val": 42})
+        assert cache.get(_k("voxcpm2", "alice"))["val"] == 42
+
+    def test_lru_access_promotes(self):
+        c = SpeakerEmbeddingCache(max_bytes=4 * 4096)
+        for k in ("a", "b", "c", "d"):
+            c.put(_k("m", k), {"emb": torch.zeros(1024, dtype=torch.float32)})
+        c.get(_k("m", "a"))
+        c.put(_k("m", "e"), {"emb": torch.zeros(1024, dtype=torch.float32)})
+        assert c.get(_k("m", "a")) is not None
+        assert c.get(_k("m", "b")) is None
+
+    def test_put_overwrites(self, cache):
+        cache.put(_k("m", "k"), {"old": True})
+        cache.put(_k("m", "k"), {"new": True})
+        assert "new" in cache.get(_k("m", "k"))
+        assert cache.stats()["entries"] == 1
+
+    def test_make_cache_key_namespaces_model_type(self):
+        k1 = SpeakerEmbeddingCache.make_cache_key("alice", model_type="voxcpm2")
+        k2 = SpeakerEmbeddingCache.make_cache_key("alice", model_type="fish_speech")
+        assert k1 != k2
+        assert k1 == ("voxcpm2", "alice", 0)
+        assert k2 == ("fish_speech", "alice", 0)
+
+    def test_make_cache_key_created_at_isolation(self):
+        k_old = SpeakerEmbeddingCache.make_cache_key("alice", model_type="voxcpm2", created_at=1712000000)
+        k_new = SpeakerEmbeddingCache.make_cache_key("alice", model_type="voxcpm2", created_at=1712000042)
+        assert k_old != k_new
+
+    def test_make_cache_key_requires_fields(self):
+        with pytest.raises(ValueError):
+            SpeakerEmbeddingCache.make_cache_key("", model_type="voxcpm2")
+        with pytest.raises(ValueError):
+            SpeakerEmbeddingCache.make_cache_key("alice", model_type="")
+
+    def test_clear_all(self, cache):
+        cache.put(_k("m", "a"), {"v": 1})
+        cache.put(_k("m", "b"), {"v": 2})
+        assert cache.clear() == 2
+        assert cache.stats()["entries"] == 0
+
+    def test_clear_matches_speaker_across_model_types(self, cache):
+        cache.put(_k("voxcpm2", "alice", 1), {"v": 1})
+        cache.put(_k("fish_speech", "alice", 2), {"v": 2})
+        cache.put(_k("cosyvoice3", "bob", 3), {"v": 3})
+        assert cache.clear("alice") == 2
+        assert cache.get(_k("voxcpm2", "alice", 1)) is None
+        assert cache.get(_k("fish_speech", "alice", 2)) is None
+        assert cache.get(_k("cosyvoice3", "bob", 3)) is not None
+
+    def test_stale_cache_on_reupload(self, cache):
+        cache.put(_k("voxcpm2", "alice", 1712000000), {"emb": torch.zeros(4), "gen": "old"})
+        assert cache.get(_k("voxcpm2", "alice", 1712000042)) is None
+
+    def test_memory_bytes(self, cache):
+        assert cache.memory_bytes() == 0
+        t = torch.zeros(1024, dtype=torch.float32)  # 4096 bytes
+        cache.put(_k("m", "k"), {"emb": t})
+        assert cache.memory_bytes() == 4096
+
+    def test_memory_bytes_ignores_non_tensors(self, cache):
+        cache.put(_k("m", "k"), {"flag": True, "name": "test", "nothing": None})
+        assert cache.memory_bytes() == 0
+
+    def test_byte_budget_evicts(self):
+        c = SpeakerEmbeddingCache(max_bytes=8192)
+        c.put(_k("m", "a"), {"emb": torch.zeros(1024, dtype=torch.float32)})
+        c.put(_k("m", "b"), {"emb": torch.zeros(1024, dtype=torch.float32)})
+        c.put(_k("m", "c"), {"emb": torch.zeros(1024, dtype=torch.float32)})
+        assert c.get(_k("m", "a")) is None
+        assert c.get(_k("m", "b")) is not None
+        assert c.get(_k("m", "c")) is not None
+        assert c.memory_bytes() <= 8192
+
+    def test_oversize_entry_skipped(self):
+        c = SpeakerEmbeddingCache(max_bytes=1024)
+        c.put(_k("m", "huge"), {"emb": torch.zeros(2048, dtype=torch.float32)})
+        assert c.get(_k("m", "huge")) is None
+        assert c.stats()["entries"] == 0
+
+    def test_stats(self, cache):
+        cache.put(_k("m", "x"), {"v": 1})
+        cache.get(_k("m", "x"))
+        cache.get(_k("m", "y"))
+        s = cache.stats()
+        assert s["hits"] == 1
+        assert s["misses"] >= 1
+        assert s["entries"] == 1
+
+    def test_thread_safety(self):
+        cache = SpeakerEmbeddingCache()
+        errors = []
+
+        def worker(tid):
+            try:
+                for i in range(50):
+                    cache.put(_k("m", f"t{tid}_v{i}"), {"tid": tid})
+                    cache.get(_k("m", f"t{tid}_v{i}"))
+            except Exception as e:
+                errors.append(e)
+
+        threads = [threading.Thread(target=worker, args=(t,)) for t in range(10)]
+        for t in threads:
+            t.start()
+        for t in threads:
+            t.join()
+        assert not errors
+        assert cache.stats()["entries"] == 500
+
+    def test_empty_speaker_name_raises_error(self, cache):
+        with pytest.raises(ValueError, match="speaker_name cannot be an empty string"):
+            cache.clear("")
+
+    def test_cpu_storage_verification(self, cache):
+        tensor = torch.randn(10, 128)
+        cache.put(_k("m", "alice"), {"emb": tensor})
+        cached = cache.get(_k("m", "alice"))
+        assert cached["emb"].device.type == "cpu"
+
+
+class TestSingleton:
+    def test_singleton_identity(self, fresh_speaker_cache):
+        a = get_speaker_cache()
+        b = get_speaker_cache()
+        assert a is b

tests/test_speaker_cache_integration.py

@@ -0,0 +1,59 @@
+"""Integration tests for the process-wide speaker cache across serving + models."""
+
+import pytest
+import torch
+
+from vllm_omni.utils.speaker_cache import SpeakerEmbeddingCache, get_speaker_cache
+
+pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
+
+
+class TestSpeakerCacheIntegration:
+    def test_delete_propagates_across_model_types(self):
+        cache = SpeakerEmbeddingCache()
+        voxcpm_key = SpeakerEmbeddingCache.make_cache_key("alice", model_type="voxcpm2")
+        fish_key = SpeakerEmbeddingCache.make_cache_key("alice", model_type="fish_speech")
+        bob_key = SpeakerEmbeddingCache.make_cache_key("bob", model_type="voxcpm2")
+
+        cache.put(voxcpm_key, {"emb": torch.zeros(4)})
+        cache.put(fish_key, {"emb": torch.zeros(4)})
+        cache.put(bob_key, {"emb": torch.zeros(4)})
+
+        removed = cache.clear("alice")
+
+        assert removed == 2
+        assert cache.get(voxcpm_key) is None
+        assert cache.get(fish_key) is None
+        assert cache.get(bob_key) is not None
+
+    def test_singleton_shared_across_call_sites(self, fresh_speaker_cache):
+        cache_a = get_speaker_cache()
+        cache_b = get_speaker_cache()
+        assert cache_a is cache_b
+        key = SpeakerEmbeddingCache.make_cache_key("carol", model_type="voxcpm2")
+        cache_a.put(key, {"tag": "from_a"})
+        got = cache_b.get(key)
+        assert got is not None
+        assert got["tag"] == "from_a"
+
+    def test_shutdown_clears_all_entries(self):
+        cache = SpeakerEmbeddingCache()
+        for i in range(3):
+            k = SpeakerEmbeddingCache.make_cache_key(f"voice{i}", model_type="voxcpm2")
+            cache.put(k, {"emb": torch.zeros(2)})
+        assert cache.stats()["entries"] == 3
+        cache.clear()
+        assert cache.stats()["entries"] == 0
+
+    def test_stale_cache_protection_delete_then_reupload(self):
+        cache = SpeakerEmbeddingCache()
+        old_key = SpeakerEmbeddingCache.make_cache_key("alice", model_type="voxcpm2", created_at=1712000000)
+        cache.put(old_key, {"emb": torch.ones(4) * 3.14, "gen": "old"})
+
+        new_key = SpeakerEmbeddingCache.make_cache_key("alice", model_type="voxcpm2", created_at=1712000042)
+        assert cache.get(new_key) is None
+        assert cache.get(old_key) is not None
+
+        cache.clear("alice")
+        assert cache.get(old_key) is None
+        assert cache.get(new_key) is None

tests/test_speaker_metadata_persistence.py

@@ -0,0 +1,100 @@
+"""Tests for speaker metadata (created_at, consent, ref_text, ...) round-tripping
+through the safetensors header.
+
+These cover the helpers that back ``_restore_uploaded_speakers`` — the logic
+that rebuilds ``uploaded_speakers`` on server start by reading the
+``.safetensors`` file written during upload.
+"""
+
+import pytest
+
+from vllm_omni.entrypoints.openai.serving_speech import OmniOpenAIServingSpeech
+
+pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
+
+
+class TestSpeakerMetadataRoundTrip:
+    def test_str_only_header(self):
+        """The safetensors metadata dict must be ``dict[str, str]`` — every value
+        serialized as a string (ints and None handled)."""
+        data = {
+            "name": "Alice",
+            "voice_name_lower": "alice",
+            "consent": "xxx",
+            "created_at": 1712345678,
+            "sample_rate": 24000,
+            "embedding_source": "audio",
+            "ref_text": None,  # None values must be dropped
+            "file_path": "/tmp/should/be/stripped.safetensors",  # re-derived, not persisted
+        }
+        header = OmniOpenAIServingSpeech._speaker_metadata_to_header(data)
+        assert all(isinstance(k, str) and isinstance(v, str) for k, v in header.items())
+        assert "ref_text" not in header  # None stripped
+        assert "file_path" not in header  # never persisted
+
+    def test_int_fields_coerce_back(self):
+        """Int fields (created_at, file_size, sample_rate, embedding_dim) survive
+        the string round-trip with their type preserved."""
+        data = {
+            "name": "Bob",
+            "voice_name_lower": "bob",
+            "consent": "yyy",
+            "created_at": 1712345678,
+            "sample_rate": 44100,
+            "file_size": 12345,
+            "embedding_dim": 1024,
+            "embedding_source": "direct",
+        }
+        header = OmniOpenAIServingSpeech._speaker_metadata_to_header(data)
+        back = OmniOpenAIServingSpeech._speaker_metadata_from_header(header, "/some/path.safetensors")
+        assert isinstance(back["created_at"], int)
+        assert back["created_at"] == 1712345678
+        assert isinstance(back["sample_rate"], int)
+        assert back["sample_rate"] == 44100
+        assert isinstance(back["file_size"], int)
+        assert isinstance(back["embedding_dim"], int)
+
+    def test_file_path_reinjected_on_load(self):
+        """file_path is not persisted in the header; restore derives it from the
+        actual file location on disk."""
+        data = {
+            "name": "Carol",
+            "voice_name_lower": "carol",
+            "consent": "zzz",
+            "created_at": 1234,
+            "embedding_source": "audio",
+        }
+        header = OmniOpenAIServingSpeech._speaker_metadata_to_header(data)
+        back = OmniOpenAIServingSpeech._speaker_metadata_from_header(header, "/real/path.safetensors")
+        assert back["file_path"] == "/real/path.safetensors"
+
+    def test_string_fields_preserved(self):
+        """ref_text, consent, speaker_description must survive unchanged."""
+        data = {
+            "name": "Dave",
+            "voice_name_lower": "dave",
+            "consent": "consent-id-42",
+            "created_at": 1,
+            "ref_text": "Hello. This is a transcript with punctuation!",
+            "speaker_description": "A warm baritone voice.",
+            "embedding_source": "audio",
+        }
+        header = OmniOpenAIServingSpeech._speaker_metadata_to_header(data)
+        back = OmniOpenAIServingSpeech._speaker_metadata_from_header(header, "/x.safetensors")
+        assert back["ref_text"] == data["ref_text"]
+        assert back["speaker_description"] == data["speaker_description"]
+        assert back["consent"] == data["consent"]
+
+    def test_malformed_int_is_left_as_string(self):
+        """If an int field somehow contains non-numeric text (manual edit,
+        corruption), the loader does not crash; it leaves the field as-is."""
+        header = {
+            "name": "Eve",
+            "voice_name_lower": "eve",
+            "consent": "x",
+            "created_at": "not-a-number",
+            "embedding_source": "audio",
+        }
+        back = OmniOpenAIServingSpeech._speaker_metadata_from_header(header, "/p.safetensors")
+        # Preserved as string rather than raising.
+        assert back["created_at"] == "not-a-number"

tests/test_voice_cache.py

@@ -1,129 +0,0 @@
-import threading
-
-import pytest
-
-from vllm_omni.utils.voice_cache import VoiceEmbeddingCache
-
-pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
-
-
-@pytest.fixture
-def cache():
-    return VoiceEmbeddingCache(max_entries=4)
-
-
-class TestVoiceEmbeddingCache:
-    def test_miss_returns_none(self, cache: VoiceEmbeddingCache):
-        assert cache.get("nonexistent") is None
-        assert cache.stats()["misses"] == 1
-
-    def test_put_and_hit(self, cache: VoiceEmbeddingCache):
-        cache.put("abc", {"val": 42})
-        result = cache.get("abc")
-        assert result is not None
-        assert result["val"] == 42
-        assert cache.stats()["hits"] == 1
-
-    def test_lru_eviction(self, cache: VoiceEmbeddingCache):
-        for i in range(5):
-            cache.put(f"key{i}", {"i": i})
-        # key0 should have been evicted (oldest, max_entries=4)
-        assert cache.get("key0") is None
-        # key1..key4 should still be present
-        for i in range(1, 5):
-            assert cache.get(f"key{i}") is not None
-        assert cache.stats()["entries"] == 4
-
-    def test_lru_access_promotes(self, cache: VoiceEmbeddingCache):
-        cache.put("a", {"v": 1})
-        cache.put("b", {"v": 2})
-        cache.put("c", {"v": 3})
-        cache.put("d", {"v": 4})
-        # Access "a" to promote it to MRU
-        cache.get("a")
-        # Insert "e" -- should evict "b" (now the oldest), not "a"
-        cache.put("e", {"v": 5})
-        assert cache.get("a") is not None
-        assert cache.get("b") is None
-
-    def test_put_overwrites(self, cache: VoiceEmbeddingCache):
-        cache.put("k", {"old": True})
-        cache.put("k", {"new": True})
-        result = cache.get("k")
-        assert result is not None
-        assert "new" in result
-        assert "old" not in result
-        assert cache.stats()["entries"] == 1
-
-    def test_make_cache_key_includes_mode(self):
-        k1 = VoiceEmbeddingCache.make_cache_key("alice", xvec_only=True)
-        k2 = VoiceEmbeddingCache.make_cache_key("alice", xvec_only=False)
-        assert k1 != k2
-        assert "xvec" in k1
-        assert "icl" in k2
-
-    def test_make_cache_key_deterministic(self):
-        k1 = VoiceEmbeddingCache.make_cache_key("bob", xvec_only=True)
-        k2 = VoiceEmbeddingCache.make_cache_key("bob", xvec_only=True)
-        assert k1 == k2
-
-    def test_make_cache_key_created_at_isolation(self):
-        """Different created_at timestamps must produce different keys (stale-cache protection)."""
-        k1 = VoiceEmbeddingCache.make_cache_key("alice", xvec_only=False, created_at=1000.0)
-        k2 = VoiceEmbeddingCache.make_cache_key("alice", xvec_only=False, created_at=2000.0)
-        assert k1 != k2
-
-    def test_stale_cache_protection(self, cache: VoiceEmbeddingCache):
-        """Re-upload (new created_at) must produce a cache miss, not a stale hit."""
-        key_old = VoiceEmbeddingCache.make_cache_key("alice", xvec_only=False, created_at=1000.0)
-        key_new = VoiceEmbeddingCache.make_cache_key("alice", xvec_only=False, created_at=2000.0)
-        cache.put(key_old, {"ref_spk_embedding": "old_emb"})
-        # Re-upload produces a new created_at → different key → cold miss
-        assert cache.get(key_new) is None
-        # Old key still in cache (not yet evicted)
-        assert cache.get(key_old) is not None
-
-    def test_cache_mode_isolation(self, cache: VoiceEmbeddingCache):
-        """xvec entry must NOT be served for an icl request (same voice)."""
-        key_xvec = VoiceEmbeddingCache.make_cache_key("alice", xvec_only=True)
-        key_icl = VoiceEmbeddingCache.make_cache_key("alice", xvec_only=False)
-        cache.put(key_xvec, {"ref_code": None, "ref_spk_embedding": "emb"})
-        # icl request should miss — different key
-        assert cache.get(key_icl) is None
-        # xvec request should hit
-        assert cache.get(key_xvec) is not None
-
-    def test_stats_counters(self, cache: VoiceEmbeddingCache):
-        cache.put("x", {"v": 1})
-        cache.get("x")  # hit
-        cache.get("x")  # hit
-        cache.get("y")  # miss
-        s = cache.stats()
-        assert s["hits"] == 2
-        assert s["misses"] == 1
-        assert s["entries"] == 1
-        assert s["max_entries"] == 4
-
-    def test_thread_safety(self):
-        cache = VoiceEmbeddingCache(max_entries=32)
-        errors = []
-
-        def worker(thread_id: int):
-            try:
-                for i in range(50):
-                    key = f"t{thread_id}_k{i}"
-                    cache.put(key, {"tid": thread_id, "i": i})
-                    cache.get(key)
-                    cache.get(f"t{(thread_id + 1) % 10}_k{i}")
-            except Exception as e:
-                errors.append(e)
-
-        threads = [threading.Thread(target=worker, args=(t,)) for t in range(10)]
-        for t in threads:
-            t.start()
-        for t in threads:
-            t.join()
-
-        assert not errors, f"Thread safety errors: {errors}"
-        s = cache.stats()
-        assert s["entries"] <= 32

vllm_omni/diffusion/models/omnivoice/pipeline_omnivoice.py

@@ -30,6 +30,7 @@ from vllm_omni.model_executor.models.omnivoice.config import OmniVoiceConfig
 from vllm_omni.model_executor.models.omnivoice.duration import RuleDurationEstimator
 from vllm_omni.model_executor.models.omnivoice.omnivoice_decoder import OmniVoiceDecoder
 from vllm_omni.model_executor.models.omnivoice.omnivoice_generator import OmniVoiceGenerator
+from vllm_omni.utils.speaker_cache import get_speaker_cache
 
 try:
     from transformers import HiggsAudioV2TokenizerModel
@@ -101,6 +102,9 @@ class OmniVoicePipeline(nn.Module, SupportAudioOutput):
         # Duration estimator
         self.duration_estimator = RuleDurationEstimator()
 
+        # Speaker cache for ref_audio_tokens
+        self._speaker_cache = get_speaker_cache()
+
         # Generation parameters
         self.num_step = self.config.num_step
         self.guidance_scale = self.config.guidance_scale
@@ -144,10 +148,12 @@ class OmniVoicePipeline(nn.Module, SupportAudioOutput):
         lang = "None"
         instruct = "None"
 
+        voice_name = None
         if isinstance(prompt, dict):
             text = prompt.get("input", prompt.get("text", str(prompt)))
             ref_audio = prompt.get("ref_audio")
             ref_text = prompt.get("ref_text")
+            voice_name = prompt.get("voice_name")
             lang = prompt.get("lang") or "None"
             instruct = prompt.get("instruct") or "None"
         else:
@@ -175,17 +181,37 @@ class OmniVoicePipeline(nn.Module, SupportAudioOutput):
         text_tokens = torch.tensor(encoding.ids, dtype=torch.long, device=device)
         text_len = text_tokens.shape[0]
 
-        # Encode reference audio tokens if provided
+        # Encode reference audio tokens if provided (with voice caching)
         ref_audio_tokens = None
         if ref_audio is not None:
             if self.audio_tokenizer is None:
                 raise RuntimeError(
                     "Voice cloning requires transformers>=5.3.0. Try: uv pip install 'transformers>=5.3.0'"
                 )
-            audio_signal, sr = ref_audio
-            if isinstance(audio_signal, np.ndarray):
-                audio_signal = torch.from_numpy(audio_signal).float()
-            ref_audio_tokens = self._encode_ref_audio(audio_signal, int(sr)).to(device)
+            # Check speaker cache first
+            _cache_key = None
+            if voice_name:
+                _cache_key = self._speaker_cache.make_cache_key(
+                    voice_name,
+                    model_type="omnivoice",
+                    created_at=int(prompt.get("voice_created_at") or 0),
+                )
+                cached = self._speaker_cache.get(_cache_key)
+                if cached is not None:
+                    ref_audio_tokens = cached["ref_audio_tokens"].to(device)
+                    _cache_key = None  # hit → don't store again
+                    logger.debug("Speaker cache HIT for OmniVoice speaker '%s'", voice_name)
+
+            if ref_audio_tokens is None:
+                audio_signal, sr = ref_audio
+                if isinstance(audio_signal, np.ndarray):
+                    audio_signal = torch.from_numpy(audio_signal).float()
+                ref_audio_tokens = self._encode_ref_audio(audio_signal, int(sr)).to(device)
+
+                # Store in cache for next request
+                if _cache_key is not None:
+                    self._speaker_cache.put(_cache_key, {"ref_audio_tokens": ref_audio_tokens.cpu()})
+                    logger.debug("Speaker cache STORE for OmniVoice speaker '%s'", voice_name)
 
         # Build conditional + unconditional batches [2, 8, max_len]
         text_ids = text_tokens.unsqueeze(0).repeat(num_cb, 1)

vllm_omni/model_executor/models/cosyvoice3/cosyvoice3.py

@@ -41,6 +41,7 @@ from vllm_omni.model_executor.models.cosyvoice3.utils import (
     extract_text_token,
 )
 from vllm_omni.model_executor.models.output_templates import OmniOutput
+from vllm_omni.utils.speaker_cache import get_speaker_cache
 
 logger = init_logger(__name__)
 
@@ -104,6 +105,7 @@ class CosyVoice3MultiModalProcessor(BaseMultiModalProcessor[CosyVoice3MultiModal
             providers=["CPUExecutionProvider"],
         )
         self._cached_model_dir = model_dir
+        self._speaker_cache = get_speaker_cache()
 
     def _call_hf_processor(
         self,
@@ -161,6 +163,28 @@ class CosyVoice3MultiModalProcessor(BaseMultiModalProcessor[CosyVoice3MultiModal
         )
         device = "cpu"
 
+        # Speaker cache: skip 3 ONNX sessions on cache hit
+        voice_name = mm_kwargs.get("voice_name")
+        cache_key = None
+        if voice_name and isinstance(voice_name, str):
+            cache_key = self._speaker_cache.make_cache_key(
+                voice_name,
+                model_type="cosyvoice3",
+                created_at=int(mm_kwargs.get("voice_created_at") or 0),
+            )
+            cached = self._speaker_cache.get(cache_key)
+            if cached is not None:
+                ft = BatchFeature(
+                    {
+                        "input_ids": input_ids,
+                        "speech_feat": cached["speech_feat"].clone(),
+                        "speech_token": cached["speech_token"].clone(),
+                        "speech_token_len": [cached["speech_token_len"].clone()],
+                        "embedding": cached["embedding"].clone(),
+                    }
+                )
+                return ft
+
         speech_token, speech_token_len = extract_speech_token(audio, self.speech_tokenizer, device)
         speech_feat, speech_feat_len = extract_speech_feat(audio, self.feat_extractor, device)
 
@@ -171,6 +195,18 @@ class CosyVoice3MultiModalProcessor(BaseMultiModalProcessor[CosyVoice3MultiModal
 
         embedding = extract_spk_embedding(audio, self.campplus_session, device)
 
+        # Cache the extracted artifacts for named speakers
+        if cache_key is not None:
+            self._speaker_cache.put(
+                cache_key,
+                {
+                    "speech_feat": speech_feat.detach().cpu(),
+                    "speech_token": speech_token.detach().cpu(),
+                    "speech_token_len": speech_token_len.detach().cpu(),
+                    "embedding": embedding.detach().cpu(),
+                },
+            )
+
         ft = BatchFeature(
             {
                 "input_ids": input_ids,

vllm_omni/model_executor/models/fish_speech/fish_speech_slow_ar.py

@@ -32,7 +32,7 @@ from vllm.model_executor.models.utils import PPMissingLayer, maybe_prefix
 from vllm.sequence import IntermediateTensors
 
 from vllm_omni.model_executor.models.output_templates import OmniOutput
-from vllm_omni.utils.voice_cache import VoiceEmbeddingCache
+from vllm_omni.utils.speaker_cache import get_speaker_cache
 
 from .configuration_fish_speech import FishSpeechConfig, FishSpeechFastARConfig, FishSpeechSlowARConfig
 from .dac_encoder import _load_dac_codec, encode_reference_audio_codes
@@ -262,7 +262,7 @@ class FishSpeechSlowARForConditionalGeneration(nn.Module):
         self.register_buffer("_semantic_allowed_mask", semantic_mask, persistent=False)
 
         # In-memory LRU cache for DAC-encoded reference audio codes.
-        self._voice_cache = VoiceEmbeddingCache()
+        self._speaker_cache = get_speaker_cache()
 
         # Tokeniser (lazy).
         self._tokenizer = None
@@ -539,37 +539,27 @@ class FishSpeechSlowARForConditionalGeneration(nn.Module):
         if not isinstance(ref_text, str) or not isinstance(text, str):
             raise ValueError("Fish Speech structured voice clone requires string text and ref_text")
 
-        # --- Voice cache: reuse DAC codes for uploaded (named) voices ---
-        _voice_cache_key: str | None = None
+        _speaker_cache_key: tuple[str, str, int] | None = None
         voice_name = info_dict.get("voice_name")
-        voice_created_at = info_dict.get("voice_created_at")
         if isinstance(voice_name, str) and voice_name:
-            _created_at = float(voice_created_at) if voice_created_at is not None else 0.0
-            if _created_at <= 0:
-                logger.warning(
-                    "Voice '%s' has no created_at timestamp; DAC code caching disabled for this request",
-                    voice_name,
+            _speaker_cache_key = self._speaker_cache.make_cache_key(
+                voice_name,
+                model_type="fish_speech",
+                created_at=int(info_dict.get("voice_created_at") or 0),
+            )
+            _cached = self._speaker_cache.get(_speaker_cache_key)
+            if _cached is not None:
+                ref_codes_fq = _cached["ref_codes_fq"].to(
+                    device=self.codebook_embeddings.weight.device,
+                    dtype=torch.long,
                 )
-            else:
-                _voice_cache_key = self._voice_cache.make_cache_key(
-                    voice_name,
-                    xvec_only=False,
-                    created_at=_created_at,
+                logger.debug("Speaker cache HIT for Fish Speech speaker '%s'", voice_name)
+                return self._apply_codebook_embeddings(
+                    tokenizer,
+                    text,
+                    ref_text,
+                    ref_codes_fq,
                 )
-                _cached = self._voice_cache.get(_voice_cache_key)
-                if _cached is not None:
-                    ref_codes_fq = _cached["ref_codes_fq"].to(
-                        device=self.codebook_embeddings.weight.device,
-                        dtype=torch.long,
-                    )
-                    _voice_cache_key = None  # hit → don't store again
-                    logger.debug("Voice cache HIT for Fish Speech voice '%s'", voice_name)
-                    return self._apply_codebook_embeddings(
-                        tokenizer,
-                        text,
-                        ref_text,
-                        ref_codes_fq,
-                    )
 
         if not isinstance(ref_audio_sr, int):
             raise ValueError("Fish Speech structured voice clone requires integer ref_audio_sr")
@@ -590,12 +580,12 @@ class FishSpeechSlowARForConditionalGeneration(nn.Module):
         )
 
         # Cache miss: store DAC codes for future reuse.
-        if _voice_cache_key is not None:
-            self._voice_cache.put(
-                _voice_cache_key,
+        if _speaker_cache_key is not None:
+            self._speaker_cache.put(
+                _speaker_cache_key,
                 {"ref_codes_fq": ref_codes_fq.detach().cpu()},
             )
-            logger.debug("Voice cache STORE for Fish Speech voice '%s'", voice_name)
+            logger.debug("Speaker cache STORE for Fish Speech speaker '%s'", voice_name)
 
         return self._apply_codebook_embeddings(tokenizer, text, ref_text, ref_codes_fq)
 

vllm_omni/model_executor/models/qwen3_tts/qwen3_tts_talker.py

@@ -29,7 +29,7 @@ from vllm.sequence import IntermediateTensors
 from vllm_omni.data_entry_keys import OmniPayload
 from vllm_omni.model_executor.models.output_templates import OmniOutput
 from vllm_omni.utils.audio import mel_filter_bank
-from vllm_omni.utils.voice_cache import VoiceEmbeddingCache
+from vllm_omni.utils.speaker_cache import get_speaker_cache
 
 from .configuration_qwen3_tts import Qwen3TTSConfig, Qwen3TTSSpeakerEncoderConfig, Qwen3TTSTalkerConfig
 from .qwen3_tts_code_predictor_vllm import Qwen3TTSTalkerCodePredictorForConditionalGenerationVLLM
@@ -428,8 +428,7 @@ class Qwen3TTSTalkerForConditionalGeneration(nn.Module):
         self._tokenizer = None
         self._speech_tokenizer: Qwen3TTSTokenizer | None = None
 
-        # In-memory LRU cache for voice extraction artifacts (Base voice clone).
-        self._voice_cache = VoiceEmbeddingCache()
+        self._speaker_cache = get_speaker_cache()
         raw_subtalker_sampling = getattr(vllm_config.model_config, "subtalker_sampling_params", None)
         self._subtalker_sampling_params: dict[str, Any] = (
             dict(raw_subtalker_sampling) if isinstance(raw_subtalker_sampling, Mapping) else {}
@@ -1364,23 +1363,36 @@ class Qwen3TTSTalkerForConditionalGeneration(nn.Module):
             in_context_mode = not xvec_only
             voice_clone_prompt = _normalize_voice_clone_prompt(info_dict.get("voice_clone_prompt"))
 
-            # Voice cache: only for uploaded voices (created_at > 0)
-            _voice_cache_key = None
+            # Speaker cache: only for uploaded (named) speakers
+            _speaker_cache_key = None
             if voice_clone_prompt is None:
                 _speaker_list = info_dict.get("speaker")
                 if isinstance(_speaker_list, list) and _speaker_list:
                     _voice_name = str(_speaker_list[0]).lower()
-                    _voice_created_at = float((info_dict.get("voice_created_at") or [0])[0])
-                    if _voice_created_at > 0:
-                        _voice_cache_key = self._voice_cache.make_cache_key(_voice_name, xvec_only, _voice_created_at)
-                    _cached = self._voice_cache.get(_voice_cache_key) if _voice_cache_key is not None else None
+                    # Per-mode namespace — xvec and icl produce different artifacts
+                    # for the same voice, so they must not share a cache slot.
+                    _mode = "xvec" if xvec_only else "icl"
+                    _voice_created_at = int((info_dict.get("voice_created_at") or [0])[0])
+                    _speaker_cache_key = self._speaker_cache.make_cache_key(
+                        _voice_name,
+                        model_type=f"qwen3_tts_{_mode}",
+                        created_at=_voice_created_at,
+                    )
+                    _cached = self._speaker_cache.get(_speaker_cache_key)
                     if _cached is not None:
+                        # Transfer cached tensors to current device
+                        ref_code_cached = _cached.get("ref_code")
+                        ref_spk_embed_cached = _cached.get("ref_spk_embedding")
+                        if isinstance(ref_code_cached, torch.Tensor):
+                            ref_code_cached = ref_code_cached.to(device=input_ids.device)
+                        if isinstance(ref_spk_embed_cached, torch.Tensor):
+                            ref_spk_embed_cached = ref_spk_embed_cached.to(device=input_ids.device)
                         voice_clone_prompt = {
-                            "ref_code": _cached.get("ref_code"),
-                            "ref_spk_embedding": _cached.get("ref_spk_embedding"),
+                            "ref_code": ref_code_cached,
+                            "ref_spk_embedding": ref_spk_embed_cached,
                             "icl_mode": _cached.get("icl_mode"),
                         }
-                        _voice_cache_key = None  # hit -> don't store again
+                        _speaker_cache_key = None  # hit → don't store again
 
             # Official implementation may pass `voice_clone_prompt.icl_mode`.
             if voice_clone_prompt is not None and "icl_mode" in voice_clone_prompt:
@@ -1432,9 +1444,9 @@ class Qwen3TTSTalkerForConditionalGeneration(nn.Module):
                 speaker_embed = self._extract_speaker_embedding(wav_np, sr).view(1, 1, -1)
 
             # Cache miss: store extraction result
-            if _voice_cache_key is not None and speaker_embed is not None:
-                self._voice_cache.put(
-                    _voice_cache_key,
+            if _speaker_cache_key is not None and speaker_embed is not None:
+                self._speaker_cache.put(
+                    _speaker_cache_key,
                     {
                         "ref_code": ref_code_prompt.detach().cpu()
                         if isinstance(ref_code_prompt, torch.Tensor)

vllm_omni/model_executor/models/voxcpm2/voxcpm2_talker.py

@@ -34,6 +34,7 @@ from vllm.multimodal.audio import AudioResampler
 from vllm.sequence import IntermediateTensors
 
 from vllm_omni.model_executor.models.output_templates import OmniOutput
+from vllm_omni.utils.speaker_cache import get_speaker_cache
 
 from .minicpm4_paged import MiniCPM4PagedForVoxCPM2, MiniCPM4PagedResidualLM
 from .voxcpm2_import_utils import import_voxcpm2_core
@@ -446,6 +447,9 @@ class VoxCPM2TalkerForConditionalGeneration(nn.Module):
         self._max_decode_steps = 2000
         self._max_batch_size = getattr(vllm_config.scheduler_config, "max_num_seqs", 4)
 
+        # Speaker cache for ref_audio_feat across requests
+        self._speaker_cache = get_speaker_cache()
+
         self._perf = _PerfTimer(enabled=_ENABLE_PROFILING)
         self._cfm_buffers: _CFMBufferManager | None = None
         self._enable_cuda_graph = True
@@ -1165,15 +1169,48 @@ class VoxCPM2TalkerForConditionalGeneration(nn.Module):
                 prompt_text = prompt_text[0] if prompt_text else None
 
             state.prompt_cache = None
+            voice_name = info_dict.get("voice_name")
+            if isinstance(voice_name, list):
+                voice_name = voice_name[0] if voice_name else None
+            _created_at = int(info_dict.get("voice_created_at") or 0)
+
             if ref_audio or (prompt_audio and prompt_text):
-                try:
-                    state.prompt_cache = self._build_prompt_cache(
-                        ref_audio=ref_audio,
-                        prompt_audio=prompt_audio,
-                        prompt_text=prompt_text,
+                # Check speaker cache for reference-only mode
+                if voice_name and ref_audio and not prompt_audio:
+                    _cache_key = self._speaker_cache.make_cache_key(
+                        voice_name, model_type="voxcpm2", created_at=_created_at
                     )
-                except Exception as e:
-                    logger.warning("build_prompt_cache failed: %s", e)
+                    cached = self._speaker_cache.get(_cache_key)
+                    if cached is not None:
+                        state.prompt_cache = {
+                            "mode": "reference",
+                            "ref_audio_feat": cached["ref_audio_feat"].clone(),
+                        }
+                        logger.debug("Speaker cache HIT for VoxCPM2 speaker '%s'", voice_name)
+
+                if state.prompt_cache is None:
+                    try:
+                        state.prompt_cache = self._build_prompt_cache(
+                            ref_audio=ref_audio,
+                            prompt_audio=prompt_audio,
+                            prompt_text=prompt_text,
+                        )
+                        if (
+                            voice_name
+                            and state.prompt_cache is not None
+                            and state.prompt_cache.get("mode") == "reference"
+                            and "ref_audio_feat" in state.prompt_cache
+                        ):
+                            _key = self._speaker_cache.make_cache_key(
+                                voice_name, model_type="voxcpm2", created_at=_created_at
+                            )
+                            self._speaker_cache.put(
+                                _key, {"ref_audio_feat": state.prompt_cache["ref_audio_feat"].cpu()}
+                            )
+                            logger.debug("Speaker cache STORE for VoxCPM2 speaker '%s'", voice_name)
+                    except Exception as e:
+                        logger.warning("build_prompt_cache failed: %s; falling back to zero-shot", e)
+                        state.prompt_cache = None
 
             inputs = self._build_prefill_inputs(token_ids, dev, req_id)
             tts = self.tts

vllm_omni/utils/speaker_cache.py

@@ -0,0 +1,136 @@
+"""Process-wide thread-safe LRU cache for speaker extraction artifacts.
+
+Keyed by ``(model_type, speaker_name, created_at)`` so each upload generation
+has its own slot. Access via :func:`get_speaker_cache`.
+"""
+
+from __future__ import annotations
+
+import threading
+from collections import OrderedDict
+from typing import Any
+
+import torch
+from vllm.logger import init_logger
+
+logger = init_logger(__name__)
+
+_MAX_BYTES = 512 * 1024**2  # 512 MiB
+
+_SINGLETON: SpeakerEmbeddingCache | None = None
+_SINGLETON_LOCK = threading.Lock()
+
+
+def _estimate_tensor_bytes(obj: object) -> int:
+    if isinstance(obj, torch.Tensor):
+        return obj.numel() * obj.element_size()
+    if isinstance(obj, dict):
+        return sum(_estimate_tensor_bytes(v) for v in obj.values())
+    if isinstance(obj, (list, tuple)):
+        return sum(_estimate_tensor_bytes(item) for item in obj)
+    return 0
+
+
+class SpeakerEmbeddingCache:
+    """Thread-safe in-memory LRU cache for speaker extraction artifacts."""
+
+    def __init__(self, *, max_bytes: int = _MAX_BYTES):
+        self._cache: OrderedDict[tuple[str, str, int], dict[str, Any]] = OrderedDict()
+        self._sizes: dict[tuple[str, str, int], int] = {}
+        self._total_bytes = 0
+        self._lock = threading.Lock()
+        self._hits = 0
+        self._misses = 0
+        self._max_bytes = max_bytes
+        logger.info("Speaker cache ready (max_bytes=%d)", self._max_bytes)
+
+    @staticmethod
+    def make_cache_key(speaker_name: str, model_type: str, created_at: int = 0) -> tuple[str, str, int]:
+        """Build a cache key. ``created_at=0`` for built-in speakers (no upload).
+
+        Names are normalized (stripped + lowercased) so delete/clear paths that
+        normalize to lowercase match entries put with mixed-case names.
+        """
+        if not speaker_name or not speaker_name.strip():
+            raise ValueError("speaker_name is required")
+        if not model_type:
+            raise ValueError("model_type is required")
+        return (model_type, speaker_name.strip().lower(), int(created_at))
+
+    def get(self, key: tuple[str, str, int]) -> dict[str, Any] | None:
+        with self._lock:
+            if key in self._cache:
+                self._cache.move_to_end(key)
+                self._hits += 1
+                return self._cache[key]
+            self._misses += 1
+            return None
+
+    def put(self, key: tuple[str, str, int], artifacts: dict[str, Any]) -> None:
+        with self._lock:
+            self._insert_locked(key, artifacts)
+
+    def _insert_locked(self, key: tuple[str, str, int], artifacts: dict[str, Any]) -> None:
+        size = _estimate_tensor_bytes(artifacts)
+        if size > self._max_bytes:
+            logger.warning("Speaker cache skip: entry %s size=%dB exceeds max_bytes=%dB", key, size, self._max_bytes)
+            return
+        if key in self._cache:
+            self._total_bytes -= self._sizes.pop(key, 0)
+            del self._cache[key]
+        self._cache[key] = artifacts
+        self._sizes[key] = size
+        self._total_bytes += size
+        self._cache.move_to_end(key)
+        while self._cache and self._total_bytes > self._max_bytes:
+            evict_key, _ = self._cache.popitem(last=False)
+            self._total_bytes -= self._sizes.pop(evict_key, 0)
+            logger.debug("Speaker cache EVICT: key=%s", evict_key)
+
+    def clear(self, speaker_name: str | None = None) -> int:
+        """Remove entries. With a name, drops matches across model types and generations."""
+        with self._lock:
+            if speaker_name is None:
+                removed = len(self._cache)
+                self._cache.clear()
+                self._sizes.clear()
+                self._total_bytes = 0
+                self._hits = 0
+                self._misses = 0
+                return removed
+
+            if not speaker_name or not speaker_name.strip():
+                raise ValueError("speaker_name cannot be an empty string")
+            normalized = speaker_name.strip().lower()
+            removed = 0
+            for k in list(self._cache.keys()):
+                if isinstance(k, tuple) and len(k) >= 2 and k[1] == normalized:
+                    self._total_bytes -= self._sizes.pop(k, 0)
+                    del self._cache[k]
+                    removed += 1
+            return removed
+
+    def memory_bytes(self) -> int:
+        with self._lock:
+            return self._total_bytes
+
+    def stats(self) -> dict[str, Any]:
+        with self._lock:
+            return {
+                "entries": len(self._cache),
+                "memory_bytes": self._total_bytes,
+                "max_bytes": self._max_bytes,
+                "memory_mb": round(self._total_bytes / (1024 * 1024), 2),
+                "hits": self._hits,
+                "misses": self._misses,
+            }
+
+
+def get_speaker_cache() -> SpeakerEmbeddingCache:
+    """Return the process-wide speaker cache singleton."""
+    global _SINGLETON
+    if _SINGLETON is None:
+        with _SINGLETON_LOCK:
+            if _SINGLETON is None:
+                _SINGLETON = SpeakerEmbeddingCache()
+    return _SINGLETON

vllm_omni/utils/voice_cache.py

@@ -1,89 +0,0 @@
-"""In-memory LRU cache for voice extraction artifacts.
-
-Keyed by voice name + extraction mode (e.g. ``"alice:icl"``).
-Only named voices are cached; inline ``ref_audio`` without a voice
-name is not cached.
-
-Usage::
-
-    key = VoiceEmbeddingCache.make_cache_key("alice", xvec_only=False)
-    cached = cache.get(key)
-    if cached is None:
-        # ... extract ...
-        cache.put(key, {"artifact": result})
-"""
-
-import os
-import threading
-from collections import OrderedDict
-from typing import Any
-
-from vllm.logger import init_logger
-
-logger = init_logger(__name__)
-
-_DEFAULT_MAX_ENTRIES = 128
-
-
-class VoiceEmbeddingCache:
-    """LRU cache for voice extraction outputs.
-
-    Each entry stores a ``dict[str, Any]`` whose contents are model-specific.
-    Thread-safe via a lightweight ``threading.Lock``.
-    """
-
-    def __init__(self, max_entries: int | None = None):
-        if max_entries is None:
-            max_entries = int(os.environ.get("VOICE_CACHE_MAX_ENTRIES", _DEFAULT_MAX_ENTRIES))
-        self._cache: OrderedDict[str, dict[str, Any]] = OrderedDict()
-        self._max_entries = max_entries
-        self._lock = threading.Lock()
-        self._hits = 0
-        self._misses = 0
-        logger.info("Voice embedding cache initialized (max_entries=%d)", max_entries)
-
-    @staticmethod
-    def make_cache_key(voice_name: str, xvec_only: bool, created_at: float = 0.0) -> str:
-        """Build a cache key from a voice name, upload timestamp, and extraction mode.
-
-        Args:
-            voice_name: The speaker/voice name (case-insensitive, lowered
-                by the caller).
-            xvec_only: True for speaker-embedding-only mode, False for
-                ICL mode (speaker embedding + ref_code).
-            created_at: Upload timestamp from metadata. Prevents stale cache
-                hits after a voice is deleted and re-uploaded with the same
-                name but different audio.
-        """
-        mode = "xvec" if xvec_only else "icl"
-        return f"{voice_name}:{created_at:.6f}:{mode}"
-
-    def get(self, key: str) -> dict[str, Any] | None:
-        """Return cached artifacts or ``None`` on miss.  Promotes to MRU on hit."""
-        with self._lock:
-            if key in self._cache:
-                self._cache.move_to_end(key)
-                self._hits += 1
-                logger.debug("Voice cache HIT (key=%s, hits=%d)", key, self._hits)
-                return self._cache[key]
-            self._misses += 1
-            return None
-
-    def put(self, key: str, artifacts: dict[str, Any]) -> None:
-        """Store *artifacts* under *key*, evicting the LRU entry if full."""
-        with self._lock:
-            self._cache[key] = artifacts
-            self._cache.move_to_end(key)
-            while len(self._cache) > self._max_entries:
-                evicted_key, _ = self._cache.popitem(last=False)
-                logger.debug("Voice cache EVICT (key=%s)", evicted_key)
-
-    def stats(self) -> dict[str, int]:
-        """Return cache statistics."""
-        with self._lock:
-            return {
-                "entries": len(self._cache),
-                "max_entries": self._max_entries,
-                "hits": self._hits,
-                "misses": self._misses,
-            }