[CI][Voxtral] Resolve failing test_prepare_speech_generation_awaits_voxtral_async (#7601)

Signed-off-by: Clodagh Walsh <clodaghwalsh17@gmail.com>

tests/entrypoints/openai_api/test_serving_speech.py

@@ -5150,6 +5150,7 @@ class TestTTSAsyncOffloading:
             models=mock_models,
             request_logger=mocker.MagicMock(),
         )
+        server.uploaded_speakers = {}
         yield server
         server.shutdown()