apply() already tokenized the prompt to build the engine prompt ids, but the
shared OmniMultiModalProcessor._apply_hf_processor_main then called
_call_hf_processor again, which rebuilt the control-token prompt and ran the
text tokenizer a second time before encoding the reference audio. Upstream's
_cached_apply_hf_processor invokes _apply_hf_processor_main for every
request, so this also happened for text-only requests and for requests whose
reference audio was already cached.
Split the old _call_hf_processor into _encode_prompt_text (used once, in
apply) and _encode_reference_audio, and override _apply_hf_processor_main to
encode the reference audio only, returning just the passthrough data when
there is no audio. _get_mm_fields_config registers only ref_audio_tokens and
ref_audio_len, so dropping input_ids from the media result changes no
consumer. The now-unreferenced _call_hf_processor is deleted; the prompt-text
kwarg transport is left as is.
Validation: ruff and pre-commit on both files; the module imports cleanly;
new tests/model_executor/models/omnivoice/test_processor_prompt_tokenization.py
covers text-only apply (one tokenizer call), the audio path (audio encoded,
no re-tokenization) and the empty-items path (no tokenizer, no encoder);
tests/model_executor/models/omnivoice (106), tests/inputs/test_mm_processor.py
and tests/entrypoints/openai_api/test_tts_{adapter,detection}.py (162) pass.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MpXe2LeRrUQGjZfKFaocMZ
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
The vLLM 0.29 rewrite left Qwen3OmniMoeThinkerMultiModalProcessor with a
_maybe_apply_prompt_updates that is identical to the one on its first base,
Omni's Qwen2_5OmniThinkerMultiModalProcessor (same body modulo docstring).
The MRO is Omni Qwen3 -> Omni Qwen2.5 -> upstream Qwen3 -> upstream Qwen2.5
-> BaseMultiModalProcessor, so deleting the override resolves to the same
code; asserted with
Qwen3OmniMoeThinkerMultiModalProcessor._maybe_apply_prompt_updates is
Qwen2_5OmniThinkerMultiModalProcessor._maybe_apply_prompt_updates.
Validation: ruff and pre-commit on the file; the MRO assertion above;
tests/model_executor/models/{qwen3_omni,qwen2_5_omni} pass (20 + 81 tests).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MpXe2LeRrUQGjZfKFaocMZ
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
After the vLLM 0.29 adaptation the override only returned
super()._cached_apply_hf_processor(inputs, timing_ctx). Delete it together
with the imports it alone used (MultiModalProcessingInfo, TimingContext).
The find-only _maybe_apply_prompt_updates override is unchanged.
Validation: ruff and pre-commit on the file; the module imports cleanly;
tests/model_executor/models/voxtral_tts passes (62 tests).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MpXe2LeRrUQGjZfKFaocMZ
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
vLLM 0.29 removed BaseMultiModalProcessor._call_hf_processor; the pipeline
now reaches the HF processor through _apply_hf_processor_main and the
_preprocess_hf_mm_data / _postprocess_hf_mm_data hooks. These three
overrides still called super()._call_hf_processor, which would raise
AttributeError if anything reached them:
- bagel: _apply_hf_processor_main already performs the image / img2img
split via self.info.ctx.call_hf_processor, so the override was unreachable.
- mimo_audio_llm: the audios -> audio rename and the sampling_rate kwarg
already live in _preprocess_hf_mm_data.
- step_audio2: the audios -> audio normalisation and the empty
audio_mels / audio_lens defaults already live in _preprocess_hf_mm_data
and _postprocess_hf_mm_data.
Caller audit: the only remaining _call_hf_processor call sites are the live
ones in vllm_omni/inputs/mm_processor.py, omnivoice.py and the GLM-TTS test.
Validation: ruff and pre-commit on the three files; the three modules
import cleanly; CPU tests in tests/inputs, tests/model_executor/models/
{mimo_audio,step_audio2} and test_upstream_prompt_update_api.py pass.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MpXe2LeRrUQGjZfKFaocMZ
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
vLLM 0.29 deleted BaseMultiModalProcessor._hf_processor_applies_updates; no
reference to the hook remains anywhere in vllm/, so these six overrides were
never called. Delete them. No behaviour change.
Validation: pre-commit on the six files; CPU tests in tests/inputs and
tests/model_executor/models/{cosyvoice3,mimo_audio,step_audio2,omnivoice},
test_glm_tts_prompt_len.py and test_upstream_prompt_update_api.py pass;
the six processor modules import cleanly.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MpXe2LeRrUQGjZfKFaocMZ
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
Reuse upstream weight mapping and multimodal processing, consolidate Omni processor bridges, and trim unused Hunyuan internals. Route no-media kwargs through the active renderer and preserve GLM generation prompt formatting.
Validation: 453 CPU tests passed, 7 skipped; processor imports, Ruff, and diff checks passed. GPU and end-to-end CI not rerun.
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
vLLM 0.29 renamed the audio attention projection from qkv to qkv_proj, and
its stacked mapping now rewrites checkpoint q_proj/k_proj/v_proj onto that
name. Omni subclasses the upstream audio attention and mirrors its __init__
to add quant_config, so it kept building self.qkv and weight loading died
with "There is no module or parameter named layers.0.self_attn.qkv_proj in
Qwen3OmniMoeAudioEncoder ... available: self_attn.qkv.weight". The inherited
forward calls self.qkv_proj too, so the old name broke both paths.
This is a second, independent cause behind the Omni Accuracy Test failure:
the shutdown-timeout signature also appeared in that job, which is why the
earlier pass attributed it solely to the shutdown fix. Both builds 3009 and
3010 carry this weight-loading error, so it predates that pass.
Not reproducible locally: it needs the 30B checkpoint. The rename mirrors
upstream Qwen3OmniMoeAudioAttention exactly, and the 20 Qwen3-Omni unit tests
still pass.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0146Xu5Sgj4vXxLxvLjV5YPj
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
Five further removals/renames, each traced from a failing 3009 job:
- StageEngineCoreProcManager mirrors the parent __init__ rather than calling
super(), and 0.29 added _request_shutdown_timeout there. The inherited
shutdown() read it, raised AttributeError, and left the engine core
subprocesses running, so the test process hung at interpreter exit until
the 15 minute job timeout. That is the whole of the Engine&Model Executor
exit-124: its pytest reported 567 passed, then sat idle for 663s. The
failure surfaced only as a StagePool warning, never as a test failure.
- PromptUpdateDetails.select_text is gone; prompt updates are token oriented
now. MiniCPM-o 4.5 kept three text-based image/video/audio replacements,
which crashed stage-0 startup and account for the whole MiniCPM-o cluster.
They encode the text and select the unk placeholder by token id instead.
- Qwen3-Omni: 0.29 marks audio_feature_lengths keep_on_cpu, while the audio
tower derives device placement from feature_lens, so the mask comparison
mixed cuda and cpu tensors. Moved explicitly, matching upstream
Qwen3OmniMoeThinker._process_audio_input.
- AttentionBackend.get_kv_cache_shape is gone; the per-layer shape comes from
the spec via compute_layer_kv_cache_shape_bytes, whose content dim is in
bytes.
- WeightsMapper.get_unstacked_mapper is renamed to get_rename_mapper.
tests/engine/test_async_omni_engine_abort.py::test_abort fails here both with
and without these changes, so it is environmental rather than a regression.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0146Xu5Sgj4vXxLxvLjV5YPj
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
Four unrelated 0.29 surface changes, each reproduced locally against the
pinned f5c3cc24 build:
- PromptUpdateDetails.is_embed now takes only the full token list. The
Bagel img2img replacement still declared (tok, seq), which crashed
stage-0 engine core startup and produced the Distributed and Qwen3-Omni
"no live replica" failures rather than anything KV related.
- dispatch_unquantized_gemm gained a linear_backend argument and is called
with it. The CPU-forcing test stubs took no arguments; they now accept
and ignore whatever upstream passes so a further signature change does
not break them again.
- VocabParallelEmbedding gained a fused kernel that engages whenever
tp_size > 1 on CUDA, and it has no CPU implementation. The encoder TP
fixtures drive CPU tensors under a faked TP group, so they pin the
portable masked path.
- The pyav video decoder backend was removed; valid options are now
opencv, torchcodec, pynvvideocodec and deepstream. The OpenAI video
endpoint hardcoded pyav, which turned every video request into a 400.
It moves to opencv, the upstream default.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0146Xu5Sgj4vXxLxvLjV5YPj
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
