Compare commits

..
Author SHA1 Message Date
e04210d690 [Hardware][MUSA] Enable Qwen3-Omni ModelOpt FP8 inference (#5671)
Signed-off-by: Xiaodong Ye <xiaodong.ye@mthreads.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
Co-authored-by: amy-why-3459 <wuhaiyan17@huawei.com>
2026-08-11 17:19:09 +08:00
11 changed files with 126 additions and 96 deletions

No files matched your search

+1 -4
View File
@@ -1,6 +1,6 @@
---
name: precheck-pr
description: Self-check your branch before creating a PR — catch dead code, prevent new model-specific Python examples, verify accuracy/perf claims, validate PR title format, and confirm merge readiness. Use when the user says "precheck", "self review", "pre-submit check", or "check my PR before I open it." Never posts to GitHub.
description: Self-check your branch before creating a PR — catch dead code, verify accuracy/perf claims, validate PR title format, and confirm merge readiness. Use when the user says "precheck", "self review", "pre-submit check", or "check my PR before I open it." Never posts to GitHub.
---
# PR Pre-Check
@@ -65,8 +65,6 @@ Ask: "Quick mode or full mode?" Then walk the checklist for the detected PR type
**Also run the Code-Quality sweep on every PR**, regardless of type or mode: the five diff-scoped checks in [references/code-quality.md](references/code-quality.md) — kwargs fragility, broad-except swallow, `Any`/wrong type hints, hot-path `.clone()`/`deepcopy`, and event-loop blocking — plus the advisory conventions (log level, structured logging, synchronization, cleanup, dependencies, naming) in [checklists.md](references/checklists.md). These count only lines the PR *adds* — the pre-existing backlog across the repo is out of scope.
**Also run the Examples-Policy check on every PR** using [references/examples-policy.md](references/examples-policy.md). Inspect only Python paths introduced by the diff. Treat a new model-, checkpoint-, vendor-, or family-specific Python example as blocking; do not report pre-existing example debt.
### Step 5: Print Report
```
@@ -79,7 +77,6 @@ Pre-check report for <branch>
───────────────── ──────
PR title format ✓
Code quality ⚠ 1 broad except, 2 Any hints
Examples policy ✗ new model-specific Python example
PR desc integrity ✓
Registry/config ✓
Dead code ⚠ 2 warnings
@@ -47,19 +47,6 @@ Diff-scoped sweep for five fragility patterns that are pervasive in the existing
Roll the ⚠/✗ counts into the report as a single **Code quality** dimension row.
---
## All PRs: Examples Policy (run on every PR, quick & full)
Inspect Python files introduced under `examples/` using
[examples-policy.md](examples-policy.md). This is a diff-scoped ratchet: existing
model-specific examples may be edited or removed without triggering the check.
- [ ] **No new model-specific Python examples:** the diff adds, copies, or renames no Python file whose path or behavior is specific to one model, checkpoint, vendor, or model family. ✗ for any violation. Route inference through a shared task/protocol entrypoint, model request behavior through `vllm_omni/model_extras`, and commands through task docs or `recipes/`.
- [ ] **New generic example code is justified:** any new task-neutral Python file under `examples/` is an actual user-facing entrypoint rather than reusable production logic, a benchmark, an app, a setup tool, or a reproducer. ⚠ when the placement is questionable.
Roll the result into the report as a separate **Examples policy** dimension.
### Conventions (eyeball — no grep)
Judgment calls that don't grep cleanly — apply them while reading the diff, not as a mechanical check:
@@ -1,68 +0,0 @@
# Examples Python Policy
Prevent new model-specific Python examples without forcing cleanup of the
existing backlog. Run this check on every PR in quick and full modes.
## Scope
Set the merge base as in the main workflow, then list Python paths introduced
by the PR, including additions, copies, and renames:
```bash
BASE="$(git merge-base HEAD origin/main)"
git diff --name-status --diff-filter=ACR --find-renames --find-copies \
"${BASE}"...HEAD -- 'examples/**/*.py'
```
Inspect destination paths for copies and renames. Existing model-specific files
that are only modified or deleted are grandfathered and must not be reported.
## Blocking classification
Mark ✗ when a newly introduced Python file does any of the following:
- Names or scopes a path to one model, checkpoint, vendor, or model family.
- Hard-codes a model identifier, model-specific prompt, request shape, output
conversion, or launch configuration behind an otherwise generic filename.
- Duplicates an existing task/protocol runner to demonstrate one model.
- Adds a per-model helper below a generic task hub such as `text_to_speech/`.
Do not infer safety from a generic filename alone. For example, a file named
`text_to_audio.py` is still model-specific if it only implements one model's
contract.
## Acceptable additions
A new Python example may pass when it is a genuinely model-neutral,
user-facing task or protocol entrypoint. It must accept model selection through
configuration and keep model-specific behavior out of the script.
Use ⚠ when the file is model-neutral but reusable logic appears misplaced under
`examples/`. Route code by purpose:
| Code purpose | Preferred location |
|---|---|
| Model prompt, defaults, request/output adaptation | `vllm_omni/model_extras/` or production model modules |
| Runnable model command and validation evidence | Task documentation or `recipes/` |
| UI or interactive application | `apps/` |
| Benchmark or evaluator | `benchmarks/` |
| Regression reproducer | `tests/` |
| Download, conversion, or setup utility | `tools/` |
If a model cannot use an existing shared runner, report the missing generic
capability. Do not solve the gap by adding a model-specific Python example.
## Report format
Add one row to the pre-check report:
```text
Examples policy ✓ no new Python example paths
Examples policy ⚠ generic helper belongs in tools/
Examples policy ✗ examples/offline_inference/new_model/end2end.py is model-specific
Examples policy ✗ examples/offline_inference/x_to_y_model_name.py is model-specific
```
For a blocker, name the path, the model-specific behavior, and the appropriate
shared runner or destination. Do not propose deleting unrelated existing
examples as part of the PR.
-8
View File
@@ -97,7 +97,6 @@ If docs and live code disagree, verify the code/tests and report the drift.
| [test-quality-evaluation.md](references/checks/test-quality-evaluation.md) | Tests change, are absent for risky code, or may not exercise production behavior. |
| [tests-docs-checklist.md](references/checks/tests-docs-checklist.md) | Coverage, CI markers, examples, user docs, or PR evidence need review. |
| [verification.md](references/checks/verification.md) | Hardware, a server, or a runnable affected path is available for active verification. |
| [examples-policy.md](../precheck-pr/references/examples-policy.md) | The PR adds, copies, or renames Python under `examples/`; apply the canonical policy shared with `precheck-pr`. |
### Delivery and reviewer coordination
@@ -153,13 +152,6 @@ contract they protect or use only the applicable evidence checks.
Apply every category in [general-checks.md](references/process/general-checks.md) before
lower-priority comments.
If the diff census contains an added, copied, or renamed Python path under
`examples/`, read and apply the canonical
[examples policy](../precheck-pr/references/examples-policy.md). Treat a new
model-specific Python example as blocking. Do not flag model-specific example
debt that the PR only modifies or removes, and do not run the rest of the
author-oriented `precheck-pr` workflow.
For each changed value or behavior, trace:
```text
+18
View File
@@ -2206,6 +2206,24 @@ class TestBaseConfigInheritance:
class TestPlatformOverrides:
"""Test platform-specific deploy config overrides."""
def test_qwen3_omni_musa_clears_parent_quantization_for_audio_stages(self):
pipeline = resolve_pipeline_config(
"qwen3_omni_moe",
Q3_OMNI_ALL_STAGES_HF_CONFIG,
)
assert isinstance(pipeline, PipelineConfig)
deploy_path = Path(get_deploy_config_path("qwen3_omni_moe.yaml"))
base_stages = merge_pipeline_deploy(pipeline, load_deploy_config(deploy_path))
assert "hf_overrides" not in base_stages[1].yaml_engine_args
assert "hf_overrides" not in base_stages[2].yaml_engine_args
musa = _apply_platform_overrides(load_deploy_config(deploy_path), platform="musa")
musa_stages = merge_pipeline_deploy(pipeline, musa)
expected = {"quantization_config": None}
assert musa_stages[1].yaml_engine_args["hf_overrides"] == expected
assert musa_stages[2].yaml_engine_args["hf_overrides"] == expected
def test_qwen3_tts_rocm_disables_code2wav_outer_cudagraph(self):
deploy_path = Path(get_deploy_config_path("qwen3_tts.yaml"))
+76 -1
View File
@@ -3,13 +3,88 @@ from types import SimpleNamespace
import pytest
import torch
from vllm.v1.cudagraph_dispatcher import CUDAGraphMode
from vllm_omni.worker.gpu_model_runner import OmniGPUModelRunner, _filter_mrope_kwargs_for_model
from vllm_omni.worker.gpu_model_runner import (
OmniGPUModelRunner,
_filter_mrope_kwargs_for_model,
)
from vllm_omni.worker.omni_connector_model_runner_mixin import OmniConnectorModelRunnerMixin
pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
def _runner_for_talker_graph_init(
*,
talker_mtp_graph_safe: bool | None,
has_separate_talker: bool = True,
model_stage: str = "talker",
) -> OmniGPUModelRunner:
runner = object.__new__(OmniGPUModelRunner)
runner.model = SimpleNamespace(
talker=object() if has_separate_talker else None,
talker_mtp=object(),
model_stage=model_stage,
)
if talker_mtp_graph_safe is not None:
runner.model.talker_mtp_graph_safe = talker_mtp_graph_safe
runner.model_config = SimpleNamespace(hf_text_config=SimpleNamespace(hidden_size=4))
runner.compilation_config = SimpleNamespace(
cudagraph_mode=CUDAGraphMode.FULL,
max_cudagraph_capture_size=1,
)
runner.vllm_config = object()
runner.max_num_reqs = 1
runner.dtype = torch.float32
runner._make_buffer = lambda *args, **kwargs: SimpleNamespace(args=args, kwargs=kwargs)
return runner
def test_talker_mtp_skips_graph_when_model_declares_unsafe(monkeypatch):
runner = _runner_for_talker_graph_init(talker_mtp_graph_safe=False)
talker_mtp = runner.model.talker_mtp
monkeypatch.setattr(
"vllm_omni.worker.gpu_model_runner.current_omni_platform.get_graph_wrapper_cls",
lambda: pytest.fail("graph wrapper must not be selected"),
)
OmniGPUModelRunner._init_talker_mtp(runner)
assert runner.talker_mtp is talker_mtp
@pytest.mark.parametrize("model_stage", ["thinker", "code2wav"])
def test_non_talker_stage_does_not_use_talker_mtp_graph(monkeypatch, model_stage: str):
runner = _runner_for_talker_graph_init(
talker_mtp_graph_safe=None,
has_separate_talker=False,
model_stage=model_stage,
)
talker_mtp = runner.model.talker_mtp
monkeypatch.setattr(
"vllm_omni.worker.gpu_model_runner.current_omni_platform.get_graph_wrapper_cls",
lambda: pytest.fail("graph wrapper must not be selected"),
)
OmniGPUModelRunner._init_talker_mtp(runner)
assert runner.talker_mtp is talker_mtp
@pytest.mark.parametrize("talker_mtp_graph_safe", [None, True])
def test_talker_mtp_uses_graph_for_legacy_or_explicit_safe_model(monkeypatch, talker_mtp_graph_safe):
runner = _runner_for_talker_graph_init(talker_mtp_graph_safe=talker_mtp_graph_safe)
wrapped = object()
monkeypatch.setattr(
"vllm_omni.worker.gpu_model_runner.current_omni_platform.get_graph_wrapper_cls",
lambda: lambda *args, **kwargs: wrapped,
)
OmniGPUModelRunner._init_talker_mtp(runner)
assert runner.talker_mtp is wrapped
class DummyBuffer:
"""A minimal buffer wrapper that exposes the `.gpu` attribute."""
+12
View File
@@ -69,6 +69,18 @@ stages:
repetition_penalty: 1.1
platforms:
# The supported Qwen3-Omni ModelOpt checkpoint quantizes the thinker only. Clear the
# checkpoint-level metadata for the BF16 audio stages on the verified MUSA
# profile before vLLM performs its normal quantization auto-detection.
musa:
stages:
- stage_id: 1
hf_overrides:
quantization_config: null
- stage_id: 2
hf_overrides:
quantization_config: null
npu:
stages:
- stage_id: 0
@@ -152,6 +152,9 @@ class Qwen3OmniMoeForConditionalGeneration(
dtype=torch.long,
)
elif self.model_stage == "talker":
# The outer wrapper exposes talker_mtp for every stage, but only
# the talker stage owns the module that the method invokes.
self.talker_mtp_graph_safe = current_omni_platform.supports_talker_mtp_graph_capture()
multimodal_config.skip_mm_profiling = True
self.has_preprocess = True
self.has_postprocess = True
+5
View File
@@ -155,6 +155,11 @@ class OmniPlatform(Platform):
"""Check if the platform supports torch.compile with inductor backend."""
raise NotImplementedError
@classmethod
def supports_talker_mtp_graph_capture(cls) -> bool:
"""Whether a model may capture its dedicated talker MTP graph."""
return True
@classmethod
def has_flash_attn_package(cls) -> bool:
"""Check if a Flash Attention package is available and usable on this platform."""
+5
View File
@@ -115,6 +115,11 @@ class MUSAOmniPlatform(OmniPlatform, MUSAPlatformBase):
"""MUSA supports torch.compile with inductor backend."""
return True
@classmethod
def supports_talker_mtp_graph_capture(cls) -> bool:
"""MUSA keeps Qwen3 talker MTP outside its dedicated FULL graph."""
return False
@classmethod
def supports_float64(cls) -> bool:
"""MUSA does not support float64 yet."""
+6 -2
View File
@@ -206,8 +206,12 @@ class OmniGPUModelRunner(GPUModelRunner):
cudagraph_mode = self.compilation_config.cudagraph_mode
assert cudagraph_mode is not None
has_separate_talker = getattr(self.model, "talker", None) is not None
talker_mtp_graph_safe = getattr(self.model, "talker_mtp_graph_safe", False)
if cudagraph_mode.has_full_cudagraphs() and (has_separate_talker or talker_mtp_graph_safe):
talker_mtp_graph_safe = getattr(self.model, "talker_mtp_graph_safe", None)
# Preserve the legacy separate-talker default only when the model has
# not declared graph safety. An explicit False must disable wrapping.
if talker_mtp_graph_safe is None:
talker_mtp_graph_safe = has_separate_talker
if cudagraph_mode.has_full_cudagraphs() and talker_mtp_graph_safe:
graph_wrapper_cls = current_omni_platform.get_graph_wrapper_cls()
self.talker_mtp = graph_wrapper_cls(talker_mtp, self.vllm_config, runtime_mode=CUDAGraphMode.FULL)
# TTS exposes mtp_hidden_size; Omni uses hf_text_config.hidden_size.