[Edge][CPU] Fused norms off by default, and register the switches this branch added
Two corrections and the paperwork they imply.
The fused-RMSNorm registration defaulted to *on*, which contradicts the
measurement it was committed with: those kernels are 3.6x and 4.5x faster in
isolation but measured 72.7 tok/s against 74.0 end to end, because inductor
already fuses the native norm into the residual add and an opaque call breaks
that fusion. The implementations stay registered, so the provider remains
selectable through `kernel_config.ir_op_priority` on a machine where the trade
lands differently; only the default ranking is left alone now, and the test
asserts both states rather than just the opt-in one.
The four switches this branch added (VLLM_OMNI_AOT_CACHE_SHIM,
_CPU_FUSED_NORMS, _CPU_RELEASE_MEMORY, _SPARK_GATE_REDUCTION) were not in the
reviewed environment-variable inventory, so
`test_statically_resolvable_environment_accesses_are_classified` failed. They
are classified now, the reviewed count moves 28 -> 32 with a note saying why,
and each has a row in docs/configuration/environment_variables.md carrying the
measurement that justifies its default.
`VLLM_OMNI_CPU_FAST_SAMPLER` and `VLLM_OMNI_STEP_STATS_SYNC` remain
unclassified and that test still fails on them. They arrived in 6ce8de1b, in
files this branch does not touch, so they are left to whoever owns that work
rather than documented from a guess.
Under `-m "core_model and cpu"`: 2593 passed before these fixes with 9
failures, of which two were mine. The rest are CUDA-only tests in a CPU venv
("Torch not compiled with CUDA enabled") plus test_qnn_export.py, which
another session is actively editing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
c440616c8b
commit
c4e61133fb
6 files changed
+107
-7
No files matched your search
@@ -79,11 +79,25 @@ def test_declines_arguments_the_kernel_cannot_take():
|
||||
assert not impl.supports_args(x, w.float(), 1e-6)
|
||||
|
||||
|
||||
def test_priority_patch_is_idempotent_and_puts_us_first():
|
||||
def test_priority_patch_is_opt_in_and_idempotent(monkeypatch):
|
||||
"""Off by default, because that is what the measurement says.
|
||||
|
||||
In isolation these kernels are 3.6x and 4.5x faster than the native path;
|
||||
in the compiled model they measured 72.7 tok/s against 74.0, because
|
||||
inductor already fuses the native norm into the residual add. So the
|
||||
registration always happens -- the provider stays selectable through
|
||||
`kernel_config.ir_op_priority` -- but the default ranking does not change
|
||||
unless it is asked for.
|
||||
"""
|
||||
if not cpu_ir_ops.CPU_FUSED_NORMS:
|
||||
pytest.skip(_REASON)
|
||||
from vllm.platforms.cpu import CpuPlatform
|
||||
|
||||
monkeypatch.delenv("VLLM_OMNI_CPU_FUSED_NORMS", raising=False)
|
||||
cpu_ir_ops.prefer_cpu_fused_norms()
|
||||
assert CpuPlatform.get_default_ir_op_priority(None).rms_norm == ["native"]
|
||||
|
||||
monkeypatch.setenv("VLLM_OMNI_CPU_FUSED_NORMS", "1")
|
||||
cpu_ir_ops.prefer_cpu_fused_norms()
|
||||
cpu_ir_ops.prefer_cpu_fused_norms() # second call must not double-wrap
|
||||
priority = CpuPlatform.get_default_ir_op_priority(None)
|
||||
|
||||
Reference in new issue
Block a user