vllm-omni

0 Fork 0

Files

T

tzhouamandClaude Fable 5.1 94485152d2 [Edge][CPU] Predictor fast-path modes: torch.ao int8 (default), torchao weight-only + compiled step, fp32 control; parity metrics

torchao's dynamic-activation int8 is numerically wrong under torch.compile on
CPU (cosine 0.02 vs fp32; caused runaway generations), so it is not offered;
weight-only + compile is faithful but slower (78 ms vs 36 ms per frame for the
15 sub-steps), so torch.ao dynamic int8 stays the default. Parity mode now also
records top-5 overlap and KL vs the bf16 model; compiled-step equivalence test.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

2026-09-10 02:44:29 +08:00

model_loader

[Ming] Cleanup Ming-family shared modules (#6119 )

2026-08-19 14:13:40 +08:00

models

[Edge][CPU] Predictor fast-path modes: torch.ao int8 (default), torchao weight-only + compiled step, fp32 control; parity metrics

2026-09-10 02:44:29 +08:00

quantization

[Quantization][Qwen3-Omni] Enable NVFP4 W4A4 serving on Blackwell (#4025 )

2026-06-18 10:01:46 +08:00

stage_input_processors

[Model] Preserve MiniCPM-o Talker full attention until capacity (#6799 )