vllm-omni

0 Fork 0

Files

T

tzhouamandClaude Fable 5.1 82955b3a22 [Edge][CPU] Default the code predictor's CPU fast path to fp32 + compiled static step; int8 opt-in

In-situ parity against the bf16 model (VLLM_OMNI_PREDICTOR_PARITY_CHECK=1):
fp32 control keeps 93.8 % top-1 / 95 % top-5 / KL 0.0015; torch.ao dynamic
int8 falls to 22 % top-1 (max logit diff 7), so per-tensor activation
quantization is not acceptable as a default. Adds the ao-nohead variant
(int8 transformer body, fp32 lm heads) for measurement.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

2026-09-10 02:54:53 +08:00

layers

Add NPU support for OmniMRotaryEmbedding (#3609 )

2026-06-18 16:58:40 +08:00

model_loader

[Ming] Cleanup Ming-family shared modules (#6119 )

2026-08-19 14:13:40 +08:00

models

[Edge][CPU] Default the code predictor's CPU fast path to fp32 + compiled static step; int8 opt-in

2026-09-10 02:54:53 +08:00

stage_input_processors

[Model] Preserve MiniCPM-o Talker full attention until capacity (#6799 )