[Model] Add native MAGI-2 Preview diffusion support (#5918)
Signed-off-by: Hongsheng Liu <liuhongsheng4@huawei.com> Signed-off-by: hsliuustc0106 <liuhongsheng4@huawei.com>
This commit is contained in:
1 parent
13c617a140
commit
d4096e01fe
78 files changed
+9896
-459
No files matched your search
@@ -9,7 +9,9 @@ Three caching strategies:
|
||||
- **TaylorSeer**: Calibration-based prediction using Taylor expansion to estimate block outputs
|
||||
- **SCM** (Step Computation Masking): Dynamic step skipping based on configurable policies
|
||||
|
||||
**Typical speedup**: 1.5-2.5x depending on model and configuration.
|
||||
Speedup is model-, step-, and policy-dependent. Treat a cache-enabled run as
|
||||
integration evidence until logs or a focused test show real hits and repeated
|
||||
timings show an improvement over the uncached baseline.
|
||||
|
||||
**Official docs**: https://docs.vllm.ai/projects/vllm-omni/en/latest/design/feature/cache_dit
|
||||
|
||||
@@ -243,8 +245,10 @@ vllm serve your-model --omni --port 8098 \
|
||||
|
||||
**Verification checklist**:
|
||||
1. Logs show "Cache-dit enabled successfully on xxx"
|
||||
2. Performance: 1.5-2x speedup vs no cache
|
||||
3. Quality: compare output with `cache_backend=None`
|
||||
2. A focused test or summary confirms at least one real cache hit
|
||||
3. Quality is compared with `cache_backend=None`
|
||||
4. Performance uses a realistic step count, repeated timings, and a policy
|
||||
whose warmup does not consume the whole run
|
||||
|
||||
## Excluded Models
|
||||
|
||||
|
||||
@@ -16,7 +16,7 @@ vllm_omni/diffusion/models/wan2_2/
|
||||
|
||||
The transformer is loaded separately via `weights_sources` + `load_weights()`. Non-transformer components (VAE, text encoder) are loaded in `__init__` via `from_pretrained()`.
|
||||
|
||||
### Custom model with external deps
|
||||
### External adapter (not native support)
|
||||
|
||||
```
|
||||
vllm_omni/diffusion/models/<name>/
|
||||
@@ -28,7 +28,10 @@ examples/offline_inference/<task>/
|
||||
└── download_<name>.py # Downloads weights and/or clones code repo
|
||||
```
|
||||
|
||||
All weights loaded eagerly in `__init__`. `load_weights()` is a no-op. External dependency imported with try/except.
|
||||
All weights are loaded eagerly in `__init__`. `load_weights()` is a no-op. The
|
||||
external dependency is imported with try/except. Use this pattern only when the
|
||||
requested scope explicitly allows an adapter; it does not satisfy a request for
|
||||
a native vLLM-Omni implementation.
|
||||
|
||||
### Custom model with ported code (e.g., BAGEL)
|
||||
|
||||
@@ -86,7 +89,7 @@ init → load ALL weights eagerly via custom helpers → load_weights() = no-op
|
||||
- `load_weights()` is `pass`
|
||||
- Weights may come from multiple HF repos in different formats (`.pth`, `.safetensors`)
|
||||
|
||||
Use this when:
|
||||
Use this only for an explicitly approved external adapter when:
|
||||
- The original model has complex, well-tested loading code you don't want to rewrite
|
||||
- Weights span multiple HF repos
|
||||
- Weight format is non-standard (e.g., a single `.pth` file, not sharded safetensors)
|
||||
@@ -112,7 +115,14 @@ Custom model `model_index.json` (minimal):
|
||||
}
|
||||
```
|
||||
|
||||
The only **required** field is `_class_name` — it must match a key in `_DIFFUSION_MODELS` in `registry.py`. Other fields are model-specific and accessible via `od_config.model_config` dict.
|
||||
The only **required** field is `_class_name` — it must match a key in
|
||||
`_DIFFUSION_MODELS` in `registry.py`. Other fields are model-specific and
|
||||
accessible via `od_config.model_config`.
|
||||
|
||||
When the released repository cannot be modified and contains neither this file
|
||||
nor a root `config.json`, use the generic native-checkpoint signature resolver.
|
||||
Match the exact Hub ID or a distinctive local file set; never infer support from
|
||||
a model-name substring.
|
||||
|
||||
## External Dependency Management
|
||||
|
||||
@@ -141,6 +151,10 @@ def download_dependency():
|
||||
|
||||
Copy essential files from the original repo into `vllm_omni/diffusion/models/<name>/`. Adapt imports to use vllm-omni utilities. Benefits: no external dependency, no git clone step. Drawback: must maintain the ported code.
|
||||
|
||||
This is the default for native model support. Keep the reference repository at
|
||||
a pinned revision for architecture analysis and golden outputs, but do not
|
||||
import it from production code.
|
||||
|
||||
## Multi-Modal Input/Output Protocols
|
||||
|
||||
Custom models that handle images, audio, or video I/O should implement protocol classes:
|
||||
|
||||
@@ -0,0 +1,131 @@
|
||||
# Native Diffusion Model Integration Checklist
|
||||
|
||||
Use this checklist for native non-Diffusers models, hybrid pipelines, and
|
||||
model-specific optimization work.
|
||||
|
||||
## 1. Freeze the contract before coding
|
||||
|
||||
Record:
|
||||
|
||||
- Reference repository and immutable revision.
|
||||
- Official checkpoint ID, local layout, license, and required auxiliary files.
|
||||
- Pipeline stages: prompt/image encoding, denoising, video/image/audio decoding,
|
||||
and muxing.
|
||||
- Token packing, RoPE coordinates, attention sinks/LSE behavior, MoE layout,
|
||||
scheduler equations, CFG formulation, dtypes, and checkpoint transforms.
|
||||
- Output geometry, frame count/FPS, audio rate/channels, supported tasks, and
|
||||
default generation steps.
|
||||
- Available local hardware versus the official qualification hardware.
|
||||
- Intended default deployment and optional optimization paths.
|
||||
|
||||
State whether the deliverable is a native implementation or an external
|
||||
adapter. For native support, use the reference as an oracle only.
|
||||
|
||||
## 2. Use phase gates
|
||||
|
||||
| Phase | Required exit evidence |
|
||||
|---|---|
|
||||
| Architecture | Component/weight inventory; unsupported reference dependencies identified; ownership decisions documented |
|
||||
| Native correctness | Strict checkpoint load; tiny forward or one-step golden vs pinned reference; output contract validated |
|
||||
| Framework integration | Registry/discovery, shared request types, existing category example, serving route, docs, and recipe work without model-name branching in shared scripts |
|
||||
| Lossless distribution | One-rank oracle vs each supported TP/SP/CFG/HSDP/VAE layout; released-checkpoint smoke for the recommended layout |
|
||||
| Memory optimization | Ordinary/layerwise/DLO behavior, peak HBM, full-tree host PSS, and stage timeline; shared backend unchanged unless a general contract is proven |
|
||||
| Lossy optimization | Cache hit demonstrated; quality delta measured; speed claim made only when warmup permits hits |
|
||||
| Full qualification | Default-resolution E2E at official step count for every supported task; media metadata and hashes recorded |
|
||||
| Review readiness | Focused tests, Ruff/format, strict docs, DCO, current-main rebase, issue/RFC for large changes, concise PR evidence and limitations |
|
||||
|
||||
Do not start performance tuning before native one-step parity. Do not claim a
|
||||
feature from topology validation alone when released weights can exercise it.
|
||||
|
||||
## 3. Keep ownership narrow
|
||||
|
||||
Default placement:
|
||||
|
||||
| Concern | Owner |
|
||||
|---|---|
|
||||
| Architecture, checkpoint remap, packed attention math, nested block alias, auxiliary lifecycle | Model package |
|
||||
| Kernel/version dispatch already useful to multiple models | Shared attention utility |
|
||||
| Request-dependent example defaults | Typed model-extra entry consumed through a registry |
|
||||
| Exact-ID/local-file detection for repositories without standard metadata | Generic native-checkpoint signature table |
|
||||
| General offload/cache/HSDP bug | Shared layer plus one focused shared regression |
|
||||
|
||||
Before changing shared code, answer:
|
||||
|
||||
1. Is this a framework contract or only this checkpoint's layout?
|
||||
2. Can the model expose an existing declarative hook instead?
|
||||
3. Is there a second consumer or a framework-level failing test?
|
||||
4. Can the shared change avoid importing or naming the model?
|
||||
|
||||
If the answers do not justify a shared change, keep it model-local.
|
||||
|
||||
## 4. Reuse examples without leaking model logic
|
||||
|
||||
- Reuse the existing `x_to_y.py` category script.
|
||||
- Resolve the pipeline class through standard metadata or native signature
|
||||
discovery, not path substrings.
|
||||
- Put dynamic model defaults and validation in `model_extras`; let shared
|
||||
examples consume only the generic typed API.
|
||||
- Do not create image/speech/default abstractions without a real consumer.
|
||||
- Add a dedicated example only when the shared request/output protocol cannot
|
||||
express the model.
|
||||
|
||||
## 5. Build a bounded evidence matrix
|
||||
|
||||
Start with the recommended resident layout. Add only supported, useful points:
|
||||
|
||||
- Single request: TP, SP, or hybrid TP/SP.
|
||||
- Throughput: DLO/DP layouts when concurrent requests are valid.
|
||||
- CFG: compare CFG-parallel output with the packed-CFG oracle only if the model
|
||||
actually uses CFG.
|
||||
- HSDP: record loading and materialization peaks; it can use more HBM than a
|
||||
resident SP layout.
|
||||
- VAE patch parallelism: compare decoded tensors or media exactly where the
|
||||
algorithm should be lossless.
|
||||
- Cache: separate cache integration from acceleration evidence.
|
||||
|
||||
With four devices, qualify four-device layouts such as DP4, DP2SP2, SP4, or
|
||||
CFG2xSP2. Do not present CFG2xSP4 or an official eight-device runtime as locally
|
||||
validated.
|
||||
|
||||
For each measured layout record:
|
||||
|
||||
- Cold E2E and warm user latency.
|
||||
- Steady-state wave time and throughput per device.
|
||||
- Prompt, denoise, video decode, audio decode, and mux times.
|
||||
- Peak allocated/reserved HBM and full-process-tree host PSS.
|
||||
- Step count, seed, output geometry, frame/FPS/audio metadata, and parity metric.
|
||||
|
||||
Plot latency versus throughput/device. Keep one-step, four-step, and full-step
|
||||
results separate because they are not quality-equivalent.
|
||||
|
||||
## 6. Keep tests proportional
|
||||
|
||||
Prefer a small set with distinct failure ownership:
|
||||
|
||||
1. Strict weight-key/dtype/load contract.
|
||||
2. Tiny native golden or one-step reference parity.
|
||||
3. One parametrized distributed-parity test covering supported layouts.
|
||||
4. Pipeline/request/output geometry and serving smoke.
|
||||
5. Shared-example/default test when the category script changes.
|
||||
6. One shared regression only for a genuine shared-layer fix.
|
||||
|
||||
Avoid duplicating the same shape or configuration assertion across
|
||||
model-specific files. Keep expensive released-checkpoint evidence in the PR or
|
||||
recipe unless the repository's CI tier explicitly requires it.
|
||||
|
||||
## 7. Write evidence, not marketing
|
||||
|
||||
In the PR description include:
|
||||
|
||||
- A compact architecture/data-flow explanation.
|
||||
- Default deployment and why it is recommended.
|
||||
- Code-change counts by component.
|
||||
- Accuracy and distributed-parity evidence tied to commands/scripts.
|
||||
- Separate Pareto figures for different step counts.
|
||||
- Supported versus validated versus planned optimizations.
|
||||
- Hardware gaps and untested official-runtime claims.
|
||||
|
||||
Call a result a bounded local qualification unless it includes repeated timing
|
||||
and variance. State whether latency includes text/image encoding and all
|
||||
decoders. Preserve the reference revision and exact commands needed to
|
||||
reproduce the evidence.
|
||||
Reference in new issue
Block a user