Files
vllm-omni/.claude/skills/add-diffusion-model/SKILL.md
T

650 lines
30 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
---
name: add-diffusion-model
description: Add a new diffusion model (text-to-image, text-to-video, image-to-video, text-to-audio, image editing) to vLLM-Omni, including Cache-DiT acceleration and parallelism support (TP, SP/USP, CFG-Parallel, HSDP). Use when integrating a new diffusion model, porting a diffusers pipeline or a custom model repo to vllm-omni, creating a new DiT transformer adapter, adding diffusion model support, or enabling multi-GPU parallelism and cache acceleration for an existing model.
---
# Adding a Diffusion Model to vLLM-Omni
## Overview
This skill guides you through adding a new diffusion model to vLLM-Omni. The model may come from HuggingFace Diffusers (structured pipeline) or from a private/custom repo. The workflow differs significantly depending on the source.
## Prerequisites
Before starting, determine:
1. **Model category**: Text-to-Image, Text-to-Video, Image-to-Video, Image Editing, Text-to-Audio, or Omni
2. **Reference source**: Diffusers pipeline, custom repo, or a combination
3. **Model HuggingFace ID** or local checkpoint path
4. **Architecture**: Scheduler, text encoder, VAE, transformer/backbone
## Step 0: Classify the Migration Path
Check the model's HF repo for `model_index.json`. This determines your path:
| Scenario | How to identify | Migration path |
|----------|----------------|----------------|
| **Already supported** | `_class_name` in `model_index.json` matches a key in `_DIFFUSION_MODELS` in `registry.py` | Skip implementation, then validate model-specific examples, tests, and docs as needed |
| **Diffusers-based** | Has standard `model_index.json` with `_diffusers_version`, subfolders for `transformer/`, `vae/`, etc. | Follow **Path A** below |
| **Custom/private repo** | No diffusers `model_index.json`, weights in non-standard format, custom model code in a separate git repo | Follow **Path B** below |
| **Hybrid** | Has some diffusers components (VAE) but custom transformer/fusion | Mix of Path A and Path B |
## Path A: Diffusers-Based Model
For models with a standard diffusers layout. See [references/transformer-adaptation.md](references/transformer-adaptation.md) for detailed code patterns.
### A1. Analyze `model_index.json`
Identify components: `transformer`, `scheduler`, `vae`, `text_encoder`, `tokenizer`.
### A2. Create model directory
```
vllm_omni/diffusion/models/your_model_name/
├── __init__.py
├── pipeline_your_model.py
└── your_model_transformer.py
```
### A3. Adapt transformer
1. Copy from diffusers source. Remove mixins (`ModelMixin`, `ConfigMixin`, `AttentionModuleMixin`).
2. Replace attention with `vllm_omni.diffusion.attention.layer.Attention` (QKV shape: `[B, seq, heads, head_dim]`).
3. Add `od_config: OmniDiffusionConfig | None = None` to `__init__`.
4. Add `load_weights()` method mapping diffusers weight names to vllm-omni names.
5. Add class attributes for acceleration features such as `_repeated_blocks` and `_layerwise_offload_blocks_attrs` (see [references/transformer-adaptation.md](references/transformer-adaptation.md) for examples).
### A4. Adapt pipeline
Inherit from `nn.Module`. The key contract:
```python
class YourPipeline(nn.Module):
def __init__(self, *, od_config: OmniDiffusionConfig, prefix: str = ""):
# Load VAE, text encoder, tokenizer via from_pretrained()
# Instantiate transformer (weights loaded later via weights_sources)
self.weights_sources = [
DiffusersPipelineLoader.ComponentSource(
model_or_path=od_config.model, subfolder="transformer",
prefix="transformer.", fall_back_to_pt=True)]
def forward(self, req: OmniDiffusionRequest) -> DiffusionOutput:
# Encode prompt → prepare latents → denoise loop → VAE decode
return DiffusionOutput(output=output)
def load_weights(self, weights):
return AutoWeightsLoader(self).load_weights(weights)
```
Add post/pre-process functions in the same pipeline file. Register them in `registry.py`.
### A4.1 Add progress bar support (recommended)
For pipelines with a standard denoising loop, prefer the existing progress bar pattern instead of hand-rolled logging.
```python
from vllm_omni.diffusion.models.progress_bar import ProgressBarMixin
class YourPipeline(nn.Module, ProgressBarMixin):
def forward(self, req: OmniDiffusionRequest) -> DiffusionOutput:
# ... prepare timesteps / latents ...
with self.progress_bar(total=len(timesteps)) as progress_bar:
for i, t in enumerate(timesteps):
# predict noise / scheduler step
latents = ...
progress_bar.update()
return DiffusionOutput(output=output)
```
For custom loop structures, follow `vllm_omni/diffusion/models/progress_bar.py` and existing pipelines using `ProgressBarMixin`.
### A5. Register, test, docs → continue at Step 4 below.
---
## Path B: Custom/Private Repo Model
For models without a diffusers pipeline — weights in custom format, model code in a private repo. Real examples: DreamID-Omni, BAGEL, HunyuanImage3. See [references/custom-model-patterns.md](references/custom-model-patterns.md) for concrete custom-model integration patterns.
### B1. Understand the reference repo
Study the original model's code to identify:
- **Model architecture files** (transformers, fusion modules, embeddings)
- **Weight format** (safetensors, `.pth`, custom checkpoint structure)
- **Weight loading helpers** (custom init functions, checkpoint loaders)
- **Pre/post-processing** (image/audio transforms, tokenization, VAE encode/decode)
- **External dependencies** (packages not on PyPI)
- **Config format** (JSON config files, hardcoded dicts)
### B2. Decide what lives WHERE
This is the key design decision for custom models. Follow these placement rules:
| Code type | Where to place | Example |
|-----------|---------------|---------|
| **Pipeline orchestration** (init, forward, denoise loop) | `vllm_omni/diffusion/models/<name>/pipeline_<name>.py` | Always required |
| **Custom transformer/backbone** (ported and adapted to vllm-omni) | `vllm_omni/diffusion/models/<name>/<name>_transformer.py` or similar | `wan2_2.py`, `fusion.py`, `bagel_transformer.py` |
| **Custom sub-models** (VAE, fusion, autoencoder) | `vllm_omni/diffusion/models/<name>/` as separate files | `autoencoder.py`, `fusion.py` |
| **External dependency code** (original repo utilities) | **External repo**, installed via download script or pip | `dreamid_omni` package via git clone |
| **Hardcoded model configs** | Module-level dicts in pipeline file | `VIDEO_CONFIG`, `AUDIO_CONFIG` dicts |
| **Download/setup script** | `examples/offline_inference/<name>/download_<name>.py` | `download_dreamid_omni.py` |
| **Custom `model_index.json`** | Generated by download script, placed at model root | Minimal: `{"_class_name": "YourPipeline", ...}` |
### B3. Handle external dependencies
If the model's code lives in a separate git repo:
**Option 1: Import with graceful fallback** (recommended for models with external utils)
```python
try:
from external_model.utils import init_vae, load_checkpoint
except ImportError:
raise ImportError(
"Failed to import from dependency 'external_model'. "
"Please run the download script first."
)
```
**Option 2: Port the code directly** (preferred when feasible)
Copy the essential model files into `vllm_omni/diffusion/models/<name>/` and adapt them. This avoids external dependencies. BAGEL does this — `autoencoder.py` and `bagel_transformer.py` are ported directly.
**Decision criteria**: Port if the code is self-contained and won't diverge. Use external deps if the model repo is actively maintained and the code is complex.
### B4. Handle custom weight loading
Custom models have two common patterns for weight loading:
**Pattern 1: Bypass standard loader** (DreamID-Omni style)
When the original model has complex custom init functions that load weights in `__init__`:
```python
class CustomPipeline(nn.Module):
def __init__(self, *, od_config, prefix=""):
super().__init__()
model = od_config.model
# Load everything eagerly in __init__ using custom helpers
self.vae = custom_init_vae(model, device=self.device)
self.text_encoder = custom_init_text_encoder(model, device=self.device)
self.transformer = CustomFusionModel(CONFIG)
load_custom_checkpoint(
self.transformer,
checkpoint_path=os.path.join(model, "model.safetensors"),
)
# NO weights_sources defined — bypasses standard loader
def load_weights(self, weights):
pass # No-op — all weights loaded in __init__
```
**Pattern 2: Use standard loader with custom `load_weights`** (BAGEL style)
When weights are in safetensors format but need name remapping:
```python
from vllm.model_executor.model_loader.weight_utils import default_weight_loader
class CustomPipeline(nn.Module):
def __init__(self, *, od_config, prefix=""):
super().__init__()
# Instantiate model architecture without weights
self.bagel = BagelModel(config)
self.vae = AutoEncoder(ae_params)
# Point loader at the safetensors in the model root
self.weights_sources = [
DiffusersPipelineLoader.ComponentSource(
model_or_path=od_config.model,
subfolder=None, # weights at root, not in subfolder
prefix="",
fall_back_to_pt=False,
)
]
def load_weights(self, weights):
# Custom name remapping for non-diffusers weight names
params = dict(self.named_parameters())
loaded = set()
for name, tensor in weights:
# Remap original weight names to vllm-omni module names
name = self._remap_weight_name(name)
if name in params:
param = params[name]
weight_loader = getattr(param, "weight_loader", default_weight_loader)
weight_loader(param, tensor)
loaded.add(name)
return loaded
```
### B5. Create the `model_index.json`
Custom models need a `model_index.json` at the model root for vllm-omni to discover them. For custom models, this is minimal:
```json
{
"_class_name": "YourModelPipeline",
"custom_key": "path/to/custom_weights.safetensors"
}
```
The `_class_name` must match a key in `_DIFFUSION_MODELS` in `registry.py`. Additional keys are model-specific (accessed via `od_config.model_config`).
If the model's weights come from multiple HF repos, write a **download script** that:
1. Downloads from each repo
2. Assembles into a single directory
3. Generates `model_index.json`
4. Installs any external dependencies (git clone + `.pth` file)
Place at: `examples/offline_inference/<name>/download_<name>.py`
### B6. Handle multi-modal inputs
If the model accepts images, audio, or other multi-modal inputs, implement the protocol classes from `vllm_omni/diffusion/models/interface.py`:
```python
from vllm_omni.diffusion.models.interface import SupportImageInput, SupportAudioInput
class MyPipeline(nn.Module, SupportImageInput, SupportAudioInput):
# Protocol markers — the engine uses these to enable proper input routing
pass
```
Preprocessing for custom models is typically done **inside `forward()`** rather than via registered pre-process functions, since the logic is often tightly coupled to the model.
### B7. Continue at Step 4 below.
---
## Common Steps (Both Paths)
### Step 4: Register Model in registry.py
Edit `vllm_omni/diffusion/registry.py`:
```python
_DIFFUSION_MODELS = {
"YourModelPipeline": ("your_model_name", "pipeline_your_model", "YourModelPipeline"),
}
_DIFFUSION_POST_PROCESS_FUNCS = {
"YourModelPipeline": "get_your_model_post_process_func", # if applicable
}
_DIFFUSION_PRE_PROCESS_FUNCS = {
"YourModelPipeline": "get_your_model_pre_process_func", # if applicable
}
```
The registry key is the `_class_name` from `model_index.json`. The tuple is `(folder_name, module_file, class_name)`.
Create `__init__.py` exporting the pipeline class and any factory functions.
### Step 5: Run, Test, Debug
Use the appropriate existing example script:
| Category | Script |
|----------|--------|
| Text-to-Image | `examples/offline_inference/text_to_image/text_to_image.py` |
| Text-to-Video | `examples/offline_inference/text_to_video/text_to_video.py` |
| Image-to-Video | `examples/offline_inference/image_to_video/image_to_video.py` |
| Image-to-Image | `examples/offline_inference/image_to_image/image_edit.py` |
| Text-to-Audio | `examples/offline_inference/text_to_audio/text_to_audio.py` |
For custom/Omni models that don't fit these categories, create a dedicated example script.
**Validation**: No errors, output is meaningful, quality matches reference implementation.
See [references/troubleshooting.md](references/troubleshooting.md) for common errors.
### Step 6: Add Example Scripts
For Omni or custom models, create:
- `examples/offline_inference/your_model_name/` — offline script + README
- `examples/online_serving/your_model_name/` — server script + client
- Download script if weights require assembly from multiple sources
### Step 7: Update Documentation
Required updates:
1. `docs/user_guide/diffusion/parallelism/overview.md` — parallelism support overview/table
2. `docs/user_guide/diffusion/cpu_offload.md` — if CPU offload supported (add to supported models table)
3. `docs/user_guide/diffusion/cache_acceleration/teacache.md` — if TeaCache supported
4. `docs/user_guide/diffusion/cache_acceleration/cache_dit.md` — if Cache-DiT supported
5. Offline example docs under `examples/offline_inference/<name>/` (`README.md` or category-specific `.md`)
6. `examples/online_serving/<name>/README.md` — online serving docs
### Step 8: Add E2E Tests
**Follow the [vllm-omni-test skill](../vllm-omni-test/SKILL.md)** for markers, file naming, Buildkite wiring, and run commands. Also read [l4_functionality_tests.inc.md](https://github.com/vllm-project/vllm-omni/blob/main/docs/contributing/ci/test_examples/l4_functionality_tests.inc.md), [test_system_overview.md](https://github.com/vllm-project/vllm-omni/blob/main/docs/contributing/ci/test_system_overview.md), and [test_writing_guide.md](https://github.com/vllm-project/vllm-omni/blob/main/docs/contributing/ci/test_writing_guide.md).
Classify the model's **CI priority** first:
| Priority | Required test levels | Files & markers |
|----------|---------------------|-----------------|
| **High** (listed in [#1832](https://github.com/vllm-project/vllm-omni/issues/1832) or on the diffusion hot path) | **L1** · **L2** online · **L3** online + offline · **L4** feature + performance | See table below |
| **Medium** (*normal priority* in L4 docs) | **L3** online + offline · **L4** feature only | Fewer L4 parametrized rows |
| **Low** | **L4** feature only | One or two `*_expansion.py` cases |
**Per-level deliverables (diffusion / `pytest.mark.diffusion`):**
| Level | Location | Marker | CI pipeline | Notes |
|-------|----------|--------|-------------|-------|
| **L1** | `tests/diffusion/models/{slug}/`, `tests/diffusion/cache/`, transformer unit tests | `core_model` + `cpu` | `test-ready.yml` | Weight remap, `_sp_plan`, cache enabler registration, shape contracts |
| **L2** | `tests/e2e/online_serving/test_{slug}.py` (and offline if the category is offline-first) | **`core_model` + `advanced_model`** (both on baseline smoke) + `diffusion` + `@hardware_test` / `hardware_marks` | `test-ready.yml` | Default deploy smoke — minimal `num_inference_steps`, single prompt |
| **L3** | `tests/e2e/online_serving/test_{slug}.py` **and** `tests/e2e/offline_inference/test_{slug}.py` when offline matters | Baseline smoke: **`core_model` + `advanced_model`**; heavier cases: `advanced_model` only (+ `diffusion`) | `test-merge.yml` **or** merged into nightly diffusion function job | Real weights, streaming/API paths, LoRA/offload smoke |
| **L4** | `tests/e2e/online_serving/test_{slug}_expansion.py` (+ offline expansion if needed) | `full_model` + `diffusion` | `test-nightly.yml` (X2I / X2V / X2A function groups) | Feature combos per [#1832](https://github.com/vllm-project/vllm-omni/issues/1832); perf → `tests/dfx/perf/tests/test_{model}_vllm_omni.json` with per-case **`mark`** (`hardware_marks` + `full_model` + `diffusion`) |
**L2 & L3 online — same file, dual marks on the baseline smoke:** The **first / simplest** case in `test_{slug}.py` (default deploy, minimal steps, single prompt) should carry **both** `@pytest.mark.core_model` and `@pytest.mark.advanced_model` on the **same** function so L2 (`test-ready.yml`) and L3 (`test-merge.yml`) share one smoke test. Heavier deploy variants or API paths in the same file use **`advanced_model` only**. When L3 moves to nightly, migrate those heavier cases into `test_{slug}_expansion.py` with `full_model` and remove the dedicated `test-merge.yml` job (see `test_longcat_image_expansion.py`, `test_qwen_image_expansion.py`).
**L4 design (high priority):** Combine multiple supported features (Cache-DiT, TP, USP, CFG, HSDP, CPU offload, quantization) into **few parametrized** `OmniServerParams` rows so each feature appears in at least one case without exploding GPU jobs. Shard single-GPU vs multi-GPU cases across the nightly X2I/X2V function steps (1-GPU vs `distributed_cuda`).
**L4 design (medium / low):** One or two parametrized rows covering the best quality/perf trade-off; skip perf JSON unless the model is high priority.
**Reference implementations:** `tests/e2e/online_serving/test_qwen_image_edit_expansion.py`, `tests/e2e/online_serving/test_longcat_image_expansion.py`, `tests/e2e/online_serving/test_hunyuan_video_15_expansion.py`.
**Keep model-specific code inside test modules — not `tests/helpers/{slug}.py`:** deploy constants, prompts, sampling dicts, and inline `request_config` / `form_data` belong in each `test_{slug}.py` and `test_{slug}_expansion.py`. Do not add per-model files under `tests/helpers/`; reuse only repo-wide harness (`mark`, `media`, `runtime`, `stage_config`, `assertions`). **L2+ online/offline e2e:** reuse or add `send_*_request` in `tests/helpers/runtime.py` — tests call the handler, not raw `omni.generate` / HTTP. See [vllm-omni-test skill](../vllm-omni-test/SKILL.md) § **Runtime send helpers**.
### Step 9: Add Cache-DiT Acceleration
Cache-DiT accelerates inference by caching intermediate computation results across denoising steps. After your model is working correctly on a single GPU, add cache-dit support.
See [references/cache-dit-patterns.md](references/cache-dit-patterns.md) for detailed code patterns.
#### 9a. Determine your model type
| Model Type | Description | Action |
|------------|-------------|--------|
| **Standard single-transformer** | One transformer with one `ModuleList` of blocks | No code needed — `CacheDiTBackend` auto-detects via `enable_cache_for_dit()` |
| **Multi-block-list** | One transformer with multiple block lists (e.g., `transformer_blocks` + `single_transformer_blocks`) | Write custom enabler with `BlockAdapter` |
| **Dual-transformer** | Two transformers (e.g., high-noise + low-noise) | Write custom enabler with `BlockAdapter` wrapping both |
#### 9b. Standard models — verify automatic support
For standard single-transformer models, test directly:
```python
omni = Omni(
model="your-model-name",
cache_backend="cache_dit",
cache_config={
"Fn_compute_blocks": 1,
"Bn_compute_blocks": 0,
"max_warmup_steps": 4,
}
)
```
Check logs for "Cache-dit enabled successfully on xxx". If it works, skip to Step 9e.
#### 9c. Custom architectures — write a custom enabler
For multi-block-list or dual-transformer models, write a custom enabler function:
```python
from cache_dit import BlockAdapter, ForwardPattern, ParamsModifier, DBCacheConfig
def enable_cache_for_your_model(pipeline, cache_config):
db_cache_config = DBCacheConfig(
num_inference_steps=None,
Fn_compute_blocks=cache_config.Fn_compute_blocks,
Bn_compute_blocks=cache_config.Bn_compute_blocks,
max_warmup_steps=cache_config.max_warmup_steps,
max_cached_steps=cache_config.max_cached_steps,
max_continuous_cached_steps=cache_config.max_continuous_cached_steps,
residual_diff_threshold=cache_config.residual_diff_threshold,
)
cache_dit.enable_cache(
BlockAdapter(
transformer=pipeline.transformer,
blocks=[
pipeline.transformer.transformer_blocks,
pipeline.transformer.single_transformer_blocks,
],
forward_pattern=[ForwardPattern.Pattern_1, ForwardPattern.Pattern_1],
params_modifiers=[ParamsModifier(...)],
),
cache_config=db_cache_config,
)
def refresh_cache_context(pipeline, num_inference_steps, verbose=True):
cache_dit.refresh_context(
pipeline.transformer, num_inference_steps=num_inference_steps, verbose=verbose
)
return refresh_cache_context
```
#### 9d. Register the custom enabler
Add your enabler to `CUSTOM_DIT_ENABLERS` in
`vllm_omni/diffusion/cache/cachedit/model_specific.py`:
```python
CUSTOM_DIT_ENABLERS = {
"Wan22Pipeline": enable_cache_for_wan22,
"LongCatImagePipeline": enable_cache_for_longcat_image,
"YourModelPipeline": enable_cache_for_your_model, # Add here
}
```
#### 9e. Test Cache-DiT
```python
omni = Omni(
model="your-model-name",
cache_backend="cache_dit",
cache_config={
"Fn_compute_blocks": 1, "Bn_compute_blocks": 0,
"max_warmup_steps": 4, "residual_diff_threshold": 0.24,
}
)
images = omni.generate("a beautiful landscape",
OmniDiffusionSamplingParams(num_inference_steps=50))
```
**Verify**: 1) logs show cache enabled, 2) 1.5-2x speedup, 3) output quality acceptable vs baseline.
If quality degrades, lower `residual_diff_threshold` (try 0.12-0.18) or increase `max_warmup_steps` (try 6-8).
---
### Step 10: Add Parallelism Support
After the model works on a single GPU, add multi-GPU parallelism. Add each type incrementally, testing after each addition.
See [references/parallelism-patterns.md](references/parallelism-patterns.md) for detailed code patterns and API reference.
**Recommended order**: TP → SP/USP → CFG Parallel → HSDP
#### 10a. Tensor Parallelism (TP)
Shards DiT linear layers across GPUs. Requires code changes in the transformer.
**What to change in the transformer**:
1. Replace `nn.Linear` with `ColumnParallelLinear` / `RowParallelLinear` / `QKVParallelLinear`
2. Update `load_weights()` to handle QKV fusion with `stacked_params_mapping`
3. Use `self.to_qkv.num_heads` (local heads) instead of total heads for split sizes
```python
from vllm.model_executor.layers.linear import (
QKVParallelLinear, RowParallelLinear, ColumnParallelLinear,
)
# Attention: QKV → RowParallel output
self.to_qkv = QKVParallelLinear(dim, head_dim, num_heads, num_kv_heads)
self.to_out = RowParallelLinear(dim, dim, input_is_parallel=True)
# FFN: ColumnParallel → RowParallel
self.w1 = ColumnParallelLinear(dim, ffn_dim)
self.w2 = RowParallelLinear(ffn_dim, dim, input_is_parallel=True)
```
**Constraints**: `num_heads % tp_size == 0` and `num_kv_heads % tp_size == 0`.
**Test**: `--tensor-parallel-size 2`
#### 10b. Sequence Parallelism (SP / USP)
Splits sequence tokens across GPUs. Non-intrusive via `_sp_plan` on the transformer class — no changes to `forward()`.
**What to change in the transformer**:
Add `_sp_plan` class attribute:
```python
from vllm_omni.diffusion.distributed.sp_plan import (
SequenceParallelInput, SequenceParallelOutput,
)
class YourTransformer(nn.Module):
_sp_plan = {
"blocks.0": {
"hidden_states": SequenceParallelInput(split_dim=1, expected_dims=3),
},
"proj_out": SequenceParallelOutput(gather_dim=1, expected_dims=3),
}
```
If inline tensor ops (e.g., `torch.cat`) exist between shard/gather points, extract them into `nn.Module` submodules so hooks can intercept them.
For RoPE that needs splitting, add an entry for the RoPE module with `split_output=True`.
**Test**: `--ulysses-degree 2` (offline) or `--usp 2` (online serving)
#### 10c. CFG Parallel
Distributes positive/negative CFG branches across 2 GPUs. Requires the pipeline to inherit `CFGParallelMixin`.
**What to change in the pipeline**:
```python
from vllm_omni.diffusion.distributed.cfg_parallel import CFGParallelMixin
class YourPipeline(nn.Module, CFGParallelMixin):
def diffuse(self, ...) -> torch.Tensor:
for i, t in enumerate(timesteps):
positive_kwargs = {...}
negative_kwargs = {...} if do_true_cfg else None
noise_pred = self.predict_noise_maybe_with_cfg(
do_true_cfg=do_true_cfg, true_cfg_scale=cfg_scale,
positive_kwargs=positive_kwargs, negative_kwargs=negative_kwargs,
)
latents = self.scheduler_step_maybe_with_cfg(
noise_pred, t, latents, do_true_cfg
)
return latents
```
Override `predict_noise()` if your transformer call is non-standard. Override `combine_cfg_noise()` for multi-output models (e.g., video + audio).
**Constraint**: Exactly 2 GPUs. Only for models using classifier-free guidance.
**Test**: `--cfg-parallel-size 2`
#### 10d. HSDP (Hybrid Sharded Data Parallel)
Shards transformer weights via PyTorch FSDP2 to reduce per-GPU VRAM. No code changes to the forward pass — just add a class attribute.
**What to change in the transformer**:
```python
class YourTransformer(nn.Module):
@staticmethod
def _is_transformer_block(name: str, module) -> bool:
return "blocks" in name and name.split(".")[-1].isdigit()
_hsdp_shard_conditions = [_is_transformer_block]
```
**Constraint**: Cannot combine with TP. For standalone HSDP, set `hsdp_shard_size` explicitly.
**Test**: `--use-hsdp` or `DiffusionParallelConfig(use_hsdp=True)`
#### 10e. Update parallelism documentation
After adding parallelism support, update:
1. `docs/user_guide/diffusion/parallelism/overview.md` — add your model to the support overview/table
2. Record which parallelism methods are supported (USP, Ring, CFG, TP, HSDP, VAE-Patch)
### Step 11: Add CPU Offload Support
Implement `SupportsComponentDiscovery` on your pipeline class to enable
`--enable-cpu-offload` and `--enable-layerwise-offload`. The protocol
declares which submodules the offloader should manage:
```python
from typing import ClassVar
from vllm_omni.diffusion.models.interface import SupportsComponentDiscovery
class YourPipeline(nn.Module, SupportsComponentDiscovery):
_dit_modules: ClassVar[list[str]] = ["transformer"]
_encoder_modules: ClassVar[list[str]] = ["text_encoder"]
_vae_modules: ClassVar[list[str]] = ["vae"]
_resident_modules: ClassVar[list[str]] = [] # optional
```
- `_dit_modules`: denoising submodules (kept on GPU during diffusion loop)
- `_encoder_modules`: encoder/vision submodules (offloaded to CPU during diffusion loop)
- `_vae_modules`: VAE(s) (handled by both sequential and layerwise backends)
- `_resident_modules`: additional modules to pin on GPU during layerwise
offloading (e.g. embedders, connectors). Only used by the layerwise
backend. Optional — defaults to `[]`.
All attribute names support dotted paths for nested submodules
(e.g. `"pipe.transformer"`, `"bagel.time_embedder"`).
Pipelines without `SupportsComponentDiscovery` fall back to scanning
well-known attribute names (`transformer`, `text_encoder`, `vae`,
etc.), which fails for non-standard names.
### Step 12: Performance Profiling
After verifying correctness and implementing parallelism/caching, profile the model's performance to identify bottlenecks and ensure optimal execution.
See the [Profiling Single-Stage Diffusion](../../../docs/contributing/profiling.md#3-profiling-single-stage-diffusion) guide for detailed instructions on:
1. Using the PyTorch profiler (`profiler: "torch"`) to capture detailed CPU/CUDA traces.
2. Using Nsight Systems (`nsys`) with `profiler: "cuda"` for low-overhead CUDA traces.
3. Controlling profiling via `omni.start_profile()` and `omni.stop_profile()`.
---
## Pre-commit conventions
New library files must pass the local gates in
[docs/contributing/README.md](../../../docs/contributing/README.md#linting).
That page is the full hook list (SPDX, forbidden imports including Hugging Face
Hub / Triton / pickle, `torch.cuda`, mypy, test marks, markdownlint, Buildkite,
shellcheck). In particular:
- SPDX copyright is `vLLM-Omni project` (stale `vLLM project` is rewritten).
- Use `import regex as re` and `pybase64` in `vllm_omni/`; do not import stdlib
`re` or `base64`. Hugging Face Hub downloads go through
`vllm.transformers_utils.repo_utils`.
- Do not add `torch.cuda.*` call sites; use `current_omni_platform`.
- New `tests/**/test_*.py` files need a CI level mark and a hardware mark.
- Do not expand `CHECK_IMPORTS[*].allowed_files` or `ALLOWED_FILES` without review.
- GitHub Actions skips SPDX/shellcheck/mypy-3.10/test-marks/markdownlint; run
`pre-commit` locally.
## Iterative Development Tips
1. **Start minimal**: Basic generation first, no parallelism/caching
2. **Use `--enforce-eager`**: Disable torch.compile during debugging
3. **Use small models**: Test with smaller variants first
4. **Check tensor shapes**: Most errors are reshape mismatches in attention
5. **Add features incrementally**: Single GPU → TP → SP → CFG → HSDP → Cache-DiT
6. **For custom models**: Get the model running with the original code first, then progressively replace components with vllm-omni equivalents
7. **Cache-DiT before parallelism tuning**: Cache-DiT is lossy — verify quality at baseline before combining with parallelism
8. **Combine lossless + lossy**: e.g., TP + SP + Cache-DiT for maximum throughput
## Reference Files
- [vllm-omni-test skill](../vllm-omni-test/SKILL.md) — L1–L4 markers, naming, Buildkite wiring, run commands
- [Transformer Adaptation](references/transformer-adaptation.md) — porting transformers from diffusers
- [Custom Model Patterns](references/custom-model-patterns.md) — patterns for non-diffusers models
- [Parallelism Patterns](references/parallelism-patterns.md) — TP, SP/USP, CFG parallel, HSDP implementation details
- [Cache-DiT Patterns](references/cache-dit-patterns.md) — cache-dit acceleration for standard and custom architectures
- [Troubleshooting](references/troubleshooting.md) — common errors and fixes