[SDXL] SDXL model enabling (#4331)

Signed-off-by: Chendi Xue <chendi.xue@intel.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
This commit is contained in:
Chendi.XueandHongsheng Liu authored and GitHub committed 2026-06-16 22:51:11 +08:00
1 parent 864d22164c
commit e7f0db1067
10 files changed
+1492 -1

No files matched your search

+1
View File
@@ -48,6 +48,7 @@ th {
| `OvisImagePipeline` | Ovis-Image | `OvisAI/Ovis-Image` | ✅︎ | ✅︎ | | ✅︎ |
| `LongcatImagePipeline` | LongCat-Image | `meituan-longcat/LongCat-Image` | ✅︎ | ✅︎ | ✅︎ | ✅︎ |
| `LongCatImageEditPipeline` | LongCat-Image-Edit | `meituan-longcat/LongCat-Image-Edit` | ✅︎ | ✅︎ | ✅︎ | ✅︎ |
| `StableDiffusionXLPipeline` | Stable-Diffusion-XL | `stabilityai/stable-diffusion-xl-base-1.0` | ✅︎ | ✅︎ | ✅︎ | ✅︎ |
| `StableDiffusion3Pipeline` | Stable-Diffusion-3 | `stabilityai/stable-diffusion-3.5-medium` | ✅︎ | ✅︎ | | ✅︎ |
| `CosyVoice3Model` | CosyVoice3 | `FunAudioLLM/Fun-CosyVoice3-0.5B-2512` | ✅︎ | ✅︎ | | ✅︎ |
| `OmniVoiceModel` | OmniVoice | `k2-fsa/OmniVoice` | ✅︎ | | | |
@@ -193,6 +193,7 @@ Factory function `get_offload_backend()` selects the appropriate backend based o
| NextStep11Pipeline | `stepfun-ai/NextStep-1.1` | `NextStepModel` | - | ✓ | `"layers"` |
| OvisImagePipeline | `AIDC-AI/Ovis-Image-7B` | `OvisImageTransformer2DModel` | - | ✓ | `"transformer"` |
| QwenImagePipeline | `Qwen/Qwen-Image` | `QwenImageTransformer2DModel` | ✓ | ✓ | `"transformer_blocks"` |
| StableDiffusionXLPipeline | `stabilityai/stable-diffusion-xl-base-1.0` | `SDXLUNet2DConditionModel` | ✓ | ✓ | `"down_blocks"`, `"up_blocks"` |
| StableDiffusion3Pipeline | `stabilityai/stable-diffusion-3.5-medium` | `SD3Transformer2DModel` | - | ✓ | `"transformer_blocks"` |
| Wan22I2VPipeline | `Wan-AI/Wan2.2-I2V-A14B-Diffusers` | `WanTransformer3DModel` | ✓ | ✓ | `"blocks"` |
| Wan22Pipeline | `Wan-AI/Wan2.2-T2V-A14B-Diffusers` | `WanTransformer3DModel` | ✓ | ✓ | `"blocks"` |
+1
View File
@@ -130,6 +130,7 @@ The following tables show which models support each feature:
| **Qwen-Image-Edit-2509** | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ (decode) | ✅ | ❌ | ❌ |
| **Qwen-Image-Layered** | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ (decode) | ❌ | ❌ |
| **SenseNova-U1** | ❌ | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ |
| **Stable-Diffusion-XL** | ❌ | ❌ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ (decode) | ❌ | ❌ |
| **Stable-Diffusion3.5** | ❌ | ✅ | ❌ | ✅ | ✅ | ❌ | ❌ | ✅ | ✅ (decode) | ❌ | ❌ |
| **Z-Image** | ✅ | ✅ | ✅ | ❓ | ✅ (TP=2 only) | ❌ | ✅ | ❌ | ✅ (decode) | ✅ | ❌ |
| **ERNIE-Image** | ❌ | ✅ | ✅ | ❓ | ✅ | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ |
+123
View File
@@ -0,0 +1,123 @@
# Stable Diffusion XL
> Text-to-image generation with Stability AI's SDXL Base 1.0
## Summary
- Vendor: Stability AI
- Model: `stabilityai/stable-diffusion-xl-base-1.0`
- Task: Text-to-image generation
- Mode: Offline inference / Online serving with the OpenAI-compatible images API
- Maintainer: Community
## When to use this recipe
Use this recipe when you want a high-quality, open-weight text-to-image model
that generates 1024x1024 images. SDXL Base 1.0 is one of the most widely used
diffusion models and produces photorealistic outputs with strong prompt
adherence. It requires ~6.5 GiB for model weights and runs inference at
~7.5 seconds per image (30 steps) on a single Intel Data Center GPU.
## References
- Model: <https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0>
- Architecture reference: <https://github.com/Stability-AI/generative-models>
- Related example under `examples/`:
[`examples/offline_inference/text_to_image/text_to_image.py`](../../examples/offline_inference/text_to_image/text_to_image.py)
## Hardware Support
## GPU
### 1x Intel Arc BMG GPU (XPU)
#### Environment
- OS: Linux
- Python: 3.10+
- Driver / runtime: Intel XPU environment with Intel Arc BMG GPU
- vLLM version: Match the repository requirements for your checkout
- vLLM-Omni version or commit: Use the commit you are deploying from
#### Command
**Offline inference:**
```bash
python examples/offline_inference/text_to_image/text_to_image.py \
--model stabilityai/stable-diffusion-xl-base-1.0 \
--prompt "a beautiful sunset over the ocean, photorealistic" \
--height 1024 --width 1024 \
--num-inference-steps 30 \
--guidance-scale 7.5 \
--seed 42 \
--output sdxl_output.png
```
**Online serving:**
```bash
vllm serve stabilityai/stable-diffusion-xl-base-1.0 --omni --port 8091
```
#### Verification
**For offline inference:** Check the output image file exists and has valid content.
**For online serving:**
```bash
curl -X POST http://localhost:8091/v1/images/generations \
-H "Content-Type: application/json" \
-d '{
"model": "stabilityai/stable-diffusion-xl-base-1.0",
"prompt": "a cute cat sitting on a windowsill, highly detailed, sharp focus",
"negative_prompt": "blurry, out of focus, low quality",
"guidance_scale": 7.5,
"n": 1,
"size": "1024x1024"
}' | python3 -c "
import json, base64, sys
data = json.load(sys.stdin)
with open('output.png', 'wb') as f:
f.write(base64.b64decode(data['data'][0]['b64_json']))
print('saved output.png')
"
```
#### Notes
- Memory usage: Model weights ~6.5 GiB, peak memory ~10.9 GiB at 1024x1024 with CFG.
- Generation time: ~7.5 seconds for 30 inference steps at 1024x1024.
- `negative_prompt` requires `guidance_scale > 1` to take effect. Recommended range: 5.0-9.0.
- Default resolution is 1024x1024. Other supported sizes: 512x512, 768x768.
- The model uses dual CLIP text encoders (ViT-L/14 + OpenCLIP ViT-bigG/14) for rich text understanding.
### 1x NVIDIA GPU (24+ GB VRAM)
#### Environment
- OS: Linux
- Python: 3.10+
- Driver / runtime: NVIDIA CUDA environment with an A100
- vLLM version: Match the repository requirements for your checkout
- vLLM-Omni version or commit: Use the commit you are deploying from
#### Command
```bash
python examples/offline_inference/text_to_image/text_to_image.py \
--model stabilityai/stable-diffusion-xl-base-1.0 \
--prompt "a beautiful sunset over the ocean, photorealistic" \
--height 1024 --width 1024 \
--num-inference-steps 30 \
--guidance-scale 7.5 \
--seed 42 \
--output sdxl_output.png
```
#### Notes
- Memory usage: ~6.5 GiB model weights, ~10 GiB peak with CFG at 1024x1024.
- Fits on a single GPU with 24 GB VRAM (e.g., RTX 3090/4090, A5000).
- For GPUs with less VRAM, use `--enable-cpu-offload` to offload text encoders during diffusion.
+59
View File
@@ -0,0 +1,59 @@
"""
Tests for Stable Diffusion XL Base 1.0 model.
Validates single-GPU text-to-image generation with CFG (guidance_scale > 1).
"""
import pytest
from tests.helpers.mark import hardware_marks
from tests.helpers.runtime import (
OmniServer,
OmniServerParams,
OpenAIClientHandler,
dummy_messages_from_mix_data,
)
SINGLE_CARD_FEATURE_MARKS = hardware_marks(res={"cuda": "L4"})
POSITIVE_PROMPT = "A serene mountain landscape at sunset, photorealistic"
NEGATIVE_PROMPT = "blurry, low quality, distorted"
def _get_diffusion_feature_cases(model: str):
return [
pytest.param(
OmniServerParams(
model=model,
),
id="default",
marks=SINGLE_CARD_FEATURE_MARKS,
),
]
@pytest.mark.advanced_model
@pytest.mark.core_model
@pytest.mark.diffusion
@pytest.mark.parametrize(
"omni_server",
_get_diffusion_feature_cases(
model="stabilityai/stable-diffusion-xl-base-1.0",
),
indirect=True,
)
def test_text_to_image_001(omni_server: OmniServer, openai_client: OpenAIClientHandler):
messages = dummy_messages_from_mix_data(content_text=POSITIVE_PROMPT)
request_config = {
"model": omni_server.model,
"messages": messages,
"extra_body": {
"height": 1024,
"width": 1024,
"num_inference_steps": 30,
"negative_prompt": NEGATIVE_PROMPT,
"guidance_scale": 7.5,
"seed": 42,
},
}
openai_client.send_diffusion_request(request_config)
+6 -1
View File
@@ -903,7 +903,12 @@ class OmniDiffusionConfig:
)
else:
tf_config_dict = get_hf_file_to_dict("transformer/config.json", self.model)
self.set_tf_model_config(TransformerConfig.from_dict(tf_config_dict))
if tf_config_dict is None:
tf_config_dict = get_hf_file_to_dict("unet/config.json", self.model)
if tf_config_dict is not None:
self.set_tf_model_config(TransformerConfig.from_dict(tf_config_dict))
else:
self.set_tf_model_config(TransformerConfig())
else:
raise FileNotFoundError("model_index.json not found")
except (AttributeError, OSError, ValueError, FileNotFoundError):
@@ -0,0 +1,2 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
@@ -0,0 +1,422 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
import logging
import os
from collections.abc import Iterable
import torch
from diffusers.image_processor import VaeImageProcessor
from diffusers.schedulers.scheduling_euler_discrete import EulerDiscreteScheduler
from diffusers.utils.torch_utils import randn_tensor
from torch import nn
from transformers import CLIPTextModel, CLIPTextModelWithProjection, CLIPTokenizer
from vllm.model_executor.models.utils import AutoWeightsLoader
from vllm_omni.diffusion.data import DiffusionOutput, OmniDiffusionConfig
from vllm_omni.diffusion.distributed.autoencoders.autoencoder_kl import DistributedAutoencoderKL
from vllm_omni.diffusion.distributed.cfg_parallel import CFGParallelMixin
from vllm_omni.diffusion.distributed.utils import get_local_device
from vllm_omni.diffusion.model_loader.diffusers_loader import DiffusersPipelineLoader
from vllm_omni.diffusion.model_loader.hub_prefetch import from_pretrained_with_prefetch, prefetch_subfolders
from vllm_omni.diffusion.models.interface import SupportsComponentDiscovery
from vllm_omni.diffusion.models.sdxl.sdxl_unet import SDXLUNet2DConditionModel
from vllm_omni.diffusion.profiler.diffusion_pipeline_profiler import DiffusionPipelineProfilerMixin
from vllm_omni.diffusion.request import OmniDiffusionRequest
logger = logging.getLogger(__name__)
def get_sdxl_image_post_process_func(od_config: OmniDiffusionConfig):
if od_config.output_type == "latent":
return lambda x: x
image_processor = VaeImageProcessor(vae_scale_factor=8)
def post_process_func(images: torch.Tensor):
if images.device.type != "cpu":
images = images.cpu()
return image_processor.postprocess(images)
return post_process_func
class StableDiffusionXLPipeline(
nn.Module, CFGParallelMixin, DiffusionPipelineProfilerMixin, SupportsComponentDiscovery
):
_dit_modules: list[str] = ["unet"]
_encoder_modules: list[str] = ["text_encoder", "text_encoder_2"]
_vae_modules: list[str] = ["vae"]
_resident_modules: list[str] = []
def __init__(
self,
*,
od_config: OmniDiffusionConfig,
prefix: str = "",
):
super().__init__()
self.od_config = od_config
self.weights_sources = [
DiffusersPipelineLoader.ComponentSource(
model_or_path=od_config.model,
subfolder="unet",
revision=None,
prefix="unet.",
fall_back_to_pt=True,
)
]
self.device = get_local_device()
model = od_config.model
local_files_only = os.path.exists(model)
dtype = od_config.dtype
sdxl_subfolders = [
"scheduler",
"tokenizer",
"tokenizer_2",
"text_encoder",
"text_encoder_2",
"vae",
]
prefetch_subfolders(model, sdxl_subfolders, local_files_only=local_files_only)
self.scheduler = EulerDiscreteScheduler.from_pretrained(
model,
subfolder="scheduler",
local_files_only=local_files_only,
torch_dtype=torch.float32,
)
self.tokenizer = CLIPTokenizer.from_pretrained(model, subfolder="tokenizer", local_files_only=local_files_only)
self.tokenizer_2 = CLIPTokenizer.from_pretrained(
model, subfolder="tokenizer_2", local_files_only=local_files_only
)
self.text_encoder = from_pretrained_with_prefetch(
CLIPTextModel.from_pretrained,
model,
subfolder="text_encoder",
prefetch_list=sdxl_subfolders,
local_files_only=local_files_only,
torch_dtype=dtype,
)
self.text_encoder_2 = from_pretrained_with_prefetch(
CLIPTextModelWithProjection.from_pretrained,
model,
subfolder="text_encoder_2",
prefetch_list=sdxl_subfolders,
local_files_only=local_files_only,
torch_dtype=dtype,
)
self.unet = SDXLUNet2DConditionModel(od_config=od_config)
self.vae = from_pretrained_with_prefetch(
DistributedAutoencoderKL.from_pretrained,
model,
subfolder="vae",
prefetch_list=sdxl_subfolders,
local_files_only=local_files_only,
torch_dtype=dtype,
).to(self.device)
self.vae_scale_factor = 8
self.image_processor = VaeImageProcessor(vae_scale_factor=self.vae_scale_factor)
self.tokenizer_max_length = 77
self.default_sample_size = 128 # 1024 / 8
self.output_type = self.od_config.output_type
self.setup_diffusion_pipeline_profiler(
enable_diffusion_pipeline_profiler=self.od_config.enable_diffusion_pipeline_profiler
)
def _get_clip_prompt_embeds(
self,
prompt: str | list[str],
tokenizer: CLIPTokenizer,
text_encoder: nn.Module,
num_images_per_prompt: int = 1,
return_pooled: bool = False,
) -> tuple[torch.Tensor, torch.Tensor | None]:
prompt = [prompt] if isinstance(prompt, str) else prompt
batch_size = len(prompt)
text_inputs = tokenizer(
prompt,
padding="max_length",
max_length=self.tokenizer_max_length,
truncation=True,
return_tensors="pt",
)
text_input_ids = text_inputs.input_ids
text_encoder_device = next(text_encoder.parameters()).device
outputs = text_encoder(text_input_ids.to(text_encoder_device), output_hidden_states=True)
prompt_embeds = outputs.hidden_states[-2].to(dtype=self.od_config.dtype, device=self.device)
_, seq_len, _ = prompt_embeds.shape
prompt_embeds = prompt_embeds.repeat(1, num_images_per_prompt, 1)
prompt_embeds = prompt_embeds.view(batch_size * num_images_per_prompt, seq_len, -1)
pooled_prompt_embeds = None
if return_pooled:
pooled_prompt_embeds = outputs[0].to(dtype=self.od_config.dtype, device=self.device)
pooled_prompt_embeds = pooled_prompt_embeds.repeat(num_images_per_prompt, 1)
pooled_prompt_embeds = pooled_prompt_embeds.view(batch_size * num_images_per_prompt, -1)
return prompt_embeds, pooled_prompt_embeds
def encode_prompt(
self,
prompt: str | list[str],
num_images_per_prompt: int = 1,
) -> tuple[torch.Tensor, torch.Tensor]:
# Encode with first CLIP (ViT-L/14, 768-dim hidden states)
prompt_embeds_1, _ = self._get_clip_prompt_embeds(
prompt,
self.tokenizer,
self.text_encoder,
num_images_per_prompt,
return_pooled=False,
)
# Encode with second CLIP (OpenCLIP ViT-bigG/14, 1280-dim hidden states + pooled)
prompt_embeds_2, pooled_prompt_embeds = self._get_clip_prompt_embeds(
prompt,
self.tokenizer_2,
self.text_encoder_2,
num_images_per_prompt,
return_pooled=True,
)
# Concatenate along feature dimension: 768 + 1280 = 2048
prompt_embeds = torch.cat([prompt_embeds_1, prompt_embeds_2], dim=-1)
return prompt_embeds, pooled_prompt_embeds
def _get_add_time_ids(
self,
original_size: tuple[int, int],
crops_coords_top_left: tuple[int, int],
target_size: tuple[int, int],
batch_size: int,
) -> torch.Tensor:
add_time_ids = torch.tensor(
[list(original_size) + list(crops_coords_top_left) + list(target_size)],
dtype=torch.float32,
device=self.device,
)
return add_time_ids.repeat(batch_size, 1)
def prepare_latents(
self,
batch_size: int,
num_channels_latents: int,
height: int,
width: int,
generator: torch.Generator | list[torch.Generator] | None,
latents: torch.Tensor | None = None,
) -> torch.Tensor:
if latents is not None:
return latents.to(device=self.device, dtype=self.od_config.dtype)
shape = (
batch_size,
num_channels_latents,
height // self.vae_scale_factor,
width // self.vae_scale_factor,
)
latents = randn_tensor(shape, generator=generator, device=self.device, dtype=self.od_config.dtype)
latents = latents * self.scheduler.init_noise_sigma
return latents
def predict_noise(self, **kwargs):
result = self.unet(**kwargs)
return result[0]
@property
def guidance_scale(self):
return self._guidance_scale
@property
def num_timesteps(self):
return self._num_timesteps
@property
def current_timestep(self):
return self._current_timestep
@property
def interrupt(self):
return self._interrupt
def diffuse(
self,
latents: torch.Tensor,
timesteps: torch.Tensor,
prompt_embeds: torch.Tensor,
added_cond_kwargs: dict,
negative_prompt_embeds: torch.Tensor | None,
negative_added_cond_kwargs: dict | None,
do_cfg: bool,
guidance_scale: float,
) -> torch.Tensor:
for _, t in enumerate(timesteps):
if self.interrupt:
continue
self._current_timestep = t
# Scale model input
latent_model_input = self.scheduler.scale_model_input(latents, t)
timestep = t.expand(latent_model_input.shape[0]).to(device=self.device, dtype=self.od_config.dtype)
positive_kwargs = {
"hidden_states": latent_model_input,
"timestep": timestep,
"encoder_hidden_states": prompt_embeds,
"added_cond_kwargs": added_cond_kwargs,
"return_dict": False,
}
if do_cfg:
negative_kwargs = {
"hidden_states": latent_model_input,
"timestep": timestep,
"encoder_hidden_states": negative_prompt_embeds,
"added_cond_kwargs": negative_added_cond_kwargs,
"return_dict": False,
}
else:
negative_kwargs = None
noise_pred = self.predict_noise_maybe_with_cfg(
do_cfg,
guidance_scale,
positive_kwargs,
negative_kwargs,
cfg_normalize=False,
)
latents = self.scheduler_step_maybe_with_cfg(noise_pred, t, latents, do_cfg)
return latents
def forward(
self,
req: OmniDiffusionRequest,
prompt: str | list[str] = "",
negative_prompt: str | list[str] = "",
height: int | None = None,
width: int | None = None,
num_inference_steps: int = 50,
num_images_per_prompt: int = 1,
generator: torch.Generator | list[torch.Generator] | None = None,
latents: torch.Tensor | None = None,
) -> DiffusionOutput:
prompt = [p if isinstance(p, str) else (p.get("prompt") or "") for p in req.prompts] or prompt
negative_prompt = [
"" if isinstance(p, str) else (p.get("negative_prompt") or "") for p in req.prompts
] or negative_prompt
height = req.sampling_params.height or self.default_sample_size * self.vae_scale_factor
width = req.sampling_params.width or self.default_sample_size * self.vae_scale_factor
num_inference_steps = req.sampling_params.num_inference_steps or num_inference_steps
generator = req.sampling_params.generator or generator
num_images_per_prompt = (
req.sampling_params.num_outputs_per_prompt
if req.sampling_params.num_outputs_per_prompt > 0
else num_images_per_prompt
)
self._guidance_scale = req.sampling_params.guidance_scale
self._current_timestep = None
self._interrupt = False
if isinstance(prompt, str):
batch_size = 1
else:
batch_size = len(prompt)
# Encode prompt
prompt_embeds, pooled_prompt_embeds = self.encode_prompt(
prompt=prompt,
num_images_per_prompt=num_images_per_prompt,
)
# Encode negative prompt for CFG
do_cfg = self.guidance_scale > 1.0
negative_prompt_embeds = None
negative_pooled_prompt_embeds = None
if do_cfg:
negative_prompt_embeds, negative_pooled_prompt_embeds = self.encode_prompt(
prompt=negative_prompt,
num_images_per_prompt=num_images_per_prompt,
)
# Prepare added conditioning
original_size = (height, width)
target_size = (height, width)
crops_coords_top_left = (0, 0)
add_time_ids = self._get_add_time_ids(
original_size,
crops_coords_top_left,
target_size,
batch_size * num_images_per_prompt,
)
added_cond_kwargs = {
"text_embeds": pooled_prompt_embeds,
"time_ids": add_time_ids,
}
negative_added_cond_kwargs = None
if do_cfg:
negative_added_cond_kwargs = {
"text_embeds": negative_pooled_prompt_embeds,
"time_ids": add_time_ids,
}
# XPU workaround: set_timesteps uses numpy internally and needs CPU tensors.
for attr in ("alphas_cumprod", "sigmas", "timesteps"):
if hasattr(self.scheduler, attr):
val = getattr(self.scheduler, attr)
if hasattr(val, "device") and val.device.type != "cpu":
setattr(self.scheduler, attr, val.cpu())
self.scheduler.set_timesteps(num_inference_steps)
self.scheduler.sigmas = self.scheduler.sigmas.to(self.device)
self.scheduler.timesteps = self.scheduler.timesteps.to(self.device)
timesteps = self.scheduler.timesteps
# Prepare latents (uses scheduler.init_noise_sigma which depends on set_timesteps)
latents = self.prepare_latents(
batch_size * num_images_per_prompt,
self.unet.in_channels,
height,
width,
generator,
latents,
)
self._num_timesteps = len(timesteps)
# Denoising loop
latents = self.diffuse(
latents=latents,
timesteps=timesteps,
prompt_embeds=prompt_embeds,
added_cond_kwargs=added_cond_kwargs,
negative_prompt_embeds=negative_prompt_embeds,
negative_added_cond_kwargs=negative_added_cond_kwargs,
do_cfg=do_cfg,
guidance_scale=self.guidance_scale,
)
self._current_timestep = None
if self.output_type == "latent":
image = latents
else:
latents = latents.to(self.vae.dtype)
latents = latents / self.vae.config.scaling_factor
image = self.vae.decode(latents, return_dict=False)[0]
return DiffusionOutput(
output=image,
stage_durations=self.stage_durations if hasattr(self, "stage_durations") else None,
)
def load_weights(self, weights: Iterable[tuple[str, torch.Tensor]]) -> set[str]:
loader = AutoWeightsLoader(self)
return loader.load_weights(weights)
@@ -0,0 +1,871 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
import math
from collections.abc import Iterable
import torch
import torch.nn as nn
import torch.nn.functional as F
from vllm.logger import init_logger
from vllm.model_executor.layers.linear import (
ColumnParallelLinear,
MergedColumnParallelLinear,
QKVParallelLinear,
RowParallelLinear,
)
from vllm.model_executor.model_loader.weight_utils import default_weight_loader
from vllm_omni.diffusion.attention.layer import Attention
from vllm_omni.diffusion.data import OmniDiffusionConfig
from vllm_omni.diffusion.distributed.sp_plan import (
SequenceParallelInput,
SequenceParallelOutput,
)
logger = init_logger(__name__)
def get_timestep_embedding(
timesteps: torch.Tensor,
embedding_dim: int,
max_period: int = 10000,
) -> torch.Tensor:
half_dim = embedding_dim // 2
exponent = (
-math.log(max_period) * torch.arange(0, half_dim, dtype=torch.float32, device=timesteps.device) / half_dim
)
emb = timesteps[:, None].float() * torch.exp(exponent)[None, :]
emb = torch.cat([torch.cos(emb), torch.sin(emb)], dim=-1)
if embedding_dim % 2 == 1:
emb = F.pad(emb, (0, 1))
return emb
class SDXLTimestepEmbedding(nn.Module):
def __init__(self, in_channels: int, time_embed_dim: int):
super().__init__()
self.linear_1 = nn.Linear(in_channels, time_embed_dim)
self.act = nn.SiLU()
self.linear_2 = nn.Linear(time_embed_dim, time_embed_dim)
def forward(self, sample: torch.Tensor) -> torch.Tensor:
sample = self.linear_1(sample)
sample = self.act(sample)
sample = self.linear_2(sample)
return sample
class SDXLAddTimestepEmbedding(nn.Module):
def __init__(self, addition_time_embed_dim: int, text_embed_dim: int, time_embed_dim: int):
super().__init__()
self.linear_1 = nn.Linear(text_embed_dim + addition_time_embed_dim * 6, time_embed_dim)
self.act = nn.SiLU()
self.linear_2 = nn.Linear(time_embed_dim, time_embed_dim)
def forward(self, text_embeds: torch.Tensor, time_ids: torch.Tensor) -> torch.Tensor:
time_embeds = get_timestep_embedding(time_ids.flatten(), 256)
time_embeds = time_embeds.reshape(text_embeds.shape[0], -1)
add_embeds = torch.cat([text_embeds, time_embeds], dim=-1)
add_embeds = add_embeds.to(dtype=self.linear_1.weight.dtype)
add_embeds = self.linear_1(add_embeds)
add_embeds = self.act(add_embeds)
add_embeds = self.linear_2(add_embeds)
return add_embeds
class SDXLResnetBlock2D(nn.Module):
def __init__(
self,
in_channels: int,
out_channels: int,
time_embed_dim: int,
groups: int = 32,
):
super().__init__()
self.norm1 = nn.GroupNorm(groups, in_channels, eps=1e-5)
self.conv1 = nn.Conv2d(in_channels, out_channels, kernel_size=3, padding=1)
self.time_emb_proj = nn.Linear(time_embed_dim, out_channels)
self.norm2 = nn.GroupNorm(groups, out_channels, eps=1e-5)
self.conv2 = nn.Conv2d(out_channels, out_channels, kernel_size=3, padding=1)
self.nonlinearity = nn.SiLU()
self.conv_shortcut = None
if in_channels != out_channels:
self.conv_shortcut = nn.Conv2d(in_channels, out_channels, kernel_size=1)
def forward(self, hidden_states: torch.Tensor, temb: torch.Tensor) -> torch.Tensor:
residual = hidden_states
hidden_states = self.norm1(hidden_states)
hidden_states = self.nonlinearity(hidden_states)
hidden_states = self.conv1(hidden_states)
temb = self.nonlinearity(temb)
temb = self.time_emb_proj(temb)
hidden_states = hidden_states + temb[:, :, None, None]
hidden_states = self.norm2(hidden_states)
hidden_states = self.nonlinearity(hidden_states)
hidden_states = self.conv2(hidden_states)
if self.conv_shortcut is not None:
residual = self.conv_shortcut(residual)
return hidden_states + residual
class SDXLGEGLU(nn.Module):
def __init__(self, dim_in: int, dim_out: int):
super().__init__()
self.proj = MergedColumnParallelLinear(dim_in, [dim_out, dim_out], bias=True)
def forward(self, hidden_states: torch.Tensor) -> torch.Tensor:
hidden_states, _ = self.proj(hidden_states)
hidden_states, gate = hidden_states.chunk(2, dim=-1)
return hidden_states * F.gelu(gate)
class SDXLFeedForward(nn.Module):
def __init__(self, dim: int, inner_dim: int):
super().__init__()
self.geglu = SDXLGEGLU(dim, inner_dim)
self.out_proj = RowParallelLinear(inner_dim, dim, bias=True)
def forward(self, hidden_states: torch.Tensor) -> torch.Tensor:
hidden_states = self.geglu(hidden_states)
hidden_states, _ = self.out_proj(hidden_states)
return hidden_states
class SDXLSelfAttention(nn.Module):
def __init__(self, dim: int, num_attention_heads: int, attention_head_dim: int, prefix: str = ""):
super().__init__()
inner_dim = num_attention_heads * attention_head_dim
self.to_qkv = QKVParallelLinear(
hidden_size=dim,
head_size=attention_head_dim,
total_num_heads=num_attention_heads,
bias=False,
)
self.to_out = nn.ModuleList(
[
RowParallelLinear(inner_dim, dim, bias=True),
]
)
self.attention = Attention(
num_heads=num_attention_heads,
head_size=attention_head_dim,
causal=False,
softmax_scale=1.0 / math.sqrt(attention_head_dim),
role="self",
prefix=prefix,
)
class SDXLCrossAttention(nn.Module):
def __init__(
self, dim: int, cross_attention_dim: int, num_attention_heads: int, attention_head_dim: int, prefix: str = ""
):
super().__init__()
inner_dim = num_attention_heads * attention_head_dim
self.to_q = ColumnParallelLinear(dim, inner_dim, bias=False)
self.to_k = ColumnParallelLinear(cross_attention_dim, inner_dim, bias=False)
self.to_v = ColumnParallelLinear(cross_attention_dim, inner_dim, bias=False)
self.to_out = nn.ModuleList(
[
RowParallelLinear(inner_dim, dim, bias=True),
]
)
self.attention = Attention(
num_heads=num_attention_heads,
head_size=attention_head_dim,
causal=False,
softmax_scale=1.0 / math.sqrt(attention_head_dim),
role="cross",
prefix=prefix,
skip_sequence_parallel=True,
)
class SDXLBasicTransformerBlock(nn.Module):
def __init__(
self,
dim: int,
num_attention_heads: int,
attention_head_dim: int,
cross_attention_dim: int,
prefix: str = "",
):
super().__init__()
self.num_attention_heads = num_attention_heads
self.attention_head_dim = attention_head_dim
self.norm1 = nn.LayerNorm(dim, elementwise_affine=True)
self.norm2 = nn.LayerNorm(dim, elementwise_affine=True)
self.norm3 = nn.LayerNorm(dim, elementwise_affine=True)
self.attn1 = SDXLSelfAttention(
dim=dim,
num_attention_heads=num_attention_heads,
attention_head_dim=attention_head_dim,
prefix=f"{prefix}.attn1",
)
self.attn2 = SDXLCrossAttention(
dim=dim,
cross_attention_dim=cross_attention_dim,
num_attention_heads=num_attention_heads,
attention_head_dim=attention_head_dim,
prefix=f"{prefix}.attn2",
)
# Feed-forward
self.ff = SDXLFeedForward(dim=dim, inner_dim=dim * 4)
def forward(
self,
hidden_states: torch.Tensor,
encoder_hidden_states: torch.Tensor,
temb: None = None,
) -> torch.Tensor:
local_num_heads = self.attn1.to_qkv.num_heads
head_dim = self.attention_head_dim
# Self-attention
norm_hidden = self.norm1(hidden_states)
qkv, _ = self.attn1.to_qkv(norm_hidden)
q, k, v = qkv.chunk(3, dim=-1)
q = q.unflatten(-1, (local_num_heads, head_dim))
k = k.unflatten(-1, (local_num_heads, head_dim))
v = v.unflatten(-1, (local_num_heads, head_dim))
attn_out = self.attn1.attention(q, k, v)
attn_out = attn_out.flatten(-2)
attn_out, _ = self.attn1.to_out[0](attn_out)
hidden_states = hidden_states + attn_out
# Cross-attention
norm_hidden = self.norm2(hidden_states)
q, _ = self.attn2.to_q(norm_hidden)
k, _ = self.attn2.to_k(encoder_hidden_states)
v, _ = self.attn2.to_v(encoder_hidden_states)
q = q.unflatten(-1, (local_num_heads, head_dim))
k = k.unflatten(-1, (local_num_heads, head_dim))
v = v.unflatten(-1, (local_num_heads, head_dim))
attn_out = self.attn2.attention(q, k, v)
attn_out = attn_out.flatten(-2)
attn_out, _ = self.attn2.to_out[0](attn_out)
hidden_states = hidden_states + attn_out
# Feed-forward
norm_hidden = self.norm3(hidden_states)
ff_out = self.ff(norm_hidden)
hidden_states = hidden_states + ff_out
return hidden_states
class SDXLTransformer2DModel(nn.Module):
def __init__(
self,
num_attention_heads: int,
attention_head_dim: int,
in_channels: int,
num_layers: int,
cross_attention_dim: int,
prefix: str = "",
):
super().__init__()
inner_dim = num_attention_heads * attention_head_dim
self.norm = nn.GroupNorm(32, in_channels, eps=1e-6, affine=True)
self.proj_in = nn.Linear(in_channels, inner_dim)
self.transformer_blocks = nn.ModuleList(
[
SDXLBasicTransformerBlock(
dim=inner_dim,
num_attention_heads=num_attention_heads,
attention_head_dim=attention_head_dim,
cross_attention_dim=cross_attention_dim,
prefix=f"{prefix}.transformer_blocks.{i}",
)
for i in range(num_layers)
]
)
self.proj_out = nn.Linear(inner_dim, in_channels)
def forward(
self,
hidden_states: torch.Tensor,
encoder_hidden_states: torch.Tensor,
) -> torch.Tensor:
batch, channels, height, width = hidden_states.shape
residual = hidden_states
hidden_states = self.norm(hidden_states)
hidden_states = hidden_states.permute(0, 2, 3, 1).reshape(batch, height * width, channels)
hidden_states = self.proj_in(hidden_states)
for block in self.transformer_blocks:
hidden_states = block(hidden_states, encoder_hidden_states)
hidden_states = self.proj_out(hidden_states)
hidden_states = hidden_states.reshape(batch, height, width, channels).permute(0, 3, 1, 2)
return hidden_states + residual
class SDXLDownsample2D(nn.Module):
def __init__(self, channels: int):
super().__init__()
self.conv = nn.Conv2d(channels, channels, kernel_size=3, stride=2, padding=1)
def forward(self, hidden_states: torch.Tensor) -> torch.Tensor:
return self.conv(hidden_states)
class SDXLUpsample2D(nn.Module):
def __init__(self, channels: int):
super().__init__()
self.conv = nn.Conv2d(channels, channels, kernel_size=3, padding=1)
def forward(self, hidden_states: torch.Tensor) -> torch.Tensor:
hidden_states = F.interpolate(hidden_states, scale_factor=2.0, mode="nearest")
return self.conv(hidden_states)
class SDXLDownBlock2D(nn.Module):
def __init__(
self,
in_channels: int,
out_channels: int,
time_embed_dim: int,
num_layers: int = 2,
add_downsample: bool = True,
):
super().__init__()
self.resnets = nn.ModuleList()
for i in range(num_layers):
res_in = in_channels if i == 0 else out_channels
self.resnets.append(SDXLResnetBlock2D(res_in, out_channels, time_embed_dim))
self.downsamplers = None
if add_downsample:
self.downsamplers = nn.ModuleList([SDXLDownsample2D(out_channels)])
def forward(
self,
hidden_states: torch.Tensor,
temb: torch.Tensor,
) -> tuple[torch.Tensor, list[torch.Tensor]]:
output_states = []
for resnet in self.resnets:
hidden_states = resnet(hidden_states, temb)
output_states.append(hidden_states)
if self.downsamplers is not None:
hidden_states = self.downsamplers[0](hidden_states)
output_states.append(hidden_states)
return hidden_states, output_states
class SDXLCrossAttnDownBlock2D(nn.Module):
def __init__(
self,
in_channels: int,
out_channels: int,
time_embed_dim: int,
num_layers: int = 2,
num_attention_heads: int = 10,
cross_attention_dim: int = 2048,
transformer_layers_per_block: int = 1,
add_downsample: bool = True,
prefix: str = "",
):
super().__init__()
attention_head_dim = out_channels // num_attention_heads
self.resnets = nn.ModuleList()
self.attentions = nn.ModuleList()
for i in range(num_layers):
res_in = in_channels if i == 0 else out_channels
self.resnets.append(SDXLResnetBlock2D(res_in, out_channels, time_embed_dim))
self.attentions.append(
SDXLTransformer2DModel(
num_attention_heads=num_attention_heads,
attention_head_dim=attention_head_dim,
in_channels=out_channels,
num_layers=transformer_layers_per_block,
cross_attention_dim=cross_attention_dim,
prefix=f"{prefix}.attentions.{i}",
)
)
self.downsamplers = None
if add_downsample:
self.downsamplers = nn.ModuleList([SDXLDownsample2D(out_channels)])
def forward(
self,
hidden_states: torch.Tensor,
temb: torch.Tensor,
encoder_hidden_states: torch.Tensor,
) -> tuple[torch.Tensor, list[torch.Tensor]]:
output_states = []
for resnet, attn in zip(self.resnets, self.attentions):
hidden_states = resnet(hidden_states, temb)
hidden_states = attn(hidden_states, encoder_hidden_states)
output_states.append(hidden_states)
if self.downsamplers is not None:
hidden_states = self.downsamplers[0](hidden_states)
output_states.append(hidden_states)
return hidden_states, output_states
class SDXLUNetMidBlock2DCrossAttn(nn.Module):
def __init__(
self,
in_channels: int,
time_embed_dim: int,
num_attention_heads: int = 20,
cross_attention_dim: int = 2048,
transformer_layers_per_block: int = 10,
prefix: str = "",
):
super().__init__()
attention_head_dim = in_channels // num_attention_heads
self.resnets = nn.ModuleList(
[
SDXLResnetBlock2D(in_channels, in_channels, time_embed_dim),
SDXLResnetBlock2D(in_channels, in_channels, time_embed_dim),
]
)
self.attentions = nn.ModuleList(
[
SDXLTransformer2DModel(
num_attention_heads=num_attention_heads,
attention_head_dim=attention_head_dim,
in_channels=in_channels,
num_layers=transformer_layers_per_block,
cross_attention_dim=cross_attention_dim,
prefix=f"{prefix}.attentions.0",
)
]
)
def forward(
self,
hidden_states: torch.Tensor,
temb: torch.Tensor,
encoder_hidden_states: torch.Tensor,
) -> torch.Tensor:
hidden_states = self.resnets[0](hidden_states, temb)
hidden_states = self.attentions[0](hidden_states, encoder_hidden_states)
hidden_states = self.resnets[1](hidden_states, temb)
return hidden_states
class SDXLUpBlock2D(nn.Module):
def __init__(
self,
in_channels: int,
prev_output_channel: int,
out_channels: int,
time_embed_dim: int,
num_layers: int = 3,
add_upsample: bool = True,
resnet_in_channels_list: list[int] | None = None,
):
super().__init__()
self.resnets = nn.ModuleList()
for i in range(num_layers):
if resnet_in_channels_list is not None:
res_in = resnet_in_channels_list[i]
else:
skip_channels = prev_output_channel if i == 0 else out_channels
res_in = (in_channels if i == 0 else out_channels) + skip_channels
self.resnets.append(SDXLResnetBlock2D(res_in, out_channels, time_embed_dim))
self.upsamplers = None
if add_upsample:
self.upsamplers = nn.ModuleList([SDXLUpsample2D(out_channels)])
def forward(
self,
hidden_states: torch.Tensor,
temb: torch.Tensor,
res_hidden_states_tuple: tuple[torch.Tensor, ...],
) -> torch.Tensor:
for resnet in self.resnets:
res_hidden = res_hidden_states_tuple[-1]
res_hidden_states_tuple = res_hidden_states_tuple[:-1]
hidden_states = torch.cat([hidden_states, res_hidden], dim=1)
hidden_states = resnet(hidden_states, temb)
if self.upsamplers is not None:
hidden_states = self.upsamplers[0](hidden_states)
return hidden_states
class SDXLCrossAttnUpBlock2D(nn.Module):
def __init__(
self,
in_channels: int,
prev_output_channel: int,
out_channels: int,
time_embed_dim: int,
num_layers: int = 3,
num_attention_heads: int = 10,
cross_attention_dim: int = 2048,
transformer_layers_per_block: int = 1,
add_upsample: bool = True,
prefix: str = "",
resnet_in_channels_list: list[int] | None = None,
):
super().__init__()
attention_head_dim = out_channels // num_attention_heads
self.resnets = nn.ModuleList()
self.attentions = nn.ModuleList()
for i in range(num_layers):
if resnet_in_channels_list is not None:
res_in = resnet_in_channels_list[i]
else:
skip_channels = prev_output_channel if i == 0 else out_channels
res_in = (in_channels if i == 0 else out_channels) + skip_channels
self.resnets.append(SDXLResnetBlock2D(res_in, out_channels, time_embed_dim))
self.attentions.append(
SDXLTransformer2DModel(
num_attention_heads=num_attention_heads,
attention_head_dim=attention_head_dim,
in_channels=out_channels,
num_layers=transformer_layers_per_block,
cross_attention_dim=cross_attention_dim,
prefix=f"{prefix}.attentions.{i}",
)
)
self.upsamplers = None
if add_upsample:
self.upsamplers = nn.ModuleList([SDXLUpsample2D(out_channels)])
def forward(
self,
hidden_states: torch.Tensor,
temb: torch.Tensor,
encoder_hidden_states: torch.Tensor,
res_hidden_states_tuple: tuple[torch.Tensor, ...],
) -> torch.Tensor:
for resnet, attn in zip(self.resnets, self.attentions):
res_hidden = res_hidden_states_tuple[-1]
res_hidden_states_tuple = res_hidden_states_tuple[:-1]
hidden_states = torch.cat([hidden_states, res_hidden], dim=1)
hidden_states = resnet(hidden_states, temb)
hidden_states = attn(hidden_states, encoder_hidden_states)
if self.upsamplers is not None:
hidden_states = self.upsamplers[0](hidden_states)
return hidden_states
def _build_sdxl_sp_plan() -> dict:
"""Build _sp_plan for all attention modules in the SDXL UNet.
Shards hidden_states at the first transformer block input and gathers
at proj_out for each SDXLTransformer2DModel instance.
"""
plan: dict = {}
attn_paths = [
# down_blocks[1]: 2 attention modules
"down_blocks.1.attentions.0",
"down_blocks.1.attentions.1",
# down_blocks[2]: 2 attention modules
"down_blocks.2.attentions.0",
"down_blocks.2.attentions.1",
# mid_block: 1 attention module
"mid_block.attentions.0",
# up_blocks[0]: 3 attention modules
"up_blocks.0.attentions.0",
"up_blocks.0.attentions.1",
"up_blocks.0.attentions.2",
# up_blocks[1]: 3 attention modules
"up_blocks.1.attentions.0",
"up_blocks.1.attentions.1",
"up_blocks.1.attentions.2",
]
for path in attn_paths:
plan[f"{path}.transformer_blocks.0"] = {
"hidden_states": SequenceParallelInput(split_dim=1, expected_dims=3, auto_pad=True),
}
plan[f"{path}.proj_out"] = SequenceParallelOutput(gather_dim=1, expected_dims=3)
return plan
class SDXLUNet2DConditionModel(nn.Module):
_repeated_blocks = ["SDXLBasicTransformerBlock", "SDXLResnetBlock2D"]
_layerwise_offload_blocks_attrs = ["down_blocks", "up_blocks"]
_sp_plan = _build_sdxl_sp_plan()
@staticmethod
def _is_shardable_block(name: str, module) -> bool:
"""Match transformer blocks and ResNet blocks for HSDP sharding."""
if not name.split(".")[-1].isdigit():
return False
return "transformer_blocks" in name or "resnets" in name
_hsdp_shard_conditions = [_is_shardable_block]
def __init__(self, *, od_config: OmniDiffusionConfig):
super().__init__()
self.od_config = od_config
# SDXL Base configuration
model_channels = 320
time_embed_dim = model_channels * 4 # 1280
block_out_channels = [320, 640, 1280]
layers_per_block = 2
transformer_layers_per_block = [1, 2, 10]
cross_attention_dim = 2048
addition_time_embed_dim = 256
num_head_channels = 64
self.in_channels = 4
self.out_channels = 4
# Input convolution
self.conv_in = nn.Conv2d(4, model_channels, kernel_size=3, padding=1)
# Time embedding
self.time_proj_dim = model_channels
self.time_embedding = SDXLTimestepEmbedding(model_channels, time_embed_dim)
# Addition embedding (for pooled text + crop/size conditioning)
self.add_embedding = SDXLAddTimestepEmbedding(
addition_time_embed_dim=addition_time_embed_dim,
text_embed_dim=1280,
time_embed_dim=time_embed_dim,
)
# Down blocks
self.down_blocks = nn.ModuleList()
# Level 0: DownBlock2D (no attention)
self.down_blocks.append(
SDXLDownBlock2D(
in_channels=model_channels,
out_channels=block_out_channels[0],
time_embed_dim=time_embed_dim,
num_layers=layers_per_block,
add_downsample=True,
)
)
# Level 1: CrossAttnDownBlock2D
self.down_blocks.append(
SDXLCrossAttnDownBlock2D(
in_channels=block_out_channels[0],
out_channels=block_out_channels[1],
time_embed_dim=time_embed_dim,
num_layers=layers_per_block,
num_attention_heads=block_out_channels[1] // num_head_channels,
cross_attention_dim=cross_attention_dim,
transformer_layers_per_block=transformer_layers_per_block[1],
add_downsample=True,
prefix="down_blocks.1",
)
)
# Level 2: CrossAttnDownBlock2D (no downsampler)
self.down_blocks.append(
SDXLCrossAttnDownBlock2D(
in_channels=block_out_channels[1],
out_channels=block_out_channels[2],
time_embed_dim=time_embed_dim,
num_layers=layers_per_block,
num_attention_heads=block_out_channels[2] // num_head_channels,
cross_attention_dim=cross_attention_dim,
transformer_layers_per_block=transformer_layers_per_block[2],
add_downsample=False,
prefix="down_blocks.2",
)
)
# Mid block
self.mid_block = SDXLUNetMidBlock2DCrossAttn(
in_channels=block_out_channels[2],
time_embed_dim=time_embed_dim,
num_attention_heads=block_out_channels[2] // num_head_channels,
cross_attention_dim=cross_attention_dim,
transformer_layers_per_block=transformer_layers_per_block[2],
prefix="mid_block",
)
# Up blocks (reversed, with skip connections)
# Skip connection channels consumed from down_block_res_samples (right to left):
# down_block_res_samples channels = [320, 320, 320, 320, 640, 640, 640, 1280, 1280]
# up_blocks[0] consumes last 3: skips are [1280, 1280, 640] (consumed from right)
# up_blocks[1] consumes next 3: skips are [640, 640, 320] (consumed from right)
# up_blocks[2] consumes next 3: skips are [320, 320, 320] (consumed from right)
self.up_blocks = nn.ModuleList()
# Level 0: CrossAttnUpBlock2D (mirrors down_blocks[2])
# Input from mid: 1280, skips (consumed right-to-left): 1280, 1280, 640
self.up_blocks.append(
SDXLCrossAttnUpBlock2D(
in_channels=block_out_channels[2],
prev_output_channel=block_out_channels[2],
out_channels=block_out_channels[2],
time_embed_dim=time_embed_dim,
num_layers=layers_per_block + 1,
num_attention_heads=block_out_channels[2] // num_head_channels,
cross_attention_dim=cross_attention_dim,
transformer_layers_per_block=transformer_layers_per_block[2],
add_upsample=True,
prefix="up_blocks.0",
resnet_in_channels_list=[1280 + 1280, 1280 + 1280, 1280 + 640],
)
)
# Level 1: CrossAttnUpBlock2D (mirrors down_blocks[1])
# Input from up_blocks[0]: 1280, skips (consumed right-to-left): 640, 640, 320
self.up_blocks.append(
SDXLCrossAttnUpBlock2D(
in_channels=block_out_channels[2],
prev_output_channel=block_out_channels[1],
out_channels=block_out_channels[1],
time_embed_dim=time_embed_dim,
num_layers=layers_per_block + 1,
num_attention_heads=block_out_channels[1] // num_head_channels,
cross_attention_dim=cross_attention_dim,
transformer_layers_per_block=transformer_layers_per_block[1],
add_upsample=True,
prefix="up_blocks.1",
resnet_in_channels_list=[1280 + 640, 640 + 640, 640 + 320],
)
)
# Level 2: UpBlock2D (mirrors down_blocks[0], no attention, no upsample)
# Input from up_blocks[1]: 640, skips (consumed right-to-left): 320, 320, 320
self.up_blocks.append(
SDXLUpBlock2D(
in_channels=block_out_channels[1],
prev_output_channel=block_out_channels[0],
out_channels=block_out_channels[0],
time_embed_dim=time_embed_dim,
num_layers=layers_per_block + 1,
add_upsample=False,
resnet_in_channels_list=[640 + 320, 320 + 320, 320 + 320],
)
)
# Output
self.conv_norm_out = nn.GroupNorm(32, block_out_channels[0], eps=1e-5)
self.conv_act = nn.SiLU()
self.conv_out = nn.Conv2d(block_out_channels[0], 4, kernel_size=3, padding=1)
def forward(
self,
hidden_states: torch.Tensor,
timestep: torch.Tensor,
encoder_hidden_states: torch.Tensor,
added_cond_kwargs: dict,
return_dict: bool = False,
) -> tuple[torch.Tensor]:
# Time embedding
dtype = hidden_states.dtype
t_emb = get_timestep_embedding(timestep, self.time_proj_dim).to(dtype=dtype)
emb = self.time_embedding(t_emb)
# Addition embedding (pooled text + size/crop)
text_embeds = added_cond_kwargs["text_embeds"].to(dtype=dtype)
time_ids = added_cond_kwargs["time_ids"]
add_emb = self.add_embedding(text_embeds, time_ids)
emb = emb + add_emb
# Input conv
hidden_states = self.conv_in(hidden_states)
# Down blocks
down_block_res_samples = [hidden_states]
for down_block in self.down_blocks:
if isinstance(down_block, SDXLCrossAttnDownBlock2D):
hidden_states, res_samples = down_block(hidden_states, emb, encoder_hidden_states)
else:
hidden_states, res_samples = down_block(hidden_states, emb)
down_block_res_samples.extend(res_samples)
# Mid block
hidden_states = self.mid_block(hidden_states, emb, encoder_hidden_states)
# Up blocks
for up_block in self.up_blocks:
# Determine how many skip connections this block needs
n_resnets = len(up_block.resnets)
res_samples = tuple(down_block_res_samples[-n_resnets:])
down_block_res_samples = down_block_res_samples[:-n_resnets]
if isinstance(up_block, SDXLCrossAttnUpBlock2D):
hidden_states = up_block(hidden_states, emb, encoder_hidden_states, res_samples)
else:
hidden_states = up_block(hidden_states, emb, res_samples)
# Output
hidden_states = self.conv_norm_out(hidden_states)
hidden_states = self.conv_act(hidden_states)
hidden_states = self.conv_out(hidden_states)
return (hidden_states,)
def load_weights(self, weights: Iterable[tuple[str, torch.Tensor]]) -> set[str]:
stacked_params_mapping = [
# Self-attention QKV fusion
(".attn1.to_qkv", ".attn1.to_q", "q"),
(".attn1.to_qkv", ".attn1.to_k", "k"),
(".attn1.to_qkv", ".attn1.to_v", "v"),
]
self.stacked_params_mapping = stacked_params_mapping
params_dict = dict(self.named_parameters())
loaded_params: set[str] = set()
for name, loaded_weight in weights:
# Handle GEGLU: diffusers stores as ff.net.0.proj.weight/bias
# MergedColumnParallelLinear needs the two halves loaded separately
if ".ff.net.0.proj." in name:
name = name.replace(".ff.net.0.proj.", ".ff.geglu.proj.")
if name in params_dict:
param = params_dict[name]
weight_loader = param.weight_loader
half = loaded_weight.shape[0] // 2
weight_loader(param, loaded_weight[:half], 0)
weight_loader(param, loaded_weight[half:], 1)
loaded_params.add(name)
continue
elif ".ff.net.2." in name:
name = name.replace(".ff.net.2.", ".ff.out_proj.")
handled = False
for param_name, weight_name, shard_id in stacked_params_mapping:
if weight_name not in name:
continue
name = name.replace(weight_name, param_name)
if name not in params_dict:
break
param = params_dict[name]
weight_loader = param.weight_loader
weight_loader(param, loaded_weight, shard_id)
loaded_params.add(name)
handled = True
break
if not handled:
if name in params_dict:
param = params_dict[name]
weight_loader = getattr(param, "weight_loader", default_weight_loader)
weight_loader(param, loaded_weight)
loaded_params.add(name)
return loaded_params
+6
View File
@@ -286,6 +286,11 @@ _DIFFUSION_MODELS = {
"pipeline_dreamzero",
"DreamZeroPipeline",
),
"StableDiffusionXLPipeline": (
"sdxl",
"pipeline_sdxl",
"StableDiffusionXLPipeline",
),
}
@@ -510,6 +515,7 @@ _DIFFUSION_POST_PROCESS_FUNCS = {
"SenseNovaU1Pipeline": "get_sensenova_u1_post_process_func",
"Cosmos3OmniDiffusersPipeline": "get_cosmos3_post_process_func",
"HiDreamImagePipeline": "get_hidream_image_post_process_func",
"StableDiffusionXLPipeline": "get_sdxl_image_post_process_func",
}
_DIFFUSION_ACTION_POST_PROCESS_FUNCS = {