[New Model]: Add sensenova u1 support (#3319)

Signed-off-by: princepride <wangzhipeng628@gmail.com>
Signed-off-by: 汪志鹏 <wangzhipeng628@gmail.com>
This commit is contained in:
汪志鹏 authored and GitHub committed 2026-05-11 22:19:00 +08:00
1 parent 2cdffcea6b
commit c9a8556c24
20 files changed
+3838 -19

No files matched your search

+1
View File
@@ -64,6 +64,7 @@ th {
| `Flux2Pipeline` | FLUX.2-dev | `black-forest-labs/FLUX.2-dev` | ✅︎ | ✅︎ | | |
| `FishSpeechSlowARForConditionalGeneration` | Fish Speech S2 Pro | `fishaudio/s2-pro` | ✅︎ | ✅︎ | | |
| `DreamIDOmniPipeline` | DreamID-Omni | `XuGuo699/DreamID-Omni` | ✅︎ | ✅︎ | | |
| `SenseNovaU1Pipeline` | SenseNova-U1 (DiT-only) | `SenseNova/SenseNova-U1-8B-MoT` | ✅︎ | | | |
| `HunyuanVideo15Pipeline` | HunyuanVideo-1.5-T2V | `hunyuanvideo-community/HunyuanVideo-1.5-Diffusers-480p_t2v`, `hunyuanvideo-community/HunyuanVideo-1.5-Diffusers-720p_t2v` | ✅︎ | ✅︎ | | |
| `HunyuanVideo15ImageToVideoPipeline` | HunyuanVideo-1.5-I2V | `hunyuanvideo-community/HunyuanVideo-1.5-Diffusers-480p_i2v`, `hunyuanvideo-community/HunyuanVideo-1.5-Diffusers-720p_i2v` | ✅︎ | ✅︎ | | |
| `VoxtralTTSForConditionalGeneration` | Voxtral TTS | `mistralai/Voxtral-4B-TTS-2603` | ✅︎ | ✅︎ | | |
+1
View File
@@ -127,6 +127,7 @@ The following tables show which models support each feature:
| **Qwen-Image-Edit** | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ (decode) | ❌ | ❌ |
| **Qwen-Image-Edit-2509** | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ (decode) | ✅ | ❌ | ❌ |
| **Qwen-Image-Layered** | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ (decode) | ❌ | ❌ |
| **SenseNova-U1** | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ |
| **Stable-Diffusion3.5** | ❌ | ✅ | ❌ | ✅ | ✅ | ❌ | ✅ | ✅ (decode) | ❌ | ❌ |
| **Z-Image** | ✅ | ✅ | ✅ | ❓ | ✅ (TP=2 only) | ✅ | ❌ | ✅ (decode) | ✅ | ❌ |
| **ERNIE-Image** | ❌ | ✅ | ✅ | ❓ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ |
@@ -0,0 +1,208 @@
# SenseNova-U1-8B-MoT
## Architecture
SenseNova-U1 is a unified Qwen3-based LLM with Mixture-of-Tokenizers (MoT) attention. Unlike two-stage pipelines (e.g., BAGEL), it handles text encoding, optional reasoning ("think mode"), and flow-matching-based image denoising entirely within a single diffusion stage.
| Feature | Description |
| :------ | :---------- |
| **Model type** | Unified LLM (single-stage diffusion pipeline) |
| **Base LLM** | Qwen3 with MoT attention and 3D RoPE |
| **Image generation** | Flow-matching Euler sampler, no separate VAE |
| **Think mode** | Optional chain-of-thought reasoning before image generation |
| **Parallelism** | Tensor Parallelism (TP) with fused QKV and fused gate/up projections |
| **Modalities** | text2img, img2img, img2text (understanding), text2text |
## Quick Start
```bash
cd examples/offline_inference/sensenova_u1
# Text-to-image
python end2end.py --prompt "A cute cat" --think
# Image-to-image editing
python end2end.py --prompt "Turn this into an oil painting" \
--image input.png --think
# Image understanding (img2text)
python end2end.py --modality img2text \
--prompt "Describe this image in detail" \
--image photo.jpg
# Text-to-text (pure chat)
python end2end.py --modality text2text \
--prompt "What is the capital of France?"
# Custom resolution (image generation)
python end2end.py --prompt "A futuristic cityscape at sunset" \
--width 2048 --height 1024 --think
```
> **Note**: Default configuration works on a single **NVIDIA A100 (80GB)** or **H100** GPU.
## Text-to-Image (t2i)
Standard text-to-image with optional think mode:
```bash
python end2end.py \
--prompt "Close portrait of an elderly woman by a farmhouse window, textured skin, gentle smile, warm natural light" \
--width 1536 --height 2720 \
--think --print-think
```
## Image-to-Image Editing (img2img)
Pass one or more `--image` paths to trigger img2img mode. The model uses the
input image(s) as visual context for editing:
```bash
# Single input image
python end2end.py \
--prompt "Add a sunset sky in the background" \
--image photo.jpg \
--width 2048 --height 2048 \
--think
# Multiple reference images
python end2end.py \
--prompt "Combine the style of Image-1 with the content of Image-2" \
--image style_ref.png content_ref.png \
--width 2048 --height 2048 \
--think
```
### Dual CFG for img2img
img2img supports dual classifier-free guidance (text CFG + image CFG):
```bash
python end2end.py \
--prompt "Make the person smile" \
--image portrait.jpg \
--cfg-scale 4.0 \
--img-cfg-scale 2.0 \
--think
```
| `img_cfg_scale` | Behavior |
| :-------------- | :------- |
| `1.0` (default) | Text-only CFG: guidance = image-condition → full-condition |
| `== cfg_scale` | Standard CFG: guidance = unconditional → full-condition |
| Other value | Dual CFG: separate text and image guidance strengths |
## Full Parameter Reference
### General Parameters
| Parameter | Default | Description |
| :-------- | :------ | :---------- |
| `--modality` | `auto` | Task modality: `auto`, `text2img`, `img2img`, `img2text`, `text2text` |
| `--prompt` | "A cute cat..." | Text prompt / editing instruction / question |
| `--image` | None | Input image path(s) for img2img or img2text |
### Image Generation Parameters (text2img / img2img)
| Parameter | Default | Description |
| :-------- | :------ | :---------- |
| `--height` | 2048 | Height of generated image (pixels) |
| `--width` | 2048 | Width of generated image (pixels) |
| `--seed` | 42 | Random seed for reproducibility |
| `--num-steps` | 50 | Number of denoising steps |
| `--cfg-scale` | 4.0 | Text classifier-free guidance scale |
| `--img-cfg-scale` | 1.0 | Image CFG scale (img2img only, 1.0 = disabled) |
| `--cfg-norm` | "none" | CFG normalization: none, global, channel, cfg_zero_star (t2i only) |
| `--timestep-shift` | 3.0 | Timestep shift for flow-matching schedule |
| `--t-eps` | 0.02 | Epsilon for timestep schedule |
| `--think` | False | Enable think mode |
| `--print-think` | False | Print think text to stdout |
### Text Generation Parameters (text2text / img2text)
| Parameter | Default | Description |
| :-------- | :------ | :---------- |
| `--max-tokens` | 512 | Maximum number of tokens to generate |
| `--do-sample` | False | Use sampling instead of greedy decoding |
| `--temperature` | 0.7 | Sampling temperature (higher = more diverse) |
### Infrastructure Parameters
| Parameter | Default | Description |
| :-------- | :------ | :---------- |
| `--model` | `SenseNova/SenseNova-U1-8B-MoT` | HuggingFace model ID or local path |
| `--output` | `.` | Output directory for saved images |
| `--tensor-parallel-size` | 1 | Number of GPUs for tensor parallelism |
| `--enforce-eager` | False | Disable torch.compile |
| `--enable-cpu-offload` | False | Enable module-wise (sequential) CPU offload to reduce peak VRAM |
## Reducing GPU Memory Usage
For hardware with limited VRAM, enable **module-wise CPU offload** with
`--enable-cpu-offload`. The pipeline implements `SupportsModuleOffload`, so the
vision encoder (`vision_model`) and the Qwen3 LLM (`language_model`) are
swapped between CPU and GPU on demand:
- During text/vision encoding, the LLM is on CPU.
- During the diffusion loop, the vision encoder is on CPU.
- Lightweight FM modules stay resident on GPU.
This lowers peak VRAM at the cost of extra CPU<->GPU transfers (use pinned
memory) — useful for running SenseNova-U1 on consumer-grade GPUs.
```bash
# Text-to-image with CPU offload
python end2end.py \
--prompt "A cute cat sitting on a windowsill" \
--width 2048 --height 2048 \
--enable-cpu-offload --think
# Image-to-image editing with CPU offload
python end2end.py \
--prompt "Turn this into an oil painting" \
--image input.png \
--width 2048 --height 2048 \
--enable-cpu-offload --think
# Image understanding (img2text) with CPU offload
python end2end.py \
--modality img2text \
--prompt "Describe this image in detail" \
--image photo.jpg \
--enable-cpu-offload
```
> **Notes**
> - CPU offload is **single-GPU only** (incompatible with `--tensor-parallel-size > 1`).
> - First-step latency is higher because of the cold-start CPU<->GPU transfers.
> - For more details and other offloading strategies, see
> [CPU Offloading for Diffusion Models](../../../docs/user_guide/diffusion/cpu_offload_diffusion.md).
## Reproducing the E2E Test
The following command reproduces the pixel-validated CI test case:
```bash
python end2end.py \
--prompt "Close portrait of an elderly woman by a farmhouse window, textured skin, gentle smile, warm natural light, emotional documentary look. The portrait should feel polished and natural, with sharp eyes, realistic skin texture, accurate facial anatomy, and premium lighting that keeps the face as the main focus." \
--width 1536 --height 2720 \
--seed 42 --num-steps 50 \
--cfg-scale 4.0 --timestep-shift 3.0 --cfg-norm none \
--think --print-think \
--output outputs
```
The corresponding pytest:
```bash
pytest -s -v tests/e2e/offline_inference/test_sensenova_u1_text2img.py \
-m "advanced_model" --run-level "advanced_model"
```
The img2img counterpart lives at
`tests/e2e/offline_inference/test_sensenova_u1_img2img.py`.
## Online Serving
For OpenAI-compatible API serving, see
[`examples/online_serving/sensenova_u1/`](../../examples/online_serving/sensenova_u1/).
@@ -0,0 +1,281 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
"""
End-to-end offline inference for SenseNova-U1-8B-MoT.
Supports four modalities: text2img, img2img, img2text, text2text.
Text-to-image:
python end2end.py --prompt "A cute cat" --think
Image-to-image (editing):
python end2end.py --prompt "Turn this into an oil painting" \
--image input.png --think
Image understanding:
python end2end.py --modality img2text \
--prompt "Describe this image" --image photo.jpg
Text chat:
python end2end.py --modality text2text \
--prompt "What is the capital of France?"
See README.md for more examples.
"""
import argparse
import os
from PIL import Image
from vllm_omni.engine.arg_utils import nullify_stage_engine_defaults
from vllm_omni.entrypoints.omni import Omni
from vllm_omni.inputs.data import OmniDiffusionSamplingParams
def parse_args():
parser = argparse.ArgumentParser(
description="SenseNova-U1 text-to-image / image-to-image / text / understanding via vLLM-Omni.",
)
parser.add_argument(
"--model",
default="SenseNova/SenseNova-U1-8B-MoT",
help="HuggingFace model ID or local path.",
)
parser.add_argument(
"--modality",
default="auto",
choices=["auto", "text2img", "img2img", "img2text", "text2text"],
help="Task modality. 'auto' infers from --image (img2img/img2text if images, else t2i/t2t).",
)
parser.add_argument(
"--prompt",
default="A cute cat sitting on a windowsill, soft natural light",
help="Text prompt for generation or editing instruction.",
)
parser.add_argument(
"--image",
nargs="+",
metavar="PATH",
default=None,
help="Input image path(s) for img2img or img2text.",
)
parser.add_argument(
"--output",
type=str,
default=".",
help="Output directory to save generated images.",
)
# Image dimensions
parser.add_argument("--height", type=int, default=2048, help="Height of generated image.")
parser.add_argument("--width", type=int, default=2048, help="Width of generated image.")
# Generation parameters
parser.add_argument("--seed", type=int, default=42, help="Random seed for deterministic results.")
parser.add_argument("--num-steps", type=int, default=50, help="Number of denoising steps.")
parser.add_argument("--cfg-scale", type=float, default=4.0, help="Text classifier-free guidance scale.")
parser.add_argument(
"--img-cfg-scale",
type=float,
default=1.0,
help="Image CFG scale for img2img (1.0 = disabled). "
"When both --cfg-scale and --img-cfg-scale differ, dual CFG is used.",
)
parser.add_argument(
"--cfg-norm",
type=str,
default="none",
choices=["none", "global", "channel", "cfg_zero_star"],
help="CFG normalization mode. cfg_zero_star is only for t2i.",
)
parser.add_argument(
"--timestep-shift",
type=float,
default=3.0,
help="Timestep shift for flow-matching schedule.",
)
parser.add_argument(
"--t-eps",
type=float,
default=0.02,
help="Epsilon for flow-matching timestep schedule.",
)
# Text generation parameters (for text2text / img2text)
parser.add_argument(
"--max-tokens",
type=int,
default=2048,
help="Max tokens for text generation (text2text / img2text).",
)
parser.add_argument(
"--do-sample",
action="store_true",
help="Use sampling instead of greedy decoding for text generation.",
)
parser.add_argument(
"--temperature",
type=float,
default=0.7,
help="Sampling temperature for text generation.",
)
# Think mode
parser.add_argument(
"--think",
action="store_true",
help="Enable think mode: the model reasons about the prompt before generating the image.",
)
parser.add_argument(
"--print-think",
action="store_true",
help="Print the generated think text to stdout.",
)
# Advanced
parser.add_argument(
"--tensor-parallel-size",
type=int,
default=1,
help="Number of GPUs for tensor parallelism.",
)
parser.add_argument(
"--enforce-eager",
action="store_true",
help="Disable torch.compile and force eager execution.",
)
parser.add_argument(
"--enable-cpu-offload",
action="store_true",
help=(
"Enable model-level (sequential) CPU offloading. "
"Mutually exclusive between vision_model and language_model: "
"only one is on GPU at a time, reducing peak VRAM at the cost "
"of extra CPU<->GPU transfers."
),
)
nullify_stage_engine_defaults(parser)
return parser.parse_args()
def _resolve_modality(args):
"""Determine the effective modality from CLI args."""
if args.modality != "auto":
return args.modality
has_images = args.image is not None
# Default: image-producing tasks for backward compatibility
return "img2img" if has_images else "text2img"
def main():
args = parse_args()
os.makedirs(args.output, exist_ok=True)
modality = _resolve_modality(args)
is_text_output = modality in ("text2text", "img2text")
has_images = modality in ("img2img", "img2text")
omni = Omni(
model=args.model,
tensor_parallel_size=args.tensor_parallel_size,
enforce_eager=args.enforce_eager,
enable_cpu_offload=args.enable_cpu_offload,
)
extra_args = {
"cfg_scale": args.cfg_scale,
"cfg_norm": args.cfg_norm,
"timestep_shift": args.timestep_shift,
"cfg_interval": (0.0, 1.0),
"batch_size": 1,
"think": args.think,
"t_eps": args.t_eps,
}
if modality == "img2img":
extra_args["img_cfg_scale"] = args.img_cfg_scale
if is_text_output:
extra_args["max_tokens"] = args.max_tokens
extra_args["do_sample"] = args.do_sample
extra_args["temperature"] = args.temperature
sampling_params = OmniDiffusionSamplingParams(
height=args.height,
width=args.width,
seed=args.seed,
num_inference_steps=args.num_steps,
extra_args=extra_args,
)
print(f"\n{'=' * 60}")
print(f"SenseNova-U1 Configuration ({modality}):")
print(f" Model : {args.model}")
if not is_text_output:
print(f" Image size : {args.width}x{args.height}")
print(f" Steps : {args.num_steps}")
print(f" CFG scale : {args.cfg_scale}")
if modality == "img2img":
print(f" Img CFG scale : {args.img_cfg_scale}")
if has_images:
print(f" Input images : {args.image}")
if is_text_output:
print(f" Max tokens : {args.max_tokens}")
print(f" Temperature : {args.temperature}")
print(f" Seed : {args.seed}")
print(f" Think mode : {args.think}")
print(f" TP size : {args.tensor_parallel_size}")
print(f"{'=' * 60}\n")
# Build prompt dict
if has_images:
if not args.image:
raise ValueError(f"{modality} requires --image.")
input_images = [Image.open(p).convert("RGB") for p in args.image]
if is_text_output:
prompt_dict = {
"prompt": args.prompt,
"multi_modal_data": {"image": input_images},
"modalities": ["text"],
}
else:
prompt_dict = {
"prompt": args.prompt,
"multi_modal_data": {"image": input_images},
"modalities": ["img2img"],
}
elif is_text_output:
prompt_dict = {"prompt": args.prompt, "modalities": ["text"]}
else:
prompt_dict = {"prompt": args.prompt, "modalities": ["image"]}
outputs = list(
omni.generate(
prompts=prompt_dict,
sampling_params_list=sampling_params,
)
)
for req_output in outputs:
custom = getattr(req_output, "_custom_output", {}) or {}
if args.print_think and custom.get("think_text"):
print(f"[Think]\n{custom['think_text']}\n")
if is_text_output:
text = custom.get("text_output", "")
if not text:
text = getattr(req_output, "text", "") or ""
print(f"[Response]\n{text}")
else:
images = getattr(req_output, "images", None) or []
if not images:
print("[Warning] No images generated.")
continue
for j, img in enumerate(images):
save_path = os.path.join(args.output, f"sensenova_u1_output_{j}.png")
img.save(save_path)
print(f"[Output] Saved {img.size[0]}x{img.size[1]} image to {save_path}")
if __name__ == "__main__":
main()
@@ -0,0 +1,163 @@
# SenseNova-U1 Online Serving
## Launch the Server
```bash
vllm serve SenseNova/SenseNova-U1-8B-MoT --omni --port 8091
```
Or use the convenience script:
```bash
cd examples/online_serving/sensenova_u1
bash run_server.sh
```
### Tensor Parallelism (TP)
```bash
vllm serve SenseNova/SenseNova-U1-8B-MoT --omni --port 8091 \
--tensor-parallel-size 2
```
## Send Requests
```bash
cd examples/online_serving/sensenova_u1
```
### Text to Image (text2img)
**Python client:**
```bash
python openai_chat_client.py \
--prompt "A beautiful sunset over mountains" \
--modality text2img \
--height 2048 --width 2048 --num-steps 50
```
**curl:**
```bash
curl http://localhost:8091/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"messages": [{"role": "user", "content": [{"type": "text", "text": "A beautiful sunset over mountains"}]}],
"modalities": ["image"],
"height": 2048,
"width": 2048,
"num_inference_steps": 50,
"seed": 42
}'
```
### Image to Image (img2img)
**Python client:**
```bash
python openai_chat_client.py \
--prompt "Turn this into an oil painting" \
--modality img2img \
--image-url /path/to/input.jpg \
--height 2048 --width 2048
```
**curl:**
```bash
IMAGE_BASE64=$(base64 -w 0 input.jpg)
cat <<EOF > payload.json
{
"messages": [{
"role": "user",
"content": [
{"type": "text", "text": "Turn this into an oil painting"},
{"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,${IMAGE_BASE64}"}}
]
}],
"modalities": ["image"],
"height": 2048,
"width": 2048,
"num_inference_steps": 50,
"seed": 42
}
EOF
curl http://localhost:8091/v1/chat/completions \
-H "Content-Type: application/json" \
-d @payload.json
```
### Image to Text (img2text)
**Python client:**
```bash
python openai_chat_client.py \
--prompt "Describe this image in detail" \
--modality img2text \
--image-url /path/to/image.jpg
```
**curl:**
```bash
IMAGE_BASE64=$(base64 -w 0 image.jpg)
cat <<EOF > payload.json
{
"messages": [{
"role": "user",
"content": [
{"type": "text", "text": "Describe this image in detail"},
{"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,${IMAGE_BASE64}"}}
]
}],
"modalities": ["text"]
}
EOF
curl http://localhost:8091/v1/chat/completions \
-H "Content-Type: application/json" \
-d @payload.json
```
### Text to Text (text2text)
**Python client:**
```bash
python openai_chat_client.py \
--prompt "What is the capital of France?" \
--modality text2text
```
**curl:**
```bash
curl http://localhost:8091/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"messages": [{"role": "user", "content": [{"type": "text", "text": "What is the capital of France?"}]}],
"modalities": ["text"]
}'
```
## Python Client Arguments
| Argument | Default | Description |
| :------- | :------ | :---------- |
| `--prompt` / `-p` | `A cute cat` | Text prompt |
| `--output` / `-o` | `sensenova_u1_output.png` | Output file path |
| `--server` / `-s` | `http://localhost:8091` | Server URL |
| `--image-url` / `-i` | `None` | Input image URL or local path (img2img/img2text) |
| `--modality` / `-m` | `text2img` | `text2img`, `img2img`, `img2text`, `text2text` |
| `--height` | `2048` | Image height (image generation only) |
| `--width` | `2048` | Image width (image generation only) |
| `--num-steps` | `50` | Number of inference steps (image generation only) |
| `--seed` | `42` | Random seed |
| `--cfg-scale` | `4.0` | CFG scale (image generation only) |
| `--think` | `False` | Enable think mode |
@@ -0,0 +1,188 @@
#!/usr/bin/env python3
"""
SenseNova-U1 OpenAI-compatible chat client for all four task modalities.
Usage:
# Text-to-image
python openai_chat_client.py --prompt "A cute cat" --modality text2img
# Image-to-image editing
python openai_chat_client.py --prompt "Turn this into an oil painting" \
--modality img2img --image-url /path/to/image.jpg
# Image understanding (img2text)
python openai_chat_client.py --prompt "Describe this image in detail" \
--modality img2text --image-url /path/to/image.jpg
# Text-to-text (chat)
python openai_chat_client.py --prompt "What is the capital of France?" \
--modality text2text
"""
import argparse
import base64
from pathlib import Path
import requests
def _encode_image(image_url: str) -> str:
"""Encode a local file or URL to a base64 data URI."""
if Path(image_url).exists():
with open(image_url, "rb") as f:
b64_data = base64.b64encode(f.read()).decode("utf-8")
return f"data:image/jpeg;base64,{b64_data}"
return image_url
def generate(
prompt: str,
server_url: str = "http://localhost:8091",
image_url: str | None = None,
modality: str = "text2img",
**kwargs: object,
) -> bytes | str | None:
"""Send a request to the SenseNova-U1 server.
All keyword arguments (height, width, seed, cfg_scale, think, …) are
forwarded as top-level fields in the request payload. The serving layer
maps standard keys (height, width, seed, num_inference_steps) to sampling
params and forwards model-specific keys (think, cfg_norm, img_cfg_scale,
…) to ``extra_args`` based on the pipeline's ``EXTRA_BODY_PARAMS``.
Returns:
Image bytes (for image outputs) or text string (for text outputs).
"""
content = [{"type": "text", "text": prompt}]
if image_url:
content.append({"type": "image_url", "image_url": {"url": _encode_image(image_url)}})
messages = [{"role": "user", "content": content}]
payload: dict = {"messages": messages}
if modality in ("text2img", "img2img"):
payload["modalities"] = ["image"]
else:
payload["modalities"] = ["text"]
for key, val in kwargs.items():
if val is not None and val is not False:
payload[key] = val
try:
print(f"Sending {modality} request to {server_url}...")
response = requests.post(
f"{server_url}/v1/chat/completions",
headers={"Content-Type": "application/json"},
json=payload,
timeout=600,
)
response.raise_for_status()
data = response.json()
# Print think text if present in response metrics
metrics = data.get("metrics") or {}
think_text = metrics.get("think_text")
if think_text:
print(f"\n[Think]\n{think_text}\n")
choices = data.get("choices", [])
for choice in choices:
choice_content = choice.get("message", {}).get("content")
if isinstance(choice_content, list) and len(choice_content) > 0:
first_item = choice_content[0]
if isinstance(first_item, dict) and "image_url" in first_item:
img_url_str = first_item["image_url"].get("url", "")
if img_url_str.startswith("data:image"):
_, b64_data = img_url_str.split(",", 1)
return base64.b64decode(b64_data)
for choice in choices:
choice_content = choice.get("message", {}).get("content")
if isinstance(choice_content, str) and choice_content:
return choice_content
print(f"Unexpected response format: {choices}")
return None
except Exception as e:
print(f"Error: {e}")
return None
def main():
parser = argparse.ArgumentParser(description="SenseNova-U1 multimodal chat client")
parser.add_argument("--prompt", "-p", default="A cute cat", help="Text prompt")
parser.add_argument("--output", "-o", default="sensenova_u1_output.png", help="Output file (for image results)")
parser.add_argument("--server", "-s", default="http://localhost:8091", help="Server URL")
parser.add_argument("--image-url", "-i", type=str, help="Input image URL or local path")
parser.add_argument(
"--modality",
"-m",
default="text2img",
choices=["text2img", "img2img", "img2text", "text2text"],
help="Task modality",
)
# Standard generation parameters
parser.add_argument("--height", type=int, default=2048, help="Image height")
parser.add_argument("--width", type=int, default=2048, help="Image width")
parser.add_argument("--num-steps", type=int, default=50, help="Inference steps")
parser.add_argument("--seed", type=int, default=42, help="Random seed")
# Model-specific parameters (forwarded via EXTRA_BODY_PARAMS)
parser.add_argument("--cfg-scale", type=float, default=4.0, help="CFG scale")
parser.add_argument("--img-cfg-scale", type=float, help="Image CFG scale (img2img)")
parser.add_argument("--cfg-norm", type=str, help="CFG normalization mode")
parser.add_argument("--timestep-shift", type=float, help="Timestep shift")
parser.add_argument("--think", action="store_true", help="Enable think mode")
args = parser.parse_args()
is_image = args.modality in ("text2img", "img2img")
print(f"Mode: {args.modality}")
if args.image_url:
print(f"Input Image: {args.image_url}")
extra: dict[str, object] = {
"seed": args.seed,
"think": args.think or None,
}
if is_image:
extra.update(
height=args.height,
width=args.width,
num_inference_steps=args.num_steps,
cfg_scale=args.cfg_scale,
)
if args.img_cfg_scale is not None:
extra["img_cfg_scale"] = args.img_cfg_scale
if args.cfg_norm is not None:
extra["cfg_norm"] = args.cfg_norm
if args.timestep_shift is not None:
extra["timestep_shift"] = args.timestep_shift
result = generate(
prompt=args.prompt,
server_url=args.server,
image_url=args.image_url,
modality=args.modality,
**extra,
)
if result:
if isinstance(result, bytes):
output_path = Path(args.output)
output_path.write_bytes(result)
print(f"Image saved to: {output_path}")
print(f"Size: {len(result) / 1024:.1f} KB")
elif isinstance(result, str):
print(f"[Response]\n{result}")
else:
print("Failed to generate response")
exit(1)
if __name__ == "__main__":
main()
@@ -0,0 +1,12 @@
#!/bin/bash
# SenseNova-U1 online serving startup script
MODEL="${MODEL:-SenseNova/SenseNova-U1-8B-MoT}"
PORT="${PORT:-8091}"
echo "Starting SenseNova-U1 server..."
echo "Model: $MODEL"
echo "Port: $PORT"
vllm serve "$MODEL" --omni \
--port "$PORT"
+187
View File
@@ -0,0 +1,187 @@
# SenseNova-U1 for unified image generation and understanding
## Summary
- Vendor: SenseNova
- Model: `SenseNova/SenseNova-U1-8B-MoT`
- Task: text2img, img2img, img2text (visual understanding), text2text (chat)
- Mode: Offline inference, Online serving (OpenAI-compatible API)
- Maintainer: Community
## When to use this recipe
Use this recipe to run SenseNova-U1-8B-MoT via vLLM-Omni. SenseNova-U1 is a
unified Qwen3-based LLM with Mixture-of-Tokenizers (MoT) attention that handles
text encoding, optional chain-of-thought reasoning, flow-matching image
denoising, and visual understanding in a single pipeline — no separate text
encoder or VAE needed. It supports four task modalities: text-to-image,
image-to-image editing (with dual CFG), image-to-text understanding, and
text-to-text chat.
## References
- Offline example:
[`examples/offline_inference/sensenova_u1/end2end.py`](../../examples/offline_inference/sensenova_u1/end2end.py)
- Online serving:
[`examples/online_serving/sensenova_u1/`](../../examples/online_serving/sensenova_u1/)
- E2E tests:
[`tests/e2e/offline_inference/test_sensenova_u1_text2img.py`](../../tests/e2e/offline_inference/test_sensenova_u1_text2img.py),
[`tests/e2e/offline_inference/test_sensenova_u1_img2img.py`](../../tests/e2e/offline_inference/test_sensenova_u1_img2img.py)
- HuggingFace model page:
[SenseNova/SenseNova-U1-8B-MoT](https://huggingface.co/SenseNova/SenseNova-U1-8B-MoT)
## Hardware Support
## GPU
### 1x H200 (144GB)
#### Environment
- OS: Linux
- Python: 3.12
- Driver / runtime: NVIDIA 590.48.01, CUDA 13.1
- vLLM-Omni version: 0.18.1.dev
#### Command
```bash
python examples/offline_inference/sensenova_u1/end2end.py \
--prompt "Close portrait of an elderly woman by a farmhouse window, textured skin, gentle smile, warm natural light, emotional documentary look. The portrait should feel polished and natural, with sharp eyes, realistic skin texture, accurate facial anatomy, and premium lighting that keeps the face as the main focus." \
--width 1536 --height 2720 \
--seed 42 --num-steps 50 \
--cfg-scale 4.0 --timestep-shift 3.0 --cfg-norm none \
--think --print-think \
--output outputs
```
#### Verification
```bash
pytest -s -v tests/e2e/offline_inference/test_sensenova_u1_text2img.py \
-m "advanced_model" --run-level "advanced_model"
```
#### Notes
- E2E latency: **32.1s** (1536×2720, 50 steps, think mode, CFG scale 4.0)
- Peak VRAM: **35.9 GB** reserved, 35.1 GB allocated
- Model loading: 32.8 GiB, 8.7s
- No deploy YAML needed — the engine auto-generates a single-stage diffusion config.
- Think mode (`--think`) is recommended for higher image quality.
#### Image-to-Image Editing (img2img)
```bash
python examples/offline_inference/sensenova_u1/end2end.py \
--prompt "Turn this into an oil painting" \
--image input.png \
--width 2048 --height 2048 \
--seed 42 --num-steps 50 \
--cfg-scale 4.0 --img-cfg-scale 1.0 --cfg-norm none \
--think --print-think \
--output outputs
```
- img2img uses dual CFG: `--cfg-scale` controls text guidance, `--img-cfg-scale`
controls image guidance (1.0 = image CFG disabled).
- Pass multiple `--image` paths for multi-reference editing.
#### Image Understanding (img2text)
```bash
python examples/offline_inference/sensenova_u1/end2end.py \
--modality img2text \
--prompt "Describe this image in detail" \
--image photo.jpg \
--max-tokens 512
```
#### Text-to-Text Chat (text2text)
```bash
python examples/offline_inference/sensenova_u1/end2end.py \
--modality text2text \
--prompt "Explain the theory of relativity in simple terms" \
--max-tokens 256
```
- For img2text and text2text, image generation parameters (height, width,
num-steps, cfg-scale) are ignored.
- Use `--do-sample --temperature 0.7` for more diverse text responses.
### 2x H200 (144GB) — TP=2
#### Environment
- OS: Linux
- Python: 3.12
- Driver / runtime: NVIDIA 590.48.01, CUDA 13.1
- vLLM-Omni version: 0.18.1.dev
#### Command
```bash
python examples/offline_inference/sensenova_u1/end2end.py \
--prompt "Close portrait of an elderly woman by a farmhouse window, textured skin, gentle smile, warm natural light, emotional documentary look. The portrait should feel polished and natural, with sharp eyes, realistic skin texture, accurate facial anatomy, and premium lighting that keeps the face as the main focus." \
--width 1536 --height 2720 \
--seed 42 --num-steps 50 \
--cfg-scale 4.0 --timestep-shift 3.0 --cfg-norm none \
--think --print-think \
--tensor-parallel-size 2 \
--output outputs
```
#### Verification
Verify the output image is generated at `outputs/sensenova_u1_output_0.png`
with the expected 1536×2720 resolution.
#### Notes
- E2E latency: **28.3s** (1536×2720, 50 steps, think mode, CFG scale 4.0)
- Peak VRAM (per GPU): **18.2 GB** reserved, 17.9 GB allocated
- Model loading: 16.5 GiB per GPU, 7.0s
- TP=2 provides ~12% speedup over TP=1; limited by serial CFG dual-forward
and communication overhead.
- The LLM transformer uses `QKVParallelLinear` and `MergedColumnParallelLinear`
for fused QKV and gate/up projections with TP support.
## Online Serving
SenseNova-U1 supports all four modalities via the OpenAI-compatible
`/v1/chat/completions` API.
### Launch
```bash
vllm serve SenseNova/SenseNova-U1-8B-MoT --omni --port 8091
```
### Send Requests
```bash
cd examples/online_serving/sensenova_u1
# Text-to-image
python openai_chat_client.py \
--prompt "A beautiful sunset" --modality text2img
# Image-to-image editing
python openai_chat_client.py \
--prompt "Turn this into an oil painting" \
--modality img2img --image-url input.jpg
# Image understanding
python openai_chat_client.py \
--prompt "Describe this image" \
--modality img2text --image-url photo.jpg
# Text chat
python openai_chat_client.py \
--prompt "What is the capital of France?" \
--modality text2text
```
For full API documentation and curl examples, see
[`examples/online_serving/sensenova_u1/README.md`](../../examples/online_serving/sensenova_u1/README.md).
@@ -0,0 +1,169 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
"""
End-to-end test for SenseNova-U1 img2img generation.
This test validates that the SenseNova-U1 model generates edited images from
an input image and text prompt that match expected reference pixel values
within a ±10 tolerance.
Equivalent to running:
python examples/offline_inference/sensenova_u1/end2end.py \
--prompt "Turn this into an oil painting" \
--image 2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg \
--width 2560 --height 1669 \
--seed 42 --num-steps 50 \
--cfg-scale 4.0 --img-cfg-scale 1.0 --cfg-norm none \
--think --print-think
"""
import os
from typing import Any
os.environ["VLLM_WORKER_MULTIPROC_METHOD"] = "spawn"
import pytest
from PIL import Image
from vllm.assets.image import ImageAsset
from tests.helpers.mark import hardware_test
from tests.helpers.runtime import OmniRunner
from vllm_omni.entrypoints.omni import Omni
from vllm_omni.inputs.data import OmniDiffusionSamplingParams
# Reference pixel data extracted from the known-good output image generated by:
# python examples/offline_inference/sensenova_u1/end2end.py \
# --prompt "Turn this into an oil painting" \
# --image 2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg \
# --width 2560 --height 1669 --seed 42 --num-steps 50 \
# --cfg-scale 4.0 --img-cfg-scale 1.0 --cfg-norm none --think
REFERENCE_PIXELS = [
{"position": (100, 100), "rgb": (95, 131, 174)},
{"position": (400, 50), "rgb": (142, 153, 174)},
{"position": (700, 100), "rgb": (127, 149, 174)},
{"position": (150, 400), "rgb": (99, 141, 183)},
{"position": (512, 336), "rgb": (184, 177, 165)},
{"position": (700, 400), "rgb": (106, 139, 168)},
{"position": (100, 600), "rgb": (170, 172, 155)},
{"position": (400, 600), "rgb": (159, 171, 169)},
{"position": (700, 600), "rgb": (139, 172, 187)},
{"position": (256, 256), "rgb": (136, 158, 180)},
]
PIXEL_TOLERANCE = 10
DEFAULT_PROMPT = "Turn this into an oil painting"
EXPECTED_OUTPUT_SIZE = (2560, 1664)
def _load_input_image() -> Image.Image:
"""Load the test input image via vllm's ImageAsset."""
return ImageAsset("2560px-Gfp-wisconsin-madison-the-nature-boardwalk").pil_image.convert("RGB")
def _build_sampling_params() -> OmniDiffusionSamplingParams:
"""Build sampling parameters for SenseNova-U1 img2img generation."""
return OmniDiffusionSamplingParams(
height=1669,
width=2560,
seed=42,
num_inference_steps=50,
extra_args={
"cfg_scale": 4.0,
"img_cfg_scale": 1.0,
"cfg_norm": "none",
"timestep_shift": 3.0,
"cfg_interval": (0.0, 1.0),
"batch_size": 1,
"think": True,
"t_eps": 0.02,
},
)
def _extract_generated_image(omni_outputs: list) -> Image.Image | None:
"""Extract the generated image from Omni outputs (single-stage DiT)."""
for req_output in omni_outputs:
if images := getattr(req_output, "images", None):
return images[0]
return None
def _validate_pixels(
image: Image.Image,
reference_pixels: list[dict[str, Any]] = REFERENCE_PIXELS,
tolerance: int = PIXEL_TOLERANCE,
) -> None:
"""Validate that image pixels match expected reference values.
Args:
image: The PIL Image to validate.
reference_pixels: List of dicts with 'position' (x, y) and 'rgb' (R, G, B).
tolerance: Maximum allowed difference per color channel.
Raises:
AssertionError: If any pixel differs beyond tolerance.
"""
for ref in reference_pixels:
x, y = ref["position"]
expected = ref["rgb"]
actual = image.getpixel((x, y))[:3]
assert all(abs(a - e) <= tolerance for a, e in zip(actual, expected)), (
f"Pixel mismatch at ({x}, {y}): expected {expected}, got {actual}"
)
def _generate_sensenova_u1_img2img(
omni: Omni,
input_image: Image.Image,
prompt: str = DEFAULT_PROMPT,
) -> Image.Image:
"""Generate an edited image using SenseNova-U1 model with img2img pipeline.
Args:
omni: The Omni instance to use for generation.
input_image: The input PIL Image for img2img.
prompt: The text prompt for image editing.
Returns:
The generated PIL Image.
Raises:
AssertionError: If no image is generated or size is incorrect.
"""
sampling_params = _build_sampling_params()
omni_outputs = list(
omni.generate(
prompts={
"prompt": prompt,
"multi_modal_data": {"image": input_image},
"modalities": ["img2img"],
},
sampling_params_list=sampling_params,
)
)
generated_image = _extract_generated_image(omni_outputs)
assert generated_image is not None, "No images generated"
assert generated_image.size == EXPECTED_OUTPUT_SIZE, f"Expected {EXPECTED_OUTPUT_SIZE}, got {generated_image.size}"
return generated_image
@pytest.mark.core_model
@pytest.mark.advanced_model
@pytest.mark.diffusion
@hardware_test(res={"cuda": "H100"})
def test_sensenova_u1_img2img(run_level):
"""Test SenseNova-U1 img2img (single-stage diffusion, no deploy YAML)."""
input_image = _load_input_image()
with OmniRunner(
"SenseNova/SenseNova-U1-8B-MoT",
stage_configs_path=None,
) as runner:
generated_image = _generate_sensenova_u1_img2img(runner.omni, input_image)
if run_level == "advanced_model":
_validate_pixels(generated_image)
@@ -0,0 +1,158 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
"""
End-to-end test for SenseNova-U1 text2img generation.
This test validates that the SenseNova-U1 model generates images that match
expected reference pixel values within a ±10 tolerance.
Equivalent to running:
python SenseNova-U1/examples/t2i/inference.py \
--model_path SenseNova/SenseNova-U1-8B-MoT \
--prompt "Close portrait of an elderly woman ..." \
--width 1536 --height 2720 \
--cfg_scale 4.0 --cfg_norm none --timestep_shift 3.0 \
--num_steps 50 --seed 42 --think
"""
import os
from typing import Any
os.environ["VLLM_WORKER_MULTIPROC_METHOD"] = "spawn"
import pytest
from PIL import Image
from tests.helpers.mark import hardware_test
from tests.helpers.runtime import OmniRunner
from vllm_omni.entrypoints.omni import Omni
from vllm_omni.inputs.data import OmniDiffusionSamplingParams
# Reference pixel data extracted from the known-good output image generated by:
# python examples/offline_inference/sensenova_u1/end2end.py \
# --prompt "Close portrait of an elderly woman ..." \
# --width 1536 --height 2720 --seed 42 --num-steps 50 \
# --cfg-scale 4.0 --timestep-shift 3.0 --cfg-norm none --think
REFERENCE_PIXELS = [
{"position": (100, 100), "rgb": (247, 249, 250)},
{"position": (768, 200), "rgb": (176, 135, 97)},
{"position": (400, 600), "rgb": (193, 188, 180)},
{"position": (1200, 2000), "rgb": (117, 98, 84)},
{"position": (750, 500), "rgb": (186, 135, 84)},
{"position": (300, 1360), "rgb": (202, 157, 107)},
{"position": (1000, 1800), "rgb": (63, 32, 15)},
{"position": (500, 2400), "rgb": (135, 116, 104)},
{"position": (768, 1360), "rgb": (70, 34, 16)},
{"position": (200, 900), "rgb": (208, 201, 191)},
]
PIXEL_TOLERANCE = 10
DEFAULT_PROMPT = (
"Close portrait of an elderly woman by a farmhouse window, textured skin, "
"gentle smile, warm natural light, emotional documentary look. The portrait "
"should feel polished and natural, with sharp eyes, realistic skin texture, "
"accurate facial anatomy, and premium lighting that keeps the face as the "
"main focus."
)
EXPECTED_OUTPUT_SIZE = (1536, 2720)
def _build_sampling_params() -> OmniDiffusionSamplingParams:
"""Build sampling parameters for SenseNova-U1 text2img generation."""
return OmniDiffusionSamplingParams(
height=EXPECTED_OUTPUT_SIZE[1],
width=EXPECTED_OUTPUT_SIZE[0],
seed=42,
num_inference_steps=50,
extra_args={
"cfg_scale": 4.0,
"cfg_norm": "none",
"timestep_shift": 3.0,
"cfg_interval": (0.0, 1.0),
"batch_size": 1,
"think": True,
"t_eps": 0.02,
},
)
def _extract_generated_image(omni_outputs: list) -> Image.Image | None:
"""Extract the generated image from Omni outputs (single-stage DiT)."""
for req_output in omni_outputs:
if images := getattr(req_output, "images", None):
return images[0]
return None
def _validate_pixels(
image: Image.Image,
reference_pixels: list[dict[str, Any]] = REFERENCE_PIXELS,
tolerance: int = PIXEL_TOLERANCE,
) -> None:
"""Validate that image pixels match expected reference values.
Args:
image: The PIL Image to validate.
reference_pixels: List of dicts with 'position' (x, y) and 'rgb' (R, G, B).
tolerance: Maximum allowed difference per color channel.
Raises:
AssertionError: If any pixel differs beyond tolerance.
"""
for ref in reference_pixels:
x, y = ref["position"]
expected = ref["rgb"]
actual = image.getpixel((x, y))[:3]
assert all(abs(a - e) <= tolerance for a, e in zip(actual, expected)), (
f"Pixel mismatch at ({x}, {y}): expected {expected}, got {actual}"
)
def _generate_sensenova_u1_image(
omni: Omni,
prompt: str = DEFAULT_PROMPT,
) -> Image.Image:
"""Generate an image using SenseNova-U1 model with configured parameters.
Args:
omni: The Omni instance to use for generation.
prompt: The text prompt for image generation.
Returns:
The generated PIL Image.
Raises:
AssertionError: If no image is generated or size is incorrect.
"""
sampling_params = _build_sampling_params()
omni_outputs = list(
omni.generate(
prompts={"prompt": prompt, "modalities": ["image"]},
sampling_params_list=sampling_params,
)
)
generated_image = _extract_generated_image(omni_outputs)
assert generated_image is not None, "No images generated"
assert generated_image.size == EXPECTED_OUTPUT_SIZE, f"Expected {EXPECTED_OUTPUT_SIZE}, got {generated_image.size}"
return generated_image
@pytest.mark.core_model
@pytest.mark.advanced_model
@pytest.mark.diffusion
@hardware_test(res={"cuda": "H100"})
def test_sensenova_u1_text2img(run_level):
"""Test SenseNova-U1 text2img (single-stage diffusion, no deploy YAML)."""
with OmniRunner(
"SenseNova/SenseNova-U1-8B-MoT",
stage_configs_path=None,
) as runner:
generated_image = _generate_sensenova_u1_image(runner.omni)
if run_level == "advanced_model":
_validate_pixels(generated_image)
+4
View File
@@ -766,6 +766,10 @@ class OmniDiffusionConfig:
self.model_class_name = "BagelPipeline"
self.set_tf_model_config(TransformerConfig())
self.update_multimodal_support()
elif model_type == "neo_chat":
self.model_class_name = "SenseNovaU1Pipeline"
self.tf_model_config = TransformerConfig()
self.update_multimodal_support()
elif model_type == "nextstep":
if self.model_class_name is None:
self.model_class_name = "NextStep11Pipeline"
+46
View File
@@ -69,6 +69,34 @@ def supports_audio_output(model_class_name: str) -> bool:
return bool(getattr(model_cls, "support_audio_output", False))
def get_extra_body_params(model_class_name: str) -> frozenset[str]:
"""Return the set of extra_body keys accepted by a pipeline.
Each pipeline can declare ``EXTRA_BODY_PARAMS: ClassVar[frozenset[str]]``
to advertise which request-level parameters should be forwarded from
``extra_body`` to ``OmniDiffusionSamplingParams.extra_args``.
Returns an empty frozenset when the pipeline does not declare any.
"""
model_cls = DiffusionModelRegistry._try_load_model_cls(model_class_name)
if model_cls is None:
return frozenset()
return frozenset(getattr(model_cls, "EXTRA_BODY_PARAMS", frozenset()))
def get_extra_output_params(model_class_name: str) -> frozenset[str]:
"""Return the set of custom_output keys to expose in API response metrics.
Each pipeline can declare ``EXTRA_OUTPUT_PARAMS: ClassVar[frozenset[str]]``
to advertise which ``DiffusionOutput.custom_output`` keys should be
copied into the response ``metrics`` dict.
Returns an empty frozenset when the pipeline does not declare any.
"""
model_cls = DiffusionModelRegistry._try_load_model_cls(model_class_name)
if model_cls is None:
return frozenset()
return frozenset(getattr(model_cls, "EXTRA_OUTPUT_PARAMS", frozenset()))
class DiffusionEngine:
"""The diffusion engine for vLLM-Omni diffusion models."""
@@ -230,6 +258,11 @@ class DiffusionEngine:
"postprocess_time_ms": postprocess_time * 1000,
}
# Detect text output: when the pipeline returns a string (e.g.,
# SenseNova-U1 / BAGEL single-stage img2text / text2text), wrap it
# as a text-type response instead of an image.
is_text_output = isinstance(output_data, str) and custom_output.get("text_output") is not None
# Handle single request or multiple requests
is_audio_output = supports_audio_output(self.od_config.model_class_name)
if is_audio_output and model_audio_sample_rate is None:
@@ -247,6 +280,19 @@ class DiffusionEngine:
prompt = request.prompts[0]
request_id = request.request_ids[0] if request.request_ids else ""
if is_text_output:
return [
OmniRequestOutput.from_diffusion(
request_id=request_id,
images=[],
prompt=prompt,
metrics=metrics,
custom_output=custom_output,
final_output_type="text",
stage_durations=output.stage_durations,
peak_memory_mb=output.peak_memory_mb,
),
]
if is_audio_output:
request_audio_payload = outputs[0] if len(outputs) == 1 else outputs
return [
@@ -0,0 +1,9 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
from .pipeline_sensenova_u1 import SenseNovaU1Pipeline, get_sensenova_u1_post_process_func
__all__ = [
"SenseNovaU1Pipeline",
"get_sensenova_u1_post_process_func",
]
File diff suppressed because it is too large. Load diff
@@ -0,0 +1,741 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
"""Qwen3 LLM with Mixture-of-Tokenizers (MoT) for SenseNova-U1.
Ported from the sensenova_u1 package with vllm tensor-parallel support:
- QKVParallelLinear for fused q/k/v projections (both und and gen paths)
- MergedColumnParallelLinear for fused gate+up projections
- RowParallelLinear for o_proj and down_proj
- VocabParallelEmbedding for token embeddings
"""
from __future__ import annotations
import copy
from dataclasses import dataclass
import torch
import torch.nn as nn
from transformers.cache_utils import DynamicCache
from vllm.model_executor.layers.linear import (
MergedColumnParallelLinear,
QKVParallelLinear,
RowParallelLinear,
)
from vllm.model_executor.layers.logits_processor import LogitsProcessor
from vllm.model_executor.layers.vocab_parallel_embedding import (
ParallelLMHead,
VocabParallelEmbedding,
)
from vllm_omni.diffusion.attention.backends.abstract import AttentionMetadata
from vllm_omni.diffusion.attention.layer import Attention
@dataclass
class SenseNovaU1ModelOutput:
last_hidden_state: torch.Tensor
past_key_values: DynamicCache | None = None
@dataclass
class SenseNovaU1CausalLMOutput:
logits: torch.Tensor | None = None
past_key_values: DynamicCache | None = None
hidden_states: torch.Tensor | None = None
inputs_embeds: torch.Tensor | None = None
# ---------------------------------------------------------------------------
# Mask / Cache utilities
# ---------------------------------------------------------------------------
def create_block_causal_mask(index: torch.Tensor):
"""Block-wise causal mask from 1D time-index. Returns (1, 1, L, L)."""
L = index.size(0)
idx_i = index.unsqueeze(1).expand(L, L)
idx_j = index.unsqueeze(0).expand(L, L)
arange = torch.arange(L, device=index.device)
mask = (idx_j == idx_i) | (arange.unsqueeze(0) <= arange.unsqueeze(1))
return torch.where(mask[None, None], 0.0, float("-inf"))
def prepare_flash_kv_cache(past_key_values, current_len: int, batch_size: int):
"""Convert prefix cache [B,H,S,D] → flash layout [B,S,H,D] and
preallocate buffers for [prefix + current] tokens."""
if past_key_values is None:
return
for layer in past_key_values.layers:
past_k, past_v = layer.keys, layer.values
if past_k is None or past_v is None:
layer.flash_prefix_len = 0
layer.flash_total_len = current_len
layer.flash_k_cache = None
layer.flash_v_cache = None
continue
past_k_flash = past_k.transpose(1, 2).contiguous()
past_v_flash = past_v.transpose(1, 2).contiguous()
prefix_len = past_k_flash.shape[1]
total_len = prefix_len + current_len
k_cache = torch.empty(
(batch_size, total_len, past_k_flash.shape[2], past_k_flash.shape[3]),
device=past_k_flash.device,
dtype=past_k_flash.dtype,
)
v_cache = torch.empty_like(k_cache)
k_cache[:, :prefix_len].copy_(past_k_flash)
v_cache[:, :prefix_len].copy_(past_v_flash)
layer.flash_prefix_len = prefix_len
layer.flash_total_len = total_len
layer.flash_k_cache = k_cache
layer.flash_v_cache = v_cache
def clear_flash_kv_cache(past_key_values):
if past_key_values is None:
return
for layer in past_key_values.layers:
for attr in ("flash_prefix_len", "flash_total_len", "flash_k_cache", "flash_v_cache"):
if hasattr(layer, attr):
delattr(layer, attr)
# ---------------------------------------------------------------------------
# RoPE
# ---------------------------------------------------------------------------
def rotate_half(x):
x1 = x[..., : x.shape[-1] // 2]
x2 = x[..., x.shape[-1] // 2 :]
return torch.cat((-x2, x1), dim=-1)
def apply_rotary_pos_emb(q, k, cos, sin, unsqueeze_dim=1):
cos = cos.unsqueeze(unsqueeze_dim)
sin = sin.unsqueeze(unsqueeze_dim)
return (q * cos) + (rotate_half(q) * sin), (k * cos) + (rotate_half(k) * sin)
def _compute_default_rope_parameters(config, device=None, **_kw):
base = config.rope_theta
partial_rotary_factor = getattr(config, "partial_rotary_factor", 1.0)
head_dim = getattr(config, "head_dim", config.hidden_size // config.num_attention_heads)
dim = int(head_dim * partial_rotary_factor)
inv_freq = 1.0 / (base ** (torch.arange(0, dim, 2, dtype=torch.int64).float().to(device) / dim))
return inv_freq, 1.0
class Qwen3RotaryEmbedding(nn.Module):
"""Custom RoPE that doubles head_dim then subsamples every-other freq."""
def __init__(self, config, device=None):
super().__init__()
self.config = config
def _rope_init_fn(cfg, dev=None):
inv_freq, attn_scaling = _compute_default_rope_parameters(cfg, dev)
cfg2 = copy.deepcopy(cfg)
head_dim = getattr(cfg2, "head_dim", cfg2.hidden_size // cfg2.num_attention_heads)
cfg2.head_dim = int(head_dim) * 2
inv_freq_full, _ = _compute_default_rope_parameters(cfg2, dev)
return inv_freq_full[::2], attn_scaling
inv_freq, self.attention_scaling = _rope_init_fn(config, device)
self.register_buffer("inv_freq", inv_freq, persistent=False)
@torch.no_grad()
def forward(self, x, position_ids):
inv_freq_expanded = self.inv_freq[None, :, None].float().expand(position_ids.shape[0], -1, 1).to(x.device)
position_ids_expanded = position_ids[:, None, :].float()
device_type = x.device.type if x.device.type != "mps" else "cpu"
with torch.autocast(device_type=device_type, enabled=False):
freqs = (inv_freq_expanded.float() @ position_ids_expanded.float()).transpose(1, 2)
emb = torch.cat((freqs, freqs), dim=-1)
cos = emb.cos() * self.attention_scaling
sin = emb.sin() * self.attention_scaling
return cos.to(dtype=x.dtype), sin.to(dtype=x.dtype)
# ---------------------------------------------------------------------------
# Norms
# ---------------------------------------------------------------------------
class Qwen3RMSNorm(nn.Module):
def __init__(self, hidden_size, eps=1e-6):
super().__init__()
self.weight = nn.Parameter(torch.ones(hidden_size))
self.variance_epsilon = eps
def forward(self, hidden_states):
input_dtype = hidden_states.dtype
hidden_states = hidden_states.to(torch.float32)
variance = hidden_states.pow(2).mean(-1, keepdim=True)
hidden_states = hidden_states * torch.rsqrt(variance + self.variance_epsilon)
return self.weight * hidden_states.to(input_dtype)
# ---------------------------------------------------------------------------
# MLP
# ---------------------------------------------------------------------------
class SenseNovaU1MLP(nn.Module):
def __init__(
self, hidden_size: int, intermediate_size: int, hidden_act: str = "silu", quant_config=None, prefix: str = ""
):
super().__init__()
self.gate_up_proj = MergedColumnParallelLinear(
hidden_size,
[intermediate_size, intermediate_size],
bias=False,
quant_config=quant_config,
prefix=f"{prefix}.gate_up_proj",
)
self.down_proj = RowParallelLinear(
intermediate_size,
hidden_size,
bias=False,
input_is_parallel=True,
quant_config=quant_config,
prefix=f"{prefix}.down_proj",
)
from transformers.activations import ACT2FN
self.act_fn = ACT2FN[hidden_act]
def forward(self, x):
gate_up, _ = self.gate_up_proj(x)
gate, up = gate_up.chunk(2, dim=-1)
x = self.act_fn(gate) * up
x, _ = self.down_proj(x)
return x
# ---------------------------------------------------------------------------
# Attention with MoT
# ---------------------------------------------------------------------------
class SenseNovaU1Attention(nn.Module):
"""Multi-head attention with Mixture-of-Tokenizers (MoT) routing.
Two parallel sets of Q/K/V/O projections:
- qkv_proj / o_proj: understanding (und) path
- qkv_proj_mot_gen / o_proj_mot_gen: generation (gen) path
3D RoPE: time (t), height (h), width (w) with separate frequencies.
"""
def __init__(self, config, layer_idx: int, quant_config=None, prefix: str = ""):
super().__init__()
self.config = config
self.layer_idx = layer_idx
self.hidden_size = config.hidden_size
self.head_dim = getattr(config, "head_dim", config.hidden_size // config.num_attention_heads)
self.total_num_heads = config.num_attention_heads
self.total_num_kv_heads = getattr(config, "num_key_value_heads", config.num_attention_heads)
self.num_kv_groups = self.total_num_heads // self.total_num_kv_heads
self.scaling = self.head_dim**-0.5
self.attention_dropout = getattr(config, "attention_dropout", 0.0)
bias = getattr(config, "attention_bias", True)
# Understanding path
self.qkv_proj = QKVParallelLinear(
self.hidden_size,
self.head_dim,
self.total_num_heads,
self.total_num_kv_heads,
bias=bias,
quant_config=quant_config,
prefix=f"{prefix}.qkv_proj",
)
self.o_proj = RowParallelLinear(
self.total_num_heads * self.head_dim,
self.hidden_size,
bias=bias,
input_is_parallel=True,
quant_config=quant_config,
prefix=f"{prefix}.o_proj",
)
# Generation (MoT) path
self.qkv_proj_mot_gen = QKVParallelLinear(
self.hidden_size,
self.head_dim,
self.total_num_heads,
self.total_num_kv_heads,
bias=bias,
quant_config=quant_config,
prefix=f"{prefix}.qkv_proj_mot_gen",
)
self.o_proj_mot_gen = RowParallelLinear(
self.total_num_heads * self.head_dim,
self.hidden_size,
bias=bias,
input_is_parallel=True,
quant_config=quant_config,
prefix=f"{prefix}.o_proj_mot_gen",
)
# Local head counts (TP-sharded)
self.num_heads = self.qkv_proj.num_heads
self.num_kv_heads = self.qkv_proj.num_kv_heads
self.q_size = self.num_heads * self.head_dim
self.kv_size = self.num_kv_heads * self.head_dim
# QK norms — understanding
self.q_norm = Qwen3RMSNorm(self.head_dim // 2, eps=config.rms_norm_eps)
self.k_norm = Qwen3RMSNorm(self.head_dim // 2, eps=config.rms_norm_eps)
self.q_norm_hw = Qwen3RMSNorm(self.head_dim // 2, eps=config.rms_norm_eps)
self.k_norm_hw = Qwen3RMSNorm(self.head_dim // 2, eps=config.rms_norm_eps)
# QK norms — generation
self.q_norm_mot_gen = Qwen3RMSNorm(self.head_dim // 2, eps=config.rms_norm_eps)
self.k_norm_mot_gen = Qwen3RMSNorm(self.head_dim // 2, eps=config.rms_norm_eps)
self.q_norm_hw_mot_gen = Qwen3RMSNorm(self.head_dim // 2, eps=config.rms_norm_eps)
self.k_norm_hw_mot_gen = Qwen3RMSNorm(self.head_dim // 2, eps=config.rms_norm_eps)
# 3D RoPE: t uses half head_dim, h/w each use quarter head_dim
t_config = copy.deepcopy(config)
t_config.head_dim = config.head_dim // 2
self.rotary_emb = Qwen3RotaryEmbedding(config=t_config)
hw_config = copy.deepcopy(config)
hw_config.head_dim = config.head_dim // 4
hw_config.rope_theta = config.rope_theta_hw
hw_config.max_position_embeddings = config.max_position_embeddings_hw
self.rotary_emb_hw = Qwen3RotaryEmbedding(config=hw_config)
self.attn = Attention(
num_heads=self.num_heads,
head_size=self.head_dim,
causal=False,
softmax_scale=self.scaling,
num_kv_heads=self.num_heads,
prefix=f"{prefix}.attn",
)
self.attn.attention = self.attn.sdpa_fallback
@staticmethod
def _align_mask_dtype(mask: torch.Tensor | None, query: torch.Tensor) -> torch.Tensor | None:
"""SDPA requires float ``attn_mask`` to match query's dtype."""
if mask is None or not mask.is_floating_point() or mask.dtype == query.dtype:
return mask
return mask.to(query.dtype)
def _run_attn(
self,
query_bhsd: torch.Tensor,
key_bhsd: torch.Tensor,
value_bhsd: torch.Tensor,
attention_mask: torch.Tensor | None,
) -> torch.Tensor:
"""Run unified attention with [B, H, S, D] inputs. Returns [B, S, H, D]."""
if self.num_kv_groups > 1:
n = self.num_kv_groups
key_bhsd = key_bhsd.repeat_interleave(n, dim=1)
value_bhsd = value_bhsd.repeat_interleave(n, dim=1)
q = query_bhsd.transpose(1, 2).contiguous()
k = key_bhsd.transpose(1, 2).contiguous()
v = value_bhsd.transpose(1, 2).contiguous()
attention_mask = self._align_mask_dtype(attention_mask, q)
attn_metadata = AttentionMetadata(attn_mask=attention_mask) if attention_mask is not None else None
return self.attn(q, k, v, attn_metadata)
def _run_attn_bshd(
self,
query_bshd: torch.Tensor,
key_bshd: torch.Tensor,
value_bshd: torch.Tensor,
attention_mask: torch.Tensor | None,
) -> torch.Tensor:
"""Run unified attention with [B, S, H, D] inputs. Returns [B, S, H, D]."""
if self.num_kv_groups > 1:
n = self.num_kv_groups
key_bshd = key_bshd.repeat_interleave(n, dim=2)
value_bshd = value_bshd.repeat_interleave(n, dim=2)
attention_mask = self._align_mask_dtype(attention_mask, query_bshd)
attn_metadata = AttentionMetadata(attn_mask=attention_mask) if attention_mask is not None else None
return self.attn(query_bshd, key_bshd, value_bshd, attn_metadata)
def _project_and_rope(self, hidden_states, indexes, qkv_proj, q_norm, k_norm, q_norm_hw, k_norm_hw):
"""Project Q/K/V via the given QKVParallelLinear and apply 3D RoPE."""
input_shape = hidden_states.shape[:-1] # (B, S)
qkv, _ = qkv_proj(hidden_states)
q, k, v = qkv.split([self.q_size, self.kv_size, self.kv_size], dim=-1)
q = q.view(*input_shape, self.num_heads, self.head_dim)
k = k.view(*input_shape, self.num_kv_heads, self.head_dim)
v = v.view(*input_shape, self.num_kv_heads, self.head_dim)
# Split head_dim into t and hw halves
q_t, q_hw = q.chunk(2, dim=-1)
k_t, k_hw = k.chunk(2, dim=-1)
# Norms then transpose to [B, H, S, D/2]
q_t = q_norm(q_t).transpose(1, 2)
k_t = k_norm(k_t).transpose(1, 2)
q_hw = q_norm_hw(q_hw).transpose(1, 2)
k_hw = k_norm_hw(k_hw).transpose(1, 2)
# Split hw into h and w quarters
q_h, q_w = q_hw.chunk(2, dim=-1)
k_h, k_w = k_hw.chunk(2, dim=-1)
# RoPE for each dimension
cos_t, sin_t = self.rotary_emb(hidden_states, indexes[0].unsqueeze(0))
q_t, k_t = apply_rotary_pos_emb(q_t, k_t, cos_t, sin_t)
cos_h, sin_h = self.rotary_emb_hw(hidden_states, indexes[1].unsqueeze(0))
q_h, k_h = apply_rotary_pos_emb(q_h, k_h, cos_h, sin_h)
cos_w, sin_w = self.rotary_emb_hw(hidden_states, indexes[2].unsqueeze(0))
q_w, k_w = apply_rotary_pos_emb(q_w, k_w, cos_w, sin_w)
# Reassemble: [B, H, S, head_dim]
query_states = torch.cat([q_t, q_h, q_w], dim=-1)
key_states = torch.cat([k_t, k_h, k_w], dim=-1)
value_states = v.transpose(1, 2) # [B, H, S, D]
return query_states, key_states, value_states
def forward_und(self, hidden_states, indexes, attention_mask, past_key_values=None, **kwargs):
"""Understanding path — unified Attention with explicit 4D mask."""
input_shape = hidden_states.shape[:-1]
query_states, key_states, value_states = self._project_and_rope(
hidden_states,
indexes,
self.qkv_proj,
self.q_norm,
self.k_norm,
self.q_norm_hw,
self.k_norm_hw,
)
update_cache = kwargs.get("update_cache", True)
if past_key_values is not None:
if update_cache:
key_states, value_states = past_key_values.update(key_states, value_states, self.layer_idx)
else:
layer = past_key_values.layers[self.layer_idx]
if layer.keys is not None:
key_states = torch.cat([layer.keys, key_states], dim=2)
value_states = torch.cat([layer.values, value_states], dim=2)
attn_output = self._run_attn(query_states, key_states, value_states, attention_mask)
attn_output = attn_output.reshape(*input_shape, -1).contiguous()
attn_output, _ = self.o_proj(attn_output)
return attn_output
def forward_gen(self, hidden_states, indexes, attention_mask, past_key_values=None, **kwargs):
"""Generation path — unified Attention, bidirectional with optional KV cache."""
input_shape = hidden_states.shape[:-1]
query_states, key_states, value_states = self._project_and_rope(
hidden_states,
indexes,
self.qkv_proj_mot_gen,
self.q_norm_mot_gen,
self.k_norm_mot_gen,
self.q_norm_hw_mot_gen,
self.k_norm_hw_mot_gen,
)
update_cache = kwargs.get("update_cache", True)
if attention_mask is None:
# Bidirectional path: no causal mask, optionally attend to a prefix.
q = query_states.transpose(1, 2).contiguous() # [B,S,H,D]
k_cur = key_states.transpose(1, 2).contiguous()
v_cur = value_states.transpose(1, 2).contiguous()
if past_key_values is not None:
if update_cache:
key_states, value_states = past_key_values.update(key_states, value_states, self.layer_idx)
k = key_states.transpose(1, 2).contiguous()
v = value_states.transpose(1, 2).contiguous()
else:
layer = past_key_values.layers[self.layer_idx]
if hasattr(layer, "flash_k_cache") and layer.flash_k_cache is not None:
prefix_len = layer.flash_prefix_len
cur_len = k_cur.shape[1]
layer.flash_k_cache[:, prefix_len : prefix_len + cur_len].copy_(k_cur)
layer.flash_v_cache[:, prefix_len : prefix_len + cur_len].copy_(v_cur)
k = layer.flash_k_cache[:, : prefix_len + cur_len]
v = layer.flash_v_cache[:, : prefix_len + cur_len]
else:
past_k = past_key_values.layers[self.layer_idx].keys
past_v = past_key_values.layers[self.layer_idx].values
if past_k is not None:
k = torch.cat([past_k.transpose(1, 2).contiguous(), k_cur], dim=1)
v = torch.cat([past_v.transpose(1, 2).contiguous(), v_cur], dim=1)
else:
k, v = k_cur, v_cur
else:
k, v = k_cur, v_cur
attn_output = self._run_attn_bshd(q, k, v, None)
attn_output = attn_output.reshape(*input_shape, -1).contiguous()
attn_output, _ = self.o_proj_mot_gen(attn_output)
return attn_output
# Masked fallback with explicit 4D additive mask
if past_key_values is not None:
if update_cache:
key_states, value_states = past_key_values.update(key_states, value_states, self.layer_idx)
else:
layer = past_key_values.layers[self.layer_idx]
if layer.keys is not None:
key_states = torch.cat([layer.keys, key_states], dim=2)
value_states = torch.cat([layer.values, value_states], dim=2)
attn_output = self._run_attn(query_states, key_states, value_states, attention_mask)
attn_output = attn_output.reshape(*input_shape, -1).contiguous()
attn_output, _ = self.o_proj_mot_gen(attn_output)
return attn_output
def forward(
self,
hidden_states,
image_gen_indicators,
exist_und,
exist_gen,
indexes,
attention_mask,
past_key_values=None,
**kwargs,
):
if exist_und and not exist_gen:
return self.forward_und(hidden_states, indexes, attention_mask, past_key_values, **kwargs)
if not exist_und and exist_gen:
return self.forward_gen(hidden_states, indexes, attention_mask, past_key_values, **kwargs)
raise NotImplementedError("Mixed und+gen tokens in a single forward not implemented for initial port")
# ---------------------------------------------------------------------------
# Decoder Layer
# ---------------------------------------------------------------------------
class SenseNovaU1DecoderLayer(nn.Module):
def __init__(self, config, layer_idx: int, quant_config=None, prefix: str = ""):
super().__init__()
self.self_attn = SenseNovaU1Attention(
config, layer_idx, quant_config=quant_config, prefix=f"{prefix}.self_attn"
)
self.mlp = SenseNovaU1MLP(
config.hidden_size,
config.intermediate_size,
config.hidden_act,
quant_config=quant_config,
prefix=f"{prefix}.mlp",
)
self.mlp_mot_gen = SenseNovaU1MLP(
config.hidden_size,
config.intermediate_size,
config.hidden_act,
quant_config=quant_config,
prefix=f"{prefix}.mlp_mot_gen",
)
self.input_layernorm = Qwen3RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
self.input_layernorm_mot_gen = Qwen3RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
self.post_attention_layernorm = Qwen3RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
self.post_attention_layernorm_mot_gen = Qwen3RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
self.attention_type = config.layer_types[layer_idx]
def _forward_und(self, hidden_states, indexes, attention_mask, past_key_values, **kwargs):
residual = hidden_states
hidden_states = self.input_layernorm(hidden_states)
hidden_states = self.self_attn(
hidden_states,
image_gen_indicators=None,
exist_und=True,
exist_gen=False,
indexes=indexes,
attention_mask=attention_mask,
past_key_values=past_key_values,
**kwargs,
)
hidden_states = residual + hidden_states
residual = hidden_states
hidden_states = self.mlp(self.post_attention_layernorm(hidden_states))
return residual + hidden_states
def _forward_gen(self, hidden_states, indexes, attention_mask, past_key_values, **kwargs):
residual = hidden_states
hidden_states = self.input_layernorm_mot_gen(hidden_states)
hidden_states = self.self_attn(
hidden_states,
image_gen_indicators=None,
exist_und=False,
exist_gen=True,
indexes=indexes,
attention_mask=attention_mask,
past_key_values=past_key_values,
**kwargs,
)
hidden_states = residual + hidden_states
residual = hidden_states
hidden_states = self.mlp_mot_gen(self.post_attention_layernorm_mot_gen(hidden_states))
return residual + hidden_states
def forward(
self,
hidden_states,
image_gen_indicators,
exist_und,
exist_gen,
indexes,
attention_mask,
past_key_values=None,
**kwargs,
):
if exist_und and not exist_gen:
return self._forward_und(hidden_states, indexes, attention_mask, past_key_values, **kwargs)
if not exist_und and exist_gen:
return self._forward_gen(hidden_states, indexes, attention_mask, past_key_values, **kwargs)
raise NotImplementedError("Mixed und+gen tokens in a single forward not implemented for initial port")
# ---------------------------------------------------------------------------
# Model (decoder stack)
# ---------------------------------------------------------------------------
class SenseNovaU1Model(nn.Module):
def __init__(self, config, quant_config=None, prefix: str = ""):
super().__init__()
self.config = config
self.embed_tokens = VocabParallelEmbedding(
config.vocab_size,
config.hidden_size,
prefix=f"{prefix}.embed_tokens",
)
self.layers = nn.ModuleList(
[
SenseNovaU1DecoderLayer(config, i, quant_config=quant_config, prefix=f"{prefix}.layers.{i}")
for i in range(config.num_hidden_layers)
]
)
self.norm = Qwen3RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
self.norm_mot_gen = Qwen3RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
def forward(
self,
input_ids=None,
image_gen_indicators=None,
indexes=None,
attention_mask=None,
past_key_values=None,
inputs_embeds=None,
use_cache=None,
**kwargs,
):
if image_gen_indicators is None:
exist_und, exist_gen = True, False
else:
exist_und = (~image_gen_indicators).any().item()
exist_gen = image_gen_indicators.any().item()
if inputs_embeds is None:
inputs_embeds = self.embed_tokens(input_ids)
if use_cache and past_key_values is None:
past_key_values = DynamicCache()
# Resolve attention mask. Callers must always provide `indexes`; this
# keeps the model stateless and safe under concurrent requests.
assert indexes is not None, "SenseNovaU1Model.forward requires explicit `indexes`."
if not isinstance(attention_mask, dict):
past_len = past_key_values.get_seq_length() if past_key_values else 0
seq_len = inputs_embeds.shape[1]
total_len = past_len + seq_len
mask = torch.zeros(1, 1, seq_len, total_len, device=inputs_embeds.device)
if seq_len > 1:
causal = torch.tril(torch.ones(seq_len, seq_len, device=inputs_embeds.device))
mask[:, :, :, past_len:] = torch.where(causal == 1, 0.0, float("-inf"))
causal_mask_mapping = {"full_attention": mask}
else:
causal_mask_mapping = attention_mask
hidden_states = inputs_embeds
for layer in self.layers:
hidden_states = layer(
hidden_states,
image_gen_indicators=image_gen_indicators,
exist_und=exist_und,
exist_gen=exist_gen,
indexes=indexes,
attention_mask=causal_mask_mapping.get(layer.attention_type, causal_mask_mapping.get("full_attention")),
past_key_values=past_key_values,
use_cache=use_cache,
**kwargs,
)
if not exist_gen:
hidden_states = self.norm(hidden_states)
else:
hidden_states = self.norm_mot_gen(hidden_states)
return SenseNovaU1ModelOutput(
last_hidden_state=hidden_states,
past_key_values=past_key_values if use_cache else None,
)
# ---------------------------------------------------------------------------
# ForCausalLM wrapper
# ---------------------------------------------------------------------------
class SenseNovaU1ForCausalLM(nn.Module):
def __init__(self, config, quant_config=None, prefix: str = ""):
super().__init__()
self.config = config
self.model = SenseNovaU1Model(config, quant_config=quant_config, prefix=f"{prefix}.model")
self.vocab_size = config.vocab_size
self.lm_head = ParallelLMHead(config.vocab_size, config.hidden_size, prefix=f"{prefix}.lm_head")
# LogitsProcessor handles the TP all-gather of vocab-sharded ParallelLMHead
# outputs so callers see full-vocab logits regardless of tp_size.
self.logits_processor = LogitsProcessor(config.vocab_size)
def forward(
self,
input_ids=None,
indexes=None,
attention_mask=None,
past_key_values=None,
inputs_embeds=None,
use_cache=None,
embed_only: bool = False,
compute_logits: bool = True,
**kwargs,
):
# Routing every access through ``forward`` keeps any externally-attached
# hooks (e.g. CPU-offload swap) firing for sub-module accesses such as
# token embedding lookup or hidden-state-only model runs. Callers should
# prefer this entry point over reaching into ``self.model`` /
# ``self.lm_head`` directly so that offloaded weights get materialised
# on the right device first.
if embed_only:
if input_ids is None:
raise ValueError("embed_only=True requires input_ids")
return SenseNovaU1CausalLMOutput(
inputs_embeds=self.model.embed_tokens(input_ids),
)
outputs = self.model(
input_ids=input_ids,
indexes=indexes,
attention_mask=attention_mask,
past_key_values=past_key_values,
inputs_embeds=inputs_embeds,
use_cache=use_cache,
**kwargs,
)
logits = self.logits_processor(self.lm_head, outputs.last_hidden_state) if compute_logits else None
return SenseNovaU1CausalLMOutput(
logits=logits,
past_key_values=outputs.past_key_values,
hidden_states=outputs.last_hidden_state,
)
def get_input_embeddings(self):
return self.model.embed_tokens
@@ -222,6 +222,13 @@ class ModelLevelOffloadBackend(OffloadBackend):
except Exception as exc:
logger.debug("Failed to move VAE to GPU: %s", exc)
# Pin resident modules on GPU (small hot submodules called inside the DiT loop).
for res, name in zip(modules.resident_modules, modules.resident_names):
try:
res.to(self.device)
except Exception as exc:
logger.warning("Failed to move resident module '%s' to GPU: %s", name, exc)
# Apply sequential offloading hooks
apply_sequential_offload(
dit_modules=modules.dits,
@@ -237,9 +244,10 @@ class ModelLevelOffloadBackend(OffloadBackend):
self.enabled = True
logger.info(
"Model-level offloading enabled: %s <-> %s (mutual exclusion)",
"Model-level offloading enabled: %s <-> %s (mutual exclusion)%s",
", ".join(modules.dit_names),
", ".join(modules.encoder_names),
f"; resident on GPU: {', '.join(modules.resident_names)}" if modules.resident_names else "",
)
def disable(self) -> None:
+7
View File
@@ -221,6 +221,11 @@ _DIFFUSION_MODELS = {
"pipeline_dreamid_omni",
"DreamIDOmniPipeline",
),
"SenseNovaU1Pipeline": (
"sensenova_u1",
"pipeline_sensenova_u1",
"SenseNovaU1Pipeline",
),
"AudioXPipeline": (
"audiox",
"pipeline_audiox",
@@ -272,6 +277,7 @@ DiffusionModelRegistry = _ModelRegistry(
_NO_CACHE_ACCELERATION = {
# Pipelines that do not support cache acceleration (cache_dit / tea_cache).
"NextStep11Pipeline",
"SenseNovaU1Pipeline",
"AudioXPipeline",
}
@@ -475,6 +481,7 @@ _DIFFUSION_POST_PROCESS_FUNCS = {
"MagiHumanPipeline": "get_magi_human_post_process_func",
"OmniVoicePipeline": "get_omnivoice_post_process_func",
"DreamIDOmniPipeline": "get_dreamid_omni_post_process_func",
"SenseNovaU1Pipeline": "get_sensenova_u1_post_process_func",
}
_DIFFUSION_PRE_PROCESS_FUNCS = {
+3 -2
View File
@@ -450,11 +450,12 @@ class OmniBase(PDDisaggregationMixin):
except Exception:
logger.exception("[%s] Finalize request handling error", self.__class__.__name__)
images = getattr(engine_outputs, "images", []) if stage_meta["final_output_type"] == "image" else []
output_type = getattr(engine_outputs, "final_output_type", stage_meta["final_output_type"])
images = getattr(engine_outputs, "images", []) if output_type == "image" else []
return OmniRequestOutput(
request_id=req_id or "",
stage_id=stage_id,
final_output_type=stage_meta["final_output_type"],
final_output_type=output_type,
request_output=engine_outputs,
images=images,
trajectory_latents=getattr(engine_outputs, "trajectory_latents", None),
+155 -16
View File
@@ -14,6 +14,7 @@ from fastapi import Request
from PIL import Image
from pydantic import TypeAdapter
from vllm_omni.diffusion.diffusion_engine import get_extra_body_params, get_extra_output_params
from vllm_omni.entrypoints.async_omni import AsyncOmni
from vllm_omni.entrypoints.openai.protocol.chat_completion import OmniChatCompletionResponse
from vllm_omni.entrypoints.utils import coerce_param_message_types
@@ -95,7 +96,9 @@ from vllm_omni.entrypoints.openai.stage_params import (
from vllm_omni.entrypoints.openai.utils import (
get_stage_type,
get_supported_speakers_from_hf_config,
is_single_stage_diffusion,
parse_lora_request,
resolve_diffusion_od_config,
validate_requested_speaker,
)
from vllm_omni.lora.request import LoRARequest
@@ -119,6 +122,8 @@ class OmniOpenAIServingChat(OpenAIServingChat, AudioMixin):
_diffusion_engine: AsyncOmni | None = None
_diffusion_model_name: str = ""
_supported_speakers: set[str] | None = None
_diffusion_extra_body_params: frozenset[str] | None = None
_diffusion_extra_output_params: frozenset[str] | None = None
@classmethod
def for_diffusion(
@@ -145,6 +150,45 @@ class OmniOpenAIServingChat(OpenAIServingChat, AudioMixin):
instance._diffusion_model_name = model_name
return instance
def _get_diffusion_extra_body_params(self) -> frozenset[str]:
"""Return the pipeline-declared ``EXTRA_BODY_PARAMS`` (cached)."""
if self._diffusion_extra_body_params is not None:
return self._diffusion_extra_body_params
params: frozenset[str] = frozenset()
try:
od_config = resolve_diffusion_od_config(self.engine_client, self._diffusion_engine)
if od_config is not None and getattr(od_config, "model_class_name", None):
params = get_extra_body_params(od_config.model_class_name)
except Exception as e:
logger.warning("Failed to read EXTRA_BODY_PARAMS from pipeline: %s", e)
self._diffusion_extra_body_params = params
return params
def _get_diffusion_extra_output_params(
self,
custom_output: dict[str, Any] | None,
) -> dict[str, Any] | None:
"""Pick pipeline-declared ``EXTRA_OUTPUT_PARAMS`` keys from *custom_output*."""
if not custom_output:
return None
if self._diffusion_extra_output_params is None:
params: frozenset[str] = frozenset()
try:
od_config = resolve_diffusion_od_config(self.engine_client, self._diffusion_engine)
if od_config is not None and getattr(od_config, "model_class_name", None):
params = get_extra_output_params(od_config.model_class_name)
except Exception as e:
logger.warning("Failed to read EXTRA_OUTPUT_PARAMS from pipeline: %s", e)
self._diffusion_extra_output_params = params
if not self._diffusion_extra_output_params:
return None
out = {k: custom_output[k] for k in self._diffusion_extra_output_params if k in custom_output}
return out or None
def _get_supported_speakers(self) -> set[str]:
"""Load supported speakers from model config (cached)."""
if self._supported_speakers is not None:
@@ -386,6 +430,34 @@ class OmniOpenAIServingChat(OpenAIServingChat, AudioMixin):
logger.warning("Failed to build image-generation prompt for omni multistage: %s", e)
_image_gen_height = None
_image_gen_width = None
elif request.modalities and ("text" in request.modalities) and is_single_stage_diffusion(self.engine_client):
# Single-stage diffusion text output (img2text / text2text).
# Build a diffusion-style prompt with modalities=["text"] so the
# pipeline routes to its text generation path.
try:
extracted_prompt, reference_images = self._extract_diffusion_prompt_and_images_from_messages(
request.messages
)
if not extracted_prompt:
return self.create_error_response("No text prompt found in messages")
tprompt: OmniTextPrompt = {"prompt": extracted_prompt}
tprompt["modalities"] = ["text"]
if reference_images:
try:
img_bytes = base64.b64decode(reference_images[0])
img = Image.open(BytesIO(img_bytes))
tprompt["multi_modal_data"] = {"image": img}
tprompt["multi_modal_uuids"] = {"image": [f"{request_id}-image-0"]}
except Exception:
pass
engine_prompts = [tprompt]
except Exception as e:
logger.warning("Failed to build text-output prompt for single-stage diffusion: %s", e)
_image_gen_height = None
_image_gen_width = None
else:
_image_gen_height = None
_image_gen_width = None
@@ -418,6 +490,10 @@ class OmniOpenAIServingChat(OpenAIServingChat, AudioMixin):
sp.extra_args["cfg_text_scale"] = cfg_text_scale
if cfg_img_scale is not None:
sp.extra_args["cfg_img_scale"] = cfg_img_scale
for _key in self._get_diffusion_extra_body_params():
_val = extra_body.get(_key)
if _val is not None:
sp.extra_args[_key] = _val
self._log_inputs(
request_id,
@@ -1610,21 +1686,37 @@ class OmniOpenAIServingChat(OpenAIServingChat, AudioMixin):
continue
if omni_outputs.final_output_type == "text":
(
choices_data,
usage,
prompt_logprobs,
prompt_token_ids,
kv_transfer_params,
) = self._create_text_choice(
request,
omni_outputs,
tokenizer,
conversation,
role,
reasoning_parser,
)
final_res = omni_outputs.request_output
if omni_outputs.request_output is not None:
(
choices_data,
usage,
prompt_logprobs,
prompt_token_ids,
kv_transfer_params,
) = self._create_text_choice(
request,
omni_outputs,
tokenizer,
conversation,
role,
reasoning_parser,
)
final_res = omni_outputs.request_output
else:
# Diffusion pipeline text output (e.g. single-stage
# img2text / text2text) — no AR request_output, so build
# a simple text choice from custom_output.
text_body = (omni_outputs.custom_output or {}).get("text_output", "")
message = ChatMessage(role=role, content=text_body)
choices_data = [
ChatCompletionResponseChoice(
index=0,
message=message,
logprobs=None,
finish_reason="stop",
stop_reason=None,
)
]
elif omni_outputs.final_output_type == "audio":
choices_data = self._create_audio_choice(omni_outputs, role, request, stream=False)
elif omni_outputs.final_output_type == "image":
@@ -1640,6 +1732,9 @@ class OmniOpenAIServingChat(OpenAIServingChat, AudioMixin):
response_metrics = {}
response_metrics.setdefault("stage_durations", omni_outputs.stage_durations or {})
response_metrics.setdefault("peak_memory_mb", float(omni_outputs.peak_memory_mb or 0.0))
extra = self._get_diffusion_extra_output_params(omni_outputs.custom_output)
if extra:
response_metrics.update(extra)
choices.extend(choices_data)
response = OmniChatCompletionResponse(
@@ -2511,6 +2606,10 @@ class OmniOpenAIServingChat(OpenAIServingChat, AudioMixin):
gen_params.extra_args["cfg_text_scale"] = cfg_text_scale
if cfg_img_scale is not None:
gen_params.extra_args["cfg_img_scale"] = cfg_img_scale
for _key in self._get_diffusion_extra_body_params():
_val = extra_body.get(_key)
if _val is not None:
gen_params.extra_args[_key] = _val
if num_frames is not None:
gen_params.num_frames = num_frames
if guidance_scale_2 is not None:
@@ -2539,6 +2638,13 @@ class OmniOpenAIServingChat(OpenAIServingChat, AudioMixin):
except Exception as e: # pragma: no cover - safeguard
logger.warning("Failed to parse LoRA request: %s", e)
# Route text modality for single-stage diffusion (img2text / text2text)
requested_modalities = extra_body.get("modalities") or []
is_text_request = "text" in requested_modalities
if is_text_request:
gen_prompt["modalities"] = ["text"]
# Add reference image if provided (from messages content)
if pil_images:
if len(pil_images) == 1:
@@ -2588,6 +2694,38 @@ class OmniOpenAIServingChat(OpenAIServingChat, AudioMixin):
result = output
if result is None:
return self._create_error_response("No output generated from AsyncOmni")
# Text output path (img2text / text2text)
if is_text_request and result.final_output_type == "text":
text_body = (result.custom_output or {}).get("text_output", "")
message = ChatMessage(role="assistant", content=text_body)
choice = ChatCompletionResponseChoice(
index=0,
message=message,
finish_reason="stop",
logprobs=None,
stop_reason=None,
)
response = OmniChatCompletionResponse(
id=request_id,
created=created_time,
model=self._diffusion_model_name,
choices=[choice],
usage=UsageInfo(
prompt_tokens=len(prompt.split()),
completion_tokens=len(text_body.split()),
total_tokens=len(prompt.split()) + len(text_body.split()),
),
metrics=self._get_diffusion_extra_output_params(result.custom_output),
)
logger.info(
"Diffusion chat completed for request %s: text output (%d chars)",
request_id,
len(text_body),
)
return response
# Image output path (text2img / img2img)
final_output_type = getattr(result, "final_output_type", "image")
# Handle nested OmniRequestOutput structure where images might be in request_output
images = getattr(result.request_output, "images", [])
@@ -2702,7 +2840,7 @@ class OmniOpenAIServingChat(OpenAIServingChat, AudioMixin):
stop_reason=None,
)
response = ChatCompletionResponse(
response = OmniChatCompletionResponse(
id=request_id,
created=created_time,
model=self._diffusion_model_name,
@@ -2712,6 +2850,7 @@ class OmniOpenAIServingChat(OpenAIServingChat, AudioMixin):
completion_tokens=1,
total_tokens=len(prompt.split()) + 1,
),
metrics=self._get_diffusion_extra_output_params(result.custom_output),
)
logger.info(
+21
View File
@@ -70,6 +70,27 @@ def get_supported_speakers_from_hf_config(hf_config: Any) -> set[str]:
return set()
def resolve_diffusion_od_config(engine_client: Any, diffusion_engine: Any = None) -> Any:
"""Resolve the OmniDiffusionConfig from the engine or diffusion engine."""
od_config = None
if hasattr(engine_client, "get_diffusion_od_config"):
od_config = engine_client.get_diffusion_od_config()
if od_config is None and diffusion_engine is not None:
if hasattr(diffusion_engine, "get_diffusion_od_config"):
od_config = diffusion_engine.get_diffusion_od_config()
else:
od_config = getattr(diffusion_engine, "od_config", None)
return od_config
def is_single_stage_diffusion(engine_client: Any) -> bool:
"""Return True if the engine is a single-stage diffusion pipeline."""
stage_configs = getattr(engine_client, "stage_configs", None) or []
if len(stage_configs) != 1:
return False
return getattr(stage_configs[0], "stage_type", None) in ("diffusion", "DIFFUSION")
def validate_requested_speaker(speaker: str | None, supported_speakers: set[str]) -> str | None:
"""Normalize and validate an optional speaker value.