[New Model]: Add sensenova u1 support (#3319)
Signed-off-by: princepride <wangzhipeng628@gmail.com> Signed-off-by: 汪志鹏 <wangzhipeng628@gmail.com>
This commit is contained in:
20 files changed
+3838
-19
No files matched your search
@@ -64,6 +64,7 @@ th {
|
||||
| `Flux2Pipeline` | FLUX.2-dev | `black-forest-labs/FLUX.2-dev` | ✅︎ | ✅︎ | | |
|
||||
| `FishSpeechSlowARForConditionalGeneration` | Fish Speech S2 Pro | `fishaudio/s2-pro` | ✅︎ | ✅︎ | | |
|
||||
| `DreamIDOmniPipeline` | DreamID-Omni | `XuGuo699/DreamID-Omni` | ✅︎ | ✅︎ | | |
|
||||
| `SenseNovaU1Pipeline` | SenseNova-U1 (DiT-only) | `SenseNova/SenseNova-U1-8B-MoT` | ✅︎ | | | |
|
||||
| `HunyuanVideo15Pipeline` | HunyuanVideo-1.5-T2V | `hunyuanvideo-community/HunyuanVideo-1.5-Diffusers-480p_t2v`, `hunyuanvideo-community/HunyuanVideo-1.5-Diffusers-720p_t2v` | ✅︎ | ✅︎ | | |
|
||||
| `HunyuanVideo15ImageToVideoPipeline` | HunyuanVideo-1.5-I2V | `hunyuanvideo-community/HunyuanVideo-1.5-Diffusers-480p_i2v`, `hunyuanvideo-community/HunyuanVideo-1.5-Diffusers-720p_i2v` | ✅︎ | ✅︎ | | |
|
||||
| `VoxtralTTSForConditionalGeneration` | Voxtral TTS | `mistralai/Voxtral-4B-TTS-2603` | ✅︎ | ✅︎ | | |
|
||||
|
||||
@@ -127,6 +127,7 @@ The following tables show which models support each feature:
|
||||
| **Qwen-Image-Edit** | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ (decode) | ❌ | ❌ |
|
||||
| **Qwen-Image-Edit-2509** | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ (decode) | ✅ | ❌ | ❌ |
|
||||
| **Qwen-Image-Layered** | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ (decode) | ❌ | ❌ |
|
||||
| **SenseNova-U1** | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ |
|
||||
| **Stable-Diffusion3.5** | ❌ | ✅ | ❌ | ✅ | ✅ | ❌ | ✅ | ✅ (decode) | ❌ | ❌ |
|
||||
| **Z-Image** | ✅ | ✅ | ✅ | ❓ | ✅ (TP=2 only) | ✅ | ❌ | ✅ (decode) | ✅ | ❌ |
|
||||
| **ERNIE-Image** | ❌ | ✅ | ✅ | ❓ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ |
|
||||
|
||||
@@ -0,0 +1,208 @@
|
||||
# SenseNova-U1-8B-MoT
|
||||
|
||||
## Architecture
|
||||
|
||||
SenseNova-U1 is a unified Qwen3-based LLM with Mixture-of-Tokenizers (MoT) attention. Unlike two-stage pipelines (e.g., BAGEL), it handles text encoding, optional reasoning ("think mode"), and flow-matching-based image denoising entirely within a single diffusion stage.
|
||||
|
||||
| Feature | Description |
|
||||
| :------ | :---------- |
|
||||
| **Model type** | Unified LLM (single-stage diffusion pipeline) |
|
||||
| **Base LLM** | Qwen3 with MoT attention and 3D RoPE |
|
||||
| **Image generation** | Flow-matching Euler sampler, no separate VAE |
|
||||
| **Think mode** | Optional chain-of-thought reasoning before image generation |
|
||||
| **Parallelism** | Tensor Parallelism (TP) with fused QKV and fused gate/up projections |
|
||||
| **Modalities** | text2img, img2img, img2text (understanding), text2text |
|
||||
|
||||
## Quick Start
|
||||
|
||||
```bash
|
||||
cd examples/offline_inference/sensenova_u1
|
||||
|
||||
# Text-to-image
|
||||
python end2end.py --prompt "A cute cat" --think
|
||||
|
||||
# Image-to-image editing
|
||||
python end2end.py --prompt "Turn this into an oil painting" \
|
||||
--image input.png --think
|
||||
|
||||
# Image understanding (img2text)
|
||||
python end2end.py --modality img2text \
|
||||
--prompt "Describe this image in detail" \
|
||||
--image photo.jpg
|
||||
|
||||
# Text-to-text (pure chat)
|
||||
python end2end.py --modality text2text \
|
||||
--prompt "What is the capital of France?"
|
||||
|
||||
# Custom resolution (image generation)
|
||||
python end2end.py --prompt "A futuristic cityscape at sunset" \
|
||||
--width 2048 --height 1024 --think
|
||||
```
|
||||
|
||||
> **Note**: Default configuration works on a single **NVIDIA A100 (80GB)** or **H100** GPU.
|
||||
|
||||
## Text-to-Image (t2i)
|
||||
|
||||
Standard text-to-image with optional think mode:
|
||||
|
||||
```bash
|
||||
python end2end.py \
|
||||
--prompt "Close portrait of an elderly woman by a farmhouse window, textured skin, gentle smile, warm natural light" \
|
||||
--width 1536 --height 2720 \
|
||||
--think --print-think
|
||||
```
|
||||
|
||||
## Image-to-Image Editing (img2img)
|
||||
|
||||
Pass one or more `--image` paths to trigger img2img mode. The model uses the
|
||||
input image(s) as visual context for editing:
|
||||
|
||||
```bash
|
||||
# Single input image
|
||||
python end2end.py \
|
||||
--prompt "Add a sunset sky in the background" \
|
||||
--image photo.jpg \
|
||||
--width 2048 --height 2048 \
|
||||
--think
|
||||
|
||||
# Multiple reference images
|
||||
python end2end.py \
|
||||
--prompt "Combine the style of Image-1 with the content of Image-2" \
|
||||
--image style_ref.png content_ref.png \
|
||||
--width 2048 --height 2048 \
|
||||
--think
|
||||
```
|
||||
|
||||
### Dual CFG for img2img
|
||||
|
||||
img2img supports dual classifier-free guidance (text CFG + image CFG):
|
||||
|
||||
```bash
|
||||
python end2end.py \
|
||||
--prompt "Make the person smile" \
|
||||
--image portrait.jpg \
|
||||
--cfg-scale 4.0 \
|
||||
--img-cfg-scale 2.0 \
|
||||
--think
|
||||
```
|
||||
|
||||
| `img_cfg_scale` | Behavior |
|
||||
| :-------------- | :------- |
|
||||
| `1.0` (default) | Text-only CFG: guidance = image-condition → full-condition |
|
||||
| `== cfg_scale` | Standard CFG: guidance = unconditional → full-condition |
|
||||
| Other value | Dual CFG: separate text and image guidance strengths |
|
||||
|
||||
## Full Parameter Reference
|
||||
|
||||
### General Parameters
|
||||
|
||||
| Parameter | Default | Description |
|
||||
| :-------- | :------ | :---------- |
|
||||
| `--modality` | `auto` | Task modality: `auto`, `text2img`, `img2img`, `img2text`, `text2text` |
|
||||
| `--prompt` | "A cute cat..." | Text prompt / editing instruction / question |
|
||||
| `--image` | None | Input image path(s) for img2img or img2text |
|
||||
|
||||
### Image Generation Parameters (text2img / img2img)
|
||||
|
||||
| Parameter | Default | Description |
|
||||
| :-------- | :------ | :---------- |
|
||||
| `--height` | 2048 | Height of generated image (pixels) |
|
||||
| `--width` | 2048 | Width of generated image (pixels) |
|
||||
| `--seed` | 42 | Random seed for reproducibility |
|
||||
| `--num-steps` | 50 | Number of denoising steps |
|
||||
| `--cfg-scale` | 4.0 | Text classifier-free guidance scale |
|
||||
| `--img-cfg-scale` | 1.0 | Image CFG scale (img2img only, 1.0 = disabled) |
|
||||
| `--cfg-norm` | "none" | CFG normalization: none, global, channel, cfg_zero_star (t2i only) |
|
||||
| `--timestep-shift` | 3.0 | Timestep shift for flow-matching schedule |
|
||||
| `--t-eps` | 0.02 | Epsilon for timestep schedule |
|
||||
| `--think` | False | Enable think mode |
|
||||
| `--print-think` | False | Print think text to stdout |
|
||||
|
||||
### Text Generation Parameters (text2text / img2text)
|
||||
|
||||
| Parameter | Default | Description |
|
||||
| :-------- | :------ | :---------- |
|
||||
| `--max-tokens` | 512 | Maximum number of tokens to generate |
|
||||
| `--do-sample` | False | Use sampling instead of greedy decoding |
|
||||
| `--temperature` | 0.7 | Sampling temperature (higher = more diverse) |
|
||||
|
||||
### Infrastructure Parameters
|
||||
|
||||
| Parameter | Default | Description |
|
||||
| :-------- | :------ | :---------- |
|
||||
| `--model` | `SenseNova/SenseNova-U1-8B-MoT` | HuggingFace model ID or local path |
|
||||
| `--output` | `.` | Output directory for saved images |
|
||||
| `--tensor-parallel-size` | 1 | Number of GPUs for tensor parallelism |
|
||||
| `--enforce-eager` | False | Disable torch.compile |
|
||||
| `--enable-cpu-offload` | False | Enable module-wise (sequential) CPU offload to reduce peak VRAM |
|
||||
|
||||
## Reducing GPU Memory Usage
|
||||
|
||||
For hardware with limited VRAM, enable **module-wise CPU offload** with
|
||||
`--enable-cpu-offload`. The pipeline implements `SupportsModuleOffload`, so the
|
||||
vision encoder (`vision_model`) and the Qwen3 LLM (`language_model`) are
|
||||
swapped between CPU and GPU on demand:
|
||||
|
||||
- During text/vision encoding, the LLM is on CPU.
|
||||
- During the diffusion loop, the vision encoder is on CPU.
|
||||
- Lightweight FM modules stay resident on GPU.
|
||||
|
||||
This lowers peak VRAM at the cost of extra CPU<->GPU transfers (use pinned
|
||||
memory) — useful for running SenseNova-U1 on consumer-grade GPUs.
|
||||
|
||||
```bash
|
||||
# Text-to-image with CPU offload
|
||||
python end2end.py \
|
||||
--prompt "A cute cat sitting on a windowsill" \
|
||||
--width 2048 --height 2048 \
|
||||
--enable-cpu-offload --think
|
||||
|
||||
# Image-to-image editing with CPU offload
|
||||
python end2end.py \
|
||||
--prompt "Turn this into an oil painting" \
|
||||
--image input.png \
|
||||
--width 2048 --height 2048 \
|
||||
--enable-cpu-offload --think
|
||||
|
||||
# Image understanding (img2text) with CPU offload
|
||||
python end2end.py \
|
||||
--modality img2text \
|
||||
--prompt "Describe this image in detail" \
|
||||
--image photo.jpg \
|
||||
--enable-cpu-offload
|
||||
```
|
||||
|
||||
> **Notes**
|
||||
> - CPU offload is **single-GPU only** (incompatible with `--tensor-parallel-size > 1`).
|
||||
> - First-step latency is higher because of the cold-start CPU<->GPU transfers.
|
||||
> - For more details and other offloading strategies, see
|
||||
> [CPU Offloading for Diffusion Models](../../../docs/user_guide/diffusion/cpu_offload_diffusion.md).
|
||||
|
||||
## Reproducing the E2E Test
|
||||
|
||||
The following command reproduces the pixel-validated CI test case:
|
||||
|
||||
```bash
|
||||
python end2end.py \
|
||||
--prompt "Close portrait of an elderly woman by a farmhouse window, textured skin, gentle smile, warm natural light, emotional documentary look. The portrait should feel polished and natural, with sharp eyes, realistic skin texture, accurate facial anatomy, and premium lighting that keeps the face as the main focus." \
|
||||
--width 1536 --height 2720 \
|
||||
--seed 42 --num-steps 50 \
|
||||
--cfg-scale 4.0 --timestep-shift 3.0 --cfg-norm none \
|
||||
--think --print-think \
|
||||
--output outputs
|
||||
```
|
||||
|
||||
The corresponding pytest:
|
||||
|
||||
```bash
|
||||
pytest -s -v tests/e2e/offline_inference/test_sensenova_u1_text2img.py \
|
||||
-m "advanced_model" --run-level "advanced_model"
|
||||
```
|
||||
|
||||
The img2img counterpart lives at
|
||||
`tests/e2e/offline_inference/test_sensenova_u1_img2img.py`.
|
||||
|
||||
## Online Serving
|
||||
|
||||
For OpenAI-compatible API serving, see
|
||||
[`examples/online_serving/sensenova_u1/`](../../examples/online_serving/sensenova_u1/).
|
||||
@@ -0,0 +1,281 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||
|
||||
"""
|
||||
End-to-end offline inference for SenseNova-U1-8B-MoT.
|
||||
|
||||
Supports four modalities: text2img, img2img, img2text, text2text.
|
||||
|
||||
Text-to-image:
|
||||
python end2end.py --prompt "A cute cat" --think
|
||||
|
||||
Image-to-image (editing):
|
||||
python end2end.py --prompt "Turn this into an oil painting" \
|
||||
--image input.png --think
|
||||
|
||||
Image understanding:
|
||||
python end2end.py --modality img2text \
|
||||
--prompt "Describe this image" --image photo.jpg
|
||||
|
||||
Text chat:
|
||||
python end2end.py --modality text2text \
|
||||
--prompt "What is the capital of France?"
|
||||
|
||||
See README.md for more examples.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import os
|
||||
|
||||
from PIL import Image
|
||||
|
||||
from vllm_omni.engine.arg_utils import nullify_stage_engine_defaults
|
||||
from vllm_omni.entrypoints.omni import Omni
|
||||
from vllm_omni.inputs.data import OmniDiffusionSamplingParams
|
||||
|
||||
|
||||
def parse_args():
|
||||
parser = argparse.ArgumentParser(
|
||||
description="SenseNova-U1 text-to-image / image-to-image / text / understanding via vLLM-Omni.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--model",
|
||||
default="SenseNova/SenseNova-U1-8B-MoT",
|
||||
help="HuggingFace model ID or local path.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--modality",
|
||||
default="auto",
|
||||
choices=["auto", "text2img", "img2img", "img2text", "text2text"],
|
||||
help="Task modality. 'auto' infers from --image (img2img/img2text if images, else t2i/t2t).",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--prompt",
|
||||
default="A cute cat sitting on a windowsill, soft natural light",
|
||||
help="Text prompt for generation or editing instruction.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--image",
|
||||
nargs="+",
|
||||
metavar="PATH",
|
||||
default=None,
|
||||
help="Input image path(s) for img2img or img2text.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--output",
|
||||
type=str,
|
||||
default=".",
|
||||
help="Output directory to save generated images.",
|
||||
)
|
||||
# Image dimensions
|
||||
parser.add_argument("--height", type=int, default=2048, help="Height of generated image.")
|
||||
parser.add_argument("--width", type=int, default=2048, help="Width of generated image.")
|
||||
|
||||
# Generation parameters
|
||||
parser.add_argument("--seed", type=int, default=42, help="Random seed for deterministic results.")
|
||||
parser.add_argument("--num-steps", type=int, default=50, help="Number of denoising steps.")
|
||||
parser.add_argument("--cfg-scale", type=float, default=4.0, help="Text classifier-free guidance scale.")
|
||||
parser.add_argument(
|
||||
"--img-cfg-scale",
|
||||
type=float,
|
||||
default=1.0,
|
||||
help="Image CFG scale for img2img (1.0 = disabled). "
|
||||
"When both --cfg-scale and --img-cfg-scale differ, dual CFG is used.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--cfg-norm",
|
||||
type=str,
|
||||
default="none",
|
||||
choices=["none", "global", "channel", "cfg_zero_star"],
|
||||
help="CFG normalization mode. cfg_zero_star is only for t2i.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--timestep-shift",
|
||||
type=float,
|
||||
default=3.0,
|
||||
help="Timestep shift for flow-matching schedule.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--t-eps",
|
||||
type=float,
|
||||
default=0.02,
|
||||
help="Epsilon for flow-matching timestep schedule.",
|
||||
)
|
||||
|
||||
# Text generation parameters (for text2text / img2text)
|
||||
parser.add_argument(
|
||||
"--max-tokens",
|
||||
type=int,
|
||||
default=2048,
|
||||
help="Max tokens for text generation (text2text / img2text).",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--do-sample",
|
||||
action="store_true",
|
||||
help="Use sampling instead of greedy decoding for text generation.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--temperature",
|
||||
type=float,
|
||||
default=0.7,
|
||||
help="Sampling temperature for text generation.",
|
||||
)
|
||||
|
||||
# Think mode
|
||||
parser.add_argument(
|
||||
"--think",
|
||||
action="store_true",
|
||||
help="Enable think mode: the model reasons about the prompt before generating the image.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--print-think",
|
||||
action="store_true",
|
||||
help="Print the generated think text to stdout.",
|
||||
)
|
||||
|
||||
# Advanced
|
||||
parser.add_argument(
|
||||
"--tensor-parallel-size",
|
||||
type=int,
|
||||
default=1,
|
||||
help="Number of GPUs for tensor parallelism.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--enforce-eager",
|
||||
action="store_true",
|
||||
help="Disable torch.compile and force eager execution.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--enable-cpu-offload",
|
||||
action="store_true",
|
||||
help=(
|
||||
"Enable model-level (sequential) CPU offloading. "
|
||||
"Mutually exclusive between vision_model and language_model: "
|
||||
"only one is on GPU at a time, reducing peak VRAM at the cost "
|
||||
"of extra CPU<->GPU transfers."
|
||||
),
|
||||
)
|
||||
|
||||
nullify_stage_engine_defaults(parser)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def _resolve_modality(args):
|
||||
"""Determine the effective modality from CLI args."""
|
||||
if args.modality != "auto":
|
||||
return args.modality
|
||||
has_images = args.image is not None
|
||||
# Default: image-producing tasks for backward compatibility
|
||||
return "img2img" if has_images else "text2img"
|
||||
|
||||
|
||||
def main():
|
||||
args = parse_args()
|
||||
os.makedirs(args.output, exist_ok=True)
|
||||
|
||||
modality = _resolve_modality(args)
|
||||
is_text_output = modality in ("text2text", "img2text")
|
||||
has_images = modality in ("img2img", "img2text")
|
||||
|
||||
omni = Omni(
|
||||
model=args.model,
|
||||
tensor_parallel_size=args.tensor_parallel_size,
|
||||
enforce_eager=args.enforce_eager,
|
||||
enable_cpu_offload=args.enable_cpu_offload,
|
||||
)
|
||||
|
||||
extra_args = {
|
||||
"cfg_scale": args.cfg_scale,
|
||||
"cfg_norm": args.cfg_norm,
|
||||
"timestep_shift": args.timestep_shift,
|
||||
"cfg_interval": (0.0, 1.0),
|
||||
"batch_size": 1,
|
||||
"think": args.think,
|
||||
"t_eps": args.t_eps,
|
||||
}
|
||||
if modality == "img2img":
|
||||
extra_args["img_cfg_scale"] = args.img_cfg_scale
|
||||
if is_text_output:
|
||||
extra_args["max_tokens"] = args.max_tokens
|
||||
extra_args["do_sample"] = args.do_sample
|
||||
extra_args["temperature"] = args.temperature
|
||||
|
||||
sampling_params = OmniDiffusionSamplingParams(
|
||||
height=args.height,
|
||||
width=args.width,
|
||||
seed=args.seed,
|
||||
num_inference_steps=args.num_steps,
|
||||
extra_args=extra_args,
|
||||
)
|
||||
|
||||
print(f"\n{'=' * 60}")
|
||||
print(f"SenseNova-U1 Configuration ({modality}):")
|
||||
print(f" Model : {args.model}")
|
||||
if not is_text_output:
|
||||
print(f" Image size : {args.width}x{args.height}")
|
||||
print(f" Steps : {args.num_steps}")
|
||||
print(f" CFG scale : {args.cfg_scale}")
|
||||
if modality == "img2img":
|
||||
print(f" Img CFG scale : {args.img_cfg_scale}")
|
||||
if has_images:
|
||||
print(f" Input images : {args.image}")
|
||||
if is_text_output:
|
||||
print(f" Max tokens : {args.max_tokens}")
|
||||
print(f" Temperature : {args.temperature}")
|
||||
print(f" Seed : {args.seed}")
|
||||
print(f" Think mode : {args.think}")
|
||||
print(f" TP size : {args.tensor_parallel_size}")
|
||||
print(f"{'=' * 60}\n")
|
||||
|
||||
# Build prompt dict
|
||||
if has_images:
|
||||
if not args.image:
|
||||
raise ValueError(f"{modality} requires --image.")
|
||||
input_images = [Image.open(p).convert("RGB") for p in args.image]
|
||||
if is_text_output:
|
||||
prompt_dict = {
|
||||
"prompt": args.prompt,
|
||||
"multi_modal_data": {"image": input_images},
|
||||
"modalities": ["text"],
|
||||
}
|
||||
else:
|
||||
prompt_dict = {
|
||||
"prompt": args.prompt,
|
||||
"multi_modal_data": {"image": input_images},
|
||||
"modalities": ["img2img"],
|
||||
}
|
||||
elif is_text_output:
|
||||
prompt_dict = {"prompt": args.prompt, "modalities": ["text"]}
|
||||
else:
|
||||
prompt_dict = {"prompt": args.prompt, "modalities": ["image"]}
|
||||
|
||||
outputs = list(
|
||||
omni.generate(
|
||||
prompts=prompt_dict,
|
||||
sampling_params_list=sampling_params,
|
||||
)
|
||||
)
|
||||
|
||||
for req_output in outputs:
|
||||
custom = getattr(req_output, "_custom_output", {}) or {}
|
||||
if args.print_think and custom.get("think_text"):
|
||||
print(f"[Think]\n{custom['think_text']}\n")
|
||||
|
||||
if is_text_output:
|
||||
text = custom.get("text_output", "")
|
||||
if not text:
|
||||
text = getattr(req_output, "text", "") or ""
|
||||
print(f"[Response]\n{text}")
|
||||
else:
|
||||
images = getattr(req_output, "images", None) or []
|
||||
if not images:
|
||||
print("[Warning] No images generated.")
|
||||
continue
|
||||
for j, img in enumerate(images):
|
||||
save_path = os.path.join(args.output, f"sensenova_u1_output_{j}.png")
|
||||
img.save(save_path)
|
||||
print(f"[Output] Saved {img.size[0]}x{img.size[1]} image to {save_path}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,163 @@
|
||||
# SenseNova-U1 Online Serving
|
||||
|
||||
## Launch the Server
|
||||
|
||||
```bash
|
||||
vllm serve SenseNova/SenseNova-U1-8B-MoT --omni --port 8091
|
||||
```
|
||||
|
||||
Or use the convenience script:
|
||||
|
||||
```bash
|
||||
cd examples/online_serving/sensenova_u1
|
||||
bash run_server.sh
|
||||
```
|
||||
|
||||
### Tensor Parallelism (TP)
|
||||
|
||||
```bash
|
||||
vllm serve SenseNova/SenseNova-U1-8B-MoT --omni --port 8091 \
|
||||
--tensor-parallel-size 2
|
||||
```
|
||||
|
||||
## Send Requests
|
||||
|
||||
```bash
|
||||
cd examples/online_serving/sensenova_u1
|
||||
```
|
||||
|
||||
### Text to Image (text2img)
|
||||
|
||||
**Python client:**
|
||||
|
||||
```bash
|
||||
python openai_chat_client.py \
|
||||
--prompt "A beautiful sunset over mountains" \
|
||||
--modality text2img \
|
||||
--height 2048 --width 2048 --num-steps 50
|
||||
```
|
||||
|
||||
**curl:**
|
||||
|
||||
```bash
|
||||
curl http://localhost:8091/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"messages": [{"role": "user", "content": [{"type": "text", "text": "A beautiful sunset over mountains"}]}],
|
||||
"modalities": ["image"],
|
||||
"height": 2048,
|
||||
"width": 2048,
|
||||
"num_inference_steps": 50,
|
||||
"seed": 42
|
||||
}'
|
||||
```
|
||||
|
||||
### Image to Image (img2img)
|
||||
|
||||
**Python client:**
|
||||
|
||||
```bash
|
||||
python openai_chat_client.py \
|
||||
--prompt "Turn this into an oil painting" \
|
||||
--modality img2img \
|
||||
--image-url /path/to/input.jpg \
|
||||
--height 2048 --width 2048
|
||||
```
|
||||
|
||||
**curl:**
|
||||
|
||||
```bash
|
||||
IMAGE_BASE64=$(base64 -w 0 input.jpg)
|
||||
|
||||
cat <<EOF > payload.json
|
||||
{
|
||||
"messages": [{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{"type": "text", "text": "Turn this into an oil painting"},
|
||||
{"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,${IMAGE_BASE64}"}}
|
||||
]
|
||||
}],
|
||||
"modalities": ["image"],
|
||||
"height": 2048,
|
||||
"width": 2048,
|
||||
"num_inference_steps": 50,
|
||||
"seed": 42
|
||||
}
|
||||
EOF
|
||||
|
||||
curl http://localhost:8091/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d @payload.json
|
||||
```
|
||||
|
||||
### Image to Text (img2text)
|
||||
|
||||
**Python client:**
|
||||
|
||||
```bash
|
||||
python openai_chat_client.py \
|
||||
--prompt "Describe this image in detail" \
|
||||
--modality img2text \
|
||||
--image-url /path/to/image.jpg
|
||||
```
|
||||
|
||||
**curl:**
|
||||
|
||||
```bash
|
||||
IMAGE_BASE64=$(base64 -w 0 image.jpg)
|
||||
|
||||
cat <<EOF > payload.json
|
||||
{
|
||||
"messages": [{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{"type": "text", "text": "Describe this image in detail"},
|
||||
{"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,${IMAGE_BASE64}"}}
|
||||
]
|
||||
}],
|
||||
"modalities": ["text"]
|
||||
}
|
||||
EOF
|
||||
|
||||
curl http://localhost:8091/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d @payload.json
|
||||
```
|
||||
|
||||
### Text to Text (text2text)
|
||||
|
||||
**Python client:**
|
||||
|
||||
```bash
|
||||
python openai_chat_client.py \
|
||||
--prompt "What is the capital of France?" \
|
||||
--modality text2text
|
||||
```
|
||||
|
||||
**curl:**
|
||||
|
||||
```bash
|
||||
curl http://localhost:8091/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"messages": [{"role": "user", "content": [{"type": "text", "text": "What is the capital of France?"}]}],
|
||||
"modalities": ["text"]
|
||||
}'
|
||||
```
|
||||
|
||||
## Python Client Arguments
|
||||
|
||||
| Argument | Default | Description |
|
||||
| :------- | :------ | :---------- |
|
||||
| `--prompt` / `-p` | `A cute cat` | Text prompt |
|
||||
| `--output` / `-o` | `sensenova_u1_output.png` | Output file path |
|
||||
| `--server` / `-s` | `http://localhost:8091` | Server URL |
|
||||
| `--image-url` / `-i` | `None` | Input image URL or local path (img2img/img2text) |
|
||||
| `--modality` / `-m` | `text2img` | `text2img`, `img2img`, `img2text`, `text2text` |
|
||||
| `--height` | `2048` | Image height (image generation only) |
|
||||
| `--width` | `2048` | Image width (image generation only) |
|
||||
| `--num-steps` | `50` | Number of inference steps (image generation only) |
|
||||
| `--seed` | `42` | Random seed |
|
||||
| `--cfg-scale` | `4.0` | CFG scale (image generation only) |
|
||||
| `--think` | `False` | Enable think mode |
|
||||
@@ -0,0 +1,188 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
SenseNova-U1 OpenAI-compatible chat client for all four task modalities.
|
||||
|
||||
Usage:
|
||||
# Text-to-image
|
||||
python openai_chat_client.py --prompt "A cute cat" --modality text2img
|
||||
|
||||
# Image-to-image editing
|
||||
python openai_chat_client.py --prompt "Turn this into an oil painting" \
|
||||
--modality img2img --image-url /path/to/image.jpg
|
||||
|
||||
# Image understanding (img2text)
|
||||
python openai_chat_client.py --prompt "Describe this image in detail" \
|
||||
--modality img2text --image-url /path/to/image.jpg
|
||||
|
||||
# Text-to-text (chat)
|
||||
python openai_chat_client.py --prompt "What is the capital of France?" \
|
||||
--modality text2text
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import base64
|
||||
from pathlib import Path
|
||||
|
||||
import requests
|
||||
|
||||
|
||||
def _encode_image(image_url: str) -> str:
|
||||
"""Encode a local file or URL to a base64 data URI."""
|
||||
if Path(image_url).exists():
|
||||
with open(image_url, "rb") as f:
|
||||
b64_data = base64.b64encode(f.read()).decode("utf-8")
|
||||
return f"data:image/jpeg;base64,{b64_data}"
|
||||
return image_url
|
||||
|
||||
|
||||
def generate(
|
||||
prompt: str,
|
||||
server_url: str = "http://localhost:8091",
|
||||
image_url: str | None = None,
|
||||
modality: str = "text2img",
|
||||
**kwargs: object,
|
||||
) -> bytes | str | None:
|
||||
"""Send a request to the SenseNova-U1 server.
|
||||
|
||||
All keyword arguments (height, width, seed, cfg_scale, think, …) are
|
||||
forwarded as top-level fields in the request payload. The serving layer
|
||||
maps standard keys (height, width, seed, num_inference_steps) to sampling
|
||||
params and forwards model-specific keys (think, cfg_norm, img_cfg_scale,
|
||||
…) to ``extra_args`` based on the pipeline's ``EXTRA_BODY_PARAMS``.
|
||||
|
||||
Returns:
|
||||
Image bytes (for image outputs) or text string (for text outputs).
|
||||
"""
|
||||
content = [{"type": "text", "text": prompt}]
|
||||
|
||||
if image_url:
|
||||
content.append({"type": "image_url", "image_url": {"url": _encode_image(image_url)}})
|
||||
|
||||
messages = [{"role": "user", "content": content}]
|
||||
payload: dict = {"messages": messages}
|
||||
|
||||
if modality in ("text2img", "img2img"):
|
||||
payload["modalities"] = ["image"]
|
||||
else:
|
||||
payload["modalities"] = ["text"]
|
||||
|
||||
for key, val in kwargs.items():
|
||||
if val is not None and val is not False:
|
||||
payload[key] = val
|
||||
|
||||
try:
|
||||
print(f"Sending {modality} request to {server_url}...")
|
||||
response = requests.post(
|
||||
f"{server_url}/v1/chat/completions",
|
||||
headers={"Content-Type": "application/json"},
|
||||
json=payload,
|
||||
timeout=600,
|
||||
)
|
||||
response.raise_for_status()
|
||||
data = response.json()
|
||||
|
||||
# Print think text if present in response metrics
|
||||
metrics = data.get("metrics") or {}
|
||||
think_text = metrics.get("think_text")
|
||||
if think_text:
|
||||
print(f"\n[Think]\n{think_text}\n")
|
||||
|
||||
choices = data.get("choices", [])
|
||||
|
||||
for choice in choices:
|
||||
choice_content = choice.get("message", {}).get("content")
|
||||
|
||||
if isinstance(choice_content, list) and len(choice_content) > 0:
|
||||
first_item = choice_content[0]
|
||||
if isinstance(first_item, dict) and "image_url" in first_item:
|
||||
img_url_str = first_item["image_url"].get("url", "")
|
||||
if img_url_str.startswith("data:image"):
|
||||
_, b64_data = img_url_str.split(",", 1)
|
||||
return base64.b64decode(b64_data)
|
||||
|
||||
for choice in choices:
|
||||
choice_content = choice.get("message", {}).get("content")
|
||||
if isinstance(choice_content, str) and choice_content:
|
||||
return choice_content
|
||||
|
||||
print(f"Unexpected response format: {choices}")
|
||||
return None
|
||||
|
||||
except Exception as e:
|
||||
print(f"Error: {e}")
|
||||
return None
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description="SenseNova-U1 multimodal chat client")
|
||||
parser.add_argument("--prompt", "-p", default="A cute cat", help="Text prompt")
|
||||
parser.add_argument("--output", "-o", default="sensenova_u1_output.png", help="Output file (for image results)")
|
||||
parser.add_argument("--server", "-s", default="http://localhost:8091", help="Server URL")
|
||||
parser.add_argument("--image-url", "-i", type=str, help="Input image URL or local path")
|
||||
parser.add_argument(
|
||||
"--modality",
|
||||
"-m",
|
||||
default="text2img",
|
||||
choices=["text2img", "img2img", "img2text", "text2text"],
|
||||
help="Task modality",
|
||||
)
|
||||
# Standard generation parameters
|
||||
parser.add_argument("--height", type=int, default=2048, help="Image height")
|
||||
parser.add_argument("--width", type=int, default=2048, help="Image width")
|
||||
parser.add_argument("--num-steps", type=int, default=50, help="Inference steps")
|
||||
parser.add_argument("--seed", type=int, default=42, help="Random seed")
|
||||
# Model-specific parameters (forwarded via EXTRA_BODY_PARAMS)
|
||||
parser.add_argument("--cfg-scale", type=float, default=4.0, help="CFG scale")
|
||||
parser.add_argument("--img-cfg-scale", type=float, help="Image CFG scale (img2img)")
|
||||
parser.add_argument("--cfg-norm", type=str, help="CFG normalization mode")
|
||||
parser.add_argument("--timestep-shift", type=float, help="Timestep shift")
|
||||
parser.add_argument("--think", action="store_true", help="Enable think mode")
|
||||
|
||||
args = parser.parse_args()
|
||||
is_image = args.modality in ("text2img", "img2img")
|
||||
|
||||
print(f"Mode: {args.modality}")
|
||||
if args.image_url:
|
||||
print(f"Input Image: {args.image_url}")
|
||||
|
||||
extra: dict[str, object] = {
|
||||
"seed": args.seed,
|
||||
"think": args.think or None,
|
||||
}
|
||||
if is_image:
|
||||
extra.update(
|
||||
height=args.height,
|
||||
width=args.width,
|
||||
num_inference_steps=args.num_steps,
|
||||
cfg_scale=args.cfg_scale,
|
||||
)
|
||||
if args.img_cfg_scale is not None:
|
||||
extra["img_cfg_scale"] = args.img_cfg_scale
|
||||
if args.cfg_norm is not None:
|
||||
extra["cfg_norm"] = args.cfg_norm
|
||||
if args.timestep_shift is not None:
|
||||
extra["timestep_shift"] = args.timestep_shift
|
||||
|
||||
result = generate(
|
||||
prompt=args.prompt,
|
||||
server_url=args.server,
|
||||
image_url=args.image_url,
|
||||
modality=args.modality,
|
||||
**extra,
|
||||
)
|
||||
|
||||
if result:
|
||||
if isinstance(result, bytes):
|
||||
output_path = Path(args.output)
|
||||
output_path.write_bytes(result)
|
||||
print(f"Image saved to: {output_path}")
|
||||
print(f"Size: {len(result) / 1024:.1f} KB")
|
||||
elif isinstance(result, str):
|
||||
print(f"[Response]\n{result}")
|
||||
else:
|
||||
print("Failed to generate response")
|
||||
exit(1)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,12 @@
|
||||
#!/bin/bash
|
||||
# SenseNova-U1 online serving startup script
|
||||
|
||||
MODEL="${MODEL:-SenseNova/SenseNova-U1-8B-MoT}"
|
||||
PORT="${PORT:-8091}"
|
||||
|
||||
echo "Starting SenseNova-U1 server..."
|
||||
echo "Model: $MODEL"
|
||||
echo "Port: $PORT"
|
||||
|
||||
vllm serve "$MODEL" --omni \
|
||||
--port "$PORT"
|
||||
@@ -0,0 +1,187 @@
|
||||
# SenseNova-U1 for unified image generation and understanding
|
||||
|
||||
## Summary
|
||||
|
||||
- Vendor: SenseNova
|
||||
- Model: `SenseNova/SenseNova-U1-8B-MoT`
|
||||
- Task: text2img, img2img, img2text (visual understanding), text2text (chat)
|
||||
- Mode: Offline inference, Online serving (OpenAI-compatible API)
|
||||
- Maintainer: Community
|
||||
|
||||
## When to use this recipe
|
||||
|
||||
Use this recipe to run SenseNova-U1-8B-MoT via vLLM-Omni. SenseNova-U1 is a
|
||||
unified Qwen3-based LLM with Mixture-of-Tokenizers (MoT) attention that handles
|
||||
text encoding, optional chain-of-thought reasoning, flow-matching image
|
||||
denoising, and visual understanding in a single pipeline — no separate text
|
||||
encoder or VAE needed. It supports four task modalities: text-to-image,
|
||||
image-to-image editing (with dual CFG), image-to-text understanding, and
|
||||
text-to-text chat.
|
||||
|
||||
## References
|
||||
|
||||
- Offline example:
|
||||
[`examples/offline_inference/sensenova_u1/end2end.py`](../../examples/offline_inference/sensenova_u1/end2end.py)
|
||||
- Online serving:
|
||||
[`examples/online_serving/sensenova_u1/`](../../examples/online_serving/sensenova_u1/)
|
||||
- E2E tests:
|
||||
[`tests/e2e/offline_inference/test_sensenova_u1_text2img.py`](../../tests/e2e/offline_inference/test_sensenova_u1_text2img.py),
|
||||
[`tests/e2e/offline_inference/test_sensenova_u1_img2img.py`](../../tests/e2e/offline_inference/test_sensenova_u1_img2img.py)
|
||||
- HuggingFace model page:
|
||||
[SenseNova/SenseNova-U1-8B-MoT](https://huggingface.co/SenseNova/SenseNova-U1-8B-MoT)
|
||||
|
||||
## Hardware Support
|
||||
|
||||
## GPU
|
||||
|
||||
### 1x H200 (144GB)
|
||||
|
||||
#### Environment
|
||||
|
||||
- OS: Linux
|
||||
- Python: 3.12
|
||||
- Driver / runtime: NVIDIA 590.48.01, CUDA 13.1
|
||||
- vLLM-Omni version: 0.18.1.dev
|
||||
|
||||
#### Command
|
||||
|
||||
```bash
|
||||
python examples/offline_inference/sensenova_u1/end2end.py \
|
||||
--prompt "Close portrait of an elderly woman by a farmhouse window, textured skin, gentle smile, warm natural light, emotional documentary look. The portrait should feel polished and natural, with sharp eyes, realistic skin texture, accurate facial anatomy, and premium lighting that keeps the face as the main focus." \
|
||||
--width 1536 --height 2720 \
|
||||
--seed 42 --num-steps 50 \
|
||||
--cfg-scale 4.0 --timestep-shift 3.0 --cfg-norm none \
|
||||
--think --print-think \
|
||||
--output outputs
|
||||
```
|
||||
|
||||
#### Verification
|
||||
|
||||
```bash
|
||||
pytest -s -v tests/e2e/offline_inference/test_sensenova_u1_text2img.py \
|
||||
-m "advanced_model" --run-level "advanced_model"
|
||||
```
|
||||
|
||||
#### Notes
|
||||
|
||||
- E2E latency: **32.1s** (1536×2720, 50 steps, think mode, CFG scale 4.0)
|
||||
- Peak VRAM: **35.9 GB** reserved, 35.1 GB allocated
|
||||
- Model loading: 32.8 GiB, 8.7s
|
||||
- No deploy YAML needed — the engine auto-generates a single-stage diffusion config.
|
||||
- Think mode (`--think`) is recommended for higher image quality.
|
||||
|
||||
#### Image-to-Image Editing (img2img)
|
||||
|
||||
```bash
|
||||
python examples/offline_inference/sensenova_u1/end2end.py \
|
||||
--prompt "Turn this into an oil painting" \
|
||||
--image input.png \
|
||||
--width 2048 --height 2048 \
|
||||
--seed 42 --num-steps 50 \
|
||||
--cfg-scale 4.0 --img-cfg-scale 1.0 --cfg-norm none \
|
||||
--think --print-think \
|
||||
--output outputs
|
||||
```
|
||||
|
||||
- img2img uses dual CFG: `--cfg-scale` controls text guidance, `--img-cfg-scale`
|
||||
controls image guidance (1.0 = image CFG disabled).
|
||||
- Pass multiple `--image` paths for multi-reference editing.
|
||||
|
||||
#### Image Understanding (img2text)
|
||||
|
||||
```bash
|
||||
python examples/offline_inference/sensenova_u1/end2end.py \
|
||||
--modality img2text \
|
||||
--prompt "Describe this image in detail" \
|
||||
--image photo.jpg \
|
||||
--max-tokens 512
|
||||
```
|
||||
|
||||
#### Text-to-Text Chat (text2text)
|
||||
|
||||
```bash
|
||||
python examples/offline_inference/sensenova_u1/end2end.py \
|
||||
--modality text2text \
|
||||
--prompt "Explain the theory of relativity in simple terms" \
|
||||
--max-tokens 256
|
||||
```
|
||||
|
||||
- For img2text and text2text, image generation parameters (height, width,
|
||||
num-steps, cfg-scale) are ignored.
|
||||
- Use `--do-sample --temperature 0.7` for more diverse text responses.
|
||||
|
||||
### 2x H200 (144GB) — TP=2
|
||||
|
||||
#### Environment
|
||||
|
||||
- OS: Linux
|
||||
- Python: 3.12
|
||||
- Driver / runtime: NVIDIA 590.48.01, CUDA 13.1
|
||||
- vLLM-Omni version: 0.18.1.dev
|
||||
|
||||
#### Command
|
||||
|
||||
```bash
|
||||
python examples/offline_inference/sensenova_u1/end2end.py \
|
||||
--prompt "Close portrait of an elderly woman by a farmhouse window, textured skin, gentle smile, warm natural light, emotional documentary look. The portrait should feel polished and natural, with sharp eyes, realistic skin texture, accurate facial anatomy, and premium lighting that keeps the face as the main focus." \
|
||||
--width 1536 --height 2720 \
|
||||
--seed 42 --num-steps 50 \
|
||||
--cfg-scale 4.0 --timestep-shift 3.0 --cfg-norm none \
|
||||
--think --print-think \
|
||||
--tensor-parallel-size 2 \
|
||||
--output outputs
|
||||
```
|
||||
|
||||
#### Verification
|
||||
|
||||
Verify the output image is generated at `outputs/sensenova_u1_output_0.png`
|
||||
with the expected 1536×2720 resolution.
|
||||
|
||||
#### Notes
|
||||
|
||||
- E2E latency: **28.3s** (1536×2720, 50 steps, think mode, CFG scale 4.0)
|
||||
- Peak VRAM (per GPU): **18.2 GB** reserved, 17.9 GB allocated
|
||||
- Model loading: 16.5 GiB per GPU, 7.0s
|
||||
- TP=2 provides ~12% speedup over TP=1; limited by serial CFG dual-forward
|
||||
and communication overhead.
|
||||
- The LLM transformer uses `QKVParallelLinear` and `MergedColumnParallelLinear`
|
||||
for fused QKV and gate/up projections with TP support.
|
||||
|
||||
## Online Serving
|
||||
|
||||
SenseNova-U1 supports all four modalities via the OpenAI-compatible
|
||||
`/v1/chat/completions` API.
|
||||
|
||||
### Launch
|
||||
|
||||
```bash
|
||||
vllm serve SenseNova/SenseNova-U1-8B-MoT --omni --port 8091
|
||||
```
|
||||
|
||||
### Send Requests
|
||||
|
||||
```bash
|
||||
cd examples/online_serving/sensenova_u1
|
||||
|
||||
# Text-to-image
|
||||
python openai_chat_client.py \
|
||||
--prompt "A beautiful sunset" --modality text2img
|
||||
|
||||
# Image-to-image editing
|
||||
python openai_chat_client.py \
|
||||
--prompt "Turn this into an oil painting" \
|
||||
--modality img2img --image-url input.jpg
|
||||
|
||||
# Image understanding
|
||||
python openai_chat_client.py \
|
||||
--prompt "Describe this image" \
|
||||
--modality img2text --image-url photo.jpg
|
||||
|
||||
# Text chat
|
||||
python openai_chat_client.py \
|
||||
--prompt "What is the capital of France?" \
|
||||
--modality text2text
|
||||
```
|
||||
|
||||
For full API documentation and curl examples, see
|
||||
[`examples/online_serving/sensenova_u1/README.md`](../../examples/online_serving/sensenova_u1/README.md).
|
||||
@@ -0,0 +1,169 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||
|
||||
"""
|
||||
End-to-end test for SenseNova-U1 img2img generation.
|
||||
|
||||
This test validates that the SenseNova-U1 model generates edited images from
|
||||
an input image and text prompt that match expected reference pixel values
|
||||
within a ±10 tolerance.
|
||||
|
||||
Equivalent to running:
|
||||
python examples/offline_inference/sensenova_u1/end2end.py \
|
||||
--prompt "Turn this into an oil painting" \
|
||||
--image 2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg \
|
||||
--width 2560 --height 1669 \
|
||||
--seed 42 --num-steps 50 \
|
||||
--cfg-scale 4.0 --img-cfg-scale 1.0 --cfg-norm none \
|
||||
--think --print-think
|
||||
"""
|
||||
|
||||
import os
|
||||
from typing import Any
|
||||
|
||||
os.environ["VLLM_WORKER_MULTIPROC_METHOD"] = "spawn"
|
||||
|
||||
import pytest
|
||||
from PIL import Image
|
||||
from vllm.assets.image import ImageAsset
|
||||
|
||||
from tests.helpers.mark import hardware_test
|
||||
from tests.helpers.runtime import OmniRunner
|
||||
from vllm_omni.entrypoints.omni import Omni
|
||||
from vllm_omni.inputs.data import OmniDiffusionSamplingParams
|
||||
|
||||
# Reference pixel data extracted from the known-good output image generated by:
|
||||
# python examples/offline_inference/sensenova_u1/end2end.py \
|
||||
# --prompt "Turn this into an oil painting" \
|
||||
# --image 2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg \
|
||||
# --width 2560 --height 1669 --seed 42 --num-steps 50 \
|
||||
# --cfg-scale 4.0 --img-cfg-scale 1.0 --cfg-norm none --think
|
||||
REFERENCE_PIXELS = [
|
||||
{"position": (100, 100), "rgb": (95, 131, 174)},
|
||||
{"position": (400, 50), "rgb": (142, 153, 174)},
|
||||
{"position": (700, 100), "rgb": (127, 149, 174)},
|
||||
{"position": (150, 400), "rgb": (99, 141, 183)},
|
||||
{"position": (512, 336), "rgb": (184, 177, 165)},
|
||||
{"position": (700, 400), "rgb": (106, 139, 168)},
|
||||
{"position": (100, 600), "rgb": (170, 172, 155)},
|
||||
{"position": (400, 600), "rgb": (159, 171, 169)},
|
||||
{"position": (700, 600), "rgb": (139, 172, 187)},
|
||||
{"position": (256, 256), "rgb": (136, 158, 180)},
|
||||
]
|
||||
|
||||
PIXEL_TOLERANCE = 10
|
||||
|
||||
DEFAULT_PROMPT = "Turn this into an oil painting"
|
||||
|
||||
EXPECTED_OUTPUT_SIZE = (2560, 1664)
|
||||
|
||||
|
||||
def _load_input_image() -> Image.Image:
|
||||
"""Load the test input image via vllm's ImageAsset."""
|
||||
return ImageAsset("2560px-Gfp-wisconsin-madison-the-nature-boardwalk").pil_image.convert("RGB")
|
||||
|
||||
|
||||
def _build_sampling_params() -> OmniDiffusionSamplingParams:
|
||||
"""Build sampling parameters for SenseNova-U1 img2img generation."""
|
||||
return OmniDiffusionSamplingParams(
|
||||
height=1669,
|
||||
width=2560,
|
||||
seed=42,
|
||||
num_inference_steps=50,
|
||||
extra_args={
|
||||
"cfg_scale": 4.0,
|
||||
"img_cfg_scale": 1.0,
|
||||
"cfg_norm": "none",
|
||||
"timestep_shift": 3.0,
|
||||
"cfg_interval": (0.0, 1.0),
|
||||
"batch_size": 1,
|
||||
"think": True,
|
||||
"t_eps": 0.02,
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def _extract_generated_image(omni_outputs: list) -> Image.Image | None:
|
||||
"""Extract the generated image from Omni outputs (single-stage DiT)."""
|
||||
for req_output in omni_outputs:
|
||||
if images := getattr(req_output, "images", None):
|
||||
return images[0]
|
||||
return None
|
||||
|
||||
|
||||
def _validate_pixels(
|
||||
image: Image.Image,
|
||||
reference_pixels: list[dict[str, Any]] = REFERENCE_PIXELS,
|
||||
tolerance: int = PIXEL_TOLERANCE,
|
||||
) -> None:
|
||||
"""Validate that image pixels match expected reference values.
|
||||
|
||||
Args:
|
||||
image: The PIL Image to validate.
|
||||
reference_pixels: List of dicts with 'position' (x, y) and 'rgb' (R, G, B).
|
||||
tolerance: Maximum allowed difference per color channel.
|
||||
|
||||
Raises:
|
||||
AssertionError: If any pixel differs beyond tolerance.
|
||||
"""
|
||||
for ref in reference_pixels:
|
||||
x, y = ref["position"]
|
||||
expected = ref["rgb"]
|
||||
actual = image.getpixel((x, y))[:3]
|
||||
assert all(abs(a - e) <= tolerance for a, e in zip(actual, expected)), (
|
||||
f"Pixel mismatch at ({x}, {y}): expected {expected}, got {actual}"
|
||||
)
|
||||
|
||||
|
||||
def _generate_sensenova_u1_img2img(
|
||||
omni: Omni,
|
||||
input_image: Image.Image,
|
||||
prompt: str = DEFAULT_PROMPT,
|
||||
) -> Image.Image:
|
||||
"""Generate an edited image using SenseNova-U1 model with img2img pipeline.
|
||||
|
||||
Args:
|
||||
omni: The Omni instance to use for generation.
|
||||
input_image: The input PIL Image for img2img.
|
||||
prompt: The text prompt for image editing.
|
||||
|
||||
Returns:
|
||||
The generated PIL Image.
|
||||
|
||||
Raises:
|
||||
AssertionError: If no image is generated or size is incorrect.
|
||||
"""
|
||||
sampling_params = _build_sampling_params()
|
||||
|
||||
omni_outputs = list(
|
||||
omni.generate(
|
||||
prompts={
|
||||
"prompt": prompt,
|
||||
"multi_modal_data": {"image": input_image},
|
||||
"modalities": ["img2img"],
|
||||
},
|
||||
sampling_params_list=sampling_params,
|
||||
)
|
||||
)
|
||||
|
||||
generated_image = _extract_generated_image(omni_outputs)
|
||||
assert generated_image is not None, "No images generated"
|
||||
assert generated_image.size == EXPECTED_OUTPUT_SIZE, f"Expected {EXPECTED_OUTPUT_SIZE}, got {generated_image.size}"
|
||||
|
||||
return generated_image
|
||||
|
||||
|
||||
@pytest.mark.core_model
|
||||
@pytest.mark.advanced_model
|
||||
@pytest.mark.diffusion
|
||||
@hardware_test(res={"cuda": "H100"})
|
||||
def test_sensenova_u1_img2img(run_level):
|
||||
"""Test SenseNova-U1 img2img (single-stage diffusion, no deploy YAML)."""
|
||||
input_image = _load_input_image()
|
||||
with OmniRunner(
|
||||
"SenseNova/SenseNova-U1-8B-MoT",
|
||||
stage_configs_path=None,
|
||||
) as runner:
|
||||
generated_image = _generate_sensenova_u1_img2img(runner.omni, input_image)
|
||||
if run_level == "advanced_model":
|
||||
_validate_pixels(generated_image)
|
||||
@@ -0,0 +1,158 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||
|
||||
"""
|
||||
End-to-end test for SenseNova-U1 text2img generation.
|
||||
|
||||
This test validates that the SenseNova-U1 model generates images that match
|
||||
expected reference pixel values within a ±10 tolerance.
|
||||
|
||||
Equivalent to running:
|
||||
python SenseNova-U1/examples/t2i/inference.py \
|
||||
--model_path SenseNova/SenseNova-U1-8B-MoT \
|
||||
--prompt "Close portrait of an elderly woman ..." \
|
||||
--width 1536 --height 2720 \
|
||||
--cfg_scale 4.0 --cfg_norm none --timestep_shift 3.0 \
|
||||
--num_steps 50 --seed 42 --think
|
||||
"""
|
||||
|
||||
import os
|
||||
from typing import Any
|
||||
|
||||
os.environ["VLLM_WORKER_MULTIPROC_METHOD"] = "spawn"
|
||||
|
||||
import pytest
|
||||
from PIL import Image
|
||||
|
||||
from tests.helpers.mark import hardware_test
|
||||
from tests.helpers.runtime import OmniRunner
|
||||
from vllm_omni.entrypoints.omni import Omni
|
||||
from vllm_omni.inputs.data import OmniDiffusionSamplingParams
|
||||
|
||||
# Reference pixel data extracted from the known-good output image generated by:
|
||||
# python examples/offline_inference/sensenova_u1/end2end.py \
|
||||
# --prompt "Close portrait of an elderly woman ..." \
|
||||
# --width 1536 --height 2720 --seed 42 --num-steps 50 \
|
||||
# --cfg-scale 4.0 --timestep-shift 3.0 --cfg-norm none --think
|
||||
REFERENCE_PIXELS = [
|
||||
{"position": (100, 100), "rgb": (247, 249, 250)},
|
||||
{"position": (768, 200), "rgb": (176, 135, 97)},
|
||||
{"position": (400, 600), "rgb": (193, 188, 180)},
|
||||
{"position": (1200, 2000), "rgb": (117, 98, 84)},
|
||||
{"position": (750, 500), "rgb": (186, 135, 84)},
|
||||
{"position": (300, 1360), "rgb": (202, 157, 107)},
|
||||
{"position": (1000, 1800), "rgb": (63, 32, 15)},
|
||||
{"position": (500, 2400), "rgb": (135, 116, 104)},
|
||||
{"position": (768, 1360), "rgb": (70, 34, 16)},
|
||||
{"position": (200, 900), "rgb": (208, 201, 191)},
|
||||
]
|
||||
|
||||
PIXEL_TOLERANCE = 10
|
||||
|
||||
DEFAULT_PROMPT = (
|
||||
"Close portrait of an elderly woman by a farmhouse window, textured skin, "
|
||||
"gentle smile, warm natural light, emotional documentary look. The portrait "
|
||||
"should feel polished and natural, with sharp eyes, realistic skin texture, "
|
||||
"accurate facial anatomy, and premium lighting that keeps the face as the "
|
||||
"main focus."
|
||||
)
|
||||
|
||||
EXPECTED_OUTPUT_SIZE = (1536, 2720)
|
||||
|
||||
|
||||
def _build_sampling_params() -> OmniDiffusionSamplingParams:
|
||||
"""Build sampling parameters for SenseNova-U1 text2img generation."""
|
||||
return OmniDiffusionSamplingParams(
|
||||
height=EXPECTED_OUTPUT_SIZE[1],
|
||||
width=EXPECTED_OUTPUT_SIZE[0],
|
||||
seed=42,
|
||||
num_inference_steps=50,
|
||||
extra_args={
|
||||
"cfg_scale": 4.0,
|
||||
"cfg_norm": "none",
|
||||
"timestep_shift": 3.0,
|
||||
"cfg_interval": (0.0, 1.0),
|
||||
"batch_size": 1,
|
||||
"think": True,
|
||||
"t_eps": 0.02,
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def _extract_generated_image(omni_outputs: list) -> Image.Image | None:
|
||||
"""Extract the generated image from Omni outputs (single-stage DiT)."""
|
||||
for req_output in omni_outputs:
|
||||
if images := getattr(req_output, "images", None):
|
||||
return images[0]
|
||||
return None
|
||||
|
||||
|
||||
def _validate_pixels(
|
||||
image: Image.Image,
|
||||
reference_pixels: list[dict[str, Any]] = REFERENCE_PIXELS,
|
||||
tolerance: int = PIXEL_TOLERANCE,
|
||||
) -> None:
|
||||
"""Validate that image pixels match expected reference values.
|
||||
|
||||
Args:
|
||||
image: The PIL Image to validate.
|
||||
reference_pixels: List of dicts with 'position' (x, y) and 'rgb' (R, G, B).
|
||||
tolerance: Maximum allowed difference per color channel.
|
||||
|
||||
Raises:
|
||||
AssertionError: If any pixel differs beyond tolerance.
|
||||
"""
|
||||
for ref in reference_pixels:
|
||||
x, y = ref["position"]
|
||||
expected = ref["rgb"]
|
||||
actual = image.getpixel((x, y))[:3]
|
||||
assert all(abs(a - e) <= tolerance for a, e in zip(actual, expected)), (
|
||||
f"Pixel mismatch at ({x}, {y}): expected {expected}, got {actual}"
|
||||
)
|
||||
|
||||
|
||||
def _generate_sensenova_u1_image(
|
||||
omni: Omni,
|
||||
prompt: str = DEFAULT_PROMPT,
|
||||
) -> Image.Image:
|
||||
"""Generate an image using SenseNova-U1 model with configured parameters.
|
||||
|
||||
Args:
|
||||
omni: The Omni instance to use for generation.
|
||||
prompt: The text prompt for image generation.
|
||||
|
||||
Returns:
|
||||
The generated PIL Image.
|
||||
|
||||
Raises:
|
||||
AssertionError: If no image is generated or size is incorrect.
|
||||
"""
|
||||
sampling_params = _build_sampling_params()
|
||||
|
||||
omni_outputs = list(
|
||||
omni.generate(
|
||||
prompts={"prompt": prompt, "modalities": ["image"]},
|
||||
sampling_params_list=sampling_params,
|
||||
)
|
||||
)
|
||||
|
||||
generated_image = _extract_generated_image(omni_outputs)
|
||||
assert generated_image is not None, "No images generated"
|
||||
assert generated_image.size == EXPECTED_OUTPUT_SIZE, f"Expected {EXPECTED_OUTPUT_SIZE}, got {generated_image.size}"
|
||||
|
||||
return generated_image
|
||||
|
||||
|
||||
@pytest.mark.core_model
|
||||
@pytest.mark.advanced_model
|
||||
@pytest.mark.diffusion
|
||||
@hardware_test(res={"cuda": "H100"})
|
||||
def test_sensenova_u1_text2img(run_level):
|
||||
"""Test SenseNova-U1 text2img (single-stage diffusion, no deploy YAML)."""
|
||||
with OmniRunner(
|
||||
"SenseNova/SenseNova-U1-8B-MoT",
|
||||
stage_configs_path=None,
|
||||
) as runner:
|
||||
generated_image = _generate_sensenova_u1_image(runner.omni)
|
||||
if run_level == "advanced_model":
|
||||
_validate_pixels(generated_image)
|
||||
@@ -766,6 +766,10 @@ class OmniDiffusionConfig:
|
||||
self.model_class_name = "BagelPipeline"
|
||||
self.set_tf_model_config(TransformerConfig())
|
||||
self.update_multimodal_support()
|
||||
elif model_type == "neo_chat":
|
||||
self.model_class_name = "SenseNovaU1Pipeline"
|
||||
self.tf_model_config = TransformerConfig()
|
||||
self.update_multimodal_support()
|
||||
elif model_type == "nextstep":
|
||||
if self.model_class_name is None:
|
||||
self.model_class_name = "NextStep11Pipeline"
|
||||
|
||||
@@ -69,6 +69,34 @@ def supports_audio_output(model_class_name: str) -> bool:
|
||||
return bool(getattr(model_cls, "support_audio_output", False))
|
||||
|
||||
|
||||
def get_extra_body_params(model_class_name: str) -> frozenset[str]:
|
||||
"""Return the set of extra_body keys accepted by a pipeline.
|
||||
|
||||
Each pipeline can declare ``EXTRA_BODY_PARAMS: ClassVar[frozenset[str]]``
|
||||
to advertise which request-level parameters should be forwarded from
|
||||
``extra_body`` to ``OmniDiffusionSamplingParams.extra_args``.
|
||||
Returns an empty frozenset when the pipeline does not declare any.
|
||||
"""
|
||||
model_cls = DiffusionModelRegistry._try_load_model_cls(model_class_name)
|
||||
if model_cls is None:
|
||||
return frozenset()
|
||||
return frozenset(getattr(model_cls, "EXTRA_BODY_PARAMS", frozenset()))
|
||||
|
||||
|
||||
def get_extra_output_params(model_class_name: str) -> frozenset[str]:
|
||||
"""Return the set of custom_output keys to expose in API response metrics.
|
||||
|
||||
Each pipeline can declare ``EXTRA_OUTPUT_PARAMS: ClassVar[frozenset[str]]``
|
||||
to advertise which ``DiffusionOutput.custom_output`` keys should be
|
||||
copied into the response ``metrics`` dict.
|
||||
Returns an empty frozenset when the pipeline does not declare any.
|
||||
"""
|
||||
model_cls = DiffusionModelRegistry._try_load_model_cls(model_class_name)
|
||||
if model_cls is None:
|
||||
return frozenset()
|
||||
return frozenset(getattr(model_cls, "EXTRA_OUTPUT_PARAMS", frozenset()))
|
||||
|
||||
|
||||
class DiffusionEngine:
|
||||
"""The diffusion engine for vLLM-Omni diffusion models."""
|
||||
|
||||
@@ -230,6 +258,11 @@ class DiffusionEngine:
|
||||
"postprocess_time_ms": postprocess_time * 1000,
|
||||
}
|
||||
|
||||
# Detect text output: when the pipeline returns a string (e.g.,
|
||||
# SenseNova-U1 / BAGEL single-stage img2text / text2text), wrap it
|
||||
# as a text-type response instead of an image.
|
||||
is_text_output = isinstance(output_data, str) and custom_output.get("text_output") is not None
|
||||
|
||||
# Handle single request or multiple requests
|
||||
is_audio_output = supports_audio_output(self.od_config.model_class_name)
|
||||
if is_audio_output and model_audio_sample_rate is None:
|
||||
@@ -247,6 +280,19 @@ class DiffusionEngine:
|
||||
prompt = request.prompts[0]
|
||||
request_id = request.request_ids[0] if request.request_ids else ""
|
||||
|
||||
if is_text_output:
|
||||
return [
|
||||
OmniRequestOutput.from_diffusion(
|
||||
request_id=request_id,
|
||||
images=[],
|
||||
prompt=prompt,
|
||||
metrics=metrics,
|
||||
custom_output=custom_output,
|
||||
final_output_type="text",
|
||||
stage_durations=output.stage_durations,
|
||||
peak_memory_mb=output.peak_memory_mb,
|
||||
),
|
||||
]
|
||||
if is_audio_output:
|
||||
request_audio_payload = outputs[0] if len(outputs) == 1 else outputs
|
||||
return [
|
||||
|
||||
@@ -0,0 +1,9 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||
|
||||
from .pipeline_sensenova_u1 import SenseNovaU1Pipeline, get_sensenova_u1_post_process_func
|
||||
|
||||
__all__ = [
|
||||
"SenseNovaU1Pipeline",
|
||||
"get_sensenova_u1_post_process_func",
|
||||
]
|
||||
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,741 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||
"""Qwen3 LLM with Mixture-of-Tokenizers (MoT) for SenseNova-U1.
|
||||
|
||||
Ported from the sensenova_u1 package with vllm tensor-parallel support:
|
||||
- QKVParallelLinear for fused q/k/v projections (both und and gen paths)
|
||||
- MergedColumnParallelLinear for fused gate+up projections
|
||||
- RowParallelLinear for o_proj and down_proj
|
||||
- VocabParallelEmbedding for token embeddings
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import copy
|
||||
from dataclasses import dataclass
|
||||
|
||||
import torch
|
||||
import torch.nn as nn
|
||||
from transformers.cache_utils import DynamicCache
|
||||
from vllm.model_executor.layers.linear import (
|
||||
MergedColumnParallelLinear,
|
||||
QKVParallelLinear,
|
||||
RowParallelLinear,
|
||||
)
|
||||
from vllm.model_executor.layers.logits_processor import LogitsProcessor
|
||||
from vllm.model_executor.layers.vocab_parallel_embedding import (
|
||||
ParallelLMHead,
|
||||
VocabParallelEmbedding,
|
||||
)
|
||||
|
||||
from vllm_omni.diffusion.attention.backends.abstract import AttentionMetadata
|
||||
from vllm_omni.diffusion.attention.layer import Attention
|
||||
|
||||
|
||||
@dataclass
|
||||
class SenseNovaU1ModelOutput:
|
||||
last_hidden_state: torch.Tensor
|
||||
past_key_values: DynamicCache | None = None
|
||||
|
||||
|
||||
@dataclass
|
||||
class SenseNovaU1CausalLMOutput:
|
||||
logits: torch.Tensor | None = None
|
||||
past_key_values: DynamicCache | None = None
|
||||
hidden_states: torch.Tensor | None = None
|
||||
inputs_embeds: torch.Tensor | None = None
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Mask / Cache utilities
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def create_block_causal_mask(index: torch.Tensor):
|
||||
"""Block-wise causal mask from 1D time-index. Returns (1, 1, L, L)."""
|
||||
L = index.size(0)
|
||||
idx_i = index.unsqueeze(1).expand(L, L)
|
||||
idx_j = index.unsqueeze(0).expand(L, L)
|
||||
arange = torch.arange(L, device=index.device)
|
||||
mask = (idx_j == idx_i) | (arange.unsqueeze(0) <= arange.unsqueeze(1))
|
||||
return torch.where(mask[None, None], 0.0, float("-inf"))
|
||||
|
||||
|
||||
def prepare_flash_kv_cache(past_key_values, current_len: int, batch_size: int):
|
||||
"""Convert prefix cache [B,H,S,D] → flash layout [B,S,H,D] and
|
||||
preallocate buffers for [prefix + current] tokens."""
|
||||
if past_key_values is None:
|
||||
return
|
||||
for layer in past_key_values.layers:
|
||||
past_k, past_v = layer.keys, layer.values
|
||||
if past_k is None or past_v is None:
|
||||
layer.flash_prefix_len = 0
|
||||
layer.flash_total_len = current_len
|
||||
layer.flash_k_cache = None
|
||||
layer.flash_v_cache = None
|
||||
continue
|
||||
past_k_flash = past_k.transpose(1, 2).contiguous()
|
||||
past_v_flash = past_v.transpose(1, 2).contiguous()
|
||||
prefix_len = past_k_flash.shape[1]
|
||||
total_len = prefix_len + current_len
|
||||
k_cache = torch.empty(
|
||||
(batch_size, total_len, past_k_flash.shape[2], past_k_flash.shape[3]),
|
||||
device=past_k_flash.device,
|
||||
dtype=past_k_flash.dtype,
|
||||
)
|
||||
v_cache = torch.empty_like(k_cache)
|
||||
k_cache[:, :prefix_len].copy_(past_k_flash)
|
||||
v_cache[:, :prefix_len].copy_(past_v_flash)
|
||||
layer.flash_prefix_len = prefix_len
|
||||
layer.flash_total_len = total_len
|
||||
layer.flash_k_cache = k_cache
|
||||
layer.flash_v_cache = v_cache
|
||||
|
||||
|
||||
def clear_flash_kv_cache(past_key_values):
|
||||
if past_key_values is None:
|
||||
return
|
||||
for layer in past_key_values.layers:
|
||||
for attr in ("flash_prefix_len", "flash_total_len", "flash_k_cache", "flash_v_cache"):
|
||||
if hasattr(layer, attr):
|
||||
delattr(layer, attr)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# RoPE
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def rotate_half(x):
|
||||
x1 = x[..., : x.shape[-1] // 2]
|
||||
x2 = x[..., x.shape[-1] // 2 :]
|
||||
return torch.cat((-x2, x1), dim=-1)
|
||||
|
||||
|
||||
def apply_rotary_pos_emb(q, k, cos, sin, unsqueeze_dim=1):
|
||||
cos = cos.unsqueeze(unsqueeze_dim)
|
||||
sin = sin.unsqueeze(unsqueeze_dim)
|
||||
return (q * cos) + (rotate_half(q) * sin), (k * cos) + (rotate_half(k) * sin)
|
||||
|
||||
|
||||
def _compute_default_rope_parameters(config, device=None, **_kw):
|
||||
base = config.rope_theta
|
||||
partial_rotary_factor = getattr(config, "partial_rotary_factor", 1.0)
|
||||
head_dim = getattr(config, "head_dim", config.hidden_size // config.num_attention_heads)
|
||||
dim = int(head_dim * partial_rotary_factor)
|
||||
inv_freq = 1.0 / (base ** (torch.arange(0, dim, 2, dtype=torch.int64).float().to(device) / dim))
|
||||
return inv_freq, 1.0
|
||||
|
||||
|
||||
class Qwen3RotaryEmbedding(nn.Module):
|
||||
"""Custom RoPE that doubles head_dim then subsamples every-other freq."""
|
||||
|
||||
def __init__(self, config, device=None):
|
||||
super().__init__()
|
||||
self.config = config
|
||||
|
||||
def _rope_init_fn(cfg, dev=None):
|
||||
inv_freq, attn_scaling = _compute_default_rope_parameters(cfg, dev)
|
||||
cfg2 = copy.deepcopy(cfg)
|
||||
head_dim = getattr(cfg2, "head_dim", cfg2.hidden_size // cfg2.num_attention_heads)
|
||||
cfg2.head_dim = int(head_dim) * 2
|
||||
inv_freq_full, _ = _compute_default_rope_parameters(cfg2, dev)
|
||||
return inv_freq_full[::2], attn_scaling
|
||||
|
||||
inv_freq, self.attention_scaling = _rope_init_fn(config, device)
|
||||
self.register_buffer("inv_freq", inv_freq, persistent=False)
|
||||
|
||||
@torch.no_grad()
|
||||
def forward(self, x, position_ids):
|
||||
inv_freq_expanded = self.inv_freq[None, :, None].float().expand(position_ids.shape[0], -1, 1).to(x.device)
|
||||
position_ids_expanded = position_ids[:, None, :].float()
|
||||
device_type = x.device.type if x.device.type != "mps" else "cpu"
|
||||
with torch.autocast(device_type=device_type, enabled=False):
|
||||
freqs = (inv_freq_expanded.float() @ position_ids_expanded.float()).transpose(1, 2)
|
||||
emb = torch.cat((freqs, freqs), dim=-1)
|
||||
cos = emb.cos() * self.attention_scaling
|
||||
sin = emb.sin() * self.attention_scaling
|
||||
return cos.to(dtype=x.dtype), sin.to(dtype=x.dtype)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Norms
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class Qwen3RMSNorm(nn.Module):
|
||||
def __init__(self, hidden_size, eps=1e-6):
|
||||
super().__init__()
|
||||
self.weight = nn.Parameter(torch.ones(hidden_size))
|
||||
self.variance_epsilon = eps
|
||||
|
||||
def forward(self, hidden_states):
|
||||
input_dtype = hidden_states.dtype
|
||||
hidden_states = hidden_states.to(torch.float32)
|
||||
variance = hidden_states.pow(2).mean(-1, keepdim=True)
|
||||
hidden_states = hidden_states * torch.rsqrt(variance + self.variance_epsilon)
|
||||
return self.weight * hidden_states.to(input_dtype)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# MLP
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class SenseNovaU1MLP(nn.Module):
|
||||
def __init__(
|
||||
self, hidden_size: int, intermediate_size: int, hidden_act: str = "silu", quant_config=None, prefix: str = ""
|
||||
):
|
||||
super().__init__()
|
||||
self.gate_up_proj = MergedColumnParallelLinear(
|
||||
hidden_size,
|
||||
[intermediate_size, intermediate_size],
|
||||
bias=False,
|
||||
quant_config=quant_config,
|
||||
prefix=f"{prefix}.gate_up_proj",
|
||||
)
|
||||
self.down_proj = RowParallelLinear(
|
||||
intermediate_size,
|
||||
hidden_size,
|
||||
bias=False,
|
||||
input_is_parallel=True,
|
||||
quant_config=quant_config,
|
||||
prefix=f"{prefix}.down_proj",
|
||||
)
|
||||
from transformers.activations import ACT2FN
|
||||
|
||||
self.act_fn = ACT2FN[hidden_act]
|
||||
|
||||
def forward(self, x):
|
||||
gate_up, _ = self.gate_up_proj(x)
|
||||
gate, up = gate_up.chunk(2, dim=-1)
|
||||
x = self.act_fn(gate) * up
|
||||
x, _ = self.down_proj(x)
|
||||
return x
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Attention with MoT
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class SenseNovaU1Attention(nn.Module):
|
||||
"""Multi-head attention with Mixture-of-Tokenizers (MoT) routing.
|
||||
|
||||
Two parallel sets of Q/K/V/O projections:
|
||||
- qkv_proj / o_proj: understanding (und) path
|
||||
- qkv_proj_mot_gen / o_proj_mot_gen: generation (gen) path
|
||||
|
||||
3D RoPE: time (t), height (h), width (w) with separate frequencies.
|
||||
"""
|
||||
|
||||
def __init__(self, config, layer_idx: int, quant_config=None, prefix: str = ""):
|
||||
super().__init__()
|
||||
self.config = config
|
||||
self.layer_idx = layer_idx
|
||||
self.hidden_size = config.hidden_size
|
||||
self.head_dim = getattr(config, "head_dim", config.hidden_size // config.num_attention_heads)
|
||||
self.total_num_heads = config.num_attention_heads
|
||||
self.total_num_kv_heads = getattr(config, "num_key_value_heads", config.num_attention_heads)
|
||||
self.num_kv_groups = self.total_num_heads // self.total_num_kv_heads
|
||||
self.scaling = self.head_dim**-0.5
|
||||
self.attention_dropout = getattr(config, "attention_dropout", 0.0)
|
||||
bias = getattr(config, "attention_bias", True)
|
||||
|
||||
# Understanding path
|
||||
self.qkv_proj = QKVParallelLinear(
|
||||
self.hidden_size,
|
||||
self.head_dim,
|
||||
self.total_num_heads,
|
||||
self.total_num_kv_heads,
|
||||
bias=bias,
|
||||
quant_config=quant_config,
|
||||
prefix=f"{prefix}.qkv_proj",
|
||||
)
|
||||
self.o_proj = RowParallelLinear(
|
||||
self.total_num_heads * self.head_dim,
|
||||
self.hidden_size,
|
||||
bias=bias,
|
||||
input_is_parallel=True,
|
||||
quant_config=quant_config,
|
||||
prefix=f"{prefix}.o_proj",
|
||||
)
|
||||
|
||||
# Generation (MoT) path
|
||||
self.qkv_proj_mot_gen = QKVParallelLinear(
|
||||
self.hidden_size,
|
||||
self.head_dim,
|
||||
self.total_num_heads,
|
||||
self.total_num_kv_heads,
|
||||
bias=bias,
|
||||
quant_config=quant_config,
|
||||
prefix=f"{prefix}.qkv_proj_mot_gen",
|
||||
)
|
||||
self.o_proj_mot_gen = RowParallelLinear(
|
||||
self.total_num_heads * self.head_dim,
|
||||
self.hidden_size,
|
||||
bias=bias,
|
||||
input_is_parallel=True,
|
||||
quant_config=quant_config,
|
||||
prefix=f"{prefix}.o_proj_mot_gen",
|
||||
)
|
||||
|
||||
# Local head counts (TP-sharded)
|
||||
self.num_heads = self.qkv_proj.num_heads
|
||||
self.num_kv_heads = self.qkv_proj.num_kv_heads
|
||||
self.q_size = self.num_heads * self.head_dim
|
||||
self.kv_size = self.num_kv_heads * self.head_dim
|
||||
|
||||
# QK norms — understanding
|
||||
self.q_norm = Qwen3RMSNorm(self.head_dim // 2, eps=config.rms_norm_eps)
|
||||
self.k_norm = Qwen3RMSNorm(self.head_dim // 2, eps=config.rms_norm_eps)
|
||||
self.q_norm_hw = Qwen3RMSNorm(self.head_dim // 2, eps=config.rms_norm_eps)
|
||||
self.k_norm_hw = Qwen3RMSNorm(self.head_dim // 2, eps=config.rms_norm_eps)
|
||||
|
||||
# QK norms — generation
|
||||
self.q_norm_mot_gen = Qwen3RMSNorm(self.head_dim // 2, eps=config.rms_norm_eps)
|
||||
self.k_norm_mot_gen = Qwen3RMSNorm(self.head_dim // 2, eps=config.rms_norm_eps)
|
||||
self.q_norm_hw_mot_gen = Qwen3RMSNorm(self.head_dim // 2, eps=config.rms_norm_eps)
|
||||
self.k_norm_hw_mot_gen = Qwen3RMSNorm(self.head_dim // 2, eps=config.rms_norm_eps)
|
||||
|
||||
# 3D RoPE: t uses half head_dim, h/w each use quarter head_dim
|
||||
t_config = copy.deepcopy(config)
|
||||
t_config.head_dim = config.head_dim // 2
|
||||
self.rotary_emb = Qwen3RotaryEmbedding(config=t_config)
|
||||
|
||||
hw_config = copy.deepcopy(config)
|
||||
hw_config.head_dim = config.head_dim // 4
|
||||
hw_config.rope_theta = config.rope_theta_hw
|
||||
hw_config.max_position_embeddings = config.max_position_embeddings_hw
|
||||
self.rotary_emb_hw = Qwen3RotaryEmbedding(config=hw_config)
|
||||
|
||||
self.attn = Attention(
|
||||
num_heads=self.num_heads,
|
||||
head_size=self.head_dim,
|
||||
causal=False,
|
||||
softmax_scale=self.scaling,
|
||||
num_kv_heads=self.num_heads,
|
||||
prefix=f"{prefix}.attn",
|
||||
)
|
||||
self.attn.attention = self.attn.sdpa_fallback
|
||||
|
||||
@staticmethod
|
||||
def _align_mask_dtype(mask: torch.Tensor | None, query: torch.Tensor) -> torch.Tensor | None:
|
||||
"""SDPA requires float ``attn_mask`` to match query's dtype."""
|
||||
if mask is None or not mask.is_floating_point() or mask.dtype == query.dtype:
|
||||
return mask
|
||||
return mask.to(query.dtype)
|
||||
|
||||
def _run_attn(
|
||||
self,
|
||||
query_bhsd: torch.Tensor,
|
||||
key_bhsd: torch.Tensor,
|
||||
value_bhsd: torch.Tensor,
|
||||
attention_mask: torch.Tensor | None,
|
||||
) -> torch.Tensor:
|
||||
"""Run unified attention with [B, H, S, D] inputs. Returns [B, S, H, D]."""
|
||||
if self.num_kv_groups > 1:
|
||||
n = self.num_kv_groups
|
||||
key_bhsd = key_bhsd.repeat_interleave(n, dim=1)
|
||||
value_bhsd = value_bhsd.repeat_interleave(n, dim=1)
|
||||
q = query_bhsd.transpose(1, 2).contiguous()
|
||||
k = key_bhsd.transpose(1, 2).contiguous()
|
||||
v = value_bhsd.transpose(1, 2).contiguous()
|
||||
attention_mask = self._align_mask_dtype(attention_mask, q)
|
||||
attn_metadata = AttentionMetadata(attn_mask=attention_mask) if attention_mask is not None else None
|
||||
return self.attn(q, k, v, attn_metadata)
|
||||
|
||||
def _run_attn_bshd(
|
||||
self,
|
||||
query_bshd: torch.Tensor,
|
||||
key_bshd: torch.Tensor,
|
||||
value_bshd: torch.Tensor,
|
||||
attention_mask: torch.Tensor | None,
|
||||
) -> torch.Tensor:
|
||||
"""Run unified attention with [B, S, H, D] inputs. Returns [B, S, H, D]."""
|
||||
if self.num_kv_groups > 1:
|
||||
n = self.num_kv_groups
|
||||
key_bshd = key_bshd.repeat_interleave(n, dim=2)
|
||||
value_bshd = value_bshd.repeat_interleave(n, dim=2)
|
||||
attention_mask = self._align_mask_dtype(attention_mask, query_bshd)
|
||||
attn_metadata = AttentionMetadata(attn_mask=attention_mask) if attention_mask is not None else None
|
||||
return self.attn(query_bshd, key_bshd, value_bshd, attn_metadata)
|
||||
|
||||
def _project_and_rope(self, hidden_states, indexes, qkv_proj, q_norm, k_norm, q_norm_hw, k_norm_hw):
|
||||
"""Project Q/K/V via the given QKVParallelLinear and apply 3D RoPE."""
|
||||
input_shape = hidden_states.shape[:-1] # (B, S)
|
||||
qkv, _ = qkv_proj(hidden_states)
|
||||
q, k, v = qkv.split([self.q_size, self.kv_size, self.kv_size], dim=-1)
|
||||
|
||||
q = q.view(*input_shape, self.num_heads, self.head_dim)
|
||||
k = k.view(*input_shape, self.num_kv_heads, self.head_dim)
|
||||
v = v.view(*input_shape, self.num_kv_heads, self.head_dim)
|
||||
|
||||
# Split head_dim into t and hw halves
|
||||
q_t, q_hw = q.chunk(2, dim=-1)
|
||||
k_t, k_hw = k.chunk(2, dim=-1)
|
||||
|
||||
# Norms then transpose to [B, H, S, D/2]
|
||||
q_t = q_norm(q_t).transpose(1, 2)
|
||||
k_t = k_norm(k_t).transpose(1, 2)
|
||||
q_hw = q_norm_hw(q_hw).transpose(1, 2)
|
||||
k_hw = k_norm_hw(k_hw).transpose(1, 2)
|
||||
|
||||
# Split hw into h and w quarters
|
||||
q_h, q_w = q_hw.chunk(2, dim=-1)
|
||||
k_h, k_w = k_hw.chunk(2, dim=-1)
|
||||
|
||||
# RoPE for each dimension
|
||||
cos_t, sin_t = self.rotary_emb(hidden_states, indexes[0].unsqueeze(0))
|
||||
q_t, k_t = apply_rotary_pos_emb(q_t, k_t, cos_t, sin_t)
|
||||
cos_h, sin_h = self.rotary_emb_hw(hidden_states, indexes[1].unsqueeze(0))
|
||||
q_h, k_h = apply_rotary_pos_emb(q_h, k_h, cos_h, sin_h)
|
||||
cos_w, sin_w = self.rotary_emb_hw(hidden_states, indexes[2].unsqueeze(0))
|
||||
q_w, k_w = apply_rotary_pos_emb(q_w, k_w, cos_w, sin_w)
|
||||
|
||||
# Reassemble: [B, H, S, head_dim]
|
||||
query_states = torch.cat([q_t, q_h, q_w], dim=-1)
|
||||
key_states = torch.cat([k_t, k_h, k_w], dim=-1)
|
||||
value_states = v.transpose(1, 2) # [B, H, S, D]
|
||||
return query_states, key_states, value_states
|
||||
|
||||
def forward_und(self, hidden_states, indexes, attention_mask, past_key_values=None, **kwargs):
|
||||
"""Understanding path — unified Attention with explicit 4D mask."""
|
||||
input_shape = hidden_states.shape[:-1]
|
||||
query_states, key_states, value_states = self._project_and_rope(
|
||||
hidden_states,
|
||||
indexes,
|
||||
self.qkv_proj,
|
||||
self.q_norm,
|
||||
self.k_norm,
|
||||
self.q_norm_hw,
|
||||
self.k_norm_hw,
|
||||
)
|
||||
update_cache = kwargs.get("update_cache", True)
|
||||
if past_key_values is not None:
|
||||
if update_cache:
|
||||
key_states, value_states = past_key_values.update(key_states, value_states, self.layer_idx)
|
||||
else:
|
||||
layer = past_key_values.layers[self.layer_idx]
|
||||
if layer.keys is not None:
|
||||
key_states = torch.cat([layer.keys, key_states], dim=2)
|
||||
value_states = torch.cat([layer.values, value_states], dim=2)
|
||||
|
||||
attn_output = self._run_attn(query_states, key_states, value_states, attention_mask)
|
||||
attn_output = attn_output.reshape(*input_shape, -1).contiguous()
|
||||
attn_output, _ = self.o_proj(attn_output)
|
||||
return attn_output
|
||||
|
||||
def forward_gen(self, hidden_states, indexes, attention_mask, past_key_values=None, **kwargs):
|
||||
"""Generation path — unified Attention, bidirectional with optional KV cache."""
|
||||
input_shape = hidden_states.shape[:-1]
|
||||
query_states, key_states, value_states = self._project_and_rope(
|
||||
hidden_states,
|
||||
indexes,
|
||||
self.qkv_proj_mot_gen,
|
||||
self.q_norm_mot_gen,
|
||||
self.k_norm_mot_gen,
|
||||
self.q_norm_hw_mot_gen,
|
||||
self.k_norm_hw_mot_gen,
|
||||
)
|
||||
update_cache = kwargs.get("update_cache", True)
|
||||
|
||||
if attention_mask is None:
|
||||
# Bidirectional path: no causal mask, optionally attend to a prefix.
|
||||
q = query_states.transpose(1, 2).contiguous() # [B,S,H,D]
|
||||
k_cur = key_states.transpose(1, 2).contiguous()
|
||||
v_cur = value_states.transpose(1, 2).contiguous()
|
||||
|
||||
if past_key_values is not None:
|
||||
if update_cache:
|
||||
key_states, value_states = past_key_values.update(key_states, value_states, self.layer_idx)
|
||||
k = key_states.transpose(1, 2).contiguous()
|
||||
v = value_states.transpose(1, 2).contiguous()
|
||||
else:
|
||||
layer = past_key_values.layers[self.layer_idx]
|
||||
if hasattr(layer, "flash_k_cache") and layer.flash_k_cache is not None:
|
||||
prefix_len = layer.flash_prefix_len
|
||||
cur_len = k_cur.shape[1]
|
||||
layer.flash_k_cache[:, prefix_len : prefix_len + cur_len].copy_(k_cur)
|
||||
layer.flash_v_cache[:, prefix_len : prefix_len + cur_len].copy_(v_cur)
|
||||
k = layer.flash_k_cache[:, : prefix_len + cur_len]
|
||||
v = layer.flash_v_cache[:, : prefix_len + cur_len]
|
||||
else:
|
||||
past_k = past_key_values.layers[self.layer_idx].keys
|
||||
past_v = past_key_values.layers[self.layer_idx].values
|
||||
if past_k is not None:
|
||||
k = torch.cat([past_k.transpose(1, 2).contiguous(), k_cur], dim=1)
|
||||
v = torch.cat([past_v.transpose(1, 2).contiguous(), v_cur], dim=1)
|
||||
else:
|
||||
k, v = k_cur, v_cur
|
||||
else:
|
||||
k, v = k_cur, v_cur
|
||||
|
||||
attn_output = self._run_attn_bshd(q, k, v, None)
|
||||
attn_output = attn_output.reshape(*input_shape, -1).contiguous()
|
||||
attn_output, _ = self.o_proj_mot_gen(attn_output)
|
||||
return attn_output
|
||||
|
||||
# Masked fallback with explicit 4D additive mask
|
||||
if past_key_values is not None:
|
||||
if update_cache:
|
||||
key_states, value_states = past_key_values.update(key_states, value_states, self.layer_idx)
|
||||
else:
|
||||
layer = past_key_values.layers[self.layer_idx]
|
||||
if layer.keys is not None:
|
||||
key_states = torch.cat([layer.keys, key_states], dim=2)
|
||||
value_states = torch.cat([layer.values, value_states], dim=2)
|
||||
|
||||
attn_output = self._run_attn(query_states, key_states, value_states, attention_mask)
|
||||
attn_output = attn_output.reshape(*input_shape, -1).contiguous()
|
||||
attn_output, _ = self.o_proj_mot_gen(attn_output)
|
||||
return attn_output
|
||||
|
||||
def forward(
|
||||
self,
|
||||
hidden_states,
|
||||
image_gen_indicators,
|
||||
exist_und,
|
||||
exist_gen,
|
||||
indexes,
|
||||
attention_mask,
|
||||
past_key_values=None,
|
||||
**kwargs,
|
||||
):
|
||||
if exist_und and not exist_gen:
|
||||
return self.forward_und(hidden_states, indexes, attention_mask, past_key_values, **kwargs)
|
||||
if not exist_und and exist_gen:
|
||||
return self.forward_gen(hidden_states, indexes, attention_mask, past_key_values, **kwargs)
|
||||
raise NotImplementedError("Mixed und+gen tokens in a single forward not implemented for initial port")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Decoder Layer
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class SenseNovaU1DecoderLayer(nn.Module):
|
||||
def __init__(self, config, layer_idx: int, quant_config=None, prefix: str = ""):
|
||||
super().__init__()
|
||||
self.self_attn = SenseNovaU1Attention(
|
||||
config, layer_idx, quant_config=quant_config, prefix=f"{prefix}.self_attn"
|
||||
)
|
||||
self.mlp = SenseNovaU1MLP(
|
||||
config.hidden_size,
|
||||
config.intermediate_size,
|
||||
config.hidden_act,
|
||||
quant_config=quant_config,
|
||||
prefix=f"{prefix}.mlp",
|
||||
)
|
||||
self.mlp_mot_gen = SenseNovaU1MLP(
|
||||
config.hidden_size,
|
||||
config.intermediate_size,
|
||||
config.hidden_act,
|
||||
quant_config=quant_config,
|
||||
prefix=f"{prefix}.mlp_mot_gen",
|
||||
)
|
||||
self.input_layernorm = Qwen3RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
|
||||
self.input_layernorm_mot_gen = Qwen3RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
|
||||
self.post_attention_layernorm = Qwen3RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
|
||||
self.post_attention_layernorm_mot_gen = Qwen3RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
|
||||
self.attention_type = config.layer_types[layer_idx]
|
||||
|
||||
def _forward_und(self, hidden_states, indexes, attention_mask, past_key_values, **kwargs):
|
||||
residual = hidden_states
|
||||
hidden_states = self.input_layernorm(hidden_states)
|
||||
hidden_states = self.self_attn(
|
||||
hidden_states,
|
||||
image_gen_indicators=None,
|
||||
exist_und=True,
|
||||
exist_gen=False,
|
||||
indexes=indexes,
|
||||
attention_mask=attention_mask,
|
||||
past_key_values=past_key_values,
|
||||
**kwargs,
|
||||
)
|
||||
hidden_states = residual + hidden_states
|
||||
residual = hidden_states
|
||||
hidden_states = self.mlp(self.post_attention_layernorm(hidden_states))
|
||||
return residual + hidden_states
|
||||
|
||||
def _forward_gen(self, hidden_states, indexes, attention_mask, past_key_values, **kwargs):
|
||||
residual = hidden_states
|
||||
hidden_states = self.input_layernorm_mot_gen(hidden_states)
|
||||
hidden_states = self.self_attn(
|
||||
hidden_states,
|
||||
image_gen_indicators=None,
|
||||
exist_und=False,
|
||||
exist_gen=True,
|
||||
indexes=indexes,
|
||||
attention_mask=attention_mask,
|
||||
past_key_values=past_key_values,
|
||||
**kwargs,
|
||||
)
|
||||
hidden_states = residual + hidden_states
|
||||
residual = hidden_states
|
||||
hidden_states = self.mlp_mot_gen(self.post_attention_layernorm_mot_gen(hidden_states))
|
||||
return residual + hidden_states
|
||||
|
||||
def forward(
|
||||
self,
|
||||
hidden_states,
|
||||
image_gen_indicators,
|
||||
exist_und,
|
||||
exist_gen,
|
||||
indexes,
|
||||
attention_mask,
|
||||
past_key_values=None,
|
||||
**kwargs,
|
||||
):
|
||||
if exist_und and not exist_gen:
|
||||
return self._forward_und(hidden_states, indexes, attention_mask, past_key_values, **kwargs)
|
||||
if not exist_und and exist_gen:
|
||||
return self._forward_gen(hidden_states, indexes, attention_mask, past_key_values, **kwargs)
|
||||
raise NotImplementedError("Mixed und+gen tokens in a single forward not implemented for initial port")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Model (decoder stack)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class SenseNovaU1Model(nn.Module):
|
||||
def __init__(self, config, quant_config=None, prefix: str = ""):
|
||||
super().__init__()
|
||||
self.config = config
|
||||
self.embed_tokens = VocabParallelEmbedding(
|
||||
config.vocab_size,
|
||||
config.hidden_size,
|
||||
prefix=f"{prefix}.embed_tokens",
|
||||
)
|
||||
self.layers = nn.ModuleList(
|
||||
[
|
||||
SenseNovaU1DecoderLayer(config, i, quant_config=quant_config, prefix=f"{prefix}.layers.{i}")
|
||||
for i in range(config.num_hidden_layers)
|
||||
]
|
||||
)
|
||||
self.norm = Qwen3RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
|
||||
self.norm_mot_gen = Qwen3RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
|
||||
|
||||
def forward(
|
||||
self,
|
||||
input_ids=None,
|
||||
image_gen_indicators=None,
|
||||
indexes=None,
|
||||
attention_mask=None,
|
||||
past_key_values=None,
|
||||
inputs_embeds=None,
|
||||
use_cache=None,
|
||||
**kwargs,
|
||||
):
|
||||
if image_gen_indicators is None:
|
||||
exist_und, exist_gen = True, False
|
||||
else:
|
||||
exist_und = (~image_gen_indicators).any().item()
|
||||
exist_gen = image_gen_indicators.any().item()
|
||||
|
||||
if inputs_embeds is None:
|
||||
inputs_embeds = self.embed_tokens(input_ids)
|
||||
|
||||
if use_cache and past_key_values is None:
|
||||
past_key_values = DynamicCache()
|
||||
|
||||
# Resolve attention mask. Callers must always provide `indexes`; this
|
||||
# keeps the model stateless and safe under concurrent requests.
|
||||
assert indexes is not None, "SenseNovaU1Model.forward requires explicit `indexes`."
|
||||
if not isinstance(attention_mask, dict):
|
||||
past_len = past_key_values.get_seq_length() if past_key_values else 0
|
||||
seq_len = inputs_embeds.shape[1]
|
||||
total_len = past_len + seq_len
|
||||
mask = torch.zeros(1, 1, seq_len, total_len, device=inputs_embeds.device)
|
||||
if seq_len > 1:
|
||||
causal = torch.tril(torch.ones(seq_len, seq_len, device=inputs_embeds.device))
|
||||
mask[:, :, :, past_len:] = torch.where(causal == 1, 0.0, float("-inf"))
|
||||
causal_mask_mapping = {"full_attention": mask}
|
||||
else:
|
||||
causal_mask_mapping = attention_mask
|
||||
|
||||
hidden_states = inputs_embeds
|
||||
for layer in self.layers:
|
||||
hidden_states = layer(
|
||||
hidden_states,
|
||||
image_gen_indicators=image_gen_indicators,
|
||||
exist_und=exist_und,
|
||||
exist_gen=exist_gen,
|
||||
indexes=indexes,
|
||||
attention_mask=causal_mask_mapping.get(layer.attention_type, causal_mask_mapping.get("full_attention")),
|
||||
past_key_values=past_key_values,
|
||||
use_cache=use_cache,
|
||||
**kwargs,
|
||||
)
|
||||
|
||||
if not exist_gen:
|
||||
hidden_states = self.norm(hidden_states)
|
||||
else:
|
||||
hidden_states = self.norm_mot_gen(hidden_states)
|
||||
|
||||
return SenseNovaU1ModelOutput(
|
||||
last_hidden_state=hidden_states,
|
||||
past_key_values=past_key_values if use_cache else None,
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# ForCausalLM wrapper
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class SenseNovaU1ForCausalLM(nn.Module):
|
||||
def __init__(self, config, quant_config=None, prefix: str = ""):
|
||||
super().__init__()
|
||||
self.config = config
|
||||
self.model = SenseNovaU1Model(config, quant_config=quant_config, prefix=f"{prefix}.model")
|
||||
self.vocab_size = config.vocab_size
|
||||
self.lm_head = ParallelLMHead(config.vocab_size, config.hidden_size, prefix=f"{prefix}.lm_head")
|
||||
# LogitsProcessor handles the TP all-gather of vocab-sharded ParallelLMHead
|
||||
# outputs so callers see full-vocab logits regardless of tp_size.
|
||||
self.logits_processor = LogitsProcessor(config.vocab_size)
|
||||
|
||||
def forward(
|
||||
self,
|
||||
input_ids=None,
|
||||
indexes=None,
|
||||
attention_mask=None,
|
||||
past_key_values=None,
|
||||
inputs_embeds=None,
|
||||
use_cache=None,
|
||||
embed_only: bool = False,
|
||||
compute_logits: bool = True,
|
||||
**kwargs,
|
||||
):
|
||||
# Routing every access through ``forward`` keeps any externally-attached
|
||||
# hooks (e.g. CPU-offload swap) firing for sub-module accesses such as
|
||||
# token embedding lookup or hidden-state-only model runs. Callers should
|
||||
# prefer this entry point over reaching into ``self.model`` /
|
||||
# ``self.lm_head`` directly so that offloaded weights get materialised
|
||||
# on the right device first.
|
||||
if embed_only:
|
||||
if input_ids is None:
|
||||
raise ValueError("embed_only=True requires input_ids")
|
||||
return SenseNovaU1CausalLMOutput(
|
||||
inputs_embeds=self.model.embed_tokens(input_ids),
|
||||
)
|
||||
|
||||
outputs = self.model(
|
||||
input_ids=input_ids,
|
||||
indexes=indexes,
|
||||
attention_mask=attention_mask,
|
||||
past_key_values=past_key_values,
|
||||
inputs_embeds=inputs_embeds,
|
||||
use_cache=use_cache,
|
||||
**kwargs,
|
||||
)
|
||||
logits = self.logits_processor(self.lm_head, outputs.last_hidden_state) if compute_logits else None
|
||||
return SenseNovaU1CausalLMOutput(
|
||||
logits=logits,
|
||||
past_key_values=outputs.past_key_values,
|
||||
hidden_states=outputs.last_hidden_state,
|
||||
)
|
||||
|
||||
def get_input_embeddings(self):
|
||||
return self.model.embed_tokens
|
||||
@@ -222,6 +222,13 @@ class ModelLevelOffloadBackend(OffloadBackend):
|
||||
except Exception as exc:
|
||||
logger.debug("Failed to move VAE to GPU: %s", exc)
|
||||
|
||||
# Pin resident modules on GPU (small hot submodules called inside the DiT loop).
|
||||
for res, name in zip(modules.resident_modules, modules.resident_names):
|
||||
try:
|
||||
res.to(self.device)
|
||||
except Exception as exc:
|
||||
logger.warning("Failed to move resident module '%s' to GPU: %s", name, exc)
|
||||
|
||||
# Apply sequential offloading hooks
|
||||
apply_sequential_offload(
|
||||
dit_modules=modules.dits,
|
||||
@@ -237,9 +244,10 @@ class ModelLevelOffloadBackend(OffloadBackend):
|
||||
self.enabled = True
|
||||
|
||||
logger.info(
|
||||
"Model-level offloading enabled: %s <-> %s (mutual exclusion)",
|
||||
"Model-level offloading enabled: %s <-> %s (mutual exclusion)%s",
|
||||
", ".join(modules.dit_names),
|
||||
", ".join(modules.encoder_names),
|
||||
f"; resident on GPU: {', '.join(modules.resident_names)}" if modules.resident_names else "",
|
||||
)
|
||||
|
||||
def disable(self) -> None:
|
||||
|
||||
@@ -221,6 +221,11 @@ _DIFFUSION_MODELS = {
|
||||
"pipeline_dreamid_omni",
|
||||
"DreamIDOmniPipeline",
|
||||
),
|
||||
"SenseNovaU1Pipeline": (
|
||||
"sensenova_u1",
|
||||
"pipeline_sensenova_u1",
|
||||
"SenseNovaU1Pipeline",
|
||||
),
|
||||
"AudioXPipeline": (
|
||||
"audiox",
|
||||
"pipeline_audiox",
|
||||
@@ -272,6 +277,7 @@ DiffusionModelRegistry = _ModelRegistry(
|
||||
_NO_CACHE_ACCELERATION = {
|
||||
# Pipelines that do not support cache acceleration (cache_dit / tea_cache).
|
||||
"NextStep11Pipeline",
|
||||
"SenseNovaU1Pipeline",
|
||||
"AudioXPipeline",
|
||||
}
|
||||
|
||||
@@ -475,6 +481,7 @@ _DIFFUSION_POST_PROCESS_FUNCS = {
|
||||
"MagiHumanPipeline": "get_magi_human_post_process_func",
|
||||
"OmniVoicePipeline": "get_omnivoice_post_process_func",
|
||||
"DreamIDOmniPipeline": "get_dreamid_omni_post_process_func",
|
||||
"SenseNovaU1Pipeline": "get_sensenova_u1_post_process_func",
|
||||
}
|
||||
|
||||
_DIFFUSION_PRE_PROCESS_FUNCS = {
|
||||
|
||||
@@ -450,11 +450,12 @@ class OmniBase(PDDisaggregationMixin):
|
||||
except Exception:
|
||||
logger.exception("[%s] Finalize request handling error", self.__class__.__name__)
|
||||
|
||||
images = getattr(engine_outputs, "images", []) if stage_meta["final_output_type"] == "image" else []
|
||||
output_type = getattr(engine_outputs, "final_output_type", stage_meta["final_output_type"])
|
||||
images = getattr(engine_outputs, "images", []) if output_type == "image" else []
|
||||
return OmniRequestOutput(
|
||||
request_id=req_id or "",
|
||||
stage_id=stage_id,
|
||||
final_output_type=stage_meta["final_output_type"],
|
||||
final_output_type=output_type,
|
||||
request_output=engine_outputs,
|
||||
images=images,
|
||||
trajectory_latents=getattr(engine_outputs, "trajectory_latents", None),
|
||||
|
||||
@@ -14,6 +14,7 @@ from fastapi import Request
|
||||
from PIL import Image
|
||||
from pydantic import TypeAdapter
|
||||
|
||||
from vllm_omni.diffusion.diffusion_engine import get_extra_body_params, get_extra_output_params
|
||||
from vllm_omni.entrypoints.async_omni import AsyncOmni
|
||||
from vllm_omni.entrypoints.openai.protocol.chat_completion import OmniChatCompletionResponse
|
||||
from vllm_omni.entrypoints.utils import coerce_param_message_types
|
||||
@@ -95,7 +96,9 @@ from vllm_omni.entrypoints.openai.stage_params import (
|
||||
from vllm_omni.entrypoints.openai.utils import (
|
||||
get_stage_type,
|
||||
get_supported_speakers_from_hf_config,
|
||||
is_single_stage_diffusion,
|
||||
parse_lora_request,
|
||||
resolve_diffusion_od_config,
|
||||
validate_requested_speaker,
|
||||
)
|
||||
from vllm_omni.lora.request import LoRARequest
|
||||
@@ -119,6 +122,8 @@ class OmniOpenAIServingChat(OpenAIServingChat, AudioMixin):
|
||||
_diffusion_engine: AsyncOmni | None = None
|
||||
_diffusion_model_name: str = ""
|
||||
_supported_speakers: set[str] | None = None
|
||||
_diffusion_extra_body_params: frozenset[str] | None = None
|
||||
_diffusion_extra_output_params: frozenset[str] | None = None
|
||||
|
||||
@classmethod
|
||||
def for_diffusion(
|
||||
@@ -145,6 +150,45 @@ class OmniOpenAIServingChat(OpenAIServingChat, AudioMixin):
|
||||
instance._diffusion_model_name = model_name
|
||||
return instance
|
||||
|
||||
def _get_diffusion_extra_body_params(self) -> frozenset[str]:
|
||||
"""Return the pipeline-declared ``EXTRA_BODY_PARAMS`` (cached)."""
|
||||
if self._diffusion_extra_body_params is not None:
|
||||
return self._diffusion_extra_body_params
|
||||
|
||||
params: frozenset[str] = frozenset()
|
||||
try:
|
||||
od_config = resolve_diffusion_od_config(self.engine_client, self._diffusion_engine)
|
||||
if od_config is not None and getattr(od_config, "model_class_name", None):
|
||||
params = get_extra_body_params(od_config.model_class_name)
|
||||
except Exception as e:
|
||||
logger.warning("Failed to read EXTRA_BODY_PARAMS from pipeline: %s", e)
|
||||
|
||||
self._diffusion_extra_body_params = params
|
||||
return params
|
||||
|
||||
def _get_diffusion_extra_output_params(
|
||||
self,
|
||||
custom_output: dict[str, Any] | None,
|
||||
) -> dict[str, Any] | None:
|
||||
"""Pick pipeline-declared ``EXTRA_OUTPUT_PARAMS`` keys from *custom_output*."""
|
||||
if not custom_output:
|
||||
return None
|
||||
|
||||
if self._diffusion_extra_output_params is None:
|
||||
params: frozenset[str] = frozenset()
|
||||
try:
|
||||
od_config = resolve_diffusion_od_config(self.engine_client, self._diffusion_engine)
|
||||
if od_config is not None and getattr(od_config, "model_class_name", None):
|
||||
params = get_extra_output_params(od_config.model_class_name)
|
||||
except Exception as e:
|
||||
logger.warning("Failed to read EXTRA_OUTPUT_PARAMS from pipeline: %s", e)
|
||||
self._diffusion_extra_output_params = params
|
||||
|
||||
if not self._diffusion_extra_output_params:
|
||||
return None
|
||||
out = {k: custom_output[k] for k in self._diffusion_extra_output_params if k in custom_output}
|
||||
return out or None
|
||||
|
||||
def _get_supported_speakers(self) -> set[str]:
|
||||
"""Load supported speakers from model config (cached)."""
|
||||
if self._supported_speakers is not None:
|
||||
@@ -386,6 +430,34 @@ class OmniOpenAIServingChat(OpenAIServingChat, AudioMixin):
|
||||
logger.warning("Failed to build image-generation prompt for omni multistage: %s", e)
|
||||
_image_gen_height = None
|
||||
_image_gen_width = None
|
||||
elif request.modalities and ("text" in request.modalities) and is_single_stage_diffusion(self.engine_client):
|
||||
# Single-stage diffusion text output (img2text / text2text).
|
||||
# Build a diffusion-style prompt with modalities=["text"] so the
|
||||
# pipeline routes to its text generation path.
|
||||
try:
|
||||
extracted_prompt, reference_images = self._extract_diffusion_prompt_and_images_from_messages(
|
||||
request.messages
|
||||
)
|
||||
if not extracted_prompt:
|
||||
return self.create_error_response("No text prompt found in messages")
|
||||
|
||||
tprompt: OmniTextPrompt = {"prompt": extracted_prompt}
|
||||
tprompt["modalities"] = ["text"]
|
||||
|
||||
if reference_images:
|
||||
try:
|
||||
img_bytes = base64.b64decode(reference_images[0])
|
||||
img = Image.open(BytesIO(img_bytes))
|
||||
tprompt["multi_modal_data"] = {"image": img}
|
||||
tprompt["multi_modal_uuids"] = {"image": [f"{request_id}-image-0"]}
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
engine_prompts = [tprompt]
|
||||
except Exception as e:
|
||||
logger.warning("Failed to build text-output prompt for single-stage diffusion: %s", e)
|
||||
_image_gen_height = None
|
||||
_image_gen_width = None
|
||||
else:
|
||||
_image_gen_height = None
|
||||
_image_gen_width = None
|
||||
@@ -418,6 +490,10 @@ class OmniOpenAIServingChat(OpenAIServingChat, AudioMixin):
|
||||
sp.extra_args["cfg_text_scale"] = cfg_text_scale
|
||||
if cfg_img_scale is not None:
|
||||
sp.extra_args["cfg_img_scale"] = cfg_img_scale
|
||||
for _key in self._get_diffusion_extra_body_params():
|
||||
_val = extra_body.get(_key)
|
||||
if _val is not None:
|
||||
sp.extra_args[_key] = _val
|
||||
|
||||
self._log_inputs(
|
||||
request_id,
|
||||
@@ -1610,21 +1686,37 @@ class OmniOpenAIServingChat(OpenAIServingChat, AudioMixin):
|
||||
continue
|
||||
|
||||
if omni_outputs.final_output_type == "text":
|
||||
(
|
||||
choices_data,
|
||||
usage,
|
||||
prompt_logprobs,
|
||||
prompt_token_ids,
|
||||
kv_transfer_params,
|
||||
) = self._create_text_choice(
|
||||
request,
|
||||
omni_outputs,
|
||||
tokenizer,
|
||||
conversation,
|
||||
role,
|
||||
reasoning_parser,
|
||||
)
|
||||
final_res = omni_outputs.request_output
|
||||
if omni_outputs.request_output is not None:
|
||||
(
|
||||
choices_data,
|
||||
usage,
|
||||
prompt_logprobs,
|
||||
prompt_token_ids,
|
||||
kv_transfer_params,
|
||||
) = self._create_text_choice(
|
||||
request,
|
||||
omni_outputs,
|
||||
tokenizer,
|
||||
conversation,
|
||||
role,
|
||||
reasoning_parser,
|
||||
)
|
||||
final_res = omni_outputs.request_output
|
||||
else:
|
||||
# Diffusion pipeline text output (e.g. single-stage
|
||||
# img2text / text2text) — no AR request_output, so build
|
||||
# a simple text choice from custom_output.
|
||||
text_body = (omni_outputs.custom_output or {}).get("text_output", "")
|
||||
message = ChatMessage(role=role, content=text_body)
|
||||
choices_data = [
|
||||
ChatCompletionResponseChoice(
|
||||
index=0,
|
||||
message=message,
|
||||
logprobs=None,
|
||||
finish_reason="stop",
|
||||
stop_reason=None,
|
||||
)
|
||||
]
|
||||
elif omni_outputs.final_output_type == "audio":
|
||||
choices_data = self._create_audio_choice(omni_outputs, role, request, stream=False)
|
||||
elif omni_outputs.final_output_type == "image":
|
||||
@@ -1640,6 +1732,9 @@ class OmniOpenAIServingChat(OpenAIServingChat, AudioMixin):
|
||||
response_metrics = {}
|
||||
response_metrics.setdefault("stage_durations", omni_outputs.stage_durations or {})
|
||||
response_metrics.setdefault("peak_memory_mb", float(omni_outputs.peak_memory_mb or 0.0))
|
||||
extra = self._get_diffusion_extra_output_params(omni_outputs.custom_output)
|
||||
if extra:
|
||||
response_metrics.update(extra)
|
||||
choices.extend(choices_data)
|
||||
|
||||
response = OmniChatCompletionResponse(
|
||||
@@ -2511,6 +2606,10 @@ class OmniOpenAIServingChat(OpenAIServingChat, AudioMixin):
|
||||
gen_params.extra_args["cfg_text_scale"] = cfg_text_scale
|
||||
if cfg_img_scale is not None:
|
||||
gen_params.extra_args["cfg_img_scale"] = cfg_img_scale
|
||||
for _key in self._get_diffusion_extra_body_params():
|
||||
_val = extra_body.get(_key)
|
||||
if _val is not None:
|
||||
gen_params.extra_args[_key] = _val
|
||||
if num_frames is not None:
|
||||
gen_params.num_frames = num_frames
|
||||
if guidance_scale_2 is not None:
|
||||
@@ -2539,6 +2638,13 @@ class OmniOpenAIServingChat(OpenAIServingChat, AudioMixin):
|
||||
except Exception as e: # pragma: no cover - safeguard
|
||||
logger.warning("Failed to parse LoRA request: %s", e)
|
||||
|
||||
# Route text modality for single-stage diffusion (img2text / text2text)
|
||||
requested_modalities = extra_body.get("modalities") or []
|
||||
is_text_request = "text" in requested_modalities
|
||||
|
||||
if is_text_request:
|
||||
gen_prompt["modalities"] = ["text"]
|
||||
|
||||
# Add reference image if provided (from messages content)
|
||||
if pil_images:
|
||||
if len(pil_images) == 1:
|
||||
@@ -2588,6 +2694,38 @@ class OmniOpenAIServingChat(OpenAIServingChat, AudioMixin):
|
||||
result = output
|
||||
if result is None:
|
||||
return self._create_error_response("No output generated from AsyncOmni")
|
||||
|
||||
# Text output path (img2text / text2text)
|
||||
if is_text_request and result.final_output_type == "text":
|
||||
text_body = (result.custom_output or {}).get("text_output", "")
|
||||
message = ChatMessage(role="assistant", content=text_body)
|
||||
choice = ChatCompletionResponseChoice(
|
||||
index=0,
|
||||
message=message,
|
||||
finish_reason="stop",
|
||||
logprobs=None,
|
||||
stop_reason=None,
|
||||
)
|
||||
response = OmniChatCompletionResponse(
|
||||
id=request_id,
|
||||
created=created_time,
|
||||
model=self._diffusion_model_name,
|
||||
choices=[choice],
|
||||
usage=UsageInfo(
|
||||
prompt_tokens=len(prompt.split()),
|
||||
completion_tokens=len(text_body.split()),
|
||||
total_tokens=len(prompt.split()) + len(text_body.split()),
|
||||
),
|
||||
metrics=self._get_diffusion_extra_output_params(result.custom_output),
|
||||
)
|
||||
logger.info(
|
||||
"Diffusion chat completed for request %s: text output (%d chars)",
|
||||
request_id,
|
||||
len(text_body),
|
||||
)
|
||||
return response
|
||||
|
||||
# Image output path (text2img / img2img)
|
||||
final_output_type = getattr(result, "final_output_type", "image")
|
||||
# Handle nested OmniRequestOutput structure where images might be in request_output
|
||||
images = getattr(result.request_output, "images", [])
|
||||
@@ -2702,7 +2840,7 @@ class OmniOpenAIServingChat(OpenAIServingChat, AudioMixin):
|
||||
stop_reason=None,
|
||||
)
|
||||
|
||||
response = ChatCompletionResponse(
|
||||
response = OmniChatCompletionResponse(
|
||||
id=request_id,
|
||||
created=created_time,
|
||||
model=self._diffusion_model_name,
|
||||
@@ -2712,6 +2850,7 @@ class OmniOpenAIServingChat(OpenAIServingChat, AudioMixin):
|
||||
completion_tokens=1,
|
||||
total_tokens=len(prompt.split()) + 1,
|
||||
),
|
||||
metrics=self._get_diffusion_extra_output_params(result.custom_output),
|
||||
)
|
||||
|
||||
logger.info(
|
||||
|
||||
@@ -70,6 +70,27 @@ def get_supported_speakers_from_hf_config(hf_config: Any) -> set[str]:
|
||||
return set()
|
||||
|
||||
|
||||
def resolve_diffusion_od_config(engine_client: Any, diffusion_engine: Any = None) -> Any:
|
||||
"""Resolve the OmniDiffusionConfig from the engine or diffusion engine."""
|
||||
od_config = None
|
||||
if hasattr(engine_client, "get_diffusion_od_config"):
|
||||
od_config = engine_client.get_diffusion_od_config()
|
||||
if od_config is None and diffusion_engine is not None:
|
||||
if hasattr(diffusion_engine, "get_diffusion_od_config"):
|
||||
od_config = diffusion_engine.get_diffusion_od_config()
|
||||
else:
|
||||
od_config = getattr(diffusion_engine, "od_config", None)
|
||||
return od_config
|
||||
|
||||
|
||||
def is_single_stage_diffusion(engine_client: Any) -> bool:
|
||||
"""Return True if the engine is a single-stage diffusion pipeline."""
|
||||
stage_configs = getattr(engine_client, "stage_configs", None) or []
|
||||
if len(stage_configs) != 1:
|
||||
return False
|
||||
return getattr(stage_configs[0], "stage_type", None) in ("diffusion", "DIFFUSION")
|
||||
|
||||
|
||||
def validate_requested_speaker(speaker: str | None, supported_speakers: set[str]) -> str | None:
|
||||
"""Normalize and validate an optional speaker value.
|
||||
|
||||
|
||||
Reference in new issue
Block a user