anodyine/vllm-omni
1
0 Fork 0
Code Issues 4 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

[Refactor]: Remove sensenova examples (#5201)

Browse Source 
Signed-off-by: princepride <wangzhipeng628@gmail.com>
This commit is contained in:
汪志鹏 authored and GitHub committed 2026-07-22 15:36:00 +00:00
1 parent 8219c01b57
commit f44649ffd0
10 files changed
+182 -604
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

examples/offline_inference/image_to_image/image_edit.py
+4
View File
@@ -97,6 +97,7 @@ import torch
from PIL import Image
from vllm_omni.diffusion.data import DiffusionParallelConfig
from vllm_omni.diffusion.utils.image_output import extract_images_from_outputs
from vllm_omni.diffusion.utils.param_utils import apply_declared_extra_args
from vllm_omni.entrypoints.omni import Omni
from vllm_omni.entrypoints.openai.stage_params import clone_sampling_params
@@ -625,6 +626,9 @@ def main():
if images:
break
if not images:
images = extract_images_from_outputs(outputs)
if not images:
raise ValueError("No images found in request_output")
examples/offline_inference/sensenova_u1/README.md
-214
View File
@@ -1,214 +0,0 @@
# SenseNova-U1: Offline inference
## Architecture
SenseNova-U1 is a unified Qwen3-based LLM with Mixture-of-Tokenizers (MoT) attention. Unlike two-stage pipelines (e.g., BAGEL), it handles text encoding, optional reasoning ("think mode"), and flow-matching-based image denoising entirely within a single diffusion stage.
| Feature | Description |
| :------ | :---------- |
| **Model type** | Unified LLM (single-stage diffusion pipeline) |
| **Base LLM** | Qwen3 with MoT attention and 3D RoPE |
| **Image generation** | Flow-matching Euler sampler, no separate VAE |
| **Think mode** | Optional chain-of-thought reasoning before image generation |
| **Parallelism** | Tensor Parallelism (TP) with fused QKV and fused gate/up projections |
| **Modalities** | text2img, img2img, img2text (understanding), text2text |
## Quick Start
```bash
cd examples/offline_inference/sensenova_u1
# Text-to-image
python end2end.py --prompt "A cute cat" --think
# Image-to-image editing
python end2end.py --prompt "Turn this into an oil painting" \
--image input.png --think
# Image understanding (img2text)
python end2end.py --modality img2text \
--prompt "Describe this image in detail" \
--image photo.jpg
# Text-to-text (pure chat)
python end2end.py --modality text2text \
--prompt "What is the capital of France?"
# Custom resolution (image generation)
python end2end.py --prompt "A futuristic cityscape at sunset" \
--width 2048 --height 1024 --think
# Text-to-image with cache acceleration (cache_dit or tea_cache)
python end2end.py --prompt "A cute cat" \
--cache-backend tea_cache
```
> **Note**: Default configuration works on a single **NVIDIA A100 (80GB)** or **H100** GPU.
## Text-to-Image (t2i)
Standard text-to-image with optional think mode:
```bash
python end2end.py \
--prompt "Close portrait of an elderly woman by a farmhouse window, textured skin, gentle smile, warm natural light" \
--width 1536 --height 2720 \
--think --print-think
```
## Image-to-Image Editing (img2img)
Pass one or more `--image` paths to trigger img2img mode. The model uses the
input image(s) as visual context for editing:
```bash
# Single input image
python end2end.py \
--prompt "Add a sunset sky in the background" \
--image photo.jpg \
--width 2048 --height 2048 \
--think
# Multiple reference images
python end2end.py \
--prompt "Combine the style of Image-1 with the content of Image-2" \
--image style_ref.png content_ref.png \
--width 2048 --height 2048 \
--think
```
### Dual CFG for img2img
img2img supports dual classifier-free guidance (text CFG + image CFG):
```bash
python end2end.py \
--prompt "Make the person smile" \
--image portrait.jpg \
--cfg-scale 4.0 \
--img-cfg-scale 2.0 \
--think
```
| `img_cfg_scale` | Behavior |
| :-------------- | :------- |
| `1.0` (default) | Text-only CFG: guidance = image-condition → full-condition |
| `== cfg_scale` | Standard CFG: guidance = unconditional → full-condition |
| Other value | Dual CFG: separate text and image guidance strengths |
## Full Parameter Reference
### General Parameters
| Parameter | Default | Description |
| :-------- | :------ | :---------- |
| `--modality` | `auto` | Task modality: `auto`, `text2img`, `img2img`, `img2text`, `text2text` |
| `--prompt` | "A cute cat..." | Text prompt / editing instruction / question |
| `--image` | None | Input image path(s) for img2img or img2text |
### Image Generation Parameters (text2img / img2img)
| Parameter | Default | Description |
| :-------- | :------ | :---------- |
| `--height` | 2048 | Height of generated image (pixels) |
| `--width` | 2048 | Width of generated image (pixels) |
| `--seed` | 42 | Random seed for reproducibility |
| `--num-steps` | 50 | Number of denoising steps |
| `--cfg-scale` | 4.0 | Text classifier-free guidance scale |
| `--img-cfg-scale` | 1.0 | Image CFG scale (img2img only, 1.0 = disabled) |
| `--cfg-norm` | "none" | CFG normalization: none, global, channel, cfg_zero_star (t2i only) |
| `--timestep-shift` | 3.0 | Timestep shift for flow-matching schedule |
| `--t-eps` | 0.02 | Epsilon for timestep schedule |
| `--think` | False | Enable think mode |
| `--print-think` | False | Print think text to stdout |
### Text Generation Parameters (text2text / img2text)
| Parameter | Default | Description |
| :-------- | :------ | :---------- |
| `--max-tokens` | 512 | Maximum number of tokens to generate |
| `--do-sample` | False | Use sampling instead of greedy decoding |
| `--temperature` | 0.7 | Sampling temperature (higher = more diverse) |
### Infrastructure Parameters
| Parameter | Default | Description |
| :-------- | :------ | :---------- |
| `--model` | `SenseNova/SenseNova-U1-8B-MoT` | HuggingFace model ID or local path |
| `--output` | `.` | Output directory for saved images |
| `--tensor-parallel-size` | 1 | Number of GPUs for tensor parallelism |
| `--enforce-eager` | False | Disable torch.compile |
| `--enable-cpu-offload` | False | Enable module-wise (sequential) CPU offload to reduce peak VRAM |
| `--cache-backend` | None | Set to `cache_dit` or `tea_cache` for image generation acceleration |
| `--enable-cache-dit-summary` | False | Print Cache-DiT cache statistics after generation |
## Reducing GPU Memory Usage
For hardware with limited VRAM, enable **module-wise CPU offload** with
`--enable-cpu-offload`. The pipeline implements `SupportsComponentDiscovery`, so the
vision encoder (`vision_model`) and the Qwen3 LLM (`language_model`) are
swapped between CPU and GPU on demand:
- During text/vision encoding, the LLM is on CPU.
- During the diffusion loop, the vision encoder is on CPU.
- Lightweight FM modules stay resident on GPU.
This lowers peak VRAM at the cost of extra CPU<->GPU transfers (use pinned
memory) — useful for running SenseNova-U1 on consumer-grade GPUs.
```bash
# Text-to-image with CPU offload
python end2end.py \
--prompt "A cute cat sitting on a windowsill" \
--width 2048 --height 2048 \
--enable-cpu-offload --think
# Image-to-image editing with CPU offload
python end2end.py \
--prompt "Turn this into an oil painting" \
--image input.png \
--width 2048 --height 2048 \
--enable-cpu-offload --think
# Image understanding (img2text) with CPU offload
python end2end.py \
--modality img2text \
--prompt "Describe this image in detail" \
--image photo.jpg \
--enable-cpu-offload
```
> **Notes**
> - CPU offload is **single-GPU only** (incompatible with `--tensor-parallel-size > 1`).
> - First-step latency is higher because of the cold-start CPU<->GPU transfers.
> - For more details and other offloading strategies, see
> [CPU Offloading for Diffusion Models](../../../docs/user_guide/diffusion/cpu_offload_diffusion.md).
## Reproducing the E2E Test
The following command reproduces the pixel-validated CI test case:
```bash
python end2end.py \
--prompt "Close portrait of an elderly woman by a farmhouse window, textured skin, gentle smile, warm natural light, emotional documentary look. The portrait should feel polished and natural, with sharp eyes, realistic skin texture, accurate facial anatomy, and premium lighting that keeps the face as the main focus." \
--width 1536 --height 2720 \
--seed 42 --num-steps 50 \
--cfg-scale 4.0 --timestep-shift 3.0 --cfg-norm none \
--think --print-think \
--output outputs
```
The corresponding pytest:
```bash
pytest -s -v tests/e2e/offline_inference/test_sensenova_u1_text2img.py \
-m "advanced_model" --run-level "advanced_model"
```
The img2img counterpart lives at
`tests/e2e/offline_inference/test_sensenova_u1_img2img.py`.
## Online Serving
For OpenAI-compatible API serving, see
[`examples/online_serving/sensenova_u1/`](../../online_serving/sensenova_u1/README.md).
examples/offline_inference/sensenova_u1/end2end.py
-307
View File
@@ -1,307 +0,0 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
"""
End-to-end offline inference for SenseNova-U1-8B-MoT.
Supports four modalities: text2img, img2img, img2text, text2text.
Text-to-image:
python end2end.py --prompt "A cute cat" --think
Image-to-image (editing):
python end2end.py --prompt "Turn this into an oil painting" \
--image input.png --think
Image understanding:
python end2end.py --modality img2text \
--prompt "Describe this image" --image photo.jpg
Text chat:
python end2end.py --modality text2text \
--prompt "What is the capital of France?"
See README.md for more examples.
"""
import argparse
import os
from PIL import Image
from vllm_omni.entrypoints.omni import Omni
from vllm_omni.inputs.data import OmniDiffusionSamplingParams
def parse_args():
parser = argparse.ArgumentParser(
description="SenseNova-U1 text-to-image / image-to-image / text / understanding via vLLM-Omni.",
)
parser.add_argument(
"--model",
default="SenseNova/SenseNova-U1-8B-MoT",
help="HuggingFace model ID or local path.",
)
parser.add_argument(
"--modality",
default="auto",
choices=["auto", "text2img", "img2img", "img2text", "text2text"],
help="Task modality. 'auto' infers from --image (img2img/img2text if images, else t2i/t2t).",
)
parser.add_argument(
"--prompt",
default="A cute cat sitting on a windowsill, soft natural light",
help="Text prompt for generation or editing instruction.",
)
parser.add_argument(
"--image",
nargs="+",
metavar="PATH",
default=None,
help="Input image path(s) for img2img or img2text.",
)
parser.add_argument(
"--output",
type=str,
default=".",
help="Output directory to save generated images.",
)
# Image dimensions
parser.add_argument("--height", type=int, default=2048, help="Height of generated image.")
parser.add_argument("--width", type=int, default=2048, help="Width of generated image.")
# Generation parameters
parser.add_argument("--seed", type=int, default=42, help="Random seed for deterministic results.")
parser.add_argument("--num-steps", type=int, default=50, help="Number of denoising steps.")
parser.add_argument("--cfg-scale", type=float, default=4.0, help="Text classifier-free guidance scale.")
parser.add_argument(
"--img-cfg-scale",
type=float,
default=1.0,
help="Image CFG scale for img2img (1.0 = disabled). "
"When both --cfg-scale and --img-cfg-scale differ, dual CFG is used.",
)
parser.add_argument(
"--cfg-norm",
type=str,
default="none",
choices=["none", "global", "channel", "cfg_zero_star"],
help="CFG normalization mode. cfg_zero_star is only for t2i.",
)
parser.add_argument(
"--timestep-shift",
type=float,
default=3.0,
help="Timestep shift for flow-matching schedule.",
)
parser.add_argument(
"--t-eps",
type=float,
default=0.02,
help="Epsilon for flow-matching timestep schedule.",
)
# Text generation parameters (for text2text / img2text)
parser.add_argument(
"--max-tokens",
type=int,
default=2048,
help="Max tokens for text generation (text2text / img2text).",
)
parser.add_argument(
"--do-sample",
action="store_true",
help="Use sampling instead of greedy decoding for text generation.",
)
parser.add_argument(
"--temperature",
type=float,
default=0.7,
help="Sampling temperature for text generation.",
)
# Think mode
parser.add_argument(
"--think",
action="store_true",
help="Enable think mode: the model reasons about the prompt before generating the image.",
)
parser.add_argument(
"--print-think",
action="store_true",
help="Print the generated think text to stdout.",
)
# Advanced
parser.add_argument(
"--tensor-parallel-size",
type=int,
default=1,
help="Number of GPUs for tensor parallelism.",
)
parser.add_argument(
"--cfg-parallel-size",
type=int,
default=1,
help="Number of GPUs for CFG parallelism.",
)
parser.add_argument(
"--enforce-eager",
action="store_true",
help="Disable torch.compile and force eager execution.",
)
parser.add_argument(
"--enable-cpu-offload",
action="store_true",
help=(
"Enable model-level (sequential) CPU offloading. "
"Mutually exclusive between vision_model and language_model: "
"only one is on GPU at a time, reducing peak VRAM at the cost "
"of extra CPU<->GPU transfers."
),
)
parser.add_argument(
"--cache-backend",
type=str,
default=None,
choices=["cache_dit", "tea_cache"],
help="Cache backend to use for image generation acceleration.",
)
parser.add_argument(
"--enable-cache-dit-summary",
action="store_true",
help="Enable cache-dit summary logging after diffusion forward passes.",
)
return parser.parse_args()
def _resolve_modality(args):
"""Determine the effective modality from CLI args."""
if args.modality != "auto":
return args.modality
has_images = args.image is not None
# Default: image-producing tasks for backward compatibility
return "img2img" if has_images else "text2img"
def main():
args = parse_args()
os.makedirs(args.output, exist_ok=True)
modality = _resolve_modality(args)
is_text_output = modality in ("text2text", "img2text")
has_images = modality in ("img2img", "img2text")
omni = Omni(
model=args.model,
tensor_parallel_size=args.tensor_parallel_size,
enforce_eager=args.enforce_eager,
enable_cpu_offload=args.enable_cpu_offload,
cache_backend=args.cache_backend,
enable_cache_dit_summary=args.enable_cache_dit_summary,
cfg_parallel_size=args.cfg_parallel_size,
)
extra_args = {
"cfg_scale": args.cfg_scale,
"cfg_norm": args.cfg_norm,
"timestep_shift": args.timestep_shift,
"cfg_interval": (0.0, 1.0),
"batch_size": 1,
"think": args.think,
"t_eps": args.t_eps,
}
if modality == "img2img":
extra_args["img_cfg_scale"] = args.img_cfg_scale
if is_text_output:
extra_args["max_tokens"] = args.max_tokens
extra_args["do_sample"] = args.do_sample
extra_args["temperature"] = args.temperature
sampling_params = OmniDiffusionSamplingParams(
height=args.height,
width=args.width,
seed=args.seed,
num_inference_steps=args.num_steps,
extra_args=extra_args,
)
print(f"\n{'=' * 60}")
print(f"SenseNova-U1 Configuration ({modality}):")
print(f" Model : {args.model}")
if not is_text_output:
print(f" Image size : {args.width}x{args.height}")
print(f" Steps : {args.num_steps}")
print(f" CFG scale : {args.cfg_scale}")
if modality == "img2img":
print(f" Img CFG scale : {args.img_cfg_scale}")
if has_images:
print(f" Input images : {args.image}")
if is_text_output:
print(f" Max tokens : {args.max_tokens}")
print(f" Temperature : {args.temperature}")
print(f" Seed : {args.seed}")
print(f" Think mode : {args.think}")
print(f" TP size : {args.tensor_parallel_size}")
print(f" CFG size : {args.cfg_parallel_size}")
print(f" Cache backend : {args.cache_backend or 'none'}")
print(f"{'=' * 60}\n")
# Build prompt dict
if has_images:
if not args.image:
raise ValueError(f"{modality} requires --image.")
input_images = [Image.open(p).convert("RGB") for p in args.image]
if is_text_output:
prompt_dict = {
"prompt": args.prompt,
"multi_modal_data": {"image": input_images},
"modalities": ["text"],
}
else:
prompt_dict = {
"prompt": args.prompt,
"multi_modal_data": {"image": input_images},
"modalities": ["img2img"],
}
elif is_text_output:
prompt_dict = {"prompt": args.prompt, "modalities": ["text"]}
else:
prompt_dict = {"prompt": args.prompt, "modalities": ["image"]}
outputs = list(
omni.generate(
prompts=prompt_dict,
sampling_params_list=sampling_params,
)
)
for req_output in outputs:
multimodal_output = getattr(req_output, "multimodal_output", {}) or {}
metadata = multimodal_output.get("metadata", {}) if isinstance(multimodal_output, dict) else {}
text_metadata = metadata.get("text", {}) if isinstance(metadata, dict) else {}
think_text = text_metadata.get("think_text") if isinstance(text_metadata, dict) else None
if args.print_think and think_text:
print(f"[Think]\n{think_text}\n")
if is_text_output:
text = multimodal_output.get("text", "") if isinstance(multimodal_output, dict) else ""
if not text:
text = text_metadata.get("text_output", "") if isinstance(text_metadata, dict) else ""
if not text:
text = getattr(req_output, "text", "") or ""
print(f"[Response]\n{text}")
else:
images = getattr(req_output, "images", None) or []
if not images:
print("[Warning] No images generated.")
continue
for j, img in enumerate(images):
save_path = os.path.join(args.output, f"sensenova_u1_output_{j}.png")
img.save(save_path)
print(f"[Output] Saved {img.size[0]}x{img.size[1]} image to {save_path}")
if __name__ == "__main__":
main()
examples/offline_inference/text_to_image/text_to_image.py
+2 -35
View File
@@ -11,6 +11,7 @@ from typing import Any
import torch
from vllm_omni.diffusion.data import logger
from vllm_omni.diffusion.utils.image_output import extract_images_from_outputs
from vllm_omni.diffusion.utils.param_utils import apply_declared_extra_args
from vllm_omni.entrypoints.omni import Omni
from vllm_omni.entrypoints.openai.stage_params import clone_sampling_params
@@ -601,11 +602,8 @@ def main():
if images:
break
# Fallback: generation-stage pipelines (e.g. MammothModa2's AR->DiT) return the
# generated image as a tensor under multimodal_output instead of populating the
# `images` field that diffusion-stage pipelines fill.
if not images:
images = _images_from_multimodal_output(outputs)
images = extract_images_from_outputs(outputs)
if not images:
raise ValueError("No images found in request_output")
@@ -624,36 +622,5 @@ def main():
print(f"Saved generated image to {save_path}")
def _images_from_multimodal_output(outputs: list[Any]) -> list[Any]:
"""Extract PIL images from multimodal_output tensors.
Generation-stage pipelines (e.g. MammothModa2's AR->DiT) return the generated
image as a tensor (normalized to [-1, 1], CHW) under ``multimodal_output``
rather than populating the ``images`` field. Convert any such tensors to PIL.
"""
from PIL import Image
pil_images: list[Any] = []
for output in outputs:
req_out = getattr(output, "request_output", output)
for completion in getattr(req_out, "outputs", None) or []:
# multimodal_output is a MultimodalPayload (a Mapping) keyed by modality,
# matching how omni examples (ming_flash_omni / magi_human / dynin) read it.
mm = getattr(completion, "multimodal_output", None) or {}
if "image" not in mm:
continue
payload = mm["image"]
for tensor in payload if isinstance(payload, list) else [payload]:
if not isinstance(tensor, torch.Tensor):
continue
img = tensor.detach().to("cpu", dtype=torch.float32)
if img.ndim == 4:
img = img[0]
img = (img / 2 + 0.5).clamp(0, 1).mul(255).to(torch.uint8)
img = img.permute(1, 2, 0).contiguous().numpy()
pil_images.append(Image.fromarray(img))
return pil_images
if __name__ == "__main__":
main()