0 Fork 0

Files

T

Ding Zuhao 1b97ff1188 [Model] Support SenseNova-U1-A3B-MoT (MoE) (#8172 )

Signed-off-by: Ding Zuhao <e1583181@u.nus.edu>

2026-09-27 08:48:08 +00:00

15 KiB

Raw Permalink Blame History

SenseNova-U1

Unified image generation and understanding

Summary

Vendor: SenseNova
Model: SenseNova/SenseNova-U1-8B-MoT
Task: text2img, img2img, img2text (visual understanding), text2text (chat)
Mode: Offline inference, Online serving (OpenAI-compatible API)
Maintainer: Community

When to use this recipe

Use this recipe to run SenseNova-U1-8B-MoT via vLLM-Omni. SenseNova-U1 is a unified Qwen3-based LLM with Mixture-of-Tokenizers (MoT) attention that handles text encoding, optional chain-of-thought reasoning, flow-matching image denoising, and visual understanding in a single pipeline — no separate text encoder or VAE needed. It supports four task modalities: text-to-image, image-to-image editing (with dual CFG), image-to-text understanding, and text-to-text chat.

References

Offline text-to-image: examples/offline_inference/text_to_image/text_to_image.py
Offline image-to-image: examples/offline_inference/image_to_image/image_edit.py
Online serving: examples/online_serving/sensenova_u1/
E2E tests: tests/e2e/offline_inference/test_sensenova_u1_text2img.py, tests/e2e/offline_inference/test_sensenova_u1_img2img.py
HuggingFace model page: SenseNova/SenseNova-U1-8B-MoT

A3B (MoE) variant

sensenova/SenseNova-U1-A3B-MoT runs on this same recipe and the same SenseNovaU1Pipeline — the checkpoint keeps model_type: neo_chat, so it resolves without --model-class-name. Only the --model id changes.

The backbone is Qwen3-MoE instead of dense Qwen3, and both MoT branches are sparse: mlp routes over 128 experts and mlp_mot_gen over 32, top-8 each, with per-expert width moe_intermediate_size=768. SenseNovaU1Config picks SenseNovaU1MoELLMConfig whenever llm_config carries num_experts, so the dense 8B path is untouched. The gen-path knobs (gen_num_experts, gen_num_experts_per_tok, gen_moe_intermediate_size) fall back to their understanding-path counterparts when absent.

Experts run through vLLM's FusedMoE, whose moe_forward op resolves its layer through vllm.forward_context rather than the diffusion runner's own context, so SenseNovaU1ForCausalLM.forward enters that context when the model has MoE layers.

Memory

The weights dominate, and every expert stays resident even though only the top-8 of each tower run per token -- the "A3B" in the name is the ~4B activated, not the 38.7B stored. Counted from the checkpoint's safetensors headers:

Component	Params	BF16
Understanding-path experts (128/layer)	28.99 B	54.0 GiB
Generation-path experts (32/layer)	7.25 B	13.5 GiB
Attention (dual tower)	1.81 B	3.4 GiB
Embedding + LM head	0.62 B	1.2 GiB
Routers, norms, vision and flow-matching heads	0.06 B	0.1 GiB
Total	38.74 B	72.2 GiB

At 1024x1024 the 72.9 GiB peak is those 72.2 GiB of weights plus about 0.7 GiB of activations and workspace. The 2048x2048 edit, the largest activation case below, peaks at 74.6 GiB, 2.4 GiB above the weights. Text-to-image only routes through the generation tower, leaving the 54 GiB of understanding-path experts idle; that tower is reached by the text prefix, think mode, img2text and text2text. One 96 GB card fits TP=1, and TP=2 halves the per-GPU footprint.

Measured (1x H20 96GB, BF16)

Fixed case used for the stage split and the torch.profiler kernel table:

Prompt: Close portrait of an elderly woman by a farmhouse window, warm natural light.
1024x1024, 50 Euler steps, think off, seed 42, cfg_scale 4.0, cfg_norm none, timestep_shift 3.0, t_eps 0.02
Python 3.12, vLLM 0.30.0, torch 2.13.0+cu130
Peak GPU memory is the reserved high-water mark the runner records (peak_memory_mb)

	Time	Memory
Weights after load	~13 s	72.2 GiB
Pipeline `forward`	7.68 s	72.9 GiB peak

--enable-diffusion-pipeline-profiler on that request:

Stage	Time	Share
Text prefix (`_t2i_prefix_forward`, understanding tower)	0.06 s	0.7%
Denoising loop (`_run_denoising_loop`, generation tower, 50 steps)	7.56 s	98.5%
Pipeline `forward`	7.68 s	100%

There is no VAE. The loop ends in unpatchify / denorm / PIL, which is inside the denoise row and is a few tens of milliseconds at 1024x1024.

A warm 8-step generate at the same resolution, wrapped in torch.profiler after one unprofiled warmup request, spent 1.19 s of self-CUDA time as:

Kernel family	Self CUDA	Share	What it is
`fused_moe_kernel`	576 ms	48%	top-8 expert GEMMs on the 32-expert generation tower
FlashAttention `flash_fwd_kernel`	222 ms	19%	dual-tower attention
`nvjet` GEMMs	~233 ms	20%	`qkv_proj` / `o_proj` (and the small dense leftovers)
`qk_norm_rope_kernel`	58 ms	5%	fused RMSNorm + RoPE
MoE routing / `act_and_mul` / elementwise	~40 ms	3%	top-k, SwiGLU, residual

So a denoise step is about half FusedMoE and about two-fifths attention + dense GEMM. --cfg-parallel-size 2 on the same 8-step t2i request (2-branch cond+uncond) is 0.95 s vs 1.48 s on one GPU (~1.56x). Img2img CFG-P=2 also completes for both branch counts at 8-step 1024x1024: 2-way (img_cfg_scale=1, cond+img_cond) 1.09 s; 3-way (cfg_scale=4, img_cfg_scale=2, cond+img_cond+uncond) 1.74 s (3 branches on 2 ranks dispatch as [[0, 2], [1]]). Cache-DiT and layerwise CPU offload do not take effect on A3B; see diffusion features note 8.

Other configurations on the same machine (second run, warm kernels):

Case	Total	Peak GPU memory
text2img 1024x1024, 50 steps, think on	9.16 s	73.0 GiB
text2img 1024x1024, 50 steps, think off, TP=2	5.64 s	36.7 GiB per GPU
img2img 2048x2048 output, 25 steps, think off	20.93 s	74.6 GiB

Think-on spends 1.37 s in _generate_think (215 tokens). At the same seed TP=2 differs from TP=1 only numerically (MAE 2.97/255, identical composition) from the changed reduction order.

python examples/offline_inference/text_to_image/text_to_image.py \
    --model sensenova/SenseNova-U1-A3B-MoT \
    --prompt "Close portrait of an elderly woman by a farmhouse window, warm natural light." \
    --width 1024 --height 1024 \
    --seed 42 --num-inference-steps 50 --cfg-scale 4.0 \
    --extra-body '{"think": false, "cfg_norm": "none", "timestep_shift": 3.0, "t_eps": 0.02}' \
    --output sensenova_a3b_t2i.png

img2text and text2text go through the server, as for the dense checkpoint:

vllm serve sensenova/SenseNova-U1-A3B-MoT --omni --port 8091

python examples/online_serving/sensenova_u1/openai_chat_client.py \
    -s http://127.0.0.1:8091 -m img2text -i input.png -p "Describe this image."

All four modalities were verified on A3B: text2img (with and without think), img2img editing, img2text and text2text.

Hardware Support

GPU

1x H200 (144GB)

Environment

OS: Linux
Python: 3.12
Driver / runtime: NVIDIA 590.48.01, CUDA 13.1
vLLM-Omni version: 0.18.1.dev

Command

python examples/offline_inference/text_to_image/text_to_image.py \
    --model SenseNova/SenseNova-U1-8B-MoT \
    --prompt "Close portrait of an elderly woman by a farmhouse window, textured skin, gentle smile, warm natural light, emotional documentary look. The portrait should feel polished and natural, with sharp eyes, realistic skin texture, accurate facial anatomy, and premium lighting that keeps the face as the main focus." \
    --width 1536 --height 2720 \
    --seed 42 --num-inference-steps 50 \
    --cfg-scale 4.0 \
    --extra-body '{"think": true, "cfg_norm": "none", "timestep_shift": 3.0, "t_eps": 0.02}' \
    --output outputs/sensenova_u1_output.png

Verification

pytest -s -v tests/e2e/offline_inference/test_sensenova_u1_text2img.py \
    -m "advanced_model" --run-level "advanced_model"

Notes

E2E latency: 32.1s (1536×2720, 50 steps, think mode, CFG scale 4.0)
Peak VRAM: 35.9 GB reserved, 35.1 GB allocated
Model loading: 32.8 GiB, 8.7s
No deploy YAML needed — the engine auto-generates a single-stage diffusion config.
Think mode (--think) is recommended for higher image quality.

Text-to-Image via the shared example

SenseNova-U1 is registered with the shared text-to-image example. Forward SenseNova-specific generation parameters as a JSON object through --extra-body:

python examples/offline_inference/text_to_image/text_to_image.py \
    --model SenseNova/SenseNova-U1-8B-MoT \
    --prompt "A beautiful sunset over mountains" \
    --width 2048 --height 2048 \
    --num-inference-steps 50 \
    --seed 42 \
    --extra-body '{"think": true, "cfg_scale": 4.0, "cfg_norm": "none", "timestep_shift": 3.0, "t_eps": 0.02}' \
    --output sensenova_text2img.png

The --extra-body keys are filtered against the model's declared extra_body_params (see vllm_omni/model_extras/sensenova_u1.py): think, cfg_scale, cfg_norm, timestep_shift, t_eps, img_cfg_scale, and max_tokens. No deploy YAML is needed — the engine auto-generates a single-stage diffusion config.

Image-to-Image Editing (img2img)

python examples/offline_inference/image_to_image/image_edit.py \
    --model SenseNova/SenseNova-U1-8B-MoT \
    --prompt "Turn this into an oil painting" \
    --image input.png \
    --resolution 2048 \
    --seed 42 --num-inference-steps 50 \
    --cfg-scale 4.0 \
    --extra-args '{"think": true, "img_cfg_scale": 1.0, "cfg_norm": "none", "timestep_shift": 3.0}' \
    --output outputs/sensenova_u1_edit.png

img2img uses dual CFG: --cfg-scale controls text guidance, while img_cfg_scale in --extra-args controls image guidance (1.0 = image CFG disabled).
Pass multiple --image paths for multi-reference editing when the underlying pipeline supports them.

Image Understanding (img2text)

vllm serve SenseNova/SenseNova-U1-8B-MoT --omni --port 8091
python examples/online_serving/sensenova_u1/openai_chat_client.py \
    --prompt "Describe this image in detail" \
    --modality img2text \
    --image-url photo.jpg

Text-to-Text Chat (text2text)

vllm serve SenseNova/SenseNova-U1-8B-MoT --omni --port 8091
python examples/online_serving/sensenova_u1/openai_chat_client.py \
    --prompt "Explain the theory of relativity in simple terms" \
    --modality text2text

For img2text and text2text, use the online chat-compatible example. The offline generic image examples intentionally focus on image-producing tasks.

2x H200 (144GB) — TP=2

Environment

OS: Linux
Python: 3.12
Driver / runtime: NVIDIA 590.48.01, CUDA 13.1
vLLM-Omni version: 0.18.1.dev

Command

python examples/offline_inference/text_to_image/text_to_image.py \
    --model SenseNova/SenseNova-U1-8B-MoT \
    --prompt "Close portrait of an elderly woman by a farmhouse window, textured skin, gentle smile, warm natural light, emotional documentary look. The portrait should feel polished and natural, with sharp eyes, realistic skin texture, accurate facial anatomy, and premium lighting that keeps the face as the main focus." \
    --width 1536 --height 2720 \
    --seed 42 --num-inference-steps 50 \
    --cfg-scale 4.0 \
    --extra-body '{"think": true, "cfg_norm": "none", "timestep_shift": 3.0, "t_eps": 0.02}' \
    --tensor-parallel-size 2 \
    --output outputs/sensenova_u1_output.png

Verification

Verify the output image is generated at outputs/sensenova_u1_output.png with the expected 1536×2720 resolution.

Notes

E2E latency: 28.3s (1536×2720, 50 steps, think mode, CFG scale 4.0)
Peak VRAM (per GPU): 18.2 GB reserved, 17.9 GB allocated
Model loading: 16.5 GiB per GPU, 7.0s
TP=2 provides ~12% speedup over TP=1; limited by serial CFG dual-forward and communication overhead.
The LLM transformer uses QKVParallelLinear and MergedColumnParallelLinear for fused QKV and gate/up projections with TP support.

1x AMD MI300X 192GB

Environment

OS: Linux 6.8.0-134-generic, x86_64
Container: official ROCm image built from docker/Dockerfile.rocm
Python: 3.12.13
PyTorch: 2.11.0+gitd0c8b1f
Driver / runtime: AMD 6.19.14.31400000 / ROCm 7.2.53211
GPU: one AMD Instinct MI300X, gfx942:sramecc+:xnack-, 191.69 GiB visible HBM
vLLM version: 0.27.0+rocm723
vLLM Omni version or commit: a704c8759c96e123c0d7c89b11f120b1c0f120cf
Installed vLLM Omni package metadata: 0.27.0rc2.dev44+g55abdade9.rocm

Command

python3 examples/offline_inference/text_to_image/text_to_image.py \
    --model SenseNova/SenseNova-U1-8B-MoT \
    --prompt "Close portrait of an elderly woman by a farmhouse window, textured skin, gentle smile, warm natural light, emotional documentary look." \
    --width 1536 \
    --height 2720 \
    --seed 42 \
    --num-inference-steps 50 \
    --cfg-scale 4.0 \
    --extra-body '{"think": true, "cfg_norm": "none", "timestep_shift": 3.0, "t_eps": 0.02}' \
    --enable-diffusion-pipeline-profiler \
    --log-stats \
    --output sensenova_u1_mi300x.png

Verification

The command completed and wrote a valid 1536 by 2720 RGB PNG.

Notes

Generation with 50 inference steps took 34.011 seconds.
Model loading used 32.774 GiB and took 13.987 seconds.
The internal profiler recorded 35.67 GB reserved and 35.10 GB allocated for the request.
The highest one second whole device memory sample was 37.93 GiB.

Online Serving

SenseNova-U1 supports all four modalities via the OpenAI-compatible /v1/chat/completions API.

Launch

vllm serve SenseNova/SenseNova-U1-8B-MoT --omni --port 8091

Send Requests

cd examples/online_serving/sensenova_u1

# Text-to-image
python openai_chat_client.py \
    --prompt "A beautiful sunset" --modality text2img

# Image-to-image editing
python openai_chat_client.py \
    --prompt "Turn this into an oil painting" \
    --modality img2img --image-url input.jpg

# Image understanding
python openai_chat_client.py \
    --prompt "Describe this image" \
    --modality img2text --image-url photo.jpg

# Text chat
python openai_chat_client.py \
    --prompt "What is the capital of France?" \
    --modality text2text

For full API documentation and curl examples, see examples/online_serving/sensenova_u1/README.md.

15 KiB Raw Permalink Blame History Unescape Escape

SenseNova-U1

Summary

When to use this recipe

References

A3B (MoE) variant

Memory

Measured (1x H20 96GB, BF16)

Hardware Support

GPU

1x H200 (144GB)

Environment

Command

Verification

Notes

Text-to-Image via the shared example

Image-to-Image Editing (img2img)

Image Understanding (img2text)

Text-to-Text Chat (text2text)

2x H200 (144GB) — TP=2

Environment

Command

Verification

Notes

1x AMD MI300X 192GB

Environment

Command

Verification

Notes

Online Serving

Launch

Send Requests

15 KiB

Raw Permalink Blame History