vllm-omni/recipes/SenseNova/SenseNova-U1.5.md

SenseNova-U1.5

Unified image generation and understanding, with an official 8-step distilled LoRA

Summary

• Vendor: SenseNova
• Model: sensenova/SenseNova-U1.5-8B-MoT
• LoRA: sensenova/SenseNova-U1.5-8B-MoT-LoRAs (SenseNova-U1.5-8B-MoT-LoRA-8step.safetensors)
• Task: text2img, img2img, img2text (visual understanding), text2text (chat)
• Mode: Offline inference, Online serving (OpenAI-compatible API)
• Maintainer: Community

When to use this recipe

U1.5 runs on the same SenseNovaU1Pipeline as U1 — the checkpoint keeps model_type: neo_chat, so it resolves without --model-class-name. Relative to U1 the config flips two fields, use_pixel_head to true (the flow-matching head becomes a ConvDecoder) and noise_scale_max_value from 8.0 to 16.0; both are already read by SenseNovaU1Config. The checkpoint is 13 shards / 50.2 GB on disk, 30.3 GB of that fp32, and loads to roughly 34 GB in bf16.

Use this recipe for U1.5 specifically, including its distilled few-step LoRA. For U1 see SenseNova-U1.

Hardware Support

GPU

1x A800 80GB

• 1x NVIDIA A800 80GB PCIe with an Intel Xeon Gold 6336Y, driver 595.84, Linux
• Python 3.12.13, vLLM 0.28.0, torch 2.13.0+cu130, CUDA 13.0, diffusers 0.40.0, transformers 5.14.1, flashinfer 0.6.16.post3, BF16, TP=1
• Peak GPU memory: 34.3 GB at 1024x1024, 36.4 GB at 1536x2720

Text-to-Image

python examples/offline_inference/text_to_image/text_to_image.py \
    --model sensenova/SenseNova-U1.5-8B-MoT \
    --prompt "Close portrait of an elderly woman by a farmhouse window, warm natural light." \
    --width 1024 --height 1024 \
    --seed 42 --num-inference-steps 50 --cfg-scale 4.0 \
    --extra-body '{"think": false, "cfg_norm": "none", "timestep_shift": 3.0, "t_eps": 0.02}' \
    --output sensenova_u15_t2i.png

Think mode ("think": true) is recommended for higher image quality.

Image-to-Image Editing

python examples/offline_inference/image_to_image/image_edit.py \
    --model sensenova/SenseNova-U1.5-8B-MoT \
    --prompt "Turn this into an oil painting" \
    --image input.png --resolution 1024 \
    --seed 42 --num-inference-steps 25 --cfg-scale 4.0 \
    --extra-args '{"think": true, "img_cfg_scale": 1.0, "cfg_norm": "none", "timestep_shift": 3.0}' \
    --output sensenova_u15_edit.png

8-step distilled LoRA

The distilled LoRA is fused into the generation tower at load time:

python examples/offline_inference/text_to_image/text_to_image.py \
    --model sensenova/SenseNova-U1.5-8B-MoT \
    --lora-path SenseNova-U1.5-8B-MoT-LoRA-8step.safetensors --lora-backend distill \
    --prompt "Close portrait of an elderly woman by a farmhouse window, warm natural light." \
    --width 1024 --height 1024 \
    --seed 42 --num-inference-steps 8 --cfg-scale 1.0 \
    --extra-body '{"think": false, "cfg_norm": "none", "timestep_shift": 3.0, "t_eps": 0.02}' \
    --output sensenova_u15_lora8.png

Use --cfg-scale 1.0 with this LoRA. It is distilled with DMD and runs without classifier-free guidance; the default 4.0 applies guidance twice and produces a blown-out, posterised image.

Online FP8 does not support the 8-step distilled LoRA. Use BF16 without --quantization fp8 for this LoRA.

Online serving

vllm serve sensenova/SenseNova-U1.5-8B-MoT --omni --port 8091

python examples/online_serving/sensenova_u1/openai_chat_client.py \
    -s http://127.0.0.1:8091 -m img2text -i input.png -p "Describe this image."

-s takes the base URL; the client appends /v1 itself.

Online FP8 quantization

--quantization fp8 quantizes only eligible attention and MLP linears in the language_model understanding and generation branches at load time. Other layers retain their existing precision.

On the tested A800 (SM80) setup with vLLM 0.29.0, VLLM_DISABLED_KERNELS=CutlassFP8ScaledMMLinearKernel is required for online FP8 to start successfully. Without it, the selected CUTLASS kernel fails during the startup dummy run with RuntimeError: cutlass_scaled_mm_sm80_epilogue. Disabling this kernel selects Marlin W8A16 (FP8 weights with BF16 activations).

VLLM_DISABLED_KERNELS=CutlassFP8ScaledMMLinearKernel \
vllm serve sensenova/SenseNova-U1.5-8B-MoT --omni \
    --quantization fp8 --port 8091

Measured BF16 vs FP8 performance (1x A800 80GB, 28 steps, P95 over 10 prompts)

vLLM 0.29.0, BF16 base dtype, seed 42, TORCH_SDPA, eager execution, TP=1, batch size 1, paged decode off, no LoRA. Each resolution uses the same 10 prompts for BF16 and FP8. Both runs set VLLM_DISABLED_KERNELS=CutlassFP8ScaledMMLinearKernel.

Resolution	Peak reserved memory (BF16 → FP8)	Step latency (BF16 → FP8)
512x512	33.166 → 18.697 GiB	63.674 → 81.136 ms
1024x1024	33.555 → 18.561 GiB	187.867 → 229.115 ms

Step latency is the P95 of per-request mean step times, excluding warmup requests; FP8 reduced memory usage but increased step latency.

Measured BF16 latency (1x A800 80GB, 25 steps, median of 3 after a warmup)

Resolution	Step latency	Total
1024x1024	209.5 ms	5.24 s
1536x1536	458.5 ms	11.46 s

At 1536x2720 with 50 steps, end-to-end is 43.0 s.

Measured latency (1x NVIDIA H200 139GB, single run, no warmup)

Reported by @hsliuustc0106 against PR head 29c090d, on one reserved CUDA device.

• Python 3.12.13, vLLM 0.28.0, torch 2.13.0+cu130, CUDA 13.0, diffusers 0.40.0, transformers 5.14.1, flashinfer 0.6.16.post3, BF16, TP=1
• sensenova/SenseNova-U1.5-8B-MoT with SenseNova-U1.5-8B-MoT-LoRA-8step.safetensors, seed 42, 1024x1024

Case	Stage latency	Peak GPU memory
50 steps, cfg 4.0, think off	9017 ms	34384 MiB
8 steps, cfg 1.0, no LoRA control	624.00 ms, 77.85 ms/step	34376 MiB
8 steps, cfg 1.0, distilled LoRA	623.85 ms, 77.98 ms/step	34364 MiB
50 steps, cfg 4.0, think on, VLLM_OMNI_SENSENOVA_PAGED_DECODE=1	81622 ms	34594 MiB

Each row runs the Text-to-Image command above with --num-inference-steps and --cfg-scale as listed and think set through --extra-body; the LoRA row adds --lora-path and --lora-backend distill.

Single runs without a preceding warmup, so the think-on row carries the first-request compile rather than a steady state. The distilled LoRA fused into 168 parameters, and its image differs from the same-seed no-LoRA control in 1,048,574 of 1,048,576 pixels, MAE 58.62.

pytest -m "cpu and not cuda" tests/diffusion/models/sensenova_u1/ tests/diffusion/lora/test_loader.py -q
pytest -m "cpu and not cuda" tests/config/test_environment_variables.py tests/diffusion/test_diffusion_worker.py -q
pytest -m cuda tests/diffusion/models/sensenova_u1/test_sensenova_u1_attention_gqa.py \
    tests/diffusion/models/sensenova_u1/test_sensenova_u1_paged_decode.py -q --tb=short

71, 13 and 9 passed. Serving reached /health; img2text returned a 1921-character description and text2text the expected answer.

Verification

pytest -q tests/diffusion/models/sensenova_u1/

Notes

• Both use_pixel_head and noise_scale_max_value come from config.json; no flag is needed.
• The checkpoint carries no configuration_neo_chat.py, so the loader logs a few "does not appear to have a file named configuration_neo_chat.py" errors during startup and then proceeds on the in-tree config. Generation is unaffected.
• The LoRA targets the generation tower only (*_mot_gen); understanding-tower weights are untouched.
• Autoregressive decode (think, text-to-text and image-to-text) runs on a paged K/V cache under a captured CUDA graph. It falls back to the ordinary cache when the device or the bundled flash_attn_varlen_func cannot support it; set VLLM_OMNI_SENSENOVA_PAGED_DECODE=0 to force that fallback.
• The first request after startup costs about 0.7 s more than the steady state whether the paged path is on or off. Measured on one A800 with the inductor, triton and vLLM compile caches all cleared, median of three runs: 718 ms above steady with the path on, 679 ms with it off.
• Each request captures its own graphs, and a think request captures twice because the sequence grows past the 512 bucket, so the capture cost is paid per request rather than once at startup.