lishunyang
69bcc6416d
Adds Lance (bytedance-research/Lance), a 3B unified AR+diffusion multimodal model on a Qwen2.5-VL backbone. Lance is BAGEL-lineage: it reuses vLLM-Omni's BAGEL Mixture-of-Transformers core verbatim and specializes only the ViT (Qwen2.5-VL vision), the VAE (Wan2.2) and the checkpoint layout. All six single-stage Lance modalities verified end-to-end on 1x B300: t2i, t2v, image_edit, video_edit, x2t_image (image understanding) and x2t_video (video understanding). 3-D mRoPE is wired through BagelRotaryEmbedding for the x2t / video / edit paths. Includes the upstream gap-analysis quality fixes (task-specific system prompts, full Qwen chat template, upstream inference defaults and aspect-ratio buckets). Ships an offline example (examples/offline_inference/lance/), an OpenAI-compatible online-serving example (examples/online_serving/lance/) and an e2e test (tests/e2e/online_serving/test_lance.py). See LANCE_PR.md for the architecture rationale, per-modality status and the two remaining follow-ups (two-stage AR+DiT, video_edit polish). Closes #3697. Signed-off-by: lishunyang <lishunyang03@gmail.com>
Lance (ByteDance) — offline inference
Lance is a 3B unified
autoregressive + diffusion multimodal model on a Qwen2.5-VL backbone. It is
BAGEL-lineage (ByteDance Mixture-of-Transformers): the released Lance_3B
checkpoint uses the same *_moe_gen MoT weight layout as BAGEL, so vLLM-Omni
implements it by reusing the BAGEL transformer core and specializing only the
ViT (Qwen2.5-VL vision), the VAE (Wan2.2) and the checkpoint layout.
This example covers all six Lance modalities from the upstream HF model card:
t2i, t2v, image_edit, video_edit, x2t_image (image understanding) and
x2t_video (video understanding).
Hardware
Single NVIDIA GPU with 16 GB+ VRAM in BF16 (we test on B300 / A100). CUDA ≥ 12.4.
Run
# Text-to-image
python examples/offline_inference/lance/end2end.py \
--model bytedance-research/Lance \
--prompts "a corgi astronaut on the moon, cinematic" \
--steps 30 --cfg-text-scale 4.0 --timestep-shift 3.5 \
--output ./out
# Text-to-video (uses the Lance_3B_Video subfolder; see ``--modality``
# choices for all six task variants)
python examples/offline_inference/lance/end2end.py \
--model bytedance-research/Lance/Lance_3B_Video --modality text2video \
--num-frames 25 --video-height 480 --video-width 768 \
--prompts "a cat playing piano, cinematic" \
--steps 30 --fps 8 --output ./out
video_edit requires --model bytedance-research/Lance/Lance_3B_Video so
the 3-D latent_pos_embed table is loaded; the other paths can point at
the top-level bytedance-research/Lance repo and resolve the right
sub-checkpoint automatically.
The HF repo bundles everything (Lance_3B/, Lance_3B_Video/,
Qwen2.5-VL-ViT/, Wan2.2_VAE.pth); no separate downloads are required.
Defaults
Matches upstream inference_lance.sh: 30 denoising steps, timestep-shift 3.5,
text CFG 4.0, seed 42, 768×768. For the understanding paths (img2text /
video2text), sampling is enabled by default at --text-temperature 0.8
because Lance's greedy decoder emits an immediate EOS for many prompts.