lishunyang
69bcc6416d
Adds Lance (bytedance-research/Lance), a 3B unified AR+diffusion multimodal model on a Qwen2.5-VL backbone. Lance is BAGEL-lineage: it reuses vLLM-Omni's BAGEL Mixture-of-Transformers core verbatim and specializes only the ViT (Qwen2.5-VL vision), the VAE (Wan2.2) and the checkpoint layout. All six single-stage Lance modalities verified end-to-end on 1x B300: t2i, t2v, image_edit, video_edit, x2t_image (image understanding) and x2t_video (video understanding). 3-D mRoPE is wired through BagelRotaryEmbedding for the x2t / video / edit paths. Includes the upstream gap-analysis quality fixes (task-specific system prompts, full Qwen chat template, upstream inference defaults and aspect-ratio buckets). Ships an offline example (examples/offline_inference/lance/), an OpenAI-compatible online-serving example (examples/online_serving/lance/) and an e2e test (tests/e2e/online_serving/test_lance.py). See LANCE_PR.md for the architecture rationale, per-modality status and the two remaining follow-ups (two-stage AR+DiT, video_edit polish). Closes #3697. Signed-off-by: lishunyang <lishunyang03@gmail.com>
32 lines
1.3 KiB
Python
32 lines
1.3 KiB
Python
# SPDX-License-Identifier: Apache-2.0
|
|
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
|
"""Lance (ByteDance) diffusion model components.
|
|
|
|
Lance is a unified autoregressive + diffusion multimodal model on a
|
|
Qwen2.5-VL-3B backbone. Architecturally it is the BAGEL family (ByteDance
|
|
Mixture-of-Transformers): the released ``Lance_3B`` checkpoint uses the exact
|
|
same ``*_moe_gen`` MoT weight layout as BAGEL, plus ``vae2llm`` / ``llm2vae`` /
|
|
``time_embedder`` / ``latent_pos_embed`` connectors. The deltas vs BAGEL are:
|
|
|
|
* backbone is Qwen2.5-VL (mRoPE) instead of Qwen2,
|
|
* understanding ViT is Qwen2.5-VL vision (not SigLIP), loaded from the base
|
|
``Qwen/Qwen2.5-VL-3B-Instruct`` rather than from the Lance checkpoint,
|
|
* VAE is Wan2.2 (reused from the vLLM-Omni WAN path) instead of the BAGEL AE,
|
|
* video path adds 3D latent position embeddings (follow-up; this module
|
|
implements the image path first).
|
|
|
|
Because Lance is BAGEL-lineage, the transformer core is reused verbatim from
|
|
``vllm_omni.diffusion.models.bagel.bagel_transformer`` and only the pipeline
|
|
wiring (ViT / VAE / checkpoint layout) is specialized here.
|
|
"""
|
|
|
|
from vllm_omni.diffusion.models.lance.pipeline_lance import (
|
|
LancePipeline,
|
|
get_lance_post_process_func,
|
|
)
|
|
|
|
__all__ = [
|
|
"LancePipeline",
|
|
"get_lance_post_process_func",
|
|
]
|