lishunyang
69bcc6416d
Adds Lance (bytedance-research/Lance), a 3B unified AR+diffusion multimodal model on a Qwen2.5-VL backbone. Lance is BAGEL-lineage: it reuses vLLM-Omni's BAGEL Mixture-of-Transformers core verbatim and specializes only the ViT (Qwen2.5-VL vision), the VAE (Wan2.2) and the checkpoint layout. All six single-stage Lance modalities verified end-to-end on 1x B300: t2i, t2v, image_edit, video_edit, x2t_image (image understanding) and x2t_video (video understanding). 3-D mRoPE is wired through BagelRotaryEmbedding for the x2t / video / edit paths. Includes the upstream gap-analysis quality fixes (task-specific system prompts, full Qwen chat template, upstream inference defaults and aspect-ratio buckets). Ships an offline example (examples/offline_inference/lance/), an OpenAI-compatible online-serving example (examples/online_serving/lance/) and an e2e test (tests/e2e/online_serving/test_lance.py). See LANCE_PR.md for the architecture rationale, per-modality status and the two remaining follow-ups (two-stage AR+DiT, video_edit polish). Closes #3697. Signed-off-by: lishunyang <lishunyang03@gmail.com>
34 lines
1.6 KiB
Python
34 lines
1.6 KiB
Python
# SPDX-License-Identifier: Apache-2.0
|
|
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
|
"""Lance AR-stage model wrapper (two-stage topology).
|
|
|
|
Lance is BAGEL-lineage, so the AR/thinker stage is the same Qwen2-MoT model
|
|
BAGEL uses. This subclass exists so the two-stage pipeline has a distinct
|
|
arch name to register against; behaviour is inherited unchanged from
|
|
:class:`OmniBagelForConditionalGeneration`.
|
|
|
|
IMPORTANT: the two-stage Lance topology is a follow-up. It additionally
|
|
requires a ``LanceConfig`` + ``LanceProcessor`` registered in the ``vllm``
|
|
package (the base ``BagelForConditionalGeneration`` resolves its HF config /
|
|
processor by ``model_type == "bagel"``; Lance ships ``model_type ==
|
|
"qwen2_5_vl"`` in ``llm_config.json`` and no BAGEL-style top-level config).
|
|
Until that lands, use the **single-stage** topology (``model_type: lance`` ->
|
|
``LANCE_SINGLE_STAGE_PIPELINE`` -> ``LancePipeline``), which is fully wired and
|
|
needs no ``vllm``-submodule changes.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
from vllm_omni.model_executor.models.bagel.bagel import (
|
|
OmniBagelForConditionalGeneration,
|
|
)
|
|
|
|
|
|
class OmniLanceForConditionalGeneration(OmniBagelForConditionalGeneration):
|
|
"""Lance AR-stage model. Identical to BAGEL's (same Qwen2-MoT weights);
|
|
distinct class only so the two-stage pipeline can register a ``lance``
|
|
arch separately. See module docstring for the two-stage follow-up."""
|
|
|
|
# Same packed→sublayer LoRA / MoT mapping as BAGEL (weight names match).
|
|
packed_modules_mapping = OmniBagelForConditionalGeneration.packed_modules_mapping
|