Resolves conflicts in:
- vllm_omni/diffusion/registry.py: keep both LancePipeline (HEAD)
and MingImagePipeline (main) entries.
- vllm_omni/diffusion/models/bagel/bagel_transformer.py: take main's
simpler cfg_batched payload; HEAD referenced undefined locals.
Also strips vllm_omni/deploy/lance.yaml -- Lance is single-stage
diffusion, so its engine knobs (pipeline=lance, enforce_eager,
trust_remote_code, max_num_seqs=1, ...) flow directly to Omni(...)
/ vllm-omni serve instead of through a deploy YAML.
create_default_diffusion builds the stage config from those kwargs.
Touched: examples/offline_inference/lance/{end2end,gradio_demo}.py,
tests/e2e/online_serving/test_lance.py, tests/helpers/stage_config.py.
Signed-off-by: lishunyang12 <lishunyang12@163.com>
- Add ``examples/offline_inference/lance/gradio_demo.py`` — a unified 7-task UI
(t2i / t2v / i2v / image_edit / video_edit / x2t_image / x2t_video) that
loads ``Lance_3B`` and ``Lance_3B_Video`` on two GPUs so image and video
tasks run concurrently. Surfaces official Lance examples per task; aspect
ratio is auto-snapped to one of upstream's 6 buckets from the input
image/video.
- Fix x2t_image / x2t_video output format: route x2t_image through a new
``_forward_x2t_image`` handler (was falling through to the img2img path
and emitting garbage), build the chat prefill in three segments
(prefix → ViT → suffix) so the ``<|im_end|>`` markers line up with
upstream's template, and start generation from a newline token instead of
``<|im_start|>`` so the assistant turn doesn't close itself with an
immediate EOS. Add a ``system_prompt`` override to ``render_lance_prompt``
so per-example QA instructions surface ("Look at the image carefully and
answer the question.") instead of the caption-style default.
- ``_forward_image_edit``: also stash the PIL image into ``custom_output``
so callers reading ``outputs[0].output`` *or* ``custom_output["image"]``
both work — the orchestrator IPC strips the bare ``output`` field across
process boundaries.
- ``pipeline_registry.py``, ``model_executor/models/lance/*``, registry
cleanup: remove the dead Lance-specific autoregressive shim
(``lance.py``) and its stage-input processor; the diffusion pipeline owns
the full flow now.
- Aspect-ratio buckets aligned with upstream's ``BucketResize`` (6 ratios:
21:9 / 16:9 / 4:3 / 1:1 / 3:4 / 9:16, sized for ``resolution_vae`` ∈
{480, 640, 768}). 3:2 / 2:3 dropped — upstream snaps those to 4:3 / 3:4.
- Gradio defaults match upstream's official scripts (cfg_text_scale=4.0,
validation_num_timesteps=30, validation_timestep_shift=3.5, seed=42); only
the demo *duration* is intentionally shorter than upstream's 121-frame
default to keep the UI snappy (user can extend to 10s via slider for
byte-identical reproduction).
Signed-off-by: lishunyang12 <lishunyang12@163.com>
Signed-off-by: WeiQing Chen <david6666666@users.noreply.github.com>
- Precision matching across the Lance transformer and pipeline so vLLM-Omni
outputs match the upstream reference end-to-end (text2img / img2img /
text2video / video2video / image and video understanding).
- Add recipes/ByteDance/Lance.md covering all single-stage modalities for
offline + online; register it in recipes/README.md.
- Offline README: document --height/--width and update the default
resolution to 1024x1024 (matches the upstream inference_lance.sh).
- Gitignore lance_compare/ (local precision-matching dump scripts; not part
of the repo).
- Remove LANCE_PR.md; the PR description lives on GitHub.
Signed-off-by: lishunyang12 <lishunyang12@163.com>
Adds Lance (bytedance-research/Lance), a 3B unified AR+diffusion
multimodal model on a Qwen2.5-VL backbone. Lance is BAGEL-lineage:
it reuses vLLM-Omni's BAGEL Mixture-of-Transformers core verbatim and
specializes only the ViT (Qwen2.5-VL vision), the VAE (Wan2.2) and the
checkpoint layout.
All six single-stage Lance modalities verified end-to-end on 1x B300:
t2i, t2v, image_edit, video_edit, x2t_image (image understanding) and
x2t_video (video understanding). 3-D mRoPE is wired through
BagelRotaryEmbedding for the x2t / video / edit paths. Includes the
upstream gap-analysis quality fixes (task-specific system prompts, full
Qwen chat template, upstream inference defaults and aspect-ratio
buckets).
Ships an offline example (examples/offline_inference/lance/), an
OpenAI-compatible online-serving example
(examples/online_serving/lance/) and an e2e test
(tests/e2e/online_serving/test_lance.py).
See LANCE_PR.md for the architecture rationale, per-modality status and
the two remaining follow-ups (two-stage AR+DiT, video_edit polish).
Closes#3697.
Signed-off-by: lishunyang <lishunyang03@gmail.com>
lishunyang12