[diffusion][feature] Add LingBot-Video T2I and TI2V generation modes (#5311)

Signed-off-by: wtz2333 <2955110911@qq.com>
Co-authored-by: Claude <noreply@anthropic.com>

examples/offline_inference/image_to_video/README.md

@@ -33,6 +33,25 @@ This folder provides a unified CLI script for image-to-video generation using vL
 
 Default model: `Wan-AI/Wan2.2-I2V-A14B-Diffusers`.
 
+### LingBot-Video TI2V
+
+LingBot-Video accepts exactly one first-frame image and uses the same dense or
+MoE checkpoint as its T2I and T2V modes:
+
+```bash
+python examples/offline_inference/image_to_video/image_to_video.py \
+  --model robbyant/lingbot-video-dense-1.3b \
+  --model-class-name LingBotVideoPipeline \
+  --image /path/to/input.png \
+  --prompt "the fox looks toward the camera" \
+  --height 192 --width 320 --num-frames 9 --num-inference-steps 2 \
+  --guidance-scale 3.0 --flow-shift 3.0 --fps 24 \
+  --output lingbot_ti2v.mp4
+```
+
+The input image keeps its original geometry until the LingBot pipeline applies
+its model-specific resize and center crop.
+
 ## Prerequisites
 
 Download the example image used in the snippets below:

examples/offline_inference/text_to_image/README.md

@@ -43,6 +43,23 @@ This folder provides several entrypoints for experimenting with text-to-image di
 
 Default model: `Qwen/Qwen-Image`
 
+### LingBot-Video
+
+LingBot-Video uses the same checkpoint for image and video generation. Run the
+shared T2I entry point with:
+
+```bash
+python examples/offline_inference/text_to_image/text_to_image.py \
+  --model robbyant/lingbot-video-dense-1.3b \
+  --prompt "a red fox standing in fresh snow" \
+  --height 192 --width 320 --num-inference-steps 2 \
+  --guidance-scale 3.0 --extra-body '{"flow_shift":3.0}' \
+  --output lingbot_t2i.png
+```
+
+The LingBot prompt builder selects the `image` output modality and requests one
+frame. Passing an input image is not supported in LingBot T2I mode.
+
 ## Quick Start
 
 ### Python API

examples/offline_inference/text_to_video/text_to_video.md

@@ -55,6 +55,21 @@ python text_to_video.py \
   --output vace_t2v_output.mp4
 ```
 
+### LingBot-Video
+
+The dedicated LingBot example explicitly selects the video output modality:
+
+```bash
+python examples/offline_inference/text_to_video/text_to_video_lingbot.py \
+  --model robbyant/lingbot-video-dense-1.3b \
+  --height 192 --width 320 --num-frames 9 --num-inference-steps 2 \
+  --output lingbot_t2v.mp4
+```
+
+Use the same command with `robbyant/lingbot-video-moe-30b-a3b` for the MoE
+checkpoint. Requested frame counts are rounded upward to the causal
+VAE's `4n+1` grid.
+
 ### LTX-2
 
 ```bash

examples/offline_inference/text_to_video/text_to_video_lingbot.py

@@ -20,9 +20,9 @@ from vllm_omni.platforms import current_omni_platform
 
 def parse_args() -> argparse.Namespace:
     parser = argparse.ArgumentParser(description="Generate text-to-video with a LingBot-Video checkpoint.")
-    parser.add_argument("--model", default="/home/models/lingbot-video-dense-1.3b")
+    parser.add_argument("--model", default="robbyant/lingbot-video-dense-1.3b")
     parser.add_argument("--prompt", default="a robotic arm picks up a red block")
-    parser.add_argument("--negative-prompt", default="")
+    parser.add_argument("--negative-prompt", default=None)
     parser.add_argument("--output", default="lingbot_video_output.mp4")
     parser.add_argument("--height", type=int, default=192)
     parser.add_argument("--width", type=int, default=320)
@@ -58,9 +58,12 @@ def _to_frame_array(video: Any) -> np.ndarray:
     if isinstance(video, torch.Tensor):
         tensor = video.detach().cpu()
         if tensor.dim() == 5:
-            tensor = tensor[0] if tensor.shape[1] not in (3, 4) else tensor[0].permute(1, 2, 3, 0)
-        elif tensor.dim() == 4 and tensor.shape[0] in (3, 4):
-            tensor = tensor.permute(1, 2, 3, 0)
+            tensor = tensor[0]
+        if tensor.dim() == 4 and tensor.shape[-1] not in (3, 4):
+            if tensor.shape[0] in (3, 4):
+                tensor = tensor.permute(1, 2, 3, 0)
+            elif tensor.shape[1] in (3, 4):
+                tensor = tensor.permute(0, 2, 3, 1)
         if tensor.is_floating_point():
             if float(tensor.min()) < 0.0:
                 tensor = tensor.clamp(-1, 1) * 0.5 + 0.5
@@ -87,8 +90,12 @@ def main() -> None:
         parallel_config=DiffusionParallelConfig(),
     )
 
-    prompt: dict[str, str] = {"prompt": args.prompt}
-    if args.negative_prompt:
+    prompt: dict[str, Any] = {
+        "prompt": args.prompt,
+        "modalities": ["video"],
+        "num_frames": args.num_frames,
+    }
+    if args.negative_prompt is not None:
         prompt["negative_prompt"] = args.negative_prompt
 
     sampling_params = OmniDiffusionSamplingParams(

examples/online_serving/lingbot_video/README.md

@@ -0,0 +1,74 @@
+# LingBot-Video
+
+LingBot-Video uses one `LingBotVideoPipeline` for text-to-image (T2I),
+text-to-video (T2V), and text-image-to-video (TI2V) generation. Both the dense
+and MoE checkpoints use the same request format.
+
+## Start the server
+
+```bash
+MODEL=robbyant/lingbot-video-dense-1.3b bash run_server.sh
+```
+
+The MoE checkpoint uses the same server and request scripts, but requires
+substantially more GPU memory:
+
+```bash
+MODEL=robbyant/lingbot-video-moe-30b-a3b bash run_server.sh
+```
+
+## Text to image
+
+The image endpoint selects T2I mode and always generates one frame:
+
+```bash
+bash run_curl_text_to_image.sh
+```
+
+The script sends a `320x192`, two-step smoke request and writes
+`lingbot_t2i.png`.
+
+## Text or text-image to video
+
+Run the video script without an image to select T2V mode:
+
+```bash
+bash run_curl_text_image_to_video.sh
+```
+
+Pass a first-frame image to the same script to select TI2V mode:
+
+```bash
+INPUT_IMAGE=/path/to/input.png bash run_curl_text_image_to_video.sh
+```
+
+The client scripts omit the optional `model` request field, so they target
+whichever dense or MoE checkpoint the server loaded. The video example uses the
+lightweight `320x192`, 9-frame, two-step configuration.
+
+Until the shared `/v1/videos` reference-image resizing is removed, TI2V target
+dimensions must be sent through `extra_params`, for example
+`{"size":"320x192"}`. Do not use the top-level `size`, `width`, or `height`
+fields for TI2V because the serving layer currently applies those dimensions
+to the reference image before the model receives it. T2V requests can continue
+to use the top-level dimension fields.
+
+LingBot video frame counts use the causal VAE `4n+1` grid. The pipeline rounds
+any requested frame count upward to the next valid value. An explicit
+`num_frames` takes precedence over `seconds`; otherwise, the server first
+resolves `seconds * fps` and the pipeline applies the same alignment.
+
+Official `resolution`/`ratio` presets can be sent through `extra_params`, for
+example `{"resolution":"720p","ratio":"16:9"}`. The `2k` and `4k` entries
+only define output dimensions; whether they run successfully depends on the
+checkpoint, GPU memory, and memory optimizations available in the deployment.
+
+For `/v1/images/generations`, the server resolves these aliases to their final
+pixel dimensions before applying `--max-generated-image-size`. Requests above
+the configured limit return HTTP 400 before engine dispatch. LingBot produces
+one output per prompt; image requests with `n>1` are also rejected with HTTP
+400.
+
+LingBot TI2V accepts exactly one image reference. Image editing, video
+references, audio references, batching, and Refiner execution are not supported
+by this pipeline mode.

examples/online_serving/lingbot_video/run_curl_text_image_to_video.sh

@@ -0,0 +1,67 @@
+#!/bin/bash
+
+set -euo pipefail
+
+if [ -n "${INPUT_IMAGE:-}" ] && [ ! -f "${INPUT_IMAGE}" ]; then
+    echo "INPUT_IMAGE does not exist: ${INPUT_IMAGE}"
+    exit 1
+fi
+
+BASE_URL="${BASE_URL:-http://localhost:8099}"
+POLL_INTERVAL="${POLL_INTERVAL:-2}"
+
+if [ -n "${INPUT_IMAGE:-}" ]; then
+    default_output="lingbot_ti2v.mp4"
+else
+    default_output="lingbot_t2v.mp4"
+fi
+OUTPUT_PATH="${OUTPUT_PATH:-${default_output}}"
+
+curl_args=(
+    -sS -X POST "${BASE_URL}/v1/videos"
+    -H "Accept: application/json"
+    -F "prompt=the subject turns toward the camera with smooth natural motion"
+    -F "num_frames=9"
+    -F "fps=24"
+    -F "num_inference_steps=2"
+    -F "guidance_scale=3.0"
+    -F "flow_shift=3.0"
+    -F "seed=42"
+)
+if [ -n "${INPUT_IMAGE:-}" ]; then
+    curl_args+=(
+        -F "input_reference=@${INPUT_IMAGE}"
+        -F 'extra_params={"size":"320x192"}'
+    )
+else
+    curl_args+=(-F "width=320" -F "height=192")
+fi
+
+create_response="$(curl "${curl_args[@]}")"
+
+video_id="$(echo "${create_response}" | jq -er '.id')"
+echo "Created video job ${video_id}"
+
+while true; do
+    status_response="$(curl -sS "${BASE_URL}/v1/videos/${video_id}")"
+    status="$(echo "${status_response}" | jq -er '.status')"
+    case "${status}" in
+        queued|in_progress)
+            sleep "${POLL_INTERVAL}"
+            ;;
+        completed)
+            break
+            ;;
+        failed)
+            echo "${status_response}" | jq .
+            exit 1
+            ;;
+        *)
+            echo "Unexpected video job status: ${status}"
+            exit 1
+            ;;
+    esac
+done
+
+curl -sS -L "${BASE_URL}/v1/videos/${video_id}/content" -o "${OUTPUT_PATH}"
+echo "Saved video to ${OUTPUT_PATH}"

examples/online_serving/lingbot_video/run_curl_text_to_image.sh

@@ -0,0 +1,20 @@
+#!/bin/bash
+
+set -euo pipefail
+
+BASE_URL="${BASE_URL:-http://localhost:8099}"
+OUTPUT_PATH="${OUTPUT_PATH:-lingbot_t2i.png}"
+
+curl -sS -X POST "${BASE_URL}/v1/images/generations" \
+    -H "Content-Type: application/json" \
+    -d "{
+        \"prompt\": \"a red fox standing in fresh snow\",
+        \"size\": \"320x192\",
+        \"num_inference_steps\": 2,
+        \"guidance_scale\": 3.0,
+        \"flow_shift\": 3.0,
+        \"seed\": 42,
+        \"response_format\": \"b64_json\"
+    }" | jq -er '.data[0].b64_json' | base64 -d > "${OUTPUT_PATH}"
+
+echo "Saved image to ${OUTPUT_PATH}"

examples/online_serving/lingbot_video/run_server.sh

@@ -0,0 +1,10 @@
+#!/bin/bash
+
+set -euo pipefail
+
+MODEL="${MODEL:-robbyant/lingbot-video-dense-1.3b}"
+PORT="${PORT:-8099}"
+
+vllm serve "${MODEL}" --omni \
+    --model-class-name LingBotVideoPipeline \
+    --port "${PORT}"