vllm-omni/recipes/Robbyant/LingBot-World-2.0.md

LingBot-World 2.0

Offline and experimental realtime interactive world generation

Summary

• Vendor: Robbyant
• Model: robbyant/lingbot-world-v2-14b-causal-fast-diffusers
• Task: image-conditioned interactive world generation
• Modes: offline trajectory replay, stepwise one-request streaming, and in-process realtime AR-Diffusion ticks (deprecated)
• Hardware validated: NVIDIA H200 and B200
• Maintainer: Community

The checkpoint is separately licensed under CC BY-NC-SA and restricted to non-commercial use. The vLLM-Omni integration code remains Apache-2.0.

Which path to use

There are three ways to drive this model, each documented in its own section below.

• Realtime stepwise — see Streaming video serving. The suggested path: one WS /v1/realtime/video session produces the whole rollout, one video chunk per AR block. Mid-session camera control uses structural SE3 session.interaction payloads; optional camera_action_script remains for request-scoped WASD scripts.
• Realtime tick (deprecated) — see Realtime in-process generation. Older one-block-per-generate() control plane with JSONL WASD frames; will be removed in the future.
• Offline — see Offline generation. Replays a fixed pose/intrinsics trajectory in one request and writes an MP4. Use it when the camera path is known up front and streaming is not needed.

Offline generation

The offline path consumes one source image and an action directory containing poses.npy with shape [frames, 4, 4] and intrinsics.npy with shape [frames, 4].

python examples/offline_inference/diffusion/lingbot_world_v2.py \
  --prompt "The camera moves slowly forward through the scene." \
  --image /path/to/first_frame.png \
  --action-dir /path/to/actions/forward \
  --num-frames 81 \
  --output lingbot_world_v2.mp4

The current causal-fast checkpoint uses four DMD sampling steps. Raw frame counts must be 9 + 12k for a non-negative integer k; for example, 129 frames produce eleven AR blocks. Both camera files must contain the same number of source frames, at least as many as requested and at most 4096. The runtime consumes only the prefix needed by the request. Longer rollouts also require sufficient device memory.

Experimental KV writeback reuse

To reuse KV from the final denoising step instead of running the additional clean-latent writeback, pass this option in the model_config supplied to Omni or AsyncOmni:

model_config = {"lingbot_reuse_last_step_kv": True}

The default is False. The value must be a boolean, not a string or integer. It is validated and fixed when the pipeline instance is constructed; changing it requires a new instance. It is not a per-request extra_args option. The previous VLLM_OMNI_LINGBOT_REUSE_LAST_STEP_KV environment variable is no longer read.

This experimental mode stores the final noisy probe's KV rather than clean x0 KV, so generated outputs can differ. The four denoising steps, chunk boundaries, and paged-context finalization remain the same.

Realtime in-process generation (deprecated)

Prefer Streaming video serving for mid-session camera control. This tick example still works and emits no runtime warning; removal is tracked as B4 of the LingBot World 2.0 roadmap (#6672).

The realtime example keeps AR-Diffusion KV and model-owned state across requests. Each JSONL line describes the prompt and/or three latent-frame camera actions applied at the next chunk boundary:

{"event_id":1,"prompt":"A road through a forest","frames":[["j"],[],[]]}
{"event_id":2,"frames":[["w"],["w"],["w"]]}
{"event_id":3,"prompt":"The road enters a snowy valley","frames":[[],[],[]]}

Run:

python examples/offline_inference/diffusion/lingbot_world_v2_realtime.py \
  --image /path/to/first_frame.png \
  --events /path/to/events.jsonl \
  --output-dir /tmp/lingbot-realtime \
  --gpu-memory-fraction 0.6

The example writes one latent tensor and one metadata JSON file per chunk. It exercises the same ARDiffusionSessionManager -> ARDiffusionOmniTickConsumer -> AsyncOmni -> ARDiffusionEngine path used by a future HTTP or WebSocket transport.

Realtime serving now goes through the generic WS /v1/realtime/video transport, which exposes no LingBot-specific event fields; the tick loop above stays available only for the older control plane.

Streaming video serving

The stepwise path keeps AR-Diffusion paged KV but issues one request for the whole rollout: prepare_encode runs once, then every AR block is four DMD steps followed by one streamed chunk. Identity metadata uses session_id = request_id with contiguous chunk_index values from zero. Camera motion can be request-scoped (camera_action_script / action_path) or updated mid-session via session.interaction.

Serve it with the AR-Diffusion deploy config, which selects the AR-Diffusion engine and enables streamed step execution:

vllm serve robbyant/lingbot-world-v2-14b-causal-fast-diffusers \
  --omni \
  --deploy-config vllm_omni/deploy/lingbot_world_v2_stepwise.yaml \
  --port 8000

--deploy-config is required: the lingbot_world pipeline deliberately registers no default deploy config, so offline replay and the tick example keep their request-mode topology when no deploy config is given, and the stepwise serving topology is only ever an explicit choice.

Clients then use the generic WebSocket protocol documented in docs/serving/streaming_video_output_api.md: session.start begins one rollout and each AR block arrives as a binary video chunk. The model is image-conditioned, so the first frame is required at image_reference as an http(s) or data: URL; an optional request-scoped pre-scripted list of WASD actions can be provided in extra_params:

{"type": "session.start", "model": "robbyant/lingbot-world-v2-14b-causal-fast-diffusers",
 "prompt": "The camera moves slowly forward through the scene.",
 "image_reference": {"image_url": "data:image/png;base64,<first frame>"},
 "width": 832, "height": 480, "num_frames": 33,
 "extra_params": {"flow_shift": 5.0,
                  "camera_action_script": [[["w"], ["w"], ["w"]], [["a"], [], []], [[], [], []]]}}

camera_action_script must hold exactly one action list per generated chunk, and a request generates ((num_frames - 1) // 4 + 1) // 3 chunks: three for num_frames: 33, seven for num_frames: 81.

Mid-session camera control uses structural SE3 on session.interaction (Unity frame: +X right, +Y up, +Z forward). WASD key tokens belong in clients and are required to be converted before submitting to the service:

{"type": "session.interaction",
 "interaction": {
   "event_id": "forward-1",
   "event": {
     "multi_modal_data": {
       "camera": {
         "mode": "velocity",
         "data": {"translation": [0.0, 0.0, 0.05], "rotation": [0.0, 0.0, 0.0, 1.0]}
       }
     }
   }
 }}

The bundled client helps translate the protocol, converting --camera-updates keystrokes to corresponding SE3 matrices:

python examples/online_serving/streaming_video_generation/streaming_video_client.py \
  --model robbyant/lingbot-world-v2-14b-causal-fast-diffusers \
  --prompt "The camera moves slowly forward through the scene." \
  --image-reference /path/to/first_frame.png \
  --width 832 --height 480 --num-frames 33 --fps 16 --seed 42 \
  --camera-updates '[{"at": 1.0, "actions": ["w"]}, {"at": 3.0, "actions": ["a"]}]' \
  --output lingbot_world_v2_stream.mp4

A served request may instead point at a pose/intrinsics trajectory with extra_params.action_path, which is resolved inside the trusted root set by model_config.lingbot_action_root or VLLM_OMNI_LINGBOT_ACTION_ROOT; a server started without that root configured accepts only camera_action_script or mid-session camera interaction.

Requested width/height must match ar_diffusion_width/ar_diffusion_height in the deploy config, because the AR cache geometry is fixed at load time. Blocks are decoded independently, so seams between chunks are possible; a session-owned streaming decoder is tracked separately.

The deprecated tick example remains available for the older one-block-per-generate() control plane.

Realtime identity and controls

• session_id identifies the persistent world and its worker-owned state.
• event_id identifies a prompt/control update and remains monotonic across reset.
• chunk_index is contiguous from zero and restarts from zero after reset.
• request_id correlates one chunk snapshot with its output metadata.
• AsyncOmni uses a separate UUID-suffixed internal engine routing ID.

The generic runtime transports controls as opaque snapshots. LingBot's adapter accepts:

• lingbot.camera_actions.v1 for per-latent-frame key states such as w, a, s, d, i, j, k, and l;
• lingbot.camera_trajectory.v1 for explicit pose/intrinsics trajectories.

Online FP8 linear layers (experimental)

Pass quantization_config={"method": "fp8"} to the existing AsyncOmni construction to use vLLM's online FP8 linear implementation. On the tested Hopper-class GPU this dispatches to CUTLASS FP8 GEMM with online per-tensor activation scaling; no LingBot-specific quantization kernel is introduced.

Only the transformer's existing vLLM parallel linear layers are eligible: self-attention, cross-attention, FFN, camera injectors, and C2WS projections. The ordinary PyTorch linear layers (including the output head and time/text embeddings), normalization, convolutions, VAE, text encoder, and AR KV cache are not converted to FP8 by this option.

Quality-sensitive projections can remain BF16 through fully qualified ignored_layers names. For example, the following configuration tests retaining the camera/C2WS path in BF16:

quantization_config = {
    "method": "fp8",
    "ignored_layers": [
        f"transformer.blocks.{i}.cam_injector_layer{j}"
        for i in range(40)
        for j in (1, 2)
    ]
    + [
        "transformer.c2ws_hidden_states_layer1",
        "transformer.c2ws_hidden_states_layer2",
    ],
}

Use transformer.blocks.<i>.self_attn.qkv to exclude the fused Q/K/V projection as a unit. The example exclusion list is an ablation, not a universal quality guarantee or a default. Validate the intended scene, seed, camera trajectory, and session length against BF16 before selecting a policy. An FP8 kernel speedup alone is not an end-to-end performance claim. The long-session E8 quality gate in #7074 remains required for Tier-2 acceptance.

Validation

Real-checkpoint validation uses 480x832 output, four DMD steps, and seed 42. The exercised matrix includes:

• TP=1 and TP=2 execution;
• two interleaved resident sessions;
• action input and prompt switching;
• seven contiguous chunks crossing the sink plus recent rolling window;
• direct versus paged replay;
• CUDA Graph execution;
• reset, close, failure cleanup, and exact metadata matching; and
• nine-frame VAE decode.

Official generate.py accuracy and performance numbers, together with generation artifacts, are recorded in the PR validation section for the exact tested commit.

Current limitations

• Only the 14B causal-fast checkpoint is supported.
• The tick control plane is internal; the public transport is the stepwise WS /v1/realtime/video path with structural SE3 mid-session camera interaction (WASD remains a client-side or camera_action_script convenience).
• Stepwise serving requires an explicit --deploy-config vllm_omni/deploy/lingbot_world_v2_stepwise.yaml; there is no default deploy config for this model.
• AR-Diffusion stages currently require one replica because session-affine routing across replicas is not implemented.
• Tick mode generates one AR block per request. Stepwise mode generates many AR blocks in one request. max_num_seqs must be one in both cases.
• Stateful streaming VAE decode is not implemented; the realtime example emits latent chunks.
• SP/USP, pipeline/CFG parallelism, HSDP, VAE parallelism, quantization methods other than online FP8, Cache-DiT, TeaCache, causal-pretrain, and the 1.3B checkpoint are not claimed.
• No AMD GPU, Ascend NPU, or Intel GPU support is claimed.

References

• Checkpoint: https://huggingface.co/robbyant/lingbot-world-v2-14b-causal-fast-diffusers
• Official implementation: https://github.com/robbyant/lingbot-world-v2
• Offline example: examples/offline_inference/diffusion/lingbot_world_v2.py
• Streaming serving client: examples/online_serving/streaming_video_generation/
• Streaming serving end-to-end test: tests/e2e/online_serving/test_lingbot_world_v2_stepwise.py
• Stepwise latent-level end-to-end test: tests/e2e/offline_inference/test_lingbot_world_v2_stepwise.py
• Realtime design: docs/design/feature/realtime_ar_diffusion.md