vllm-omni/recipes/meituan-longcat/LongCat-Video-Avatar-1.5.md

LongCat-Video-Avatar-1.5

Audio-driven avatar video generation (AT2V / AI2V)

Summary

• Vendor: meituan-longcat
• Model: meituan-longcat/LongCat-Video-Avatar-1.5
• Task: Audio-to-video (AT2V) and audio-and-image-to-video (AI2V) avatar generation
• Mode: Offline inference
• Maintainer: Community

When to use this recipe

Use this recipe to drive a talking or singing avatar from a speech clip, either from text alone (AT2V) or from a reference image (AI2V), with the native LongCat-Video-Avatar-1.5 pipeline in vLLM-Omni. It covers:

1. Single-speaker AT2V / AI2V — one 93-frame segment from one audio track.
2. AVC continuation — chained segments that cover a longer audio track.
3. Multi-speaker AI2V — several audio tracks placed on a reference image through speaker bounding boxes.

The example saves an mp4 with the generated frames muxed with the input audio.

References

• Upstream model card: https://huggingface.co/meituan-longcat/LongCat-Video-Avatar-1.5
• Upstream repository and example assets: https://github.com/meituan-longcat/LongCat-Video
• Related example under examples/: longcat_video/end2end.py

Hardware Support

CUDA

1× NVIDIA H100 (80 GB)

Environment

• OS: Linux
• Python: 3.10+
• Driver: NVIDIA driver with CUDA 12.x
• vLLM version: Match the repository requirements for your checkout
• vLLM-Omni version or commit: Use the commit you are deploying from

Prerequisites

Install the LongCat Avatar optional runtime dependencies:

pip install "vllm-omni[longcat-video-avatar]"

The example uses the imageio_ffmpeg executable installed by vLLM-Omni's imageio[ffmpeg] dependency.

Command

Audio-to-video (AT2V):

python examples/offline_inference/longcat_video/end2end.py \
  --model meituan-longcat/LongCat-Video-Avatar-1.5 \
  --stage at2v \
  --audio /path/to/speech.wav \
  --prompt "A person speaks calmly while facing the camera." \
  --num-frames 93 \
  --num-inference-steps 8 \
  --output longcat_avatar_at2v.mp4

Audio-and-image-to-video (AI2V):

python examples/offline_inference/longcat_video/end2end.py \
  --model meituan-longcat/LongCat-Video-Avatar-1.5 \
  --stage ai2v \
  --audio /path/to/speech.wav \
  --image /path/to/reference.jpg \
  --prompt "A person speaks calmly while facing the camera." \
  --num-frames 93 \
  --num-inference-steps 8 \
  --output longcat_avatar_ai2v.mp4

Official Example Assets

Download the official Avatar example assets:

mkdir -p longcat_avatar_assets/assets/avatar/single longcat_avatar_assets/assets/avatar/multi

curl -L -o longcat_avatar_assets/assets/avatar/single/man.mp3 \
  https://raw.githubusercontent.com/meituan-longcat/LongCat-Video/main/assets/avatar/single/man.mp3
curl -L -o longcat_avatar_assets/assets/avatar/single/man.png \
  https://raw.githubusercontent.com/meituan-longcat/LongCat-Video/main/assets/avatar/single/man.png
curl -L -o longcat_avatar_assets/assets/avatar/single_example_1.json \
  https://raw.githubusercontent.com/meituan-longcat/LongCat-Video/main/assets/avatar/single_example_1.json

curl -L -o longcat_avatar_assets/assets/avatar/multi/sing.png \
  https://raw.githubusercontent.com/meituan-longcat/LongCat-Video/main/assets/avatar/multi/sing.png
curl -L -o longcat_avatar_assets/assets/avatar/multi/sing_man.WAV \
  https://raw.githubusercontent.com/meituan-longcat/LongCat-Video/main/assets/avatar/multi/sing_man.WAV
curl -L -o longcat_avatar_assets/assets/avatar/multi/sing_woman.WAV \
  https://raw.githubusercontent.com/meituan-longcat/LongCat-Video/main/assets/avatar/multi/sing_woman.WAV
curl -L -o longcat_avatar_assets/assets/avatar/multi/introduce.png \
  https://raw.githubusercontent.com/meituan-longcat/LongCat-Video/main/assets/avatar/multi/introduce.png
curl -L -o longcat_avatar_assets/assets/avatar/multi/introduce_man.mp3 \
  https://raw.githubusercontent.com/meituan-longcat/LongCat-Video/main/assets/avatar/multi/introduce_man.mp3
curl -L -o longcat_avatar_assets/assets/avatar/multi/introduce_woman.mp3 \
  https://raw.githubusercontent.com/meituan-longcat/LongCat-Video/main/assets/avatar/multi/introduce_woman.mp3
curl -L -o longcat_avatar_assets/assets/avatar/multi_example_1.json \
  https://raw.githubusercontent.com/meituan-longcat/LongCat-Video/main/assets/avatar/multi_example_1.json
curl -L -o longcat_avatar_assets/assets/avatar/multi_example_2.json \
  https://raw.githubusercontent.com/meituan-longcat/LongCat-Video/main/assets/avatar/multi_example_2.json

export LONGCAT_VIDEO_ASSET_DIR="$PWD/longcat_avatar_assets/assets/avatar"
export AVATAR_MODEL=meituan-longcat/LongCat-Video-Avatar-1.5
export PROMPT="$(python -c 'import json, os; print(json.load(open(os.environ["LONGCAT_VIDEO_ASSET_DIR"] + "/single_example_1.json"))["prompt"])')"

--base-model-dir points at a local LongCat-Video base checkpoint for the tokenizer, text encoder and VAE; omit it to download those components from Hugging Face.

export BASE_MODEL_DIR=/cache/longcat_models/LongCat-Video

Official Asset Smoke Test

Audio-to-video with the official audio:

python examples/offline_inference/longcat_video/end2end.py \
  --model "$AVATAR_MODEL" \
  --base-model-dir "$BASE_MODEL_DIR" \
  --stage at2v \
  --audio "$LONGCAT_VIDEO_ASSET_DIR/single/man.mp3" \
  --prompt "$PROMPT" \
  --num-frames 93 \
  --num-inference-steps 8 \
  --output official_asset_longcat_avatar_at2v.mp4

Audio-and-image-to-video with the official audio and image:

python examples/offline_inference/longcat_video/end2end.py \
  --model "$AVATAR_MODEL" \
  --base-model-dir "$BASE_MODEL_DIR" \
  --stage ai2v \
  --audio "$LONGCAT_VIDEO_ASSET_DIR/single/man.mp3" \
  --image "$LONGCAT_VIDEO_ASSET_DIR/single/man.png" \
  --prompt "$PROMPT" \
  --num-frames 93 \
  --num-inference-steps 8 \
  --output official_asset_longcat_avatar_ai2v.mp4

Official Single-Speaker AVC Continuation

--num-segments chains AVC segments; auto covers the full input audio.

python examples/offline_inference/longcat_video/end2end.py \
  --model "$AVATAR_MODEL" \
  --base-model-dir "$BASE_MODEL_DIR" \
  --input-json "$LONGCAT_VIDEO_ASSET_DIR/single_example_1.json" \
  --num-frames 93 \
  --num-cond-frames 13 \
  --num-segments 5 \
  --num-inference-steps 8 \
  --output official_asset_longcat_avatar_single_example_1_ai2v_5seg.mp4

Replace --num-segments 5 with --num-segments auto to run until the full official audio is covered. That is a slow archival validation path rather than a regular smoke test: about 24m28s on a Modal H100 for the full single_example_1.json audio with 8 inference steps.

Official Multi-Speaker AI2V

The multi-speaker cases use several audio tracks with speaker masks and optional bounding boxes. The command below generates the first 93-frame segment, matching the first call to the official generate_ai2v().

python examples/offline_inference/longcat_video/end2end.py \
  --model "$AVATAR_MODEL" \
  --base-model-dir "$BASE_MODEL_DIR" \
  --input-json "$LONGCAT_VIDEO_ASSET_DIR/multi_example_1.json" \
  --num-frames 93 \
  --num-inference-steps 8 \
  --output official_asset_longcat_avatar_multi_example_1_ai2v.mp4

Use multi_example_2.json for the sequential-speaking case, and add --num-cond-frames 13 --num-segments auto to cover the full audio with AVC continuation.

Verification

ffprobe -hide_banner longcat_avatar_ai2v.mp4

The output must report one video stream and one audio stream: the example muxes the generated frames with the input audio.

Notes

Support matrix

Mode	Speakers	Image required	AVC continuation
AT2V	Single	No	Yes
AI2V	Single	Yes	Yes
AT2V	Multi	Not supported	Not supported
AI2V	Multi	Yes	Yes

Multi-speaker Avatar requires AI2V because speaker bounding boxes and masks are defined on a reference image; AT2V with several tracks fails with an explicit error. AVC continuation uses full-audio embeddings only when --num-segments is greater than 1 or set to auto; single-segment AT2V/AI2V keeps the shorter first-segment audio path.

Sampling parameters

Parameter	LongCat pipeline value	Description
--num-frames	num_frames	Frames generated per segment. The default is 93 frames, or about 3.72 seconds at 25 FPS. LongCat normalizes other values to the temporal-VAE-compatible 4k+1 form.
--num-inference-steps	steps	Denoising steps. The distilled Avatar path uses 8 steps.
--fps	save_fps	Output frame rate, and the rate used to align audio embeddings with video frames. The default is 25 FPS.
sampling_params.guidance_scale	text_guidance_scale	Classifier-free guidance applied to the text prompt.
sampling_params.guidance_scale_2	audio_guidance_scale	Classifier-free guidance applied to the speech audio.

With the distilled LoRA enabled (the default), the pipeline uses 8 inference steps and sets both guidance scales to 1.0. With --no-use-distill, the example sets both to 4.0. For AVC continuation every segment holds num_frames frames, but each segment after the first reuses num_cond_frames conditioning frames, so the unique frame count is num_frames + (num_segments - 1) * (num_frames - num_cond_frames).

Key flags

• --stage: at2v for audio-to-video, ai2v for audio-and-image-to-video.
• --resolution: 480p or 720p.
• --input-json: official LongCat Avatar JSON case, used for the single-speaker and multi-speaker AI2V examples.
• --num-segments: number of AVC segments, or auto to cover the full audio.
• --num-cond-frames: previous-frame conditioning window for AVC continuation.
• --use-int8 / --no-use-int8: load the official INT8 Avatar DiT weights by default, or full precision weights.
• --use-distill / --no-use-distill: enable the official distilled LoRA path by default.
• --build-components-on-gpu: build large Avatar components directly on GPU for faster startup, at a higher peak VRAM cost. Disabled by default.
• --base-model-dir: local LongCat-Video base model directory for the tokenizer, text encoder and VAE components.

Memory usage

Measured on a Modal H100 running the official 93-frame AI2V example with the INT8 DiT, distilled LoRA and 8 inference steps:

Component build	CLI option	Peak GPU memory	Model loading time
CPU (default)	None	About 41.0 GiB	About 130s
GPU	--build-components-on-gpu	About 56.8 GiB	About 33s

The reported peak covers the end-to-end run. Building components on CPU only reduces initialization memory; denoising still runs on GPU. These numbers are memory guidance rather than a performance benchmark.

Known limitations

• Single GPU only. Cache acceleration, sequence/CFG/tensor parallelism, HSDP, CPU offload, VAE patch parallelism and quantization are not wired up for this pipeline yet; see the diffusion feature matrix in docs/user_guide/diffusion_features.md.
• Offline inference only. Online serving is not supported yet.