[Model] Adapt Wan2.2-I2V-A14B via LightX2V offline conversion path (#2134)
Signed-off-by: Celeste-jq <591998922@qq.com> Co-authored-by: Canlin Guo <canlinguosdu@gmail.com>
This commit is contained in:
1 parent
cb4d13a658
commit
8097747a5d
14 files changed
+804
-25
No files matched your search
@@ -56,6 +56,92 @@ outputs = omni.generate(
|
||||
!!! note "Server-side Path Requirement"
|
||||
The LoRA adapter path (`local_path`) must be readable on the **server** machine. If your client and server are on different machines, ensure the LoRA adapter is accessible via a shared mount or copied to the server.
|
||||
|
||||
## Wan2.2 LightX2V Offline Assembly
|
||||
|
||||
This workflow is LoRA-adjacent: it uses external LightX2V conversion plus
|
||||
`Wan2.2-Distill-Loras` to bake converted Wan2.2 I2V checkpoints into a local
|
||||
Diffusers directory, instead of loading LoRA adapters at runtime.
|
||||
|
||||
### Required assets
|
||||
|
||||
- Base model: `Wan-AI/Wan2.2-I2V-A14B`
|
||||
- Diffusers skeleton: `Wan-AI/Wan2.2-I2V-A14B-Diffusers`
|
||||
- Optional external converter from the LightX2V project (not shipped in this repository)
|
||||
- Optional LoRA weights: `lightx2v/Wan2.2-Distill-Loras`
|
||||
|
||||
### Step 1: Optional - convert high/low-noise DiT weights with LightX2V
|
||||
|
||||
Install or clone LightX2V from the upstream repository
|
||||
(`https://github.com/ModelTC/LightX2V`). After cloning, the converter used
|
||||
below is available at `<lightx2v_root>/tools/convert/converter.py`.
|
||||
|
||||
```bash
|
||||
python /path/to/lightx2v/tools/convert/converter.py \
|
||||
--source /path/to/Wan2.2-I2V-A14B/high_noise_model \
|
||||
--output /tmp/wan22_lightx2v/high_noise_out \
|
||||
--output_ext .safetensors \
|
||||
--output_name diffusion_pytorch_model \
|
||||
--model_type wan_dit \
|
||||
--direction forward \
|
||||
--lora_path /path/to/wan2.2_i2v_A14b_high_noise_lora_rank64_lightx2v_4step_1022.safetensors \
|
||||
--lora_key_convert auto \
|
||||
--single_file
|
||||
|
||||
python /path/to/lightx2v/tools/convert/converter.py \
|
||||
--source /path/to/Wan2.2-I2V-A14B/low_noise_model \
|
||||
--output /tmp/wan22_lightx2v/low_noise_out \
|
||||
--output_ext .safetensors \
|
||||
--output_name diffusion_pytorch_model \
|
||||
--model_type wan_dit \
|
||||
--direction forward \
|
||||
--lora_path /path/to/wan2.2_i2v_A14b_low_noise_lora_rank64_lightx2v_4step_1022.safetensors \
|
||||
--lora_key_convert auto \
|
||||
--single_file
|
||||
```
|
||||
|
||||
If you are not using LightX2V, skip this step and either keep the original
|
||||
Diffusers weights from the skeleton or point Step 2 at any other converted
|
||||
`transformer/` and `transformer_2/` checkpoints.
|
||||
|
||||
### Step 2: Assemble a final Diffusers-style directory
|
||||
|
||||
```bash
|
||||
python tools/wan22/assemble_wan22_i2v_diffusers.py \
|
||||
--diffusers-skeleton /path/to/Wan2.2-I2V-A14B-Diffusers \
|
||||
--transformer-weight /tmp/wan22_lightx2v/high_noise_out \
|
||||
--transformer-2-weight /tmp/wan22_lightx2v/low_noise_out \
|
||||
--output-dir /path/to/Wan2.2-I2V-A14B-Custom-Diffusers \
|
||||
--asset-mode symlink \
|
||||
--overwrite
|
||||
```
|
||||
|
||||
`--transformer-weight` and `--transformer-2-weight` are optional. If you omit
|
||||
them, the tool keeps the original weights from the Diffusers skeleton.
|
||||
|
||||
### Step 3: Run offline inference
|
||||
|
||||
```bash
|
||||
python examples/offline_inference/image_to_video/image_to_video.py \
|
||||
--model /path/to/Wan2.2-I2V-A14B-Custom-Diffusers \
|
||||
--image /path/to/input.jpg \
|
||||
--prompt "A cat playing with yarn" \
|
||||
--num-frames 81 \
|
||||
--num-inference-steps 4 \
|
||||
--tensor-parallel-size 4 \
|
||||
--height 480 \
|
||||
--width 832 \
|
||||
--flow-shift 12 \
|
||||
--sample-solver euler \
|
||||
--guidance-scale 1.0 \
|
||||
--guidance-scale-high 1.0 \
|
||||
--boundary-ratio 0.875
|
||||
```
|
||||
|
||||
Notes:
|
||||
|
||||
- This route avoids runtime LoRA loading changes in vLLM-Omni when you choose to bake converted weights into a local Diffusers directory.
|
||||
- Output quality and speed depend on the replacement checkpoints and sampling params you choose.
|
||||
|
||||
|
||||
## See Also
|
||||
|
||||
|
||||
@@ -62,12 +62,13 @@ Key arguments:
|
||||
- `--negative-prompt`: Optional list of artifacts to suppress.
|
||||
- `--boundary-ratio`: Boundary split ratio for two-stage MoE models.
|
||||
- `--flow-shift`: Scheduler flow shift (5.0 for 720p, 12.0 for 480p).
|
||||
- `--sample-solver`: Wan2.2 sampling solver. Use `unipc` for the default multistep solver, or `euler` for Lightning/Distill checkpoints.
|
||||
- `--num-inference-steps`: Number of denoising steps (default 50).
|
||||
- `--fps`: Frames per second for the saved MP4 (requires `diffusers` export_to_video).
|
||||
- `--output`: Path to save the generated video.
|
||||
- `--vae-use-slicing`: Enable VAE slicing for memory optimization.
|
||||
- `--vae-use-tiling`: Enable VAE tiling for memory optimization.
|
||||
- `--cfg-parallel-size`: set it to 2 to enable CFG Parallel. See more examples in [`user_guide`](https://github.com/vllm-project/vllm-omni/tree/main/docs/user_guide/diffusion/parallelism_acceleration.md#cfg-parallel).
|
||||
- `--cfg-parallel-size`: set it to 2 to enable CFG Parallel. See more examples in [`user_guide`](https://github.com/vllm-project/vllm-omni/tree/main/docs/user_guide/diffusion/parallelism/cfg_parallel.md).
|
||||
- `--tensor-parallel-size`: tensor parallel size (effective for models that support TP, e.g. LTX2).
|
||||
- `--enable-cpu-offload`: enable CPU offloading for diffusion models.
|
||||
- `--use-hsdp`: Enable Hybrid Sharded Data Parallel to shard model weights across GPUs.
|
||||
@@ -78,6 +79,9 @@ Key arguments:
|
||||
|
||||
> ℹ️ If you encounter OOM errors, try using `--vae-use-slicing` and `--vae-use-tiling` to reduce memory usage.
|
||||
|
||||
For Wan2.2 LightX2V-converted local Diffusers directories and related LoRA
|
||||
assets, see the [LoRA guide](../../diffusion/lora.md#wan22-lightx2v-offline-assembly).
|
||||
|
||||
## Example materials
|
||||
|
||||
??? abstract "image_to_video.py"
|
||||
|
||||
Reference in new issue
Block a user