Compare commits

...
Author SHA1 Message Date
Hongsheng Liu 40284854bb [Model] Minimize LTX-2.5 adapter changes
Signed-off-by: Hongsheng Liu <liuhongsheng4@huawei.com>
2026-08-12 12:33:36 +08:00
Hongsheng Liu 30c72afdda [Model] Address LTX-2.5 review feedback
Signed-off-by: Hongsheng Liu <liuhongsheng4@huawei.com>
2026-08-12 11:58:58 +08:00
Hongsheng Liu 4b541e4d56 [Model] Add LTX-2.5 Diffusers backend support
Signed-off-by: Hongsheng Liu <liuhongsheng4@huawei.com>
2026-08-12 08:58:20 +08:00
5 changed files with 395 additions and 3 deletions

No files matched your search

@@ -28,6 +28,7 @@ However, as we strive to ensure output similarity between vLLM-Omni's diffuser b
- Qwen/Qwen-Image - Qwen/Qwen-Image
- Tongyi-MAI/Z-Image-Turbo - Tongyi-MAI/Z-Image-Turbo
- Wan2.2-I2V-A14B-Diffusers - Wan2.2-I2V-A14B-Diffusers
- Lightricks/LTX-2.5-Diffusers (source-preview dependencies)
If you find that a model not listed above also produces different outputs from running diffusers model directly. If you find that a model not listed above also produces different outputs from running diffusers model directly.
Please consider file an issue. Please consider file an issue.
@@ -116,6 +117,36 @@ The model loading and inference strictly follows the diffusers library, and they
Users are encouraged to double-check the model pipeline's interface in [diffusers' official documentation](https://huggingface.co/docs/diffusers/api/pipelines/overview). Users are encouraged to double-check the model pipeline's interface in [diffusers' official documentation](https://huggingface.co/docs/diffusers/api/pipelines/overview).
Some particular examples are below. Some particular examples are below.
#### LTX-2.5 (source preview)
Use the gated Diffusers checkpoint
[`Lightricks/LTX-2.5-Diffusers`](https://huggingface.co/Lightricks/LTX-2.5-Diffusers);
[`Lightricks/LTX-2.5`](https://huggingface.co/Lightricks/LTX-2.5) contains
component weights and is not directly loadable by this backend. Accept the
model terms, authenticate, and install the source-preview dependencies:
```bash
hf auth login
python -m pip install --upgrade 'transformers>=5.10.1' \
'diffusers @ git+https://github.com/huggingface/diffusers.git@7564fb016dabda0c943416190fc92398c50b1b20'
```
```bash
vllm serve Lightricks/LTX-2.5-Diffusers \
--omni \
--diffusion-load-format diffusers \
--dtype bfloat16 \
--enable-cpu-offload \
--vae-use-tiling
```
The converted repository loads the distilled transformer. When no `sigmas`,
`timesteps`, or `num_inference_steps` is supplied, the adapter applies the
official eight-sigma unguided schedule and returns synchronized 48 kHz audio.
Support is limited to one-stage distilled T2V; `transformer_full`, two-stage
upsampling, image/video conditioning, and the diffusion video decoder are not
supported. The general Diffusers backend [limitations](#limitations) apply.
#### Wan Series #### Wan Series
The Wan series video generation models takes `boundary_ratio` and `flow_shift` during model initialization ([ref](https://huggingface.co/docs/diffusers/api/pipelines/wan)), not during inference. The Wan series video generation models takes `boundary_ratio` and `flow_shift` during model initialization ([ref](https://huggingface.co/docs/diffusers/api/pipelines/wan)), not during inference.
@@ -25,6 +25,7 @@ However, as we strive to ensure output similarity between vLLM-Omni's diffuser b
- Qwen/Qwen-Image - Qwen/Qwen-Image
- Tongyi-MAI/Z-Image-Turbo - Tongyi-MAI/Z-Image-Turbo
- Wan2.2-I2V-A14B-Diffusers - Wan2.2-I2V-A14B-Diffusers
- Lightricks/LTX-2.5-Diffusers (source-preview dependencies)
If you find that a model not listed above also produces different outputs from running diffusers model directly. If you find that a model not listed above also produces different outputs from running diffusers model directly.
Please consider file an issue or submit a PR to fix. Please consider file an issue or submit a PR to fix.
@@ -118,6 +119,36 @@ The model loading and inference strictly follows the diffusers library, and they
Users are encouraged to double-check the model pipeline's interface in [diffusers' official documentation](https://huggingface.co/docs/diffusers/api/pipelines/overview). Users are encouraged to double-check the model pipeline's interface in [diffusers' official documentation](https://huggingface.co/docs/diffusers/api/pipelines/overview).
Some particular examples are below. Some particular examples are below.
#### LTX-2.5 (source preview)
Use the gated Diffusers checkpoint
[`Lightricks/LTX-2.5-Diffusers`](https://huggingface.co/Lightricks/LTX-2.5-Diffusers);
[`Lightricks/LTX-2.5`](https://huggingface.co/Lightricks/LTX-2.5) contains
component weights and is not directly loadable by this backend. Accept the
model terms, authenticate, and install the source-preview dependencies:
```bash
hf auth login
python -m pip install --upgrade 'transformers>=5.10.1' \
'diffusers @ git+https://github.com/huggingface/diffusers.git@7564fb016dabda0c943416190fc92398c50b1b20'
```
```bash
vllm serve Lightricks/LTX-2.5-Diffusers \
--omni \
--diffusion-load-format diffusers \
--dtype bfloat16 \
--enable-cpu-offload \
--vae-use-tiling
```
The converted repository loads the distilled transformer. When no `sigmas`,
`timesteps`, or `num_inference_steps` is supplied, the adapter applies the
official eight-sigma unguided schedule and returns synchronized 48 kHz audio.
Support is limited to one-stage distilled T2V; `transformer_full`, two-stage
upsampling, image/video conditioning, and the diffusion video decoder are not
supported. The general Diffusers backend [limitations](#limitations) apply.
#### Wan Series #### Wan Series
The Wan series video generation models takes `boundary_ratio` and `flow_shift` during model initialization ([ref](https://huggingface.co/docs/diffusers/api/pipelines/wan)), not during inference. The Wan series video generation models takes `boundary_ratio` and `flow_shift` during model initialization ([ref](https://huggingface.co/docs/diffusers/api/pipelines/wan)), not during inference.
@@ -17,6 +17,10 @@ from vllm_omni.diffusion.data import (
OmniDiffusionConfig, OmniDiffusionConfig,
) )
from vllm_omni.diffusion.models.diffusers_adapter import DiffusersAdapterPipeline from vllm_omni.diffusion.models.diffusers_adapter import DiffusersAdapterPipeline
from vllm_omni.diffusion.models.diffusers_adapter.pipeline_utils import (
LTX25_DISTILLED_SIGMAS,
get_pipeline_utils,
)
from vllm_omni.diffusion.request import OmniDiffusionRequest from vllm_omni.diffusion.request import OmniDiffusionRequest
from vllm_omni.diffusion.worker.request_batch import DiffusionRequestBatch from vllm_omni.diffusion.worker.request_batch import DiffusionRequestBatch
from vllm_omni.inputs.data import OmniDiffusionSamplingParams from vllm_omni.inputs.data import OmniDiffusionSamplingParams
@@ -180,6 +184,32 @@ class TestPipelineArgumentsHandling:
with pytest.raises(NotImplementedError): with pytest.raises(NotImplementedError):
DiffusersAdapterPipeline(od_config=od_config) DiffusersAdapterPipeline(od_config=od_config)
def test_ltx25_original_repo_rejected_before_loading(self, mocker):
mock_from_pretrained = mocker.patch(
"vllm_omni.diffusion.models.diffusers_adapter.pipeline_diffusers_adapter.DiffusionPipeline.from_pretrained"
)
adapter = DiffusersAdapterPipeline(od_config=_make_od_config(model="Lightricks/LTX-2.5"))
with pytest.raises(ValueError, match="Lightricks/LTX-2.5-Diffusers"):
adapter.load_weights()
mock_from_pretrained.assert_not_called()
def test_ltx25_requires_upstream_diffusers_feature(self, mocker):
mocker.patch(
"vllm_omni.diffusion.models.diffusers_adapter.pipeline_utils._supports_ltx25_diffusers",
return_value=False,
)
mock_from_pretrained = mocker.patch(
"vllm_omni.diffusion.models.diffusers_adapter.pipeline_diffusers_adapter.DiffusionPipeline.from_pretrained"
)
adapter = DiffusersAdapterPipeline(od_config=_make_od_config(model="Lightricks/LTX-2.5-Diffusers"))
with pytest.raises(ImportError, match="7564fb016dabda0c943416190fc92398c50b1b20"):
adapter.load_weights()
mock_from_pretrained.assert_not_called()
def test_adapter_load_weights_injects_supported_quantization_config(self, mocker): def test_adapter_load_weights_injects_supported_quantization_config(self, mocker):
( (
FakePipelineQuantizationConfig, FakePipelineQuantizationConfig,
@@ -374,6 +404,28 @@ class TestPipelineArgumentsHandling:
assert isinstance(output, DiffusionOutput) assert isinstance(output, DiffusionOutput)
assert output.output == raw_output assert output.output == raw_output
def test_ltx25_preserves_video_and_singular_audio_output(self, tmp_path):
frames = torch.zeros(1, 2, 3, 4, 3)
audio = torch.zeros(1, 48000)
(tmp_path / "model_index.json").write_text(
'{"_class_name":"LTX2Pipeline","duration_head":["diffusers","LTX2DurationHead"]}',
encoding="utf-8",
)
od_config = _make_od_config(model=str(tmp_path))
adapter = DiffusersAdapterPipeline(od_config=od_config)
adapter._pipeline = SimpleNamespace(vocoder=SimpleNamespace(config=SimpleNamespace(output_sampling_rate=48000)))
adapter._pipeline_utils = get_pipeline_utils("LTX2Pipeline")
output = adapter._wrap_output(SimpleNamespace(frames=frames, audio=audio))
assert isinstance(output.output, dict)
assert output.output["payload"]["video"] is frames
assert output.output["payload"]["audio"] is audio
assert output.output["metadata"] == {"audio": {"sample_rate": 48000}}
frames_only = adapter._wrap_output(SimpleNamespace(frames=frames, audio=None))
assert frames_only.output is frames
def test_adapter_build_call_kwargs(self, mocker): def test_adapter_build_call_kwargs(self, mocker):
class MockPipeline: class MockPipeline:
def __call__( def __call__(
@@ -629,6 +681,97 @@ class TestPipelineArgumentsHandling:
with pytest.raises(ValueError): with pytest.raises(ValueError):
pipeline.forward(DiffusionRequestBatch(requests=[problematic_request])) pipeline.forward(DiffusionRequestBatch(requests=[problematic_request]))
def test_ltx25_uses_distilled_sampling_defaults_from_renamed_local_directory(self, mocker, tmp_path):
class LTX2Pipeline:
def __call__(
self,
prompt=None,
num_inference_steps=None,
sigmas=None,
timesteps=None,
guidance_scale=None,
audio_guidance_scale=None,
stg_scale=None,
audio_stg_scale=None,
modality_scale=None,
audio_modality_scale=None,
guidance_rescale=None,
audio_guidance_rescale=None,
num_frames=None,
frame_rate=None,
output_type=None,
generator=None,
):
return None
def to(self, device):
return self
# Local model directories do not retain their Hub repository name.
# Detect 2.5 from its pipeline component metadata instead.
(tmp_path / "model_index.json").write_text(
'{"_class_name":"LTX2Pipeline","duration_head":["diffusers","LTX2DurationHead"]}',
encoding="utf-8",
)
enable_tiling = mocker.Mock()
loaded_pipeline = LTX2Pipeline()
loaded_pipeline.vae = SimpleNamespace(enable_tiling=enable_tiling)
mocker.patch(
"vllm_omni.diffusion.models.diffusers_adapter.pipeline_utils._supports_ltx25_diffusers",
return_value=True,
)
mocker.patch(
"vllm_omni.diffusion.models.diffusers_adapter.pipeline_diffusers_adapter.DiffusionPipeline.from_pretrained",
return_value=loaded_pipeline,
)
adapter = DiffusersAdapterPipeline(
od_config=_make_od_config(
model=str(tmp_path),
vae_use_tiling=True,
)
)
adapter.load_weights()
enable_tiling.assert_called_once_with()
sampling = OmniDiffusionSamplingParams(
num_frames=121,
frame_rate=24.0,
seed=42,
generator_device="cpu",
)
kwargs = adapter._build_call_kwargs(DiffusionRequestBatch(requests=[_make_request(sampling_params=sampling)]))
assert kwargs["sigmas"] == LTX25_DISTILLED_SIGMAS
assert kwargs["num_inference_steps"] == 8
assert kwargs["guidance_scale"] == 1.0
assert kwargs["audio_guidance_scale"] == 1.0
assert kwargs["stg_scale"] == 0.0
assert kwargs["audio_stg_scale"] == 0.0
assert kwargs["modality_scale"] == 1.0
assert kwargs["audio_modality_scale"] == 1.0
assert kwargs["guidance_rescale"] == 0.0
assert kwargs["audio_guidance_rescale"] == 0.0
explicit_steps = OmniDiffusionSamplingParams(num_inference_steps=12, generator_device="cpu")
step_kwargs = adapter._build_call_kwargs(
DiffusionRequestBatch(requests=[_make_request(sampling_params=explicit_steps)])
)
assert step_kwargs["num_inference_steps"] == 12
assert "sigmas" not in step_kwargs
custom_sigmas = [1.0, 0.5]
explicit_schedule = OmniDiffusionSamplingParams(
num_inference_steps=2,
sigmas=custom_sigmas,
generator_device="cpu",
)
schedule_kwargs = adapter._build_call_kwargs(
DiffusionRequestBatch(requests=[_make_request(sampling_params=explicit_schedule)])
)
assert schedule_kwargs["num_inference_steps"] == 2
assert schedule_kwargs["sigmas"] is custom_sigmas
@pytest.mark.advanced_model @pytest.mark.advanced_model
@hardware_test(res={"cuda": "L4"}, num_cards=1) @hardware_test(res={"cuda": "L4"}, num_cards=1)
@@ -23,7 +23,11 @@ from diffusers.pipelines.pipeline_utils import DiffusionPipeline
from torch import nn from torch import nn
from vllm_omni.diffusion.data import DiffusionOutput, OmniDiffusionConfig from vllm_omni.diffusion.data import DiffusionOutput, OmniDiffusionConfig
from vllm_omni.diffusion.models.diffusers_adapter.pipeline_utils import BasePipelineUtils, get_pipeline_utils from vllm_omni.diffusion.models.diffusers_adapter.pipeline_utils import (
BasePipelineUtils,
get_pipeline_utils,
validate_model_compatibility,
)
from vllm_omni.diffusion.models.diffusers_adapter.quantization_utils import ( from vllm_omni.diffusion.models.diffusers_adapter.quantization_utils import (
apply_diffusers_quantization_config, apply_diffusers_quantization_config,
convert_diffusers_quantization_config, convert_diffusers_quantization_config,
@@ -110,6 +114,7 @@ class DiffusersAdapterPipeline(nn.Module, DiffusionPipelineProfilerMixin):
model_id = self.od_config.model model_id = self.od_config.model
dtype = self.od_config.dtype dtype = self.od_config.dtype
validate_model_compatibility(self.od_config)
load_kwargs = { load_kwargs = {
"torch_dtype": dtype, "torch_dtype": dtype,
@@ -130,6 +135,8 @@ class DiffusersAdapterPipeline(nn.Module, DiffusionPipelineProfilerMixin):
logger.debug(f"Loading diffusers pipeline with kwargs: {load_kwargs}") logger.debug(f"Loading diffusers pipeline with kwargs: {load_kwargs}")
self._pipeline = DiffusionPipeline.from_pretrained(model_id, **load_kwargs) self._pipeline = DiffusionPipeline.from_pretrained(model_id, **load_kwargs)
if pipeline_class_name is None:
self._pipeline_utils = get_pipeline_utils(self._pipeline.__class__.__name__)
self._pipeline_utils.apply_post_load_updates(self._pipeline, self.od_config) self._pipeline_utils.apply_post_load_updates(self._pipeline, self.od_config)
self._pipeline.to(self.device) self._pipeline.to(self.device)
@@ -146,14 +153,14 @@ class DiffusersAdapterPipeline(nn.Module, DiffusionPipelineProfilerMixin):
# VAE slicing and tiling: try-catch because not all models have VAE # VAE slicing and tiling: try-catch because not all models have VAE
if self.od_config.vae_use_slicing: if self.od_config.vae_use_slicing:
try: try:
self._pipeline.enable_vae_slicing() self._pipeline_utils.enable_vae_optimization(self._pipeline, "slicing")
except Exception as e: except Exception as e:
logger.warning( logger.warning(
f"Failed to enable VAE slicing for diffusers pipeline {self._pipeline.__class__.__name__}: {e}" f"Failed to enable VAE slicing for diffusers pipeline {self._pipeline.__class__.__name__}: {e}"
) )
if self.od_config.vae_use_tiling: if self.od_config.vae_use_tiling:
try: try:
self._pipeline.enable_vae_tiling() self._pipeline_utils.enable_vae_optimization(self._pipeline, "tiling")
except Exception as e: except Exception as e:
logger.warning( logger.warning(
f"Failed to enable VAE tiling for diffusers pipeline {self._pipeline.__class__.__name__}: {e}" f"Failed to enable VAE tiling for diffusers pipeline {self._pipeline.__class__.__name__}: {e}"
@@ -393,6 +400,8 @@ class DiffusersAdapterPipeline(nn.Module, DiffusionPipelineProfilerMixin):
else: else:
kwargs["generator"] = torch.Generator(device=sampling.generator_device) kwargs["generator"] = torch.Generator(device=sampling.generator_device)
self._pipeline_utils.update_call_kwargs(self.od_config, kwargs)
logger.info( logger.info(
"Calling diffusers pipeline with kwargs: %s", DiffusersAdapterPipeline._summarize_call_kwargs_value(kwargs) "Calling diffusers pipeline with kwargs: %s", DiffusersAdapterPipeline._summarize_call_kwargs_value(kwargs)
) )
@@ -497,6 +506,8 @@ class DiffusersAdapterPipeline(nn.Module, DiffusionPipelineProfilerMixin):
""" """
from vllm_omni.diffusion.data import DiffusionOutput from vllm_omni.diffusion.data import DiffusionOutput
output = self._pipeline_utils.normalize_output(self._pipeline, self.od_config, output)
if hasattr(output, "images"): if hasattr(output, "images"):
# Preserve diffusers image format (`output_type`) # Preserve diffusers image format (`output_type`)
return DiffusionOutput(output=output.images) return DiffusionOutput(output=output.images)
@@ -1,6 +1,10 @@
# SPDX-License-Identifier: Apache-2.0 # SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project # SPDX-FileCopyrightText: Copyright contributors to the vLLM project
import inspect
import json
import logging
from pathlib import Path
from typing import Any from typing import Any
from diffusers.pipelines.pipeline_utils import DiffusionPipeline from diffusers.pipelines.pipeline_utils import DiffusionPipeline
@@ -8,6 +12,74 @@ from diffusers.pipelines.pipeline_utils import DiffusionPipeline
from vllm_omni.diffusion.data import OmniDiffusionConfig from vllm_omni.diffusion.data import OmniDiffusionConfig
from vllm_omni.inputs.data import OmniDiffusionSamplingParams from vllm_omni.inputs.data import OmniDiffusionSamplingParams
logger = logging.getLogger(__name__)
LTX25_ORIGINAL_MODEL_ID = "Lightricks/LTX-2.5"
LTX25_DIFFUSERS_MODEL_ID = "Lightricks/LTX-2.5-Diffusers"
LTX25_DIFFUSERS_COMMIT = "7564fb016dabda0c943416190fc92398c50b1b20"
# Diffusers' public LTX-2.5 schedule intentionally excludes the scheduler's
# terminal zero. FlowMatchEulerDiscreteScheduler appends that internally.
LTX25_DISTILLED_SIGMAS = [1.0, 0.99375, 0.9875, 0.98125, 0.975, 0.909375, 0.725, 0.421875]
_LTX25_DISTILLED_GUIDANCE_DEFAULTS: dict[str, float] = {
"guidance_scale": 1.0,
"audio_guidance_scale": 1.0,
"stg_scale": 0.0,
"audio_stg_scale": 0.0,
"modality_scale": 1.0,
"audio_modality_scale": 1.0,
"guidance_rescale": 0.0,
"audio_guidance_rescale": 0.0,
}
def _supports_ltx25_diffusers() -> bool:
"""Check the standard-pipeline capabilities added with LTX-2.5."""
try:
import diffusers
import transformers
pipeline_class = diffusers.LTX2Pipeline
init_parameters = inspect.signature(pipeline_class.__init__).parameters
return (
"duration_head" in init_parameters
and "prompt_enhancer" in init_parameters
and hasattr(transformers, "Gemma4UnifiedForConditionalGeneration")
)
except (AttributeError, ImportError, RuntimeError):
return False
def is_ltx25_diffusers_model(od_config: OmniDiffusionConfig) -> bool:
"""Return whether config targets the official converted LTX-2.5 repo."""
normalized_model = str(od_config.model).rstrip("/\\").replace("\\", "/").lower()
if normalized_model == LTX25_DIFFUSERS_MODEL_ID.lower():
return True
# A downloaded checkpoint may be served from any directory name. The
# duration-head component is the pipeline-level marker introduced for
# LTX-2.5; older LTX-2.0/2.3 indexes do not contain it.
model_path = Path(str(od_config.model)).expanduser()
subfolder = od_config.diffusers_load_kwargs.get("subfolder")
if subfolder is not None:
model_path /= str(subfolder)
try:
with (model_path / "model_index.json").open(encoding="utf-8") as model_index_file:
model_index = json.load(model_index_file)
except (OSError, TypeError, ValueError):
return False
if not isinstance(model_index, dict):
return False
duration_head = model_index.get("duration_head")
return (
model_index.get("_class_name") == "LTX2Pipeline"
and isinstance(duration_head, list)
and bool(duration_head)
and duration_head[0] is not None
)
class BasePipelineUtils: class BasePipelineUtils:
"""No-op hooks for pipeline-specific diffusers adapter behavior.""" """No-op hooks for pipeline-specific diffusers adapter behavior."""
@@ -18,9 +90,112 @@ class BasePipelineUtils:
def apply_post_load_updates(self, pipeline: DiffusionPipeline, od_config: OmniDiffusionConfig) -> None: def apply_post_load_updates(self, pipeline: DiffusionPipeline, od_config: OmniDiffusionConfig) -> None:
pass pass
def enable_vae_optimization(self, pipeline: DiffusionPipeline, optimization: str) -> None:
getattr(pipeline, f"enable_vae_{optimization}")()
def validate_runtime_sampling_params(self, sampling: OmniDiffusionSamplingParams) -> None: def validate_runtime_sampling_params(self, sampling: OmniDiffusionSamplingParams) -> None:
pass pass
def update_call_kwargs(
self,
od_config: OmniDiffusionConfig,
call_kwargs: dict[str, Any],
) -> None:
pass
def normalize_output(
self,
pipeline: DiffusionPipeline,
od_config: OmniDiffusionConfig,
output: Any,
) -> Any:
return output
def validate_model_compatibility(od_config: OmniDiffusionConfig) -> None:
"""Reject known model/dependency combinations before loading weights."""
normalized_model = str(od_config.model).rstrip("/\\").replace("\\", "/").lower()
if normalized_model == LTX25_ORIGINAL_MODEL_ID.lower():
raise ValueError(
f"{LTX25_ORIGINAL_MODEL_ID} is a component-weight repository without a Diffusers "
f"model_index.json. Use the official converted repository {LTX25_DIFFUSERS_MODEL_ID} "
"with --diffusion-load-format diffusers."
)
if is_ltx25_diffusers_model(od_config) and not _supports_ltx25_diffusers():
raise ImportError(
f"{LTX25_DIFFUSERS_MODEL_ID} requires unreleased LTX-2.5 support from Diffusers and "
"Gemma 4 Unified support from Transformers. Install the source-preview dependencies with "
"`python -m pip install --upgrade 'transformers>=5.10.1' "
f"'diffusers @ git+https://github.com/huggingface/diffusers.git@{LTX25_DIFFUSERS_COMMIT}'`."
)
class LTX2PipelineUtils(BasePipelineUtils):
"""Diffusers call policy for the official default LTX-2.5 checkpoint."""
def enable_vae_optimization(self, pipeline: DiffusionPipeline, optimization: str) -> None:
pipeline_method = getattr(pipeline, f"enable_vae_{optimization}", None)
if callable(pipeline_method):
pipeline_method()
else:
getattr(pipeline.vae, f"enable_{optimization}")()
def update_call_kwargs(
self,
od_config: OmniDiffusionConfig,
call_kwargs: dict[str, Any],
) -> None:
if not is_ltx25_diffusers_model(od_config):
return
has_sigmas = call_kwargs.get("sigmas") is not None
has_timesteps = call_kwargs.get("timesteps") is not None
has_step_count = call_kwargs.get("num_inference_steps") is not None
if not has_sigmas and not has_timesteps:
if has_step_count:
logger.warning(
"LTX-2.5 distilled inference received num_inference_steps without sigmas. "
"Diffusers will use a generic schedule; omit num_inference_steps to use the "
"official distilled schedule."
)
else:
call_kwargs["sigmas"] = list(LTX25_DISTILLED_SIGMAS)
call_kwargs["num_inference_steps"] = len(LTX25_DISTILLED_SIGMAS)
for key, value in _LTX25_DISTILLED_GUIDANCE_DEFAULTS.items():
if call_kwargs.get(key) is None:
call_kwargs[key] = value
def normalize_output(
self,
pipeline: DiffusionPipeline,
od_config: OmniDiffusionConfig,
output: Any,
) -> Any:
if not is_ltx25_diffusers_model(od_config):
return output
frames = getattr(output, "frames", None)
audio = getattr(output, "audio", None)
if frames is None or audio is None:
return output
envelope: dict[str, Any] = {"payload": {"video": frames, "audio": audio}}
config = getattr(getattr(pipeline, "vocoder", None), "config", None)
value = (
config.get("output_sampling_rate")
if hasattr(config, "get")
else getattr(config, "output_sampling_rate", None)
)
try:
sample_rate = None if isinstance(value, bool) else int(value)
except (TypeError, ValueError):
sample_rate = None
if sample_rate is not None and sample_rate > 0:
envelope["metadata"] = {"audio": {"sample_rate": sample_rate}}
return envelope
class WanPipelineUtils(BasePipelineUtils): class WanPipelineUtils(BasePipelineUtils):
def update_load_kwargs(self, od_config: OmniDiffusionConfig, load_kwargs: dict[str, Any]) -> None: def update_load_kwargs(self, od_config: OmniDiffusionConfig, load_kwargs: dict[str, Any]) -> None:
@@ -49,6 +224,7 @@ class WanPipelineUtils(BasePipelineUtils):
PIPELINE_UTILS_REGISTRY: dict[str, type[BasePipelineUtils]] = { PIPELINE_UTILS_REGISTRY: dict[str, type[BasePipelineUtils]] = {
"LTX2Pipeline": LTX2PipelineUtils,
"WanPipeline": WanPipelineUtils, "WanPipeline": WanPipelineUtils,
"WanImageToVideoPipeline": WanPipelineUtils, "WanImageToVideoPipeline": WanPipelineUtils,
"WanVACEPipeline": WanPipelineUtils, "WanVACEPipeline": WanPipelineUtils,