[Feature][MiniMax-H3] Support the full LightX2V Turbo LoRA matrix (#7062)
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
1 parent
039808e0d9
commit
af74a5a155
10 files changed
+557
-141
No files matched your search
@@ -249,8 +249,14 @@ MiniMax-H3 supports two few-step mechanisms that must not be conflated:
|
||||
- **Checkpoint-pinned schedule**: a merged release writes `base_schedule` into
|
||||
`model_index.json`. Requests must pass `num_inference_steps` as the interval
|
||||
count (for example `4` for `[1.0, 0.7, 0.4, 0.15, 0.0]`).
|
||||
- **Runtime Turbo LoRA**: LightX2V Turbo artifacts request five sigma points and
|
||||
enforce `flow_shift=6`, `audio_flow_shift=3`.
|
||||
- **Runtime Turbo LoRA**: each LightX2V Turbo artifact carries its own contract,
|
||||
read from its filename. A four-step artifact requests five sigma points and an
|
||||
eight-step one requests nine; the 768p retrains enforce `flow_shift=6` and the
|
||||
544p artifacts `flow_shift=12`. `audio_flow_shift=3` across the family. A
|
||||
request that does not match the loaded artifact is rejected by name. Alpha
|
||||
comes from the artifact's metadata, or 8 when it declares none. `ref2v`
|
||||
artifacts require `--task-type ref2va`; `fl2v` artifacts serve T2VA and FL2VA.
|
||||
The ComfyUI exports use a fused-QKV layout and are not supported.
|
||||
- **Runtime native LoRA**: FlashGen-style artifacts declare
|
||||
`key_format=minimax-h3-native` and embed `base_schedule` in safetensors
|
||||
metadata. When active, the adapter schedule overrides the base checkpoint and
|
||||
|
||||
@@ -68,16 +68,24 @@ The `/v1/videos` request schema and `extra_params.task` values (`t2va`,
|
||||
|
||||
## Turbo LoRA
|
||||
|
||||
MiniMax-H3 Turbo uses five sigma points, `flow_shift=6`, and
|
||||
`audio_flow_shift=3`. Start Turbo deployments with the dedicated defaults so
|
||||
requests that omit sampling controls do not inherit the 50-step base schedule:
|
||||
The deploy config below carries the four-step 768p contract -- five sigma
|
||||
points, `flow_shift=6`, `audio_flow_shift=3` -- so requests that omit sampling
|
||||
controls do not inherit the 50-step base schedule:
|
||||
|
||||
```bash
|
||||
vllm-omni serve MiniMaxAI/MiniMax-H3 \
|
||||
--omni \
|
||||
--lora-path /path/to/MiniMax-H3-Turbo \
|
||||
--lora-path /path/to/minimax_h3_fl2v_turbo_4step_v1.0_768p_bf16.safetensors \
|
||||
--deploy-config vllm_omni/deploy/minimax_h3_disaggregated_turbo.yaml
|
||||
```
|
||||
|
||||
`--lora-path` must name one artifact, or a directory holding exactly one; the
|
||||
Turbo repository holds several and a directory of them is rejected as
|
||||
ambiguous. Serving a different artifact means overriding those defaults with
|
||||
that artifact's own contract -- see the
|
||||
[Turbo LoRA table](MiniMax-H3.md#turbo-lora) for every published row.
|
||||
|
||||
The base deployment intentionally retains 50 inference steps for non-LoRA
|
||||
quality. Turbo LoRA supports T2VA and FL2VA requests, not Ref2VA.
|
||||
quality. This disaggregated topology is validated on the `fl2v` artifacts,
|
||||
which serve T2VA and FL2VA; the `ref2v` artifacts need a `--task-type ref2va`
|
||||
server.
|
||||
@@ -869,13 +869,51 @@ balanced switch order.
|
||||
|
||||
### Turbo LoRA
|
||||
|
||||
Only the native Diffusers 4-step FL2VA/T2VA v1.0 artifact is supported:
|
||||
The eight Diffusers-layout LightX2V Turbo artifacts are supported. The
|
||||
filename records the contract, so the server reads the step count, task family
|
||||
and flow shift from it and validates each request against the artifact that is
|
||||
loaded. It does not rewrite request or deploy-config sampling values: the
|
||||
request must carry that artifact's own settings, listed here, or it is
|
||||
rejected.
|
||||
|
||||
```text
|
||||
lightx2v/Minimax-h3-Turbo/minimax_h3_fl2v_turbo_4step_v1.0_768p_bf16.safetensors
|
||||
```
|
||||
| Artifact | Task | Forwards | `num_inference_steps` | `flow_shift` | declared `alpha` |
|
||||
| --- | --- | ---: | ---: | ---: | ---: |
|
||||
| `minimax_h3_fl2v_turbo_4step_v0.1.safetensors` | T2VA / FL2VA | 4 | 5 | 12 | none -> 8 |
|
||||
| `minimax_h3_fl2v_turbo_4step_v1.0_768p_bf16.safetensors` | T2VA / FL2VA | 4 | 5 | 6 | 128 |
|
||||
| `minimax_h3_fl2v_turbo_4step_v1.1_768p_bf16.safetensors` | T2VA / FL2VA | 4 | 5 | 6 | 128 |
|
||||
| `minimax_h3_fl2v_turbo_4step_v1.2_768p_bf16.safetensors` | T2VA / FL2VA | 4 | 5 | 6 | 8 |
|
||||
| `minimax_h3_fl2v_turbo_8step_v1.0_bf16.safetensors` | T2VA / FL2VA | 8 | 9 | 12 | 8 |
|
||||
| `minimax_h3_fl2v_turbo_8step_v1.0_768p_bf16.safetensors` | T2VA / FL2VA | 8 | 9 | 6 | 8 |
|
||||
| `minimax_h3_ref2v_turbo_4step_v0.1_bf16.safetensors` | Ref2VA | 4 | 5 | 12 | 8 |
|
||||
| `minimax_h3_ref2v_turbo_8step_v1.0_768p_bf16.safetensors` | Ref2VA | 8 | 9 | 6 | 8 |
|
||||
|
||||
Download only that file:
|
||||
`audio_flow_shift` is `3.0` across the family. Each row is the complete
|
||||
published filename; use it verbatim as `TURBO_FILE` below.
|
||||
|
||||
Alpha needs no manual compensation: the server reads it from the artifact's
|
||||
metadata, falling back to 8 with a warning for
|
||||
`minimax_h3_fl2v_turbo_4step_v0.1`, the one artifact that declares none. The
|
||||
request-level `scale` is a further multiplier on top of it.
|
||||
|
||||
> [!NOTE]
|
||||
> Every artifact is rank 128 and the delta is applied at `scale * alpha /
|
||||
> rank`, so the `alpha=8` rows drive at 1/16 the strength of the `alpha=128`
|
||||
> rows. 8 is the default of LightX2V's reference script, which never reads the
|
||||
> metadata; its documented `v0.1` command does not override that default.
|
||||
|
||||
Every artifact except `minimax_h3_fl2v_turbo_4step_v0.1` also ships a
|
||||
`_comfyui_` export of the same weights. Those fuse Q/K/V into one projection
|
||||
and are **not** supported; downloading one is refused by name. Take the
|
||||
Diffusers file. The filename is the contract, so do not rename an artifact
|
||||
either -- a renamed file is rejected rather than served on a guess.
|
||||
|
||||
FL2VA artifacts serve `t2va` and `fl2va` on any FL2VA or combined server.
|
||||
Ref2VA artifacts require
|
||||
`--task-type ref2va`: a combined server serves `ref2va` from a second DiT that
|
||||
the adapter cannot bind to, so loading one there is refused rather than silently
|
||||
running an undistilled model on the few-step schedule.
|
||||
|
||||
Download the artifact you want:
|
||||
|
||||
```bash
|
||||
export TURBO_DIR=/path/to/minimax-h3-turbo
|
||||
@@ -884,10 +922,19 @@ hf download lightx2v/Minimax-h3-Turbo "${TURBO_FILE}" --local-dir "${TURBO_DIR}"
|
||||
export TURBO_LORA="${TURBO_DIR}/${TURBO_FILE}"
|
||||
```
|
||||
|
||||
`--lora-path` accepts one artifact, or a directory holding exactly one.
|
||||
|
||||
> [!IMPORTANT]
|
||||
> This changes earlier behaviour. `--lora-path /path/to/minimax-h3-turbo`
|
||||
> pointing at a full clone of the Turbo repository used to select the v1.0 768p
|
||||
> file implicitly; a directory holding several recognized artifacts is now
|
||||
> rejected as ambiguous. Name the artifact instead:
|
||||
> `--lora-path /path/to/minimax-h3-turbo/minimax_h3_fl2v_turbo_4step_v1.0_768p_bf16.safetensors`.
|
||||
|
||||
Start from a non-offloaded or DLO FL2VA server command and add
|
||||
`--task-type fl2va --lora-backend peft --lora-path "${TURBO_LORA}"`.
|
||||
`--lora-path` preloads the adapter; each request still activates it and uses
|
||||
the published sampling settings:
|
||||
`--lora-path` preloads the adapter; each request still activates it and must
|
||||
carry that artifact's sampling settings:
|
||||
|
||||
```bash
|
||||
-F 'num_inference_steps=5' \
|
||||
@@ -896,14 +943,25 @@ the published sampling settings:
|
||||
-F "lora={\"name\":\"h3-turbo-v1.0\",\"path\":\"${TURBO_LORA}\",\"scale\":1.0}"
|
||||
```
|
||||
|
||||
For FL2VA, change `task` and add `input_reference` as shown above. The 8-step,
|
||||
ComfyUI, Ref2VA, and v1.1 artifacts are not supported. This integration is
|
||||
dynamic-only and does not support prefusion or LoRA composition. DLO is
|
||||
Switching to another FL2VA artifact means repointing `TURBO_FILE`, which moves
|
||||
both `--lora-path` and the request's `lora.path`, and carrying that row's
|
||||
`num_inference_steps` and `flow_shift`: `9` and `6` for `8step_v1.0_768p`, `9`
|
||||
and `12` for the 544p `8step_v1.0`. A request that does not match the loaded
|
||||
artifact is rejected, so a mismatch cannot silently degrade output.
|
||||
|
||||
The two `ref2v` rows are not served by this FL2VA command. Start a
|
||||
`--task-type ref2va` server, take a request from
|
||||
[Ref2VA](#3-ref2va-image-only-imageaudio-or-mixed-references) and override
|
||||
`num_inference_steps` and `flow_shift` with that row's values; those examples
|
||||
already send `audio_flow_shift=3.0`.
|
||||
|
||||
For FL2VA, change `task` and add `input_reference` as shown above. This
|
||||
integration is dynamic-only and does not support prefusion or LoRA composition. DLO is
|
||||
supported by keeping the request-switchable LoRA A/B buffers resident on the
|
||||
accelerator while DLO streams only the base blocks; budget for this additional
|
||||
fixed HBM usage. Model-level and standard layerwise offload remain unsupported.
|
||||
The five requested sigma points produce the four denoiser evaluations expected
|
||||
by the Turbo artifact.
|
||||
The requested sigma points always number one more than the artifact's denoiser
|
||||
evaluations.
|
||||
|
||||
### FlashGen native LoRA
|
||||
|
||||
|
||||
@@ -13,6 +13,8 @@ import pytest
|
||||
import torch
|
||||
import torch.nn as nn
|
||||
|
||||
from vllm_omni.diffusion.models.minimax_h3.lora import TurboSpec, parse_turbo_filename
|
||||
|
||||
pytestmark = [pytest.mark.core_model, pytest.mark.cpu, pytest.mark.diffusion]
|
||||
|
||||
_ItemT = TypeVar("_ItemT")
|
||||
@@ -24,6 +26,13 @@ def _append_and_return(items: list[_ItemT], item: _ItemT, result: _ResultT) -> _
|
||||
return result
|
||||
|
||||
|
||||
def _turbo_spec(filename: str) -> TurboSpec:
|
||||
"""Build the spec a loaded artifact of this name would carry."""
|
||||
spec = parse_turbo_filename(filename)
|
||||
assert spec is not None
|
||||
return spec
|
||||
|
||||
|
||||
def test_h3_prepares_resolved_cache_state_immediately_before_denoise():
|
||||
from vllm_omni.diffusion.models.minimax_h3 import MiniMaxH3Pipeline
|
||||
from vllm_omni.diffusion.request import OmniDiffusionRequest
|
||||
@@ -244,9 +253,12 @@ def test_combined_task_inference_and_transformer_routing():
|
||||
assert pipeline._resolve_task(None, {"image": object()}) == "fl2va"
|
||||
assert pipeline._resolve_task(None, {"audio": object()}) == "ref2va"
|
||||
assert pipeline._resolve_task(None, {"video": object()}) == "ref2va"
|
||||
assert pipeline._resolve_task("fl2va", {"image": object()}, has_turbo_lora=True) == "fl2va"
|
||||
with pytest.raises(OmniClientError, match="supports T2VA/FL2VA requests only"):
|
||||
pipeline._resolve_task("ref2va", {}, has_turbo_lora=True)
|
||||
fl2v_spec = _turbo_spec("minimax_h3_fl2v_turbo_4step_v1.0_768p_bf16.safetensors")
|
||||
ref2v_spec = _turbo_spec("minimax_h3_ref2v_turbo_8step_v1.0_768p_bf16.safetensors")
|
||||
assert pipeline._resolve_task("fl2va", {"image": object()}, turbo_spec=fl2v_spec) == "fl2va"
|
||||
with pytest.raises(OmniClientError, match="serves"):
|
||||
pipeline._resolve_task("ref2va", {}, turbo_spec=fl2v_spec)
|
||||
assert pipeline._resolve_task("ref2va", {"image": object()}, turbo_spec=ref2v_spec) == "ref2va"
|
||||
pipeline.partition = "ref2va"
|
||||
pipeline.supported_tasks = frozenset({"ref2va"})
|
||||
assert pipeline._resolve_task(None, {"image": object()}) == "ref2va"
|
||||
|
||||
@@ -303,7 +303,7 @@ def _pipeline_stub(fusion=None):
|
||||
torch.nn.Module.__init__(pipeline)
|
||||
pipeline.partition = "combined"
|
||||
pipeline.supported_tasks = frozenset({"t2va", "fl2va", "ref2va"})
|
||||
pipeline._turbo_lora_adapter_ids = set()
|
||||
pipeline._turbo_lora_specs = {}
|
||||
pipeline._fasth3 = fusion
|
||||
pipeline.od_config = SimpleNamespace()
|
||||
pipeline.default_video_shift, pipeline.default_audio_shift = 12.0, 3.0
|
||||
|
||||
@@ -13,7 +13,9 @@ from vllm.lora.lora_weights import PackedLoRALayerWeights
|
||||
from vllm_omni.diffusion.lora.manager import DiffusionLoRAManager
|
||||
from vllm_omni.diffusion.models.minimax_h3 import lora as lora_module
|
||||
from vllm_omni.diffusion.models.minimax_h3.lora import (
|
||||
TurboSpec,
|
||||
load_minimax_h3_turbo_lora,
|
||||
parse_turbo_filename,
|
||||
)
|
||||
from vllm_omni.errors import OmniClientError
|
||||
from vllm_omni.lora.request import LoRARequest
|
||||
@@ -46,7 +48,7 @@ def _request(path) -> LoRARequest:
|
||||
def _write_tiny_turbo(
|
||||
path,
|
||||
*,
|
||||
alpha: str = "128",
|
||||
alpha: str | None = "128",
|
||||
key_format: str = "minimax-h3-diffusers",
|
||||
omit_target: str | None = None,
|
||||
shape_overrides: dict[str, tuple[int, int]] | None = None,
|
||||
@@ -88,11 +90,17 @@ def _write_tiny_turbo(
|
||||
)
|
||||
else:
|
||||
tensors[b_name] = torch.ones(overrides.get(b_name, (output_dim, rank)))
|
||||
save_file(
|
||||
tensors,
|
||||
str(path),
|
||||
metadata={"alpha": alpha, "key_format": key_format},
|
||||
)
|
||||
metadata = {"key_format": key_format}
|
||||
if alpha is not None:
|
||||
metadata["alpha"] = alpha
|
||||
save_file(tensors, str(path), metadata=metadata)
|
||||
|
||||
|
||||
def _spec(filename: str = "minimax_h3_fl2v_turbo_4step_v1.0_768p_bf16.safetensors") -> TurboSpec:
|
||||
"""Build the spec a loaded artifact of this name would carry."""
|
||||
spec = parse_turbo_filename(filename)
|
||||
assert spec is not None
|
||||
return spec
|
||||
|
||||
|
||||
def test_h3_turbo_loads_through_legacy_lora_model_and_swaps_ffn(tmp_path):
|
||||
@@ -107,9 +115,12 @@ def test_h3_turbo_loads_through_legacy_lora_model_and_swaps_ffn(tmp_path):
|
||||
)
|
||||
|
||||
assert loaded is not None
|
||||
lora_model, peft_helper = loaded
|
||||
lora_model, peft_helper, spec = loaded
|
||||
assert peft_helper.r == 128
|
||||
assert peft_helper.lora_alpha == 128
|
||||
assert spec.denoise_steps == 4
|
||||
assert spec.sigma_points == 5
|
||||
assert spec.task_family == "fl2v"
|
||||
assert len(lora_model.loras) == 312
|
||||
assert "blocks.0.attn.to_q" in lora_model.loras
|
||||
assert "blocks.0.mlp.fc1" in lora_model.loras
|
||||
@@ -152,7 +163,8 @@ def test_legacy_manager_uses_the_h3_model_loader_without_changing_its_interface(
|
||||
|
||||
class _Pipeline:
|
||||
def _load_diffusion_lora_adapter(self, **kwargs):
|
||||
return load_minimax_h3_turbo_lora(partition="fl2va", **kwargs)
|
||||
lora_model, peft_helper, _spec_unused = load_minimax_h3_turbo_lora(partition="fl2va", **kwargs)
|
||||
return lora_model, peft_helper
|
||||
|
||||
manager = object.__new__(DiffusionLoRAManager)
|
||||
manager.pipeline = _Pipeline()
|
||||
@@ -168,22 +180,30 @@ def test_legacy_manager_uses_the_h3_model_loader_without_changing_its_interface(
|
||||
assert "blocks.0.mlp.fc1" in lora_model.loras
|
||||
|
||||
|
||||
def test_h3_turbo_rejects_wrong_alpha_and_ref2va(tmp_path):
|
||||
wrong_alpha = tmp_path / "wrong_alpha" / "minimax_h3_fl2v_turbo_4step_v1.0_768p_bf16.safetensors"
|
||||
wrong_alpha.parent.mkdir()
|
||||
_write_tiny_turbo(wrong_alpha, alpha="8")
|
||||
with pytest.raises(ValueError, match="requires alpha=128"):
|
||||
load_minimax_h3_turbo_lora(
|
||||
partition="fl2va",
|
||||
lora_request=_request(wrong_alpha),
|
||||
lora_path=wrong_alpha,
|
||||
dtype=torch.float32,
|
||||
)
|
||||
def test_h3_turbo_honours_the_declared_alpha_and_task_family(tmp_path):
|
||||
"""Alpha comes from the artifact; the family decides which server serves it."""
|
||||
|
||||
eight_step_alpha = tmp_path / "alpha8" / "minimax_h3_fl2v_turbo_8step_v1.0_768p_bf16.safetensors"
|
||||
eight_step_alpha.parent.mkdir()
|
||||
_write_tiny_turbo(eight_step_alpha, alpha="8")
|
||||
loaded = load_minimax_h3_turbo_lora(
|
||||
partition="fl2va",
|
||||
lora_request=_request(eight_step_alpha),
|
||||
lora_path=eight_step_alpha,
|
||||
dtype=torch.float32,
|
||||
)
|
||||
assert loaded is not None
|
||||
_, peft_helper, spec = loaded
|
||||
# The eight-step artifact is a 1/16-strength adapter; pinning alpha to the
|
||||
# four-step value would over-drive it by 16x.
|
||||
assert peft_helper.lora_alpha == 8
|
||||
assert spec.alpha == 8
|
||||
assert spec.sigma_points == 9
|
||||
|
||||
valid = tmp_path / "valid" / "minimax_h3_fl2v_turbo_4step_v1.0_768p_bf16.safetensors"
|
||||
valid.parent.mkdir()
|
||||
_write_tiny_turbo(valid)
|
||||
with pytest.raises(ValueError, match="supports FL2VA/T2VA only"):
|
||||
with pytest.raises(ValueError, match="Ref2VA-only server cannot serve it"):
|
||||
load_minimax_h3_turbo_lora(
|
||||
partition="ref2va",
|
||||
lora_request=_request(valid),
|
||||
@@ -192,6 +212,43 @@ def test_h3_turbo_rejects_wrong_alpha_and_ref2va(tmp_path):
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("partition", ["fl2va", "combined"])
|
||||
def test_h3_turbo_ref2v_artifact_needs_a_ref2va_only_server(tmp_path, partition):
|
||||
"""A combined server routes ref2va to ``transformers_ref``, which the
|
||||
adapter cannot bind to: the LoRA target pattern only injects into
|
||||
``transformer``. Admitting it there would run an undistilled DiT on the
|
||||
artifact's few-step schedule with no error anywhere."""
|
||||
|
||||
path = tmp_path / "minimax_h3_ref2v_turbo_8step_v1.0_768p_bf16.safetensors"
|
||||
_write_tiny_turbo(path)
|
||||
|
||||
with pytest.raises(ValueError, match="start the server with --task-type ref2va"):
|
||||
load_minimax_h3_turbo_lora(
|
||||
partition=partition,
|
||||
lora_request=_request(path),
|
||||
lora_path=path,
|
||||
dtype=torch.float32,
|
||||
)
|
||||
|
||||
|
||||
def test_h3_turbo_ref2v_artifact_loads_on_a_ref2va_server(tmp_path):
|
||||
path = tmp_path / "minimax_h3_ref2v_turbo_8step_v1.0_768p_bf16.safetensors"
|
||||
_write_tiny_turbo(path)
|
||||
|
||||
loaded = load_minimax_h3_turbo_lora(
|
||||
partition="ref2va",
|
||||
lora_request=_request(path),
|
||||
lora_path=path,
|
||||
dtype=torch.float32,
|
||||
)
|
||||
|
||||
assert loaded is not None
|
||||
spec = loaded[2]
|
||||
assert spec.task_family == "ref2v"
|
||||
assert spec.supported_tasks == frozenset({"ref2va"})
|
||||
assert spec.sigma_points == 9
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"offload_mode",
|
||||
[
|
||||
@@ -214,6 +271,8 @@ def test_h3_turbo_rejects_offload_modes(tmp_path, offload_mode):
|
||||
|
||||
|
||||
def test_h3_turbo_allows_distributed_layerwise_offload(monkeypatch):
|
||||
"""DLO keeps the LoRA buffers resident, so it is not an unsupported mode."""
|
||||
|
||||
from vllm_omni.diffusion.models.minimax_h3 import MiniMaxH3Pipeline
|
||||
from vllm_omni.diffusion.models.minimax_h3 import (
|
||||
pipeline_minimax_h3 as pipeline_module,
|
||||
@@ -222,7 +281,7 @@ def test_h3_turbo_allows_distributed_layerwise_offload(monkeypatch):
|
||||
pipeline = object.__new__(MiniMaxH3Pipeline)
|
||||
torch.nn.Module.__init__(pipeline)
|
||||
pipeline.partition = "fl2va"
|
||||
pipeline._turbo_lora_adapter_ids = set()
|
||||
pipeline._turbo_lora_specs = {}
|
||||
pipeline._native_lora_adapter_ids = set()
|
||||
pipeline._lora_sigma_schedules = {}
|
||||
pipeline.od_config = SimpleNamespace(
|
||||
@@ -231,10 +290,11 @@ def test_h3_turbo_allows_distributed_layerwise_offload(monkeypatch):
|
||||
enable_distributed_layerwise_offload=True,
|
||||
)
|
||||
captured = {}
|
||||
spec = _spec()
|
||||
|
||||
def load_turbo(**kwargs):
|
||||
captured.update(kwargs)
|
||||
return object(), object()
|
||||
return object(), object(), spec
|
||||
|
||||
monkeypatch.setattr(pipeline_module, "load_minimax_h3_turbo_lora", load_turbo)
|
||||
loaded = pipeline._load_diffusion_lora_adapter(
|
||||
@@ -245,6 +305,10 @@ def test_h3_turbo_allows_distributed_layerwise_offload(monkeypatch):
|
||||
|
||||
assert loaded is not None
|
||||
assert captured["unsupported_offload_mode"] is None
|
||||
# The pipeline passes its own partition through and keeps the spec that
|
||||
# decides the request contract.
|
||||
assert captured["partition"] == "fl2va"
|
||||
assert pipeline._turbo_lora_specs == {1: spec}
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
@@ -268,7 +332,7 @@ def test_h3_turbo_offload_compatibility_follows_dit_selection(
|
||||
pipeline = object.__new__(MiniMaxH3Pipeline)
|
||||
torch.nn.Module.__init__(pipeline)
|
||||
pipeline.partition = "fl2va"
|
||||
pipeline._turbo_lora_adapter_ids = set()
|
||||
pipeline._turbo_lora_specs = {}
|
||||
pipeline._native_lora_adapter_ids = set()
|
||||
pipeline._lora_sigma_schedules = {}
|
||||
pipeline.od_config = SimpleNamespace(
|
||||
@@ -281,10 +345,11 @@ def test_h3_turbo_offload_compatibility_follows_dit_selection(
|
||||
pin_cpu_memory=True,
|
||||
)
|
||||
captured = {}
|
||||
spec = _spec()
|
||||
|
||||
def load_turbo(**kwargs):
|
||||
captured.update(kwargs)
|
||||
return object(), object()
|
||||
return object(), object(), spec
|
||||
|
||||
monkeypatch.setattr(pipeline_module, "load_minimax_h3_turbo_lora", load_turbo)
|
||||
|
||||
@@ -296,31 +361,84 @@ def test_h3_turbo_offload_compatibility_follows_dit_selection(
|
||||
assert captured["unsupported_offload_mode"] == expected_mode
|
||||
|
||||
|
||||
def test_h3_turbo_accepts_only_the_declared_v1_artifact(tmp_path):
|
||||
unsupported = tmp_path / "minimax_h3_fl2v_turbo_8step_v1.0_bf16.safetensors"
|
||||
_write_tiny_turbo(unsupported)
|
||||
def test_h3_turbo_uses_the_reference_alpha_when_the_artifact_declares_none(tmp_path):
|
||||
"""``4step_v0.1`` ships no alpha. LightX2V's reference script applies
|
||||
``scale * alpha / rank`` with alpha defaulting to 8, so the fallback must be
|
||||
8; alpha == rank would drive the adapter 16x too strongly."""
|
||||
|
||||
with pytest.raises(ValueError, match="supports only"):
|
||||
path = tmp_path / "minimax_h3_fl2v_turbo_4step_v0.1.safetensors"
|
||||
_write_tiny_turbo(path, alpha=None)
|
||||
|
||||
loaded = load_minimax_h3_turbo_lora(
|
||||
partition="fl2va",
|
||||
lora_request=_request(path),
|
||||
lora_path=path,
|
||||
dtype=torch.float32,
|
||||
)
|
||||
|
||||
assert loaded is not None
|
||||
_, peft_helper, spec = loaded
|
||||
assert spec.alpha == 8
|
||||
assert peft_helper.lora_alpha == 8
|
||||
assert spec.alpha / spec.rank == 0.0625
|
||||
|
||||
|
||||
def test_h3_turbo_declines_the_comfyui_export(tmp_path):
|
||||
"""The ComfyUI exports fuse Q/K/V, so they are not this loader's format."""
|
||||
|
||||
path = tmp_path / "minimax_h3_fl2v_turbo_4step_v1.0_768p_comfyui_bf16.safetensors"
|
||||
_write_tiny_turbo(path)
|
||||
|
||||
assert (
|
||||
load_minimax_h3_turbo_lora(
|
||||
partition="fl2va",
|
||||
lora_request=_request(unsupported),
|
||||
lora_path=unsupported,
|
||||
lora_request=_request(path),
|
||||
lora_path=path,
|
||||
dtype=torch.float32,
|
||||
)
|
||||
is None
|
||||
)
|
||||
|
||||
supported = tmp_path / "minimax_h3_fl2v_turbo_4step_v1.0_768p_bf16.safetensors"
|
||||
_write_tiny_turbo(supported)
|
||||
# Directory resolution selects the declared artifact even when another
|
||||
# v1.0 checkpoint is present beside it.
|
||||
assert (
|
||||
|
||||
def test_h3_turbo_accepts_every_published_artifact_name(tmp_path):
|
||||
"""Each published filename loads and carries its own sampler contract."""
|
||||
|
||||
for index, (name, steps, shift) in enumerate(
|
||||
(
|
||||
("minimax_h3_fl2v_turbo_4step_v0.1.safetensors", 4, 12.0),
|
||||
("minimax_h3_fl2v_turbo_4step_v1.1_768p_bf16.safetensors", 4, 6.0),
|
||||
("minimax_h3_fl2v_turbo_8step_v1.0_bf16.safetensors", 8, 12.0),
|
||||
("minimax_h3_fl2v_turbo_8step_v1.0_768p_bf16.safetensors", 8, 6.0),
|
||||
)
|
||||
):
|
||||
directory = tmp_path / f"artifact{index}"
|
||||
directory.mkdir()
|
||||
path = directory / name
|
||||
_write_tiny_turbo(path)
|
||||
loaded = load_minimax_h3_turbo_lora(
|
||||
partition="fl2va",
|
||||
lora_request=_request(path),
|
||||
lora_path=path,
|
||||
dtype=torch.float32,
|
||||
)
|
||||
assert loaded is not None, name
|
||||
spec = loaded[2]
|
||||
assert spec.denoise_steps == steps
|
||||
assert spec.sigma_points == steps + 1
|
||||
assert spec.video_shift == shift
|
||||
|
||||
|
||||
def test_h3_turbo_directory_with_several_artifacts_is_ambiguous(tmp_path):
|
||||
_write_tiny_turbo(tmp_path / "minimax_h3_fl2v_turbo_4step_v1.0_768p_bf16.safetensors")
|
||||
_write_tiny_turbo(tmp_path / "minimax_h3_fl2v_turbo_8step_v1.0_768p_bf16.safetensors")
|
||||
|
||||
with pytest.raises(ValueError, match="point --lora-path at one file"):
|
||||
load_minimax_h3_turbo_lora(
|
||||
partition="fl2va",
|
||||
lora_request=_request(tmp_path),
|
||||
lora_path=tmp_path,
|
||||
dtype=torch.float32,
|
||||
)
|
||||
is not None
|
||||
)
|
||||
|
||||
|
||||
def test_h3_turbo_rejects_a_truncated_declared_artifact(tmp_path):
|
||||
@@ -352,7 +470,7 @@ def test_non_h3_checkpoint_falls_back_to_the_generic_peft_loader(tmp_path):
|
||||
|
||||
declared = tmp_path / "minimax_h3_fl2v_turbo_4step_v1.0_768p_bf16.safetensors"
|
||||
_write_tiny_turbo(declared, key_format="other")
|
||||
with pytest.raises(ValueError, match="requires safetensors metadata"):
|
||||
with pytest.raises(ValueError, match="expected 'minimax-h3-diffusers'"):
|
||||
load_minimax_h3_turbo_lora(
|
||||
partition="fl2va",
|
||||
lora_request=_request(declared),
|
||||
@@ -371,7 +489,7 @@ def test_only_an_active_recognized_turbo_adapter_restricts_ref2va(monkeypatch):
|
||||
torch.nn.Module.__init__(pipeline)
|
||||
pipeline.partition = "combined"
|
||||
pipeline.supported_tasks = frozenset({"t2va", "fl2va", "ref2va"})
|
||||
pipeline._turbo_lora_adapter_ids = set()
|
||||
pipeline._turbo_lora_specs = {}
|
||||
pipeline._native_lora_adapter_ids = set()
|
||||
pipeline._lora_sigma_schedules = {}
|
||||
request = _request("generic-peft")
|
||||
@@ -388,14 +506,14 @@ def test_only_an_active_recognized_turbo_adapter_restricts_ref2va(monkeypatch):
|
||||
return pipeline._resolve_task(
|
||||
"ref2va",
|
||||
{},
|
||||
has_turbo_lora=pipeline._has_active_turbo_lora(sampling),
|
||||
turbo_spec=pipeline._active_turbo_spec(sampling),
|
||||
)
|
||||
|
||||
recognized = (object(), object())
|
||||
recognized = (object(), object(), _spec())
|
||||
monkeypatch.setattr(pipeline_module, "load_minimax_h3_turbo_lora", lambda **_: recognized)
|
||||
assert load() is recognized
|
||||
assert load() == recognized[:2]
|
||||
assert resolve(0.0) == "ref2va"
|
||||
with pytest.raises(OmniClientError, match="supports T2VA/FL2VA requests only"):
|
||||
with pytest.raises(OmniClientError, match="serves"):
|
||||
resolve(1.0)
|
||||
|
||||
# Simulate manager eviction followed by a generic PEFT adapter reusing the
|
||||
@@ -410,7 +528,7 @@ def test_h3_turbo_requires_all_loaded_targets_to_bind():
|
||||
|
||||
pipeline = object.__new__(MiniMaxH3Pipeline)
|
||||
torch.nn.Module.__init__(pipeline)
|
||||
pipeline._turbo_lora_adapter_ids = {1}
|
||||
pipeline._turbo_lora_specs = {1: _spec()}
|
||||
lora_model = SimpleNamespace(id=1, loras={"to_q": object(), "to_k": object()})
|
||||
|
||||
pipeline._validate_diffusion_lora_binding(
|
||||
@@ -448,7 +566,7 @@ def test_h3_turbo_rejects_unsupported_sampling(num_inference_steps, extra_args,
|
||||
)
|
||||
|
||||
with pytest.raises(OmniClientError, match=error):
|
||||
pipeline._validate_turbo_sampling(sampling)
|
||||
pipeline._validate_turbo_sampling(sampling, _spec())
|
||||
|
||||
|
||||
def test_h3_turbo_accepts_five_sigma_points_for_four_nfe():
|
||||
@@ -461,5 +579,6 @@ def test_h3_turbo_accepts_five_sigma_points_for_four_nfe():
|
||||
SimpleNamespace(
|
||||
num_inference_steps=5,
|
||||
extra_args={"flow_shift": 6.0, "audio_flow_shift": 3.0},
|
||||
)
|
||||
),
|
||||
_spec(),
|
||||
)
|
||||
@@ -309,7 +309,7 @@ def test_pipeline_native_schedule_and_task_validation(monkeypatch):
|
||||
torch.nn.Module.__init__(pipeline)
|
||||
pipeline.partition = "fl2va"
|
||||
pipeline.supported_tasks = frozenset({"t2va", "fl2va", "ref2va"})
|
||||
pipeline._turbo_lora_adapter_ids = set()
|
||||
pipeline._turbo_lora_specs = {}
|
||||
pipeline._native_lora_adapter_ids = {7}
|
||||
pipeline._lora_sigma_schedules = {7: DMD2SigmaSchedule.from_positions([1.0, 0.7, 0.4, 0.15, 0.0])}
|
||||
pipeline._base_schedule_by_partition = {"fl2va": None}
|
||||
@@ -330,7 +330,7 @@ def test_pipeline_native_schedule_and_task_validation(monkeypatch):
|
||||
pipeline._resolve_task(
|
||||
"fl2va",
|
||||
{},
|
||||
has_turbo_lora=False,
|
||||
turbo_spec=None,
|
||||
has_native_lora=True,
|
||||
)
|
||||
|
||||
@@ -418,7 +418,7 @@ def test_pipeline_replaces_native_classification_after_reload(monkeypatch, tmp_p
|
||||
enable_layerwise_offload=False,
|
||||
enable_distributed_layerwise_offload=False,
|
||||
)
|
||||
pipeline._turbo_lora_adapter_ids = set()
|
||||
pipeline._turbo_lora_specs = {}
|
||||
pipeline._native_lora_adapter_ids = set()
|
||||
pipeline._lora_sigma_schedules = {}
|
||||
|
||||
@@ -472,7 +472,7 @@ def test_h3_native_allows_distributed_layerwise_offload(monkeypatch):
|
||||
enable_layerwise_offload=False,
|
||||
enable_distributed_layerwise_offload=True,
|
||||
)
|
||||
pipeline._turbo_lora_adapter_ids = set()
|
||||
pipeline._turbo_lora_specs = {}
|
||||
pipeline._native_lora_adapter_ids = set()
|
||||
pipeline._lora_sigma_schedules = {}
|
||||
captured: dict[str, object] = {}
|
||||
@@ -612,7 +612,7 @@ def test_legacy_manager_uses_native_loader(tmp_path):
|
||||
enable_layerwise_offload=False,
|
||||
enable_distributed_layerwise_offload=False,
|
||||
)
|
||||
pipeline._turbo_lora_adapter_ids = set()
|
||||
pipeline._turbo_lora_specs = {}
|
||||
pipeline._native_lora_adapter_ids = set()
|
||||
pipeline._lora_sigma_schedules = {}
|
||||
return pipeline._load_diffusion_lora_adapter(**kwargs)
|
||||
@@ -634,7 +634,7 @@ def test_pipeline_schedule_inactive_when_scale_zero():
|
||||
pipeline = object.__new__(MiniMaxH3Pipeline)
|
||||
torch.nn.Module.__init__(pipeline)
|
||||
pipeline.partition = "fl2va"
|
||||
pipeline._turbo_lora_adapter_ids = set()
|
||||
pipeline._turbo_lora_specs = {}
|
||||
pipeline._native_lora_adapter_ids = {7}
|
||||
pipeline._lora_sigma_schedules = {7: DMD2SigmaSchedule.from_positions([1.0, 0.7, 0.4, 0.15, 0.0])}
|
||||
pipeline._base_schedule_by_partition = {"fl2va": None}
|
||||
@@ -664,7 +664,7 @@ def test_pipeline_schedule_falls_back_after_eviction(monkeypatch, tmp_path):
|
||||
enable_layerwise_offload=False,
|
||||
enable_distributed_layerwise_offload=False,
|
||||
)
|
||||
pipeline._turbo_lora_adapter_ids = set()
|
||||
pipeline._turbo_lora_specs = {}
|
||||
pipeline._native_lora_adapter_ids = set()
|
||||
pipeline._lora_sigma_schedules = {}
|
||||
pipeline._base_schedule_by_partition = {"fl2va": None}
|
||||
|
||||
@@ -0,0 +1,86 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
|
||||
|
||||
"""Contract coverage for the LightX2V MiniMax-H3 Turbo artifact matrix."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
|
||||
from vllm_omni.diffusion.models.minimax_h3.lora import (
|
||||
_TURBO_AUDIO_SHIFT,
|
||||
_TURBO_VIDEO_SHIFT_544P,
|
||||
_TURBO_VIDEO_SHIFT_768P,
|
||||
TurboSpec,
|
||||
parse_turbo_filename,
|
||||
)
|
||||
|
||||
pytestmark = [pytest.mark.core_model, pytest.mark.diffusion, pytest.mark.cpu]
|
||||
|
||||
# The eight Diffusers-layout artifacts published at lightx2v/Minimax-h3-Turbo,
|
||||
# with the contract each one implies. The ComfyUI export of an artifact is a
|
||||
# fused-QKV re-packing of the same weights and is not served.
|
||||
PUBLISHED = [
|
||||
("minimax_h3_fl2v_turbo_4step_v0.1.safetensors", "fl2v", 4, _TURBO_VIDEO_SHIFT_544P),
|
||||
("minimax_h3_fl2v_turbo_4step_v1.0_768p_bf16.safetensors", "fl2v", 4, _TURBO_VIDEO_SHIFT_768P),
|
||||
("minimax_h3_fl2v_turbo_4step_v1.1_768p_bf16.safetensors", "fl2v", 4, _TURBO_VIDEO_SHIFT_768P),
|
||||
("minimax_h3_fl2v_turbo_4step_v1.2_768p_bf16.safetensors", "fl2v", 4, _TURBO_VIDEO_SHIFT_768P),
|
||||
("minimax_h3_fl2v_turbo_8step_v1.0_bf16.safetensors", "fl2v", 8, _TURBO_VIDEO_SHIFT_544P),
|
||||
("minimax_h3_fl2v_turbo_8step_v1.0_768p_bf16.safetensors", "fl2v", 8, _TURBO_VIDEO_SHIFT_768P),
|
||||
("minimax_h3_ref2v_turbo_4step_v0.1_bf16.safetensors", "ref2v", 4, _TURBO_VIDEO_SHIFT_544P),
|
||||
("minimax_h3_ref2v_turbo_8step_v1.0_768p_bf16.safetensors", "ref2v", 8, _TURBO_VIDEO_SHIFT_768P),
|
||||
]
|
||||
|
||||
|
||||
@pytest.mark.parametrize(("filename", "task", "steps", "video_shift"), PUBLISHED)
|
||||
def test_every_published_artifact_is_recognised(filename: str, task: str, steps: int, video_shift: float) -> None:
|
||||
spec = parse_turbo_filename(filename)
|
||||
assert spec is not None, filename
|
||||
assert spec.filename == filename
|
||||
assert spec.task_family == task
|
||||
assert spec.denoise_steps == steps
|
||||
assert spec.video_shift == video_shift
|
||||
assert spec.audio_shift == _TURBO_AUDIO_SHIFT
|
||||
# Until the loader reads metadata, alpha is LightX2V's reference default.
|
||||
assert spec.rank == 128
|
||||
assert spec.alpha == 8.0
|
||||
|
||||
|
||||
def test_all_eight_artifacts_are_covered() -> None:
|
||||
assert len(PUBLISHED) == 8
|
||||
assert len({name for name, *_ in PUBLISHED}) == 8
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"filename",
|
||||
[
|
||||
"adapter_model.safetensors",
|
||||
"minimax_h3_fl2v_turbo_v1.0_768p_bf16.safetensors",
|
||||
"minimax_h3_t2v_turbo_4step_v1.0.safetensors",
|
||||
"minimax_h3_fl2v_turbo_0step_v1.0.safetensors",
|
||||
# The ComfyUI exports are a different tensor layout, not a variant of
|
||||
# the supported one; the loader refuses them by name.
|
||||
"minimax_h3_fl2v_turbo_4step_v1.0_768p_comfyui_bf16.safetensors",
|
||||
"minimax_h3_ref2v_turbo_8step_v1.0_768p_comfyui_bf16.safetensors",
|
||||
],
|
||||
)
|
||||
def test_unrelated_names_are_rejected(filename: str) -> None:
|
||||
assert parse_turbo_filename(filename) is None
|
||||
|
||||
|
||||
def _spec(filename: str) -> TurboSpec:
|
||||
spec = parse_turbo_filename(filename)
|
||||
assert spec is not None
|
||||
return spec
|
||||
|
||||
|
||||
def test_sigma_points_follow_the_step_count() -> None:
|
||||
assert _spec("minimax_h3_fl2v_turbo_4step_v1.0_768p_bf16.safetensors").sigma_points == 5
|
||||
assert _spec("minimax_h3_fl2v_turbo_8step_v1.0_768p_bf16.safetensors").sigma_points == 9
|
||||
|
||||
|
||||
def test_task_family_decides_supported_tasks() -> None:
|
||||
fl2v = _spec("minimax_h3_fl2v_turbo_4step_v1.0_768p_bf16.safetensors")
|
||||
ref2v = _spec("minimax_h3_ref2v_turbo_8step_v1.0_768p_bf16.safetensors")
|
||||
assert fl2v.supported_tasks == frozenset({"t2va", "fl2va"})
|
||||
assert ref2v.supported_tasks == frozenset({"ref2va"})
|
||||
@@ -1,20 +1,25 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
|
||||
|
||||
"""Loader for the LightX2V Turbo MiniMax-H3 LoRA.
|
||||
"""Loader for the LightX2V Turbo MiniMax-H3 LoRA family.
|
||||
|
||||
Keyed by ``key_format=minimax-h3-diffusers``. The native FlashGen contract lives
|
||||
in ``.npu.lora``; the two artifacts differ in rank, target naming and QKV
|
||||
layout, so each owns its own parsing and packing.
|
||||
The published artifacts differ in task family, step count, training resolution
|
||||
and LoRA alpha, so :class:`TurboSpec` carries the contract of the file actually
|
||||
loaded and the pipeline validates each request against it. Only the
|
||||
Diffusers-PEFT exports are served. The native FlashGen contract lives in
|
||||
``.npu.lora``.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import math
|
||||
from dataclasses import dataclass, replace
|
||||
from pathlib import Path
|
||||
|
||||
import regex as re
|
||||
import torch
|
||||
from safetensors import safe_open
|
||||
from vllm.logger import init_logger
|
||||
from vllm.lora.lora_model import LoRAModel
|
||||
from vllm.lora.lora_weights import PackedLoRALayerWeights
|
||||
from vllm.lora.peft_helper import PEFTHelper
|
||||
@@ -22,12 +27,79 @@ from vllm.model_executor.models.utils import WeightsMapper
|
||||
|
||||
from vllm_omni.lora.request import LoRARequest
|
||||
|
||||
logger = init_logger(__name__)
|
||||
|
||||
_TURBO_RANK = 128
|
||||
_TURBO_ALPHA = 128
|
||||
_TURBO_HIDDEN_SIZE = 5376
|
||||
_TURBO_ATTENTION_INNER_SIZE = 7168
|
||||
_TURBO_FFN_HIDDEN_SIZE = 14336
|
||||
_TURBO_FILENAME = "minimax_h3_fl2v_turbo_4step_v1.0_768p_bf16.safetensors"
|
||||
# ``minimax_h3_<task>_turbo_<n>step_v<major.minor>[_768p][_bf16]``. The name is
|
||||
# the only place the sampler contract is recorded.
|
||||
_TURBO_NAME_RE = re.compile(
|
||||
r"^minimax_h3_(?P<task>fl2v|ref2v)_turbo_(?P<steps>\d+)step"
|
||||
r"_v\d+\.\d+(?P<res>_768p)?(?:_bf16)?\.safetensors$"
|
||||
)
|
||||
# The 768p retrains moved to a shorter video flow shift; the 544p artifacts keep
|
||||
# the base model's.
|
||||
_TURBO_VIDEO_SHIFT_768P = 6.0
|
||||
_TURBO_VIDEO_SHIFT_544P = 12.0
|
||||
_TURBO_AUDIO_SHIFT = 3.0
|
||||
# Only ``4step_v0.1`` declares no alpha. LightX2V's reference script never reads
|
||||
# the metadata: it applies ``scale * alpha / rank`` with ``--lora-alpha``
|
||||
# defaulting to 8, and its documented v0.1 command does not override that. So an
|
||||
# artifact that declares nothing is driven at 8, not at its rank.
|
||||
_TURBO_DEFAULT_ALPHA = 8.0
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class TurboSpec:
|
||||
"""The sampler contract one Turbo artifact was distilled for."""
|
||||
|
||||
filename: str
|
||||
task_family: str
|
||||
"""``fl2v`` (serves t2va and fl2va) or ``ref2v`` (serves ref2va)."""
|
||||
denoise_steps: int
|
||||
video_shift: float
|
||||
audio_shift: float
|
||||
rank: int
|
||||
alpha: float
|
||||
|
||||
@property
|
||||
def sigma_points(self) -> int:
|
||||
"""Sigma points the API contract expects: one more than the forwards."""
|
||||
return self.denoise_steps + 1
|
||||
|
||||
@property
|
||||
def supported_tasks(self) -> frozenset[str]:
|
||||
if self.task_family == "ref2v":
|
||||
return frozenset({"ref2va"})
|
||||
return frozenset({"t2va", "fl2va"})
|
||||
|
||||
|
||||
def parse_turbo_filename(name: str) -> TurboSpec | None:
|
||||
"""Return the contract a Turbo filename encodes, or ``None``.
|
||||
|
||||
``alpha`` is the rank-matched default; the loader replaces it with the
|
||||
artifact's declared value.
|
||||
"""
|
||||
|
||||
match = _TURBO_NAME_RE.match(name)
|
||||
if match is None:
|
||||
return None
|
||||
steps = int(match.group("steps"))
|
||||
if steps <= 0:
|
||||
return None
|
||||
return TurboSpec(
|
||||
filename=name,
|
||||
task_family=match.group("task"),
|
||||
denoise_steps=steps,
|
||||
video_shift=_TURBO_VIDEO_SHIFT_768P if match.group("res") else _TURBO_VIDEO_SHIFT_544P,
|
||||
audio_shift=_TURBO_AUDIO_SHIFT,
|
||||
rank=_TURBO_RANK,
|
||||
alpha=_TURBO_DEFAULT_ALPHA,
|
||||
)
|
||||
|
||||
|
||||
_LORA_A_SUFFIX = ".lora_A.default.weight"
|
||||
_LORA_B_SUFFIX = ".lora_B.default.weight"
|
||||
_TURBO_TARGETS = frozenset({"to_q", "to_k", "to_v", "out_proj", "fc1", "fc2"})
|
||||
@@ -81,11 +153,20 @@ def _select_turbo_file(artifact_path: str | Path) -> Path | None:
|
||||
if not path.is_dir():
|
||||
return None
|
||||
|
||||
candidate = path / _TURBO_FILENAME
|
||||
return candidate if candidate.is_file() else None
|
||||
candidates = sorted(child for child in path.glob("*.safetensors") if parse_turbo_filename(child.name))
|
||||
if len(candidates) == 1:
|
||||
return candidates[0]
|
||||
if len(candidates) > 1:
|
||||
raise ValueError(
|
||||
f"{path} holds {len(candidates)} MiniMax-H3 Turbo artifacts "
|
||||
f"({[c.name for c in candidates[:4]]}); point --lora-path at one file."
|
||||
)
|
||||
return None
|
||||
|
||||
|
||||
def _validate_and_convert_tensors(checkpoint) -> dict[str, torch.Tensor]:
|
||||
"""Validate a Diffusers-layout Turbo tensor set and pack it for the manager."""
|
||||
|
||||
tensors: dict[str, torch.Tensor] = {}
|
||||
pairs: dict[str, set[str]] = {}
|
||||
raw_targets: set[str] = set()
|
||||
@@ -134,7 +215,7 @@ def _validate_and_convert_tensors(checkpoint) -> dict[str, torch.Tensor]:
|
||||
unexpected = sorted(raw_targets - _TURBO_EXPECTED_RAW_TARGETS)
|
||||
if missing or unexpected:
|
||||
raise ValueError(
|
||||
"MiniMax-H3 Turbo target set does not match the supported v1.0 artifact: "
|
||||
"MiniMax-H3 Turbo target set does not match the published artifact layout: "
|
||||
f"missing={len(missing)} {missing[:5]}, unexpected={len(unexpected)} {unexpected[:5]}"
|
||||
)
|
||||
return tensors
|
||||
@@ -164,39 +245,61 @@ def load_minimax_h3_turbo_lora(
|
||||
lora_path: str | Path,
|
||||
dtype: torch.dtype,
|
||||
unsupported_offload_mode: str | None = None,
|
||||
) -> tuple[LoRAModel, PEFTHelper] | None:
|
||||
"""Load the published LightX2V Turbo v1.0 through the legacy manager."""
|
||||
) -> tuple[LoRAModel, PEFTHelper, TurboSpec] | None:
|
||||
"""Load any published LightX2V Turbo artifact through the legacy manager."""
|
||||
|
||||
lora_file = _select_turbo_file(lora_path)
|
||||
if lora_file is None:
|
||||
return None
|
||||
spec = parse_turbo_filename(lora_file.name)
|
||||
if spec is None:
|
||||
# Not a Turbo artifact by name; leave it to the next loader.
|
||||
return None
|
||||
with safe_open(lora_file, framework="pt", device="cpu") as checkpoint:
|
||||
# Three published artifacts declare no key_format, so absence is not an
|
||||
# error; a declaration naming a different format is.
|
||||
metadata = checkpoint.metadata() or {}
|
||||
if metadata.get("key_format") != "minimax-h3-diffusers":
|
||||
if lora_file.name == _TURBO_FILENAME:
|
||||
raise ValueError(
|
||||
"MiniMax-H3 Turbo v1.0 requires safetensors metadata key_format='minimax-h3-diffusers'"
|
||||
)
|
||||
return None
|
||||
if lora_file.name != _TURBO_FILENAME:
|
||||
raise ValueError(f"MiniMax-H3 Turbo supports only {_TURBO_FILENAME!r}, got {lora_file.name!r}")
|
||||
declared_format = metadata.get("key_format")
|
||||
if declared_format not in (None, "minimax-h3-diffusers"):
|
||||
raise ValueError(
|
||||
f"{lora_file.name} declares key_format={declared_format!r}, expected 'minimax-h3-diffusers'"
|
||||
)
|
||||
raw_alpha = metadata.get("alpha")
|
||||
try:
|
||||
alpha = float(raw_alpha) if raw_alpha is not None else math.nan
|
||||
except ValueError as exc:
|
||||
raise ValueError(f"MiniMax-H3 Turbo alpha must be numeric, got {raw_alpha!r}") from exc
|
||||
if alpha != _TURBO_ALPHA:
|
||||
raise ValueError(f"MiniMax-H3 Turbo v1.0 requires alpha={_TURBO_ALPHA}, got {raw_alpha!r}")
|
||||
if partition == "ref2va":
|
||||
raise ValueError("MiniMax-H3 Turbo LoRA supports FL2VA/T2VA only")
|
||||
if raw_alpha is None:
|
||||
logger.warning(
|
||||
"MiniMax-H3 Turbo artifact %s declares no alpha; using the LightX2V reference "
|
||||
"default alpha=%g (scale %g). Override with the request-level LoRA scale.",
|
||||
spec.filename,
|
||||
spec.alpha,
|
||||
spec.alpha / spec.rank,
|
||||
)
|
||||
else:
|
||||
try:
|
||||
spec = replace(spec, alpha=float(raw_alpha))
|
||||
except (TypeError, ValueError) as exc:
|
||||
raise ValueError(f"MiniMax-H3 Turbo alpha must be numeric, got {raw_alpha!r}") from exc
|
||||
if not math.isfinite(spec.alpha) or spec.alpha <= 0:
|
||||
raise ValueError(f"MiniMax-H3 Turbo alpha must be a positive number, got {raw_alpha!r}")
|
||||
|
||||
if partition == "ref2va" and spec.task_family == "fl2v":
|
||||
raise ValueError(f"{spec.filename} is an FL2VA/T2VA Turbo artifact; a Ref2VA-only server cannot serve it.")
|
||||
# ``combined`` serves ref2va from ``transformers_ref``, but the LoRA
|
||||
# target pattern only injects into ``transformer``: the adapter would
|
||||
# bind to the stack that never runs.
|
||||
if spec.task_family == "ref2v" and partition != "ref2va":
|
||||
raise ValueError(
|
||||
f"{spec.filename} is a Ref2VA Turbo artifact; start the server with --task-type ref2va "
|
||||
f"(task_type={partition!r} serves ref2va from a DiT the adapter cannot bind to)"
|
||||
)
|
||||
if unsupported_offload_mode is not None:
|
||||
raise ValueError(f"MiniMax-H3 Turbo dynamic LoRA does not support {unsupported_offload_mode}")
|
||||
|
||||
tensors = _validate_and_convert_tensors(checkpoint)
|
||||
|
||||
peft_helper = PEFTHelper.from_dict(
|
||||
{
|
||||
"r": _TURBO_RANK,
|
||||
"lora_alpha": _TURBO_ALPHA,
|
||||
"r": spec.rank,
|
||||
"lora_alpha": spec.alpha,
|
||||
"target_modules": _TURBO_TARGET_PATTERN,
|
||||
}
|
||||
)
|
||||
@@ -209,4 +312,16 @@ def load_minimax_h3_turbo_lora(
|
||||
weights_mapper=_TURBO_WEIGHTS_MAPPER,
|
||||
)
|
||||
_pack_h3_turbo_fc1(lora_model)
|
||||
return lora_model, peft_helper
|
||||
logger.info(
|
||||
"MiniMax-H3 Turbo adapter %s: task=%s, %d denoiser forwards (%d sigma points), "
|
||||
"rank=%d alpha=%g, flow_shift=%g/%g",
|
||||
spec.filename,
|
||||
spec.task_family,
|
||||
spec.denoise_steps,
|
||||
spec.sigma_points,
|
||||
spec.rank,
|
||||
spec.alpha,
|
||||
spec.video_shift,
|
||||
spec.audio_shift,
|
||||
)
|
||||
return lora_model, peft_helper, spec
|
||||
@@ -125,7 +125,7 @@ from .denoise_loop import (
|
||||
)
|
||||
from .encoder import MiniMaxH3Qwen3VLEncoder
|
||||
from .fasth3 import FastH3WeightFusion, resolve_fasth3_fusion
|
||||
from .lora import load_minimax_h3_turbo_lora
|
||||
from .lora import TurboSpec, load_minimax_h3_turbo_lora
|
||||
from .minimax_h3_transformer import (
|
||||
MiniMaxH3Attention,
|
||||
MiniMaxH3DiTModel,
|
||||
@@ -188,9 +188,6 @@ MINIMAX_H3_MAX_REFERENCE_IMAGE_BYTES = 30 * 1024 * 1024
|
||||
MINIMAX_H3_REFERENCE_IMAGE_FORMATS = frozenset({"jpeg", "png", "webp", "heic", "heif"})
|
||||
MINIMAX_H3_MIN_OUTPUT_SECONDS = 4.0
|
||||
MINIMAX_H3_MAX_OUTPUT_SECONDS = 15.0
|
||||
MINIMAX_H3_TURBO_SIGMA_POINTS = 5
|
||||
MINIMAX_H3_TURBO_VIDEO_SHIFT = 6.0
|
||||
MINIMAX_H3_TURBO_AUDIO_SHIFT = 3.0
|
||||
MINIMAX_H3_DOWNLOAD_PATTERNS = [
|
||||
"FL2VA/**",
|
||||
"Ref2VA/model_index.json",
|
||||
@@ -706,7 +703,7 @@ class MiniMaxH3Pipeline(
|
||||
) -> tuple[LoRAModel, PEFTHelper] | None:
|
||||
# A cache eviction may be followed by a different adapter reusing the
|
||||
# same client-supplied ID. Every real load replaces the classification.
|
||||
self._turbo_lora_adapter_ids.discard(lora_request.lora_int_id)
|
||||
self._turbo_lora_specs.pop(lora_request.lora_int_id, None)
|
||||
self._native_lora_adapter_ids.discard(lora_request.lora_int_id)
|
||||
self._lora_sigma_schedules.pop(lora_request.lora_int_id, None)
|
||||
od_config = getattr(self, "od_config", None)
|
||||
@@ -726,8 +723,9 @@ class MiniMaxH3Pipeline(
|
||||
unsupported_offload_mode=" or ".join(offload_modes) or None,
|
||||
)
|
||||
if loaded is not None:
|
||||
self._turbo_lora_adapter_ids.add(lora_request.lora_int_id)
|
||||
return loaded
|
||||
lora_model, peft_helper, turbo_spec = loaded
|
||||
self._turbo_lora_specs[lora_request.lora_int_id] = turbo_spec
|
||||
return lora_model, peft_helper
|
||||
|
||||
# Selection is by the artifact's safetensors ``key_format``, not by the
|
||||
# running platform: the native loader is checkpoint-format parsing with
|
||||
@@ -753,7 +751,7 @@ class MiniMaxH3Pipeline(
|
||||
lora_model: LoRAModel,
|
||||
bound_lora_names: frozenset[str],
|
||||
) -> None:
|
||||
if lora_model.id in self._turbo_lora_adapter_ids:
|
||||
if lora_model.id in self._turbo_lora_specs:
|
||||
missing = sorted(set(lora_model.loras) - bound_lora_names)
|
||||
if missing:
|
||||
raise ValueError(
|
||||
@@ -770,14 +768,6 @@ class MiniMaxH3Pipeline(
|
||||
f"bound={len(bound_lora_names)}/{len(lora_model.loras)}, missing={missing[:5]}"
|
||||
)
|
||||
|
||||
def _has_active_turbo_lora(self, sampling: Any) -> bool:
|
||||
lora_request = sampling.lora_request
|
||||
return (
|
||||
lora_request is not None
|
||||
and not math.isclose(0.0, float(sampling.lora_scale))
|
||||
and lora_request.lora_int_id in self._turbo_lora_adapter_ids
|
||||
)
|
||||
|
||||
def _has_active_native_lora(self, sampling: Any) -> bool:
|
||||
lora_request = sampling.lora_request
|
||||
return (
|
||||
@@ -837,27 +827,46 @@ class MiniMaxH3Pipeline(
|
||||
return adapter_schedule
|
||||
return self._base_schedule_for_task(task)
|
||||
|
||||
def _validate_turbo_sampling(self, sampling: Any) -> None:
|
||||
def _active_turbo_spec(self, sampling: Any) -> TurboSpec | None:
|
||||
"""Return the spec of the Turbo adapter this request actually applies.
|
||||
|
||||
A recognized adapter at scale 0 contributes nothing, so it neither
|
||||
constrains the task nor imposes its sampler contract.
|
||||
"""
|
||||
|
||||
lora_request = sampling.lora_request
|
||||
if lora_request is None or math.isclose(0.0, float(sampling.lora_scale)):
|
||||
return None
|
||||
return self._turbo_lora_specs.get(lora_request.lora_int_id)
|
||||
|
||||
def _validate_turbo_sampling(self, sampling: Any, spec: TurboSpec) -> None:
|
||||
"""Hold a request to the contract of the artifact that is loaded.
|
||||
|
||||
Sigma-point count and both flow shifts vary across the Turbo family, so
|
||||
each is checked against the adapter's own spec rather than a single
|
||||
published configuration.
|
||||
"""
|
||||
|
||||
extra = sampling.extra_args or {}
|
||||
sigma_points = sampling.num_inference_steps
|
||||
if sigma_points != MINIMAX_H3_TURBO_SIGMA_POINTS:
|
||||
if sigma_points != spec.sigma_points:
|
||||
raise OmniClientError(
|
||||
"MiniMax-H3 Turbo requires num_inference_steps=5 (five sigma points produce four denoiser evaluations)"
|
||||
f"{spec.filename} is a {spec.denoise_steps}-step artifact and requires "
|
||||
f"num_inference_steps={spec.sigma_points} "
|
||||
f"({spec.sigma_points} sigma points produce {spec.denoise_steps} denoiser evaluations)"
|
||||
)
|
||||
try:
|
||||
video_shift = float(extra.get("flow_shift", self.default_video_shift))
|
||||
except (TypeError, ValueError) as exc:
|
||||
raise OmniClientError(f"MiniMax-H3 Turbo requires flow_shift={MINIMAX_H3_TURBO_VIDEO_SHIFT:g}") from exc
|
||||
if not math.isclose(video_shift, MINIMAX_H3_TURBO_VIDEO_SHIFT):
|
||||
raise OmniClientError(f"MiniMax-H3 Turbo requires flow_shift={MINIMAX_H3_TURBO_VIDEO_SHIFT:g}")
|
||||
raise OmniClientError(f"{spec.filename} requires flow_shift={spec.video_shift:g}") from exc
|
||||
if not math.isclose(video_shift, spec.video_shift):
|
||||
raise OmniClientError(f"{spec.filename} requires flow_shift={spec.video_shift:g}")
|
||||
try:
|
||||
audio_shift = float(extra.get("audio_flow_shift", self.default_audio_shift))
|
||||
except (TypeError, ValueError) as exc:
|
||||
raise OmniClientError(
|
||||
f"MiniMax-H3 Turbo requires audio_flow_shift={MINIMAX_H3_TURBO_AUDIO_SHIFT:g}"
|
||||
) from exc
|
||||
if not math.isclose(audio_shift, MINIMAX_H3_TURBO_AUDIO_SHIFT):
|
||||
raise OmniClientError(f"MiniMax-H3 Turbo requires audio_flow_shift={MINIMAX_H3_TURBO_AUDIO_SHIFT:g}")
|
||||
raise OmniClientError(f"{spec.filename} requires audio_flow_shift={spec.audio_shift:g}") from exc
|
||||
if not math.isclose(audio_shift, spec.audio_shift):
|
||||
raise OmniClientError(f"{spec.filename} requires audio_flow_shift={spec.audio_shift:g}")
|
||||
|
||||
def adopt_cache_dit_backend(self, backend: CacheDiTBackend) -> None:
|
||||
"""Adopt runner-installed generic Cache-DiT for request transitions."""
|
||||
@@ -890,7 +899,7 @@ class MiniMaxH3Pipeline(
|
||||
getattr(od_config, "task_type", None),
|
||||
str(od_config.model),
|
||||
)
|
||||
self._turbo_lora_adapter_ids: set[int] = set()
|
||||
self._turbo_lora_specs: dict[int, TurboSpec] = {}
|
||||
self._native_lora_adapter_ids: set[int] = set()
|
||||
self._lora_sigma_schedules: dict[int, DMD2SigmaSchedule] = {}
|
||||
model_root = _resolve_minimax_h3_model_root(
|
||||
@@ -1177,7 +1186,7 @@ class MiniMaxH3Pipeline(
|
||||
requested: str | None,
|
||||
multi_modal_data: dict[str, Any],
|
||||
*,
|
||||
has_turbo_lora: bool = False,
|
||||
turbo_spec: TurboSpec | None = None,
|
||||
has_native_lora: bool = False,
|
||||
) -> str:
|
||||
if requested is None:
|
||||
@@ -1196,8 +1205,11 @@ class MiniMaxH3Pipeline(
|
||||
raise OmniClientError(
|
||||
f"checkpoint partition {self.partition!r} supports {sorted(self.supported_tasks)}, got task={task!r}"
|
||||
)
|
||||
if task == "ref2va" and has_turbo_lora:
|
||||
raise OmniClientError("MiniMax-H3 Turbo LoRA supports T2VA/FL2VA requests only")
|
||||
if turbo_spec is not None and task not in turbo_spec.supported_tasks:
|
||||
raise OmniClientError(
|
||||
f"{turbo_spec.filename} is a {turbo_spec.task_family} Turbo artifact and serves "
|
||||
f"{sorted(turbo_spec.supported_tasks)}, got task={task!r}"
|
||||
)
|
||||
if has_native_lora and task != "t2va":
|
||||
raise OmniClientError("MiniMax-H3 native LoRA supports T2VA requests only")
|
||||
if self._fasth3 is not None:
|
||||
@@ -2216,16 +2228,16 @@ class MiniMaxH3Pipeline(
|
||||
quality = sampling.quality
|
||||
logger.debug("MiniMax H3 request quality=%s", quality)
|
||||
extra = sampling.extra_args or {}
|
||||
has_turbo_lora = self._has_active_turbo_lora(sampling)
|
||||
turbo_spec = self._active_turbo_spec(sampling)
|
||||
has_native_lora = self._has_active_native_lora(sampling)
|
||||
task = self._resolve_task(
|
||||
extra.get("task"),
|
||||
multi_modal_data,
|
||||
has_turbo_lora=has_turbo_lora,
|
||||
turbo_spec=turbo_spec,
|
||||
has_native_lora=has_native_lora,
|
||||
)
|
||||
if has_turbo_lora:
|
||||
self._validate_turbo_sampling(sampling)
|
||||
if turbo_spec is not None:
|
||||
self._validate_turbo_sampling(sampling, turbo_spec)
|
||||
if has_native_lora:
|
||||
self._validate_native_sampling(sampling, task=task)
|
||||
if self._fasth3 is not None:
|
||||
|
||||
Reference in new issue
Block a user