Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
 Hongsheng Liu
|
0db924bc54 | ||
|
Hongsheng Liu
|
39e1c70aea | ||
|
Hongsheng Liu
|
b9572edb81 | ||
|
Hongsheng Liu
|
a4516ea5db | ||
|
Hongsheng Liu
|
280c4f56f6 | ||
|
|
e04210d690 |
No files matched your search
+25
-31
@@ -40,30 +40,16 @@ nav:
|
||||
- Models:
|
||||
- models/supported_models.md
|
||||
- Features:
|
||||
- Overview: features/README.md
|
||||
- Runtime and Stage Execution:
|
||||
- Execution Modes and Streaming: user_guide/diffusion/execution_modes.md
|
||||
- Sleep Mode: features/sleep_mode.md
|
||||
- Quantization:
|
||||
- Overview: user_guide/quantization/overview.md
|
||||
- Online Quantization: user_guide/quantization/online.md
|
||||
- Quantized KV Cache: user_guide/quantization/quantized_kvcache.md
|
||||
- FP8 W8A8: user_guide/quantization/fp8.md
|
||||
- Int8 W8A8: user_guide/quantization/int8.md
|
||||
- BitsAndBytes W4: user_guide/quantization/bitsandbytes.md
|
||||
- MXFP8 W8A8: user_guide/quantization/mxfp8.md
|
||||
- MXFP4 W4A4: user_guide/quantization/mxfp4.md
|
||||
- ModelOpt: user_guide/quantization/modelopt.md
|
||||
- AutoRound: user_guide/quantization/autoround.md
|
||||
- msModelSlim: user_guide/quantization/msmodelslim.md
|
||||
- Diffusion Acceleration:
|
||||
- Overview and Compatibility:
|
||||
- Overview: user_guide/diffusion_features.md
|
||||
- Feature Compatibility: user_guide/feature_compatibility.md
|
||||
- CPU Offloading: user_guide/diffusion/cpu_offload.md
|
||||
- Sleep Mode: features/sleep_mode.md
|
||||
- Diffusion Features:
|
||||
- Overview: user_guide/diffusion_features.md
|
||||
- Feature Compatibility: user_guide/feature_compatibility.md
|
||||
- Regional Compilation: user_guide/diffusion/regional_compilation.md
|
||||
- Cache Acceleration:
|
||||
- TeaCache: user_guide/diffusion/cache_acceleration/teacache.md
|
||||
- Cache-DiT: user_guide/diffusion/cache_acceleration/cache_dit.md
|
||||
- Attention Backends: user_guide/diffusion/attention_backends.md
|
||||
- Frame Interpolation: user_guide/diffusion/frame_interpolation.md
|
||||
- Parallelism:
|
||||
- Overview: user_guide/diffusion/parallelism/overview.md
|
||||
- CFG Parallel: user_guide/diffusion/parallelism/cfg_parallel.md
|
||||
@@ -72,15 +58,23 @@ nav:
|
||||
- Sequence Parallel: user_guide/diffusion/parallelism/sequence_parallel.md
|
||||
- Tensor Parallel: user_guide/diffusion/parallelism/tensor_parallel.md
|
||||
- VAE Parallelism: user_guide/diffusion/parallelism/vae_parallelism.md
|
||||
- Attention Backends: user_guide/diffusion/attention_backends.md
|
||||
- Regional Compilation: user_guide/diffusion/regional_compilation.md
|
||||
- Frame Interpolation: user_guide/diffusion/frame_interpolation.md
|
||||
- Startup and Loading: user_guide/diffusion/startup_and_loading.md
|
||||
- CPU Offloading: user_guide/diffusion/cpu_offload.md
|
||||
- LoRA: user_guide/diffusion/lora.md
|
||||
- Experimental:
|
||||
- Session State Manager: features/session_state_manager.md
|
||||
- Integrations:
|
||||
- Custom Diffusion Pipeline: features/custom_pipeline.md
|
||||
- Custom Pipeline: features/custom_pipeline.md
|
||||
- Execution Modes: user_guide/diffusion/execution_modes.md
|
||||
- Quantization:
|
||||
- Quantization: user_guide/quantization/overview.md
|
||||
- Online Quantization: user_guide/quantization/online.md
|
||||
- Quantized KV Cache: user_guide/quantization/quantized_kvcache.md
|
||||
- FP8 W8A8: user_guide/quantization/fp8.md
|
||||
- Int8 W8A8: user_guide/quantization/int8.md
|
||||
- BitsAndBytes W4: user_guide/quantization/bitsandbytes.md
|
||||
- MXFP8 W8A8: user_guide/quantization/mxfp8.md
|
||||
- MXFP4 W4A4: user_guide/quantization/mxfp4.md
|
||||
- ModelOpt: user_guide/quantization/modelopt.md
|
||||
- GGUF: user_guide/quantization/gguf.md
|
||||
- AutoRound: user_guide/quantization/autoround.md
|
||||
- msModelSlim: user_guide/quantization/msmodelslim.md
|
||||
- ComfyUI: features/comfyui.md
|
||||
- Fault Tolerance:
|
||||
- Failure Modes: user_guide/fault_injection_reliability_matrix.md
|
||||
@@ -114,7 +108,6 @@ nav:
|
||||
- Shared Memory Connector: design/feature/omni_connectors/shared_memory_connector.md
|
||||
- Yuanrong Store Connector: design/feature/omni_connectors/yuanrong_connector.md
|
||||
- Yuanrong Transfer Engine Connector: design/feature/omni_connectors/yuanrong_transfer_engine_connector.md
|
||||
- Quantization: design/feature/quantization.md
|
||||
- Diffusion Acceleration:
|
||||
- Parallelism:
|
||||
- CFG-Parallel: design/feature/cfg_parallel.md
|
||||
@@ -124,8 +117,9 @@ nav:
|
||||
- Sequence Parallel: design/feature/sequence_parallel.md
|
||||
- Tensor Parallel: design/feature/tensor_parallel.md
|
||||
- VAE Patch Parallelism: design/feature/vae_parallel.md
|
||||
- Attention Optimization:
|
||||
- Attention Backends:
|
||||
- Skip-Softmax: design/feature/skip_softmax.md
|
||||
- Quantization: design/feature/quantization.md
|
||||
- Cache-DiT: design/feature/cache_dit.md
|
||||
- TeaCache: design/feature/teacache.md
|
||||
- Diffusion Continuous Batching: design/feature/diffusion_continuous_batching.md
|
||||
|
||||
+5
-13
@@ -11,10 +11,6 @@ in the active navigation.
|
||||
|
||||
## Feature Design Documents
|
||||
|
||||
For user-facing configuration and current compatibility, see the
|
||||
[Features overview](../features/README.md). A design document defines an
|
||||
implementation contract; it is not, by itself, a general support claim.
|
||||
|
||||
### Runtime and stage execution
|
||||
|
||||
- [Disaggregated Inference](feature/disaggregated_inference.md)
|
||||
@@ -34,10 +30,6 @@ implementation contract; it is not, by itself, a general support claim.
|
||||
- [Yuanrong Store Connector](feature/omni_connectors/yuanrong_connector.md)
|
||||
- [Yuanrong Transfer Engine Connector](feature/omni_connectors/yuanrong_transfer_engine_connector.md)
|
||||
|
||||
### Quantization
|
||||
|
||||
- [Quantization](feature/quantization.md)
|
||||
|
||||
### Diffusion acceleration
|
||||
|
||||
#### Parallelism
|
||||
@@ -50,14 +42,14 @@ implementation contract; it is not, by itself, a general support claim.
|
||||
- [Tensor Parallel](feature/tensor_parallel.md)
|
||||
- [VAE Patch Parallelism](feature/vae_parallel.md)
|
||||
|
||||
#### Attention optimization
|
||||
|
||||
The [Diffusion Attention Backends](../user_guide/diffusion/attention_backends.md)
|
||||
guide lists all selectable backends and their platform defaults. Skip-Softmax
|
||||
is the backend-related optimization with a standalone feature-design contract:
|
||||
#### Attention Backends
|
||||
|
||||
- [Skip-Softmax](feature/skip_softmax.md)
|
||||
|
||||
#### Quantization
|
||||
|
||||
- [Quantization](feature/quantization.md)
|
||||
|
||||
- [Cache-DiT](feature/cache_dit.md)
|
||||
- [TeaCache](feature/teacache.md)
|
||||
- [Diffusion Continuous Batching](feature/diffusion_continuous_batching.md)
|
||||
|
||||
@@ -1,67 +0,0 @@
|
||||
# Features
|
||||
|
||||
Use this section to choose and configure vLLM-Omni runtime and optimization
|
||||
features. The navigation follows the active
|
||||
[Feature Design](../design/index.md#feature-design-documents) taxonomy whenever
|
||||
an implementation contract has a user-facing workflow.
|
||||
|
||||
!!! note "User guides, design documents, and recipes"
|
||||
|
||||
User guides explain how to enable a feature and where it is supported.
|
||||
Design documents define internal contracts and do not, by themselves,
|
||||
imply general or production support. For model-specific launch flags and
|
||||
hardware requirements, use the linked examples and validated recipes from
|
||||
[Supported Models](../models/supported_models.md).
|
||||
|
||||
## Runtime and Stage Execution
|
||||
|
||||
| Goal | User guide | Related design contract |
|
||||
| --- | --- | --- |
|
||||
| Choose serial, batched, step-wise, or streaming diffusion execution | [Execution Modes and Streaming](../user_guide/diffusion/execution_modes.md) | [Diffusion Continuous Batching](../design/feature/diffusion_continuous_batching.md), [Async Diffusion Output](../design/feature/async_diffusion_output.md) |
|
||||
| Reclaim stage memory without restarting the server | [Sleep Mode](sleep_mode.md) | Runtime lifecycle behavior is documented in the user guide |
|
||||
|
||||
Some runtime designs are deliberately not promoted as standalone User Guide
|
||||
features yet:
|
||||
|
||||
- [Disaggregated Inference](../design/feature/disaggregated_inference.md) and
|
||||
[OmniConnector implementations](../design/index.md#communication) describe
|
||||
topology and transport contracts. Practical deployment configuration remains
|
||||
under [Pipeline and deploy configurations](../configuration/stage_configs.md).
|
||||
- [Async Chunk](../design/feature/async_chunk.md) and
|
||||
[Async Omni Output Materialization](../design/feature/omni_async_output_materialization.md)
|
||||
are model- and pipeline-dependent. Use the selected model's deploy
|
||||
configuration and recipe for the supported settings.
|
||||
- [Automatic Prefix Caching](../design/feature/prefix_caching.md) remains a
|
||||
design-level contract until its user-facing configuration and compatibility
|
||||
surface is consolidated.
|
||||
|
||||
## Quantization
|
||||
|
||||
Quantization is a cross-model feature rather than a diffusion-only
|
||||
optimization. The unified [`quantization_config` guide](../user_guide/quantization/overview.md)
|
||||
covers diffusion-only models, multi-stage omni/TTS models, and multi-stage
|
||||
diffusion models. Its [design contract](../design/feature/quantization.md)
|
||||
defines the shared configuration and backend extension points.
|
||||
|
||||
## Diffusion Acceleration
|
||||
|
||||
| Goal | User guide | Related design contract |
|
||||
| --- | --- | --- |
|
||||
| Compare acceleration methods and supported combinations | [Overview](../user_guide/diffusion_features.md), [Feature Compatibility](../user_guide/feature_compatibility.md) | [Diffusion acceleration designs](../design/index.md#diffusion-acceleration) |
|
||||
| Move weights between host and device memory | [CPU Offloading](../user_guide/diffusion/cpu_offload.md) | [Distributed Layerwise Offload](../design/feature/distributed_layerwise_offload.md) |
|
||||
| Reuse denoising computation | [Cache Acceleration](../user_guide/diffusion/cache_acceleration/cache_dit.md) | [Cache-DiT](../design/feature/cache_dit.md), [TeaCache](../design/feature/teacache.md) |
|
||||
| Distribute diffusion work across devices | [Parallelism](../user_guide/diffusion/parallelism/overview.md) | [Parallelism designs](../design/index.md#parallelism) |
|
||||
| Select dense, sparse, or quantized attention paths | [Attention Backends](../user_guide/diffusion/attention_backends.md) | [Skip-Softmax](../design/feature/skip_softmax.md) |
|
||||
| Compile repeated diffusion regions | [Regional Compilation](../user_guide/diffusion/regional_compilation.md) | User-facing optimization guide |
|
||||
| Add generated video frames | [Frame Interpolation](../user_guide/diffusion/frame_interpolation.md) | User-facing extension guide |
|
||||
| Reduce diffusion model startup time | [Startup and Loading](../user_guide/diffusion/startup_and_loading.md) | User-facing loading guide |
|
||||
| Apply diffusion adapters | [LoRA](../user_guide/diffusion/lora.md) | User-facing extension guide |
|
||||
|
||||
The [Pipeline Parallelism guide](../user_guide/diffusion/parallelism/pipeline_parallel.md)
|
||||
remains available by direct link, but it is not promoted in the primary User
|
||||
Guide navigation until its user-facing support and placement are settled.
|
||||
|
||||
## Experimental
|
||||
|
||||
[Session State Manager](session_state_manager.md) is opt-in and experimental.
|
||||
Its APIs and compatibility may change without notice.
|
||||
@@ -1,60 +0,0 @@
|
||||
# Diffusion Startup and Loading
|
||||
|
||||
Large diffusion models can take several minutes to load at startup. vLLM-Omni
|
||||
loads safetensors shards in parallel to reduce this initialization time.
|
||||
|
||||
Multi-thread weight loading is enabled by default with four threads. No
|
||||
configuration is needed for the default behavior.
|
||||
|
||||
## Configuration
|
||||
|
||||
| Parameter | CLI flag | Default | Description |
|
||||
| --- | --- | --- | --- |
|
||||
| `enable_multithread_weight_load` | `--disable-multithread-weight-load` | `True` | Pass the flag to disable multi-thread loading |
|
||||
| `num_weight_load_threads` | `--num-weight-load-threads` | `4` | Number of parallel weight-loading threads |
|
||||
|
||||
!!! tip
|
||||
|
||||
The default balances startup speed and disk I/O contention. Fast NVMe
|
||||
storage may benefit from more threads, while network storage or hard disks
|
||||
may not.
|
||||
|
||||
## Online Serving
|
||||
|
||||
```bash
|
||||
# Default: multi-thread loading with four threads
|
||||
vllm serve Qwen/Qwen-Image --omni --port 8091
|
||||
|
||||
# Increase the thread count
|
||||
vllm serve Wan-AI/Wan2.2-I2V-A14B-Diffusers --omni \
|
||||
--num-weight-load-threads 8
|
||||
|
||||
# Disable multi-thread loading
|
||||
vllm serve Qwen/Qwen-Image --omni --disable-multithread-weight-load
|
||||
```
|
||||
|
||||
## Offline Inference
|
||||
|
||||
```python
|
||||
from vllm_omni import Omni
|
||||
|
||||
# Default: multi-thread loading with four threads
|
||||
omni = Omni(model="Qwen/Qwen-Image")
|
||||
|
||||
# Increase the thread count
|
||||
omni = Omni(
|
||||
model="Wan-AI/Wan2.2-I2V-A14B-Diffusers",
|
||||
num_weight_load_threads=8,
|
||||
)
|
||||
```
|
||||
|
||||
## Reference Benchmarks
|
||||
|
||||
The following measurements were collected on NVIDIA H800 hardware. Treat them
|
||||
as reference results rather than a guarantee for other storage or hardware
|
||||
configurations.
|
||||
|
||||
| Model | Sequential loading | Multi-thread loading | Speedup |
|
||||
| --- | ---: | ---: | ---: |
|
||||
| **Qwen/Qwen-Image** (53.7 GiB) | 168 s | 27 s | **6.2x** |
|
||||
| **Wan-AI/Wan2.2-I2V-A14B-Diffusers** (64.5 GiB) | 283 s | 56 s | **5.1x** |
|
||||
@@ -49,7 +49,7 @@ Parallelism methods distribute computation across GPUs without quality loss (mat
|
||||
|
||||
| Method | Description | Best For |
|
||||
|--------|-------------|----------|
|
||||
| **[Multi-Thread Weight Loading](diffusion/startup_and_loading.md)** | Loads safetensors shards in parallel using a thread pool | All diffusion models; reduces startup from minutes to seconds |
|
||||
| **[Multi-Thread Weight Loading](#multi-thread-weight-loading)** | Loads safetensors shards in parallel using a thread pool | All diffusion models; reduces startup from minutes to seconds |
|
||||
|
||||
**Note:** Some acceleration methods can be combined together for optimized performance. See [Feature Compatibility Table](#feature-compatibility) and [Feature Compatibility Tutorial](feature_compatibility.md) for detailed configuration examples.
|
||||
|
||||
@@ -100,9 +100,7 @@ The following tables show which models support each feature:
|
||||
|
||||
- **🔀SP (Ulysses & Ring)**: Includes both Ulysses-SP and Ring-Attention methods
|
||||
- ✅ = Fully supported
|
||||
- ✅* = Supported with the constraint listed below the table
|
||||
- ❌ = Not supported
|
||||
- ❓ = Not verified; not recommended
|
||||
|
||||
> Notes:
|
||||
|
||||
@@ -121,7 +119,7 @@ The following tables show which models support each feature:
|
||||
| **FLUX.2-dev** | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ |
|
||||
| **GLM-Image** | ❌ | ❌ | ❌ | ✅ | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ |
|
||||
| **Hidream-I1-Full** | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ |
|
||||
| **HunyuanImage3** | ❌ | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅* |
|
||||
| **HunyuanImage3** | ❌ | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ |
|
||||
| **Krea 2** | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ✅ (decode) | ❌ | ❌ |
|
||||
| **LongCat-Image** | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ |
|
||||
| **LongCat-Image-Edit** | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ |
|
||||
@@ -147,7 +145,6 @@ The following tables show which models support each feature:
|
||||
> 2. `Tongyi-MAI/Z-Image-Turbo` and `SII-GAIR/daVinci-MagiHuman-Base-1080p` are distilled models with minimal NFEs; CFG-Parallel is not necessary.
|
||||
> 3. Cosmos3 T2I uses `Cosmos3OmniDiffusersPipeline` with `modalities=["image"]`. Model-level CPU offload swaps the nested UND reasoner and GEN generator pathways; layerwise offload remains available for blockwise GEN/UND offload.
|
||||
> 4. Krea 2 currently supports single-GPU inference plus LoRA, Cache-DiT, HSDP, CPU/layerwise offload, and VAE-patch-parallel (decode). TP/SP/CFG-Parallel are not yet wired. The few-step distilled (Turbo) checkpoint uses `is_distilled=true` (fixed timestep shift `mu=1.15`); generate at 2048x2048 by default with `num_inference_steps≈8` and `guidance_scale=0`. The Raw checkpoint uses 1024x1024, `num_inference_steps=28`, and `guidance_scale=4.5`.
|
||||
> 5. HunyuanImage3 supports step execution. Multi-request step execution requires `TORCH_SDPA`; see [Diffusion Execution Modes](diffusion/execution_modes.md#step-execution).
|
||||
|
||||
### VideoGen
|
||||
|
||||
@@ -158,16 +155,13 @@ The following tables show which models support each feature:
|
||||
| **Wan2.1-VACE** | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ (decode) | ❌ | ❌ |
|
||||
| **LTX-2** | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ (decode) | ❌ | ❌ |
|
||||
| **LTX-2.3** | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ (decode) | ❌ | ❌ |
|
||||
| **Helios** | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ❌ | ❌ | ✅* |
|
||||
| **Helios** | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ |
|
||||
| **HunyuanVideo-1.5 T2V I2V** | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ (encode/decode) | ✅ | ❌ |
|
||||
| **DreamID-Omni** | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ |
|
||||
| **Cosmos3** | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ (encode/decode) | ✅ | ❌ |
|
||||
| **LongCat-Video-Avatar-1.5** | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ |
|
||||
| **MiniMax-H3** | ✅ (FL2VA) | ✅ | ✅ | ❌ | ✅ (DiT/TE) | ❌ | ✅ | ✅ | ✅ (tile) | ✅ (DiT) | ❌ |
|
||||
|
||||
> **Step execution note:** Helios supports single-request step execution only;
|
||||
> use `max_num_seqs=1`.
|
||||
|
||||
**Frame Interpolation Support**
|
||||
|
||||
- **Supported**: Wan2.2 text-to-video, image-to-video, and TI2V pipelines
|
||||
@@ -196,7 +190,7 @@ The following tables show which models support each feature:
|
||||
| **🔀Ring-Attn** | ✅ | ✅ | ✅ | | | | | | | | | | | |
|
||||
| **🔀CFG-Parallel** | ✅ | ✅ | ✅ | ✅ | | | | | | | | | | |
|
||||
| **🔀Tensor Parallel** | ✅ | ✅ | ✅ | ✅ | ✅ | | | | | | | | | |
|
||||
| **🔀HSDP** | ❓ | ❓ | ✅ | ❓ | ✅ | ❌ | | | | | | | | |
|
||||
| **🔀HSDP** | ❓ | ❓ | ❓ | ❓ | ❓ | ❌ | | | | | | | | |
|
||||
| **🔀Expert Parallel** | ❓ | ❓ | ❓ | ❓ | ❓ | ❓ | ❓ | | | | | | | |
|
||||
| **💾CPU Offloading (Layerwise)** | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | | | | | | |
|
||||
| **💾CPU Offloading (Module-wise)** | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❓ | ❓ | ❌ | | | | | |
|
||||
@@ -207,43 +201,97 @@ The following tables show which models support each feature:
|
||||
|
||||
!!! info
|
||||
|
||||
1. HSDP can be combined with Ulysses-SP or CFG-Parallel. Tensor Parallel
|
||||
and HSDP are not compatible; other HSDP combinations in the table
|
||||
remain unverified.
|
||||
1. Tensor Parallel and HSDP are not compatible.
|
||||
2. TeaCache and Cache-DiT are not compatible.
|
||||
3. CPU Offloading (Layerwise) and CPU Offloading (Module-wise) are not compatible.
|
||||
4. The CPU Offloading (Layerwise) row describes local layerwise offload.
|
||||
Multi-device Distributed Layerwise Offload has a separate topology and
|
||||
compatibility matrix in the [CPU Offloading guide](diffusion/cpu_offload.md#distributed-layerwise-offloading).
|
||||
5. The compatibility matrix uses FP8 as the representative quantization method.
|
||||
4. CPU Offloading (Layerwise) supports single-card for now.
|
||||
5. Using FP8-Quant as an example of qunatization methods.
|
||||
6. Step Execution is not compatible with any diffusion cache backend. LoRA is supported, but each scheduled batch must use a single adapter (requests with different `lora_request` or `lora_scale` are kept in separate batches).
|
||||
|
||||
|
||||
## Multi-Thread Weight Loading
|
||||
|
||||
The loading guide now lives at [Diffusion Startup and
|
||||
Loading](diffusion/startup_and_loading.md). This heading remains so existing
|
||||
links to this section continue to work.
|
||||
Large diffusion models can take several minutes to load weights at startup (e.g., ~3 min for Qwen-Image, ~5 min for Wan2.2 I2V 14B). Multi-thread weight loading speeds up this process by loading safetensors shards in parallel using a thread pool instead of sequentially.
|
||||
|
||||
This optimization is **enabled by default** with 4 threads. No configuration is needed for the default behavior.
|
||||
|
||||
### Configuration
|
||||
|
||||
| Parameter | CLI Flag | Default | Description |
|
||||
|-----------|----------|---------|-------------|
|
||||
| `enable_multithread_weight_load` | `--disable-multithread-weight-load` | `True` (enabled) | Pass the flag to disable multi-thread loading |
|
||||
| `num_weight_load_threads` | `--num-weight-load-threads` | `4` | Number of threads for parallel weight loading |
|
||||
|
||||
!!! tip
|
||||
The default of 4 threads balances speed and disk I/O contention. On fast NVMe storage you may benefit from more threads (e.g., 8). On HDD or network storage, the default of 4 avoids saturating I/O bandwidth.
|
||||
|
||||
### Online Serving
|
||||
|
||||
```bash
|
||||
# Default (multi-thread enabled, 4 threads)
|
||||
vllm serve Qwen/Qwen-Image --omni --port 8091
|
||||
|
||||
# Custom thread count
|
||||
vllm serve Wan-AI/Wan2.2-I2V-A14B-Diffusers --omni --num-weight-load-threads 8
|
||||
|
||||
# Disable multi-thread loading
|
||||
vllm serve Qwen/Qwen-Image --omni --disable-multithread-weight-load
|
||||
```
|
||||
|
||||
### Offline Inference
|
||||
|
||||
```python
|
||||
from vllm_omni import Omni
|
||||
|
||||
# Default (multi-thread enabled, 4 threads)
|
||||
omni = Omni(model="Qwen/Qwen-Image")
|
||||
|
||||
# Custom thread count
|
||||
omni = Omni(
|
||||
model="Wan-AI/Wan2.2-I2V-A14B-Diffusers",
|
||||
num_weight_load_threads=8,
|
||||
)
|
||||
```
|
||||
|
||||
### Benchmarks
|
||||
|
||||
Measured on NVIDIA H800:
|
||||
|
||||
| Model | Before | After | Speedup |
|
||||
|-------|--------|-------|---------|
|
||||
| **Qwen/Qwen-Image** (53.7 GiB) | 168s | 27s | **6.2x** |
|
||||
| **Wan-AI/Wan2.2-I2V-A14B-Diffusers** (64.5 GiB) | 283s | 56s | **5.1x** |
|
||||
|
||||
## Learn More
|
||||
|
||||
The Diffusion Acceleration navigation groups the remaining guides as follows:
|
||||
**Cache Acceleration:**
|
||||
|
||||
| Area | Guide |
|
||||
| --- | --- |
|
||||
| Compatibility | [Feature Compatibility](feature_compatibility.md) |
|
||||
| CPU offloading | [CPU Offloading](diffusion/cpu_offload.md) |
|
||||
| Cache acceleration | [TeaCache](diffusion/cache_acceleration/teacache.md), [Cache-DiT](diffusion/cache_acceleration/cache_dit.md) |
|
||||
| Parallelism | [Parallelism Overview](diffusion/parallelism/overview.md) |
|
||||
| Attention | [Attention Backends](diffusion/attention_backends.md) |
|
||||
| Compilation | [Regional Compilation](diffusion/regional_compilation.md) |
|
||||
| Video extension | [Frame Interpolation](diffusion/frame_interpolation.md) |
|
||||
| Startup | [Startup and Loading](diffusion/startup_and_loading.md) |
|
||||
| Adapters | [LoRA](diffusion/lora.md) |
|
||||
- **[TeaCache Configuration Guide](diffusion/cache_acceleration/teacache.md)** - Parameter tuning, performance tips, troubleshooting
|
||||
- **[Cache-DiT Advanced Guide](diffusion/cache_acceleration/cache_dit.md)** - DBCache, TaylorSeer, SCM techniques and optimization
|
||||
|
||||
Related cross-model and runtime features are documented separately:
|
||||
**Parallelism Methods:**
|
||||
|
||||
- [Quantization](quantization/overview.md) covers diffusion-only models,
|
||||
multi-stage omni/TTS models, and multi-stage diffusion models.
|
||||
- [Execution Modes and Streaming](diffusion/execution_modes.md) covers the
|
||||
diffusion runtime, including batching, step execution, and streaming output.
|
||||
- **[Parallelism Overview](diffusion/parallelism/overview.md)** - Tensor Parallelism, Sequence Parallelism, CFG Parallelism, Pipeline Parallelism, HSDP, and Expert Parallelism
|
||||
|
||||
**Memory Optimization:**
|
||||
|
||||
- **[CPU Offload Guide](diffusion/cpu_offload.md)** - Offload model components to CPU, reduce GPU memory usage
|
||||
- **[VAE Parallelism Guide](diffusion/parallelism/vae_parallelism.md)** - Distribute VAE decode work across GPUs for high-resolution images and videos
|
||||
- **[Quantization](quantization/overview.md)** - Quantization methods for diffusion, multi-stage omni/TTS, and multi-stage diffusion models
|
||||
|
||||
**Extensions:**
|
||||
|
||||
- **[LoRA Inference Guide](diffusion/lora.md)** - Low-Rank Adaptation for style customization and fine-tuning
|
||||
- **[Frame Interpolation Guide](diffusion/frame_interpolation.md)** - Worker-side post-generation video frame interpolation for smoother motion
|
||||
|
||||
**Execution Modes:**
|
||||
|
||||
- **[Diffusion Execution Modes](diffusion/execution_modes.md)** - Configure request batching, step execution, continuous batching, and streaming output
|
||||
|
||||
**Startup Optimization:**
|
||||
|
||||
- **[Multi-Thread Weight Loading](#multi-thread-weight-loading)** - Speed up model startup by loading safetensors shards in parallel
|
||||
|
||||
**Advanced Topics:**
|
||||
|
||||
- **[Feature Compatibility](feature_compatibility.md)** - How to combine multiple features for maximum performance
|
||||
@@ -61,7 +61,9 @@ from vllm_omni.diffusion.utils.param_utils import apply_declared_extra_args
|
||||
from vllm_omni.entrypoints.omni import Omni
|
||||
from vllm_omni.inputs.data import OmniDiffusionSamplingParams
|
||||
from vllm_omni.model_extras import (
|
||||
build_image_to_video_prompt,
|
||||
build_image_to_video_prompt as build_model_image_to_video_prompt,
|
||||
)
|
||||
from vllm_omni.model_extras import (
|
||||
get_extra_body_params,
|
||||
get_model_class_name,
|
||||
)
|
||||
@@ -83,6 +85,22 @@ def parse_json_object(value: str, flag_name: str = "argument") -> dict[str, Any]
|
||||
parse_profiler_config = functools.partial(parse_json_object, flag_name="--profiler-config")
|
||||
|
||||
|
||||
def build_image_to_video_prompt(
|
||||
prompt: str,
|
||||
negative_prompt: str | None,
|
||||
media_inputs: dict[str, Any],
|
||||
) -> dict[str, Any]:
|
||||
"""Build the canonical request envelope for the shared I2V example."""
|
||||
result: dict[str, Any] = {
|
||||
"prompt": prompt,
|
||||
"modalities": ["video"],
|
||||
"multi_modal_data": media_inputs,
|
||||
}
|
||||
if negative_prompt is not None:
|
||||
result["negative_prompt"] = negative_prompt
|
||||
return result
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(
|
||||
description=(
|
||||
@@ -522,10 +540,13 @@ def main():
|
||||
# Preserve the historical empty-prompt behavior for non-LTX examples.
|
||||
negative_prompt = ""
|
||||
prompt_dict = build_image_to_video_prompt(
|
||||
model_class_name=model_class_name,
|
||||
prompt=args.prompt,
|
||||
negative_prompt=negative_prompt,
|
||||
media_inputs=media_inputs,
|
||||
)
|
||||
prompt_dict = build_model_image_to_video_prompt(
|
||||
model_class_name=model_class_name,
|
||||
prompt=prompt_dict,
|
||||
height=height,
|
||||
width=width,
|
||||
num_frames=num_frames,
|
||||
|
||||
@@ -57,8 +57,9 @@ python examples/offline_inference/text_to_image/text_to_image.py \
|
||||
--output lingbot_t2i.png
|
||||
```
|
||||
|
||||
The LingBot prompt builder selects the `image` output modality and requests one
|
||||
frame. Passing an input image is not supported in LingBot T2I mode.
|
||||
The shared T2I entry point selects the `image` output modality. The LingBot
|
||||
pipeline validates that T2I requests produce one frame and do not include an
|
||||
input image.
|
||||
|
||||
## Quick Start
|
||||
|
||||
|
||||
@@ -21,7 +21,9 @@ from vllm_omni.inputs.data import OmniDiffusionSamplingParams
|
||||
from vllm_omni.lora.request import LoRARequest
|
||||
from vllm_omni.lora.utils import stable_lora_int_id
|
||||
from vllm_omni.model_extras import (
|
||||
build_text_to_image_prompt,
|
||||
build_text_to_image_prompt as build_model_text_to_image_prompt,
|
||||
)
|
||||
from vllm_omni.model_extras import (
|
||||
get_extra_body_params,
|
||||
get_model_class_name,
|
||||
should_init_extra_args_for_non_diffusion_stages,
|
||||
@@ -56,6 +58,17 @@ def parse_json_object(value: str, flag_name: str = "argument") -> dict[str, Any]
|
||||
parse_profiler_config = functools.partial(parse_json_object, flag_name="--profiler-config")
|
||||
|
||||
|
||||
def build_text_to_image_prompt(prompt: str, negative_prompt: str | None) -> dict[str, Any]:
|
||||
"""Build the canonical request envelope for the shared T2I example."""
|
||||
result: dict[str, Any] = {
|
||||
"prompt": prompt,
|
||||
"modalities": ["image"],
|
||||
}
|
||||
if negative_prompt is not None:
|
||||
result["negative_prompt"] = negative_prompt
|
||||
return result
|
||||
|
||||
|
||||
def _normalize_images_for_save(images: list[Any]) -> list[Any]:
|
||||
"""Convert NumPy diffusion outputs to PIL images before saving."""
|
||||
normalized = []
|
||||
@@ -505,9 +518,12 @@ def main():
|
||||
generation_start = time.perf_counter()
|
||||
|
||||
prompt_dict = build_text_to_image_prompt(
|
||||
model_class_name=model_class_name,
|
||||
prompt=args.prompt,
|
||||
negative_prompt=args.negative_prompt,
|
||||
)
|
||||
prompt_dict = build_model_text_to_image_prompt(
|
||||
model_class_name=model_class_name,
|
||||
prompt=prompt_dict,
|
||||
height=args.height,
|
||||
width=args.width,
|
||||
)
|
||||
|
||||
@@ -1,6 +1,9 @@
|
||||
# Image-To-Video
|
||||
|
||||
This example demonstrates how to deploy the Wan2.2 image-to-video model for online video generation using vLLM-Omni.
|
||||
This example demonstrates online image-to-video generation with vLLM-Omni.
|
||||
The startup script defaults to Wan2.2, but `MODEL` can select any supported
|
||||
image-to-video model. The existing curl helper remains a concrete Wan2.2
|
||||
request through the standard Videos API.
|
||||
|
||||
## Start Server
|
||||
|
||||
@@ -20,9 +23,7 @@ bash run_server.sh
|
||||
|
||||
The script allows overriding:
|
||||
- `MODEL` (default: `Wan-AI/Wan2.2-I2V-A14B-Diffusers`)
|
||||
- `PORT` (default: `8091`)
|
||||
- `BOUNDARY_RATIO` (default: `0.875`)
|
||||
- `FLOW_SHIFT` (default: `12.0`)
|
||||
- `PORT` (default: `8099`)
|
||||
- `CACHE_BACKEND` (default: `none`)
|
||||
- `ENABLE_CACHE_DIT_SUMMARY` (default: `0`)
|
||||
|
||||
|
||||
@@ -1,12 +1,12 @@
|
||||
#!/bin/bash
|
||||
# Wan2.2 image-to-video server start script
|
||||
# Shared image-to-video server start script
|
||||
|
||||
MODEL="${MODEL:-Wan-AI/Wan2.2-I2V-A14B-Diffusers}"
|
||||
PORT="${PORT:-8099}"
|
||||
CACHE_BACKEND="${CACHE_BACKEND:-none}"
|
||||
ENABLE_CACHE_DIT_SUMMARY="${ENABLE_CACHE_DIT_SUMMARY:-0}"
|
||||
|
||||
echo "Starting Wan2.2 I2V server..."
|
||||
echo "Starting image-to-video server..."
|
||||
echo "Model: $MODEL"
|
||||
echo "Port: $PORT"
|
||||
echo "Cache backend: $CACHE_BACKEND"
|
||||
|
||||
@@ -1,6 +1,9 @@
|
||||
# Text-To-Image
|
||||
|
||||
This example demonstrates how to deploy Qwen-Image model for online image generation service using vLLM-Omni.
|
||||
This example demonstrates online text-to-image generation with vLLM-Omni. The
|
||||
startup script and Python clients default to Qwen-Image, but `MODEL` can select
|
||||
any supported text-to-image model. The existing curl helper remains a concrete
|
||||
Qwen-Image request through the standard Images API.
|
||||
|
||||
## Start Server
|
||||
|
||||
@@ -226,8 +229,8 @@ count, use `size` and `n` rather than `height`, `width`, or
|
||||
| `height` | int | None | Image height in pixels |
|
||||
| `width` | int | None | Image width in pixels |
|
||||
| `size` | str | None | Image size (e.g., "1024x1024") |
|
||||
| `num_inference_steps` | int | 50 | Number of denoising steps |
|
||||
| `true_cfg_scale` | float | 4.0 | Qwen-Image CFG scale |
|
||||
| `num_inference_steps` | int | model | Number of denoising steps |
|
||||
| `true_cfg_scale` | float | model | True CFG scale when supported |
|
||||
| `seed` | int | None | Random seed (reproducible) |
|
||||
| `negative_prompt` | str | None | Negative prompt |
|
||||
| `num_outputs_per_prompt` | int | 1 | Number of images to generate |
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Qwen-Image Gradio Demo for online serving.
|
||||
Shared text-to-image Gradio demo for online serving.
|
||||
|
||||
Usage:
|
||||
python gradio_demo.py [--server http://localhost:8091] [--port 7860]
|
||||
@@ -77,9 +77,9 @@ def generate_image(
|
||||
def create_demo(server_url: str):
|
||||
"""Create Gradio demo interface."""
|
||||
|
||||
with gr.Blocks(title="Qwen-Image Demo") as demo:
|
||||
gr.Markdown("# Qwen-Image Online Generation")
|
||||
gr.Markdown("Generate images using Qwen-Image model")
|
||||
with gr.Blocks(title="Text-to-Image Demo") as demo:
|
||||
gr.Markdown("# Text-to-Image Online Generation")
|
||||
gr.Markdown("Generate images using the model configured on the server")
|
||||
|
||||
with gr.Row():
|
||||
with gr.Column(scale=1):
|
||||
@@ -115,7 +115,6 @@ def create_demo(server_url: str):
|
||||
label="Inference Steps",
|
||||
minimum=10,
|
||||
maximum=100,
|
||||
# Default steps aligned with run_curl_text_to_image.sh to 100
|
||||
value=100,
|
||||
step=5,
|
||||
)
|
||||
@@ -173,7 +172,7 @@ def create_demo(server_url: str):
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description="Qwen-Image Gradio Demo")
|
||||
parser = argparse.ArgumentParser(description="Shared text-to-image Gradio demo")
|
||||
parser.add_argument("--server", default="http://localhost:8091", help="Server URL")
|
||||
parser.add_argument("--port", type=int, default=7860, help="Gradio port")
|
||||
parser.add_argument("--share", action="store_true", help="Create public link")
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Qwen-Image OpenAI-compatible image generation client.
|
||||
Shared OpenAI-compatible text-to-image generation client.
|
||||
|
||||
Usage:
|
||||
python openai_chat_client.py --prompt "A beautiful landscape" --output output.png
|
||||
@@ -39,7 +39,7 @@ def generate_image(
|
||||
height: Image height in pixels
|
||||
width: Image width in pixels
|
||||
steps: Number of diffusion steps
|
||||
true_cfg_scale: Qwen-Image CFG scale
|
||||
true_cfg_scale: True CFG scale for models that support it
|
||||
seed: Random seed
|
||||
negative_prompt: Negative prompt
|
||||
num_outputs_per_prompt: Number of images to generate
|
||||
@@ -70,7 +70,7 @@ def generate_image(
|
||||
payload["num_inference_steps"] = steps
|
||||
if true_cfg_scale is not None:
|
||||
payload["true_cfg_scale"] = true_cfg_scale
|
||||
if negative_prompt:
|
||||
if negative_prompt is not None:
|
||||
payload["negative_prompt"] = negative_prompt
|
||||
if seed is not None:
|
||||
payload["seed"] = seed
|
||||
@@ -115,15 +115,15 @@ def generate_image(
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description="Qwen-Image chat client")
|
||||
parser = argparse.ArgumentParser(description="Shared text-to-image client")
|
||||
parser.add_argument("--prompt", "-p", default="a cup of coffee on the table", help="Text prompt")
|
||||
parser.add_argument("--output", "-o", default="qwen_image_output.png", help="Output file")
|
||||
parser.add_argument("--output", "-o", default="text_to_image_output.png", help="Output file")
|
||||
parser.add_argument("--server", "-s", default="http://localhost:8091", help="Server URL")
|
||||
parser.add_argument("--height", type=int, default=1024, help="Image height")
|
||||
parser.add_argument("--width", type=int, default=1024, help="Image width")
|
||||
parser.add_argument("--steps", type=int, default=50, help="Inference steps")
|
||||
parser.add_argument("--cfg-scale", type=float, default=4.0, help="True CFG scale")
|
||||
parser.add_argument("--seed", type=int, default=0, help="Random seed")
|
||||
parser.add_argument("--height", type=int, help="Image height; defaults to the model setting")
|
||||
parser.add_argument("--width", type=int, help="Image width; defaults to the model setting")
|
||||
parser.add_argument("--steps", type=int, help="Inference steps; defaults to the model setting")
|
||||
parser.add_argument("--cfg-scale", type=float, help="True CFG scale when supported")
|
||||
parser.add_argument("--seed", type=int, help="Random seed")
|
||||
parser.add_argument("--negative", help="Negative prompt")
|
||||
|
||||
parser.add_argument("--lora-path", default=None, help="Server-local LoRA adapter folder (PEFT format)")
|
||||
|
||||
@@ -1,10 +1,10 @@
|
||||
#!/bin/bash
|
||||
# Qwen-Image online serving startup script
|
||||
# Shared text-to-image online serving startup script
|
||||
|
||||
MODEL="${MODEL:-Qwen/Qwen-Image}"
|
||||
PORT="${PORT:-8091}"
|
||||
|
||||
echo "Starting Qwen-Image server..."
|
||||
echo "Starting text-to-image server..."
|
||||
echo "Model: $MODEL"
|
||||
echo "Port: $PORT"
|
||||
|
||||
|
||||
@@ -2206,6 +2206,24 @@ class TestBaseConfigInheritance:
|
||||
class TestPlatformOverrides:
|
||||
"""Test platform-specific deploy config overrides."""
|
||||
|
||||
def test_qwen3_omni_musa_clears_parent_quantization_for_audio_stages(self):
|
||||
pipeline = resolve_pipeline_config(
|
||||
"qwen3_omni_moe",
|
||||
Q3_OMNI_ALL_STAGES_HF_CONFIG,
|
||||
)
|
||||
assert isinstance(pipeline, PipelineConfig)
|
||||
deploy_path = Path(get_deploy_config_path("qwen3_omni_moe.yaml"))
|
||||
|
||||
base_stages = merge_pipeline_deploy(pipeline, load_deploy_config(deploy_path))
|
||||
assert "hf_overrides" not in base_stages[1].yaml_engine_args
|
||||
assert "hf_overrides" not in base_stages[2].yaml_engine_args
|
||||
|
||||
musa = _apply_platform_overrides(load_deploy_config(deploy_path), platform="musa")
|
||||
musa_stages = merge_pipeline_deploy(pipeline, musa)
|
||||
expected = {"quantization_config": None}
|
||||
assert musa_stages[1].yaml_engine_args["hf_overrides"] == expected
|
||||
assert musa_stages[2].yaml_engine_args["hf_overrides"] == expected
|
||||
|
||||
def test_qwen3_tts_rocm_disables_code2wav_outer_cudagraph(self):
|
||||
deploy_path = Path(get_deploy_config_path("qwen3_tts.yaml"))
|
||||
|
||||
|
||||
@@ -0,0 +1,47 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
from PIL import Image
|
||||
|
||||
from examples.offline_inference.image_to_video.image_to_video import build_image_to_video_prompt
|
||||
from examples.offline_inference.text_to_image.text_to_image import build_text_to_image_prompt
|
||||
|
||||
pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("negative_prompt", "expected_negative_prompt"),
|
||||
[(None, None), ("", ""), ("blurry", "blurry")],
|
||||
)
|
||||
def test_text_to_image_builds_canonical_prompt(
|
||||
negative_prompt: str | None,
|
||||
expected_negative_prompt: str | None,
|
||||
) -> None:
|
||||
result = build_text_to_image_prompt("a red fox", negative_prompt)
|
||||
|
||||
assert result["prompt"] == "a red fox"
|
||||
assert result["modalities"] == ["image"]
|
||||
if expected_negative_prompt is None:
|
||||
assert "negative_prompt" not in result
|
||||
else:
|
||||
assert result["negative_prompt"] == expected_negative_prompt
|
||||
|
||||
|
||||
def test_image_to_video_builds_canonical_prompt() -> None:
|
||||
image = Image.new("RGB", (32, 16), "red")
|
||||
|
||||
result = build_image_to_video_prompt(
|
||||
prompt="the fox turns toward the camera",
|
||||
negative_prompt="flicker",
|
||||
media_inputs={"image": image},
|
||||
)
|
||||
|
||||
assert result == {
|
||||
"prompt": "the fox turns toward the camera",
|
||||
"modalities": ["video"],
|
||||
"multi_modal_data": {"image": image},
|
||||
"negative_prompt": "flicker",
|
||||
}
|
||||
@@ -3,6 +3,8 @@
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
from PIL import Image
|
||||
|
||||
@@ -135,67 +137,6 @@ def test_lingbot_extra_registry_declares_request_params() -> None:
|
||||
assert should_init_extra_args_for_non_diffusion_stages("LingBotVideoPipeline") is False
|
||||
|
||||
|
||||
@pytest.mark.core_model
|
||||
@pytest.mark.cpu
|
||||
def test_lingbot_text_to_image_prompt_builder_preserves_empty_negative_prompt() -> None:
|
||||
assert build_text_to_image_prompt(
|
||||
"LingBotVideoPipeline",
|
||||
prompt="a red fox in fresh snow",
|
||||
negative_prompt="",
|
||||
height=192,
|
||||
width=320,
|
||||
) == {
|
||||
"prompt": "a red fox in fresh snow",
|
||||
"modalities": ["image"],
|
||||
"negative_prompt": "",
|
||||
}
|
||||
|
||||
|
||||
@pytest.mark.core_model
|
||||
@pytest.mark.cpu
|
||||
def test_lingbot_image_to_video_prompt_builder() -> None:
|
||||
image = Image.new("RGB", (320, 192), "red")
|
||||
result = build_image_to_video_prompt(
|
||||
"LingBotVideoPipeline",
|
||||
prompt="the fox looks toward the camera",
|
||||
negative_prompt=None,
|
||||
media_inputs={"image": image},
|
||||
height=192,
|
||||
width=320,
|
||||
num_frames=9,
|
||||
)
|
||||
assert result == {
|
||||
"prompt": "the fox looks toward the camera",
|
||||
"modalities": ["video"],
|
||||
"multi_modal_data": {"image": image},
|
||||
"num_frames": 9,
|
||||
}
|
||||
|
||||
|
||||
@pytest.mark.core_model
|
||||
@pytest.mark.cpu
|
||||
@pytest.mark.parametrize(
|
||||
"media_inputs",
|
||||
[
|
||||
{},
|
||||
{"image": "input.png"},
|
||||
{"image": [Image.new("RGB", (16, 16))]},
|
||||
{"image": Image.new("RGB", (16, 16)), "video": b"video"},
|
||||
],
|
||||
ids=["missing", "path", "multiple", "extra-modality"],
|
||||
)
|
||||
def test_lingbot_image_to_video_prompt_builder_rejects_invalid_media(
|
||||
media_inputs: dict[str, object],
|
||||
) -> None:
|
||||
with pytest.raises(ValueError, match="exactly one PIL image"):
|
||||
build_image_to_video_prompt(
|
||||
"LingBotVideoPipeline",
|
||||
prompt="move",
|
||||
negative_prompt=None,
|
||||
media_inputs=media_inputs,
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.core_model
|
||||
@pytest.mark.cpu
|
||||
@pytest.mark.parametrize("pipeline_name", ["Cosmos3OmniDiffusersPipeline", "Cosmos3OmniPipeline"])
|
||||
@@ -342,8 +283,11 @@ def test_ming_flash_omni_declared_extra_args_route_into_sampling_params() -> Non
|
||||
def test_ming_flash_omni_text_to_image_prompt_builder() -> None:
|
||||
assert build_text_to_image_prompt(
|
||||
"MingImagePipeline",
|
||||
prompt="Please draw a cute cat.",
|
||||
negative_prompt="ugly, blurry",
|
||||
{
|
||||
"prompt": "Please draw a cute cat.",
|
||||
"modalities": ["image"],
|
||||
"negative_prompt": "ugly, blurry",
|
||||
},
|
||||
height=512,
|
||||
width=768,
|
||||
) == {
|
||||
@@ -360,8 +304,7 @@ def test_ming_flash_omni_text_to_image_prompt_builder() -> None:
|
||||
# pipeline's 1024x1024 default still applies.
|
||||
assert build_text_to_image_prompt(
|
||||
"MingImagePipeline",
|
||||
prompt="Draw a poster.",
|
||||
negative_prompt=None,
|
||||
{"prompt": "Draw a poster.", "modalities": ["image"]},
|
||||
) == {
|
||||
"prompt": "Draw a poster.",
|
||||
"modalities": ["image"],
|
||||
@@ -391,28 +334,6 @@ def test_ming_flash_omni_image_to_image_prompt_builder() -> None:
|
||||
assert "negative_prompt" not in result
|
||||
|
||||
|
||||
@pytest.mark.core_model
|
||||
@pytest.mark.cpu
|
||||
@pytest.mark.parametrize("pipeline_name", ["Cosmos3OmniDiffusersPipeline", "Cosmos3OmniPipeline"])
|
||||
def test_cosmos3_text_to_image_prompt_builder_selects_image_modality(pipeline_name: str) -> None:
|
||||
assert build_text_to_image_prompt(
|
||||
pipeline_name,
|
||||
prompt="a red sports car at golden hour",
|
||||
negative_prompt="blurry, distorted",
|
||||
height=1024,
|
||||
width=1024,
|
||||
) == {
|
||||
"prompt": "a red sports car at golden hour",
|
||||
"modalities": ["image"],
|
||||
"negative_prompt": "blurry, distorted",
|
||||
}
|
||||
assert build_text_to_image_prompt(
|
||||
pipeline_name,
|
||||
prompt="a red sports car",
|
||||
negative_prompt=None,
|
||||
) == {"prompt": "a red sports car", "modalities": ["image"]}
|
||||
|
||||
|
||||
@pytest.mark.core_model
|
||||
@pytest.mark.cpu
|
||||
def test_audiox_extra_registry_declares_request_and_response_params() -> None:
|
||||
@@ -505,8 +426,11 @@ def test_unknown_pipeline_has_empty_extra_registry() -> None:
|
||||
def test_bagel_text_to_image_prompt_builder() -> None:
|
||||
assert build_text_to_image_prompt(
|
||||
"BagelPipeline",
|
||||
prompt="a cat",
|
||||
negative_prompt="blurry",
|
||||
{
|
||||
"prompt": "a cat",
|
||||
"modalities": ["image"],
|
||||
"negative_prompt": "blurry",
|
||||
},
|
||||
height=512,
|
||||
width=768,
|
||||
) == {
|
||||
@@ -543,14 +467,17 @@ def test_bagel_image_to_image_prompt_builder() -> None:
|
||||
|
||||
@pytest.mark.core_model
|
||||
@pytest.mark.cpu
|
||||
def test_unknown_pipeline_uses_default_text_to_image_prompt() -> None:
|
||||
assert build_text_to_image_prompt(
|
||||
"UnknownPipeline",
|
||||
prompt="a cat",
|
||||
negative_prompt=None,
|
||||
height=512,
|
||||
width=512,
|
||||
) == {"prompt": "a cat"}
|
||||
def test_unknown_pipeline_preserves_canonical_text_to_image_prompt() -> None:
|
||||
canonical_prompt = {"prompt": "a cat", "modalities": ["image"]}
|
||||
assert (
|
||||
build_text_to_image_prompt(
|
||||
"UnknownPipeline",
|
||||
canonical_prompt,
|
||||
height=512,
|
||||
width=512,
|
||||
)
|
||||
is canonical_prompt
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.core_model
|
||||
@@ -569,12 +496,14 @@ def test_unknown_pipeline_uses_default_image_to_image_prompt() -> None:
|
||||
}
|
||||
|
||||
|
||||
def _build_vace_prompt(media_inputs: dict[str, object], *, num_frames: int = 5) -> dict:
|
||||
def _build_vace_prompt(media_inputs: dict[str, Any], *, num_frames: int = 5) -> dict:
|
||||
return build_image_to_video_prompt(
|
||||
"WanVACEPipeline",
|
||||
prompt="a bird flying",
|
||||
negative_prompt=None,
|
||||
media_inputs=media_inputs,
|
||||
{
|
||||
"prompt": "a bird flying",
|
||||
"modalities": ["video"],
|
||||
"multi_modal_data": media_inputs,
|
||||
},
|
||||
height=16,
|
||||
width=320,
|
||||
num_frames=num_frames,
|
||||
@@ -597,7 +526,7 @@ def _build_vace_prompt(media_inputs: dict[str, object], *, num_frames: int = 5)
|
||||
],
|
||||
ids=["i2v", "v2lf", "flf2v", "inpaint", "r2v"],
|
||||
)
|
||||
def test_vace_image_to_video_prompt_builder(media_inputs: dict[str, object]) -> None:
|
||||
def test_vace_image_to_video_prompt_builder(media_inputs: dict[str, Any]) -> None:
|
||||
result = _build_vace_prompt(media_inputs)
|
||||
mmd = result["multi_modal_data"]
|
||||
if "reference_images" in mmd:
|
||||
@@ -616,7 +545,7 @@ def test_vace_image_to_video_prompt_builder(media_inputs: dict[str, object]) ->
|
||||
({"control_image": Image.new("RGB", (320, 16))}, "Unsupported VACE media input"),
|
||||
],
|
||||
)
|
||||
def test_vace_rejects_invalid_media_combinations(media_inputs: dict[str, object], message: str) -> None:
|
||||
def test_vace_rejects_invalid_media_combinations(media_inputs: dict[str, Any], message: str) -> None:
|
||||
with pytest.raises(ValueError, match=message):
|
||||
_build_vace_prompt(media_inputs)
|
||||
|
||||
@@ -664,8 +593,7 @@ def test_mammothmoda2_extra_registry_declares_request_and_response_params() -> N
|
||||
|
||||
wrapper_prompt = build_text_to_image_prompt(
|
||||
"MammothModa2ForConditionalGeneration",
|
||||
prompt="a cat",
|
||||
negative_prompt=None,
|
||||
{"prompt": "a cat", "modalities": ["image"]},
|
||||
height=256,
|
||||
width=256,
|
||||
)
|
||||
@@ -679,8 +607,11 @@ def test_mammothmoda2_text_to_image_prompt_builder() -> None:
|
||||
# prompt is ignored (MammothModa2 t2i uses CFG, not an explicit negative path).
|
||||
assert build_text_to_image_prompt(
|
||||
"MammothModa2DiTPipeline",
|
||||
prompt="a cat",
|
||||
negative_prompt="blurry",
|
||||
{
|
||||
"prompt": "a cat",
|
||||
"modalities": ["image"],
|
||||
"negative_prompt": "blurry",
|
||||
},
|
||||
height=512,
|
||||
width=768,
|
||||
) == {
|
||||
|
||||
@@ -3,13 +3,88 @@ from types import SimpleNamespace
|
||||
|
||||
import pytest
|
||||
import torch
|
||||
from vllm.v1.cudagraph_dispatcher import CUDAGraphMode
|
||||
|
||||
from vllm_omni.worker.gpu_model_runner import OmniGPUModelRunner, _filter_mrope_kwargs_for_model
|
||||
from vllm_omni.worker.gpu_model_runner import (
|
||||
OmniGPUModelRunner,
|
||||
_filter_mrope_kwargs_for_model,
|
||||
)
|
||||
from vllm_omni.worker.omni_connector_model_runner_mixin import OmniConnectorModelRunnerMixin
|
||||
|
||||
pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
|
||||
|
||||
|
||||
def _runner_for_talker_graph_init(
|
||||
*,
|
||||
talker_mtp_graph_safe: bool | None,
|
||||
has_separate_talker: bool = True,
|
||||
model_stage: str = "talker",
|
||||
) -> OmniGPUModelRunner:
|
||||
runner = object.__new__(OmniGPUModelRunner)
|
||||
runner.model = SimpleNamespace(
|
||||
talker=object() if has_separate_talker else None,
|
||||
talker_mtp=object(),
|
||||
model_stage=model_stage,
|
||||
)
|
||||
if talker_mtp_graph_safe is not None:
|
||||
runner.model.talker_mtp_graph_safe = talker_mtp_graph_safe
|
||||
runner.model_config = SimpleNamespace(hf_text_config=SimpleNamespace(hidden_size=4))
|
||||
runner.compilation_config = SimpleNamespace(
|
||||
cudagraph_mode=CUDAGraphMode.FULL,
|
||||
max_cudagraph_capture_size=1,
|
||||
)
|
||||
runner.vllm_config = object()
|
||||
runner.max_num_reqs = 1
|
||||
runner.dtype = torch.float32
|
||||
runner._make_buffer = lambda *args, **kwargs: SimpleNamespace(args=args, kwargs=kwargs)
|
||||
return runner
|
||||
|
||||
|
||||
def test_talker_mtp_skips_graph_when_model_declares_unsafe(monkeypatch):
|
||||
runner = _runner_for_talker_graph_init(talker_mtp_graph_safe=False)
|
||||
talker_mtp = runner.model.talker_mtp
|
||||
monkeypatch.setattr(
|
||||
"vllm_omni.worker.gpu_model_runner.current_omni_platform.get_graph_wrapper_cls",
|
||||
lambda: pytest.fail("graph wrapper must not be selected"),
|
||||
)
|
||||
|
||||
OmniGPUModelRunner._init_talker_mtp(runner)
|
||||
|
||||
assert runner.talker_mtp is talker_mtp
|
||||
|
||||
|
||||
@pytest.mark.parametrize("model_stage", ["thinker", "code2wav"])
|
||||
def test_non_talker_stage_does_not_use_talker_mtp_graph(monkeypatch, model_stage: str):
|
||||
runner = _runner_for_talker_graph_init(
|
||||
talker_mtp_graph_safe=None,
|
||||
has_separate_talker=False,
|
||||
model_stage=model_stage,
|
||||
)
|
||||
talker_mtp = runner.model.talker_mtp
|
||||
monkeypatch.setattr(
|
||||
"vllm_omni.worker.gpu_model_runner.current_omni_platform.get_graph_wrapper_cls",
|
||||
lambda: pytest.fail("graph wrapper must not be selected"),
|
||||
)
|
||||
|
||||
OmniGPUModelRunner._init_talker_mtp(runner)
|
||||
|
||||
assert runner.talker_mtp is talker_mtp
|
||||
|
||||
|
||||
@pytest.mark.parametrize("talker_mtp_graph_safe", [None, True])
|
||||
def test_talker_mtp_uses_graph_for_legacy_or_explicit_safe_model(monkeypatch, talker_mtp_graph_safe):
|
||||
runner = _runner_for_talker_graph_init(talker_mtp_graph_safe=talker_mtp_graph_safe)
|
||||
wrapped = object()
|
||||
monkeypatch.setattr(
|
||||
"vllm_omni.worker.gpu_model_runner.current_omni_platform.get_graph_wrapper_cls",
|
||||
lambda: lambda *args, **kwargs: wrapped,
|
||||
)
|
||||
|
||||
OmniGPUModelRunner._init_talker_mtp(runner)
|
||||
|
||||
assert runner.talker_mtp is wrapped
|
||||
|
||||
|
||||
class DummyBuffer:
|
||||
"""A minimal buffer wrapper that exposes the `.gpu` attribute."""
|
||||
|
||||
|
||||
@@ -69,6 +69,18 @@ stages:
|
||||
repetition_penalty: 1.1
|
||||
|
||||
platforms:
|
||||
# The supported Qwen3-Omni ModelOpt checkpoint quantizes the thinker only. Clear the
|
||||
# checkpoint-level metadata for the BF16 audio stages on the verified MUSA
|
||||
# profile before vLLM performs its normal quantization auto-detection.
|
||||
musa:
|
||||
stages:
|
||||
- stage_id: 1
|
||||
hf_overrides:
|
||||
quantization_config: null
|
||||
- stage_id: 2
|
||||
hf_overrides:
|
||||
quantization_config: null
|
||||
|
||||
npu:
|
||||
stages:
|
||||
- stage_id: 0
|
||||
|
||||
@@ -152,6 +152,9 @@ class Qwen3OmniMoeForConditionalGeneration(
|
||||
dtype=torch.long,
|
||||
)
|
||||
elif self.model_stage == "talker":
|
||||
# The outer wrapper exposes talker_mtp for every stage, but only
|
||||
# the talker stage owns the module that the method invokes.
|
||||
self.talker_mtp_graph_safe = current_omni_platform.supports_talker_mtp_graph_capture()
|
||||
multimodal_config.skip_mm_profiling = True
|
||||
self.has_preprocess = True
|
||||
self.has_postprocess = True
|
||||
|
||||
@@ -3,32 +3,6 @@
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
|
||||
def build_text_to_image_prompt(
|
||||
prompt: str,
|
||||
negative_prompt: str | None,
|
||||
height: int | None = None,
|
||||
width: int | None = None,
|
||||
) -> dict[str, Any]:
|
||||
"""Build a Cosmos3 T2I prompt by selecting image output modality.
|
||||
|
||||
Cosmos3 uses the same pipeline class for T2I and video modes. The OpenAI
|
||||
image endpoint selects T2I by adding ``modalities=["image"]``. ``height``
|
||||
and ``width`` are accepted for registry compatibility; sizing travels
|
||||
through sampling params instead of the prompt payload.
|
||||
"""
|
||||
del height, width
|
||||
text_prompt: dict[str, Any] = {
|
||||
"prompt": prompt,
|
||||
"modalities": ["image"],
|
||||
}
|
||||
if negative_prompt is not None:
|
||||
text_prompt["negative_prompt"] = negative_prompt
|
||||
return text_prompt
|
||||
|
||||
|
||||
COSMOS3_EXTRA_BODY_PARAMS = frozenset(
|
||||
{
|
||||
"flow_shift",
|
||||
|
||||
@@ -3,11 +3,6 @@
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Mapping
|
||||
from typing import Any
|
||||
|
||||
from PIL import Image
|
||||
|
||||
LINGBOT_VIDEO_EXTRA_BODY_PARAMS = frozenset(
|
||||
{
|
||||
"batch_cfg",
|
||||
@@ -24,43 +19,3 @@ LINGBOT_VIDEO_EXTRA_BODY_PARAMS = frozenset(
|
||||
"t_thresh",
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
def build_text_to_image_prompt(
|
||||
prompt: str,
|
||||
negative_prompt: str | None,
|
||||
height: int | None = None,
|
||||
width: int | None = None,
|
||||
) -> dict[str, Any]:
|
||||
del height, width
|
||||
result: dict[str, Any] = {
|
||||
"prompt": prompt,
|
||||
"modalities": ["image"],
|
||||
}
|
||||
if negative_prompt is not None:
|
||||
result["negative_prompt"] = negative_prompt
|
||||
return result
|
||||
|
||||
|
||||
def build_image_to_video_prompt(
|
||||
prompt: str,
|
||||
negative_prompt: str | None,
|
||||
media_inputs: Mapping[str, Any],
|
||||
height: int | None = None,
|
||||
width: int | None = None,
|
||||
num_frames: int | None = None,
|
||||
) -> dict[str, Any]:
|
||||
del height, width
|
||||
if set(media_inputs) != {"image"} or not isinstance(media_inputs["image"], Image.Image):
|
||||
raise ValueError("LingBot text-image-to-video requires exactly one PIL image input.")
|
||||
|
||||
result: dict[str, Any] = {
|
||||
"prompt": prompt,
|
||||
"modalities": ["video"],
|
||||
"multi_modal_data": {"image": media_inputs["image"]},
|
||||
}
|
||||
if num_frames is not None:
|
||||
result["num_frames"] = num_frames
|
||||
if negative_prompt is not None:
|
||||
result["negative_prompt"] = negative_prompt
|
||||
return result
|
||||
@@ -28,21 +28,12 @@ from vllm_omni.model_extras.cosmos3 import (
|
||||
COSMOS3_EXTRA_BODY_PARAMS,
|
||||
COSMOS3_EXTRA_OUTPUT_PARAMS,
|
||||
)
|
||||
from vllm_omni.model_extras.cosmos3 import (
|
||||
build_text_to_image_prompt as build_cosmos3_text_to_image_prompt,
|
||||
)
|
||||
from vllm_omni.model_extras.helios import (
|
||||
HELIOS_EXTRA_BODY_PARAMS,
|
||||
HELIOS_EXTRA_OUTPUT_PARAMS,
|
||||
)
|
||||
from vllm_omni.model_extras.hunyuan_image3 import build_x_to_text_prompt as build_hunyuan_x_to_text_prompt
|
||||
from vllm_omni.model_extras.lingbot_video import LINGBOT_VIDEO_EXTRA_BODY_PARAMS
|
||||
from vllm_omni.model_extras.lingbot_video import (
|
||||
build_image_to_video_prompt as build_lingbot_image_to_video_prompt,
|
||||
)
|
||||
from vllm_omni.model_extras.lingbot_video import (
|
||||
build_text_to_image_prompt as build_lingbot_text_to_image_prompt,
|
||||
)
|
||||
from vllm_omni.model_extras.ltx2 import LTX_EXTRA_BODY_PARAMS, LTX_EXTRA_OUTPUT_PARAMS
|
||||
from vllm_omni.model_extras.magi_human import (
|
||||
MAGI_HUMAN_EXTRA_BODY_PARAMS,
|
||||
@@ -147,18 +138,6 @@ def build_x_to_text_prompt(
|
||||
return builder(model, prompt, has_image)
|
||||
|
||||
|
||||
def default_text_to_image_prompt(
|
||||
prompt: str,
|
||||
negative_prompt: str | None,
|
||||
height: int | None = None,
|
||||
width: int | None = None,
|
||||
) -> dict[str, Any]:
|
||||
result: dict[str, Any] = {"prompt": prompt}
|
||||
if negative_prompt is not None:
|
||||
result["negative_prompt"] = negative_prompt
|
||||
return result
|
||||
|
||||
|
||||
def default_image_to_image_prompt(
|
||||
prompt: str,
|
||||
negative_prompt: str | None,
|
||||
@@ -175,20 +154,6 @@ def default_image_to_image_prompt(
|
||||
return result
|
||||
|
||||
|
||||
def default_image_to_video_prompt(
|
||||
prompt: str,
|
||||
negative_prompt: str | None,
|
||||
media_inputs: Mapping[str, Any],
|
||||
height: int | None = None,
|
||||
width: int | None = None,
|
||||
num_frames: int | None = None,
|
||||
) -> dict[str, Any]:
|
||||
del height, width, num_frames
|
||||
if set(media_inputs) != {"image"} or not isinstance(media_inputs["image"], Image.Image):
|
||||
raise ValueError("This model only supports a single --image input in the shared image-to-video example.")
|
||||
return default_image_to_image_prompt(prompt, negative_prompt, media_inputs["image"])
|
||||
|
||||
|
||||
_EXTRA_SPECS: dict[str, dict[str, Any]] = {
|
||||
"AudioXPipeline": {
|
||||
"extra_body_params": AUDIOX_EXTRA_BODY_PARAMS,
|
||||
@@ -208,12 +173,11 @@ _EXTRA_SPECS: dict[str, dict[str, Any]] = {
|
||||
"Cosmos3OmniDiffusersPipeline": {
|
||||
"extra_body_params": COSMOS3_EXTRA_BODY_PARAMS,
|
||||
"extra_output_params": COSMOS3_EXTRA_OUTPUT_PARAMS,
|
||||
"text_to_image_prompt_builder": build_cosmos3_text_to_image_prompt,
|
||||
# The shared T2I example already supplies modalities=["image"].
|
||||
},
|
||||
"Cosmos3OmniPipeline": {
|
||||
"extra_body_params": COSMOS3_EXTRA_BODY_PARAMS,
|
||||
"extra_output_params": COSMOS3_EXTRA_OUTPUT_PARAMS,
|
||||
"text_to_image_prompt_builder": build_cosmos3_text_to_image_prompt,
|
||||
},
|
||||
"MagiHumanPipeline": {
|
||||
"extra_body_params": MAGI_HUMAN_EXTRA_BODY_PARAMS,
|
||||
@@ -229,8 +193,8 @@ _EXTRA_SPECS: dict[str, dict[str, Any]] = {
|
||||
},
|
||||
"LingBotVideoPipeline": {
|
||||
"extra_body_params": LINGBOT_VIDEO_EXTRA_BODY_PARAMS,
|
||||
"text_to_image_prompt_builder": build_lingbot_text_to_image_prompt,
|
||||
"image_to_video_prompt_builder": build_lingbot_image_to_video_prompt,
|
||||
# Shared T2I/I2V envelopes select the output modality. LingBot's
|
||||
# pipeline owns model-specific validation and normalization.
|
||||
},
|
||||
**{
|
||||
model_class_name: {
|
||||
@@ -263,7 +227,7 @@ _EXTRA_SPECS: dict[str, dict[str, Any]] = {
|
||||
}
|
||||
|
||||
# Multi-stage discovery reports the top-level wrapper rather than its DiT
|
||||
# submodule, so both names must resolve to the same request adapters.
|
||||
# submodule, so both names must resolve to the same request builders.
|
||||
_EXTRA_SPECS["MammothModa2ForConditionalGeneration"] = _EXTRA_SPECS["MammothModa2DiTPipeline"]
|
||||
_EXTRA_SPECS["Mammothmoda2Model"] = _EXTRA_SPECS["MammothModa2DiTPipeline"]
|
||||
|
||||
@@ -305,18 +269,21 @@ def should_init_extra_args_for_non_diffusion_stages(model_class_name: str | None
|
||||
|
||||
def build_text_to_image_prompt(
|
||||
model_class_name: str | None,
|
||||
prompt: str,
|
||||
negative_prompt: str | None,
|
||||
prompt: dict[str, Any],
|
||||
height: int | None = None,
|
||||
width: int | None = None,
|
||||
) -> dict[str, Any]:
|
||||
"""Build a model-specific T2I prompt from an example-owned envelope."""
|
||||
spec = _get_spec(model_class_name)
|
||||
builder: TextToImagePromptBuilder = (
|
||||
spec.get("text_to_image_prompt_builder", default_text_to_image_prompt)
|
||||
if spec is not None
|
||||
else default_text_to_image_prompt
|
||||
builder: TextToImagePromptBuilder | None = spec.get("text_to_image_prompt_builder") if spec else None
|
||||
if builder is None:
|
||||
return prompt
|
||||
return builder(
|
||||
prompt=str(prompt["prompt"]),
|
||||
negative_prompt=prompt.get("negative_prompt"),
|
||||
height=height,
|
||||
width=width,
|
||||
)
|
||||
return builder(prompt, negative_prompt, height, width)
|
||||
|
||||
|
||||
def build_image_to_image_prompt(
|
||||
@@ -338,17 +305,24 @@ def build_image_to_image_prompt(
|
||||
|
||||
def build_image_to_video_prompt(
|
||||
model_class_name: str | None,
|
||||
prompt: str,
|
||||
negative_prompt: str | None,
|
||||
media_inputs: Mapping[str, Any],
|
||||
prompt: dict[str, Any],
|
||||
height: int | None = None,
|
||||
width: int | None = None,
|
||||
num_frames: int | None = None,
|
||||
) -> dict[str, Any]:
|
||||
"""Build a model-specific I2V prompt from an example-owned envelope."""
|
||||
spec = _get_spec(model_class_name)
|
||||
builder: ImageToVideoPromptBuilder = (
|
||||
spec.get("image_to_video_prompt_builder", default_image_to_video_prompt)
|
||||
if spec is not None
|
||||
else default_image_to_video_prompt
|
||||
builder: ImageToVideoPromptBuilder | None = spec.get("image_to_video_prompt_builder") if spec else None
|
||||
if builder is None:
|
||||
return prompt
|
||||
media_inputs = prompt.get("multi_modal_data") or {}
|
||||
if not isinstance(media_inputs, Mapping):
|
||||
raise TypeError("Canonical I2V prompt multi_modal_data must be a mapping.")
|
||||
return builder(
|
||||
prompt=str(prompt["prompt"]),
|
||||
negative_prompt=prompt.get("negative_prompt"),
|
||||
media_inputs=media_inputs,
|
||||
height=height,
|
||||
width=width,
|
||||
num_frames=num_frames,
|
||||
)
|
||||
return builder(prompt, negative_prompt, media_inputs, height, width, num_frames)
|
||||
@@ -155,6 +155,11 @@ class OmniPlatform(Platform):
|
||||
"""Check if the platform supports torch.compile with inductor backend."""
|
||||
raise NotImplementedError
|
||||
|
||||
@classmethod
|
||||
def supports_talker_mtp_graph_capture(cls) -> bool:
|
||||
"""Whether a model may capture its dedicated talker MTP graph."""
|
||||
return True
|
||||
|
||||
@classmethod
|
||||
def has_flash_attn_package(cls) -> bool:
|
||||
"""Check if a Flash Attention package is available and usable on this platform."""
|
||||
|
||||
@@ -115,6 +115,11 @@ class MUSAOmniPlatform(OmniPlatform, MUSAPlatformBase):
|
||||
"""MUSA supports torch.compile with inductor backend."""
|
||||
return True
|
||||
|
||||
@classmethod
|
||||
def supports_talker_mtp_graph_capture(cls) -> bool:
|
||||
"""MUSA keeps Qwen3 talker MTP outside its dedicated FULL graph."""
|
||||
return False
|
||||
|
||||
@classmethod
|
||||
def supports_float64(cls) -> bool:
|
||||
"""MUSA does not support float64 yet."""
|
||||
|
||||
@@ -206,8 +206,12 @@ class OmniGPUModelRunner(GPUModelRunner):
|
||||
cudagraph_mode = self.compilation_config.cudagraph_mode
|
||||
assert cudagraph_mode is not None
|
||||
has_separate_talker = getattr(self.model, "talker", None) is not None
|
||||
talker_mtp_graph_safe = getattr(self.model, "talker_mtp_graph_safe", False)
|
||||
if cudagraph_mode.has_full_cudagraphs() and (has_separate_talker or talker_mtp_graph_safe):
|
||||
talker_mtp_graph_safe = getattr(self.model, "talker_mtp_graph_safe", None)
|
||||
# Preserve the legacy separate-talker default only when the model has
|
||||
# not declared graph safety. An explicit False must disable wrapping.
|
||||
if talker_mtp_graph_safe is None:
|
||||
talker_mtp_graph_safe = has_separate_talker
|
||||
if cudagraph_mode.has_full_cudagraphs() and talker_mtp_graph_safe:
|
||||
graph_wrapper_cls = current_omni_platform.get_graph_wrapper_cls()
|
||||
self.talker_mtp = graph_wrapper_cls(talker_mtp, self.vllm_config, runtime_mode=CUDAGraphMode.FULL)
|
||||
# TTS exposes mtp_hidden_size; Omni uses hf_text_config.hidden_size.
|
||||
|
||||
Reference in new issue
Block a user