Merge main and preserve event-driven defaults in shared engine bases
Signed-off-by: Sy03 <1370724210@qq.com>
This commit is contained in:
commit
d2bb944863
371 files changed
+43235
-38379
No files matched your search
@@ -131,13 +131,13 @@ for the validated batching, attention, and quantization combinations.
|
||||
|
||||
### MiniMax-H3: conditioning, denoising, and decode
|
||||
|
||||
MiniMax-H3 illustrates logical component and phase boundaries between text
|
||||
encoding, task-specific denoising, and media decoding:
|
||||
`Text Encoder → DiT → VAE Decoder`. The current registered topology co-locates
|
||||
these components in one diffusion stage. These boundaries are useful for
|
||||
profiling and offload; independent placement would require a future
|
||||
multi-stage `PipelineConfig`. The stage mapping follows the implementation and
|
||||
serving recipe; the [release overview](https://minimaxi.com/blog/minimax-h3)
|
||||
MiniMax-H3 combines text and reference-media encoding, task-specific denoising,
|
||||
and media decoding. Standard serving co-locates these components in one diffusion
|
||||
stage. The opt-in
|
||||
[disaggregated topology](https://github.com/vllm-project/vllm-omni/blob/main/recipes/MiniMaxAI/MiniMax-H3-Disaggregated.md)
|
||||
runs Qwen and the video/audio VAE encoders in Stage 0, then sends their conditioning
|
||||
to a Stage 1 containing the DiTs and VAE decoders. The stage mapping follows the
|
||||
implementation and serving recipe; the [release overview](https://minimaxi.com/blog/minimax-h3)
|
||||
notes that a detailed technical report is forthcoming.
|
||||
|
||||
#### Model architecture by component
|
||||
@@ -149,16 +149,15 @@ notes that a detailed technical report is forthcoming.
|
||||
| **VAE Decoder** | Video VAE and audio VAE decode the generated latents and assemble the synchronized output | Decoded video frames and stereo waveform, with FPS and audio sample rate metadata |
|
||||
|
||||
Both DiT partitions remain behind one task-selection point; `task` chooses
|
||||
which partition runs. In the current implementation, the three logical
|
||||
components execute inside one diffusion stage. This is different from
|
||||
Qwen3-Omni's three independently scheduled stages, even though both models
|
||||
produce synchronized audio and video-like outputs.
|
||||
which partition runs.
|
||||
|
||||
#### Representative component execution
|
||||
|
||||
The following table describes the standard single-stage deployment.
|
||||
|
||||
| Logical component | Batching | Attention and execution | Parallelism | Quantization |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| **Text Encoder** | The current co-located pipeline follows the diffusion request scheduler; independent prompt/reference batching would require a separate multi-stage topology | Qwen3-VL attention and multimodal preprocessing | `--text-encoder-tp-size N` shards the encoder across the first `N` DiT ranks; independent placement is not part of the current topology | BF16/FP32 baseline; the H3 DiT FP8 path does not quantize the text encoder |
|
||||
| **Text Encoder** | Follows the diffusion request scheduler | Qwen3-VL attention and multimodal preprocessing | `--text-encoder-tp-size N` shards the encoder across the first `N` DiT ranks | BF16/FP32 baseline; the H3 DiT FP8 path does not quantize the text encoder |
|
||||
| **DiT** | The current H3 implementation executes one generation request per diffusion batch; use concurrency for service-level throughput | cuDNN attention for the validated two-GPU consumer profile; TRTLLM attention or FlashAttention-4 on supported Blackwell profiles | TP2 plus text-encoder TP and Ulysses/VAE parallel groups on multi-GPU profiles; DLO/CPU offload trade memory for transfer time | Online FP8 applies to eligible DiT linears and is incompatible with layerwise offload |
|
||||
| **VAE Decoder** | Decode follows each generation request; tile/patch work can be distributed even when denoising is not batched | VAE decode kernels rather than DiT attention | VAE patch parallelism and native tiled decode within the diffusion stage | BF16/FP32 baseline; the documented H3 FP8 path leaves both VAEs unchanged |
|
||||
|
||||
|
||||
@@ -76,6 +76,34 @@ preserve common `self` or `cross` policies. A model also decides whether its
|
||||
path is compatible with automatic TRTLLM selection, because only the model
|
||||
knows whether masking and packing satisfy the kernel contract.
|
||||
|
||||
### Model-owned implementation specializations
|
||||
|
||||
Use `Attention(..., impl_overrides={backend_name: implementation_class})`
|
||||
when a model needs different computation within an existing backend contract.
|
||||
The model defines the implementation in its own directory and passes it when
|
||||
constructing the layer:
|
||||
|
||||
```python
|
||||
# MyModelVSAImpl is a model-owned subclass of FastVideoVSAImpl.
|
||||
self.attention = Attention(
|
||||
num_heads=num_heads,
|
||||
head_size=head_size,
|
||||
softmax_scale=head_size**-0.5,
|
||||
causal=False,
|
||||
role="self",
|
||||
impl_overrides={"FASTVIDEO_VSA": MyModelVSAImpl},
|
||||
)
|
||||
```
|
||||
|
||||
The override is applied after role/platform selection, keyed by the selected
|
||||
backend's `get_name()`. It must subclass the **selected implementation**;
|
||||
incompatible platform implementations raise `TypeError`. Unselected overrides
|
||||
have no effect. Backend capabilities, constructor options, and shared
|
||||
parallel dispatch remain unchanged and must be honored by the specialization.
|
||||
|
||||
Use a new backend for a new provider/capability contract. `custom_attention`
|
||||
instead owns communication and requires `skip_sequence_parallel=True`.
|
||||
|
||||
## Adding or changing a backend
|
||||
|
||||
An implementation change is complete only when it:
|
||||
@@ -91,8 +119,9 @@ An implementation change is complete only when it:
|
||||
|
||||
## FastVideo VSA model metadata
|
||||
|
||||
FastVideo VSA support is scoped to `FastVideo/FastWan2.2-TI2V-5B-Diffusers`, whose text-to-video and image-to-video modes both use `Wan22Pipeline`. It operates on a flattened DiT sequence but partitions tokens in
|
||||
the original latent video grid. Wan integrations therefore attach the
|
||||
FastWan's text-to-video and image-to-video modes both use `Wan22Pipeline`.
|
||||
It operates on a flattened DiT sequence but partitions tokens in the original
|
||||
latent video grid. Wan integrations therefore attach the
|
||||
post-patch `(T, H, W)` shape as `vsa_dit_seq_shape` attention metadata. The
|
||||
backend validates that `T * H * W` equals the sequence length, derives the
|
||||
runtime block count, and selects the configured top-k key/value blocks for
|
||||
@@ -104,3 +133,8 @@ the checkpoint contains those weights. Checkpoints without the projection do
|
||||
not allocate or execute it. When top-k selects every block, native checkpoints
|
||||
route to SDPA, while FastVideo DMD checkpoints preserve the VSA all-block path
|
||||
to retain checkpoint semantics.
|
||||
|
||||
MiniMax-H3 supplies `MiniMaxH3VSAImpl` through `impl_overrides`.
|
||||
`models/minimax_h3/attention/fastvideo_h3.py` owns its prefix layout, video tiling, and
|
||||
learned compression gate. Pooling, block-map construction, and tile64 provider
|
||||
calls are shared through `attention/ops/block_sparse.py`.
|
||||
+359
-1056
File diff suppressed because it is too large.
Load diff
@@ -16,6 +16,7 @@ required_reviewers:
|
||||
- "@yenuo26"
|
||||
- "@NickCao"
|
||||
primary_code_paths:
|
||||
- vllm_omni/engine/omni_engine_base.py
|
||||
- vllm_omni/engine/async_omni_engine.py
|
||||
- vllm_omni/engine/async_engine_utils.py
|
||||
- vllm_omni/engine/orchestrator.py
|
||||
@@ -28,8 +29,12 @@ related_code_paths:
|
||||
- vllm_omni/engine/stage_pool.py
|
||||
- vllm_omni/outputs/output_processor.py
|
||||
- vllm_omni/distributed/omni_coordinator/**
|
||||
# Duplex kernel design is owned by ../fullduplex.md; listed here because the
|
||||
# orchestrator hosts the duplex control plane.
|
||||
# The duplex siblings (DuplexOmniEngine / DuplexOrchestrator) and the
|
||||
# engine-resident session framework are owned by ../fullduplex.md; listed
|
||||
# here because they fill the template seams of OmniEngineBase and
|
||||
# OrchestratorBase.
|
||||
- vllm_omni/engine/duplex_omni_engine.py
|
||||
- vllm_omni/engine/duplex_orchestrator.py
|
||||
- vllm_omni/engine/duplex/**
|
||||
depends_on:
|
||||
- input_output_modality_contracts.md
|
||||
@@ -48,6 +53,8 @@ validation_paths:
|
||||
- tests/engine/test_cfg_companion_lifecycle.py
|
||||
- tests/engine/test_rpc_result_router.py
|
||||
- tests/engine/duplex/**
|
||||
- tests/engine/test_duplex_orchestrator.py
|
||||
- tests/engine/test_duplex_omni_engine.py
|
||||
- tests/engine/test_duplex_import_boundary.py
|
||||
upstream_refs:
|
||||
- vllm.v1.engine.EngineCoreRequest
|
||||
|
||||
@@ -19,7 +19,9 @@ required_reviewers:
|
||||
primary_code_paths:
|
||||
- vllm_omni/entrypoints/omni.py
|
||||
- vllm_omni/entrypoints/async_omni.py
|
||||
- vllm_omni/entrypoints/async_omni_base.py
|
||||
- vllm_omni/entrypoints/omni_base.py
|
||||
- vllm_omni/entrypoints/duplex_omni.py
|
||||
- vllm_omni/entrypoints/cli/**
|
||||
- vllm_omni/entrypoints/openai/**
|
||||
- vllm_omni/entrypoints/openpi/**
|
||||
@@ -27,14 +29,13 @@ primary_code_paths:
|
||||
- vllm_omni/entrypoints/stage_utils.py
|
||||
- vllm_omni/entrypoints/utils.py
|
||||
- vllm_omni/entrypoints/duplex/**
|
||||
- vllm_omni/entrypoints/duplex_request_client.py
|
||||
- vllm_omni/clients/**
|
||||
primary_path_exceptions:
|
||||
- path: vllm_omni/entrypoints/openai/errors.py
|
||||
owner: error_contracts.md
|
||||
- path: vllm_omni/entrypoints/duplex/**
|
||||
owner: ../fullduplex.md
|
||||
- path: vllm_omni/entrypoints/duplex_request_client.py
|
||||
- path: vllm_omni/entrypoints/duplex_omni.py
|
||||
owner: ../fullduplex.md
|
||||
- path: vllm_omni/clients/**
|
||||
owner: ../fullduplex.md
|
||||
@@ -42,7 +43,9 @@ related_code_paths:
|
||||
- vllm_omni/errors.py
|
||||
- vllm_omni/inputs/**
|
||||
- vllm_omni/outputs/**
|
||||
- vllm_omni/engine/omni_engine_base.py
|
||||
- vllm_omni/engine/async_omni_engine.py
|
||||
- vllm_omni/engine/duplex_omni_engine.py
|
||||
- vllm_omni/config/**
|
||||
- vllm_omni/deploy/**
|
||||
depends_on:
|
||||
@@ -53,7 +56,9 @@ validation_paths:
|
||||
- tests/entrypoints/test_omni_entrypoints.py
|
||||
- tests/entrypoints/test_async_omni.py
|
||||
- tests/entrypoints/test_async_omni_pause_sleep_routing.py
|
||||
- tests/entrypoints/test_async_omni_duplex.py
|
||||
- tests/entrypoints/test_duplex_omni.py
|
||||
- tests/entrypoints/duplex/**
|
||||
- tests/entrypoints/openai_api/test_duplex_api_server.py
|
||||
- tests/entrypoints/test_serve.py
|
||||
- tests/entrypoints/test_stream_finish_reason.py
|
||||
- tests/entrypoints/openai/**
|
||||
|
||||
Reference in new issue
Block a user