Compare commits
7
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
a5ee63fb1f | ||
|
|
3335e76c97 | ||
|
|
9167599b3c | ||
|
|
99f946210b | ||
|
|
ec4e12f379 | ||
|
|
d272d57bef | ||
|
|
bc95e7415f |
No files matched your search
@@ -309,7 +309,7 @@ For Omni or custom models, create:
|
|||||||
|
|
||||||
Required updates:
|
Required updates:
|
||||||
1. `docs/user_guide/diffusion/parallelism/overview.md` — parallelism support overview/table
|
1. `docs/user_guide/diffusion/parallelism/overview.md` — parallelism support overview/table
|
||||||
2. `docs/user_guide/diffusion/cpu_offload_diffusion.md` — if CPU offload supported (add to supported models table)
|
2. `docs/user_guide/diffusion/cpu_offload.md` — if CPU offload supported (add to supported models table)
|
||||||
3. `docs/user_guide/diffusion/cache_acceleration/teacache.md` — if TeaCache supported
|
3. `docs/user_guide/diffusion/cache_acceleration/teacache.md` — if TeaCache supported
|
||||||
4. `docs/user_guide/diffusion/cache_acceleration/cache_dit.md` — if Cache-DiT supported
|
4. `docs/user_guide/diffusion/cache_acceleration/cache_dit.md` — if Cache-DiT supported
|
||||||
5. Offline example docs under `examples/offline_inference/<name>/` (`README.md` or category-specific `.md`)
|
5. Offline example docs under `examples/offline_inference/<name>/` (`README.md` or category-specific `.md`)
|
||||||
|
|||||||
+43
-25
@@ -68,12 +68,12 @@ nav:
|
|||||||
- Sequence Parallel: user_guide/diffusion/parallelism/sequence_parallel.md
|
- Sequence Parallel: user_guide/diffusion/parallelism/sequence_parallel.md
|
||||||
- Tensor Parallel: user_guide/diffusion/parallelism/tensor_parallel.md
|
- Tensor Parallel: user_guide/diffusion/parallelism/tensor_parallel.md
|
||||||
- VAE Parallelism: user_guide/diffusion/parallelism/vae_parallelism.md
|
- VAE Parallelism: user_guide/diffusion/parallelism/vae_parallelism.md
|
||||||
- CPU Offloading: user_guide/diffusion/cpu_offload_diffusion.md
|
- CPU Offloading: user_guide/diffusion/cpu_offload.md
|
||||||
- LoRA: user_guide/diffusion/lora.md
|
- LoRA: user_guide/diffusion/lora.md
|
||||||
- Custom Pipeline: features/custom_pipeline.md
|
- Custom Pipeline: features/custom_pipeline.md
|
||||||
- Execution Modes: user_guide/diffusion/execution_modes.md
|
- Execution Modes: user_guide/diffusion/execution_modes.md
|
||||||
- Quantization:
|
- Quantization:
|
||||||
- Overview: user_guide/quantization/overview.md
|
- Quantization: user_guide/quantization/overview.md
|
||||||
- Online Quantization: user_guide/quantization/online.md
|
- Online Quantization: user_guide/quantization/online.md
|
||||||
- Quantized KV Cache: user_guide/quantization/quantized_kvcache.md
|
- Quantized KV Cache: user_guide/quantization/quantized_kvcache.md
|
||||||
- FP8 W8A8: user_guide/quantization/fp8.md
|
- FP8 W8A8: user_guide/quantization/fp8.md
|
||||||
@@ -101,31 +101,49 @@ nav:
|
|||||||
- CI: contributing/ci
|
- CI: contributing/ci
|
||||||
- Design Documents:
|
- Design Documents:
|
||||||
- design/index.md
|
- design/index.md
|
||||||
- design/architecture_overview.md
|
- Architecture Documents:
|
||||||
|
- Architecture Overview: design/architecture_overview.md
|
||||||
- Feature Design:
|
- Feature Design:
|
||||||
- design/feature/disaggregated_inference.md
|
- Runtime and Stage Execution:
|
||||||
- design/feature/ray_based_execution.md
|
- Disaggregated Inference: design/feature/disaggregated_inference.md
|
||||||
- design/feature/omni_connectors/
|
- Async Chunk: design/feature/async_chunk.md
|
||||||
- design/feature/prefix_caching.md
|
- Async Diffusion Output: design/feature/async_diffusion_output.md
|
||||||
- design/feature/cfg_parallel.md
|
- Async Omni Output Materialization: design/feature/omni_async_output_materialization.md
|
||||||
- design/feature/expert_parallel.md
|
- Automatic Prefix Caching: design/feature/prefix_caching.md
|
||||||
- design/feature/sequence_parallel.md
|
- Communication:
|
||||||
- design/feature/tensor_parallel.md
|
- OmniConnector Implementations:
|
||||||
- design/feature/vae_parallel.md
|
- Mooncake Store Connector: design/feature/omni_connectors/mooncake_store_connector.md
|
||||||
- design/feature/hsdp.md
|
- Mooncake Transfer Engine Connector: design/feature/omni_connectors/mooncake_transfer_engine_connector.md
|
||||||
- design/feature/cache_dit.md
|
- Mori Transfer Engine Connector: design/feature/omni_connectors/mori_transfer_engine_connector.md
|
||||||
- design/feature/teacache.md
|
- Shared Memory Connector: design/feature/omni_connectors/shared_memory_connector.md
|
||||||
- design/feature/skip_softmax.md
|
- Yuanrong Store Connector: design/feature/omni_connectors/yuanrong_connector.md
|
||||||
- design/feature/async_chunk.md
|
- Yuanrong Transfer Engine Connector: design/feature/omni_connectors/yuanrong_transfer_engine_connector.md
|
||||||
- design/feature/omni_async_output_materialization.md
|
- Diffusion Acceleration:
|
||||||
- design/feature/vae_parallel.md
|
- Parallelism:
|
||||||
- design/feature/diffusion_continuous_batching.md
|
- CFG-Parallel: design/feature/cfg_parallel.md
|
||||||
- design/feature/distributed_layerwise_offload.md
|
- Expert Parallel: design/feature/expert_parallel.md
|
||||||
|
- Hybrid Sharded Data Parallel (HSDP): design/feature/hsdp.md
|
||||||
|
- Pipeline Parallel: design/feature/pipeline_parallel.md
|
||||||
|
- Sequence Parallel: design/feature/sequence_parallel.md
|
||||||
|
- Tensor Parallel: design/feature/tensor_parallel.md
|
||||||
|
- VAE Patch Parallelism: design/feature/vae_parallel.md
|
||||||
|
- Attention Backends:
|
||||||
|
- Skip-Softmax: design/feature/skip_softmax.md
|
||||||
|
- Quantization: design/feature/quantization.md
|
||||||
|
- Cache-DiT: design/feature/cache_dit.md
|
||||||
|
- TeaCache: design/feature/teacache.md
|
||||||
|
- Diffusion Continuous Batching: design/feature/diffusion_continuous_batching.md
|
||||||
|
- Distributed Layerwise Offload: design/feature/distributed_layerwise_offload.md
|
||||||
|
- Infrastructure and Performance:
|
||||||
|
- Prometheus Metrics: design/metrics.md
|
||||||
|
- Speech Generation Performance Optimizations: design/qwen3_omni_tts_performance_optimization.md
|
||||||
- Module Design:
|
- Module Design:
|
||||||
- design/module/ar_module.md
|
- Runtime Modules:
|
||||||
- design/module/dit_module.md
|
- AR Module: design/module/ar_module.md
|
||||||
- design/module/entrypoint_module.md
|
- DIT Module: design/module/dit_module.md
|
||||||
- design/module/async_omni_architecture.md
|
- Orchestration:
|
||||||
|
- Entrypoint Module: design/module/entrypoint_module.md
|
||||||
|
- AsyncOmni Architecture: design/module/async_omni_architecture.md
|
||||||
- Docs Guide: contributing/DOCS_GUIDE.md
|
- Docs Guide: contributing/DOCS_GUIDE.md
|
||||||
- API Reference:
|
- API Reference:
|
||||||
- api/README.md
|
- api/README.md
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
**Scope**
|
**Scope**
|
||||||
|
|
||||||
For diffusion models, the L4 functionality test covers all or common *diffusion features* that are supported by this model, including several [parallelism acceleration methods](https://docs.vllm.ai/projects/vllm-omni/en/latest/user_guide/diffusion/parallelism_acceleration/), [CPU offloading](https://docs.vllm.ai/projects/vllm-omni/en/latest/user_guide/diffusion/cpu_offload_diffusion/), [TeaCache](https://docs.vllm.ai/projects/vllm-omni/en/latest/user_guide/diffusion/teacache/) and [Cache-DiT](https://docs.vllm.ai/projects/vllm-omni/en/latest/user_guide/diffusion/cache_dit_acceleration/) cache backends, [quantization methods](https://docs.vllm.ai/projects/vllm-omni/en/latest/user_guide/quantization/overview/).
|
For diffusion models, the L4 functionality test covers all or common *diffusion features* that are supported by this model, including several [parallelism acceleration methods](https://docs.vllm.ai/projects/vllm-omni/en/latest/user_guide/diffusion/parallelism_acceleration/), [CPU offloading](https://docs.vllm.ai/projects/vllm-omni/en/latest/user_guide/diffusion/cpu_offload/), [TeaCache](https://docs.vllm.ai/projects/vllm-omni/en/latest/user_guide/diffusion/teacache/) and [Cache-DiT](https://docs.vllm.ai/projects/vllm-omni/en/latest/user_guide/diffusion/cache_dit_acceleration/) cache backends, [quantization methods](https://docs.vllm.ai/projects/vllm-omni/en/latest/user_guide/quantization/overview/).
|
||||||
|
|
||||||
**Test Case Design**
|
**Test Case Design**
|
||||||
|
|
||||||
|
|||||||
@@ -824,7 +824,7 @@ omni = Omni(model="your-model",
|
|||||||
|
|
||||||
### CPU Offload
|
### CPU Offload
|
||||||
|
|
||||||
See detailed guide: [CPU Offloading for Diffusion Models](../../user_guide/diffusion/cpu_offload_diffusion.md)
|
See detailed guide: [CPU Offloading for Diffusion Models](../../user_guide/diffusion/cpu_offload.md)
|
||||||
|
|
||||||
vLLM-Omni provides two offloading strategies to reduce GPU memory usage:
|
vLLM-Omni provides two offloading strategies to reduce GPU memory usage:
|
||||||
|
|
||||||
|
|||||||
@@ -7,7 +7,7 @@ either shard those host-side weights across an existing parallel group or keep
|
|||||||
the standard loader's rank-local weights and avoid an additional collective.
|
the standard loader's rank-local weights and avoid an additional collective.
|
||||||
|
|
||||||
For user-facing commands, see the
|
For user-facing commands, see the
|
||||||
[distributed layerwise offloading guide](../../user_guide/diffusion/cpu_offload_diffusion.md)
|
[distributed layerwise offloading guide](../../user_guide/diffusion/cpu_offload.md)
|
||||||
and the [Cosmos3 recipe](https://github.com/vllm-project/vllm-omni/blob/main/recipes/cosmos3/Cosmos3-DistOffload.md).
|
and the [Cosmos3 recipe](https://github.com/vllm-project/vllm-omni/blob/main/recipes/cosmos3/Cosmos3-DistOffload.md).
|
||||||
|
|
||||||
## Status
|
## Status
|
||||||
|
|||||||
@@ -0,0 +1,249 @@
|
|||||||
|
# Quantization
|
||||||
|
|
||||||
|
This document describes the quantization architecture in vLLM-Omni and the
|
||||||
|
extension points for contributors. For user-facing configuration, supported
|
||||||
|
hardware, and method-specific instructions, see the
|
||||||
|
[quantization user guide](../../user_guide/quantization/overview.md).
|
||||||
|
|
||||||
|
## Goals and scope
|
||||||
|
|
||||||
|
The design has four goals:
|
||||||
|
|
||||||
|
1. Provide one `quantization_config` entry point for diffusion-only,
|
||||||
|
multi-stage omni/TTS, and multi-stage diffusion models.
|
||||||
|
2. Reuse vLLM's quantization registry and `QuantizationConfig` contract where
|
||||||
|
a vLLM backend already provides the required behavior.
|
||||||
|
3. Allow different components of one pipeline to use different quantization
|
||||||
|
configurations.
|
||||||
|
4. Preserve the distinction between online quantization of a BF16/FP16
|
||||||
|
checkpoint and loading a pre-quantized checkpoint with serialized scales.
|
||||||
|
|
||||||
|
This document covers configuration resolution, component routing, model
|
||||||
|
integration, backend extension, and validation. Kernel implementations and
|
||||||
|
user-facing support matrices are outside its scope.
|
||||||
|
|
||||||
|
## Architecture
|
||||||
|
|
||||||
|
The quantization path is split across the following layers:
|
||||||
|
|
||||||
|
| Layer | Responsibility | Source |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| Public API | Exposes `build_quant_config`, component routing, and backend registration | `vllm_omni/quantization/__init__.py` |
|
||||||
|
| Factory | Normalizes method names, selects Omni overrides or vLLM registry backends, detects ModelOpt configs, and reconciles checkpoint metadata | `vllm_omni/quantization/factory.py` |
|
||||||
|
| Component router | Resolves a quantization config from a layer prefix using longest-prefix matching | `vllm_omni/quantization/component_config.py` |
|
||||||
|
| Method configs | Implement `QuantizationConfig` and select the platform-specific linear method | `vllm_omni/quantization/*_config.py` |
|
||||||
|
| Diffusion configuration | Stores the canonical runtime config and propagates checkpoint configuration | `vllm_omni/diffusion/data.py` |
|
||||||
|
| Worker and model integration | Passes the resolved config into vLLM's model configuration and model layers | `vllm_omni/diffusion/worker/diffusion_worker.py` and `vllm_omni/diffusion/models/` |
|
||||||
|
|
||||||
|
The runtime contract remains vLLM's `QuantizationConfig`:
|
||||||
|
|
||||||
|
```text
|
||||||
|
user/CLI or model config
|
||||||
|
|
|
||||||
|
v
|
||||||
|
OmniDiffusionConfig.quantization_config
|
||||||
|
|
|
||||||
|
+--> build_quant_config()
|
||||||
|
| |
|
||||||
|
| +--> Omni override
|
||||||
|
| +--> vLLM registry backend
|
||||||
|
| +--> ComponentQuantizationConfig
|
||||||
|
|
|
||||||
|
+--> checkpoint metadata reconciliation
|
||||||
|
|
|
||||||
|
+--> diffusion worker / vLLM config
|
||||||
|
|
|
||||||
|
+--> model layers call get_quant_method(layer, prefix)
|
||||||
|
```
|
||||||
|
|
||||||
|
The model layer receives either a method implementation or `None`. Returning
|
||||||
|
`None` leaves that layer unquantized.
|
||||||
|
|
||||||
|
## Configuration resolution
|
||||||
|
|
||||||
|
`build_quant_config()` accepts the following forms:
|
||||||
|
|
||||||
|
| Input | Resolution |
|
||||||
|
| --- | --- |
|
||||||
|
| `None` or `"none"` | Disable quantization. |
|
||||||
|
| Method string, such as `"fp8"` | Resolve an Omni override first, then the vLLM quantization registry. |
|
||||||
|
| Flat dictionary with `method` or `quant_method` | Normalize the method and construct its `QuantizationConfig`. |
|
||||||
|
| Per-component dictionary | Construct a `ComponentQuantizationConfig`. |
|
||||||
|
| Existing `QuantizationConfig` | Pass through without rebuilding it. |
|
||||||
|
|
||||||
|
Method aliases are normalized case-insensitively, with `-` and `_` treated as
|
||||||
|
equivalent. `auto-round` and `auto_round` use the Omni INC/AutoRound adapter.
|
||||||
|
ModelOpt checkpoint metadata is detected from its method, producer metadata, or
|
||||||
|
`quant_algo`, then mapped to the corresponding vLLM ModelOpt configuration.
|
||||||
|
|
||||||
|
The factory uses Omni-specific overrides for methods whose diffusion behavior
|
||||||
|
or platform dispatch is not provided by the vLLM registry. The current override
|
||||||
|
set includes INT8, BitsAndBytes, MXFP8, MXFP4, MXFP4 dual-scale, and INC/
|
||||||
|
AutoRound. Other methods, such as FP8, GGUF, and ModelOpt, are resolved through
|
||||||
|
vLLM's registry when their configuration is compatible.
|
||||||
|
|
||||||
|
## Per-component routing
|
||||||
|
|
||||||
|
Multi-stage models do not necessarily want one quantization policy for every
|
||||||
|
module. A component map expresses that policy without requiring each pipeline
|
||||||
|
to construct separate vLLM configurations:
|
||||||
|
|
||||||
|
```python
|
||||||
|
quantization_config = {
|
||||||
|
"transformer": {"method": "fp8"},
|
||||||
|
"vae": None,
|
||||||
|
"default": None,
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
`ComponentQuantizationConfig` applies the following rules:
|
||||||
|
|
||||||
|
1. Component keys are interpreted as runtime layer-prefixes.
|
||||||
|
2. The longest matching prefix wins.
|
||||||
|
3. `None` disables quantization for the matching component.
|
||||||
|
4. `default`, when present, handles prefixes with no explicit match.
|
||||||
|
5. A matching child config delegates to its normal
|
||||||
|
`get_quant_method(layer, prefix)` implementation.
|
||||||
|
|
||||||
|
Prefix matching happens after the model's weight-name mapping. New model
|
||||||
|
integrations must verify the actual runtime prefixes, especially when a
|
||||||
|
`WeightsMapper` changes checkpoint names. A prefix mismatch can silently route
|
||||||
|
a layer to the default configuration, so component-routing tests should cover
|
||||||
|
both positive matches and fall-through behavior.
|
||||||
|
|
||||||
|
## Online and pre-quantized checkpoints
|
||||||
|
|
||||||
|
The same method name can represent two different loading paths:
|
||||||
|
|
||||||
|
- **Online**: the source checkpoint contains BF16/FP16 weights; the method
|
||||||
|
creates quantized weights or scales while the model is loaded.
|
||||||
|
- **Pre-quantized**: the checkpoint contains quantized weights and the metadata
|
||||||
|
required by the method; the loader consumes those serialized tensors.
|
||||||
|
|
||||||
|
Method configuration owns the mode flag, for example
|
||||||
|
`is_checkpoint_int8_serialized` or `is_checkpoint_mxfp8_serialized`. The
|
||||||
|
configuration's `get_quant_method()` selects the corresponding linear method.
|
||||||
|
|
||||||
|
For pipelines with transformer-specific `config.json` files, such as cascade
|
||||||
|
models, `resolve_quant_config_from_disk()` reconciles the active configuration
|
||||||
|
with each transformer's checkpoint metadata. Its invariants are:
|
||||||
|
|
||||||
|
- If no active configuration exists, valid checkpoint metadata can be used for
|
||||||
|
auto-detection.
|
||||||
|
- A method mismatch is an error rather than an implicit conversion.
|
||||||
|
- Serialized-checkpoint flags rebuild an online configuration into the
|
||||||
|
matching offline configuration.
|
||||||
|
- Checkpoint-specific `ignored_layers` are preserved by rebuilding the active
|
||||||
|
configuration when they differ.
|
||||||
|
- AutoRound MXFP8 metadata (`data_type="mx_fp"`) selects its offline path.
|
||||||
|
|
||||||
|
This prevents a model from interpreting serialized weights with an online
|
||||||
|
loader or silently applying the wrong method to one transformer in a
|
||||||
|
multi-transformer pipeline.
|
||||||
|
|
||||||
|
## Model and layer integration
|
||||||
|
|
||||||
|
Diffusion model constructors receive the resolved configuration and pass it to
|
||||||
|
quantizable vLLM layers. Each method configuration is responsible for:
|
||||||
|
|
||||||
|
- identifying supported layers in `get_quant_method()`;
|
||||||
|
- creating the correct parameter shapes and loaders;
|
||||||
|
- processing weights after loading when conversion is required; and
|
||||||
|
- applying the platform-specific quantized operation at runtime.
|
||||||
|
|
||||||
|
Model code must keep precision-sensitive or unsupported modules unquantized.
|
||||||
|
The shared `safe_quant_config()` helper is used by model integrations for
|
||||||
|
normalization and modulation layers where a generic FP8 configuration would be
|
||||||
|
unsafe. Text encoders, audio/vision encoders, VAEs, schedulers, and tokenizers
|
||||||
|
are not implicitly quantized just because the main transformer is quantized;
|
||||||
|
they require explicit component routing and method support.
|
||||||
|
|
||||||
|
Weight quantization and quantized KV cache are separate paths. The former is
|
||||||
|
resolved through `quantization_config`; runtime attention/KV-cache quantization
|
||||||
|
is configured by the diffusion attention configuration and must be validated
|
||||||
|
independently.
|
||||||
|
|
||||||
|
## Platform and parallelism boundaries
|
||||||
|
|
||||||
|
Platform checks belong in the method implementation rather than in the common
|
||||||
|
factory. This keeps configuration parsing portable while allowing CUDA, NPU,
|
||||||
|
and XPU methods to select different kernels or reject unsupported execution at
|
||||||
|
model construction time. The user guide is the source of truth for the current
|
||||||
|
hardware support matrix.
|
||||||
|
|
||||||
|
Quantization methods must also respect the weight layout produced by the
|
||||||
|
configured parallelism strategy:
|
||||||
|
|
||||||
|
- Tensor-parallel layers may receive partitioned weight shapes and loaders.
|
||||||
|
- Sequence-parallel execution changes activation movement but does not by
|
||||||
|
itself change the quantization configuration.
|
||||||
|
- Data-parallel replicas build their own method state; quantization metadata
|
||||||
|
must be available consistently on every rank.
|
||||||
|
- `WeightsMapper` transformations must be applied before matching ignored-layer
|
||||||
|
lists or component prefixes.
|
||||||
|
|
||||||
|
Distributed layerwise offload has an additional boundary: its sharded mmap
|
||||||
|
AllGather path is not compatible with online quantization because the path
|
||||||
|
expects already materialized checkpoint weights. Use the rank-local
|
||||||
|
`--dlo-no-use-allgather` path or a compatible pre-quantized checkpoint when
|
||||||
|
combining these features. See the [DLO design document](distributed_layerwise_offload.md)
|
||||||
|
for the full compatibility matrix.
|
||||||
|
|
||||||
|
## Adding a quantization backend
|
||||||
|
|
||||||
|
Contributors should follow this sequence:
|
||||||
|
|
||||||
|
1. Check whether an existing vLLM backend can provide the required weight
|
||||||
|
format and kernel behavior. Prefer reuse over a duplicate Omni backend.
|
||||||
|
2. If Omni-specific behavior is required, implement a `QuantizationConfig` and
|
||||||
|
its `QuantizeMethodBase`/linear method in `vllm_omni/quantization/`.
|
||||||
|
3. Implement `get_name()`, supported activation dtypes, minimum capability,
|
||||||
|
checkpoint parsing, ignored-layer handling, and `get_quant_method()`.
|
||||||
|
4. Add a lazy factory override in `factory.py`. Keep optional platform
|
||||||
|
dependencies out of package import paths and normalize aliases to one
|
||||||
|
canonical method name.
|
||||||
|
5. If the method is pre-quantized, define the checkpoint metadata and the
|
||||||
|
serialized loading path before adding an online path.
|
||||||
|
6. Add model-specific component routing only when the model contains stages
|
||||||
|
that must use different policies.
|
||||||
|
7. Document user-visible configuration in the quantization user guide and keep
|
||||||
|
implementation rationale here.
|
||||||
|
|
||||||
|
The public factory registration hook,
|
||||||
|
`register_quantization_override(method, builder)`, is intended for integrations
|
||||||
|
that need to register an Omni-compatible builder without changing the vLLM
|
||||||
|
registry.
|
||||||
|
|
||||||
|
## Validation
|
||||||
|
|
||||||
|
Validation should cover the complete path, not only whether a config object can
|
||||||
|
be instantiated:
|
||||||
|
|
||||||
|
| Area | Required checks |
|
||||||
|
| --- | --- |
|
||||||
|
| Factory | Strings, aliases, flat dictionaries, invalid inputs, passthrough, and component maps |
|
||||||
|
| Method config | Online/offline selection, ignored layers, platform guards, and weight loading |
|
||||||
|
| Checkpoint metadata | Auto-detection, method mismatch errors, serialized flags, and per-transformer metadata |
|
||||||
|
| Model integration | Correct quantization scope, unquantized components, and parallel weight layouts |
|
||||||
|
| Numerical quality | Same-seed BF16 versus quantized output comparison at representative resolutions and step counts |
|
||||||
|
| Performance | Peak device memory, generation latency, and any backend-specific throughput metric |
|
||||||
|
|
||||||
|
The unit tests live under `tests/diffusion/quantization/`. The trajectory
|
||||||
|
similarity tool in
|
||||||
|
`vllm_omni/quantization/tools/compare_diffusion_trajectory_similarity.py`
|
||||||
|
provides a repeatable quality and memory comparison for diffusion models.
|
||||||
|
Method support and feature combinations remain model- and hardware-dependent;
|
||||||
|
passing factory tests alone does not establish production support.
|
||||||
|
|
||||||
|
## Design invariants
|
||||||
|
|
||||||
|
- `quantization_config` is the single runtime source of truth after config
|
||||||
|
normalization.
|
||||||
|
- A checkpoint method mismatch fails loudly instead of risking weight
|
||||||
|
corruption.
|
||||||
|
- Components without an explicit supported quantization path remain in their
|
||||||
|
original precision.
|
||||||
|
- Optional backend dependencies are imported lazily.
|
||||||
|
- Method implementations own platform-specific capability checks and kernels.
|
||||||
|
- User-facing support claims require model, hardware, numerical, and memory
|
||||||
|
validation in addition to configuration-level tests.
|
||||||
@@ -1,59 +0,0 @@
|
|||||||
# Distributed utils
|
|
||||||
|
|
||||||
This directory (vllm_omni/distributed/ray_utils) contains utilities for distributed execution in vllm-omni, supporting both **Ray** and **Multiprocessing** backends.
|
|
||||||
## 1. Installation
|
|
||||||
```bash
|
|
||||||
pip install "ray[default]"
|
|
||||||
```
|
|
||||||
## 2. Ray Utils
|
|
||||||
|
|
||||||
The `ray_utils` module provides helper functions for managing Ray clusters and actors, which is essential for:
|
|
||||||
* **Multi-node deployment**: Running pipeline stages across different physical machines.
|
|
||||||
* **Resource management**: Efficient GPU/CPU allocation.
|
|
||||||
|
|
||||||
### 2.1 Basic Usage
|
|
||||||
|
|
||||||
To use the Ray backend, specify `worker_backend="ray"` when initializing the engine.
|
|
||||||
|
|
||||||
**Command Line Example:**
|
|
||||||
```bash
|
|
||||||
vllm serve Qwen/Qwen2.5-Omni-7B \
|
|
||||||
--omni \
|
|
||||||
--port 8091 \
|
|
||||||
--worker-backend ray \
|
|
||||||
--ray-address auto
|
|
||||||
```
|
|
||||||
|
|
||||||
### 2.2 Cluster Setup
|
|
||||||
|
|
||||||
**Step 1: Start Head Node**
|
|
||||||
Run this on your primary machine:
|
|
||||||
```bash
|
|
||||||
ray start --head --port=6399
|
|
||||||
```
|
|
||||||
|
|
||||||
**Step 2: Connect Worker Nodes**
|
|
||||||
Run this on each worker machine:
|
|
||||||
```bash
|
|
||||||
ray start --address=<HEAD_NODE_IP>:6399
|
|
||||||
```
|
|
||||||
|
|
||||||
> **Tip**: For a complete cluster setup script, refer to the vLLM example:
|
|
||||||
> [run_cluster.sh](https://github.com/vllm-project/vllm/blob/main/examples/online_serving/run_cluster.sh)
|
|
||||||
|
|
||||||
### 2.3 Distributed Connector Support
|
|
||||||
|
|
||||||
When running on Ray, the system automatically adapts its communication strategy:
|
|
||||||
|
|
||||||
* **Cross-Node**: Recommended to use `MooncakeTransferEngineConnector` (RDMA, fastest) or `MooncakeStoreConnector` (TCP fallback).
|
|
||||||
* **Same-Node**: Can still use `SharedMemoryConnector` for efficiency, or Ray's native object store (plasma).
|
|
||||||
* **SHM payload routing**: `SharedMemoryConnector` always stores payloads in `/dev/shm`.
|
|
||||||
|
|
||||||
### 2.4 Internal Helpers
|
|
||||||
|
|
||||||
* **`initialize_ray_cluster`**: Connects to an existing Ray cluster or starts a local one.
|
|
||||||
|
|
||||||
## 3. Troubleshooting
|
|
||||||
|
|
||||||
* **Connection Issues**: Ensure the Ray head node is accessible and ports (default 6399 in this example) are open.
|
|
||||||
* **Version Mismatch**: Ensure all nodes run the same version of Ray and Python.
|
|
||||||
+50
-3
@@ -1,6 +1,8 @@
|
|||||||
# Design Documents
|
# Design Documents
|
||||||
|
|
||||||
This section contains design documents and architecture specifications for vLLM-Omni.
|
This section contains design documents and architecture specifications for
|
||||||
|
vLLM-Omni. The sidebar groups documents by the system concern they describe,
|
||||||
|
while the source paths remain stable so existing links continue to work.
|
||||||
|
|
||||||
## Architecture Documents
|
## Architecture Documents
|
||||||
|
|
||||||
@@ -8,18 +10,63 @@ This section contains design documents and architecture specifications for vLLM-
|
|||||||
|
|
||||||
## Feature Design Documents
|
## Feature Design Documents
|
||||||
|
|
||||||
|
### Runtime and stage execution
|
||||||
|
|
||||||
- [Disaggregated Inference](feature/disaggregated_inference.md)
|
- [Disaggregated Inference](feature/disaggregated_inference.md)
|
||||||
- [Ray-based Execution](feature/ray_based_execution.md)
|
- [Async Chunk](feature/async_chunk.md)
|
||||||
|
- [Async Diffusion Output](feature/async_diffusion_output.md)
|
||||||
- [Async Omni Output Materialization](feature/omni_async_output_materialization.md)
|
- [Async Omni Output Materialization](feature/omni_async_output_materialization.md)
|
||||||
|
- [Automatic Prefix Caching in Omni Models](feature/prefix_caching.md)
|
||||||
|
|
||||||
|
### Communication
|
||||||
|
|
||||||
|
#### OmniConnector implementations
|
||||||
|
|
||||||
|
- [Mooncake Store Connector](feature/omni_connectors/mooncake_store_connector.md)
|
||||||
|
- [Mooncake Transfer Engine Connector](feature/omni_connectors/mooncake_transfer_engine_connector.md)
|
||||||
|
- [Mori Transfer Engine Connector](feature/omni_connectors/mori_transfer_engine_connector.md)
|
||||||
|
- [Shared Memory Connector](feature/omni_connectors/shared_memory_connector.md)
|
||||||
|
- [Yuanrong Store Connector](feature/omni_connectors/yuanrong_connector.md)
|
||||||
|
- [Yuanrong Transfer Engine Connector](feature/omni_connectors/yuanrong_transfer_engine_connector.md)
|
||||||
|
|
||||||
|
### Diffusion acceleration
|
||||||
|
|
||||||
|
#### Parallelism
|
||||||
|
|
||||||
|
- [CFG-Parallel](feature/cfg_parallel.md)
|
||||||
|
- [Expert Parallel](feature/expert_parallel.md)
|
||||||
|
- [Hybrid Sharded Data Parallel (HSDP)](feature/hsdp.md)
|
||||||
|
- [Pipeline Parallel](feature/pipeline_parallel.md)
|
||||||
|
- [Sequence Parallel](feature/sequence_parallel.md)
|
||||||
|
- [Tensor Parallel](feature/tensor_parallel.md)
|
||||||
|
- [VAE Patch Parallelism](feature/vae_parallel.md)
|
||||||
|
|
||||||
|
#### Attention Backends
|
||||||
|
|
||||||
|
- [Skip-Softmax](feature/skip_softmax.md)
|
||||||
|
|
||||||
|
#### Quantization
|
||||||
|
|
||||||
|
- [Quantization](feature/quantization.md)
|
||||||
|
|
||||||
|
- [Cache-DiT](feature/cache_dit.md)
|
||||||
|
- [TeaCache](feature/teacache.md)
|
||||||
- [Diffusion Continuous Batching](feature/diffusion_continuous_batching.md)
|
- [Diffusion Continuous Batching](feature/diffusion_continuous_batching.md)
|
||||||
- [Distributed Layerwise Offload](feature/distributed_layerwise_offload.md)
|
- [Distributed Layerwise Offload](feature/distributed_layerwise_offload.md)
|
||||||
|
|
||||||
## Infrastructure Design Documents
|
## Infrastructure and Performance
|
||||||
|
|
||||||
- [Prometheus Metrics](metrics.md)
|
- [Prometheus Metrics](metrics.md)
|
||||||
|
- [Speech Generation Performance Optimizations](qwen3_omni_tts_performance_optimization.md)
|
||||||
|
|
||||||
## Module Design Documents
|
## Module Design Documents
|
||||||
|
|
||||||
|
### Runtime Modules
|
||||||
|
|
||||||
- [AR Module](module/ar_module.md)
|
- [AR Module](module/ar_module.md)
|
||||||
- [DIT Module](module/dit_module.md)
|
- [DIT Module](module/dit_module.md)
|
||||||
|
|
||||||
|
### Orchestration
|
||||||
|
|
||||||
- [Entrypoint Module](module/entrypoint_module.md)
|
- [Entrypoint Module](module/entrypoint_module.md)
|
||||||
|
- [AsyncOmni Architecture (Qwen3-Omni Example)](module/async_omni_architecture.md)
|
||||||
File renamed without changes.
@@ -59,7 +59,7 @@ Memory optimization methods help reduce GPU memory usage, enabling inference on
|
|||||||
|
|
||||||
| Method | Description | Best For |
|
| Method | Description | Best For |
|
||||||
|--------|-------------|----------|
|
|--------|-------------|----------|
|
||||||
| **[CPU Offload](diffusion/cpu_offload_diffusion.md)** | Offloads model components to CPU memory | Limited VRAM, large models on consumer GPUs |
|
| **[CPU Offload](diffusion/cpu_offload.md)** | Offloads model components to CPU memory | Limited VRAM, large models on consumer GPUs |
|
||||||
| **[Quantization](quantization/overview.md)** | Reduces transformer stages from BF16 to FP8/INT8/etc. | Limited VRAM, minimal accuracy loss |
|
| **[Quantization](quantization/overview.md)** | Reduces transformer stages from BF16 to FP8/INT8/etc. | Limited VRAM, minimal accuracy loss |
|
||||||
| **[VAE Parallelism](diffusion/parallelism/vae_parallelism.md)** | Distributes VAE decode work across GPUs | High-resolution generation with reduced VAE memory peak |
|
| **[VAE Parallelism](diffusion/parallelism/vae_parallelism.md)** | Distributes VAE decode work across GPUs | High-resolution generation with reduced VAE memory peak |
|
||||||
|
|
||||||
@@ -104,8 +104,8 @@ The following tables show which models support each feature:
|
|||||||
|
|
||||||
> Notes:
|
> Notes:
|
||||||
|
|
||||||
> 1. CPU Offload has two methods: Module-wise (default for models with DiT + text encoder) and Layerwise. The tables below show **Layerwise support** only. Split models like Cosmos3 (no separate text encoder) swap their reasoner/generator components for module-wise offload; see the [CPU Offload Guide](diffusion/cpu_offload_diffusion.md).
|
> 1. CPU Offload has two methods: Module-wise (default for models with DiT + text encoder) and Layerwise. The tables below show **Layerwise support** only. Split models like Cosmos3 (no separate text encoder) swap their reasoner/generator components for module-wise offload; see the [CPU Offload Guide](diffusion/cpu_offload.md).
|
||||||
> 2. The **💾Quantization** column is collapsed for readability. See [Quantization Overview](quantization/overview.md) for per-method and per-model support details.
|
> 2. The **💾Quantization** column is collapsed for readability. See [Quantization](quantization/overview.md) for per-method and per-model support details.
|
||||||
|
|
||||||
### ImageGen
|
### ImageGen
|
||||||
|
|
||||||
@@ -274,9 +274,9 @@ Measured on NVIDIA H800:
|
|||||||
|
|
||||||
**Memory Optimization:**
|
**Memory Optimization:**
|
||||||
|
|
||||||
- **[CPU Offload Guide](diffusion/cpu_offload_diffusion.md)** - Offload model components to CPU, reduce GPU memory usage
|
- **[CPU Offload Guide](diffusion/cpu_offload.md)** - Offload model components to CPU, reduce GPU memory usage
|
||||||
- **[VAE Parallelism Guide](diffusion/parallelism/vae_parallelism.md)** - Distribute VAE decode work across GPUs for high-resolution images and videos
|
- **[VAE Parallelism Guide](diffusion/parallelism/vae_parallelism.md)** - Distribute VAE decode work across GPUs for high-resolution images and videos
|
||||||
- **[Quantization Overview](quantization/overview.md)** - Overview of quantization methods for diffusion, multi-stage omni/TTS, and multi-stage diffusion models
|
- **[Quantization](quantization/overview.md)** - Quantization methods for diffusion, multi-stage omni/TTS, and multi-stage diffusion models
|
||||||
|
|
||||||
**Extensions:**
|
**Extensions:**
|
||||||
|
|
||||||
|
|||||||
@@ -5,6 +5,9 @@ path. The same configuration entrypoint is used across diffusion-only models,
|
|||||||
multi-stage omni/TTS models, and multi-stage diffusion models, but each model
|
multi-stage omni/TTS models, and multi-stage diffusion models, but each model
|
||||||
type has a different quantization scope.
|
type has a different quantization scope.
|
||||||
|
|
||||||
|
For the internal architecture and backend extension points, see the
|
||||||
|
[quantization design document](../../design/feature/quantization.md).
|
||||||
|
|
||||||
## Quantization Modes
|
## Quantization Modes
|
||||||
|
|
||||||
| Mode | Guide | Description | Methods |
|
| Mode | Guide | Description | Methods |
|
||||||
|
|||||||
@@ -75,6 +75,8 @@ plugins:
|
|||||||
redirect_maps:
|
redirect_maps:
|
||||||
user_guide/diffusion/request_batching.md: user_guide/diffusion/execution_modes.md
|
user_guide/diffusion/request_batching.md: user_guide/diffusion/execution_modes.md
|
||||||
user_guide/diffusion/step_execution.md: user_guide/diffusion/execution_modes.md
|
user_guide/diffusion/step_execution.md: user_guide/diffusion/execution_modes.md
|
||||||
|
user_guide/diffusion/cpu_offload_diffusion.md: user_guide/diffusion/cpu_offload.md
|
||||||
|
user_guide/quantization/index.md: user_guide/quantization/overview.md
|
||||||
design/feature/diffusion_request_level_batching.md: design/feature/diffusion_continuous_batching.md
|
design/feature/diffusion_request_level_batching.md: design/feature/diffusion_continuous_batching.md
|
||||||
design/feature/diffusion_step_execution.md: design/feature/diffusion_continuous_batching.md
|
design/feature/diffusion_step_execution.md: design/feature/diffusion_continuous_batching.md
|
||||||
- glightbox
|
- glightbox
|
||||||
|
|||||||
Reference in new issue
Block a user