Compare commits
7
Commits
4d617b6ed9
...
a5ee63fb1f
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
a5ee63fb1f | ||
|
|
3335e76c97 | ||
|
|
9167599b3c | ||
|
|
99f946210b | ||
|
|
ec4e12f379 | ||
|
|
d272d57bef | ||
|
|
bc95e7415f |
No files matched your search
@@ -309,7 +309,7 @@ For Omni or custom models, create:
|
||||
|
||||
Required updates:
|
||||
1. `docs/user_guide/diffusion/parallelism/overview.md` — parallelism support overview/table
|
||||
2. `docs/user_guide/diffusion/cpu_offload_diffusion.md` — if CPU offload supported (add to supported models table)
|
||||
2. `docs/user_guide/diffusion/cpu_offload.md` — if CPU offload supported (add to supported models table)
|
||||
3. `docs/user_guide/diffusion/cache_acceleration/teacache.md` — if TeaCache supported
|
||||
4. `docs/user_guide/diffusion/cache_acceleration/cache_dit.md` — if Cache-DiT supported
|
||||
5. Offline example docs under `examples/offline_inference/<name>/` (`README.md` or category-specific `.md`)
|
||||
|
||||
+43
-25
@@ -68,12 +68,12 @@ nav:
|
||||
- Sequence Parallel: user_guide/diffusion/parallelism/sequence_parallel.md
|
||||
- Tensor Parallel: user_guide/diffusion/parallelism/tensor_parallel.md
|
||||
- VAE Parallelism: user_guide/diffusion/parallelism/vae_parallelism.md
|
||||
- CPU Offloading: user_guide/diffusion/cpu_offload_diffusion.md
|
||||
- CPU Offloading: user_guide/diffusion/cpu_offload.md
|
||||
- LoRA: user_guide/diffusion/lora.md
|
||||
- Custom Pipeline: features/custom_pipeline.md
|
||||
- Execution Modes: user_guide/diffusion/execution_modes.md
|
||||
- Quantization:
|
||||
- Overview: user_guide/quantization/overview.md
|
||||
- Quantization: user_guide/quantization/overview.md
|
||||
- Online Quantization: user_guide/quantization/online.md
|
||||
- Quantized KV Cache: user_guide/quantization/quantized_kvcache.md
|
||||
- FP8 W8A8: user_guide/quantization/fp8.md
|
||||
@@ -101,31 +101,49 @@ nav:
|
||||
- CI: contributing/ci
|
||||
- Design Documents:
|
||||
- design/index.md
|
||||
- design/architecture_overview.md
|
||||
- Architecture Documents:
|
||||
- Architecture Overview: design/architecture_overview.md
|
||||
- Feature Design:
|
||||
- design/feature/disaggregated_inference.md
|
||||
- design/feature/ray_based_execution.md
|
||||
- design/feature/omni_connectors/
|
||||
- design/feature/prefix_caching.md
|
||||
- design/feature/cfg_parallel.md
|
||||
- design/feature/expert_parallel.md
|
||||
- design/feature/sequence_parallel.md
|
||||
- design/feature/tensor_parallel.md
|
||||
- design/feature/vae_parallel.md
|
||||
- design/feature/hsdp.md
|
||||
- design/feature/cache_dit.md
|
||||
- design/feature/teacache.md
|
||||
- design/feature/skip_softmax.md
|
||||
- design/feature/async_chunk.md
|
||||
- design/feature/omni_async_output_materialization.md
|
||||
- design/feature/vae_parallel.md
|
||||
- design/feature/diffusion_continuous_batching.md
|
||||
- design/feature/distributed_layerwise_offload.md
|
||||
- Runtime and Stage Execution:
|
||||
- Disaggregated Inference: design/feature/disaggregated_inference.md
|
||||
- Async Chunk: design/feature/async_chunk.md
|
||||
- Async Diffusion Output: design/feature/async_diffusion_output.md
|
||||
- Async Omni Output Materialization: design/feature/omni_async_output_materialization.md
|
||||
- Automatic Prefix Caching: design/feature/prefix_caching.md
|
||||
- Communication:
|
||||
- OmniConnector Implementations:
|
||||
- Mooncake Store Connector: design/feature/omni_connectors/mooncake_store_connector.md
|
||||
- Mooncake Transfer Engine Connector: design/feature/omni_connectors/mooncake_transfer_engine_connector.md
|
||||
- Mori Transfer Engine Connector: design/feature/omni_connectors/mori_transfer_engine_connector.md
|
||||
- Shared Memory Connector: design/feature/omni_connectors/shared_memory_connector.md
|
||||
- Yuanrong Store Connector: design/feature/omni_connectors/yuanrong_connector.md
|
||||
- Yuanrong Transfer Engine Connector: design/feature/omni_connectors/yuanrong_transfer_engine_connector.md
|
||||
- Diffusion Acceleration:
|
||||
- Parallelism:
|
||||
- CFG-Parallel: design/feature/cfg_parallel.md
|
||||
- Expert Parallel: design/feature/expert_parallel.md
|
||||
- Hybrid Sharded Data Parallel (HSDP): design/feature/hsdp.md
|
||||
- Pipeline Parallel: design/feature/pipeline_parallel.md
|
||||
- Sequence Parallel: design/feature/sequence_parallel.md
|
||||
- Tensor Parallel: design/feature/tensor_parallel.md
|
||||
- VAE Patch Parallelism: design/feature/vae_parallel.md
|
||||
- Attention Backends:
|
||||
- Skip-Softmax: design/feature/skip_softmax.md
|
||||
- Quantization: design/feature/quantization.md
|
||||
- Cache-DiT: design/feature/cache_dit.md
|
||||
- TeaCache: design/feature/teacache.md
|
||||
- Diffusion Continuous Batching: design/feature/diffusion_continuous_batching.md
|
||||
- Distributed Layerwise Offload: design/feature/distributed_layerwise_offload.md
|
||||
- Infrastructure and Performance:
|
||||
- Prometheus Metrics: design/metrics.md
|
||||
- Speech Generation Performance Optimizations: design/qwen3_omni_tts_performance_optimization.md
|
||||
- Module Design:
|
||||
- design/module/ar_module.md
|
||||
- design/module/dit_module.md
|
||||
- design/module/entrypoint_module.md
|
||||
- design/module/async_omni_architecture.md
|
||||
- Runtime Modules:
|
||||
- AR Module: design/module/ar_module.md
|
||||
- DIT Module: design/module/dit_module.md
|
||||
- Orchestration:
|
||||
- Entrypoint Module: design/module/entrypoint_module.md
|
||||
- AsyncOmni Architecture: design/module/async_omni_architecture.md
|
||||
- Docs Guide: contributing/DOCS_GUIDE.md
|
||||
- API Reference:
|
||||
- api/README.md
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
**Scope**
|
||||
|
||||
For diffusion models, the L4 functionality test covers all or common *diffusion features* that are supported by this model, including several [parallelism acceleration methods](https://docs.vllm.ai/projects/vllm-omni/en/latest/user_guide/diffusion/parallelism_acceleration/), [CPU offloading](https://docs.vllm.ai/projects/vllm-omni/en/latest/user_guide/diffusion/cpu_offload_diffusion/), [TeaCache](https://docs.vllm.ai/projects/vllm-omni/en/latest/user_guide/diffusion/teacache/) and [Cache-DiT](https://docs.vllm.ai/projects/vllm-omni/en/latest/user_guide/diffusion/cache_dit_acceleration/) cache backends, [quantization methods](https://docs.vllm.ai/projects/vllm-omni/en/latest/user_guide/quantization/overview/).
|
||||
For diffusion models, the L4 functionality test covers all or common *diffusion features* that are supported by this model, including several [parallelism acceleration methods](https://docs.vllm.ai/projects/vllm-omni/en/latest/user_guide/diffusion/parallelism_acceleration/), [CPU offloading](https://docs.vllm.ai/projects/vllm-omni/en/latest/user_guide/diffusion/cpu_offload/), [TeaCache](https://docs.vllm.ai/projects/vllm-omni/en/latest/user_guide/diffusion/teacache/) and [Cache-DiT](https://docs.vllm.ai/projects/vllm-omni/en/latest/user_guide/diffusion/cache_dit_acceleration/) cache backends, [quantization methods](https://docs.vllm.ai/projects/vllm-omni/en/latest/user_guide/quantization/overview/).
|
||||
|
||||
**Test Case Design**
|
||||
|
||||
|
||||
@@ -824,7 +824,7 @@ omni = Omni(model="your-model",
|
||||
|
||||
### CPU Offload
|
||||
|
||||
See detailed guide: [CPU Offloading for Diffusion Models](../../user_guide/diffusion/cpu_offload_diffusion.md)
|
||||
See detailed guide: [CPU Offloading for Diffusion Models](../../user_guide/diffusion/cpu_offload.md)
|
||||
|
||||
vLLM-Omni provides two offloading strategies to reduce GPU memory usage:
|
||||
|
||||
|
||||
@@ -7,7 +7,7 @@ either shard those host-side weights across an existing parallel group or keep
|
||||
the standard loader's rank-local weights and avoid an additional collective.
|
||||
|
||||
For user-facing commands, see the
|
||||
[distributed layerwise offloading guide](../../user_guide/diffusion/cpu_offload_diffusion.md)
|
||||
[distributed layerwise offloading guide](../../user_guide/diffusion/cpu_offload.md)
|
||||
and the [Cosmos3 recipe](https://github.com/vllm-project/vllm-omni/blob/main/recipes/cosmos3/Cosmos3-DistOffload.md).
|
||||
|
||||
## Status
|
||||
|
||||
@@ -0,0 +1,249 @@
|
||||
# Quantization
|
||||
|
||||
This document describes the quantization architecture in vLLM-Omni and the
|
||||
extension points for contributors. For user-facing configuration, supported
|
||||
hardware, and method-specific instructions, see the
|
||||
[quantization user guide](../../user_guide/quantization/overview.md).
|
||||
|
||||
## Goals and scope
|
||||
|
||||
The design has four goals:
|
||||
|
||||
1. Provide one `quantization_config` entry point for diffusion-only,
|
||||
multi-stage omni/TTS, and multi-stage diffusion models.
|
||||
2. Reuse vLLM's quantization registry and `QuantizationConfig` contract where
|
||||
a vLLM backend already provides the required behavior.
|
||||
3. Allow different components of one pipeline to use different quantization
|
||||
configurations.
|
||||
4. Preserve the distinction between online quantization of a BF16/FP16
|
||||
checkpoint and loading a pre-quantized checkpoint with serialized scales.
|
||||
|
||||
This document covers configuration resolution, component routing, model
|
||||
integration, backend extension, and validation. Kernel implementations and
|
||||
user-facing support matrices are outside its scope.
|
||||
|
||||
## Architecture
|
||||
|
||||
The quantization path is split across the following layers:
|
||||
|
||||
| Layer | Responsibility | Source |
|
||||
| --- | --- | --- |
|
||||
| Public API | Exposes `build_quant_config`, component routing, and backend registration | `vllm_omni/quantization/__init__.py` |
|
||||
| Factory | Normalizes method names, selects Omni overrides or vLLM registry backends, detects ModelOpt configs, and reconciles checkpoint metadata | `vllm_omni/quantization/factory.py` |
|
||||
| Component router | Resolves a quantization config from a layer prefix using longest-prefix matching | `vllm_omni/quantization/component_config.py` |
|
||||
| Method configs | Implement `QuantizationConfig` and select the platform-specific linear method | `vllm_omni/quantization/*_config.py` |
|
||||
| Diffusion configuration | Stores the canonical runtime config and propagates checkpoint configuration | `vllm_omni/diffusion/data.py` |
|
||||
| Worker and model integration | Passes the resolved config into vLLM's model configuration and model layers | `vllm_omni/diffusion/worker/diffusion_worker.py` and `vllm_omni/diffusion/models/` |
|
||||
|
||||
The runtime contract remains vLLM's `QuantizationConfig`:
|
||||
|
||||
```text
|
||||
user/CLI or model config
|
||||
|
|
||||
v
|
||||
OmniDiffusionConfig.quantization_config
|
||||
|
|
||||
+--> build_quant_config()
|
||||
| |
|
||||
| +--> Omni override
|
||||
| +--> vLLM registry backend
|
||||
| +--> ComponentQuantizationConfig
|
||||
|
|
||||
+--> checkpoint metadata reconciliation
|
||||
|
|
||||
+--> diffusion worker / vLLM config
|
||||
|
|
||||
+--> model layers call get_quant_method(layer, prefix)
|
||||
```
|
||||
|
||||
The model layer receives either a method implementation or `None`. Returning
|
||||
`None` leaves that layer unquantized.
|
||||
|
||||
## Configuration resolution
|
||||
|
||||
`build_quant_config()` accepts the following forms:
|
||||
|
||||
| Input | Resolution |
|
||||
| --- | --- |
|
||||
| `None` or `"none"` | Disable quantization. |
|
||||
| Method string, such as `"fp8"` | Resolve an Omni override first, then the vLLM quantization registry. |
|
||||
| Flat dictionary with `method` or `quant_method` | Normalize the method and construct its `QuantizationConfig`. |
|
||||
| Per-component dictionary | Construct a `ComponentQuantizationConfig`. |
|
||||
| Existing `QuantizationConfig` | Pass through without rebuilding it. |
|
||||
|
||||
Method aliases are normalized case-insensitively, with `-` and `_` treated as
|
||||
equivalent. `auto-round` and `auto_round` use the Omni INC/AutoRound adapter.
|
||||
ModelOpt checkpoint metadata is detected from its method, producer metadata, or
|
||||
`quant_algo`, then mapped to the corresponding vLLM ModelOpt configuration.
|
||||
|
||||
The factory uses Omni-specific overrides for methods whose diffusion behavior
|
||||
or platform dispatch is not provided by the vLLM registry. The current override
|
||||
set includes INT8, BitsAndBytes, MXFP8, MXFP4, MXFP4 dual-scale, and INC/
|
||||
AutoRound. Other methods, such as FP8, GGUF, and ModelOpt, are resolved through
|
||||
vLLM's registry when their configuration is compatible.
|
||||
|
||||
## Per-component routing
|
||||
|
||||
Multi-stage models do not necessarily want one quantization policy for every
|
||||
module. A component map expresses that policy without requiring each pipeline
|
||||
to construct separate vLLM configurations:
|
||||
|
||||
```python
|
||||
quantization_config = {
|
||||
"transformer": {"method": "fp8"},
|
||||
"vae": None,
|
||||
"default": None,
|
||||
}
|
||||
```
|
||||
|
||||
`ComponentQuantizationConfig` applies the following rules:
|
||||
|
||||
1. Component keys are interpreted as runtime layer-prefixes.
|
||||
2. The longest matching prefix wins.
|
||||
3. `None` disables quantization for the matching component.
|
||||
4. `default`, when present, handles prefixes with no explicit match.
|
||||
5. A matching child config delegates to its normal
|
||||
`get_quant_method(layer, prefix)` implementation.
|
||||
|
||||
Prefix matching happens after the model's weight-name mapping. New model
|
||||
integrations must verify the actual runtime prefixes, especially when a
|
||||
`WeightsMapper` changes checkpoint names. A prefix mismatch can silently route
|
||||
a layer to the default configuration, so component-routing tests should cover
|
||||
both positive matches and fall-through behavior.
|
||||
|
||||
## Online and pre-quantized checkpoints
|
||||
|
||||
The same method name can represent two different loading paths:
|
||||
|
||||
- **Online**: the source checkpoint contains BF16/FP16 weights; the method
|
||||
creates quantized weights or scales while the model is loaded.
|
||||
- **Pre-quantized**: the checkpoint contains quantized weights and the metadata
|
||||
required by the method; the loader consumes those serialized tensors.
|
||||
|
||||
Method configuration owns the mode flag, for example
|
||||
`is_checkpoint_int8_serialized` or `is_checkpoint_mxfp8_serialized`. The
|
||||
configuration's `get_quant_method()` selects the corresponding linear method.
|
||||
|
||||
For pipelines with transformer-specific `config.json` files, such as cascade
|
||||
models, `resolve_quant_config_from_disk()` reconciles the active configuration
|
||||
with each transformer's checkpoint metadata. Its invariants are:
|
||||
|
||||
- If no active configuration exists, valid checkpoint metadata can be used for
|
||||
auto-detection.
|
||||
- A method mismatch is an error rather than an implicit conversion.
|
||||
- Serialized-checkpoint flags rebuild an online configuration into the
|
||||
matching offline configuration.
|
||||
- Checkpoint-specific `ignored_layers` are preserved by rebuilding the active
|
||||
configuration when they differ.
|
||||
- AutoRound MXFP8 metadata (`data_type="mx_fp"`) selects its offline path.
|
||||
|
||||
This prevents a model from interpreting serialized weights with an online
|
||||
loader or silently applying the wrong method to one transformer in a
|
||||
multi-transformer pipeline.
|
||||
|
||||
## Model and layer integration
|
||||
|
||||
Diffusion model constructors receive the resolved configuration and pass it to
|
||||
quantizable vLLM layers. Each method configuration is responsible for:
|
||||
|
||||
- identifying supported layers in `get_quant_method()`;
|
||||
- creating the correct parameter shapes and loaders;
|
||||
- processing weights after loading when conversion is required; and
|
||||
- applying the platform-specific quantized operation at runtime.
|
||||
|
||||
Model code must keep precision-sensitive or unsupported modules unquantized.
|
||||
The shared `safe_quant_config()` helper is used by model integrations for
|
||||
normalization and modulation layers where a generic FP8 configuration would be
|
||||
unsafe. Text encoders, audio/vision encoders, VAEs, schedulers, and tokenizers
|
||||
are not implicitly quantized just because the main transformer is quantized;
|
||||
they require explicit component routing and method support.
|
||||
|
||||
Weight quantization and quantized KV cache are separate paths. The former is
|
||||
resolved through `quantization_config`; runtime attention/KV-cache quantization
|
||||
is configured by the diffusion attention configuration and must be validated
|
||||
independently.
|
||||
|
||||
## Platform and parallelism boundaries
|
||||
|
||||
Platform checks belong in the method implementation rather than in the common
|
||||
factory. This keeps configuration parsing portable while allowing CUDA, NPU,
|
||||
and XPU methods to select different kernels or reject unsupported execution at
|
||||
model construction time. The user guide is the source of truth for the current
|
||||
hardware support matrix.
|
||||
|
||||
Quantization methods must also respect the weight layout produced by the
|
||||
configured parallelism strategy:
|
||||
|
||||
- Tensor-parallel layers may receive partitioned weight shapes and loaders.
|
||||
- Sequence-parallel execution changes activation movement but does not by
|
||||
itself change the quantization configuration.
|
||||
- Data-parallel replicas build their own method state; quantization metadata
|
||||
must be available consistently on every rank.
|
||||
- `WeightsMapper` transformations must be applied before matching ignored-layer
|
||||
lists or component prefixes.
|
||||
|
||||
Distributed layerwise offload has an additional boundary: its sharded mmap
|
||||
AllGather path is not compatible with online quantization because the path
|
||||
expects already materialized checkpoint weights. Use the rank-local
|
||||
`--dlo-no-use-allgather` path or a compatible pre-quantized checkpoint when
|
||||
combining these features. See the [DLO design document](distributed_layerwise_offload.md)
|
||||
for the full compatibility matrix.
|
||||
|
||||
## Adding a quantization backend
|
||||
|
||||
Contributors should follow this sequence:
|
||||
|
||||
1. Check whether an existing vLLM backend can provide the required weight
|
||||
format and kernel behavior. Prefer reuse over a duplicate Omni backend.
|
||||
2. If Omni-specific behavior is required, implement a `QuantizationConfig` and
|
||||
its `QuantizeMethodBase`/linear method in `vllm_omni/quantization/`.
|
||||
3. Implement `get_name()`, supported activation dtypes, minimum capability,
|
||||
checkpoint parsing, ignored-layer handling, and `get_quant_method()`.
|
||||
4. Add a lazy factory override in `factory.py`. Keep optional platform
|
||||
dependencies out of package import paths and normalize aliases to one
|
||||
canonical method name.
|
||||
5. If the method is pre-quantized, define the checkpoint metadata and the
|
||||
serialized loading path before adding an online path.
|
||||
6. Add model-specific component routing only when the model contains stages
|
||||
that must use different policies.
|
||||
7. Document user-visible configuration in the quantization user guide and keep
|
||||
implementation rationale here.
|
||||
|
||||
The public factory registration hook,
|
||||
`register_quantization_override(method, builder)`, is intended for integrations
|
||||
that need to register an Omni-compatible builder without changing the vLLM
|
||||
registry.
|
||||
|
||||
## Validation
|
||||
|
||||
Validation should cover the complete path, not only whether a config object can
|
||||
be instantiated:
|
||||
|
||||
| Area | Required checks |
|
||||
| --- | --- |
|
||||
| Factory | Strings, aliases, flat dictionaries, invalid inputs, passthrough, and component maps |
|
||||
| Method config | Online/offline selection, ignored layers, platform guards, and weight loading |
|
||||
| Checkpoint metadata | Auto-detection, method mismatch errors, serialized flags, and per-transformer metadata |
|
||||
| Model integration | Correct quantization scope, unquantized components, and parallel weight layouts |
|
||||
| Numerical quality | Same-seed BF16 versus quantized output comparison at representative resolutions and step counts |
|
||||
| Performance | Peak device memory, generation latency, and any backend-specific throughput metric |
|
||||
|
||||
The unit tests live under `tests/diffusion/quantization/`. The trajectory
|
||||
similarity tool in
|
||||
`vllm_omni/quantization/tools/compare_diffusion_trajectory_similarity.py`
|
||||
provides a repeatable quality and memory comparison for diffusion models.
|
||||
Method support and feature combinations remain model- and hardware-dependent;
|
||||
passing factory tests alone does not establish production support.
|
||||
|
||||
## Design invariants
|
||||
|
||||
- `quantization_config` is the single runtime source of truth after config
|
||||
normalization.
|
||||
- A checkpoint method mismatch fails loudly instead of risking weight
|
||||
corruption.
|
||||
- Components without an explicit supported quantization path remain in their
|
||||
original precision.
|
||||
- Optional backend dependencies are imported lazily.
|
||||
- Method implementations own platform-specific capability checks and kernels.
|
||||
- User-facing support claims require model, hardware, numerical, and memory
|
||||
validation in addition to configuration-level tests.
|
||||
@@ -1,59 +0,0 @@
|
||||
# Distributed utils
|
||||
|
||||
This directory (vllm_omni/distributed/ray_utils) contains utilities for distributed execution in vllm-omni, supporting both **Ray** and **Multiprocessing** backends.
|
||||
## 1. Installation
|
||||
```bash
|
||||
pip install "ray[default]"
|
||||
```
|
||||
## 2. Ray Utils
|
||||
|
||||
The `ray_utils` module provides helper functions for managing Ray clusters and actors, which is essential for:
|
||||
* **Multi-node deployment**: Running pipeline stages across different physical machines.
|
||||
* **Resource management**: Efficient GPU/CPU allocation.
|
||||
|
||||
### 2.1 Basic Usage
|
||||
|
||||
To use the Ray backend, specify `worker_backend="ray"` when initializing the engine.
|
||||
|
||||
**Command Line Example:**
|
||||
```bash
|
||||
vllm serve Qwen/Qwen2.5-Omni-7B \
|
||||
--omni \
|
||||
--port 8091 \
|
||||
--worker-backend ray \
|
||||
--ray-address auto
|
||||
```
|
||||
|
||||
### 2.2 Cluster Setup
|
||||
|
||||
**Step 1: Start Head Node**
|
||||
Run this on your primary machine:
|
||||
```bash
|
||||
ray start --head --port=6399
|
||||
```
|
||||
|
||||
**Step 2: Connect Worker Nodes**
|
||||
Run this on each worker machine:
|
||||
```bash
|
||||
ray start --address=<HEAD_NODE_IP>:6399
|
||||
```
|
||||
|
||||
> **Tip**: For a complete cluster setup script, refer to the vLLM example:
|
||||
> [run_cluster.sh](https://github.com/vllm-project/vllm/blob/main/examples/online_serving/run_cluster.sh)
|
||||
|
||||
### 2.3 Distributed Connector Support
|
||||
|
||||
When running on Ray, the system automatically adapts its communication strategy:
|
||||
|
||||
* **Cross-Node**: Recommended to use `MooncakeTransferEngineConnector` (RDMA, fastest) or `MooncakeStoreConnector` (TCP fallback).
|
||||
* **Same-Node**: Can still use `SharedMemoryConnector` for efficiency, or Ray's native object store (plasma).
|
||||
* **SHM payload routing**: `SharedMemoryConnector` always stores payloads in `/dev/shm`.
|
||||
|
||||
### 2.4 Internal Helpers
|
||||
|
||||
* **`initialize_ray_cluster`**: Connects to an existing Ray cluster or starts a local one.
|
||||
|
||||
## 3. Troubleshooting
|
||||
|
||||
* **Connection Issues**: Ensure the Ray head node is accessible and ports (default 6399 in this example) are open.
|
||||
* **Version Mismatch**: Ensure all nodes run the same version of Ray and Python.
|
||||
+50
-3
@@ -1,6 +1,8 @@
|
||||
# Design Documents
|
||||
|
||||
This section contains design documents and architecture specifications for vLLM-Omni.
|
||||
This section contains design documents and architecture specifications for
|
||||
vLLM-Omni. The sidebar groups documents by the system concern they describe,
|
||||
while the source paths remain stable so existing links continue to work.
|
||||
|
||||
## Architecture Documents
|
||||
|
||||
@@ -8,18 +10,63 @@ This section contains design documents and architecture specifications for vLLM-
|
||||
|
||||
## Feature Design Documents
|
||||
|
||||
### Runtime and stage execution
|
||||
|
||||
- [Disaggregated Inference](feature/disaggregated_inference.md)
|
||||
- [Ray-based Execution](feature/ray_based_execution.md)
|
||||
- [Async Chunk](feature/async_chunk.md)
|
||||
- [Async Diffusion Output](feature/async_diffusion_output.md)
|
||||
- [Async Omni Output Materialization](feature/omni_async_output_materialization.md)
|
||||
- [Automatic Prefix Caching in Omni Models](feature/prefix_caching.md)
|
||||
|
||||
### Communication
|
||||
|
||||
#### OmniConnector implementations
|
||||
|
||||
- [Mooncake Store Connector](feature/omni_connectors/mooncake_store_connector.md)
|
||||
- [Mooncake Transfer Engine Connector](feature/omni_connectors/mooncake_transfer_engine_connector.md)
|
||||
- [Mori Transfer Engine Connector](feature/omni_connectors/mori_transfer_engine_connector.md)
|
||||
- [Shared Memory Connector](feature/omni_connectors/shared_memory_connector.md)
|
||||
- [Yuanrong Store Connector](feature/omni_connectors/yuanrong_connector.md)
|
||||
- [Yuanrong Transfer Engine Connector](feature/omni_connectors/yuanrong_transfer_engine_connector.md)
|
||||
|
||||
### Diffusion acceleration
|
||||
|
||||
#### Parallelism
|
||||
|
||||
- [CFG-Parallel](feature/cfg_parallel.md)
|
||||
- [Expert Parallel](feature/expert_parallel.md)
|
||||
- [Hybrid Sharded Data Parallel (HSDP)](feature/hsdp.md)
|
||||
- [Pipeline Parallel](feature/pipeline_parallel.md)
|
||||
- [Sequence Parallel](feature/sequence_parallel.md)
|
||||
- [Tensor Parallel](feature/tensor_parallel.md)
|
||||
- [VAE Patch Parallelism](feature/vae_parallel.md)
|
||||
|
||||
#### Attention Backends
|
||||
|
||||
- [Skip-Softmax](feature/skip_softmax.md)
|
||||
|
||||
#### Quantization
|
||||
|
||||
- [Quantization](feature/quantization.md)
|
||||
|
||||
- [Cache-DiT](feature/cache_dit.md)
|
||||
- [TeaCache](feature/teacache.md)
|
||||
- [Diffusion Continuous Batching](feature/diffusion_continuous_batching.md)
|
||||
- [Distributed Layerwise Offload](feature/distributed_layerwise_offload.md)
|
||||
|
||||
## Infrastructure Design Documents
|
||||
## Infrastructure and Performance
|
||||
|
||||
- [Prometheus Metrics](metrics.md)
|
||||
- [Speech Generation Performance Optimizations](qwen3_omni_tts_performance_optimization.md)
|
||||
|
||||
## Module Design Documents
|
||||
|
||||
### Runtime Modules
|
||||
|
||||
- [AR Module](module/ar_module.md)
|
||||
- [DIT Module](module/dit_module.md)
|
||||
|
||||
### Orchestration
|
||||
|
||||
- [Entrypoint Module](module/entrypoint_module.md)
|
||||
- [AsyncOmni Architecture (Qwen3-Omni Example)](module/async_omni_architecture.md)
|
||||
File renamed without changes.
@@ -59,7 +59,7 @@ Memory optimization methods help reduce GPU memory usage, enabling inference on
|
||||
|
||||
| Method | Description | Best For |
|
||||
|--------|-------------|----------|
|
||||
| **[CPU Offload](diffusion/cpu_offload_diffusion.md)** | Offloads model components to CPU memory | Limited VRAM, large models on consumer GPUs |
|
||||
| **[CPU Offload](diffusion/cpu_offload.md)** | Offloads model components to CPU memory | Limited VRAM, large models on consumer GPUs |
|
||||
| **[Quantization](quantization/overview.md)** | Reduces transformer stages from BF16 to FP8/INT8/etc. | Limited VRAM, minimal accuracy loss |
|
||||
| **[VAE Parallelism](diffusion/parallelism/vae_parallelism.md)** | Distributes VAE decode work across GPUs | High-resolution generation with reduced VAE memory peak |
|
||||
|
||||
@@ -104,8 +104,8 @@ The following tables show which models support each feature:
|
||||
|
||||
> Notes:
|
||||
|
||||
> 1. CPU Offload has two methods: Module-wise (default for models with DiT + text encoder) and Layerwise. The tables below show **Layerwise support** only. Split models like Cosmos3 (no separate text encoder) swap their reasoner/generator components for module-wise offload; see the [CPU Offload Guide](diffusion/cpu_offload_diffusion.md).
|
||||
> 2. The **💾Quantization** column is collapsed for readability. See [Quantization Overview](quantization/overview.md) for per-method and per-model support details.
|
||||
> 1. CPU Offload has two methods: Module-wise (default for models with DiT + text encoder) and Layerwise. The tables below show **Layerwise support** only. Split models like Cosmos3 (no separate text encoder) swap their reasoner/generator components for module-wise offload; see the [CPU Offload Guide](diffusion/cpu_offload.md).
|
||||
> 2. The **💾Quantization** column is collapsed for readability. See [Quantization](quantization/overview.md) for per-method and per-model support details.
|
||||
|
||||
### ImageGen
|
||||
|
||||
@@ -274,9 +274,9 @@ Measured on NVIDIA H800:
|
||||
|
||||
**Memory Optimization:**
|
||||
|
||||
- **[CPU Offload Guide](diffusion/cpu_offload_diffusion.md)** - Offload model components to CPU, reduce GPU memory usage
|
||||
- **[CPU Offload Guide](diffusion/cpu_offload.md)** - Offload model components to CPU, reduce GPU memory usage
|
||||
- **[VAE Parallelism Guide](diffusion/parallelism/vae_parallelism.md)** - Distribute VAE decode work across GPUs for high-resolution images and videos
|
||||
- **[Quantization Overview](quantization/overview.md)** - Overview of quantization methods for diffusion, multi-stage omni/TTS, and multi-stage diffusion models
|
||||
- **[Quantization](quantization/overview.md)** - Quantization methods for diffusion, multi-stage omni/TTS, and multi-stage diffusion models
|
||||
|
||||
**Extensions:**
|
||||
|
||||
|
||||
@@ -5,6 +5,9 @@ path. The same configuration entrypoint is used across diffusion-only models,
|
||||
multi-stage omni/TTS models, and multi-stage diffusion models, but each model
|
||||
type has a different quantization scope.
|
||||
|
||||
For the internal architecture and backend extension points, see the
|
||||
[quantization design document](../../design/feature/quantization.md).
|
||||
|
||||
## Quantization Modes
|
||||
|
||||
| Mode | Guide | Description | Methods |
|
||||
|
||||
@@ -75,6 +75,8 @@ plugins:
|
||||
redirect_maps:
|
||||
user_guide/diffusion/request_batching.md: user_guide/diffusion/execution_modes.md
|
||||
user_guide/diffusion/step_execution.md: user_guide/diffusion/execution_modes.md
|
||||
user_guide/diffusion/cpu_offload_diffusion.md: user_guide/diffusion/cpu_offload.md
|
||||
user_guide/quantization/index.md: user_guide/quantization/overview.md
|
||||
design/feature/diffusion_request_level_batching.md: design/feature/diffusion_continuous_batching.md
|
||||
design/feature/diffusion_step_execution.md: design/feature/diffusion_continuous_batching.md
|
||||
- glightbox
|
||||
|
||||
Reference in new issue
Block a user