From 303edff9302a4121409ee9ded29a886baab18b66 Mon Sep 17 00:00:00 2001 From: Wang fuyin Date: Tue, 22 Sep 2026 10:53:37 +0800 Subject: [PATCH] [Doc][skip ci] Add design doc for scheduler-managed diffusion paged KV cache (#6891) Signed-off-by: Acerak01-fy --- docs/.nav.yml | 3 + .../feature/diffusion_paged_kv_cache.md | 346 ++++++++++++++++++ docs/design/index.md | 4 + docs/user_guide/diffusion/execution_modes.md | 7 +- docs/user_guide/diffusion/paged_kv_cache.md | 172 +++++++++ docs/user_guide/diffusion_features.md | 1 + recipes/Tencent/HunyuanImage-3.0-Instruct.md | 50 +++ 7 files changed, 582 insertions(+), 1 deletion(-) create mode 100644 docs/design/feature/diffusion_paged_kv_cache.md create mode 100644 docs/user_guide/diffusion/paged_kv_cache.md diff --git a/docs/.nav.yml b/docs/.nav.yml index da55af813..ec428d76b 100644 --- a/docs/.nav.yml +++ b/docs/.nav.yml @@ -80,6 +80,8 @@ nav: - Cache Acceleration: - TeaCache: user_guide/diffusion/cache_acceleration/teacache.md - Cache-DiT: user_guide/diffusion/cache_acceleration/cache_dit.md + - KV Cache: + - Scheduler-Managed Paged KV Cache: user_guide/diffusion/paged_kv_cache.md - Parallelism: - Overview: user_guide/diffusion/parallelism/overview.md - CFG Parallel: user_guide/diffusion/parallelism/cfg_parallel.md @@ -153,6 +155,7 @@ nav: - Attention Optimization: - Backend Selection: design/feature/attention_backend_selection.md - Skip-Softmax: design/feature/skip_softmax.md + - Scheduler-Managed Paged KV Cache: design/feature/diffusion_paged_kv_cache.md - CPU Offloading: - Overview: design/feature/offloader/README.md - Model-Level: design/feature/offloader/module_offload.md diff --git a/docs/design/feature/diffusion_paged_kv_cache.md b/docs/design/feature/diffusion_paged_kv_cache.md new file mode 100644 index 000000000..b05709194 --- /dev/null +++ b/docs/design/feature/diffusion_paged_kv_cache.md @@ -0,0 +1,346 @@ +# Scheduler-Managed Paged KV Cache + +This document describes the design and lifecycle of Scheduler-managed paged KV cache for diffusion DiT stages. + +Paged KV cache is a scheduler-managed KV-cache and memory-management +mechanism. It is separate from attention backend selection: native attention +backends such as FlashAttention or Ascend FIA perform the attention computation, +while paged KV provides the block layout and metadata consumed by those +backends. + +The current implementation requires the `FLASH_ATTN` selector; other +attention backends do not currently support `paged_scheduler`. + +For operator-facing configuration and examples, see the +[Paged KV Cache user guide](../../user_guide/diffusion/paged_kv_cache.md). + +## Table of Contents + +1. [Overview](#overview) +2. [Motivation](#motivation) +3. [Architecture](#architecture) +4. [Execution Modes](#execution-modes) +5. [Design](#design) +6. [Model and Platform Integration](#model-and-platform-integration) +7. [Configuration and Compatibility](#configuration-and-compatibility) +8. [Limitations](#limitations) +9. [Related Files](#related-files) + +## Overview + +Scheduler-managed paged KV provides a page-based memory and data-movement +foundation for diffusion DiT stages. It makes KV capacity explicit, preserves +stable prefixes across denoising steps, and defines page-granular destinations +for future AR-to-DiT transfer. HunyuanImage-3.0 is the current request-mode +reference on CUDA and Ascend NPU; the supported boundaries and follow-up work +are described below. + +## Motivation + +Paged KV is introduced for three related goals: + +1. **Memory management:** make diffusion KV capacity visible to the Scheduler + so logical requests can reserve and release fixed-size blocks while the + Worker owns one reusable physical pool. This improves capacity planning and + memory utilization, and can increase request concurrency and throughput when + the dense path would otherwise reserve separate full tensors. +2. **Transfer optimization:** give a DiT stage stable destination page slots so + a future AR-to-DiT connector can transfer missing KV blocks directly, + instead of gathering a complete tensor into a temporary receiver buffer. +3. **Prefix reuse:** keep the stable prompt/reference prefix resident while + only the timestep/image target is rewritten at each denoising step. The same + page-level contract also provides the foundation for canonical prefix-cache + reuse when a model supplies cacheable block identities. + +Request-local memory management and prefix reuse are implemented by the local +request path. Cross-stage transfer and cross-request canonical prefix caching +use the same page interfaces but require model/connector integration that is +outside the current Hunyuan implementation. + +The legacy dense path keeps complete model-owned tensors for each request. The +paged path instead maps requests to Scheduler blocks and reuses a Worker page +pool, making capacity and fragmentation explicit. This can improve HBM +utilization and concurrency, although it does not guarantee lower latency for +every shape. + +## Architecture + +The control plane describes and reserves logical sequences. The data plane +installs the resulting snapshot on each Worker and runs the native attention +backend. + +```mermaid +flowchart LR + subgraph CP[Control plane] + R[Diffusion request] --> P[Model preprocessing
layout, CFG rows, spans] + P --> S[Diffusion Scheduler
logical sequences and blocks] + S --> M[DiffusionKVMetadata] + end + M -->|immutable snapshot| X[Executor / RPC] + subgraph DP[Data plane: each Worker rank] + W[Model runner
physical pages and block tables] + W --> A[Paged attention adapter] + A --> C[CUDA FlashAttention / FA3] + A --> N[Ascend FIA] + W --> O[Denoising output] + end + X --> W +``` + +### Ownership boundary + +| Component | Owns | Does not own | +| --- | --- | --- | +| Model preprocessing | Lengths, positions, CFG count, and attention spans | Physical blocks or Worker state | +| Diffusion Scheduler | Request lifecycle, logical sequences, admission, and release | Device tensors or native kernel metadata | +| Executor/RPC | Transport of the immutable allocation snapshot | Allocation decisions or page contents | +| Worker/model runner | Rank-local KV tensors, rows, block tables, slots, and active metadata | Scheduler block lifetime | +| Paged attention adapter | Layout translation, slot metadata, and backend dispatch | Capacity management or request admission | +| Platform backend | Native page geometry, metadata construction, and kernel selection | Model-specific token semantics | + +`DiffusionKVCacheManager` is the request-facing Scheduler facade: it groups and +atomically allocates the internal sequences for one request, then publishes +`DiffusionKVMetadata` over the Worker-owned physical pool. + +### Request lifecycle + +1. **Admission:** preprocessing creates one logical sequence per CFG branch; + the Scheduler reserves all required blocks atomically. +2. **Activation:** the Worker validates the metadata generation, binds rows, + stages block tables, and installs native attention metadata. +3. **Execution:** the first denoising step writes the complete sequence once; + later steps write only the changing target span and read retained pages. +4. **Release:** completion, cancellation, errors, and reinitialization release + Scheduler blocks and clear Worker rows. + +Dense requests do not install paged metadata. Paged requests reject legacy +dense KV payloads or stale/incomplete native metadata instead of silently +falling back to dense execution. + +### AR-to-DiT transfer contract + +When an AR stage and a DiT stage are deployed separately, the previous +model-owned transfer commonly gathers and flattens a complete KV tensor into a +temporary receiver buffer before the DiT model can use it. + +The Scheduler-managed contract gives the DiT Worker a destination page layout +before transfer. The current feature implements this DiT-side destination +contract (reservation, layout, and generation/readiness metadata); it does not +execute connector transfers or import AR KV. A future connector can target the +reserved slots directly: + +```mermaid +flowchart LR + A[AR Worker
source pages] --> X[Connector
missing blocks] + S[DiT Scheduler
reserve destination blocks] --> D[DiT Worker
destination pages] + X --> D + D --> F[Native DiT attention] +``` + +The diagram is a target connector boundary, not an enabled runtime path in this +feature. The expected benefits of a future connector using this contract are: + +| Concern | Previous model-owned transfer | Page-native destination contract | +| --- | --- | --- | +| Transfer unit | Packed or complete KV tensor | Individual missing page blocks | +| Receiver storage | Temporary contiguous tensor, then model cache | Scheduler-reserved Worker pages | +| Data movement | Gather/flatten plus another copy or concatenation | Direct write to destination slots when a connector is enabled | +| Readiness | Implicit runner-side synchronization | Explicit generation and ready metadata | +| Denoising reuse | Reassemble the stable prefix in the model | Native attention reads resident pages | +| Cleanup | Transfer and model-cache lifetimes are coupled | Source lease, destination reservation, and request release are explicit | + +Reserving destination pages up front supplies stable physical targets and an +explicit generation/ready state, so a future connector can move only missing +blocks and pass the same metadata to native attention. Imported AR KV is not +enabled for `paged_scheduler` in this PR; DreamZero and LingBot-World use the +separate `ar_diffusion_kv` contract. + +## Execution Modes + +Paged KV is a cache feature; request batching and step execution are separate +engine modes. The terminology follows the +[diffusion continuous batching design](diffusion_continuous_batching.md): + +| Execution path | Configuration | Scheduler batch unit | Hunyuan `paged_scheduler` status | +| --- | --- | --- | --- | +| Request execution | `step_execution=false`, `max_num_seqs=1` | One complete denoising request | Current supported path | +| Request-level batching | `step_execution=false`, `max_num_seqs>1` | Multiple compatible public requests in one forward | Not supported by the current `paged_scheduler` integration | +| Step execution | `step_execution=true`, `max_num_seqs=1` | One request advances one denoising step per scheduler tick | Paged path rejects it | +| Step continuous batching | `step_execution=true`, `max_num_seqs>1` | Compatible active step states advance together | Separate feature; paged Hunyuan is not implemented | + +`max_num_seqs` limits public requests. CFG branches are internal rows of one +request and are controlled by `diffusion_kv_max_rows_per_request`: a request +without CFG uses one row, while the standard positive/negative CFG request uses +two. Atomic CFG admission does not mean that CFG is required. + +## Design + +### Startup and cache sizing + +Before real requests are admitted, the engine and Workers: + +1. Resolve `diffusion_kv_mode` and prepare a maximum-shape profile request. +2. Register paged attention layers and collect a native `KVCacheSpec` per KV + group. +3. Run the marked memory profile to determine non-KV memory. This probe is not + a latency sample. On NPU, it may use SDPA only when MindIE-SD is + unavailable; this is the sole intentional fallback and applies only to this + startup profile. +4. Build the native cache configuration and resolve block geometry. +5. Allocate rank-local physical pages and create the Scheduler facade over the + same geometry. + +`kv_cache_memory_bytes` is a physical KV-pool budget per Worker rank, not a +request token count. When omitted, the normal utilization path sizes the pool +after subtracting profiled non-KV memory. Reserved pool memory can exceed the +live payload because it also reflects capacity and fragmentation. + +### Logical layout + +Each sequence has a stable prefix and a changing target: + +```text +|-------------------------- allocated seq_len --------------------------| +|---------------- prefix_len ----------------|---- target_len ----|unused| + retained across steps rewritten each step +``` + +The runner exposes these boundaries to the adapter: + +| Phase | `query_len` | `seq_len` | `kv_start_pos` | +| --- | ---: | ---: | ---: | +| First denoising/prefill | `seq_len` | `seq_len` | `0` | +| Later denoising steps | `target_len` | `prefix_len + target_len` | `prefix_len` | + +For `block_size`, token position `p` maps to the native slot +`block_id(p // block_size) * block_size + p % block_size`. The Scheduler owns +the block IDs; the Worker turns them into rank-local tables and slots. + +### Dense and paged representation + +Both modes implement Hunyuan's mixed causal/full attention. They differ in KV +representation and backend input, not in the attention result they describe: + +| Aspect | `dense_legacy` | `paged_scheduler` | +| --- | --- | --- | +| KV owner | Model-owned contiguous cache | Worker-owned physical pages | +| Attention input | Contiguous K/V with a dense mask and/or span metadata | Block tables, slots, lengths, and `full_attn_spans` | +| Prefix reuse | Model-owned prefix tensor; cache scope depends on the dense path | Stable prefix pages are retained across denoising steps; canonical cross-request reuse requires cache identities | +| Scheduler state | No diffusion page reservation | Logical sequences, admission, generation, and release | + +The current implementation guarantees prefix reuse between the first and later +denoising steps. Native vLLM cross-request prefix caching is not enabled by +this Hunyuan integration: the diffusion KV manager uses +`enable_caching=False`, and Hunyuan does not publish canonical block hashes. + +### Attention execution + +The two paths preserve the same mixed causal/full spans. In the paged path, +the adapter converts each row into aligned native segments: + +| Segment | Query input | KV visibility | Causal flag | +| --- | --- | --- | --- | +| Causal `[s, e)` | `Q[s:e]` | `K[:e]`, `V[:e]` | `true` | +| Full `[a, b)` | Query overlap | `K[:b]`, `V[:b]` | `false` | + +The K/V update is deliberately outside the segment loop: + +```text +Q/K/V projection + -> write the current K/V span once per layer + -> run all causal/full native segments + -> restore output order +``` + +CUDA keeps cache-update ownership in its native paged-attention contract. +Ascend prewrites the normal-layout K/V span once, then FIA segment calls read +the persistent pages without receiving K/V again. Output is restored to the +original token order before projection, residual, and MLP layers consume it. + +Dense does not imply one universal kernel call. Hunyuan's CUDA dense path can +also use the shared piecewise FlashAttention helper for aligned regions. In +the paged path, the same regions are described as native segments that read +persistent pages. The two modes can therefore have different kernel names and +call counts while preserving the same mask semantics. + +### Piecewise planning + +The common planner supports both row layouts: + +| Row layout | Preparation and output handling | +| --- | --- | +| Homogeneous ranges, such as CFG rows from one request | Keep the batched layout, use contiguous views/direct output-buffer writes, and avoid the large indexed output scatter | +| Heterogeneous lengths or offsets | Gather valid tokens with `index_select`, run native segments, and scatter them back with `index_copy_` | + +The homogeneous fast path applies to rows in one attention invocation; it does +not enable arbitrary public-request batching or change Scheduler allocation. + +## Model and Platform Integration + +### HunyuanImage-3.0 + +Hunyuan is the reference integration because its self-attention mixes causal +and full regions and its generated-image region changes across denoising +steps. It creates one logical row per conditional or unconditional CFG branch +and supports strict Ulysses SP in request mode. Its runner owns row creation, +phase activation, and page metadata; the model boundary supplies token layout +and attention spans without allocating Scheduler blocks. + +### Platform matrix + +| Model/integration | Platform | Execution | Backend | Status | +| --- | --- | --- | --- | --- | +| HunyuanImage-3.0 | NVIDIA CUDA | Request-level | `FLASH_ATTN` -> native FlashAttention/FA3 | Implemented and validated | +| HunyuanImage-3.0 | Ascend NPU | Request-level | `FLASH_ATTN` -> Ascend FIA | Implemented and validated | +| DreamZero / LingBot-World | Platform-specific | Separate `ar_diffusion_kv` contract | Imported AR-KV path | Not `paged_scheduler` | +| Other diffusion models | Any | N/A | N/A | No paged integration yet | + +The logical `FLASH_ATTN` selector is the only current paged-KV capability +advertised by `FlashAttentionBackend`. Other diffusion backends do not +transparently fall back to dense attention for a paged request. + +## Configuration and Compatibility + +`paged_scheduler` requires a native cache configuration, a positive row limit, +and a prepared memory-profile request. The main fields are: + +| Field | Meaning | +| --- | --- | +| `diffusion_kv_mode` | `dense_legacy` (default) or `paged_scheduler` | +| `diffusion_kv_max_rows_per_request` | Worker row capacity for one public request, including CFG branches | +| `kv_cache_memory_bytes` | Optional explicit physical KV-pool budget per Worker rank | +| `gpu_memory_utilization` | Automatic pool sizing when no byte budget is supplied | +| `diffusion_attention_backend` | Must resolve to `FLASH_ATTN` for the current paged implementation | + +The user guide contains the stage YAML and `vllm serve --omni` example. +Use request execution for the current Hunyuan integration; the unsupported +batching and step modes are summarized in [Execution Modes](#execution-modes). + +The current paged KV format is unquantized BF16 with identity Q/K/V scales. +Model weight quantization, CPU offload, layerwise offload, and distributed +layerwise offload (DLO) have not been validated with this path. Leave those +options disabled for a validated paged setup; use `dense_legacy` when they are +required. + +## Limitations + +- Hunyuan `paged_scheduler` is request-level only; request batching and step + execution are not implemented (see [Execution Modes](#execution-modes)). +- Only strict Ulysses SP and the current two-branch CFG layout are supported; + Ring, AllGather-KV, and independent Hunyuan KV contexts are not implemented. +- Imported AR-to-DiT page transfer and cross-request canonical prefix caching + are follow-up integrations; `ar_diffusion_kv` is a separate contract. +- Quantized KV and CPU/DLO offload combinations are untested. +- For a formal `paged_scheduler` request, missing native metadata or backend + support is an explicit error, not a dense or SDPA fallback. The marked + startup memory profile is the only intentional exception. + +## Related Files + +- Scheduler and allocation: [`diffusion_kv/manager.py`](gh-file:vllm_omni/diffusion/diffusion_kv/manager.py), [`diffusion_kv/request.py`](gh-file:vllm_omni/diffusion/diffusion_kv/request.py), [`diffusion_kv/metadata.py`](gh-file:vllm_omni/diffusion/diffusion_kv/metadata.py) +- Scheduler and runtime: [`sched/base_scheduler.py`](gh-file:vllm_omni/diffusion/sched/base_scheduler.py), [`diffusion_kv/config.py`](gh-file:vllm_omni/diffusion/diffusion_kv/config.py), [`forward_context.py`](gh-file:vllm_omni/diffusion/forward_context.py), [`vllm_config.py`](gh-file:vllm_omni/diffusion/vllm_config.py) +- Executor boundary: [`executor/abstract.py`](gh-file:vllm_omni/diffusion/executor/abstract.py), [`executor/uniproc_executor.py`](gh-file:vllm_omni/diffusion/executor/uniproc_executor.py) +- Worker data plane: [`diffusion_kv/initialization.py`](gh-file:vllm_omni/diffusion/diffusion_kv/initialization.py), [`diffusion_kv/model_runner_backend.py`](gh-file:vllm_omni/diffusion/diffusion_kv/model_runner_backend.py), [`worker/diffusion_model_runner.py`](gh-file:vllm_omni/diffusion/worker/diffusion_model_runner.py) +- Attention adapter and planner: [`diffusion_kv/paged_attention_adapter.py`](gh-file:vllm_omni/diffusion/diffusion_kv/paged_attention_adapter.py), [`attention/layer.py`](gh-file:vllm_omni/diffusion/attention/layer.py), [`attention/backends/flash_attn.py`](gh-file:vllm_omni/diffusion/attention/backends/flash_attn.py), [`attention/backends/utils/piecewise_attn.py`](gh-file:vllm_omni/diffusion/attention/backends/utils/piecewise_attn.py) +- Model boundary: [`models/hunyuan_image3/request_layout.py`](gh-file:vllm_omni/diffusion/models/hunyuan_image3/request_layout.py), [`models/hunyuan_image3/pipeline_hunyuan_image3.py`](gh-file:vllm_omni/diffusion/models/hunyuan_image3/pipeline_hunyuan_image3.py), [`models/hunyuan_image3/hunyuan_image3_transformer.py`](gh-file:vllm_omni/diffusion/models/hunyuan_image3/hunyuan_image3_transformer.py) +- Platform hooks: [`platforms/interface.py`](gh-file:vllm_omni/platforms/interface.py), [`platforms/cuda/platform.py`](gh-file:vllm_omni/platforms/cuda/platform.py), [`platforms/npu/platform.py`](gh-file:vllm_omni/platforms/npu/platform.py) diff --git a/docs/design/index.md b/docs/design/index.md index f70571cb5..99646dc23 100644 --- a/docs/design/index.md +++ b/docs/design/index.md @@ -57,6 +57,10 @@ implementation contract; it is not, by itself, a general support claim. - [Tensor Parallel](feature/tensor_parallel.md) - [VAE Patch Parallelism](feature/vae_parallel.md) +#### KV cache and memory management + +- [Scheduler-Managed Paged KV Cache for Diffusion DiT Stages](feature/diffusion_paged_kv_cache.md) + #### Attention optimization The [Diffusion Attention Backends](../user_guide/diffusion/attention_backends.md) diff --git a/docs/user_guide/diffusion/execution_modes.md b/docs/user_guide/diffusion/execution_modes.md index 5eee3c4d9..c9d79640f 100644 --- a/docs/user_guide/diffusion/execution_modes.md +++ b/docs/user_guide/diffusion/execution_modes.md @@ -108,7 +108,12 @@ set `DIFFUSION_ATTENTION_BACKEND=TORCH_SDPA` or configure `diffusion_attention_config.default.backend=TORCH_SDPA` before using `--max-num-seqs >1`. See the [HunyuanImage-3.0 recipe](https://github.com/vllm-project/vllm-omni/blob/main/recipes/Tencent/HunyuanImage-3.0-Instruct.md) -for its validated configuration. Helios supports single-request step execution only: use +for its validated configuration. For HunyuanImage3, this step-execution +support applies to the `dense_legacy` path; `paged_scheduler` currently +supports request-level execution only. See the +[Scheduler-Managed Paged KV Cache guide](paged_kv_cache.md) for its required +backend and configuration. Helios supports single-request step +execution only: use `--step-execution --max-num-seqs 1` for Helios. MiniMax H3 supports step-wise continuous batching by packing co-batched requests into one sequence that keeps a separate attention document per request; that layout needs a backend which diff --git a/docs/user_guide/diffusion/paged_kv_cache.md b/docs/user_guide/diffusion/paged_kv_cache.md new file mode 100644 index 000000000..777213e77 --- /dev/null +++ b/docs/user_guide/diffusion/paged_kv_cache.md @@ -0,0 +1,172 @@ +# Scheduler-Managed Paged KV Cache + +Scheduler-managed paged KV cache stores diffusion DiT key/value tensors in +Worker-owned pages. It is useful when a denoising request should keep its +stable prompt/reference prefix in device memory while rewriting only the +changing image span on later steps. The current HunyuanImage-3.0 integration +is request-level only; it does not silently change an unsupported request to +dense attention. + +Paging changes KV storage and scheduling ownership; it does not change the +model's causal/full attention semantics. + +## Choose a mode + +| Need | Mode | +| --- | --- | +| Broadest compatibility or step execution | `dense_legacy` (default) | +| HunyuanImage-3.0 request-mode serving with Scheduler-managed KV pages | `paged_scheduler` | + +Use `dense_legacy` when the model, attention backend, or offload combination +has not been validated for paged KV. HunyuanImage-3.0 paged KV currently accepts +request-level execution only; set `step_execution: false`. + +## Supported scope + +| Model | Platform | Execution | Backend | Status | +| --- | --- | --- | --- | --- | +| HunyuanImage-3.0 | NVIDIA CUDA | Request-level | `FLASH_ATTN` (native FlashAttention/FA3) | Implemented and validated | +| HunyuanImage-3.0 | Ascend NPU | Request-level | `FLASH_ATTN` (Ascend FIA) | Implemented and validated | +| Other diffusion models | Any | N/A | Any | Not integrated or tested | + +Only the logical `FLASH_ATTN` backend currently advertises paged-KV support. +`TORCH_SDPA`, `CUDNN_ATTN`, `FLASHINFER_ATTN`, SAGE, Hub, and other diffusion +backends do not transparently fall back to dense execution for a paged request. + +Strict Ulysses SP and two-branch CFG parallel are supported in HunyuanImage-3.0 +request mode when the selected device topology has been validated. Ring SP, +AllGather-KV SP, Hunyuan paged step execution, and independent public-request +batching are not supported by this path. DreamZero and LingBot-World use the +separate `ar_diffusion_kv` imported-AR-KV contract. + +## Configure + +The diffusion KV fields are stage settings. The two +`diffusion_kv_*` fields are diffusion-specific engine fields and are forwarded +through `engine_extras`; `kv_cache_memory_bytes` is a standard stage field and +is placed directly under the stage. + +Set `pipeline: hunyuan_image3_dit` to select the standalone DiT pipeline, +where stage `0` is the diffusion stage. + +```yaml +pipeline: hunyuan_image3_dit + +stages: + - stage_id: 0 + max_num_seqs: 1 + enforce_eager: true + diffusion_attention_backend: FLASH_ATTN + step_execution: false + kv_cache_memory_bytes: 536870912 # 512 MiB per Worker/rank + engine_extras: + diffusion_kv_mode: paged_scheduler + diffusion_kv_max_rows_per_request: 2 +``` + +| Field | Required | Meaning | +| --- | --- | --- | +| `diffusion_kv_mode` | Yes | `paged_scheduler` enables Scheduler-managed pages; `dense_legacy` is the default | +| `diffusion_kv_max_rows_per_request` | Yes for paged | Positive Worker row limit, including all internal CFG branches; use `1` without CFG and `2` for standard CFG | +| `kv_cache_memory_bytes` | Optional | Explicit physical KV-pool budget per Worker/rank; omit it to use automatic sizing | +| `gpu_memory_utilization` | Optional | Automatic pool-sizing fraction when no explicit byte budget is set | +| `max_num_seqs` | Recommended `1` | Public request capacity; it is separate from the internal CFG row limit | +| `step_execution` | Must be `false` | Hunyuan paged KV is request-level only | +| `diffusion_attention_backend` | `FLASH_ATTN` | Required logical backend for the current paged implementation | + +`kv_cache_memory_bytes` describes pool capacity, not live payload. Reserved +memory can therefore exceed the memory currently used by active requests. +The Scheduler allocates all rows belonging to one request together: a request +with CFG disabled has one row, while standard two-branch CFG has two rows. CFG +rows are internal branches, not independent public requests. + +Paged KV is currently unquantized BF16. `diffusion_kv_cache_dtype` does not +enable quantized paged KV, and model weight quantization has not been validated +in combination with this mode. CPU offload, layerwise offload, and distributed +layerwise offload (DLO) are currently untested with paged KV. Leave these +options disabled for paged KV; use `dense_legacy` when offload is required. + +## Hunyuan request-mode example + +Create a small deploy overlay (for example, +`hunyuan_image3_paged.yaml`) with the stage settings above, then start the +validated Hunyuan request-mode path: + +```bash +vllm serve tencent/HunyuanImage-3.0-Instruct \ + --omni \ + --trust-remote-code \ + --port 8091 \ + --deploy-config /path/to/hunyuan_image3_paged.yaml +``` + +The base Hunyuan DiT deploy file can be reused and overridden from the command +line when a separate overlay is inconvenient: + +```bash +vllm serve tencent/HunyuanImage-3.0-Instruct \ + --omni \ + --trust-remote-code \ + --deploy-config vllm_omni/deploy/hunyuan_image3_dit.yaml \ + --stage-overrides \ + '{"0":{"diffusion_kv_mode":"paged_scheduler","diffusion_kv_max_rows_per_request":2,"kv_cache_memory_bytes":536870912,"step_execution":false,"diffusion_attention_backend":"FLASH_ATTN"}}' +``` + +After the server is ready, send the normal Hunyuan image-generation request: + +```bash +curl -s http://localhost:8091/v1/chat/completions \ + -H "Content-Type: application/json" \ + -d '{ + "modalities": ["image"], + "messages": [{"role": "user", "content": "A red ceramic vase on a wooden table"}], + "extra_body": { + "height": 1024, + "width": 1024, + "num_inference_steps": 20, + "guidance_scale": 5.0, + "seed": 1234 + } + }' | jq -r '.choices[0].message.content[0].image_url.url' \ + | cut -d',' -f2- | base64 -d > hunyuan_image3_paged.png +``` + +The same configuration is shown in the +[HunyuanImage-3.0 recipe](https://github.com/vllm-project/vllm-omni/blob/main/recipes/Tencent/HunyuanImage-3.0-Instruct.md). + +Confirm from startup and worker logs that the resolved backend is +`FLASH_ATTN` and that the paged native path is active. A paged request must +fail explicitly if its Worker metadata or native backend is unavailable; it +must not silently switch to dense attention. + +## Parallelism + +Hunyuan request-mode paged KV can be combined with tensor parallelism, strict +Ulysses SP, or two-way CFG parallel when the selected topology has been +validated. Configure the topology in the stage's existing `parallel_config`: + +| Setting | User-visible effect | Paged-KV requirement | +| --- | --- | --- | +| `tensor_parallel_size` | Shards model weights across ranks | Device count must match the stage topology | +| `sequence_parallel_size` + strict Ulysses | Shards the sequence across ranks | Ring and AllGather-KV modes are not supported | +| `cfg_parallel_size: 2` | Places the positive and negative branches on separate ranks | Set `diffusion_kv_max_rows_per_request` to at least `2` | + +`max_num_seqs` controls public request concurrency and is independent of the +internal CFG row limit. Keep `max_num_seqs: 1` for the current Hunyuan paged +path; increasing it does not enable independent request batching. + +## Troubleshooting + +- `paged_scheduler requires diffusion_kv_max_rows_per_request`: add a positive + stage value and include all CFG rows. +- Unsupported backend error: select `FLASH_ATTN`; the other attention backends + currently do not implement the paged-KV capability. +- Step-mode rejection: remove `step_execution: true` and use request-level + execution, or select `dense_legacy` for step execution. +- Imported AR-KV rejection: `paged_scheduler` does not consume the + `ar_diffusion_kv` contract used by DreamZero or LingBot-World. +- Missing image output: include `"modalities": ["image"]` when using the chat + completions endpoint, and decode the returned data URI as shown above. + +For the ownership and lifecycle contract, see the +[Scheduler-managed paged KV design](../../design/feature/diffusion_paged_kv_cache.md). diff --git a/docs/user_guide/diffusion_features.md b/docs/user_guide/diffusion_features.md index 703a83f38..b27e80ccc 100644 --- a/docs/user_guide/diffusion_features.md +++ b/docs/user_guide/diffusion_features.md @@ -248,6 +248,7 @@ The Diffusion Acceleration navigation groups the remaining guides as follows: | Compatibility | [Feature Compatibility](feature_compatibility.md) | | CPU offloading | [CPU Offloading](diffusion/cpu_offload.md) | | Cache acceleration | [TeaCache](diffusion/cache_acceleration/teacache.md), [Cache-DiT](diffusion/cache_acceleration/cache_dit.md) | +| KV cache paging | [Scheduler-Managed Paged KV Cache](diffusion/paged_kv_cache.md) | | Parallelism | [Parallelism Overview](diffusion/parallelism/overview.md) | | Attention | [Attention Backends](diffusion/attention_backends.md) | | Compilation | [Regional Compilation](diffusion/regional_compilation.md) | diff --git a/recipes/Tencent/HunyuanImage-3.0-Instruct.md b/recipes/Tencent/HunyuanImage-3.0-Instruct.md index eb5ccedf6..d7e0d2d77 100644 --- a/recipes/Tencent/HunyuanImage-3.0-Instruct.md +++ b/recipes/Tencent/HunyuanImage-3.0-Instruct.md @@ -297,6 +297,56 @@ of: the TP=4 configuration before increasing batch size. GPU memory utilization is not a useful primary tuning knob for this DiT-only recipe. +### Scheduler-managed paged KV (request mode) + +HunyuanImage-3 can use Scheduler-managed paged KV in request-level execution. +The current paged implementation uses unquantized BF16 KV and requires the +logical `FLASH_ATTN` backend. The following command reuses the standard DiT +deploy file and supplies the paged-only stage settings with +`--stage-overrides`: + +For the mode-selection guide, configuration field reference, and parallelism +limits, see the [Scheduler-Managed Paged KV Cache user guide](../../docs/user_guide/diffusion/paged_kv_cache.md). + +```bash +vllm serve tencent/HunyuanImage-3.0-Instruct \ + --omni \ + --trust-remote-code \ + --port 8091 \ + --deploy-config vllm_omni/deploy/hunyuan_image3_dit.yaml \ + --stage-overrides \ + '{"0":{"diffusion_kv_mode":"paged_scheduler","diffusion_kv_max_rows_per_request":2,"kv_cache_memory_bytes":536870912,"step_execution":false,"diffusion_attention_backend":"FLASH_ATTN"}}' +``` + +Here `kv_cache_memory_bytes` reserves a 512 MiB physical KV pool per Worker +and rank, and `diffusion_kv_max_rows_per_request=2` reserves the two internal +rows used by standard positive/negative CFG. Use `1` when CFG is disabled. +Keep `max_num_seqs=1`: this paged Hunyuan path does not currently batch +independent public requests. + +Generate an image after `/health` reports ready: + +```bash +curl -s http://localhost:8091/v1/chat/completions \ + -H "Content-Type: application/json" \ + -d '{ + "modalities": ["image"], + "messages": [{"role": "user", "content": "A red ceramic vase on a wooden table"}], + "extra_body": { + "height": 1024, + "width": 1024, + "num_inference_steps": 20, + "guidance_scale": 5.0, + "seed": 1234 + } + }' | jq -r '.choices[0].message.content[0].image_url.url' \ + | cut -d',' -f2- | base64 -d > hunyuan_image3_paged.png +``` + +This mode is also implemented on Ascend NPU, where `FLASH_ATTN` resolves to +Ascend FIA. Hunyuan paged step execution, quantized paged KV, and CPU/DLO +offload combinations are not validated; use `dense_legacy` for those cases. + ### 2x B200 ModelOpt mixed FP8/NVFP4 #### Environment