Generalize AR-Diffusion KV session capabilities (#5271)

Signed-off-by: Jack47 <scsvip@gmail.com>
This commit is contained in:
Jack Chen authored and GitHub committed 2026-07-23 12:37:32 +08:00
1 parent 00bdb5dcd7
commit 96235da468
18 files changed
+1808 -800

No files matched your search

@@ -0,0 +1,165 @@
# AR-Diffusion pipeline capability
> **Status:** experimental and non-normative. This note documents the current
> `experimental/ar_diffusion` contract. It follows the ownership and lifecycle
> direction in [#5137](https://github.com/vllm-project/vllm-omni/issues/5137),
> but it is not yet part of the stable module-design hierarchy.
`ARDiffusionEngine` is an opt-in diffusion backend for pipelines that keep
autoregressive attention KV across requests. The engine only selects
`ARDiffusionModelRunner`; the runner owns paged KV pools, session lookup, LRU
eviction, and failure cleanup.
A pipeline opts in by implementing `SupportsARDiffusionPipeline`. Its immutable
`ARDiffusionKVCacheSpec` reports worker-local KV geometry and policy:
- self-attention layers, TP-local KV heads, head size, and tokens per latent frame;
- latent frames committed by one model block, sliding window, and attention sink;
- logical KV branches mapped to worker-local storage slots;
- named fixed-length cross-attention KV, model-specific scratch geometry, and
the pipeline's requested session-capacity upper bound.
The runner binds an `ARDiffusionKVState` only around one `forward()` call. The
pipeline may use it during that context but must not retain it. The session
survives subsequent requests with the same `session_id`. Request reset, explicit
close, LRU eviction, and forward exceptions all release runner-owned KV and call
the pipeline's reset/close lifecycle hook. Cross-attention KV is allocated lazily
per session and logical KV branch, and is released on the same path. A pipeline
publishes one named cache transactionally by yielding exactly one `(k, v)` pair
per layer:
```python
state.populate_cross_attention("main", "text", project_text_kv_by_layer())
```
The cache becomes readable only after the iterable completes successfully.
Logical branches remain independent even when they share a worker-local
`local_index`; that index only maps lazily executed self-attention and scratch
work to worker-local capacity slots. It does not define cross-attention cache
ownership.
Requests select the session with `extra_args["session_id"]`; setting `reset=True`
replaces it before the forward, while `close_session=True` releases it after a
successful forward. The runner also exposes `reset_session()` and
`close_session()` for an owning serving layer.
## Current execution limits
The experimental runner supports request-mode execution with
`max_num_seqs=1`. It rejects step execution and request-batch execution at load
time, and its inherited batch/step entry points also fail explicitly. Supporting
those paths requires a batch-aware state-binding contract rather than silently
bypassing `bind_ar_diffusion_state()`.
The current implementation retains one session per model instance. A request
for a different `session_id` closes the resident session through the same LRU
lifecycle used by explicit close and failure cleanup before allocating the new
one. Self-attention and cross-attention memory are therefore sized for one
resident session even if the pipeline declares a larger `session_capacity`.
This prioritizes rollout throughput and prevents block-pool exhaustion before a
count-based eviction can run. Multi-session residency can be added later with a
memory-backed capacity calculation or allocation-pressure eviction.
Scratch storage is also capability-driven. `frames_per_block` reserves space for
an uncommitted current video block, while `max_scratch_tokens_per_branch`
declares the maximum model-specific tokens, such as action/state registers, that
must coexist with it. `AR_DIFFUSION_KV_SCRATCH_BLOCKS_PER_BRANCH` may increase
that derived minimum for deployment experiments but cannot reduce it.
## DreamZero adapter
DreamZero implements the capability directly on `DreamZeroPipeline`. Its adapter
reads DreamZero transformer geometry, describes the positive and negative CFG
branches, binds the state with a context manager, and translates the existing
boolean CFG calls to the explicit `"positive"` and `"negative"` KV branch names.
Its optional `ar_diffusion_warmup_requests()` provider owns the robot observation
schema and the one-frame/four-frame warmup convention; none of those details are
present in the generic runner.
Code migrating from the former experimental KV state API should use the
mechanical mapping below:
```python
# Before: state.pos, state.neg, state.get_kv_caches(is_negative, ...)
positive = state.adapter("positive")
negative = state.adapter("negative")
contexts = state.get_kv_caches("negative", seq_len=frame_tokens, commit_current=True)
```
Direct experimental `ARDiffusionKVCache` construction also migrates
mechanically: replace `local_branches` with explicit `kv_branches`,
`num_frame_per_block` with `frames_per_block`, and the fixed text/image length
arguments with a named `cross_attention_lengths` mapping plus
`session_capacity`. Normal engine users do not construct this object directly.
Cross-attention writers migrate from incremental publication:
```python
# Before
for layer_index, (key, value) in enumerate(project_text_kv_by_layer()):
state.write_cross_attention_kv("text", layer_index, branch, key, value)
state.mark_cross_attention_populated(branch, "text")
# After
state.populate_cross_attention(branch, "text", project_text_kv_by_layer())
```
The iterable is consumed once and must yield exactly `num_layers` pairs. Failed
or incomplete projection leaves the previous complete cache unchanged, or
leaves the cache absent on its first population.
## Single-KV-branch pipeline sketch
A future single-branch causal DMD pipeline can opt in without changes to the
runner or KV core:
```python
class WorldPipeline:
def ar_diffusion_kv_cache_spec(self):
return ARDiffusionKVCacheSpec(
num_layers=self.transformer.num_layers,
num_kv_heads=self.transformer.tp_local_kv_heads,
head_size=self.transformer.head_size,
tokens_per_frame=self.transformer.tokens_per_latent_frame,
frames_per_block=3,
window_frames=self.transformer.sliding_window_frames,
sink_frames=self.transformer.sink_frames,
kv_branches=(ARDiffusionKVBranchSpec("main", 0),),
cross_attention=(ARDiffusionCrossAttentionKVSpec("text", self.text_length),),
max_scratch_tokens_per_branch=self.transformer.max_register_tokens,
session_capacity=1,
)
@contextmanager
def bind_ar_diffusion_state(self, session_id, state):
self.current_ar_state = state
try:
yield
finally:
self.current_ar_state = None
def reset_ar_diffusion_session(self, session_id):
self.model_states.pop(session_id, None)
def close_ar_diffusion_session(self, session_id):
self.model_states.pop(session_id, None)
```
The model writes/reads named cross-attention KV through the bound state and uses
the `"main"` KV branch for paged self-attention. Pipelines without a warmup
provider are loaded normally and skip AR rollout warmup.
## Relationship to the multi-stage KV manager RFC
[#5244](https://github.com/vllm-project/vllm-omni/issues/5244) describes a
future multi-stage KV transfer manager, connector integration, and cache-aware
scheduling. This capability has a narrower scope: it owns model-local paged KV
and session lifecycle inside one AR-Diffusion runner. It does not define
cross-stage transfer, connector behavior, or scheduler-visible cache affinity.
When the experimental runtime migrates toward #5244, the implementation should
reconcile its current full `KVCacheManager` use with the RFC's proposed
`BlockPool`-oriented DiT manager. The pipeline capability should remain the
model-facing geometry and state-binding boundary where possible, while the
experimental `ARDiffusionKVCache` class itself should not be treated as a stable
public API.
@@ -0,0 +1,317 @@
# SPDX-License-Identifier: Apache-2.0
"""CPU contracts for capability-driven AR-Diffusion sessions."""
from __future__ import annotations
from collections import OrderedDict
from contextlib import contextmanager
from types import SimpleNamespace
import pytest
import torch
from vllm_omni.diffusion.worker.diffusion_model_runner import DiffusionModelRunner
from vllm_omni.experimental.ar_diffusion.capability import (
ARDiffusionCrossAttentionKVSpec,
ARDiffusionKVBranchSpec,
ARDiffusionKVCacheSpec,
)
from vllm_omni.experimental.ar_diffusion.kv_cache import ARDiffusionKVConfig
from vllm_omni.experimental.ar_diffusion.runner import ARDiffusionModelRunner
BLOCK = 16
POS = "positive"
NEG = "negative"
pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
def lingbot_like_spec(*, capacity: int = 2) -> ARDiffusionKVCacheSpec:
"""Single-kv_branch causal DMD: 3 latent frames/block + sink/window + text KV."""
return ARDiffusionKVCacheSpec(
num_layers=2,
num_kv_heads=4, # already TP-local
head_size=64,
tokens_per_frame=BLOCK,
frames_per_block=3,
window_frames=5,
sink_frames=1,
kv_branches=(ARDiffusionKVBranchSpec("main", 0),),
session_capacity=capacity,
cross_attention=(ARDiffusionCrossAttentionKVSpec("text", 8),),
)
def dreamzero_like_spec(*, capacity: int = 2) -> ARDiffusionKVCacheSpec:
return ARDiffusionKVCacheSpec(
num_layers=2,
num_kv_heads=4,
head_size=64,
tokens_per_frame=BLOCK,
frames_per_block=4,
window_frames=6,
kv_branches=(ARDiffusionKVBranchSpec(POS, 0), ARDiffusionKVBranchSpec(NEG, 1)),
session_capacity=capacity,
cross_attention=(ARDiffusionCrossAttentionKVSpec("text", 8),),
)
class CapablePipeline:
def __init__(self, spec: ARDiffusionKVCacheSpec) -> None:
self.spec = spec
self.bound_state = None
self.binds: list[str] = []
self.resets: list[str] = []
self.closes: list[str] = []
def ar_diffusion_kv_cache_spec(self) -> ARDiffusionKVCacheSpec:
return self.spec
@contextmanager
def bind_ar_diffusion_state(self, session_id: str, state):
assert self.bound_state is None
self.bound_state = state
self.binds.append(session_id)
try:
yield
finally:
self.bound_state = None
def reset_ar_diffusion_session(self, session_id: str) -> None:
self.resets.append(session_id)
def close_ar_diffusion_session(self, session_id: str) -> None:
self.closes.append(session_id)
class WarmupPipeline(CapablePipeline):
def __init__(self, spec: ARDiffusionKVCacheSpec, requests: list[object]) -> None:
super().__init__(spec)
self.requests = requests
def ar_diffusion_warmup_requests(self, session_id: str):
assert session_id == ARDiffusionModelRunner._WARMUP_SID
return iter(self.requests)
class BatchCapablePipeline(CapablePipeline):
supports_request_batch = True
def make_runner(
pipeline: object,
*,
available_bytes: int = 1 << 28,
step_execution: bool = False,
) -> ARDiffusionModelRunner:
runner = object.__new__(ARDiffusionModelRunner)
runner.od_config = SimpleNamespace(
max_num_seqs=1,
dtype=torch.float32,
enforce_eager=True,
step_execution=step_execution,
)
runner.device = torch.device("cpu")
runner.pipeline = pipeline
runner.ar_diffusion_kv_config = ARDiffusionKVConfig(enable=True)
runner.kv_cache = None
runner._ar_diffusion_capability = None
runner._ar_diffusion_kv_cache_spec = None
runner._sessions = OrderedDict()
runner._session_capacity = 0
runner._perf_e2e_times = []
runner._preallocate_kv_cache(available_bytes=available_bytes)
return runner
def commit_one_frame(runner: ARDiffusionModelRunner, session_id: str, kv_branch: str):
state = runner._get_or_create_session(session_id)
ctx = state.get_kv_caches(kv_branch, seq_len=BLOCK, commit_current=True)[0].forward_ctx
ctx.ensure_video_slots(torch.device("cpu"))
state.commit_paged_context(kv_branch)
return state
def test_ar_runner_rejects_pipeline_without_capability():
runner = object.__new__(ARDiffusionModelRunner)
runner.od_config = SimpleNamespace(max_num_seqs=1)
runner.pipeline = object()
with pytest.raises(TypeError, match="SupportsARDiffusionPipeline"):
runner._preallocate_kv_cache(available_bytes=1 << 20)
def test_lingbot_like_single_branch_session_reuse_reset_and_close():
pipeline = CapablePipeline(lingbot_like_spec())
runner = make_runner(pipeline)
kv = runner.kv_cache
assert kv is not None
assert kv.num_local_kv_branches == 1
assert kv.frames_per_block == 3
assert kv.spec.window_chunks == 5
assert kv.spec.sink_chunks == 1
assert kv.cross_attention_lengths == {"text": 8}
first = commit_one_frame(runner, "s1", "main")
assert runner._get_or_create_session("s1") is first
k = torch.randn(1, 8, 4, 64)
v = torch.randn(1, 8, 4, 64)
first.populate_cross_attention("main", "text", [(k, v)] * first.num_layers)
assert first.get_cross_attention_kv("main", "text")[0]["k"].shape == k.shape
runner.reset_session("s1")
assert "s1" not in runner._sessions
assert "s1" not in kv._cross_sessions
assert pipeline.resets == ["s1"]
second = runner._get_or_create_session("s1")
assert second is not first
runner.close_session("s1")
assert "s1" not in runner._sessions
assert pipeline.closes == ["s1"]
def test_lingbot_like_sink_survives_sliding_window_eviction():
runner = make_runner(CapablePipeline(lingbot_like_spec()))
kv = runner.kv_cache
assert kv is not None
for _ in range(8):
state = commit_one_frame(runner, "s1", "main")
table = kv.block_table(state.adapter("main"))
assert table[0] != kv.null_block_id
assert table[1] == kv.null_block_id
ctx = state.get_kv_caches("main", seq_len=BLOCK, commit_current=False)[0].forward_ctx
visible, _ = ctx.video_block_table(torch.device("cpu"))
assert visible[0] == table[0]
assert len(visible) == 6 # sink + recent window, including current scratch
def test_dreamzero_like_two_branches_are_independent():
runner = make_runner(CapablePipeline(dreamzero_like_spec()))
kv = runner.kv_cache
assert kv is not None and kv.num_local_kv_branches == 2
state = commit_one_frame(runner, "s1", POS)
assert len(kv.window_block_ids(state.adapter(POS))) == 1
assert kv.window_block_ids(state.adapter(NEG)) == []
def test_lru_eviction_releases_blocks_and_notifies_pipeline():
pipeline = CapablePipeline(lingbot_like_spec(capacity=8))
runner = make_runner(pipeline)
kv = runner.kv_cache
assert kv is not None
assert runner._session_capacity == 1
assert kv.session_capacity == 1
free_total = kv.manager.block_pool.get_num_free_blocks()
old = commit_one_frame(runner, "old", "main")
k = torch.randn(1, 8, 4, 64)
old.populate_cross_attention("main", "text", [(k, k)] * old.num_layers)
assert kv.manager.block_pool.get_num_free_blocks() < free_total
runner._get_or_create_session("new")
assert tuple(runner._sessions) == ("new",)
assert pipeline.closes == ["old"]
assert "old" not in kv._cross_sessions
assert kv.manager.block_pool.get_num_free_blocks() == free_total
def test_forward_exception_releases_pending_allocation_and_model_state(monkeypatch):
pipeline = CapablePipeline(lingbot_like_spec())
runner = make_runner(pipeline)
kv = runner.kv_cache
assert kv is not None
free_total = kv.manager.block_pool.get_num_free_blocks()
def boom(self, req, kv_prefetch_job=None):
state = pipeline.bound_state
ctx = state.get_kv_caches("main", seq_len=BLOCK, commit_current=True)[0].forward_ctx
ctx.ensure_video_slots(torch.device("cpu"))
raise RuntimeError("layer exploded")
monkeypatch.setattr(DiffusionModelRunner, "execute_model", boom)
request = SimpleNamespace(sampling_params=SimpleNamespace(extra_args={"session_id": "broken"}))
with pytest.raises(RuntimeError, match="layer exploded"):
runner.execute_model(request)
assert pipeline.bound_state is None
assert pipeline.closes == ["broken"]
assert not runner._sessions
assert not kv._adapters
assert kv.manager.block_pool.get_num_free_blocks() == free_total
def test_synchronize_exception_uses_forward_cleanup_path(monkeypatch):
pipeline = CapablePipeline(lingbot_like_spec())
runner = make_runner(pipeline)
kv = runner.kv_cache
assert kv is not None
free_total = kv.manager.block_pool.get_num_free_blocks()
def return_after_allocation(self, req, kv_prefetch_job=None):
state = pipeline.bound_state
ctx = state.get_kv_caches("main", seq_len=BLOCK, commit_current=True)[0].forward_ctx
ctx.ensure_video_slots(torch.device("cpu"))
return object()
def synchronize_boom(device):
raise RuntimeError("asynchronous kernel failed")
monkeypatch.setattr(DiffusionModelRunner, "execute_model", return_after_allocation)
monkeypatch.setattr(torch.accelerator, "synchronize", synchronize_boom)
runner.device = torch.device("cuda")
request = SimpleNamespace(sampling_params=SimpleNamespace(extra_args={"session_id": "broken"}))
with pytest.raises(RuntimeError, match="asynchronous kernel failed"):
runner.execute_model(request)
assert pipeline.bound_state is None
assert pipeline.closes == ["broken"]
assert not runner._sessions
assert not kv._adapters
assert not runner._perf_e2e_times
assert kv.manager.block_pool.get_num_free_blocks() == free_total
def test_ar_runner_rejects_step_and_request_batch_modes():
with pytest.raises(ValueError, match="step_execution=True"):
make_runner(CapablePipeline(lingbot_like_spec()), step_execution=True)
with pytest.raises(ValueError, match="request-batch execution"):
make_runner(BatchCapablePipeline(lingbot_like_spec()))
def test_ar_runner_defensively_rejects_inherited_batch_and_step_entrypoints():
runner = object.__new__(ARDiffusionModelRunner)
with pytest.raises(RuntimeError, match="request-batch execution"):
runner.execute_model_batch(None, None)
with pytest.raises(RuntimeError, match="step execution"):
runner.execute_stepwise(None)
def test_model_specific_warmup_provider_is_consumed(monkeypatch):
requests = [object(), object()]
pipeline = WarmupPipeline(lingbot_like_spec(), requests)
runner = make_runner(pipeline)
seen: list[object] = []
monkeypatch.setattr(runner, "execute_model", seen.append)
runner._warmup_ar_rollout()
assert seen == requests
assert pipeline.closes == [runner._WARMUP_SID]
def test_pipeline_without_warmup_provider_is_safely_skipped(monkeypatch):
pipeline = CapablePipeline(lingbot_like_spec())
runner = make_runner(pipeline)
execute = SimpleNamespace(called=False)
def fail_if_called(request):
execute.called = True
monkeypatch.setattr(runner, "execute_model", fail_if_called)
runner._warmup_ar_rollout()
assert execute.called is False
+66 -18
View File
@@ -12,6 +12,7 @@ from vllm.v1.kv_cache_interface import KVCacheSpecKind, get_kv_cache_spec_kind
from vllm.v1.kv_cache_spec_registry import KVCacheSpecRegistry
from vllm.v1.request import RequestStatus
from vllm_omni.experimental.ar_diffusion.capability import ARDiffusionKVBranchSpec
from vllm_omni.experimental.ar_diffusion.kv_cache import (
ARDiffusionKVCache,
ARDiffusionKVConfig,
@@ -194,16 +195,29 @@ def test_cross_attn_pool_deducted_from_self_attn_budget():
block_size=BLOCK,
max_model_len=4096,
available_bytes=avail,
cross_attn_length=L,
kv_branches=(ARDiffusionKVBranchSpec("positive", 0), ARDiffusionKVBranchSpec("negative", 0)),
session_capacity=1,
cross_attention_lengths={"text": L},
device=torch.device("cpu"),
)
cross_bytes = 2 * 2 * L * 4 * 64 * torch.float16.itemsize * 2 # K+V, pos+neg, layers
expected = compute_num_blocks(avail - cross_bytes, 0.5, kv.spec.page_size_bytes * 2)
assert kv.num_blocks == expected
assert expected > 2 * (2 + 1) + 2 # above the min-blocks floor, so the deduction is what's tested
assert expected > 1 * (2 + 1) + 2 # above the local-slot floor, so the cross deduction is what's tested
def _make_kv(*, local_branches, num_frame_per_block=2, window_chunks=9):
def _make_kv(
*,
local_branches,
num_frame_per_block=2,
window_chunks=9,
max_scratch_tokens_per_branch=0,
):
kv_branches = (
(ARDiffusionKVBranchSpec("positive", 0), ARDiffusionKVBranchSpec("negative", 0))
if local_branches == 1
else (ARDiffusionKVBranchSpec("positive", 0), ARDiffusionKVBranchSpec("negative", 1))
)
return ARDiffusionKVCache(
ARDiffusionKVConfig(enable=True, chunk_size=BLOCK, window_chunks=window_chunks),
num_layers=1,
@@ -214,41 +228,75 @@ def _make_kv(*, local_branches, num_frame_per_block=2, window_chunks=9):
max_model_len=4096,
available_bytes=1 << 16, # tiny -> the floor binds
device=torch.device("cpu"),
local_branches=local_branches,
num_frame_per_block=num_frame_per_block,
kv_branches=kv_branches,
session_capacity=1,
frames_per_block=num_frame_per_block,
max_scratch_tokens_per_branch=max_scratch_tokens_per_branch,
)
def test_pool_floor_is_branch_aware():
"""CFG-parallel rank (one local branch) sizes for one window + in-flight chunk;
a single-process run (both branches) sizes for two. Scratch scales the same way."""
"""CFG-parallel rank (one local kv_branch) sizes for one window + in-flight chunk;
a single-process run (both kv_branches) sizes for two. Scratch scales the same way."""
one = _make_kv(local_branches=1)
two = _make_kv(local_branches=2)
assert one.managed_num_blocks == 1 * (9 + 2) + 2 # 13
assert two.managed_num_blocks == 2 * (9 + 2) + 2 # 24
assert one.scratch_num_blocks == one.scratch_blocks_per_branch
assert two.scratch_num_blocks == 2 * two.scratch_blocks_per_branch
assert one.num_blocks_total == 13 + one.scratch_blocks_per_branch
assert one.scratch_num_blocks == one.scratch_blocks_per_kv_branch
assert two.scratch_num_blocks == 2 * two.scratch_blocks_per_kv_branch
assert one.scratch_blocks_per_kv_branch == 2
assert one.num_blocks_total == 13 + one.scratch_blocks_per_kv_branch
def test_scratch_capacity_is_derived_from_declared_geometry(monkeypatch):
monkeypatch.delenv("AR_DIFFUSION_KV_SCRATCH_BLOCKS_PER_BRANCH", raising=False)
kv = _make_kv(
local_branches=1,
num_frame_per_block=6,
max_scratch_tokens_per_branch=BLOCK + 1,
)
assert kv.scratch_blocks_per_kv_branch == 8 # six video blocks + two auxiliary blocks
def test_scratch_env_override_cannot_reduce_declared_minimum(monkeypatch):
monkeypatch.setenv("AR_DIFFUSION_KV_SCRATCH_BLOCKS_PER_BRANCH", "1")
kv = _make_kv(
local_branches=1,
num_frame_per_block=6,
max_scratch_tokens_per_branch=BLOCK + 1,
)
assert kv.scratch_blocks_per_kv_branch == 8
def test_scratch_maps_to_slot_zero_with_one_local_branch():
"""A CFG-parallel rank runs exactly one branch: whichever CFG side it is,
"""A CFG-parallel rank runs exactly one kv_branch: whichever CFG side it is,
its scratch lands in the rank's single slot (no dead second slot)."""
one = _make_kv(local_branches=1)
assert one.scratch_block_ids(True, 0, 2) == one.scratch_block_ids(False, 0, 2)
assert one.scratch_block_ids("positive", 0, 2) == one.scratch_block_ids("negative", 0, 2)
two = _make_kv(local_branches=2)
assert two.scratch_block_ids(True, 0, 2) != two.scratch_block_ids(False, 0, 2)
assert two.scratch_block_ids("positive", 0, 2) != two.scratch_block_ids("negative", 0, 2)
def test_scratch_exhaustion_still_raises():
one = _make_kv(local_branches=1)
cap = one.scratch_blocks_per_branch
cap = one.scratch_blocks_per_kv_branch
with pytest.raises(RuntimeError, match="scratch blocks exhausted"):
one.scratch_block_ids(False, 0, cap + 1)
one.scratch_block_ids("positive", 0, cap + 1)
def test_invalid_local_branches_rejected():
with pytest.raises(ValueError, match="local_branches"):
_make_kv(local_branches=3)
def test_non_contiguous_branch_indices_rejected():
with pytest.raises(ValueError, match="contiguous"):
ARDiffusionKVCache(
ARDiffusionKVConfig(enable=True, chunk_size=BLOCK, window_chunks=2),
num_layers=1,
num_kv_heads=4,
head_size=64,
dtype=torch.float32,
block_size=BLOCK,
max_model_len=4096,
available_bytes=1 << 16,
kv_branches=(ARDiffusionKVBranchSpec("main", 1),),
session_capacity=1,
)
+18 -4
View File
@@ -4,9 +4,18 @@
import pytest
import torch
from vllm_omni.experimental.ar_diffusion.capability import ARDiffusionKVBranchSpec
from vllm_omni.experimental.ar_diffusion.kv_cache import ARDiffusionKVCache, ARDiffusionKVConfig
DIMS = dict(num_layers=2, num_kv_heads=4, head_size=64, dtype=torch.float16, block_size=16)
DIMS = dict(
num_layers=2,
num_kv_heads=4,
head_size=64,
dtype=torch.float16,
block_size=16,
kv_branches=(ARDiffusionKVBranchSpec("positive", 0), ARDiffusionKVBranchSpec("negative", 1)),
session_capacity=2,
)
def make_cache(*, chunk_size=16, window_chunks=2, available_bytes=1 << 24):
@@ -67,7 +76,7 @@ def test_num_computed_advances_per_chunk():
def test_state_close_frees_both_branch_blocks():
"""ARDiffusionKVState.close() returns both CFG branches' pool blocks to the pool.
"""ARDiffusionKVState.close() returns both CFG kv_branches' pool blocks to the pool.
This is the primitive the runner's LRU eviction relies on: when a session is
evicted, close() must free the blocks both adapters hold, or session churn
@@ -80,12 +89,17 @@ def test_state_close_frees_both_branch_blocks():
pos = kv.begin_request("bde__s")
neg = kv.begin_request("bde__s__neg")
state = ARDiffusionKVState(kv, pos, neg, num_layers=kv.num_layers)
state = ARDiffusionKVState(
kv,
"s",
{"positive": pos, "negative": neg},
num_layers=kv.num_layers,
)
for _ in range(3):
for adapter in (pos, neg):
kv.allocate_chunk(adapter)
kv.commit_chunk(adapter)
# Both branches hold resident blocks now.
# Both kv_branches hold resident blocks now.
assert kv.manager.block_pool.get_num_free_blocks() < free_total
state.close()
+38 -31
View File
@@ -11,6 +11,7 @@ from unittest.mock import MagicMock
import pytest
import torch
from vllm_omni.experimental.ar_diffusion.capability import ARDiffusionKVBranchSpec
from vllm_omni.experimental.ar_diffusion.kv_cache import (
ARDiffusionKVCache,
ARDiffusionKVConfig,
@@ -24,6 +25,8 @@ from vllm_omni.experimental.ar_diffusion.kv_cache.state import ARDiffusionKVStat
BLOCK = 16
N_HEADS = 4
HEAD_DIM = 64
POS = "positive"
NEG = "negative"
def make_state(*, num_layers=1, window_chunks=2, dtype=torch.float32, device=torch.device("cpu")):
@@ -37,11 +40,15 @@ def make_state(*, num_layers=1, window_chunks=2, dtype=torch.float32, device=tor
block_size=BLOCK,
max_model_len=4096,
available_bytes=1 << 26,
kv_branches=(ARDiffusionKVBranchSpec(POS, 0), ARDiffusionKVBranchSpec(NEG, 1)),
session_capacity=2,
frames_per_block=2,
max_scratch_tokens_per_branch=BLOCK,
device=device,
)
pos = kv.begin_request("r-pos")
neg = kv.begin_request("r-neg")
return kv, ARDiffusionKVState(kv, pos, neg, num_layers=num_layers)
return kv, ARDiffusionKVState(kv, "s1", {POS: pos, NEG: neg}, num_layers=num_layers)
def _dense_attention(query: torch.Tensor, key: torch.Tensor, value: torch.Tensor) -> torch.Tensor:
@@ -79,51 +86,51 @@ def _commit_video_span(
kv: ARDiffusionKVCache,
st: ARDiffusionKVState,
*,
is_negative: bool,
kv_branch: str,
n_chunks: int,
dtype: torch.dtype,
device: torch.device,
) -> tuple[torch.Tensor, torch.Tensor]:
ctx = st.get_kv_caches(is_negative, seq_len=n_chunks * BLOCK, commit_current=True)[0].forward_ctx
ctx = st.get_kv_caches(kv_branch, seq_len=n_chunks * BLOCK, commit_current=True)[0].forward_ctx
ctx.ensure_video_slots(device)
k = torch.randn(1, n_chunks * BLOCK, N_HEADS, HEAD_DIM, dtype=dtype, device=device)
v = torch.randn(1, n_chunks * BLOCK, N_HEADS, HEAD_DIM, dtype=dtype, device=device)
kv._k_pools[0][ctx.current_video_slot_mapping] = k[0]
kv._v_pools[0][ctx.current_video_slot_mapping] = v[0]
st.commit_paged_context(is_negative)
st.commit_paged_context(kv_branch)
return k, v
def test_paged_context_allocates_lazily_and_commits_after_forward():
_, st = make_state()
contexts = st.get_kv_caches(False, seq_len=BLOCK, commit_current=True)
contexts = st.get_kv_caches(POS, seq_len=BLOCK, commit_current=True)
ctx = contexts[0].forward_ctx
assert isinstance(contexts[0], ARDiffusionPagedLayerContext)
assert st.pos.completed_chunks == 0
assert st.adapter(POS).completed_chunks == 0
assert ctx.current_video_slot_mapping is None
ctx.ensure_video_slots(torch.device("cpu"))
assert st.pos.completed_chunks == 0
assert st.adapter(POS).completed_chunks == 0
assert len(ctx.current_video_block_ids) == 1
st.commit_paged_context(False)
assert st.pos.completed_chunks == 1
assert st._committed[False] == BLOCK
st.commit_paged_context(POS)
assert st.adapter(POS).completed_chunks == 1
assert st._committed[POS] == BLOCK
def test_scratch_video_and_action_blocks_do_not_commit():
kv, st = make_state()
ctx = st.get_kv_caches(False, seq_len=2 * BLOCK, commit_current=False)[0].forward_ctx
ctx = st.get_kv_caches(POS, seq_len=2 * BLOCK, commit_current=False)[0].forward_ctx
ctx.ensure_video_slots(torch.device("cpu"))
ctx.ensure_action_slots(3, torch.device("cpu"))
assert ctx.current_video_block_ids == kv.scratch_block_ids(False, 0, 2)
assert ctx.action_scratch_block_ids == kv.scratch_block_ids(False, 2, 1)
st.commit_paged_context(False)
assert st.pos.completed_chunks == 0
assert st._committed[False] == 0
assert ctx.current_video_block_ids == kv.scratch_block_ids(POS, 0, 2)
assert ctx.action_scratch_block_ids == kv.scratch_block_ids(POS, 2, 1)
st.commit_paged_context(POS)
assert st.adapter(POS).completed_chunks == 0
assert st._committed[POS] == 0
def test_pipeline_kv_get_paged_path_has_no_gather_backend():
@@ -155,7 +162,7 @@ def test_paged_attention_matches_dense_reference_cpu(history_chunks, action_len,
k, v = _commit_video_span(
kv,
st,
is_negative=False,
kv_branch=POS,
n_chunks=history_chunks,
dtype=dtype,
device=device,
@@ -163,7 +170,7 @@ def test_paged_attention_matches_dense_reference_cpu(history_chunks, action_len,
history_k_parts.append(k)
history_v_parts.append(v)
ctx = st.get_kv_caches(False, seq_len=BLOCK, commit_current=commit_current)[0].forward_ctx
ctx = st.get_kv_caches(POS, seq_len=BLOCK, commit_current=commit_current)[0].forward_ctx
ctx.ensure_video_slots(device)
current_k = torch.randn(1, BLOCK, N_HEADS, HEAD_DIM, dtype=dtype, device=device)
current_v = torch.randn(1, BLOCK, N_HEADS, HEAD_DIM, dtype=dtype, device=device)
@@ -212,9 +219,9 @@ def test_paged_attention_matches_dense_reference_cpu(history_chunks, action_len,
torch.testing.assert_close(paged, ref, rtol=1e-5, atol=1e-5)
before = st.pos.completed_chunks
st.commit_paged_context(False)
assert st.pos.completed_chunks == before + (1 if commit_current else 0)
before = st.adapter(POS).completed_chunks
st.commit_paged_context(POS)
assert st.adapter(POS).completed_chunks == before + (1 if commit_current else 0)
@pytest.mark.skipif(not _cuda_flash_attn_usable(), reason="usable CUDA FlashAttention is required")
@@ -231,13 +238,13 @@ def test_paged_attention_matches_dense_reference_gpu(history_chunks, action_len,
history_k, history_v = _commit_video_span(
kv,
st,
is_negative=False,
kv_branch=POS,
n_chunks=history_chunks,
dtype=dtype,
device=device,
)
layer_ctx = st.get_kv_caches(False, seq_len=BLOCK, commit_current=commit_current)[0]
layer_ctx = st.get_kv_caches(POS, seq_len=BLOCK, commit_current=commit_current)[0]
ctx = layer_ctx.forward_ctx
current_k = torch.randn(1, BLOCK, N_HEADS, HEAD_DIM, dtype=dtype, device=device)
current_v = torch.randn(1, BLOCK, N_HEADS, HEAD_DIM, dtype=dtype, device=device)
@@ -294,13 +301,13 @@ def test_block_table_padded_to_fixed_width():
device = torch.device("cpu")
kv, st = make_state(window_chunks=2)
ctx1 = st.get_kv_caches(False, seq_len=BLOCK, commit_current=True)[0].forward_ctx
ctx1 = st.get_kv_caches(POS, seq_len=BLOCK, commit_current=True)[0].forward_ctx
ctx1.prepare(device=device, action_len=0, query_len=BLOCK)
st.commit_paged_context(False)
st.commit_paged_context(POS)
ctx2 = st.get_kv_caches(False, seq_len=BLOCK, commit_current=True)[0].forward_ctx
ctx2 = st.get_kv_caches(POS, seq_len=BLOCK, commit_current=True)[0].forward_ctx
ctx2.prepare(device=device, action_len=0, query_len=BLOCK)
st.commit_paged_context(False)
st.commit_paged_context(POS)
# 1-block vs 2-block visible history: same table width, same max_seq_len.
assert ctx1.block_table.shape == ctx2.block_table.shape
@@ -315,7 +322,7 @@ def test_block_table_padded_to_fixed_width():
def test_prepare_is_idempotent_and_layers_share_metadata():
device = torch.device("cpu")
_, st = make_state(num_layers=2)
contexts = st.get_kv_caches(False, seq_len=BLOCK, commit_current=False)
contexts = st.get_kv_caches(POS, seq_len=BLOCK, commit_current=False)
fctx = contexts[0].forward_ctx
fctx.prepare(device=device, action_len=0, query_len=BLOCK)
table = fctx.block_table
@@ -335,7 +342,7 @@ def test_prepare_is_idempotent_and_layers_share_metadata():
def test_layer_inputs_before_prepare_raises():
_, st = make_state()
layer_ctx = st.get_kv_caches(False, seq_len=BLOCK, commit_current=False)[0]
layer_ctx = st.get_kv_caches(POS, seq_len=BLOCK, commit_current=False)[0]
with pytest.raises(RuntimeError, match="before prepare"):
layer_ctx.to_layer_inputs()
@@ -363,7 +370,7 @@ def test_custom_op_compiles_fullgraph_without_recompile_on_value_change():
kv, st = make_state(num_layers=2, window_chunks=2)
def run_one_forward(commit):
contexts = st.get_kv_caches(False, seq_len=BLOCK, commit_current=commit)
contexts = st.get_kv_caches(POS, seq_len=BLOCK, commit_current=commit)
fctx = contexts[0].forward_ctx
fctx.prepare(device=device, action_len=0, query_len=BLOCK)
q = torch.randn(BLOCK, N_HEADS, HEAD_DIM)
@@ -374,7 +381,7 @@ def test_custom_op_compiles_fullgraph_without_recompile_on_value_change():
# block-forward code object in production).
for layer_ctx in contexts:
out = compiled(layer_ctx.to_layer_inputs(), q, k, v)
st.commit_paged_context(False)
st.commit_paged_context(POS)
return out
torch._dynamo.reset()
+153 -98
View File
@@ -4,6 +4,7 @@
import pytest
import torch
from vllm_omni.experimental.ar_diffusion.capability import ARDiffusionKVBranchSpec
from vllm_omni.experimental.ar_diffusion.kv_cache import (
ARDiffusionKVCache,
ARDiffusionKVConfig,
@@ -14,9 +15,11 @@ from vllm_omni.experimental.ar_diffusion.kv_cache.state import ARDiffusionKVStat
BLOCK = 16
N_HEADS = 4
HEAD_DIM = 64
POS = "positive"
NEG = "negative"
def make_state(num_layers=1, window_chunks=4, cross_attn_length=0):
def make_state(num_layers=1, window_chunks=4, cross_attn_length=0, shared_local_index=False):
cfg = ARDiffusionKVConfig(enable=True, chunk_size=BLOCK, window_chunks=window_chunks)
kv = ARDiffusionKVCache(
cfg,
@@ -27,24 +30,29 @@ def make_state(num_layers=1, window_chunks=4, cross_attn_length=0):
block_size=BLOCK,
max_model_len=4096,
available_bytes=1 << 24,
cross_attn_length=cross_attn_length,
kv_branches=(
ARDiffusionKVBranchSpec(POS, 0),
ARDiffusionKVBranchSpec(NEG, 0 if shared_local_index else 1),
),
session_capacity=2,
cross_attention_lengths={"text": cross_attn_length} if cross_attn_length else None,
device=torch.device("cpu"),
)
pos = kv.begin_request("r-pos")
neg = kv.begin_request("r-neg")
return kv, ARDiffusionKVState(kv, pos, neg, num_layers=num_layers)
return kv, ARDiffusionKVState(kv, "s1", {POS: pos, NEG: neg}, num_layers=num_layers)
def _prepare_and_commit(st: ARDiffusionKVState, is_negative: bool, n_chunks: int) -> None:
ctx = st.get_kv_caches(is_negative, seq_len=n_chunks * BLOCK, commit_current=True)[0].forward_ctx
def _prepare_and_commit(st: ARDiffusionKVState, kv_branch: str, n_chunks: int) -> None:
ctx = st.get_kv_caches(kv_branch, seq_len=n_chunks * BLOCK, commit_current=True)[0].forward_ctx
ctx.ensure_video_slots(torch.device("cpu"))
st.commit_paged_context(is_negative)
st.commit_paged_context(kv_branch)
def test_get_returns_paged_layer_contexts_when_nothing_committed():
_, st = make_state(num_layers=2)
for neg in (False, True):
contexts = st.get_kv_caches(neg, seq_len=BLOCK, commit_current=False)
for kv_branch in (POS, NEG):
contexts = st.get_kv_caches(kv_branch, seq_len=BLOCK, commit_current=False)
assert len(contexts) == 2
assert all(isinstance(ctx, ARDiffusionPagedLayerContext) for ctx in contexts)
assert contexts[0].history_block_ids == []
@@ -52,43 +60,43 @@ def test_get_returns_paged_layer_contexts_when_nothing_committed():
def test_managed_paged_context_commits_only_after_forward():
kv, st = make_state(num_layers=1)
contexts = st.get_kv_caches(False, seq_len=2 * BLOCK, commit_current=True)
contexts = st.get_kv_caches(POS, seq_len=2 * BLOCK, commit_current=True)
ctx = contexts[0].forward_ctx
assert st.pos.completed_chunks == 0
assert kv.window_block_ids(st.pos) == []
assert st.adapter(POS).completed_chunks == 0
assert kv.window_block_ids(st.adapter(POS)) == []
ctx.ensure_video_slots(torch.device("cpu"))
assert st.pos.completed_chunks == 0
assert st.adapter(POS).completed_chunks == 0
assert len(ctx.current_video_block_ids) == 2
st.commit_paged_context(False)
st.commit_paged_context(POS)
assert st.pos.completed_chunks == 2
assert st._committed[False] == 2 * BLOCK
assert len(kv.window_block_ids(st.pos)) == 2
assert st.adapter(POS).completed_chunks == 2
assert st._committed[POS] == 2 * BLOCK
assert len(kv.window_block_ids(st.adapter(POS))) == 2
def test_scratch_paged_context_does_not_commit():
kv, st = make_state(num_layers=1)
contexts = st.get_kv_caches(False, seq_len=BLOCK, commit_current=False)
contexts = st.get_kv_caches(POS, seq_len=BLOCK, commit_current=False)
ctx = contexts[0].forward_ctx
ctx.ensure_video_slots(torch.device("cpu"))
assert ctx.current_video_block_ids == kv.scratch_block_ids(False, 0, 1)
st.commit_paged_context(False)
assert ctx.current_video_block_ids == kv.scratch_block_ids(POS, 0, 1)
st.commit_paged_context(POS)
assert st.pos.completed_chunks == 0
assert st._committed[False] == 0
assert kv.window_block_ids(st.pos) == []
assert st.adapter(POS).completed_chunks == 0
assert st._committed[POS] == 0
assert kv.window_block_ids(st.adapter(POS)) == []
def test_eviction_bounds_resident_window():
kv, st = make_state(num_layers=1, window_chunks=3)
_prepare_and_commit(st, False, 3)
assert len(kv.window_block_ids(st.pos)) == 3
_prepare_and_commit(st, POS, 3)
assert len(kv.window_block_ids(st.adapter(POS))) == 3
_prepare_and_commit(st, False, 2)
ctx = st.get_kv_caches(False, seq_len=BLOCK, commit_current=False)[0].forward_ctx
_prepare_and_commit(st, POS, 2)
ctx = st.get_kv_caches(POS, seq_len=BLOCK, commit_current=False)[0].forward_ctx
visible_blocks, video_len = ctx.video_block_table(torch.device("cpu"))
assert len(visible_blocks) == 3
assert video_len == 3 * BLOCK
@@ -96,53 +104,47 @@ def test_eviction_bounds_resident_window():
def test_branches_are_independent():
kv, st = make_state()
_prepare_and_commit(st, False, 1)
assert len(kv.window_block_ids(st.pos)) == 1
assert kv.window_block_ids(st.neg) == []
_prepare_and_commit(st, POS, 1)
assert len(kv.window_block_ids(st.adapter(POS))) == 1
assert kv.window_block_ids(st.adapter(NEG)) == []
def test_reset_clears_session_window():
kv, st = make_state(num_layers=1, window_chunks=4)
_prepare_and_commit(st, False, 2)
_prepare_and_commit(st, True, 1)
assert st._committed[False] == 2 * BLOCK and st._committed[True] == BLOCK
_prepare_and_commit(st, POS, 2)
_prepare_and_commit(st, NEG, 1)
assert st._committed[POS] == 2 * BLOCK and st._committed[NEG] == BLOCK
free_before = kv.manager.block_pool.get_num_free_blocks()
st.reset()
assert st._committed == {False: 0, True: 0}
assert st._paged_pending == {False: None, True: None}
assert kv.window_block_ids(st.pos) == []
assert kv.window_block_ids(st.neg) == []
assert st._committed == {POS: 0, NEG: 0}
assert st._paged_pending == {POS: None, NEG: None}
assert kv.window_block_ids(st.adapter(POS)) == []
assert kv.window_block_ids(st.adapter(NEG)) == []
assert kv.manager.block_pool.get_num_free_blocks() > free_before
_prepare_and_commit(st, False, 1)
assert len(kv.window_block_ids(st.pos)) == 1
_prepare_and_commit(st, POS, 1)
assert len(kv.window_block_ids(st.adapter(POS))) == 1
def test_get_cross_kv_caches_raises_before_populate():
"""Engine ownership guard: a cross-attn read before _kv_populate_cross must fail loud."""
_, st = make_state() # cross_attn_length == 0, nothing populated
for neg in (False, True):
with pytest.raises(RuntimeError, match="cross-attn read before"):
st.get_cross_kv_caches(neg)
# Guards the full AND: the populated flag alone is not enough without a pool.
st._cross_text_populated[False] = True
with pytest.raises(RuntimeError, match="cross-attn read before"):
st.get_cross_kv_caches(False)
def test_get_cross_attention_kv_raises_before_populate():
_, st = make_state(cross_attn_length=8)
for kv_branch in (POS, NEG):
with pytest.raises(RuntimeError, match="before it was populated"):
st.get_cross_attention_kv(kv_branch, "text")
def test_get_cross_kv_caches_returns_pool_dicts_when_populated():
def test_get_cross_attention_kv_returns_session_pool_dicts_when_populated():
L = 8
kv, st = make_state(num_layers=2, cross_attn_length=L)
written = []
for i in range(2):
for _ in range(2):
k = torch.randn(1, L, N_HEADS, HEAD_DIM)
v = torch.randn(1, L, N_HEADS, HEAD_DIM)
kv.write_cross_kv(i, False, k, v)
written.append((k, v))
st._cross_text_populated[False] = True
st.populate_cross_attention(POS, "text", written)
out = st.get_cross_kv_caches(False)
out = st.get_cross_attention_kv(POS, "text")
assert len(out) == 2
for i, (k, v) in enumerate(written):
assert out[i]["is_init"] is True
@@ -151,6 +153,99 @@ def test_get_cross_kv_caches_returns_pool_dicts_when_populated():
assert torch.equal(out[i]["v"], v)
def test_window_reset_can_keep_named_cross_attention_cache():
kv, st = make_state(num_layers=1, cross_attn_length=8)
k = torch.randn(1, 8, N_HEADS, HEAD_DIM)
v = torch.randn(1, 8, N_HEADS, HEAD_DIM)
st.populate_cross_attention(POS, "text", [(k, v)])
old_adapter = st.adapter(POS)
st.reset(keep_cross_attention=("text",))
assert st.adapter(POS) is not old_adapter
assert torch.equal(st.get_cross_attention_kv(POS, "text")[0]["k"], k)
assert "s1" in kv._cross_sessions
st.reset()
assert "s1" not in kv._cross_sessions
assert not st.is_cross_attention_populated(POS, "text")
def test_cross_attention_uses_logical_branch_when_local_slot_is_shared():
_, st = make_state(num_layers=2, cross_attn_length=8, shared_local_index=True)
assert st.kv_cache.num_local_kv_branches == 1
pos_layers = [
(torch.full((1, 8, N_HEADS, HEAD_DIM), 1.0), torch.full((1, 8, N_HEADS, HEAD_DIM), 2.0)) for _ in range(2)
]
neg_layers = [
(torch.full((1, 8, N_HEADS, HEAD_DIM), 3.0), torch.full((1, 8, N_HEADS, HEAD_DIM), 4.0)) for _ in range(2)
]
st.populate_cross_attention(POS, "text", pos_layers)
st.populate_cross_attention(NEG, "text", neg_layers)
for layer in st.get_cross_attention_kv(POS, "text"):
assert torch.count_nonzero(layer["k"] != 1.0) == 0
assert torch.count_nonzero(layer["v"] != 2.0) == 0
for layer in st.get_cross_attention_kv(NEG, "text"):
assert torch.count_nonzero(layer["k"] != 3.0) == 0
assert torch.count_nonzero(layer["v"] != 4.0) == 0
def test_cross_attention_population_is_transactional():
kv, st = make_state(num_layers=2, cross_attn_length=8)
original = [
(torch.full((1, 8, N_HEADS, HEAD_DIM), value), torch.full((1, 8, N_HEADS, HEAD_DIM), -value))
for value in (1.0, 2.0)
]
replacement = (
torch.full((1, 8, N_HEADS, HEAD_DIM), 9.0),
torch.full((1, 8, N_HEADS, HEAD_DIM), -9.0),
)
def interrupted_population():
yield replacement
raise RuntimeError("projection failed")
with pytest.raises(RuntimeError, match="projection failed"):
st.populate_cross_attention(POS, "text", interrupted_population())
assert not st.is_cross_attention_populated(POS, "text")
assert "s1" not in kv._cross_sessions
st.populate_cross_attention(POS, "text", original)
with pytest.raises(RuntimeError, match="projection failed"):
st.populate_cross_attention(POS, "text", interrupted_population())
out = st.get_cross_attention_kv(POS, "text")
for layer_index, layer in enumerate(out):
assert torch.equal(layer["k"], original[layer_index][0])
assert torch.equal(layer["v"], original[layer_index][1])
for invalid_layers in (original[:1], [*original, replacement]):
with pytest.raises(ValueError, match="expected 2 layers"):
st.populate_cross_attention(POS, "text", invalid_layers)
out = st.get_cross_attention_kv(POS, "text")
for layer_index, layer in enumerate(out):
assert torch.equal(layer["k"], original[layer_index][0])
assert torch.equal(layer["v"], original[layer_index][1])
invalid_shape = (torch.empty(1, 7, N_HEADS, HEAD_DIM), original[0][1])
with pytest.raises(ValueError, match="expected k/v shape"):
st.populate_cross_attention(POS, "text", [invalid_shape, original[1]])
out = st.get_cross_attention_kv(POS, "text")
assert torch.equal(out[0]["k"], original[0][0])
def test_closed_state_cannot_repopulate_cross_attention():
kv, st = make_state(num_layers=1, cross_attn_length=8)
layer = (torch.ones(1, 8, N_HEADS, HEAD_DIM), torch.ones(1, 8, N_HEADS, HEAD_DIM))
st.close()
with pytest.raises(RuntimeError, match="is closed"):
st.populate_cross_attention(POS, "text", [layer])
assert "s1" not in kv._cross_sessions
def test_kv_create_is_noop_engine_owns_allocation():
from unittest.mock import MagicMock
@@ -166,57 +261,17 @@ def test_kv_create_is_noop_engine_owns_allocation():
def test_get_kv_cache_requires_frame_aligned_seqlen():
_, st = make_state()
with pytest.raises(AssertionError, match="frame-aligned"):
st.get_kv_caches(False, seq_len=BLOCK + 1, commit_current=True)
st.get_kv_caches(POS, seq_len=BLOCK + 1, commit_current=True)
def test_prepare_one_branch_allocates_nothing_for_the_other():
"""CFG-parallel laziness: a rank prepares only the branch it runs."""
"""CFG-parallel laziness: a rank prepares only the kv_branch it runs."""
kv, st = make_state(num_layers=1, window_chunks=4)
pos_ctx = st.get_kv_caches(False, seq_len=BLOCK, commit_current=True)[0].forward_ctx
neg_ctx = st.get_kv_caches(True, seq_len=BLOCK, commit_current=True)[0].forward_ctx
pos_ctx = st.get_kv_caches(POS, seq_len=BLOCK, commit_current=True)[0].forward_ctx
neg_ctx = st.get_kv_caches(NEG, seq_len=BLOCK, commit_current=True)[0].forward_ctx
pos_ctx.prepare(device=torch.device("cpu"), action_len=0, query_len=BLOCK)
assert pos_ctx._allocated_video
assert not neg_ctx._allocated_video
assert kv.window_block_ids(st.neg) == []
def test_failed_forward_tears_down_session(monkeypatch):
"""Review follow-up (hsliuustc0106): a forward that dies partway must not
leave allocated-but-uncommitted KV behind — the session is torn down (pool
blocks freed, model-local state dropped) and the error propagates."""
from types import SimpleNamespace
from vllm_omni.diffusion.worker.diffusion_model_runner import DiffusionModelRunner
from vllm_omni.experimental.ar_diffusion.runner import ARDiffusionModelRunner
kv, st = make_state(num_layers=1, window_chunks=4)
_prepare_and_commit(st, False, 2) # session owns pool blocks
free_before_failure = kv.manager.block_pool.get_num_free_blocks()
runner = object.__new__(ARDiffusionModelRunner)
runner.kv_cache = kv
runner._ar_diffusion_states = __import__("collections").OrderedDict({"s1": st})
runner._max_ar_diffusion_states = 4
runner.pipeline = SimpleNamespace(_states={"s1": object()}, _ar_diffusion_kv_state=None)
runner.device = None
runner._perf_e2e_times = []
def boom(self, req, kv_prefetch_job=None):
raise RuntimeError("layer 17 exploded")
monkeypatch.setattr(DiffusionModelRunner, "execute_model", boom)
req = SimpleNamespace(
request_id="r0",
sampling_params=SimpleNamespace(extra_args={"session_id": "s1"}),
)
with pytest.raises(RuntimeError, match="layer 17 exploded"):
ARDiffusionModelRunner.execute_model(runner, req)
assert "s1" not in runner._ar_diffusion_states
assert "s1" not in runner.pipeline._states
assert runner.pipeline._ar_diffusion_kv_state is None
# close() freed the session's resident blocks.
assert kv.manager.block_pool.get_num_free_blocks() > free_before_failure
assert kv.window_block_ids(st.adapter(NEG)) == []
+49 -7
View File
@@ -2,6 +2,7 @@
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
from collections import OrderedDict
from types import SimpleNamespace
import pytest
@@ -14,7 +15,6 @@ pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
def _empty_pipeline() -> DreamZeroPipeline:
pipeline = DreamZeroPipeline.__new__(DreamZeroPipeline)
pipeline._states = OrderedDict()
pipeline._max_session_states = 2
return pipeline
@@ -32,17 +32,59 @@ def test_dreamzero_pipeline_state_is_session_keyed() -> None:
assert session_b.call_count == 3
def test_dreamzero_pipeline_state_lru_caps_retained_sessions() -> None:
def test_dreamzero_pipeline_state_follows_runner_lifecycle_notifications() -> None:
pipeline = _empty_pipeline()
session_a = pipeline._get_or_create_state("session-a")
pipeline._get_or_create_state("session-b")
assert pipeline._get_or_create_state("session-a") is session_a
session_b = pipeline._get_or_create_state("session-b")
pipeline.state = session_b
pipeline._get_or_create_state("session-c")
pipeline.close_ar_diffusion_session("session-a")
assert pipeline.state is session_b
pipeline.reset_ar_diffusion_session("session-b")
assert list(pipeline._states) == ["session-a", "session-c"]
assert "session-b" not in pipeline._states
assert not pipeline._states
assert pipeline.state is None
assert pipeline._get_or_create_state("session-a") is not session_a
assert pipeline._get_or_create_state("session-b") is not session_b
def test_close_session_clears_active_state_alias() -> None:
pipeline = _empty_pipeline()
session_a = pipeline._get_or_create_state("session-a")
pipeline.state = session_a
pipeline.close_ar_diffusion_session("session-a")
assert not pipeline._states
assert pipeline.state is None
def test_dreamzero_warmup_provider_builds_session_scoped_requests() -> None:
provider = SimpleNamespace(
ar_diffusion_kv_cache_spec=lambda: SimpleNamespace(window_frames=5, frames_per_block=4),
od_config=SimpleNamespace(
ar_diffusion_kv_config=None,
model_config={"policy_server_config": {"image_resolution": [8, 16]}},
),
_ar_warmup_robot_obs=DreamZeroPipeline._ar_warmup_robot_obs,
)
requests = list(DreamZeroPipeline.ar_diffusion_warmup_requests(provider, "warmup-session"))
assert [request.prompt for request in requests] == ["warmup", "warmup"]
assert all(request.sampling_params.extra_args["session_id"] == "warmup-session" for request in requests)
assert requests[0].sampling_params.extra_args["robot_obs"]["observation/exterior_image_0_left"].shape == (
8,
16,
3,
)
assert requests[1].sampling_params.extra_args["robot_obs"]["observation/exterior_image_0_left"].shape == (
4,
8,
16,
3,
)
def test_dreamzero_state_owns_no_kv_caches() -> None:
@@ -11,10 +11,12 @@ from __future__ import annotations
import copy
import json
import logging
import math
import os
import re as re_module
from collections import OrderedDict
from collections.abc import Iterable
from contextlib import contextmanager
import numpy as np
import torch
@@ -52,7 +54,14 @@ from vllm_omni.diffusion.models.dreamzero.utils import (
DEFAULT_SIGMA_SHIFT,
)
from vllm_omni.diffusion.models.schedulers.scheduling_flow_unipc_multistep import FlowUniPCMultistepScheduler
from vllm_omni.diffusion.request import OmniDiffusionRequest
from vllm_omni.diffusion.worker.request_batch import DiffusionRequestBatch
from vllm_omni.experimental.ar_diffusion.capability import (
ARDiffusionCrossAttentionKVSpec,
ARDiffusionKVBranchSpec,
ARDiffusionKVCacheSpec,
)
from vllm_omni.inputs.data import OmniDiffusionSamplingParams
logger = logging.getLogger(__name__)
MAX_DREAMZERO_SESSIONS = 64
@@ -92,18 +101,128 @@ class DreamZeroPipeline(nn.Module, CFGParallelMixin):
"""DreamZero world model pipeline.
Multi-output: predict_noise() returns (video_pred, action_pred).
CFG: video gets standard CFG, action takes positive branch only.
CFG: video gets standard CFG, action takes the positive KV branch only.
KV is managed by the AR-Diffusion engine: ``self._ar_diffusion_kv_state`` is set by the
runner before ``forward()`` and the pipeline routes all KV access (get / update /
commit / reset) through the pool-backed state. Purely duck-typed (no engine import).
KV is managed by the AR-Diffusion engine through the explicit capability
methods below. The runner binds one session state only for ``forward()``.
"""
_ar_diffusion_kv_state = None # set by the runner before each forward
_POSITIVE_BRANCH = "positive"
_NEGATIVE_BRANCH = "negative"
_ar_diffusion_kv_state = None
state: DreamZeroState | None
def ar_diffusion_kv_cache_spec(self) -> ARDiffusionKVCacheSpec:
"""Describe DreamZero's local KV geometry to the generic runner."""
transformer = self.transformer
frame_tokens = int(transformer.frame_seqlen)
max_attention_tokens = int(transformer.blocks[0].self_attn.max_attention_size)
cfg_world = int(get_classifier_free_guidance_world_size())
negative_local_index = 0 if cfg_world >= 2 else 1
cross_attention = [
ARDiffusionCrossAttentionKVSpec("text", int(transformer.text_len)),
]
if transformer.model_type == "i2v":
cross_attention.append(ARDiffusionCrossAttentionKVSpec("image", 257))
return ARDiffusionKVCacheSpec(
num_layers=int(transformer.num_layers),
num_kv_heads=int(transformer.blocks[0].self_attn.tp_num_heads),
head_size=int(transformer.dim // transformer.num_heads),
tokens_per_frame=frame_tokens,
frames_per_block=int(transformer.num_frame_per_block),
window_frames=max_attention_tokens // frame_tokens,
kv_branches=(
ARDiffusionKVBranchSpec(self._POSITIVE_BRANCH, 0),
ARDiffusionKVBranchSpec(self._NEGATIVE_BRANCH, negative_local_index),
),
session_capacity=MAX_DREAMZERO_SESSIONS,
cross_attention=tuple(cross_attention),
max_scratch_tokens_per_branch=int(transformer.num_action_per_block + transformer.num_state_per_block),
)
@contextmanager
def bind_ar_diffusion_state(self, session_id, state):
"""Bind runner-owned KV only for the duration of one forward."""
if self._ar_diffusion_kv_state is not None:
raise RuntimeError("DreamZero AR-Diffusion state is already bound")
if state.session_id != session_id:
raise ValueError(f"DreamZero bound session mismatch: {state.session_id!r} != {session_id!r}")
self._ar_diffusion_kv_state = state
try:
yield
finally:
self._ar_diffusion_kv_state = None
def reset_ar_diffusion_session(self, session_id: str) -> None:
"""Reset DreamZero-owned state after the runner releases session KV."""
self._drop_ar_diffusion_session_state(session_id)
def close_ar_diffusion_session(self, session_id: str) -> None:
"""Drop DreamZero-owned state after close, eviction, or failed forward."""
self._drop_ar_diffusion_session_state(session_id)
def _drop_ar_diffusion_session_state(self, session_id: str) -> None:
"""Remove model state and clear the compatibility alias when it points there."""
removed = self._states.pop(str(session_id or "default"), None)
if removed is not None and getattr(self, "state", None) is removed:
self.state = None
@staticmethod
def _ar_warmup_robot_obs(height: int, width: int, n_frames: int, session_id: str) -> dict:
image = (
np.zeros((height, width, 3), dtype=np.uint8)
if n_frames == 1
else np.zeros((n_frames, height, width, 3), dtype=np.uint8)
)
return {
"observation/exterior_image_0_left": image,
"observation/exterior_image_1_left": image,
"observation/wrist_image_left": image,
"observation/joint_position": np.zeros(7, dtype=np.float32),
"observation/cartesian_position": np.zeros(6, dtype=np.float32),
"observation/gripper_position": np.zeros(1, dtype=np.float32),
"prompt": "warmup",
"session_id": session_id,
}
def ar_diffusion_warmup_requests(self, session_id: str) -> Iterable[OmniDiffusionRequest]:
"""Yield DreamZero-valid requests covering each resident window shape."""
spec = self.ar_diffusion_kv_cache_spec()
n_forwards = 1 + math.ceil(max(0, spec.window_frames - 1) / spec.frames_per_block)
raw_kv_config = getattr(self.od_config, "ar_diffusion_kv_config", None)
if raw_kv_config is None and isinstance(self.od_config.model_config, dict):
raw_kv_config = self.od_config.model_config.get("ar_diffusion_kv_config")
capture_reset = (
bool(raw_kv_config.get("warmup_capture_reset", False))
if isinstance(raw_kv_config, dict)
else bool(getattr(raw_kv_config, "warmup_capture_reset", False))
)
if capture_reset:
n_forwards += 1
policy_config = self.od_config.model_config.get("policy_server_config", {})
height, width = (int(value) for value in policy_config.get("image_resolution", [180, 320]))
for index in range(n_forwards):
n_frames = 1 if index == 0 else 4
robot_obs = self._ar_warmup_robot_obs(height, width, n_frames, session_id)
sampling_params = OmniDiffusionSamplingParams(
extra_args={
"reset": index == 0,
"session_id": session_id,
"robot_obs": robot_obs,
}
)
yield OmniDiffusionRequest(
prompt="warmup",
sampling_params=sampling_params,
request_id=f"ardiffusion-warmup-{index}",
)
def _ar_branch(self, is_negative: bool) -> str:
return self._NEGATIVE_BRANCH if is_negative else self._POSITIVE_BRANCH
def _kv_get(self, state, is_negative, seq_len=None, update_kv_cache=False):
return self._ar_diffusion_kv_state.get_kv_caches(
is_negative,
self._ar_branch(is_negative),
seq_len=seq_len,
commit_current=update_kv_cache,
)
@@ -115,11 +234,18 @@ class DreamZeroPipeline(nn.Module, CFGParallelMixin):
return
def _kv_commit(self, is_negative: bool):
self._ar_diffusion_kv_state.commit_paged_context(is_negative)
self._ar_diffusion_kv_state.commit_paged_context(self._ar_branch(is_negative))
def _kv_get_cross(self, state, is_negative):
"""Cross-attn cache from the engine pool (text k/v + I2V image k_img/v_img)."""
return self._ar_diffusion_kv_state.get_cross_kv_caches(is_negative)
kv_branch = self._ar_branch(is_negative)
text_caches = self._ar_diffusion_kv_state.get_cross_attention_kv(kv_branch, "text")
if "image" in self._ar_diffusion_kv_state.kv_cache.cross_attention_lengths:
image_caches = self._ar_diffusion_kv_state.get_cross_attention_kv(kv_branch, "image")
for text_cache, image_cache in zip(text_caches, image_caches, strict=True):
text_cache["k_img"] = image_cache["k"]
text_cache["v_img"] = image_cache["v"]
return text_caches
def _kv_populate_cross(self, context: torch.Tensor, clip_feature, is_negative: bool) -> None:
"""Eagerly project cross-attn K/V for all layers into the AR-Diffusion pool.
@@ -133,32 +259,41 @@ class DreamZeroPipeline(nn.Module, CFGParallelMixin):
after the image is encoded so ``clip_feature`` is available.
"""
s = self._ar_diffusion_kv_state
need_text = not s._cross_text_populated.get(is_negative, False)
kv_branch = self._ar_branch(is_negative)
need_text = not s.is_cross_attention_populated(kv_branch, "text")
need_img = (
clip_feature is not None
and getattr(self.transformer, "model_type", "t2v") == "i2v"
and not s._cross_img_populated.get(is_negative, False)
and self.transformer.model_type == "i2v"
and not s.is_cross_attention_populated(kv_branch, "image")
)
if not need_text and not need_img:
return
projected = self.transformer.text_embedding(context) if need_text else None
img_ctx = self.transformer.img_emb(clip_feature) if need_img else None
for i, block in enumerate(self.transformer.blocks):
ca = block.cross_attn
n, d = ca.tp_num_heads, ca.head_dim
k = v = None
if projected is not None:
k = ca.norm_k(ca.k(projected)).unflatten(2, (n, d))
v = ca.v(projected).unflatten(2, (n, d))
k_img = v_img = None
if img_ctx is not None:
k_img = ca.norm_k_img(ca.k_img(img_ctx)).unflatten(2, (n, d))
v_img = ca.v_img(img_ctx).unflatten(2, (n, d))
s.kv_cache.write_cross_kv(i, is_negative, k, v, k_img, v_img)
if need_text:
s._cross_text_populated[is_negative] = True
if need_img:
s._cross_img_populated[is_negative] = True
if projected is not None:
def text_layer_kv():
for block in self.transformer.blocks:
ca = block.cross_attn
n, d = ca.tp_num_heads, ca.head_dim
yield (
ca.norm_k(ca.k(projected)).unflatten(2, (n, d)),
ca.v(projected).unflatten(2, (n, d)),
)
s.populate_cross_attention(kv_branch, "text", text_layer_kv())
if img_ctx is not None:
def image_layer_kv():
for block in self.transformer.blocks:
ca = block.cross_attn
n, d = ca.tp_num_heads, ca.head_dim
yield (
ca.norm_k_img(ca.k_img(img_ctx)).unflatten(2, (n, d)),
ca.v_img(img_ctx).unflatten(2, (n, d)),
)
s.populate_cross_attention(kv_branch, "image", image_layer_kv())
logger.info(
"AR-Diffusion CROSS POPULATE [%s]: %d layers, text=%s img=%s",
"neg" if is_negative else "pos",
@@ -179,7 +314,8 @@ class DreamZeroPipeline(nn.Module, CFGParallelMixin):
image half repopulates on the restart forward.
"""
state.reset(clear_video_latents=clear_video_latents)
self._ar_diffusion_kv_state.reset(keep_cross_text=not clear_video_latents)
keep_cross = ("text",) if not clear_video_latents else ()
self._ar_diffusion_kv_state.reset(keep_cross_attention=keep_cross)
def __init__(self, *, od_config: OmniDiffusionConfig, prefix: str = "") -> None:
"""Initialize pipeline components.
@@ -298,7 +434,6 @@ class DreamZeroPipeline(nn.Module, CFGParallelMixin):
)
self._states: OrderedDict[str, DreamZeroState] = OrderedDict()
self._max_session_states = MAX_DREAMZERO_SESSIONS
self.state = self._get_or_create_state("default")
# DiT step cache is configured by StepCacheBackend
@@ -373,9 +508,6 @@ class DreamZeroPipeline(nn.Module, CFGParallelMixin):
if state is None:
state = DreamZeroState()
self._states[session_key] = state
max_states = getattr(self, "_max_session_states", MAX_DREAMZERO_SESSIONS)
while len(self._states) > max_states:
self._states.popitem(last=False)
else:
self._states.move_to_end(session_key)
return state
@@ -536,6 +668,10 @@ class DreamZeroPipeline(nn.Module, CFGParallelMixin):
if not torch.cuda.is_available():
return
state = self.state
if state is None:
state = self._get_or_create_state("default")
self.state = state
device = next(self.text_encoder.parameters()).device
with torch.inference_mode():
try:
@@ -547,18 +683,18 @@ class DreamZeroPipeline(nn.Module, CFGParallelMixin):
try:
image = torch.zeros(1, 1, 3, 180, 320, dtype=torch.bfloat16, device=device)
self._encode_image(image, self.num_frames, 180, 320, state=self.state)
self._encode_image(image, self.num_frames, 180, 320, state=state)
except Exception as exc:
logger.warning("DreamZero compile warmup (image_encoder) skipped: %s", exc)
try:
self.state.reset_vae_encoder_stream()
state.reset_vae_encoder_stream()
dummy_video = torch.zeros(1, 3, 1, 180, 320, dtype=torch.bfloat16, device=device)
self._vae_stream_seed(self.state, dummy_video[:, :, :1])
self._vae_stream_seed(state, dummy_video[:, :, :1])
for _ in range(4):
self._vae_stream_append_frame(self.state, dummy_video[:, :, :1])
self._vae_stream_append_frame(state, dummy_video[:, :, :1])
self._vae_stream_get_observation_latents(
self.state,
state,
self.num_frame_per_block,
dtype=torch.bfloat16,
)
@@ -1034,7 +1170,7 @@ class DreamZeroPipeline(nn.Module, CFGParallelMixin):
state_features = kwargs.get("state_features")
embodiment_id = kwargs.get("embodiment_id")
# Shared kwargs for predict_noise (both cond & uncond branches)
# Shared kwargs for predict_noise (both conditional and unconditional branches)
common_kwargs = dict(
seq_len=seq_len,
current_start_frame=state.current_start_frame,
@@ -1236,21 +1372,14 @@ class DreamZeroPipeline(nn.Module, CFGParallelMixin):
text_tokens = text_inputs["input_ids"].to(device)
attention_mask = text_inputs["attention_mask"].to(device)
# Explicit reset from OpenPI serving is carried by `extra_args["reset"]`
# on the next inference request after websocket reset/session switch.
if extra_args.get("reset", False):
# Explicit request reset is handled by ARDiffusionModelRunner before it
# binds this forward. Model-detected prompt/window resets still happen
# here because they depend on tokenized DreamZero state.
reset_reason = state.reset_reason(text_tokens, 0, self.transformer.local_attn_size)
if reset_reason == "session":
self._kv_reset(state)
else:
# Auto-reset based on model state (before accumulation). Both a "session"
# reset and an "inference" (window-boundary) reset clear the model-local
# KV window, so route both through _kv_reset to drop the AR-Diffusion pool window
# too and stay parity-exact; "inference" keeps the accumulated video
# latents (reset_inference_state == reset(clear_video_latents=False)).
reset_reason = state.reset_reason(text_tokens, 0, self.transformer.local_attn_size)
if reset_reason == "session":
self._kv_reset(state)
elif reset_reason == "inference":
self._kv_reset(state, clear_video_latents=False)
elif reset_reason == "inference":
self._kv_reset(state, clear_video_latents=False)
state.language = text_tokens
# Frame accumulation: stitched single frame -> multi-frame video
@@ -1280,7 +1409,7 @@ class DreamZeroPipeline(nn.Module, CFGParallelMixin):
if state.prompt_embeds is None:
state.prompt_embeds = self._encode_text(text_tokens, attention_mask)
prompt_embeds = state.prompt_embeds
# Negative prompt for CFG uncond branch (model constant)
# Negative prompt for the unconditional CFG branch (model constant)
negative_prompt_embeds = None
if self.cfg_scale > 1.0:
if self._negative_prompt_embeds_cache is None:
@@ -2,11 +2,24 @@
"""AR-Diffusion Engine (AR-Diffusion).
The AR-Diffusion engine: a ``DiffusionEngine`` subclass that adds engine-level KV
cache management for autoregressive / chunked "world-model" diffusion models
(DreamZero and the AR-DiT family). Selected via
cache management for autoregressive / chunked diffusion models. Selected via
``OmniDiffusionConfig.engine_backend = "ar_diffusion"``.
"""
from vllm_omni.experimental.ar_diffusion.capability import (
ARDiffusionCrossAttentionKVSpec,
ARDiffusionKVBranchSpec,
ARDiffusionKVCacheSpec,
SupportsARDiffusionPipeline,
SupportsARDiffusionWarmup,
)
from vllm_omni.experimental.ar_diffusion.engine import ARDiffusionEngine
__all__ = ["ARDiffusionEngine"]
__all__ = [
"ARDiffusionKVBranchSpec",
"ARDiffusionCrossAttentionKVSpec",
"ARDiffusionEngine",
"ARDiffusionKVCacheSpec",
"SupportsARDiffusionPipeline",
"SupportsARDiffusionWarmup",
]
@@ -0,0 +1,169 @@
# SPDX-License-Identifier: Apache-2.0
"""Typed pipeline contract for the model-neutral AR-Diffusion runtime."""
from __future__ import annotations
from collections.abc import Iterable
from contextlib import AbstractContextManager
from dataclasses import dataclass
from typing import TYPE_CHECKING, Protocol, runtime_checkable
if TYPE_CHECKING:
from vllm_omni.diffusion.request import OmniDiffusionRequest
from vllm_omni.experimental.ar_diffusion.kv_cache.state import ARDiffusionKVState
@dataclass(frozen=True)
class ARDiffusionKVBranchSpec:
"""One logical model KV branch and its worker-local storage slot.
Distinct KV branches may share ``local_index`` when distributed execution
guarantees that at most one of them runs on a worker. For example, two CFG
branches use indices ``0, 1`` without CFG parallelism and may both use index
``0`` when each worker executes only one branch.
"""
name: str
local_index: int
def __post_init__(self) -> None:
if not self.name:
raise ValueError("AR-Diffusion KV branch names must be non-empty")
if self.local_index < 0:
raise ValueError(f"AR-Diffusion KV branch local_index must be non-negative, got {self.local_index}")
@dataclass(frozen=True)
class ARDiffusionCrossAttentionKVSpec:
"""A fixed-length, per-layer cross-attention KV allocation."""
name: str
num_tokens: int
def __post_init__(self) -> None:
if not self.name:
raise ValueError("AR-Diffusion cross-attention cache names must be non-empty")
if self.num_tokens <= 0:
raise ValueError(f"AR-Diffusion cross-attention num_tokens must be positive, got {self.num_tokens}")
@dataclass(frozen=True)
class ARDiffusionKVCacheSpec:
"""KV geometry and session policy supplied by an AR-Diffusion pipeline.
``tokens_per_frame`` is also the paged KV block size. A model forward may
commit ``frames_per_block`` such blocks at once. ``window_frames`` is the
recent sliding tail and ``sink_frames`` is the separately retained prefix;
both are expressed in the same frame/block unit.
``max_scratch_tokens_per_branch`` is the maximum non-video KV (for example,
action/state registers) that must coexist with an uncommitted video block.
``session_capacity`` is the pipeline's upper bound; a runner may retain
fewer sessions when its current memory and execution policy is stricter.
The runner owns all pool allocations and session lifetime. The pipeline
owns its non-KV model state and receives reset/close notifications through
:class:`SupportsARDiffusionPipeline`.
"""
num_layers: int
num_kv_heads: int
head_size: int
tokens_per_frame: int
frames_per_block: int
window_frames: int
kv_branches: tuple[ARDiffusionKVBranchSpec, ...]
session_capacity: int
sink_frames: int = 0
reset_at_boundary: bool = False
cross_attention: tuple[ARDiffusionCrossAttentionKVSpec, ...] = ()
max_model_len: int = 1 << 20
max_scratch_tokens_per_branch: int = 0
def __post_init__(self) -> None:
positive_fields = {
"num_layers": self.num_layers,
"num_kv_heads": self.num_kv_heads,
"head_size": self.head_size,
"tokens_per_frame": self.tokens_per_frame,
"frames_per_block": self.frames_per_block,
"window_frames": self.window_frames,
"session_capacity": self.session_capacity,
"max_model_len": self.max_model_len,
}
for name, value in positive_fields.items():
if value <= 0:
raise ValueError(f"AR-Diffusion {name} must be positive, got {value}")
if self.sink_frames < 0:
raise ValueError(f"AR-Diffusion sink_frames must be non-negative, got {self.sink_frames}")
if self.max_scratch_tokens_per_branch < 0:
raise ValueError(
"AR-Diffusion max_scratch_tokens_per_branch must be non-negative, "
f"got {self.max_scratch_tokens_per_branch}"
)
if not self.kv_branches:
raise ValueError("AR-Diffusion requires at least one KV branch")
kv_branch_names = [kv_branch.name for kv_branch in self.kv_branches]
if len(kv_branch_names) != len(set(kv_branch_names)):
raise ValueError(f"AR-Diffusion KV branch names must be unique, got {kv_branch_names}")
local_indices = {kv_branch.local_index for kv_branch in self.kv_branches}
if local_indices != set(range(max(local_indices) + 1)):
raise ValueError(
f"AR-Diffusion KV branch local_index values must be contiguous from zero, got {sorted(local_indices)}"
)
cross_names = [cache.name for cache in self.cross_attention]
if len(cross_names) != len(set(cross_names)):
raise ValueError(f"AR-Diffusion cross-attention cache names must be unique, got {cross_names}")
@property
def num_local_kv_branches(self) -> int:
"""Number of KV branch storage slots allocated on this worker."""
return max(kv_branch.local_index for kv_branch in self.kv_branches) + 1
@property
def cross_attention_lengths(self) -> dict[str, int]:
return {cache.name: cache.num_tokens for cache in self.cross_attention}
@runtime_checkable
class SupportsARDiffusionPipeline(Protocol):
"""Required pipeline capability for :class:`ARDiffusionModelRunner`.
A session begins when the runner first sees its ``session_id`` and persists
across requests until reset, explicit close, LRU eviction, or a failed
forward. ``bind_ar_diffusion_state`` exposes the runner-owned KV state only
for the duration of one request. The pipeline must not retain the state
after the context exits.
"""
def ar_diffusion_kv_cache_spec(self) -> ARDiffusionKVCacheSpec:
"""Return immutable KV geometry and session policy for this worker."""
...
def bind_ar_diffusion_state(
self,
session_id: str,
state: ARDiffusionKVState,
) -> AbstractContextManager[None]:
"""Bind ``state`` to model execution for one request."""
...
def reset_ar_diffusion_session(self, session_id: str) -> None:
"""Reset model-owned state for ``session_id`` after KV release."""
...
def close_ar_diffusion_session(self, session_id: str) -> None:
"""Drop model-owned state after close, eviction, or failed forward."""
...
@runtime_checkable
class SupportsARDiffusionWarmup(Protocol):
"""Optional provider of model-valid warmup requests.
Pipelines without this capability are loaded without an AR rollout warmup.
"""
def ar_diffusion_warmup_requests(self, session_id: str) -> Iterable[OmniDiffusionRequest]:
"""Yield requests for compiled shapes, each carrying ``session_id``."""
...
@@ -20,8 +20,7 @@ class ARDiffusionEngine(DiffusionEngine):
(world models, AR-DiT) that materialize persistent attention KV. It reuses
vLLM's paged KV stack (``KVCacheManager`` / ``BlockPool`` / ``BlockTables``)
as a library, driven from the engine rather than hand-rolled inside each
model. See ``BDE_doc/diffusion_kv_cache_management.md`` for the design and
``BDE_doc/dreamzero_kv_phase1_plan.md`` for the rollout.
model.
It is selected per model via ``OmniDiffusionConfig.engine_backend = "ar_diffusion"``
(resolved by :meth:`DiffusionEngine.make_engine`), so models that do not opt
@@ -32,7 +31,7 @@ class ARDiffusionEngine(DiffusionEngine):
actual KV *body* is :class:`~vllm_omni.experimental.ar_diffusion.kv_cache.manager.ARDiffusionKVCache`, owned
by :class:`~vllm_omni.experimental.ar_diffusion.runner.ARDiffusionModelRunner`. ``ARDiffusionEngine`` itself is the
thin selection / injection seam; it wires the AR-Diffusion executor → worker → runner
so DreamZero's rollout runs against the runner-owned KV cache.
so capable pipelines run against runner-owned KV session state.
"""
#: Workers of this engine build the AR-Diffusion runner (unless the config
@@ -3,7 +3,7 @@
Thin glue over vLLM's paged KV stack (``KVCacheManager`` / ``BlockPool`` /
``SlidingWindowManager``), used by the AR-Diffusion Engine to manage KV for
AR-diffusion models. See ``BDE_doc/dreamzero_kv_phase1_plan.md``.
AR-diffusion models.
Layout: ``config`` (public knob) · ``paged`` (engine-generic paging mechanics +
chunk-window eviction spec) · ``manager`` (the ARDiffusionKVCache orchestrator + its
@@ -15,14 +15,13 @@ class ARDiffusionKVConfig:
"""
enable: bool = False
# Persistent KV tokens materialized per AR chunk. For DreamZero this is
# ``num_frame_per_block * frame_seqlen``.
# Persistent KV tokens materialized per paged cache block.
chunk_size: int = 0
# Resident window in chunks. ``None`` means full attention (no eviction).
window_chunks: int | None = None
# Protected leading chunks (attention sink); never evicted.
sink_chunks: int = 0
# DreamZero-style window reset vs. VGGT-style sliding replace.
# Boundary reset vs. sliding replacement.
reset_at_boundary: bool = False
# Fraction of free device memory budgeted for the AR-Diffusion KV pool.
gpu_memory_fraction: float = 0.1
@@ -1,20 +1,19 @@
# SPDX-License-Identifier: Apache-2.0
"""ARDiffusionKVCache — the engine-level KV cache orchestrator for one AR-Diffusion model.
"""Engine-level KV cache orchestration for AR-Diffusion models.
This is the *body* of AR-Diffusion's KV management: it owns a vLLM ``KVCacheManager`` (a
single chunk-window group) and the per-request adapter lifecycle, and exposes the
per-chunk operations a rollout needs — allocate, slot mapping, commit, window
lookup, free. It lives in the model runner (worker / GPU side), co-located with
the model and the KV tensors; the DreamZero pipeline calls these methods during a
rollout. The main-process ``ARDiffusionEngine`` only selects the engine and is otherwise
thin.
the model and the KV tensors. The main-process ``ARDiffusionEngine`` only
selects the engine and is otherwise thin.
"""
from __future__ import annotations
import inspect
import os
from collections.abc import Sequence
from collections.abc import Collection, Iterable, Sequence
import torch
from vllm.logger import init_logger
@@ -27,6 +26,7 @@ from vllm.v1.kv_cache_interface import (
)
from vllm.v1.request import RequestStatus
from vllm_omni.experimental.ar_diffusion.capability import ARDiffusionKVBranchSpec
from vllm_omni.experimental.ar_diffusion.kv_cache.config import ARDiffusionKVConfig
from vllm_omni.experimental.ar_diffusion.kv_cache.paged import (
ChunkWindowSpec,
@@ -48,7 +48,7 @@ class ARDiffusionRequestAdapter:
real ``KVCacheManager`` against this adapter so the surface cannot silently
drift across vLLM versions.
A AR-Diffusion request advances one *chunk* at a time: ``allocate_slots`` is called
An AR-Diffusion request advances one *chunk* at a time: ``allocate_slots`` is called
once per chunk and ``num_computed_tokens`` advances only when a chunk is
committed (:meth:`on_chunk_committed`), so the ``T`` denoise steps of a chunk
reuse the same slots.
@@ -145,7 +145,7 @@ def build_kv_manager(
class ARDiffusionKVCache:
"""Owns the paged KV pool + per-request lifecycle for a AR-Diffusion model.
"""Own the paged KV pool and KV-branch-local storage for one model.
Build once per loaded model (dimensions known); then per request:
``begin_request`` → per chunk (``allocate_chunk`` → ``chunk_write_slots`` →
@@ -163,11 +163,12 @@ class ARDiffusionKVCache:
block_size: int,
max_model_len: int,
available_bytes: int,
cross_attn_length: int = 0,
cross_attn_img_length: int = 0,
kv_branches: tuple[ARDiffusionKVBranchSpec, ...],
session_capacity: int,
cross_attention_lengths: dict[str, int] | None = None,
device: torch.device | None = None,
local_branches: int = 2,
num_frame_per_block: int = 1,
frames_per_block: int = 1,
max_scratch_tokens_per_branch: int = 0,
) -> None:
if not config.enable:
raise ValueError("ARDiffusionKVCache built with a disabled ARDiffusionKVConfig")
@@ -175,77 +176,63 @@ class ARDiffusionKVCache:
raise ValueError("Phase 1 requires a bounded window (window_chunks)")
if config.chunk_size <= 0:
raise ValueError("ARDiffusionKVConfig.chunk_size must be set (> 0)")
if local_branches not in (1, 2):
raise ValueError(f"local_branches must be 1 or 2, got {local_branches}")
if not kv_branches:
raise ValueError("ARDiffusionKVCache requires at least one KV branch")
if session_capacity <= 0:
raise ValueError(f"session_capacity must be positive, got {session_capacity}")
kv_branch_names = [kv_branch.name for kv_branch in kv_branches]
if len(kv_branch_names) != len(set(kv_branch_names)):
raise ValueError(f"ARDiffusionKVCache KV branch names must be unique, got {kv_branch_names}")
local_indices = {kv_branch.local_index for kv_branch in kv_branches}
if local_indices != set(range(max(local_indices) + 1)):
raise ValueError(
"ARDiffusionKVCache KV branch local_index values must be contiguous from zero, "
f"got {sorted(local_indices)}"
)
self.config = config
# How many CFG branches allocate from THIS rank's pool. Under
# CFG-parallel x2 each rank executes exactly one branch (rank0 pos,
# rank1 neg) and the other branch's lazy contexts never allocate, so
# sizing for both would leave ~half the pool as idle capacity. A
# single-process run (TP1 / offline) executes both branches -> 2.
self.local_branches = local_branches
self.num_frame_per_block = max(1, int(num_frame_per_block))
self.kv_branches = kv_branches
self.session_capacity = session_capacity
self._kv_branch_local_indices = {kv_branch.name: kv_branch.local_index for kv_branch in kv_branches}
self.num_local_kv_branches = max(local_indices) + 1
if frames_per_block <= 0:
raise ValueError(f"frames_per_block must be positive, got {frames_per_block}")
if max_scratch_tokens_per_branch < 0:
raise ValueError(f"max_scratch_tokens_per_branch must be non-negative, got {max_scratch_tokens_per_branch}")
self.frames_per_block = int(frames_per_block)
self.block_size = block_size
self.num_layers = num_layers
self.num_kv_heads = num_kv_heads
self.head_size = head_size
self.dtype = dtype
self.cross_attn_length = cross_attn_length
self.cross_attention_lengths = dict(cross_attention_lengths or {})
invalid_cross = {name: length for name, length in self.cross_attention_lengths.items() if length <= 0}
if invalid_cross:
raise ValueError(f"cross_attention_lengths must be positive, got {invalid_cross}")
self.device = device or torch.device("cpu")
self._allocate_tensors = device is not None
self._adapters: dict[str, ARDiffusionRequestAdapter] = {}
# -- cross-attention pool (static, one-time-fill) -----------------------
# Cross-attn KV is computed once from the text encoder and never changes.
# Allocate a per-layer contiguous tensor for each branch: (2, text_len,
# kv_heads, head_dim) where dim-0 = [pos, neg]. Allocated separately
# from the self-attn block pool (static, small — ~400 MiB for
# DreamZero I2V). Both pools draw from the same GPU free memory but
# the cross-attn pool is sized directly rather than via the block pool
# budget (no eviction/chunk lifecycle needed).
self.cross_attn_img_length = cross_attn_img_length
self._cross_k: list[torch.Tensor] = []
self._cross_v: list[torch.Tensor] = []
# I2V image-token cross-attn pool. Like the text k/v, the image-token
# k_img/v_img are session-invariant (the conditioning image doesn't change),
# so they are cached once and read every denoise step — see
# WanI2VCrossAttention (#4154 caches these model-side too). Empty for T2V.
self._cross_k_img: list[torch.Tensor] = []
self._cross_v_img: list[torch.Tensor] = []
# Named cross-attention K/V is allocated lazily per session and released
# with that session. Reserve its worst-case capacity when sizing the
# paged self-attention pool so the two stores share one memory budget.
self._cross_sessions: dict[
str,
dict[str, dict[str, tuple[list[torch.Tensor], list[torch.Tensor]]]],
] = {}
# Bytes the cross-attn pools consume directly (K+V, pos+neg, all layers).
# Deducted from the self-attn paged-pool budget below so the two
# allocations together stay within the GPU free-memory budget.
def _cross_pool_bytes(length: int) -> int:
return 2 * 2 * length * num_kv_heads * head_size * dtype.itemsize * num_layers
return int(2 * len(self.kv_branches) * length * num_kv_heads * head_size * dtype.itemsize * num_layers)
cross_total_bytes = 0
if device is not None and cross_attn_length > 0:
cross_shape = (2, cross_attn_length, num_kv_heads, head_size)
cross_bytes = _cross_pool_bytes(cross_attn_length)
cross_total_bytes += cross_bytes
for _ in range(num_layers):
self._cross_k.append(torch.empty(cross_shape, dtype=dtype, device=device))
self._cross_v.append(torch.empty(cross_shape, dtype=dtype, device=device))
cross_bytes_per_session = sum(_cross_pool_bytes(length) for length in self.cross_attention_lengths.values())
cross_total_bytes = cross_bytes_per_session * session_capacity
if cross_total_bytes:
_log.info(
"AR-Diffusion cross-attn pool: %d layers × (%d tok × %d heads × %d) = %.1f MiB",
num_layers,
cross_attn_length,
num_kv_heads,
head_size,
cross_bytes / (1024 * 1024),
"AR-Diffusion cross-attn reservation: %.1f MiB/session × %d sessions = %.1f MiB",
cross_bytes_per_session / (1024 * 1024),
session_capacity,
cross_total_bytes / (1024 * 1024),
)
if cross_attn_img_length > 0:
img_shape = (2, cross_attn_img_length, num_kv_heads, head_size)
cross_total_bytes += _cross_pool_bytes(cross_attn_img_length)
for _ in range(num_layers):
self._cross_k_img.append(torch.empty(img_shape, dtype=dtype, device=device))
self._cross_v_img.append(torch.empty(img_shape, dtype=dtype, device=device))
_log.info(
"AR-Diffusion cross-attn IMG pool: %d layers × %d img-tok (I2V)",
num_layers,
cross_attn_img_length,
)
self.spec = ChunkWindowSpec(
block_size=block_size,
@@ -260,46 +247,51 @@ class ARDiffusionKVCache:
)
# Each pool block spans all layers' K/V, so size against the per-layer
# page size times the layer count.
# Size the self-attn pool against the memory left after the cross-attn
# pools (allocated above), so cross + self-attn stays within the budget.
# Size the self-attn pool against memory reserved for the maximum number
# of lazily allocated cross-attention sessions.
num_blocks = compute_num_blocks(
max(0, available_bytes - cross_total_bytes),
config.gpu_memory_fraction,
self.spec.page_size_bytes * num_layers,
)
# Floor: one forward needs the resident window plus the in-flight chunk
# (num_frame_per_block frame-blocks) for every branch THIS rank runs,
# (frames_per_block frame-blocks) for every KV branch THIS rank runs,
# with a little eviction-transient headroom. The memory-fraction
# heuristic can under-size this once block_size grows — e.g. frame-granular
# paging at the true frame_seqlen makes each block larger and the pool
# fewer-blocks — so guarantee the minimum the rollout cannot run without,
# otherwise allocate_chunk hits an exhausted pool mid-forward.
min_blocks = self.local_branches * (config.window_chunks + self.num_frame_per_block) + 2
resident_blocks = config.sink_chunks + config.window_chunks
min_blocks = self.num_local_kv_branches * (resident_blocks + self.frames_per_block) + 2
if num_blocks < min_blocks:
_log.warning(
"AR-Diffusion KV pool: memory-fraction sizing gave %d blocks; raising to the %d-block "
"floor (%d local CFG branch(es) x (window_chunks=%d + num_frame_per_block=%d) + 2 headroom)",
"floor (%d local KV branch(es) x (sink_chunks=%d + window_chunks=%d "
"+ frames_per_block=%d) + 2 headroom)",
num_blocks,
min_blocks,
self.local_branches,
self.num_local_kv_branches,
config.sink_chunks,
config.window_chunks,
self.num_frame_per_block,
self.frames_per_block,
)
num_blocks = min_blocks
layer_names = [f"ar_diffusion.layer.{i}" for i in range(num_layers)]
self.manager = build_kv_manager(self.spec, layer_names, num_blocks, max_model_len)
self.managed_num_blocks = num_blocks
self.num_blocks = num_blocks
# Scratch blocks are outside KVCacheManager ownership. They hold current
# denoise-step video KV when update_kv_cache=False and action/state KV in
# both modes; they are reused every forward and never committed.
# Default 4: measured high-water is num_frame_per_block video blocks (2)
# + 1 action block per branch per forward, +1 margin.
scratch_per_branch = int(os.environ.get("AR_DIFFUSION_KV_SCRATCH_BLOCKS_PER_BRANCH", "4"))
if scratch_per_branch <= 0:
raise ValueError("AR_DIFFUSION_KV_SCRATCH_BLOCKS_PER_BRANCH must be positive")
self.scratch_blocks_per_branch = scratch_per_branch
self.scratch_num_blocks = self.local_branches * scratch_per_branch
# Scratch blocks are outside KVCacheManager ownership. A non-committing
# forward needs one block per current frame plus space for any
# model-declared action/state tokens that coexist with video KV.
declared_scratch_blocks = (max_scratch_tokens_per_branch + block_size - 1) // block_size
minimum_scratch_blocks = self.frames_per_block + declared_scratch_blocks
override = os.environ.get("AR_DIFFUSION_KV_SCRATCH_BLOCKS_PER_BRANCH")
override_blocks = int(override) if override is not None else 0
if override_blocks < 0:
raise ValueError("AR_DIFFUSION_KV_SCRATCH_BLOCKS_PER_BRANCH must be non-negative")
scratch_per_kv_branch = max(minimum_scratch_blocks, override_blocks)
self.scratch_blocks_per_kv_branch = scratch_per_kv_branch
self.scratch_num_blocks = self.num_local_kv_branches * scratch_per_kv_branch
self.num_blocks_total = self.managed_num_blocks + self.scratch_num_blocks
self.null_block_id = self.manager.block_pool.null_block.block_id
@@ -322,49 +314,147 @@ class ARDiffusionKVCache:
# Cross-attn KV is static once populated — write once (from text encoder),
# read many (every denoising step). Not managed through the paged block pool.
def write_cross_kv(
def _kv_branch_index(self, kv_branch: str) -> int:
try:
return self._kv_branch_local_indices[kv_branch]
except KeyError as exc:
expected = tuple(self._kv_branch_local_indices)
raise KeyError(f"Unknown AR-Diffusion KV branch {kv_branch!r}; expected {expected}") from exc
def _cross_attention_pool(
self,
layer_idx: int,
is_negative: bool,
k: torch.Tensor | None,
v: torch.Tensor | None,
k_img: torch.Tensor | None = None,
v_img: torch.Tensor | None = None,
session_id: str,
cache_name: str,
kv_branch: str,
) -> tuple[list[torch.Tensor], list[torch.Tensor]]:
if cache_name not in self.cross_attention_lengths:
raise KeyError(
f"Unknown AR-Diffusion cross-attention cache {cache_name!r}; "
f"expected {tuple(self.cross_attention_lengths)}"
)
self._kv_branch_index(kv_branch)
session = self._cross_sessions.get(session_id)
if session is None:
raise RuntimeError(
f"AR-Diffusion cross-attention cache {cache_name!r} for session {session_id!r} "
"was read before it was populated"
)
pool = session.get(cache_name, {}).get(kv_branch)
if pool is None:
raise RuntimeError(
f"AR-Diffusion cross-attention cache {cache_name!r} for session {session_id!r} "
f"and KV branch {kv_branch!r} was read before it was populated"
)
return pool
def is_cross_attention_populated(self, session_id: str, cache_name: str, kv_branch: str) -> bool:
"""Whether a complete logical-branch cache has been published."""
if cache_name not in self.cross_attention_lengths:
raise KeyError(
f"Unknown AR-Diffusion cross-attention cache {cache_name!r}; "
f"expected {tuple(self.cross_attention_lengths)}"
)
self._kv_branch_index(kv_branch)
session = self._cross_sessions.get(session_id)
return session is not None and kv_branch in session.get(cache_name, {})
def populate_cross_attention(
self,
session_id: str,
cache_name: str,
kv_branch: str,
layer_kv: Iterable[tuple[torch.Tensor, torch.Tensor]],
) -> None:
"""Write one layer's cross-attn K/V into the pool.
"""Atomically populate one logical branch of a named cross-attention cache.
``k`` / ``v``: ``(B, text_len, tp_num_heads, head_dim)``; only batch-0
is copied (B=1 for inference). ``None`` skips the text half (window
restarts keep the still-valid text K/V and rewrite only the image half).
The ``is_negative`` flag selects the correct CFG branch slot. ``k_img`` /
``v_img`` (I2V image tokens, ``(B, 257, ...)``) are written when the
image pool is allocated.
``layer_kv`` must yield exactly one ``(k, v)`` pair per model layer.
Inputs have shape ``(B, length, local_kv_heads, head_size)``; batch zero
is copied because AR-Diffusion currently supports one sequence per
forward. The new cache is published only after every layer is copied,
so failed projection/copy work cannot expose partially initialized KV.
"""
branch = 1 if is_negative else 0
if k is not None:
self._cross_k[layer_idx][branch].copy_(k[0])
self._cross_v[layer_idx][branch].copy_(v[0])
if k_img is not None and self._cross_k_img:
self._cross_k_img[layer_idx][branch].copy_(k_img[0])
self._cross_v_img[layer_idx][branch].copy_(v_img[0])
try:
length = self.cross_attention_lengths[cache_name]
except KeyError as exc:
raise KeyError(
f"Unknown AR-Diffusion cross-attention cache {cache_name!r}; "
f"expected {tuple(self.cross_attention_lengths)}"
) from exc
self._kv_branch_index(kv_branch)
session = self._cross_sessions.get(session_id)
if session is None and len(self._cross_sessions) >= self.session_capacity:
raise RuntimeError(
"AR-Diffusion cross-attention session capacity exhausted; "
"the runner must evict a session before allocating another"
)
if not self._allocate_tensors:
raise RuntimeError("AR-Diffusion cross-attention tensors require a configured pool device")
def read_cross_kv(self, layer_idx: int, is_negative: bool) -> dict:
"""Return a pool-backed cross-attn cache dict for one layer.
shape = (length, self.num_kv_heads, self.head_size)
expected_input_shape = (1, *shape)
k_pool = [torch.empty(shape, dtype=self.dtype, device=self.device) for _ in range(self.num_layers)]
v_pool = [torch.empty(shape, dtype=self.dtype, device=self.device) for _ in range(self.num_layers)]
populated_layers = 0
for layer_idx, (k, v) in enumerate(layer_kv):
if layer_idx >= self.num_layers:
raise ValueError(
f"AR-Diffusion cross-attention cache {cache_name!r} expected {self.num_layers} layers, "
f"got more than {self.num_layers}"
)
if not isinstance(k, torch.Tensor) or not isinstance(v, torch.Tensor):
raise ValueError(
f"AR-Diffusion cross-attention cache {cache_name!r} layer {layer_idx} "
f"must yield torch.Tensor k/v, got {type(k).__name__} and {type(v).__name__}"
)
if tuple(k.shape) != expected_input_shape or tuple(v.shape) != expected_input_shape:
raise ValueError(
f"AR-Diffusion cross-attention cache {cache_name!r} layer {layer_idx} expected "
f"k/v shape {expected_input_shape}, got {tuple(k.shape)} and {tuple(v.shape)}"
)
k_pool[layer_idx].copy_(k[0])
v_pool[layer_idx].copy_(v[0])
populated_layers += 1
if populated_layers != self.num_layers:
raise ValueError(
f"AR-Diffusion cross-attention cache {cache_name!r} expected {self.num_layers} layers, "
f"got {populated_layers}"
)
The dict matches the ``{"is_init": True, "k": Tensor, "v": Tensor}``
convention the cross-attention module expects — it reads from the pool
slice rather than from the lazy-initialised model-local dict. For I2V,
``k_img`` / ``v_img`` are added so the image-token cache (added by #4154)
reads from the pool too.
"""
branch = 1 if is_negative else 0
k = self._cross_k[layer_idx][branch].unsqueeze(0) # (1, L, heads, dim)
v = self._cross_v[layer_idx][branch].unsqueeze(0)
cache = {"is_init": True, "k": k, "v": v}
if self._cross_k_img:
cache["k_img"] = self._cross_k_img[layer_idx][branch].unsqueeze(0)
cache["v_img"] = self._cross_v_img[layer_idx][branch].unsqueeze(0)
return cache
if session is None:
session = {}
self._cross_sessions[session_id] = session
session.setdefault(cache_name, {})[kv_branch] = (k_pool, v_pool)
def read_cross_attention_kv(
self,
session_id: str,
cache_name: str,
layer_idx: int,
kv_branch: str,
) -> dict[str, torch.Tensor | bool]:
"""Return a model-facing K/V dict for one named cross-attention pool."""
k_pool, v_pool = self._cross_attention_pool(session_id, cache_name, kv_branch)
return {
"is_init": True,
"k": k_pool[layer_idx].unsqueeze(0),
"v": v_pool[layer_idx].unsqueeze(0),
}
def retain_cross_attention(self, session_id: str, cache_names: Collection[str]) -> None:
"""Release named cross-attention caches not retained by an internal reset."""
session = self._cross_sessions.get(session_id)
if session is None:
return
keep = set(cache_names)
for cache_name in tuple(session):
if cache_name not in keep:
del session[cache_name]
if not session:
self._cross_sessions.pop(session_id, None)
def release_cross_attention(self, session_id: str) -> None:
"""Release every named cross-attention allocation for one session."""
self._cross_sessions.pop(session_id, None)
# -- request lifecycle ---------------------------------------------------
@@ -431,20 +521,19 @@ class ARDiffusionKVCache:
self.block_size,
)
def scratch_block_ids(self, is_negative: bool, start: int, count: int) -> list[int]:
"""Return branch-local scratch block ids outside manager ownership."""
def scratch_block_ids(self, kv_branch: str, start: int, count: int) -> list[int]:
"""Return KV-branch-local scratch block ids outside manager ownership."""
if count < 0 or start < 0:
raise ValueError(f"scratch start/count must be non-negative, got start={start}, count={count}")
if start + count > self.scratch_blocks_per_branch:
if start + count > self.scratch_blocks_per_kv_branch:
raise RuntimeError(
"AR-Diffusion paged attention scratch blocks exhausted: "
f"need [{start}, {start + count}) of {self.scratch_blocks_per_branch}. "
"Increase AR_DIFFUSION_KV_SCRATCH_BLOCKS_PER_BRANCH."
f"need [{start}, {start + count}) of {self.scratch_blocks_per_kv_branch}. "
"Declare max_scratch_tokens_per_branch in the pipeline capability "
"or increase AR_DIFFUSION_KV_SCRATCH_BLOCKS_PER_BRANCH."
)
# With one local branch (CFG-parallel x2) the rank's only branch maps to
# slot 0 whichever CFG side it is; the other branch never allocates here.
branch_offset = self.scratch_blocks_per_branch if (is_negative and self.local_branches == 2) else 0
base = self.managed_num_blocks + branch_offset + start
kv_branch_offset = self.scratch_blocks_per_kv_branch * self._kv_branch_index(kv_branch)
base = self.managed_num_blocks + kv_branch_offset + start
return list(range(base, base + count))
def key_cache(self, layer_idx: int) -> torch.Tensor:
@@ -455,13 +544,13 @@ class ARDiffusionKVCache:
def window_block_ids(self, adapter: ARDiffusionRequestAdapter) -> list[int]:
"""Resident (non-null) managed blocks visible to paged attention."""
return resident_block_ids(self.block_table(adapter), self.null_block_id)
return [int(block_id) for block_id in resident_block_ids(self.block_table(adapter), self.null_block_id)]
def commit_chunk(self, adapter: ARDiffusionRequestAdapter) -> None:
"""Advance the adapter by one chunk after its K/V is written.
This standalone primitive is used by low-level manager tests. In the
DreamZero paged-attention path, :meth:`ARDiffusionKVState.commit_paged_context`
This standalone primitive is used by low-level manager tests. The
paged-attention path uses :meth:`ARDiffusionKVState.commit_paged_context`,
advances the adapter only after the forward succeeds. Call once per
committed chunk, not per denoise step.
"""
@@ -153,7 +153,7 @@ def chunk_window_skipped_tokens(
Pure function so the eviction policy is unit-testable without constructing a
manager. Two strategies:
- ``reset_at_boundary`` (DreamZero): at each chunk boundary everything past
- ``reset_at_boundary``: at each chunk boundary everything past
the sink is dropped.
- otherwise (VGGT-style sliding replace): keep the last ``window`` tokens
(plus the sink); the skip count snaps down to a chunk boundary so a chunk
@@ -184,6 +184,25 @@ class ChunkWindowManager(SlidingWindowManager):
reset_at_boundary=spec.reset_at_boundary,
)
def remove_skipped_blocks(
self,
request_id: str,
total_computed_tokens: int,
num_prompt_tokens: int | None = None,
) -> None:
"""Free the middle gap while preserving the leading attention sink."""
del num_prompt_tokens
num_skipped_tokens = self.get_num_skipped_tokens(total_computed_tokens)
if num_skipped_tokens <= 0:
return
sink_blocks = self.kv_cache_spec.sink_chunks
num_skipped_blocks = num_skipped_tokens // self.block_size
self._remove_blocks_in_range(
request_id,
sink_blocks,
sink_blocks + num_skipped_blocks,
)
# Register so KVCacheManager resolves ChunkWindowSpec to ChunkWindowManager.
# Dispatch walks the spec's MRO, so without explicit registration the subclass
@@ -1,5 +1,5 @@
# SPDX-License-Identifier: Apache-2.0
"""Paged self-attention helpers for AR-Diffusion DreamZero KV reuse."""
"""Paged self-attention helpers for AR-Diffusion KV reuse."""
from __future__ import annotations
@@ -10,7 +10,7 @@ import torch
from vllm_omni.experimental.ar_diffusion.kv_cache.paged import compute_slot_mapping
# Set by ARDiffusionPagedForwardContext.prepare() before each branch forward and
# Set by ARDiffusionPagedForwardContext.prepare() before each KV branch forward and
# read by the fused write+attend custom op below. The pools are process-lifetime
# allocations owned by one ARDiffusionKVCache per worker process (CFG-parallel /
# TP ranks are separate processes), mirroring vLLM's forward-context pattern for
@@ -39,7 +39,7 @@ class ARDiffusionPagedLayerInputs(NamedTuple):
A NamedTuple of plain tensors + ints so ``torch.compile`` treats every field
as a pytree graph input (no object-attribute guards, no recompiles when only
tensor *values* change). All layers of one branch forward share the same
tensor *values* change). All layers of one KV branch forward share the same
metadata tensor objects, built once by ``prepare()``.
``layer_idx`` is a 0-dim CPU tensor, NOT a python int: all 40 DiT blocks
@@ -62,11 +62,11 @@ class ARDiffusionPagedLayerInputs(NamedTuple):
@dataclass
class ARDiffusionPagedForwardContext:
"""Mutable branch-level state shared by all layer contexts in one forward."""
"""Mutable KV-branch state shared by all layer contexts in one forward."""
kv_cache: Any
adapter: Any
is_negative: bool
kv_branch: str
history_block_ids: list[int]
seq_len: int
commit_current: bool
@@ -80,7 +80,7 @@ class ARDiffusionPagedForwardContext:
_allocated_video: bool = False
_committed: bool = False
_action_len: int = 0
# Set once by prepare(); shared by all layers of the branch forward.
# Set once by prepare(); shared by all layers of the KV branch forward.
block_table: torch.Tensor | None = None
query_start_loc: torch.Tensor | None = None
seq_lens: torch.Tensor | None = None
@@ -102,7 +102,7 @@ class ARDiffusionPagedForwardContext:
return self.seq_len // self.block_size
def ensure_video_slots(self, device: torch.device) -> None:
"""Allocate/write targets for the current video tokens, once per branch."""
"""Allocate/write targets for the current video tokens, once per KV branch."""
if self._allocated_video:
return
@@ -116,7 +116,7 @@ class ARDiffusionPagedForwardContext:
positions = torch.arange(start, start + self.seq_len, dtype=torch.long)
self.current_video_slot_mapping = compute_slot_mapping(table, positions, self.block_size).to(device=device)
else:
self.current_video_block_ids = self.kv_cache.scratch_block_ids(self.is_negative, 0, n_blocks)
self.current_video_block_ids = self.kv_cache.scratch_block_ids(self.kv_branch, 0, n_blocks)
positions = torch.arange(self.seq_len, dtype=torch.long)
self.current_video_slot_mapping = compute_slot_mapping(
self.current_video_block_ids,
@@ -140,7 +140,7 @@ class ARDiffusionPagedForwardContext:
action_blocks = (action_len + self.block_size - 1) // self.block_size
scratch_offset = 0 if self.commit_current else len(self.current_video_block_ids)
self.action_scratch_block_ids = self.kv_cache.scratch_block_ids(
self.is_negative,
self.kv_branch,
scratch_offset,
action_blocks,
)
@@ -161,8 +161,15 @@ class ARDiffusionPagedForwardContext:
)
all_video_blocks = self.history_block_ids + self.current_video_block_ids
max_video_blocks = self.max_video_tokens // self.block_size
visible_video_blocks = all_video_blocks[-max_video_blocks:] if max_video_blocks else []
video_len = min(len(all_video_blocks) * self.block_size, self.max_video_tokens)
sink_blocks = int(self.kv_cache.spec.sink_chunks)
if len(all_video_blocks) <= max_video_blocks:
visible_video_blocks = all_video_blocks
else:
tail_blocks = max_video_blocks - sink_blocks
visible_video_blocks = all_video_blocks[:sink_blocks]
if tail_blocks:
visible_video_blocks += all_video_blocks[-tail_blocks:]
video_len = len(visible_video_blocks) * self.block_size
return visible_video_blocks, video_len
def build_block_table(
@@ -201,9 +208,9 @@ class ARDiffusionPagedForwardContext:
return block_table, query_start_loc, seq_lens, self.query_len, max_seq_len
def prepare(self, device: torch.device, action_len: int, query_len: int) -> None:
"""Host-side, once-per-branch-forward setup (called OUTSIDE torch.compile).
"""Host-side, once-per-KV-branch setup (called OUTSIDE torch.compile).
Allocates the current video/action slots (still lazy: only the branch a
Allocates the current video/action slots (still lazy: only the KV branch a
CFG-parallel rank actually runs reaches its ``_forward_blocks``), builds
the padded block-table metadata ONCE for all layers, and publishes the
pool registry for the fused custom op. The compiled per-layer code then
@@ -245,7 +252,7 @@ class ARDiffusionPagedForwardContext:
@dataclass
class ARDiffusionPagedLayerContext:
"""Layer-specific handle passed through DreamZero's existing ``kv_cache`` slot."""
"""Layer-specific handle passed through a model's ``kv_cache`` slot."""
is_ar_diffusion_paged_context: ClassVar[bool] = True
layer_idx: int
@@ -260,8 +267,8 @@ class ARDiffusionPagedLayerContext:
return self.forward_ctx.adapter
@property
def is_negative(self) -> bool:
return self.forward_ctx.is_negative
def kv_branch(self) -> str:
return self.forward_ctx.kv_branch
@property
def history_block_ids(self) -> list[int]:
@@ -1,8 +1,12 @@
# SPDX-License-Identifier: Apache-2.0
"""DreamZero-facing AR-Diffusion KV bridge backed by paged self-attention."""
"""Model-facing session state backed by paged AR-Diffusion KV storage."""
from __future__ import annotations
from collections.abc import Collection, Iterable, Mapping
from typing import TYPE_CHECKING
import torch
from vllm.logger import init_logger
from vllm_omni.experimental.ar_diffusion.kv_cache.paged_attention import (
@@ -10,67 +14,79 @@ from vllm_omni.experimental.ar_diffusion.kv_cache.paged_attention import (
ARDiffusionPagedLayerContext,
)
if TYPE_CHECKING:
from vllm_omni.experimental.ar_diffusion.kv_cache.manager import (
ARDiffusionKVCache,
ARDiffusionRequestAdapter,
)
_log = init_logger(__name__)
class ARDiffusionKVState:
"""Per-request bridge between a DreamZero rollout and the AR-Diffusion KV pool.
"""Runner-owned KV state for one session and any number of KV branches.
The self-attention path is paged-only: ``get_kv_caches`` returns lightweight
per-layer contexts, and each attention layer writes current K/V into the
pool before calling the FlashAttention block-table kernel. No resident
window is gathered into a contiguous tensor.
KV branches are addressed by the names in ``ARDiffusionKVCacheSpec``. Each
branch owns an independent request adapter and therefore independent
resident self-attention blocks. Named cross-attention allocations are also
session-scoped and are released by :meth:`close`.
"""
def __init__(self, kv_cache, pos_adapter, neg_adapter, num_layers: int) -> None:
"""One state per DreamZero session, holding BOTH CFG branches' adapters.
``pos``/``neg`` are the classifier-free-guidance branches: ``pos`` is
the conditional pass (real prompt), ``neg`` the unconditional pass
(negative prompt). Each owns an independent request adapter — and
therefore independent pool blocks — because the two branches produce
different K/V for the same frames. Every ``is_negative`` flag in this
module selects between them (``False`` -> ``pos``, ``True`` -> ``neg``).
"""
def __init__(
self,
kv_cache: ARDiffusionKVCache,
session_id: str,
adapters: Mapping[str, ARDiffusionRequestAdapter],
*,
num_layers: int,
) -> None:
expected = tuple(kv_branch.name for kv_branch in kv_cache.kv_branches)
if set(adapters) != set(expected):
raise ValueError(
"AR-Diffusion session adapters must match the configured KV branches; "
f"expected {expected}, got {tuple(adapters)}"
)
self.kv_cache = kv_cache
self.pos = pos_adapter
self.neg = neg_adapter
self.session_id = session_id
self.adapters = dict(adapters)
self.num_layers = num_layers
self._committed: dict[bool, int] = {False: 0, True: 0}
self._paged_pending: dict[bool, ARDiffusionPagedForwardContext | None] = {False: None, True: None}
# Cross-attn KV is populated once after text encoding (eager), then
# read every denoising step. Tracked per half: the text K/V depend only
# on the session prompt (survive window-boundary resets), while the I2V
# image-token K/V come from the current observation's CLIP features and
# must be re-projected on every window restart.
self._cross_text_populated: dict[bool, bool] = {False: False, True: False}
self._cross_img_populated: dict[bool, bool] = {False: False, True: False}
self._committed: dict[str, int] = dict.fromkeys(expected, 0)
self._paged_pending: dict[str, ARDiffusionPagedForwardContext | None] = dict.fromkeys(expected)
self._closed = False
def _adapter(self, is_negative: bool):
return self.neg if is_negative else self.pos
@property
def kv_branch_names(self) -> tuple[str, ...]:
return tuple(self.adapters)
def adapter(self, kv_branch: str) -> ARDiffusionRequestAdapter:
"""Return the request adapter for one logical KV branch."""
if self._closed:
raise RuntimeError(f"AR-Diffusion session {self.session_id!r} is closed")
try:
return self.adapters[kv_branch]
except KeyError as exc:
raise KeyError(f"Unknown AR-Diffusion KV branch {kv_branch!r}; expected {self.kv_branch_names}") from exc
def get_kv_caches(
self,
is_negative: bool,
kv_branch: str,
seq_len: int | None = None,
commit_current: bool = False,
) -> list[ARDiffusionPagedLayerContext]:
if seq_len is None:
raise ValueError("AR-Diffusion paged self-attention requires seq_len in get_kv_caches()")
return self.prepare_paged_context(is_negative, seq_len, commit_current)
return self.prepare_paged_context(kv_branch, seq_len, commit_current)
def prepare_paged_context(
self,
is_negative: bool,
kv_branch: str,
seq_len: int,
commit_current: bool,
) -> list[ARDiffusionPagedLayerContext]:
"""Return per-layer paged attention contexts for one branch forward.
"""Return per-layer paged attention contexts for one KV branch forward.
Allocation is lazy: constructing both CFG branches' kwargs must not
allocate manager blocks for a branch that a CFG-parallel rank will not run.
The first self-attn layer that consumes the context allocates the current
video slots, then all layers share that branch-level state.
Allocation is lazy so distributed workers allocate only for KV branches
they actually execute.
"""
cs = self.kv_cache.spec.chunk_size
if int(seq_len) % cs != 0:
@@ -78,25 +94,26 @@ class ARDiffusionKVState:
f"AR-Diffusion expects frame-aligned seq_len (multiple of chunk_size={cs}), got {seq_len}"
)
pending = self._paged_pending.get(is_negative)
pending = self._paged_pending.get(kv_branch)
if pending is not None and pending.commit_current and pending._allocated_video and not pending._committed:
raise RuntimeError("AR-Diffusion paged context replaced before its managed current chunk was committed")
adapter = self._adapter(is_negative)
branch = "neg" if is_negative else "pos"
adapter = self.adapter(kv_branch)
forward_ctx = ARDiffusionPagedForwardContext(
kv_cache=self.kv_cache,
adapter=adapter,
is_negative=is_negative,
kv_branch=kv_branch,
history_block_ids=self.kv_cache.window_block_ids(adapter),
seq_len=int(seq_len),
commit_current=bool(commit_current),
max_video_tokens=int(self.kv_cache.spec.sliding_window),
max_video_tokens=int(
self.kv_cache.spec.sliding_window + self.kv_cache.spec.sink_chunks * self.kv_cache.spec.chunk_size
),
)
self._paged_pending[is_negative] = forward_ctx
self._paged_pending[kv_branch] = forward_ctx
_log.debug(
"AR-Diffusion GET [%s] source=paged-attn layers=%d history_blocks=%d seq_len=%d commit_current=%s",
branch,
"AR-Diffusion GET [%s] source=paged-attn layers=%d history_blocks=%d seq_len=%d commit_current=%s",
kv_branch,
self.num_layers,
len(forward_ctx.history_block_ids),
int(seq_len),
@@ -104,80 +121,94 @@ class ARDiffusionKVState:
)
return [ARDiffusionPagedLayerContext(layer_idx=i, forward_ctx=forward_ctx) for i in range(self.num_layers)]
def commit_paged_context(self, is_negative: bool) -> None:
"""Commit the managed current video blocks after a successful forward."""
ctx = self._paged_pending.get(is_negative)
def commit_paged_context(self, kv_branch: str) -> None:
"""Commit managed current blocks after one successful KV branch forward."""
self.adapter(kv_branch)
ctx = self._paged_pending.get(kv_branch)
if ctx is None:
return
branch = "neg" if is_negative else "pos"
if ctx.commit_current and ctx._allocated_video:
n_chunks = ctx.seq_len // self.kv_cache.spec.chunk_size
for _ in range(n_chunks):
ctx.adapter.on_chunk_committed()
self._committed[is_negative] += ctx.seq_len
self._committed[kv_branch] += ctx.seq_len
_log.debug(
"AR-Diffusion COMMIT [%s] paged-attn new_tokens=%d chunks=%d resident=%d/%d",
branch,
"AR-Diffusion COMMIT [%s] new_tokens=%d chunks=%d resident=%d/%d",
kv_branch,
ctx.seq_len,
n_chunks,
len(self.kv_cache.window_block_ids(ctx.adapter)),
self.kv_cache.spec.window_chunks,
)
ctx.mark_committed()
self._paged_pending[is_negative] = None
self._paged_pending[kv_branch] = None
def get_cross_kv_caches(self, is_negative: bool) -> list[dict]:
"""Return pool-backed cross-attn cache dicts.
def is_cross_attention_populated(self, kv_branch: str, cache_name: str) -> bool:
self.adapter(kv_branch)
return self.kv_cache.is_cross_attention_populated(self.session_id, cache_name, kv_branch)
Under AR-Diffusion the cross-attn pool is always populated by ``_kv_populate_cross``
before the first read, so this must never be reached unpopulated — if it
were, the model would lazily project and write cross KV itself, violating
engine ownership. Fail loud rather than fall back to the model.
def populate_cross_attention(
self,
kv_branch: str,
cache_name: str,
layer_kv: Iterable[tuple[torch.Tensor, torch.Tensor]],
) -> None:
"""Publish one named logical-branch cache after all layers are written.
The iterable is consumed once in layer order. If projection, validation,
or copying fails, the previous complete cache (if any) remains visible.
"""
img_ok = not self.kv_cache._cross_k_img or self._cross_img_populated[is_negative]
if not (self._cross_text_populated[is_negative] and img_ok and self.kv_cache.cross_attn_length > 0):
self.adapter(kv_branch)
self.kv_cache.populate_cross_attention(self.session_id, cache_name, kv_branch, layer_kv)
def get_cross_attention_kv(self, kv_branch: str, cache_name: str) -> list[dict[str, torch.Tensor | bool]]:
"""Return all layers for one populated named cross-attention cache."""
if not self.is_cross_attention_populated(kv_branch, cache_name):
raise RuntimeError(
f"AR-Diffusion cross-attn read before _kv_populate_cross (neg={is_negative}, "
f"cross_attn_length={self.kv_cache.cross_attn_length}, "
f"text={self._cross_text_populated[is_negative]} "
f"img={self._cross_img_populated[is_negative]}) — the engine must own all cross KV"
f"AR-Diffusion cross-attention cache {cache_name!r} for KV branch {kv_branch!r} "
"was read before it was populated"
)
return [self.kv_cache.read_cross_kv(i, is_negative) for i in range(self.num_layers)]
return [
self.kv_cache.read_cross_attention_kv(self.session_id, cache_name, i, kv_branch)
for i in range(self.num_layers)
]
def close(self) -> None:
"""Final teardown: free both branches' resident pool blocks."""
self.kv_cache.end_request(self.pos)
self.kv_cache.end_request(self.neg)
self._paged_pending = {False: None, True: None}
"""Release all self- and cross-attention storage owned by this session."""
if self._closed:
return
for adapter in self.adapters.values():
self.kv_cache.end_request(adapter)
self.kv_cache.release_cross_attention(self.session_id)
self._paged_pending = dict.fromkeys(self.adapters)
self._closed = True
def reset(self, *, keep_cross_text: bool = False) -> None:
"""Drop this session's KV — mirrors the model-local ``state.reset()``.
def reset(self, *, keep_cross_attention: Collection[str] = ()) -> None:
"""Release resident blocks and reopen this session with fresh adapters.
DreamZero resets at the attention-window boundary (``should_reset``); the
model starts a fresh sliding window, so AR-Diffusion must free the resident pool
blocks and start a fresh adapter for each branch. The ``ARDiffusionKVState``
object (and thus ``pipeline._ar_diffusion_kv_state``) is preserved across the reset
so the runner's session mapping stays valid — only the pool-backed state
is recycled.
Args:
keep_cross_text: On window ("inference") resets the session prompt is
unchanged, so the text cross-attn K/V in the static pool buffers are
still valid — keep them and only force the image half to repopulate.
Session resets (prompt change / explicit reset) pass ``False``.
``keep_cross_attention`` names allocations whose conditioning remains
valid across a model-internal window reset. Ordinary session reset
should leave it empty.
"""
pos_id, neg_id = self.pos.request_id, self.neg.request_id
self.close()
self.pos = self.kv_cache.begin_request(pos_id)
self.neg = self.kv_cache.begin_request(neg_id)
self._committed = {False: 0, True: 0}
self._paged_pending = {False: None, True: None}
if not keep_cross_text:
self._cross_text_populated = {False: False, True: False}
self._cross_img_populated = {False: False, True: False}
unknown = set(keep_cross_attention) - set(self.kv_cache.cross_attention_lengths)
if unknown:
raise KeyError(f"Unknown AR-Diffusion cross-attention caches to keep: {sorted(unknown)}")
request_ids = {kv_branch: adapter.request_id for kv_branch, adapter in self.adapters.items()}
if keep_cross_attention:
for adapter in self.adapters.values():
self.kv_cache.end_request(adapter)
self.kv_cache.retain_cross_attention(self.session_id, keep_cross_attention)
else:
self.close()
self.adapters = {
kv_branch: self.kv_cache.begin_request(request_id) for kv_branch, request_id in request_ids.items()
}
self._committed = dict.fromkeys(self.adapters, 0)
self._paged_pending = dict.fromkeys(self.adapters)
self._closed = False
_log.info(
"AR-Diffusion RESET [%s/%s] session KV cleared (window boundary; cross text %s)",
pos_id,
neg_id,
"kept" if keep_cross_text else "cleared",
"AR-Diffusion RESET session=%s KV branches=%s kept_cross=%s",
self.session_id,
self.kv_branch_names,
sorted(keep_cross_attention),
)
+237 -332
View File
@@ -1,32 +1,23 @@
# SPDX-License-Identifier: Apache-2.0
"""ARDiffusionModelRunner — the diffusion model runner for the AR-Diffusion engine.
Subclasses ``DiffusionModelRunner`` and owns the engine-level KV cache
(:class:`ARDiffusionKVCache`). It brackets a request's rollout with the KV lifecycle
(``begin_request`` / ``end_request``) and exposes the live ``ARDiffusionKVCache`` so the
model pipeline (DreamZero) can drive the per-chunk allocate / slot-mapping /
gather / commit operations during ``pipeline.forward``.
Selecting the AR-Diffusion engine enables KV management (no env gate); optional KV
parameters come from the deploy yaml's ``model_config["ar_diffusion_kv_config"]``.
"""
from __future__ import annotations
import dataclasses
import math
import time
from collections import OrderedDict
import numpy as np
import torch
from vllm.logger import init_logger
from vllm_omni.diffusion.data import DiffusionOutput, OmniDiffusionConfig
from vllm_omni.diffusion.models.dreamzero.pipeline_dreamzero import MAX_DREAMZERO_SESSIONS
from vllm_omni.diffusion.request import OmniDiffusionRequest
from vllm_omni.diffusion.sched.interface import KVPrefetchJob
from vllm_omni.diffusion.sched.interface import DiffusionSchedulerOutput, KVPrefetchJob
from vllm_omni.diffusion.worker.diffusion_model_runner import DiffusionModelRunner
from vllm_omni.diffusion.worker.utils import BatchRunnerOutput
from vllm_omni.experimental.ar_diffusion.capability import (
ARDiffusionKVCacheSpec,
SupportsARDiffusionPipeline,
SupportsARDiffusionWarmup,
)
from vllm_omni.experimental.ar_diffusion.kv_cache.config import ARDiffusionKVConfig
from vllm_omni.experimental.ar_diffusion.kv_cache.manager import ARDiffusionKVCache
from vllm_omni.experimental.ar_diffusion.kv_cache.state import ARDiffusionKVState
@@ -35,14 +26,7 @@ logger = init_logger(__name__)
def resolve_ar_diffusion_kv_config(od_config: OmniDiffusionConfig) -> ARDiffusionKVConfig:
"""Resolve the AR-Diffusion KV config from the deploy/command arg.
Selecting the AR-Diffusion engine implies KV management, so the result is always
enabled. Optional overrides (``window_chunks`` / ``gpu_memory_fraction`` /
``sink_chunks`` / ``reset_at_boundary``) may be supplied via the deploy yaml as
``model_config["ar_diffusion_kv_config"]``. ``chunk_size`` / ``window_chunks`` are
finalized from the model geometry at load (see ``_preallocate_kv_cache``).
"""
"""Resolve optional deployment overrides and enable AR-Diffusion KV."""
raw = getattr(od_config, "ar_diffusion_kv_config", None)
if raw is None:
model_config = getattr(od_config, "model_config", None)
@@ -56,359 +40,280 @@ def resolve_ar_diffusion_kv_config(od_config: OmniDiffusionConfig) -> ARDiffusio
class ARDiffusionModelRunner(DiffusionModelRunner):
def __init__(self, vllm_config, od_config: OmniDiffusionConfig, device) -> None:
"""Own AR KV pools and session lifetime for a capable diffusion pipeline.
The runner never inspects model internals. At load time it consumes an
:class:`ARDiffusionKVCacheSpec`; during execution it binds a runner-owned
:class:`ARDiffusionKVState` through the pipeline's context manager. Sessions
persist across requests and end on explicit close, LRU eviction, or failed
forward. A request with ``extra_args["reset"]`` releases the old session
before executing with a fresh one.
"""
_WARMUP_SID = "__ardiffusion_warmup__"
_MAX_RESIDENT_SESSIONS = 1
def __init__(self, vllm_config: object, od_config: OmniDiffusionConfig, device: torch.device) -> None:
super().__init__(vllm_config, od_config, device)
self.ar_diffusion_kv_config = resolve_ar_diffusion_kv_config(od_config)
# Built after the model is loaded (dimensions known); stays None while KV
# management is disabled.
self.kv_cache: ARDiffusionKVCache | None = None
# DreamZero KV is session-scoped (persists across a session's forwards),
# so AR-Diffusion KV state is keyed by session_id and reused, not created per request.
# Bounded by the same LRU cap as DreamZeroPipeline._states: an evicted
# session's ARDiffusionKVState owns pool blocks (two CFG adapters), so without a
# bound, session-id churn in a long-running server would leak pool ownership
# until the KV pool is exhausted. Evicting frees those blocks (state.close()).
self._ar_diffusion_states: OrderedDict[str, ARDiffusionKVState] = OrderedDict()
self._max_ar_diffusion_states = MAX_DREAMZERO_SESSIONS
# Per-request server-side forward E2E times (seconds), recorded in
# execute_model. This is the worker-side compute time — the analog of the
# upstream server's per-request E2E — and excludes the engine<->worker IPC
# that the client's omni.generate wall time includes. Read (and cleared)
# by the ar_diffusion_perf_stats worker RPC for the perf-compare summary.
self._ar_diffusion_capability: SupportsARDiffusionPipeline | None = None
self._ar_diffusion_kv_cache_spec: ARDiffusionKVCacheSpec | None = None
self._sessions: OrderedDict[str, ARDiffusionKVState] = OrderedDict()
self._session_capacity = 0
self._perf_e2e_times: list[float] = []
def build_kv_cache(
self,
*,
num_layers: int,
num_kv_heads: int,
head_size: int,
dtype,
block_size: int,
max_model_len: int,
available_bytes: int,
cross_attn_length: int = 0,
cross_attn_img_length: int = 0,
local_branches: int = 2,
num_frame_per_block: int = 1,
) -> None:
"""Construct the ARDiffusionKVCache (preallocating GPU pools on ``self.device``).
@staticmethod
def _require_capability(pipeline: object) -> SupportsARDiffusionPipeline:
if not isinstance(pipeline, SupportsARDiffusionPipeline):
raise TypeError(
"ARDiffusionEngine requires the loaded pipeline to implement "
"SupportsARDiffusionPipeline: ar_diffusion_kv_cache_spec(), "
"bind_ar_diffusion_state(), reset_ar_diffusion_session(), and "
"close_ar_diffusion_session(). Select the default DiffusionEngine "
"or add an explicit AR-Diffusion capability adapter to the model."
)
return pipeline
No-op when KV management is disabled.
"""
if not self.ar_diffusion_kv_config.enable:
return
self.kv_cache = ARDiffusionKVCache(
self.ar_diffusion_kv_config,
num_layers=num_layers,
num_kv_heads=num_kv_heads,
head_size=head_size,
dtype=dtype,
block_size=block_size,
max_model_len=max_model_len,
available_bytes=available_bytes,
cross_attn_length=cross_attn_length,
cross_attn_img_length=cross_attn_img_length,
local_branches=local_branches,
num_frame_per_block=num_frame_per_block,
device=self.device,
)
logger.info(
"AR-Diffusion KV cache enabled: %d blocks, chunk_size=%d, window_chunks=%s, "
"layers=%d kv_heads=%d head_dim=%d block_size=%d cross_attn_len=%d device=%s",
self.kv_cache.num_blocks,
self.ar_diffusion_kv_config.chunk_size,
self.ar_diffusion_kv_config.window_chunks,
num_layers,
num_kv_heads,
head_size,
block_size,
cross_attn_length,
self.device,
)
# -- preallocation at load -------------------------------------------------
def load_model(self, *args, **kwargs):
def load_model(self, *args: object, **kwargs: object) -> None:
super().load_model(*args, **kwargs)
if self.ar_diffusion_kv_config.enable and self.pipeline is not None:
self._preallocate_kv_cache()
# Pre-capture the CUDA graphs for every window-fill shape at load time
# (only when compiling/cuda-graph is on) so serving is fast from chunk 0.
if not self.od_config.enforce_eager and self.ar_diffusion_kv_config.warmup_cudagraph:
self._warmup_ar_rollout()
if self.pipeline is None:
return
self._preallocate_kv_cache()
if not self.od_config.enforce_eager and self.ar_diffusion_kv_config.warmup_cudagraph:
self._warmup_ar_rollout()
def _infer_frame_seqlen(self) -> int:
"""frame_seqlen = (H//8)*(W//8)//4 from the configured image_resolution."""
mc = getattr(self.od_config, "model_config", None) or {}
psc = (mc.get("policy_server_config") if isinstance(mc, dict) else None) or {}
res = psc.get("image_resolution", [180, 320])
h, w = int(res[0]), int(res[1])
return (h // 8) * (w // 8) // 4
def _available_memory_bytes(self) -> int:
if self.device is None or torch.device(self.device).type != "cuda":
raise RuntimeError("AR-Diffusion KV preallocation currently requires a CUDA device")
return int(torch.cuda.mem_get_info(self.device)[0])
def _preallocate_kv_cache(self) -> None:
"""Build the KV pool once, from the loaded DreamZero transformer geometry."""
# The paged self-attention path carries one rollout per request: slot
# mappings/adapters are single-sequence and K/V writes address batch
# index 0 only. Reject a multi-sequence config at init instead of
# failing (or silently dropping sequences) mid-forward.
def _preallocate_kv_cache(self, *, available_bytes: int | None = None) -> None:
"""Build pools solely from the pipeline capability and runner config."""
if bool(getattr(self.od_config, "step_execution", False)):
raise ValueError(
"ARDiffusionModelRunner currently supports request-mode execution only; "
"step_execution=True would bypass per-request AR session binding."
)
max_num_seqs = int(getattr(self.od_config, "max_num_seqs", 1) or 1)
if max_num_seqs > 1:
raise ValueError(
"AR-Diffusion paged KV supports max_num_seqs=1 (single-sequence "
f"rollouts); got max_num_seqs={max_num_seqs}. Per-sequence "
"adapters/slot maps are not implemented."
f"rollouts); got max_num_seqs={max_num_seqs}."
)
t = self.pipeline.transformer
num_layers = int(t.num_layers)
num_kv_heads = int(getattr(t.blocks[0].self_attn, "tp_num_heads", t.num_heads))
head_size = int(t.dim // t.num_heads)
num_frame_per_block = int(t.num_frame_per_block)
local_attn_size = int(t.local_attn_size)
# The loaded transformer is the source of truth for frame_seqlen (it derives
# max_attention_size from it). The deploy-config inference is only a
# cross-check — warn on drift rather than silently sizing chunks wrong.
frame_seqlen = int(getattr(t, "frame_seqlen", 0)) or self._infer_frame_seqlen()
inferred = self._infer_frame_seqlen()
if frame_seqlen != inferred:
logger.warning(
"AR-Diffusion frame_seqlen mismatch: transformer=%d deploy-config=%d; using transformer",
frame_seqlen,
inferred,
capability = self._require_capability(self.pipeline)
if bool(getattr(self.pipeline, "supports_request_batch", False)):
raise ValueError(
"ARDiffusionModelRunner currently supports one request at a time; "
"request-batch execution would bypass per-request AR session binding."
)
spec = capability.ar_diffusion_kv_cache_spec()
if not isinstance(spec, ARDiffusionKVCacheSpec):
raise TypeError(
f"ar_diffusion_kv_cache_spec() must return ARDiffusionKVCacheSpec, got {type(spec).__name__}"
)
# The model's own attention window, in tokens (= the [-max_attention_size:]
# slice it applies). Read it directly rather than recomputing.
max_attention_size = int(t.blocks[0].self_attn.max_attention_size)
# Cross-attn text length — static pool for the text-encoder output.
# The cached k/v spans the full text sequence (the image tokens
# prepended in _forward_blocks are stripped inside the cross-attn
# forward but are not part of the cached k/v).
cross_attn_length = int(getattr(t, "text_len", 0))
# I2V image-token cross-attn length: the model splits context[:, :257] as the
# image tokens (the img_emb(clip_feature) output), caching k_img/v_img. T2V: 0.
cross_attn_img_length = 257 if getattr(t, "model_type", "t2v") == "i2v" else 0
# Frame-granular paging: 1 block = 1 frame = frame_seqlen tokens, so the
# resident window matches max_attention_size exactly (it need not be a whole
# number of num_frame_per_block causal blocks).
chunk_size = frame_seqlen
window_chunks = self.ar_diffusion_kv_config.window_chunks or (max_attention_size // frame_seqlen)
self.ar_diffusion_kv_config = dataclasses.replace(
self.ar_diffusion_kv_config, chunk_size=chunk_size, window_chunks=window_chunks
config = dataclasses.replace(
self.ar_diffusion_kv_config,
chunk_size=spec.tokens_per_frame,
window_chunks=self.ar_diffusion_kv_config.window_chunks or spec.window_frames,
sink_chunks=self.ar_diffusion_kv_config.sink_chunks or spec.sink_frames,
reset_at_boundary=self.ar_diffusion_kv_config.reset_at_boundary or spec.reset_at_boundary,
)
free_bytes = torch.cuda.mem_get_info(self.device)[0]
# Under CFG-parallel each rank executes exactly ONE branch (rank0 pos,
# rank1 neg; the other branch's lazy contexts never allocate on this
# rank), so its pool only needs one branch's capacity. A single-process
# run (cfg world 1) executes both branches from one pool.
try:
from vllm_omni.diffusion.distributed.parallel_state import (
get_classifier_free_guidance_world_size,
)
cfg_world = int(get_classifier_free_guidance_world_size())
except Exception:
cfg_world = 1
local_branches = 1 if cfg_world >= 2 else 2
logger.info(
"AR-Diffusion preallocating (paged): frame_seqlen=%d num_frame_per_block=%d "
"local_attn_size=%d -> chunk_size=%d window_chunks=%d (window=%d tokens)",
frame_seqlen,
num_frame_per_block,
local_attn_size,
chunk_size,
window_chunks,
window_chunks * chunk_size,
)
self.build_kv_cache(
num_layers=num_layers,
num_kv_heads=num_kv_heads,
head_size=head_size,
self.ar_diffusion_kv_config = config
self._ar_diffusion_capability = capability
self._ar_diffusion_kv_cache_spec = spec
self._session_capacity = min(spec.session_capacity, self._MAX_RESIDENT_SESSIONS)
self.kv_cache = ARDiffusionKVCache(
config,
num_layers=spec.num_layers,
num_kv_heads=spec.num_kv_heads,
head_size=spec.head_size,
dtype=self.od_config.dtype,
block_size=chunk_size, # one pool block per chunk
max_model_len=1 << 20,
available_bytes=free_bytes,
cross_attn_length=cross_attn_length,
cross_attn_img_length=cross_attn_img_length,
local_branches=local_branches,
num_frame_per_block=num_frame_per_block,
block_size=spec.tokens_per_frame,
max_model_len=spec.max_model_len,
available_bytes=self._available_memory_bytes() if available_bytes is None else available_bytes,
kv_branches=spec.kv_branches,
session_capacity=self._session_capacity,
cross_attention_lengths=spec.cross_attention_lengths,
frames_per_block=spec.frames_per_block,
max_scratch_tokens_per_branch=spec.max_scratch_tokens_per_branch,
device=self.device,
)
logger.info(
"AR-Diffusion KV cache: blocks=%d layers=%d local_kv_heads=%d head_size=%d "
"tokens/frame=%d frames/block=%d window=%d sink=%d kv_branches=%s cross=%s "
"resident_capacity=%d requested_capacity=%d",
self.kv_cache.num_blocks,
spec.num_layers,
spec.num_kv_heads,
spec.head_size,
spec.tokens_per_frame,
spec.frames_per_block,
config.window_chunks,
config.sink_chunks,
[(kv_branch.name, kv_branch.local_index) for kv_branch in spec.kv_branches],
spec.cross_attention_lengths,
self._session_capacity,
spec.session_capacity,
)
def _new_session_state(self, session_id: str) -> ARDiffusionKVState:
if self.kv_cache is None or self._ar_diffusion_kv_cache_spec is None:
raise RuntimeError("AR-Diffusion session requested before KV cache initialization")
adapters = {
kv_branch.name: self.kv_cache.begin_request(f"ar::{session_id}::{kv_branch.name}")
for kv_branch in self._ar_diffusion_kv_cache_spec.kv_branches
}
return ARDiffusionKVState(
self.kv_cache,
session_id,
adapters,
num_layers=self._ar_diffusion_kv_cache_spec.num_layers,
)
def _release_session(
self,
session_id: str,
*,
reset_model: bool,
reason: str,
suppress_errors: bool = False,
) -> None:
"""One release path for reset, close, eviction, and failed forwards."""
state = self._sessions.pop(session_id, None)
errors: list[Exception] = []
if state is not None:
try:
state.close()
except Exception as exc: # noqa: BLE001 - notify the pipeline even if pool cleanup fails
errors.append(exc)
capability = self._ar_diffusion_capability
if capability is not None:
try:
if reset_model:
capability.reset_ar_diffusion_session(session_id)
else:
capability.close_ar_diffusion_session(session_id)
except Exception as exc: # noqa: BLE001 - preserve all lifecycle cleanup attempts
errors.append(exc)
logger.debug("AR-Diffusion released session=%s reason=%s", session_id, reason)
if errors:
if suppress_errors:
logger.warning(
"AR-Diffusion session=%s cleanup after %s had %d error(s): %s",
session_id,
reason,
len(errors),
errors,
)
else:
raise errors[0]
def reset_session(self, session_id: str) -> None:
"""Release KV and notify the pipeline to reset model-owned state."""
self._release_session(session_id, reset_model=True, reason="reset")
def close_session(self, session_id: str) -> None:
"""Release KV and notify the pipeline to drop model-owned state."""
self._release_session(session_id, reset_model=False, reason="close")
def _get_or_create_session(self, session_id: str) -> ARDiffusionKVState:
state = self._sessions.get(session_id)
if state is None:
while len(self._sessions) >= self._session_capacity:
oldest = next(iter(self._sessions))
self._release_session(oldest, reset_model=False, reason="lru_eviction")
state = self._new_session_state(session_id)
self._sessions[session_id] = state
self._sessions.move_to_end(session_id)
return state
@staticmethod
def _request_session(req: OmniDiffusionRequest) -> tuple[str, dict]:
extra_args = req.sampling_params.extra_args or {}
return str(extra_args.get("session_id") or "default"), extra_args
def execute_model(
self,
req: OmniDiffusionRequest,
kv_prefetch_job: KVPrefetchJob | None = None,
) -> DiffusionOutput:
# KV disabled -> base behavior, unchanged.
if self.kv_cache is None:
return super().execute_model(req, kv_prefetch_job=kv_prefetch_job)
capability = self._ar_diffusion_capability
if capability is None:
raise RuntimeError("AR-Diffusion capability missing after KV cache initialization")
kv = self.kv_cache
# DreamZero KV is session-scoped (the model state persists across a
# session's forwards), so the AR-Diffusion KV state is keyed by session_id and
# reused — matching how pipeline.forward resolves the model-local state.
extra_args = req.sampling_params.extra_args or {}
session_id = str(extra_args.get("session_id") or "default")
state = self._ar_diffusion_states.get(session_id)
if state is None:
pos = kv.begin_request(f"bde__{session_id}")
neg = kv.begin_request(f"bde__{session_id}__neg")
state = ARDiffusionKVState(kv, pos, neg, num_layers=kv.num_layers)
self._ar_diffusion_states[session_id] = state
# Evict the least-recently-used session(s) past the cap, freeing their
# pool blocks — mirrors DreamZeroPipeline._states so the AR-Diffusion pool can't
# outlive the model-local state map it shadows.
while len(self._ar_diffusion_states) > self._max_ar_diffusion_states:
old_id, old_state = self._ar_diffusion_states.popitem(last=False)
old_state.close()
logger.debug("AR-Diffusion evicted session=%s (LRU); freed pool blocks", old_id)
# Track recency on every access (hit or miss) so the LRU order is correct.
self._ar_diffusion_states.move_to_end(session_id)
logger.debug(
"AR-Diffusion execute_model: req=%s session=%s chunk_size=%d window_chunks=%s num_blocks=%d",
req.request_id,
session_id,
kv.spec.chunk_size,
kv.config.window_chunks,
kv.num_blocks,
)
self.pipeline._ar_diffusion_kv_state = state
# Time the worker-side forward (true per-request compute E2E). The CUDA
# sync below makes the stop reflect completed GPU work, not just dispatch.
_e2e_t0 = time.perf_counter()
session_id, extra_args = self._request_session(req)
if extra_args.get("reset", False):
self.reset_session(session_id)
state = self._get_or_create_session(session_id)
started = time.perf_counter()
try:
out = super().execute_model(req, kv_prefetch_job=kv_prefetch_job)
with capability.bind_ar_diffusion_state(session_id, state):
output = super().execute_model(req, kv_prefetch_job=kv_prefetch_job)
if self.device is not None and torch.device(self.device).type == "cuda":
torch.accelerator.synchronize(self.device)
except Exception:
# Transactional containment: a forward that died partway may have
# written some layers' K/V into allocated-but-uncommitted blocks
# and advanced model-local state. Tear the whole session down
# (engine pool blocks freed via close(); model-local state dropped)
# so the next request with this session id starts clean instead of
# tripping the pending-commit guard or reading half-written KV.
broken = self._ar_diffusion_states.pop(session_id, None)
if broken is not None:
broken.close()
states = getattr(self.pipeline, "_states", None)
if states is not None:
states.pop(session_id, None)
self._release_session(
session_id,
reset_model=False,
reason="forward_exception",
suppress_errors=True,
)
logger.warning(
"AR-Diffusion execute_model failed for session=%s; session state torn down "
"(pool blocks freed) — the next request starts a fresh session",
"AR-Diffusion forward failed for session=%s; KV and model state were released",
session_id,
)
raise
finally:
self.pipeline._ar_diffusion_kv_state = None
if self.device is not None and torch.cuda.is_available():
torch.accelerator.synchronize(self.device)
self._perf_e2e_times.append(time.perf_counter() - _e2e_t0)
return out
self._perf_e2e_times.append(time.perf_counter() - started)
if extra_args.get("close_session", False):
self.close_session(session_id)
return output
# -- cuda-graph warm-up ----------------------------------------------------
def execute_model_batch(
self,
scheduler_output: DiffusionSchedulerOutput,
od_config: OmniDiffusionConfig,
) -> BatchRunnerOutput:
"""Reject request batching until batch-aware AR state binding exists."""
raise RuntimeError(
"ARDiffusionModelRunner does not support request-batch execution; use request mode with max_num_seqs=1."
)
#: Dedicated session id for the warm-up rollout; cleaned from both state maps.
_WARMUP_SID = "__ardiffusion_warmup__"
def _synth_robot_obs(self, h: int, w: int, n_frames: int) -> dict:
"""Zero-filled roboarena observation matching the client's obs layout.
Mirrors ``examples/.../client_schedule.make_obs_from_video`` (2 ext + wrist
cameras, 7/6/1 state vectors) so it survives ``_transform_robot_obs`` exactly
like a real request. One frame -> ``(H,W,3)``; chunk -> ``(n_frames,H,W,3)``.
"""
img = np.zeros((h, w, 3), dtype=np.uint8) if n_frames == 1 else np.zeros((n_frames, h, w, 3), dtype=np.uint8)
return {
"observation/exterior_image_0_left": img,
"observation/exterior_image_1_left": img,
"observation/wrist_image_left": img,
"observation/joint_position": np.zeros(7, dtype=np.float32),
"observation/cartesian_position": np.zeros(6, dtype=np.float32),
"observation/gripper_position": np.zeros(1, dtype=np.float32),
"prompt": "warmup",
"session_id": self._WARMUP_SID,
}
def execute_stepwise(self, scheduler_output: DiffusionSchedulerOutput) -> BatchRunnerOutput:
"""Reject step execution until step-aware AR state binding exists."""
raise RuntimeError(
"ARDiffusionModelRunner does not support step execution; use request mode with step_execution=False."
)
def _warmup_ar_rollout(self) -> None:
"""Drive a synthetic rollout through one window-fill so every DiT graph is
captured at load (off the serving hot path). Frees the synthetic session and
leaves the KV pool exactly as found. Never raises — falls back to lazy capture.
"""
from vllm_omni.inputs.data import OmniDiffusionSamplingParams
kv = self.kv_cache
if kv is None:
"""Run model-provided warmup requests, or safely skip when absent."""
if self.kv_cache is None or self.pipeline is None:
return
if not isinstance(self.pipeline, SupportsARDiffusionWarmup):
logger.info("AR-Diffusion pipeline provides no warmup requests; skipping rollout warmup")
return
sid = self._WARMUP_SID
t = self.pipeline.transformer
nfpb = int(t.num_frame_per_block)
window_chunks = int(kv.config.window_chunks or 0)
if window_chunks <= 0:
return
# Distinct DiT window shapes are walked by one fill: prefill + ceil((W-1)/nfpb)
# chunk forwards. +1 optionally to also capture the reset-cycle forward.
n_forwards = 1 + math.ceil(max(0, window_chunks - 1) / nfpb)
if self.ar_diffusion_kv_config.warmup_capture_reset:
n_forwards += 1
res = self._image_resolution()
h, w = int(res[0]), int(res[1])
free_before = kv.manager.block_pool.get_num_free_blocks()
# The step cache is left active: the DiT input shape is per-chunk (the resident
# window length), and the first denoise step of every chunk always computes, so
# every window shape is captured regardless of how many later steps are skipped.
logger.info(
"AR-Diffusion cudagraph warm-up: up to %d synthetic forwards (window_chunks=%d, nfpb=%d)",
n_forwards,
window_chunks,
nfpb,
)
free_before = self.kv_cache.manager.block_pool.get_num_free_blocks()
try:
for i in range(n_forwards):
n_frames = 1 if i == 0 else 4 # client convention: 1-frame prefill, 4-frame chunks
obs = self._synth_robot_obs(h, w, n_frames)
sp = OmniDiffusionSamplingParams(extra_args={"reset": i == 0, "session_id": sid, "robot_obs": obs})
req = OmniDiffusionRequest(prompts=["warmup"], sampling_params=sp, request_id=f"ardiffusion-warmup-{i}")
self.execute_model(req)
# Stop once the resident window is full — the remaining shapes are
# already captured (the window caps/resets through the same set).
ar_state = self._ar_diffusion_states.get(sid)
if (
not self.ar_diffusion_kv_config.warmup_capture_reset
and ar_state is not None
and len(kv.window_block_ids(ar_state.pos)) >= window_chunks
):
break
except Exception as e: # noqa: BLE001 — warm-up must never break model load
logger.warning("AR-Diffusion cudagraph warm-up failed (%s); using lazy capture.", e)
for request in self.pipeline.ar_diffusion_warmup_requests(sid):
self.execute_model(request)
except Exception as exc: # noqa: BLE001 - warmup must not make model load fail
logger.warning("AR-Diffusion rollout warmup failed (%s); using lazy capture", exc)
finally:
# Free the synthetic session from BOTH maps (KV pool + pipeline state).
ar_state = self._ar_diffusion_states.pop(sid, None)
if ar_state is not None:
try:
ar_state.close() # return KV blocks to the BlockPool
except Exception as e: # noqa: BLE001
logger.warning("AR-Diffusion warm-up: state.close() failed (%s)", e)
try:
self.pipeline._states.pop(sid, None)
except Exception: # noqa: BLE001
pass
# Drop any per-forward timings the warm-up recorded so serving stats start clean.
self._release_session(
sid,
reset_model=False,
reason="warmup_complete",
suppress_errors=True,
)
self._perf_e2e_times.clear()
free_after = kv.manager.block_pool.get_num_free_blocks()
free_after = self.kv_cache.manager.block_pool.get_num_free_blocks()
if free_after != free_before:
logger.warning(
"AR-Diffusion warm-up: KV pool not restored (free %d -> %d) — possible leak",
"AR-Diffusion warmup did not restore the KV pool (free %d -> %d)",
free_before,
free_after,
)
else:
logger.info(
"AR-Diffusion cudagraph warm-up complete; KV pool restored (%d free blocks)",
free_after,
)
def _image_resolution(self) -> list[int]:
mc = getattr(self.od_config, "model_config", None) or {}
psc = (mc.get("policy_server_config") if isinstance(mc, dict) else None) or {}
return psc.get("image_resolution", [180, 320])