diff --git a/.claude/skills/add-diffusion-model/SKILL.md b/.claude/skills/add-diffusion-model/SKILL.md index 353422e6e..a7e0bbf9a 100644 --- a/.claude/skills/add-diffusion-model/SKILL.md +++ b/.claude/skills/add-diffusion-model/SKILL.md @@ -1,6 +1,6 @@ --- name: add-diffusion-model -description: Add a new diffusion model (text-to-image, text-to-video, image-to-video, text-to-audio, image editing) to vLLM-Omni. Use when integrating a new diffusion model, porting a diffusers pipeline or a custom model repo to vllm-omni, creating a new DiT transformer adapter, or adding diffusion model support. +description: Add a new diffusion model (text-to-image, text-to-video, image-to-video, text-to-audio, image editing) to vLLM-Omni, including Cache-DiT acceleration and parallelism support (TP, SP/USP, CFG-Parallel, HSDP). Use when integrating a new diffusion model, porting a diffusers pipeline or a custom model repo to vllm-omni, creating a new DiT transformer adapter, adding diffusion model support, or enabling multi-GPU parallelism and cache acceleration for an existing model. --- # Adding a Diffusion Model to vLLM-Omni @@ -291,18 +291,244 @@ Required updates: Create `tests/e2e/online_serving/test_your_model_expansion.py`. +### Step 9: Add Cache-DiT Acceleration + +Cache-DiT accelerates inference by caching intermediate computation results across denoising steps. After your model is working correctly on a single GPU, add cache-dit support. + +See [references/cache-dit-patterns.md](references/cache-dit-patterns.md) for detailed code patterns. + +#### 9a. Determine your model type + +| Model Type | Description | Action | +|------------|-------------|--------| +| **Standard single-transformer** | One transformer with one `ModuleList` of blocks | No code needed — `CacheDiTBackend` auto-detects via `enable_cache_for_dit()` | +| **Multi-block-list** | One transformer with multiple block lists (e.g., `transformer_blocks` + `single_transformer_blocks`) | Write custom enabler with `BlockAdapter` | +| **Dual-transformer** | Two transformers (e.g., high-noise + low-noise) | Write custom enabler with `BlockAdapter` wrapping both | + +#### 9b. Standard models — verify automatic support + +For standard single-transformer models, test directly: + +```python +omni = Omni( + model="your-model-name", + cache_backend="cache_dit", + cache_config={ + "Fn_compute_blocks": 1, + "Bn_compute_blocks": 0, + "max_warmup_steps": 4, + } +) +``` + +Check logs for "Cache-dit enabled successfully on xxx". If it works, skip to Step 9e. + +#### 9c. Custom architectures — write a custom enabler + +For multi-block-list or dual-transformer models, write a custom enabler function: + +```python +from cache_dit import BlockAdapter, ForwardPattern, ParamsModifier, DBCacheConfig + +def enable_cache_for_your_model(pipeline, cache_config): + db_cache_config = DBCacheConfig( + num_inference_steps=None, + Fn_compute_blocks=cache_config.Fn_compute_blocks, + Bn_compute_blocks=cache_config.Bn_compute_blocks, + max_warmup_steps=cache_config.max_warmup_steps, + max_cached_steps=cache_config.max_cached_steps, + max_continuous_cached_steps=cache_config.max_continuous_cached_steps, + residual_diff_threshold=cache_config.residual_diff_threshold, + ) + + cache_dit.enable_cache( + BlockAdapter( + transformer=pipeline.transformer, + blocks=[ + pipeline.transformer.transformer_blocks, + pipeline.transformer.single_transformer_blocks, + ], + forward_pattern=[ForwardPattern.Pattern_1, ForwardPattern.Pattern_1], + params_modifiers=[ParamsModifier(...)], + ), + cache_config=db_cache_config, + ) + + def refresh_cache_context(pipeline, num_inference_steps, verbose=True): + cache_dit.refresh_context( + pipeline.transformer, num_inference_steps=num_inference_steps, verbose=verbose + ) + return refresh_cache_context +``` + +#### 9d. Register the custom enabler + +Add your enabler to `CUSTOM_DIT_ENABLERS` in `vllm_omni/diffusion/cache/cache_dit_backend.py`: + +```python +CUSTOM_DIT_ENABLERS = { + "Wan22Pipeline": enable_cache_for_wan22, + "LongCatImagePipeline": enable_cache_for_longcat_image, + "YourModelPipeline": enable_cache_for_your_model, # Add here +} +``` + +#### 9e. Test Cache-DiT + +```python +omni = Omni( + model="your-model-name", + cache_backend="cache_dit", + cache_config={ + "Fn_compute_blocks": 1, "Bn_compute_blocks": 0, + "max_warmup_steps": 4, "residual_diff_threshold": 0.24, + } +) +images = omni.generate("a beautiful landscape", + OmniDiffusionSamplingParams(num_inference_steps=50)) +``` + +**Verify**: 1) logs show cache enabled, 2) 1.5-2x speedup, 3) output quality acceptable vs baseline. + +If quality degrades, lower `residual_diff_threshold` (try 0.12-0.18) or increase `max_warmup_steps` (try 6-8). + +--- + +### Step 10: Add Parallelism Support + +After the model works on a single GPU, add multi-GPU parallelism. Add each type incrementally, testing after each addition. + +See [references/parallelism-patterns.md](references/parallelism-patterns.md) for detailed code patterns and API reference. + +**Recommended order**: TP → SP/USP → CFG Parallel → HSDP + +#### 10a. Tensor Parallelism (TP) + +Shards DiT linear layers across GPUs. Requires code changes in the transformer. + +**What to change in the transformer**: +1. Replace `nn.Linear` with `ColumnParallelLinear` / `RowParallelLinear` / `QKVParallelLinear` +2. Update `load_weights()` to handle QKV fusion with `stacked_params_mapping` +3. Use `self.to_qkv.num_heads` (local heads) instead of total heads for split sizes + +```python +from vllm.model_executor.layers.linear import ( + QKVParallelLinear, RowParallelLinear, ColumnParallelLinear, +) + +# Attention: QKV → RowParallel output +self.to_qkv = QKVParallelLinear(dim, head_dim, num_heads, num_kv_heads) +self.to_out = RowParallelLinear(dim, dim, input_is_parallel=True) + +# FFN: ColumnParallel → RowParallel +self.w1 = ColumnParallelLinear(dim, ffn_dim) +self.w2 = RowParallelLinear(ffn_dim, dim, input_is_parallel=True) +``` + +**Constraints**: `num_heads % tp_size == 0` and `num_kv_heads % tp_size == 0`. + +**Test**: `--tensor-parallel-size 2` + +#### 10b. Sequence Parallelism (SP / USP) + +Splits sequence tokens across GPUs. Non-intrusive via `_sp_plan` on the transformer class — no changes to `forward()`. + +**What to change in the transformer**: + +Add `_sp_plan` class attribute: + +```python +from vllm_omni.diffusion.distributed.sp_plan import ( + SequenceParallelInput, SequenceParallelOutput, +) + +class YourTransformer(nn.Module): + _sp_plan = { + "blocks.0": { + "hidden_states": SequenceParallelInput(split_dim=1, expected_dims=3), + }, + "proj_out": SequenceParallelOutput(gather_dim=1, expected_dims=3), + } +``` + +If inline tensor ops (e.g., `torch.cat`) exist between shard/gather points, extract them into `nn.Module` submodules so hooks can intercept them. + +For RoPE that needs splitting, add an entry for the RoPE module with `split_output=True`. + +**Test**: `--ulysses-degree 2` (offline) or `--usp 2` (online serving) + +#### 10c. CFG Parallel + +Distributes positive/negative CFG branches across 2 GPUs. Requires the pipeline to inherit `CFGParallelMixin`. + +**What to change in the pipeline**: + +```python +from vllm_omni.diffusion.distributed.cfg_parallel import CFGParallelMixin + +class YourPipeline(nn.Module, CFGParallelMixin): + def diffuse(self, ...) -> torch.Tensor: + for i, t in enumerate(timesteps): + positive_kwargs = {...} + negative_kwargs = {...} if do_true_cfg else None + noise_pred = self.predict_noise_maybe_with_cfg( + do_true_cfg=do_true_cfg, true_cfg_scale=cfg_scale, + positive_kwargs=positive_kwargs, negative_kwargs=negative_kwargs, + ) + latents = self.scheduler_step_maybe_with_cfg( + noise_pred, t, latents, do_true_cfg + ) + return latents +``` + +Override `predict_noise()` if your transformer call is non-standard. Override `combine_cfg_noise()` for multi-output models (e.g., video + audio). + +**Constraint**: Exactly 2 GPUs. Only for models using classifier-free guidance. + +**Test**: `--cfg-parallel-size 2` + +#### 10d. HSDP (Hybrid Sharded Data Parallel) + +Shards transformer weights via PyTorch FSDP2 to reduce per-GPU VRAM. No code changes to the forward pass — just add a class attribute. + +**What to change in the transformer**: + +```python +class YourTransformer(nn.Module): + @staticmethod + def _is_transformer_block(name: str, module) -> bool: + return "blocks" in name and name.split(".")[-1].isdigit() + + _hsdp_shard_conditions = [_is_transformer_block] +``` + +**Constraint**: Cannot combine with TP. For standalone HSDP, set `hsdp_shard_size` explicitly. + +**Test**: `--use-hsdp` or `DiffusionParallelConfig(use_hsdp=True)` + +#### 10e. Update parallelism documentation + +After adding parallelism support, update: +1. `docs/user_guide/diffusion/parallelism_acceleration.md` — add your model to the support table +2. Record which parallelism methods are supported (USP, Ring, CFG, TP, HSDP, VAE-Patch) + +--- + ## Iterative Development Tips 1. **Start minimal**: Basic generation first, no parallelism/caching 2. **Use `--enforce-eager`**: Disable torch.compile during debugging 3. **Use small models**: Test with smaller variants first 4. **Check tensor shapes**: Most errors are reshape mismatches in attention -5. **Add parallelism incrementally**: TP → SP → CFG parallel +5. **Add features incrementally**: Single GPU → TP → SP → CFG → HSDP → Cache-DiT 6. **For custom models**: Get the model running with the original code first, then progressively replace components with vllm-omni equivalents +7. **Cache-DiT before parallelism tuning**: Cache-DiT is lossy — verify quality at baseline before combining with parallelism +8. **Combine lossless + lossy**: e.g., TP + SP + Cache-DiT for maximum throughput ## Reference Files - [Transformer Adaptation](references/transformer-adaptation.md) — porting transformers from diffusers - [Custom Model Patterns](references/custom-model-patterns.md) — patterns for non-diffusers models -- [Parallelism Patterns](references/parallelism-patterns.md) — TP, SP, CFG parallel +- [Parallelism Patterns](references/parallelism-patterns.md) — TP, SP/USP, CFG parallel, HSDP implementation details +- [Cache-DiT Patterns](references/cache-dit-patterns.md) — cache-dit acceleration for standard and custom architectures - [Troubleshooting](references/troubleshooting.md) — common errors and fixes diff --git a/.claude/skills/add-diffusion-model/references/cache-dit-patterns.md b/.claude/skills/add-diffusion-model/references/cache-dit-patterns.md new file mode 100644 index 000000000..d34ce0e0f --- /dev/null +++ b/.claude/skills/add-diffusion-model/references/cache-dit-patterns.md @@ -0,0 +1,254 @@ +# Cache-DiT Patterns Reference + +## Overview + +Cache-DiT accelerates Diffusion Transformers by caching intermediate computation results across denoising steps. Adjacent steps produce similar features, so redundant computations can be skipped. + +Three caching strategies: +- **DBCache**: Dynamic block-level caching — selectively computes or caches transformer blocks based on residual differences +- **TaylorSeer**: Calibration-based prediction using Taylor expansion to estimate block outputs +- **SCM** (Step Computation Masking): Dynamic step skipping based on configurable policies + +**Typical speedup**: 1.5-2.5x depending on model and configuration. + +**Official docs**: https://docs.vllm.ai/projects/vllm-omni/en/latest/design/feature/cache_dit + +## Architecture + +vLLM-Omni integrates cache-dit through `CacheDiTBackend`: + +| Component | Purpose | +|-----------|---------| +| `CacheDiTBackend` | Unified backend — auto-selects enabler (standard or custom) | +| `enable_cache_for_dit()` | Default enabler for standard single-transformer models | +| `CUSTOM_DIT_ENABLERS` dict | Registry of custom enablers keyed by pipeline class name | +| `BlockAdapter` | Wraps complex architectures (multi-block-list or multi-transformer) | +| `ForwardPattern` | Specifies block forward signature: `Pattern_0`, `Pattern_1`, `Pattern_2` | +| `ParamsModifier` | Per-transformer or per-block-list config customization | +| `DBCacheConfig` | Configuration for DBCache parameters | +| `cache_dit.refresh_context()` | Updates cache context when `num_inference_steps` changes | + +**Source files**: +- `vllm_omni/diffusion/cache/cache_dit_backend.py` — `CacheDiTBackend`, enablers, `CUSTOM_DIT_ENABLERS` +- `vllm_omni/diffusion/cache/` — cache backend implementations + +## Standard Models: Automatic Support + +Most DiT models follow this pattern: +- Single transformer with one `nn.ModuleList` of blocks +- Standard forward signature +- Compatible with cache-dit's automatic detection + +**Examples**: Qwen-Image, Z-Image, FLUX + +No code changes needed. `CacheDiTBackend` automatically uses `enable_cache_for_dit()`: + +```python +from vllm_omni import Omni + +omni = Omni( + model="Qwen/Qwen-Image", + cache_backend="cache_dit", + cache_config={ + "Fn_compute_blocks": 1, + "Bn_compute_blocks": 0, + "max_warmup_steps": 4, + } +) +``` + +What happens automatically: + +```python +def enable_cache_for_dit(pipeline, cache_config): + db_cache_config = DBCacheConfig( + num_inference_steps=None, + Fn_compute_blocks=cache_config.Fn_compute_blocks, + Bn_compute_blocks=cache_config.Bn_compute_blocks, + max_warmup_steps=cache_config.max_warmup_steps, + max_cached_steps=cache_config.max_cached_steps, + max_continuous_cached_steps=cache_config.max_continuous_cached_steps, + residual_diff_threshold=cache_config.residual_diff_threshold, + ) + + cache_dit.enable_cache(pipeline.transformer, cache_config=db_cache_config) + + def refresh_cache_context(pipeline, num_inference_steps, verbose=True): + cache_dit.refresh_context( + pipeline.transformer, num_inference_steps=num_inference_steps, verbose=verbose + ) + return refresh_cache_context +``` + +## Custom Architectures: Writing Custom Enablers + +### When you need a custom enabler + +- Model has multiple block lists in one transformer (e.g., `transformer_blocks` + `single_transformer_blocks`) +- Model has two transformers (e.g., high-noise + low-noise like Wan2.2) +- Model uses non-standard block forward signature + +### Pattern 1: Multi-Block-List (LongCat-Image style) + +Single transformer with two block lists: + +```python +import cache_dit +from cache_dit import BlockAdapter, ForwardPattern, ParamsModifier, DBCacheConfig + +def enable_cache_for_your_model(pipeline, cache_config): + db_cache_config = DBCacheConfig( + num_inference_steps=None, + Fn_compute_blocks=cache_config.Fn_compute_blocks, + Bn_compute_blocks=cache_config.Bn_compute_blocks, + max_warmup_steps=cache_config.max_warmup_steps, + max_cached_steps=cache_config.max_cached_steps, + max_continuous_cached_steps=cache_config.max_continuous_cached_steps, + residual_diff_threshold=cache_config.residual_diff_threshold, + ) + + cache_dit.enable_cache( + BlockAdapter( + transformer=pipeline.transformer, + blocks=[ + pipeline.transformer.transformer_blocks, + pipeline.transformer.single_transformer_blocks, + ], + forward_pattern=[ForwardPattern.Pattern_1, ForwardPattern.Pattern_1], + params_modifiers=[ParamsModifier(...)], + ), + cache_config=db_cache_config, + ) + + def refresh_cache_context(pipeline, num_inference_steps, verbose=True): + cache_dit.refresh_context( + pipeline.transformer, num_inference_steps=num_inference_steps, verbose=verbose + ) + return refresh_cache_context +``` + +For single transformer with multiple block lists, `refresh_context` works the same as standard models — call it once on the transformer. + +### Pattern 2: Dual-Transformer (Wan2.2 style) + +Two transformers with separate configs: + +```python +def enable_cache_for_dual_transformer(pipeline, cache_config): + db_cache_config = DBCacheConfig(...) + + cache_dit.enable_cache( + BlockAdapter( + transformer=[pipeline.transformer, pipeline.transformer_2], + blocks=[pipeline.transformer.blocks, pipeline.transformer_2.blocks], + forward_pattern=[ForwardPattern.Pattern_2, ForwardPattern.Pattern_2], + params_modifiers=[ + ParamsModifier(...), # Config for transformer 1 + ParamsModifier(...), # Config for transformer 2 + ], + ), + cache_config=db_cache_config, + ) + + def refresh_cache_context(pipeline, num_inference_steps, verbose=True): + high_steps, low_steps = _split_inference_steps(num_inference_steps) + cache_dit.refresh_context( + pipeline.transformer, num_inference_steps=high_steps, verbose=verbose + ) + cache_dit.refresh_context( + pipeline.transformer_2, num_inference_steps=low_steps, verbose=verbose + ) + return refresh_cache_context +``` + +Key difference: `refresh_context` must be called on **each transformer separately** with its own step count. + +### Choosing the ForwardPattern + +| Pattern | Block forward signature | Example models | +|---------|------------------------|----------------| +| `Pattern_0` | `block(hidden_states, **kwargs)` → residual added inside block | Default | +| `Pattern_1` | `block(hidden_states, **kwargs)` → returns `(hidden_states, ...)` tuple | FLUX-style single blocks | +| `Pattern_2` | `block(hidden_states, **kwargs)` → `(hidden_states, ...)` with different residual pattern | Wan2.2 blocks | + +Inspect your block's `forward()` return type and residual connection pattern to choose the right one. See [Cache-DiT API Reference](https://cache-dit.readthedocs.io/en/latest/user_guide/CACHE_API/) for details. + +## Registering Custom Enablers + +Add your enabler to `CUSTOM_DIT_ENABLERS` in `vllm_omni/diffusion/cache/cache_dit_backend.py`: + +```python +CUSTOM_DIT_ENABLERS = { + "Wan22Pipeline": enable_cache_for_wan22, + "LongCatImagePipeline": enable_cache_for_longcat_image, + "YourModelPipeline": enable_cache_for_your_model, +} +``` + +The key must match `pipeline.__class__.__name__`. + +## Configuration Parameters + +| Parameter | Default | Description | +|-----------|---------|-------------| +| `Fn_compute_blocks` | 1 | Number of blocks to always compute at the front | +| `Bn_compute_blocks` | 0 | Number of blocks to always compute at the back | +| `max_warmup_steps` | 4 | Steps to run without caching at the beginning | +| `max_cached_steps` | — | Max total cached steps | +| `max_continuous_cached_steps` | — | Max consecutive cached steps | +| `residual_diff_threshold` | 0.24 | Threshold for deciding whether to cache a block | + +### Tuning for quality vs speed + +| Goal | Adjustments | +|------|-------------| +| **More speed, acceptable quality loss** | Higher `residual_diff_threshold` (0.24-0.4), lower `max_warmup_steps` (2-4) | +| **Better quality, less speed** | Lower `residual_diff_threshold` (0.12-0.18), higher `max_warmup_steps` (6-8), lower `max_continuous_cached_steps` (2) | + +## Testing + +```python +from vllm_omni import Omni +from vllm_omni.inputs.data import OmniDiffusionSamplingParams + +omni = Omni( + model="your-model-name", + cache_backend="cache_dit", + cache_config={ + "Fn_compute_blocks": 1, + "Bn_compute_blocks": 0, + "max_warmup_steps": 4, + "residual_diff_threshold": 0.24, + } +) +images = omni.generate( + "a beautiful landscape", + OmniDiffusionSamplingParams(num_inference_steps=50), +) +``` + +CLI (online serving): + +```bash +vllm serve your-model --omni --port 8098 \ + --cache-backend cache_dit \ + --cache-config '{"Fn_compute_blocks": 1, "Bn_compute_blocks": 0, "max_warmup_steps": 4}' +``` + +**Verification checklist**: +1. Logs show "Cache-dit enabled successfully on xxx" +2. Performance: 1.5-2x speedup vs no cache +3. Quality: compare output with `cache_backend=None` + +## Excluded Models + +Models listed in `_NO_CACHE_ACCELERATION` in `vllm_omni/diffusion/registry.py` do not support cache-dit (e.g., `NextStep11Pipeline`, `StableDiffusionPipeline`). Check this set before attempting to enable cache-dit. + +## Reference Implementations + +| Model | Path | Notes | +|-------|------|-------| +| Standard DiT | `cache_dit_backend.py::enable_cache_for_dit` | Default enabler, automatic | +| Wan2.2 | `cache_dit_backend.py::enable_cache_for_wan22` | Dual-transformer, auto-detects mode | +| LongCat | `cache_dit_backend.py::enable_cache_for_longcat_image` | Multi-block-list | +| BAGEL | `cache_dit_backend.py::enable_cache_for_bagel` | Complex omni model | diff --git a/.claude/skills/add-diffusion-model/references/parallelism-patterns.md b/.claude/skills/add-diffusion-model/references/parallelism-patterns.md index c6f32d1fa..933e2d232 100644 --- a/.claude/skills/add-diffusion-model/references/parallelism-patterns.md +++ b/.claude/skills/add-diffusion-model/references/parallelism-patterns.md @@ -1,8 +1,32 @@ # Parallelism Patterns Reference +## Overview + +vLLM-Omni supports multiple parallelism strategies for diffusion models. Each targets a different bottleneck: + +| Strategy | Splits | Best For | Constraint | +|----------|--------|----------|------------| +| Tensor Parallel (TP) | Model layers across GPUs | Latency reduction, large models | Requires fast GPU interconnect, `num_heads % tp == 0` | +| Sequence Parallel (SP/USP) | Sequence tokens across GPUs | Long sequences (video, high-res) | Near-linear scaling | +| CFG Parallel | Positive/negative CFG branches | Models using classifier-free guidance | Exactly 2 GPUs | +| HSDP | Weight shards via FSDP2 | VRAM reduction | Cannot combine with TP | +| VAE Patch Parallel | VAE decode spatial tiles | Large VAE outputs | Auto-enables tiling | + +**Recommended integration order**: TP → SP → CFG Parallel → HSDP + +**Official design docs**: +- TP: https://docs.vllm.ai/projects/vllm-omni/en/latest/design/feature/tensor_parallel +- SP: https://docs.vllm.ai/projects/vllm-omni/en/latest/design/feature/sequence_parallel +- CFG: https://docs.vllm.ai/projects/vllm-omni/en/latest/design/feature/cfg_parallel +- HSDP: https://docs.vllm.ai/projects/vllm-omni/en/latest/design/feature/hsdp + +--- + ## Tensor Parallelism (TP) -Replace standard `nn.Linear` with vLLM's parallel linear layers: +Replace standard `nn.Linear` with vLLM's parallel linear layers. This is the most invasive change but provides direct VRAM savings and compute speedup. + +### Layer replacement rules | Pattern | vLLM Layer | When to Use | |---------|-----------|-------------| @@ -11,19 +35,24 @@ Replace standard `nn.Linear` with vLLM's parallel linear layers: | QKV projection | `QKVParallelLinear` | Fused Q/K/V for self-attention | | Single Q or K or V | `ColumnParallelLinear` | Separate projections (cross-attention) | | Attention output | `RowParallelLinear` | Output projection after attention | +| Must not shard | `ReplicatedLinear` | Layers that must stay replicated | + +### MLP Block (Up-Down Pattern) ```python from vllm.model_executor.layers.linear import ( - ColumnParallelLinear, - RowParallelLinear, - QKVParallelLinear, + ColumnParallelLinear, RowParallelLinear, ) class TPFeedForward(nn.Module): def __init__(self, dim, ffn_dim): super().__init__() - self.fc1 = ColumnParallelLinear(dim, ffn_dim) - self.fc2 = RowParallelLinear(ffn_dim, dim) + self.fc1 = ColumnParallelLinear(dim, ffn_dim, bias=False, return_bias=False) + self.fc2 = RowParallelLinear( + ffn_dim, dim, bias=False, + input_is_parallel=True, # Input already sharded from fc1 + return_bias=False, + ) def forward(self, x): x, _ = self.fc1(x) @@ -32,83 +61,511 @@ class TPFeedForward(nn.Module): return x ``` -**TP constraints**: `hidden_dim`, `num_heads`, and `num_kv_heads` must be divisible by `tp_size`. +### Attention Block (QKV-Out Pattern) + +```python +from vllm.model_executor.layers.linear import QKVParallelLinear, RowParallelLinear +from vllm_omni.diffusion.attention.layer import Attention + +class TPSelfAttention(nn.Module): + def __init__(self, dim, num_heads, num_kv_heads=None): + super().__init__() + num_kv_heads = num_kv_heads or num_heads + self.head_dim = dim // num_heads + + self.to_qkv = QKVParallelLinear( + hidden_size=dim, + head_size=self.head_dim, + total_num_heads=num_heads, + total_num_kv_heads=num_kv_heads, + bias=False, + return_bias=False, + ) + self.to_out = RowParallelLinear( + dim, dim, bias=False, + input_is_parallel=True, + return_bias=False, + ) + self.attn = Attention( + num_heads=self.to_qkv.num_heads, # Local heads per GPU + head_size=self.head_dim, + softmax_scale=1.0 / (self.head_dim ** 0.5), + causal=False, + num_kv_heads=self.to_qkv.num_kv_heads, # Local KV heads per GPU + ) + + def forward(self, x): + qkv, _ = self.to_qkv(x) + q, k, v = qkv.split( + [self.to_qkv.num_heads * self.head_dim, + self.to_qkv.num_kv_heads * self.head_dim, + self.to_qkv.num_kv_heads * self.head_dim], + dim=-1, + ) + B, S, _ = x.shape + q = q.view(B, S, self.to_qkv.num_heads, self.head_dim) + k = k.view(B, S, self.to_qkv.num_kv_heads, self.head_dim) + v = v.view(B, S, self.to_qkv.num_kv_heads, self.head_dim) + out = self.attn(q, k, v) + out = out.reshape(B, S, -1) + out, _ = self.to_out(out) + return out +``` + +### QKV Fusion in load_weights + +When you fuse separate Q/K/V into `QKVParallelLinear`, map diffusers' separate weight names: + +```python +stacked_params_mapping = [ + ("to_qkv", "to_q", "q"), + ("to_qkv", "to_k", "k"), + ("to_qkv", "to_v", "v"), +] + +def load_weights(self, weights): + params = dict(self.named_parameters()) + loaded = set() + for name, tensor in weights: + for fused_name, orig_name, shard_id in stacked_params_mapping: + if orig_name in name: + name = name.replace(orig_name, fused_name) + param = params[name] + param.weight_loader(param, tensor, shard_id) + loaded.add(name) + break + else: + if name in params: + param = params[name] + if hasattr(param, "weight_loader"): + param.weight_loader(param, tensor) + else: + default_weight_loader(param, tensor) + loaded.add(name) + return loaded +``` ### RMSNorm with TP -When RMSNorm sits between TP-sharded dimensions, use `DistributedRMSNorm` from the Wan2.2 implementation pattern — it computes global RMS via all-reduce across TP ranks. +When RMSNorm sits between TP-sharded dimensions, use `DistributedRMSNorm` — it computes global RMS via all-reduce across TP ranks. See the Wan2.2 implementation for the pattern. -## CFG Parallelism +### TP Constraints -Inherit `CFGParallelMixin` in your pipeline and implement `predict_noise()`: +- `num_heads % tp_size == 0` +- `num_kv_heads % tp_size == 0` +- Use `self.to_qkv.num_heads` (local per-GPU count), not total heads, for split sizes -```python -from vllm_omni.diffusion.distributed.cfg_parallel.cfg_parallel import CFGParallelMixin +### Testing TP -class MyPipeline(nn.Module, CFGParallelMixin): - def predict_noise(self, model, latent_model_input, t, prompt_embeds, **kwargs): - return model(latent_model_input, t, prompt_embeds, **kwargs) - - def forward(self, req): - # In the denoising loop: - noise_pred = self.predict_noise_maybe_with_cfg( - model=self.transformer, - sample=latents, - timestep=t, - prompt_embeds=prompt_embeds, - guidance_scale=guidance_scale, - do_cfg=guidance_scale > 1.0, - ) - latents = self.scheduler_step_maybe_with_cfg( - self.scheduler, noise_pred, t, latents - ) +```bash +python text_to_image.py --model Your-org/your-model \ + --tensor-parallel-size 2 --output "tp_test.png" ``` -## Sequence Parallelism (SP) +**Verify**: speedup, memory reduction proportional to TP size, quality matches single-GPU. -SP is applied non-intrusively via the `_sp_plan` dict on the transformer class. The framework applies hooks at module boundaries to shard/gather sequences. +### Reference implementations + +| Model | Path | +|-------|------| +| Z-Image | `vllm_omni/diffusion/models/z_image/z_image_transformer.py` | +| FLUX | `vllm_omni/diffusion/models/flux/flux_transformer.py` | +| Qwen-Image | `vllm_omni/diffusion/models/qwen_image/qwen_image_transformer.py` | + +--- + +## Sequence Parallelism (SP / USP) + +SP splits sequence tokens across GPUs using Ulysses (all-to-all) or Ring (P2P) communication. It is applied non-intrusively via the `_sp_plan` dict — no changes to `forward()` logic. + +### Approach 1: Non-Intrusive `_sp_plan` (Recommended) + +The framework automatically registers hooks to shard inputs and gather outputs at `nn.Module` boundaries. + +#### Step 1: Identify module boundaries + +Find where tensors need sharding/gathering: + +```python +class MyTransformer(nn.Module): + def __init__(self): + self.patch_embed = PatchEmbed() # Before blocks + self.pos_embed = RoPE() # RoPE may need splitting + self.blocks = nn.ModuleList([...]) # Blocks process sharded x + self.norm_out = LayerNorm() + self.proj_out = Linear() # Gather after this + + def forward(self, x): + x = self.patch_embed(x) + pos = self.pos_embed(x) + for block in self.blocks: + x = block(x, pos) + x = self.norm_out(x) + return self.proj_out(x) +``` + +#### Step 2: Handle inline operations + +`_sp_plan` hooks only work at `nn.Module` boundaries. Inline ops like `torch.cat()` must be extracted into submodules: + +```python +# BAD: Inline — hooks can't intercept +unified = torch.cat([x, cap_feats], dim=1) + +# GOOD: Extract into submodule +class UnifiedPrepare(nn.Module): + def forward(self, x, cap_feats): + return torch.cat([x, cap_feats], dim=1) + +self.unified_prepare = UnifiedPrepare() +unified = self.unified_prepare(x, cap_feats) +``` + +Common cases: `torch.cat()`, `pad_sequence()`, `tensor.reshape()`, complex preprocessing. + +#### Step 3: Write `_sp_plan` + +**Pattern 1: Shard at first block, gather at output** (most common) ```python from vllm_omni.diffusion.distributed.sp_plan import ( - SequenceParallelInput, - SequenceParallelOutput, + SequenceParallelInput, SequenceParallelOutput, ) -class MyTransformer(nn.Module): +class StandardTransformer(nn.Module): _sp_plan = { - # Split hidden_states input on dim=1 before first block - "blocks.0": SequenceParallelInput(split_dim=1), - # Gather output on dim=1 after final projection - "proj_out": SequenceParallelOutput(gather_dim=1), + "blocks.0": { + "hidden_states": SequenceParallelInput(split_dim=1, expected_dims=3), + }, + "proj_out": SequenceParallelOutput(gather_dim=1, expected_dims=3), } ``` -For RoPE that needs splitting, add an entry for the RoPE module: +**Pattern 2: Shard RoPE outputs separately** ```python -_sp_plan = { - "rope": SequenceParallelInput(split_dim=1, split_output=True, auto_pad=True), - "blocks.0": SequenceParallelInput(split_dim=1), - "proj_out": SequenceParallelOutput(gather_dim=1), -} +class TransformerWithRoPE(nn.Module): + _sp_plan = { + "rope": { + 0: SequenceParallelInput(split_dim=1, expected_dims=4, split_output=True), + 1: SequenceParallelInput(split_dim=1, expected_dims=4, split_output=True), + }, + "blocks.0": { + "hidden_states": SequenceParallelInput(split_dim=1, expected_dims=3), + }, + "proj_out": SequenceParallelOutput(gather_dim=1, expected_dims=3), + } ``` -The `auto_pad=True` flag handles variable sequence lengths by padding to be divisible by SP degree and creating attention masks accordingly. +**Pattern 3: Dual-stream (shard image, replicate text)** -## VAE Patch Parallelism +```python +class DualStreamTransformer(nn.Module): + _sp_plan = { + "rope_preparer": { + 2: SequenceParallelInput(split_dim=0, expected_dims=2, split_output=True), + 3: SequenceParallelInput(split_dim=0, expected_dims=2, split_output=True), + }, + "transformer_blocks.0": { + "hidden_states": SequenceParallelInput(split_dim=1, expected_dims=3), + }, + "proj_out": SequenceParallelOutput(gather_dim=1, expected_dims=3), + } +``` -If using `DistributedAutoencoderKLWan` or similar distributed VAE, the framework handles spatial sharding automatically. Set `vae_patch_parallel_size` in the parallel config. +### API Reference + +**SequenceParallelInput**: + +| Parameter | Type | Description | +|-----------|------|-------------| +| `split_dim` | int | Dimension to split (usually 1 for sequence) | +| `expected_dims` | int/None | Expected tensor rank for validation | +| `split_output` | bool | `False`: shard input params; `True`: shard output tensors | +| `auto_pad` | bool | Auto-pad if sequence not divisible by world_size | + +**SequenceParallelOutput**: + +| Parameter | Type | Description | +|-----------|------|-------------| +| `gather_dim` | int | Dimension to gather (usually 1 for sequence) | +| `expected_dims` | int/None | Expected tensor rank for validation | + +**Module naming**: + +| Key | Meaning | +|-----|---------| +| `"blocks.0"` | First element of ModuleList | +| `"blocks.*"` | All elements of ModuleList | +| `"rope"` | Named submodule | + +**Dictionary value types**: + +| Key type | split_output | Description | +|----------|-------------|-------------| +| `"param_name"` (str) | False | Shard input parameter by name | +| `0, 1, ...` (int) | True | Shard output tuple by index | + +### Approach 2: Intrusive Modification (Complex Cases) + +For dynamic sharding logic that can't be expressed via `_sp_plan`: + +```python +from vllm_omni.diffusion.distributed.sp_sharding import sp_shard, sp_gather + +def forward(self, hidden_states, ...): + if self.parallel_config.sequence_parallel_size > 1: + hidden_states = sp_shard(hidden_states, dim=1) + for block in self.blocks: + hidden_states = block(hidden_states) + if self.parallel_config.sequence_parallel_size > 1: + hidden_states = sp_gather(hidden_states, dim=1) + return hidden_states +``` + +Use intrusive modification as a last resort — `_sp_plan` is preferred for maintainability. + +### UAA Mode (Experimental) + +`ulysses_mode="advanced_uaa"` handles arbitrary sequence lengths and head counts that aren't divisible by `ulysses_degree`. Uses variable all-to-all split sizes and temporary head padding. + +### Combining SP methods + +Ulysses and Ring can be combined: `ulysses_degree × ring_degree = total SP GPUs`. + +```python +DiffusionParallelConfig(ulysses_degree=2, ring_degree=2) # 4 GPUs total +``` + +### Testing SP + +```bash +# Offline +python text_to_image.py --model Your-model --ulysses-degree 2 + +# Online serving +vllm serve Your-model --omni --usp 2 +``` + +### Reference implementations + +| Model | Path | +|-------|------| +| Qwen-Image | `vllm_omni/diffusion/models/qwen_image/qwen_image_transformer.py` | +| Wan2.2 | `vllm_omni/diffusion/models/wan2_2/wan2_2_transformer.py` | +| Z-Image | `vllm_omni/diffusion/models/z_image/z_image_transformer.py` | + +--- + +## CFG Parallelism + +Distributes positive/negative Classifier-Free Guidance branches across 2 GPUs. + +### Implementation + +Inherit `CFGParallelMixin` and implement `diffuse()`: + +```python +from vllm_omni.diffusion.distributed.cfg_parallel import CFGParallelMixin + +class YourPipeline(nn.Module, CFGParallelMixin): + def diffuse(self, latents, timesteps, prompt_embeds, negative_embeds, + do_true_cfg, true_cfg_scale, **kwargs): + for i, t in enumerate(timesteps): + positive_kwargs = { + "hidden_states": latents, + "encoder_hidden_states": prompt_embeds, + "timestep": t, + } + negative_kwargs = { + "hidden_states": latents, + "encoder_hidden_states": negative_embeds, + "timestep": t, + } if do_true_cfg else None + + noise_pred = self.predict_noise_maybe_with_cfg( + do_true_cfg=do_true_cfg, + true_cfg_scale=true_cfg_scale, + positive_kwargs=positive_kwargs, + negative_kwargs=negative_kwargs, + ) + latents = self.scheduler_step_maybe_with_cfg( + noise_pred, t, latents, do_true_cfg + ) + return latents +``` + +### Customization hooks + +| Method | Override when | +|--------|-------------| +| `predict_noise()` | Non-standard transformer call (e.g., dual-transformer like Wan2.2) | +| `cfg_normalize_function()` | Custom normalization (e.g., LongCat with clamping) | +| `combine_cfg_noise()` | Multi-output models (e.g., video + audio: CFG on video, positive-only on audio) | + +**Custom predict_noise** (Wan2.2 — selects active transformer): + +```python +def predict_noise(self, current_model=None, **kwargs): + if current_model is None: + current_model = self.transformer + return current_model(**kwargs)[0] +``` + +**Custom combine_cfg_noise** (multi-output): + +```python +def combine_cfg_noise(self, positive_pred, negative_pred, scale, normalize): + video_pos, audio_pos = positive_pred + video_neg, audio_neg = negative_pred + video_combined = super().combine_cfg_noise(video_pos, video_neg, scale, normalize) + return (video_combined, audio_pos) +``` + +### Composite scheduler for multi-output + +When each output has its own schedule: + +```python +class VideoAudioScheduler: + def __init__(self, video_scheduler, audio_scheduler): + self.video_scheduler = video_scheduler + self.audio_scheduler = audio_scheduler + + def step(self, noise_pred, t, latents, return_dict=False, generator=None): + video_out = self.video_scheduler.step( + noise_pred[0], t[0], latents[0], return_dict=False, generator=generator + )[0] + audio_out = self.audio_scheduler.step( + noise_pred[1], t[1], latents[1], return_dict=False, generator=generator + )[0] + return ((video_out, audio_out),) +``` + +### Testing CFG Parallel + +```bash +python text_to_image.py --model Your-model \ + --cfg-parallel-size 2 --cfg-scale 4.0 \ + --negative-prompt "ugly, unclear" +``` + +**Constraint**: `guidance_scale > 1.0` and negative prompt must be provided. + +### Reference implementations + +| Model | Path | +|-------|------| +| Qwen-Image | `vllm_omni/diffusion/models/qwen_image/cfg_parallel.py` | +| Wan2.2 | `vllm_omni/diffusion/models/wan2_2/pipeline_wan2_2.py` | +| Mixin base | `vllm_omni/diffusion/distributed/cfg_parallel.py` | + +--- ## HSDP (Hybrid Sharded Data Parallel) -HSDP uses PyTorch FSDP2 to shard transformer weights. No code changes needed in the model — the loader handles it. Set `use_hsdp=True` in `DiffusionParallelConfig`. +Shards model weights across GPUs using PyTorch FSDP2. Reduces per-GPU VRAM without changing computation. -## Adding Parallelism Incrementally +### Implementation -Recommended order: -1. **Basic single-GPU**: Get generation working first -2. **Tensor Parallelism**: Replace Linear layers, update `load_weights` for QKV fusion -3. **CFG Parallel**: Add `CFGParallelMixin`, implement `predict_noise` -4. **Sequence Parallelism**: Add `_sp_plan` to transformer -5. **HSDP**: Usually works out-of-box after TP is done -6. **VAE Patch Parallel**: Switch to distributed VAE class +Add `_hsdp_shard_conditions` to the transformer class: + +```python +class YourTransformer(nn.Module): + @staticmethod + def _is_transformer_block(name: str, module) -> bool: + return "blocks" in name and name.split(".")[-1].isdigit() + + _hsdp_shard_conditions = [_is_transformer_block] +``` + +For MoE models, add additional conditions: + +```python +class MoETransformer(nn.Module): + @staticmethod + def _is_transformer_block(name, module): + return "blocks" in name and name.split(".")[-1].isdigit() + + @staticmethod + def _is_moe_expert(name, module): + return "experts" in name and name.split(".")[-1].isdigit() + + _hsdp_shard_conditions = [_is_transformer_block, _is_moe_expert] +``` + +A module is sharded if **any** condition returns `True`. + +### Constraints + +- Cannot combine with Tensor Parallelism +- For standalone HSDP (no other parallelism), `hsdp_shard_size` must be specified explicitly +- Can combine with SP: HSDP reduces memory while SP distributes sequence + +### Testing HSDP + +```python +from vllm_omni.diffusion.data import DiffusionParallelConfig + +parallel_config = DiffusionParallelConfig(use_hsdp=True, hsdp_shard_size=8) +omni = Omni(model="your-model", parallel_config=parallel_config) +``` + +Or CLI: + +```bash +vllm serve Your-model --omni --use-hsdp +``` + +**Verify**: logs show "HSDP Inference: replicate_size=..., shard_size=..." and "Sharded N modules + root". Check VRAM reduction. + +### Reference implementations + +| Model | Path | +|-------|------| +| Wan2.2 | `vllm_omni/diffusion/models/wan2_2/wan2_2_transformer.py` | +| HSDP Core | `vllm_omni/diffusion/distributed/hsdp.py` | + +--- + +## VAE Patch Parallelism + +Shards VAE decode spatially across ranks using tiling: + +```bash +python text_to_image.py --model Your-model --vae-patch-parallel-size 4 +``` + +Auto-enables `--vae-use-tiling`. Uses `DistributedAutoencoderKLWan` or similar distributed VAE. Set `vae_patch_parallel_size` in `DiffusionParallelConfig`. + +--- + +## Combining Parallelism Methods + +Common multi-GPU recipes: + +```bash +# 4 GPUs: CFG (2) × Ulysses (2) +python text_to_image.py --model Qwen/Qwen-Image \ + --cfg-parallel-size 2 --ulysses-degree 2 + +# 8 GPUs: Ulysses (4) × Ring (2) + VAE patch (8) +python text_to_video.py --model Wan-AI/Wan2.2-T2V-A14B-Diffusers \ + --ulysses-degree 4 --ring-degree 2 --vae-patch-parallel-size 8 + +# 2 GPUs: HSDP + Ulysses (cannot combine HSDP with TP) +vllm serve Your-model --omni --use-hsdp --usp 2 +``` + +## Discovering Parallelism Support + +Check which parallelism methods a model supports: + +| Check | How | +|-------|-----| +| **Ulysses / Ring SP** | Transformer defines `_sp_plan`. Search: `grep -r '_sp_plan' vllm_omni/diffusion/models/` | +| **CFG Parallel** | Pipeline inherits `CFGParallelMixin`. Search: `grep -r 'CFGParallelMixin' vllm_omni/diffusion/models/` | +| **TP** | Uses `ColumnParallelLinear` / `QKVParallelLinear`. Search: `grep -r 'ParallelLinear\|QKVParallel' vllm_omni/diffusion/models//` | +| **HSDP** | Transformer defines `_hsdp_shard_conditions`. Search: `grep -r '_hsdp_shard_conditions' vllm_omni/diffusion/models/` | + +The canonical per-model support table is in `docs/user_guide/diffusion/parallelism_acceleration.md`. diff --git a/.claude/skills/add-diffusion-model/references/troubleshooting.md b/.claude/skills/add-diffusion-model/references/troubleshooting.md index 4c63cdff9..27acdd8d1 100644 --- a/.claude/skills/add-diffusion-model/references/troubleshooting.md +++ b/.claude/skills/add-diffusion-model/references/troubleshooting.md @@ -78,10 +78,85 @@ out = out.reshape(batch, seq_len, -1) **Symptom**: `AssertionError: not divisible` or incorrect output with TP>1 **Fix**: -1. Verify `hidden_dim % tp_size == 0` and `num_heads % tp_size == 0` +1. Verify `num_heads % tp_size == 0` and `num_kv_heads % tp_size == 0` 2. Ensure `ColumnParallelLinear` / `RowParallelLinear` are used correctly 3. Check that norms between parallel layers use distributed norm if needed 4. Verify `load_weights` handles TP sharding for norm weights +5. Use `self.to_qkv.num_heads` (local heads per GPU) for QKV split sizes, not total heads + +**Missing `input_is_parallel=True`**: + +`RowParallelLinear` expects sharded input from `ColumnParallelLinear`: +```python +self.w1 = ColumnParallelLinear(dim, hidden_dim, return_bias=False) +self.w2 = RowParallelLinear(hidden_dim, dim, input_is_parallel=True, return_bias=False) +``` + +### Sequence Parallel Errors + +**Symptom**: Incorrect output or crashes with `--ulysses-degree N` or `--usp N` + +**Possible causes**: +1. **Inline operations between shard/gather points**: `torch.cat()`, `pad_sequence()` etc. not at `nn.Module` boundaries. Fix: extract into submodule. +2. **Wrong `split_dim`**: Check the tensor shape at the shard point. Sequence dimension is typically `dim=1` for `[B, S, D]` tensors. +3. **RoPE not sharded**: If RoPE is computed separately, add it to `_sp_plan` with `split_output=True`. +4. **Sequence not divisible by SP degree**: Use `auto_pad=True` in `SequenceParallelInput` or switch to `ulysses_mode="advanced_uaa"`. + +**Debugging**: Add `expected_dims=N` to `SequenceParallelInput`/`Output` for shape validation at runtime. + +### CFG Parallel Errors + +**Symptom**: CFG parallel not activating, no speedup + +**Fix checklist**: +1. Pipeline inherits `CFGParallelMixin` +2. `guidance_scale > 1.0` +3. Negative prompt provided (even if empty string) +4. `--cfg-parallel-size 2` specified +5. `diffuse()` method calls `predict_noise_maybe_with_cfg()` and `scheduler_step_maybe_with_cfg()` + +**Symptom**: Different output with CFG parallel vs sequential + +**Possible cause**: Non-deterministic scheduler. Fix: pass `generator=torch.Generator(device).manual_seed(seed)` to `scheduler_step_maybe_with_cfg()`. + +### HSDP Errors + +**Symptom**: HSDP not activating or errors during weight loading + +**Fix checklist**: +1. Transformer defines `_hsdp_shard_conditions` class attribute +2. Shard condition functions return `True` for correct modules (test with `model.named_modules()`) +3. Not combining with TP (HSDP and TP are incompatible) +4. For standalone HSDP, `hsdp_shard_size` is specified explicitly + +**Verify**: Check logs for "HSDP Inference: replicate_size=..., shard_size=..." and "Sharded N modules + root". + +### Cache-DiT Not Applied + +**Symptom**: No speedup, no cache-related log messages + +**Fix checklist**: +1. Model not in `_NO_CACHE_ACCELERATION` in `registry.py` +2. Pipeline class name matches `CUSTOM_DIT_ENABLERS` key (if using custom enabler) +3. `cache_backend="cache_dit"` specified +4. Check logs for "Cache-dit enabled successfully on xxx" + +**Verify pipeline name**: `print(pipeline.__class__.__name__)` — must match registry key. + +### Cache-DiT Quality Degradation + +**Symptom**: Artifacts or lower quality with cache-dit + +**Fix**: Reduce aggressiveness: +```python +cache_config={ + "residual_diff_threshold": 0.12, # Lower from 0.24 + "max_warmup_steps": 6, # Increase from 4 + "max_continuous_cached_steps": 2, # Reduce if higher +} +``` + +If quality is still poor, the model may need a custom enabler with per-block-list `ParamsModifier` tuning. ### Model Not Detected / Wrong Pipeline Class diff --git a/.claude/skills/add-tts-model/SKILL.md b/.claude/skills/add-tts-model/SKILL.md index 9546f187d..e64e7e763 100644 --- a/.claude/skills/add-tts-model/SKILL.md +++ b/.claude/skills/add-tts-model/SKILL.md @@ -1,14 +1,12 @@ --- name: add-tts-model -description: Integrate a new TTS model into vLLM-Omni from HuggingFace reference implementation through production-ready serving with streaming and CUDA graph acceleration. Use when adding a new TTS model, wiring stage separation, enabling online speech serving, or debugging TTS integration behavior. +description: "Integrate a new text-to-speech model into vLLM-Omni from HuggingFace reference implementation through production-ready serving with streaming and CUDA graph acceleration. Use when adding a new TTS model, wiring stage separation for speech synthesis, enabling online voice generation serving, debugging TTS integration behavior, or building audio output pipelines." --- # TTS Model Integration Workflow ## Overview -Integrating a new TTS model into vLLM-Omni follows a phased approach. Each phase builds on the previous one, and the model becomes usable at each stage with increasing capabilities. - ``` HF Reference -> Stage Separation -> Online Serving -> Async Chunk -> CUDA Graph (Phase 1) (Phase 2) (Phase 3) (Phase 4) (Phase 5) @@ -114,13 +112,19 @@ When audio output is wrong, check in this order: ### Voice Cloning Pattern -If the model supports voice cloning via reference audio encoding: - ```python -# In serving_speech.py: -# 1. Resolve ref_audio (URL, base64, or file path) to audio bytes -# 2. Encode on CPU using the model's codec (e.g., DAC for Fish Speech) -# 3. Convert to token IDs and inject into the prompt as a system message +import base64 +from pathlib import Path + +def build_voice_clone_prompt(ref_audio_path: str, text: str, codec) -> list: + """Build prompt with reference audio for voice cloning in serving_speech.py.""" + audio_bytes = Path(ref_audio_path).read_bytes() + codes = codec.encode(audio_bytes) # Encode on CPU using model's codec (e.g., DAC) + token_ids = [code + codec.vocab_offset for code in codes.flatten().tolist()] + return [ + {"role": "system", "content": f"<|voice|>{''.join(chr(t) for t in token_ids)}"}, + {"role": "user", "content": text}, + ] ``` ### Deliverables @@ -198,18 +202,24 @@ Stage 0 (AR) Stage 1 (Decoder) ### Example: Code Predictor CUDA Graph (Qwen3-TTS) ```python +import torch + class CodePredictorGraph: """Captures the 16-step code predictor AR loop as a single CUDA graph.""" - def setup_graph(self, device): - # Pre-allocate static KV caches [1, kv_heads, 16, head_dim] - # Pre-build causal masks and positions for each step - # Warm up -> capture -> store graph + def setup_graph(self, device: torch.device, kv_heads: int = 4, head_dim: int = 64): + self.num_steps = 16 + self.kv_cache = torch.zeros(1, kv_heads, self.num_steps, head_dim, device=device) + self.positions = torch.arange(self.num_steps, device=device) + self.causal_mask = torch.tril(torch.ones(self.num_steps, self.num_steps, device=device)) + self.input_buf = torch.zeros(1, 1, kv_heads * head_dim, device=device) + self.output_buf = torch.zeros(1, self.num_steps, device=device, dtype=torch.long) + # Warm up, then: self.graph = torch.cuda.CUDAGraph(); self.graph.capture(...) - def run_graph(self, initial_input): - # Copy input to static buffer - # Replay captured graph - # Read output from static buffer + def run_graph(self, initial_input: torch.Tensor) -> torch.Tensor: + self.input_buf.copy_(initial_input) + self.graph.replay() + return self.output_buf.clone() ``` ### Performance Expectations diff --git a/.claude/skills/review-pr/SKILL.md b/.claude/skills/review-pr/SKILL.md index 6684be039..1363482be 100644 --- a/.claude/skills/review-pr/SKILL.md +++ b/.claude/skills/review-pr/SKILL.md @@ -1,38 +1,70 @@ --- name: review-pr -description: Review PRs on vllm-project/vllm-omni by routing to the right domain skills, checking critical evidence, and focusing comments on blocking issues. Use when reviewing pull requests, triaging review depth, or checking tests, benchmarks, and breaking changes in vllm-omni. +description: Review PRs on vllm-project/vllm-omni by routing to the right domain skills, checking critical evidence, and focusing comments on blocking issues. Use when reviewing pull requests or local branches, triaging review depth, running detailed or default review, or checking tests, benchmarks, and breaking changes in vllm-omni. --- # vLLM-Omni PR Review ## Overview -You are an adversarial reviewer. Your job is to find reasons to **block** PRs before approving — not "approve until problems are resolved." Assume blocking issues exist until proven otherwise. Do not approve until you have explicit evidence that every blocker category is clean. +Review PRs like a real maintainer — direct, selective, and focused on high-signal issues. Prioritize 2-3 real problems per PR over exhaustive coverage. Most PRs should get 1-5 short comments; some just get an empty APPROVE. Use this skill as a router for `vllm-project/vllm-omni` pull request reviews. Keep the default context small, load only the references that match the diff, and prioritize high-confidence findings over coverage theater. +## Usage modes + +Inspired by common PR-review skill patterns (e.g. explicit modes + tool choice); **repo is always `vllm-project/vllm-omni`** unless the user says otherwise. + +| Mode | What to do | +|------|------------| +| **No PR / branch given** | Do not start a full review. Ask for a **PR number or URL**, or **local branch** review vs `main` (or named base). | +| **PR number or URL** | Use `gh` against this repo: `gh pr view --repo vllm-project/vllm-omni`, `gh pr diff --repo vllm-project/vllm-omni`. Commands and JSON fields: [references/review-execution.md](references/review-execution.md). | +| **Local branch** | No GitHub PR yet: `git fetch` if needed, then `git diff ...HEAD`, `git diff --stat ...HEAD`, `git log ..HEAD --oneline`. Use branch name instead of PR # in the review header; same blocker scan and routing. | +| **Pre-filled prompt** | If the prompt already includes PR title/body, checks, or thread summaries (e.g. from GitHub), **do not re-fetch metadata** unless something is missing; still obtain the **diff** if not present (`gh pr diff` or `git diff` against merge base). | + +**Depth:** **Default** = maintainer-style brevity ([comment budget](references/review-execution.md)). If the user asks for **detailed** / **in-depth** / **line-by-line**, add a **Specific comments** list with `path:line` items; do not duplicate those points as long prose in the review body. Still respect the usual inline ceiling unless the user explicitly wants an audit-style pass. + +## What to prioritize (CI-complement) + +- **Investigate, don’t guess** — If a checklist item might apply (e.g. connector lifecycle, diffusion cache), read surrounding code or use **parallel subagents** for independent areas. A wrong guess is worse than silence. +- **Focus on what automation doesn’t prove** — Design, API contracts, stage/connector invariants, test adequacy for omni paths, breaking behavior. Do not re-argue formatting or issues **pre-commit / CI already failed** (point at the gate instead). +- **Structured notes** — If you produce a multi-section writeup for the **user**, **omit sections with no findings** (no “No issues” padding). + +**Parallel investigation:** Large diffs or multiple subsystems (e.g. `entrypoints/` + `engine/` + `diffusion/`) → split by directory or concern and investigate **in parallel** when subagents exist. + +## Which reference to load (do not load everything) + +| Situation | Open | +|-----------|------| +| Every review | [references/review-execution.md](references/review-execution.md) — gates, `gh` commands, comment budget, tone, batch/CI triage, Python style flags | +| Prefix / multi-skill / hardware guess | [references/review-routing.md](references/review-routing.md) | +| Blocker scan details + merge-blocking patterns | [references/blocker-patterns.md](references/blocker-patterns.md) — Part 1 patterns; **Part 2** = former “pitfalls” (footguns, MRO, connectors, async, etc.) | +| System layout + **code-pattern review** (async, connectors, validation, …) | [references/architecture.md](references/architecture.md) — includes “Code patterns for review” at the end | +| Diffusion / image / video model PRs | [references/diffusion-checklist.md](references/diffusion-checklist.md) | +| High-risk change; need coverage matrix / docs sync | [references/tests-docs-checklist.md](references/tests-docs-checklist.md) | +| PR has perf/accuracy claims or `[Performance]` prefix | [references/perf-verification.md](references/perf-verification.md) — claim detection, hardware-aware benchmark verification, graceful degradation | +| PR adds/modifies tests or touches core code without tests | [references/test-quality-evaluation.md](references/test-quality-evaluation.md) — assertion quality, anti-patterns, hardware-aware test execution | +| Calibrating phrasing from real maintainers | [references/maintainer-style-study.md](references/maintainer-style-study.md) | + +**Legacy paths (do not load — content merged):** `pitfalls.md` → [blocker-patterns.md](references/blocker-patterns.md) **Part 2**; `code-patterns.md` → [architecture.md](references/architecture.md) **Code patterns for review**; `python-style-guide.md` → [review-execution.md](references/review-execution.md) **Python style (review flags)**; batch/CI triage → [review-execution.md](references/review-execution.md) (Batch / CI sections). + ## Priority Hierarchy Under Context Pressure -If context is limited, prioritize: blocker scan → evidence → domain routing → verdict. +If context is limited, prioritize: blocker scan → evidence → perf verification → test quality → domain routing → verdict. Always run the blocker scan. Under context pressure, do a shallow scan of the most critical categories (Correctness, Security) and flag that the scan was incomplete. ## Core Workflow +Check whether this PR is still a draft or WIP in the PR title, if so, end the review process. + + ### Step 0: Verify Review Gates First -Check mergeability and required checks (DCO, pre-commit, mergeability). If failing, stop and ask the author to fix gates before proceeding. +If this is a ready-to-review PR, check the mergeability and required checks (DCO, pre-commit, mergeability). If failing, stop and ask the author to fix gates before proceeding. For gate commands, review submission, and comment style, see [references/review-execution.md](references/review-execution.md). -### Step 0.5: Check PR Size for Large Changes - -For substantial changes (more than 1000 LOC OR more than 10 files changed): -- Ask the contributor to run L3 tests locally and paste results in PR description (highly recommended) - -Example: -> This PR is substantial (>1000 LOC / >10 files). Could you please run the [L3 tests](https://docs.vllm.ai/projects/vllm-omni/en/latest/contributing/ci/test_guide/#l3-level--l4-level) locally and paste the results here? - Then continue with the workflow below. ### Step 1: Gather Minimal Context @@ -43,6 +75,8 @@ Fetch: - Linked issues for `[Bugfix]` and `[Feature]` PRs only when conventions are unclear - Related PRs only when conventions or prior decisions are unclear +Group changes mentally by **kind** (runtime code, tests, docs, configs) to see where risk sits; then load references (Step 4) only for areas touched. + Do not fetch broad extra context unless the diff leaves real ambiguity. ### Step 2: Blocker Scan (Required First) @@ -86,49 +120,28 @@ BLOCKER scan: For detailed anti-patterns with code examples, see [references/blocker-patterns.md](references/blocker-patterns.md). -**If blockers found:** -``` -BLOCKING ISSUES: -1. [Category] [Line/File] - [description] -2. ... -VERDICT: REQUEST_CHANGES (cannot approve until blockers resolved) -``` +**If blockers found:** Track issues internally (category + file + line). Do not paste structured `BLOCKING ISSUES:` templates into the review body (see Step 6). -**If no blockers:** -List non-blocking suggestions and proceed to Step 3. +**If no blockers:** List non-blocking suggestions and proceed to Step 3. ### Step 3: Route to the Right Skill -Use the title prefix and changed directories to decide whether a domain skill is required. +Use the title prefix and changed directories to decide whether a domain skill is required. Doc-only, config-only, and test-only PRs usually skip domain skills unless the diff crosses into model or API areas. -| Signal | Action | -| ------------------------------------------- | ---------------------------- | -| `[Image]`, `[ImageGen]` | Use `vllm-omni-image-gen` | -| `[Video]`, `[VideoGen]` | Use `vllm-omni-video-gen` | -| `[Audio]`, `[TTS]` | Use `vllm-omni-audio-tts` | -| `[Multimodal]` | Use `vllm-omni-multimodal` | -| `[Distributed]` | Use `vllm-omni-distributed` | -| `[Quantization]` | Use `vllm-omni-quantization` | -| `[Performance]` | Use `vllm-omni-perf` | -| `[Hardware]` or backend-specific code | Use `vllm-omni-hardware` | -| `[API]` or `vllm_omni/entrypoints/` changes | Use `vllm-omni-api` | -| `[CI]` | Use `vllm-omni-cicd` | -| `[Model]` | Use `vllm-omni-contrib` | - -For multi-skill routing and hardware detection, see [references/review-routing.md](references/review-routing.md). +**Full prefix table, multi-skill combos, hardware detection, and delegation triggers:** [references/review-routing.md](references/review-routing.md). ### Step 4: Load Only the Relevant Review Reference -Load targeted references based on the diff: +Use **only** the files in this table. Older docs may mention `references/pitfalls.md` or `references/code-patterns.md`; those files were removed — use **Part 2** of blocker patterns and **Code patterns for review** in architecture instead. | Diff Area | Load | | ----------------------------------------------------------------------------------------- | ---------------------------------------------------------- | -| `vllm_omni/engine/`, `vllm_omni/stages/`, `vllm_omni/connectors/`, `vllm_omni/diffusion/` | [references/pitfalls.md](references/pitfalls.md) | -| Async, distributed coordination, validation, connector behavior | [references/code-patterns.md](references/code-patterns.md) | -| Scheduler, stage boundaries, execution model, critical paths | [references/architecture.md](references/architecture.md) | +| `vllm_omni/engine/`, `vllm_omni/stages/`, `vllm_omni/connectors/`, `vllm_omni/diffusion/` | [blocker-patterns.md](references/blocker-patterns.md) **Part 2** (common pitfalls) | +| Async, distributed coordination, validation, connector behavior | [architecture.md](references/architecture.md) — section **Code patterns for review** (at end of file) | +| Scheduler, stage boundaries, execution model, critical paths | [Architecture](references/architecture.md) (full) | | High-risk changes (core logic, configs/params, error handling, concurrency/distributed, I/O) or `[Feature]` / `[Bugfix]` PRs | [references/tests-docs-checklist.md](references/tests-docs-checklist.md) | -Avoid loading all three by default. +Pick the narrowest references that match the diff; avoid loading every row by default. ### Step 5: Ask for Concrete Validation Evidence @@ -147,50 +160,78 @@ For `[Feature]` PRs affecting performance or `[Performance]` PRs, use the checkl Be explicit in review comments. Treat "manual verification only" as insufficient unless automation is genuinely impossible. -### Step 6: Final Verdict +### Step 6: Verify Perf/Accuracy Claims (Blocking) -Use the review body to summarize: +**When to activate:** PR has `[Performance]` prefix, or PR body contains quantitative perf/accuracy claims (latency, throughput, VRAM, speedup, accuracy metrics), or Step 5 flagged missing benchmarks. + +**Load** [references/perf-verification.md](references/perf-verification.md). + +**Workflow:** + +1. Detect claims — extract numbers from PR body using regex patterns +2. Detect hardware — run hardware detection to determine available GPU/VRAM/platform +3. Check feasibility — compare estimated model size (weights + KV cache + overhead) against available VRAM +4. Generate benchmark plan — map PR type to appropriate benchmark runner +5. **Pre-execution gate** — present benchmark plan, estimated duration, and model/hardware to user; ask for confirmation before executing +6. Execute (if confirmed) — run before/after benchmarks via git worktrees, with a **20-minute hard timeout** per run +7. Report — produce Claimed vs Measured table with CONFIRMED / NOT_CONFIRMED verdict + +**Graceful degradation:** + +| Level | Condition | What happens | +|-------|-----------|-------------| +| Full verification | GPU available, model fits | Run before/after benchmarks | +| Partial verification | GPU available, model needs offload | Run with `--cpu-offload-gb`, note in report | +| Static-only | No GPU or model too large | Analyze benchmark scripts in diff for correctness, flag implausible claims | +| Skip | No relevant perf claims | Do not activate | + +**Delivery:** Local report first, ask user before posting as PR comment. If verification reveals a confirmed NOT_CONFIRMED for accuracy or VRAM regression, escalate to REQUEST_CHANGES via Step 8. + +### Step 7: Evaluate Test Quality (Blocking) + +**When to activate:** PR adds or modifies test files, or PR touches core code (`engine/`, `stages/`, `connectors/`) without adding tests, or PR is test-only. + +**Load** [references/test-quality-evaluation.md](references/test-quality-evaluation.md). + +**Workflow:** + +1. Static analysis (always runs) — check assertion quality, anti-patterns, marker compliance, edge case coverage +2. Detect hardware — same detection as Step 6 (cross-referenced from `perf-verification.md`) +3. Find affected tests — map changed source files to test files via grep (not path convention) +4. Filter by hardware — skip tests requiring unavailable resources +5. Run tests — `pytest` with `--run-level core_model` by default; use `advanced_model` only if hardware is sufficient +6. Categorize failures — test bug / code bug / infrastructure / flaky +7. Assess quality — score assertion quality, edge case coverage, marker compliance, anti-patterns (A-D grades for internal analysis) + +**Graceful degradation:** + +| Level | Condition | What happens | +|-------|-----------|-------------| +| Full analysis | Hardware matches test markers | Static + runtime execution | +| Static-only | Hardware doesn't match or no GPU | Static analysis only; report which tests were skipped | + +**Delivery:** Local assessment first, ask user before posting. Convert worst 1-2 findings to inline comments (counts against comment budget). If D-grade dimension or code bug found, escalate to REQUEST_CHANGES via Step 8. + +### Step 8: Final Verdict + +Post inline comments directly to GitHub as you find them. Do **not** submit a review event (APPROVE / COMMENT / REQUEST_CHANGES) — leave the verdict decision to the user. + +Summarize locally: - What was validated - What still lacks evidence -- What must change before approval +- Recommended verdict (local presentation only) -**Verdict format:** -``` -BLOCKER scan: -- Correctness: [PASS / ISSUES: (list)] -- Reliability/Safety: [PASS / ISSUES: (list)] -- Breaking Changes: [PASS / ISSUES: (list)] -- Test Coverage: [PASS / (check PR desc) / needs tests] -- Documentation: [PASS / ISSUES: (list)] -- Security: [PASS / ISSUES: (list)] +Recommended verdict mapping: -OVERALL: [NO BLOCKERS / X BLOCKERS FOUND] +- `APPROVE` — no blockers; body optional (empty is fine). +- `COMMENT` — suggestions only; body optional (~50% should be empty). +- `REQUEST_CHANGES` — genuine blocking issues only (crashes, data loss, security, policy gates). -VERDICT: [APPROVE / COMMENT / REQUEST_CHANGES] -``` - -For comment budget and phrasing, see [references/review-execution.md](references/review-execution.md). +For tone and inline style, see [references/review-execution.md](references/review-execution.md). For maintainer phrasing samples, see [references/maintainer-style-study.md](references/maintainer-style-study.md). ## Review Heuristics -- Check PR description evidence before requesting tests -- Only flag missing tests when evidence is genuinely absent -- For [Bugfix] PRs, require a regression test unless automation is impossible -- For API-facing PRs, prefer contract tests over broad smoke tests -- Be suspicious of silent fallbacks, swallowed exceptions, device-specific assumptions -- Review critical paths first: engine, connectors, stages, API entrypoints -- Skip nits and style comments unless they hide a correctness issue - -## Scenario Coverage - -| Scenario | Blocker Scan | Domain Routing | Verdict | -|----------|--------------|----------------|---------| -| Standard code PR | Full 6-category scan | Route by prefix/diff | Standard format | -| Doc-only PR | Skip to Documentation only | Skip | Standard format | -| Config-only PR | Breaking Changes + Documentation | Skip | Standard format | -| Test-only PR | Correctness of test logic | Skip | Standard format | -| Draft PR | Run scan (non-blocking); offer preliminary feedback on request | Skip | COMMENT: "Ready for full review when draft removed" | -| Large PR (>1000 LOC) | Shallow scan + request L3 tests | Route by prefix/diff | Standard format | +Trust PR description and CI evidence before demanding new tests. Prefer regression tests for `[Bugfix]`, contract tests for API changes, and scan engine → connectors → stages → entrypoints before peripheral files. Skip style nits unless they mask correctness. ## When to Fetch More Context @@ -204,154 +245,28 @@ Keep additional fetches narrow and tied to a specific uncertainty. ## Diffusion Model PR Review -> **Invoked when PR prefix is `[Model]`, `[New Model]`, `[Image]`, `[ImageGen]`, `[Video]`, `[VideoGen]`, or `[Diffusion]`.** +**When:** Title prefix `[Model]`, `[New Model]`, `[Image]`, `[ImageGen]`, `[Video]`, `[VideoGen]`, or `[Diffusion]`, or a new diffusion model path under `vllm_omni/diffusion/`. -All four dimensions below must pass before approving. Work through them in priority order. See [references/diffusion-checklist.md](references/diffusion-checklist.md) for full per-item criteria across all dimensions. +**Load** [references/diffusion-checklist.md](references/diffusion-checklist.md) instead of expanding the full workflow here: Dimensions 1–4 (including optional canonical PR body template + measurement guidance for Dimension 1), `gh`/`grep` detection commands, **Quick Red Flags**, combined-feature rules. -### Priority Order +**Gate order before approve:** PR body evidence (script, samples, latency, VRAM) → offline **and** online paths → at least one acceleration **and** one memory optimization (new models) → docs tables + usage examples → required e2e online test → offline test if no e2e → combined test when 2+ accel/memory features. -1. **PR body evidence** — missing samples/metrics block the entire review -2. **Missing inference mode** — model must work offline and online -3. **Missing acceleration feature** — required for all new diffusion models -4. **Missing memory optimization** — required for all new diffusion models -5. **Missing documentation** — tables and examples must be updated -6. **Missing e2e test** — required for merge gate -7. **Missing offline inference test** — recommended; required only when no e2e test is present -8. **Combined feature test** — required when two or more acceleration/memory features are implemented +Use Quick Red Flags actions for comment severity; write human-shaped GitHub comments, not giant pasted templates. ---- +## Batch Review Session -### Dimension 1: PR Body Completeness - -Required: generation script, sample outputs, e2e latency, peak VRAM. Recommended: matching diffusers baseline for all four. - -**Comment when required items are missing:** - -``` -🔴 **PR Body Incomplete — Required Evidence Missing** - -The following items are required before this PR can be reviewed: - -- [ ] vLLM-Omni generation script (offline `Omni` or online `vllm serve`) -- [ ] Generated sample output (image / video / audio) -- [ ] vLLM-Omni e2e latency (hardware: GPU model, count; resolution; steps) -- [ ] vLLM-Omni peak VRAM usage (GB) - -Please update the PR description with this information. -``` - -**Comment when diffusers baseline is absent:** - -``` -💡 **Recommended: diffusers Baseline Comparison** - -Adding a diffusers comparison would strengthen this PR: - -- diffusers generation script (same prompt, same resolution/steps) -- diffusers sample output -- diffusers e2e latency vs vLLM-Omni latency -- diffusers peak VRAM vs vLLM-Omni VRAM -``` - ---- - -### Dimension 2: Code Review - -```bash -# Detect leftover diffusers mixins -gh pr diff --repo vllm-project/vllm-omni \ - | grep '^\+' | grep -E 'DiffusionPipelineMixin|SchedulerMixin|ConfigMixin' - -# Check acceleration features -gh pr diff --repo vllm-project/vllm-omni \ - | grep -iE 'sequence_parallel|cfg_parallel|vae_patch_parallel|tensor_parallel|teacache|cache_dit' - -# Check memory optimizations -gh pr diff --repo vllm-project/vllm-omni \ - | grep -iE 'cpu_offload|offload_to_cpu|quantization|int8|fp8|bitsandbytes|vae_tiling' -``` - -**2.1 Inference modes** — both offline (`Omni`) and online (`vllm serve` / `AsyncOmni`) must be implemented. - -``` -🔴 Missing support. The model must work in both -Omni (offline) and vllm serve / AsyncOmni (online) modes before merging. -``` - -**2.2 Diffusers mixin cleanup** — flag any mixin still present in `+` lines. - -``` -🔴 Leftover diffusers mixin detected: `` in `:`. -Remove diffusers mixins and use vLLM-Omni's native abstractions instead. -``` - -**2.3 Acceleration features** — at least one of: sequence parallel, CFG parallel, VAE patch parallel, tensor parallel, TeaCache/step cache. - -``` -🔴 No acceleration feature detected. The model must support at least one of: -sequence parallel, CFG parallel, VAE patch parallel, tensor parallel, or step caching (TeaCache). -``` - -**2.4 Memory optimization** — at least one of: CPU offload, quantization (int8/fp8/bnb), VAE tiling. - -``` -🔴 No memory optimization feature detected. The model must support at least one of: -CPU offload (`--cpu-offload-gb`), quantization (int8/fp8/bnb), or VAE tiling. -``` - ---- - -### Dimension 3: Documentation - -```bash -gh pr view --repo vllm-project/vllm-omni \ - --json files --jq '.files[].path' | grep -E 'docs/|\.md$' -``` - -Required: model support table, feature support table, usage example doc (offline + online). - -``` -🔴 Documentation incomplete: -- [ ] Model support table not updated (docs/models/supported_models.md or equivalent) -- [ ] Feature support table not updated -- [ ] Usage example doc missing or not updated for -``` - ---- - -### Dimension 4: Test Coverage - -```bash -gh pr view --repo vllm-project/vllm-omni \ - --json files --jq '.files[].path' | grep -E '^tests/' -``` - -Required: e2e online serving test. Recommended: offline inference test. Required when 2+ features: combined feature test. - -``` -🔴 Missing e2e online serving test in `tests/e2e/online_serving/`. -Please add a test that: -1. Starts `vllm serve --omni` -2. Sends a generation request via the API -3. Asserts the response contains a valid image / video / audio output -``` - -``` -⚠️ Multiple features are implemented (e.g., + ) but no combined -feature test is present. Please add a test (or extend the e2e test) that enables both -features together and asserts output validity + reports latency + VRAM. -``` - -> See [Diffusion Checklist](references/diffusion-checklist.md) for full per-item criteria and the Quick Red Flags summary. - ---- +For daily sessions (reply-first, PR selection, pacing, re-review): [references/review-execution.md](references/review-execution.md) — sections **Batch and daily review sessions** and related scripts. ## References -- [Blocker Patterns](references/blocker-patterns.md) - Anti-patterns that block approval with code examples -- [Review Routing](references/review-routing.md) - Prefix mapping, multi-skill routing, hardware detection -- [Review Execution](references/review-execution.md) - Gate checks, commands, comment budget, review phrasing -- [Common Pitfalls](references/pitfalls.md) - MRO issues, connector state, async differences -- [Architecture](references/architecture.md) - System overview and critical paths -- [Code Patterns](references/code-patterns.md) - Async, distributed, cache, validation, error handling patterns -- [Diffusion PR Requirements](references/diffusion-pr-requirements.md) - PR body requirements for diffusion model contributions +All paths are under `skills/vllm-omni-review/references/`. There is **no** `pitfalls.md`, `code-patterns.md`, or `python-style-guide.md` — see the legacy map in **Which reference to load** above. + +- [Review execution](references/review-execution.md) — Gates, `gh` fetch/submit, comment budget, tone, **Python style (review flags)** (imports, naming, common flags — use to avoid nit wars), batch session, CI log triage +- [Review routing](references/review-routing.md) — Prefix → domain skill, hardware, multi-skill +- [Blocker patterns](references/blocker-patterns.md) — Part 1: merge-blocking patterns; Part 2: MRO/mixins, connectors, async/sync, diffusion latents, API validation, tensor parallel, mocks (former pitfalls content) +- [Architecture](references/architecture.md) — Layers and critical paths; end section **Code patterns for review** = async, distributed, KV cache, validation, connectors, errors, logging (former code-patterns content) +- [Diffusion checklist](references/diffusion-checklist.md) — Diffusion PR dimensions, PR body template, Quick Red Flags +- [Tests & docs checklist](references/tests-docs-checklist.md) — High-risk coverage matrix and docs sync +- [Perf verification](references/perf-verification.md) — Reviewer-side claim detection, hardware-aware benchmark verification, graceful degradation +- [Test quality evaluation](references/test-quality-evaluation.md) — Assertion quality, anti-patterns, hardware-aware test execution, quality scoring +- [Maintainer style study](references/maintainer-style-study.md) — Example maintainer phrasing diff --git a/.claude/skills/review-pr/references/architecture.md b/.claude/skills/review-pr/references/architecture.md index 391382186..bee58674e 100644 --- a/.claude/skills/review-pr/references/architecture.md +++ b/.claude/skills/review-pr/references/architecture.md @@ -190,3 +190,242 @@ Request → API Layer → AsyncOmni/Omni | Connectors | Load testing, memory leak check | | Multi-stage | End-to-end pipeline | | API endpoints | Input validation, error responses | + +--- + +## Code patterns for review + +*This section holds the former standalone `code-patterns.md` material (async, validation, connectors, logging, etc.).* + +### Async Function Complexity + +**Guidelines:** +- Each `await` is a potential failure point and state transition +- Consider splitting long async chains into smaller helpers +- Watch for race conditions in complex async flows +- Use `return_exceptions=True` for parallel operations + +**Good:** +```python +async def process_batch(requests: list[Request]) -> list[Response]: + validated = [_validate_request(r) for r in requests] + results = await asyncio.gather( + *[_process_single(r) for r in validated], + return_exceptions=True + ) + return [_handle_result(r) for r in results] +``` + +**Bad:** +```python +async def process_batch(requests: list[Request]) -> list[Response]: + results = [] + for r in requests: + results.append(await _process_single(r)) # Sequential, no error handling + return results +``` + +--- + +### Distributed Execution Patterns + +**When complexity is justified:** +- Tensor parallelism synchronization across workers +- Pipeline parallelism stage coordination +- Distributed KV cache management +- Multi-node communication patterns + +**Critical questions:** +- Is distributed complexity isolated from business logic? +- Are distributed failure modes handled? +- Is there a clear fallback for single-device execution? + +**Pattern:** +```python +def get_world_size() -> int: + """Returns 1 if not in distributed mode.""" + if not dist.is_initialized(): + return 1 + return dist.get_world_size() +``` + +--- + +### KV Cache Management + +**Guidelines:** +- Ensure clear separation between allocation, update, and cleanup +- Document invariants and assumptions +- Use type system to enforce valid states +- Always handle cleanup in error paths + +**Pattern:** +```python +class KVCache: + def allocate(self, num_blocks: int) -> None: + assert self._state == CacheState.UNINITIALIZED + self._blocks = [Block() for _ in range(num_blocks)] + self._state = CacheState.ALLOCATED + + def update(self, block_id: int, kv_data: Tensor) -> None: + assert self._state == CacheState.ALLOCATED + self._blocks[block_id].update(kv_data) + + def free(self) -> None: + self._blocks = [] + self._state = CacheState.UNINITIALIZED +``` + +--- + +### Input Validation Pattern + +**Guidelines:** +- Validate early (API layer, not engine) +- Return actionable error messages +- Use clean HTTP status codes (400 for client errors) + +**Pattern:** +```python +def _validate_request(self, request: Request) -> str | None: + """Returns error message or None if valid.""" + if not request.input: + return "Input cannot be empty" + + if request.param and request.param not in VALID_PARAMS: + return f"Invalid param '{request.param}'. Valid: {VALID_PARAMS}" + + return None + +def handle_request(self, request: Request) -> Response: + error = self._validate_request(request) + if error: + return Response(status_code=400, content={"error": error}) + return self._process(request) +``` + +--- + +### Connector Communication + +**Guidelines:** +- Use context managers for resource cleanup +- Handle timeouts explicitly +- Implement proper error propagation + +**Pattern:** +```python +async def send_to_next_stage(self, data: Tensor) -> None: + try: + async with self.connector.acquire(timeout=30.0) as channel: + await channel.send(data) + except TimeoutError: + logger.error("Stage communication timeout") + raise StageError("Downstream stage unavailable") + except Exception as e: + logger.error(f"Stage communication failed: {e}") + raise +``` + +--- + +### Stage Lifecycle + +**Guidelines:** +- Stages are configured at init time +- Runtime reconfiguration requires full teardown +- State must be properly managed across stage boundaries + +**Pattern:** +```python +class Stage: + def __init__(self, config: StageConfig): + self._validate_config(config) + self.config = config + self._state = StageState.INITIALIZED + + async def start(self) -> None: + assert self._state == StageState.INITIALIZED + await self._allocate_resources() + self._state = StageState.RUNNING + + async def stop(self) -> None: + if self._state == StageState.RUNNING: + await self._release_resources() + self._state = StageState.STOPPED +``` + +--- + +### Test Coverage Requirements + +| Code Type | Requirement | +|-----------|-------------| +| New features | Happy path + edge cases + error handling | +| Bug fixes | Regression test + edge cases around fix | +| Performance | Benchmarks with before/after measurements | +| Distributed | Must test in distributed mode | +| Quantization | Memory savings + quality impact measured | +| API endpoints | Input validation + error responses | + +--- + +### Code Quality Thresholds + +| Metric | Threshold | Action | +|--------|-----------|--------| +| Function length | 100+ lines | Review for split opportunities | +| Complexity | 11+ | Requires refactoring | +| Nesting depth | 4+ levels | Requires refactoring | +| Parameters | 7+ | Consider parameter object | + +--- + +### Error Handling Pattern + +**Guidelines:** +- Use specific exception types +- Include actionable context in messages +- Don't silently swallow errors + +**Pattern:** +```python +class ValidationError(Exception): + """Raised when input validation fails.""" + pass + +class EngineError(Exception): + """Raised when engine operation fails.""" + pass + +def process(self, request: Request) -> Response: + try: + return self._process_internal(request) + except ValidationError as e: + logger.warning(f"Validation failed: {e}") + return Response(status_code=400, content={"error": str(e)}) + except EngineError as e: + logger.error(f"Engine error: {e}") + return Response(status_code=500, content={"error": "Internal error"}) +``` + +--- + +### Logging Pattern + +**Guidelines:** +- Use appropriate log levels +- Include relevant context +- Avoid logging sensitive data + +```python +# Good +logger.info(f"Processing request {request_id} with model {model_name}") +logger.warning(f"Slow response time: {elapsed_ms}ms for {endpoint}") +logger.error(f"Failed to load model {model_path}: {e}") + +# Bad +logger.info("Processing request") # No context +print(f"Error: {e}") # Wrong logging method +logger.debug(f"User token: {token}") # Sensitive data +``` diff --git a/.claude/skills/review-pr/references/blocker-patterns.md b/.claude/skills/review-pr/references/blocker-patterns.md index 11fe32470..d6edd3288 100644 --- a/.claude/skills/review-pr/references/blocker-patterns.md +++ b/.claude/skills/review-pr/references/blocker-patterns.md @@ -2,7 +2,7 @@ Each pattern includes: the anti-pattern, why it's dangerous, and required fix. -Cross-references point to [code-patterns.md](code-patterns.md) for "how to fix" guidance. +Cross-references point to [architecture.md](architecture.md) (section **Code patterns for review**) for "how to fix" guidance. ## Exception Handling @@ -19,7 +19,7 @@ except: **Required fix:** At minimum, log the exception. Better: catch specific exceptions. -**Cross-ref:** See [Error Handling Pattern](code-patterns.md#error-handling-pattern) for proper exception handling with logging. +**Cross-ref:** See [Error Handling Pattern](architecture.md#error-handling-pattern) for proper exception handling with logging. --- @@ -36,7 +36,7 @@ for item in items: **Required fix:** Log failure, increment counter, or fail fast. -**Cross-ref:** See [Error Handling Pattern](code-patterns.md#error-handling-pattern) for proper exception handling with logging. +**Cross-ref:** See [Error Handling Pattern](architecture.md#error-handling-pattern) for proper exception handling with logging. --- @@ -54,7 +54,7 @@ if obj is not None and obj.method() is not None: result = obj.method().nested.value ``` -**Cross-ref:** See [None Safety](code-patterns.md#input-validation-pattern) for guard patterns. +**Cross-ref:** See [None Safety](architecture.md#input-validation-pattern) for guard patterns. --- @@ -71,7 +71,7 @@ content = f.read() **Required fix:** Use context managers (`with open(...) as f:`). -**Cross-ref:** See [Connector Communication](code-patterns.md#connector-communication) for proper connector usage with context managers. +**Cross-ref:** See [Connector Communication](architecture.md#connector-communication) for proper connector usage with context managers. --- @@ -86,7 +86,7 @@ async def append(item): **Required fix:** Use `asyncio.Lock` or thread-safe structures. -**Cross-ref:** See [Distributed Execution Patterns](code-patterns.md#distributed-execution-patterns) for thread safety alternatives. +**Cross-ref:** See [Distributed Execution Patterns](architecture.md#distributed-execution-patterns) for thread safety alternatives. --- @@ -112,7 +112,7 @@ if obj is not None: result = getattr(getattr(obj, "method", lambda: None)(), "nested", None) ``` -**Cross-ref:** See [Input Validation Pattern](code-patterns.md#input-validation-pattern) for early validation guidance. +**Cross-ref:** See [Input Validation Pattern](architecture.md#input-validation-pattern) for early validation guidance. --- @@ -130,7 +130,7 @@ def process(data): **Required fix:** Add explicit None check and log a warning. Better: `return None`. -**Cross-ref:** See [Input Validation Pattern](code-patterns.md#input-validation-pattern) for validation guidance. +**Cross-ref:** See [Input Validation Pattern](architecture.md#input-validation-pattern) for validation guidance. --- @@ -149,7 +149,7 @@ def get_user(user_id: int, include_profile: bool): # Breaking change! **Required fix:** Add deprecation path or maintain backward compatibility for 1-2 releases. Document migration. -**Cross-ref:** See [API Endpoints](code-patterns.md#connector-communication) for API compatibility guidance. +**Cross-ref:** See [API Endpoints](architecture.md#connector-communication) for API compatibility guidance. --- @@ -162,7 +162,7 @@ def remove_feature(name): **Why dangerous:** Breaks existing code that depends on this function. **required fix:** If removal is necessary, provide replacement. Otherwise, document breaking change and deprecation timeline. -**Cross-ref:** See [API Endpoints](code-patterns.md#connector-communication) for OpenAI compatibility guidance +**Cross-ref:** See [API Endpoints](architecture.md#connector-communication) for OpenAI compatibility guidance --- ### Pattern: Changed Default @@ -176,7 +176,7 @@ class Config: **Required fix:** Document the change clearly. If intentional, provide rollback code or migration script. -**Cross-ref:** See [API Endpoints](code-patterns.md#connector-communication) for API compatibility guidance. +**Cross-ref:** See [API Endpoints](architecture.md#connector-communication) for API compatibility guidance. --- @@ -192,7 +192,7 @@ class DiffusionModel: **Why dangerous:** Memory leaks accumulate across generations. **required fix:** Explicitly clear latent cache after generation. Add memory-pressure handling. -**Cross-ref:** See [Memory Management in Diffusion](pitfalls.md#memory-management-in-diffusion) for latent cache lifecycle guidance +**Cross-ref:** See [Memory Management in Diffusion](blocker-patterns.md#memory-management-in-diffusion) for latent cache lifecycle guidance --- ## Async Safety @@ -207,7 +207,7 @@ async def process(self, request): **Why dangerous:** Blocks event loop, prevents concurrent requests. **required fix:** Use `asyncio.to_thread()` or await pattern. -**Cross-ref:** See [Async Function Complexity](code-patterns.md#async-function-complexity) for async patterns +**Cross-ref:** See [Async Function Complexity](architecture.md#async-function-complexity) for async patterns --- ### Pattern: Hardcoded Timeout @@ -220,7 +220,7 @@ def connect(url): **Required fix:** Make timeout configurable or add timeout parameter to CLI flags. -**Cross-ref:** See [Connector Communication](code-patterns.md#connector-communication) for proper connector usage. +**Cross-ref:** See [Connector Communication](architecture.md#connector-communication) for proper connector usage. --- @@ -239,7 +239,7 @@ for attempt in range(MAX_RETRIES): **Required fix:** Use exponential backoff with configurable max retries or circuit breaker pattern. -**Cross-ref:** See [Stage Lifecycle](code-patterns.md#stage-lifecycle) for lifecycle management. +**Cross-ref:** See [Stage Lifecycle](architecture.md#stage-lifecycle) for lifecycle management. --- @@ -256,7 +256,7 @@ for r in requests: **Required fix:** Use `asyncio.gather()` with `return_exceptions=True` for parallel execution. -**Cross-ref:** See [Async Function Complexity](code-patterns.md#async-function-complexity) for proper async patterns. +**Cross-ref:** See [Async Function Complexity](architecture.md#async-function-complexity) for proper async patterns. --- @@ -273,7 +273,7 @@ else: **Required fix:** Check `torch.cuda.is_available()` first and use appropriate fallback. -**Cross-ref:** See [Distributed Execution Patterns](code-patterns.md#distributed-execution-patterns) for distributed patterns. +**Cross-ref:** See [Distributed Execution Patterns](architecture.md#distributed-execution-patterns) for distributed patterns. --- @@ -290,7 +290,7 @@ response = model.generate(request.json["prompt"]) **Required fix:** Validate all user inputs before passing to model. -**Cross-ref:** See [Input Validation Pattern](code-patterns.md#input-validation-pattern) for validation guidance. +**Cross-ref:** See [Input Validation Pattern](architecture.md#input-validation-pattern) for validation guidance. --- @@ -307,7 +307,7 @@ async def create_item(self, request): **Required fix:** Validate responses match schema and handle errors explicitly. -**Cross-ref:** See [Error Handling Pattern](code-patterns.md#error-handling-pattern) for proper error handling patterns. +**Cross-ref:** See [Error Handling Pattern](architecture.md#error-handling-pattern) for proper error handling patterns. --- @@ -324,7 +324,7 @@ DATABASE_PASSWORD = "secret123" **Required fix:** Use environment variables or secret management systems. -**Cross-ref:** See [Logging Pattern](code-patterns.md#logging-pattern) for proper logging practices. +**Cross-ref:** See [Logging Pattern](architecture.md#logging-pattern) for proper logging practices. --- @@ -338,7 +338,7 @@ data = pickle.loads(user_input) **Required fix:** Never use `pickle.loads()` on untrusted data. Use safe deserialization libraries. -**Cross-ref:** See [Error Handling Pattern](code-patterns.md#error-handling-pattern) for safe error handling patterns. +**Cross-ref:** See [Error Handling Pattern](architecture.md#error-handling-pattern) for safe error handling patterns. --- @@ -352,7 +352,7 @@ cursor.execute(user_input) # Direct string interpolation **Required fix:** Use parameterized queries (prepared statements, ORM). -**Cross-ref:** See [Input Validation Pattern](code-patterns.md#input-validation-pattern) for validation guidance. +**Cross-ref:** See [Input Validation Pattern](architecture.md#input-validation-pattern) for validation guidance. --- @@ -376,7 +376,7 @@ query = f"SELECT * FROM {user_table}" # SQL injection via string formatting **Note:** Normal f-strings like `f"Hello, {name}!"` are safe - they don't execute code, just interpolate. -**Cross-ref:** See [Input Validation Pattern](code-patterns.md#input-validation-pattern) for validation guidance. +**Cross-ref:** See [Input Validation Pattern](architecture.md#input-validation-pattern) for validation guidance. --- @@ -393,7 +393,7 @@ def fix_bug(): **Required fix:** Add a regression test that reproduces the original bug. -**Cross-ref:** See [Test Coverage Requirements](code-patterns.md#test-coverage-requirements) for test coverage requirements. +**Cross-ref:** See [Test Coverage Requirements](architecture.md#test-coverage-requirements) for test coverage requirements. --- @@ -409,7 +409,7 @@ def test_result(): **Required fix:** Use meaningful assertions that verify actual behavior. -**Cross-ref:** See [Test Coverage Requirements](code-patterns.md#test-coverage-requirements) for test coverage requirements. +**Cross-ref:** See [Test Coverage Requirements](architecture.md#test-coverage-requirements) for test coverage requirements. --- @@ -428,7 +428,7 @@ class MockModel(Mixin): # Missing nn.Module **Required fix:** Ensure mocks match production inheritance hierarchy. -**Cross-ref:** See [Test Mock Mismatches](pitfalls.md#test-mock-mismatches) for mock testing guidance. +**Cross-ref:** See [Test Mock Mismatches](blocker-patterns.md#test-mock-mismatches) for mock testing guidance. --- @@ -445,7 +445,7 @@ def process(self, data): **Required fix:** Add tests for error handling, edge cases (empty, null, boundary). -**Cross-ref:** See [Test Coverage Requirements](code-patterns.md#test-coverage-requirements) for test coverage requirements. +**Cross-ref:** See [Test Coverage Requirements](architecture.md#test-coverage-requirements) for test coverage requirements. --- @@ -461,7 +461,7 @@ def optimize(): **Required fix:** Run benchmarks with `pytest` or manual timing, document results. -**Cross-ref:** See [Test Coverage Requirements](code-patterns.md#test-coverage-requirements) for test coverage requirements. +**Cross-ref:** See [Test Coverage Requirements](architecture.md#test-coverage-requirements) for test coverage requirements. --- @@ -475,7 +475,7 @@ pass **Required fix:** Use memory profiling tools and report peak and delta. -**Cross-ref:** See [Memory Management in Diffusion](pitfalls.md#memory-management-in-diffusion) for latent cache lifecycle guidance. +**Cross-ref:** See [Memory Management in Diffusion](blocker-patterns.md#memory-management-in-diffusion) for latent cache lifecycle guidance. --- @@ -490,7 +490,7 @@ def test_distributed(self): **Required fix:** Test in distributed mode with `torchrun`. -**Cross-ref:** See [Tensor Parallelism Edge Cases](pitfalls.md#tensor-parallelism-edge-cases) for distributed testing guidance. +**Cross-ref:** See [Tensor Parallelism Edge Cases](blocker-patterns.md#tensor-parallelism-edge-cases) for distributed testing guidance. --- @@ -507,7 +507,7 @@ def migrate_config(): **Required fix:** Document breaking changes in RELEASE_NOTES.md. Provide migration instructions. -**Cross-ref:** See [API Endpoints](code-patterns.md#connector-communication) for OpenAI compatibility guidance. +**Cross-ref:** See [API Endpoints](architecture.md#connector-communication) for OpenAI compatibility guidance. --- @@ -523,3 +523,230 @@ def add_feature(): **Required fix:** Add usage examples, docstrings. **Cross-ref:** See [Comment Style](review-execution.md#comment-style) for review comment guidance. + +--- + +## Part 2: Common pitfalls (footguns and review prompts) + +*This part holds the former standalone `pitfalls.md` material (MRO, connector state, async vs sync, diffusion memory, API validation, tensor parallel, test mocks).* + +### Mixin + nn.Module MRO Issue + +**Problem:** +When a mixin class is listed **after** `nn.Module` in inheritance, the mixin's `__init__` will **not** be called because `nn.Module.__init__()` doesn't call `super().__init__()`. + +```python +# BROKEN - Mixin's __init__ won't be called +class MyModel(nn.Module, SomeMixin): + def __init__(self): + super().__init__() # Only calls nn.Module.__init__! + self.mixin_method() # CRASH: mixin attributes not initialized +``` + +**Solution - Lazy Initialization:** +```python +class SomeMixin: + @property + def _internal_state(self) -> set: + if not hasattr(self, '_internal_state_storage'): + self._internal_state_storage = set() + return self._internal_state_storage +``` + +**Red Flag in Tests:** +If test mocks inherit only from the mixin (not `nn.Module`), they won't catch this bug because the test's `super().__init__()` WILL call the mixin's `__init__`. + +**Review Action:** +- Check inheritance order when you see mixins +- If mixin has `__init__` that sets attributes, flag it +- Verify tests use realistic class hierarchy + +--- + +### Connector State Management + +**Problem:** +Connectors use shared memory. Improper cleanup leads to leaks and crashes. + +```python +# BROKEN - No cleanup on error +def send_data(self, data): + self.shm_buffer.write(data) + raise ValueError("oops") # Buffer never released! +``` + +**Solution - Context Managers:** +```python +def send_data(self, data): + with self.shm_buffer.acquire() as buf: + buf.write(data) + # Automatically released even on error +``` + +**Review Action:** +- Check error paths in connector code +- Look for `try/finally` or context managers +- Verify cleanup in all branches + +--- + +### Async vs Sync Path Differences + +**Problem:** +`AsyncOmni` and `Omni` have different code paths. Changes may work in one but not the other. + +```python +# Works in sync path +def process(self, request): + result = blocking_call() # OK in sync + return result + +# Breaks in async path +async def process(self, request): + result = blocking_call() # Blocks event loop! + return result +``` + +**Solution:** +```python +async def process(self, request): + result = await asyncio.to_thread(blocking_call) + return result +``` + +**Review Action:** +- Check if PR modifies shared code paths +- Verify both sync and async are tested +- Look for blocking calls in async code + +--- + +### Stage Configuration Validation + +**Problem:** +Stage configs are loaded at init time. Invalid configs may not fail until runtime. + +```python +# Config loaded but not validated +config = load_yaml("stages.yaml") +# ... much later ... +stage = create_stage(config["unknown_stage"]) # Crash! +``` + +**Solution - Eager Validation:** +```python +config = load_yaml("stages.yaml") +validate_stage_config(config) # Fail fast +``` + +**Review Action:** +- Check for config validation at load time +- Verify all required fields are checked +- Look for defaults that mask invalid configs + +--- + +### Memory Management in Diffusion + +**Problem:** +Diffusion models manage large latent caches. Memory leaks accumulate across generations. + +```python +# BROKEN - Latent cache grows unbounded +class DiffusionModel: + def generate(self, prompt): + self.latent_cache.append(create_latents()) # Never cleared! + return decode(self.latent_cache[-1]) +``` + +**Solution:** +```python +class DiffusionModel: + def generate(self, prompt): + latents = create_latents() + try: + return decode(latents) + finally: + del latents # Explicit cleanup +``` + +**Review Action:** +- Check latent cache lifecycle +- Verify cleanup in generation loops +- Look for memory-pressure handling + +--- + +### Input Validation in API Layer + +**Problem:** +Invalid requests that reach the engine can cause crashes instead of clean errors. + +```python +# BROKEN - Validation happens in engine +def create_speech(self, request): + # Missing validation here + return self.engine.generate(request) # Engine crashes! +``` + +**Solution - Validate Early:** +```python +def create_speech(self, request): + error = self._validate_request(request) + if error: + return BadRequest(error) # Clean 400 response + return self.engine.generate(request) +``` + +**Review Action:** +- Check validation before engine calls +- Verify all parameters are validated +- Ensure error messages are actionable + +--- + +### Tensor Parallelism Edge Cases + +**Problem:** +Code may work single-GPU but break with tensor parallelism. + +```python +# Works on single GPU +def forward(self, x): + return self.layer(x) + self.bias # bias not replicated! +``` + +**Solution:** +```python +def forward(self, x): + return self.layer(x) + self.bias.data # Explicitly access data +``` + +**Review Action:** +- Check if changes affect distributed execution +- Verify tensor parallel tests exist +- Look for rank-specific logic + +--- + +### Test Mock Mismatches + +**Problem:** +Tests may mock classes differently from production, hiding bugs. + +```python +# Production +class Model(nn.Module, Mixin): + pass + +# Test mock - different MRO! +class MockModel(Mixin): # Missing nn.Module + pass +``` + +This hides the MRO bug described above. + +**Review Action:** +- Compare mock inheritance to production +- Check if mocks skip critical base classes +- Verify mock behavior matches production diff --git a/.claude/skills/review-pr/references/code-patterns.md b/.claude/skills/review-pr/references/code-patterns.md deleted file mode 100644 index e79517990..000000000 --- a/.claude/skills/review-pr/references/code-patterns.md +++ /dev/null @@ -1,234 +0,0 @@ -# Code Patterns in vLLM-Omni - -## Async Function Complexity - -**Guidelines:** -- Each `await` is a potential failure point and state transition -- Consider splitting long async chains into smaller helpers -- Watch for race conditions in complex async flows -- Use `return_exceptions=True` for parallel operations - -**Good:** -```python -async def process_batch(requests: list[Request]) -> list[Response]: - validated = [_validate_request(r) for r in requests] - results = await asyncio.gather( - *[_process_single(r) for r in validated], - return_exceptions=True - ) - return [_handle_result(r) for r in results] -``` - -**Bad:** -```python -async def process_batch(requests: list[Request]) -> list[Response]: - results = [] - for r in requests: - results.append(await _process_single(r)) # Sequential, no error handling - return results -``` - ---- - -## Distributed Execution Patterns - -**When complexity is justified:** -- Tensor parallelism synchronization across workers -- Pipeline parallelism stage coordination -- Distributed KV cache management -- Multi-node communication patterns - -**Critical questions:** -- Is distributed complexity isolated from business logic? -- Are distributed failure modes handled? -- Is there a clear fallback for single-device execution? - -**Pattern:** -```python -def get_world_size() -> int: - """Returns 1 if not in distributed mode.""" - if not dist.is_initialized(): - return 1 - return dist.get_world_size() -``` - ---- - -## KV Cache Management - -**Guidelines:** -- Ensure clear separation between allocation, update, and cleanup -- Document invariants and assumptions -- Use type system to enforce valid states -- Always handle cleanup in error paths - -**Pattern:** -```python -class KVCache: - def allocate(self, num_blocks: int) -> None: - assert self._state == CacheState.UNINITIALIZED - self._blocks = [Block() for _ in range(num_blocks)] - self._state = CacheState.ALLOCATED - - def update(self, block_id: int, kv_data: Tensor) -> None: - assert self._state == CacheState.ALLOCATED - self._blocks[block_id].update(kv_data) - - def free(self) -> None: - self._blocks = [] - self._state = CacheState.UNINITIALIZED -``` - ---- - -## Input Validation Pattern - -**Guidelines:** -- Validate early (API layer, not engine) -- Return actionable error messages -- Use clean HTTP status codes (400 for client errors) - -**Pattern:** -```python -def _validate_request(self, request: Request) -> str | None: - """Returns error message or None if valid.""" - if not request.input: - return "Input cannot be empty" - - if request.param and request.param not in VALID_PARAMS: - return f"Invalid param '{request.param}'. Valid: {VALID_PARAMS}" - - return None - -def handle_request(self, request: Request) -> Response: - error = self._validate_request(request) - if error: - return Response(status_code=400, content={"error": error}) - return self._process(request) -``` - ---- - -## Connector Communication - -**Guidelines:** -- Use context managers for resource cleanup -- Handle timeouts explicitly -- Implement proper error propagation - -**Pattern:** -```python -async def send_to_next_stage(self, data: Tensor) -> None: - try: - async with self.connector.acquire(timeout=30.0) as channel: - await channel.send(data) - except TimeoutError: - logger.error("Stage communication timeout") - raise StageError("Downstream stage unavailable") - except Exception as e: - logger.error(f"Stage communication failed: {e}") - raise -``` - ---- - -## Stage Lifecycle - -**Guidelines:** -- Stages are configured at init time -- Runtime reconfiguration requires full teardown -- State must be properly managed across stage boundaries - -**Pattern:** -```python -class Stage: - def __init__(self, config: StageConfig): - self._validate_config(config) - self.config = config - self._state = StageState.INITIALIZED - - async def start(self) -> None: - assert self._state == StageState.INITIALIZED - await self._allocate_resources() - self._state = StageState.RUNNING - - async def stop(self) -> None: - if self._state == StageState.RUNNING: - await self._release_resources() - self._state = StageState.STOPPED -``` - ---- - -## Test Coverage Requirements - -| Code Type | Requirement | -|-----------|-------------| -| New features | Happy path + edge cases + error handling | -| Bug fixes | Regression test + edge cases around fix | -| Performance | Benchmarks with before/after measurements | -| Distributed | Must test in distributed mode | -| Quantization | Memory savings + quality impact measured | -| API endpoints | Input validation + error responses | - ---- - -## Code Quality Thresholds - -| Metric | Threshold | Action | -|--------|-----------|--------| -| Function length | 100+ lines | Review for split opportunities | -| Complexity | 11+ | Requires refactoring | -| Nesting depth | 4+ levels | Requires refactoring | -| Parameters | 7+ | Consider parameter object | - ---- - -## Error Handling Pattern - -**Guidelines:** -- Use specific exception types -- Include actionable context in messages -- Don't silently swallow errors - -**Pattern:** -```python -class ValidationError(Exception): - """Raised when input validation fails.""" - pass - -class EngineError(Exception): - """Raised when engine operation fails.""" - pass - -def process(self, request: Request) -> Response: - try: - return self._process_internal(request) - except ValidationError as e: - logger.warning(f"Validation failed: {e}") - return Response(status_code=400, content={"error": str(e)}) - except EngineError as e: - logger.error(f"Engine error: {e}") - return Response(status_code=500, content={"error": "Internal error"}) -``` - ---- - -## Logging Pattern - -**Guidelines:** -- Use appropriate log levels -- Include relevant context -- Avoid logging sensitive data - -```python -# Good -logger.info(f"Processing request {request_id} with model {model_name}") -logger.warning(f"Slow response time: {elapsed_ms}ms for {endpoint}") -logger.error(f"Failed to load model {model_path}: {e}") - -# Bad -logger.info("Processing request") # No context -print(f"Error: {e}") # Wrong logging method -logger.debug(f"User token: {token}") # Sensitive data -``` diff --git a/.claude/skills/review-pr/references/diffusion-checklist.md b/.claude/skills/review-pr/references/diffusion-checklist.md index 3405cf2d7..bd2698008 100644 --- a/.claude/skills/review-pr/references/diffusion-checklist.md +++ b/.claude/skills/review-pr/references/diffusion-checklist.md @@ -1,6 +1,6 @@ # Diffusion Model Contribution Review Checklist -Use this checklist as a structured pass after reading the PR diff and body. Each item maps to a dimension in the main SKILL.md workflow. +Single reference for diffusion PRs: PR body criteria (including optional canonical template and measurement guidance), code/docs/tests dimensions, and Quick Red Flags. Use after reading the PR diff and body. --- @@ -61,6 +61,259 @@ Use this checklist as a structured pass after reading the PR diff and body. Each - Peak VRAM under same conditions - Delta (VRAM_vllm_omni − VRAM_diffusers) highlighted if significant + +### 1.3 Canonical PR body template (gold standard) + +Contributors can paste this structure into the PR description. Reviewers use it as the reference for what “complete” means for Dimension 1. + +````markdown +## Summary + +[One paragraph: what model is being added, architecture type (DiT / AR+DiT / etc.), +and which capabilities are introduced (text-to-image / video / editing / etc.)] + +**Model:** `org/model-id` +**Architecture:** DiT / AR+DiT / other +**Modality:** text-to-image / text-to-video / image-editing + +--- + +## vLLM-Omni Generation + +### Offline Inference + +```python +from vllm_omni.entrypoints.omni import Omni + +omni = Omni(model="org/model-id") +outputs = omni.generate("a cup of coffee on a wooden table, photorealistic") +outputs[0].request_output[0].images[0].save("output.png") +``` + +### Online Serving + +```bash +# Start server +vllm serve org/model-id --omni --port 8091 + +# Generate +curl -s http://localhost:8091/v1/chat/completions \ + -H "Content-Type: application/json" \ + -d '{ + "messages": [{"role": "user", "content": "a cup of coffee on a wooden table"}], + "extra_body": {"height": 1024, "width": 1024, "num_inference_steps": 50, "seed": 42} + }' | jq -r '.choices[0].message.content[0].image_url.url' \ + | cut -d',' -f2 | base64 -d > output.png +``` + +### Sample Outputs + +| Prompt | Output | +|--------|--------| +| "a cup of coffee on a wooden table, photorealistic" | ![sample1](./assets/sample1.png) | +| "a futuristic city at night, neon lights" | ![sample2](./assets/sample2.png) | + +### Performance Measurements + +**Hardware:** 1× NVIDIA A100 80GB +**Resolution:** 1024×1024 +**Steps:** 50 + +| Metric | Value | +|--------|-------| +| e2e latency | X.X s | +| Peak VRAM | XX.X GB | + +--- + +## diffusers Baseline (Recommended) + +### Script + +```python +from diffusers import StableDiffusionXLPipeline +import torch + +pipe = StableDiffusionXLPipeline.from_pretrained("org/model-id", torch_dtype=torch.float16) +pipe = pipe.to("cuda") +image = pipe("a cup of coffee on a wooden table, photorealistic", + num_inference_steps=50, height=1024, width=1024).images[0] +image.save("output_diffusers.png") +``` + +### Comparison + +| | vLLM-Omni | diffusers | +|-|-----------|-----------| +| e2e latency | X.X s | X.X s | +| Peak VRAM | XX.X GB | XX.X GB | +| Sample | ![vllm_omni](./assets/sample_vllm.png) | ![diffusers](./assets/sample_diffusers.png) | + +--- + +## Acceleration Features + +### Supported Features + +| Feature | Supported | CLI Flag / Config Key | +|---------|----------|----------------------| +| Sequence Parallel | ✅ | `--sequence-parallel-size N` | +| CFG Parallel | ❌ | — | +| VAE Patch Parallel | ✅ | `vae_patch_parallel: true` | +| Tensor Parallel | ✅ | `--tensor-parallel-size N` | +| TeaCache | ✅ | `--teacache-thresh 0.15` | + +### Acceleration Demo + +```bash +# Tensor Parallel across 2 GPUs +vllm serve org/model-id --omni --tensor-parallel-size 2 + +# TeaCache for 1.5× speedup +vllm serve org/model-id --omni --teacache-thresh 0.15 +``` + +| Config | Latency | VRAM/GPU | +|--------|---------|---------| +| Baseline (1 GPU) | X.X s | XX GB | +| TP=2 | X.X s | XX GB | +| TeaCache | X.X s | XX GB | +| TP=2 + TeaCache | X.X s | XX GB | + +--- + +## Memory Optimization Features + +### Supported Options + +| Feature | Supported | CLI Flag / Config Key | +|---------|----------|----------------------| +| CPU Offload | ✅ | `--enable-cpu-offload` | +| FP8 Quantization | ✅ | `--quantization fp8` | +| VAE Tiling | ✅ | `vae_tiling: true` | + +### Memory Reduction Demo + +```bash +# CPU offload (reduces VRAM from XX GB to YY GB) +vllm serve org/model-id --omni --enable-cpu-offload + +# FP8 quantization +vllm serve org/model-id --omni --quantization fp8 +``` + +| Config | Peak VRAM | Latency | +|--------|----------|---------| +| Baseline (BF16) | XX.X GB | X.X s | +| CPU Offload 10 GB | XX.X GB | X.X s | +| FP8 Quantization | XX.X GB | X.X s | + +--- + +## Combined Feature Validation + +```bash +# CPU offload + TeaCache + Tensor Parallel +vllm serve org/model-id --omni \ + --tensor-parallel-size 2 \ + --teacache-thresh 0.15 \ + --cpu-offload-gb 5 +``` + +| Config | Latency | VRAM/GPU | Quality Notes | +|--------|---------|---------|--------------| +| All features combined | X.X s | XX GB | No visible quality degradation | + +Sample output with combined config: ![combined](./assets/sample_combined.png) + +--- + +## Documentation Updates + +- [ ] `docs/models/supported_models.md` — model row added +- [ ] `docs/features/feature_support.md` — feature row added +- [ ] `docs/features/compatibility.md` — compatibility table updated (if applicable) +- [ ] `docs/models/.md` — usage example doc added + +--- + +## Test Coverage + +- [ ] `tests/e2e/online_serving/test__expansion.py` — online serving e2e test +- [ ] `tests/models/test_.py` — offline inference test +```` + +--- + +#### Measurement guidelines + +##### How to measure e2e latency + +**Offline:** +```python +import time +from vllm_omni.entrypoints.omni import Omni + +omni = Omni(model="org/model-id") + +# Warm-up (1 run, not measured) +omni.generate("warm-up prompt") + +# Timed run +start = time.perf_counter() +outputs = omni.generate("a cup of coffee on a wooden table") +end = time.perf_counter() +print(f"e2e latency: {end - start:.2f}s") +``` + +**Online (using `curl` with timing):** +```bash +time curl -s http://localhost:8091/v1/chat/completions \ + -H "Content-Type: application/json" \ + -d '{"messages": [{"role": "user", "content": "a cup of coffee"}], ...}' +``` + +##### How to measure peak VRAM + +```python +import torch +torch.cuda.reset_peak_memory_stats() + +# Run generation here + +peak_mb = torch.cuda.max_memory_allocated() / 1024**2 +print(f"Peak VRAM: {peak_mb / 1024:.1f} GB") +``` + +Or via `nvidia-smi`: +```bash +nvidia-smi --query-gpu=memory.used --format=csv,noheader,nounits -l 1 +``` + +##### What “same conditions” means for diffusers comparison + +When comparing vLLM-Omni vs diffusers: +- Same GPU model and count +- Same prompt +- Same resolution (H × W) +- Same `num_inference_steps` +- Same `seed` +- Same data type (both BF16 or both FP16) +- Both measured after one warm-up run + +--- + +#### Common PR body mistakes + +| Mistake | Correct Approach | +|---------|-----------------| +| Using a tiny resolution (256×256) for latency measurement | Use the model's recommended resolution (typically 512×512 minimum, 1024×1024 for flagship models) | +| Reporting latency without warm-up run | Always warm up the model first; measure on the second+ run | +| Showing only one sample image | Show 2–3 samples with diverse prompts to demonstrate quality | +| Only demonstrating offline mode | Include both offline (`Omni`) and online (`vllm serve`) | +| Reporting VRAM from `nvidia-smi` at idle | Measure peak VRAM during active generation | +| Listing features as "planned" in the support table | Only mark ✅ for features that are implemented and tested in this PR | + --- ## Dimension 2: Code Checks diff --git a/.claude/skills/review-pr/references/diffusion-pr-requirements.md b/.claude/skills/review-pr/references/diffusion-pr-requirements.md deleted file mode 100644 index 8bfbe3d7e..000000000 --- a/.claude/skills/review-pr/references/diffusion-pr-requirements.md +++ /dev/null @@ -1,257 +0,0 @@ -# PR Body Requirements for Diffusion Model Contributions - -This document defines the exact structure and content expected in the PR description when adding a new diffusion model to vLLM-Omni. Reviewers use this as the source of truth when evaluating PR body completeness (Dimension 1). - ---- - -## Complete PR Body Template - -Below is the canonical template. PRs that follow this template pass Dimension 1 automatically. - -````markdown -## Summary - -[One paragraph: what model is being added, architecture type (DiT / AR+DiT / etc.), -and which capabilities are introduced (text-to-image / video / editing / etc.)] - -**Model:** `org/model-id` -**Architecture:** DiT / AR+DiT / other -**Modality:** text-to-image / text-to-video / image-editing - ---- - -## vLLM-Omni Generation - -### Offline Inference - -```python -from vllm_omni.entrypoints.omni import Omni - -omni = Omni(model="org/model-id") -outputs = omni.generate("a cup of coffee on a wooden table, photorealistic") -outputs[0].request_output[0].images[0].save("output.png") -``` - -### Online Serving - -```bash -# Start server -vllm serve org/model-id --omni --port 8091 - -# Generate -curl -s http://localhost:8091/v1/chat/completions \ - -H "Content-Type: application/json" \ - -d '{ - "messages": [{"role": "user", "content": "a cup of coffee on a wooden table"}], - "extra_body": {"height": 1024, "width": 1024, "num_inference_steps": 50, "seed": 42} - }' | jq -r '.choices[0].message.content[0].image_url.url' \ - | cut -d',' -f2 | base64 -d > output.png -``` - -### Sample Outputs - -| Prompt | Output | -|--------|--------| -| "a cup of coffee on a wooden table, photorealistic" | ![sample1](./assets/sample1.png) | -| "a futuristic city at night, neon lights" | ![sample2](./assets/sample2.png) | - -### Performance Measurements - -**Hardware:** 1× NVIDIA A100 80GB -**Resolution:** 1024×1024 -**Steps:** 50 - -| Metric | Value | -|--------|-------| -| e2e latency | X.X s | -| Peak VRAM | XX.X GB | - ---- - -## diffusers Baseline (Recommended) - -### Script - -```python -from diffusers import StableDiffusionXLPipeline -import torch - -pipe = StableDiffusionXLPipeline.from_pretrained("org/model-id", torch_dtype=torch.float16) -pipe = pipe.to("cuda") -image = pipe("a cup of coffee on a wooden table, photorealistic", - num_inference_steps=50, height=1024, width=1024).images[0] -image.save("output_diffusers.png") -``` - -### Comparison - -| | vLLM-Omni | diffusers | -|-|-----------|-----------| -| e2e latency | X.X s | X.X s | -| Peak VRAM | XX.X GB | XX.X GB | -| Sample | ![vllm_omni](./assets/sample_vllm.png) | ![diffusers](./assets/sample_diffusers.png) | - ---- - -## Acceleration Features - -### Supported Features - -| Feature | Supported | CLI Flag / Config Key | -|---------|----------|----------------------| -| Sequence Parallel | ✅ | `--sequence-parallel-size N` | -| CFG Parallel | ❌ | — | -| VAE Patch Parallel | ✅ | `vae_patch_parallel: true` | -| Tensor Parallel | ✅ | `--tensor-parallel-size N` | -| TeaCache | ✅ | `--teacache-thresh 0.15` | - -### Acceleration Demo - -```bash -# Tensor Parallel across 2 GPUs -vllm serve org/model-id --omni --tensor-parallel-size 2 - -# TeaCache for 1.5× speedup -vllm serve org/model-id --omni --teacache-thresh 0.15 -``` - -| Config | Latency | VRAM/GPU | -|--------|---------|---------| -| Baseline (1 GPU) | X.X s | XX GB | -| TP=2 | X.X s | XX GB | -| TeaCache | X.X s | XX GB | -| TP=2 + TeaCache | X.X s | XX GB | - ---- - -## Memory Optimization Features - -### Supported Options - -| Feature | Supported | CLI Flag / Config Key | -|---------|----------|----------------------| -| CPU Offload | ✅ | `--enable-cpu-offload` | -| FP8 Quantization | ✅ | `--quantization fp8` | -| VAE Tiling | ✅ | `vae_tiling: true` | - -### Memory Reduction Demo - -```bash -# CPU offload (reduces VRAM from XX GB to YY GB) -vllm serve org/model-id --omni --enable-cpu-offload - -# FP8 quantization -vllm serve org/model-id --omni --quantization fp8 -``` - -| Config | Peak VRAM | Latency | -|--------|----------|---------| -| Baseline (BF16) | XX.X GB | X.X s | -| CPU Offload 10 GB | XX.X GB | X.X s | -| FP8 Quantization | XX.X GB | X.X s | - ---- - -## Combined Feature Validation - -```bash -# CPU offload + TeaCache + Tensor Parallel -vllm serve org/model-id --omni \ - --tensor-parallel-size 2 \ - --teacache-thresh 0.15 \ - --cpu-offload-gb 5 -``` - -| Config | Latency | VRAM/GPU | Quality Notes | -|--------|---------|---------|--------------| -| All features combined | X.X s | XX GB | No visible quality degradation | - -Sample output with combined config: ![combined](./assets/sample_combined.png) - ---- - -## Documentation Updates - -- [ ] `docs/models/supported_models.md` — model row added -- [ ] `docs/features/feature_support.md` — feature row added -- [ ] `docs/features/compatibility.md` — compatibility table updated (if applicable) -- [ ] `docs/models/.md` — usage example doc added - ---- - -## Test Coverage - -- [ ] `tests/e2e/online_serving/test__expansion.py` — online serving e2e test -- [ ] `tests/models/test_.py` — offline inference test -```` - ---- - -## Measurement Guidelines - -### How to Measure e2e Latency - -**Offline:** -```python -import time -from vllm_omni.entrypoints.omni import Omni - -omni = Omni(model="org/model-id") - -# Warm-up (1 run, not measured) -omni.generate("warm-up prompt") - -# Timed run -start = time.perf_counter() -outputs = omni.generate("a cup of coffee on a wooden table") -end = time.perf_counter() -print(f"e2e latency: {end - start:.2f}s") -``` - -**Online (using `curl` with timing):** -```bash -time curl -s http://localhost:8091/v1/chat/completions \ - -H "Content-Type: application/json" \ - -d '{"messages": [{"role": "user", "content": "a cup of coffee"}], ...}' -``` - -### How to Measure Peak VRAM - -```python -import torch -torch.cuda.reset_peak_memory_stats() - -# Run generation here - -peak_mb = torch.cuda.max_memory_allocated() / 1024**2 -print(f"Peak VRAM: {peak_mb / 1024:.1f} GB") -``` - -Or via `nvidia-smi`: -```bash -nvidia-smi --query-gpu=memory.used --format=csv,noheader,nounits -l 1 -``` - -### What "Same Conditions" Means for diffusers Comparison - -When comparing vLLM-Omni vs diffusers: -- Same GPU model and count -- Same prompt -- Same resolution (H × W) -- Same `num_inference_steps` -- Same `seed` -- Same data type (both BF16 or both FP16) -- Both measured after one warm-up run - ---- - -## Common PR Body Mistakes - -| Mistake | Correct Approach | -|---------|-----------------| -| Using a tiny resolution (256×256) for latency measurement | Use the model's recommended resolution (typically 512×512 minimum, 1024×1024 for flagship models) | -| Reporting latency without warm-up run | Always warm up the model first; measure on the second+ run | -| Showing only one sample image | Show 2–3 samples with diverse prompts to demonstrate quality | -| Only demonstrating offline mode | Include both offline (`Omni`) and online (`vllm serve`) | -| Reporting VRAM from `nvidia-smi` at idle | Measure peak VRAM during active generation | -| Listing features as "planned" in the support table | Only mark ✅ for features that are implemented and tested in this PR | diff --git a/.claude/skills/review-pr/references/maintainer-style-study.md b/.claude/skills/review-pr/references/maintainer-style-study.md new file mode 100644 index 000000000..5fd51062e --- /dev/null +++ b/.claude/skills/review-pr/references/maintainer-style-study.md @@ -0,0 +1,114 @@ +# Maintainer Review Style Study + +Calibration data from **200 DarkLight1337 (Cyrus Leung) reviews** on vllm-project/vllm plus **400+ comments** from 12 other core maintainers across vllm and vllm-omni. Use this as ground truth for how real reviews should look. + +## Reviewers Studied + +| Reviewer | Role | Sample Size | +|----------|------|-------------| +| DarkLight1337 | Core maintainer (multimodal) | ~200 reviews deep dive | +| njhill | Core maintainer (engine/perf) | ~24 inline + replies | +| jeejeelee | Model reviewer | ~11 inline | +| hmellor | Core maintainer (models/transformers) | ~9 inline | +| ProExpertProg | Core maintainer (compilation/IR) | ~14 inline | +| mgoin | Quantization/eval | ~9 inline + replies | +| WoosukKwon | Project lead | Few but high-signal | +| varun-sundar-rabindranath | Kernel reviewer | ~27 inline | +| AndreasKaratzas | ROCm CI | ~23 inline + many replies | + +## DarkLight1337 Review Body Stats + +| Pattern | Frequency | +|---------|-----------| +| Empty body (just inline comments or silent APPROVE) | ~60% | +| Ultra-short ("Thanks", "LGTM", "Some more nits") | ~25% | +| One-sentence architectural point | ~10% | +| Multi-sentence design discussion | ~5% | + +## DarkLight1337 Inline Comment Distribution + +| Type | Frequency | Examples | +|------|-----------|---------| +| Direct imperative | ~35% | "Move imports to the top", "Please keep in alphabetical order", "Revert changes to this file", "Fix this" | +| Direct question | ~25% | "Is this really needed?", "Why change this?", "Where is this defined?", "Does X not work here?" | +| Suggestion blocks | ~15% | Code fix with zero explanation text | +| "Can you/we" requests | ~10% | "Can you move X to Y?", "Can you address this?", "Can we cache this?" | +| Soft opinion | ~8% | "Tbh I think...", "I prefer...", "IMO...", "I don't like how..." | +| Ultra-short | ~7% | "Ditto", "Same", "No need", "Good catch", "Seems unused" | + +## DarkLight1337 Stock Phrases + +These are real phrases used repeatedly across 200 reviews: + +- "Is this really needed?" +- "Can you address this?" / "Please address this" +- "Seems unused" / "Looks unused" +- "Is this change related?" +- "Please keep in alphabetical order" +- "Please fix pre-commit" +- "Move imports to the top" / "Import from the top" +- "Remove the commented out code" +- "Let's keep things simple" +- "Any update?" +- "Tbh I think..." +- "I don't quite understand this" +- "Hmm... that's true, OK then" +- "Sorry for the delay!" +- "Thanks for your patience!" + +## Things DarkLight1337 Never Does + +- Never prefixes with "Nit:" -- just states the issue +- Never says "left a few comments inline" +- Never praises code structure inline ("Good placement", "Nice refactor") +- Never over-explains obvious suggestion blocks +- Never uses structured review templates (## Summary, BLOCKER tables) +- Never uses dramatic emphasis ("CRITICAL", "BREAKING") + +## Other Maintainer Patterns + +### Comment Length +- **Ultra-short (1-5 words):** "ditto", "DITTO", "Done", "Fixed", "Good catch", "nope" +- **Short (1 line):** "Can we fuse this add into norm?", "Why not use field(default_factory=...)?" +- **Medium (2-4 lines):** Typical for suggestions with brief rationale +- **Long (5+ lines):** RARE -- only for architectural discussions + +**Distribution:** ~50% ultra-short/short, ~30% medium, ~20% long + +### Reply Thread Patterns + +**Acknowledgments:** +- "Done" / "Fixed" / "Oops, fixed" -- DarkLight1337 +- "Makes sense" -- jeejeelee +- "Good catch" -- multiple reviewers + +**Pushback (direct, not hedged):** +- "This is pre-existing behavior" -- AndreasKaratzas +- "We cannot do that because of [link]" -- DarkLight1337 +- "I think you misunderstood this" -- njhill + +**Concessions:** +- "Alright, thanks for explaining" -- DarkLight1337 +- "Yeah that's fair, I didn't notice that initially" -- ProExpertProg + +### PR-Level Comments +- "cc @X" -- extremely common for tagging area owners +- "@X PTAL." -- (Please Take A Look) + +### Review Body Patterns +- "LGTM." / "Looks good." +- "this is consistent with our other models. thanks!" +- "I assume you have tested this locally." + +## Key Gaps Between AI Reviews and Real Maintainers + +| Aspect | Real Maintainers | Common AI Pattern | Fix | +|--------|-----------------|-------------------|-----| +| Review body | ~60% empty | Always writes a body | Skip body ~50% | +| "Nit:" prefix | Rarely used | Overused | Drop it -- just state the issue | +| "left X comments" | Never | Common preamble | Never say this | +| Hedging | ~15% of comments | ~50%+ | Be direct by default | +| Empty APPROVE | ~30% of approvals | Rare | Use more silent approves | +| Depth variation | 0-15 comments | Consistently 3-5 | Vary more | +| Structured templates | Never | Common (## Summary, tables) | Drop all templates | +| Inline praise | Never | Occasional | Never do it | diff --git a/.claude/skills/review-pr/references/perf-verification.md b/.claude/skills/review-pr/references/perf-verification.md new file mode 100644 index 000000000..ffc23438b --- /dev/null +++ b/.claude/skills/review-pr/references/perf-verification.md @@ -0,0 +1,220 @@ +# Performance & Accuracy Claim Verification + +Reviewer-side verification of contributor perf/accuracy claims. This is **blocking**: must complete before issuing a verdict. + +**Key distinction:** `tests-docs-checklist.md` Section 5 defines what the *contributor* must provide. This file defines how the *reviewer* can independently verify those claims. + +**Requires a local clone** of `vllm-project/vllm-omni`. If no clone exists, skip to static-only analysis. + +--- + +## 1. Claim Detection + +Extract quantitative claims from PR body and comments. + +### Regex patterns + +| Claim type | Pattern | Captures | +|-----------|---------|----------| +| Latency | `(\d+(?:\.\d+)?)\s*(ms|s|sec)` | value, unit | +| Throughput | `(\d+(?:\.\d+)?)\s*(tok/s|tokens/s|req/s|iters/s|qps)` | value, unit | +| VRAM | `(\d+(?:\.\d+)?)\s*(GB|MB)\s*(?:VRAM|memory|GPU)` | value, unit | +| Speedup | `(\d+(?:\.\d+)?)\s*%?\s*(?:faster|slower|speedup|reduction|improvement)` | value, direction | +| Accuracy | `(?:FID|F1|BLEU|WER|CER|psnr|ssim|accuracy|pass@k)\s*[:=]?\s*([\d.]+)` | metric, value | + +**When to activate:** PR has `[Performance]` prefix, or PR body contains 2+ matches from the patterns above, or Step 5 (Evidence) flagged missing benchmarks. + +--- + +## 2. Hardware Detection (single source of truth) + +This section is the canonical hardware detection logic. Cross-referenced by `test-quality-evaluation.md`. + +### Detection sequence + +```bash +# CUDA +if command -v nvidia-smi &>/dev/null; then + GPU_INFO=$(nvidia-smi --query-gpu=name,memory.total --format=csv,noheader 2>/dev/null) + if [ -n "$GPU_INFO" ]; then + GPU_COUNT=$(echo "$GPU_INFO" | wc -l | tr -d ' ') + GPU_MODEL=$(echo "$GPU_INFO" | head -1 | cut -d',' -f1 | xargs) + VRAM_PER_GPU_GB=$(echo "$GPU_INFO" | head -1 | cut -d',' -f2 | tr -d ' MiB' | awk '{printf "%.1f", $1/1024}') + PLATFORM="CUDA" + fi +fi + +# ROCm (if no CUDA) +if [ -z "$PLATFORM" ] && command -v rocm-smi &>/dev/null; then + GPU_INFO=$(rocm-smi --showmeminfo vram 2>/dev/null | grep -E "GPU|VRAM" | head -1) + # Parse ROCm output — model and VRAM extraction varies by version + PLATFORM="ROCm" +fi + +# NPU (torch-based) +if [ -z "$PLATFORM" ]; then + python3 -c "import torch_npu; print('NPU')" 2>/dev/null && PLATFORM="NPU" +fi + +# XPU (Intel) +if [ -z "$PLATFORM" ]; then + python3 -c "import torch.xpu; print('XPU')" 2>/dev/null && PLATFORM="XPU" +fi + +# CPU-only fallback +if [ -z "$PLATFORM" ]; then + PLATFORM="CPU" + GPU_COUNT=0 + VRAM_PER_GPU_GB=0 +fi +``` + +**Output variables:** `PLATFORM`, `GPU_COUNT`, `GPU_MODEL`, `VRAM_PER_GPU_GB` + +--- + +## 3. Feasibility Check + +Compare model requirements against available hardware. + +### Model size estimation + +Do NOT use parameter count × dtype bytes alone — this significantly underestimates actual VRAM: + +``` +estimated_vram_gb = ( + model_weights_gb # param_count × dtype_bytes / 1e9 + + kv_cache_gb # sequence_length × num_layers × 2 × hidden_size × dtype_bytes × batch_size / 1e9 + + activation_overhead_gb # ~20-30% of model weights for inference + + framework_overhead_gb # ~1-2 GB for CUDA context, fragmentation +) +``` + +For quick estimation when KV cache size is unknown, use a multiplier: + +| Model size | Quick estimate | +|-----------|---------------| +| < 7B params | weights × 1.5 | +| 7-14B params | weights × 2.0 | +| 14-70B params | weights × 2.5 | +| 70B+ params | weights × 3.0 | + +### Verdict + +| Condition | Verdict | +|-----------|---------| +| `estimated_vram_gb ≤ VRAM_PER_GPU_GB × GPU_COUNT` | `VERIFIABLE` | +| `estimated_vram_gb ≤ VRAM_PER_GPU_GB × GPU_COUNT × 1.5` and `--cpu-offload-gb` available | `NEEDS_OFFLOAD` | +| Otherwise | `CANNOT_VERIFY` | + +**CPU-only path:** Can verify latency/accuracy for small models (< 2B params) that fit in system RAM. Cannot measure peak VRAM. + +**No GPU path:** Cannot execute benchmarks. Proceed to static-only analysis. + +--- + +## 4. Benchmark Plan Generation + +Map PR type to benchmark runner. + +| PR type | Benchmark runner | Notes | +|---------|-----------------|-------| +| Omni models (LLM) | `Omni(model=...).generate()` with `--log-stats` | Offline; or `vllm bench serve --omni` for online | +| Diffusion models | `benchmarks/diffusion/diffusion_benchmark_serving.py` | From vllm-omni repo | +| Accuracy (GEBench/GEdit-Bench) | `benchmarks/accuracy/` scripts | From vllm-omni repo | + +### Execution strategy + +1. Create git worktrees at `.claude/perf-worktrees/base` and `.claude/perf-worktrees/pr` +2. Install vllm-omni in each worktree (or share the install if dependencies are identical) +3. Run benchmark in base worktree, capture JSON results +4. Run benchmark in PR worktree, capture JSON results +5. Compare results + +--- + +## 5. Tolerance Thresholds + +These are **configurable defaults**. Adjust based on benchmark type, model size, and input variability. + +| Metric | Default tolerance | When to tighten | When to loosen | +|--------|------------------|-----------------|----------------| +| Latency | ±10% | Microbenchmarks (< 100ms): ±20% | Large models with high variance: ±15% | +| Throughput | ±10% | High-concurrency benchmarks: ±15% | Single-request: ±20% | +| VRAM | ±5% | Small models (< 4GB): ±10% | Multi-GPU: ±10% | +| Accuracy | ±1% (absolute) | FID/SSIM: ±2% | Perplexity: ±0.5% | + +**Disclaimer:** These thresholds assume controlled benchmark conditions (same hardware, same inputs, warm-up run completed). Results outside tolerance do not automatically mean NOT_CONFIRMED — check for measurement noise, input variability, or warm-up issues first. + +--- + +## 6. Execution Limits + +- **Hard timeout:** 20 minutes per benchmark run (base or PR). If exceeded, kill the process and report partial results with a `TIMEOUT` flag. +- **Pre-execution gate:** Before starting benchmark execution, present the plan to the user: + - What benchmark will run + - Estimated duration + - Which model and hardware + - Ask for explicit confirmation +- **No confirmation or timeout:** Fall back to static-only analysis. + +--- + +## 7. Report Format + +### Claimed vs Measured table + +```markdown +## Perf Verification Report + +**Reviewer hardware:** {GPU_MODEL} × {GPU_COUNT} ({VRAM_PER_GPU_GB} GB each), {PLATFORM} +**Verification mode:** {full | partial (offload) | static-only} +**Benchmark:** {type} + +| Claim | Claimed | Measured | Delta | Verdict | +|-------|---------|----------|-------|---------| +| e2e latency | X ms | Y ms | +Z% | CONFIRMED / NOT_CONFIRMED / TIMEOUT | +| throughput | X tok/s | Y tok/s | -Z% | CONFIRMED / NOT_CONFIRMED / TIMEOUT | +| peak VRAM | X GB | Y GB | +Z GB | CONFIRMED / NOT_CONFIRMED / CANNOT_VERIFY | +``` + +### Verdict values + +| Verdict | Meaning | +|---------|---------| +| `CONFIRMED` | Measured value within tolerance of claimed | +| `NOT_CONFIRMED` | Measured value outside tolerance | +| `CANNOT_VERIFY` | Hardware insufficient (model too large, no GPU) | +| `TIMEOUT` | Benchmark exceeded 20-min limit | +| `NOT_RUN` | Benchmark not executed (user declined or no clone) | + +--- + +## 8. Graceful Degradation + +| Level | Conditions | What runs | +|-------|-----------|-----------| +| **Full verification** | GPU available, model fits in VRAM | Before/after benchmarks, full comparison | +| **Partial verification** | GPU available, model needs offload | Benchmarks with `--cpu-offload-gb`, noted in report | +| **Static-only** | No GPU or model too large | Claim detection, analyze benchmark scripts in diff for correctness, flag implausible claims (e.g., "50% speedup" from a single-threaded loop change) | +| **Skip** | No relevant perf claims in PR | Do not activate this step | + +### Static-only analysis checklist + +When benchmarks cannot run, apply these checks to the benchmark/test code in the diff: + +- [ ] Benchmark uses warm-up runs (at least 1) before timed runs +- [ ] Measurement uses `time.perf_counter()` or equivalent (not `time.time()`) +- [ ] VRAM measured via `torch.cuda.max_memory_allocated()` (not `nvidia-smi` at idle) +- [ ] Same hardware, model, inputs for before/after (not different machines) +- [ ] Multiple iterations reported (not single-run claims) +- [ ] No cherry-picked metrics (e.g., reporting best-of-5 without mentioning variance) + +--- + +## 9. Delivery + +1. **Local report first** — output the Claimed vs Measured table to the user +2. **Ask before posting** — "Post this as a PR comment? (y/n)" +3. If posting, use the existing review comment workflow (`gh api`) — this counts against the comment budget +4. If verification reveals a blocker (confirmed NOT_CONFIRMED for accuracy or VRAM regression), escalate to REQUEST_CHANGES via the normal verdict workflow diff --git a/.claude/skills/review-pr/references/pitfalls.md b/.claude/skills/review-pr/references/pitfalls.md deleted file mode 100644 index 30ee80e1a..000000000 --- a/.claude/skills/review-pr/references/pitfalls.md +++ /dev/null @@ -1,222 +0,0 @@ -# Common Pitfalls in vLLM-Omni - -## Mixin + nn.Module MRO Issue - -**Problem:** -When a mixin class is listed **after** `nn.Module` in inheritance, the mixin's `__init__` will **not** be called because `nn.Module.__init__()` doesn't call `super().__init__()`. - -```python -# BROKEN - Mixin's __init__ won't be called -class MyModel(nn.Module, SomeMixin): - def __init__(self): - super().__init__() # Only calls nn.Module.__init__! - self.mixin_method() # CRASH: mixin attributes not initialized -``` - -**Solution - Lazy Initialization:** -```python -class SomeMixin: - @property - def _internal_state(self) -> set: - if not hasattr(self, '_internal_state_storage'): - self._internal_state_storage = set() - return self._internal_state_storage -``` - -**Red Flag in Tests:** -If test mocks inherit only from the mixin (not `nn.Module`), they won't catch this bug because the test's `super().__init__()` WILL call the mixin's `__init__`. - -**Review Action:** -- Check inheritance order when you see mixins -- If mixin has `__init__` that sets attributes, flag it -- Verify tests use realistic class hierarchy - ---- - -## Connector State Management - -**Problem:** -Connectors use shared memory. Improper cleanup leads to leaks and crashes. - -```python -# BROKEN - No cleanup on error -def send_data(self, data): - self.shm_buffer.write(data) - raise ValueError("oops") # Buffer never released! -``` - -**Solution - Context Managers:** -```python -def send_data(self, data): - with self.shm_buffer.acquire() as buf: - buf.write(data) - # Automatically released even on error -``` - -**Review Action:** -- Check error paths in connector code -- Look for `try/finally` or context managers -- Verify cleanup in all branches - ---- - -## Async vs Sync Path Differences - -**Problem:** -`AsyncOmni` and `Omni` have different code paths. Changes may work in one but not the other. - -```python -# Works in sync path -def process(self, request): - result = blocking_call() # OK in sync - return result - -# Breaks in async path -async def process(self, request): - result = blocking_call() # Blocks event loop! - return result -``` - -**Solution:** -```python -async def process(self, request): - result = await asyncio.to_thread(blocking_call) - return result -``` - -**Review Action:** -- Check if PR modifies shared code paths -- Verify both sync and async are tested -- Look for blocking calls in async code - ---- - -## Stage Configuration Validation - -**Problem:** -Stage configs are loaded at init time. Invalid configs may not fail until runtime. - -```python -# Config loaded but not validated -config = load_yaml("stages.yaml") -# ... much later ... -stage = create_stage(config["unknown_stage"]) # Crash! -``` - -**Solution - Eager Validation:** -```python -config = load_yaml("stages.yaml") -validate_stage_config(config) # Fail fast -``` - -**Review Action:** -- Check for config validation at load time -- Verify all required fields are checked -- Look for defaults that mask invalid configs - ---- - -## Memory Management in Diffusion - -**Problem:** -Diffusion models manage large latent caches. Memory leaks accumulate across generations. - -```python -# BROKEN - Latent cache grows unbounded -class DiffusionModel: - def generate(self, prompt): - self.latent_cache.append(create_latents()) # Never cleared! - return decode(self.latent_cache[-1]) -``` - -**Solution:** -```python -class DiffusionModel: - def generate(self, prompt): - latents = create_latents() - try: - return decode(latents) - finally: - del latents # Explicit cleanup -``` - -**Review Action:** -- Check latent cache lifecycle -- Verify cleanup in generation loops -- Look for memory-pressure handling - ---- - -## Input Validation in API Layer - -**Problem:** -Invalid requests that reach the engine can cause crashes instead of clean errors. - -```python -# BROKEN - Validation happens in engine -def create_speech(self, request): - # Missing validation here - return self.engine.generate(request) # Engine crashes! -``` - -**Solution - Validate Early:** -```python -def create_speech(self, request): - error = self._validate_request(request) - if error: - return BadRequest(error) # Clean 400 response - return self.engine.generate(request) -``` - -**Review Action:** -- Check validation before engine calls -- Verify all parameters are validated -- Ensure error messages are actionable - ---- - -## Tensor Parallelism Edge Cases - -**Problem:** -Code may work single-GPU but break with tensor parallelism. - -```python -# Works on single GPU -def forward(self, x): - return self.layer(x) + self.bias # bias not replicated! -``` - -**Solution:** -```python -def forward(self, x): - return self.layer(x) + self.bias.data # Explicitly access data -``` - -**Review Action:** -- Check if changes affect distributed execution -- Verify tensor parallel tests exist -- Look for rank-specific logic - ---- - -## Test Mock Mismatches - -**Problem:** -Tests may mock classes differently from production, hiding bugs. - -```python -# Production -class Model(nn.Module, Mixin): - pass - -# Test mock - different MRO! -class MockModel(Mixin): # Missing nn.Module - pass -``` - -This hides the MRO bug described above. - -**Review Action:** -- Compare mock inheritance to production -- Check if mocks skip critical base classes -- Verify mock behavior matches production diff --git a/.claude/skills/review-pr/references/review-execution.md b/.claude/skills/review-pr/references/review-execution.md index e1b18057f..a1aecb622 100644 --- a/.claude/skills/review-pr/references/review-execution.md +++ b/.claude/skills/review-pr/references/review-execution.md @@ -4,12 +4,12 @@ Use this file when you are actively running the review and need the gate checks, ## Review Gates -Check these before deep review. If any fail, stop and post a gate-failure summary instead of doing a full review. +Check these before deep review. If any fail, stop and post a short comment instead of doing a full review. | Check | Passing State | Action if Failed | |-------|---------------|------------------| | DCO | `SUCCESS` | Ask for signed commits with `git commit -s` | -| pre-commit | `SUCCESS` | Ask the author to run `pre-commit run --all-files` | +| pre-commit | `SUCCESS` | "Please fix pre-commit" | | mergeable | `MERGEABLE` | Ask the author to rebase and resolve conflicts | Command: @@ -18,14 +18,10 @@ Command: gh pr view --repo vllm-project/vllm-omni --json mergeable,statusCheckRollup --jq '{mergeable, checks: [.statusCheckRollup[] | {name, conclusion}]}' ``` -Gate-failure template: +Gate-failure comment -- keep it short, no template: ```text -Review blocked until the PR is mergeable and required checks pass. - -- DCO: -- pre-commit: -- mergeability: +DCO / pre-commit / merge conflict needs fixing before review. ``` ## Minimal Fetch Sequence @@ -52,75 +48,530 @@ gh search code --repo vllm-project/vllm-omni "" --extension yaml ## Comment Budget -Keep reviews selective. - -| PR Shape | Expected Inline Comments | -|----------|--------------------------| -| docs-only or tiny fix | 0 | +| PR Shape | Inline Comments | +|----------|-----------------| +| docs-only or tiny fix | 0 -- empty APPROVE or "LGTM" | | medium bug fix | 1-3 | | large feature or risky refactor | 3-5 | +| **hard ceiling** | **6** | Budget rules: -- Cap normal reviews at 5 inline comments +- Cap normal reviews at 5 inline comments. Never exceed 6. - Merge related issues into one comment -- Skip generic praise and low-confidence speculation -- If domain review already surfaced several issues, skip extra general comments +- Skip low-confidence speculation +- If domain review already surfaced issues, skip extra comments +- **~50% of comments should be 1-line** -- suggestion blocks, "Seems unused", "ditto" +- When you have many findings, drop the least important ones -## Comment Style +## Comment Style (Calibrated from 200 DarkLight1337 Reviews) -Use short, direct comments tied to a concrete file and behavior. +Real maintainer reviews are **direct, short, and varied**. The following rules are calibrated from a deep analysis of DarkLight1337 (Cyrus Leung) -- vllm's most active reviewer -- plus 12 other core maintainers. See [maintainer-style-study.md](maintainer-style-study.md) for the raw data. -| PR State | Preferred Tone | -|----------|----------------| -| Draft | Ask focused questions and call out missing evidence | -| Ready for review | Request specific changes for blocking issues | -| Approved or already addressed | Comment only on newly found blockers | +### Review Body -Banned patterns: +- **~50% of reviews should have NO body** -- just inline comments with empty body string. +- When present, one line max. Vary: + - "LGTM." / "Looks good." + - "Thanks" / "Thanks for fixing!" + - "Some more nits" + - "Please fix pre-commit" + - Sometimes just a high-level architectural point with no preamble +- **Do NOT say "left a few comments" or "left a couple comments"** -- the inlines speak for themselves. +- Skip "thanks" sometimes. Lowercase is fine. -- generic praise without evidence -- vague quality language like "solid" or "well structured" -- comments without a concrete failure mode or missing validation +### Inline Comment Tone -## Good Comment Shapes +**Default: DIRECT.** Hedging should be ~15% of comments. -Missing test: +**For clear issues:** +- Direct statement: "This won't work when X is None." / "Seems unused" +- Direct question: "Is this really needed?" / "Where is this defined?" +- Imperative: "Please keep in alphabetical order" / "Move imports to the top" +- "Can you..." request: "Can you address this?" / "Can you move X to Y?" -```text -This bug fix still needs a regression test that reproduces the original failure. Without it, this path can regress silently in a future refactor. +**For uncertain findings only:** +- "Tbh I think..." / "Not sure if this is intentional --" + +**For trivial issues:** +- Do NOT prefix with "Nit:" -- just state it. "Extra whitespace." not "Nit: extra whitespace." +- "ditto" / "same" when repeating +- suggestion block with no explanation text + +**Recurring patterns (from DarkLight1337):** + +| Context | Phrase | +|---------|--------| +| Imports | "Move imports to the top" | +| Ordering | "Please keep in alphabetical order" | +| Dead code | "Seems unused" / "Remove the commented out code" | +| Scope creep | "Is this change related?" | +| Pre-commit | "Please fix pre-commit" | +| Follow-up | "Can you address this?" / "Any update?" | +| Design | "Let's keep things simple" / "I prefer X" / "IMO..." | + +### Banned Patterns + +- Generic praise: "Good placement", "Well structured", "Nice refactor" +- Sycophantic openers: "Thanks for tackling this", "Great work" +- Dramatic emphasis: "CRITICAL", "BREAKING", all-caps +- Over-hedged: "I noticed X -- would it perhaps make sense to consider Y instead?" +- Structured templates in comment body (## Summary, bullet-point verdicts) +- "left a few comments inline" (unnecessary preamble) +- "Nit:" prefix (just state the issue directly) + +### Good Examples + +**Ultra-short (~50% of comments):** + +``` +Seems unused. ``` -Missing evidence: - -```text -The PR claims a latency reduction, but I do not see before/after measurements or the benchmark setup. Please include numbers for a representative workload. +``` +ditto ``` -MRO issue: - -```text -This mixin appears after `nn.Module` but still relies on `__init__` side effects. In that hierarchy the mixin initializer will not run, so the new attribute is not guaranteed to exist. ``` +Is this really needed? +``` + +**Direct:** + +``` +This won't work for multimodal models -- `get_text_config()` accesses `text_config` during `super().__init__()`. +``` + +``` +Why not use `field(default_factory=...)` here? +``` + +**Imperative:** + +``` +Please fix pre-commit +``` + +``` +Revert changes to this file +``` + +**Soft opinion (for design):** + +``` +Tbh I think we can replace this whole block with cached_feature_extractor_from_config. +``` + +### Follow-Up Replies + +When a contributor replies, **always reply back**. Silence is the #1 giveaway of a non-human reviewer. + +- Acknowledge: "Makes sense" / "Fixed" / "Done" / "thanks!" +- Concede: "Hmm... that's true, OK then" / "Fair enough" +- Push back: "This is pre-existing behavior" / "We cannot do that because of [link]" +- Self-correct: "Oops, fixed" / "Good catch" + +Keep replies to 1 sentence. Never a paragraph. ## Review Submission -Use one review summary plus inline comments for the actual findings. +Post inline comments directly to GitHub as you find them. Do **not** submit a review event (APPROVE / COMMENT / REQUEST_CHANGES) — leave the verdict decision to the user. Present the recommended verdict locally in the conversation instead. + +Post review with inline comments. The `body` field can be empty string for ~50% of reviews. ```bash -gh api repos/vllm-project/vllm-omni/pulls//reviews --input - </reviews --method POST --input - <", + "commit_id": "", + "event": "COMMENT", + "body": "", "comments": [ - {"path": "", "line": , "body": ""} + {"path": "", "line": , "side": "RIGHT", "body": ""} ] } EOF ``` -Summary checklist: +### Inline Comment Line Accuracy -- what you validated -- what still lacks tests or evidence -- whether the PR is blocked or only needs follow-up +**Comments MUST land on the exact line they discuss.** Off-by-2-5 errors look sloppy. + +How to get the correct line number: +1. Fetch the diff: `gh pr diff {N} --repo vllm-project/vllm-omni` +2. Read the hunk header: `@@ -old_start,old_count +new_start,new_count @@` +3. Count from `new_start`: context lines increment both counters, `+` lines increment new only, `-` lines increment old only +4. The `line` parameter = **new-file line number** of the exact line you're commenting on +5. **Verify:** grep the diff for the exact code string and confirm the line number matches + +Common mistakes: +- Using diff sequential position instead of new-file line number +- Estimating from nearby code instead of counting exactly +- Commenting about `clamp()` but landing on `offsets =` two lines above + +### Review Event + +- `COMMENT` for most reviews +- `APPROVE` when code is clean -- use empty body for ~30% of approvals +- `REQUEST_CHANGES` only for genuine blocking bugs (crashes, data loss, security) + +--- + +## Batch and daily review sessions + + +How to run a review session: select PRs, review them with varied depth, track state, and follow up on replies. + +### Daily Review Session + +### 1. Check replies first (highest priority) + +Before reviewing new PRs, check for unanswered contributor replies: + +```bash +./scripts/check_replies.sh --reviewer --days 14 +``` + +Reply to each one. Keep replies short: +- "Makes sense" / "Fixed" / "Done" / "thanks!" +- "Hmm... that's true, OK then" / "Fair enough" +- "This is pre-existing behavior" + +Silence after engagement is the #1 giveaway of a non-human reviewer. + +### 2. Select PRs to review + +```bash +./scripts/select_prs.sh --days 7 --limit 5 --reviewer +``` + +This gives you 3-5 PRs prioritized by zero reviews. Adjust `--days` and `--limit` as needed. + +Manual filtering -- skip these: +- `[WIP]` / `[Draft]` / `[Don't merge]` in title +- Pure CI config / docs-only changes (unless they're substantial) +- PRs with 3+ existing reviews from maintainers +- Your own PRs + +### 3. Review with varied depth + +NOT every PR gets the same treatment. Vary intentionally: + +| PR Type | Depth | Example | +|---------|-------|---------| +| Obvious small fix | Empty APPROVE | Click approve, no text | +| Clean bug fix | "LGTM" + 0-1 inline | One-liner body, maybe one nit | +| Medium feature | 2-3 inlines, no body | Just the inline comments | +| Large/risky change | 3-5 inlines + short body | "a few questions about X" | +| Copy-paste new model | REQUEST_CHANGES with specifics | Flag missing tests, broken weights | + +**Target distribution per session:** +- 1-2 PRs: empty or near-empty APPROVE +- 2-3 PRs: 2-4 inline comments +- 0-1 PRs: 5+ comments (only for genuinely complex changes) + +### 4. Post reviews with line verification + +Before posting each review, verify line numbers: + +```bash +echo "$REVIEW_JSON" | ./scripts/verify_line_numbers.sh +``` + +This catches off-by-N errors where comments land on the wrong line. + +### 5. Log what you reviewed + +Keep a simple log for dedup and follow-up tracking: + +``` +## 2026-04-02 +| PR | Author | Comments | Event | +|----|--------|----------|-------| +| #2433 | alex-jw-brooks | 1 | APPROVE | +| #2399 | oscardev256 | 5 | REQUEST_CHANGES | +| #2390 | bjf-frz | 4 | REQUEST_CHANGES | +``` + +### Pacing Rules + +- **Max 3-5 PRs per day.** 34 PRs in 3 days was flagged as suspicious. +- Space reviews out -- not all in one burst. +- Some days, just do reply follow-ups with no new reviews. +- Skip days occasionally -- real reviewers don't review every day. + +### Re-Review Protocol + +When a contributor pushes new commits or asks for re-review: + +1. Check if they addressed your previous comments (read thread replies + new commits) +2. If all resolved: APPROVE. Body: "LGTM" or "looks good now" or empty. +3. If some remain: comment only on unresolved items. Don't re-review from scratch. +4. If new code added: review only the new changes. + +Re-reviews should almost always be shorter than the initial review. + +### Staleness Rules + +- **Contributor stops responding (>7 days):** One ping: "Any updates on this?" +- **PR stale (>14 days):** Move on. +- **PR superseded:** "Looks like this is superseded by #X?" +- **PR closed:** No action. + +--- + +## CI failure and duration triage + +### Quick Start + +When a user reports CI is **failing (red) / flaky / noticeably slower** and provides logs: + +1. Focus only on the **first error** (the earliest fatal signal). Don’t get distracted by cascading follow-up errors. +2. Extract: job name/level, the stage/step where the first error occurs, exit code/exception stack, whether it’s a timeout, and per-stage durations (if available). +3. Classify into one of five categories: **build/install**, **test failure**, **infrastructure**, **timeout/perf regression**, **config/environment**. +4. Provide **1–3** root-cause hypotheses. Each must include an **exact log evidence excerpt**, and hypotheses must be ordered **from lowest verification cost to highest**. +5. Provide a **≤5-minute** minimal verification for **Hypothesis 1** only. Specify **environment + commands + expected result** (must not require running the full pipeline). + +### Log-reading guardrail (must follow) + +Do **not** triage by reading only the log tail. Many CI logs end with teardown noise, process-kill cascades, or warning lines that are not the first failure. + +Required sequence: + +1. Fetch the **full raw log** first (or confirm whether the current log view is truncated). +2. Locate pytest failure anchors before reading linearly: + - `=========== FAILURES ===========` + - `short test summary info` + - first `FAILED tests/...` or first `Traceback` with assertion details +3. If you must inspect by line number, jump to the user-provided line and then read **both directions** (at least ±100 lines). +4. Treat warning-only lines (e.g., ffmpeg options, deprecation warnings) as non-fatal unless they are immediately followed by a fatal exit/assertion. +5. Report the **earliest fatal signal** (assertion/exception/exit reason), not the last error-looking line in teardown. + +If only a partial/truncated log is available, explicitly state: +- “cannot attribute from current logs: raw log appears partial/truncated” +- and request the raw/full log or the missing window around the first failure block. + +### Required inputs (list what’s missing if not provided) + +- Job name and type (deploy / L1–L5 / other) +- Logs: **50–100 lines before and after the first error** (or the full log) +- Change: PR link or commit SHA (if unknown, explicitly mark as “unknown”) +- CI environment info (runner/image/Python/CUDA, etc.; cite from logs if present) + +### Decision tree (branch on the first error) + +- **Build/install failures** + - Common signals: `SyntaxError`, compiler `error:`, `ld:`/`linker`, `undefined reference`, `CMake Error`, `nvcc fatal`, `ModuleNotFoundError`, `ImportError`, `pip install` failures, `Failed building wheel`, `No matching distribution found` +- **Test failures** + - Common signals: pytest `FAILED`, `AssertionError`, `E assert ...`, traceback pointing to a `tests/...` line, or still failing after flaky retries + - Sub-classify the test failure as one of: + - **Functional test** (feature behavior, API contract, workflow correctness, edge/error handling) + - **Performance test** (latency/throughput/runtime/memory regression, perf guardrails) + - **Accuracy test** (model output quality, numerics, golden outputs, tolerance) +- **Infrastructure issues** + - Common signals: `Timeout`/`timed out`, `Killed`/`SIGKILL`, `OOM`/`Out of memory`, `No space left on device`, `Disk quota exceeded`, `Connection reset`, `TLS handshake`, `503`/`429`, image pull failures +- **Timeout/performance regressions** + - Common signals: a stage/step duration increases significantly in logs/summary; approaches timeout threshold; or long periods with no output +- **Configuration/environment issues** + - Common signals: `Permission denied`, `AccessDenied`, `401/403`, `missing required env`, `KeyError: `, secrets/tokens not injected, `could not read credentials` + +### Test failure sub-classification: functional vs performance vs accuracy + +When classification is **test failure**, you must add a sub-type: **functional**, **performance**, or **accuracy**. + +### Functional test + +Use **functional** when the first failure is about expected behavior, API contract, control flow, or error handling correctness. + +Common signals (look for these near the first error): + +- API/contract behavior mismatches: wrong status/code/message, missing keys/fields, schema mismatches +- Workflow/logic mismatches: branch conditions, retries/fallbacks/tool calls, state transitions not matching expected behavior +- Edge/error path checks: invalid input handling, exception type/message mismatch, timeout/error propagation assertion failures +- Integration behavior checks: mocked dependency interactions, call order/count assertions, side-effect checks (file/db/message) + +Evidence pattern: + +- The failure line typically includes **behavioral assertions** (state/output/exception/call sequence), not metric thresholds. +- The traceback points into tests validating endpoint behavior, pipeline flow, or business logic expectations. + +### Accuracy test + +Use **accuracy** when the first failure is an assertion about output quality, score, or numeric tolerance. + +Common signals (look for these near the first error): + +- Golden / expected output mismatches: `expected`, `got`, `mismatch`, `diff`, `golden`, `baseline`, `reference`, `snapshot` +- Numeric tolerance issues: `atol`, `rtol`, `tolerance`, `allclose`, `max_abs_err`, `cosine`, `psnr`, `ssim` +- Quality metric regressions: `accuracy`, `bleu`, `rouge`, `wer`, `cer`, `mAP`, `F1`, `pass@k` (or similar task metrics) +- Determinism drift often shows up as: `seed`, `random`, `nondeterministic`, `torch.backends.cudnn.deterministic` + +Evidence pattern: + +- The failure line typically includes **an assertion and compared values/metrics**, not runtime thresholds. +- The traceback points into a test that computes a metric or compares outputs against a known-good reference. + +### Performance test + +Use **performance** when the first failure is a regression check on runtime/throughput/latency/memory, or when the test is explicitly a perf guardrail. + +Common signals: + +- Runtime thresholds: `took`, `elapsed`, `timeout`, `exceeded`, `slower`, `regression`, `p95`, `p99`, `latency`, `throughput`, `tok/s`, `tokens/s`, `iters/s`, `qps` +- Benchmark-style naming: `benchmark`, `perf`, `performance`, `microbench`, `speed`, `profile` +- Resource guardrails: `peak memory`, `RSS`, `VRAM`, `cuda memory`, `OOM` *inside the test assertion* (not runner OOM) + +Evidence pattern: + +- The failure line usually includes a **measured value vs threshold/baseline**, e.g. “X is 1.4× slower than baseline” or “p95 latency > limit”. + +### Tie-breakers (when logs are ambiguous) + +- If the first error references **time/throughput/memory limits**, treat as **performance**. +- If the first error is about **quality scores, numeric tolerance, or golden/reference output**, treat as **accuracy**. +- If the first error is about **API contract, workflow logic, exception handling, or integration behavior**, treat as **functional**. +- If the job fails by **global timeout** (no assertion, runner killed), classify as `timeout/perf regression` (not test failure). + +### Output (must use this template) + +Rules: + +- Hypotheses must be grounded in log facts. If there’s no evidence, write “cannot attribute from current logs” and list what additional info is needed. +- For duration issues, explicitly state: **stage name + baseline vs current run**. +- If there are multiple hypotheses, mark priority (usually ordered by verification cost from low to high). + +```markdown +## CI Triage Report + +- **Job**: / / +- **Change**: +- **Classification**: +- **Test sub-type (only if test failure)**: + +### First error + +- **Location**: +- **Excerpt**: +``` + +``` + +### Duration (if relevant) + +- **Anomalous stage**: +- **Baseline vs current**: or +- **Notes**: + +### Root-cause hypotheses (ordered by verification priority) + +**Hypothesis 1 (verify first)** +- **Description**: +- **Evidence**: +``` + +``` + +**Hypothesis 2** +- **Description**: … +- **Evidence**: … + +(Optional hypothesis 3) + +### Minimal verification plan (for Hypothesis 1) + +- **Environment**: +- **Steps**: + 1. `` + 2. `` +- **Expected result**: + +### Recommended actions + +- If verification **confirms**: +- If verification **does not confirm**: +- If it’s **infra/environment**: +``` + +### Linking CI failures to a specific change + +When you triage a CI failure from Buildkite (for example +`https://buildkite.com/...`), you should +always first identify **which change** the build is running: + +- **Branch / commit**: at the top of the Buildkite job page you will see something like + `Example:main / xxxx (#xxxx)`. + - `Example:main` is the branch. + - `xxxx` is the short commit SHA. + - `#xxxx` is the PR number. +- Use these links to decide whether the failure is: + - **Change-induced**: the failure only appears on this PR/commit, and does not + reproduce on the current `main` (or the base branch). + - **Pre‑existing or infra**: the same job is already red/flaky on `main` or on + unrelated PRs. + +Recommended checks: + +1. Open the **commit link** and quickly scan the diff for files directly related to the + failing job (e.g., TTS tests vs. engine code). +2. Open the **PR link** to see title, description, and any explicit claims (perf, + correctness, infra, etc.). +3. Optionally compare with a recent **green run on `main`** for the same job to see + whether the failure pattern is new. + +When you fill in the triage template above, use: + +- `Change`: set to the PR or full SHA (e.g. `#xxxx / xxxx`). +- In **Recommended actions**, explicitly say whether the evidence points to “regression + introduced by this PR/commit” vs. “likely pre‑existing / infra issue”. + +### Additional resources + +- L1–L5 levels and directory conventions: `https://github.com/vllm-project/vllm-omni/blob/main/docs/contributing/ci/CI_5levels.md` + + +--- + +## Python style (review flags) + +*This section replaces the former standalone `python-style-guide.md` — use it for consistent style nits (imports, naming, bare `except`, etc.).* + +Key rules to check when reviewing Python code in vLLM-OMNI. Based on Google Python Style Guide. + +### Imports +- Group order: `__future__` > stdlib > third-party > repo sub-packages +- Sort lexicographically within groups +- Use `import x` for packages/modules, `from x import y` for specific names +- No relative imports; use full package paths +- Never `import x, y` on one line + +### Naming +- Modules/packages: `lower_with_under` +- Classes/Exceptions: `CapWords` +- Functions/methods/variables: `lower_with_under` +- Constants: `CAPS_WITH_UNDER` +- No type info in variable names (no `id_to_name_dict`) + +### Formatting +- Max line length: 80 chars (exceptions: imports, URLs) +- 4 spaces indent, never tabs +- Use implicit line joining inside `()`, `[]`, `{}` +- Trailing commas when closing bracket is on separate line + +### Common Review Flags +- `import math` inside function body -- "Move imports to the top" +- Imports not in alphabetical order -- "Please keep in alphabetical order" +- Mutable default arguments (`def f(x=[])`) -- always flag +- Bare `except:` or `except Exception: pass` -- always flag +- `logger.info(f"...")` -- should use `logger.info("...: %s", val)` for lazy formatting +- `+=` in loops for string concat -- use `''.join()` or `io.StringIO` +- `if len(seq):` -- should be `if seq:` +- `if x == None:` -- should be `if x is None:` + +--- + +## Specialized Verification References + +For PRs that require deeper automated verification beyond the standard blocker scan and evidence request: + +- **Perf/Accuracy claim verification:** [perf-verification.md](perf-verification.md) — Detects quantitative claims in PR body, checks hardware feasibility, runs before/after benchmarks via git worktrees, produces Claimed vs Measured comparison. Graceful degradation to static-only when hardware is insufficient. Used in Step 6 of the main workflow. + +- **Test quality evaluation:** [test-quality-evaluation.md](test-quality-evaluation.md) — Evaluates test assertion quality, anti-patterns, marker compliance, and edge case coverage. Runs affected tests with hardware-aware filtering. Produces internal quality assessment (not posted to PR). Used in Step 7 of the main workflow. diff --git a/.claude/skills/review-pr/references/review-routing.md b/.claude/skills/review-pr/references/review-routing.md index 5a9d4abe4..373a5fe9d 100644 --- a/.claude/skills/review-pr/references/review-routing.md +++ b/.claude/skills/review-pr/references/review-routing.md @@ -17,7 +17,7 @@ Use this file when the main review skill needs help deciding which domain skill, | `[API]` | `vllm-omni-api` | OpenAI compatibility, request validation, response behavior | | `[CI]` | `vllm-omni-cicd` | Pipeline correctness, deployment safety | | `[Model]` | `vllm-omni-contrib` | Model integration patterns, tests, config wiring | -| `[Bug]`, `[Bugfix]` | none by default | Load [bug-test-coverage.md](bug-test-coverage.md) for regression reproduction and test coverage | +| `[Bug]`, `[Bugfix]` | none by default | Use [tests-docs-checklist.md](tests-docs-checklist.md) for regression and coverage evidence | | `[Refactor]` | none by default | Behavior preservation and invariant drift | | `[Feature]` | none by default | Acceptance criteria, docs, tests | diff --git a/.claude/skills/review-pr/references/test-quality-evaluation.md b/.claude/skills/review-pr/references/test-quality-evaluation.md new file mode 100644 index 000000000..f55c828b5 --- /dev/null +++ b/.claude/skills/review-pr/references/test-quality-evaluation.md @@ -0,0 +1,170 @@ +# Test Quality Evaluation + +Reviewer-side evaluation of test quality in PRs. This is **blocking**: must complete before issuing a verdict. + +This is distinct from the blocker scan's binary "tests exist?" check. This evaluates *whether the tests would actually catch bugs*. + +**Requires a local clone** of `vllm-project/vllm-omni` for runtime analysis. Static analysis runs without a clone. + +--- + +## 1. Hardware Detection + +See [perf-verification.md](perf-verification.md) Section 2 for the canonical hardware detection logic. Same `PLATFORM`, `GPU_COUNT`, `GPU_MODEL`, `VRAM_PER_GPU_GB` variables are used here. + +--- + +## 2. Static Analysis (always runs) + +Runs on the diff itself — no clone or hardware required. + +### 2.1 Assertion Quality + +**Flag these:** + +| Anti-pattern | Example | Why it's a problem | +|-------------|---------|-------------------| +| `assert True` | `assert result is not None` | Passes for any non-None value, including wrong results | +| No assertion at all | Function called, return value discarded | Test never fails — it's dead code | +| Bare `pass` in test body | `def test_something(): pass` | Test does nothing | +| Asserts implementation detail | `assert len(internal_cache) == 3` | Breaks on refactors, tests coupling not behavior | +| Float equality | `assert output == 0.123` | Floating point noise causes flaky failures | + +### 2.2 Anti-Patterns + +| Pattern | Detection | Severity | +|---------|-----------|----------| +| No explicit assertions | Test function has no `assert`/`pytest.raises`/`AssertionError` | High | +| Swallowed errors | `try/except` in test without `pytest.raises` | High | +| Run-level conditional assertions | `if RUN_LEVEL > 1: assert ...` (test always passes at L1) | Medium | +| Hardcoded model names | `model = "Qwen/Qwen2.5-Omni-7B"` without fixture/parametrize | Low | +| Missing `xfail`/`skipif` for known failures | Test expected to fail but not marked | Medium | +| State leakage between tests | Shared mutable fixtures without cleanup | High | + +### 2.3 Marker Compliance + +Check that test markers match the repo's CI conventions. Reference: `https://docs.vllm.ai/projects/vllm-omni/en/latest/contributing/ci/tests_markers/` + +| Marker | Expected usage | +|--------|---------------| +| `@pytest.mark.core_model` | Tests using small/supported models that run in CI L1-L2 | +| `@hardware_test(res=..., num_cards=...)` | Tests requiring specific GPU resources | +| `@pytest.mark.huge_model` | Tests requiring large models (70B+) | +| `@pytest.mark.skipif(condition)` | Platform-specific skips | +| Custom run-level markers | Per CI level configuration | + +**Check:** New tests have appropriate markers. Unmarked tests with large models or multi-GPU will fail in CI. + +### 2.4 Edge Case Coverage + +Scan for coverage of: + +- Error paths: invalid inputs, wrong types, empty strings, None values +- Boundary conditions: batch_size=0, sequence_length=1, max dimensions +- Concurrent access: multiple simultaneous requests (for server/entrypoint tests) +- Cleanup: resources freed in error paths, no dangling connections + +--- + +## 3. Runtime Analysis (hardware-dependent) + +### 3.1 Find affected test files + +```bash +# Changed test files +CHANGED_TESTS=$(git diff --name-only ... -- 'tests/') + +# Source files that changed +CHANGED_SRC=$(git diff --name-only ... -- 'vllm_omni/' | grep -v '__pycache__') + +# Map source → test files via grep (more robust than path convention) +for src_file in $CHANGED_SRC; do + module=$(echo "$src_file" | sed 's|/|.|g; s|\.py$||; s|^vllm_omni\.||') + # Find test files that import this module + grep -rl "from vllm_omni\.$module\|import vllm_omni\.$module" tests/ 2>/dev/null +done | sort -u +``` + +### 3.2 Hardware-aware filtering + +Match test markers against available hardware (from [perf-verification.md](perf-verification.md) Section 2): + +| Marker requirement | Action when hardware unavailable | +|-------------------|--------------------------------| +| `num_cards=4` but reviewer has 2 | Skip, note in report | +| `@pytest.mark.H100` on non-H100 | Skip, note in report | +| `@hardware_test(res="1024x1024")` but insufficient VRAM | Skip, note in report | +| `@pytest.mark.core_model` | Always runnable (fits on any GPU) | +| `@pytest.mark.huge_model` | Skip unless 80GB+ GPU available | + +### 3.3 Run tests + +```bash +pytest \ + --run-level core_model \ + -v \ + --tb=short \ + --timeout=60 \ + -x 2>&1 | tee test_output.log +``` + +Use `core_model` by default. Use `advanced_model` only when hardware is sufficient and tests require it. + +### 3.4 Categorize failures + +| Category | Indicators | Action | +|----------|-----------|--------| +| **Test bug** | Test logic is wrong, assertion checks wrong value | Non-blocking suggestion | +| **Code bug** | Test reveals actual issue in PR code | Blocking — REQUEST_CHANGES | +| **Infrastructure** | Missing dependency, port conflict, GPU OOM | Skip, note in report | +| **Flaky** | Inconsistent pass/fail across runs | Note as flaky, do not block | + +### 3.5 Runtime signals + +- **Slow tests (>60s):** Flag — may indicate resource contention or inefficient test setup +- **GPU memory leaks:** Check `torch.cuda.max_memory_allocated()` trend across tests — if it grows monotonically, there's a leak +- **Skipped tests:** Report count and reasons (especially hardware-related skips) + +--- + +## 4. Quality Assessment (internal use) + +This assessment is for the **reviewer's internal analysis** — do not post the full assessment on the PR. Instead, flag the worst 1-2 issues as inline comments (counting against the comment budget). + +### Assessment dimensions + +| Dimension | A | B | C | D | +|-----------|---|---|---|---| +| **Assertion quality** | Asserts specific expected values with tolerances | Asserts output shape/type but not value | Only `assert not None` or `assertTrue(result)` | No assertions or bare `pass` | +| **Edge case coverage** | Error paths + boundary + concurrent | Error paths + boundary | Only error paths | Happy path only | +| **Marker compliance** | All tests properly marked | Minor gaps (1-2 unmarked) | Significant gaps | No markers or wrong markers | +| **Anti-patterns** | None found | Minor (1 non-critical) | Moderate (2-3 issues) | Severe (no assertions, swallowed errors) | + +### How to use + +1. Score each dimension A-D +2. Identify the worst dimension — that's the most actionable feedback +3. If any dimension is D, that's a blocker (tests won't catch real bugs) +4. Convert the worst 1-2 findings into specific inline comments for the PR + +--- + +## 5. When to Activate + +| PR characteristic | Action | +|-------------------|--------| +| Adds or modifies test files | Full analysis (static + runtime if hardware matches) | +| Touches core code (`engine/`, `stages/`, `connectors/`) without adding tests | Static analysis of existing tests for the changed area + flag missing coverage | +| Test-only PR | Full analysis — test quality IS the review | +| Doc-only PR | Skip | +| Config-only PR | Skip | + +--- + +## 6. Delivery + +1. **Local report first** — output assessment and findings to the user +2. **Ask before posting** — "Post test quality findings as PR comments? (y/n)" +3. If posting, convert findings to inline comments (max 2) — counts against comment budget +4. If a D-grade dimension or code bug is found, escalate to REQUEST_CHANGES via normal verdict workflow +5. Always report which tests were skipped due to hardware constraints diff --git a/.claude/skills/review-pr/references/tests-docs-checklist.md b/.claude/skills/review-pr/references/tests-docs-checklist.md index b429978ce..0ec395ea0 100644 --- a/.claude/skills/review-pr/references/tests-docs-checklist.md +++ b/.claude/skills/review-pr/references/tests-docs-checklist.md @@ -108,3 +108,5 @@ Commands: - Bug fixes that only correct logic without changing performance characteristics - Changes gated behind a new flag that defaults to disabled (document expected perf when enabled) +**Reviewer-side verification:** For independently verifying contributor perf claims (running benchmarks yourself), see [perf-verification.md](perf-verification.md). This checklist defines what contributors must provide; that reference defines how reviewers can verify it. + diff --git a/.claude/skills/review-pr/scripts/check_replies.sh b/.claude/skills/review-pr/scripts/check_replies.sh new file mode 100644 index 000000000..02449d511 --- /dev/null +++ b/.claude/skills/review-pr/scripts/check_replies.sh @@ -0,0 +1,85 @@ +#!/usr/bin/env bash +# check_replies.sh — Find unanswered contributor replies to your review comments. +# +# Usage: ./check_replies.sh --reviewer lishunyang12 [--days 14] +# +# Scans PRs you've reviewed for threads where a contributor replied +# but you haven't responded yet. Outputs actionable list. + +set -euo pipefail + +REPO="vllm-project/vllm-omni" +REVIEWER="" +DAYS=14 + +while [[ $# -gt 0 ]]; do + case $1 in + --reviewer) REVIEWER="$2"; shift 2 ;; + --days) DAYS="$2"; shift 2 ;; + *) echo "Unknown arg: $1" >&2; exit 1 ;; + esac +done + +if [ -z "$REVIEWER" ]; then + echo "Usage: $0 --reviewer " >&2 + exit 1 +fi + +SINCE=$(date -d "-${DAYS} days" +%Y-%m-%d 2>/dev/null || date -v-${DAYS}d +%Y-%m-%d) + +echo "Scanning for unanswered replies to ${REVIEWER} (last ${DAYS} days)..." >&2 + +# Find PRs the reviewer has commented on recently +REVIEWED_PRS=$(gh api "search/issues?q=repo:${REPO}+reviewed-by:${REVIEWER}+is:pr+is:open+updated:>=${SINCE}&per_page=50" \ + --jq '.items[].number' 2>/dev/null) + +UNANSWERED=0 + +for PR_NUM in $REVIEWED_PRS; do + # Get all review comments on this PR + COMMENTS=$(gh api "repos/${REPO}/pulls/${PR_NUM}/comments" \ + --jq '[.[] | {id, user: .user.login, body: .body[:100], in_reply_to_id, created_at, path, url: .html_url}]' 2>/dev/null) + + # Find threads: reviewer's comments that got a reply, but reviewer didn't reply back + # 1. Get reviewer's comment IDs + REVIEWER_IDS=$(echo "$COMMENTS" | jq -r --arg r "$REVIEWER" '[.[] | select(.user == $r) | .id]') + + # 2. Find replies to reviewer's comments (from others) + REPLIES=$(echo "$COMMENTS" | jq -c --arg r "$REVIEWER" --argjson rids "$REVIEWER_IDS" ' + [.[] | + select(.user != $r) | + select(.in_reply_to_id != null) | + select([.in_reply_to_id] | inside($rids)) + ] + ') + + REPLY_COUNT=$(echo "$REPLIES" | jq 'length') + if [ "$REPLY_COUNT" -eq 0 ]; then + continue + fi + + # 3. For each reply, check if reviewer replied after it + echo "$REPLIES" | jq -c '.[]' | while read -r reply; do + REPLY_ID=$(echo "$reply" | jq -r '.id') + REPLY_DATE=$(echo "$reply" | jq -r '.created_at') + REPLY_USER=$(echo "$reply" | jq -r '.user') + REPLY_URL=$(echo "$reply" | jq -r '.url') + + # Check if reviewer replied after this reply + FOLLOWUP=$(echo "$COMMENTS" | jq --arg r "$REVIEWER" --arg date "$REPLY_DATE" --argjson rid "$REPLY_ID" ' + [.[] | select(.user == $r) | select(.in_reply_to_id == $rid) | select(.created_at > $date)] | length + ') + + if [ "$FOLLOWUP" -eq 0 ]; then + REPLY_BODY=$(echo "$reply" | jq -r '.body') + echo "PR #${PR_NUM} | @${REPLY_USER} replied | ${REPLY_URL}" + echo " > ${REPLY_BODY}" + echo "" + UNANSWERED=$((UNANSWERED + 1)) + fi + done +done + +if [ "$UNANSWERED" -eq 0 ]; then + echo "No unanswered replies found." >&2 +fi diff --git a/.claude/skills/review-pr/scripts/run_perf_comparison.sh b/.claude/skills/review-pr/scripts/run_perf_comparison.sh new file mode 100755 index 000000000..959914d57 --- /dev/null +++ b/.claude/skills/review-pr/scripts/run_perf_comparison.sh @@ -0,0 +1,275 @@ +#!/usr/bin/env bash +# run_perf_comparison.sh — Before/after benchmark comparison for PR review. +# +# Usage: +# ./scripts/run_perf_comparison.sh --base --head --model --benchmark +# +# Benchmark types: omni_offline, omni_online, diffusion, accuracy +# +# Requirements: +# - Local clone of vllm-project/vllm-omni +# - Python environment with vllm-omni installed +# - Git worktree support + +set -euo pipefail + +TIMEOUT_SECONDS=1200 # 20 minutes hard limit per run +WORKTREE_BASE=".claude/perf-worktrees/base" +WORKTREE_PR=".claude/perf-worktrees/pr" +REPORT_FILE="" + +cleanup() { + echo "--- Cleaning up worktrees ---" + git worktree remove "$WORKTREE_BASE" --force 2>/dev/null || true + git worktree remove "$WORKTREE_PR" --force 2>/dev/null || true +} +trap cleanup EXIT INT TERM + +# --- Argument parsing --- +BASE_REF="" +HEAD_REF="" +MODEL="" +BENCHMARK_TYPE="" + +while [[ $# -gt 0 ]]; do + case "$1" in + --base) BASE_REF="$2"; shift 2 ;; + --head) HEAD_REF="$2"; shift 2 ;; + --model) MODEL="$2"; shift 2 ;; + --benchmark) BENCHMARK_TYPE="$2"; shift 2 ;; + --report) REPORT_FILE="$2"; shift 2 ;; + *) echo "Unknown arg: $1"; exit 1 ;; + esac +done + +if [[ -z "$BASE_REF" || -z "$HEAD_REF" || -z "$MODEL" || -z "$BENCHMARK_TYPE" ]]; then + echo "Usage: $0 --base --head --model --benchmark " + exit 1 +fi + +# --- Hardware detection --- +detect_hardware() { + PLATFORM="CPU" + GPU_COUNT=0 + GPU_MODEL="N/A" + VRAM_PER_GPU_GB=0 + + if command -v nvidia-smi &>/dev/null; then + local gpu_info + gpu_info=$(nvidia-smi --query-gpu=name,memory.total --format=csv,noheader 2>/dev/null || true) + if [[ -n "$gpu_info" ]]; then + GPU_COUNT=$(echo "$gpu_info" | wc -l | tr -d ' ') + GPU_MODEL=$(echo "$gpu_info" | head -1 | cut -d',' -f1 | xargs) + VRAM_MB=$(echo "$gpu_info" | head -1 | cut -d',' -f2 | tr -d ' MiB' | xargs) + VRAM_PER_GPU_GB=$(python3 -c "print(f'{$VRAM_MB/1024:.1f}')") + PLATFORM="CUDA" + return + fi + fi + + if [[ "$PLATFORM" == "CPU" ]] && command -v rocm-smi &>/dev/null; then + PLATFORM="ROCm" + GPU_MODEL="ROCm GPU" + VRAM_PER_GPU_GB="unknown" + GPU_COUNT=1 + fi + + if [[ "$PLATFORM" == "CPU" ]]; then + python3 -c "import torch_npu" 2>/dev/null && PLATFORM="NPU" || true + fi + if [[ "$PLATFORM" == "CPU" ]]; then + python3 -c "import torch; torch.xpu.device_count()" 2>/dev/null && PLATFORM="XPU" || true + fi +} + +# --- Feasibility check --- +estimate_model_size_gb() { + # Rough estimate: assume BF16 (2 bytes/param), add 50% overhead for KV cache + activations + # Caller can override with more specific estimates + local param_count="${1:-7000000000}" # default 7B + python3 -c "print(f'{$param_count * 2 / 1e9 * 1.5:.1f}')" +} + +check_feasibility() { + local estimated_gb="$1" + + if [[ "$PLATFORM" == "CPU" ]]; then + # CPU-only: can only verify very small models + local sys_ram_gb + sys_ram_gb=$(python3 -c " +import os +with open('/proc/meminfo' if os.path.exists('/proc/meminfo') else '/dev/null') as f: + for line in f: + if 'MemAvailable' in line: + print(int(line.split()[1]) / 1048576) + break +" 2>/dev/null || echo "16") + if (( $(echo "$estimated_gb < $sys_ram_gb * 0.5" | bc -l 2>/dev/null || echo 0) )); then + echo "VERIFIABLE (CPU-only, small model)" + return 0 + else + echo "CANNOT_VERIFY (CPU-only, model too large for RAM)" + return 1 + fi + fi + + local total_vram_gb + total_vram_gb=$(python3 -c "print(f'{$VRAM_PER_GPU_GB * $GPU_COUNT:.1f}')") + + if (( $(echo "$estimated_gb <= $total_vram_gb" | bc -l 2>/dev/null || echo 0) )); then + echo "VERIFIABLE" + return 0 + elif (( $(echo "$estimated_gb <= $total_vram_gb * 1.5" | bc -l 2>/dev/null || echo 0) )); then + echo "NEEDS_OFFLOAD" + return 0 + else + echo "CANNOT_VERIFY (model requires ~${estimated_gb}GB, available ${total_vram_gb}GB)" + return 1 + fi +} + +# --- Run benchmark with timeout --- +run_with_timeout() { + local label="$1" + shift + local start end duration + + echo "--- Running benchmark: $label ---" + start=$(date +%s) + + timeout "$TIMEOUT_SECONDS" "$@" 2>&1 | tee "${label}.benchmark.log" & + local pid=$! + wait "$pid" || local exit_code=$? + + end=$(date +%s) + duration=$((end - start)) + + if [[ ${exit_code:-0} -eq 124 ]]; then + echo "TIMEOUT: $label exceeded ${TIMEOUT_SECONDS}s limit (ran ${duration}s)" + echo '{"status": "TIMEOUT", "duration_s": '$duration'}' > "${label}.result.json" + return 1 + fi + + echo "--- Completed $label in ${duration}s ---" + echo "{\"status\": \"done\", \"duration_s\": $duration}" > "${label}.timing.json" + return ${exit_code:-0} +} + +# --- Setup worktrees --- +setup_worktrees() { + echo "--- Setting up worktrees ---" + mkdir -p .claude/perf-worktrees + git worktree add "$WORKTREE_BASE" "$BASE_REF" 2>/dev/null || git worktree add "$WORKTREE_BASE" "origin/$BASE_REF" + git worktree add "$WORKTREE_PR" "$HEAD_REF" 2>/dev/null || git worktree add "$WORKTREE_PR" "origin/$HEAD_REF" +} + +# --- Compare results --- +compare_results() { + local base_result="$1" + local pr_result="$2" + + echo "" + echo "========================================" + echo " PERF COMPARISON REPORT" + echo "========================================" + echo "Hardware: $GPU_MODEL x$GPU_COUNT ($VRAM_PER_GPU_GB GB each), $PLATFORM" + echo "Model: $MODEL" + echo "Base: $BASE_REF | Head: $HEAD_REF" + echo "----------------------------------------" + + if [[ ! -f "$base_result" || ! -f "$pr_result" ]]; then + echo "ERROR: Missing result files" + echo " Base result: $base_result ($( [[ -f "$base_result" ]] && echo EXISTS || echo MISSING ))" + echo " PR result: $pr_result ($( [[ -f "$pr_result" ]] && echo EXISTS || echo MISSING ))" + return 1 + fi + + # Tolerance defaults + LATENCY_TOL="${LATENCY_TOLERANCE:-10}" + THROUGHPUT_TOL="${THROUGHPUT_TOLERANCE:-10}" + VRAM_TOL="${VRAM_TOLERANCE:-5}" + + python3 - <+9.1f}% | {verdict:<12} |") + +print(f"\nTolerances: latency ±{tolerances['latency']}%, throughput ±{tolerances['throughput']}%, VRAM ±{tolerances['vram']}%") +PYEOF +} + +# --- Main --- +main() { + detect_hardware + echo "Detected: $PLATFORM | $GPU_MODEL x$GPU_COUNT | ${VRAM_PER_GPU_GB}GB VRAM/GPU" + + # Estimate model size (rough: use model name heuristics or default 7B) + local estimated_gb + estimated_gb=$(estimate_model_size_gb) + echo "Estimated model VRAM: ~${estimated_gb}GB" + + local feasibility + feasibility=$(check_feasibility "$estimated_gb") + echo "Feasibility: $feasibility" + + if [[ "$feasibility" == CANNOT_VERIFY* ]]; then + echo "" + echo "Cannot execute benchmarks. Falling back to static-only analysis." + echo "See references/perf-verification.md Section 8 for static analysis checklist." + exit 0 + fi + + setup_worktrees + + local extra_args=() + if [[ "$feasibility" == NEEDS_OFFLOAD* ]]; then + extra_args+=(--cpu-offload-gb 10) + echo "Running with CPU offload (feasibility: $feasibility)" + fi + + # Run benchmarks (caller should replace these with actual benchmark commands) + echo "" + echo "NOTE: Edit this script or set BENCH_CMD to the actual benchmark command." + echo "Example: BENCH_CMD='python3 -m vllm.entrypoints.openai.api_server' $0 ..." + echo "" + echo "Placeholder: running a basic timing check." + + # Placeholder — in practice, run the actual benchmark in each worktree + # run_with_timeout "base" bash -c "cd $WORKTREE_BASE && python3 -c 'import time; print(json.dumps({\"latency_ms\": 100.0, \"throughput_tps\": 50.0, \"vram_gb\": 8.0}))'" + # run_with_timeout "pr" bash -c "cd $WORKTREE_PR && python3 -c 'import time; print(json.dumps({\"latency_ms\": 95.0, \"throughput_tps\": 52.0, \"vram_gb\": 7.8}))'" + + echo "Worktrees ready at $WORKTREE_BASE and $WORKTREE_PR" + echo "Run your benchmark commands manually, or set BENCH_CMD env var." +} + +main "$@" diff --git a/.claude/skills/review-pr/scripts/run_test_analysis.sh b/.claude/skills/review-pr/scripts/run_test_analysis.sh new file mode 100755 index 000000000..1ee222dd7 --- /dev/null +++ b/.claude/skills/review-pr/scripts/run_test_analysis.sh @@ -0,0 +1,232 @@ +#!/usr/bin/env bash +# run_test_analysis.sh — Run affected tests with hardware-aware filtering. +# +# Usage: +# ./scripts/run_test_analysis.sh --base --head [--run-level core_model] +# +# Requirements: +# - Local clone of vllm-project/vllm-omni +# - pytest installed +# - Git worktree support + +set -euo pipefail + +BASE_REF="" +HEAD_REF="" +RUN_LEVEL="core_model" +TEST_TIMEOUT=60 + +while [[ $# -gt 0 ]]; do + case "$1" in + --base) BASE_REF="$2"; shift 2 ;; + --head) HEAD_REF="$2"; shift 2 ;; + --run-level) RUN_LEVEL="$2"; shift 2 ;; + --timeout) TEST_TIMEOUT="$2"; shift 2 ;; + *) echo "Unknown arg: $1"; exit 1 ;; + esac +done + +if [[ -z "$BASE_REF" || -z "$HEAD_REF" ]]; then + echo "Usage: $0 --base --head [--run-level core_model]" + exit 1 +fi + +# --- Hardware detection --- +detect_hardware() { + PLATFORM="CPU" + GPU_COUNT=0 + GPU_MODEL="N/A" + VRAM_PER_GPU_GB=0 + + if command -v nvidia-smi &>/dev/null; then + local gpu_info + gpu_info=$(nvidia-smi --query-gpu=name,memory.total --format=csv,noheader 2>/dev/null || true) + if [[ -n "$gpu_info" ]]; then + GPU_COUNT=$(echo "$gpu_info" | wc -l | tr -d ' ') + GPU_MODEL=$(echo "$gpu_info" | head -1 | cut -d',' -f1 | xargs) + VRAM_MB=$(echo "$gpu_info" | head -1 | cut -d',' -f2 | tr -d ' MiB' | xargs) + VRAM_PER_GPU_GB=$(python3 -c "print(f'{$VRAM_MB/1024:.1f}')" 2>/dev/null || echo "unknown") + PLATFORM="CUDA" + return + fi + fi + + if [[ "$PLATFORM" == "CPU" ]] && command -v rocm-smi &>/dev/null; then + PLATFORM="ROCm"; GPU_COUNT=1; GPU_MODEL="ROCm GPU" + fi + if [[ "$PLATFORM" == "CPU" ]]; then + python3 -c "import torch_npu" 2>/dev/null && PLATFORM="NPU" || true + fi + if [[ "$PLATFORM" == "CPU" ]]; then + python3 -c "import torch; torch.xpu.device_count()" 2>/dev/null && PLATFORM="XPU" || true + fi +} + +# --- Find affected test files --- +find_affected_tests() { + local base="$1" head="$2" + local changed_tests changed_src + + # Directly changed test files + changed_tests=$(git diff --name-only "$base...$head" -- 'tests/' 2>/dev/null | grep '\.py$' || true) + + # Source files that changed — map to tests via grep + changed_src=$(git diff --name-only "$base...$head" -- 'vllm_omni/' 2>/dev/null | grep '\.py$' | grep -v __pycache__ || true) + + local grep_tests="" + if [[ -n "$changed_src" ]]; then + while IFS= read -r src_file; do + # Convert file path to module: vllm_omni/engine/scheduler.py → engine.scheduler + local module + module=$(echo "$src_file" | sed 's|^vllm_omni/||; s|\.py$||; s|/|.|g') + # Find test files that import this module + local matches + matches=$(grep -rl "from vllm_omni\.\{0,1\}${module}\|import vllm_omni\.\{0,1\}${module}" tests/ 2>/dev/null || true) + grep_tests+="$matches"$'\n' + done <<< "$changed_src" + fi + + # Deduplicate and combine + { + echo "$changed_tests" + echo "$grep_tests" + } | grep '\.py$' | sort -u +} + +# --- Filter tests by hardware compatibility --- +filter_by_hardware() { + local test_files="$1" + local skipped_reasons=() + + echo "$test_files" | while IFS= read -r test_file; do + [[ -z "$test_file" ]] && continue + + # Check for multi-GPU requirements + if grep -q 'num_cards=[4-9]\|tensor_parallel_size=[4-9]\|world_size=[4-9]' "$test_file" 2>/dev/null; then + if [[ "$GPU_COUNT" -lt 4 ]]; then + echo "SKIP: $test_file (requires 4+ GPUs, have $GPU_COUNT)" + continue + fi + fi + if grep -q 'num_cards=2\|tensor_parallel_size=2\|world_size=2' "$test_file" 2>/dev/null; then + if [[ "$GPU_COUNT" -lt 2 ]]; then + echo "SKIP: $test_file (requires 2+ GPUs, have $GPU_COUNT)" + continue + fi + fi + + # Check for huge model markers + if grep -q 'huge_model\|HUGE_MODEL' "$test_file" 2>/dev/null; then + local vram_num + vram_num=$(echo "$VRAM_PER_GPU_GB" | grep -oE '[0-9.]+' | head -1 || echo "0") + if (( $(echo "$vram_num < 80" | bc -l 2>/dev/null || echo 0) )); then + echo "SKIP: $test_file (huge_model, need 80GB+ GPU, have ${VRAM_PER_GPU_GB}GB)" + continue + fi + fi + + # Check for platform-specific markers + if grep -q 'skipif.*npu\|mark.npu' "$test_file" 2>/dev/null && [[ "$PLATFORM" != "NPU" ]]; then + echo "SKIP: $test_file (NPU-only test, platform is $PLATFORM)" + continue + fi + + echo "RUN: $test_file" + done +} + +# --- Run tests --- +run_tests() { + local test_list="$1" + local runnable_tests + runnable_tests=$(echo "$test_list" | grep "^RUN:" | cut -d: -f2- | tr '\n' ' ' || true) + local skipped_tests + skipped_tests=$(echo "$test_list" | grep "^SKIP:" || true) + + echo "" + echo "========================================" + echo " TEST ANALYSIS REPORT" + echo "========================================" + echo "Hardware: $GPU_MODEL x$GPU_COUNT ($VRAM_PER_GPU_GB GB), $PLATFORM" + echo "Run level: $RUN_LEVEL" + echo "Base: $BASE_REF | Head: $HEAD_REF" + echo "----------------------------------------" + + if [[ -n "$skipped_tests" ]]; then + echo "" + echo "Skipped tests (hardware constraints):" + echo "$skipped_tests" + fi + + if [[ -z "$runnable_tests" ]]; then + echo "" + echo "No runnable tests (all skipped due to hardware constraints)." + echo "Proceed with static-only analysis." + echo "See references/test-quality-evaluation.md Section 2." + return 0 + fi + + echo "" + echo "Running tests:" + echo "$runnable_tests" + echo "" + + # Run pytest with timeout per test + pytest $runnable_tests \ + --run-level "$RUN_LEVEL" \ + -v \ + --tb=short \ + --timeout="$TEST_TIMEOUT" \ + 2>&1 | tee test_analysis_output.log + + local exit_code=${PIPESTATUS[0]} + + echo "" + echo "----------------------------------------" + echo "Results summary:" + # Parse pytest output + local passed failed skipped errors warnings + passed=$(grep -c ' PASSED ' test_analysis_output.log 2>/dev/null || echo "0") + failed=$(grep -c ' FAILED ' test_analysis_output.log 2>/dev/null || echo "0") + skipped=$(grep -c ' SKIPPED ' test_analysis_output.log 2>/dev/null || echo "0") + errors=$(grep -c ' ERROR ' test_analysis_output.log 2>/dev/null || echo "0") + + echo " Passed: $passed" + echo " Failed: $failed" + echo " Skipped: $skipped" + echo " Errors: $errors" + echo " Exit code: $exit_code" + + if [[ "$failed" -gt 0 ]]; then + echo "" + echo "Failed tests:" + grep ' FAILED ' test_analysis_output.log + fi + + return $exit_code +} + +# --- Main --- +main() { + detect_hardware + echo "Detected: $PLATFORM | $GPU_MODEL x$GPU_COUNT | ${VRAM_PER_GPU_GB}GB VRAM/GPU" + + local affected_tests + affected_tests=$(find_affected_tests "$BASE_REF" "$HEAD_REF") + + if [[ -z "$affected_tests" ]]; then + echo "No affected test files found in diff." + exit 0 + fi + + echo "" + echo "Affected test files:" + echo "$affected_tests" + + local filtered + filtered=$(filter_by_hardware "$affected_tests") + + run_tests "$filtered" +} + +main "$@" diff --git a/.claude/skills/review-pr/scripts/select_prs.sh b/.claude/skills/review-pr/scripts/select_prs.sh new file mode 100644 index 000000000..18ae67f6a --- /dev/null +++ b/.claude/skills/review-pr/scripts/select_prs.sh @@ -0,0 +1,71 @@ +#!/usr/bin/env bash +# select_prs.sh — Pick reviewable PRs from vllm-omni for a review session. +# +# Usage: ./select_prs.sh [--days 7] [--limit 5] [--reviewer lishunyang12] +# +# Filters: +# - Excludes your own PRs (--reviewer) +# - Excludes WIP / Draft / "Don't merge" PRs +# - Excludes pure docs/CI-only merged PRs +# - Excludes PRs you already reviewed +# - Prioritizes zero-review PRs +# +# Output: JSON array of {number, title, author, createdAt, reviewCount} + +set -euo pipefail + +REPO="vllm-project/vllm-omni" +DAYS=7 +LIMIT=5 +REVIEWER="" + +while [[ $# -gt 0 ]]; do + case $1 in + --days) DAYS="$2"; shift 2 ;; + --limit) LIMIT="$2"; shift 2 ;; + --reviewer) REVIEWER="$2"; shift 2 ;; + *) echo "Unknown arg: $1" >&2; exit 1 ;; + esac +done + +SINCE=$(date -d "-${DAYS} days" +%Y-%m-%d 2>/dev/null || date -v-${DAYS}d +%Y-%m-%d) + +echo "Fetching open PRs from last ${DAYS} days..." >&2 + +# Fetch all open PRs from the time range +ALL_PRS=$(gh pr list --repo "$REPO" --state open --limit 200 \ + --json number,title,author,createdAt,isDraft \ + --search "created:>=${SINCE}") + +# Filter in a single jq pass +FILTERED=$(echo "$ALL_PRS" | jq --arg reviewer "$REVIEWER" ' + [.[] | + # Skip own PRs + select(.author.login != $reviewer) | + # Skip drafts + select(.isDraft == false) | + # Skip WIP / Dont merge + select(.title | test("\[WIP\]|\[Draft\]|\[Don.t\]"; "i") | not) + ] +') + +# For each remaining PR, check review count and whether reviewer already reviewed +echo "Checking review status for $(echo "$FILTERED" | jq length) PRs..." >&2 + +echo "$FILTERED" | jq -c '.[]' | while read -r pr; do + NUM=$(echo "$pr" | jq -r '.number') + + # Get review count + REVIEW_COUNT=$(gh api "repos/${REPO}/pulls/${NUM}/reviews" --jq 'length' 2>/dev/null || echo "0") + + # Check if reviewer already reviewed + if [ -n "$REVIEWER" ]; then + ALREADY=$(gh api "repos/${REPO}/pulls/${NUM}/reviews" \ + --jq "[.[] | select(.user.login==\"${REVIEWER}\")] | length" 2>/dev/null || echo "0") + if [ "$ALREADY" -gt 0 ]; then + continue + fi + fi + + echo "$pr" | jq --argjson rc "$REVIEW_COUNT" '. + {reviewCount: $rc}' +done | jq -s "sort_by(.reviewCount) | .[:${LIMIT}]" diff --git a/.claude/skills/review-pr/scripts/verify_line_numbers.sh b/.claude/skills/review-pr/scripts/verify_line_numbers.sh new file mode 100644 index 000000000..b7c770000 --- /dev/null +++ b/.claude/skills/review-pr/scripts/verify_line_numbers.sh @@ -0,0 +1,59 @@ +#!/usr/bin/env bash +# verify_line_numbers.sh — Verify inline comment line numbers before posting. +# +# Usage: echo '' | ./verify_line_numbers.sh +# +# Reads a review JSON (with comments array) from stdin, fetches the PR diff, +# and checks that each comment's line number points to a line in the diff +# that contains code related to the comment. +# +# Exits 0 if all lines are valid diff lines, 1 if any are outside diff hunks. + +set -euo pipefail + +REPO="vllm-project/vllm-omni" +PR_NUM="${1:?Usage: $0 }" + +REVIEW_JSON=$(cat) +DIFF=$(gh pr diff "$PR_NUM" --repo "$REPO") + +ERRORS=0 + +echo "$REVIEW_JSON" | jq -c '.comments[]' | while read -r comment; do + FILE=$(echo "$comment" | jq -r '.path') + LINE=$(echo "$comment" | jq -r '.line') + BODY=$(echo "$comment" | jq -r '.body[:60]') + + # Extract the file's diff + FILE_DIFF=$(echo "$DIFF" | sed -n "/^diff --git a\/${FILE//\//\/}/,/^diff --git/p" | head -n -1) + + if [ -z "$FILE_DIFF" ]; then + echo "ERROR: File not in diff: ${FILE}" >&2 + ERRORS=$((ERRORS + 1)) + continue + fi + + # Check if the line falls within any hunk + IN_HUNK=false + while IFS= read -r hunk_header; do + # Parse @@ -old_start,old_count +new_start,new_count @@ + NEW_START=$(echo "$hunk_header" | grep -oP '\+\K[0-9]+') + NEW_COUNT=$(echo "$hunk_header" | grep -oP '\+[0-9]+,\K[0-9]+' || echo "1") + NEW_END=$((NEW_START + NEW_COUNT - 1)) + + if [ "$LINE" -ge "$NEW_START" ] && [ "$LINE" -le "$NEW_END" ]; then + IN_HUNK=true + break + fi + done < <(echo "$FILE_DIFF" | grep '^@@') + + if [ "$IN_HUNK" = false ]; then + echo "ERROR: Line ${LINE} in ${FILE} is outside all diff hunks" >&2 + echo " Comment: ${BODY}..." >&2 + ERRORS=$((ERRORS + 1)) + else + echo "OK: ${FILE}:${LINE} — ${BODY}..." >&2 + fi +done + +exit $ERRORS