Files
vllm-omni/.claude/skills/add-tts-model/references/cuda-graph-example.md
T

43 lines
1.7 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# CUDA Graph Example: Qwen3-TTS Code Predictor
Reference sketch for capturing the 16-step code-predictor AR loop as a single
CUDA graph. Adapt the shapes, number of steps, and KV-head layout to your
model.
```python
import torch
class CodePredictorGraph:
"""Captures the 16-step code predictor AR loop as a single CUDA graph."""
def setup_graph(self, device: torch.device, kv_heads: int = 4, head_dim: int = 64):
self.num_steps = 16
self.kv_cache = torch.zeros(1, kv_heads, self.num_steps, head_dim, device=device)
self.positions = torch.arange(self.num_steps, device=device)
self.causal_mask = torch.tril(torch.ones(self.num_steps, self.num_steps, device=device))
self.input_buf = torch.zeros(1, 1, kv_heads * head_dim, device=device)
self.output_buf = torch.zeros(1, self.num_steps, device=device, dtype=torch.long)
# Warm up, then: self.graph = torch.cuda.CUDAGraph(); self.graph.capture(...)
def run_graph(self, initial_input: torch.Tensor) -> torch.Tensor:
self.input_buf.copy_(initial_input)
self.graph.replay()
return self.output_buf.clone()
```
## Performance expectations (Qwen3-TTS code predictor)
- **3–5× speedup** on the graphed component.
- Effective only for fixed batch sizes (typically `batch_size=1`).
- Fall back to eager for any shape/config that wasn't captured — do not try
to recapture per request.
## Graph-safety constraints
- `torch.argmax` instead of `torch.multinomial`.
- Fixed batch size.
- No Python control flow that branches on tensor values inside the captured
region (use `torch.where` / masks).
- No `.item()`, `.cpu()`, `.tolist()` — each would break the capture or
cause a GPU→CPU sync during replay.