39 lines
1.4 KiB
Markdown
39 lines
1.4 KiB
Markdown
# Regional Compilation
|
|
|
|
Regional compilation applies `torch.compile` to the repeated transformer blocks
|
|
declared by a diffusion model. It is the default compilation scope when
|
|
diffusion inference runs without `--enforce-eager`.
|
|
|
|
## Configuration
|
|
|
|
Dynamic compilation is enabled by default so the compiled regions can handle
|
|
mixed resolutions. For a fixed-shape workload, disable it explicitly:
|
|
|
|
```bash
|
|
vllm serve <model> --omni --no-diffusion-compile-dynamic
|
|
```
|
|
|
|
The equivalent per-stage deploy configuration is:
|
|
|
|
```yaml
|
|
stages:
|
|
- stage_id: 0
|
|
diffusion_compile_granularity: regional
|
|
diffusion_compile_dynamic: false
|
|
```
|
|
|
|
For an experimental whole-transformer compile scope, set
|
|
`--diffusion-compile-granularity full` or use
|
|
`diffusion_compile_granularity: full` in the deploy configuration. Full scope may
|
|
still contain graph breaks; it does not force one graph. It is rejected when
|
|
HSDP, sequence parallelism, CPU offload, or layerwise offload is enabled. Use
|
|
regional scope with those features.
|
|
|
|
These settings control the generic model-runner compilation path. Pipelines
|
|
that provide their own `setup_compile()` implementation manage their compilation
|
|
policy independently. Compilation is lazy, so backend or graph errors can first
|
|
surface on the initial request.
|
|
|
|
Use `--enforce-eager` to disable the model runner's generic compile setup.
|
|
Pipelines that compile internally define their own eager-mode behavior.
|