[perf] Add configurable diffusion compile granularity (#4603)
Signed-off-by: TaffyOfficial <2324465096@qq.com> Co-authored-by: TaffyOfficial <2324465096@qq.com>
This commit is contained in:
1 parent
84bbd57e4d
commit
93786c9cd3
15 files changed
+359
-15
No files matched your search
@@ -0,0 +1,38 @@
|
||||
# Regional Compilation
|
||||
|
||||
Regional compilation applies `torch.compile` to the repeated transformer blocks
|
||||
declared by a diffusion model. It is the default compilation scope when
|
||||
diffusion inference runs without `--enforce-eager`.
|
||||
|
||||
## Configuration
|
||||
|
||||
Dynamic compilation is enabled by default so the compiled regions can handle
|
||||
mixed resolutions. For a fixed-shape workload, disable it explicitly:
|
||||
|
||||
```bash
|
||||
vllm serve <model> --omni --no-diffusion-compile-dynamic
|
||||
```
|
||||
|
||||
The equivalent per-stage deploy configuration is:
|
||||
|
||||
```yaml
|
||||
stages:
|
||||
- stage_id: 0
|
||||
diffusion_compile_granularity: regional
|
||||
diffusion_compile_dynamic: false
|
||||
```
|
||||
|
||||
For an experimental whole-transformer compile scope, set
|
||||
`--diffusion-compile-granularity full` or use
|
||||
`diffusion_compile_granularity: full` in the stage configuration. Full scope may
|
||||
still contain graph breaks; it does not force one graph. It is rejected when
|
||||
HSDP, sequence parallelism, CPU offload, or layerwise offload is enabled. Use
|
||||
regional scope with those features.
|
||||
|
||||
These settings control the generic model-runner compilation path. Pipelines
|
||||
that provide their own `setup_compile()` implementation manage their compilation
|
||||
policy independently. Compilation is lazy, so backend or graph errors can first
|
||||
surface on the initial request.
|
||||
|
||||
Use `--enforce-eager` to disable the model runner's generic compile setup.
|
||||
Pipelines that compile internally define their own eager-mode behavior.
|
||||
Reference in new issue
Block a user