feat: configure Torch Dynamo recompile limit from environment (#7243)
Signed-off-by: Xiaodong Ye <xiaodong.ye@mthreads.com>
This commit is contained in:
1 parent
d418004b6b
commit
6715c57b2d
6 files changed
+111
-1
No files matched your search
@@ -130,6 +130,22 @@ settings.
|
||||
Omni-owned variable, even though vLLM-Omni supplies a persistent default when
|
||||
it is unset.
|
||||
|
||||
### Torch compilation
|
||||
|
||||
| Name | Type and default | Applies to and read time | Precedence and invalid values | Lifecycle |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| `VLLM_OMNI_TORCH_DYNAMO_RECOMPILE_LIMIT` | Positive integer; unset preserves the current Torch setting | Process default for Torch Dynamo; read when `vllm_omni` is imported | Sets `torch._dynamo.config.recompile_limit` to the requested value. A non-positive or non-integer value raises `ValueError`. Later backend-specific overrides still take precedence. | Diagnostic |
|
||||
|
||||
Set this variable before launching Omni, for example:
|
||||
|
||||
```bash
|
||||
export VLLM_OMNI_TORCH_DYNAMO_RECOMPILE_LIMIT=64
|
||||
```
|
||||
|
||||
The setting applies across model families and platforms. vLLM compilation
|
||||
contexts and backends can subsequently apply their own limits; this variable
|
||||
does not replace those backend-specific policies.
|
||||
|
||||
## Per-stage environment
|
||||
|
||||
Deploy configurations can set arbitrary environment keys for one stage:
|
||||
|
||||
Reference in new issue
Block a user