[Misc] Add standalone safetensors retention diagnostic (#7145)
Signed-off-by: Hongsheng Liu <liuhongsheng4@huawei.com>
This commit is contained in:
1 parent
ce27ee6038
commit
f11bf1a05d
3 files changed
+182
No files matched your search
@@ -42,6 +42,11 @@ V1 does not include:
|
||||
|
||||
## Motivation and use cases
|
||||
|
||||
For investigating CPU memory retained by dependency tensor materialization,
|
||||
see the [standalone safetensors diagnostic](https://github.com/vllm-project/vllm-omni/blob/main/benchmarks/host_weight_runtime/README.md).
|
||||
It distinguishes repeated dependency calls from reuse of cached views and does
|
||||
not establish per-request HWR leakage.
|
||||
|
||||
Model loading can create the same final host representation repeatedly. This
|
||||
is especially expensive when loading performs checkpoint decoding, tensor
|
||||
renaming, TP slicing, quantization, packing, or scale construction before GPU
|
||||
|
||||
Reference in new issue
Block a user