wtz2333andhsliuustc0106
5a6a93dc5f
Signed-off-by: hsliuustc0106 <liuhongsheng4@huawei.com> Signed-off-by: wtz2333 <2955110911@qq.com> Co-authored-by: hsliuustc0106 <liuhongsheng4@huawei.com>
63 lines
2.4 KiB
Markdown
63 lines
2.4 KiB
Markdown
# LingBot-Video Benchmarks
|
|
|
|
These manual benchmarks compare the upstream LingBot-Video implementation with
|
|
vLLM-Omni. Run all commands from the repository root.
|
|
|
|
| Benchmark | Script | Purpose |
|
|
|-----------|--------|---------|
|
|
| Dense pipeline parity | `dense_pipeline_parity.py` | Compare generated videos and report MAE, MSE, PSNR, latency, and optional steady-state timings |
|
|
| MoE numerical parity | `moe_transformer_parity.py` | Check bitwise parity for the sparse MoE block and the full MoE transformer |
|
|
|
|
Both benchmarks require a local LingBot-Video checkout and local model files.
|
|
They do not run in CI.
|
|
|
|
## Dense pipeline parity
|
|
|
|
The dense benchmark launches the upstream Diffusers inference script and the
|
|
vLLM-Omni offline example with identical prompts and sampling parameters. It
|
|
then compares the decoded MP4 frames.
|
|
|
|
```bash
|
|
CUDA_VISIBLE_DEVICES=0 \
|
|
python benchmarks/lingbot_video/dense_pipeline_parity.py \
|
|
--model /path/to/lingbot-video-dense-1.3b \
|
|
--official-repo /path/to/lingbot-video \
|
|
--output-dir /tmp/lingbot_dense_parity
|
|
```
|
|
|
|
Use `--runs N` to additionally measure repeated in-process vLLM-Omni
|
|
requests. The video comparison reports MAE, MSE, and PSNR; it is an end-to-end
|
|
pipeline diagnostic rather than a bitwise transformer check.
|
|
|
|
## MoE numerical parity
|
|
|
|
The lightweight block comparison uses deterministic weights and inputs to
|
|
cover correction-bias routing, group-limited top-k, routed experts, FP32
|
|
scatter-weighted restore, padding masks, and the shared expert.
|
|
|
|
```bash
|
|
CUDA_VISIBLE_DEVICES=0 \
|
|
python benchmarks/lingbot_video/moe_transformer_parity.py \
|
|
--scope block \
|
|
--official-repo /path/to/lingbot-video \
|
|
--output-json /tmp/lingbot_moe_block_parity.json
|
|
```
|
|
|
|
The full comparison loads the upstream and vLLM-Omni 30B transformers
|
|
sequentially and compares their output tensors. It requires a CUDA device with
|
|
`torch._grouped_mm` and enough memory for one 30B BF16 transformer.
|
|
|
|
```bash
|
|
CUDA_VISIBLE_DEVICES=0 \
|
|
python benchmarks/lingbot_video/moe_transformer_parity.py \
|
|
--scope transformer \
|
|
--official-repo /path/to/lingbot-video \
|
|
--model /path/to/lingbot-video-moe-30b-a3b \
|
|
--output-json /tmp/lingbot_moe_transformer_parity.json
|
|
```
|
|
|
|
For the correctness oracle, the upstream transformer uses
|
|
`diffusers:_native_math` and vLLM-Omni uses
|
|
`TORCH_SDPA + SDPBackend.MATH`. The command exits with a nonzero status unless
|
|
the selected comparisons are bitwise equal.
|