66 lines
2.9 KiB
Markdown
66 lines
2.9 KiB
Markdown
|
# Qwen2.5-Omni embedding microbenchmark
|
||
|
|
|
||
|
|
Measure the host-observed latency of `embed_input_ids` before and after the
|
||
|
|
redundant text embedding removal in PR #7477. Both variants use the same real
|
||
|
|
checkpoint embedding table, dependencies, inputs and batch shapes. The baseline
|
||
|
|
wrapper is extracted from the specified Git commit; the fixed wrapper comes from
|
||
|
|
the current checkout. No full model is initialized.
|
||
|
|
|
||
|
|
## Run
|
||
|
|
|
||
|
|
Use Linux, a CUDA GPU, the project dependencies, and a local
|
||
|
|
`Qwen/Qwen2.5-Omni-7B` checkpoint with its safetensors index and weights.
|
||
|
|
From the repository root:
|
||
|
|
|
||
|
|
```bash
|
||
|
|
git fetch origin 01a2f93256975c7ff9565c5414b0fe0225bc4765
|
||
|
|
PYTHONPATH="$PWD" OMP_NUM_THREADS=1 python benchmarks/qwen2_5_omni/benchmark_embeddings.py \
|
||
|
|
--model /path/to/Qwen2.5-Omni-7B \
|
||
|
|
--baseline-ref 01a2f93256975c7ff9565c5414b0fe0225bc4765 \
|
||
|
|
--cpu-core 2 --lengths 1172 --warmup 50 --blocks 20 --iterations 20 \
|
||
|
|
--output-dir /tmp/qwen25-embedding-results
|
||
|
|
```
|
||
|
|
|
||
|
|
Choose a CPU core in your allowed affinity set. For the original CUDA length
|
||
|
|
sweep, use `--lengths 512 1172 4096 16384`.
|
||
|
|
|
||
|
|
## Method
|
||
|
|
|
||
|
|
- BF16 checkpoint embeddings, TP=1, CUDA int32 token IDs, CPU multimodal masks.
|
||
|
|
- One pinned CPU core, one PyTorch thread, garbage collection disabled.
|
||
|
|
- 50 warmups per variant, then 20 paired blocks of 20 calls with balanced,
|
||
|
|
randomized variant order. Summary: median of block medians.
|
||
|
|
- Host latency includes synchronization inside the wrapper. Explicit GPU
|
||
|
|
synchronization before and after the timed call is excluded from host latency.
|
||
|
|
Calling-thread CPU time and synchronized completion time are recorded separately.
|
||
|
|
- CPU frequency is sampled at block boundaries; clocks are not locked.
|
||
|
|
- Validation checks one text embedding call for the fixed wrapper, two for the
|
||
|
|
multimodal baseline, and `torch.equal` for identical inputs and batch shapes.
|
||
|
|
This does not establish batch invariance or full-model output equality.
|
||
|
|
|
||
|
|
Interleaved audio/video uses a valid synthetic multimodal mask. This benchmark
|
||
|
|
measures wrapper latency, not E2E throughput.
|
||
|
|
|
||
|
|
## Reference results
|
||
|
|
|
||
|
|
Original environment: H100 80GB; Python 3.12.3; vLLM 0.29.0+cu129;
|
||
|
|
PyTorch 2.13.0+cu129; Transformers 5.14.0; Qwen2.5-Omni-7B BF16.
|
||
|
|
At 1172 tokens, the September 13 measurements were:
|
||
|
|
|
||
|
|
| Input | Before (μs) | After (μs) | Host latency reduction |
|
||
|
|
| --- | ---: | ---: | ---: |
|
||
|
|
| Text | 18.81 | 18.66 | 0.80% |
|
||
|
|
| Audio | 104.32 | 85.30 | 18.23% |
|
||
|
|
| Image | 104.72 | 86.27 | 17.62% |
|
||
|
|
| Video | 104.73 | 85.76 | 18.11% |
|
||
|
|
| Audio + image + video | 567.92 | 549.54 | 3.24% |
|
||
|
|
| Interleaved audio/video | 622.76 | 597.84 | 4.00% |
|
||
|
|
|
||
|
|
Results vary between runs and environments. These local savings did not produce
|
||
|
|
a consistent E2E throughput improvement in the separate repeated E2E experiment.
|
||
|
|
|
||
|
|
## Output
|
||
|
|
|
||
|
|
The output directory contains `metadata.json`, `micro-raw.json`,
|
||
|
|
`validation.json`, and `summary.json`. Raw records include per-call timings,
|
||
|
|
block timestamps and CPU frequency samples. Timer overhead is measured separately.
|