174 lines
7.9 KiB
Markdown
174 lines
7.9 KiB
Markdown
|
|
# Metrics
|
|
|
|
You can use these metrics in production to monitor the health and performance of the vLLM-omni system. Typical scenarios include:
|
|
|
|
- **Performance Monitoring**: Track throughput (e.g., `e2e_avg_tokens_per_s`), latency (e.g., `e2e_total_ms`), and resource utilization to verify that the system meets expected standards.
|
|
|
|
- **Debugging and Troubleshooting**: Use detailed per-request metrics to diagnose issues, such as high transfer times or unexpected token counts.
|
|
|
|
## How to Enable and View Metrics
|
|
|
|
### Start the Service with Metrics Logging
|
|
|
|
```bash
|
|
vllm serve /workspace/models/Qwen3-Omni-30B-A3B-Instruct --omni --port 8014 --log-stats
|
|
```
|
|
|
|
### Send a Request
|
|
|
|
```bash
|
|
python openai_chat_completion_client_for_multimodal_generation.py --query-type use_image
|
|
```
|
|
|
|
### What You Will See
|
|
|
|
With `--log-stats` enabled, the server will output detailed metrics logs after each request. Example output:
|
|
|
|
|
|
#### Overall Summary
|
|
|
|
| Field | Value |
|
|
|-----------------------------|--------------|
|
|
| e2e_requests | 1 |
|
|
| e2e_wall_time_ms | 41,299.190 |
|
|
| e2e_total_tokens | 5,202 |
|
|
| e2e_avg_time_per_request_ms | 41,299.190 |
|
|
| e2e_avg_tokens_per_s | 125.959 |
|
|
| e2e_stage_0_wall_time_ms | 10,192.289 |
|
|
| e2e_stage_1_wall_time_ms | 30,541.409 |
|
|
| e2e_stage_2_wall_time_ms | 207.496 |
|
|
|
|
#### RequestE2EStats
|
|
|
|
| Field | Value |
|
|
|-------------------------|------------|
|
|
| e2e_total_ms | 41,299.133 |
|
|
| e2e_total_tokens | 5,202 |
|
|
| transfers_total_time_ms | 245.895 |
|
|
| transfers_total_kbytes | 138,089.939|
|
|
|
|
#### StageRequestStats
|
|
|
|
| Field | 0 | 1 | 2 |
|
|
|------------------------|--------|--------|--------|
|
|
| audio_generated_frames | 0 | 0 | 525,525|
|
|
| batch_id | 38 | 274 | 0 |
|
|
| batch_size | 1 | 1 | 1 |
|
|
| num_tokens_in | 4,860 | 4,826 | 4,384 |
|
|
| num_tokens_out | 67 | 275 | 0 |
|
|
| postprocess_time_ms | 256.158| 0.491 | 0.000 |
|
|
| stage_gen_time_ms | 9,910.007|30,379.198|160.745|
|
|
|
|
#### TransferEdgeStats
|
|
|
|
| Field | 0->1 | 1->2 |
|
|
|---------------------|-------------|------------|
|
|
| size_kbytes | 109,277.349 | 28,812.591 |
|
|
| tx_time_ms | 78.701 | 18.790 |
|
|
| rx_decode_time_ms | 111.865 | 31.706 |
|
|
| in_flight_time_ms | 2.015 | 2.819 |
|
|
|
|
|
|
These logs include:
|
|
|
|
- **Overall summary**: total requests, wall time, average tokens/sec, etc.
|
|
|
|
- **E2E table**: per-request latency and token counts.
|
|
|
|
- **Stage table**: per-stage batch and timing details.
|
|
|
|
- **Transfer table**: data transfer and timing for each edge.
|
|
|
|
You can use these logs to monitor system health, debug performance, and analyze request-level metrics as described above.
|
|
|
|
|
|
## Metrics Scope: Offline vs Online Inference
|
|
|
|
For **offline inference** (batch mode), the summary includes both system-level metrics (aggregated across all requests) and per-request metrics. In this case, `e2e_requests` can be greater than 1, reflecting multiple completed requests in a batch.
|
|
|
|
For **online inference** (serving mode), the summary is always per-request. `e2e_requests` is always 1, and only request-level metrics are reported for each completion.
|
|
|
|
---
|
|
|
|
## Parameter Details
|
|
|
|
### Summary Metrics
|
|
|
|
| Field | Meaning |
|
|
|---------------------------|----------------------------------------------------------------------------------------------|
|
|
| `e2e_requests` | Number of completed requests. |
|
|
| `e2e_wall_time_ms` | Wall-clock time span from run start to last completion, in ms. |
|
|
| `e2e_total_tokens` | Total tokens counted across all completed requests (stage0 input + all stage outputs). |
|
|
| `e2e_avg_time_per_request_ms` | Average wall time per request: `e2e_wall_time_ms / e2e_requests`. |
|
|
| `e2e_avg_tokens_per_s` | Average token throughput over wall time: `e2e_total_tokens * 1000 / e2e_wall_time_ms`. |
|
|
| `e2e_stage_{i}_wall_time_ms` | Wall-clock time span for stage i, in ms. Each stage's wall time is reported as a separate field, e.g., `e2e_stage_0_wall_time_ms`, `e2e_stage_1_wall_time_ms`, etc. |
|
|
|
|
---
|
|
|
|
### E2E Table (per request)
|
|
|
|
| Field | Meaning |
|
|
|---------------------------|-----------------------------------------------------------------------|
|
|
| `e2e_total_ms` | End-to-end latency in ms. |
|
|
| `e2e_total_tokens` | Total tokens for the request (stage0 input + all stage outputs). |
|
|
| `transfers_total_time_ms` | Sum of transfer edge `total_time_ms` for this request. |
|
|
| `transfers_total_kbytes` | Sum of transfer kbytes for this request. |
|
|
|
|
|
|
---
|
|
|
|
### Stage Table (per stage event / request)
|
|
|
|
| Field | Meaning |
|
|
|---------------------------|-------------------------------------------------------------------------------------------------|
|
|
| `batch_id` | Batch index. |
|
|
| `batch_size` | Batch size. |
|
|
| `num_tokens_in` | Input tokens to the stage. |
|
|
| `num_tokens_out` | Output tokens from the stage. |
|
|
| `stage_gen_time_ms` | Stage compute time in ms, excluding postprocessing time (reported separately as `postprocess_time_ms`). |
|
|
| `image_num` | Number of images generated (for diffusion/image stages). |
|
|
| `resolution` | Image resolution (for diffusion/image stages). |
|
|
| `postprocess_time_ms` | Diffusion/image: post-processing time in ms. |
|
|
|
|
---
|
|
|
|
### Transfer Table (per edge / request)
|
|
|
|
| Field | Meaning |
|
|
|----------------------|---------------------------------------------------------------------------|
|
|
| `size_kbytes` | Total kbytes transferred. |
|
|
| `tx_time_ms` | Sender transfer time in ms. |
|
|
| `rx_decode_time_ms` | Receiver decode time in ms. |
|
|
| `in_flight_time_ms` | In-flight time in ms. |
|
|
|
|
|
|
### Expectation of the Numbers (Verification)
|
|
|
|
**Formulas:**
|
|
|
|
- `e2e_total_tokens = Stage0's num_tokens_in + sum(all stages' num_tokens_out)`
|
|
|
|
- `transfers_total_time_ms = sum(tx_time_ms + rx_decode_time_ms + in_flight_time_ms)` for every edge
|
|
|
|
**Using the example above:**
|
|
|
|
**e2e_total_tokens**
|
|
|
|
- Stage0's `num_tokens_in`: **4,860**
|
|
- Stage0's `num_tokens_out`: **67**
|
|
- Stage1's `num_tokens_out`: **275**
|
|
- Stage2's `num_tokens_out`: **0**
|
|
|
|
so `e2e_total_tokens = 4,860 + 67 + 275 + 0 = 5,202`, which matches the table value `e2e_total_tokens`.
|
|
|
|
**transfers_total_time_ms**
|
|
|
|
For each edge:
|
|
|
|
- 0->1: tx_time_ms (**78.701**) + rx_decode_time_ms (**111.865**) + in_flight_time_ms (**2.015**) = **192.581**
|
|
|
|
- 1->2: tx_time_ms (**18.790**) + rx_decode_time_ms (**31.706**) + in_flight_time_ms (**2.819**) = **53.315**
|
|
|
|
192.581 + 53.315 = **245.896** = transfers_total_time_ms, which matches the calculation (difference is due to rounding)
|