[Doc] Document async Omni output materialization (#5610)
Signed-off-by: Chenguang ZHENG <645327136@qq.com>
This commit is contained in:
Chenguang Zheng authored and GitHub committed
1 parent
30cba84222
commit
d5d47e9be1
4 files changed
+538
No files matched your search
@@ -118,6 +118,7 @@ nav:
|
||||
- design/feature/teacache.md
|
||||
- design/feature/skip_softmax.md
|
||||
- design/feature/async_chunk.md
|
||||
- design/feature/omni_async_output_materialization.md
|
||||
- design/feature/vae_parallel.md
|
||||
- design/feature/diffusion_step_execution.md
|
||||
- design/feature/diffusion_request_level_batching.md
|
||||
|
||||
@@ -0,0 +1,386 @@
|
||||
# Async Omni Output Materialization
|
||||
|
||||
## Table of Contents
|
||||
|
||||
1. [Overview](#overview)
|
||||
2. [Performance](#performance)
|
||||
3. [Architecture](#architecture)
|
||||
4. [Enabling the Feature](#enabling-the-feature)
|
||||
5. [Compatibility and Fallbacks](#compatibility-and-fallbacks)
|
||||
6. [Related Files](#related-files)
|
||||
|
||||
## Overview
|
||||
|
||||
Async Omni output materialization moves CPU-side construction of
|
||||
`OmniModelRunnerOutput` out of the autoregressive (AR) decode critical path.
|
||||
It complements [Async Chunk](async_chunk.md): async chunk pipelines partial
|
||||
outputs between stages, while async output materialization prevents the
|
||||
construction of those partial outputs from blocking the next decode step.
|
||||
|
||||
An AR stage must return sampled token IDs quickly because the scheduler needs
|
||||
them for the next decode step. However, an Omni stage can also produce hidden
|
||||
states, multimodal tensors, streaming inter-stage payloads, and connector
|
||||
metadata. Building these payloads can require device-to-host (D2H) copies,
|
||||
tensor slicing, flattening, and Python object construction.
|
||||
|
||||
Without this feature, all of that work runs inline in `sample_tokens()`:
|
||||
|
||||
```text
|
||||
sample tokens
|
||||
-> update decode state
|
||||
-> copy hidden and multimodal outputs to CPU
|
||||
-> build per-request payloads
|
||||
-> build the streaming wire payload
|
||||
-> collect connector signals
|
||||
-> return sampled tokens
|
||||
-> launch the next decode step
|
||||
```
|
||||
|
||||
With async output materialization, only work required by the next decode step
|
||||
stays on the critical path:
|
||||
|
||||
```text
|
||||
sample tokens
|
||||
-> update decode state
|
||||
-> snapshot output state and start asynchronous D2H copies
|
||||
-> register sampled tokens
|
||||
-> launch the next decode step
|
||||
|
||||
background output path
|
||||
-> wait for the payload snapshot
|
||||
-> build per-request payloads
|
||||
-> build the streaming wire payload
|
||||
-> collect connector signals
|
||||
-> construct OmniModelRunnerOutput
|
||||
```
|
||||
|
||||
This allows payload construction for step `N` to overlap with GPU decode work
|
||||
for step `N + 1`. It does not change model computation or generated output.
|
||||
|
||||
!!! note "Platform and validation scope"
|
||||
The performance and correctness validation in this document covers CUDA
|
||||
and ROCm. This is a validation scope, not a runtime platform guard: XPU
|
||||
inherits `GPUARModelRunner`, and MUSA selects `GPUARWorker`, so those
|
||||
backends may enter the asynchronous path when its other guards pass. XPU
|
||||
and MUSA have not been validated for this optimization. Ascend NPU uses a
|
||||
separate `NPUARModelRunner` path that still constructs
|
||||
`OmniModelRunnerOutput` and drains connector output synchronously before
|
||||
wrapping the result as an asynchronous runner output.
|
||||
|
||||
<p align="center">
|
||||
<img
|
||||
alt="Decode step gap before and after async output materialization"
|
||||
src="../../../source/architecture/qwen3-omni-async-output-step-gap.svg"
|
||||
width="100%"
|
||||
>
|
||||
</p>
|
||||
|
||||
<p align="center">
|
||||
<em>
|
||||
Figure 1: Moving payload construction off the decode path reduces the
|
||||
observed gap between consecutive Talker steps from about 2.8 ms to 41 µs.
|
||||
Adapted from the
|
||||
<a href="https://vllm.ai/blog/2026-07-01-qwen3-omni-optimization">
|
||||
Qwen3-Omni optimization blog
|
||||
</a>.
|
||||
</em>
|
||||
</p>
|
||||
|
||||
For Qwen3-Omni, the feature is enabled for both AR stages:
|
||||
|
||||
- **Thinker**: Defers hidden-state and multimodal payload materialization.
|
||||
- **Talker**: Runs its decode-state postprocess eagerly, then defers codec
|
||||
payload materialization. Because Code2Wav consumes codec codes rather than
|
||||
Talker hidden states, the Talker also skips unnecessary hidden-state D2H.
|
||||
- **Code2Wav**: Does not use this path because it is a generation stage rather
|
||||
than an AR stage.
|
||||
|
||||
Qwen3-TTS uses the same mechanism in its AR Talker stage. The Talker updates
|
||||
the decode state it needs for the next step before returning, then constructs
|
||||
the codec payload for Code2Wav in the background.
|
||||
|
||||
## Performance
|
||||
|
||||
In the controlled Qwen3-Omni optimization sweep, enabling async output
|
||||
materialization on top of CUDA Graph and async chunk produced the following
|
||||
results at concurrency 64:
|
||||
|
||||
| Configuration | Request throughput | Mean audio TTFP | Mean audio RTF |
|
||||
|---|---:|---:|---:|
|
||||
| CUDA Graph + async chunk | 9.3 req/s | 655 ms | 0.63 |
|
||||
| + async output materialization | 11.3 req/s | 631 ms | 0.47 |
|
||||
| Change | **+22%** | **-4%** | **-25%** |
|
||||
|
||||
The sweep used `Qwen/Qwen3-Omni-30B-A3B-Instruct`, 640 Seed-TTS English
|
||||
prompts, and one replica each for Thinker, Talker, and Code2Wav. Async output
|
||||
materialization recovers throughput by removing CPU payload construction from
|
||||
the decode critical path without regressing time to first audio packet (TTFP).
|
||||
See the
|
||||
[Qwen3-Omni optimization blog](https://vllm.ai/blog/2026-07-01-qwen3-omni-optimization)
|
||||
for the complete benchmark methodology and results.
|
||||
|
||||
The implementation was also validated against the synchronous safe-copy path.
|
||||
The text and audio output hashes matched, confirming that overlap does not
|
||||
change the generated result. See
|
||||
[PR #4476](https://github.com/vllm-project/vllm-omni/pull/4476) for the
|
||||
correctness and profiling details.
|
||||
|
||||
## Architecture
|
||||
|
||||
### Output Lifecycle
|
||||
|
||||
`OmniAsyncGPUModelRunnerOutput` extends vLLM's
|
||||
`AsyncGPUModelRunnerOutput`. It preserves the existing asynchronous
|
||||
sampled-token feedback while adding a background builder for the complete Omni
|
||||
output.
|
||||
|
||||
The output lifecycle is:
|
||||
|
||||
1. `GPUARModelRunner.sample_tokens()` samples tokens and performs the
|
||||
bookkeeping required by the next decode step.
|
||||
2. The runner snapshots step-local metadata such as request IDs, token spans,
|
||||
scheduler output, and query start locations.
|
||||
3. CUDA payload tensors are cloned before reusable model or CUDA Graph buffers
|
||||
can be overwritten.
|
||||
4. The cloned payload is copied to pinned CPU memory on a dedicated CUDA
|
||||
stream. A CUDA event records when the copy is ready.
|
||||
5. `OmniAsyncGPUModelRunnerOutput` starts a background thread that waits for
|
||||
the payload event and builds `OmniModelRunnerOutput`.
|
||||
6. The AR runner immediately registers the sampled-token CPU copy with the
|
||||
input batch and returns, allowing async scheduling to advance.
|
||||
7. When the engine calls `get_output()`, it joins the background builder,
|
||||
propagates any builder exception, and finalizes sampled tokens and logprobs
|
||||
through the upstream output implementation.
|
||||
|
||||
```text
|
||||
GPU / decode path Background output path
|
||||
|
||||
forward + sample
|
||||
|
|
||||
+-- clone output tensors
|
||||
+-- enqueue D2H copy --------------> wait for D2H event
|
||||
+-- register sampled tokens |
|
||||
+-- return async output +-- slice per request
|
||||
| +-- build multimodal payloads
|
||||
next decode step +-- partition streaming payloads
|
||||
+-- build the wire payload
|
||||
+-- drain connector signals
|
||||
+-- build OmniModelRunnerOutput
|
||||
```
|
||||
|
||||
### Safe State Snapshots
|
||||
|
||||
The next scheduler step can mutate runner state while the background builder is
|
||||
still active. Most step-local builder inputs are therefore detached snapshots.
|
||||
The snapshotted state includes:
|
||||
|
||||
- scheduler token counts and speculative-token metadata
|
||||
- request IDs and request-to-batch-index mappings
|
||||
- sampled token IDs, logprobs, and prompt logprobs
|
||||
- query start locations and scheduled-token spans
|
||||
- hidden states and multimodal output tensors
|
||||
- KV-connector and encoder-cache outputs captured for the step
|
||||
|
||||
CUDA tensor cloning is required because CUDA Graph and model output buffers can
|
||||
be reused by the next forward pass. The dedicated copy stream and pinned host
|
||||
buffers make the D2H transfer asynchronous; the snapshot retains its cloned
|
||||
CUDA sources until the transfer event completes.
|
||||
|
||||
### Live Runner State and Ownership
|
||||
|
||||
The complete builder input is not a detached snapshot. Connector output state
|
||||
remains runner-owned: `get_omni_connector_output()` drains live, per-cycle
|
||||
connector signals after the streaming payload is built. Correctness therefore
|
||||
depends on that builder being the only consumer that drains connector signals
|
||||
for its output cycle. Receive-side connector state written by the background
|
||||
receiver is coordinated by the connector mixin's `_lock`.
|
||||
`OmniAsyncGPUModelRunnerOutput.get_output()` joins the builder and is the
|
||||
completion and exception boundary before the materialized output is consumed.
|
||||
|
||||
The shared output builder also contains a full-payload accumulation branch
|
||||
that reads `self.requests`. That branch is not reachable when async Omni output
|
||||
materialization is enabled: this feature requires `async_chunk`, while
|
||||
`should_accumulate_full_payload_output()` returns `False` whenever
|
||||
`async_chunk` is enabled. The feature path partitions each step's payload into
|
||||
streaming inter-stage and client handoffs instead.
|
||||
|
||||
### Output Builder
|
||||
|
||||
The background builder performs the work that previously ran inline:
|
||||
|
||||
- resolves which requests need downstream payloads
|
||||
- converts hidden and multimodal tensors into per-request CPU payloads
|
||||
- applies model-specific payload processing
|
||||
- partitions per-step payloads into inter-stage and client streams
|
||||
- creates the tensor-only `multimodal_outputs` wire payload
|
||||
- drains live connector readiness signals after building the streaming payload
|
||||
- constructs the final `OmniModelRunnerOutput`
|
||||
|
||||
Input-side connector operations remain synchronous with model execution. In
|
||||
particular, receiving inputs and flushing previously completed connector
|
||||
outputs are not deferred because they affect scheduler-visible request state.
|
||||
|
||||
### Qwen3-Omni Stage Behavior
|
||||
|
||||
| Stage | Async output behavior | Reason |
|
||||
|---|---|---|
|
||||
| Thinker | Snapshots hidden states and multimodal outputs; builds the downstream payload in the background | The Talker needs the payload, but the next Thinker decode step only needs sampled-token feedback |
|
||||
| Talker | Runs lightweight postprocess eagerly; snapshots codec outputs; omits hidden states from the downstream payload | `hidden_states.last` is needed by the next Talker step, while Code2Wav only needs codec codes |
|
||||
| Code2Wav | Uses the normal generation-stage output path | Code2Wav is not executed by `GPUARModelRunner` |
|
||||
|
||||
The Qwen3-TTS Talker follows the same pattern as the Qwen3-Omni Talker:
|
||||
postprocess runs eagerly because the next decode step needs its updated state,
|
||||
while codec payload construction runs asynchronously. Its Code2Wav stage also
|
||||
uses the normal generation-stage path.
|
||||
|
||||
## Enabling the Feature
|
||||
|
||||
There is no separate `async_omni_output` command-line or YAML option. The
|
||||
feature is selected automatically for model stages that opt in when the runtime
|
||||
conditions are safe. On the validated CUDA and ROCm configurations, using the
|
||||
bundled deployment profile is enough to turn it on for the supported
|
||||
Qwen3-Omni and Qwen3-TTS pipelines. The selection logic has no CUDA/ROCm-only
|
||||
guard; see [Compatibility and Fallbacks](#compatibility-and-fallbacks) for the
|
||||
other platform paths. The same profile does not enable background Omni
|
||||
materialization on Ascend NPU.
|
||||
|
||||
### Example 1: Qwen3-Omni
|
||||
|
||||
Start the full Qwen3-Omni pipeline:
|
||||
|
||||
```bash
|
||||
vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct \
|
||||
--omni \
|
||||
--port 8091
|
||||
```
|
||||
|
||||
The model registry automatically loads
|
||||
`vllm_omni/deploy/qwen3_omni_moe.yaml`. Its relevant settings are:
|
||||
|
||||
```yaml
|
||||
async_chunk: true
|
||||
|
||||
stages:
|
||||
- stage_id: 0 # Thinker
|
||||
enable_prefix_caching: false
|
||||
# async_scheduling defaults to true for AR stages
|
||||
|
||||
- stage_id: 1 # Talker
|
||||
enable_prefix_caching: false
|
||||
# async_scheduling defaults to true for AR stages
|
||||
|
||||
- stage_id: 2 # Code2Wav
|
||||
enable_prefix_caching: false
|
||||
async_scheduling: false
|
||||
```
|
||||
|
||||
With this profile, async output materialization activates automatically for:
|
||||
|
||||
- **Stage 0, Thinker**: Defers hidden-state and multimodal payload
|
||||
construction.
|
||||
- **Stage 1, Talker**: Defers codec payload construction after eagerly
|
||||
updating its decode state.
|
||||
|
||||
Stage 2 does not use this feature because Code2Wav is not an AR stage.
|
||||
|
||||
### Example 2: Qwen3-TTS
|
||||
|
||||
Start Qwen3-TTS with any checkpoint that uses the Qwen3-TTS Talker pipeline.
|
||||
For example:
|
||||
|
||||
```bash
|
||||
vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice \
|
||||
--omni \
|
||||
--port 8091
|
||||
```
|
||||
|
||||
The model registry automatically loads `vllm_omni/deploy/qwen3_tts.yaml`. Its
|
||||
relevant settings are:
|
||||
|
||||
```yaml
|
||||
async_chunk: true
|
||||
|
||||
stages:
|
||||
- stage_id: 0 # Talker
|
||||
async_scheduling: true
|
||||
enable_prefix_caching: false
|
||||
|
||||
- stage_id: 1 # Code2Wav
|
||||
enable_prefix_caching: false
|
||||
```
|
||||
|
||||
Async output materialization activates only for Stage 0, the AR Talker. The
|
||||
Talker keeps the state required by its next decode step on the GPU, skips an
|
||||
unnecessary hidden-state D2H copy, and builds the codec payload for Code2Wav in
|
||||
the background. Stage 1 uses the generation-stage output path.
|
||||
|
||||
The same behavior applies to the Base and VoiceDesign Qwen3-TTS checkpoints
|
||||
because they use the same Talker implementation.
|
||||
|
||||
!!! warning
|
||||
Do not pass `--no-async-chunk` or enable prefix caching when you want this
|
||||
optimization. Either change causes the runner to fall back to synchronous
|
||||
output construction. Enabling async chunk alone cannot activate the feature
|
||||
for a model that has not opted into async Omni output.
|
||||
|
||||
!!! note "Prefix cache compatibility"
|
||||
Async Omni output materialization and Omni prefix caching cannot currently
|
||||
run together. `_should_use_async_omni_output()` returns `False` whenever
|
||||
`self.omni_prefix_cache` is present, so prefix caching selects synchronous
|
||||
output materialization. Supporting both features requires snapshotting or
|
||||
otherwise synchronizing prefix-cache merge and update state before the
|
||||
background output path can consume it safely.
|
||||
|
||||
## Compatibility and Fallbacks
|
||||
|
||||
`GPUARModelRunner` uses the async output path only when all of the following
|
||||
runtime conditions hold:
|
||||
|
||||
| Requirement | Reason |
|
||||
|---|---|
|
||||
| AR async scheduling is enabled | The optimization relies on the scheduler advancing while the prior output is materialized |
|
||||
| `async_chunk` is enabled | The feature targets incremental downstream Omni payloads |
|
||||
| The model stage opts in with `use_async_omni_output` | Models must declare that their output lifecycle is safe to defer |
|
||||
| Omni prefix cache is disabled | Prefix-cache merge and update ordering currently requires synchronous materialization |
|
||||
| Speculative decoding is disabled | Speculative output state is not included in this deferred path |
|
||||
| Routed-expert output is disabled | Routed-expert extraction currently requires the synchronous path |
|
||||
| Postprocess is absent or explicitly runs eagerly | State needed by the next decode step must be updated before the runner returns |
|
||||
|
||||
These checks are evaluated per stage on runners based on `GPUARModelRunner`.
|
||||
An unsupported combination does not prevent serving; it only falls back to
|
||||
synchronous output construction for that stage. CUDA and ROCm are the validated
|
||||
platforms. XPU inherits `GPUARModelRunner`, and MUSA selects `GPUARWorker`, so
|
||||
they may select the async path when these checks pass, but this optimization is
|
||||
not yet validated on either backend.
|
||||
|
||||
Ascend NPU does not use this selection path. `NPUARModelRunner.sample_tokens()`
|
||||
fully constructs `OmniModelRunnerOutput`, calls
|
||||
`get_omni_connector_output()`, and only then creates the asynchronous wrapper,
|
||||
so Omni payload materialization remains synchronous.
|
||||
|
||||
!!! note
|
||||
`use_async_omni_output`,
|
||||
`eager_omni_postprocess_before_async_output`, and
|
||||
`omni_pooler_payload_include_hidden` are model implementation contracts,
|
||||
not user-facing configuration fields.
|
||||
|
||||
## Related Files
|
||||
|
||||
- `vllm_omni/worker/gpu_ar_model_runner.py`: Async output object, safe tensor
|
||||
snapshots, compatibility guards, and deferred Omni output builder.
|
||||
- `vllm_omni/platforms/npu/worker/npu_ar_model_runner.py`: Separate Ascend NPU
|
||||
runner, where Omni output materialization remains synchronous.
|
||||
- `vllm_omni/model_executor/models/qwen3_omni/qwen3_omni.py`: Thinker and
|
||||
Talker opt-in behavior.
|
||||
- `vllm_omni/deploy/qwen3_omni_moe.yaml`: Default Qwen3-Omni deployment
|
||||
settings.
|
||||
- `vllm_omni/model_executor/models/qwen3_tts/qwen3_tts_talker.py`: Qwen3-TTS
|
||||
Talker opt-in and eager postprocess behavior.
|
||||
- `vllm_omni/deploy/qwen3_tts.yaml`: Default Qwen3-TTS deployment settings.
|
||||
- `tests/worker/test_gpu_ar_model_runner.py`: Snapshot, guard, connector
|
||||
ordering, and background error-propagation tests.
|
||||
- [Async Chunk](async_chunk.md): Inter-stage chunking and scheduling design.
|
||||
- [Qwen3-Omni optimization blog](https://vllm.ai/blog/2026-07-01-qwen3-omni-optimization):
|
||||
Optimization context and controlled performance results.
|
||||
- [PR #4476](https://github.com/vllm-project/vllm-omni/pull/4476): Feature
|
||||
implementation, validation, and review history.
|
||||
@@ -10,6 +10,7 @@ This section contains design documents and architecture specifications for vLLM-
|
||||
|
||||
- [Disaggregated Inference](feature/disaggregated_inference.md)
|
||||
- [Ray-based Execution](feature/ray_based_execution.md)
|
||||
- [Async Omni Output Materialization](feature/omni_async_output_materialization.md)
|
||||
- [Adding Step Execution Support for Diffusion Pipelines](feature/diffusion_step_execution.md)
|
||||
- [Request-Level Batching for Diffusion](feature/diffusion_request_level_batching.md)
|
||||
- [Continuous Batching for Step-Wise Diffusion](feature/diffusion_continuous_batching.md)
|
||||
|
||||
@@ -0,0 +1,150 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" width="1024" height="600" viewBox="0 0 1024 600" role="img" aria-labelledby="title desc">
|
||||
<title id="title">Qwen3-Omni Talker step gap before and after asynchronous output materialization</title>
|
||||
<desc id="desc">Before the optimization, synchronous D2H copy, per-request payload construction, streaming payload handoff, and connector-signal drain separate consecutive Talker forward and sample steps by about 2.8 milliseconds. After the optimization, that Omni output work runs on a background builder while the next Talker step executes, leaving a boundary of about 41 microseconds for snapshot, D2H enqueue, state update, and sampled-token feedback.</desc>
|
||||
<defs>
|
||||
<marker id="time-arrow" viewBox="0 0 10 10" refX="8" refY="5" markerWidth="7" markerHeight="7" orient="auto">
|
||||
<path d="M 0 0 L 10 5 L 0 10 z" fill="#64748b"/>
|
||||
</marker>
|
||||
<marker id="red-end" viewBox="0 0 10 10" refX="8" refY="5" markerWidth="5" markerHeight="5" orient="auto">
|
||||
<path d="M 0 0 L 10 5 L 0 10 z" fill="#b91c1c"/>
|
||||
</marker>
|
||||
<marker id="red-start" viewBox="0 0 10 10" refX="2" refY="5" markerWidth="5" markerHeight="5" orient="auto">
|
||||
<path d="M 10 0 L 0 5 L 10 10 z" fill="#b91c1c"/>
|
||||
</marker>
|
||||
<marker id="blue-end" viewBox="0 0 10 10" refX="8" refY="5" markerWidth="6" markerHeight="6" orient="auto">
|
||||
<path d="M 0 0 L 10 5 L 0 10 z" fill="#2563eb"/>
|
||||
</marker>
|
||||
<style>
|
||||
.canvas { fill: #ffffff; stroke: #e5e7eb; }
|
||||
.title { font: 700 22px system-ui, -apple-system, Segoe UI, sans-serif; fill: #111827; }
|
||||
.subtitle { font: 500 13px system-ui, -apple-system, Segoe UI, sans-serif; fill: #64748b; }
|
||||
.phase { font: 700 16px system-ui, -apple-system, Segoe UI, sans-serif; fill: #1f2937; }
|
||||
.phase-sub { font: 500 12px system-ui, -apple-system, Segoe UI, sans-serif; fill: #64748b; }
|
||||
.lane { font: 600 12px system-ui, -apple-system, Segoe UI, sans-serif; fill: #374151; }
|
||||
.block-title { font: 700 10px system-ui, -apple-system, Segoe UI, sans-serif; letter-spacing: .25px; fill: #ffffff; }
|
||||
.block-op { font: 500 9.5px system-ui, -apple-system, Segoe UI, sans-serif; fill: #ffffff; }
|
||||
.axis { font: 500 11px system-ui, -apple-system, Segoe UI, sans-serif; fill: #94a3b8; }
|
||||
.measure { font: 700 12px system-ui, -apple-system, Segoe UI, sans-serif; fill: #b91c1c; }
|
||||
.inset-title { font: 700 11px system-ui, -apple-system, Segoe UI, sans-serif; fill: #475569; }
|
||||
.inset-step { font: 600 10px system-ui, -apple-system, Segoe UI, sans-serif; fill: #ffffff; }
|
||||
.note-title { font: 700 12px system-ui, -apple-system, Segoe UI, sans-serif; fill: #1e3a8a; }
|
||||
.note-text { font: 500 10.5px system-ui, -apple-system, Segoe UI, sans-serif; fill: #475569; }
|
||||
.stat-cap { font: 600 11px system-ui, -apple-system, Segoe UI, sans-serif; fill: #64748b; }
|
||||
.stat-big { font: 800 16px system-ui, -apple-system, Segoe UI, sans-serif; fill: #111827; }
|
||||
.stat-sub { font: 500 11px system-ui, -apple-system, Segoe UI, sans-serif; fill: #64748b; }
|
||||
.before-card { fill: #eef4ff; stroke: #c7dbf6; }
|
||||
.after-card { fill: #eafaf0; stroke: #bfe7cf; }
|
||||
.talker { fill: #16a34a; }
|
||||
.sync-output { fill: #ea580c; }
|
||||
.async-output { fill: #2563eb; }
|
||||
.zoom-card { fill: #fff7f7; stroke: #f3caca; }
|
||||
.note-card { fill: #eff6ff; stroke: #bfdbfe; }
|
||||
.stat-blue { fill: #eef4ff; stroke: #c7dbf6; }
|
||||
.stat-green { fill: #eafaf0; stroke: #bfe7cf; }
|
||||
.stat-gray { fill: #f4f5f7; stroke: #e2e5ea; }
|
||||
.guide { stroke: #e2e8f0; stroke-width: 1; }
|
||||
.time-line { stroke: #64748b; stroke-width: 1.3; fill: none; marker-end: url(#time-arrow); }
|
||||
.measure-line { stroke: #b91c1c; stroke-width: 1.4; marker-start: url(#red-start); marker-end: url(#red-end); }
|
||||
.zoom-lead { stroke: #dc2626; stroke-width: 1.15; stroke-dasharray: 4 3; fill: none; }
|
||||
.handoff { stroke: #2563eb; stroke-width: 1.3; stroke-dasharray: 3 2; fill: none; marker-end: url(#blue-end); }
|
||||
</style>
|
||||
</defs>
|
||||
|
||||
<rect x="14" y="14" width="996" height="572" rx="14" class="canvas"/>
|
||||
|
||||
<text x="44" y="52" class="title">Async output: what moves off the Talker decode path</text>
|
||||
<text x="44" y="74" class="subtitle">Qwen3-Omni Talker stage · consecutive decode steps in a fixed profiler window</text>
|
||||
|
||||
<!-- ===================== BEFORE ===================== -->
|
||||
<rect x="44" y="94" width="144" height="110" rx="10" class="before-card"/>
|
||||
<text x="60" y="120" class="phase">Before</text>
|
||||
<text x="60" y="139" class="phase-sub">Omni output built</text>
|
||||
<text x="60" y="156" class="phase-sub">synchronously on the</text>
|
||||
<text x="60" y="173" class="phase-sub">AR critical path</text>
|
||||
|
||||
<text x="206" y="122" class="lane">AR path</text>
|
||||
<line x1="300" y1="88" x2="300" y2="210" class="guide"/>
|
||||
|
||||
<rect x="300" y="104" width="190" height="54" rx="5" class="talker"/>
|
||||
<text x="395" y="124" text-anchor="middle" class="block-title">TALKER STAGE · STEP N</text>
|
||||
<text x="395" y="145" text-anchor="middle" class="block-op">forward → sample</text>
|
||||
|
||||
<rect x="490" y="104" width="250" height="54" rx="5" class="sync-output"/>
|
||||
<text x="615" y="120" text-anchor="middle" class="block-title">SYNC OMNI OUTPUT · STEP N</text>
|
||||
<text x="615" y="138" text-anchor="middle" class="block-op">D2H → per-request slice / payload build</text>
|
||||
<text x="615" y="152" text-anchor="middle" class="block-op">streaming handoff → connector-signal drain</text>
|
||||
|
||||
<rect x="740" y="104" width="220" height="54" rx="5" class="talker"/>
|
||||
<text x="850" y="124" text-anchor="middle" class="block-title">TALKER STAGE · STEP N+1</text>
|
||||
<text x="850" y="145" text-anchor="middle" class="block-op">forward → sample</text>
|
||||
|
||||
<line x1="490" y1="174" x2="740" y2="174" class="measure-line"/>
|
||||
<text x="615" y="194" text-anchor="middle" class="measure">~2.8 ms GPU gap: output work is inline</text>
|
||||
|
||||
<!-- ===================== AFTER ===================== -->
|
||||
<rect x="44" y="218" width="144" height="140" rx="10" class="after-card"/>
|
||||
<text x="60" y="244" class="phase">After</text>
|
||||
<text x="60" y="263" class="phase-sub">Only step feedback</text>
|
||||
<text x="60" y="280" class="phase-sub">stays inline; payload</text>
|
||||
<text x="60" y="297" class="phase-sub">work runs in the</text>
|
||||
<text x="60" y="314" class="phase-sub">background</text>
|
||||
|
||||
<text x="206" y="254" class="lane">Talker GPU</text>
|
||||
<text x="206" y="326" class="lane">Output builder</text>
|
||||
<line x1="300" y1="212" x2="300" y2="370" class="guide"/>
|
||||
|
||||
<rect x="300" y="226" width="235" height="54" rx="5" class="talker"/>
|
||||
<text x="418" y="246" text-anchor="middle" class="block-title">TALKER STAGE · STEP N</text>
|
||||
<text x="418" y="267" text-anchor="middle" class="block-op">forward → sample</text>
|
||||
|
||||
<rect x="540" y="226" width="235" height="54" rx="5" class="talker"/>
|
||||
<text x="658" y="246" text-anchor="middle" class="block-title">TALKER STAGE · STEP N+1</text>
|
||||
<text x="658" y="267" text-anchor="middle" class="block-op">forward → sample</text>
|
||||
|
||||
<path d="M 535 280 L 548 298" class="handoff"/>
|
||||
<rect x="540" y="298" width="360" height="58" rx="5" class="async-output"/>
|
||||
<text x="720" y="315" text-anchor="middle" class="block-title">BACKGROUND OMNI OUTPUT · STEP N</text>
|
||||
<text x="720" y="334" text-anchor="middle" class="block-op">wait for D2H → per-request slice / payload build</text>
|
||||
<text x="720" y="349" text-anchor="middle" class="block-op">streaming handoff → connector-signal drain</text>
|
||||
|
||||
<line x1="300" y1="366" x2="930" y2="366" class="time-line"/>
|
||||
<text x="922" y="382" text-anchor="end" class="axis">time</text>
|
||||
|
||||
<!-- leads from the real 41 us boundary to the magnified inset -->
|
||||
<line x1="535" y1="280" x2="485" y2="424" class="zoom-lead"/>
|
||||
<line x1="540" y1="280" x2="525" y2="424" class="zoom-lead"/>
|
||||
|
||||
<!-- ===================== 41 µs BOUNDARY ===================== -->
|
||||
<rect x="300" y="386" width="400" height="92" rx="10" class="zoom-card"/>
|
||||
<text x="316" y="404" class="inset-title">Zoom · Talker step boundary (magnified)</text>
|
||||
|
||||
<rect x="320" y="416" width="165" height="26" rx="4" class="talker"/>
|
||||
<rect x="525" y="416" width="155" height="26" rx="4" class="talker"/>
|
||||
<text x="403" y="433" text-anchor="middle" class="inset-step">Talker step N</text>
|
||||
<text x="603" y="433" text-anchor="middle" class="inset-step">Talker step N+1</text>
|
||||
|
||||
<line x1="485" y1="452" x2="525" y2="452" class="measure-line"/>
|
||||
<text x="505" y="469" text-anchor="middle" class="measure">~41 µs</text>
|
||||
|
||||
<rect x="720" y="386" width="240" height="92" rx="10" class="note-card"/>
|
||||
<text x="736" y="405" class="note-title">Inline at the ~41 µs boundary</text>
|
||||
<text x="736" y="424" class="note-text">• eager Talker state update</text>
|
||||
<text x="736" y="441" class="note-text">• output snapshot + D2H enqueue</text>
|
||||
<text x="736" y="458" class="note-text">• sampled-token feedback</text>
|
||||
|
||||
<!-- ===================== SUMMARY ===================== -->
|
||||
<rect x="44" y="494" width="296" height="70" rx="9" class="stat-blue"/>
|
||||
<text x="60" y="514" class="stat-cap">Step-to-step gap</text>
|
||||
<text x="60" y="538" class="stat-big">2.8 ms → 41 µs</text>
|
||||
<text x="60" y="555" class="stat-sub">~68× tighter</text>
|
||||
|
||||
<rect x="356" y="494" width="296" height="70" rx="9" class="stat-green"/>
|
||||
<text x="372" y="514" class="stat-cap">Talker GPU</text>
|
||||
<text x="372" y="538" class="stat-big">next step starts immediately</text>
|
||||
<text x="372" y="555" class="stat-sub">decode workers stay busy</text>
|
||||
|
||||
<rect x="668" y="494" width="312" height="70" rx="9" class="stat-gray"/>
|
||||
<text x="684" y="514" class="stat-cap">Background builder</text>
|
||||
<text x="684" y="538" class="stat-big">materializes Omni output</text>
|
||||
<text x="684" y="555" class="stat-sub">payload work overlaps the next Talker step</text>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 9.8 KiB |
Reference in new issue
Block a user