[Perf] Support continuous batching for MammothModa2 (#7954)
Signed-off-by: nagisa-kun <1434936049@qq.com> Signed-off-by: nagisa <1434936049@qq.com>
This commit is contained in:
1 parent
c9c478dd07
commit
b9cf74352a
5 files changed
+684
-72
No files matched your search
@@ -18,12 +18,24 @@ The generic example formats the AR prompt, drives the AR → DiT stage pipeline,
|
||||
and forwards MammothModa2-specific generation parameters through the
|
||||
pipeline-declared `extra_body` contract.
|
||||
|
||||
MammothModa2's DiT stage runs in the shared diffusion runtime in request mode.
|
||||
The first integration intentionally supports one request and one image per
|
||||
forward only (`max_num_seqs: 1`, `num_outputs_per_prompt: 1`). Request-level
|
||||
batching, step execution, continuous batching, compilation, quantization,
|
||||
parallelism, and offload are not enabled by this recipe. TeaCache acceleration
|
||||
is supported for the DiT stage.
|
||||
MammothModa2's DiT stage runs in the shared diffusion runtime. The default
|
||||
`mammoth_moda2.yaml` uses request-level batching with up to eight compatible
|
||||
requests (`max_num_seqs: 8`) and one image per request
|
||||
(`num_outputs_per_prompt: 1`). TeaCache and Cache-DiT are optional alternative
|
||||
cache backends in request mode. Enable TeaCache with `--cache-backend tea_cache`
|
||||
as shown below, or enable Cache-DiT using the commented settings in that YAML.
|
||||
Compilation, DiT quantization, parallelism, and offload are not enabled by
|
||||
these presets.
|
||||
|
||||
To use step execution and continuous batching, select
|
||||
[`mammoth_moda2_step.yaml`](../../vllm_omni/deploy/mammoth_moda2_step.yaml)
|
||||
with `--deploy-config vllm_omni/deploy/mammoth_moda2_step.yaml` in the
|
||||
text-to-image commands below. This preset inherits the same stage placement
|
||||
and capacity, enables step execution for Stage 1, and disables request-batch
|
||||
admission waiting and diffusion cache acceleration. Compatible requests can
|
||||
join between denoising steps and finish independently, including requests
|
||||
with different inference-step counts. Step mode cannot be combined with
|
||||
TeaCache, Cache-DiT, or other diffusion cache backends.
|
||||
|
||||
Image size, seed, guidance, and denoising steps use the standard diffusion
|
||||
request fields. `cfg_range` remains a MammothModa2-specific `extra_body`
|
||||
|
||||
Reference in new issue
Block a user