anodyine/vllm-omni
1
0 Fork 0
Code Issues 4 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

[Docs] Remove stale stage config references (#6270)

Browse Source 
Signed-off-by: zwhzzz0821 <2831474076@qq.com>
This commit is contained in:
Yukim1 authored and GitHub committed 2026-08-20 07:09:12 +08:00
1 parent efaf4276f9
commit 6f55a06fab
85 files changed
+566 -794
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/contributing/model/adding_omni_model.md
+25 -14
View File
@@ -9,7 +9,7 @@ This guide walks through the process of adding a new multi-stage model to vLLM-O
3. [Step-by-Step Implementation](#step-by-step-implementation)
4. [Key Components](#key-components)
5. [Model Registration](#model-registration)
6. [Stage Configuration](#stage-configuration)
6. [Pipeline and Deploy Configuration](#pipeline-and-deploy-configuration)
7. [Stage Input Processors](#stage-input-processors)
8. [Testing](#testing)
9. [Adding a Model Recipe](#adding-a-model-recipe)
@@ -316,18 +316,28 @@ _OMNI_MODELS = {
The registry uses lazy loading, so the model class is imported only when needed.
## Stage Configuration
## Pipeline and Deploy Configuration
Create a YAML configuration file in `vllm_omni/deploy/`. For a complete example, see the [Qwen3-Omni configuration file](gh-file:vllm_omni/deploy/qwen3_omni_moe.yaml).
Define fixed topology in `vllm_omni/model_executor/models/<model>/pipeline.py`
with `PipelineConfig` and `StagePipelineConfig`, then register it in
`vllm_omni/config/pipeline_registry.py`. For a complete example, see the
[Qwen3-Omni pipeline](gh-file:vllm_omni/model_executor/models/qwen3_omni/pipeline.py).
### Key Configuration Fields
### Pipeline topology fields
- **`model_stage`**: Which stage to run ("thinker", "talker", "code2wav", etc.)
- **`model_arch`**: The model architecture name (must match registry)
- **`engine_input_source`**: List of stage IDs that provide input to this stage
- **`custom_process_input_func`**: Function to process inputs from previous stages
- **`final_output`**: Whether this stage produces the final output (True/False)
- **`final_output_type`**: Type of final output ("text", "audio", "image", etc.)
- **`model_stage`**: Logical stage name ("thinker", "talker", "code2wav", etc.)
- **`execution_type`**: Runtime family (`LLM_AR`, `LLM_GENERATION`, or `DIFFUSION`)
- **`model_arch`**: Stage architecture name, when different from the pipeline default
- **`input_sources`**: Upstream stage IDs that provide input to this stage
- **`custom_process_input_func`**: Function that processes inputs from previous stages
- **`final_output`**: Whether this stage produces user-facing output
- **`final_output_type`**: Output modality ("text", "audio", "image", etc.)
- **`owns_tokenizer`**: Whether this stage owns the pipeline tokenizer
Create the matching runtime defaults in `vllm_omni/deploy/<model>.yaml` and set
`PipelineConfig.default_deploy_config_name` to that file. The deploy YAML owns
GPU placement, memory sizing, parallelism, connectors, and sampling defaults;
it must not redefine topology fields.
## Stage Input Processors
@@ -374,7 +384,7 @@ The stage transition process follows these steps:
2. **Transition Detection**: The orchestrator checks if there's a next stage and calls `process_engine_inputs()` on it
3. **Input Processing**: The stage input processor configured in stage YAML (under `vllm_omni/model_executor/stage_input_processors/`) handles the transition:
3. **Input Processing**: The stage input processor registered by the pipeline (implemented under `vllm_omni/model_executor/stage_input_processors/`) handles the transition:
```python
def process_engine_inputs(
self, stage_list: list[Any], prompt: OmniTokensPrompt | TextPrompt = None
@@ -577,7 +587,7 @@ Adding a new model to vLLM-Omni involves:
1. **Create model directory structure** with stage implementations
2. **Implement unified model class** that orchestrates stages
3. **Register model** in `registry.py`
4. **Create stage configuration** YAML file
4. **Define pipeline topology and deploy defaults** in `pipeline.py` and `vllm_omni/deploy/`
5. **Implement stage input processors** for stage transitions
6. **Write tests** to verify functionality
7. **Add model recipe** to the [vllm-project/recipes](https://github.com/vllm-project/recipes) repository (see [Adding a Model Recipe](#adding-a-model-recipe) section)
@@ -590,7 +600,8 @@ For a complete reference implementation, see:
- **Thinker**: `vllm_omni/model_executor/models/qwen3_omni/qwen3_omni_moe_thinker.py`
- **Talker**: `vllm_omni/model_executor/models/qwen3_omni/qwen3_omni_moe_talker.py`
- **Code2Wav**: `vllm_omni/model_executor/models/qwen3_omni/qwen3_omni_code2wav.py`
- **Stage config**: `vllm_omni/deploy/qwen3_omni_moe.yaml`
- **Pipeline topology**: `vllm_omni/model_executor/models/qwen3_omni/pipeline.py`
- **Deploy config**: `vllm_omni/deploy/qwen3_omni_moe.yaml`
- **Input processors**: `vllm_omni/model_executor/stage_input_processors/qwen3_omni.py`
- **Registry**: `vllm_omni/model_executor/models/registry.py`
- **Testing**: `vllm_omni/tests/e2e/offline_inference/test_qwen3_omni.py`
@@ -598,4 +609,4 @@ For a complete reference implementation, see:
For more information, see:
- [Architecture Overview](../../design/architecture_overview.md)
- [Supported Models](../../models/supported_models.md)
- [Stage Configuration Guide](../../configuration/stage_configs.md)
- [Pipeline and Deploy Configuration Guide](../../configuration/stage_configs.md)
docs/contributing/model/adding_tts_model.md
+92 -103
View File
@@ -20,7 +20,7 @@ pattern, refer to MOSS-TTS-Nano.
4. [Step-by-Step Implementation](#step-by-step-implementation)
5. [Key Components](#key-components)
6. [Model Registration](#model-registration)
7. [Stage Configuration](#stage-configuration)
7. [Pipeline and Deploy Configuration](#pipeline-and-deploy-configuration)
8. [Stage Input Processors](#stage-input-processors)
9. [Online Serving Integration](#online-serving-integration)
10. [Single-Stage Models](#single-stage-models)
@@ -427,112 +427,93 @@ _OMNI_MODELS = {
The registry uses lazy loading - model classes are only imported when needed.
## Stage Configuration
## Pipeline and Deploy Configuration
Each stage has a `worker_type` that determines how it is scheduled:
Define fixed topology in `models/<model>/pipeline.py`. The execution type
selects the scheduler and worker family:
- `worker_type: ar` - autoregressive stage, uses `OmniARScheduler` with PagedAttention
- `worker_type: generation` - non-AR stage (e.g. decoder), uses `OmniGenerationScheduler`
- `StageExecutionType.LLM_AR` uses the autoregressive runtime with PagedAttention.
- `StageExecutionType.LLM_GENERATION` uses the generation runtime for stages such as audio decoders.
Key configuration fields:
Key topology fields belong to `StagePipelineConfig`:
| Field | Description |
|-------|-------------|
| `model_stage` | Which stage to initialize (`ar_stage`, `decoder`, etc.) |
| `model_arch` | Architecture name, must match `registry.py` |
| `engine_input_source` | List of upstream stage IDs that provide input (e.g. `[0]`) |
| `engine_output_type` | Output type: `latent` for intermediate, `audio` for final |
| `custom_process_next_stage_input_func` | Async chunk processor function path (streaming only) |
| `final_output` | Whether this stage produces the final user-facing output |
| `final_output_type` | Type of final output (`audio`, `text`, etc.) |
| `model_stage` | Logical stage name (`ar_stage`, `decoder`, etc.). |
| `execution_type` | Runtime family for this stage. |
| `input_sources` | Upstream stage IDs that provide input. |
| `model_arch` | Stage-specific architecture name, when it differs from the pipeline default. |
| `engine_output_type` | Output representation such as `latent` or `audio`. |
| `custom_process_next_stage_input_func` | Processor for full-payload stage handoff. |
| `async_chunk_process_next_stage_input_func` | Processor for streaming chunk handoff. |
| `final_output` / `final_output_type` | Whether this stage produces user-facing output and its modality. |
| `owns_tokenizer` | Whether this stage supplies the pipeline tokenizer. |
!!! note
New in-tree models should define frozen topology in `models/<model>/pipeline.py`,
register it in `vllm_omni/config/pipeline_registry.py`, and put deployment
knobs in `vllm_omni/deploy/<model>.yaml`. Use `--deploy-config` to load a
custom deployment YAML.
For example, a two-stage TTS topology can be defined as:
### Batch mode
```python
from vllm_omni.config.stage_config import (
PipelineConfig,
StageExecutionType,
StagePipelineConfig,
)
```yaml
# stage_configs/your_model_name.yaml
_PROC = "vllm_omni.model_executor.stage_input_processors.your_model_name"
stage_args:
- stage_id: 0
stage_type: llm
runtime:
devices: "0"
engine_args:
model_stage: ar_stage
max_num_seqs: 64
model_arch: YourTTSModelForConditionalGeneration
worker_type: ar
scheduler_cls: vllm_omni.core.sched.omni_ar_scheduler.OmniARScheduler
engine_output_type: latent
default_sampling_params:
temperature: 0.9
top_k: 50
max_tokens: 2048
- stage_id: 1
stage_type: llm
runtime:
devices: "0"
engine_args:
model_stage: decoder
model_arch: YourTTSModelForConditionalGeneration
worker_type: generation
scheduler_cls: vllm_omni.core.sched.omni_generation_scheduler.OmniGenerationScheduler
engine_output_type: audio
engine_input_source: [0]
final_output: true
final_output_type: audio
YOUR_TTS_PIPELINE = PipelineConfig(
model_type="your_model_name",
default_deploy_config_name="your_model_name.yaml",
model_arch="YourTTSModelForConditionalGeneration",
stages=(
StagePipelineConfig(
stage_id=0,
model_stage="ar_stage",
execution_type=StageExecutionType.LLM_AR,
input_sources=(),
owns_tokenizer=True,
engine_output_type="latent",
custom_process_next_stage_input_func=f"{_PROC}.ar2decoder",
async_chunk_process_next_stage_input_func=f"{_PROC}.ar2decoder_async_chunk",
),
StagePipelineConfig(
stage_id=1,
model_stage="decoder",
execution_type=StageExecutionType.LLM_GENERATION,
input_sources=(0,),
model_arch="YourTTSDecoder",
engine_output_type="audio",
final_output=True,
final_output_type="audio",
),
),
)
```
### Streaming mode (async_chunk)
Add `async_chunk: true` at the top level and specify `custom_process_next_stage_input_func`
on Stage 0 to define how intermediate outputs are chunked and forwarded:
Register the pipeline in `vllm_omni/config/pipeline_registry.py`. Runtime
placement and sizing belong in `vllm_omni/deploy/your_model_name.yaml`:
```yaml
# stage_configs/your_model_name_async_chunk.yaml
async_chunk: true
stage_args:
stages:
- stage_id: 0
stage_type: llm
runtime:
devices: "0"
engine_args:
model_stage: ar_stage
max_num_seqs: 64
model_arch: YourTTSModelForConditionalGeneration
worker_type: ar
scheduler_cls: vllm_omni.core.sched.omni_ar_scheduler.OmniARScheduler
engine_output_type: latent
custom_process_next_stage_input_func: >
vllm_omni.model_executor.stage_input_processors.your_model_name.ar2decoder_async_chunk
devices: "0"
max_num_seqs: 64
default_sampling_params:
temperature: 0.9
top_k: 50
max_tokens: 2048
- stage_id: 1
stage_type: llm
runtime:
devices: "0"
engine_args:
model_stage: decoder
model_arch: YourTTSModelForConditionalGeneration
worker_type: generation
scheduler_cls: vllm_omni.core.sched.omni_generation_scheduler.OmniGenerationScheduler
engine_output_type: audio
engine_input_source: [0]
final_output: true
final_output_type: audio
devices: "0"
max_num_seqs: 1
```
Set `async_chunk: false` for end-to-end handoff. The registered pipeline
selects the full-payload or async chunk processor; processor paths and topology
must not be moved into the deploy YAML. Use `--deploy-config` to load a custom
deployment overlay.
## Stage Input Processors
Stage input processors convert Stage 0 outputs into Stage 1 inputs. Create yours in
@@ -832,32 +813,40 @@ vllm_omni/deploy/your_model_name.yaml
No stage input processor is needed.
### Stage config
### Pipeline and deploy config
Use a single stage with `worker_type: ar`. The `is_comprehension: true` field and the
top-level `async_chunk: false` are required — omitting them causes silent
misclassification in the serving layer. Set `max_num_seqs` to at least 4 for
concurrent production use.
Use a single `LLM_AR` stage that owns the tokenizer and produces final audio:
```python
YOUR_TTS_PIPELINE = PipelineConfig(
model_type="your_model_name",
default_deploy_config_name="your_model_name.yaml",
model_arch="YourModelForCausalLM",
stages=(
StagePipelineConfig(
stage_id=0,
model_stage="your_model_stage_key",
execution_type=StageExecutionType.LLM_AR,
input_sources=(),
owns_tokenizer=True,
engine_output_type="audio",
final_output=True,
final_output_type="audio",
),
),
)
```
The deploy YAML controls runtime behavior. Set `max_num_seqs` to at least 4
for concurrent production use:
```yaml
# stage_configs/your_model_name.yaml
async_chunk: false
stage_args:
stages:
- stage_id: 0
stage_type: llm
is_comprehension: true # required for serving_speech.py dispatch
runtime:
devices: "0"
engine_args:
model_stage: your_model_stage_key
model_arch: YourModelForCausalLM
worker_type: ar
scheduler_cls: vllm_omni.core.sched.omni_ar_scheduler.OmniARScheduler
engine_output_type: audio
max_num_seqs: 4 # min 4 for concurrent requests; default 1 causes gaps
final_output: true
final_output_type: audio
devices: "0"
max_num_seqs: 4
```
### Generator-based streaming pattern
@@ -1031,4 +1020,4 @@ For more information, see:
- [Architecture Overview](../../design/architecture_overview.md)
- [Async Chunk Design](../../design/feature/async_chunk.md)
- [Stage Configuration Guide](../../configuration/stage_configs.md)
- [Pipeline and Deploy Configuration Guide](../../configuration/stage_configs.md)