vllm-omni/docs/architecture/high_level_arch_design.md

vLLM-omni Architecture Design Document

Introduction

This document outlines the architectural design for vLLM-omni, a new repository developed under the vLLM project. vLLM-omni supports multi-modality models with non-autoregressive structures and non-textual outputs, extending vLLM beyond traditional text-based, autoregressive generation.

Goals

The primary goals for vLLM-omni are:

• Support for Non-textual Output: Enables the integration, efficient processing and output of various data types, including but not limited to, images, audio, and video, alongside text.
• Non-Autoregressive Structure: Support model structure beyond autoregressive, especially Diffusion Transformer (DiT), which is widely used in visual and audio generation.
• Integration with vLLM Core: Maintain compatibility and leverage existing vLLM key modules and optimizations where applicable.
• Extensibility: Design a modular and flexible architecture that can easily accommodate new modalities, model architectures, and output formats.

Expected Gains

• It breaks through the limitation of vLLM that only supports autoregressive text generation models, and provides architectural support for popular model structures, especially multimodal generation.
• By supporting new output formats and new model architectures, developers can easily connect to mature ecosystems such as Higress Gateway and Ray, improving the ease of deployment and efficiency of corresponding models.
• Integrating efficient cache management, multiple parallel strategies, disaggregation from vLLM, to improve inference efficiency of corresponding models.

Non-Autoregressive Structure

• Diffusion Transformer: initialized with a random latent noise, the transformer model iteratively predicts latent output for multiple steps, under control of sampling timestamps. Finally, it is decoded by the decoder of Variational AutoEncoder (VAE). It is not supported by vLLM now.
• Autoregressive: dominant generation paradigm for text, now supported by vLLM. It generates tokens conditioned on previous tokens one by one. The efficient KV Cache management provided by vLLM can accelerate inference effectively.
  

	Autoregressive	DiT
Scenarios	Text generation	Multimodal generation
Generation	Token-by-token, left to right	Continuous vector, iteratively refine
Key advantages	Smooth, good local consistency	Global consistency, diversity and controlability
Efficiency	More efficient with KV cache	Need generation repeatedly, slow
Error accumulation	Easy to accumulate	Can fix possibly
Key challenges	Limited global planning, long context generation	Large global computational cost
Sequence length	Varied length	Fixed length
Attention mask	Lower triangular	Full bi-directional matrix
Parallel strategy	TP, DP, PP, .etc	DP, CP, .etc
vLLM support	Supported	Unsupported

Targeted models to support

According to analysis for current popular open-source models, most of them have the combination of AR+DiT. Specifically, they can be further categorized into 3 types below:

• DiT as a main structure, with AR as text encoder (e.g.: Qwen-Image)
  A popular model for image generation and editing. Main visual generation models are similar to the structure, such as Flux.
  
• AR as a main structure, with DiT as multi-modal generator (e.g. BAGEL)
  A unified multimodal comprehension and generation model. Visual generation can utilize CoT text generation output. Main unified multimodal models are similar to the structure.
  
• Extension, multiple AR+DiT (e.g. Qwen-omni)
  A new model for multimodal input and output. It designs the Thinker-Talker-Codec structure, which is a double-AR+DiT format.
  

Architectural Overview

vLLM-omni will import core modules from existing vLLM framework and introduce several new components and modifications to achieve its goals. The core architecture will be based on a modular design, allowing for independent development and integration of different modality handlers and output generators.

General ideas

• It divides models in vLLM-omni into multiple stages, which are processed by different Engine Cores. Each stage can choose AR Engine Core for AR model part, and Diffusion Engine Core for diffusion model part.
• Designed a specialized diffusion engine core for DiT or dLLM.
• Plug-in with external DiT acceleration modules such as xDiT, Cache-DiT.
• Output linked with Higress API, Gradio for online demo, ComfyUI for less-code process.

Key Components

• Processor: Input processing, tokenization or encoding
• Scheduler: for order and queue of incoming requests
• Executor: Prepare model input and run
• Multimodal output processor: support multimodal outputs
• Hidden State Output processor: support intermediate hidden vectors as request output
• Engine Core: accelerate AR modules, imported from vLLM main repo
• Diffusion Engine Core: Wrap and accelerate DiT modules.
• Diffusion model: Wrap up diffusion models with vLLM format

Examples data flow of specific models

1. Qwen-omni
   
2. Qwen-Image
   

Overview of function call dependency

Roadmap

• AsyncLLM:

• Initializer: resource allocation for different stages (2025 H2)
• Hidden state Output Processor (2025 H2)
• Multimodal Output Processor (2025 H2)
• Interface for API, Gradio, ComfyUI

• OmniEngine:

• Basic Diffusion Engine core: Diffusion step manager, Cache manager (2025 H2)
• Pipeline of different Engine Cores (2025 H2)
• Communication between Engine Cores via AsyncLLM (2025 H2)

• Model support (Included but not limited to):

• DiT: Qwen-Image (2025 H2), Flux-Kontext
• Unified-MLLM: BAGEL (2025 H2), OmniGen2, Janus-Pro
• Extensions of AR+DiT: Qwen-omni (2025 H2)

• Optimization

• Integration of other DiT acceleration modules, such as xDiT, Cache-DiT
• Disaggregation for AR and DiT
• Integration with vLLM's existing KV cache for potential future hybrid models.
• Support for streaming multi-modal inputs and outputs

Contact

For any questions or feedback regarding this design document, please contact
Han Gao: hgaoaf@connect.ust.hk,
Hongsheng Liu: hsliuustc@gmail.com