[Docs] optimize and supplement docs system (#86)
Signed-off-by: Gaohan123 <gaohan19@huawei.com>
This commit is contained in:
Gao Han authored and GitHub committed
1 parent
38f58064eb
commit
db81ee9f69
75 files changed
+2452
-512
No files matched your search
@@ -0,0 +1,38 @@
|
||||
nav:
|
||||
- Home: README.md
|
||||
- User Guide:
|
||||
- Getting Started:
|
||||
- getting_started/quickstart.md
|
||||
- getting_started/installation
|
||||
- Examples:
|
||||
- examples/README.md
|
||||
- Offline Inference:
|
||||
- Qwen2.5-Omni: user_guide/examples/offline_inference/qwen2_5_omni.md
|
||||
- Online Serving: user_guide/examples/online_serving/README.md
|
||||
- General:
|
||||
- usage/*
|
||||
- Configuration:
|
||||
- configuration/README.md
|
||||
- configuration/*
|
||||
- Models:
|
||||
- models/supported_models.md
|
||||
- Features: features
|
||||
- Developer Guide:
|
||||
- contributing/README.md
|
||||
- General:
|
||||
- glob: contributing/*
|
||||
flatten_single_child_sections: true
|
||||
- Model Implementation:
|
||||
- contributing/model/README.md
|
||||
- CI: contributing/ci
|
||||
- Design Documents: design
|
||||
- Docs Guide: contributing/DOCS_GUIDE.md
|
||||
- API Reference:
|
||||
- api/README.md
|
||||
- api/vllm_omni
|
||||
- CLI Reference: cli
|
||||
- Community:
|
||||
- community/*
|
||||
- Blog: https://blog.vllm.ai
|
||||
- Forum: https://discuss.vllm.ai
|
||||
- Slack: https://slack.vllm.ai
|
||||
@@ -0,0 +1,60 @@
|
||||
---
|
||||
hide:
|
||||
- navigation
|
||||
- toc
|
||||
---
|
||||
|
||||
# Welcome to vLLM-Omni
|
||||
|
||||
<p align="center">
|
||||
<picture>
|
||||
<source media="(prefers-color-scheme: dark)" src="./source/logos/vllm-omni-logo-text-dark.png">
|
||||
<img alt="vllm-omni" src="./source/logos/vllm-omni-logo-text-dark.png" width=55%>
|
||||
</picture>
|
||||
</p>
|
||||
<h3 align="center">
|
||||
Easy, fast, and cheap omni-modality model serving for everyone
|
||||
</h3>
|
||||
|
||||
<p style="text-align:center">
|
||||
<script async defer src="https://buttons.github.io/buttons.js"></script>
|
||||
<a class="github-button" href="https://github.com/vllm-project/vllm-omni" data-show-count="true" data-size="large" aria-label="Star">Star</a>
|
||||
<a class="github-button" href="https://github.com/vllm-project/vllm-omni/subscription" data-show-count="true" data-icon="octicon-eye" data-size="large" aria-label="Watch">Watch</a>
|
||||
<a class="github-button" href="https://github.com/vllm-project/vllm-omni/fork" data-show-count="true" data-icon="octicon-repo-forked" data-size="large" aria-label="Fork">Fork</a>
|
||||
</p>
|
||||
|
||||
|
||||
## About
|
||||
|
||||
[vLLM](https://github.com/vllm-project/vllm) was originally designed to support large language models for text-based autoregressive generation tasks. vLLM-Omni extends its support for omni-modality model inference and serving:
|
||||
|
||||
- **Omni-modality**: Text, image, video, and audio data processing
|
||||
- **Non-autoregressive Architectures**: extend the AR support of vLLM to Diffusion Transformers (DiT) and other parallel generation models
|
||||
- **Heterogeneous outputs**: from traditional text generation to multimodal outputs
|
||||
|
||||
<p align="center">
|
||||
<picture>
|
||||
<img alt="vllm-omni" src="./source/architecture/omni-modality model architecture.png" width=55%>
|
||||
</picture>
|
||||
</p>
|
||||
|
||||
vLLM-omni is fast with:
|
||||
|
||||
- Seamless AR support for efficient KV Cache management with vLLM
|
||||
- Pipelined stage execution overlapping
|
||||
- Fully disaggregation based on omniConnector and dynamic resource allocation across stages
|
||||
|
||||
vLLM-omni is flexible and easy to use with:
|
||||
|
||||
- Seamless integration with popular Hugging Face models
|
||||
- High-throughput serving with various decoding algorithms, including *parallel sampling*, *beam search*, and more
|
||||
- Tensor, pipeline, data and expert parallelism support for distributed inference
|
||||
- Streaming outputs
|
||||
- OpenAI-compatible API server
|
||||
|
||||
vLLM-Omni seamlessly supports most popular open-source models on HuggingFace, including:
|
||||
|
||||
- Multi-modality generation models (e.g. Qwen-image)
|
||||
- Omni-modality models (e.g. Qwen-omni)
|
||||
|
||||
For more information, checkout the following:
|
||||
@@ -0,0 +1,88 @@
|
||||
# Summary
|
||||
|
||||
## Entry Points
|
||||
|
||||
Main entry points for vLLM-omni inference and serving.
|
||||
|
||||
- [vllm_omni.entrypoints.async_omni_llm.AsyncOmniLLM][]
|
||||
- [vllm_omni.entrypoints.async_omni_llm.AsyncOmniStageLLM][]
|
||||
- [vllm_omni.entrypoints.chat_utils.OmniAsyncMultiModalContentParser][]
|
||||
- [vllm_omni.entrypoints.chat_utils.OmniAsyncMultiModalItemTracker][]
|
||||
- [vllm_omni.entrypoints.chat_utils.parse_chat_messages_futures][]
|
||||
- [vllm_omni.entrypoints.cli.serve.OmniServeCommand][]
|
||||
- [vllm_omni.entrypoints.log_utils.OrchestratorMetrics][]
|
||||
- [vllm_omni.entrypoints.omni_llm.OmniLLM][]
|
||||
- [vllm_omni.entrypoints.omni_llm.OmniStageLLM][]
|
||||
- [vllm_omni.entrypoints.omni_stage.OmniStage][]
|
||||
- [vllm_omni.entrypoints.openai.serving_chat.OmniOpenAIServingChat][]
|
||||
|
||||
## Inputs
|
||||
|
||||
Input data structures for multi-modal inputs.
|
||||
|
||||
- [vllm_omni.inputs.data.OmniEmbedsPrompt][]
|
||||
- [vllm_omni.inputs.data.OmniTokenInputs][]
|
||||
- [vllm_omni.inputs.data.OmniTokensPrompt][]
|
||||
- [vllm_omni.inputs.parse.parse_singleton_prompt_omni][]
|
||||
- [vllm_omni.inputs.preprocess.OmniInputPreprocessor][]
|
||||
|
||||
## Engine
|
||||
|
||||
Engine classes for offline and online inference.
|
||||
|
||||
- [vllm_omni.engine.AdditionalInformationEntry][]
|
||||
- [vllm_omni.engine.AdditionalInformationPayload][]
|
||||
- [vllm_omni.engine.OmniEngineCoreRequest][]
|
||||
- [vllm_omni.engine.PromptEmbedsPayload][]
|
||||
- [vllm_omni.engine.arg_utils.AsyncOmniEngineArgs][]
|
||||
- [vllm_omni.engine.arg_utils.OmniEngineArgs][]
|
||||
- [vllm_omni.engine.output_processor.MultimodalOutputProcessor][]
|
||||
- [vllm_omni.engine.output_processor.OmniRequestState][]
|
||||
- [vllm_omni.engine.processor.OmniProcessor][]
|
||||
|
||||
## Core
|
||||
|
||||
Core scheduling and caching components.
|
||||
|
||||
- [vllm_omni.core.dit_cache_manager.DiTCacheManager][]
|
||||
- [vllm_omni.core.sched.diffusion_scheduler.DiffusionScheduler][]
|
||||
- [vllm_omni.core.sched.output.OmniNewRequestData][]
|
||||
- [vllm_omni.core.sched.scheduler.OmniScheduler][]
|
||||
|
||||
## Model Executor
|
||||
|
||||
Model execution components.
|
||||
|
||||
- [vllm_omni.model_executor.models.qwen2_5_omni.qwen2_5_omni.OmniOutput][]
|
||||
- [vllm_omni.model_executor.models.qwen2_5_omni.qwen2_5_omni.Qwen2_5OmniForConditionalGeneration][]
|
||||
- [vllm_omni.model_executor.models.qwen2_5_omni.qwen2_5_omni_talker.Qwen2_5OmniTalkerForConditionalGeneration][]
|
||||
- [vllm_omni.model_executor.models.qwen2_5_omni.qwen2_5_omni_thinker.Qwen2_5OmniAudioFeatureInputs][]
|
||||
- [vllm_omni.model_executor.models.qwen2_5_omni.qwen2_5_omni_thinker.Qwen2_5OmniConditionalGenerationMixin][]
|
||||
- [vllm_omni.model_executor.models.qwen2_5_omni.qwen2_5_omni_thinker.Qwen2_5OmniThinkerDummyInputsBuilder][]
|
||||
- [vllm_omni.model_executor.models.qwen2_5_omni.qwen2_5_omni_thinker.Qwen2_5OmniThinkerForConditionalGeneration][]
|
||||
- [vllm_omni.model_executor.models.qwen2_5_omni.qwen2_5_omni_thinker.Qwen2_5OmniThinkerMultiModalDataParser][]
|
||||
- [vllm_omni.model_executor.models.qwen2_5_omni.qwen2_5_omni_thinker.Qwen2_5OmniThinkerMultiModalProcessor][]
|
||||
- [vllm_omni.model_executor.models.qwen2_5_omni.qwen2_5_omni_thinker.Qwen2_5OmniThinkerProcessingInfo][]
|
||||
- [vllm_omni.model_executor.models.qwen2_5_omni.qwen2_5_omni_token2wav.Qwen2_5OmniToken2WavBigVGANModel][]
|
||||
- [vllm_omni.model_executor.models.qwen2_5_omni.qwen2_5_omni_token2wav.Qwen2_5OmniToken2WavDiTModel][]
|
||||
- [vllm_omni.model_executor.models.qwen2_5_omni.qwen2_5_omni_token2wav.Qwen2_5OmniToken2WavForConditionalGenerationVLLM][]
|
||||
- [vllm_omni.model_executor.models.qwen2_5_omni.qwen2_5_omni_token2wav.Qwen2_5OmniToken2WavModel][]
|
||||
- [vllm_omni.model_executor.models.qwen2_5_omni.qwen2_old.Qwen2EmbeddingModel][]
|
||||
- [vllm_omni.model_executor.models.qwen2_5_omni.qwen2_old.Qwen2ForCausalLM][]
|
||||
- [vllm_omni.model_executor.models.qwen2_5_omni.qwen2_old.Qwen2Model][]
|
||||
|
||||
## Configuration
|
||||
|
||||
Configuration classes.
|
||||
|
||||
- [vllm_omni.config.model.OmniModelConfig][]
|
||||
|
||||
## Workers
|
||||
|
||||
Worker classes and model runners for distributed inference.
|
||||
|
||||
- [vllm_omni.worker.gpu_ar_model_runner.GPUARModelRunner][]
|
||||
- [vllm_omni.worker.gpu_ar_worker.GPUARWorker][]
|
||||
- [vllm_omni.worker.gpu_diffusion_model_runner.GPUDiffusionModelRunner][]
|
||||
- [vllm_omni.worker.gpu_diffusion_worker.GPUDiffusionWorker][]
|
||||
- [vllm_omni.worker.gpu_model_runner.OmniGPUModelRunner][]
|
||||
@@ -1,68 +0,0 @@
|
||||
# Summary
|
||||
|
||||
## Entry Points
|
||||
|
||||
Main entry points for vLLM-omni inference and serving.
|
||||
|
||||
- [vllm_omni.entrypoints.omni_llm.OmniLLM][]
|
||||
- [vllm_omni.entrypoints.omni_llm.OmniStageLLM][]
|
||||
- [vllm_omni.entrypoints.omni_stage.OmniStage][]
|
||||
- [vllm_omni.entrypoints.cli.serve.OmniServeCommand][]
|
||||
|
||||
## Inputs
|
||||
|
||||
Input data structures for multi-modal inputs.
|
||||
|
||||
- [vllm_omni.inputs.data.OmniTokenInputs][]
|
||||
- [vllm_omni.inputs.data.OmniTokensPrompt][]
|
||||
- [vllm_omni.inputs.data.OmniEmbedsPrompt][]
|
||||
- [vllm_omni.inputs.parse.parse_singleton_prompt_omni][]
|
||||
- [vllm_omni.inputs.preprocess.OmniInputPreprocessor][]
|
||||
|
||||
## Engine
|
||||
|
||||
Engine classes for offline and online inference.
|
||||
|
||||
- [vllm_omni.engine.arg_utils.OmniEngineArgs][]
|
||||
- [vllm_omni.engine.processor.OmniProcessor][]
|
||||
- [vllm_omni.engine.output_processor.OmniRequestState][]
|
||||
- [vllm_omni.engine.output_processor.MultimodalOutputProcessor][]
|
||||
- [vllm_omni.engine.PromptEmbedsPayload][]
|
||||
- [vllm_omni.engine.AdditionalInformationPayload][]
|
||||
- [vllm_omni.engine.OmniEngineCoreRequest][]
|
||||
|
||||
## Core
|
||||
|
||||
Core scheduling and caching components.
|
||||
|
||||
- [vllm_omni.core.sched.scheduler.OmniScheduler][]
|
||||
- [vllm_omni.core.sched.diffusion_scheduler.DiffusionScheduler][]
|
||||
- [vllm_omni.core.sched.output.OmniNewRequestData][]
|
||||
- [vllm_omni.core.dit_cache_manager.DiTCacheManager][]
|
||||
|
||||
## Model Executor
|
||||
|
||||
Model execution components.
|
||||
|
||||
- [vllm_omni.model_executor.models.qwen2_5_omni.Qwen2_5OmniForConditionalGeneration][]
|
||||
- [vllm_omni.model_executor.models.qwen2_5_omni_talker.Qwen2_5OmniTalkerForConditionalGeneration][]
|
||||
- [vllm_omni.model_executor.models.qwen2_5_omni_thinker.Qwen2_5OmniThinkerForConditionalGeneration][]
|
||||
- [vllm_omni.model_executor.models.qwen2_5_omni_token2wav.Qwen2_5OmniToken2WavForConditionalGenerationVLLM][]
|
||||
- [vllm_omni.model_executor.layers.mrope.MRotaryEmbedding][]
|
||||
- [vllm_omni.model_executor.models.registry.OmniModelRegistry][]
|
||||
|
||||
## Configuration
|
||||
|
||||
Configuration classes.
|
||||
|
||||
- [vllm_omni.config.OmniModelConfig][]
|
||||
|
||||
## Workers
|
||||
|
||||
Worker classes and model runners for distributed inference.
|
||||
|
||||
- [vllm_omni.worker.gpu_ar_worker.GPUARWorker][]
|
||||
- [vllm_omni.worker.gpu_diffusion_worker.GPUDiffusionWorker][]
|
||||
- [vllm_omni.worker.gpu_model_runner.OmniGPUModelRunner][]
|
||||
- [vllm_omni.worker.gpu_ar_model_runner.GPUARModelRunner][]
|
||||
- [vllm_omni.worker.gpu_diffusion_model_runner.GPUDiffusionModelRunner][]
|
||||
@@ -0,0 +1,24 @@
|
||||
# vLLM-Omni CLI Guide
|
||||
|
||||
The CLI for vLLM-Omni inherits from vllm with some additional arguments.
|
||||
|
||||
## serve
|
||||
|
||||
Starts the vLLM-Omni OpenAI Compatible API server.
|
||||
|
||||
Start with a model:
|
||||
|
||||
```bash
|
||||
vllm serve Qwen/Qwen2.5-Omni-7B --omni
|
||||
```
|
||||
|
||||
Specify the port:
|
||||
|
||||
```bash
|
||||
vllm serve Qwen/Qwen2.5-Omni-7B --omni --port 8091
|
||||
```
|
||||
|
||||
If you have custom stage configs file, launch the server with command below
|
||||
```bash
|
||||
vllm serve Qwen/Qwen2.5-Omni-7B --omni --stage-configs-path /path/to/stage_configs_file
|
||||
```
|
||||
@@ -0,0 +1,6 @@
|
||||
# Contact Us
|
||||
|
||||
- For technical questions and feature requests, please use GitHub [Issues](https://github.com/vllm-project/vllm-omni/issues)
|
||||
- For discussing with fellow users, please use the [vLLM Forum](https://discuss.vllm.ai/)
|
||||
- For coordinating contributions and development, please use [Slack](https://slack.vllm.ai/)
|
||||
- For security disclosures, please use GitHub's [Security Advisories](https://github.com/vllm-project/vllm/security/advisories) feature
|
||||
@@ -0,0 +1 @@
|
||||
# Meetups
|
||||
@@ -0,0 +1,11 @@
|
||||
# Configuration Options
|
||||
|
||||
This section lists the most common options for running vLLM-Omni.
|
||||
|
||||
For options within a vLLM Engine. Please refer to [vLLM Configuration](https://docs.vllm.ai/en/v0.11.0/configuration/index.html)
|
||||
|
||||
Currently, the main options are maintained by stage configs for each model.
|
||||
|
||||
For specific example, please refer to [Qwen2.5-omni stage config](stage_configs/qwen2_5_omni.yaml)
|
||||
|
||||
For introduction, please check [Introduction for stage config](./stage_configs.md)
|
||||
@@ -0,0 +1,267 @@
|
||||
# Stage configs for vLLM-Omni
|
||||
|
||||
In vLLM-Omni, it divides models in vLLM-omni into multiple stages, which are processed by different LLMEngines, DiffusionEngines or other kinds of engines. According to different types of stages, such as AR stage or DiT stage, each stage can choose corresponding schedulers, model workers to load into Engines with a plug-in way.
|
||||
|
||||
Therefore, the stage configs for a model have several main functions:
|
||||
|
||||
- Claim partition of stages and corresponding class implementation in the folder model_executor/models.
|
||||
- The disaggregated configuration for each stage and the communication topology among them.
|
||||
- Engine arguments for each engine within the stage.
|
||||
- Input and output dependencies for each stage.
|
||||
- Default input parameters.
|
||||
|
||||
If users want to modify some part of it. The custom stage_configs file can be input as input argument in both online and offline. Just like examples below:
|
||||
|
||||
For offline (Assume necessary dependencies have ben imported):
|
||||
```python
|
||||
model_name = "Qwen/Qwen2.5-Omni-7B"
|
||||
omni_llm = OmniLLM(model=model_name, stage_configs_path="/path/to/custom_stage_configs.yaml")
|
||||
```
|
||||
|
||||
For online serving:
|
||||
```bash
|
||||
vllm serve Qwen/Qwen2.5-Omni-7B --omni --port 8091 --stage-configs-path /path/to/stage_configs_file
|
||||
```
|
||||
|
||||
Below is a specific example of stage_configs.yaml in Qwen2.5-omni.
|
||||
```python
|
||||
# stage config for running qwen2.5-omni with architecture of OmniLLM.
|
||||
stage_args:
|
||||
- stage_id: 0 # mark the unique id for each stage
|
||||
runtime: # The disaggregated configuration
|
||||
process: true # Run this stage in a separate process
|
||||
devices: "0" # Visible devices for this stage (CUDA_VISIBLE_DEVICES/torch.cuda.set_device)
|
||||
max_batch_size: 1 # the batch_size for offline inference
|
||||
engine_args: # Engine arguments for a certain engine
|
||||
model_stage: thinker
|
||||
model_arch: Qwen2_5OmniForConditionalGeneration # The model implementation registered in model_executor/models/registry.py
|
||||
worker_cls: vllm_omni.worker.gpu_ar_worker.GPUARWorker # The specific worker used
|
||||
scheduler_cls: vllm_omni.core.sched.scheduler.OmniScheduler # The specific scehduler used
|
||||
gpu_memory_utilization: 0.8 # The gpu memory allocation for the stage within a single chip
|
||||
enforce_eager: true # Now we only support eager mode
|
||||
trust_remote_code: true # Needed by huggingface config parsing
|
||||
engine_output_type: latent # It claims that the stage will input latent hiddenstates besides token ids
|
||||
enable_prefix_caching: false # For request with hiddenstates output, the prefix caching is not supported now
|
||||
is_comprehension: true # If the stage is a text or multimodal comprehension module. If it is, the AsyncOmniLLM will use its tokenizer as default
|
||||
final_output: true # If the stage has output as part of final outputs. If it is false, which means that the stage only works as a intermediate role.
|
||||
final_output_type: text # What is the final output type. It can be text and audio now.
|
||||
default_sampling_params: # sampling parameters for the stage. Their meaning aligns with vLLM.
|
||||
temperature: 0.0
|
||||
top_p: 1.0
|
||||
top_k: -1
|
||||
max_tokens: 2048
|
||||
seed: 42
|
||||
detokenize: True
|
||||
repetition_penalty: 1.1
|
||||
- stage_id: 1
|
||||
runtime:
|
||||
process: true
|
||||
devices: "1"
|
||||
max_batch_size: 3
|
||||
engine_args:
|
||||
model_stage: talker
|
||||
model_arch: Qwen2_5OmniForConditionalGeneration
|
||||
worker_cls: vllm_omni.worker.gpu_ar_worker.GPUARWorker
|
||||
scheduler_cls: vllm_omni.core.sched.scheduler.OmniScheduler
|
||||
gpu_memory_utilization: 0.8
|
||||
enforce_eager: true
|
||||
trust_remote_code: true
|
||||
enable_prefix_caching: false
|
||||
engine_output_type: latent
|
||||
engine_input_source: [0]
|
||||
custom_process_input_func: vllm_omni.model_executor.stage_input_processors.qwen2_5_omni.thinker2talker
|
||||
default_sampling_params:
|
||||
temperature: 0.9
|
||||
top_p: 0.8
|
||||
top_k: 40
|
||||
max_tokens: 2048
|
||||
seed: 42
|
||||
detokenize: True
|
||||
repetition_penalty: 1.05
|
||||
stop_token_ids: [8294]
|
||||
- stage_id: 2
|
||||
runtime:
|
||||
process: true
|
||||
devices: "0" # Example: use a different GPU than the previous stage; use "0" if single GPU
|
||||
max_batch_size: 1
|
||||
engine_args:
|
||||
model_stage: code2wav
|
||||
model_arch: Qwen2_5OmniForConditionalGeneration
|
||||
worker_cls: vllm_omni.worker.gpu_diffusion_worker.GPUDiffusionWorker
|
||||
scheduler_cls: vllm_omni.core.sched.diffusion_scheduler.DiffusionScheduler
|
||||
gpu_memory_utilization: 0.15
|
||||
enforce_eager: true
|
||||
trust_remote_code: true
|
||||
enable_prefix_caching: false
|
||||
engine_output_type: audio
|
||||
engine_input_source: [1]
|
||||
final_output: true
|
||||
final_output_type: audio
|
||||
default_sampling_params:
|
||||
temperature: 0.0
|
||||
top_p: 1.0
|
||||
top_k: -1
|
||||
max_tokens: 2048
|
||||
seed: 42
|
||||
detokenize: True
|
||||
repetition_penalty: 1.1
|
||||
|
||||
# Top-level runtime config (concise): default windows and stage edges
|
||||
runtime:
|
||||
enabled: true
|
||||
defaults:
|
||||
window_size: -1 # Simplified: trigger downstream only after full upstream completion
|
||||
max_inflight: 1 # Simplified: process serially within each stage
|
||||
edges:
|
||||
- from: 0 # thinker → talker: trigger only after receiving full input (-1)
|
||||
to: 1
|
||||
window_size: -1
|
||||
- from: 1 # talker → code2wav: trigger only after receiving full input (-1)
|
||||
to: 2
|
||||
window_size: -1
|
||||
|
||||
```
|
||||
|
||||
## Stage Configuration Arguments
|
||||
|
||||
Each stage in the `stage_args` list contains the following configuration options:
|
||||
|
||||
### `stage_id`
|
||||
|
||||
A unique identifier for each stage in the multi-stage pipeline. Stages are numbered sequentially starting from 0, and this ID is used to reference stages in inter-stage dependencies (e.g., `engine_input_source`).
|
||||
|
||||
### `runtime`
|
||||
|
||||
Configuration for disaggregated execution of the stage, controlling how the stage is deployed and executed.
|
||||
|
||||
#### `runtime.process`
|
||||
|
||||
Whether to run this stage in a separate process. When set to `true`, the stage will be executed in an isolated process, enabling better resource isolation and parallel execution across different stages. This is essential for multi-GPU deployments where different stages run on different devices.
|
||||
|
||||
Default: `true`
|
||||
|
||||
#### `runtime.devices`
|
||||
|
||||
Visible devices for this stage, specified as a string. This controls which GPU devices are available to the stage process, similar to setting `CUDA_VISIBLE_DEVICES` or using `torch.cuda.set_device()`. For example, `"0"` uses GPU 0, `"1"` uses GPU 1, and `"0,1"` makes both GPUs 0 and 1 visible.
|
||||
|
||||
Default: `"0"`
|
||||
|
||||
#### `runtime.max_batch_size`
|
||||
|
||||
The maximum batch size for offline inference in this stage. This limits how many sequences can be processed together in a single batch during offline inference operations.
|
||||
|
||||
Default: `1`
|
||||
|
||||
### `engine_args`
|
||||
|
||||
Engine arguments for configuring the LLM engine, diffusion engine, or other engine types used by this stage.
|
||||
|
||||
#### `engine_args.model_stage`
|
||||
|
||||
The name identifier for this model stage within the multi-stage architecture. This is used internally to distinguish different stages of the same model (e.g., "thinker", "talker", "code2wav" in Qwen2.5-Omni).
|
||||
|
||||
#### `engine_args.model_arch`
|
||||
|
||||
The model architecture class name that is registered in `model_executor/models/registry.py`. This specifies which model implementation to use for this stage. The class must be registered in the model registry for vLLM-Omni to locate and instantiate it.
|
||||
|
||||
#### `engine_args.worker_cls`
|
||||
|
||||
The specific worker class to use for this stage. This determines how the model computations are executed. Examples include `vllm_omni.worker.gpu_ar_worker.GPUARWorker` for autoregressive stages and `vllm_omni.worker.gpu_diffusion_worker.GPUDiffusionWorker` for diffusion-based stages.
|
||||
|
||||
#### `engine_args.scheduler_cls`
|
||||
|
||||
The scheduler class to use for this stage. The scheduler manages request queuing, batching, and execution order. Examples include `vllm_omni.core.sched.scheduler.OmniScheduler` for standard stages and `vllm_omni.core.sched.diffusion_scheduler.DiffusionScheduler` for diffusion stages.
|
||||
|
||||
#### `engine_args.gpu_memory_utilization`
|
||||
|
||||
The fraction of GPU memory to allocate for this stage within a single GPU chip. This is a value between 0.0 and 1.0, where 0.8 means 80% of the GPU memory will be used by this stage. This allows fine-grained control over memory allocation when multiple stages share the same GPU or when reserving memory for other operations.
|
||||
|
||||
Default: `0.8`
|
||||
|
||||
#### `engine_args.enforce_eager`
|
||||
|
||||
Whether to enforce eager execution mode. When set to `true`, the engine will run in eager mode without using CUDA graphs or other compilation optimizations. Currently, vLLM-Omni only supports eager mode.
|
||||
|
||||
Default: `true`
|
||||
|
||||
#### `engine_args.trust_remote_code`
|
||||
|
||||
Whether to trust remote code when loading models from Hugging Face. This is required for models that use custom code in their configuration files. Set to `true` when loading models that require custom model implementations.
|
||||
|
||||
Default: `true`
|
||||
|
||||
#### `engine_args.engine_output_type`
|
||||
|
||||
Specifies the type of output produced by this stage's engine. This determines what kind of data flows to downstream stages. Possible values include `latent` (hidden states), `text` (tokenized text), and `audio` (audio waveforms). When set to `latent`, the stage outputs latent hidden states in addition to token IDs, which are consumed by downstream stages.
|
||||
|
||||
Default: `latent`
|
||||
|
||||
#### `engine_args.enable_prefix_caching`
|
||||
|
||||
Whether to enable prefix caching for this stage. Prefix caching can improve performance by caching KV cache for common prompt prefixes. However, for requests that output hidden states (when `engine_output_type` is `latent`), prefix caching is not currently supported and should be set to `false`.
|
||||
|
||||
Default: `false`
|
||||
|
||||
### `is_comprehension`
|
||||
|
||||
Whether this stage is a text or multimodal comprehension module. When set to `true`, the stage acts as a comprehension module that processes input text or multimodal content. If this is the first comprehension stage, `AsyncOmniLLM` will use its tokenizer as the default tokenizer for the entire pipeline.
|
||||
|
||||
Default: `true`
|
||||
|
||||
### `final_output`
|
||||
|
||||
Whether this stage produces output that is part of the final outputs returned to the user. When set to `false`, the stage only works as an intermediate stage, processing data that flows to downstream stages but not contributing directly to the final response.
|
||||
|
||||
Default: `true`
|
||||
|
||||
### `final_output_type`
|
||||
|
||||
The type of final output produced by this stage. This specifies what format the output will be in when returned to the user. Currently supported values are `text` (for text generation) and `audio` (for audio generation).
|
||||
|
||||
Default: `text`
|
||||
|
||||
### `default_sampling_params`
|
||||
|
||||
Default sampling parameters for this stage. These parameters control the generation behavior and align with vLLM's sampling parameter semantics. These defaults are used when no explicit sampling parameters are provided in the request.
|
||||
|
||||
#### `default_sampling_params.temperature`
|
||||
|
||||
Sampling temperature for controlling randomness. Lower values (e.g., 0.0) make the output more deterministic and focused, while higher values increase randomness.
|
||||
|
||||
Default: `0.0`
|
||||
|
||||
#### `default_sampling_params.top_p`
|
||||
|
||||
Nucleus sampling parameter. Only tokens with cumulative probability mass up to `top_p` are considered. This helps filter out low-probability tokens while maintaining diversity.
|
||||
|
||||
Default: `1.0`
|
||||
|
||||
#### `default_sampling_params.top_k`
|
||||
|
||||
Top-k sampling parameter. Only the top `k` most likely tokens are considered. Set to `-1` to disable top-k filtering and consider all tokens.
|
||||
|
||||
Default: `-1`
|
||||
|
||||
#### `default_sampling_params.max_tokens`
|
||||
|
||||
Maximum number of tokens to generate in this stage. This limits the length of the output sequence.
|
||||
|
||||
Default: `2048`
|
||||
|
||||
#### `default_sampling_params.seed`
|
||||
|
||||
Random seed for reproducible generation. When set, the random number generator will be initialized with this seed to ensure consistent outputs across runs.
|
||||
|
||||
Default: `42`
|
||||
|
||||
#### `default_sampling_params.detokenize`
|
||||
|
||||
Whether to detokenize the output tokens into text. When set to `true`, token IDs are converted back to readable text strings.
|
||||
|
||||
Default: `True`
|
||||
|
||||
#### `default_sampling_params.repetition_penalty`
|
||||
|
||||
Penalty applied to tokens that have already appeared in the generated sequence. Values greater than 1.0 discourage repetition, while values less than 1.0 encourage it. A value of 1.0 applies no penalty.
|
||||
|
||||
Default: `1.1`
|
||||
@@ -0,0 +1,94 @@
|
||||
# stage config for running qwen2.5-omni with architecture of OmniLLM.
|
||||
stage_args:
|
||||
- stage_id: 0
|
||||
runtime:
|
||||
process: true # Run this stage in a separate process
|
||||
devices: "0" # Visible devices for this stage (CUDA_VISIBLE_DEVICES/torch.cuda.set_device)
|
||||
max_batch_size: 1
|
||||
engine_args:
|
||||
model_stage: thinker
|
||||
model_arch: Qwen2_5OmniForConditionalGeneration
|
||||
worker_cls: vllm_omni.worker.gpu_ar_worker.GPUARWorker
|
||||
scheduler_cls: vllm_omni.core.sched.scheduler.OmniScheduler
|
||||
gpu_memory_utilization: 0.8
|
||||
enforce_eager: true # Now we only support eager mode
|
||||
trust_remote_code: true
|
||||
engine_output_type: latent
|
||||
enable_prefix_caching: false
|
||||
is_comprehension: true
|
||||
final_output: true
|
||||
final_output_type: text
|
||||
default_sampling_params:
|
||||
temperature: 0.0
|
||||
top_p: 1.0
|
||||
top_k: -1
|
||||
max_tokens: 2048
|
||||
seed: 42
|
||||
detokenize: True
|
||||
repetition_penalty: 1.1
|
||||
- stage_id: 1
|
||||
runtime:
|
||||
process: true
|
||||
devices: "1"
|
||||
max_batch_size: 3
|
||||
engine_args:
|
||||
model_stage: talker
|
||||
model_arch: Qwen2_5OmniForConditionalGeneration
|
||||
worker_cls: vllm_omni.worker.gpu_ar_worker.GPUARWorker
|
||||
scheduler_cls: vllm_omni.core.sched.scheduler.OmniScheduler
|
||||
gpu_memory_utilization: 0.8
|
||||
enforce_eager: true
|
||||
trust_remote_code: true
|
||||
enable_prefix_caching: false
|
||||
engine_output_type: latent
|
||||
engine_input_source: [0]
|
||||
custom_process_input_func: vllm_omni.model_executor.stage_input_processors.qwen2_5_omni.thinker2talker
|
||||
default_sampling_params:
|
||||
temperature: 0.9
|
||||
top_p: 0.8
|
||||
top_k: 40
|
||||
max_tokens: 2048
|
||||
seed: 42
|
||||
detokenize: True
|
||||
repetition_penalty: 1.05
|
||||
stop_token_ids: [8294]
|
||||
- stage_id: 2
|
||||
runtime:
|
||||
process: true
|
||||
devices: "0" # Example: use a different GPU than the previous stage; use "0" if single GPU
|
||||
max_batch_size: 1
|
||||
engine_args:
|
||||
model_stage: code2wav
|
||||
model_arch: Qwen2_5OmniForConditionalGeneration
|
||||
worker_cls: vllm_omni.worker.gpu_diffusion_worker.GPUDiffusionWorker
|
||||
scheduler_cls: vllm_omni.core.sched.diffusion_scheduler.DiffusionScheduler
|
||||
gpu_memory_utilization: 0.15
|
||||
enforce_eager: true
|
||||
trust_remote_code: true
|
||||
enable_prefix_caching: false
|
||||
engine_output_type: audio
|
||||
engine_input_source: [1]
|
||||
final_output: true
|
||||
final_output_type: audio
|
||||
default_sampling_params:
|
||||
temperature: 0.0
|
||||
top_p: 1.0
|
||||
top_k: -1
|
||||
max_tokens: 2048
|
||||
seed: 42
|
||||
detokenize: True
|
||||
repetition_penalty: 1.1
|
||||
|
||||
# Top-level runtime config (concise): default windows and stage edges
|
||||
runtime:
|
||||
enabled: true
|
||||
defaults:
|
||||
window_size: -1 # Simplified: trigger downstream only after full upstream completion
|
||||
max_inflight: 1 # Simplified: process serially within each stage
|
||||
edges:
|
||||
- from: 0 # thinker → talker: trigger only after receiving full input (-1)
|
||||
to: 1
|
||||
window_size: -1
|
||||
- from: 1 # talker → code2wav: trigger only after receiving full input (-1)
|
||||
to: 2
|
||||
window_size: -1
|
||||
File renamed without changes.
@@ -4,7 +4,7 @@ Thank you for your interest in contributing to vLLM-omni! This document provides
|
||||
|
||||
## Getting Started
|
||||
|
||||
### Development Environment
|
||||
### Development Environment for vLLM
|
||||
|
||||
vLLM-omni uses `uv` as the environment manager, to create and manage Python environments. Please follow the documentation to install `uv`. After installing `uv`, you can create a new Python environment using the following commands:
|
||||
|
||||
@@ -19,8 +19,24 @@ Install vLLM from source:
|
||||
git clone https://github.com/vllm-project/vllm.git
|
||||
cd vllm
|
||||
git checkout v0.11.0
|
||||
VLLM_USE_PRECOMPILED=1 uv pip install -e .
|
||||
```
|
||||
Set up environment variables to get pre-built wheels. If there are internet problems, just download the whl file manually. And set VLLM_PRECOMPILED_WHEEL_LOCATION as your local absolute path of whl file.
|
||||
```bash
|
||||
export VLLM_PRECOMPILED_WHEEL_LOCATION=https://github.com/vllm-project/vllm/releases/download/v0.11.0/vllm-0.11.0-cp38-abi3-manylinux1_x86_64.whl
|
||||
```
|
||||
Install vllm with command below (If you have no existing PyTorch).
|
||||
```bash
|
||||
uv pip install --editable .
|
||||
```
|
||||
|
||||
Install vllm with command below (If you already have PyTorch).
|
||||
```bash
|
||||
python use_existing_torch.py
|
||||
uv pip install -r requirements/build.txt
|
||||
uv pip install --no-build-isolation --editable .
|
||||
```
|
||||
|
||||
### Development Environment for vLLM-Omni
|
||||
|
||||
Install vLLM-omni from source:
|
||||
|
||||
@@ -49,7 +65,7 @@ vLLM-omni's `pre-commit` hooks will now run automatically every time you commit.
|
||||
|
||||
```bash
|
||||
pre-commit run # runs on staged files
|
||||
pre-commit run -a # runs on all files (short for --all-files)
|
||||
pre-commit run --show-diff-on-failure --color=always --all-files # runs on all files (short for --all-files)
|
||||
```
|
||||
|
||||
### Documentation
|
||||
@@ -163,7 +179,7 @@ The goal of the vLLM-omni team is to be a _transparent reviewing machine_. We wo
|
||||
|
||||
## Additional Resources
|
||||
|
||||
- [Design Documents](design_documents/index.md) - Architecture and design documentation
|
||||
- [Design Documents](../design/index.md) - Architecture and design documentation
|
||||
|
||||
## Thank You
|
||||
|
||||
|
||||
@@ -0,0 +1,4 @@
|
||||
# CI Failures
|
||||
|
||||
What should I do when a CI job fails on my PR, but I don't think my PR caused
|
||||
the failure?
|
||||
@@ -0,0 +1 @@
|
||||
# Summary
|
||||
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
@@ -0,0 +1,6 @@
|
||||
# Examples
|
||||
|
||||
vLLM-Omni's examples are split into two categories:
|
||||
|
||||
- If you are using vLLM-Omni from within Python code, see the *Offline Inference* section.
|
||||
- If you are using vLLM-Omni from an HTTP application or client, see the *Online Serving* section.
|
||||
@@ -0,0 +1,3 @@
|
||||
# Features
|
||||
|
||||
vLLM-Omni introduces new features to support multimodal comprehension and generation models inference and serving.
|
||||
@@ -0,0 +1,4 @@
|
||||
nav:
|
||||
- README.md
|
||||
- gpu.md
|
||||
- npu.md
|
||||
@@ -0,0 +1,7 @@
|
||||
# Installation
|
||||
|
||||
vLLM supports the following hardware platforms:
|
||||
|
||||
- [GPU](gpu.md)
|
||||
- [NVIDIA CUDA](gpu.md#nvidia-cuda)
|
||||
- [Ascend NPU](npu.md)
|
||||
@@ -0,0 +1,49 @@
|
||||
# GPU
|
||||
|
||||
vLLM-Omni is a Python library that supports the following GPU variants. Select your GPU type to see vendor specific instructions:
|
||||
|
||||
[](){ #nvidia-cuda }
|
||||
|
||||
=== "NVIDIA CUDA"
|
||||
|
||||
--8<-- "docs/getting_started/installation/gpu/cuda.inc.md:installation"
|
||||
|
||||
## Requirements
|
||||
|
||||
- OS: Linux
|
||||
- Python: 3.12
|
||||
|
||||
!!! note
|
||||
vLLM-Omni does not support Windows natively.
|
||||
|
||||
=== "NVIDIA CUDA"
|
||||
|
||||
--8<-- "docs/getting_started/installation/gpu/cuda.inc.md:requirements"
|
||||
|
||||
## Set up using Python
|
||||
|
||||
### Create a new Python environment
|
||||
|
||||
--8<-- "docs/getting_started/installation/python_env_setup.inc.md"
|
||||
|
||||
=== "NVIDIA CUDA"
|
||||
|
||||
--8<-- "docs/getting_started/installation/gpu/cuda.inc.md:set-up-using-python"
|
||||
|
||||
### Pre-built wheels
|
||||
|
||||
=== "NVIDIA CUDA"
|
||||
|
||||
--8<-- "docs/getting_started/installation/gpu/cuda.inc.md:pre-built-wheels"
|
||||
|
||||
[](){ #build-from-source }
|
||||
|
||||
### Build wheel from source
|
||||
|
||||
=== "NVIDIA CUDA"
|
||||
|
||||
--8<-- "docs/getting_started/installation/gpu/cuda.inc.md:build-wheel-from-source"
|
||||
|
||||
## Set up using Docker
|
||||
|
||||
### Pre-built images
|
||||
@@ -0,0 +1,73 @@
|
||||
# --8<-- [start:installation]
|
||||
|
||||
vLLM-Omni mainly contains python implementations for framework and models.
|
||||
|
||||
# --8<-- [end:installation]
|
||||
# --8<-- [start:requirements]
|
||||
|
||||
- GPU: compute capability 7.0 or higher (e.g., V100, T4, RTX20xx, A100, L4, H100, etc.)
|
||||
|
||||
# --8<-- [end:requirements]
|
||||
# --8<-- [start:set-up-using-python]
|
||||
|
||||
vLLM-Omni depends vLLM. So please follow instructions below mainly for vLLM.
|
||||
|
||||
!!! note
|
||||
PyTorch installed via `conda` will statically link `NCCL` library, which can cause issues when vLLM tries to use `NCCL`. See <gh-issue:8420> for more details.
|
||||
|
||||
In order to be performant, vLLM has to compile many cuda kernels. The compilation unfortunately introduces binary incompatibility with other CUDA versions and PyTorch versions, even for the same PyTorch version with different building configurations.
|
||||
|
||||
Therefore, it is recommended to install vLLM and vLLM-Omni with a **fresh new** environment. If either you have a different CUDA version or you want to use an existing PyTorch installation, you need to build vLLM from source. See [build-from-source-vllm](https://docs.vllm.ai/en/stable/getting_started/installation/gpu/#build-wheel-from-source) for more details.
|
||||
|
||||
# --8<-- [start:pre-built-wheels]
|
||||
|
||||
## Installation of vLLM
|
||||
|
||||
Now we build it based on vLLM v0.11.0. Please install it with command below.
|
||||
```bash
|
||||
uv pip install vllm==0.11.0 --torch-backend=auto
|
||||
```
|
||||
|
||||
## Installation of vLLM-Omni
|
||||
|
||||
```bash
|
||||
uv pip install vllm-omni
|
||||
```
|
||||
|
||||
# --8<-- [end:pre-built-wheels]
|
||||
|
||||
# --8<-- [start:build-wheel-from-source]
|
||||
|
||||
## Installation of vLLM
|
||||
If you want to check or debug with source code of vLLM, install stable release version of vllm with 0.11.0 from source with pre-built wheel file.
|
||||
|
||||
```bash
|
||||
git clone https://github.com/vllm-project/vllm.git
|
||||
cd vllm
|
||||
git checkout v0.11.0
|
||||
```
|
||||
Set up environment variables to get pre-built wheels. If there are internet problems, just download the whl file manually. And set VLLM_PRECOMPILED_WHEEL_LOCATION as your local absolute path of whl file.
|
||||
```bash
|
||||
export VLLM_PRECOMPILED_WHEEL_LOCATION=https://github.com/vllm-project/vllm/releases/download/v0.11.0/vllm-0.11.0-cp38-abi3-manylinux1_x86_64.whl
|
||||
```
|
||||
Install vllm with command below (If you have no existing PyTorch).
|
||||
```bash
|
||||
uv pip install --editable .
|
||||
```
|
||||
|
||||
Install vllm with command below (If you already have PyTorch).
|
||||
```bash
|
||||
python use_existing_torch.py
|
||||
uv pip install -r requirements/build.txt
|
||||
uv pip install --no-build-isolation --editable .
|
||||
```
|
||||
|
||||
## Installation of vLLM-omni
|
||||
Install additional requirements for vllm-omni
|
||||
```bash
|
||||
git clone https://github.com/vllm-project/vllm-omni.git
|
||||
cd vllm_omni
|
||||
uv pip install -e .
|
||||
```
|
||||
|
||||
# --8<-- [end:build-wheel-from-source]
|
||||
@@ -0,0 +1,3 @@
|
||||
# Ascend-NPU
|
||||
|
||||
vLLM-Omni is a Python library that supports the following NPU variants. Select your NPU type to see vendor specific instructions:
|
||||
@@ -0,0 +1,5 @@
|
||||
# --8<-- [start:installation]
|
||||
|
||||
vLLM-Omni mainly contains python implementations for framework and models.
|
||||
|
||||
# --8<-- [end:installation]
|
||||
@@ -0,0 +1,6 @@
|
||||
It's recommended to use [uv](https://docs.astral.sh/uv/), a very fast Python environment manager, to create and manage Python environments. Please follow the [documentation](https://docs.astral.sh/uv/#getting-started) to install `uv`. After installing `uv`, you can create a new Python environment using the following commands:
|
||||
|
||||
```bash
|
||||
uv venv --python 3.12 --seed
|
||||
source .venv/bin/activate
|
||||
```
|
||||
@@ -0,0 +1,23 @@
|
||||
# Quickstart
|
||||
|
||||
This guide will help you quickly get started with vLLM-Omni to perform:
|
||||
|
||||
- Offline batched inference
|
||||
- Online serving using OpenAI-compatible server
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- OS: Linux
|
||||
- Python: 3.12
|
||||
|
||||
## Installation
|
||||
|
||||
Please refer to [installation](installation/README.md)
|
||||
|
||||
## Offline Inference
|
||||
|
||||
Please refer to [offline inference](../user_guide/examples/offline_inference/qwen2_5_omni.md)
|
||||
|
||||
## Online Serving with OpenAI-Completions API
|
||||
|
||||
Please refer to [online serving](../user_guide/examples/online_serving/README.md)
|
||||
@@ -1,32 +0,0 @@
|
||||
# Welcome to vLLM-omni
|
||||
|
||||
**vLLM-omni** extends vLLM capabilities to support multi-modality model inference and serving, particularly focusing on non-autoregressive structures and non-textual outputs. It aims to support:
|
||||
|
||||
- **Multi-modal Models**: Text, image, video, audio, and sensor data processing
|
||||
- **Non-autoregressive Architectures**: Diffusion Transformers (DiT) and other parallel generation models
|
||||
- **Heterogeneous Outputs**: Beyond traditional text generation to multimodal outputs
|
||||
|
||||
## <span class="twemoji">🎯</span> Key Features
|
||||
|
||||
### Multi-Engine Support
|
||||
|
||||
- **Autoregressive Engine**: Traditional text generation with enhanced KV-caching
|
||||
- **Diffusion Engine**: Support for DiT models and iterative generation
|
||||
- **Hybrid Engine**: Combined AR+DiT processing pipelines
|
||||
|
||||
### Modality Processing
|
||||
|
||||
- **Text**: Advanced tokenization and embedding generation
|
||||
- **Image**: Vision encoder integration (CLIP, etc.)
|
||||
- **Audio**: Speech processing and audio embedding
|
||||
|
||||
## <span class="twemoji">🤖</span> Supported Models
|
||||
|
||||
- Qwen-omni (Thinker-Talker-Codec structure)
|
||||
- HunyunaImage 3.0 (Ongoing)
|
||||
- Qwen-Image (Ongoing)
|
||||
|
||||
## <span class="twemoji">📚</span> Documentation Navigation
|
||||
|
||||
- To run open-source models on vLLM-Omni, we recommend starting with the [:material-code-tags: User Quide](user_guide/getting_started/quickstart.md)
|
||||
- To develop and contribute to vLLM-Omni, we recommend starting with the [:material-tools: Developer Guide](contributing/README.md)
|
||||
@@ -0,0 +1,268 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-omni project
|
||||
"""
|
||||
Hook to automatically generate docs/api/README.md from the codebase.
|
||||
|
||||
This script scans the vllm_omni module for public classes and functions,
|
||||
categorizes them, and generates a summary README file.
|
||||
"""
|
||||
|
||||
import ast
|
||||
import logging
|
||||
from pathlib import Path
|
||||
|
||||
logger = logging.getLogger("mkdocs")
|
||||
|
||||
ROOT_DIR = Path(__file__).parent.parent.parent.parent
|
||||
API_README_PATH = ROOT_DIR / "docs" / "api" / "README.md"
|
||||
|
||||
# Category mappings: module prefix -> category name and description
|
||||
CATEGORIES = {
|
||||
"entrypoints": {
|
||||
"name": "Entry Points",
|
||||
"description": "Main entry points for vLLM-omni inference and serving.",
|
||||
},
|
||||
"inputs": {
|
||||
"name": "Inputs",
|
||||
"description": "Input data structures for multi-modal inputs.",
|
||||
},
|
||||
"engine": {
|
||||
"name": "Engine",
|
||||
"description": "Engine classes for offline and online inference.",
|
||||
},
|
||||
"core": {
|
||||
"name": "Core",
|
||||
"description": "Core scheduling and caching components.",
|
||||
},
|
||||
"model_executor": {
|
||||
"name": "Model Executor",
|
||||
"description": "Model execution components.",
|
||||
},
|
||||
"config": {
|
||||
"name": "Configuration",
|
||||
"description": "Configuration classes.",
|
||||
},
|
||||
"worker": {
|
||||
"name": "Workers",
|
||||
"description": "Worker classes and model runners for distributed inference.",
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
class APIVisitor(ast.NodeVisitor):
|
||||
"""AST visitor to extract public classes and module-level functions."""
|
||||
|
||||
def __init__(self, module_path: str):
|
||||
self.module_path = module_path
|
||||
self.classes: list[str] = []
|
||||
self.functions: list[str] = []
|
||||
self._class_stack: list[str] = [] # Track nested class definitions
|
||||
|
||||
def visit_ClassDef(self, node: ast.ClassDef) -> None:
|
||||
"""Visit class definitions."""
|
||||
if not node.name.startswith("_"):
|
||||
self.classes.append(f"{self.module_path}.{node.name}")
|
||||
# Track that we're entering a class
|
||||
self._class_stack.append(node.name)
|
||||
self.generic_visit(node)
|
||||
# Remove from stack when done visiting
|
||||
self._class_stack.pop()
|
||||
|
||||
def visit_FunctionDef(self, node: ast.FunctionDef) -> None:
|
||||
"""Visit function definitions - only collect module-level functions."""
|
||||
# Only collect if we're not inside a class (stack is empty)
|
||||
if not self._class_stack and not node.name.startswith("_"):
|
||||
self.functions.append(f"{self.module_path}.{node.name}")
|
||||
self.generic_visit(node)
|
||||
|
||||
def visit_AsyncFunctionDef(self, node: ast.AsyncFunctionDef) -> None:
|
||||
"""Visit async function definitions - only collect module-level functions."""
|
||||
# Only collect if we're not inside a class (stack is empty)
|
||||
if not self._class_stack and not node.name.startswith("_"):
|
||||
self.functions.append(f"{self.module_path}.{node.name}")
|
||||
self.generic_visit(node)
|
||||
|
||||
|
||||
def parse_file_for_symbols(file_path: Path, module_path: str) -> tuple[list[str], list[str]]:
|
||||
"""
|
||||
Parse a Python file and extract public classes and functions.
|
||||
|
||||
Returns:
|
||||
Tuple of (classes, functions)
|
||||
"""
|
||||
try:
|
||||
# If this is __init__.py, use parent module path
|
||||
if file_path.name == "__init__.py":
|
||||
# Remove .__init__ from module path
|
||||
if module_path.endswith(".__init__"):
|
||||
module_path = module_path[:-9]
|
||||
|
||||
with open(file_path, encoding="utf-8") as f:
|
||||
content = f.read()
|
||||
|
||||
tree = ast.parse(content, filename=str(file_path))
|
||||
visitor = APIVisitor(module_path)
|
||||
visitor.visit(tree)
|
||||
|
||||
return visitor.classes, visitor.functions
|
||||
except Exception as e:
|
||||
logger.debug(f"Could not parse {file_path}: {e}")
|
||||
return [], []
|
||||
|
||||
|
||||
def scan_package(package_name: str = "vllm_omni") -> dict[str, list[str]]:
|
||||
"""
|
||||
Scan the vllm_omni package and categorize public symbols.
|
||||
|
||||
Returns:
|
||||
Dict mapping category names to lists of symbol full names
|
||||
"""
|
||||
categorized: dict[str, list[str]] = {cat["name"]: [] for cat in CATEGORIES.values()}
|
||||
|
||||
try:
|
||||
# Find the package directory
|
||||
package_path = ROOT_DIR / package_name
|
||||
if not package_path.exists():
|
||||
logger.warning(f"Package path not found: {package_path}")
|
||||
return categorized
|
||||
|
||||
# Walk through all Python files
|
||||
for py_file in package_path.rglob("*.py"):
|
||||
# Skip __init__.py and private modules
|
||||
if py_file.name.startswith("_") and py_file.name != "__init__.py":
|
||||
continue
|
||||
|
||||
# Get module path
|
||||
relative_path = py_file.relative_to(ROOT_DIR)
|
||||
module_path = str(relative_path.with_suffix("")).replace("/", ".").replace("\\", ".")
|
||||
|
||||
# Handle __init__.py - use parent module path
|
||||
if py_file.name == "__init__.py":
|
||||
# Remove .__init__ from module path
|
||||
if module_path.endswith(".__init__"):
|
||||
module_path = module_path[:-9]
|
||||
|
||||
# Determine category from module path
|
||||
category = None
|
||||
for prefix, cat_info in CATEGORIES.items():
|
||||
if prefix in module_path:
|
||||
category = cat_info["name"]
|
||||
break
|
||||
|
||||
if not category:
|
||||
continue
|
||||
|
||||
# Parse file for symbols
|
||||
classes, functions = parse_file_for_symbols(py_file, module_path)
|
||||
|
||||
# Filter out internal implementation classes
|
||||
# Skip classes that look like internal components (DiT layers, etc.)
|
||||
internal_patterns = [
|
||||
"Block",
|
||||
"Layer",
|
||||
"Net",
|
||||
"Embedding",
|
||||
"Norm",
|
||||
"Activation",
|
||||
"Solver",
|
||||
"Pooling",
|
||||
"Attention",
|
||||
"MLP",
|
||||
"DecoderLayer",
|
||||
"InputEmbedding",
|
||||
"TimestepEmbedding",
|
||||
"CodecEmbedding",
|
||||
"DownSample",
|
||||
"UpSample",
|
||||
"Res2Net",
|
||||
"SqueezeExcitation",
|
||||
"TimeDelay",
|
||||
"TorchActivation",
|
||||
"SnakeBeta",
|
||||
"SinusPosition",
|
||||
"RungeKutta",
|
||||
"AMPBlock",
|
||||
"AdaLayerNorm",
|
||||
]
|
||||
|
||||
# Add classes (filter out internal ones)
|
||||
for class_name in classes:
|
||||
class_short_name = class_name.split(".")[-1]
|
||||
# Skip if it matches internal patterns (unless it's a main model class)
|
||||
if any(pattern in class_short_name for pattern in internal_patterns):
|
||||
# But include main model classes
|
||||
if not any(
|
||||
main_class in class_short_name
|
||||
for main_class in [
|
||||
"ForConditionalGeneration",
|
||||
"Model",
|
||||
"Registry",
|
||||
"Worker",
|
||||
"Runner",
|
||||
"Scheduler",
|
||||
"Manager",
|
||||
"Processor",
|
||||
"Config",
|
||||
]
|
||||
):
|
||||
continue
|
||||
categorized[category].append(class_name)
|
||||
|
||||
# Add important functions (parse, preprocess, etc.)
|
||||
for func_name in functions:
|
||||
# Include functions that match certain patterns
|
||||
if any(keyword in func_name.lower() for keyword in ["parse", "preprocess"]):
|
||||
categorized[category].append(func_name)
|
||||
|
||||
# Sort symbols within each category
|
||||
for category in categorized:
|
||||
categorized[category].sort()
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Error scanning package: {e}", exc_info=True)
|
||||
|
||||
return categorized
|
||||
|
||||
|
||||
def generate_readme(categorized: dict[str, list[str]]) -> str:
|
||||
"""Generate the API README markdown content."""
|
||||
lines = ["# Summary", ""]
|
||||
|
||||
# Generate sections for each category
|
||||
for prefix, cat_info in CATEGORIES.items():
|
||||
category_name = cat_info["name"]
|
||||
description = cat_info["description"]
|
||||
symbols = categorized.get(category_name, [])
|
||||
|
||||
if not symbols:
|
||||
continue
|
||||
|
||||
lines.append(f"## {category_name}")
|
||||
lines.append("")
|
||||
lines.append(description)
|
||||
lines.append("")
|
||||
|
||||
for symbol in symbols:
|
||||
lines.append(f"- [{symbol}][]")
|
||||
|
||||
lines.append("")
|
||||
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def on_startup(command, dirty: bool):
|
||||
"""MkDocs hook entry point."""
|
||||
logger.info("Generating API README documentation")
|
||||
|
||||
# Scan the package
|
||||
categorized = scan_package()
|
||||
|
||||
# Generate README content
|
||||
content = generate_readme(categorized)
|
||||
|
||||
# Write to file
|
||||
API_README_PATH.parent.mkdir(parents=True, exist_ok=True)
|
||||
with open(API_README_PATH, "w", encoding="utf-8") as f:
|
||||
f.write(content)
|
||||
|
||||
logger.info(f"API README generated: {API_README_PATH.relative_to(ROOT_DIR)}")
|
||||
@@ -0,0 +1,121 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||
"""
|
||||
This is basically a port of MyST parser’s external URL resolution mechanism
|
||||
(https://myst-parser.readthedocs.io/en/latest/syntax/cross-referencing.html#customising-external-url-resolution)
|
||||
to work with MkDocs.
|
||||
|
||||
It allows Markdown authors to use GitHub shorthand links like:
|
||||
|
||||
- [Text](gh-issue:123)
|
||||
- <gh-pr:456>
|
||||
- [File](gh-file:path/to/file.py#L10)
|
||||
|
||||
These are automatically rewritten into fully qualified GitHub URLs pointing to
|
||||
issues, pull requests, files, directories, or projects in the
|
||||
`vllm-project/vllm-omni` repository.
|
||||
|
||||
The goal is to simplify cross-referencing common GitHub resources
|
||||
in project docs.
|
||||
"""
|
||||
|
||||
import regex as re
|
||||
from mkdocs.config.defaults import MkDocsConfig
|
||||
from mkdocs.structure.files import Files
|
||||
from mkdocs.structure.pages import Page
|
||||
|
||||
|
||||
def on_page_markdown(markdown: str, *, page: Page, config: MkDocsConfig, files: Files) -> str:
|
||||
"""
|
||||
Custom MkDocs plugin hook to rewrite special GitHub reference links
|
||||
in Markdown.
|
||||
|
||||
This function scans the given Markdown content for specially formatted
|
||||
GitHub shorthand links, such as:
|
||||
- `[Link text](gh-issue:123)`
|
||||
- `<gh-pr:456>`
|
||||
|
||||
And rewrites them into fully-qualified GitHub URLs with GitHub icons:
|
||||
- `[:octicons-mark-github-16: Link text](https://github.com/vllm-project/vllm/issues/123)`
|
||||
- `[:octicons-mark-github-16: Pull Request #456](https://github.com/vllm-project/vllm/pull/456)`
|
||||
|
||||
Supported shorthand types:
|
||||
- `gh-issue`
|
||||
- `gh-pr`
|
||||
- `gh-project`
|
||||
- `gh-dir`
|
||||
- `gh-file`
|
||||
|
||||
Args:
|
||||
markdown (str): The raw Markdown content of the page.
|
||||
page (Page): The MkDocs page object being processed.
|
||||
config (MkDocsConfig): The MkDocs site configuration.
|
||||
files (Files): The collection of files in the MkDocs build.
|
||||
|
||||
Returns:
|
||||
str: The updated Markdown content with GitHub shorthand links replaced.
|
||||
"""
|
||||
gh_icon = ":octicons-mark-github-16:"
|
||||
gh_url = "https://github.com"
|
||||
repo_url = f"{gh_url}/vllm-project/vllm-omni"
|
||||
org_url = f"{gh_url}/orgs/vllm-project"
|
||||
|
||||
# Mapping of shorthand types to their corresponding GitHub base URLs
|
||||
urls = {
|
||||
"issue": f"{repo_url}/issues",
|
||||
"pr": f"{repo_url}/pull",
|
||||
"project": f"{org_url}/projects",
|
||||
"dir": f"{repo_url}/tree/main",
|
||||
"file": f"{repo_url}/blob/main",
|
||||
}
|
||||
|
||||
# Default title prefixes for auto links
|
||||
titles = {
|
||||
"issue": "Issue #",
|
||||
"pr": "Pull Request #",
|
||||
"project": "Project #",
|
||||
"dir": "",
|
||||
"file": "",
|
||||
}
|
||||
|
||||
# Regular expression to match GitHub shorthand links
|
||||
scheme = r"gh-(?P<type>.+?):(?P<path>.+?)(#(?P<fragment>.+?))?"
|
||||
inline_link = re.compile(r"\[(?P<title>[^\[]+?)\]\(" + scheme + r"\)")
|
||||
auto_link = re.compile(f"<{scheme}>")
|
||||
|
||||
def replace_inline_link(match: re.Match) -> str:
|
||||
"""
|
||||
Replaces a matched inline-style GitHub shorthand link
|
||||
with a full Markdown link.
|
||||
|
||||
Example:
|
||||
[My issue](gh-issue:123) → [:octicons-mark-github-16: My issue](https://github.com/vllm-project/vllm/issues/123)
|
||||
"""
|
||||
url = f"{urls[match.group('type')]}/{match.group('path')}"
|
||||
if fragment := match.group("fragment"):
|
||||
url += f"#{fragment}"
|
||||
|
||||
return f"[{gh_icon} {match.group('title')}]({url})"
|
||||
|
||||
def replace_auto_link(match: re.Match) -> str:
|
||||
"""
|
||||
Replaces a matched autolink-style GitHub shorthand
|
||||
with a full Markdown link.
|
||||
|
||||
Example:
|
||||
<gh-pr:456> → [:octicons-mark-github-16: Pull Request #456](https://github.com/vllm-project/vllm/pull/456)
|
||||
"""
|
||||
type = match.group("type")
|
||||
path = match.group("path")
|
||||
title = f"{titles[type]}{path}"
|
||||
url = f"{urls[type]}/{path}"
|
||||
if fragment := match.group("fragment"):
|
||||
url += f"#{fragment}"
|
||||
|
||||
return f"[{gh_icon} {title}]({url})"
|
||||
|
||||
# Replace both inline and autolinks
|
||||
markdown = inline_link.sub(replace_inline_link, markdown)
|
||||
markdown = auto_link.sub(replace_auto_link, markdown)
|
||||
|
||||
return markdown
|
||||
@@ -0,0 +1,47 @@
|
||||
/**
|
||||
* edit_and_feedback.js
|
||||
*
|
||||
* Enhances MkDocs Material docs pages by:
|
||||
*
|
||||
* 1. Adding a "Question? Give us feedback" link
|
||||
* below the "Edit" button.
|
||||
*
|
||||
* - The link opens a GitHub issue with a template,
|
||||
* auto-filled with the current page URL and path.
|
||||
*
|
||||
* 2. Ensuring the edit button opens in a new tab
|
||||
* with target="_blank" and rel="noopener".
|
||||
*/
|
||||
document.addEventListener("DOMContentLoaded", function () {
|
||||
const url = window.location.href;
|
||||
const page = document.body.dataset.mdUrl || location.pathname;
|
||||
|
||||
const feedbackLink = document.createElement("a");
|
||||
feedbackLink.href = `https://github.com/vllm-project/vllm-omni/issues/new?template=100-documentation.yml&title=${encodeURIComponent(
|
||||
`[Docs] Feedback for \`${page}\``
|
||||
)}&body=${encodeURIComponent(`📄 **Reference:**\n${url}\n\n📝 **Feedback:**\n_Your response_`)}`;
|
||||
feedbackLink.target = "_blank";
|
||||
feedbackLink.rel = "noopener";
|
||||
feedbackLink.title = "Provide feedback";
|
||||
feedbackLink.className = "md-content__button";
|
||||
feedbackLink.innerHTML = `
|
||||
<svg
|
||||
xmlns="http://www.w3.org/2000/svg"
|
||||
height="24px"
|
||||
viewBox="0 -960 960 960"
|
||||
width="24px"
|
||||
fill="currentColor"
|
||||
>
|
||||
<path d="M280-280h280v-80H280v80Zm0-160h400v-80H280v80Zm0-160h400v-80H280v80Zm-80 480q-33 0-56.5-23.5T120-200v-560q0-33 23.5-56.5T200-840h560q33 0 56.5 23.5T840-760v560q0 33-23.5 56.5T760-120H200Zm0-80h560v-560H200v560Zm0-560v560-560Z"/>
|
||||
</svg>
|
||||
`;
|
||||
|
||||
const editButton = document.querySelector('.md-content__button[href*="edit"]');
|
||||
|
||||
if (editButton && editButton.parentNode) {
|
||||
editButton.insertAdjacentElement("beforebegin", feedbackLink);
|
||||
|
||||
editButton.setAttribute("target", "_blank");
|
||||
editButton.setAttribute("rel", "noopener");
|
||||
}
|
||||
});
|
||||
@@ -0,0 +1,20 @@
|
||||
// Enables MathJax rendering
|
||||
window.MathJax = {
|
||||
tex: {
|
||||
inlineMath: [["\\(", "\\)"]],
|
||||
displayMath: [["\\[", "\\]"]],
|
||||
processEscapes: true,
|
||||
processEnvironments: true
|
||||
},
|
||||
options: {
|
||||
ignoreHtmlClass: ".*|",
|
||||
processHtmlClass: "arithmatex"
|
||||
}
|
||||
};
|
||||
|
||||
document$.subscribe(() => {
|
||||
MathJax.startup.output.clearCache()
|
||||
MathJax.typesetClear()
|
||||
MathJax.texReset()
|
||||
MathJax.typesetPromise()
|
||||
})
|
||||
@@ -0,0 +1,56 @@
|
||||
/**
|
||||
* slack_and_forum.js
|
||||
*
|
||||
* Adds a custom Slack and Forum button to the MkDocs Material header.
|
||||
*
|
||||
*/
|
||||
|
||||
window.addEventListener('DOMContentLoaded', () => {
|
||||
const headerInner = document.querySelector('.md-header__inner');
|
||||
|
||||
if (headerInner) {
|
||||
const slackButton = document.createElement('button');
|
||||
slackButton.className = 'slack-button';
|
||||
slackButton.title = 'Join us on Slack';
|
||||
slackButton.style.border = 'none';
|
||||
slackButton.style.background = 'transparent';
|
||||
slackButton.style.cursor = 'pointer';
|
||||
|
||||
slackButton.innerHTML = `
|
||||
<img src="https://a.slack-edge.com/80588/marketing/img/icons/icon_slack_hash_colored.png"
|
||||
style="height: 1.1rem;"
|
||||
alt="Slack">
|
||||
`;
|
||||
|
||||
slackButton.addEventListener('click', () => {
|
||||
window.open('https://slack.vllm.ai', '_blank', 'noopener');
|
||||
});
|
||||
|
||||
const forumButton = document.createElement('button');
|
||||
forumButton.className = 'forum-button';
|
||||
forumButton.title = 'Join the Forum';
|
||||
forumButton.style.border = 'none';
|
||||
forumButton.style.background = 'transparent';
|
||||
forumButton.style.cursor = 'pointer';
|
||||
|
||||
forumButton.innerHTML = `
|
||||
<svg
|
||||
xmlns="http://www.w3.org/2000/svg"
|
||||
viewBox="0 -960 960 960"
|
||||
fill="currentColor"
|
||||
>
|
||||
<path d="M817.85-198.15 698.46-317.54H320q-24.48 0-41.47-16.99T261.54-376v-11.69h424.61q25.39 0 43.47-18.08 18.07-18.08 18.07-43.46v-268.92h11.69q24.48 0 41.47 16.99 17 16.99 17 41.47v461.54ZM179.08-434.69l66.84-66.85h363.31q10.77 0 17.69-6.92 6.93-6.92 6.93-17.69v-246.77q0-10.77-6.93-17.7-6.92-6.92-17.69-6.92H203.69q-10.77 0-17.69 6.92-6.92 6.93-6.92 17.7v338.23Zm-36.93 89.46v-427.69q0-25.39 18.08-43.46 18.08-18.08 43.46-18.08h405.54q25.39 0 43.46 18.08 18.08 18.07 18.08 43.46v246.77q0 25.38-18.08 43.46-18.07 18.07-43.46 18.07H261.54L142.15-345.23Zm36.93-180.92V-797.54v271.39Z"/>
|
||||
</svg>
|
||||
`;
|
||||
|
||||
forumButton.addEventListener('click', () => {
|
||||
window.open('https://discuss.vllm.ai/', '_blank', 'noopener');
|
||||
});
|
||||
|
||||
const githubSource = document.querySelector('.md-header__source');
|
||||
if (githubSource) {
|
||||
githubSource.parentNode.insertBefore(slackButton, githubSource.nextSibling);
|
||||
githubSource.parentNode.insertBefore(forumButton, slackButton.nextSibling);
|
||||
}
|
||||
}
|
||||
});
|
||||
@@ -0,0 +1 @@
|
||||
{% extends "base.html" %}
|
||||
@@ -0,0 +1,21 @@
|
||||
<!-- Enables the use of toc_depth in document frontmatter https://github.com/squidfunk/mkdocs-material/issues/4827#issuecomment-1869812019 -->
|
||||
<li class="md-nav__item">
|
||||
<a href="{{ toc_item.url }}" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
{{ toc_item.title }}
|
||||
</span>
|
||||
</a>
|
||||
|
||||
<!-- Table of contents list -->
|
||||
{% if toc_item.children %}
|
||||
<nav class="md-nav" aria-label="{{ toc_item.title | striptags }}">
|
||||
<ul class="md-nav__list">
|
||||
{% for toc_item in toc_item.children %}
|
||||
{% if not page.meta.toc_depth or toc_item.level <= page.meta.toc_depth %}
|
||||
{% include "partials/toc-item.html" %}
|
||||
{% endif %}
|
||||
{% endfor %}
|
||||
</ul>
|
||||
</nav>
|
||||
{% endif %}
|
||||
</li>
|
||||
@@ -1,55 +1,218 @@
|
||||
/* Custom CSS for vLLM-omni documentation - based on vLLM's extra.css */
|
||||
|
||||
/* Improve code block readability */
|
||||
.highlight {
|
||||
border-radius: 4px;
|
||||
/* Warning for latest docs */
|
||||
.md-banner {
|
||||
background-color: var(--md-warning-bg-color);
|
||||
color: var(--md-warning-fg-color);
|
||||
}
|
||||
|
||||
/* Style API documentation */
|
||||
.md-typeset .doc-contents {
|
||||
margin-bottom: 1.5em;
|
||||
/* https://christianoliff.com/blog/styling-external-links-with-an-icon-in-css/ */
|
||||
a:not(:has(svg)):not(.md-icon):not(.autorefs-external) {
|
||||
align-items: center;
|
||||
|
||||
&[href^="//"]::after,
|
||||
&[href^="http://"]::after,
|
||||
&[href^="https://"]::after {
|
||||
content: "";
|
||||
width: 12px;
|
||||
height: 12px;
|
||||
margin-left: 4px;
|
||||
background-image: url("data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' width='16' height='16' stroke='gray' viewBox='0 0 16 16'%3E%3Cpath fill-rule='evenodd' d='M8.636 3.5a.5.5 0 0 0-.5-.5H1.5A1.5 1.5 0 0 0 0 4.5v10A1.5 1.5 0 0 0 1.5 16h10a1.5 1.5 0 0 0 1.5-1.5V7.864a.5.5 0 0 0-1 0V14.5a.5.5 0 0 1-.5.5h-10a.5.5 0 0 1-.5-.5v-10a.5.5 0 0 1 .5-.5h6.636a.5.5 0 0 0 .5-.5z'/%3E%3Cpath fill-rule='evenodd' d='M16 .5a.5.5 0 0 0-.5-.5h-5a.5.5 0 0 0 0 1h3.793L6.146 9.146a.5.5 0 1 0 .708.708L15 1.707V5.5a.5.5 0 0 0 1 0v-5z'/%3E%3C/svg%3E");
|
||||
background-position: center;
|
||||
background-repeat: no-repeat;
|
||||
background-size: contain;
|
||||
display: inline-block;
|
||||
}
|
||||
}
|
||||
|
||||
/* Improve table readability */
|
||||
.md-typeset table {
|
||||
display: table;
|
||||
width: 100%;
|
||||
a[href*="localhost"]::after,
|
||||
a[href*="127.0.0.1"]::after,
|
||||
a[href*="org.readthedocs.build"]::after,
|
||||
a[href*="docs.vllm.ai"]::after {
|
||||
display: none !important;
|
||||
}
|
||||
|
||||
/* Light mode: darker section titles (from vLLM) */
|
||||
/* Light mode: darker section titles */
|
||||
body[data-md-color-scheme="default"] .md-nav__item--section > label.md-nav__link .md-ellipsis {
|
||||
color: rgba(0, 0, 0, 0.7) !important;
|
||||
font-weight: 700;
|
||||
}
|
||||
|
||||
/* Dark mode: lighter gray section titles (from vLLM) */
|
||||
/* Dark mode: lighter gray section titles */
|
||||
body[data-md-color-scheme="slate"] .md-nav__item--section > label.md-nav__link .md-ellipsis {
|
||||
color: rgba(255, 255, 255, 0.75) !important;
|
||||
font-weight: 700;
|
||||
}
|
||||
|
||||
/* Navigation: Expand first level of API Reference by default, collapse deeper levels */
|
||||
/* Show direct children of API Reference section (vllm_omni's children: config, core, entrypoints, etc.) */
|
||||
.md-nav__item--nested[data-md-state] > .md-nav {
|
||||
display: none;
|
||||
/* Custom admonitions */
|
||||
:root {
|
||||
--md-admonition-icon--announcement: url('data:image/svg+xml;charset=utf-8,<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 16 16" width="16" height="16"><path d="M3.25 9a.75.75 0 0 1 .75.75c0 2.142.456 3.828.733 4.653a.122.122 0 0 0 .05.064.212.212 0 0 0 .117.033h1.31c.085 0 .18-.042.258-.152a.45.45 0 0 0 .075-.366A16.743 16.743 0 0 1 6 9.75a.75.75 0 0 1 1.5 0c0 1.588.25 2.926.494 3.85.293 1.113-.504 2.4-1.783 2.4H4.9c-.686 0-1.35-.41-1.589-1.12A16.4 16.4 0 0 1 2.5 9.75.75.75 0 0 1 3.25 9Z"></path><path d="M0 6a4 4 0 0 1 4-4h2.75a.75.75 0 0 1 .75.75v6.5a.75.75 0 0 1-.75.75H4a4 4 0 0 1-4-4Zm4-2.5a2.5 2.5 0 1 0 0 5h2v-5Z"></path><path d="M15.59.082A.75.75 0 0 1 16 .75v10.5a.75.75 0 0 1-1.189.608l-.002-.001h.001l-.014-.01a5.775 5.775 0 0 0-.422-.25 10.63 10.63 0 0 0-1.469-.64C11.576 10.484 9.536 10 6.75 10a.75.75 0 0 1 0-1.5c2.964 0 5.174.516 6.658 1.043.423.151.787.302 1.092.443V2.014c-.305.14-.669.292-1.092.443C11.924 2.984 9.713 3.5 6.75 3.5a.75.75 0 0 1 0-1.5c2.786 0 4.826-.484 6.155-.957.665-.236 1.154-.47 1.47-.64.144-.077.284-.161.421-.25l.014-.01a.75.75 0 0 1 .78-.061Z"></path></svg>');
|
||||
--md-admonition-icon--important: url('data:image/svg+xml;charset=utf-8,<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 16 16" width="16" height="16"><path d="M4.47.22A.749.749 0 0 1 5 0h6c.199 0 .389.079.53.22l4.25 4.25c.141.14.22.331.22.53v6a.749.749 0 0 1-.22.53l-4.25 4.25A.749.749 0 0 1 11 16H5a.749.749 0 0 1-.53-.22L.22 11.53A.749.749 0 0 1 0 11V5c0-.199.079-.389.22-.53Zm.84 1.28L1.5 5.31v5.38l3.81 3.81h5.38l3.81-3.81V5.31L10.69 1.5ZM8 4a.75.75 0 0 1 .75.75v3.5a.75.75 0 0 1-1.5 0v-3.5A.75.75 0 0 1 8 4Zm0 8a1 1 0 1 1 0-2 1 1 0 0 1 0 2Z"></path></svg>');
|
||||
--md-admonition-icon--code: url('data:image/svg+xml;charset=utf-8,<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 16 16"><path d="m11.28 3.22 4.25 4.25a.75.75 0 0 1 0 1.06l-4.25 4.25a.749.749 0 0 1-1.275-.326.75.75 0 0 1 .215-.734L13.94 8l-3.72-3.72a.749.749 0 0 1 .326-1.275.75.75 0 0 1 .734.215m-6.56 0a.75.75 0 0 1 1.042.018.75.75 0 0 1 .018 1.042L2.06 8l3.72 3.72a.749.749 0 0 1-.326 1.275.75.75 0 0 1-.734-.215L.47 8.53a.75.75 0 0 1 0-1.06Z"/></svg>');
|
||||
--md-admonition-icon--console: url('data:image/svg+xml;charset=utf-8,<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 16 16"><path d="M0 2.75C0 1.784.784 1 1.75 1h12.5c.966 0 1.75.784 1.75 1.75v10.5A1.75 1.75 0 0 1 14.25 15H1.75A1.75 1.75 0 0 1 0 13.25Zm1.75-.25a.25.25 0 0 0-.25.25v10.5c0 .138.112.25.25.25h12.5a.25.25 0 0 0 .25-.25V2.75a.25.25 0 0 0-.25-.25ZM7.25 8a.75.75 0 0 1-.22.53l-2.25 2.25a.749.749 0 0 1-1.275-.326.75.75 0 0 1 .215-.734L5.44 8 3.72 6.28a.749.749 0 0 1 .326-1.275.75.75 0 0 1 .734.215l2.25 2.25c.141.14.22.331.22.53m1.5 1.5h3a.75.75 0 0 1 0 1.5h-3a.75.75 0 0 1 0-1.5"/></svg>');
|
||||
}
|
||||
|
||||
/* Expand active items */
|
||||
.md-nav__item--nested.md-nav__item--active > .md-nav {
|
||||
display: block;
|
||||
.md-typeset .admonition.announcement,
|
||||
.md-typeset details.announcement {
|
||||
border-color: rgb(255, 110, 66);
|
||||
}
|
||||
.md-typeset .admonition.important,
|
||||
.md-typeset details.important {
|
||||
border-color: rgb(239, 85, 82);
|
||||
}
|
||||
.md-typeset .admonition.code,
|
||||
.md-typeset details.code {
|
||||
border-color: #64dd17
|
||||
}
|
||||
.md-typeset .admonition.console,
|
||||
.md-typeset details.console {
|
||||
border-color: #64dd17
|
||||
}
|
||||
|
||||
/* For API Reference: show first level children (config, core, entrypoints, etc.) by default */
|
||||
.md-nav__item--section:has([href*="api/vllm_omni/index"]) > .md-nav > .md-nav__list > .md-nav__item--nested > .md-nav {
|
||||
display: block;
|
||||
.md-typeset .announcement > .admonition-title,
|
||||
.md-typeset .announcement > summary {
|
||||
background-color: rgb(255, 110, 66, 0.1);
|
||||
}
|
||||
.md-typeset .important > .admonition-title,
|
||||
.md-typeset .important > summary {
|
||||
background-color: rgb(239, 85, 82, 0.1);
|
||||
}
|
||||
.md-typeset .code > .admonition-title,
|
||||
.md-typeset .code > summary {
|
||||
background-color: #64dd171a;
|
||||
}
|
||||
.md-typeset .console > .admonition-title,
|
||||
.md-typeset .console > summary {
|
||||
background-color: #64dd171a;
|
||||
}
|
||||
|
||||
/* But keep deeper levels collapsed */
|
||||
.md-nav__item--section:has([href*="api/vllm_omni/index"]) > .md-nav > .md-nav__list > .md-nav__item--nested > .md-nav > .md-nav__list > .md-nav__item--nested > .md-nav {
|
||||
display: none;
|
||||
.md-typeset .announcement > .admonition-title::before,
|
||||
.md-typeset .announcement > summary::before {
|
||||
background-color: rgb(239, 85, 82);
|
||||
-webkit-mask-image: var(--md-admonition-icon--announcement);
|
||||
mask-image: var(--md-admonition-icon--announcement);
|
||||
}
|
||||
.md-typeset .important > .admonition-title::before,
|
||||
.md-typeset .important > summary::before {
|
||||
background-color: rgb(239, 85, 82);
|
||||
-webkit-mask-image: var(--md-admonition-icon--important);
|
||||
mask-image: var(--md-admonition-icon--important);
|
||||
}
|
||||
.md-typeset .code > .admonition-title::before,
|
||||
.md-typeset .code > summary::before {
|
||||
background-color: #64dd17;
|
||||
-webkit-mask-image: var(--md-admonition-icon--code);
|
||||
mask-image: var(--md-admonition-icon--code);
|
||||
}
|
||||
.md-typeset .console > .admonition-title::before,
|
||||
.md-typeset .console > summary::before {
|
||||
background-color: #64dd17;
|
||||
-webkit-mask-image: var(--md-admonition-icon--console);
|
||||
mask-image: var(--md-admonition-icon--console);
|
||||
}
|
||||
|
||||
/* Expand deeper levels only when active */
|
||||
.md-nav__item--section:has([href*="api/vllm_omni/index"]) > .md-nav > .md-nav__list > .md-nav__item--nested > .md-nav > .md-nav__list > .md-nav__item--nested.md-nav__item--active > .md-nav {
|
||||
display: block;
|
||||
/* Make label fully visible on hover */
|
||||
.md-content__button[href*="edit"]:hover::after {
|
||||
opacity: 1;
|
||||
}
|
||||
|
||||
/* Hide edit button on generated docs/examples pages */
|
||||
@media (min-width: 960px) {
|
||||
.md-content__button[href*="docs/examples/"] {
|
||||
display: none !important;
|
||||
}
|
||||
}
|
||||
|
||||
.md-content__button-wrapper {
|
||||
position: absolute;
|
||||
top: 0.6rem;
|
||||
right: 0.8rem;
|
||||
display: flex;
|
||||
flex-direction: row;
|
||||
align-items: center;
|
||||
gap: 0.4rem;
|
||||
z-index: 1;
|
||||
}
|
||||
|
||||
.md-content__button-wrapper a {
|
||||
display: inline-flex;
|
||||
align-items: center;
|
||||
justify-content: center;
|
||||
height: 24px;
|
||||
width: 24px;
|
||||
color: var(--md-default-fg-color);
|
||||
text-decoration: none;
|
||||
}
|
||||
|
||||
.md-content__button-wrapper a:hover {
|
||||
color: var(--md-accent-fg-color);
|
||||
}
|
||||
|
||||
/* Slack and Forum css */
|
||||
.slack-button,
|
||||
.forum-button {
|
||||
display: inline-flex;
|
||||
align-items: center;
|
||||
justify-content: center;
|
||||
margin-left: 0.4rem;
|
||||
height: 24px;
|
||||
}
|
||||
|
||||
.slack-button img {
|
||||
height: 18px;
|
||||
filter: none !important;
|
||||
}
|
||||
|
||||
.slack-button:hover,
|
||||
.forum-button:hover {
|
||||
opacity: 0.7;
|
||||
}
|
||||
|
||||
.forum-button svg {
|
||||
height: 28px;
|
||||
opacity: 0.9;
|
||||
transform: translateY(2px);
|
||||
}
|
||||
|
||||
/* For logo css */
|
||||
[data-md-color-scheme="default"] .logo-dark {
|
||||
display: none;
|
||||
}
|
||||
|
||||
[data-md-color-scheme="slate"] .logo-light {
|
||||
display: none;
|
||||
}
|
||||
|
||||
/* Outline for content tabs */
|
||||
.md-typeset .tabbed-set {
|
||||
border: 0.075rem solid var(--md-default-fg-color);
|
||||
border-radius: 0.2rem;
|
||||
}
|
||||
|
||||
.md-typeset .tabbed-content {
|
||||
padding: 0 0.6em;
|
||||
}
|
||||
|
||||
/* Hide link icon in header logo */
|
||||
.md-header__button.md-logo :is(img, svg) {
|
||||
pointer-events: none;
|
||||
}
|
||||
|
||||
.md-header__button.md-logo::after {
|
||||
display: none !important;
|
||||
}
|
||||
|
||||
/* Hide link icons in content tabs (tabbed content) */
|
||||
.md-typeset .tabbed-set > label::after,
|
||||
.md-typeset .tabbed-set > input:checked + label::after {
|
||||
display: none !important;
|
||||
}
|
||||
|
||||
/* Hide link icons in navigation tabs */
|
||||
.md-nav__link[href]::after,
|
||||
.md-nav__item--nested > .md-nav__link::after {
|
||||
display: none !important;
|
||||
}
|
||||
|
||||
/* Hide link icons in top navigation tabs */
|
||||
.md-tabs__link::after {
|
||||
display: none !important;
|
||||
}
|
||||
@@ -0,0 +1,34 @@
|
||||
# Supported Models
|
||||
|
||||
vLLM-Omni supports unified multimodal comprehension and generation models across various tasks.
|
||||
|
||||
## Model Implementation
|
||||
|
||||
If vLLM-Omni natively supports a model, its implementation can be found in <gh-file:vllm-omni/model_executor/models>.
|
||||
|
||||
## List of Supported Models for Nvidia GPU
|
||||
|
||||
<style>
|
||||
th {
|
||||
white-space: nowrap;
|
||||
min-width: 0 !important;
|
||||
}
|
||||
</style>
|
||||
|
||||
| Architecture | Models | Example HF Models |
|
||||
|--------------|--------|-------------------|
|
||||
| `Qwen2_5OmniForConditionalGeneration` | Qwen2.5-Omni | `Qwen/Qwen2.5-Omni-7B`, `Qwen/Qwen2.5-Omni-3B`|
|
||||
|
||||
|
||||
## List of Supported Models for Ascend NPU
|
||||
|
||||
<style>
|
||||
th {
|
||||
white-space: nowrap;
|
||||
min-width: 0 !important;
|
||||
}
|
||||
</style>
|
||||
|
||||
| Architecture | Models | Example HF Models |
|
||||
|--------------|--------|-------------------|
|
||||
| `Qwen2_5OmniForConditionalGeneration` | Qwen2.5-Omni | `Qwen/Qwen2.5-Omni-7B`, `Qwen/Qwen2.5-Omni-3B`|
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 17 KiB |
@@ -0,0 +1,13 @@
|
||||
# Frequently Asked Questions
|
||||
|
||||
> Q: How many chips do I need to infer a model in vLLM-Omni?
|
||||
|
||||
A: Now, we support natively disaggregated deployment for different model stages within a model. There is a restriction that one chip can only has one AutoRegressive model stage. This is because the unified KV cache management of vLLM. Stages of other types can coexist with in a chip. The restriction will be resolved in later version.
|
||||
|
||||
> Q: When trying to run examples, I encounter error about backend of librosa or soundfile. How to solve it?
|
||||
|
||||
A: If you encounter error about backend of librosa, try to install ffmpeg with command below.
|
||||
```
|
||||
sudo apt update
|
||||
sudo apt install ffmpeg
|
||||
```
|
||||
@@ -1,8 +0,0 @@
|
||||
# Examples
|
||||
|
||||
vLLM-omni's examples are split into two categories:
|
||||
|
||||
- If you are using vLLM-omni from within Python code, see the *Offline Inference* section.
|
||||
- If you are using vLLM-omni from an HTTP application or client, see the *Online Serving* section.
|
||||
|
||||
For detailed example documentation, check the [examples directory](https://github.com/vllm-project/vllm-omni/tree/main/examples) in the repository.
|
||||
@@ -1,6 +1,6 @@
|
||||
# Offline Example of vLLM-omni for Qwen2.5-omni
|
||||
|
||||
Source <https://github.com/vllm-project/vllm-omni/tree/main/examples\offline_inference\qwen2_5_omni>.
|
||||
Source <https://github.com/vllm-project/vllm-omni/tree/main/examples/offline_inference/qwen2_5_omni>.
|
||||
|
||||
|
||||
## 🛠️ Installation
|
||||
@@ -36,29 +36,29 @@ Then run the command below.
|
||||
bash run_single_prompt.sh
|
||||
```
|
||||
|
||||
### FAQ
|
||||
|
||||
If you encounter error about backend of librosa, try to install ffmpeg with command below.
|
||||
```
|
||||
sudo apt update
|
||||
sudo apt install ffmpeg
|
||||
```
|
||||
|
||||
## Example materials
|
||||
|
||||
??? abstract "end2end.py"
|
||||
``````py
|
||||
--8<-- "examples\offline_inference\qwen2_5_omni\end2end.py"
|
||||
--8<-- "examples/offline_inference/qwen2_5_omni/end2end.py"
|
||||
``````
|
||||
??? abstract "extract_prompts.py"
|
||||
``````py
|
||||
--8<-- "examples\offline_inference\qwen2_5_omni\extract_prompts.py"
|
||||
``````
|
||||
??? abstract "processing_omni.py"
|
||||
``````py
|
||||
--8<-- "examples\offline_inference\qwen2_5_omni\processing_omni.py"
|
||||
--8<-- "examples/offline_inference/qwen2_5_omni/extract_prompts.py"
|
||||
``````
|
||||
??? abstract "run_multiple_prompts.sh"
|
||||
``````sh
|
||||
--8<-- "examples\offline_inference\qwen2_5_omni\run_multiple_prompts.sh"
|
||||
--8<-- "examples/offline_inference/qwen2_5_omni/run_multiple_prompts.sh"
|
||||
``````
|
||||
??? abstract "run_single_prompt.sh"
|
||||
``````sh
|
||||
--8<-- "examples\offline_inference\qwen2_5_omni\run_single_prompt.sh"
|
||||
``````
|
||||
??? abstract "utils.py"
|
||||
``````py
|
||||
--8<-- "examples\offline_inference\qwen2_5_omni\utils.py"
|
||||
--8<-- "examples/offline_inference/qwen2_5_omni/run_single_prompt.sh"
|
||||
``````
|
||||
@@ -1,97 +0,0 @@
|
||||
# Offline Example of vLLM-omni for Qwen2.5-omni
|
||||
|
||||
Source <https://github.com/vllm-project/vllm-omni/tree/main/examples/offline_inference/qwen_2_5_omni>.
|
||||
|
||||
|
||||
## 🛠️ Installation
|
||||
|
||||
### Environment setup
|
||||
|
||||
Use Docker to keep consistent basic environment (Optional, Recommended)
|
||||
```bash
|
||||
docker run --gpus all --ipc=host --network=host -v $source_dir:$container_dir --name $container_name -it nvcr.io/nvidia/pytorch:25.01-py3 bash
|
||||
```
|
||||
|
||||
Set up basic uv environment
|
||||
```bash
|
||||
pip install uv
|
||||
uv venv --python 3.12 --seed
|
||||
source .venv/bin/activate
|
||||
```
|
||||
|
||||
### Installation of vLLM (for users)
|
||||
|
||||
Now we build it based on vLLM v0.11.0. Please install it with command below.
|
||||
```bash
|
||||
uv pip install vllm==0.11.0 --torch-backend=auto
|
||||
```
|
||||
|
||||
### Installation of vLLM (for developers)
|
||||
|
||||
If you want to check or debug with source code of vLLM, install stable release version of vllm with 0.11.0 from source with pre-built wheel file.
|
||||
|
||||
```bash
|
||||
git clone https://github.com/vllm-project/vllm.git
|
||||
cd vllm
|
||||
git checkout v0.11.0
|
||||
```
|
||||
Set up environment variables to get pre-built wheels. If there are internet problems, just download the whl file manually. And set VLLM_PRECOMPILED_WHEEL_LOCATION as your local absolute path of whl file.
|
||||
```bash
|
||||
export VLLM_PRECOMPILED_WHEEL_LOCATION=https://github.com/vllm-project/vllm/releases/download/v0.11.0/vllm-0.11.0-cp38-abi3-manylinux1_x86_64.whl
|
||||
```
|
||||
Install vllm with command below (If you have no existing PyTorch).
|
||||
```bash
|
||||
uv pip install --editable .
|
||||
```
|
||||
|
||||
Install vllm with command below (If you already have PyTorch).
|
||||
```bash
|
||||
python use_existing_torch.py
|
||||
uv pip install -r requirements/build.txt
|
||||
uv pip install --no-build-isolation --editable .
|
||||
```
|
||||
|
||||
### Verification for successful installation of vLLM
|
||||
Just run the command below. If no error, it demonstrates that the installation is successful.
|
||||
```bash
|
||||
python -c "import vllm._C"
|
||||
```
|
||||
|
||||
### Installation of vLLM-omni
|
||||
Install additional requirements for vllm-omni
|
||||
```bash
|
||||
git clone https://github.com/vllm-project/vllm-omni.git
|
||||
cd vllm_omni
|
||||
uv pip install -e .
|
||||
```
|
||||
|
||||
## Run examples (Qwen2.5-omni)
|
||||
|
||||
Get into the example folder
|
||||
```bash
|
||||
cd examples/offline_inference/qwen_2_5_omni
|
||||
```
|
||||
Then run
|
||||
```bash
|
||||
bash run.sh
|
||||
```
|
||||
The output audio is saved in ./output_audio
|
||||
|
||||
## Example materials
|
||||
|
||||
??? abstract "end2end.py"
|
||||
``````py
|
||||
--8<-- "examples/offline_inference/qwen_2_5_omni/end2end.py"
|
||||
``````
|
||||
??? abstract "processing_omni.py"
|
||||
``````py
|
||||
--8<-- "examples/offline_inference/qwen_2_5_omni/processing_omni.py"
|
||||
``````
|
||||
??? abstract "run.sh"
|
||||
``````sh
|
||||
--8<-- "examples/offline_inference/qwen_2_5_omni/run.sh"
|
||||
``````
|
||||
??? abstract "utils.py"
|
||||
``````py
|
||||
--8<-- "examples/offline_inference/qwen_2_5_omni/utils.py"
|
||||
``````
|
||||
@@ -1,6 +1,6 @@
|
||||
# Online serving Example of vLLM-omni for Qwen2.5-omni
|
||||
|
||||
Source <https://github.com/vllm-project/vllm-omni/blob/main/examples\online_serving\README.md>.
|
||||
Source <https://github.com/vllm-project/vllm-omni/blob/main/examples/online_serving/README.md>.
|
||||
|
||||
|
||||
## 🛠️ Installation
|
||||
@@ -26,10 +26,55 @@ cd examples/online_serving
|
||||
|
||||
Send request via python
|
||||
```bash
|
||||
python openai_chat_completion_client_for_multimodal_generation.py
|
||||
python openai_chat_completion_client_for_multimodal_generation.py --query-type mixed_modalities
|
||||
```
|
||||
|
||||
Send request via curl
|
||||
```bash
|
||||
bash run_curl_multimodal_generation.sh
|
||||
bash run_curl_multimodal_generation.sh mixed_modalities
|
||||
```
|
||||
|
||||
### FAQ
|
||||
|
||||
If you encounter error about backend of librosa, try to install ffmpeg with command below.
|
||||
```
|
||||
sudo apt update
|
||||
sudo apt install ffmpeg
|
||||
```
|
||||
|
||||
## Run Local Web UI Demo
|
||||
|
||||
This Web UI demo allows users to interact with the model through a web browser.
|
||||
|
||||
### Running Gradio Demo
|
||||
|
||||
Once vllm and vllm-omni are installed, you can launch the web service built on AsyncOmniLLM by
|
||||
|
||||
```bash
|
||||
python gradio_demo.py --model Qwen/Qwen2.5-Omni-7B --port 7861
|
||||
```
|
||||
|
||||
Then open `http://localhost:7861/` on your local browser to interact with the web UI.
|
||||
|
||||
|
||||
### Options
|
||||
|
||||
The gradio demo also supports running with an existing API server and can be customized with the following arguments.
|
||||
|
||||
|
||||
```bash
|
||||
python gradio_demo.py \
|
||||
--model Qwen/Qwen2.5-Omni-7B \
|
||||
--use-api-server \
|
||||
--api-base http://localhost:8091/v1 \
|
||||
--ip 127.0.0.1 \
|
||||
--port 7861
|
||||
```
|
||||
|
||||
- `--model`: Model name
|
||||
- `--use-api-server`: If set, connect to an existing vLLM HTTP API server instead of running AsyncOmniLLM locally.
|
||||
- `--api-base`: Base URL for vllm serve (only used when `use-api-server` is set, default: http://localhost:8091/v1)
|
||||
- `--ip`: Host/IP for Gradio server (default: 127.0.0.1)
|
||||
- `--port`: Port for Gradio server (default: 7861)
|
||||
- `--stage-configs-path`: Path to custom stage configs YAML file (optional)
|
||||
- `--share`: Share the Gradio demo publicly (creates a public link)
|
||||
@@ -0,0 +1,7 @@
|
||||
# Gradio Demo
|
||||
|
||||
Source <https://github.com/vllm-project/vllm-omni/blob/main/examples/online_serving/gradio_demo.py>.
|
||||
|
||||
``````py
|
||||
--8<-- "examples/online_serving/gradio_demo.py"
|
||||
``````
|
||||
@@ -1,5 +0,0 @@
|
||||
# Online Serving
|
||||
|
||||
## Available Examples
|
||||
|
||||
Online serving is under active development. Coming soon.
|
||||
+2
-2
@@ -1,7 +1,7 @@
|
||||
# OpenAI Chat Completion Client For Multimodal Generation
|
||||
|
||||
Source <https://github.com/vllm-project/vllm-omni/blob/main/examples\online_serving\openai_chat_completion_client_for_multimodal_generation.py>.
|
||||
Source <https://github.com/vllm-project/vllm-omni/blob/main/examples/online_serving/openai_chat_completion_client_for_multimodal_generation.py>.
|
||||
|
||||
``````py
|
||||
--8<-- "examples\online_serving\openai_chat_completion_client_for_multimodal_generation.py"
|
||||
--8<-- "examples/online_serving/openai_chat_completion_client_for_multimodal_generation.py"
|
||||
``````
|
||||
@@ -1,55 +0,0 @@
|
||||
# Quickstart
|
||||
|
||||
This guide will help you get started with vLLM-omni.
|
||||
|
||||
## Installation
|
||||
|
||||
Use Docker to keep consistent basic environment (Optional, Recommended)
|
||||
|
||||
```bash
|
||||
docker run --gpus all --ipc=host --network=host -v $source_dir:$container_dir --rm --name $container_name -it nvcr.io/nvidia/pytorch:25.01-py3 bash
|
||||
```
|
||||
|
||||
Set up basic uv environment
|
||||
|
||||
```bash
|
||||
pip install uv
|
||||
uv venv --python 3.12 --seed
|
||||
source .venv/bin/activate
|
||||
```
|
||||
|
||||
Install vLLM
|
||||
|
||||
```bash
|
||||
uv pip install vllm==0.11.0 --torch-backend=auto
|
||||
```
|
||||
|
||||
Install vLLM-omni
|
||||
|
||||
```bash
|
||||
git clone https://github.com/vllm-project/vllm-omni.git
|
||||
cd vllm_omni
|
||||
uv pip install -e .
|
||||
```
|
||||
|
||||
## Run examples (Qwen2.5-omni)
|
||||
|
||||
Get into the example folder
|
||||
|
||||
```bash
|
||||
cd examples/offline_inference/qwen_2_5_omni
|
||||
```
|
||||
|
||||
Then run.
|
||||
|
||||
```bash
|
||||
bash run.sh
|
||||
```
|
||||
|
||||
The output audio is saved in ./output_audio
|
||||
|
||||
## Next Steps
|
||||
|
||||
- Read the [architecture documentation](../../contributing/design_documents/vllm_omni_design.md)
|
||||
- Check out the [API reference](../../api/overview.md)
|
||||
- Explore the [examples](../examples/index.md)
|
||||
Reference in new issue
Block a user