Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
f5f78e46ff | ||
|
|
acde431668 | ||
|
|
f144f5effe | ||
|
|
a6b5519616 | ||
|
|
0f7c9c755d | ||
|
|
af9784c03f | ||
|
|
74eedd3ef1 | ||
|
|
a19098233b | ||
|
|
b313279af7 | ||
|
|
4d89ebaa07 | ||
|
|
617b1d6053 | ||
|
|
ca37f221a8 | ||
|
|
2b5a3dd6b3 | ||
|
|
7543f2ff3b | ||
|
|
6ac85d4607 | ||
|
|
4725593a4b | ||
|
|
c51b4fd7b8 | ||
|
|
2ee4a07adb | ||
|
|
1d0a97f0d1 | ||
|
|
2fdc647ed7 | ||
|
|
4027534dbf |
No files matched your search
@@ -54,7 +54,6 @@ steps:
|
||||
- docker#v5.2.0:
|
||||
image: public.ecr.aws/q9t5s3a7/vllm-ci-test-repo:$BUILDKITE_COMMIT
|
||||
always-pull: true
|
||||
shm-size: "8gb"
|
||||
propagate-environment: true
|
||||
shm-size: "8gb"
|
||||
environment:
|
||||
@@ -62,6 +61,42 @@ steps:
|
||||
volumes:
|
||||
- "/fsx/hf_cache:/fsx/hf_cache"
|
||||
|
||||
- label: ":full_moon: Diffusion Model Test with H100"
|
||||
timeout_in_minutes: 60
|
||||
depends_on: upload-nightly-pipeline
|
||||
# if: build.env("NIGHTLY") == "1"
|
||||
commands:
|
||||
- export VLLM_WORKER_MULTIPROC_METHOD=spawn
|
||||
- pytest -s -v tests/e2e/online_serving/test_*_expansion.py -m "advanced_model and diffusion and H100" --run-level "advanced_model"
|
||||
agents:
|
||||
queue: "mithril-h100-pool"
|
||||
plugins:
|
||||
- kubernetes:
|
||||
podSpec:
|
||||
containers:
|
||||
- image: 936637512419.dkr.ecr.us-west-2.amazonaws.com/vllm-ci-pull-through-cache/q9t5s3a7/vllm-ci-test-repo:$BUILDKITE_COMMIT
|
||||
resources:
|
||||
limits:
|
||||
nvidia.com/gpu: 2
|
||||
volumeMounts:
|
||||
- name: devshm
|
||||
mountPath: /dev/shm
|
||||
- name: hf-cache
|
||||
mountPath: /root/.cache/huggingface
|
||||
env:
|
||||
- name: HF_HOME
|
||||
value: /root/.cache/huggingface
|
||||
nodeSelector:
|
||||
node.kubernetes.io/instance-type: gpu-h100-sxm
|
||||
volumes:
|
||||
- name: devshm
|
||||
emptyDir:
|
||||
medium: Memory
|
||||
- name: hf-cache
|
||||
hostPath:
|
||||
path: /mnt/hf-cache
|
||||
type: DirectoryOrCreate
|
||||
|
||||
- label: ":full_moon: Qwen3-TTS Non-Async-Chunk E2E Test"
|
||||
timeout_in_minutes: 30
|
||||
depends_on: upload-nightly-pipeline
|
||||
|
||||
@@ -9,7 +9,7 @@ Easy, fast, and cheap omni-modality model serving for everyone
|
||||
</h3>
|
||||
|
||||
<p align="center">
|
||||
| <a href="https://vllm-omni.readthedocs.io/en/latest/"><b>Documentation</b></a> | <a href="https://discuss.vllm.ai"><b>User Forum</b></a> | <a href="https://slack.vllm.ai"><b>Developer Slack</b></a> | <a href="docs/assets/WeChat.jpg"><b>WeChat</b></a> | <a href="https://arxiv.org/abs/2602.02204"><b>Paper</b></a> | <a href="https://drive.google.com/file/d/12k_DOo2xoEROvd4xQxYNuEi6C2GECoRx/view?usp=drive_link"><b>Slides</b></a> |
|
||||
| <a href="https://vllm-omni.readthedocs.io/en/latest/"><b>Documentation</b></a> | <a href="https://discuss.vllm.ai"><b>User Forum</b></a> | <a href="https://slack.vllm.ai"><b>Developer Slack</b></a> | <a href="docs/assets/WeChat.jpg"><b>WeChat</b></a> | <a href="https://arxiv.org/abs/2602.02204"><b>Paper</b></a> | <a href="https://docs.google.com/presentation/d/1XJWgv79lORl8rbaVvp2d5Sqs6ZEBgAgj/edit?usp=sharing&ouid=110473603432222024453&rtpof=true&sd=true"><b>Slides</b></a> |
|
||||
</p>
|
||||
|
||||
|
||||
|
||||
@@ -47,6 +47,7 @@ If no, check your shell running the ComfyUI process. There may be some error mes
|
||||
This extension offers the following nodes based on the output modalities (at **ComfyUI sidebar -> Node Library**):
|
||||
|
||||
- **Generate Image** for text-to-image and image-to-image tasks
|
||||
- **Generate Video** for text-to-video and image-to-video tasks
|
||||
- **Multimodality Understanding** for multimodality-to-text and multimodality-to-audio tasks
|
||||
- **TTS** and **TTS Voice Clone** for TTS tasks
|
||||
|
||||
@@ -59,7 +60,7 @@ To build a simple workflow yourself,
|
||||
|
||||
- Drag a generation node onto the canvas.
|
||||
- Depending on your need, grab built-in multimedia file loader nodes, such as **image->Load Image**, **image->video->Load Video**, **audio->Load Audio**
|
||||
- Depending on your need, grab built-in multimedia file preview nodes, such as **image->Preview Image**, **image->video->Save Video**, **audio->Preview Audio**. For text output, you can install [ComfyUI-Custom-Scripts plugin](https://github.com/pythongosssss/ComfyUI-Custom-Scripts/) and grab its **utils->Show Text 🐍** node.
|
||||
- Depending on your need, grab built-in multimedia file preview nodes, such as **image->Preview Image**, **image->video->Save Video**, **audio->Preview Audio**, **utils->Preview as Text**.
|
||||
- If you want to tune sampling parameters, grab corresponding nodes from **vLLM-Omni-> Sampling Params**.
|
||||
- For multi-stage models, you can connect multiple **AR Sampling Params** and **Diffusion Sampling Params** nodes to a **Multi-Stage Sampling Params List** node, and connect this node to the generation node.
|
||||
- For some multi-stage models like BAGEL, [only one stage's sampling parameters are exposed and tunable via vLLM-Omni's online serving API](https://docs.vllm.ai/projects/vllm-omni/en/latest/user_guide/examples/online_serving/bagel/). Thus, these models are treated as single-stage ones. Please check the vLLM-Omni documentation on how to correctly set each model's sampling parameters.
|
||||
@@ -104,6 +105,20 @@ You can configure per-stage sampling parameters for multi-stage models.
|
||||
> [!TIP]
|
||||
> The node automatically choose text-to-image or image-to-image API endpoints depending on whether you connect an image input or not.
|
||||
|
||||
### Text-to-video and image-to-video generation (e.g., Wan)
|
||||
|
||||
(Also available at **ComfyUI sidebar->Template->vLLM-Omni->vLLM-Omni Video Generation**)
|
||||
|
||||
<p align="center">
|
||||
<picture>
|
||||
<source media="(prefers-color-scheme: dark)" src="https://raw.githubusercontent.com/vllm-project/vllm-omni/refs/heads/main/apps/ComfyUI-vLLM-Omni/docs/images/comfyui-video-generation.jpg">
|
||||
<img alt="vLLM-Omni video generation" src="https://raw.githubusercontent.com/vllm-project/vllm-omni/refs/heads/main/apps/ComfyUI-vLLM-Omni/docs/images/comfyui-video-generation.jpg" width=55%>
|
||||
</picture>
|
||||
</p>
|
||||
|
||||
> [!TIP]
|
||||
> The node automatically choose text-to-video or image-to-video API endpoints depending on whether you connect an image input or not.
|
||||
|
||||
### TTS (e.g., Qwen TTS series)
|
||||
|
||||
(Also available at **ComfyUI sidebar->Template->vLLM-Omni->vLLM-Omni TTS**)
|
||||
|
||||
@@ -14,17 +14,21 @@ from .comfyui_vllm_omni.nodes import (
|
||||
VLLMOmniARSampling,
|
||||
VLLMOmniDiffusionSampling,
|
||||
VLLMOmniGenerateImage,
|
||||
VLLMOmniGenerateVideo,
|
||||
VLLMOmniQwenTTSParams,
|
||||
VLLMOmniRemoteLoRA,
|
||||
VLLMOmniSamplingParamsList,
|
||||
VLLMOmniTTS,
|
||||
VLLMOmniUnderstanding,
|
||||
VLLMOmniVoiceClone,
|
||||
VLLMOmniWanParams,
|
||||
)
|
||||
|
||||
# A dictionary that contains all nodes you want to export with their names
|
||||
NODE_CLASS_MAPPINGS = {
|
||||
# === Generation ===
|
||||
"VLLMOmniGenerateImage": VLLMOmniGenerateImage,
|
||||
"VLLMOmniGenerateVideo": VLLMOmniGenerateVideo,
|
||||
"VLLMOmniUnderstanding": VLLMOmniUnderstanding,
|
||||
"VLLMOmniTTS": VLLMOmniTTS,
|
||||
"VLLMOmniVoiceClone": VLLMOmniVoiceClone,
|
||||
@@ -32,13 +36,16 @@ NODE_CLASS_MAPPINGS = {
|
||||
"VLLMOmniARSampling": VLLMOmniARSampling,
|
||||
"VLLMOmniDiffusionSampling": VLLMOmniDiffusionSampling,
|
||||
"VLLMOmniSamplingParamsList": VLLMOmniSamplingParamsList,
|
||||
"VLLMOmniRemoteLoRA": VLLMOmniRemoteLoRA,
|
||||
"VLLMOmniQwenTTSParams": VLLMOmniQwenTTSParams,
|
||||
"VLLMOmniWanParams": VLLMOmniWanParams,
|
||||
}
|
||||
|
||||
# A dictionary that contains the friendly/humanly readable titles for the nodes
|
||||
NODE_DISPLAY_NAME_MAPPINGS = {
|
||||
# === Generation ===
|
||||
"VLLMOmniGenerateImage": "Generate Image",
|
||||
"VLLMOmniGenerateVideo": "Generate Video",
|
||||
"VLLMOmniUnderstanding": "Multimodality Understanding",
|
||||
"VLLMOmniTTS": "TTS (Text to Speech)",
|
||||
"VLLMOmniVoiceClone": "TTS Voice Cloning",
|
||||
@@ -46,7 +53,9 @@ NODE_DISPLAY_NAME_MAPPINGS = {
|
||||
"VLLMOmniARSampling": "AR Sampling Params",
|
||||
"VLLMOmniDiffusionSampling": "Diffusion Sampling Params",
|
||||
"VLLMOmniSamplingParamsList": "Multi-Stage Sampling Params List",
|
||||
"VLLMOmniRemoteLoRA": "LoRA",
|
||||
"VLLMOmniQwenTTSParams": "Qwen TTS Params",
|
||||
"VLLMOmniWanParams": "Wan Video Params",
|
||||
}
|
||||
|
||||
WEB_DIRECTORY = "./web"
|
||||
@@ -11,6 +11,7 @@ from .utils.types import (
|
||||
AutoregressionSamplingParams,
|
||||
DiffusionSamplingParams,
|
||||
QwenTTSModelSpecificParams,
|
||||
WanModelSpecificParams,
|
||||
)
|
||||
from .utils.validators import (
|
||||
add_sampling_parameters_to_stage,
|
||||
@@ -56,6 +57,7 @@ class VLLMOmniGenerateImage(_VLLMOmniGenerateBase):
|
||||
# "video": ("VIDEO",),
|
||||
# "audio": ("AUDIO",),
|
||||
"sampling_params": ("SAMPLING_PARAMS",),
|
||||
"lora": ("REMOTE_LORA",),
|
||||
},
|
||||
}
|
||||
|
||||
@@ -76,9 +78,11 @@ class VLLMOmniGenerateImage(_VLLMOmniGenerateBase):
|
||||
audio: AudioInput | None = None, # Hidden & unused
|
||||
video: VideoInput | None = None, # Hidden & unused
|
||||
sampling_params: dict | list[dict] | None = None,
|
||||
lora: dict | None = None,
|
||||
**kwargs,
|
||||
):
|
||||
logger.info("Uncaught kwargs: %s", kwargs)
|
||||
if kwargs:
|
||||
logger.info("Uncaught kwargs: %s", kwargs)
|
||||
logger.debug("Got sampling params: %s", sampling_params)
|
||||
validate_model_and_sampling_params_types(model, sampling_params)
|
||||
if image is None and mask is not None:
|
||||
@@ -105,6 +109,7 @@ class VLLMOmniGenerateImage(_VLLMOmniGenerateBase):
|
||||
height=height,
|
||||
negative_prompt=negative_prompt,
|
||||
sampling_params=sampling_params,
|
||||
lora=lora,
|
||||
)
|
||||
return (output,)
|
||||
elif image is not None and audio is None and video is None:
|
||||
@@ -119,6 +124,7 @@ class VLLMOmniGenerateImage(_VLLMOmniGenerateBase):
|
||||
negative_prompt=negative_prompt,
|
||||
mask=mask,
|
||||
sampling_params=sampling_params,
|
||||
lora=lora,
|
||||
)
|
||||
return (output,)
|
||||
|
||||
@@ -136,11 +142,91 @@ class VLLMOmniGenerateImage(_VLLMOmniGenerateBase):
|
||||
audio=audio,
|
||||
video=video,
|
||||
sampling_params=sampling_params,
|
||||
lora=lora,
|
||||
)
|
||||
|
||||
return (output,)
|
||||
|
||||
|
||||
class VLLMOmniGenerateVideo(_VLLMOmniGenerateBase):
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
return {
|
||||
"required": {
|
||||
"url": ("STRING", {"default": "http://localhost:8000/v1"}),
|
||||
"model": ("STRING", {"default": "Wan-AI/Wan2.2-T2V-A14B-Diffusers"}),
|
||||
"prompt": ("STRING", {"multiline": True}),
|
||||
"negative_prompt": ("STRING", {"multiline": True, "default": ""}),
|
||||
"width": ("INT", {"default": 832, "min": 1}),
|
||||
"height": ("INT", {"default": 480, "min": 1}),
|
||||
"fps": ("INT", {"default": 16, "min": 1}),
|
||||
"num_frames": ("INT", {"default": 41, "min": 1}),
|
||||
},
|
||||
"optional": {
|
||||
"image": ("IMAGE",),
|
||||
"sampling_params": ("SAMPLING_PARAMS",),
|
||||
"lora": ("REMOTE_LORA",),
|
||||
"model_params": ("VIDEO_PARAMS",),
|
||||
},
|
||||
}
|
||||
|
||||
RETURN_TYPES = ("VIDEO",)
|
||||
RETURN_NAMES = ("video",)
|
||||
FUNCTION = "generate"
|
||||
|
||||
async def generate(
|
||||
self,
|
||||
url: str,
|
||||
model: str,
|
||||
prompt: str,
|
||||
width: int,
|
||||
height: int,
|
||||
fps: int,
|
||||
num_frames: int,
|
||||
negative_prompt: str | None = None,
|
||||
image: torch.Tensor | None = None,
|
||||
sampling_params: dict | list[dict] | None = None,
|
||||
model_params: dict | None = None,
|
||||
lora: dict | None = None,
|
||||
**kwargs,
|
||||
):
|
||||
if kwargs:
|
||||
logger.info("Uncaught kwargs: %s", kwargs)
|
||||
logger.debug("Got sampling params: %s", sampling_params)
|
||||
logger.debug("Got model params: %s", model_params)
|
||||
validate_model_and_sampling_params_types(model, sampling_params)
|
||||
|
||||
# Currently, all video generation models are single-stage diffusion models
|
||||
if isinstance(sampling_params, list):
|
||||
if len(sampling_params) != 1:
|
||||
raise ValueError(
|
||||
"Video generation expects a single sampling params group. "
|
||||
"Please provide one Diffusion sampling node."
|
||||
)
|
||||
sampling_params = sampling_params[0]
|
||||
|
||||
if sampling_params is not None:
|
||||
sampling_params.pop("type", None) # internal fields
|
||||
if model_params is not None:
|
||||
model_params.pop("type", None) # internal fields
|
||||
|
||||
client = VLLMOmniClient(url)
|
||||
output = await client.generate_video(
|
||||
model=model,
|
||||
prompt=prompt,
|
||||
image=image, # image present => i2v, absent => t2v
|
||||
width=width,
|
||||
height=height,
|
||||
num_frames=num_frames,
|
||||
fps=fps,
|
||||
negative_prompt=negative_prompt,
|
||||
sampling_params=sampling_params,
|
||||
lora=lora,
|
||||
model_params=model_params,
|
||||
)
|
||||
return (output,)
|
||||
|
||||
|
||||
class VLLMOmniUnderstanding(_VLLMOmniGenerateBase):
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
@@ -188,7 +274,8 @@ class VLLMOmniUnderstanding(_VLLMOmniGenerateBase):
|
||||
use_audio_in_video: bool = True,
|
||||
**kwargs,
|
||||
) -> tuple[str, AudioInput]:
|
||||
logger.info("Uncaught kwargs: %s", kwargs)
|
||||
if kwargs:
|
||||
logger.info("Uncaught kwargs: %s", kwargs)
|
||||
logger.debug("Got sampling params: %s", sampling_params)
|
||||
validate_model_and_sampling_params_types(model, sampling_params)
|
||||
|
||||
@@ -547,6 +634,52 @@ class VLLMOmniSamplingParamsList:
|
||||
return (params,)
|
||||
|
||||
|
||||
class VLLMOmniRemoteLoRA:
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
return {
|
||||
"required": {
|
||||
"local_path": ("STRING", {"default": ""}),
|
||||
"name": ("STRING", {"default": ""}),
|
||||
"scale": (
|
||||
"FLOAT",
|
||||
{"default": 1.0, "min": 0.0, "max": 1.0, "step": 0.1},
|
||||
),
|
||||
"int_id": (
|
||||
"INT",
|
||||
{
|
||||
"default": 0,
|
||||
"min": 0,
|
||||
"step": 1,
|
||||
"tooltip": "0 means it is not set and the server can derive it.",
|
||||
},
|
||||
),
|
||||
}
|
||||
}
|
||||
|
||||
RETURN_TYPES = ("REMOTE_LORA",)
|
||||
RETURN_NAMES = ("lora",)
|
||||
FUNCTION = "get_lora"
|
||||
CATEGORY = "vLLM-Omni"
|
||||
|
||||
@classmethod
|
||||
def VALIDATE_INPUTS(cls, local_path, name) -> str | Literal[True]:
|
||||
if not local_path.strip() or not name.strip():
|
||||
return "Both local_path and name must be provided."
|
||||
return True
|
||||
|
||||
def get_lora(self, local_path: str, name: str, scale: float, int_id: int):
|
||||
local_path = local_path.strip()
|
||||
name = name.strip()
|
||||
lora = {
|
||||
"local_path": local_path or None,
|
||||
"name": name or None,
|
||||
"scale": float(scale),
|
||||
"int_id": int(int_id) if int_id > 0 else None,
|
||||
}
|
||||
return (lora,)
|
||||
|
||||
|
||||
class VLLMOmniQwenTTSParams:
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
@@ -572,3 +705,32 @@ class VLLMOmniQwenTTSParams:
|
||||
|
||||
def get_params(self, **kwargs):
|
||||
return (QwenTTSModelSpecificParams(kwargs),)
|
||||
|
||||
|
||||
class VLLMOmniWanParams:
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
return {
|
||||
"required": {
|
||||
"guidance_scale_2": (
|
||||
"FLOAT",
|
||||
{"default": 4.0, "min": 0.0, "max": 20.0, "step": 0.1},
|
||||
),
|
||||
"boundary_ratio": (
|
||||
"FLOAT",
|
||||
{"default": 0.875, "min": 0.0, "max": 1.0, "step": 0.001},
|
||||
),
|
||||
"flow_shift": (
|
||||
"FLOAT",
|
||||
{"default": 5.0, "min": 0.0, "max": 100.0, "step": 0.1},
|
||||
),
|
||||
}
|
||||
}
|
||||
|
||||
RETURN_TYPES = ("VIDEO_PARAMS",)
|
||||
RETURN_NAMES = ("Wan video params",)
|
||||
FUNCTION = "get_params"
|
||||
CATEGORY = "vLLM-Omni/Video Params"
|
||||
|
||||
def get_params(self, **kwargs):
|
||||
return (WanModelSpecificParams(kwargs),)
|
||||
@@ -6,6 +6,7 @@ The image generation part is derived from dougbtv/comfyui-vllm-omni by Doug (@do
|
||||
Original source at https://github.com/dougbtv/comfyui-vllm-omni, distributed under the MIT License.
|
||||
"""
|
||||
|
||||
import json
|
||||
from typing import Any
|
||||
|
||||
import aiohttp
|
||||
@@ -17,6 +18,7 @@ from .format import (
|
||||
audio_to_base64,
|
||||
base64_to_audio,
|
||||
base64_to_image_tensor,
|
||||
base64_to_video,
|
||||
bytes_to_audio,
|
||||
image_tensor_to_base64,
|
||||
image_tensor_to_png_bytes,
|
||||
@@ -30,7 +32,7 @@ logger = get_logger(__name__)
|
||||
|
||||
|
||||
class VLLMOmniClient:
|
||||
def __init__(self, base_url: str, timeout: float = 300.0):
|
||||
def __init__(self, base_url: str, timeout: float | None = None):
|
||||
self.base_url = base_url
|
||||
self.timeout = aiohttp.ClientTimeout(total=timeout)
|
||||
|
||||
@@ -43,12 +45,13 @@ class VLLMOmniClient:
|
||||
height: int,
|
||||
negative_prompt: str | None = None,
|
||||
sampling_params: dict | None = None,
|
||||
lora: dict | None = None,
|
||||
) -> torch.Tensor:
|
||||
"""Run text-to-image generation via DALLE API"""
|
||||
await self._check_model_exist(model)
|
||||
|
||||
size = f"{width}x{height}"
|
||||
payload = {
|
||||
payload: dict[str, Any] = {
|
||||
"model": model,
|
||||
"prompt": prompt,
|
||||
"size": size,
|
||||
@@ -58,6 +61,8 @@ class VLLMOmniClient:
|
||||
payload["negative_prompt"] = negative_prompt
|
||||
if sampling_params is not None:
|
||||
payload.update(sampling_params)
|
||||
if lora is not None:
|
||||
payload["lora"] = lora
|
||||
logger.debug("img gen payload: %s", payload)
|
||||
|
||||
url = self.base_url + "/images/generations"
|
||||
@@ -110,6 +115,7 @@ class VLLMOmniClient:
|
||||
negative_prompt: str | None = None,
|
||||
mask: torch.Tensor | None = None,
|
||||
sampling_params: dict | None = None,
|
||||
lora: dict | None = None,
|
||||
) -> torch.Tensor:
|
||||
"""Run image editing via DALLE API"""
|
||||
await self._check_model_exist(model)
|
||||
@@ -131,6 +137,8 @@ class VLLMOmniClient:
|
||||
if sampling_params is not None:
|
||||
for k, v in sampling_params.items():
|
||||
form.add_field(k, str(v))
|
||||
if lora is not None:
|
||||
form.add_field("lora", json.dumps(lora, ensure_ascii=False))
|
||||
if mask is not None:
|
||||
mask_filename = "mask.png"
|
||||
form.add_field(
|
||||
@@ -183,6 +191,7 @@ class VLLMOmniClient:
|
||||
audio: AudioInput | None = None,
|
||||
video: VideoInput | None = None,
|
||||
sampling_params: dict | list[dict] | None = None,
|
||||
lora: dict | None = None,
|
||||
) -> torch.Tensor:
|
||||
payload = VLLMOmniClient._prepare_chat_completion_messages(
|
||||
model=model,
|
||||
@@ -193,6 +202,8 @@ class VLLMOmniClient:
|
||||
video=video,
|
||||
sampling_params=sampling_params,
|
||||
modalities=["image"],
|
||||
# === below are additional `extra_body` fields, handled by **kwargs ===
|
||||
lora=lora,
|
||||
)
|
||||
choices = await self._generate_base_chat_completion(model, payload)
|
||||
|
||||
@@ -206,6 +217,79 @@ class VLLMOmniClient:
|
||||
|
||||
return torch.stack(image_tensors, dim=0)
|
||||
|
||||
async def generate_video(
|
||||
self,
|
||||
*,
|
||||
model: str,
|
||||
prompt: str,
|
||||
width: int,
|
||||
height: int,
|
||||
num_frames: int,
|
||||
fps: int,
|
||||
negative_prompt: str | None = None,
|
||||
image: torch.Tensor | None = None,
|
||||
sampling_params: dict | None = None,
|
||||
model_params: dict | None = None,
|
||||
lora: dict | None = None,
|
||||
**extra_body,
|
||||
) -> VideoInput:
|
||||
form = aiohttp.FormData()
|
||||
form.add_field("model", model)
|
||||
form.add_field("prompt", prompt)
|
||||
form.add_field("width", str(width))
|
||||
form.add_field("height", str(height))
|
||||
form.add_field("num_frames", str(num_frames))
|
||||
form.add_field("fps", str(fps))
|
||||
if negative_prompt:
|
||||
form.add_field("negative_prompt", negative_prompt)
|
||||
if sampling_params is not None:
|
||||
for k, v in sampling_params.items():
|
||||
form.add_field(k, str(v))
|
||||
if model_params is not None:
|
||||
for k, v in model_params.items():
|
||||
form.add_field(k, str(v))
|
||||
if lora is not None:
|
||||
form.add_field("lora", json.dumps(lora, ensure_ascii=False))
|
||||
if extra_body:
|
||||
form.add_field("extra_body", json.dumps(extra_body, ensure_ascii=False))
|
||||
|
||||
if image is not None:
|
||||
image_filename = "image.png" # Required for multipart form
|
||||
form.add_field(
|
||||
"input_reference",
|
||||
image_tensor_to_png_bytes(image, image_filename),
|
||||
filename=image_filename,
|
||||
content_type="image/png",
|
||||
)
|
||||
|
||||
url = self.base_url + "/videos"
|
||||
async with aiohttp.ClientSession(timeout=self.timeout) as session:
|
||||
try:
|
||||
async with session.post(url, data=form) as response:
|
||||
if not response.ok:
|
||||
error_text = await response.text()
|
||||
raise (ValueError if response.status < 500 else RuntimeError)(
|
||||
f"vLLM-Omni API returned status {response.status}: {error_text}"
|
||||
)
|
||||
|
||||
try:
|
||||
data = await response.json()
|
||||
except aiohttp.ContentTypeError as e:
|
||||
raise RuntimeError(f"Invalid JSON response from vLLM-Omni: {e}")
|
||||
except aiohttp.ClientError as e:
|
||||
raise RuntimeError(f"Network error connecting to vLLM-Omni at {url}: {e}")
|
||||
|
||||
if "data" not in data:
|
||||
raise RuntimeError("API response missing 'data' field - expected OpenAI DALL-E format")
|
||||
if not data["data"]:
|
||||
raise RuntimeError("API returned empty data array")
|
||||
try:
|
||||
base64_str = data["data"][0]["b64_json"]
|
||||
except (KeyError, IndexError):
|
||||
raise RuntimeError("API response missing 'b64_json' field in first data item")
|
||||
|
||||
return base64_to_video(base64_str)
|
||||
|
||||
async def generate_understanding_chat_completion(
|
||||
self,
|
||||
*,
|
||||
|
||||
@@ -6,12 +6,17 @@ Original source at https://github.com/dougbtv/comfyui-vllm-omni, distributed und
|
||||
|
||||
import base64
|
||||
import mimetypes
|
||||
from fractions import Fraction
|
||||
from io import BytesIO
|
||||
|
||||
import av
|
||||
import numpy as np
|
||||
import torch
|
||||
from av.audio.frame import AudioFrame
|
||||
from av.audio.resampler import AudioResampler
|
||||
from av.video.frame import VideoFrame
|
||||
from comfy_api.input import AudioInput, VideoInput
|
||||
from comfy_api.latest import InputImpl, Types
|
||||
from comfy_extras import nodes_audio
|
||||
from PIL import Image
|
||||
|
||||
@@ -138,6 +143,71 @@ def video_to_base64(video: VideoInput, filename: str = "video.mp4") -> str:
|
||||
return f"data:{mime_type};base64,{base64_str}"
|
||||
|
||||
|
||||
def base64_to_video(base64_str: str) -> VideoInput:
|
||||
if base64_str.startswith("data:video"):
|
||||
_, base64_str = base64_str.split(",", 1)
|
||||
|
||||
try:
|
||||
video_bytes = base64.b64decode(base64_str)
|
||||
except Exception as e:
|
||||
raise ValueError(f"Invalid base64 string: {e}")
|
||||
|
||||
video_buffer = BytesIO(video_bytes)
|
||||
|
||||
try:
|
||||
with av.open(video_buffer, mode="r") as container:
|
||||
video_stream = next((s for s in container.streams if s.type == "video"), None)
|
||||
if video_stream is None:
|
||||
raise ValueError("No video stream found in decoded payload.")
|
||||
|
||||
frames = []
|
||||
for frame in container.decode(video_stream):
|
||||
if not isinstance(frame, VideoFrame):
|
||||
continue
|
||||
image = frame.to_ndarray(format="rgb24")
|
||||
frames.append(torch.from_numpy(image).float() / 255.0)
|
||||
|
||||
if len(frames) == 0:
|
||||
raise ValueError("No video frames found in decoded payload.")
|
||||
|
||||
images = torch.stack(frames, dim=0)
|
||||
frame_rate = Fraction(video_stream.average_rate) if video_stream.average_rate else Fraction(1)
|
||||
|
||||
audio: AudioInput | None = None
|
||||
if len(container.streams.audio):
|
||||
audio_stream = container.streams.audio[-1]
|
||||
audio_frames = []
|
||||
resampler = AudioResampler(format="fltp")
|
||||
for frame in container.decode(audio_stream):
|
||||
if not isinstance(frame, AudioFrame):
|
||||
continue
|
||||
resampled = resampler.resample(frame)
|
||||
if not isinstance(resampled, list):
|
||||
resampled = [resampled]
|
||||
for audio_frame in resampled:
|
||||
if audio_frame is not None:
|
||||
audio_frames.append(audio_frame.to_ndarray())
|
||||
|
||||
if len(audio_frames) > 0:
|
||||
audio_data = np.concatenate(audio_frames, axis=1)
|
||||
sample_rate = int(audio_stream.sample_rate) if audio_stream.sample_rate else 1
|
||||
audio = {
|
||||
"waveform": torch.from_numpy(audio_data).unsqueeze(0),
|
||||
"sample_rate": sample_rate,
|
||||
}
|
||||
|
||||
components = Types.VideoComponents(
|
||||
images=images,
|
||||
frame_rate=frame_rate,
|
||||
audio=audio,
|
||||
metadata=container.metadata if container.metadata else None,
|
||||
)
|
||||
except Exception as e:
|
||||
raise RuntimeError(f"Failed to decode video: {e}")
|
||||
|
||||
return InputImpl.VideoFromComponents(components)
|
||||
|
||||
|
||||
def audio_to_bytes(audio: AudioInput, filename: str = "audio.mp3", quality: str = "128k") -> BytesIO:
|
||||
waveform = audio["waveform"][0] # Shape: (C, T)
|
||||
sample_rate = audio["sample_rate"]
|
||||
|
||||
@@ -23,6 +23,10 @@ class QwenTTSModelSpecificParams(dict):
|
||||
pass
|
||||
|
||||
|
||||
class WanModelSpecificParams(dict):
|
||||
pass
|
||||
|
||||
|
||||
class ModelMode(Enum):
|
||||
IMAGE_GENERATION = auto()
|
||||
VIDEO_GENERATION = auto()
|
||||
|
||||
@@ -16,7 +16,7 @@ def validate_model_and_sampling_params_types(
|
||||
# Skip if no spec or no sampling params
|
||||
pipeline_spec, _ = lookup_model_spec(model_name)
|
||||
if pipeline_spec is None:
|
||||
logger.info("skipping sampling params check because spec is not found")
|
||||
logger.info(f"skipping sampling params check because spec for {model_name} is not found")
|
||||
return
|
||||
if sampling_param_list is None:
|
||||
return
|
||||
|
||||
|
After Width: | Height: | Size: 129 KiB |
@@ -0,0 +1,513 @@
|
||||
{
|
||||
"id": "3aea1d08-7de5-4997-ba71-b2f66ad72a21",
|
||||
"revision": 0,
|
||||
"last_node_id": 14,
|
||||
"last_link_id": 28,
|
||||
"nodes": [
|
||||
{
|
||||
"id": 10,
|
||||
"type": "MarkdownNote",
|
||||
"pos": [
|
||||
227.99306819575278,
|
||||
-231.24143306069843
|
||||
],
|
||||
"size": [
|
||||
240.3326478922474,
|
||||
136.3505820791881
|
||||
],
|
||||
"flags": {},
|
||||
"order": 0,
|
||||
"mode": 0,
|
||||
"inputs": [],
|
||||
"outputs": [],
|
||||
"title": "Note: Task and Input",
|
||||
"properties": {},
|
||||
"widgets_values": [
|
||||
"vLLM-Omni nodes are categorized based on the output modality. The \"Generate Video\" node supports both text-to-video generation and image-to-video generation. The node will create corresponding payload depending on whether an input image is present or not."
|
||||
],
|
||||
"color": "#432",
|
||||
"bgcolor": "#000"
|
||||
},
|
||||
{
|
||||
"id": 4,
|
||||
"type": "LoadImage",
|
||||
"pos": [
|
||||
496.31859627609606,
|
||||
-229.71277089860084
|
||||
],
|
||||
"size": [
|
||||
270,
|
||||
314
|
||||
],
|
||||
"flags": {},
|
||||
"order": 1,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"localized_name": "image",
|
||||
"name": "image",
|
||||
"type": "COMBO",
|
||||
"widget": {
|
||||
"name": "image"
|
||||
},
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"localized_name": "choose file to upload",
|
||||
"name": "upload",
|
||||
"type": "IMAGEUPLOAD",
|
||||
"widget": {
|
||||
"name": "upload"
|
||||
},
|
||||
"link": null
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"localized_name": "IMAGE",
|
||||
"name": "IMAGE",
|
||||
"type": "IMAGE",
|
||||
"links": [
|
||||
25
|
||||
]
|
||||
},
|
||||
{
|
||||
"localized_name": "MASK",
|
||||
"name": "MASK",
|
||||
"type": "MASK",
|
||||
"links": null
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "LoadImage"
|
||||
},
|
||||
"widgets_values": [
|
||||
"cute-cat.jpg",
|
||||
"image"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 8,
|
||||
"type": "VLLMOmniDiffusionSampling",
|
||||
"pos": [
|
||||
465.5140218220927,
|
||||
148.80072646828987
|
||||
],
|
||||
"size": [
|
||||
284.205078125,
|
||||
226
|
||||
],
|
||||
"flags": {},
|
||||
"order": 2,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"localized_name": "n",
|
||||
"name": "n",
|
||||
"type": "INT",
|
||||
"widget": {
|
||||
"name": "n"
|
||||
},
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"localized_name": "num_inference_steps",
|
||||
"name": "num_inference_steps",
|
||||
"type": "INT",
|
||||
"widget": {
|
||||
"name": "num_inference_steps"
|
||||
},
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"localized_name": "guidance_scale",
|
||||
"name": "guidance_scale",
|
||||
"type": "FLOAT",
|
||||
"widget": {
|
||||
"name": "guidance_scale"
|
||||
},
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"localized_name": "true_cfg_scale",
|
||||
"name": "true_cfg_scale",
|
||||
"type": "FLOAT",
|
||||
"widget": {
|
||||
"name": "true_cfg_scale"
|
||||
},
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"localized_name": "vae_use_slicing",
|
||||
"name": "vae_use_slicing",
|
||||
"type": "BOOLEAN",
|
||||
"widget": {
|
||||
"name": "vae_use_slicing"
|
||||
},
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"localized_name": "vae_use_tiling",
|
||||
"name": "vae_use_tiling",
|
||||
"type": "BOOLEAN",
|
||||
"widget": {
|
||||
"name": "vae_use_tiling"
|
||||
},
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"localized_name": "seed",
|
||||
"name": "seed",
|
||||
"type": "INT",
|
||||
"widget": {
|
||||
"name": "seed"
|
||||
},
|
||||
"link": null
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"localized_name": "diffusion sampling params",
|
||||
"name": "diffusion sampling params",
|
||||
"type": "SAMPLING_PARAMS",
|
||||
"links": [
|
||||
27
|
||||
]
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "VLLMOmniDiffusionSampling"
|
||||
},
|
||||
"widgets_values": [
|
||||
1,
|
||||
50,
|
||||
1,
|
||||
1,
|
||||
false,
|
||||
false,
|
||||
1992,
|
||||
"randomize"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 14,
|
||||
"type": "VLLMOmniWanParams",
|
||||
"pos": [
|
||||
518.6005521845747,
|
||||
428.4455663719664
|
||||
],
|
||||
"size": [
|
||||
270,
|
||||
106
|
||||
],
|
||||
"flags": {},
|
||||
"order": 3,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"localized_name": "guidance_scale_2",
|
||||
"name": "guidance_scale_2",
|
||||
"type": "FLOAT",
|
||||
"widget": {
|
||||
"name": "guidance_scale_2"
|
||||
},
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"localized_name": "boundary_ratio",
|
||||
"name": "boundary_ratio",
|
||||
"type": "FLOAT",
|
||||
"widget": {
|
||||
"name": "boundary_ratio"
|
||||
},
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"localized_name": "flow_shift",
|
||||
"name": "flow_shift",
|
||||
"type": "FLOAT",
|
||||
"widget": {
|
||||
"name": "flow_shift"
|
||||
},
|
||||
"link": null
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"localized_name": "Wan video params",
|
||||
"name": "Wan video params",
|
||||
"type": "VIDEO_PARAMS",
|
||||
"links": [
|
||||
28
|
||||
]
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "VLLMOmniWanParams"
|
||||
},
|
||||
"widgets_values": [
|
||||
4,
|
||||
0.875,
|
||||
5
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 12,
|
||||
"type": "VLLMOmniGenerateVideo",
|
||||
"pos": [
|
||||
827.2566336087867,
|
||||
-73.77449831827545
|
||||
],
|
||||
"size": [
|
||||
400,
|
||||
346
|
||||
],
|
||||
"flags": {},
|
||||
"order": 4,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"localized_name": "image",
|
||||
"name": "image",
|
||||
"shape": 7,
|
||||
"type": "IMAGE",
|
||||
"link": 25
|
||||
},
|
||||
{
|
||||
"localized_name": "sampling_params",
|
||||
"name": "sampling_params",
|
||||
"shape": 7,
|
||||
"type": "SAMPLING_PARAMS",
|
||||
"link": 27
|
||||
},
|
||||
{
|
||||
"localized_name": "lora",
|
||||
"name": "lora",
|
||||
"shape": 7,
|
||||
"type": "REMOTE_LORA",
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"localized_name": "model_params",
|
||||
"name": "model_params",
|
||||
"shape": 7,
|
||||
"type": "VIDEO_PARAMS",
|
||||
"link": 28
|
||||
},
|
||||
{
|
||||
"localized_name": "url",
|
||||
"name": "url",
|
||||
"type": "STRING",
|
||||
"widget": {
|
||||
"name": "url"
|
||||
},
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"localized_name": "model",
|
||||
"name": "model",
|
||||
"type": "STRING",
|
||||
"widget": {
|
||||
"name": "model"
|
||||
},
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"localized_name": "prompt",
|
||||
"name": "prompt",
|
||||
"type": "STRING",
|
||||
"widget": {
|
||||
"name": "prompt"
|
||||
},
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"localized_name": "negative_prompt",
|
||||
"name": "negative_prompt",
|
||||
"type": "STRING",
|
||||
"widget": {
|
||||
"name": "negative_prompt"
|
||||
},
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"localized_name": "width",
|
||||
"name": "width",
|
||||
"type": "INT",
|
||||
"widget": {
|
||||
"name": "width"
|
||||
},
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"localized_name": "height",
|
||||
"name": "height",
|
||||
"type": "INT",
|
||||
"widget": {
|
||||
"name": "height"
|
||||
},
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"localized_name": "fps",
|
||||
"name": "fps",
|
||||
"type": "INT",
|
||||
"widget": {
|
||||
"name": "fps"
|
||||
},
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"localized_name": "num_frames",
|
||||
"name": "num_frames",
|
||||
"type": "INT",
|
||||
"widget": {
|
||||
"name": "num_frames"
|
||||
},
|
||||
"link": null
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"localized_name": "video",
|
||||
"name": "video",
|
||||
"type": "VIDEO",
|
||||
"links": [
|
||||
26
|
||||
]
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "VLLMOmniGenerateVideo"
|
||||
},
|
||||
"widgets_values": [
|
||||
"http://localhost:8000/v1",
|
||||
"Wan-AI/Wan2.2-TI2V-5B-Diffusers",
|
||||
"Make the cat in the attached video yawning. Eye closed and mouth wide open. Keep the original realistic style.",
|
||||
"Cartoonish",
|
||||
900,
|
||||
1200,
|
||||
12,
|
||||
41
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 13,
|
||||
"type": "SaveVideo",
|
||||
"pos": [
|
||||
1257.5441369501657,
|
||||
-44.12956394419863
|
||||
],
|
||||
"size": [
|
||||
374,
|
||||
372.34896826553796
|
||||
],
|
||||
"flags": {},
|
||||
"order": 5,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"localized_name": "video",
|
||||
"name": "video",
|
||||
"type": "VIDEO",
|
||||
"link": 26
|
||||
},
|
||||
{
|
||||
"localized_name": "filename_prefix",
|
||||
"name": "filename_prefix",
|
||||
"type": "STRING",
|
||||
"widget": {
|
||||
"name": "filename_prefix"
|
||||
},
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"localized_name": "format",
|
||||
"name": "format",
|
||||
"type": "COMBO",
|
||||
"widget": {
|
||||
"name": "format"
|
||||
},
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"localized_name": "codec",
|
||||
"name": "codec",
|
||||
"type": "COMBO",
|
||||
"widget": {
|
||||
"name": "codec"
|
||||
},
|
||||
"link": null
|
||||
}
|
||||
],
|
||||
"outputs": [],
|
||||
"properties": {},
|
||||
"widgets_values": [
|
||||
"video/ComfyUI",
|
||||
"auto",
|
||||
"auto"
|
||||
]
|
||||
}
|
||||
],
|
||||
"links": [
|
||||
[
|
||||
25,
|
||||
4,
|
||||
0,
|
||||
12,
|
||||
0,
|
||||
"IMAGE"
|
||||
],
|
||||
[
|
||||
26,
|
||||
12,
|
||||
0,
|
||||
13,
|
||||
0,
|
||||
"VIDEO"
|
||||
],
|
||||
[
|
||||
27,
|
||||
8,
|
||||
0,
|
||||
12,
|
||||
1,
|
||||
"SAMPLING_PARAMS"
|
||||
],
|
||||
[
|
||||
28,
|
||||
14,
|
||||
0,
|
||||
12,
|
||||
3,
|
||||
"VIDEO_PARAMS"
|
||||
]
|
||||
],
|
||||
"groups": [
|
||||
{
|
||||
"id": 1,
|
||||
"title": "Input",
|
||||
"bounding": [
|
||||
213.1706010087147,
|
||||
-313.20095750667554,
|
||||
560.8246471437027,
|
||||
407.4246532472182
|
||||
],
|
||||
"color": "#3f789e",
|
||||
"font_size": 24,
|
||||
"flags": {}
|
||||
}
|
||||
],
|
||||
"config": {},
|
||||
"extra": {
|
||||
"workflowRendererVersion": "LG",
|
||||
"ds": {
|
||||
"scale": 1.1469075819486894,
|
||||
"offset": [
|
||||
101.7632950847089,
|
||||
492.29121889347806
|
||||
]
|
||||
}
|
||||
},
|
||||
"version": 0.4
|
||||
}
|
||||
@@ -0,0 +1,100 @@
|
||||
# Qwen3-TTS Benchmark
|
||||
|
||||
Benchmarks for Qwen3-TTS text-to-speech models, comparing vLLM-Omni streaming serving against HuggingFace Transformers offline inference.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
```bash
|
||||
pip install matplotlib aiohttp soundfile numpy tqdm
|
||||
pip install qwen_tts # for HF baseline
|
||||
```
|
||||
|
||||
## Quick Start
|
||||
|
||||
Run the full benchmark (vllm-omni + HF baseline) with a single command:
|
||||
|
||||
```bash
|
||||
cd benchmarks/qwen3-tts
|
||||
bash run_benchmark.sh
|
||||
```
|
||||
|
||||
Results (JSON + PNG plots) are saved to `results/`.
|
||||
|
||||
### Common options
|
||||
|
||||
```bash
|
||||
# Only vllm-omni (skip HF baseline)
|
||||
bash run_benchmark.sh --async-only
|
||||
|
||||
# Only HF baseline
|
||||
bash run_benchmark.sh --hf-only
|
||||
|
||||
# Use a different model (e.g. 1.7B)
|
||||
MODEL=Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice bash run_benchmark.sh --async-only
|
||||
|
||||
# Use batch_size=4 config for higher throughput
|
||||
STAGE_CONFIG=vllm_omni/configs/qwen3_tts_bs4.yaml bash run_benchmark.sh --async-only
|
||||
|
||||
# Custom GPU, prompt count, concurrency levels
|
||||
GPU_DEVICE=1 NUM_PROMPTS=20 CONCURRENCY="1 4" bash run_benchmark.sh
|
||||
```
|
||||
|
||||
## Manual Steps
|
||||
|
||||
### 1) Start the vLLM-Omni server
|
||||
|
||||
```bash
|
||||
CUDA_VISIBLE_DEVICES=0 python -m vllm_omni.entrypoints.cli.main serve \
|
||||
"Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice" \
|
||||
--omni --host 127.0.0.1 --port 8000 \
|
||||
--stage-configs-path benchmarks/qwen3-tts/vllm_omni/configs/qwen3_tts_bs1.yaml \
|
||||
--trust-remote-code
|
||||
```
|
||||
|
||||
### 2) Run online serving benchmark
|
||||
|
||||
```bash
|
||||
python vllm_omni/bench_tts_serve.py \
|
||||
--port 8000 \
|
||||
--num-prompts 50 \
|
||||
--max-concurrency 1 4 10 \
|
||||
--config-name "async_chunk" \
|
||||
--result-dir results/
|
||||
```
|
||||
|
||||
### 3) Run HuggingFace baseline
|
||||
|
||||
```bash
|
||||
python transformers/bench_tts_hf.py \
|
||||
--model "Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice" \
|
||||
--num-prompts 50 \
|
||||
--gpu-device 0 \
|
||||
--result-dir results/
|
||||
```
|
||||
|
||||
### 4) Generate comparison plots
|
||||
|
||||
```bash
|
||||
python plot_results.py \
|
||||
--results results/bench_async_chunk_*.json results/bench_hf_transformers_*.json \
|
||||
--labels "vllm-omni" "hf_transformers" \
|
||||
--output results/comparison.png
|
||||
```
|
||||
|
||||
## Stage Configs
|
||||
|
||||
| Config | Batch Size | Description |
|
||||
|--------|:----------:|-------------|
|
||||
| `vllm_omni/configs/qwen3_tts_bs1.yaml` | 1 | Single-request processing (lower latency) |
|
||||
| `vllm_omni/configs/qwen3_tts_bs4.yaml` | 4 | Concurrent request processing (higher throughput) |
|
||||
|
||||
Both configs use a 2-stage pipeline (Talker -> Code2Wav) with `async_chunk` streaming enabled. The `SharedMemoryConnector` streams codec frames (25-frame chunks with 25-frame context overlap) between stages.
|
||||
|
||||
The model is specified via the CLI `--model` flag (or `MODEL` env var in `run_benchmark.sh`), so the same configs work for both the 0.6B and 1.7B model variants.
|
||||
|
||||
## Metrics
|
||||
|
||||
- **TTFP (Time to First Audio Packet)**: Time from request to first audio chunk (streaming latency)
|
||||
- **E2E (End-to-End Latency)**: Total time from request to complete audio response
|
||||
- **RTF (Real-Time Factor)**: E2E latency / audio duration. RTF < 1.0 means faster-than-real-time synthesis
|
||||
- **Throughput**: Total audio seconds generated per wall-clock second
|
||||
@@ -0,0 +1,254 @@
|
||||
"""Plot Qwen3-TTS benchmark results.
|
||||
|
||||
Generates comparison bar charts similar to the async_chunk design doc:
|
||||
- TTFP (Time-to-First-Packet) across concurrency levels
|
||||
- E2E latency across concurrency levels
|
||||
- RTF (Real-Time Factor) across concurrency levels
|
||||
|
||||
Usage:
|
||||
# Compare two configs (async_chunk vs no_async_chunk):
|
||||
python plot_results.py \
|
||||
--results results/bench_async_chunk_*.json results/bench_no_async_chunk_*.json \
|
||||
--labels "async_chunk" "no_async_chunk" \
|
||||
--output results/qwen3_tts_benchmark.png
|
||||
|
||||
# Single config:
|
||||
python plot_results.py \
|
||||
--results results/bench_async_chunk_*.json \
|
||||
--labels "async_chunk" \
|
||||
--output results/qwen3_tts_benchmark.png
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
import matplotlib.pyplot as plt
|
||||
import numpy as np
|
||||
|
||||
|
||||
def load_results(result_files: list[str]) -> list[list[dict]]:
|
||||
"""Load benchmark results from JSON files."""
|
||||
all_results = []
|
||||
for f in result_files:
|
||||
with open(f) as fh:
|
||||
data = json.load(fh)
|
||||
all_results.append(data)
|
||||
return all_results
|
||||
|
||||
|
||||
def plot_comparison(
|
||||
all_results: list[list[dict]],
|
||||
labels: list[str],
|
||||
output_path: str,
|
||||
title_prefix: str = "Qwen3-TTS",
|
||||
):
|
||||
"""Generate comparison bar charts."""
|
||||
n_configs = len(all_results)
|
||||
|
||||
# Collect concurrency levels present in ALL configs (skip missing data)
|
||||
all_concurrencies = [set(r["concurrency"] for r in results) for results in all_results]
|
||||
concurrencies = sorted(set.union(*all_concurrencies))
|
||||
|
||||
# Build data arrays, using None for missing concurrency levels
|
||||
ttfp_data = {label: [] for label in labels}
|
||||
e2e_data = {label: [] for label in labels}
|
||||
rtf_data = {label: [] for label in labels}
|
||||
throughput_data = {label: [] for label in labels}
|
||||
|
||||
for results, label in zip(all_results, labels):
|
||||
conc_map = {r["concurrency"]: r for r in results}
|
||||
for c in concurrencies:
|
||||
r = conc_map.get(c)
|
||||
ttfp_data[label].append(r["mean_ttfp_ms"] if r else None)
|
||||
e2e_data[label].append(r["mean_e2e_ms"] if r else None)
|
||||
rtf_data[label].append(r["mean_rtf"] if r else None)
|
||||
throughput_data[label].append(r["audio_throughput"] if r else None)
|
||||
|
||||
fig, axes = plt.subplots(2, 2, figsize=(14, 10))
|
||||
fig.suptitle(f"{title_prefix} Performance Benchmark", fontsize=16, fontweight="bold")
|
||||
|
||||
x = np.arange(len(concurrencies))
|
||||
width = 0.35 if n_configs == 2 else 0.5
|
||||
if n_configs > 1:
|
||||
offsets = np.linspace(-width / 2 * (n_configs - 1), width / 2 * (n_configs - 1), n_configs)
|
||||
else:
|
||||
offsets = [0]
|
||||
|
||||
colors = ["#2196F3", "#FF5722", "#4CAF50", "#FFC107"]
|
||||
|
||||
def plot_metric(ax, data_dict, ylabel, title, fmt=".1f"):
|
||||
bars = []
|
||||
for i, (label, values) in enumerate(data_dict.items()):
|
||||
# Replace None with 0 for plotting, but track which are missing
|
||||
plot_values = [v if v is not None else 0 for v in values]
|
||||
color = colors[i % len(colors)]
|
||||
bar = ax.bar(x + offsets[i], plot_values, width, label=label, color=color, alpha=0.85)
|
||||
bars.append(bar)
|
||||
# Add value labels on bars (skip None/missing data)
|
||||
max_val = max((v for v in values if v is not None), default=1)
|
||||
for rect, val in zip(bar, values):
|
||||
if val is not None and val > 0:
|
||||
ax.text(
|
||||
rect.get_x() + rect.get_width() / 2,
|
||||
rect.get_height() + max_val * 0.02,
|
||||
f"{val:{fmt}}",
|
||||
ha="center",
|
||||
va="bottom",
|
||||
fontsize=9,
|
||||
fontweight="bold",
|
||||
)
|
||||
ax.set_xlabel("Concurrency", fontsize=12)
|
||||
ax.set_ylabel(ylabel, fontsize=12)
|
||||
ax.set_title(title, fontsize=13, fontweight="bold")
|
||||
ax.set_xticks(x)
|
||||
ax.set_xticklabels([str(c) for c in concurrencies])
|
||||
ax.legend(fontsize=10)
|
||||
ax.grid(axis="y", alpha=0.3)
|
||||
ax.set_axisbelow(True)
|
||||
|
||||
plot_metric(axes[0, 0], ttfp_data, "TTFP (ms)", "Time to First Audio Packet (TTFP)")
|
||||
plot_metric(axes[0, 1], e2e_data, "E2E Latency (ms)", "End-to-End Latency (E2E)")
|
||||
plot_metric(axes[1, 0], rtf_data, "RTF", "Real-Time Factor (RTF)", fmt=".3f")
|
||||
plot_metric(axes[1, 1], throughput_data, "Audio-sec / Wall-sec", "Audio Throughput", fmt=".2f")
|
||||
|
||||
plt.tight_layout()
|
||||
plt.savefig(output_path, dpi=150, bbox_inches="tight")
|
||||
print(f"Plot saved to {output_path}")
|
||||
plt.close()
|
||||
|
||||
|
||||
def plot_single_summary(results: list[dict], label: str, output_path: str):
|
||||
"""Generate a single-config summary with percentile breakdown."""
|
||||
concurrencies = [r["concurrency"] for r in results]
|
||||
|
||||
fig, axes = plt.subplots(1, 3, figsize=(16, 5))
|
||||
fig.suptitle(f"Qwen3-TTS Benchmark - {label}", fontsize=15, fontweight="bold")
|
||||
|
||||
# TTFP breakdown
|
||||
ax = axes[0]
|
||||
means = [r["mean_ttfp_ms"] for r in results]
|
||||
medians = [r["median_ttfp_ms"] for r in results]
|
||||
p90s = [r["p90_ttfp_ms"] for r in results]
|
||||
p99s = [r["p99_ttfp_ms"] for r in results]
|
||||
x = np.arange(len(concurrencies))
|
||||
w = 0.2
|
||||
ax.bar(x - 1.5 * w, means, w, label="mean", color="#2196F3")
|
||||
ax.bar(x - 0.5 * w, medians, w, label="median", color="#4CAF50")
|
||||
ax.bar(x + 0.5 * w, p90s, w, label="p90", color="#FF9800")
|
||||
ax.bar(x + 1.5 * w, p99s, w, label="p99", color="#F44336")
|
||||
ax.set_xticks(x)
|
||||
ax.set_xticklabels([str(c) for c in concurrencies])
|
||||
ax.set_xlabel("Concurrency")
|
||||
ax.set_ylabel("TTFP (ms)")
|
||||
ax.set_title("Time to First Audio Packet")
|
||||
ax.legend(fontsize=9)
|
||||
ax.grid(axis="y", alpha=0.3)
|
||||
|
||||
# E2E breakdown
|
||||
ax = axes[1]
|
||||
means = [r["mean_e2e_ms"] for r in results]
|
||||
medians = [r["median_e2e_ms"] for r in results]
|
||||
p90s = [r["p90_e2e_ms"] for r in results]
|
||||
p99s = [r["p99_e2e_ms"] for r in results]
|
||||
ax.bar(x - 1.5 * w, means, w, label="mean", color="#2196F3")
|
||||
ax.bar(x - 0.5 * w, medians, w, label="median", color="#4CAF50")
|
||||
ax.bar(x + 0.5 * w, p90s, w, label="p90", color="#FF9800")
|
||||
ax.bar(x + 1.5 * w, p99s, w, label="p99", color="#F44336")
|
||||
ax.set_xticks(x)
|
||||
ax.set_xticklabels([str(c) for c in concurrencies])
|
||||
ax.set_xlabel("Concurrency")
|
||||
ax.set_ylabel("E2E Latency (ms)")
|
||||
ax.set_title("End-to-End Latency")
|
||||
ax.legend(fontsize=9)
|
||||
ax.grid(axis="y", alpha=0.3)
|
||||
|
||||
# RTF
|
||||
ax = axes[2]
|
||||
means = [r["mean_rtf"] for r in results]
|
||||
medians = [r["median_rtf"] for r in results]
|
||||
ax.bar(x - 0.15, means, 0.3, label="mean", color="#2196F3")
|
||||
ax.bar(x + 0.15, medians, 0.3, label="median", color="#4CAF50")
|
||||
ax.set_xticks(x)
|
||||
ax.set_xticklabels([str(c) for c in concurrencies])
|
||||
ax.set_xlabel("Concurrency")
|
||||
ax.set_ylabel("RTF")
|
||||
ax.set_title("Real-Time Factor")
|
||||
ax.legend(fontsize=9)
|
||||
ax.grid(axis="y", alpha=0.3)
|
||||
|
||||
plt.tight_layout()
|
||||
plt.savefig(output_path, dpi=150, bbox_inches="tight")
|
||||
print(f"Plot saved to {output_path}")
|
||||
plt.close()
|
||||
|
||||
|
||||
def print_comparison_table(all_results: list[list[dict]], labels: list[str]):
|
||||
"""Print a markdown-formatted comparison table."""
|
||||
concurrencies = sorted(set(r["concurrency"] for r in all_results[0]))
|
||||
|
||||
print("\n## Benchmark Results\n")
|
||||
header = "| Metric | Concurrency |"
|
||||
sep = "| --- | --- |"
|
||||
for label in labels:
|
||||
header += f" {label} |"
|
||||
sep += " --- |"
|
||||
print(header)
|
||||
print(sep)
|
||||
|
||||
for metric, key, fmt in [
|
||||
("TTFP (ms)", "mean_ttfp_ms", ".1f"),
|
||||
("E2E (ms)", "mean_e2e_ms", ".1f"),
|
||||
("RTF", "mean_rtf", ".3f"),
|
||||
("Throughput (audio-s/s)", "audio_throughput", ".2f"),
|
||||
]:
|
||||
for c in concurrencies:
|
||||
row = f"| {metric} | {c} |"
|
||||
for results in all_results:
|
||||
conc_map = {r["concurrency"]: r for r in results}
|
||||
val = conc_map.get(c, {}).get(key, 0)
|
||||
row += f" {val:{fmt}} |"
|
||||
print(row)
|
||||
|
||||
# Improvement calculation (only if 2 configs)
|
||||
if len(all_results) == 2:
|
||||
print(f"\n## Improvement ({labels[0]} vs {labels[1]})\n")
|
||||
print("| Metric | Concurrency | Improvement |")
|
||||
print("| --- | --- | --- |")
|
||||
for metric, key in [("TTFP", "mean_ttfp_ms"), ("E2E", "mean_e2e_ms"), ("RTF", "mean_rtf")]:
|
||||
for c in concurrencies:
|
||||
m0 = {r["concurrency"]: r for r in all_results[0]}
|
||||
m1 = {r["concurrency"]: r for r in all_results[1]}
|
||||
v0 = m0.get(c, {}).get(key, 0)
|
||||
v1 = m1.get(c, {}).get(key, 0)
|
||||
if v1 > 0:
|
||||
pct = (v1 - v0) / v1 * 100
|
||||
print(f"| {metric} | {c} | {pct:+.1f}% |")
|
||||
|
||||
|
||||
def parse_args():
|
||||
parser = argparse.ArgumentParser(description="Plot Qwen3-TTS benchmark results")
|
||||
parser.add_argument(
|
||||
"--results", type=str, nargs="+", required=True, help="Path(s) to result JSON files (one per config)"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--labels", type=str, nargs="+", required=True, help="Labels for each config (must match --results count)"
|
||||
)
|
||||
parser.add_argument("--output", type=str, default="results/qwen3_tts_benchmark.png", help="Output image path")
|
||||
parser.add_argument("--title", type=str, default="Qwen3-TTS", help="Title prefix for the plot")
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
args = parse_args()
|
||||
assert len(args.results) == len(args.labels), "--results and --labels must have the same count"
|
||||
|
||||
all_results = load_results(args.results)
|
||||
print_comparison_table(all_results, args.labels)
|
||||
|
||||
Path(args.output).parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
if len(all_results) == 1:
|
||||
plot_single_summary(all_results[0], args.labels[0], args.output)
|
||||
else:
|
||||
plot_comparison(all_results, args.labels, args.output, title_prefix=args.title)
|
||||
@@ -0,0 +1,3 @@
|
||||
# Benchmark results are machine-specific - do not commit
|
||||
*
|
||||
!.gitignore
|
||||
@@ -0,0 +1,272 @@
|
||||
#!/bin/bash
|
||||
# Qwen3-TTS Benchmark Runner
|
||||
#
|
||||
# Compares vllm-omni streaming serving vs HuggingFace transformers offline inference.
|
||||
# Produces JSON results and comparison plots.
|
||||
#
|
||||
# Usage:
|
||||
# # Full comparison (vllm-omni + HF):
|
||||
# bash run_benchmark.sh
|
||||
#
|
||||
# # Only vllm-omni async_chunk config:
|
||||
# bash run_benchmark.sh --async-only
|
||||
#
|
||||
# # Only HuggingFace baseline:
|
||||
# bash run_benchmark.sh --hf-only
|
||||
#
|
||||
# # vllm-omni only (skip HF):
|
||||
# bash run_benchmark.sh --skip-hf
|
||||
#
|
||||
# # Custom settings:
|
||||
# GPU_DEVICE=1 NUM_PROMPTS=20 CONCURRENCY="1 4" bash run_benchmark.sh
|
||||
#
|
||||
# # Use 1.7B model:
|
||||
# MODEL=Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice bash run_benchmark.sh --async-only
|
||||
#
|
||||
# # Use batch_size=4 config:
|
||||
# STAGE_CONFIG=vllm_omni/configs/qwen3_tts_bs4.yaml bash run_benchmark.sh --async-only
|
||||
#
|
||||
# Environment variables:
|
||||
# GPU_DEVICE - GPU index to use (default: 0)
|
||||
# NUM_PROMPTS - Number of prompts per concurrency level (default: 50)
|
||||
# CONCURRENCY - Space-separated concurrency levels (default: "1 4 10")
|
||||
# MODEL - Model name (default: Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice)
|
||||
# PORT - Server port (default: 8000)
|
||||
# GPU_MEM_TALKER - gpu_memory_utilization for talker stage (default: 0.3)
|
||||
# GPU_MEM_CODE2WAV - gpu_memory_utilization for code2wav stage (default: 0.2)
|
||||
# STAGE_CONFIG - Path to stage config YAML (default: configs/qwen3_tts_bs1.yaml)
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
PROJECT_ROOT="$(cd "${SCRIPT_DIR}/../.." && pwd)"
|
||||
|
||||
# Defaults
|
||||
GPU_DEVICE="${GPU_DEVICE:-0}"
|
||||
NUM_PROMPTS="${NUM_PROMPTS:-50}"
|
||||
CONCURRENCY="${CONCURRENCY:-1 4 10}"
|
||||
MODEL="${MODEL:-Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice}"
|
||||
PORT="${PORT:-8000}"
|
||||
GPU_MEM_TALKER="${GPU_MEM_TALKER:-0.3}"
|
||||
GPU_MEM_CODE2WAV="${GPU_MEM_CODE2WAV:-0.2}"
|
||||
NUM_WARMUPS="${NUM_WARMUPS:-3}"
|
||||
STAGE_CONFIG="${STAGE_CONFIG:-vllm_omni/configs/qwen3_tts_bs1.yaml}"
|
||||
RESULT_DIR="${SCRIPT_DIR}/results"
|
||||
TIMESTAMP="$(date +%Y%m%d_%H%M%S)"
|
||||
|
||||
# Parse args
|
||||
RUN_ASYNC=true
|
||||
RUN_HF=true
|
||||
for arg in "$@"; do
|
||||
case "$arg" in
|
||||
--async-only) RUN_HF=false ;;
|
||||
--hf-only) RUN_ASYNC=false ;;
|
||||
--skip-hf) RUN_HF=false ;;
|
||||
esac
|
||||
done
|
||||
|
||||
mkdir -p "${RESULT_DIR}"
|
||||
|
||||
echo "============================================================"
|
||||
echo " Qwen3-TTS Benchmark"
|
||||
echo "============================================================"
|
||||
echo " GPU: ${GPU_DEVICE}"
|
||||
echo " Model: ${MODEL}"
|
||||
echo " Prompts: ${NUM_PROMPTS}"
|
||||
echo " Concurrency: ${CONCURRENCY}"
|
||||
echo " Port: ${PORT}"
|
||||
echo " Stage config: ${STAGE_CONFIG}"
|
||||
echo " Results: ${RESULT_DIR}"
|
||||
echo "============================================================"
|
||||
|
||||
# Prepare stage config with correct GPU device and memory settings
|
||||
prepare_config() {
|
||||
local config_template="$1"
|
||||
local config_name="$2"
|
||||
local output_path="${RESULT_DIR}/${config_name}_stage_config.yaml"
|
||||
|
||||
# Use sed to patch GPU device and memory utilization
|
||||
sed \
|
||||
-e "s/devices: \"0\"/devices: \"${GPU_DEVICE}\"/g" \
|
||||
-e "s/gpu_memory_utilization: 0.3/gpu_memory_utilization: ${GPU_MEM_TALKER}/g" \
|
||||
-e "s/gpu_memory_utilization: 0.2/gpu_memory_utilization: ${GPU_MEM_CODE2WAV}/g" \
|
||||
"${config_template}" > "${output_path}"
|
||||
|
||||
echo "${output_path}"
|
||||
}
|
||||
|
||||
# Start server and wait for it to be ready
|
||||
start_server() {
|
||||
local stage_config="$1"
|
||||
local config_name="$2"
|
||||
local log_file="${RESULT_DIR}/server_${config_name}_${TIMESTAMP}.log"
|
||||
|
||||
echo ""
|
||||
echo "Starting server with config: ${config_name}"
|
||||
echo " Stage config: ${stage_config}"
|
||||
echo " Log file: ${log_file}"
|
||||
|
||||
VLLM_WORKER_MULTIPROC_METHOD=spawn \
|
||||
CUDA_VISIBLE_DEVICES="${GPU_DEVICE}" \
|
||||
python -m vllm_omni.entrypoints.cli.main serve "${MODEL}" \
|
||||
--omni \
|
||||
--host 127.0.0.1 \
|
||||
--port "${PORT}" \
|
||||
--stage-configs-path "${stage_config}" \
|
||||
--stage-init-timeout 120 \
|
||||
--trust-remote-code \
|
||||
--disable-log-stats \
|
||||
> "${log_file}" 2>&1 &
|
||||
|
||||
SERVER_PID=$!
|
||||
echo " Server PID: ${SERVER_PID}"
|
||||
|
||||
# Wait for server to be ready
|
||||
echo " Waiting for server to be ready..."
|
||||
local max_wait=300
|
||||
local waited=0
|
||||
while [ ${waited} -lt ${max_wait} ]; do
|
||||
if curl -sf "http://127.0.0.1:${PORT}/v1/models" > /dev/null 2>&1; then
|
||||
echo " Server is ready! (waited ${waited}s)"
|
||||
return 0
|
||||
fi
|
||||
# Check if process is still alive
|
||||
if ! kill -0 ${SERVER_PID} 2>/dev/null; then
|
||||
echo " ERROR: Server process died. Check log: ${log_file}"
|
||||
tail -20 "${log_file}"
|
||||
return 1
|
||||
fi
|
||||
sleep 2
|
||||
waited=$((waited + 2))
|
||||
done
|
||||
|
||||
echo " ERROR: Server did not start within ${max_wait}s. Check log: ${log_file}"
|
||||
kill ${SERVER_PID} 2>/dev/null || true
|
||||
return 1
|
||||
}
|
||||
|
||||
# Stop the server
|
||||
stop_server() {
|
||||
if [ -n "${SERVER_PID:-}" ]; then
|
||||
echo " Stopping server (PID: ${SERVER_PID})..."
|
||||
kill ${SERVER_PID} 2>/dev/null || true
|
||||
wait ${SERVER_PID} 2>/dev/null || true
|
||||
# Kill any remaining child processes on the port
|
||||
local pids
|
||||
pids=$(lsof -ti:${PORT} 2>/dev/null || true)
|
||||
if [ -n "${pids}" ]; then
|
||||
echo " Cleaning up remaining processes on port ${PORT}..."
|
||||
echo "${pids}" | xargs kill -9 2>/dev/null || true
|
||||
fi
|
||||
echo " Server stopped."
|
||||
SERVER_PID=""
|
||||
fi
|
||||
}
|
||||
|
||||
# Cleanup on exit
|
||||
trap 'stop_server' EXIT
|
||||
|
||||
# Run benchmark for a given config
|
||||
run_bench() {
|
||||
local config_name="$1"
|
||||
local config_template="$2"
|
||||
|
||||
echo ""
|
||||
echo "============================================================"
|
||||
echo " Benchmarking: ${config_name}"
|
||||
echo "============================================================"
|
||||
|
||||
local stage_config
|
||||
stage_config=$(prepare_config "${config_template}" "${config_name}")
|
||||
|
||||
start_server "${stage_config}" "${config_name}"
|
||||
|
||||
# Convert concurrency string to args
|
||||
local conc_args=""
|
||||
for c in ${CONCURRENCY}; do
|
||||
conc_args="${conc_args} ${c}"
|
||||
done
|
||||
|
||||
cd "${PROJECT_ROOT}"
|
||||
python "${SCRIPT_DIR}/vllm_omni/bench_tts_serve.py" \
|
||||
--host 127.0.0.1 \
|
||||
--port "${PORT}" \
|
||||
--num-prompts "${NUM_PROMPTS}" \
|
||||
--max-concurrency ${conc_args} \
|
||||
--num-warmups "${NUM_WARMUPS}" \
|
||||
--config-name "${config_name}" \
|
||||
--result-dir "${RESULT_DIR}"
|
||||
|
||||
stop_server
|
||||
|
||||
# Allow GPU memory to settle
|
||||
sleep 5
|
||||
}
|
||||
|
||||
# Run vllm-omni benchmark
|
||||
if [ "${RUN_ASYNC}" = true ]; then
|
||||
run_bench "async_chunk" "${SCRIPT_DIR}/${STAGE_CONFIG}"
|
||||
fi
|
||||
|
||||
# Run HuggingFace baseline benchmark
|
||||
if [ "${RUN_HF}" = true ]; then
|
||||
echo ""
|
||||
echo "============================================================"
|
||||
echo " Benchmarking: HuggingFace transformers (offline)"
|
||||
echo "============================================================"
|
||||
|
||||
cd "${PROJECT_ROOT}"
|
||||
python "${SCRIPT_DIR}/transformers/bench_tts_hf.py" \
|
||||
--model "${MODEL}" \
|
||||
--num-prompts "${NUM_PROMPTS}" \
|
||||
--num-warmups "${NUM_WARMUPS}" \
|
||||
--gpu-device "${GPU_DEVICE}" \
|
||||
--config-name "hf_transformers" \
|
||||
--result-dir "${RESULT_DIR}"
|
||||
|
||||
# Allow GPU memory to settle
|
||||
sleep 5
|
||||
fi
|
||||
|
||||
# Plot results
|
||||
echo ""
|
||||
echo "============================================================"
|
||||
echo " Generating plots..."
|
||||
echo "============================================================"
|
||||
|
||||
RESULT_FILES=""
|
||||
LABELS=""
|
||||
|
||||
if [ "${RUN_ASYNC}" = true ]; then
|
||||
ASYNC_FILE=$(ls -t "${RESULT_DIR}"/bench_async_chunk_*.json 2>/dev/null | head -1)
|
||||
if [ -n "${ASYNC_FILE}" ]; then
|
||||
RESULT_FILES="${ASYNC_FILE}"
|
||||
LABELS="async_chunk"
|
||||
fi
|
||||
fi
|
||||
|
||||
if [ "${RUN_HF}" = true ]; then
|
||||
HF_FILE=$(ls -t "${RESULT_DIR}"/bench_hf_transformers_*.json 2>/dev/null | head -1)
|
||||
if [ -n "${HF_FILE}" ]; then
|
||||
if [ -n "${RESULT_FILES}" ]; then
|
||||
RESULT_FILES="${RESULT_FILES} ${HF_FILE}"
|
||||
LABELS="${LABELS} hf_transformers"
|
||||
else
|
||||
RESULT_FILES="${HF_FILE}"
|
||||
LABELS="hf_transformers"
|
||||
fi
|
||||
fi
|
||||
fi
|
||||
|
||||
if [ -n "${RESULT_FILES}" ]; then
|
||||
python "${SCRIPT_DIR}/plot_results.py" \
|
||||
--results ${RESULT_FILES} \
|
||||
--labels ${LABELS} \
|
||||
--output "${RESULT_DIR}/qwen3_tts_benchmark_${TIMESTAMP}.png"
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo "============================================================"
|
||||
echo " Benchmark complete!"
|
||||
echo " Results: ${RESULT_DIR}"
|
||||
echo "============================================================"
|
||||
@@ -0,0 +1,281 @@
|
||||
"""Benchmark Qwen3-TTS using HuggingFace transformers (qwen_tts library).
|
||||
|
||||
Measures E2E latency, RTF, and audio duration for offline (non-serving) inference.
|
||||
Results are saved in the same JSON format as bench_tts_serve.py for unified plotting.
|
||||
|
||||
Usage:
|
||||
python bench_tts_hf.py \
|
||||
--model Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice \
|
||||
--num-prompts 50 \
|
||||
--num-warmups 3 \
|
||||
--gpu-device 0 \
|
||||
--result-dir results/
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import time
|
||||
from dataclasses import asdict, dataclass, field
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
|
||||
import numpy as np
|
||||
import soundfile as sf
|
||||
import torch
|
||||
|
||||
PROMPTS = [
|
||||
"Hello, welcome to the voice synthesis benchmark test.",
|
||||
"She said she would be here by noon, but nobody showed up.",
|
||||
"The quick brown fox jumps over the lazy dog near the riverbank.",
|
||||
"I can't believe how beautiful the sunset looks from up here on the mountain.",
|
||||
"Please remember to bring your identification documents to the appointment tomorrow morning.",
|
||||
"Have you ever wondered what it would be like to travel through time and visit ancient civilizations?",
|
||||
"The restaurant on the corner serves the best pasta I have ever tasted in my entire life.",
|
||||
"After the meeting, we should discuss the quarterly results and plan for the next phase.",
|
||||
"Learning a new language takes patience, practice, and a genuine curiosity about other cultures.",
|
||||
"The train leaves at half past seven, so we need to arrive at the station before then.",
|
||||
"Could you please turn down the music a little bit, I'm trying to concentrate on my work.",
|
||||
"It was a dark and stormy night when the old lighthouse keeper heard a knock at the door.",
|
||||
]
|
||||
|
||||
|
||||
@dataclass
|
||||
class BenchmarkResult:
|
||||
config_name: str = ""
|
||||
concurrency: int = 1 # always 1 for offline
|
||||
num_prompts: int = 0
|
||||
completed: int = 0
|
||||
failed: int = 0
|
||||
duration_s: float = 0.0
|
||||
# TTFP stats - not applicable for HF offline, set to E2E for compatibility
|
||||
mean_ttfp_ms: float = 0.0
|
||||
median_ttfp_ms: float = 0.0
|
||||
std_ttfp_ms: float = 0.0
|
||||
p90_ttfp_ms: float = 0.0
|
||||
p95_ttfp_ms: float = 0.0
|
||||
p99_ttfp_ms: float = 0.0
|
||||
# E2E stats (ms)
|
||||
mean_e2e_ms: float = 0.0
|
||||
median_e2e_ms: float = 0.0
|
||||
std_e2e_ms: float = 0.0
|
||||
p90_e2e_ms: float = 0.0
|
||||
p95_e2e_ms: float = 0.0
|
||||
p99_e2e_ms: float = 0.0
|
||||
# RTF stats
|
||||
mean_rtf: float = 0.0
|
||||
median_rtf: float = 0.0
|
||||
std_rtf: float = 0.0
|
||||
p99_rtf: float = 0.0
|
||||
# Audio stats
|
||||
mean_audio_duration_s: float = 0.0
|
||||
total_audio_duration_s: float = 0.0
|
||||
audio_throughput: float = 0.0
|
||||
request_throughput: float = 0.0
|
||||
# Per-request details
|
||||
per_request: list = field(default_factory=list)
|
||||
|
||||
|
||||
def run_benchmark(args):
|
||||
from qwen_tts import Qwen3TTSModel
|
||||
|
||||
device = f"cuda:{args.gpu_device}"
|
||||
print(f"Loading model: {args.model} on {device}")
|
||||
model = Qwen3TTSModel.from_pretrained(
|
||||
args.model,
|
||||
device_map=device,
|
||||
dtype=torch.bfloat16,
|
||||
)
|
||||
print("Model loaded.")
|
||||
|
||||
# Build prompt list
|
||||
prompts = [PROMPTS[i % len(PROMPTS)] for i in range(args.num_prompts)]
|
||||
|
||||
# Warmup
|
||||
if args.num_warmups > 0:
|
||||
print(f"Warming up with {args.num_warmups} requests...")
|
||||
for i in range(args.num_warmups):
|
||||
p = PROMPTS[i % len(PROMPTS)]
|
||||
wavs, sr = model.generate_custom_voice(
|
||||
text=p,
|
||||
language=args.language,
|
||||
speaker=args.voice,
|
||||
)
|
||||
# Sync GPU
|
||||
torch.cuda.synchronize(device)
|
||||
print("Warmup done.")
|
||||
|
||||
# Benchmark
|
||||
print(f"Running {args.num_prompts} requests sequentially...")
|
||||
e2e_times = []
|
||||
rtfs = []
|
||||
audio_durations = []
|
||||
per_request = []
|
||||
failed = 0
|
||||
|
||||
audio_dir = None
|
||||
if args.save_audio:
|
||||
audio_dir = Path(args.result_dir) / "audio_hf"
|
||||
audio_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
total_start = time.perf_counter()
|
||||
|
||||
for i, prompt in enumerate(prompts):
|
||||
try:
|
||||
torch.cuda.synchronize(device)
|
||||
st = time.perf_counter()
|
||||
|
||||
wavs, sr = model.generate_custom_voice(
|
||||
text=prompt,
|
||||
language=args.language,
|
||||
speaker=args.voice,
|
||||
)
|
||||
|
||||
torch.cuda.synchronize(device)
|
||||
elapsed = time.perf_counter() - st
|
||||
|
||||
# Compute audio duration
|
||||
audio_samples = wavs[0]
|
||||
if isinstance(audio_samples, torch.Tensor):
|
||||
audio_samples = audio_samples.cpu().numpy()
|
||||
audio_dur = len(audio_samples) / sr
|
||||
|
||||
rtf = elapsed / audio_dur if audio_dur > 0 else 0.0
|
||||
|
||||
e2e_times.append(elapsed)
|
||||
rtfs.append(rtf)
|
||||
audio_durations.append(audio_dur)
|
||||
per_request.append(
|
||||
{
|
||||
"e2e_ms": elapsed * 1000,
|
||||
"ttfp_ms": elapsed * 1000, # no streaming, TTFP = E2E
|
||||
"rtf": rtf,
|
||||
"audio_duration_s": audio_dur,
|
||||
"prompt": prompt,
|
||||
}
|
||||
)
|
||||
|
||||
if audio_dir:
|
||||
sf.write(str(audio_dir / f"output_{i:04d}.wav"), audio_samples, sr)
|
||||
|
||||
if (i + 1) % 10 == 0 or i == 0:
|
||||
print(
|
||||
f" [{i + 1}/{args.num_prompts}] e2e={elapsed * 1000:.0f}ms rtf={rtf:.3f} audio={audio_dur:.2f}s"
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
print(f" [{i + 1}/{args.num_prompts}] FAILED: {e}")
|
||||
failed += 1
|
||||
|
||||
total_duration = time.perf_counter() - total_start
|
||||
completed = len(e2e_times)
|
||||
|
||||
# Compute stats
|
||||
result = BenchmarkResult(
|
||||
config_name=args.config_name,
|
||||
concurrency=1,
|
||||
num_prompts=args.num_prompts,
|
||||
completed=completed,
|
||||
failed=failed,
|
||||
duration_s=total_duration,
|
||||
)
|
||||
|
||||
if e2e_times:
|
||||
e2e_ms = [t * 1000 for t in e2e_times]
|
||||
|
||||
result.mean_e2e_ms = float(np.mean(e2e_ms))
|
||||
result.median_e2e_ms = float(np.median(e2e_ms))
|
||||
result.std_e2e_ms = float(np.std(e2e_ms))
|
||||
result.p90_e2e_ms = float(np.percentile(e2e_ms, 90))
|
||||
result.p95_e2e_ms = float(np.percentile(e2e_ms, 95))
|
||||
result.p99_e2e_ms = float(np.percentile(e2e_ms, 99))
|
||||
|
||||
# For HF offline, TTFP = E2E (no streaming)
|
||||
result.mean_ttfp_ms = result.mean_e2e_ms
|
||||
result.median_ttfp_ms = result.median_e2e_ms
|
||||
result.std_ttfp_ms = result.std_e2e_ms
|
||||
result.p90_ttfp_ms = result.p90_e2e_ms
|
||||
result.p95_ttfp_ms = result.p95_e2e_ms
|
||||
result.p99_ttfp_ms = result.p99_e2e_ms
|
||||
|
||||
result.mean_rtf = float(np.mean(rtfs))
|
||||
result.median_rtf = float(np.median(rtfs))
|
||||
result.std_rtf = float(np.std(rtfs))
|
||||
result.p99_rtf = float(np.percentile(rtfs, 99))
|
||||
|
||||
result.mean_audio_duration_s = float(np.mean(audio_durations))
|
||||
result.total_audio_duration_s = float(np.sum(audio_durations))
|
||||
result.audio_throughput = result.total_audio_duration_s / total_duration
|
||||
result.request_throughput = completed / total_duration
|
||||
result.per_request = per_request
|
||||
|
||||
# Print summary in standardized performance template
|
||||
W = 50
|
||||
print("")
|
||||
print(f"{'=' * W}")
|
||||
print(f"{'Serving Benchmark Result':^{W}}")
|
||||
print(f"{'=' * W}")
|
||||
print(f"{'Successful requests:':<40}{completed:<10}")
|
||||
print(f"{'Failed requests:':<40}{failed:<10}")
|
||||
print(f"{'Maximum request concurrency:':<40}{1:<10}")
|
||||
print(f"{'Benchmark duration (s):':<40}{total_duration:<10.2f}")
|
||||
print(f"{'Request throughput (req/s):':<40}{result.request_throughput:<10.2f}")
|
||||
print(f"{'-' * W}")
|
||||
print(f"{'End-to-end Latency':^{W}}")
|
||||
print(f"{'-' * W}")
|
||||
print(f"{'Mean E2EL (ms):':<40}{result.mean_e2e_ms:<10.2f}")
|
||||
print(f"{'Median E2EL (ms):':<40}{result.median_e2e_ms:<10.2f}")
|
||||
print(f"{'P99 E2EL (ms):':<40}{result.p99_e2e_ms:<10.2f}")
|
||||
print(f"{'=' * W}")
|
||||
print(f"{'Audio Result':^{W}}")
|
||||
print(f"{'=' * W}")
|
||||
print(f"{'Total audio duration generated (s):':<40}{result.total_audio_duration_s:<10.2f}")
|
||||
print(f"{'Audio throughput (audio duration/s):':<40}{result.audio_throughput:<10.2f}")
|
||||
print(f"{'-' * W}")
|
||||
print(f"{'Time to First Packet':^{W}}")
|
||||
print(f"{'-' * W}")
|
||||
print(f"{'Mean AUDIO_TTFP (ms):':<40}{result.mean_ttfp_ms:<10.2f}")
|
||||
print(f"{'Median AUDIO_TTFP (ms):':<40}{result.median_ttfp_ms:<10.2f}")
|
||||
print(f"{'P99 AUDIO_TTFP (ms):':<40}{result.p99_ttfp_ms:<10.2f}")
|
||||
print(f"{'-' * W}")
|
||||
print(f"{'Real Time Factor':^{W}}")
|
||||
print(f"{'-' * W}")
|
||||
print(f"{'Mean AUDIO_RTF:':<40}{result.mean_rtf:<10.3f}")
|
||||
print(f"{'Median AUDIO_RTF:':<40}{result.median_rtf:<10.3f}")
|
||||
print(f"{'P99 AUDIO_RTF:':<40}{result.p99_rtf:<10.3f}")
|
||||
print(f"{'=' * W}")
|
||||
print("")
|
||||
|
||||
# Save results (as a list with single concurrency=1 entry, matching serve format)
|
||||
result_dir = Path(args.result_dir)
|
||||
result_dir.mkdir(parents=True, exist_ok=True)
|
||||
timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
|
||||
result_file = result_dir / f"bench_{args.config_name}_{timestamp}.json"
|
||||
|
||||
with open(result_file, "w") as f:
|
||||
json.dump([asdict(result)], f, indent=2)
|
||||
print(f"Results saved to {result_file}")
|
||||
|
||||
return result
|
||||
|
||||
|
||||
def parse_args():
|
||||
parser = argparse.ArgumentParser(description="Qwen3-TTS HuggingFace Benchmark")
|
||||
parser.add_argument(
|
||||
"--model", type=str, default="Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice", help="HuggingFace model name or path"
|
||||
)
|
||||
parser.add_argument("--num-prompts", type=int, default=50)
|
||||
parser.add_argument("--num-warmups", type=int, default=3)
|
||||
parser.add_argument("--gpu-device", type=int, default=0)
|
||||
parser.add_argument("--voice", type=str, default="Vivian")
|
||||
parser.add_argument("--language", type=str, default="English")
|
||||
parser.add_argument(
|
||||
"--config-name", type=str, default="hf_transformers", help="Label for this config (used in filenames)"
|
||||
)
|
||||
parser.add_argument("--result-dir", type=str, default="results")
|
||||
parser.add_argument("--save-audio", action="store_true", help="Save generated audio files")
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
args = parse_args()
|
||||
run_benchmark(args)
|
||||
@@ -0,0 +1,348 @@
|
||||
"""Benchmark client for Qwen3-TTS via /v1/audio/speech endpoint.
|
||||
|
||||
Measures TTFP (Time-to-First-Packet), E2E latency, and RTF (Real-Time Factor)
|
||||
across configurable concurrency levels. Saves results as JSON for plotting.
|
||||
|
||||
Usage:
|
||||
python bench_tts_serve.py \
|
||||
--host 127.0.0.1 --port 8000 \
|
||||
--num-prompts 50 \
|
||||
--max-concurrency 1 4 10 \
|
||||
--result-dir results/
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import asyncio
|
||||
import json
|
||||
import time
|
||||
from dataclasses import asdict, dataclass, field
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
|
||||
import aiohttp
|
||||
import numpy as np
|
||||
from tqdm.asyncio import tqdm
|
||||
|
||||
PROMPTS = [
|
||||
"Hello, welcome to the voice synthesis benchmark test.",
|
||||
"She said she would be here by noon, but nobody showed up.",
|
||||
"The quick brown fox jumps over the lazy dog near the riverbank.",
|
||||
"I can't believe how beautiful the sunset looks from up here on the mountain.",
|
||||
"Please remember to bring your identification documents to the appointment tomorrow morning.",
|
||||
"Have you ever wondered what it would be like to travel through time and visit ancient civilizations?",
|
||||
"The restaurant on the corner serves the best pasta I have ever tasted in my entire life.",
|
||||
"After the meeting, we should discuss the quarterly results and plan for the next phase.",
|
||||
"Learning a new language takes patience, practice, and a genuine curiosity about other cultures.",
|
||||
"The train leaves at half past seven, so we need to arrive at the station before then.",
|
||||
"Could you please turn down the music a little bit, I'm trying to concentrate on my work.",
|
||||
"It was a dark and stormy night when the old lighthouse keeper heard a knock at the door.",
|
||||
]
|
||||
|
||||
|
||||
@dataclass
|
||||
class RequestResult:
|
||||
success: bool = False
|
||||
ttfp: float = 0.0 # Time to first audio packet (seconds)
|
||||
e2e: float = 0.0 # End-to-end latency (seconds)
|
||||
audio_bytes: int = 0 # Total audio bytes received
|
||||
audio_duration: float = 0.0 # Audio duration in seconds (estimated from PCM)
|
||||
rtf: float = 0.0 # Real-time factor = e2e / audio_duration
|
||||
prompt: str = ""
|
||||
error: str = ""
|
||||
|
||||
|
||||
@dataclass
|
||||
class BenchmarkResult:
|
||||
config_name: str = ""
|
||||
concurrency: int = 0
|
||||
num_prompts: int = 0
|
||||
completed: int = 0
|
||||
failed: int = 0
|
||||
duration_s: float = 0.0
|
||||
# TTFP stats (ms)
|
||||
mean_ttfp_ms: float = 0.0
|
||||
median_ttfp_ms: float = 0.0
|
||||
std_ttfp_ms: float = 0.0
|
||||
p90_ttfp_ms: float = 0.0
|
||||
p95_ttfp_ms: float = 0.0
|
||||
p99_ttfp_ms: float = 0.0
|
||||
# E2E stats (ms)
|
||||
mean_e2e_ms: float = 0.0
|
||||
median_e2e_ms: float = 0.0
|
||||
std_e2e_ms: float = 0.0
|
||||
p90_e2e_ms: float = 0.0
|
||||
p95_e2e_ms: float = 0.0
|
||||
p99_e2e_ms: float = 0.0
|
||||
# RTF stats
|
||||
mean_rtf: float = 0.0
|
||||
median_rtf: float = 0.0
|
||||
std_rtf: float = 0.0
|
||||
p99_rtf: float = 0.0
|
||||
# Audio stats
|
||||
mean_audio_duration_s: float = 0.0
|
||||
total_audio_duration_s: float = 0.0
|
||||
audio_throughput: float = 0.0 # audio_duration / wall_time
|
||||
request_throughput: float = 0.0 # requests / second
|
||||
# Per-request details
|
||||
per_request: list = field(default_factory=list)
|
||||
|
||||
|
||||
def pcm_bytes_to_duration(num_bytes: int, sample_rate: int = 24000, sample_width: int = 2) -> float:
|
||||
"""Convert raw PCM byte count to duration in seconds."""
|
||||
num_samples = num_bytes / sample_width
|
||||
return num_samples / sample_rate
|
||||
|
||||
|
||||
async def send_tts_request(
|
||||
session: aiohttp.ClientSession,
|
||||
api_url: str,
|
||||
prompt: str,
|
||||
voice: str = "vivian",
|
||||
language: str = "English",
|
||||
pbar: tqdm | None = None,
|
||||
) -> RequestResult:
|
||||
"""Send a streaming TTS request and measure latency metrics."""
|
||||
payload = {
|
||||
"input": prompt,
|
||||
"voice": voice,
|
||||
"language": language,
|
||||
"stream": True,
|
||||
"response_format": "pcm",
|
||||
}
|
||||
|
||||
result = RequestResult(prompt=prompt)
|
||||
st = time.perf_counter()
|
||||
|
||||
try:
|
||||
async with session.post(api_url, json=payload) as response:
|
||||
if response.status != 200:
|
||||
result.error = f"HTTP {response.status}: {await response.text()}"
|
||||
result.success = False
|
||||
return result
|
||||
|
||||
first_chunk = True
|
||||
total_bytes = 0
|
||||
|
||||
async for chunk in response.content.iter_any():
|
||||
if first_chunk and len(chunk) > 0:
|
||||
result.ttfp = time.perf_counter() - st
|
||||
first_chunk = False
|
||||
total_bytes += len(chunk)
|
||||
|
||||
result.e2e = time.perf_counter() - st
|
||||
result.audio_bytes = total_bytes
|
||||
result.audio_duration = pcm_bytes_to_duration(total_bytes)
|
||||
|
||||
if result.audio_duration > 0:
|
||||
result.rtf = result.e2e / result.audio_duration
|
||||
result.success = True
|
||||
|
||||
except Exception as e:
|
||||
result.error = str(e)
|
||||
result.success = False
|
||||
result.e2e = time.perf_counter() - st
|
||||
|
||||
if pbar:
|
||||
pbar.update(1)
|
||||
return result
|
||||
|
||||
|
||||
async def run_benchmark(
|
||||
host: str,
|
||||
port: int,
|
||||
num_prompts: int,
|
||||
max_concurrency: int,
|
||||
num_warmups: int = 3,
|
||||
voice: str = "vivian",
|
||||
language: str = "English",
|
||||
) -> BenchmarkResult:
|
||||
"""Run benchmark at a given concurrency level."""
|
||||
api_url = f"http://{host}:{port}/v1/audio/speech"
|
||||
|
||||
connector = aiohttp.TCPConnector(
|
||||
limit=max_concurrency,
|
||||
limit_per_host=max_concurrency,
|
||||
keepalive_timeout=60,
|
||||
)
|
||||
session = aiohttp.ClientSession(
|
||||
connector=connector,
|
||||
timeout=aiohttp.ClientTimeout(total=600),
|
||||
)
|
||||
|
||||
# Warmup
|
||||
if num_warmups > 0:
|
||||
print(f" Warming up with {num_warmups} requests...")
|
||||
warmup_tasks = []
|
||||
for i in range(num_warmups):
|
||||
prompt = PROMPTS[i % len(PROMPTS)]
|
||||
warmup_tasks.append(send_tts_request(session, api_url, prompt, voice, language))
|
||||
await asyncio.gather(*warmup_tasks)
|
||||
print(" Warmup done.")
|
||||
|
||||
# Build request list
|
||||
request_prompts = [PROMPTS[i % len(PROMPTS)] for i in range(num_prompts)]
|
||||
|
||||
# Run benchmark
|
||||
print(f" Running {num_prompts} requests with concurrency={max_concurrency}...")
|
||||
semaphore = asyncio.Semaphore(max_concurrency)
|
||||
pbar = tqdm(total=num_prompts, desc=f" concurrency={max_concurrency}")
|
||||
|
||||
async def limited_request(prompt):
|
||||
async with semaphore:
|
||||
return await send_tts_request(session, api_url, prompt, voice, language, pbar)
|
||||
|
||||
start_time = time.perf_counter()
|
||||
tasks = [asyncio.create_task(limited_request(p)) for p in request_prompts]
|
||||
results: list[RequestResult] = await asyncio.gather(*tasks)
|
||||
duration = time.perf_counter() - start_time
|
||||
pbar.close()
|
||||
|
||||
await session.close()
|
||||
|
||||
# Compute stats
|
||||
successful = [r for r in results if r.success]
|
||||
failed = [r for r in results if not r.success]
|
||||
|
||||
bench = BenchmarkResult(
|
||||
concurrency=max_concurrency,
|
||||
num_prompts=num_prompts,
|
||||
completed=len(successful),
|
||||
failed=len(failed),
|
||||
duration_s=duration,
|
||||
)
|
||||
|
||||
if successful:
|
||||
ttfps = [r.ttfp * 1000 for r in successful] # convert to ms
|
||||
e2es = [r.e2e * 1000 for r in successful]
|
||||
rtfs = [r.rtf for r in successful]
|
||||
audio_durs = [r.audio_duration for r in successful]
|
||||
|
||||
bench.mean_ttfp_ms = float(np.mean(ttfps))
|
||||
bench.median_ttfp_ms = float(np.median(ttfps))
|
||||
bench.std_ttfp_ms = float(np.std(ttfps))
|
||||
bench.p90_ttfp_ms = float(np.percentile(ttfps, 90))
|
||||
bench.p95_ttfp_ms = float(np.percentile(ttfps, 95))
|
||||
bench.p99_ttfp_ms = float(np.percentile(ttfps, 99))
|
||||
|
||||
bench.mean_e2e_ms = float(np.mean(e2es))
|
||||
bench.median_e2e_ms = float(np.median(e2es))
|
||||
bench.std_e2e_ms = float(np.std(e2es))
|
||||
bench.p90_e2e_ms = float(np.percentile(e2es, 90))
|
||||
bench.p95_e2e_ms = float(np.percentile(e2es, 95))
|
||||
bench.p99_e2e_ms = float(np.percentile(e2es, 99))
|
||||
|
||||
bench.mean_rtf = float(np.mean(rtfs))
|
||||
bench.median_rtf = float(np.median(rtfs))
|
||||
bench.std_rtf = float(np.std(rtfs))
|
||||
bench.p99_rtf = float(np.percentile(rtfs, 99))
|
||||
|
||||
bench.mean_audio_duration_s = float(np.mean(audio_durs))
|
||||
bench.total_audio_duration_s = float(np.sum(audio_durs))
|
||||
bench.audio_throughput = bench.total_audio_duration_s / duration
|
||||
bench.request_throughput = len(successful) / duration
|
||||
|
||||
bench.per_request = [
|
||||
{
|
||||
"ttfp_ms": r.ttfp * 1000,
|
||||
"e2e_ms": r.e2e * 1000,
|
||||
"rtf": r.rtf,
|
||||
"audio_duration_s": r.audio_duration,
|
||||
"prompt": r.prompt,
|
||||
}
|
||||
for r in successful
|
||||
]
|
||||
|
||||
# Print summary in standardized performance template
|
||||
W = 50
|
||||
print("")
|
||||
print(f"{'=' * W}")
|
||||
print(f"{'Serving Benchmark Result':^{W}}")
|
||||
print(f"{'=' * W}")
|
||||
print(f"{'Successful requests:':<40}{bench.completed:<10}")
|
||||
print(f"{'Failed requests:':<40}{bench.failed:<10}")
|
||||
print(f"{'Maximum request concurrency:':<40}{max_concurrency:<10}")
|
||||
print(f"{'Benchmark duration (s):':<40}{duration:<10.2f}")
|
||||
print(f"{'Request throughput (req/s):':<40}{bench.request_throughput:<10.2f}")
|
||||
print(f"{'-' * W}")
|
||||
print(f"{'End-to-end Latency':^{W}}")
|
||||
print(f"{'-' * W}")
|
||||
print(f"{'Mean E2EL (ms):':<40}{bench.mean_e2e_ms:<10.2f}")
|
||||
print(f"{'Median E2EL (ms):':<40}{bench.median_e2e_ms:<10.2f}")
|
||||
print(f"{'P99 E2EL (ms):':<40}{bench.p99_e2e_ms:<10.2f}")
|
||||
print(f"{'=' * W}")
|
||||
print(f"{'Audio Result':^{W}}")
|
||||
print(f"{'=' * W}")
|
||||
print(f"{'Total audio duration generated (s):':<40}{bench.total_audio_duration_s:<10.2f}")
|
||||
print(f"{'Audio throughput (audio duration/s):':<40}{bench.audio_throughput:<10.2f}")
|
||||
print(f"{'-' * W}")
|
||||
print(f"{'Time to First Packet':^{W}}")
|
||||
print(f"{'-' * W}")
|
||||
print(f"{'Mean AUDIO_TTFP (ms):':<40}{bench.mean_ttfp_ms:<10.2f}")
|
||||
print(f"{'Median AUDIO_TTFP (ms):':<40}{bench.median_ttfp_ms:<10.2f}")
|
||||
print(f"{'P99 AUDIO_TTFP (ms):':<40}{bench.p99_ttfp_ms:<10.2f}")
|
||||
print(f"{'-' * W}")
|
||||
print(f"{'Real Time Factor':^{W}}")
|
||||
print(f"{'-' * W}")
|
||||
print(f"{'Mean AUDIO_RTF:':<40}{bench.mean_rtf:<10.3f}")
|
||||
print(f"{'Median AUDIO_RTF:':<40}{bench.median_rtf:<10.3f}")
|
||||
print(f"{'P99 AUDIO_RTF:':<40}{bench.p99_rtf:<10.3f}")
|
||||
print(f"{'=' * W}")
|
||||
print("")
|
||||
|
||||
if failed:
|
||||
for r in failed[:3]:
|
||||
print(f" [ERROR] {r.error[:200]}")
|
||||
|
||||
return bench
|
||||
|
||||
|
||||
async def main(args):
|
||||
all_results = []
|
||||
|
||||
for concurrency in args.max_concurrency:
|
||||
result = await run_benchmark(
|
||||
host=args.host,
|
||||
port=args.port,
|
||||
num_prompts=args.num_prompts,
|
||||
max_concurrency=concurrency,
|
||||
num_warmups=args.num_warmups,
|
||||
voice=args.voice,
|
||||
language=args.language,
|
||||
)
|
||||
result.config_name = args.config_name
|
||||
all_results.append(asdict(result))
|
||||
|
||||
# Save results
|
||||
result_dir = Path(args.result_dir)
|
||||
result_dir.mkdir(parents=True, exist_ok=True)
|
||||
timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
|
||||
result_file = result_dir / f"bench_{args.config_name}_{timestamp}.json"
|
||||
|
||||
with open(result_file, "w") as f:
|
||||
json.dump(all_results, f, indent=2)
|
||||
print(f"Results saved to {result_file}")
|
||||
|
||||
return all_results
|
||||
|
||||
|
||||
def parse_args():
|
||||
parser = argparse.ArgumentParser(description="Qwen3-TTS Benchmark Client")
|
||||
parser.add_argument("--host", type=str, default="127.0.0.1")
|
||||
parser.add_argument("--port", type=int, default=8000)
|
||||
parser.add_argument("--num-prompts", type=int, default=50, help="Number of prompts per concurrency level")
|
||||
parser.add_argument( # noqa: E501
|
||||
"--max-concurrency", type=int, nargs="+", default=[1, 4, 10], help="Concurrency levels to test"
|
||||
)
|
||||
parser.add_argument("--num-warmups", type=int, default=3)
|
||||
parser.add_argument("--voice", type=str, default="vivian")
|
||||
parser.add_argument("--language", type=str, default="English")
|
||||
parser.add_argument(
|
||||
"--config-name", type=str, default="async_chunk", help="Label for this config (used in filenames)"
|
||||
)
|
||||
parser.add_argument("--result-dir", type=str, default="results")
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
args = parse_args()
|
||||
asyncio.run(main(args))
|
||||
@@ -0,0 +1,96 @@
|
||||
# Qwen3-TTS batch_size=1 config (streaming with async_chunk)
|
||||
# 2-stage pipeline: Talker -> Code2Wav
|
||||
async_chunk: true
|
||||
stage_args:
|
||||
- stage_id: 0
|
||||
stage_type: llm
|
||||
runtime:
|
||||
devices: "0"
|
||||
max_batch_size: 1
|
||||
engine_args:
|
||||
model_stage: qwen3_tts
|
||||
model_arch: Qwen3TTSTalkerForConditionalGeneration
|
||||
hf_overrides:
|
||||
architectures: [Qwen3TTSTalkerForConditionalGeneration]
|
||||
worker_type: ar
|
||||
scheduler_cls: vllm_omni.core.sched.omni_ar_scheduler.OmniARScheduler
|
||||
enforce_eager: false
|
||||
trust_remote_code: true
|
||||
async_scheduling: false
|
||||
enable_prefix_caching: false
|
||||
engine_output_type: latent
|
||||
gpu_memory_utilization: 0.3
|
||||
distributed_executor_backend: "mp"
|
||||
max_num_batched_tokens: 512
|
||||
max_model_len: 4096
|
||||
custom_process_next_stage_input_func: vllm_omni.model_executor.stage_input_processors.qwen3_tts.talker2code2wav_async_chunk
|
||||
output_connectors:
|
||||
to_stage_1: connector_of_shared_memory
|
||||
default_sampling_params:
|
||||
temperature: 0.9
|
||||
top_k: 50
|
||||
max_tokens: 4096
|
||||
seed: 42
|
||||
detokenize: false
|
||||
repetition_penalty: 1.05
|
||||
stop_token_ids: [2150]
|
||||
|
||||
- stage_id: 1
|
||||
stage_type: llm
|
||||
runtime:
|
||||
devices: "0"
|
||||
max_batch_size: 1
|
||||
engine_args:
|
||||
model_stage: code2wav
|
||||
model_arch: Qwen3TTSCode2Wav
|
||||
hf_overrides:
|
||||
architectures: [Qwen3TTSCode2Wav]
|
||||
worker_type: generation
|
||||
scheduler_cls: vllm_omni.core.sched.omni_generation_scheduler.OmniGenerationScheduler
|
||||
enforce_eager: true
|
||||
trust_remote_code: true
|
||||
async_scheduling: false
|
||||
enable_prefix_caching: false
|
||||
engine_output_type: audio
|
||||
gpu_memory_utilization: 0.2
|
||||
distributed_executor_backend: "mp"
|
||||
max_num_batched_tokens: 8192
|
||||
max_model_len: 32768
|
||||
engine_input_source: [0]
|
||||
final_output: true
|
||||
final_output_type: audio
|
||||
input_connectors:
|
||||
from_stage_0: connector_of_shared_memory
|
||||
tts_args:
|
||||
max_instructions_length: 500
|
||||
default_sampling_params:
|
||||
temperature: 0.0
|
||||
top_p: 1.0
|
||||
top_k: -1
|
||||
max_tokens: 65536
|
||||
seed: 42
|
||||
detokenize: true
|
||||
repetition_penalty: 1.0
|
||||
|
||||
runtime:
|
||||
enabled: true
|
||||
defaults:
|
||||
window_size: -1
|
||||
max_inflight: 1
|
||||
|
||||
connectors:
|
||||
connector_of_shared_memory:
|
||||
name: SharedMemoryConnector
|
||||
extra:
|
||||
shm_threshold_bytes: 65536
|
||||
codec_streaming: true
|
||||
connector_get_sleep_s: 0.01
|
||||
connector_get_max_wait_first_chunk: 3000
|
||||
connector_get_max_wait: 300
|
||||
codec_chunk_frames: 25
|
||||
codec_left_context_frames: 25
|
||||
|
||||
edges:
|
||||
- from: 0
|
||||
to: 1
|
||||
window_size: -1
|
||||
@@ -0,0 +1,97 @@
|
||||
# Qwen3-TTS batch_size=4 config (streaming with async_chunk)
|
||||
# Enables concurrent request processing with max_inflight=4
|
||||
# 2-stage pipeline: Talker -> Code2Wav
|
||||
async_chunk: true
|
||||
stage_args:
|
||||
- stage_id: 0
|
||||
stage_type: llm
|
||||
runtime:
|
||||
devices: "0"
|
||||
max_batch_size: 4
|
||||
engine_args:
|
||||
model_stage: qwen3_tts
|
||||
model_arch: Qwen3TTSTalkerForConditionalGeneration
|
||||
hf_overrides:
|
||||
architectures: [Qwen3TTSTalkerForConditionalGeneration]
|
||||
worker_type: ar
|
||||
scheduler_cls: vllm_omni.core.sched.omni_ar_scheduler.OmniARScheduler
|
||||
enforce_eager: false
|
||||
trust_remote_code: true
|
||||
async_scheduling: false
|
||||
enable_prefix_caching: false
|
||||
engine_output_type: latent
|
||||
gpu_memory_utilization: 0.3
|
||||
distributed_executor_backend: "mp"
|
||||
max_num_batched_tokens: 512
|
||||
max_model_len: 4096
|
||||
custom_process_next_stage_input_func: vllm_omni.model_executor.stage_input_processors.qwen3_tts.talker2code2wav_async_chunk
|
||||
output_connectors:
|
||||
to_stage_1: connector_of_shared_memory
|
||||
default_sampling_params:
|
||||
temperature: 0.9
|
||||
top_k: 50
|
||||
max_tokens: 4096
|
||||
seed: 42
|
||||
detokenize: false
|
||||
repetition_penalty: 1.05
|
||||
stop_token_ids: [2150]
|
||||
|
||||
- stage_id: 1
|
||||
stage_type: llm
|
||||
runtime:
|
||||
devices: "0"
|
||||
max_batch_size: 4
|
||||
engine_args:
|
||||
model_stage: code2wav
|
||||
model_arch: Qwen3TTSCode2Wav
|
||||
hf_overrides:
|
||||
architectures: [Qwen3TTSCode2Wav]
|
||||
worker_type: generation
|
||||
scheduler_cls: vllm_omni.core.sched.omni_generation_scheduler.OmniGenerationScheduler
|
||||
enforce_eager: true
|
||||
trust_remote_code: true
|
||||
async_scheduling: false
|
||||
enable_prefix_caching: false
|
||||
engine_output_type: audio
|
||||
gpu_memory_utilization: 0.2
|
||||
distributed_executor_backend: "mp"
|
||||
max_num_batched_tokens: 8192
|
||||
max_model_len: 32768
|
||||
engine_input_source: [0]
|
||||
final_output: true
|
||||
final_output_type: audio
|
||||
input_connectors:
|
||||
from_stage_0: connector_of_shared_memory
|
||||
tts_args:
|
||||
max_instructions_length: 500
|
||||
default_sampling_params:
|
||||
temperature: 0.0
|
||||
top_p: 1.0
|
||||
top_k: -1
|
||||
max_tokens: 65536
|
||||
seed: 42
|
||||
detokenize: true
|
||||
repetition_penalty: 1.0
|
||||
|
||||
runtime:
|
||||
enabled: true
|
||||
defaults:
|
||||
window_size: -1
|
||||
max_inflight: 4
|
||||
|
||||
connectors:
|
||||
connector_of_shared_memory:
|
||||
name: SharedMemoryConnector
|
||||
extra:
|
||||
shm_threshold_bytes: 65536
|
||||
codec_streaming: true
|
||||
connector_get_sleep_s: 0.01
|
||||
connector_get_max_wait_first_chunk: 3000
|
||||
connector_get_max_wait: 300
|
||||
codec_chunk_frames: 25
|
||||
codec_left_context_frames: 25
|
||||
|
||||
edges:
|
||||
- from: 0
|
||||
to: 1
|
||||
window_size: -1
|
||||
@@ -60,5 +60,5 @@ vLLM-Omni seamlessly supports most popular open-source models on HuggingFace, in
|
||||
|
||||
For more information, checkout the following:
|
||||
|
||||
- [vllm-omni architecture design and recent roadmaps](https://docs.google.com/presentation/d/1qv4qMW1rKAqDREMXiUDLIgqqHQe7TDPj/edit?usp=sharing&ouid=110473603432222024453&rtpof=true&sd=true)
|
||||
- [vllm-omni architecture design and recent roadmaps](https://docs.google.com/presentation/d/1XJWgv79lORl8rbaVvp2d5Sqs6ZEBgAgj/edit?usp=sharing&ouid=110473603432222024453&rtpof=true&sd=true)
|
||||
- [vllm-omni announcement blogpost](https://blog.vllm.ai/2025/11/30/vllm-omni.html)
|
||||
|
Before Width: | Height: | Size: 125 KiB After Width: | Height: | Size: 98 KiB |
@@ -508,7 +508,9 @@ L3 level testing executes after code is merged into the main branch. Its core pu
|
||||
**2.4.5 Request Execution**
|
||||
|
||||
```python
|
||||
openai_client.send_request(request_config, request_num=1)
|
||||
openai_client.send_omni_request(request_config, request_num=1) # for omni-understanding models
|
||||
# or
|
||||
openai_client.send_diffusion_request(request_config, request_num=1) # for diffusion models
|
||||
```
|
||||
|
||||
**Explanation**:
|
||||
@@ -517,7 +519,7 @@ L3 level testing executes after code is merged into the main branch. Its core pu
|
||||
|
||||
**Single Request**: The comment clearly states this is a single-request completion test. For concurrent testing, it can be extended to multiple requests using request_num = n.
|
||||
|
||||
**Implicit Validation**: The send_request method internally includes validation logic dynamically selected based on the --run-level parameter: core_model performs basic validation, while advanced_model performs deep validation.
|
||||
**Implicit Validation**: The `send_omni_request` and `send_diffusion_request` methods internally includes validation logic dynamically selected based on the --run-level parameter: core_model performs basic validation, while advanced_model performs deep validation.
|
||||
|
||||
- ***Run Command***: `pytest -s -v /tests/e2e/online_serving/test_{model_name}.py -m advanced_model --run-level=advanced_model`
|
||||
|
||||
|
||||
@@ -32,7 +32,7 @@ Defined in `pyproject.toml`:
|
||||
| `slow` | Slow tests (may skip in quick CI) |
|
||||
| `benchmark` | Benchmark tests |
|
||||
|
||||
\* Means those markers are auto-added, and they will be added by the `@hardware_test` decorator.
|
||||
\* Means those markers are auto-added by `@hardware_test` (parametrization decorator) or `hardware_marks` (only returning the list of marks for flexibility).
|
||||
|
||||
### Example usage for markers
|
||||
|
||||
@@ -49,6 +49,7 @@ from tests.utils import hardware_test
|
||||
def test_video_to_audio()
|
||||
...
|
||||
```
|
||||
|
||||
### Decorator: `@hardware_test`
|
||||
|
||||
This decorator is intended to make hardware-aware, cross-platform test authoring easier and more robust for CI/CD environments. The `hardware_test` decorator in `vllm-omni/tests/utils.py` performs the following actions:
|
||||
@@ -98,6 +99,24 @@ This decorator is intended to make hardware-aware, cross-platform test authoring
|
||||
- ROCm only: `pytest -m "distributed_rocm and MI325"`
|
||||
- NPU only: `pytest -m "distributed_npu"`
|
||||
|
||||
### Function: `hardware_marks`
|
||||
|
||||
`hardware_marks` returns a list of pytest mark objects with the same signature as `@hardware_test`. Use it when you need more flexibility, such as attaching hardware marks to individual `pytest.param` entries rather than an entire test function.
|
||||
|
||||
```python
|
||||
from tests.utils import hardware_marks
|
||||
|
||||
MULTI_CARD_MARKS = hardware_marks(
|
||||
res={"cuda": "H100", "rocm": "MI325", "npu": "A2"}, num_cards=2
|
||||
)
|
||||
|
||||
@pytest.mark.parametrize("omni_server", [
|
||||
pytest.param(OmniServerParams(...), id="case_001", marks=MULTI_CARD_MARKS),
|
||||
], indirect=True)
|
||||
def test_feature(omni_server):
|
||||
...
|
||||
```
|
||||
|
||||
## Add Support for a New Platform
|
||||
|
||||
If you want to add support for a new platform (e.g., "tpu" for a new accelerator), follow these steps:
|
||||
@@ -132,12 +151,13 @@ If you want to add support for a new platform (e.g., "tpu" for a new accelerator
|
||||
# Optionally: add skipif_tpu when implemented
|
||||
return [test_platform, test_resource, test_distributed]
|
||||
```
|
||||
3. **Update `hardware_test` to recognize your new platform**:
|
||||
In the relevant place (see the `hardware_test` implementation), add:
|
||||
3. **Update `hardware_marks` to recognize your new platform**:
|
||||
In the relevant place (see the `hardware_marks` implementation), add:
|
||||
```python
|
||||
if platform == "tpu":
|
||||
marks = tpu_marks(res=resource, num_cards=cards)
|
||||
```
|
||||
(`hardware_test` calls `hardware_marks` internally, so both will pick up the change.)
|
||||
4. **(Recommended) Add a test using your new markers**:
|
||||
```python
|
||||
@hardware_test(
|
||||
@@ -151,7 +171,7 @@ If you want to add support for a new platform (e.g., "tpu" for a new accelerator
|
||||
**Summary**:
|
||||
- Add pytest markers for your new platform/resources
|
||||
- Implement a marker function (`xxx_marks`)
|
||||
- Plug into `hardware_test`
|
||||
- You're done: tests decorated with `@hardware_test` using your platform now automatically get the correct markers, distribution, and isolation!
|
||||
- Plug into `hardware_marks`
|
||||
- You're done: tests using `@hardware_test` or `hardware_marks` with your platform now automatically get the correct markers, distribution, and isolation!
|
||||
|
||||
See code in `vllm-omni/tests/utils.py` for existing examples (`cuda_marks`, `rocm_marks`, `npu_marks`).
|
||||
@@ -158,6 +158,7 @@ vllm_omni/ tests/
|
||||
├── test_diffusion_layerwise_offload.py
|
||||
├── test_diffusion_lora.py
|
||||
├── test_sequence_parallel.py
|
||||
├── test_qwen_image_edit_expansion.py
|
||||
└── stage_configs/
|
||||
├── qwen2_5_omni_ci.yaml
|
||||
├── qwen3_omni_ci.yaml
|
||||
|
||||
@@ -651,8 +651,27 @@ For a fair comparison, keep the same **prompt**, **seed**, **resolution**, **num
|
||||
|
||||
#### 5.2 Functionality Check in CI
|
||||
|
||||
To ensure project maintainability and sustainable development, we encourage contributors to submit test code (unit tests, system tests, or end-to-end tests) alongside their code changes.
|
||||
For comprehensive testing guidelines, please refer to the [Test File Structure and Style Guide](../ci/tests_style.md).
|
||||
To ensure project maintainability and sustainable development, please submit test code (unit tests, system tests, or end-to-end tests) alongside their code changes.
|
||||
|
||||
For comprehensive testing guidelines and the definition of test levels (L1-L5), please refer to the [Test File Structure and Style Guide](../ci/tests_style.md).
|
||||
The following tests are required to add:
|
||||
|
||||
- L4 test of the model's full *functionality* (i.e., all the *diffusion features* that are supported by this model), including several [parallelism acceleration methods](https://docs.vllm.ai/projects/vllm-omni/en/latest/user_guide/diffusion/parallelism_acceleration/), [CPU offloading](https://docs.vllm.ai/projects/vllm-omni/en/latest/user_guide/diffusion/cpu_offload_diffusion/), [TeaCache](https://docs.vllm.ai/projects/vllm-omni/en/latest/user_guide/diffusion/teacache/) and [Cache-DiT](https://docs.vllm.ai/projects/vllm-omni/en/latest/user_guide/diffusion/cache_dit_acceleration/) cache backends, [quantization methods](https://docs.vllm.ai/projects/vllm-omni/en/latest/user_guide/diffusion/quantization/overview/).
|
||||
- Test cases: Currently all the features are available in online serving mode. Hence, only need to add `tests/e2e/online_serving/test_{model}_expansion.py`. The following test cases shall cover all features:
|
||||
- 1 GPU: TeaCache & GGUF (or fallback to FP8, or disable it) & Layer-wise CPU offloading (or fallback to Module-wise)
|
||||
- 2 GPUs: Cache-DiT & FP8 (or fallback to GGUF, or disable it) & Ulysses = 2
|
||||
- 2 GPUs: Cache-DiT & GGUF (or fallback to FP8, or disable it) & Ring = 2
|
||||
- 2 GPUs: TeaCache & FP8 (or fallback to GGUF, or disable it) & CFG Parallel = 2
|
||||
- 2 GPUs: Cache-DiT & FP8 (or fallback to GGUF, or disable it) & Tensor Parallel = 2 & VAE Patch Parallel = 2
|
||||
- 2 GPUs: Cache-DiT & GGUF (or fallback to FP8, or disable it) & HSDP = 2 & VAE Patch Parallel = 2
|
||||
- Validation: test that the multimodal output files of your model have the correct shapes.
|
||||
- Test marks: always add `advanced_model` and `diffusion`. Add `parallel` and GPU-related marks if needed. Ref: [Markers for Tests](https://docs.vllm.ai/projects/vllm-omni/en/latest/contributing/ci/tests_markers/)
|
||||
- To maximize code reuse, you may refer to
|
||||
- `tests/conftest.py` for `omni_server` and `openai_client` fixtures, `generate_synthetic_image` and `assert_XXX_valid` helper.
|
||||
- `tests/utils.py` for `@hardware_test(...)` and `hardware_marks`.
|
||||
- [Parametrizing tests (pytest doc)](https://docs.pytest.org/en/stable/example/parametrize.html) to reuse test function implementation for different cases.
|
||||
- Doc: add a concise dostring for each test function.
|
||||
- Reference L4 test implementation: [tests/e2e/online_serving/test_qwen_image_edit_expansion.py](https://github.com/vllm-project/vllm-omni/blob/main/tests/e2e/online_serving/test_qwen_image_edit_expansion.py).
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -19,7 +19,7 @@ The `async_chunk` feature enables asynchronous, chunked processing of data acros
|
||||
|
||||
For qwen3-omni:
|
||||
- **Thinker → Talker**: Per decode step (typically chunk_size=1)
|
||||
- **Talker → Code2Wav**: Accumulated to `codec_chunk_frames` (default=25) before sending. Set `initial_codec_chunk_frames` to emit smaller chunks during the initial phase for reduced TTFA
|
||||
- **Talker → Code2Wav**: Accumulated to `codec_chunk_frames` (default=25) before sending. During the initial phase, a dynamic initial chunk size (IC) is automatically selected based on server load to reduce TTFA. Use the per-request `initial_codec_chunk_frames` API field to override.
|
||||
- **Code2Wav**: Streaming decode with code2wav chunk_size
|
||||
|
||||
With `async_chunk`:
|
||||
@@ -67,6 +67,13 @@ Enabling **async_chunk** (False→True) sharply reduces time-to-first-audio (TTF
|
||||
</picture>
|
||||
</p>
|
||||
|
||||
<p align="center">
|
||||
<picture>
|
||||
<source media="(prefers-color-scheme: dark)" src="https://raw.githubusercontent.com/vllm-project/vllm-omni/refs/heads/main/docs/source/performance/qwen3-omni_e2e_performance.png">
|
||||
<img alt="E2E Performance Data Comparison" src="https://raw.githubusercontent.com/vllm-project/vllm-omni/refs/heads/main/docs/source/performance/qwen3-omni_e2e_performance.png" width=100%>
|
||||
</picture>
|
||||
</p>
|
||||
|
||||
## Architecture
|
||||
### Data Flow
|
||||
|
||||
|
||||
@@ -47,7 +47,8 @@ If no, check your shell running the ComfyUI process. There may be some error mes
|
||||
This extension offers the following nodes based on the output modalities:
|
||||
|
||||
- **Generate Image** for text-to-image and image-to-image tasks
|
||||
- **Multimodality Comprehension** for multimodality-to-text and multimodality-to-audio tasks
|
||||
- **Generate Video** for text-to-video and image-to-video tasks
|
||||
- **Multimodality Understanding** for multimodality-to-text and multimodality-to-audio tasks
|
||||
- **TTS** and **TTS Voice Clone** for TTS tasks
|
||||
|
||||
This extension also offers example workflows (at **ComfyUI sidebar -> Templates -> vLLM-Omni**)
|
||||
@@ -59,7 +60,7 @@ To build a simple workflow yourself,
|
||||
|
||||
- Drag a generation node onto the canvas.
|
||||
- Depending on your need, grab built-in multimedia file loader nodes, such as **image->Load Image**, **image->video->Load Video**, **audio->Load Audio**
|
||||
- Depending on your need, grab built-in multimedia file preview nodes, such as **image->Preview Image**, **image->video->Save Video**, **audio->Preview Audio**. For text output, you can install [ComfyUI-Custom-Scripts plugin](https://github.com/pythongosssss/ComfyUI-Custom-Scripts/) and grab its **utils->Show Text 🐍** node.
|
||||
- Depending on your need, grab built-in multimedia file preview nodes, such as **image->Preview Image**, **image->video->Save Video**, **audio->Preview Audio**, **utils->Preview as Text**.
|
||||
- If you want to tune sampling parameters, grab corresponding nodes from **vLLM-Omni-> Sampling Params**.
|
||||
- For multi-stage models, you can connect multiple **AR Sampling Params** and **Diffusion Sampling Params** nodes to a **Multi-Stage Sampling Params List** node, and connect this node to the generation node.
|
||||
- For some multi-stage models like BAGEL, [only one stage's sampling parameters are exposed and tunable via vLLM-Omni's online serving API](https://docs.vllm.ai/projects/vllm-omni/en/latest/user_guide/examples/online_serving/bagel/). Thus, these models are treated as single-stage ones. Please check the vLLM-Omni documentation on how to correctly set each model's sampling parameters.
|
||||
|
||||
@@ -132,9 +132,19 @@ class Example:
|
||||
".dat",
|
||||
".db",
|
||||
".sqlite", # Data files
|
||||
".pyc",
|
||||
".pyo",
|
||||
".pyd", # Python compiled
|
||||
".npy",
|
||||
".npz",
|
||||
".pkl",
|
||||
".pickle", # Serialized data
|
||||
}
|
||||
excluded_dirs = {"__pycache__", ".git", "node_modules", ".tox", ".mypy_cache"}
|
||||
|
||||
def is_other_file(file: Path) -> bool:
|
||||
if any(part in excluded_dirs for part in file.parts):
|
||||
return False
|
||||
return file.is_file() and file != self.main_file and file.suffix.lower() not in binary_extensions
|
||||
|
||||
return [file for file in self.path.rglob("*") if is_other_file(file)]
|
||||
|
||||
@@ -49,6 +49,7 @@ th {
|
||||
|`GlmImageForConditionalGeneration` | GLM-Image | `zai-org/GLM-Image` |
|
||||
|`NextStep11Pipeline` | NextStep-1.1 | `stepfun-ai/NextStep-1.1` |
|
||||
|`MiMoAudioForConditionalGeneration` | MiMo-Audio-7B-Instruct | `XiaomiMiMo/MiMo-Audio-7B-Instruct` |
|
||||
|`Flux2Pipeline` | FLUX.2-dev | `black-forest-labs/FLUX.2-dev` |
|
||||
|
||||
|
||||
## List of Supported Models for NPU
|
||||
|
||||
@@ -96,7 +96,7 @@ Content-Type: application/json
|
||||
| `language` | string | "Auto" | Language (see supported languages below) |
|
||||
| `instructions` | string | "" | Voice style/emotion instructions |
|
||||
| `max_new_tokens` | integer | 2048 | Maximum tokens to generate |
|
||||
| `initial_codec_chunk_frames` | integer | null | Initial chunk size for reduced TTFA (overrides stage config) |
|
||||
| `initial_codec_chunk_frames` | integer | null | Per-request initial chunk size override for TTFA tuning. When null, IC is computed dynamically based on server load. |
|
||||
|
||||
**Supported languages:** Auto, Chinese, English, Japanese, Korean, German, French, Russian, Portuguese, Spanish, Italian
|
||||
|
||||
|
||||
|
After Width: | Height: | Size: 34 KiB |
|
Before Width: | Height: | Size: 36 KiB After Width: | Height: | Size: 37 KiB |
|
Before Width: | Height: | Size: 33 KiB After Width: | Height: | Size: 45 KiB |
@@ -18,23 +18,27 @@ The following parallelism methods are currently supported in vLLM-Omni:
|
||||
|
||||
6. [HSDP](#hsdp): Hybrid Sharded Data Parallel shards model weights across GPUs using PyTorch FSDP2. This reduces per-GPU memory usage, enabling inference of large models on GPUs with limited memory.
|
||||
|
||||
7. [Expert Parallel](#expert-parallelism): Expert Parallelism shards the Experts of a Mixture-of-Experts (MoE) layer across multiple devices. During the forward, a gating mechanism routes tokens to their designated experts, necessitating cross-cards communication(all-to-all) to dispatch tokens to the correct ranks and combine the results. This parallelism allows for massive scaling of model parameters without a proportional increase in the computational load per device.
|
||||
|
||||
The following table shows which models are currently supported by parallelism method:
|
||||
|
||||
### ImageGen
|
||||
|
||||
| Model | Model Identifier | Ulysses-SP | Ring-SP | CFG-Parallel | Tensor-Parallel | VAE-Patch-Parallel |
|
||||
|--------------------------|--------------------------------------|:----------:|:-------:|:------------:|:---------------:|:------------------:|
|
||||
| **LongCat-Image** | `meituan-longcat/LongCat-Image` | ✅ | ✅ | ❌ | ✅ | ❌ |
|
||||
| **LongCat-Image-Edit** | `meituan-longcat/LongCat-Image-Edit` | ✅ | ✅ | ❌ | ✅ | ❌ |
|
||||
| **Ovis-Image** | `OvisAI/Ovis-Image` | ❌ | ❌ | ❌ | ❌ | ❌ |
|
||||
| **Qwen-Image** | `Qwen/Qwen-Image` | ✅ | ✅ | ✅ | ✅ | ✅ |
|
||||
| **Qwen-Image-Edit** | `Qwen/Qwen-Image-Edit` | ✅ | ✅ | ✅ | ✅ | ❌ |
|
||||
| **Qwen-Image-Edit-2509** | `Qwen/Qwen-Image-Edit-2509` | ✅ | ✅ | ✅ | ✅ | ❌ |
|
||||
| **Qwen-Image-Layered** | `Qwen/Qwen-Image-Layered` | ✅ | ✅ | ✅ | ✅ | ❌ |
|
||||
| **Z-Image** | `Tongyi-MAI/Z-Image-Turbo` | ✅ | ✅ | ❌ | ✅ (TP=2 only) | ✅ |
|
||||
| **Stable-Diffusion3.5** | `stabilityai/stable-diffusion-3.5` | ❌ | ❌ | ❌ | ✅ | ✅ |
|
||||
| **FLUX.2-klein** | `black-forest-labs/FLUX.2-klein-4B` | ❌ | ❌ | ❌ | ✅ | ❌ |
|
||||
| **FLUX.1-dev** | `black-forest-labs/FLUX.1-dev` | ❌ | ❌ | ✅ | ✅ | ❌ |
|
||||
| Model | Model Identifier | Ulysses-SP | Ring-SP | CFG-Parallel | Tensor-Parallel | VAE-Patch-Parallel | Expert-Parallel |
|
||||
|--------------------------|--------------------------------------|:----------:|:-------:|:------------:|:---------------:|:------------------:|:---------------:|
|
||||
| **LongCat-Image** | `meituan-longcat/LongCat-Image` | ✅ | ✅ | ❌ | ✅ | ❌ | N/A |
|
||||
| **LongCat-Image-Edit** | `meituan-longcat/LongCat-Image-Edit` | ✅ | ✅ | ❌ | ✅ | ❌ | N/A |
|
||||
| **Ovis-Image** | `OvisAI/Ovis-Image` | ❌ | ❌ | ❌ | ❌ | ❌ | N/A |
|
||||
| **Qwen-Image** | `Qwen/Qwen-Image` | ✅ | ✅ | ✅ | ✅ | ✅ | N/A |
|
||||
| **Qwen-Image-Edit** | `Qwen/Qwen-Image-Edit` | ✅ | ✅ | ✅ | ✅ | ❌ | N/A |
|
||||
| **Qwen-Image-Edit-2509** | `Qwen/Qwen-Image-Edit-2509` | ✅ | ✅ | ✅ | ✅ | ❌ | N/A |
|
||||
| **Qwen-Image-Layered** | `Qwen/Qwen-Image-Layered` | ✅ | ✅ | ✅ | ✅ | ❌ | N/A |
|
||||
| **Z-Image** | `Tongyi-MAI/Z-Image-Turbo` | ✅ | ✅ | ❌ | ✅ (TP=2 only) | ✅ | N/A |
|
||||
| **Stable-Diffusion3.5** | `stabilityai/stable-diffusion-3.5` | ❌ | ❌ | ❌ | ✅ | ✅ | N/A |
|
||||
| **FLUX.2-klein** | `black-forest-labs/FLUX.2-klein-4B` | ❌ | ❌ | ❌ | ✅ | ❌ | N/A |
|
||||
| **FLUX.1-dev** | `black-forest-labs/FLUX.1-dev` | ❌ | ❌ | ✅ | ✅ | ❌ | N/A |
|
||||
| **FLUX.2-dev** | `black-forest-labs/FLUX.2-dev` | ❌ | ❌ | ❌ | ✅ | ❌ | N/A |
|
||||
| **HunyuanImage3.0** | `tencent/HunyuanImage-3.0`, `tencent/HunyuanImage-3.0-Instruct` | ❌ | ❌ | ❌ | ✅ | ❌ | ✅ |
|
||||
|
||||
!!! note "TP Limitations for Diffusion Models"
|
||||
We currently implement Tensor Parallelism (TP) only for the DiT (Diffusion Transformer) blocks. This is because the `text_encoder` component in vLLM-Omni uses the original Transformers implementation, which does not yet support TP.
|
||||
@@ -414,3 +418,28 @@ vllm serve Wan-AI/Wan2.2-T2V-A14B-Diffusers --omni --port 8091 --use-hsdp --usp
|
||||
#### Adding HSDP Support to New Models
|
||||
|
||||
For detailed instructions on adding HSDP support to new models, see the [HSDP Contributing Guide](../../design/feature/hsdp.md).
|
||||
|
||||
### Expert Parallelism
|
||||
|
||||
Unlike Tensor Parallelism which shards every layer's weights, EP only shards the MoE expert MLP blocks. This significantly reduces the memory footprint of MoE models (e.g., HunyuanImage3.0) while maintaining constant dense-equivalent compute efficiency. Expert Parallelism is enabled via `DiffusionParallelConfig.enable_expert_parallel`. And `self.ep = tp * sp * cfg * dp` for now, so at least one of TP/SP/CFG/DP should set when EP enabled.
|
||||
|
||||
#### Offline Inference
|
||||
|
||||
```python
|
||||
from vllm_omni import Omni
|
||||
from vllm_omni.diffusion.data import DiffusionParallelConfig
|
||||
|
||||
omni = Omni(
|
||||
model="tencent/HunyuanImage-3.0",
|
||||
parallel_config=DiffusionParallelConfig(tensor_parallel_size=8, enable_expert_parallel=True),
|
||||
)
|
||||
|
||||
outputs = omni.generate(
|
||||
"A brown and white dog is running on the grass",
|
||||
OmniDiffusionSamplingParams(
|
||||
num_inference_steps=50,
|
||||
width=1024,
|
||||
height=1024,
|
||||
),
|
||||
)
|
||||
```
|
||||
@@ -69,6 +69,7 @@ The following table shows which models are currently supported by each accelerat
|
||||
| **FLUX.1-dev** | `black-forest-labs/FLUX.1-dev` | ❌ | ✅ | ❌ | ❌ | ✅ | ✅ | ❌ |
|
||||
| **NextStep-1.1** | `stepfun-ai/NextStep-1.1` | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ❌ |
|
||||
| **FLUX.2-klein** | `black-forest-labs/FLUX.2-klein-4B` | ❌ | ✅ | ❌ | ❌ | ❌ | ✅ | ❌ |
|
||||
| **FLUX.2-dev** | `black-forest-labs/FLUX.2-dev` | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ |
|
||||
|
||||
### VideoGen
|
||||
|
||||
|
||||
@@ -99,7 +99,8 @@ python end2end.py --query-type CustomVoice --streaming --output-dir /tmp/out_str
|
||||
```
|
||||
|
||||
Each Code2Wav chunk is logged as it arrives (default 25 frames; configurable via `codec_chunk_frames`
|
||||
and `initial_codec_chunk_frames` in the stage config). The final WAV file is written once generation
|
||||
in the stage config). The initial chunk size is dynamically selected based on server load for reduced
|
||||
TTFA, and can be overridden per-request via the `initial_codec_chunk_frames` API field. The final WAV file is written once generation
|
||||
completes. This demonstrates that audio data is available progressively rather than only at the end.
|
||||
|
||||
> **Note:** Streaming uses `AsyncOmni` internally. The non-streaming path (`Omni`) is unchanged.
|
||||
|
||||
@@ -17,6 +17,31 @@ Please refer to [README.md](https://github.com/vllm-project/vllm-omni/tree/main/
|
||||
| `Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice` | CustomVoice | Smaller/faster variant |
|
||||
| `Qwen/Qwen3-TTS-12Hz-0.6B-Base` | Base | Smaller/faster variant for voice cloning |
|
||||
|
||||
## Gradio Demo
|
||||
|
||||
Two interactive Gradio demos are available, both supporting all 3 task types:
|
||||
|
||||
| Demo | File | Transport | Streaming Quality |
|
||||
| ---- | ---- | --------- | ----------------- |
|
||||
| Standard | `gradio_demo.py` | HTTP chunked | May have small gaps between chunks |
|
||||
| FastRTC | `gradio_fastrtc_demo.py` | WebRTC | Gapless streaming (requires `pip install fastrtc`) |
|
||||
|
||||
```bash
|
||||
# Option 1: Launch server + Standard Gradio together
|
||||
./run_gradio_demo.sh # CustomVoice (default)
|
||||
./run_gradio_demo.sh --task-type VoiceDesign # VoiceDesign
|
||||
./run_gradio_demo.sh --task-type Base # Voice cloning
|
||||
|
||||
# Option 2: If server is already running
|
||||
python gradio_demo.py --api-base http://localhost:8000
|
||||
|
||||
# Option 3: FastRTC demo (gapless streaming)
|
||||
pip install fastrtc
|
||||
python gradio_fastrtc_demo.py --api-base http://localhost:8000
|
||||
```
|
||||
|
||||
Then open http://localhost:7860 in your browser.
|
||||
|
||||
## Run examples (Qwen3-TTS)
|
||||
|
||||
### Launch the Server
|
||||
@@ -112,7 +137,7 @@ The Python client supports the following command-line arguments:
|
||||
- `--voice`: Speaker/voice name (default: `vivian`). Options: `vivian`, `ryan`, `aiden`, etc.
|
||||
- `--language`: Language. Options: `Auto`, `Chinese`, `English`, `Japanese`, `Korean`, `German`, `French`, `Russian`, `Portuguese`, `Spanish`, `Italian`
|
||||
- `--instructions`: Voice style/emotion instructions
|
||||
- `--ref-audio`: Reference audio file path or URL for voice cloning (Base task). Local paths are automatically base64-encoded by the client before sending to the server.
|
||||
- `--ref-audio`: Reference audio file path or URL for voice cloning (Base task)
|
||||
- `--ref-text`: Reference audio transcript for voice cloning (Base task)
|
||||
- `--response-format`: Audio output format (default: `wav`). Options: `wav`, `mp3`, `flac`, `pcm`, `aac`, `opus`
|
||||
- `--output` (or `-o`): Output audio file path (default: `tts_output.wav`)
|
||||
@@ -187,6 +212,59 @@ sudo apt install ffmpeg
|
||||
|
||||
## API Reference
|
||||
|
||||
### Voices Endpoint
|
||||
|
||||
#### GET /v1/audio/voices
|
||||
|
||||
List all available voices/speakers from the loaded model, including both built-in model voices and uploaded custom voices.
|
||||
|
||||
**Response Example:**
|
||||
```json
|
||||
{
|
||||
"voices": ["vivian", "ryan", "custom_voice_1"],
|
||||
"uploaded_voices": [
|
||||
{
|
||||
"name": "custom_voice_1",
|
||||
"consent": "user_consent_id",
|
||||
"created_at": 1738660000,
|
||||
"file_size": 1024000,
|
||||
"mime_type": "audio/wav"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
#### POST /v1/audio/voices
|
||||
|
||||
Upload a new voice sample for voice cloning in Base task TTS requests.
|
||||
|
||||
**Form Parameters:**
|
||||
- `audio_sample` (required): Audio file (max 10MB, supported formats: wav, mp3, flac, ogg, aac, webm, mp4)
|
||||
- `consent` (required): Consent recording ID
|
||||
- `name` (required): Name for the new voice
|
||||
|
||||
**Response Example:**
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
"voice": {
|
||||
"name": "custom_voice_1",
|
||||
"consent": "user_consent_id",
|
||||
"created_at": 1738660000,
|
||||
"mime_type": "audio/wav",
|
||||
"file_size": 1024000
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Usage Example:**
|
||||
```bash
|
||||
curl -X POST http://localhost:8000/v1/audio/voices \
|
||||
-F "audio_sample=@/path/to/voice_sample.wav" \
|
||||
-F "consent=user_consent_id" \
|
||||
-F "name=custom_voice_1"
|
||||
```
|
||||
|
||||
### Endpoint
|
||||
|
||||
```
|
||||
@@ -196,20 +274,6 @@ Content-Type: application/json
|
||||
|
||||
This endpoint follows the [OpenAI Audio Speech API](https://platform.openai.com/docs/api-reference/audio/createSpeech) format with additional Qwen3-TTS parameters.
|
||||
|
||||
### Voices Endpoint
|
||||
|
||||
```
|
||||
GET /v1/audio/voices
|
||||
```
|
||||
|
||||
Lists available voices for the loaded model:
|
||||
|
||||
```json
|
||||
{
|
||||
"voices": ["aiden", "dylan", "eric", "one_anna", "ryan", "serena", "sohee", "uncle_fu", "vivian"]
|
||||
}
|
||||
```
|
||||
|
||||
### Request Body
|
||||
|
||||
```json
|
||||
@@ -220,7 +284,7 @@ Lists available voices for the loaded model:
|
||||
"task_type": "CustomVoice",
|
||||
"language": "Auto",
|
||||
"instructions": "Optional style instructions",
|
||||
"ref_audio": "HTTP URL, base64 data URL, or file:// URI for voice cloning",
|
||||
"ref_audio": "URL or base64 for voice cloning",
|
||||
"ref_text": "Reference audio transcript",
|
||||
"x_vector_only_mode": false,
|
||||
"max_new_tokens": 2048
|
||||
@@ -253,6 +317,7 @@ Returns binary audio data with appropriate `Content-Type` header (e.g., `audio/w
|
||||
| `language` | string | "Auto" | Language (see supported languages below) |
|
||||
| `instructions` | string | "" | Voice style/emotion instructions |
|
||||
| `max_new_tokens` | int | 2048 | Maximum tokens to generate |
|
||||
| `initial_codec_chunk_frames` | int | null | Per-request initial chunk size override for TTFA tuning. When null, IC is computed dynamically based on server load. |
|
||||
| `stream` | bool | false | Stream raw PCM chunks as they are decoded (requires `response_format="pcm"`) |
|
||||
|
||||
**Supported languages:** Auto, Chinese, English, Japanese, Korean, German, French, Russian, Portuguese, Spanish, Italian
|
||||
@@ -261,7 +326,7 @@ Returns binary audio data with appropriate `Content-Type` header (e.g., `audio/w
|
||||
|
||||
| Parameter | Type | Required | Description |
|
||||
| -------------------- | ------ | -------- | -------------------------------------------- |
|
||||
| `ref_audio` | string | **Yes** | Reference audio (HTTP URL, base64 data URL, or `file://` URI with `--allowed-local-media-path`) |
|
||||
| `ref_audio` | string | **Yes** | Reference audio (URL or base64 data URL) |
|
||||
| `ref_text` | string | No | Transcript of reference audio (for ICL mode) |
|
||||
| `x_vector_only_mode` | bool | No | Use speaker embedding only (no ICL) |
|
||||
|
||||
@@ -301,11 +366,27 @@ curl -X POST http://localhost:8091/v1/audio/speech \
|
||||
|
||||
## Example materials
|
||||
|
||||
??? abstract "gradio_demo.py"
|
||||
``````py
|
||||
--8<-- "examples/online_serving/qwen3_tts/gradio_demo.py"
|
||||
``````
|
||||
??? abstract "gradio_fastrtc_demo.py"
|
||||
``````py
|
||||
--8<-- "examples/online_serving/qwen3_tts/gradio_fastrtc_demo.py"
|
||||
``````
|
||||
??? abstract "openai_speech_client.py"
|
||||
``````py
|
||||
--8<-- "examples/online_serving/qwen3_tts/openai_speech_client.py"
|
||||
``````
|
||||
??? abstract "run_gradio_demo.sh"
|
||||
``````sh
|
||||
--8<-- "examples/online_serving/qwen3_tts/run_gradio_demo.sh"
|
||||
``````
|
||||
??? abstract "run_server.sh"
|
||||
``````sh
|
||||
--8<-- "examples/online_serving/qwen3_tts/run_server.sh"
|
||||
``````
|
||||
??? abstract "tts_common.py"
|
||||
``````py
|
||||
--8<-- "examples/online_serving/qwen3_tts/tts_common.py"
|
||||
``````
|
||||
@@ -208,6 +208,11 @@ def parse_args() -> argparse.Namespace:
|
||||
default=1,
|
||||
help="Number of GPUs used for tensor parallelism (TP) inside the DiT.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--enable-expert-parallel",
|
||||
action="store_true",
|
||||
help="Enable expert parallelism for MoE layers.",
|
||||
)
|
||||
parser.add_argument("--layers", type=int, default=4, help="Number of layers to decompose the input image into.")
|
||||
parser.add_argument(
|
||||
"--resolution",
|
||||
@@ -348,6 +353,7 @@ def main():
|
||||
ring_degree=args.ring_degree,
|
||||
cfg_parallel_size=args.cfg_parallel_size,
|
||||
tensor_parallel_size=args.tensor_parallel_size,
|
||||
enable_expert_parallel=args.enable_expert_parallel,
|
||||
)
|
||||
|
||||
# Configure cache based on backend type
|
||||
@@ -402,7 +408,7 @@ def main():
|
||||
else:
|
||||
print(f" Input image size: {input_image.size}")
|
||||
print(
|
||||
f" Parallel configuration: ulysses_degree={args.ulysses_degree}, ring_degree={args.ring_degree}, cfg_parallel_size={args.cfg_parallel_size}, tensor_parallel_size={args.tensor_parallel_size}"
|
||||
f" Parallel configuration: ulysses_degree={args.ulysses_degree}, ring_degree={args.ring_degree}, cfg_parallel_size={args.cfg_parallel_size}, tensor_parallel_size={args.tensor_parallel_size}, enable_expert_parallel: {args.enable_expert_parallel}"
|
||||
)
|
||||
print(f"{'=' * 60}\n")
|
||||
|
||||
|
||||
@@ -96,7 +96,8 @@ python end2end.py --query-type CustomVoice --streaming --output-dir /tmp/out_str
|
||||
```
|
||||
|
||||
Each Code2Wav chunk is logged as it arrives (default 25 frames; configurable via `codec_chunk_frames`
|
||||
and `initial_codec_chunk_frames` in the stage config). The final WAV file is written once generation
|
||||
in the stage config). The initial chunk size is dynamically selected based on server load for reduced
|
||||
TTFA, and can be overridden per-request via the `initial_codec_chunk_frames` API field. The final WAV file is written once generation
|
||||
completes. This demonstrates that audio data is available progressively rather than only at the end.
|
||||
|
||||
> **Note:** Streaming uses `AsyncOmni` internally. The non-streaming path (`Omni`) is unchanged.
|
||||
|
||||
@@ -106,6 +106,23 @@ python text_to_image.py \
|
||||
--seed 42
|
||||
```
|
||||
|
||||
### Flux.2-dev Models
|
||||
To start Flux.2-dev with a single GPU, cpu-offload must be enabled because the total size of its weights exceeds the 80GB memory capacity of the GPU.
|
||||
```bash
|
||||
python examples/offline_inference/text_to_image/text_to_image.py \
|
||||
--model black-forest-labs/FLUX.2-dev \
|
||||
--prompt "a lovely bunny holding a sign that says 'vllm-omni'" \
|
||||
--seed 42 \
|
||||
--tensor-parallel-size 1 \
|
||||
--num-images-per-prompt 1 \
|
||||
--num-inference-steps 50 \
|
||||
--guidance-scale 4.0 \
|
||||
--height 1024 \
|
||||
--width 1024 \
|
||||
--enable-cpu-offload \
|
||||
--output flux2-dev.png
|
||||
```
|
||||
|
||||
### Key Arguments
|
||||
|
||||
**Common arguments:**
|
||||
|
||||
@@ -170,6 +170,11 @@ def parse_args() -> argparse.Namespace:
|
||||
default=1,
|
||||
help="Number of GPUs used for tensor parallelism (TP) inside the DiT.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--enable-expert-parallel",
|
||||
action="store_true",
|
||||
help="Enable expert parallelism for MoE layers.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--lora-path",
|
||||
type=str,
|
||||
@@ -258,6 +263,7 @@ def main():
|
||||
cfg_parallel_size=args.cfg_parallel_size,
|
||||
tensor_parallel_size=args.tensor_parallel_size,
|
||||
vae_patch_parallel_size=args.vae_patch_parallel_size,
|
||||
enable_expert_parallel=args.enable_expert_parallel,
|
||||
)
|
||||
|
||||
# Check if profiling is requested via environment variable
|
||||
@@ -323,7 +329,7 @@ def main():
|
||||
print(
|
||||
f" Parallel configuration: tensor_parallel_size={args.tensor_parallel_size}, "
|
||||
f"ulysses_degree={args.ulysses_degree}, ring_degree={args.ring_degree}, cfg_parallel_size={args.cfg_parallel_size}, "
|
||||
f"vae_patch_parallel_size={args.vae_patch_parallel_size}"
|
||||
f"vae_patch_parallel_size={args.vae_patch_parallel_size}, enable_expert_parallel={args.enable_expert_parallel}."
|
||||
)
|
||||
print(f" CPU offload: {args.enable_cpu_offload}")
|
||||
print(f" Image size: {args.width}x{args.height}")
|
||||
|
||||
@@ -127,6 +127,11 @@ def parse_args() -> argparse.Namespace:
|
||||
default=1,
|
||||
help="Number of GPUs used for VAE patch/tile parallelism (decode).",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--enable-expert-parallel",
|
||||
action="store_true",
|
||||
help="Enable expert parallelism for MoE layers.",
|
||||
)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
@@ -160,6 +165,7 @@ def main():
|
||||
cfg_parallel_size=args.cfg_parallel_size,
|
||||
tensor_parallel_size=args.tensor_parallel_size,
|
||||
vae_patch_parallel_size=args.vae_patch_parallel_size,
|
||||
enable_expert_parallel=args.enable_expert_parallel,
|
||||
)
|
||||
|
||||
# Check if profiling is requested via environment variable
|
||||
@@ -193,7 +199,7 @@ def main():
|
||||
print(
|
||||
f" Parallel configuration: ulysses_degree={args.ulysses_degree}, ring_degree={args.ring_degree},"
|
||||
f" cfg_parallel_size={args.cfg_parallel_size}, tensor_parallel_size={args.tensor_parallel_size},"
|
||||
f" vae_patch_parallel_size={args.vae_patch_parallel_size}"
|
||||
f" vae_patch_parallel_size={args.vae_patch_parallel_size}, enable_expert_parallel={args.enable_expert_parallel}"
|
||||
)
|
||||
print(f" Video size: {args.width}x{args.height}")
|
||||
print(f"{'=' * 60}\n")
|
||||
|
||||
@@ -14,6 +14,31 @@ Please refer to [README.md](https://github.com/vllm-project/vllm-omni/tree/main/
|
||||
| `Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice` | CustomVoice | Smaller/faster variant |
|
||||
| `Qwen/Qwen3-TTS-12Hz-0.6B-Base` | Base | Smaller/faster variant for voice cloning |
|
||||
|
||||
## Gradio Demo
|
||||
|
||||
Two interactive Gradio demos are available, both supporting all 3 task types:
|
||||
|
||||
| Demo | File | Transport | Streaming Quality |
|
||||
| ---- | ---- | --------- | ----------------- |
|
||||
| Standard | `gradio_demo.py` | HTTP chunked | May have small gaps between chunks |
|
||||
| FastRTC | `gradio_fastrtc_demo.py` | WebRTC | Gapless streaming (requires `pip install fastrtc`) |
|
||||
|
||||
```bash
|
||||
# Option 1: Launch server + Standard Gradio together
|
||||
./run_gradio_demo.sh # CustomVoice (default)
|
||||
./run_gradio_demo.sh --task-type VoiceDesign # VoiceDesign
|
||||
./run_gradio_demo.sh --task-type Base # Voice cloning
|
||||
|
||||
# Option 2: If server is already running
|
||||
python gradio_demo.py --api-base http://localhost:8000
|
||||
|
||||
# Option 3: FastRTC demo (gapless streaming)
|
||||
pip install fastrtc
|
||||
python gradio_fastrtc_demo.py --api-base http://localhost:8000
|
||||
```
|
||||
|
||||
Then open http://localhost:7860 in your browser.
|
||||
|
||||
## Run examples (Qwen3-TTS)
|
||||
|
||||
### Launch the Server
|
||||
@@ -289,7 +314,7 @@ Returns binary audio data with appropriate `Content-Type` header (e.g., `audio/w
|
||||
| `language` | string | "Auto" | Language (see supported languages below) |
|
||||
| `instructions` | string | "" | Voice style/emotion instructions |
|
||||
| `max_new_tokens` | int | 2048 | Maximum tokens to generate |
|
||||
| `initial_codec_chunk_frames` | int | null | Initial chunk size for reduced TTFA (overrides stage config) |
|
||||
| `initial_codec_chunk_frames` | int | null | Per-request initial chunk size override for TTFA tuning. When null, IC is computed dynamically based on server load. |
|
||||
| `stream` | bool | false | Stream raw PCM chunks as they are decoded (requires `response_format="pcm"`) |
|
||||
|
||||
**Supported languages:** Auto, Chinese, English, Japanese, Korean, German, French, Russian, Portuguese, Spanish, Italian
|
||||
|
||||
@@ -0,0 +1,404 @@
|
||||
"""Gradio demo for Qwen3-TTS online serving via /v1/audio/speech API.
|
||||
|
||||
Uses Gradio's built-in audio component for playback. Supports streaming
|
||||
(progressive PCM chunks) and non-streaming (full audio download) modes.
|
||||
|
||||
For gapless streaming playback, see gradio_fastrtc_demo.py which uses WebRTC.
|
||||
|
||||
Supports all 3 task types:
|
||||
- CustomVoice: Predefined speaker with optional style instructions
|
||||
- VoiceDesign: Natural language voice description
|
||||
- Base: Voice cloning from reference audio (upload or URL)
|
||||
|
||||
Usage:
|
||||
# Start the server first (see run_server.sh), then:
|
||||
python gradio_demo.py --api-base http://localhost:8000
|
||||
|
||||
# Or use run_gradio_demo.sh to start both server and demo together.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import io
|
||||
|
||||
import gradio as gr
|
||||
import httpx
|
||||
import numpy as np
|
||||
import soundfile as sf
|
||||
from tts_common import (
|
||||
PCM_SAMPLE_RATE,
|
||||
SUPPORTED_LANGUAGES,
|
||||
TASK_TYPES,
|
||||
add_common_args,
|
||||
build_payload,
|
||||
fetch_voices,
|
||||
on_task_type_change,
|
||||
stream_pcm_chunks,
|
||||
)
|
||||
|
||||
|
||||
def generate_speech(
|
||||
api_base: str,
|
||||
text: str,
|
||||
task_type: str,
|
||||
voice: str,
|
||||
language: str,
|
||||
instructions: str,
|
||||
ref_audio: tuple | None,
|
||||
ref_audio_url: str,
|
||||
ref_text: str,
|
||||
x_vector_only: bool,
|
||||
response_format: str,
|
||||
speed: float,
|
||||
):
|
||||
"""Non-streaming: call /v1/audio/speech and return audio."""
|
||||
payload = build_payload(
|
||||
text,
|
||||
task_type,
|
||||
voice,
|
||||
language,
|
||||
instructions,
|
||||
ref_audio,
|
||||
ref_audio_url,
|
||||
ref_text,
|
||||
x_vector_only,
|
||||
response_format,
|
||||
speed,
|
||||
stream=False,
|
||||
)
|
||||
|
||||
try:
|
||||
with httpx.Client(timeout=300.0) as client:
|
||||
resp = client.post(
|
||||
f"{api_base}/v1/audio/speech",
|
||||
json=payload,
|
||||
headers={
|
||||
"Content-Type": "application/json",
|
||||
"Authorization": "Bearer EMPTY",
|
||||
},
|
||||
)
|
||||
except httpx.TimeoutException:
|
||||
raise gr.Error("Request timed out. The server may be busy.")
|
||||
except httpx.ConnectError:
|
||||
raise gr.Error(f"Cannot connect to server at {api_base}. Make sure the vLLM server is running.")
|
||||
|
||||
if resp.status_code != 200:
|
||||
raise gr.Error(f"Server error ({resp.status_code}): {resp.text}")
|
||||
|
||||
content_type = resp.headers.get("content-type", "")
|
||||
if "application/json" in content_type:
|
||||
try:
|
||||
error_data = resp.json()
|
||||
raise gr.Error(f"Server error: {error_data}")
|
||||
except ValueError:
|
||||
pass
|
||||
|
||||
try:
|
||||
if response_format == "pcm":
|
||||
audio_np = np.frombuffer(resp.content, dtype=np.int16).astype(np.float32) / 32767.0
|
||||
return (PCM_SAMPLE_RATE, audio_np)
|
||||
audio_np, sample_rate = sf.read(io.BytesIO(resp.content))
|
||||
if audio_np.ndim > 1:
|
||||
audio_np = audio_np[:, 0]
|
||||
return (sample_rate, audio_np.astype(np.float32))
|
||||
except Exception as e:
|
||||
raise gr.Error(f"Failed to decode audio response: {e}")
|
||||
|
||||
|
||||
def generate_speech_stream(
|
||||
api_base: str,
|
||||
text: str,
|
||||
task_type: str,
|
||||
voice: str,
|
||||
language: str,
|
||||
instructions: str,
|
||||
ref_audio: tuple | None,
|
||||
ref_audio_url: str,
|
||||
ref_text: str,
|
||||
x_vector_only: bool,
|
||||
response_format: str,
|
||||
speed: float,
|
||||
chunk_seconds: float = 0.25,
|
||||
):
|
||||
"""Streaming: yield individual PCM chunks for progressive playback."""
|
||||
payload = build_payload(
|
||||
text,
|
||||
task_type,
|
||||
voice,
|
||||
language,
|
||||
instructions,
|
||||
ref_audio,
|
||||
ref_audio_url,
|
||||
ref_text,
|
||||
x_vector_only,
|
||||
response_format,
|
||||
speed,
|
||||
stream=True,
|
||||
)
|
||||
|
||||
min_chunk_samples = int(PCM_SAMPLE_RATE * chunk_seconds)
|
||||
# Pre-buffer 2 chunks before first playback to avoid the gap
|
||||
# between chunk 1 and chunk 2 during progressive streaming.
|
||||
prebuffer_chunks = 2
|
||||
pending: list[np.ndarray] = []
|
||||
pending_len = 0
|
||||
chunks_yielded = 0
|
||||
prebuffer: list[np.ndarray] = []
|
||||
|
||||
try:
|
||||
for samples in stream_pcm_chunks(api_base, payload):
|
||||
float_samples = samples.astype(np.float32) / 32767.0
|
||||
pending.append(float_samples)
|
||||
pending_len += len(float_samples)
|
||||
if pending_len >= min_chunk_samples:
|
||||
audio_chunk = np.concatenate(pending)
|
||||
pending.clear()
|
||||
pending_len = 0
|
||||
|
||||
if chunks_yielded < prebuffer_chunks:
|
||||
prebuffer.append(audio_chunk)
|
||||
chunks_yielded += 1
|
||||
if chunks_yielded == prebuffer_chunks:
|
||||
yield (PCM_SAMPLE_RATE, np.concatenate(prebuffer))
|
||||
prebuffer.clear()
|
||||
else:
|
||||
yield (PCM_SAMPLE_RATE, audio_chunk)
|
||||
# Flush: prebuffer (if stream was shorter than 2 chunks) + remaining pending
|
||||
remaining = prebuffer + pending
|
||||
if remaining:
|
||||
yield (PCM_SAMPLE_RATE, np.concatenate(remaining))
|
||||
except httpx.TimeoutException:
|
||||
raise gr.Error("Request timed out. The server may be busy.")
|
||||
except httpx.ConnectError:
|
||||
raise gr.Error(f"Cannot connect to server at {api_base}. Make sure the vLLM server is running.")
|
||||
|
||||
|
||||
def on_stream_change(stream: bool):
|
||||
"""When streaming is enabled, lock format to PCM and disable speed."""
|
||||
if stream:
|
||||
return (
|
||||
gr.update(value="pcm", interactive=False),
|
||||
gr.update(interactive=False),
|
||||
)
|
||||
return (
|
||||
gr.update(value="wav", interactive=True),
|
||||
gr.update(interactive=True),
|
||||
)
|
||||
|
||||
|
||||
def build_interface(api_base: str, stream_chunk_seconds: float = 0.25):
|
||||
"""Build the Gradio interface."""
|
||||
voices = fetch_voices(api_base)
|
||||
|
||||
css = """
|
||||
#generate-btn button { width: 100%; }
|
||||
.task-info { padding: 8px 12px; border-radius: 6px;
|
||||
background: #f0f4ff; margin-bottom: 8px; }
|
||||
"""
|
||||
|
||||
with gr.Blocks(css=css, title="Qwen3-TTS Demo") as demo:
|
||||
gr.Markdown("# Qwen3-TTS Online Serving Demo")
|
||||
gr.Markdown(f"**Server:** `{api_base}`")
|
||||
|
||||
with gr.Row():
|
||||
# Left column: inputs
|
||||
with gr.Column(scale=3):
|
||||
text_input = gr.Textbox(
|
||||
label="Text to Synthesize",
|
||||
placeholder="Enter text here, e.g., Hello, how are you?",
|
||||
lines=4,
|
||||
)
|
||||
|
||||
with gr.Row():
|
||||
task_type = gr.Radio(
|
||||
choices=TASK_TYPES,
|
||||
value="CustomVoice",
|
||||
label="Task Type",
|
||||
scale=2,
|
||||
)
|
||||
language = gr.Dropdown(
|
||||
choices=SUPPORTED_LANGUAGES,
|
||||
value="Auto",
|
||||
label="Language",
|
||||
scale=1,
|
||||
)
|
||||
|
||||
voice = gr.Dropdown(
|
||||
choices=voices,
|
||||
value=voices[0] if voices else None,
|
||||
label="Speaker",
|
||||
visible=True,
|
||||
)
|
||||
|
||||
instructions = gr.Textbox(
|
||||
label="Instructions",
|
||||
placeholder=("e.g., Speak with excitement / A warm, friendly female voice"),
|
||||
lines=2,
|
||||
visible=True,
|
||||
info="Optional style/emotion instructions",
|
||||
)
|
||||
|
||||
ref_audio = gr.Audio(
|
||||
label="Reference Audio (upload for voice cloning)",
|
||||
type="numpy",
|
||||
sources=["upload", "microphone"],
|
||||
visible=False,
|
||||
)
|
||||
ref_audio_url = gr.Textbox(
|
||||
label="Reference Audio URL",
|
||||
placeholder=("https://example.com/reference.wav (alternative to uploading)"),
|
||||
lines=1,
|
||||
visible=False,
|
||||
)
|
||||
ref_text = gr.Textbox(
|
||||
label="Reference Audio Transcript",
|
||||
placeholder=("Transcript of the reference audio (optional, improves quality)"),
|
||||
lines=2,
|
||||
visible=False,
|
||||
)
|
||||
x_vector_only = gr.Checkbox(
|
||||
label="Use x-vector only",
|
||||
value=False,
|
||||
visible=False,
|
||||
info=("Skip reference transcript, use speaker embedding only (lower quality)"),
|
||||
)
|
||||
|
||||
with gr.Row():
|
||||
response_format = gr.Dropdown(
|
||||
choices=["wav", "mp3", "flac", "pcm", "aac", "opus"],
|
||||
value="wav",
|
||||
label="Audio Format",
|
||||
scale=1,
|
||||
)
|
||||
speed = gr.Slider(
|
||||
minimum=0.25,
|
||||
maximum=4.0,
|
||||
value=1.0,
|
||||
step=0.05,
|
||||
label="Speed",
|
||||
scale=1,
|
||||
)
|
||||
stream_checkbox = gr.Checkbox(
|
||||
label="Stream output",
|
||||
value=False,
|
||||
info=(
|
||||
"Enable streaming (PCM format, speed disabled). "
|
||||
"For gapless streaming, use gradio_fastrtc_demo.py"
|
||||
),
|
||||
scale=1,
|
||||
)
|
||||
|
||||
generate_btn = gr.Button(
|
||||
"Generate Speech",
|
||||
variant="primary",
|
||||
size="lg",
|
||||
elem_id="generate-btn",
|
||||
)
|
||||
|
||||
# Right column: output
|
||||
with gr.Column(scale=2):
|
||||
audio_output = gr.Audio(
|
||||
label="Generated Audio",
|
||||
interactive=False,
|
||||
streaming=True,
|
||||
autoplay=True,
|
||||
)
|
||||
gr.Markdown(
|
||||
"### Task Types\n"
|
||||
"- **CustomVoice**: Use a predefined speaker "
|
||||
"(Vivian, Ryan, etc.) with optional style instructions\n"
|
||||
"- **VoiceDesign**: Describe the desired voice in natural "
|
||||
"language (instructions required)\n"
|
||||
"- **Base**: Clone a voice from reference audio "
|
||||
"(upload a file or provide a URL)"
|
||||
)
|
||||
|
||||
# Dynamic UI updates
|
||||
task_type.change(
|
||||
fn=on_task_type_change,
|
||||
inputs=[task_type],
|
||||
outputs=[
|
||||
voice,
|
||||
instructions,
|
||||
ref_audio,
|
||||
ref_audio_url,
|
||||
ref_text,
|
||||
x_vector_only,
|
||||
],
|
||||
)
|
||||
|
||||
stream_checkbox.change(
|
||||
fn=on_stream_change,
|
||||
inputs=[stream_checkbox],
|
||||
outputs=[response_format, speed],
|
||||
)
|
||||
|
||||
all_inputs = [
|
||||
text_input,
|
||||
task_type,
|
||||
voice,
|
||||
language,
|
||||
instructions,
|
||||
ref_audio,
|
||||
ref_audio_url,
|
||||
ref_text,
|
||||
x_vector_only,
|
||||
response_format,
|
||||
speed,
|
||||
]
|
||||
|
||||
def dispatch(stream_enabled, *args):
|
||||
if stream_enabled:
|
||||
yield from generate_speech_stream(
|
||||
api_base,
|
||||
*args,
|
||||
chunk_seconds=stream_chunk_seconds,
|
||||
)
|
||||
else:
|
||||
yield generate_speech(api_base, *args)
|
||||
|
||||
generate_btn.click(
|
||||
fn=dispatch,
|
||||
inputs=[stream_checkbox] + all_inputs,
|
||||
outputs=[audio_output],
|
||||
)
|
||||
|
||||
demo.queue()
|
||||
return demo
|
||||
|
||||
|
||||
def parse_args():
|
||||
parser = argparse.ArgumentParser(description="Gradio demo for Qwen3-TTS online serving.")
|
||||
add_common_args(parser)
|
||||
parser.add_argument(
|
||||
"--stream-chunk-seconds",
|
||||
type=float,
|
||||
default=0.25,
|
||||
help=(
|
||||
"Seconds of audio to buffer per streaming chunk (default: 0.25). "
|
||||
"Lower = smoother but more overhead, higher = chunkier."
|
||||
),
|
||||
)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def main():
|
||||
args = parse_args()
|
||||
|
||||
print(f"Connecting to vLLM server at: {args.api_base}")
|
||||
demo = build_interface(
|
||||
args.api_base,
|
||||
stream_chunk_seconds=args.stream_chunk_seconds,
|
||||
)
|
||||
|
||||
try:
|
||||
demo.launch(
|
||||
server_name=args.host,
|
||||
server_port=args.port,
|
||||
share=args.share,
|
||||
)
|
||||
except KeyboardInterrupt:
|
||||
print("\nShutting down...")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,274 @@
|
||||
"""Gradio demo for Qwen3-TTS using FastRTC for gapless streaming audio.
|
||||
|
||||
Uses WebRTC via FastRTC to deliver audio chunks as a continuous real-time
|
||||
stream, avoiding the inter-chunk gaps present in Gradio's built-in streaming.
|
||||
|
||||
Requires: pip install fastrtc
|
||||
|
||||
Supports all 3 task types:
|
||||
- CustomVoice: Predefined speaker with optional style instructions
|
||||
- VoiceDesign: Natural language voice description
|
||||
- Base: Voice cloning from reference audio (URL only in this demo)
|
||||
|
||||
Usage:
|
||||
# Start the server first (see run_server.sh), then:
|
||||
python gradio_fastrtc_demo.py --api-base http://localhost:8000
|
||||
|
||||
# With public sharing:
|
||||
python gradio_fastrtc_demo.py --api-base http://localhost:8000 --share
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import logging
|
||||
from collections.abc import Generator
|
||||
|
||||
import gradio as gr
|
||||
import numpy as np
|
||||
from fastrtc import WebRTC
|
||||
from tts_common import (
|
||||
PCM_SAMPLE_RATE,
|
||||
SUPPORTED_LANGUAGES,
|
||||
TASK_TYPES,
|
||||
add_common_args,
|
||||
build_payload,
|
||||
fetch_voices,
|
||||
stream_pcm_chunks,
|
||||
)
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def make_tts_generator(api_base: str):
|
||||
"""Create a generator function for FastRTC's mode='receive'.
|
||||
|
||||
FastRTC calls this with the additional_inputs values whenever the user
|
||||
clicks "Start Stream". It must be a synchronous generator yielding
|
||||
(sample_rate, int16_ndarray) tuples.
|
||||
"""
|
||||
|
||||
def tts_generate(
|
||||
text: str,
|
||||
task_type: str,
|
||||
voice: str,
|
||||
language: str,
|
||||
instructions: str,
|
||||
ref_audio_url: str,
|
||||
ref_text: str,
|
||||
x_vector_only: bool,
|
||||
) -> Generator[tuple[int, np.ndarray], None, None]:
|
||||
if not text or not text.strip():
|
||||
return
|
||||
|
||||
payload = build_payload(
|
||||
text,
|
||||
task_type,
|
||||
voice,
|
||||
language,
|
||||
instructions,
|
||||
ref_audio=None, # WebRTC UI doesn't support file upload
|
||||
ref_audio_url=ref_audio_url,
|
||||
ref_text=ref_text,
|
||||
x_vector_only=x_vector_only,
|
||||
stream=True,
|
||||
)
|
||||
|
||||
try:
|
||||
for samples in stream_pcm_chunks(api_base, payload):
|
||||
yield (PCM_SAMPLE_RATE, samples)
|
||||
except Exception:
|
||||
logger.exception("TTS streaming error")
|
||||
|
||||
return tts_generate
|
||||
|
||||
|
||||
def build_interface(api_base: str):
|
||||
"""Build a custom Gradio Blocks UI matching gradio_demo.py layout."""
|
||||
voices = fetch_voices(api_base)
|
||||
handler = make_tts_generator(api_base)
|
||||
|
||||
css = """
|
||||
#generate-btn button { width: 100%; }
|
||||
.task-info { padding: 8px 12px; border-radius: 6px;
|
||||
background: #f0f4ff; margin-bottom: 8px; }
|
||||
"""
|
||||
|
||||
with gr.Blocks(css=css, title="Qwen3-TTS (FastRTC)") as demo:
|
||||
gr.Markdown("# Qwen3-TTS Online Serving Demo (FastRTC)")
|
||||
gr.Markdown(f"**Server:** `{api_base}` | **Transport:** WebRTC (gapless streaming)")
|
||||
|
||||
with gr.Row():
|
||||
# Left column: inputs
|
||||
with gr.Column(scale=3):
|
||||
text_input = gr.Textbox(
|
||||
label="Text to Synthesize",
|
||||
placeholder="Enter text here, e.g., Hello, how are you?",
|
||||
lines=4,
|
||||
)
|
||||
|
||||
with gr.Row():
|
||||
task_type = gr.Radio(
|
||||
choices=TASK_TYPES,
|
||||
value="CustomVoice",
|
||||
label="Task Type",
|
||||
scale=2,
|
||||
)
|
||||
language = gr.Dropdown(
|
||||
choices=SUPPORTED_LANGUAGES,
|
||||
value="Auto",
|
||||
label="Language",
|
||||
scale=1,
|
||||
)
|
||||
|
||||
voice = gr.Dropdown(
|
||||
choices=voices,
|
||||
value=voices[0] if voices else None,
|
||||
label="Speaker",
|
||||
visible=True,
|
||||
)
|
||||
|
||||
instructions = gr.Textbox(
|
||||
label="Instructions",
|
||||
placeholder=("e.g., Speak with excitement / A warm, friendly female voice"),
|
||||
lines=2,
|
||||
visible=True,
|
||||
info="Optional style/emotion instructions",
|
||||
)
|
||||
|
||||
# Base (voice clone) controls
|
||||
# Note: file upload is not supported with WebRTC,
|
||||
# use a URL instead.
|
||||
ref_audio_url = gr.Textbox(
|
||||
label="Reference Audio URL",
|
||||
placeholder=("https://example.com/reference.wav (alternative to uploading)"),
|
||||
lines=1,
|
||||
visible=False,
|
||||
)
|
||||
ref_text = gr.Textbox(
|
||||
label="Reference Audio Transcript",
|
||||
placeholder=("Transcript of the reference audio (optional, improves quality)"),
|
||||
lines=2,
|
||||
visible=False,
|
||||
)
|
||||
x_vector_only = gr.Checkbox(
|
||||
label="Use x-vector only",
|
||||
value=False,
|
||||
visible=False,
|
||||
info=("Skip reference transcript, use speaker embedding only (lower quality)"),
|
||||
)
|
||||
|
||||
generate_btn = gr.Button(
|
||||
"Generate Speech",
|
||||
variant="primary",
|
||||
size="lg",
|
||||
elem_id="generate-btn",
|
||||
)
|
||||
|
||||
# Right column: output
|
||||
with gr.Column(scale=2):
|
||||
webrtc_output = WebRTC(
|
||||
label="Generated Audio (WebRTC)",
|
||||
mode="receive",
|
||||
modality="audio",
|
||||
)
|
||||
gr.Markdown(
|
||||
"### Task Types\n"
|
||||
"- **CustomVoice**: Use a predefined speaker "
|
||||
"(Vivian, Ryan, etc.) with optional style instructions\n"
|
||||
"- **VoiceDesign**: Describe the desired voice in natural "
|
||||
"language (instructions required)\n"
|
||||
"- **Base**: Clone a voice from reference audio "
|
||||
"(provide a URL)"
|
||||
)
|
||||
|
||||
# Dynamic UI updates for task type
|
||||
# FastRTC doesn't support file upload, so we only toggle
|
||||
# 5 components (no ref_audio)
|
||||
def on_task_type_change_fastrtc(task_type: str):
|
||||
if task_type == "CustomVoice":
|
||||
return (
|
||||
gr.update(visible=True), # voice
|
||||
gr.update(visible=True, info="Optional style/emotion instructions"),
|
||||
gr.update(visible=False), # ref_audio_url
|
||||
gr.update(visible=False), # ref_text
|
||||
gr.update(visible=False), # x_vector_only
|
||||
)
|
||||
elif task_type == "VoiceDesign":
|
||||
return (
|
||||
gr.update(visible=False),
|
||||
gr.update(visible=True, info="Required: describe the voice style"),
|
||||
gr.update(visible=False),
|
||||
gr.update(visible=False),
|
||||
gr.update(visible=False),
|
||||
)
|
||||
elif task_type == "Base":
|
||||
return (
|
||||
gr.update(visible=False),
|
||||
gr.update(visible=False),
|
||||
gr.update(visible=True),
|
||||
gr.update(visible=True),
|
||||
gr.update(visible=True),
|
||||
)
|
||||
return (
|
||||
gr.update(visible=True),
|
||||
gr.update(visible=True),
|
||||
gr.update(visible=False),
|
||||
gr.update(visible=False),
|
||||
gr.update(visible=False),
|
||||
)
|
||||
|
||||
task_type.change(
|
||||
fn=on_task_type_change_fastrtc,
|
||||
inputs=[task_type],
|
||||
outputs=[
|
||||
voice,
|
||||
instructions,
|
||||
ref_audio_url,
|
||||
ref_text,
|
||||
x_vector_only,
|
||||
],
|
||||
)
|
||||
|
||||
# Wire up WebRTC streaming
|
||||
all_inputs = [
|
||||
text_input,
|
||||
task_type,
|
||||
voice,
|
||||
language,
|
||||
instructions,
|
||||
ref_audio_url,
|
||||
ref_text,
|
||||
x_vector_only,
|
||||
]
|
||||
|
||||
webrtc_output.stream(
|
||||
fn=handler,
|
||||
inputs=all_inputs,
|
||||
outputs=[webrtc_output],
|
||||
trigger=generate_btn.click,
|
||||
)
|
||||
|
||||
demo.queue()
|
||||
return demo
|
||||
|
||||
|
||||
def parse_args():
|
||||
parser = argparse.ArgumentParser(description="Gradio FastRTC demo for Qwen3-TTS (gapless streaming).")
|
||||
add_common_args(parser)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def main():
|
||||
args = parse_args()
|
||||
logging.basicConfig(level=logging.INFO)
|
||||
print(f"Connecting to vLLM server at: {args.api_base}")
|
||||
|
||||
demo = build_interface(args.api_base)
|
||||
demo.launch(
|
||||
server_name=args.host,
|
||||
server_port=args.port,
|
||||
share=args.share,
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,203 @@
|
||||
#!/bin/bash
|
||||
# Launch both vLLM server and Gradio demo for Qwen3-TTS
|
||||
#
|
||||
# Usage:
|
||||
# ./run_gradio_demo.sh # Default: CustomVoice
|
||||
# ./run_gradio_demo.sh --task-type VoiceDesign # VoiceDesign model
|
||||
# ./run_gradio_demo.sh --task-type Base --gradio-port 7861
|
||||
#
|
||||
# Options:
|
||||
# --task-type TYPE Task type: CustomVoice, VoiceDesign, Base (default: CustomVoice)
|
||||
# --server-port PORT Port for vLLM server (default: 8000)
|
||||
# --gradio-port PORT Port for Gradio demo (default: 7860)
|
||||
# --server-host HOST Host for vLLM server (default: 0.0.0.0)
|
||||
# --gradio-ip IP IP for Gradio demo (default: 127.0.0.1)
|
||||
# --share Share Gradio demo publicly
|
||||
|
||||
set -e
|
||||
|
||||
# Default values
|
||||
TASK_TYPE="CustomVoice"
|
||||
SERVER_PORT=8000
|
||||
GRADIO_PORT=7860
|
||||
SERVER_HOST="0.0.0.0"
|
||||
GRADIO_IP="127.0.0.1"
|
||||
GRADIO_SHARE=false
|
||||
|
||||
# Parse command line arguments
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case $1 in
|
||||
--task-type)
|
||||
TASK_TYPE="$2"
|
||||
shift 2
|
||||
;;
|
||||
--server-port)
|
||||
SERVER_PORT="$2"
|
||||
shift 2
|
||||
;;
|
||||
--gradio-port)
|
||||
GRADIO_PORT="$2"
|
||||
shift 2
|
||||
;;
|
||||
--server-host)
|
||||
SERVER_HOST="$2"
|
||||
shift 2
|
||||
;;
|
||||
--gradio-ip)
|
||||
GRADIO_IP="$2"
|
||||
shift 2
|
||||
;;
|
||||
--share)
|
||||
GRADIO_SHARE=true
|
||||
shift
|
||||
;;
|
||||
--help)
|
||||
echo "Usage: $0 [OPTIONS]"
|
||||
echo ""
|
||||
echo "Options:"
|
||||
echo " --task-type TYPE Task type: CustomVoice, VoiceDesign, Base (default: CustomVoice)"
|
||||
echo " --server-port PORT Port for vLLM server (default: 8000)"
|
||||
echo " --gradio-port PORT Port for Gradio demo (default: 7860)"
|
||||
echo " --server-host HOST Host for vLLM server (default: 0.0.0.0)"
|
||||
echo " --gradio-ip IP IP for Gradio demo (default: 127.0.0.1)"
|
||||
echo " --share Share Gradio demo publicly"
|
||||
echo ""
|
||||
exit 0
|
||||
;;
|
||||
*)
|
||||
echo "Unknown option: $1"
|
||||
echo "Use --help for usage information"
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
done
|
||||
|
||||
# Map task type to model
|
||||
case "$TASK_TYPE" in
|
||||
CustomVoice)
|
||||
MODEL="Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice"
|
||||
;;
|
||||
VoiceDesign)
|
||||
MODEL="Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign"
|
||||
;;
|
||||
Base)
|
||||
MODEL="Qwen/Qwen3-TTS-12Hz-1.7B-Base"
|
||||
;;
|
||||
*)
|
||||
echo "Unknown task type: $TASK_TYPE"
|
||||
echo "Supported: CustomVoice, VoiceDesign, Base"
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
API_BASE="http://localhost:${SERVER_PORT}"
|
||||
|
||||
echo "=========================================="
|
||||
echo "Qwen3-TTS Gradio Demo"
|
||||
echo "=========================================="
|
||||
echo "Task Type : $TASK_TYPE"
|
||||
echo "Model : $MODEL"
|
||||
echo "Server : http://${SERVER_HOST}:${SERVER_PORT}"
|
||||
echo "Gradio : http://${GRADIO_IP}:${GRADIO_PORT}"
|
||||
echo "=========================================="
|
||||
|
||||
# Cleanup on exit
|
||||
cleanup() {
|
||||
echo ""
|
||||
echo "Shutting down..."
|
||||
if [ -n "$SERVER_PID" ]; then
|
||||
echo "Stopping vLLM server (PID: $SERVER_PID)..."
|
||||
kill "$SERVER_PID" 2>/dev/null || true
|
||||
wait "$SERVER_PID" 2>/dev/null || true
|
||||
fi
|
||||
if [ -n "$GRADIO_PID" ]; then
|
||||
echo "Stopping Gradio demo (PID: $GRADIO_PID)..."
|
||||
kill "$GRADIO_PID" 2>/dev/null || true
|
||||
wait "$GRADIO_PID" 2>/dev/null || true
|
||||
fi
|
||||
echo "Cleanup complete"
|
||||
exit 0
|
||||
}
|
||||
trap cleanup SIGINT SIGTERM
|
||||
|
||||
# Start vLLM server
|
||||
echo ""
|
||||
echo "Starting vLLM server..."
|
||||
LOG_FILE="/tmp/vllm_tts_server_${SERVER_PORT}.log"
|
||||
|
||||
vllm-omni serve "$MODEL" \
|
||||
--stage-configs-path vllm_omni/model_executor/stage_configs/qwen3_tts.yaml \
|
||||
--host "$SERVER_HOST" \
|
||||
--port "$SERVER_PORT" \
|
||||
--gpu-memory-utilization 0.9 \
|
||||
--trust-remote-code \
|
||||
--enforce-eager \
|
||||
--omni 2>&1 | tee "$LOG_FILE" &
|
||||
SERVER_PID=$!
|
||||
|
||||
# Wait for server startup
|
||||
echo ""
|
||||
echo "Waiting for vLLM server to be ready..."
|
||||
STARTUP_FLAG="/tmp/vllm_tts_startup_flag_${SERVER_PORT}.tmp"
|
||||
rm -f "$STARTUP_FLAG"
|
||||
|
||||
(
|
||||
tail -f "$LOG_FILE" 2>/dev/null | grep -m 1 "Application startup complete" > /dev/null && touch "$STARTUP_FLAG"
|
||||
) &
|
||||
TAIL_PID=$!
|
||||
|
||||
MAX_WAIT=300
|
||||
ELAPSED=0
|
||||
while [ $ELAPSED -lt $MAX_WAIT ]; do
|
||||
if [ -f "$STARTUP_FLAG" ]; then
|
||||
kill "$TAIL_PID" 2>/dev/null || true
|
||||
wait "$TAIL_PID" 2>/dev/null || true
|
||||
echo ""
|
||||
echo "vLLM server is ready!"
|
||||
break
|
||||
fi
|
||||
if ! kill -0 "$SERVER_PID" 2>/dev/null; then
|
||||
kill "$TAIL_PID" 2>/dev/null || true
|
||||
echo ""
|
||||
echo "Error: vLLM server failed to start"
|
||||
exit 1
|
||||
fi
|
||||
sleep 1
|
||||
ELAPSED=$((ELAPSED + 1))
|
||||
done
|
||||
|
||||
rm -f "$STARTUP_FLAG"
|
||||
|
||||
if [ $ELAPSED -ge $MAX_WAIT ]; then
|
||||
kill "$TAIL_PID" 2>/dev/null || true
|
||||
echo "Error: Server startup timed out after ${MAX_WAIT}s"
|
||||
kill "$SERVER_PID" 2>/dev/null || true
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Start Gradio demo
|
||||
echo ""
|
||||
echo "Starting Gradio demo..."
|
||||
cd "$SCRIPT_DIR"
|
||||
GRADIO_CMD=("python" "gradio_demo.py" "--api-base" "$API_BASE" "--ip" "$GRADIO_IP" "--port" "$GRADIO_PORT")
|
||||
if [ "$GRADIO_SHARE" = true ]; then
|
||||
GRADIO_CMD+=("--share")
|
||||
fi
|
||||
|
||||
"${GRADIO_CMD[@]}" &
|
||||
GRADIO_PID=$!
|
||||
|
||||
echo ""
|
||||
echo "=========================================="
|
||||
echo "Both services are running!"
|
||||
echo "=========================================="
|
||||
echo "vLLM Server : http://${SERVER_HOST}:${SERVER_PORT}"
|
||||
echo "Gradio Demo : http://${GRADIO_IP}:${GRADIO_PORT}"
|
||||
echo ""
|
||||
echo "Press Ctrl+C to stop both services"
|
||||
echo "=========================================="
|
||||
echo ""
|
||||
|
||||
wait $SERVER_PID $GRADIO_PID || true
|
||||
cleanup
|
||||
@@ -0,0 +1,218 @@
|
||||
"""Shared constants, helpers, and payload building for Qwen3-TTS Gradio demos."""
|
||||
|
||||
import base64
|
||||
import io
|
||||
|
||||
import gradio as gr
|
||||
import httpx
|
||||
import numpy as np
|
||||
import soundfile as sf
|
||||
|
||||
SUPPORTED_LANGUAGES = [
|
||||
"Auto",
|
||||
"Chinese",
|
||||
"English",
|
||||
"Japanese",
|
||||
"Korean",
|
||||
"German",
|
||||
"French",
|
||||
"Russian",
|
||||
"Portuguese",
|
||||
"Spanish",
|
||||
"Italian",
|
||||
]
|
||||
|
||||
TASK_TYPES = ["CustomVoice", "VoiceDesign", "Base"]
|
||||
|
||||
PCM_SAMPLE_RATE = 24000
|
||||
|
||||
DEFAULT_API_BASE = "http://localhost:8000"
|
||||
|
||||
|
||||
def fetch_voices(api_base: str) -> list[str]:
|
||||
"""Fetch available voices from the server."""
|
||||
try:
|
||||
with httpx.Client(timeout=10.0) as client:
|
||||
resp = client.get(
|
||||
f"{api_base}/v1/audio/voices",
|
||||
headers={"Authorization": "Bearer EMPTY"},
|
||||
)
|
||||
if resp.status_code == 200:
|
||||
data = resp.json()
|
||||
return data.get("voices", ["Vivian", "Ryan"])
|
||||
except Exception:
|
||||
pass
|
||||
return ["Vivian", "Ryan"]
|
||||
|
||||
|
||||
def encode_audio_to_base64(audio_data: tuple) -> str:
|
||||
"""Encode Gradio audio input (sample_rate, numpy_array) to base64 data URL."""
|
||||
sample_rate, audio_np = audio_data
|
||||
|
||||
if audio_np.dtype != np.int16:
|
||||
if audio_np.dtype in (np.float32, np.float64):
|
||||
audio_np = np.clip(audio_np, -1.0, 1.0)
|
||||
audio_np = (audio_np * 32767).astype(np.int16)
|
||||
else:
|
||||
audio_np = audio_np.astype(np.int16)
|
||||
|
||||
buf = io.BytesIO()
|
||||
sf.write(buf, audio_np, sample_rate, format="WAV")
|
||||
wav_b64 = base64.b64encode(buf.getvalue()).decode("utf-8")
|
||||
return f"data:audio/wav;base64,{wav_b64}"
|
||||
|
||||
|
||||
def build_payload(
|
||||
text: str,
|
||||
task_type: str,
|
||||
voice: str,
|
||||
language: str,
|
||||
instructions: str,
|
||||
ref_audio: tuple | None,
|
||||
ref_audio_url: str,
|
||||
ref_text: str,
|
||||
x_vector_only: bool,
|
||||
response_format: str = "pcm",
|
||||
speed: float = 1.0,
|
||||
stream: bool = True,
|
||||
) -> dict:
|
||||
"""Build the /v1/audio/speech request payload.
|
||||
|
||||
Raises gr.Error for invalid input so callers don't need to validate.
|
||||
"""
|
||||
if not text or not text.strip():
|
||||
raise gr.Error("Please enter text to synthesize.")
|
||||
|
||||
payload: dict = {
|
||||
"input": text.strip(),
|
||||
"response_format": "pcm" if stream else response_format,
|
||||
"stream": stream,
|
||||
}
|
||||
if not stream:
|
||||
payload["speed"] = speed
|
||||
|
||||
if task_type:
|
||||
payload["task_type"] = task_type
|
||||
if language:
|
||||
payload["language"] = language
|
||||
|
||||
if task_type == "CustomVoice":
|
||||
if voice:
|
||||
payload["voice"] = voice
|
||||
if instructions and instructions.strip():
|
||||
payload["instructions"] = instructions.strip()
|
||||
|
||||
elif task_type == "VoiceDesign":
|
||||
if not instructions or not instructions.strip():
|
||||
raise gr.Error("VoiceDesign task requires voice style instructions.")
|
||||
payload["instructions"] = instructions.strip()
|
||||
|
||||
elif task_type == "Base":
|
||||
ref_audio_url_stripped = ref_audio_url.strip() if ref_audio_url else ""
|
||||
if ref_audio_url_stripped:
|
||||
payload["ref_audio"] = ref_audio_url_stripped
|
||||
elif ref_audio is not None:
|
||||
payload["ref_audio"] = encode_audio_to_base64(ref_audio)
|
||||
else:
|
||||
raise gr.Error("Base (voice clone) task requires reference audio. Upload a file or provide a URL.")
|
||||
if ref_text and ref_text.strip():
|
||||
payload["ref_text"] = ref_text.strip()
|
||||
if x_vector_only:
|
||||
payload["x_vector_only_mode"] = True
|
||||
|
||||
return payload
|
||||
|
||||
|
||||
def on_task_type_change(task_type: str):
|
||||
"""Update UI visibility based on selected task type."""
|
||||
if task_type == "CustomVoice":
|
||||
return (
|
||||
gr.update(visible=True), # voice dropdown
|
||||
gr.update(visible=True, info="Optional style/emotion instructions"),
|
||||
gr.update(visible=False), # ref_audio
|
||||
gr.update(visible=False), # ref_audio_url
|
||||
gr.update(visible=False), # ref_text
|
||||
gr.update(visible=False), # x_vector_only
|
||||
)
|
||||
elif task_type == "VoiceDesign":
|
||||
return (
|
||||
gr.update(visible=False),
|
||||
gr.update(visible=True, info="Required: describe the voice style"),
|
||||
gr.update(visible=False),
|
||||
gr.update(visible=False),
|
||||
gr.update(visible=False),
|
||||
gr.update(visible=False),
|
||||
)
|
||||
elif task_type == "Base":
|
||||
return (
|
||||
gr.update(visible=False),
|
||||
gr.update(visible=False),
|
||||
gr.update(visible=True),
|
||||
gr.update(visible=True),
|
||||
gr.update(visible=True),
|
||||
gr.update(visible=True),
|
||||
)
|
||||
return (
|
||||
gr.update(visible=True),
|
||||
gr.update(visible=True),
|
||||
gr.update(visible=False),
|
||||
gr.update(visible=False),
|
||||
gr.update(visible=False),
|
||||
gr.update(visible=False),
|
||||
)
|
||||
|
||||
|
||||
def stream_pcm_chunks(api_base: str, payload: dict):
|
||||
"""Stream raw PCM bytes from the server, yielding int16 numpy arrays.
|
||||
|
||||
Handles odd-byte boundaries between network chunks.
|
||||
"""
|
||||
leftover = b""
|
||||
with httpx.Client(timeout=300.0) as client:
|
||||
with client.stream(
|
||||
"POST",
|
||||
f"{api_base}/v1/audio/speech",
|
||||
json=payload,
|
||||
headers={
|
||||
"Content-Type": "application/json",
|
||||
"Authorization": "Bearer EMPTY",
|
||||
},
|
||||
) as resp:
|
||||
if resp.status_code != 200:
|
||||
resp.read()
|
||||
raise gr.Error(f"Server error ({resp.status_code}): {resp.text}")
|
||||
for chunk in resp.iter_bytes():
|
||||
if not chunk:
|
||||
continue
|
||||
raw = leftover + chunk
|
||||
usable = len(raw) - (len(raw) % 2)
|
||||
leftover = raw[usable:]
|
||||
if usable == 0:
|
||||
continue
|
||||
yield np.frombuffer(raw[:usable], dtype=np.int16).copy()
|
||||
|
||||
|
||||
def add_common_args(parser):
|
||||
"""Add CLI arguments shared by both demos."""
|
||||
parser.add_argument(
|
||||
"--api-base",
|
||||
default=DEFAULT_API_BASE,
|
||||
help=f"Base URL for the vLLM API server (default: {DEFAULT_API_BASE}).",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--host",
|
||||
default="0.0.0.0",
|
||||
help="Host/IP for Gradio server (default: 0.0.0.0).",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--port",
|
||||
type=int,
|
||||
default=7860,
|
||||
help="Port for Gradio server (default: 7860).",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--share",
|
||||
action="store_true",
|
||||
help="Share the Gradio demo publicly.",
|
||||
)
|
||||
return parser
|
||||
@@ -3,13 +3,16 @@ from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from tests.conftest import OmniServerParams
|
||||
from tests.utils import hardware_test
|
||||
|
||||
models = ["Qwen/Qwen2.5-Omni-7B"]
|
||||
stage_configs = [str(Path(__file__).parent.parent / "e2e" / "stage_configs" / "qwen2_5_omni_ci.yaml")]
|
||||
|
||||
# Create parameter combinations for model and stage config
|
||||
test_params = [(model, stage_config) for model in models for stage_config in stage_configs]
|
||||
test_params = [
|
||||
OmniServerParams(model=model, stage_config_path=stage_config) for model in models for stage_config in stage_configs
|
||||
]
|
||||
|
||||
|
||||
@pytest.mark.core_model
|
||||
|
||||
@@ -63,6 +63,12 @@ def _setup_comfyui_test_environment():
|
||||
mock_comfy_api_input.AudioInput = AudioInput
|
||||
mock_comfy_api_input.VideoInput = VideoInput
|
||||
mock_comfy_api.input = mock_comfy_api_input
|
||||
mock_comfy_api_latest = MagicMock()
|
||||
mock_comfy_api_latest.Types.VideoComponents = MagicMock(side_effect=lambda **kwargs: kwargs)
|
||||
mock_comfy_api_latest.InputImpl.VideoFromComponents = MagicMock(
|
||||
side_effect=lambda _: VideoInput(b"mock_video_from_components")
|
||||
)
|
||||
mock_comfy_api.latest = mock_comfy_api_latest
|
||||
|
||||
def mock_load(_: str | BinaryIO):
|
||||
"""Mock nodes_audio.load that returns a waveform tensor (channels, samples) and sample rate."""
|
||||
@@ -78,5 +84,6 @@ def _setup_comfyui_test_environment():
|
||||
# Install mock modules BEFORE importing any comfyui_vllm_omni code
|
||||
sys.modules["comfy_api"] = mock_comfy_api
|
||||
sys.modules["comfy_api.input"] = mock_comfy_api_input
|
||||
sys.modules["comfy_api.latest"] = mock_comfy_api_latest
|
||||
sys.modules["comfy_extras"] = mock_comfy_extras
|
||||
sys.modules["comfy_extras.nodes_audio"] = mock_nodes_audio
|
||||
@@ -20,11 +20,12 @@ import torch
|
||||
from comfy_api.input import AudioInput, VideoInput
|
||||
from comfyui_vllm_omni.nodes import (
|
||||
VLLMOmniGenerateImage,
|
||||
VLLMOmniGenerateVideo,
|
||||
VLLMOmniTTS,
|
||||
VLLMOmniUnderstanding,
|
||||
VLLMOmniVoiceClone,
|
||||
)
|
||||
from comfyui_vllm_omni.utils.types import AutoregressionSamplingParams, DiffusionSamplingParams
|
||||
from comfyui_vllm_omni.utils.types import AutoregressionSamplingParams, DiffusionSamplingParams, WanModelSpecificParams
|
||||
from PIL import Image
|
||||
from vllm import SamplingParams
|
||||
from vllm.outputs import CompletionOutput, RequestOutput
|
||||
@@ -34,6 +35,8 @@ from vllm_omni.entrypoints.cli.serve import OmniServeCommand
|
||||
from vllm_omni.inputs.data import OmniSamplingParams
|
||||
from vllm_omni.outputs import OmniRequestOutput
|
||||
|
||||
pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
|
||||
|
||||
|
||||
class ServerCase(NamedTuple):
|
||||
"""Parametrizing the model to serve."""
|
||||
@@ -49,6 +52,7 @@ class SamplingCase(NamedTuple):
|
||||
|
||||
kind: "SamplingKind"
|
||||
sampling_params: dict | list[dict] | None
|
||||
lora: dict | None = None
|
||||
|
||||
|
||||
class SamplingKind(StrEnum):
|
||||
@@ -58,11 +62,17 @@ class SamplingKind(StrEnum):
|
||||
UNDERSTANDING_AR_LIST = auto()
|
||||
TTS_NONE = auto()
|
||||
TTS_DIFFUSION_SINGLE = auto()
|
||||
VIDEO_NONE = auto()
|
||||
VIDEO_DIFFUSION_SINGLE = auto()
|
||||
|
||||
|
||||
# Pre-defined arguments to be used in function calls during the tests
|
||||
IMAGE_WIDTH = 64
|
||||
IMAGE_HEIGHT = 64
|
||||
VIDEO_WIDTH = 32
|
||||
VIDEO_HEIGHT = 32
|
||||
VIDEO_FPS = 8
|
||||
VIDEO_NUM_FRAMES = 5
|
||||
DIFFUSION_SINGLE_SAMPLING_PARAMS = DiffusionSamplingParams(
|
||||
{
|
||||
"n": 2,
|
||||
@@ -102,6 +112,10 @@ AR_LIST_SAMPLING_PARAMS = [
|
||||
),
|
||||
]
|
||||
|
||||
VIDEO_MODEL_PARAMS = WanModelSpecificParams({"guidance_scale_2": 5.0, "boundary_ratio": 0.98, "flow_shift": 12.0})
|
||||
|
||||
LORA_PARAMS = {"local_path": "test_lora_path", "name": "test_name", "scale": 0.7, "int_id": 10}
|
||||
|
||||
|
||||
def _build_image_output(size: tuple[int, int] = (IMAGE_WIDTH, IMAGE_HEIGHT), color: str = "red") -> Image.Image:
|
||||
return Image.new("RGB", size, color=color)
|
||||
@@ -181,6 +195,16 @@ def _build_diffusion_image_output_for_images_endpoint() -> OmniRequestOutput:
|
||||
)
|
||||
|
||||
|
||||
def _build_diffusion_video_output() -> OmniRequestOutput:
|
||||
# Small video: VIDEO_NUM_FRAMES frames of (VIDEO_HEIGHT x VIDEO_WIDTH) RGB, shape (F, H, W, C)
|
||||
video_frames = torch.zeros((VIDEO_NUM_FRAMES, VIDEO_HEIGHT, VIDEO_WIDTH, 3), dtype=torch.float32)
|
||||
return OmniRequestOutput.from_diffusion(
|
||||
request_id="test_req_video",
|
||||
images=[video_frames],
|
||||
final_output_type="video",
|
||||
)
|
||||
|
||||
|
||||
def _build_diffusion_image_output_for_chat_endpoint() -> OmniRequestOutput:
|
||||
request_output = MagicMock()
|
||||
request_output.images = [_build_image_output(color="blue")]
|
||||
@@ -193,12 +217,40 @@ def _build_diffusion_image_output_for_chat_endpoint() -> OmniRequestOutput:
|
||||
)
|
||||
|
||||
|
||||
def _assert_sampling_param_values(received: OmniSamplingParams, expected: dict[str, Any]):
|
||||
def _assert_sampling_param_values(
|
||||
received: OmniSamplingParams, expected: dict[str, Any], expected_lora: dict | None = None
|
||||
):
|
||||
for key, expected_value in expected.items():
|
||||
actual_value = getattr(received, key, None)
|
||||
actual_value = getattr(received, key)
|
||||
assert actual_value == expected_value, (
|
||||
f"Expected sampling param '{key}'={expected_value}, got {actual_value}. The received sampling params: {received}"
|
||||
)
|
||||
if expected_lora:
|
||||
assert received.lora_request.lora_name == expected_lora["name"], (
|
||||
f"Expected lora name={(expected_lora['name'])}, got {received.lora_request.lora_name}. The received sampling params: {received}"
|
||||
)
|
||||
assert received.lora_request.lora_int_id == expected_lora["int_id"], (
|
||||
f"Expected lora int_id={expected_lora['int_id']}, got {received.lora_request.lora_int_id}. The received sampling params: {received}"
|
||||
)
|
||||
assert received.lora_request.lora_path == expected_lora["local_path"], (
|
||||
f"Expected lora path={expected_lora['local_path']}, got {received.lora_request.lora_path}. The received sampling params: {received}"
|
||||
)
|
||||
assert received.lora_scale == expected_lora["scale"], (
|
||||
f"Expected lora scale={expected_lora['scale']}, got {received.lora_scale}. The received sampling params: {received}"
|
||||
)
|
||||
|
||||
|
||||
def _assert_model_param_values(received: OmniSamplingParams, expected: dict):
|
||||
for key, expected_value in expected.items():
|
||||
try:
|
||||
actual_value = getattr(received, key)
|
||||
expected_param_name = key
|
||||
except AttributeError:
|
||||
actual_value = received.extra_args.get(key, None)
|
||||
expected_param_name = f'extra_args["{key}"]'
|
||||
assert actual_value == expected_value, (
|
||||
f"Expected model param '{expected_param_name}'={expected_value}, got {actual_value}. The received sampling params: {received}"
|
||||
)
|
||||
|
||||
|
||||
def _build_mock_outputs(outputs: Iterable[OmniRequestOutput], sampling_case: SamplingCase, server_case: ServerCase):
|
||||
@@ -235,6 +287,7 @@ def _build_mock_outputs(outputs: Iterable[OmniRequestOutput], sampling_case: Sam
|
||||
"height": IMAGE_HEIGHT,
|
||||
**expected,
|
||||
},
|
||||
LORA_PARAMS,
|
||||
)
|
||||
elif sampling_case.kind is SamplingKind.UNDERSTANDING_NONE:
|
||||
assert len(received_sampling_params_list) == 3
|
||||
@@ -244,6 +297,33 @@ def _build_mock_outputs(outputs: Iterable[OmniRequestOutput], sampling_case: Sam
|
||||
_assert_sampling_param_values(received_sampling_params_list[i], expected)
|
||||
elif sampling_case.kind in {SamplingKind.TTS_NONE, SamplingKind.TTS_DIFFUSION_SINGLE}:
|
||||
assert len(received_sampling_params_list) == 1
|
||||
elif sampling_case.kind is SamplingKind.VIDEO_NONE:
|
||||
assert len(received_sampling_params_list) == 1
|
||||
_assert_sampling_param_values(
|
||||
received_sampling_params_list[0],
|
||||
{
|
||||
"width": VIDEO_WIDTH,
|
||||
"height": VIDEO_HEIGHT,
|
||||
"num_frames": VIDEO_NUM_FRAMES,
|
||||
"fps": VIDEO_FPS,
|
||||
},
|
||||
)
|
||||
elif sampling_case.kind is SamplingKind.VIDEO_DIFFUSION_SINGLE:
|
||||
assert len(received_sampling_params_list) == 1
|
||||
expected = DIFFUSION_SINGLE_SAMPLING_PARAMS.copy()
|
||||
expected["num_outputs_per_prompt"] = expected.pop("n") # convert from n to num_outputs_per_prompt
|
||||
_assert_sampling_param_values(
|
||||
received_sampling_params_list[0],
|
||||
{
|
||||
"width": VIDEO_WIDTH,
|
||||
"height": VIDEO_HEIGHT,
|
||||
"num_frames": VIDEO_NUM_FRAMES,
|
||||
"fps": VIDEO_FPS,
|
||||
**expected,
|
||||
},
|
||||
LORA_PARAMS,
|
||||
)
|
||||
_assert_model_param_values(received_sampling_params_list[0], VIDEO_MODEL_PARAMS)
|
||||
else:
|
||||
raise AssertionError(f"Unknown sampling case: {sampling_case.kind}")
|
||||
|
||||
@@ -401,9 +481,15 @@ def api_server(unused_tcp_port_factory, server_case: ServerCase, mock_async_omni
|
||||
@pytest.mark.parametrize(
|
||||
"sampling_case",
|
||||
[
|
||||
pytest.param(SamplingCase(kind=SamplingKind.IMAGE_NONE, sampling_params=None), id="no-sampling-params"),
|
||||
pytest.param(
|
||||
SamplingCase(kind=SamplingKind.IMAGE_DIFFUSION_SINGLE, sampling_params=DIFFUSION_SINGLE_SAMPLING_PARAMS),
|
||||
SamplingCase(kind=SamplingKind.IMAGE_NONE, sampling_params=None, lora=None), id="no-sampling-params"
|
||||
),
|
||||
pytest.param(
|
||||
SamplingCase(
|
||||
kind=SamplingKind.IMAGE_DIFFUSION_SINGLE,
|
||||
sampling_params=DIFFUSION_SINGLE_SAMPLING_PARAMS,
|
||||
lora=LORA_PARAMS,
|
||||
),
|
||||
id="single-diffusion-sampling-params",
|
||||
),
|
||||
],
|
||||
@@ -423,7 +509,8 @@ async def test_image_generation_node(api_server: str, model: str, image_input: b
|
||||
kwargs["image"] = torch.zeros((1, IMAGE_WIDTH, IMAGE_HEIGHT, 3), dtype=torch.float32)
|
||||
if sampling_case.sampling_params is not None:
|
||||
kwargs["sampling_params"] = sampling_case.sampling_params
|
||||
print(f"!!!!!! Calling {model} node.generate with kwargs: {sampling_case.sampling_params}")
|
||||
if sampling_case.lora:
|
||||
kwargs["lora"] = sampling_case.lora
|
||||
|
||||
result = await node.generate(**kwargs)
|
||||
|
||||
@@ -565,3 +652,75 @@ async def test_tts_nodes(api_server: str, node_cls, call_kwargs: dict, sampling_
|
||||
assert result[0]["sample_rate"] == 24000
|
||||
assert isinstance(result[0]["waveform"], torch.Tensor)
|
||||
assert result[0]["waveform"].shape == (1, 1, 24000)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize(
|
||||
"server_case,model,image_input",
|
||||
[
|
||||
pytest.param(
|
||||
ServerCase(
|
||||
served_model="Wan-AI/Wan2.2-T2V-A14B-Diffusers",
|
||||
stage_list=["diffusion"],
|
||||
stage_configs=[{"stage_type": "diffusion"}],
|
||||
outputs=[_build_diffusion_video_output()],
|
||||
),
|
||||
"Wan-AI/Wan2.2-T2V-A14B-Diffusers",
|
||||
False,
|
||||
id="text-to-video",
|
||||
),
|
||||
pytest.param(
|
||||
ServerCase(
|
||||
served_model="Wan-AI/Wan2.2-I2V-A14B-Diffusers",
|
||||
stage_list=["diffusion"],
|
||||
stage_configs=[{"stage_type": "diffusion"}],
|
||||
outputs=[_build_diffusion_video_output()],
|
||||
),
|
||||
"Wan-AI/Wan2.2-I2V-A14B-Diffusers",
|
||||
True,
|
||||
id="image-to-video",
|
||||
),
|
||||
],
|
||||
indirect=["server_case"],
|
||||
)
|
||||
@pytest.mark.parametrize(
|
||||
"sampling_case",
|
||||
[
|
||||
pytest.param(SamplingCase(kind=SamplingKind.VIDEO_NONE, sampling_params=None), id="no-sampling-params"),
|
||||
pytest.param(
|
||||
SamplingCase(
|
||||
kind=SamplingKind.VIDEO_DIFFUSION_SINGLE,
|
||||
sampling_params=DIFFUSION_SINGLE_SAMPLING_PARAMS,
|
||||
lora=LORA_PARAMS,
|
||||
),
|
||||
id="single-diffusion-sampling-params",
|
||||
),
|
||||
],
|
||||
indirect=["sampling_case"],
|
||||
)
|
||||
async def test_video_generation_node(api_server: str, model: str, image_input: bool, sampling_case: SamplingCase):
|
||||
node = VLLMOmniGenerateVideo()
|
||||
|
||||
kwargs = {
|
||||
"url": api_server,
|
||||
"model": model,
|
||||
"prompt": "A beautiful sunset timelapse",
|
||||
"negative_prompt": "",
|
||||
"width": VIDEO_WIDTH,
|
||||
"height": VIDEO_HEIGHT,
|
||||
"fps": VIDEO_FPS,
|
||||
"num_frames": VIDEO_NUM_FRAMES,
|
||||
"model_params": VIDEO_MODEL_PARAMS,
|
||||
}
|
||||
if image_input:
|
||||
kwargs["image"] = torch.zeros((1, VIDEO_HEIGHT, VIDEO_WIDTH, 3), dtype=torch.float32)
|
||||
if sampling_case.sampling_params is not None:
|
||||
kwargs["sampling_params"] = sampling_case.sampling_params
|
||||
if sampling_case.lora:
|
||||
kwargs["lora"] = sampling_case.lora
|
||||
|
||||
result = await node.generate(**kwargs)
|
||||
|
||||
assert isinstance(result, tuple)
|
||||
assert len(result) == 1
|
||||
assert isinstance(result[0], VideoInput)
|
||||
@@ -18,16 +18,21 @@ import subprocess
|
||||
import sys
|
||||
import threading
|
||||
import time
|
||||
from collections.abc import Generator
|
||||
from dataclasses import dataclass
|
||||
from io import BytesIO
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
from typing import Any, NamedTuple
|
||||
|
||||
import imageio.v3 as iio
|
||||
import numpy as np
|
||||
import psutil
|
||||
import pytest
|
||||
import soundfile as sf
|
||||
import torch
|
||||
import yaml
|
||||
from openai import OpenAI
|
||||
from openai import OpenAI, omit
|
||||
from PIL import Image
|
||||
from vllm import TextPrompt
|
||||
from vllm.distributed.parallel_state import cleanup_dist_env_and_memory
|
||||
from vllm.logger import init_logger
|
||||
@@ -44,6 +49,65 @@ PromptImageInput = list[Any] | Any | None
|
||||
PromptVideoInput = list[Any] | Any | None
|
||||
|
||||
|
||||
class OmniServerParams(NamedTuple):
|
||||
model: str
|
||||
port: int | None = None
|
||||
stage_config_path: str | None = None
|
||||
server_args: list[str] | None = None
|
||||
|
||||
|
||||
def assert_image_valid(image: Path | Image.Image, *, width: int | None = None, height: int | None = None):
|
||||
"""Assert the file is a loadable image with optional exact dimensions."""
|
||||
if isinstance(image, Path):
|
||||
assert image.exists(), f"Image not found: {image}"
|
||||
image = Image.open(image)
|
||||
image.load()
|
||||
assert image.width > 0 and image.height > 0
|
||||
if width is not None:
|
||||
assert image.width == width, f"Expected width={width}, got {image.width} in {image.name}"
|
||||
if height is not None:
|
||||
assert image.height == height, f"Expected height={height}, got {image.height} in {image.name}"
|
||||
return image
|
||||
|
||||
|
||||
def assert_video_valid(frames: Path | np.ndarray, *, width: int, height: int, num_frames: int) -> None:
|
||||
"""Assert the MP4 has the expected resolution and exact frame count."""
|
||||
if isinstance(frames, Path):
|
||||
assert frames.exists(), f"Video not found: {frames}"
|
||||
frames = iio.imread(str(frames), plugin="pyav", index=None)
|
||||
assert frames.shape[0] == num_frames, f"Expected {num_frames} frames, got {frames.shape[0]}"
|
||||
assert frames.shape[1] == height, f"Expected height={height}, got {frames.shape[1]}"
|
||||
assert frames.shape[2] == width, f"Expected width={width}, got {frames.shape[2]}"
|
||||
|
||||
|
||||
def assert_audio_valid(path: Path, *, sample_rate: int, channels: int, duration_s: float) -> None:
|
||||
"""Assert the WAV has the expected sample rate, channel count, and duration."""
|
||||
|
||||
assert path.exists(), f"Audio not found: {path}"
|
||||
info = sf.info(str(path))
|
||||
assert info.samplerate == sample_rate, f"Expected sample_rate={sample_rate}, got {info.samplerate}"
|
||||
assert info.channels == channels, f"Expected {channels} channel(s), got {info.channels}"
|
||||
expected_frames = int(duration_s * sample_rate)
|
||||
assert info.frames == expected_frames, (
|
||||
f"Expected {expected_frames} frames ({duration_s}s @ {sample_rate} Hz), got {info.frames}"
|
||||
)
|
||||
|
||||
|
||||
def decode_b64_image(b64: str):
|
||||
img = Image.open(BytesIO(base64.b64decode(b64)))
|
||||
img.load()
|
||||
return img
|
||||
|
||||
|
||||
@pytest.fixture(scope="session")
|
||||
def model_prefix() -> str:
|
||||
"""Optional model-path prefix from MODEL_PREFIX env var.
|
||||
Useful if models are downloaded to non-default local directories.
|
||||
"""
|
||||
prefix = os.environ.get("MODEL_PREFIX", "")
|
||||
return f"{prefix.rstrip('/')}/" if prefix else ""
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def default_vllm_config():
|
||||
"""Set a default VllmConfig for all tests.
|
||||
@@ -1151,7 +1215,9 @@ def pytest_addoption(parser):
|
||||
|
||||
|
||||
@pytest.fixture(scope="session")
|
||||
def run_level(request):
|
||||
def run_level(request) -> str:
|
||||
"""A command-line argument that specifies the level of tests to run in this session.
|
||||
See https://docs.vllm.ai/projects/vllm-omni/en/latest/contributing/ci/CI_5levels/"""
|
||||
return request.config.getoption("--run-level")
|
||||
|
||||
|
||||
@@ -1159,15 +1225,17 @@ _omni_server_lock = threading.Lock()
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def omni_server(request, run_level):
|
||||
def omni_server(request: pytest.FixtureRequest, run_level: str, model_prefix: str) -> Generator[OmniServer, Any, None]:
|
||||
"""Start vLLM-Omni server as a subprocess with actual model weights.
|
||||
Uses session scope so the server starts only once for the entire test session.
|
||||
Multi-stage initialization can take 10-20+ minutes.
|
||||
"""
|
||||
with _omni_server_lock:
|
||||
*port, model, stage_config_path = request.param
|
||||
port = port[0] if port else None
|
||||
if run_level == "advanced_model":
|
||||
params: OmniServerParams = request.param
|
||||
model = model_prefix + params.model
|
||||
port = params.port
|
||||
stage_config_path = params.stage_config_path
|
||||
if run_level == "advanced_model" and stage_config_path is not None:
|
||||
stage_config_path = modify_stage_config(
|
||||
stage_config_path,
|
||||
deletes={
|
||||
@@ -1179,7 +1247,11 @@ def omni_server(request, run_level):
|
||||
},
|
||||
)
|
||||
|
||||
server_args = ["--stage-configs-path", stage_config_path, "--stage-init-timeout", "120"]
|
||||
server_args = params.server_args or []
|
||||
server_args = ["--stage-init-timeout", "120", *server_args]
|
||||
if stage_config_path is not None:
|
||||
server_args += ["--stage-configs-path", stage_config_path]
|
||||
|
||||
with OmniServer(model, server_args, port=port) if port else OmniServer(model, server_args) as server:
|
||||
print("OmniServer started successfully")
|
||||
yield server
|
||||
@@ -1199,6 +1271,17 @@ class OmniResponse:
|
||||
error_message: str | None = None
|
||||
|
||||
|
||||
@dataclass
|
||||
class DiffusionResponse:
|
||||
text_content: str | None = None
|
||||
images: list[Image.Image] | None = None
|
||||
audios: list[Any] | None = None
|
||||
videos: list[Any] | None = None
|
||||
e2e_latency: float | None = None
|
||||
success: bool = False
|
||||
error_message: str | None = None
|
||||
|
||||
|
||||
def assert_omni_response(response: OmniResponse, request_config: dict[str, Any], run_level):
|
||||
"""
|
||||
Validate response results.
|
||||
@@ -1247,6 +1330,51 @@ def assert_omni_response(response: OmniResponse, request_config: dict[str, Any],
|
||||
print(f"similarity is: {response.similarity}")
|
||||
|
||||
|
||||
def assert_diffusion_response(response: DiffusionResponse, request_config: dict[str, Any], run_level: str = None):
|
||||
"""
|
||||
Validate diffusion response results.
|
||||
|
||||
Args:
|
||||
response: DiffusionResponse object. Any not-None content will be validated based on the request_config.
|
||||
request_config: Request configuration dictionary containing parameters like model, messages, extra_body.
|
||||
When validating a certain modality, the corresponding params in request_config['extra_body'] must present.
|
||||
It will be used to check against the multimedia file in the response.
|
||||
run_level: Test run level (e.g., "core_model", "advanced_model")
|
||||
|
||||
Raises:
|
||||
AssertionError: When the response does not meet validation criteria
|
||||
KeyError: When the request_config does not contain necessary parameters for validation
|
||||
"""
|
||||
assert response.success, "The request failed."
|
||||
|
||||
e2e_latency = response.e2e_latency
|
||||
if e2e_latency is not None:
|
||||
print(f"the avg e2e is: {e2e_latency}")
|
||||
|
||||
extra_body = request_config.get("extra_body", {})
|
||||
|
||||
num_outputs_per_prompt = extra_body.get("num_outputs_per_prompt", 1)
|
||||
|
||||
if response.images is not None:
|
||||
assert len(response.images) > 0, "No images in response"
|
||||
assert len(response.images) == num_outputs_per_prompt, (
|
||||
f"Expected {num_outputs_per_prompt} images, got {len(response.images)}"
|
||||
)
|
||||
if run_level == "advanced_model":
|
||||
expected_width = extra_body["width"] # intend to raise KeyError
|
||||
expected_height = extra_body["height"] # intend to raise KeyError
|
||||
for img in response.images:
|
||||
assert_image_valid(img, width=expected_width, height=expected_height)
|
||||
if response.videos is not None:
|
||||
raise NotImplementedError(
|
||||
"Video validation is not implemented yet"
|
||||
) # consider using assert_video_valid defined above
|
||||
if response.audios is not None:
|
||||
raise NotImplementedError(
|
||||
"Audio validation is not implemented yet"
|
||||
) # consider using assert_audio_valid defined above
|
||||
|
||||
|
||||
class OpenAIClientHandler:
|
||||
"""
|
||||
OpenAI client handler class, encapsulating both streaming and non-streaming response processing logic.
|
||||
@@ -1269,7 +1397,7 @@ class OpenAIClientHandler:
|
||||
self.client = OpenAI(base_url=f"http://{host}:{port}/v1", api_key=api_key)
|
||||
self.run_level = run_level
|
||||
|
||||
def _process_stream_response(self, chat_completion) -> OmniResponse:
|
||||
def _process_stream_omni_response(self, chat_completion) -> OmniResponse:
|
||||
"""
|
||||
Process streaming responses.
|
||||
|
||||
@@ -1330,7 +1458,7 @@ class OpenAIClientHandler:
|
||||
|
||||
return result
|
||||
|
||||
def _process_non_stream_response(self, chat_completion) -> OmniResponse:
|
||||
def _process_non_stream_omni_response(self, chat_completion) -> OmniResponse:
|
||||
"""
|
||||
Process non-streaming responses.
|
||||
|
||||
@@ -1384,7 +1512,46 @@ class OpenAIClientHandler:
|
||||
|
||||
return result
|
||||
|
||||
def send_request(self, request_config: dict[str, Any], request_num: int = 1) -> list[OmniResponse]:
|
||||
def _process_diffusion_response(self, chat_completion) -> DiffusionResponse:
|
||||
"""
|
||||
Process diffusion responses (image generation/editing).
|
||||
|
||||
Args:
|
||||
chat_completion: OpenAI response object
|
||||
|
||||
Returns:
|
||||
DiffusionResponse: Processed response object
|
||||
"""
|
||||
result = DiffusionResponse()
|
||||
start_time = time.perf_counter()
|
||||
|
||||
try:
|
||||
images = []
|
||||
# [TODO] reading video and audio output from API response for later validation
|
||||
|
||||
for choice in chat_completion.choices:
|
||||
if hasattr(choice.message, "content") and choice.message.content is not None:
|
||||
content = choice.message.content
|
||||
if isinstance(content, list):
|
||||
for item in content:
|
||||
if hasattr(item, "image_url") and item.image_url is not None:
|
||||
image_url = item.image_url.url
|
||||
if image_url.startswith("data:image"):
|
||||
b64_data = image_url.split(",", 1)[1]
|
||||
img = decode_b64_image(b64_data)
|
||||
images.append(img)
|
||||
|
||||
result.e2e_latency = time.perf_counter() - start_time
|
||||
result.images = images if images else None
|
||||
result.success = True
|
||||
|
||||
except Exception as e:
|
||||
result.error_message = f"Diffusion response processing error: {str(e)}"
|
||||
print(f"Error: {result.error_message}")
|
||||
|
||||
return result
|
||||
|
||||
def send_omni_request(self, request_config: dict[str, Any], request_num: int = 1) -> list[OmniResponse]:
|
||||
"""
|
||||
Send OpenAI requests.
|
||||
|
||||
@@ -1410,9 +1577,9 @@ class OpenAIClientHandler:
|
||||
)
|
||||
|
||||
if stream:
|
||||
response = self._process_stream_response(chat_completion)
|
||||
response = self._process_stream_omni_response(chat_completion)
|
||||
else:
|
||||
response = self._process_non_stream_response(chat_completion)
|
||||
response = self._process_non_stream_omni_response(chat_completion)
|
||||
|
||||
assert_omni_response(response, request_config, run_level=self.run_level)
|
||||
responses.append(response)
|
||||
@@ -1438,25 +1605,82 @@ class OpenAIClientHandler:
|
||||
chat_completion = future.result()
|
||||
|
||||
if stream:
|
||||
response = self._process_stream_response(chat_completion)
|
||||
response = self._process_stream_omni_response(chat_completion)
|
||||
else:
|
||||
response = self._process_non_stream_response(chat_completion)
|
||||
response = self._process_non_stream_omni_response(chat_completion)
|
||||
|
||||
assert_omni_response(response, request_config, run_level=self.run_level)
|
||||
responses.append(response)
|
||||
|
||||
return responses
|
||||
|
||||
def send_diffusion_request(self, request_config: dict[str, Any], request_num: int = 1) -> list[OmniResponse]:
|
||||
"""
|
||||
Send OpenAI requests for diffusion models.
|
||||
|
||||
Args:
|
||||
request_config: Request configuration dictionary containing parameters like model, messages
|
||||
request_num: Number of requests to send concurrently, defaults to 1 (single request)
|
||||
Returns:
|
||||
List[OmniResponse]: List of response objects
|
||||
"""
|
||||
responses = []
|
||||
stream = request_config.get("stream", False)
|
||||
modalities = request_config.get("modalities", omit) # Most diffusion models don't require modalities param
|
||||
extra_body = request_config.get("extra_body", None)
|
||||
|
||||
if stream:
|
||||
raise NotImplementedError("Streaming is not currently implemented for diffusion model e2e test")
|
||||
|
||||
if request_num == 1:
|
||||
# Send single request
|
||||
chat_completion = self.client.chat.completions.create(
|
||||
model=request_config.get("model"),
|
||||
messages=request_config.get("messages"),
|
||||
extra_body=extra_body,
|
||||
modalities=modalities,
|
||||
)
|
||||
|
||||
response = self._process_diffusion_response(chat_completion)
|
||||
assert_diffusion_response(response, request_config, run_level=self.run_level)
|
||||
responses.append(response)
|
||||
|
||||
else:
|
||||
# Send concurrent requests
|
||||
with concurrent.futures.ThreadPoolExecutor(max_workers=request_num) as executor:
|
||||
futures = []
|
||||
|
||||
# Submit all request tasks
|
||||
for _ in range(request_num):
|
||||
future = executor.submit(
|
||||
self.client.chat.completions.create,
|
||||
model=request_config.get("model"),
|
||||
messages=request_config.get("messages"),
|
||||
modalities=modalities,
|
||||
extra_body=extra_body,
|
||||
)
|
||||
futures.append(future)
|
||||
|
||||
# Process completed tasks
|
||||
for future in concurrent.futures.as_completed(futures):
|
||||
chat_completion = future.result()
|
||||
response = self._process_diffusion_response(chat_completion)
|
||||
assert_diffusion_response(response, request_config, run_level=self.run_level)
|
||||
responses.append(response)
|
||||
|
||||
return responses
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def openai_client(omni_server, run_level):
|
||||
"""Create OpenAIClientHandler fixture"""
|
||||
def openai_client(omni_server: OmniServer, run_level: str):
|
||||
"""Create OpenAIClientHandler fixture to facilitate communication with OmniServer
|
||||
with encapsulated request sending, concurrent requests, response handling, and validation."""
|
||||
return OpenAIClientHandler(host=omni_server.host, port=omni_server.port, api_key="EMPTY", run_level=run_level)
|
||||
|
||||
|
||||
class OmniRunner:
|
||||
"""
|
||||
Test runner for Omni models.
|
||||
Offline test runner for Omni models.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
@@ -1727,9 +1951,10 @@ class OmniRunner:
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def omni_runner(request):
|
||||
def omni_runner(request, model_prefix):
|
||||
with _omni_server_lock:
|
||||
model, stage_config_path = request.param
|
||||
model = model_prefix + model
|
||||
with OmniRunner(model, seed=42, stage_configs_path=stage_config_path, stage_init_timeout=300) as runner:
|
||||
print("OmniRunner started successfully")
|
||||
yield runner
|
||||
|
||||
@@ -1,5 +1,29 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||
|
||||
"""Tests for Ulysses + Ring sequence-parallel attention correctness.
|
||||
|
||||
What is tested
|
||||
--------------
|
||||
* ``test_sequence_parallel`` verifies that the ``Attention`` layer produces
|
||||
numerically equivalent results whether the sequence is processed on a single
|
||||
rank (baseline, SP=1) or sharded across multiple ranks via Ulysses/Ring SP.
|
||||
The test spawns two separate multi-process runs with ``torch.multiprocessing.spawn``:
|
||||
1. Baseline – world_size=1, ulysses_degree=1, ring_degree=1.
|
||||
2. SP run – world_size=ulysses_degree*ring_degree, each rank holds a
|
||||
contiguous slice of the full sequence.
|
||||
After both runs, rank-0 output tensors are compared element-wise with a
|
||||
tolerance appropriate for the dtype (bfloat16).
|
||||
|
||||
SP-plan hooks are NOT applied in this test
|
||||
------------------------------------------
|
||||
``ForwardContext.sp_plan_hooks_applied`` remains ``False``. As a result,
|
||||
``ForwardContext.sp_active`` falls back to the "no hooks" branch: SP is
|
||||
considered active whenever ``parallel_config.sequence_parallel_size > 1``.
|
||||
This makes the test self-contained and suitable for standalone CI runs that
|
||||
do not exercise the full model-registry pipeline.
|
||||
"""
|
||||
|
||||
import os
|
||||
import pickle
|
||||
import tempfile
|
||||
@@ -27,6 +51,11 @@ def update_environment_variables(envs_dict: dict[str, str]):
|
||||
os.environ[k] = v
|
||||
|
||||
|
||||
def seed_everything(seed: int):
|
||||
torch.manual_seed(seed)
|
||||
current_omni_platform.manual_seed(seed)
|
||||
|
||||
|
||||
class TestAttentionModel(torch.nn.Module):
|
||||
"""Test model using Attention layer."""
|
||||
|
||||
@@ -138,11 +167,10 @@ class TestMultiLayerAttentionModel(torch.nn.Module):
|
||||
@pytest.mark.parametrize("num_heads", [8])
|
||||
@pytest.mark.parametrize("head_size", [8])
|
||||
@pytest.mark.parametrize("causal", [False])
|
||||
@pytest.mark.parametrize("dtype", [torch.float16, torch.bfloat16]) # [torch.float16, torch.bfloat16]
|
||||
@pytest.mark.parametrize("dtype", [torch.bfloat16])
|
||||
@pytest.mark.parametrize("use_sync", [False])
|
||||
@pytest.mark.parametrize("dynamic", [False])
|
||||
@pytest.mark.parametrize("use_compile", [False])
|
||||
@pytest.mark.parametrize("attn_backend", ["sdpa", "flash_attn"])
|
||||
def test_sequence_parallel(
|
||||
ulysses_degree: int,
|
||||
ring_degree: int,
|
||||
@@ -156,7 +184,6 @@ def test_sequence_parallel(
|
||||
seq_len: int,
|
||||
num_heads: int,
|
||||
head_size: int,
|
||||
attn_backend: str,
|
||||
):
|
||||
"""Test Ulysses attention by comparing with and without SP enabled."""
|
||||
sequence_parallel_size = ulysses_degree * ring_degree
|
||||
@@ -200,7 +227,6 @@ def test_sequence_parallel(
|
||||
model_state_file,
|
||||
input_data_file,
|
||||
True, # is_baseline
|
||||
attn_backend,
|
||||
),
|
||||
nprocs=1,
|
||||
)
|
||||
@@ -228,7 +254,6 @@ def test_sequence_parallel(
|
||||
model_state_file,
|
||||
input_data_file,
|
||||
False, # is_baseline
|
||||
attn_backend,
|
||||
),
|
||||
nprocs=sequence_parallel_size,
|
||||
)
|
||||
@@ -324,12 +349,11 @@ def ulysses_attention_on_test_model(
|
||||
model_state_file: str,
|
||||
input_data_file: str,
|
||||
is_baseline: bool,
|
||||
attn_backend: str,
|
||||
):
|
||||
"""Run Ulysses attention test on a test model and save results for comparison."""
|
||||
# Use fixed seed for reproducibility across baseline and SP runs
|
||||
RANDOM_SEED = 42
|
||||
current_omni_platform.seed_everything(RANDOM_SEED)
|
||||
seed_everything(RANDOM_SEED)
|
||||
|
||||
mode_str = "Baseline (no SP)" if is_baseline else f"SP (ulysses={ulysses_degree}, ring={ring_degree})"
|
||||
print(f"\n[{mode_str}] Rank {local_rank}/{world_size} - Random seed set to {RANDOM_SEED}")
|
||||
@@ -366,7 +390,6 @@ def ulysses_attention_on_test_model(
|
||||
model="test_model",
|
||||
dtype=dtype,
|
||||
parallel_config=parallel_config,
|
||||
attention_backend=attn_backend, # Set the attention backend here
|
||||
)
|
||||
|
||||
# Initialize model parallel
|
||||
@@ -411,11 +434,6 @@ def ulysses_attention_on_test_model(
|
||||
with open(model_state_file, "wb") as f:
|
||||
pickle.dump(model_state, f)
|
||||
|
||||
# Generate and save full input data with fixed seed
|
||||
# Reinitialize RNG to ensure reproducibility
|
||||
torch.manual_seed(42)
|
||||
if torch.cuda.is_available():
|
||||
torch.cuda.manual_seed_all(42)
|
||||
full_hidden_states = torch.randn(
|
||||
(batch_size, seq_len, hidden_size),
|
||||
dtype=dtype,
|
||||
|
||||
@@ -0,0 +1,244 @@
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
import pytest
|
||||
import torch
|
||||
|
||||
from vllm_omni.diffusion.models.flux2.flux2_transformer import (
|
||||
Flux2PosEmbed,
|
||||
Flux2Transformer2DModel,
|
||||
)
|
||||
|
||||
|
||||
# Initialize TP group before tests
|
||||
@pytest.fixture(scope="function", autouse=True)
|
||||
def setup_tp_group():
|
||||
"""Set up TP group for each test function"""
|
||||
with patch("vllm.model_executor.layers.linear.get_tensor_model_parallel_world_size", return_value=2):
|
||||
with patch("vllm.distributed.parallel_state.get_tp_group") as mock_get_tp_group:
|
||||
mock_tp_group = MagicMock()
|
||||
mock_tp_group.world_size = 2
|
||||
mock_get_tp_group.return_value = mock_tp_group
|
||||
yield
|
||||
|
||||
|
||||
class TestFlux2TransformerWeightLoading:
|
||||
"""Test Flux2Transformer weight loading functionality"""
|
||||
|
||||
def test_weight_loading_tp2(self, setup_tp_group):
|
||||
"""Verify weights load correctly with TP=2"""
|
||||
# Prepare test data
|
||||
model = Flux2Transformer2DModel(
|
||||
num_layers=2,
|
||||
num_single_layers=1,
|
||||
num_attention_heads=48,
|
||||
attention_head_dim=128,
|
||||
joint_attention_dim=15360,
|
||||
)
|
||||
|
||||
# Mock TP=2 weight loading
|
||||
mock_weights = []
|
||||
|
||||
# 1. Test regular weight loading
|
||||
mock_weights.append(("x_embedder.weight", torch.randn(6144, 128)))
|
||||
mock_weights.append(("context_embedder.weight", torch.randn(6144, 15360)))
|
||||
mock_weights.append(("proj_out.weight", torch.randn(128, 6144)))
|
||||
|
||||
# 2. Test stacked_params_mapping weight loading
|
||||
# Full weights - load_weights handles sharding internally
|
||||
to_qkv_weight = torch.randn(18432, 6144) # Full weights
|
||||
mock_weights.append(("transformer_blocks.0.attn.to_qkv.weight", to_qkv_weight))
|
||||
|
||||
# Add_kv_proj weights
|
||||
add_kv_proj_weight = torch.randn(18432, 6144) # Full weights
|
||||
mock_weights.append(("transformer_blocks.0.attn.add_kv_proj.weight", add_kv_proj_weight))
|
||||
|
||||
# 3. Test single block weight loading
|
||||
single_block_qkv_mlp_weight = torch.randn(18432 + 18432 * 3, 6144) # Full weights
|
||||
mock_weights.append(("single_transformer_blocks.0.attn.to_qkv_mlp_proj.weight", single_block_qkv_mlp_weight))
|
||||
|
||||
# Execute weight loading
|
||||
loaded_params = model.load_weights(mock_weights)
|
||||
|
||||
# Verify
|
||||
assert len(loaded_params) > 0, "Parameters should be loaded"
|
||||
|
||||
# Verify stacked_params_mapping is correctly set
|
||||
assert model.stacked_params_mapping is not None
|
||||
# Should have 6 mappings: 3 for to_qkv + 3 for add_kv_proj
|
||||
assert len(model.stacked_params_mapping) == 6, "Should have 6 mappings"
|
||||
|
||||
# Verify weights are correctly loaded to corresponding layers
|
||||
assert hasattr(model.transformer_blocks[0].attn.to_qkv, "weight")
|
||||
# With TP=2, weight dimension on each GPU should be 18432/2 = 9216
|
||||
assert model.transformer_blocks[0].attn.to_qkv.weight.shape == (9216, 6144), (
|
||||
f"With TP=2, to_qkv weight dimension should be (9216, 6144), got {model.transformer_blocks[0].attn.to_qkv.weight.shape}"
|
||||
)
|
||||
|
||||
|
||||
class TestFlux2RopePositionEmbedding:
|
||||
"""Test Flux2 RoPE position embedding functionality"""
|
||||
|
||||
def test_rope_position_embedding(self):
|
||||
"""Verify RoPE produces correct embeddings for 4D coordinates"""
|
||||
# Prepare test data - use model default configuration
|
||||
# axes_dims_rope default is (32, 32, 32, 32)
|
||||
# get_1d_rotary_pos_embed outputs half the dimension for real/imag parts
|
||||
# So actual output dimension should be (16+16+16+16) = 64
|
||||
axes_dims = (32, 32, 32, 32) # Use model default
|
||||
rope_theta = 2000 # Model default is 2000
|
||||
pos_embed = Flux2PosEmbed(theta=rope_theta, axes_dim=axes_dims)
|
||||
|
||||
# Create test IDs
|
||||
seq_len = 10
|
||||
ids = torch.randint(0, 100, (seq_len, 4)) # [S, 4]
|
||||
|
||||
# Forward pass
|
||||
freqs_cos, freqs_sin = pos_embed(ids)
|
||||
|
||||
# Verify output shape - based on model config, expected dimension is 64
|
||||
# Each axes_dim=32 outputs 16-dim real part, sum of 4 dimensions = 64
|
||||
expected_dim = sum(axes_dims) // 2 # 128/2 = 64
|
||||
assert freqs_cos.shape == (seq_len, expected_dim), (
|
||||
f"Expected shape {(seq_len, expected_dim)}, got {freqs_cos.shape}"
|
||||
)
|
||||
assert freqs_sin.shape == (seq_len, expected_dim), (
|
||||
f"Expected shape {(seq_len, expected_dim)}, got {freqs_sin.shape}"
|
||||
)
|
||||
|
||||
# Verify output type - NPU may use float32, other devices use float64
|
||||
assert freqs_cos.dtype in [torch.float32, torch.float64], "Should be float type"
|
||||
assert freqs_sin.dtype in [torch.float32, torch.float64], "Should be float type"
|
||||
|
||||
# Verify value range
|
||||
assert torch.all(freqs_cos >= -1) and torch.all(freqs_cos <= 1), "cos values should be in [-1, 1]"
|
||||
assert torch.all(freqs_sin >= -1) and torch.all(freqs_sin <= 1), "sin values should be in [-1, 1]"
|
||||
|
||||
# Verify trigonometric relationship: cos^2 + sin^2 ≈ 1
|
||||
cos_sq_sin_sq = freqs_cos**2 + freqs_sin**2
|
||||
assert torch.allclose(cos_sq_sin_sq, torch.ones_like(cos_sq_sin_sq), atol=1e-6), "cos^2 + sin^2 should ≈ 1"
|
||||
|
||||
# Verify different positions produce different embeddings
|
||||
ids_diff = torch.randint(0, 100, (seq_len, 4))
|
||||
freqs_cos_diff, freqs_sin_diff = pos_embed(ids_diff)
|
||||
assert not torch.allclose(freqs_cos, freqs_cos_diff), "Different positions should produce different embeddings"
|
||||
|
||||
# Verify same positions produce same embeddings
|
||||
ids_same = ids.clone()
|
||||
freqs_cos_same, freqs_sin_same = pos_embed(ids_same)
|
||||
assert torch.allclose(freqs_cos, freqs_cos_same), "Same positions should produce same embeddings"
|
||||
assert torch.allclose(freqs_sin, freqs_sin_same), "Same positions should produce same embeddings"
|
||||
|
||||
|
||||
class TestFlux2PackedModuleMapping:
|
||||
"""Test Flux2 packed module mapping functionality"""
|
||||
|
||||
def test_packed_module_mapping(self, setup_tp_group):
|
||||
"""Verify to_qkv packing matches HF checkpoint"""
|
||||
model = Flux2Transformer2DModel(
|
||||
num_layers=1,
|
||||
num_single_layers=0,
|
||||
num_attention_heads=48,
|
||||
attention_head_dim=128,
|
||||
joint_attention_dim=15360,
|
||||
)
|
||||
|
||||
# Verify stacked_params_mapping is correctly initialized
|
||||
model.load_weights([]) # Trigger stacked_params_mapping initialization
|
||||
assert model.stacked_params_mapping is not None
|
||||
|
||||
# Verify mapping configuration
|
||||
expected_mappings = [
|
||||
(".to_qkv.", ".to_q.", "q"),
|
||||
(".to_qkv.", ".to_k.", "k"),
|
||||
(".to_qkv.", ".to_v.", "v"),
|
||||
(".add_kv_proj", ".add_q_proj", "q"),
|
||||
(".add_kv_proj", ".add_k_proj", "k"),
|
||||
(".add_kv_proj", ".add_v_proj", "v"),
|
||||
]
|
||||
assert model.stacked_params_mapping == expected_mappings
|
||||
|
||||
# Create mock HF checkpoint weights
|
||||
hf_weights = []
|
||||
|
||||
# Mock HF separated Q/K/V weights
|
||||
attn_block = model.transformer_blocks[0].attn
|
||||
head_dim = 128
|
||||
num_heads = 48
|
||||
hidden_size = 6144
|
||||
# Full weight dimension
|
||||
full_shard_size = num_heads * head_dim # 6144
|
||||
|
||||
# Q projection weights (full weights)
|
||||
q_weight = torch.randn(full_shard_size, hidden_size)
|
||||
hf_weights.append(("transformer_blocks.0.attn.to_q.weight", q_weight))
|
||||
|
||||
# K projection weights (full weights)
|
||||
k_weight = torch.randn(full_shard_size, hidden_size)
|
||||
hf_weights.append(("transformer_blocks.0.attn.to_k.weight", k_weight))
|
||||
|
||||
# V projection weights (full weights)
|
||||
v_weight = torch.randn(full_shard_size, hidden_size)
|
||||
hf_weights.append(("transformer_blocks.0.attn.to_v.weight", v_weight))
|
||||
|
||||
# Mock HF separated add_q/k/v projection weights
|
||||
add_q_weight = torch.randn(full_shard_size, hidden_size)
|
||||
hf_weights.append(("transformer_blocks.0.attn.add_q_proj.weight", add_q_weight))
|
||||
|
||||
add_k_weight = torch.randn(full_shard_size, hidden_size)
|
||||
hf_weights.append(("transformer_blocks.0.attn.add_k_proj.weight", add_k_weight))
|
||||
|
||||
add_v_weight = torch.randn(full_shard_size, hidden_size)
|
||||
hf_weights.append(("transformer_blocks.0.attn.add_v_proj.weight", add_v_weight))
|
||||
|
||||
# Execute weight loading
|
||||
loaded_params = model.load_weights(hf_weights)
|
||||
|
||||
# Verify weights are loaded
|
||||
assert len(loaded_params) > 0
|
||||
|
||||
# Verify final QKV weights are correctly combined (considering TP sharding)
|
||||
# With TP=2, dimension on each GPU should be half of full dimension
|
||||
expected_qkv_shape = (full_shard_size * 3 // 2, hidden_size) # 18432/2 = 9216
|
||||
assert attn_block.to_qkv.weight.shape == expected_qkv_shape, (
|
||||
f"to_qkv weight dimension should be {expected_qkv_shape}, got {attn_block.to_qkv.weight.shape}"
|
||||
)
|
||||
|
||||
expected_add_kv_shape = (full_shard_size * 3 // 2, hidden_size)
|
||||
assert attn_block.add_kv_proj.weight.shape == expected_add_kv_shape, (
|
||||
f"add_kv_proj weight dimension should be {expected_add_kv_shape}, got {attn_block.add_kv_proj.weight.shape}"
|
||||
)
|
||||
|
||||
def test_packed_mapping_edge_cases(self, setup_tp_group):
|
||||
"""Test edge cases for packed mapping"""
|
||||
model = Flux2Transformer2DModel(
|
||||
num_layers=1,
|
||||
num_single_layers=1,
|
||||
num_attention_heads=48,
|
||||
attention_head_dim=128,
|
||||
joint_attention_dim=15360,
|
||||
)
|
||||
model.load_weights([])
|
||||
|
||||
# Test invalid weight names
|
||||
invalid_weights = [("invalid.weight", torch.randn(10, 10))]
|
||||
loaded_params = model.load_weights(invalid_weights)
|
||||
assert len(loaded_params) == 0, "Should not load invalid weights"
|
||||
|
||||
# Test to_out weight renaming
|
||||
to_out_weight = torch.randn(6144, 6144)
|
||||
weights = [("transformer_blocks.0.attn.to_out.0.weight", to_out_weight)]
|
||||
loaded_params = model.load_weights(weights)
|
||||
|
||||
# Check if to_out related weights are loaded
|
||||
to_out_loaded = any("to_out" in p for p in loaded_params)
|
||||
assert to_out_loaded, "to_out weights should be correctly renamed and loaded"
|
||||
|
||||
# Test partial weight loading
|
||||
partial_weights = [
|
||||
("x_embedder.weight", torch.randn(6144, 128)),
|
||||
("transformer_blocks.0.attn.to_q.weight", torch.randn(6144, 6144)), # Full weights
|
||||
]
|
||||
loaded_params = model.load_weights(partial_weights)
|
||||
assert len(loaded_params) == 2, "Should load two weights"
|
||||
assert "x_embedder.weight" in loaded_params
|
||||
assert any("to_qkv" in p for p in loaded_params), "to_q should be mapped to to_qkv"
|
||||
@@ -0,0 +1,58 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||
|
||||
from time import time
|
||||
|
||||
from vllm_omni.distributed.omni_coordinator import (
|
||||
InstanceInfo,
|
||||
RandomBalancer,
|
||||
StageStatus,
|
||||
)
|
||||
|
||||
|
||||
def test_load_balancer_select_returns_valid_index():
|
||||
"""Verify RandomBalancer.select() returns a valid index for instances."""
|
||||
# Task structure mirrors async_omni; RandomBalancer ignores task contents.
|
||||
task: dict = {
|
||||
"request_id": "test",
|
||||
"engine_inputs": None,
|
||||
"sampling_params": None,
|
||||
}
|
||||
|
||||
now = time()
|
||||
instances = [
|
||||
InstanceInfo(
|
||||
input_addr="tcp://host:10001",
|
||||
output_addr="tcp://host:10001-out",
|
||||
stage_id=0,
|
||||
status=StageStatus.UP,
|
||||
queue_length=0,
|
||||
last_heartbeat=now,
|
||||
registered_at=now,
|
||||
),
|
||||
InstanceInfo(
|
||||
input_addr="tcp://host:10002",
|
||||
output_addr="tcp://host:10002-out",
|
||||
stage_id=0,
|
||||
status=StageStatus.UP,
|
||||
queue_length=1,
|
||||
last_heartbeat=now,
|
||||
registered_at=now,
|
||||
),
|
||||
InstanceInfo(
|
||||
input_addr="tcp://host:10003",
|
||||
output_addr="tcp://host:10003-out",
|
||||
stage_id=1,
|
||||
status=StageStatus.UP,
|
||||
queue_length=2,
|
||||
last_heartbeat=now,
|
||||
registered_at=now,
|
||||
),
|
||||
]
|
||||
|
||||
balancer = RandomBalancer()
|
||||
|
||||
index = balancer.select(task, instances)
|
||||
|
||||
assert isinstance(index, int)
|
||||
assert 0 <= index < len(instances)
|
||||
@@ -0,0 +1,119 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||
|
||||
import json
|
||||
import time
|
||||
|
||||
import pytest
|
||||
import zmq
|
||||
|
||||
from vllm_omni.distributed.omni_coordinator import (
|
||||
InstanceList,
|
||||
OmniCoordClientForHub,
|
||||
)
|
||||
|
||||
|
||||
def _bind_pub() -> tuple[zmq.Context, zmq.Socket, str]:
|
||||
ctx = zmq.Context.instance()
|
||||
pub = ctx.socket(zmq.PUB)
|
||||
pub.bind("tcp://127.0.0.1:*")
|
||||
endpoint = pub.getsockopt(zmq.LAST_ENDPOINT).decode("ascii")
|
||||
return ctx, pub, endpoint
|
||||
|
||||
|
||||
def _wait_for_condition(cond, timeout: float = 2.0, interval: float = 0.01) -> bool:
|
||||
start = time.time()
|
||||
while time.time() - start < timeout:
|
||||
if cond():
|
||||
return True
|
||||
time.sleep(interval)
|
||||
return False
|
||||
|
||||
|
||||
def test_hub_client_caches_instance_list_from_pub():
|
||||
"""Verify OmniCoordClientForHub receives instance list updates from OmniCoordinator and caches for get_instance_list()."""
|
||||
ctx, pub, endpoint = _bind_pub()
|
||||
|
||||
client = OmniCoordClientForHub(endpoint)
|
||||
# ZMQ PUB/SUB slow-joiner: allow SUB to finish connecting before first send
|
||||
time.sleep(0.2)
|
||||
|
||||
now = time.time()
|
||||
instances_payload = [
|
||||
{
|
||||
"input_addr": "tcp://stage:10001",
|
||||
"output_addr": "tcp://stage:10001-out",
|
||||
"stage_id": 0,
|
||||
"status": "up",
|
||||
"queue_length": 0,
|
||||
"last_heartbeat": now,
|
||||
"registered_at": now,
|
||||
},
|
||||
{
|
||||
"input_addr": "tcp://stage:10002",
|
||||
"output_addr": "tcp://stage:10002-out",
|
||||
"stage_id": 0,
|
||||
"status": "up",
|
||||
"queue_length": 1,
|
||||
"last_heartbeat": now,
|
||||
"registered_at": now,
|
||||
},
|
||||
{
|
||||
"input_addr": "tcp://stage:10003",
|
||||
"output_addr": "tcp://stage:10003-out",
|
||||
"stage_id": 1,
|
||||
"status": "error",
|
||||
"queue_length": 5,
|
||||
"last_heartbeat": now,
|
||||
"registered_at": now,
|
||||
},
|
||||
]
|
||||
|
||||
payload = {"instances": instances_payload, "timestamp": now}
|
||||
pub.send(json.dumps(payload).encode("utf-8"))
|
||||
|
||||
assert _wait_for_condition(lambda: len(client.get_instance_list().instances) == 3)
|
||||
|
||||
inst_list = client.get_instance_list()
|
||||
assert isinstance(inst_list, InstanceList)
|
||||
assert len(inst_list.instances) == 3
|
||||
|
||||
for src, inst in zip(instances_payload, inst_list.instances, strict=True):
|
||||
assert inst.input_addr == src["input_addr"]
|
||||
assert inst.output_addr == src["output_addr"]
|
||||
assert inst.stage_id == src["stage_id"]
|
||||
assert inst.status.value == src["status"]
|
||||
|
||||
stage0 = client.get_instances_for_stage(0)
|
||||
stage1 = client.get_instances_for_stage(1)
|
||||
|
||||
assert all(inst.stage_id == 0 for inst in stage0.instances)
|
||||
assert all(inst.stage_id == 1 for inst in stage1.instances)
|
||||
|
||||
# Send an updated list with fewer instances and verify cache refresh.
|
||||
updated_payload = {
|
||||
"instances": instances_payload[:2],
|
||||
"timestamp": now + 1.0,
|
||||
}
|
||||
pub.send(json.dumps(updated_payload).encode("utf-8"))
|
||||
|
||||
assert _wait_for_condition(lambda: len(client.get_instance_list().instances) == 2)
|
||||
updated_list = client.get_instance_list()
|
||||
assert len(updated_list.instances) == 2
|
||||
|
||||
client.close()
|
||||
pub.close(0)
|
||||
ctx.term()
|
||||
|
||||
|
||||
def test_hub_client_close_closes_sub_socket():
|
||||
"""Verify OmniCoordClientForHub.close() marks client as closed; second close raises."""
|
||||
ctx, pub, endpoint = _bind_pub()
|
||||
client = OmniCoordClientForHub(endpoint)
|
||||
client.close()
|
||||
|
||||
with pytest.raises(RuntimeError, match="already closed"):
|
||||
client.close()
|
||||
|
||||
pub.close(0)
|
||||
ctx.term()
|
||||
@@ -0,0 +1,110 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||
|
||||
import json
|
||||
|
||||
import zmq
|
||||
|
||||
from vllm_omni.distributed.omni_coordinator import (
|
||||
OmniCoordClientForStage,
|
||||
StageStatus,
|
||||
)
|
||||
|
||||
|
||||
def _bind_router() -> tuple[zmq.Context, zmq.Socket, str]:
|
||||
ctx = zmq.Context.instance()
|
||||
router = ctx.socket(zmq.ROUTER)
|
||||
router.bind("tcp://127.0.0.1:*")
|
||||
endpoint = router.getsockopt(zmq.LAST_ENDPOINT).decode("ascii")
|
||||
return ctx, router, endpoint
|
||||
|
||||
|
||||
def _recv_event(router: zmq.Socket) -> dict:
|
||||
frames = router.recv_multipart()
|
||||
# ROUTER adds identity frame; the last frame is the payload.
|
||||
payload = frames[-1]
|
||||
return json.loads(payload.decode("utf-8"))
|
||||
|
||||
|
||||
def test_stage_client_auto_register_on_init():
|
||||
"""Verify OmniCoordClientForStage automatically sends initial registration/status-up event when created."""
|
||||
ctx, router, endpoint = _bind_router()
|
||||
|
||||
input_addr = "tcp://stage:10001"
|
||||
output_addr = "tcp://stage:10001-out"
|
||||
stage_id = 0
|
||||
|
||||
client = OmniCoordClientForStage(endpoint, input_addr, output_addr, stage_id)
|
||||
|
||||
event = _recv_event(router)
|
||||
|
||||
assert event["event_type"] == "update"
|
||||
assert event["status"] == StageStatus.UP.value
|
||||
assert event["stage_id"] == stage_id
|
||||
assert event["input_addr"] == input_addr
|
||||
assert event["output_addr"] == output_addr
|
||||
|
||||
client.close()
|
||||
router.close(0)
|
||||
ctx.term()
|
||||
|
||||
|
||||
def test_stage_client_update_info_sends_correct_event():
|
||||
"""Verify OmniCoordClientForStage.update_info() sends status/load update events with expected fields."""
|
||||
ctx, router, endpoint = _bind_router()
|
||||
|
||||
input_addr = "tcp://stage:10002"
|
||||
output_addr = "tcp://stage:10002-out"
|
||||
stage_id = 1
|
||||
|
||||
client = OmniCoordClientForStage(endpoint, input_addr, output_addr, stage_id)
|
||||
|
||||
# Discard initial registration event.
|
||||
_recv_event(router)
|
||||
|
||||
client.update_info(status=StageStatus.ERROR)
|
||||
client.update_info(queue_length=10)
|
||||
|
||||
first = _recv_event(router)
|
||||
second = _recv_event(router)
|
||||
|
||||
assert first["status"] == StageStatus.ERROR.value
|
||||
assert first["stage_id"] == stage_id
|
||||
assert first["input_addr"] == input_addr
|
||||
assert first["output_addr"] == output_addr
|
||||
|
||||
assert second["queue_length"] == 10
|
||||
assert second["stage_id"] == stage_id
|
||||
assert second["input_addr"] == input_addr
|
||||
assert second["output_addr"] == output_addr
|
||||
|
||||
client.close()
|
||||
router.close(0)
|
||||
ctx.term()
|
||||
|
||||
|
||||
def test_stage_client_close_sends_down_status():
|
||||
"""Verify close() sends final status-down event before closing underlying socket."""
|
||||
ctx, router, endpoint = _bind_router()
|
||||
|
||||
input_addr = "tcp://stage:10003"
|
||||
output_addr = "tcp://stage:10003-out"
|
||||
stage_id = 2
|
||||
|
||||
client = OmniCoordClientForStage(endpoint, input_addr, output_addr, stage_id)
|
||||
|
||||
# Discard initial registration event.
|
||||
_recv_event(router)
|
||||
|
||||
client.close()
|
||||
|
||||
event = _recv_event(router)
|
||||
assert event["status"] == StageStatus.DOWN.value
|
||||
assert event["stage_id"] == stage_id
|
||||
assert event["input_addr"] == input_addr
|
||||
assert event["output_addr"] == output_addr
|
||||
|
||||
assert client._socket.closed # DEALER socket no longer usable after close
|
||||
|
||||
router.close(0)
|
||||
ctx.term()
|
||||
@@ -0,0 +1,209 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||
|
||||
import json
|
||||
import time
|
||||
|
||||
import zmq
|
||||
from vllm.v1.utils import get_engine_client_zmq_addr
|
||||
|
||||
from vllm_omni.distributed.omni_coordinator import (
|
||||
OmniCoordClientForStage,
|
||||
OmniCoordinator,
|
||||
StageStatus,
|
||||
)
|
||||
|
||||
|
||||
def _recv_instance_list(sub: zmq.Socket, timeout_ms: int = 2000) -> dict | None:
|
||||
"""Receive InstanceList JSON from SUB socket. Returns None on timeout."""
|
||||
sub.setsockopt(zmq.RCVTIMEO, timeout_ms)
|
||||
try:
|
||||
data = sub.recv()
|
||||
return json.loads(data.decode("utf-8"))
|
||||
except zmq.Again:
|
||||
return None
|
||||
|
||||
|
||||
def _wait_for_instance_list(
|
||||
sub: zmq.Socket,
|
||||
expected_count: int,
|
||||
timeout: float = 3.0,
|
||||
) -> dict | None:
|
||||
"""Wait until received InstanceList with expected_count active instances."""
|
||||
start = time.time()
|
||||
while time.time() - start < timeout:
|
||||
msg = _recv_instance_list(sub, timeout_ms=500)
|
||||
if msg is not None and len(msg.get("instances", [])) == expected_count:
|
||||
return msg
|
||||
return None
|
||||
|
||||
|
||||
def test_omni_coordinator_registration_broadcast():
|
||||
"""Verify that after multiple OmniCoordClientForStage instances register,
|
||||
OmniCoordinator publishes an InstanceList containing all registered instances.
|
||||
"""
|
||||
router_addr = get_engine_client_zmq_addr(
|
||||
local_only=False,
|
||||
host="127.0.0.1",
|
||||
port=0,
|
||||
)
|
||||
pub_addr = get_engine_client_zmq_addr(
|
||||
local_only=False,
|
||||
host="127.0.0.1",
|
||||
port=0,
|
||||
)
|
||||
coordinator = OmniCoordinator(
|
||||
router_zmq_addr=router_addr,
|
||||
pub_zmq_addr=pub_addr,
|
||||
heartbeat_timeout=1000.0,
|
||||
)
|
||||
|
||||
sub_ctx = zmq.Context.instance()
|
||||
sub = sub_ctx.socket(zmq.SUB)
|
||||
sub.connect(pub_addr)
|
||||
sub.setsockopt(zmq.SUBSCRIBE, b"")
|
||||
|
||||
# ZMQ PUB/SUB slow-joiner: allow SUB to connect before clients register.
|
||||
time.sleep(0.3)
|
||||
|
||||
# Create 3 stage clients; each auto-registers on init.
|
||||
clients = [
|
||||
OmniCoordClientForStage(router_addr, "tcp://stage:10001", "tcp://stage:10001-out", 0),
|
||||
OmniCoordClientForStage(router_addr, "tcp://stage:10002", "tcp://stage:10002-out", 0),
|
||||
OmniCoordClientForStage(router_addr, "tcp://stage:10003", "tcp://stage:10003-out", 1),
|
||||
]
|
||||
|
||||
msg = _wait_for_instance_list(sub, expected_count=3)
|
||||
assert msg is not None, "Expected InstanceList with 3 instances"
|
||||
assert len(msg["instances"]) == 3
|
||||
assert isinstance(msg["timestamp"], (int, float))
|
||||
|
||||
input_addrs = {inst["input_addr"] for inst in msg["instances"]}
|
||||
assert "tcp://stage:10001" in input_addrs
|
||||
assert "tcp://stage:10002" in input_addrs
|
||||
assert "tcp://stage:10003" in input_addrs
|
||||
|
||||
for c in clients:
|
||||
c.close()
|
||||
coordinator.close()
|
||||
sub.close(0)
|
||||
sub_ctx.term()
|
||||
|
||||
|
||||
def test_omni_coordinator_heartbeat_timeout_handling():
|
||||
"""Verify that when a stage instance stops sending heartbeats,
|
||||
OmniCoordinator marks it as unhealthy and excludes it from the active list.
|
||||
"""
|
||||
router_addr = get_engine_client_zmq_addr(
|
||||
local_only=False,
|
||||
host="127.0.0.1",
|
||||
port=0,
|
||||
)
|
||||
pub_addr = get_engine_client_zmq_addr(
|
||||
local_only=False,
|
||||
host="127.0.0.1",
|
||||
port=0,
|
||||
)
|
||||
coordinator = OmniCoordinator(
|
||||
router_zmq_addr=router_addr,
|
||||
pub_zmq_addr=pub_addr,
|
||||
heartbeat_timeout=5.0,
|
||||
)
|
||||
|
||||
sub_ctx = zmq.Context.instance()
|
||||
sub = sub_ctx.socket(zmq.SUB)
|
||||
sub.connect(pub_addr)
|
||||
sub.setsockopt(zmq.SUBSCRIBE, b"")
|
||||
|
||||
time.sleep(0.3)
|
||||
|
||||
# A and B: real clients that send heartbeats every 5s.
|
||||
client_a = OmniCoordClientForStage(router_addr, "tcp://stage:a", "tcp://stage:a-out", 0)
|
||||
client_b = OmniCoordClientForStage(router_addr, "tcp://stage:b", "tcp://stage:b-out", 0)
|
||||
|
||||
# C: raw DEALER that sends only registration, no heartbeat.
|
||||
dealer_ctx = zmq.Context.instance()
|
||||
dealer_c = dealer_ctx.socket(zmq.DEALER)
|
||||
dealer_c.connect(router_addr)
|
||||
reg_event = {
|
||||
"input_addr": "tcp://stage:c",
|
||||
"output_addr": "tcp://stage:c-out",
|
||||
"stage_id": 0,
|
||||
"event_type": "update",
|
||||
"status": StageStatus.UP.value,
|
||||
"queue_length": 0,
|
||||
}
|
||||
dealer_c.send(json.dumps(reg_event).encode("utf-8"))
|
||||
|
||||
msg = _wait_for_instance_list(sub, expected_count=3)
|
||||
assert msg is not None, "Expected initial 3 instances"
|
||||
assert len(msg["instances"]) == 3
|
||||
|
||||
# Wait for heartbeat timeout (timeout=5s, check interval ~2.5s).
|
||||
time.sleep(8.0)
|
||||
|
||||
# Receive the update (C should be ERROR and excluded from active list).
|
||||
msg_after_timeout = _wait_for_instance_list(sub, expected_count=2, timeout=5.0)
|
||||
assert msg_after_timeout is not None, "Expected InstanceList with 2 instances after timeout"
|
||||
instances = msg_after_timeout.get("instances", [])
|
||||
input_addrs = {inst["input_addr"] for inst in instances}
|
||||
|
||||
assert "tcp://stage:a" in input_addrs
|
||||
assert "tcp://stage:b" in input_addrs
|
||||
assert "tcp://stage:c" not in input_addrs
|
||||
|
||||
client_a.close()
|
||||
client_b.close()
|
||||
dealer_c.close(0)
|
||||
coordinator.close()
|
||||
sub.close(0)
|
||||
dealer_ctx.term()
|
||||
sub_ctx.term()
|
||||
|
||||
|
||||
def test_omni_coordinator_instance_shutdown_handling():
|
||||
"""Verify that when a stage instance sends status='down',
|
||||
OmniCoordinator removes it from the active list and broadcasts an updated list.
|
||||
"""
|
||||
router_addr = get_engine_client_zmq_addr(
|
||||
local_only=False,
|
||||
host="127.0.0.1",
|
||||
port=0,
|
||||
)
|
||||
pub_addr = get_engine_client_zmq_addr(
|
||||
local_only=False,
|
||||
host="127.0.0.1",
|
||||
port=0,
|
||||
)
|
||||
coordinator = OmniCoordinator(
|
||||
router_zmq_addr=router_addr,
|
||||
pub_zmq_addr=pub_addr,
|
||||
heartbeat_timeout=1000.0,
|
||||
)
|
||||
|
||||
sub_ctx = zmq.Context.instance()
|
||||
sub = sub_ctx.socket(zmq.SUB)
|
||||
sub.connect(pub_addr)
|
||||
sub.setsockopt(zmq.SUBSCRIBE, b"")
|
||||
|
||||
time.sleep(0.3) # PUB/SUB slow-joiner
|
||||
|
||||
client = OmniCoordClientForStage(router_addr, "tcp://stage:shutdown", "tcp://stage:shutdown-out", 0)
|
||||
|
||||
msg = _wait_for_instance_list(sub, expected_count=1)
|
||||
assert msg is not None
|
||||
assert len(msg["instances"]) == 1
|
||||
assert msg["instances"][0]["input_addr"] == "tcp://stage:shutdown"
|
||||
|
||||
# Send down status (simulating graceful shutdown).
|
||||
client.update_info(status=StageStatus.DOWN)
|
||||
|
||||
# Receive updated list (should have 0 active instances).
|
||||
msg = _wait_for_instance_list(sub, expected_count=0)
|
||||
assert msg is not None
|
||||
assert len(msg["instances"]) == 0
|
||||
|
||||
client.close()
|
||||
coordinator.close()
|
||||
sub.close(0)
|
||||
sub_ctx.term()
|
||||
@@ -6,7 +6,7 @@ import pytest
|
||||
import torch
|
||||
from vllm.distributed.parallel_state import cleanup_dist_env_and_memory
|
||||
|
||||
from tests.utils import GPUMemoryMonitor, hardware_test
|
||||
from tests.utils import DeviceMemoryMonitor, hardware_test
|
||||
from vllm_omni.inputs.data import OmniDiffusionSamplingParams
|
||||
from vllm_omni.platforms import current_omni_platform
|
||||
|
||||
@@ -23,9 +23,9 @@ models = ["riverclouds/qwen_image_random"]
|
||||
def inference(model_name: str, offload: bool = True):
|
||||
gc.collect()
|
||||
current_omni_platform.empty_cache()
|
||||
device_index = torch.cuda.current_device()
|
||||
torch.cuda.reset_peak_memory_stats(device=device_index)
|
||||
monitor = GPUMemoryMonitor(device_index=device_index, interval=0.02)
|
||||
device_index = current_omni_platform.current_device()
|
||||
current_omni_platform.reset_peak_memory_stats()
|
||||
monitor = DeviceMemoryMonitor(device_index=device_index, interval=0.02)
|
||||
monitor.start()
|
||||
m = Omni(
|
||||
model=model_name,
|
||||
@@ -33,7 +33,7 @@ def inference(model_name: str, offload: bool = True):
|
||||
# cache_backend="cache_dit",
|
||||
enable_cpu_offload=offload,
|
||||
)
|
||||
torch.cuda.reset_peak_memory_stats(device=device_index)
|
||||
current_omni_platform.reset_peak_memory_stats()
|
||||
height = 256
|
||||
width = 256
|
||||
|
||||
@@ -44,7 +44,7 @@ def inference(model_name: str, offload: bool = True):
|
||||
width=width,
|
||||
num_inference_steps=9,
|
||||
guidance_scale=0.0,
|
||||
generator=torch.Generator("cuda").manual_seed(42),
|
||||
generator=torch.Generator(device=current_omni_platform.device_type).manual_seed(42),
|
||||
),
|
||||
)
|
||||
peak = monitor.peak_used_mb
|
||||
|
||||
@@ -5,7 +5,7 @@ import pytest
|
||||
import torch
|
||||
from vllm.distributed.parallel_state import cleanup_dist_env_and_memory
|
||||
|
||||
from tests.utils import GPUMemoryMonitor
|
||||
from tests.utils import DeviceMemoryMonitor
|
||||
from vllm_omni.inputs.data import OmniDiffusionSamplingParams
|
||||
from vllm_omni.platforms import current_omni_platform
|
||||
|
||||
@@ -28,11 +28,9 @@ def run_inference(
|
||||
layerwise_offload: bool = False,
|
||||
num_inference_steps: int = 3,
|
||||
) -> float:
|
||||
# For now, only support on GPU, so apply torch.cuda operations here
|
||||
# NPU / ROCm platforms are expected to be detected and skipped this test function
|
||||
current_omni_platform.empty_cache()
|
||||
device_index = torch.cuda.current_device()
|
||||
monitor = GPUMemoryMonitor(device_index=device_index, interval=0.02)
|
||||
device_index = current_omni_platform.current_device()
|
||||
monitor = DeviceMemoryMonitor(device_index=device_index, interval=0.02)
|
||||
monitor.start()
|
||||
|
||||
m = Omni(
|
||||
@@ -44,7 +42,7 @@ def run_inference(
|
||||
flow_shift=5.0,
|
||||
)
|
||||
|
||||
torch.cuda.reset_peak_memory_stats(device=device_index)
|
||||
current_omni_platform.reset_peak_memory_stats()
|
||||
|
||||
# Refer to tests/e2e/offline_inference/test_t2v_model.py
|
||||
# Use minimal settings for testing
|
||||
@@ -57,7 +55,7 @@ def run_inference(
|
||||
OmniDiffusionSamplingParams(
|
||||
height=height,
|
||||
width=width,
|
||||
generator=torch.Generator("cuda").manual_seed(42),
|
||||
generator=torch.Generator(device=current_omni_platform.device_type).manual_seed(42),
|
||||
guidance_scale=1.0,
|
||||
num_inference_steps=num_inference_steps,
|
||||
num_frames=num_frames,
|
||||
|
||||
@@ -0,0 +1,248 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||
|
||||
"""
|
||||
System test for Expert Parallel (EP).
|
||||
|
||||
Tests verify that EP inference produces correct outputs compared to baseline.
|
||||
"""
|
||||
|
||||
import gc
|
||||
import os
|
||||
import time
|
||||
from typing import NamedTuple
|
||||
|
||||
import numpy as np
|
||||
import pytest
|
||||
import torch
|
||||
import torch.distributed as dist
|
||||
from PIL import Image
|
||||
|
||||
from tests.utils import hardware_test
|
||||
from vllm_omni import Omni
|
||||
from vllm_omni.diffusion.data import DiffusionParallelConfig
|
||||
from vllm_omni.inputs.data import OmniDiffusionSamplingParams
|
||||
from vllm_omni.platforms import current_omni_platform
|
||||
|
||||
# Test configuration
|
||||
MODELS = ["tencent/HunyuanImage-3.0"]
|
||||
PROMPT = "A brown and white dog is running on the grass"
|
||||
DEFAULT_HEIGHT = 1024
|
||||
DEFAULT_WIDTH = 1024
|
||||
DEFAULT_SEED = 1234
|
||||
DEFAULT_STEPS = 50
|
||||
DEFAULT_GUIDANCE_SCALE = 0.0
|
||||
COS_SIM_MEAN_THRESHOLD = 1e-2
|
||||
COS_SIM_MAX_THRESHOLD = 1e-2
|
||||
MSE_THRESHOLD = 1e-2
|
||||
DIFF_MEAN_THRESHOLD = 5e-2
|
||||
DIFF_MAX_THRESHOLD = 1
|
||||
TOTAL_CARD_NUM = 8
|
||||
|
||||
|
||||
class InferenceResult(NamedTuple):
|
||||
"""Result of an inference run."""
|
||||
|
||||
images: list[Image.Image]
|
||||
elapsed_ms: float
|
||||
|
||||
|
||||
def _cleanup_distributed():
|
||||
"""Clean up distributed environment and GPU resources."""
|
||||
if dist.is_initialized():
|
||||
dist.destroy_process_group()
|
||||
|
||||
for key in ["MASTER_ADDR", "MASTER_PORT", "RANK", "WORLD_SIZE", "LOCAL_RANK"]:
|
||||
os.environ.pop(key, None)
|
||||
|
||||
gc.collect()
|
||||
if current_omni_platform.is_available():
|
||||
current_omni_platform.empty_cache()
|
||||
current_omni_platform.synchronize()
|
||||
|
||||
time.sleep(5)
|
||||
|
||||
|
||||
def _diff_metrics(a: Image.Image, b: Image.Image) -> tuple[float, float, float, float, float]:
|
||||
"""Return (mean_abs_diff, max_abs_diff) over RGB pixels in [0, 1]."""
|
||||
ta = torch.from_numpy(np.asarray(a.convert("RGB"), dtype=np.float32) / 255.0)
|
||||
tb = torch.from_numpy(np.asarray(b.convert("RGB"), dtype=np.float32) / 255.0)
|
||||
assert ta.shape == tb.shape, f"Image shapes differ: {ta.shape} vs {tb.shape}"
|
||||
abs_diff = torch.abs(ta - tb)
|
||||
|
||||
vec_a = ta.reshape(-1)
|
||||
vec_b = tb.reshape(-1)
|
||||
cos_sim = torch.nn.functional.cosine_similarity(vec_a, vec_b, dim=0, eps=1e-8)
|
||||
mse = torch.nn.functional.mse_loss(ta, tb)
|
||||
return abs_diff.mean().item(), abs_diff.max().item(), cos_sim.mean().item(), cos_sim.max().item(), mse.item()
|
||||
|
||||
|
||||
def _run_inference(
|
||||
model_name: str,
|
||||
tensor_parallel_size: int,
|
||||
enable_expert_parallel: bool,
|
||||
guidance_scale: float = DEFAULT_GUIDANCE_SCALE,
|
||||
height: int = DEFAULT_HEIGHT,
|
||||
width: int = DEFAULT_WIDTH,
|
||||
seed: int = DEFAULT_SEED,
|
||||
warmup: bool = True,
|
||||
) -> InferenceResult:
|
||||
"""Run inference with specified configuration.
|
||||
|
||||
Args:
|
||||
warmup: If True, run one warmup iteration before the timed run.
|
||||
"""
|
||||
parallel_config = DiffusionParallelConfig(
|
||||
tensor_parallel_size=tensor_parallel_size,
|
||||
enable_expert_parallel=enable_expert_parallel,
|
||||
)
|
||||
omni = Omni(model=model_name, parallel_config=parallel_config)
|
||||
|
||||
try:
|
||||
# Warmup run (not timed)
|
||||
if warmup:
|
||||
_ = omni.generate(
|
||||
PROMPT,
|
||||
OmniDiffusionSamplingParams(
|
||||
height=height,
|
||||
width=width,
|
||||
num_inference_steps=DEFAULT_STEPS,
|
||||
guidance_scale=guidance_scale,
|
||||
generator=torch.Generator(current_omni_platform.device_type).manual_seed(seed),
|
||||
num_outputs_per_prompt=1,
|
||||
),
|
||||
)
|
||||
|
||||
# Timed run
|
||||
start = time.time()
|
||||
outputs = omni.generate(
|
||||
PROMPT,
|
||||
OmniDiffusionSamplingParams(
|
||||
height=height,
|
||||
width=width,
|
||||
num_inference_steps=DEFAULT_STEPS,
|
||||
guidance_scale=guidance_scale,
|
||||
generator=torch.Generator(current_omni_platform.device_type).manual_seed(seed),
|
||||
num_outputs_per_prompt=1,
|
||||
),
|
||||
)
|
||||
elapsed_ms = (time.time() - start) * 1000
|
||||
|
||||
return InferenceResult(
|
||||
images=outputs[0].request_output[0].images,
|
||||
elapsed_ms=elapsed_ms,
|
||||
)
|
||||
finally:
|
||||
omni.close()
|
||||
_cleanup_distributed()
|
||||
|
||||
|
||||
# EP configurations: (tensor_parallel_size, enable_ep, height, width, is_perf_test)
|
||||
EP_TEST_CONFIG = [
|
||||
(TOTAL_CARD_NUM, True, DEFAULT_HEIGHT, DEFAULT_WIDTH, True),
|
||||
]
|
||||
|
||||
|
||||
@pytest.mark.core_model
|
||||
@pytest.mark.diffusion
|
||||
@pytest.mark.parallel
|
||||
@hardware_test(res={"cuda": "H100"}, num_cards={"cuda": TOTAL_CARD_NUM})
|
||||
@pytest.mark.parametrize("model_name", MODELS)
|
||||
def test_ep(model_name):
|
||||
baseline_cache: dict[str, InferenceResult] = {}
|
||||
results: list[dict] = []
|
||||
# baseline_key = PROMPT + "_" + str(DEFAULT_HEIGHT) + "*" + str(DEFAULT_WIDTH) + "_" + str(DEFAULT_SEED) + "_" + str(DEFAULT_STEPS)
|
||||
# baseline_cache[baseline_key] = _run_inference(
|
||||
# model_name=model_name,
|
||||
# tensor_parallel_size=TOTAL_CARD_NUM,
|
||||
# enable_expert_parallel=False,
|
||||
# warmup=True,
|
||||
# )
|
||||
device_count = current_omni_platform.get_device_count()
|
||||
|
||||
print("\n" + "=" * 90)
|
||||
print(f"Sequence Parallel Test - Model: {model_name}")
|
||||
print(f"Available GPUs: {device_count}")
|
||||
print("=" * 90)
|
||||
|
||||
for tensor_parallel_size, enable_ep, height, width, is_perf_test in EP_TEST_CONFIG:
|
||||
cache_key = PROMPT + "_" + str(height) + "*" + str(width) + "_" + str(DEFAULT_SEED) + "_" + str(DEFAULT_STEPS)
|
||||
if cache_key not in baseline_cache:
|
||||
print(
|
||||
f"\n--- Running baseline{{ prompt: {PROMPT}, resolution: {height}x{width}, seed: {DEFAULT_SEED}, steps: {DEFAULT_STEPS} }}---"
|
||||
)
|
||||
baseline = _run_inference(
|
||||
model_name=model_name,
|
||||
tensor_parallel_size=TOTAL_CARD_NUM,
|
||||
enable_expert_parallel=False,
|
||||
warmup=True,
|
||||
)
|
||||
assert len(baseline.images) == 1
|
||||
baseline_cache[cache_key] = baseline
|
||||
print(f"[baseline] {height}x{width}: {baseline.elapsed_ms:.0f}ms")
|
||||
else:
|
||||
baseline = baseline_cache[cache_key]
|
||||
|
||||
print(f"\n--- Running test: {{ enable_ep: {enable_ep} }} ---")
|
||||
infer_result = _run_inference(
|
||||
model_name=model_name,
|
||||
tensor_parallel_size=tensor_parallel_size,
|
||||
enable_expert_parallel=enable_ep,
|
||||
warmup=is_perf_test,
|
||||
)
|
||||
assert len(infer_result.images) == 1
|
||||
mean_diff, max_diff, cos_sim_mean, cos_sim_max, mse = _diff_metrics(baseline.images[0], infer_result.images[0])
|
||||
print(f"{cos_sim_mean}, {cos_sim_max}, {mse}")
|
||||
|
||||
# Build result entry
|
||||
result = {
|
||||
"prompt": PROMPT,
|
||||
"enable_ep": enable_ep,
|
||||
"height": height,
|
||||
"width": width,
|
||||
"baseline_ms": baseline.elapsed_ms,
|
||||
"ep_ms": infer_result.elapsed_ms,
|
||||
"mean_diff": mean_diff,
|
||||
"max_diff": max_diff,
|
||||
"cos_sim_mean": cos_sim_mean,
|
||||
"cos_sim_max": cos_sim_max,
|
||||
"mse": mse,
|
||||
"is_perf_test": is_perf_test,
|
||||
}
|
||||
results.append(result)
|
||||
|
||||
# Output based on test type
|
||||
if is_perf_test:
|
||||
speedup = baseline.elapsed_ms / infer_result.elapsed_ms if infer_result.elapsed_ms > 0 else 0
|
||||
result["speedup"] = speedup
|
||||
print(
|
||||
f"[enable_ep: {enable_ep}] {tensor_parallel_size} GPUs | "
|
||||
f"baseline: {baseline.elapsed_ms:.0f}ms, ep: {infer_result.elapsed_ms:.0f}ms, "
|
||||
f"speedup: {speedup:.2f}x"
|
||||
)
|
||||
else:
|
||||
print(
|
||||
f"[enable_ep: {enable_ep}] {tensor_parallel_size} GPUs | ep: {infer_result.elapsed_ms:.0f}ms (correctness only)"
|
||||
)
|
||||
|
||||
print(
|
||||
f"[enable_ep: {enable_ep}] diff: [mean={mean_diff:.6e}, max={max_diff:.6e}], cos_sim: [mean={cos_sim_mean:.6e}, max={cos_sim_max:.6e}], mse: {mse:.6e}"
|
||||
)
|
||||
assert mean_diff <= DIFF_MEAN_THRESHOLD and max_diff <= DIFF_MAX_THRESHOLD, (
|
||||
f"[enable_ep: {enable_ep}] output differs from baseline: mean={mean_diff:.6e}, max={max_diff:.6e}"
|
||||
)
|
||||
|
||||
print("\n" + "=" * 90)
|
||||
print("SUMMARY")
|
||||
print("=" * 90)
|
||||
print(f"{'Mode':<15} {'GPUs':<6} {'Size':<10} {'Baseline':<12} {'EP':<12} {'Speedup':<10} {'Status'}")
|
||||
print("-" * 90)
|
||||
for r in results:
|
||||
speedup_str = f"{r['speedup']:.2f}x" if r.get("speedup") else "N/A"
|
||||
baseline_str = f"{r['baseline_ms']:.0f}ms" if r["is_perf_test"] else "N/A"
|
||||
status = "PASS" if r["mean_diff"] <= DIFF_MEAN_THRESHOLD else "FAIL"
|
||||
print(
|
||||
f"{r['prompt']:<15.10} {r['enable_ep']:<6} {r['height']}x{r['width']:<5} "
|
||||
f"{baseline_str:<12} {r['ep_ms']:.0f}ms{'':<7} {speedup_str:<10} {status}"
|
||||
)
|
||||
print("=" * 90)
|
||||
@@ -22,7 +22,7 @@ import torch
|
||||
from PIL import Image
|
||||
from vllm.distributed.parallel_state import cleanup_dist_env_and_memory
|
||||
|
||||
from tests.utils import GPUMemoryMonitor, hardware_test
|
||||
from tests.utils import DeviceMemoryMonitor, hardware_test
|
||||
from vllm_omni import Omni
|
||||
from vllm_omni.diffusion.data import DiffusionParallelConfig
|
||||
from vllm_omni.inputs.data import OmniDiffusionSamplingParams
|
||||
@@ -54,12 +54,13 @@ def _pil_to_float_rgb_tensor(img: Image.Image) -> torch.Tensor:
|
||||
|
||||
|
||||
def _diff_metrics(a: Image.Image, b: Image.Image) -> tuple[float, float]:
|
||||
"""Return (mean_abs_diff, max_abs_diff) over RGB pixels in [0, 1]."""
|
||||
"""Return (mean_abs_diff, p99_abs_diff) over RGB pixels in [0, 1]."""
|
||||
ta = _pil_to_float_rgb_tensor(a)
|
||||
tb = _pil_to_float_rgb_tensor(b)
|
||||
assert ta.shape == tb.shape, f"Image shapes differ: {ta.shape} vs {tb.shape}"
|
||||
abs_diff = torch.abs(ta - tb)
|
||||
return abs_diff.mean().item(), abs_diff.max().item()
|
||||
p99_abs_diff = torch.quantile(abs_diff.flatten(), 0.99).item()
|
||||
return abs_diff.mean().item(), p99_abs_diff
|
||||
|
||||
|
||||
def _extract_single_image(outputs) -> Image.Image:
|
||||
@@ -93,9 +94,9 @@ def _run_zimage_generate(
|
||||
if num_requests < 2:
|
||||
raise ValueError("num_requests must be >= 2 (1 warmup + >=1 timed)")
|
||||
|
||||
torch.cuda.empty_cache()
|
||||
device_index = torch.cuda.current_device()
|
||||
monitor = GPUMemoryMonitor(device_index=device_index, interval=0.02)
|
||||
current_omni_platform.empty_cache()
|
||||
device_index = current_omni_platform.current_device()
|
||||
monitor = DeviceMemoryMonitor(device_index=device_index, interval=0.02)
|
||||
monitor.start()
|
||||
m = Omni(
|
||||
model=_get_zimage_model(),
|
||||
@@ -161,8 +162,8 @@ def _run_zimage_generate(
|
||||
def test_zimage_tensor_parallel_tp2(tmp_path: Path):
|
||||
if current_omni_platform.is_npu() or current_omni_platform.is_rocm():
|
||||
pytest.skip("Z-Image TP e2e test is only supported on CUDA for now.")
|
||||
if not torch.cuda.is_available() or torch.cuda.device_count() < 2:
|
||||
pytest.skip("Z-Image TP=2 requires >= 2 CUDA devices.")
|
||||
if not current_omni_platform.is_available() or current_omni_platform.device_count() < 2:
|
||||
pytest.skip("Z-Image TP=2 requires >= 2 devices.")
|
||||
|
||||
enforce_eager = False
|
||||
|
||||
@@ -196,18 +197,18 @@ def test_zimage_tensor_parallel_tp2(tmp_path: Path):
|
||||
assert tp1_img.width == width and tp1_img.height == height
|
||||
assert tp2_img.width == width and tp2_img.height == height
|
||||
|
||||
mean_abs_diff, max_abs_diff = _diff_metrics(tp1_img, tp2_img)
|
||||
mean_abs_diff, p99_abs_diff = _diff_metrics(tp1_img, tp2_img)
|
||||
mean_threshold = 3e-2
|
||||
max_threshold = 5e-1
|
||||
p99_threshold = 2.5e-1
|
||||
print(
|
||||
"Z-Image TP image diff stats (TP=1 vs TP=2): "
|
||||
f"mean_abs_diff={mean_abs_diff:.6e}, max_abs_diff={max_abs_diff:.6e}; "
|
||||
f"thresholds: mean<={mean_threshold:.6e}, max<={max_threshold:.6e}; "
|
||||
f"mean_abs_diff={mean_abs_diff:.6e}, p99_abs_diff={p99_abs_diff:.6e}; "
|
||||
f"thresholds: mean<={mean_threshold:.6e}, p99<={p99_threshold:.6e}; "
|
||||
f"tp1_img={tp1_path}, tp2_img={tp2_path}"
|
||||
)
|
||||
assert mean_abs_diff <= mean_threshold and max_abs_diff <= max_threshold, (
|
||||
f"Image diff exceeded threshold: mean_abs_diff={mean_abs_diff:.6e}, max_abs_diff={max_abs_diff:.6e} "
|
||||
f"(thresholds: mean<={mean_threshold:.6e}, max<={max_threshold:.6e})"
|
||||
assert mean_abs_diff <= mean_threshold and p99_abs_diff <= p99_threshold, (
|
||||
f"Image diff exceeded threshold: mean_abs_diff={mean_abs_diff:.6e}, p99_abs_diff={p99_abs_diff:.6e} "
|
||||
f"(thresholds: mean<={mean_threshold:.6e}, p99<={p99_threshold:.6e})"
|
||||
)
|
||||
|
||||
print(f"Z-Image TP perf (lower is better): tp1_time_s={tp1_time_s:.6f}, tp2_time_s={tp2_time_s:.6f}")
|
||||
@@ -223,8 +224,8 @@ def test_zimage_tensor_parallel_tp2(tmp_path: Path):
|
||||
def test_zimage_vae_patch_parallel_tp2(tmp_path: Path):
|
||||
if current_omni_platform.is_npu() or current_omni_platform.is_rocm():
|
||||
pytest.skip("Z-Image VAE patch parallel e2e test is only supported on CUDA for now.")
|
||||
if not torch.cuda.is_available() or torch.cuda.device_count() < 2:
|
||||
pytest.skip("Z-Image VAE patch parallel TP=2 requires >= 2 CUDA devices.")
|
||||
if not current_omni_platform.is_available() or current_omni_platform.device_count() < 2:
|
||||
pytest.skip("Z-Image VAE patch parallel TP=2 requires >= 2 devices.")
|
||||
|
||||
enforce_eager = False
|
||||
|
||||
@@ -262,16 +263,16 @@ def test_zimage_vae_patch_parallel_tp2(tmp_path: Path):
|
||||
baseline_img.save(baseline_path)
|
||||
pp2_img.save(pp2_path)
|
||||
|
||||
mean_abs_diff, max_abs_diff = _diff_metrics(baseline_img, pp2_img)
|
||||
mean_abs_diff, p99_abs_diff = _diff_metrics(baseline_img, pp2_img)
|
||||
mean_threshold = 5e-3
|
||||
max_threshold = 1e-1
|
||||
p99_threshold = 1e-1
|
||||
print(
|
||||
"Z-Image VAE patch parallel image diff stats (TP=2, pp=1 vs pp=2): "
|
||||
f"mean_abs_diff={mean_abs_diff:.6e}, max_abs_diff={max_abs_diff:.6e}; "
|
||||
f"thresholds: mean<={mean_threshold:.6e}, max<={max_threshold:.6e}; "
|
||||
f"mean_abs_diff={mean_abs_diff:.6e}, p99_abs_diff={p99_abs_diff:.6e}; "
|
||||
f"thresholds: mean<={mean_threshold:.6e}, p99<={p99_threshold:.6e}; "
|
||||
f"pp1_img={baseline_path}, pp2_img={pp2_path}"
|
||||
)
|
||||
assert mean_abs_diff <= mean_threshold and max_abs_diff <= max_threshold, (
|
||||
f"Image diff exceeded threshold: mean_abs_diff={mean_abs_diff:.6e}, max_abs_diff={max_abs_diff:.6e} "
|
||||
f"(thresholds: mean<={mean_threshold:.6e}, max<={max_threshold:.6e})"
|
||||
assert mean_abs_diff <= mean_threshold and p99_abs_diff <= p99_threshold, (
|
||||
f"Image diff exceeded threshold: mean_abs_diff={mean_abs_diff:.6e}, p99_abs_diff={p99_abs_diff:.6e} "
|
||||
f"(thresholds: mean<={mean_threshold:.6e}, p99<={p99_threshold:.6e})"
|
||||
)
|
||||
@@ -8,6 +8,7 @@ from pathlib import Path
|
||||
import pytest
|
||||
|
||||
from tests.conftest import (
|
||||
OmniServerParams,
|
||||
dummy_messages_from_mix_data,
|
||||
generate_synthetic_audio,
|
||||
generate_synthetic_image,
|
||||
@@ -51,7 +52,9 @@ else:
|
||||
stage_configs = [get_chunk_config()]
|
||||
|
||||
# Create parameter combinations for model and stage config
|
||||
test_params = [(model, stage_config) for model in models for stage_config in stage_configs]
|
||||
test_params = [
|
||||
OmniServerParams(model=model, stage_config_path=stage_config) for model in models for stage_config in stage_configs
|
||||
]
|
||||
|
||||
|
||||
def get_system_prompt():
|
||||
@@ -120,7 +123,7 @@ def test_mix_to_text_audio_001(omni_server, openai_client) -> None:
|
||||
}
|
||||
|
||||
# Test single completion
|
||||
openai_client.send_request(request_config)
|
||||
openai_client.send_omni_request(request_config)
|
||||
|
||||
|
||||
@pytest.mark.advanced_model
|
||||
@@ -146,4 +149,4 @@ def test_text_to_text_001(omni_server, openai_client) -> None:
|
||||
"key_words": {"text": ["beijing"]},
|
||||
}
|
||||
|
||||
openai_client.send_request(request_config, request_num=get_max_batch_size())
|
||||
openai_client.send_omni_request(request_config, request_num=get_max_batch_size())
|
||||
@@ -12,6 +12,7 @@ from pathlib import Path
|
||||
import pytest
|
||||
|
||||
from tests.conftest import (
|
||||
OmniServerParams,
|
||||
dummy_messages_from_mix_data,
|
||||
generate_synthetic_audio,
|
||||
generate_synthetic_image,
|
||||
@@ -50,7 +51,9 @@ def get_chunk_config(default_path):
|
||||
default_path = str(Path(__file__).parent.parent / "stage_configs" / "qwen3_omni_ci.yaml")
|
||||
stage_configs = [default_path, get_chunk_config(default_path)]
|
||||
# Create parameter combinations for model and stage config
|
||||
test_params = [(model, stage_config) for model in models for stage_config in stage_configs]
|
||||
test_params = [
|
||||
OmniServerParams(model=model, stage_config_path=stage_config) for model in models for stage_config in stage_configs
|
||||
]
|
||||
|
||||
|
||||
def get_system_prompt():
|
||||
@@ -106,7 +109,7 @@ def test_text_to_audio_001(omni_server, openai_client) -> None:
|
||||
"key_words": {"text": ["beijing"]},
|
||||
}
|
||||
|
||||
openai_client.send_request(request_config)
|
||||
openai_client.send_omni_request(request_config)
|
||||
|
||||
|
||||
@pytest.mark.advanced_model
|
||||
@@ -128,7 +131,7 @@ def test_text_to_text_audio_001(omni_server, openai_client) -> None:
|
||||
"key_words": {"text": ["beijing"]},
|
||||
}
|
||||
|
||||
openai_client.send_request(request_config, request_num=get_max_batch_size())
|
||||
openai_client.send_omni_request(request_config, request_num=get_max_batch_size())
|
||||
|
||||
|
||||
@pytest.mark.advanced_model
|
||||
@@ -153,7 +156,7 @@ def test_image_to_text_001(omni_server, openai_client) -> None:
|
||||
"key_words": {"image": IMAGE_KEY},
|
||||
}
|
||||
|
||||
openai_client.send_request(request_config)
|
||||
openai_client.send_omni_request(request_config)
|
||||
|
||||
|
||||
@pytest.mark.advanced_model
|
||||
@@ -177,7 +180,7 @@ def test_image_to_audio_001(omni_server, openai_client) -> None:
|
||||
"key_words": {"image": IMAGE_KEY},
|
||||
}
|
||||
|
||||
openai_client.send_request(request_config)
|
||||
openai_client.send_omni_request(request_config)
|
||||
|
||||
|
||||
@pytest.mark.advanced_model
|
||||
@@ -201,7 +204,7 @@ def test_image_to_text_audio_001(omni_server, openai_client) -> None:
|
||||
"key_words": {"image": IMAGE_KEY},
|
||||
}
|
||||
|
||||
openai_client.send_request(request_config, request_num=get_max_batch_size())
|
||||
openai_client.send_omni_request(request_config, request_num=get_max_batch_size())
|
||||
|
||||
|
||||
@pytest.mark.advanced_model
|
||||
@@ -225,7 +228,7 @@ def test_video_to_text_001(omni_server, openai_client) -> None:
|
||||
"key_words": {"video": VIDEO_KEY},
|
||||
}
|
||||
|
||||
openai_client.send_request(request_config)
|
||||
openai_client.send_omni_request(request_config)
|
||||
|
||||
|
||||
@pytest.mark.advanced_model
|
||||
@@ -249,7 +252,7 @@ def test_video_to_audio_001(omni_server, openai_client) -> None:
|
||||
"key_words": {"video": VIDEO_KEY},
|
||||
}
|
||||
|
||||
openai_client.send_request(request_config)
|
||||
openai_client.send_omni_request(request_config)
|
||||
|
||||
|
||||
@pytest.mark.advanced_model
|
||||
@@ -273,7 +276,7 @@ def test_video_to_text_audio_001(omni_server, openai_client) -> None:
|
||||
"key_words": {"video": VIDEO_KEY},
|
||||
}
|
||||
|
||||
openai_client.send_request(request_config, request_num=get_max_batch_size())
|
||||
openai_client.send_omni_request(request_config, request_num=get_max_batch_size())
|
||||
|
||||
|
||||
@pytest.mark.advanced_model
|
||||
@@ -298,7 +301,7 @@ def test_text_audio_to_text_audio_001(omni_server, openai_client) -> None:
|
||||
"key_words": {"audio": AUDIO_KEY},
|
||||
}
|
||||
|
||||
openai_client.send_request(request_config)
|
||||
openai_client.send_omni_request(request_config)
|
||||
|
||||
|
||||
@pytest.mark.advanced_model
|
||||
@@ -324,7 +327,7 @@ def test_text_image_to_text_audio_001(omni_server, openai_client) -> None:
|
||||
"key_words": {"image": IMAGE_KEY},
|
||||
}
|
||||
|
||||
openai_client.send_request(request_config)
|
||||
openai_client.send_omni_request(request_config)
|
||||
|
||||
|
||||
@pytest.mark.advanced_model
|
||||
@@ -351,7 +354,7 @@ def test_text_video_to_text_audio_001(omni_server, openai_client) -> None:
|
||||
"key_words": {"video": VIDEO_KEY},
|
||||
}
|
||||
|
||||
openai_client.send_request(request_config)
|
||||
openai_client.send_omni_request(request_config)
|
||||
|
||||
|
||||
@pytest.mark.skip(reason="There is a known issue with shape mismatch error.")
|
||||
@@ -383,4 +386,4 @@ def test_mix_to_text_audio_001(omni_server, openai_client) -> None:
|
||||
"stream": True,
|
||||
"key_words": {"audio": AUDIO_KEY, "image": IMAGE_KEY, "video": VIDEO_KEY},
|
||||
}
|
||||
openai_client.send_request(request_config, request_num=get_max_batch_size())
|
||||
openai_client.send_omni_request(request_config, request_num=get_max_batch_size())
|
||||
@@ -0,0 +1,164 @@
|
||||
"""
|
||||
Comprehensive tests of diffusion features that are available in online serving mode
|
||||
and are supported by the following models:
|
||||
- Qwen-Image-Edit: single image input
|
||||
- Qwen-Image-Edit-2509: two image inputs
|
||||
"""
|
||||
|
||||
import pytest
|
||||
|
||||
from tests.conftest import (
|
||||
OmniServer,
|
||||
OmniServerParams,
|
||||
OpenAIClientHandler,
|
||||
dummy_messages_from_mix_data,
|
||||
generate_synthetic_image,
|
||||
)
|
||||
from tests.utils import hardware_marks
|
||||
|
||||
EDIT_PROMPT = "Transform this modern, geometrist image into a Vincent van Gogh style impressionist painting."
|
||||
MULTI_EDIT_PROMPT = (
|
||||
"Transform the first image into a Dadaism collage art. "
|
||||
"Transform the second image into a Vincent van Gogh style painting. "
|
||||
"Then juxtapose the two transformed images into a single artwork for visual contrast."
|
||||
)
|
||||
NEGATIVE_PROMPT = "blurry, low quality, modern, geometrist"
|
||||
SINGLE_CARD_FEATURE_MARKS = hardware_marks(res={"cuda": "H100"})
|
||||
PARALLEL_FEATURE_MARKS = hardware_marks(res={"cuda": "H100"}, num_cards=2)
|
||||
|
||||
|
||||
# This test file targets two models, so I write a helper function.
|
||||
# If a similar test only involves one model, one can just define a global list variable.
|
||||
def _get_diffusion_feature_cases(model: str):
|
||||
return [
|
||||
pytest.param(
|
||||
OmniServerParams(
|
||||
model=model,
|
||||
server_args=[
|
||||
"--cache-backend",
|
||||
"tea_cache", # [TODO] may consider changing to cache_dit after #1779 is resolved. Currently cache_dit and layerwise offload cannot work together.
|
||||
"--enable-layerwise-offload",
|
||||
],
|
||||
),
|
||||
id="single_card_001",
|
||||
marks=SINGLE_CARD_FEATURE_MARKS,
|
||||
),
|
||||
pytest.param(
|
||||
OmniServerParams(
|
||||
model=model,
|
||||
server_args=[
|
||||
"--cache-backend",
|
||||
"cache_dit",
|
||||
"--ulysses-degree",
|
||||
"2",
|
||||
],
|
||||
),
|
||||
id="parallel_001",
|
||||
marks=PARALLEL_FEATURE_MARKS,
|
||||
),
|
||||
pytest.param(
|
||||
OmniServerParams(
|
||||
model=model,
|
||||
server_args=[
|
||||
"--cache-backend",
|
||||
"cache_dit",
|
||||
"--ring",
|
||||
"2",
|
||||
],
|
||||
),
|
||||
id="parallel_002",
|
||||
marks=PARALLEL_FEATURE_MARKS,
|
||||
),
|
||||
pytest.param(
|
||||
OmniServerParams(
|
||||
model=model,
|
||||
server_args=[
|
||||
"--cache-backend",
|
||||
"tea_cache",
|
||||
"--cfg-parallel-size",
|
||||
"2",
|
||||
],
|
||||
),
|
||||
id="parallel_003",
|
||||
marks=PARALLEL_FEATURE_MARKS,
|
||||
),
|
||||
pytest.param(
|
||||
OmniServerParams(
|
||||
model=model,
|
||||
server_args=[
|
||||
"--cache-backend",
|
||||
"cache_dit",
|
||||
"--tensor-parallel-size",
|
||||
"2",
|
||||
"--vae-patch-parallel-size",
|
||||
"2",
|
||||
"--vae-use-tiling",
|
||||
],
|
||||
),
|
||||
id="parallel_004",
|
||||
marks=PARALLEL_FEATURE_MARKS,
|
||||
),
|
||||
]
|
||||
|
||||
|
||||
@pytest.mark.advanced_model
|
||||
@pytest.mark.diffusion
|
||||
@pytest.mark.parametrize(
|
||||
"omni_server",
|
||||
_get_diffusion_feature_cases("Qwen/Qwen-Image-Edit"),
|
||||
indirect=True,
|
||||
)
|
||||
def test_qwen_image_edit(omni_server: OmniServer, openai_client: OpenAIClientHandler):
|
||||
"""Test all diffusion features with Qwen-Image-Edit in regular end-user scenarios."""
|
||||
image_data_url = f"data:image/jpeg;base64,{generate_synthetic_image(512, 512)['base64']}"
|
||||
|
||||
messages = dummy_messages_from_mix_data(image_data_url=image_data_url, content_text=EDIT_PROMPT)
|
||||
|
||||
# CFG parallel is only activated when a negative prompt and true_cfg_scale > 1.0 are both present
|
||||
request_config = {
|
||||
"model": omni_server.model,
|
||||
"messages": messages,
|
||||
"extra_body": {
|
||||
"height": 512,
|
||||
"width": 512,
|
||||
"num_inference_steps": 2,
|
||||
"negative_prompt": NEGATIVE_PROMPT,
|
||||
"true_cfg_scale": 4.0,
|
||||
"seed": 42,
|
||||
},
|
||||
}
|
||||
|
||||
openai_client.send_diffusion_request(request_config)
|
||||
|
||||
|
||||
@pytest.mark.advanced_model
|
||||
@pytest.mark.diffusion
|
||||
@pytest.mark.parametrize(
|
||||
"omni_server",
|
||||
_get_diffusion_feature_cases("Qwen/Qwen-Image-Edit-2509"),
|
||||
indirect=True,
|
||||
)
|
||||
def test_qwen_image_edit_2509(omni_server: OmniServer, openai_client: OpenAIClientHandler):
|
||||
"""Test all diffusion features with Qwen-Image-Edit-2509 in regular end-user scenarios."""
|
||||
image_data_url_1 = f"data:image/jpeg;base64,{generate_synthetic_image(512, 512)['base64']}"
|
||||
image_data_url_2 = f"data:image/jpeg;base64,{generate_synthetic_image(512, 512)['base64']}"
|
||||
|
||||
messages = dummy_messages_from_mix_data(
|
||||
image_data_url=[image_data_url_1, image_data_url_2], content_text=MULTI_EDIT_PROMPT
|
||||
)
|
||||
|
||||
# CFG parallel is only activated when a negative prompt and true_cfg_scale > 1.0 are both present
|
||||
request_config = {
|
||||
"model": omni_server.model,
|
||||
"messages": messages,
|
||||
"extra_body": {
|
||||
"height": 512,
|
||||
"width": 512,
|
||||
"num_inference_steps": 2,
|
||||
"negative_prompt": NEGATIVE_PROMPT,
|
||||
"true_cfg_scale": 4.0,
|
||||
"seed": 42,
|
||||
},
|
||||
}
|
||||
|
||||
openai_client.send_diffusion_request(request_config)
|
||||
@@ -136,7 +136,6 @@ def mock_async_diffusion(mocker: MockerFixture):
|
||||
|
||||
async def generate(**kwargs):
|
||||
# Return n PIL images wrapped in result object
|
||||
print("!!!!!!!!!!!!!!!!!!!!! kwargs", kwargs)
|
||||
n = kwargs["sampling_params_list"][0].num_outputs_per_prompt
|
||||
mock.captured_sampling_params_list = kwargs["sampling_params_list"]
|
||||
mock.captured_prompt = kwargs["prompt"]
|
||||
|
||||
@@ -1017,3 +1017,27 @@ class TestStreamingResponse:
|
||||
response = client.post("/v1/audio/speech", json={"input": "Hello", "response_format": "wav"})
|
||||
assert response.status_code == 200
|
||||
assert "audio/wav" in response.headers["content-type"]
|
||||
|
||||
|
||||
class TestAsyncOmniSupportedTasks:
|
||||
"""Test that AsyncOmni reports correct supported tasks based on output modalities."""
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_tts_only_no_generate_task(self):
|
||||
"""TTS-only models (audio output, no text) should not include 'generate'."""
|
||||
from vllm_omni.entrypoints.async_omni import AsyncOmni
|
||||
|
||||
omni = AsyncOmni.__new__(AsyncOmni)
|
||||
omni.output_modalities = [None, "audio"]
|
||||
tasks = await omni.get_supported_tasks()
|
||||
assert "generate" not in tasks
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_omni_model_includes_generate(self):
|
||||
"""Models with text output (e.g. Qwen3-Omni) should include 'generate'."""
|
||||
from vllm_omni.entrypoints.async_omni import AsyncOmni
|
||||
|
||||
omni = AsyncOmni.__new__(AsyncOmni)
|
||||
omni.output_modalities = ["text", None, "audio"]
|
||||
tasks = await omni.get_supported_tasks()
|
||||
assert "generate" in tasks
|
||||
@@ -12,7 +12,7 @@ from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from tests.conftest import convert_audio_file_to_text, cosine_similarity_text
|
||||
from tests.conftest import OmniServerParams, convert_audio_file_to_text, cosine_similarity_text
|
||||
from tests.utils import hardware_test
|
||||
|
||||
models = ["Qwen/Qwen2.5-Omni-7B"]
|
||||
@@ -22,7 +22,11 @@ stage_configs = [str(Path(__file__).parent.parent.parent / "e2e" / "stage_config
|
||||
|
||||
example_dir = str(Path(__file__).parent.parent.parent.parent / "examples" / "online_serving" / "qwen2_5_omni")
|
||||
# Create parameter combinations for model and stage config
|
||||
test_params = [(8091, model, stage_config) for model in models for stage_config in stage_configs]
|
||||
test_params = [
|
||||
OmniServerParams(model=model, port=8091, stage_config_path=stage_config)
|
||||
for model in models
|
||||
for stage_config in stage_configs
|
||||
]
|
||||
|
||||
|
||||
def run_cmd(command):
|
||||
|
||||
@@ -12,7 +12,7 @@ from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from tests.conftest import convert_audio_file_to_text, cosine_similarity_text
|
||||
from tests.conftest import OmniServerParams, convert_audio_file_to_text, cosine_similarity_text
|
||||
from tests.utils import hardware_test
|
||||
|
||||
models = ["Qwen/Qwen3-Omni-30B-A3B-Instruct"]
|
||||
@@ -22,7 +22,11 @@ stage_configs = [str(Path(__file__).parent.parent.parent / "e2e" / "stage_config
|
||||
|
||||
example_dir = str(Path(__file__).parent.parent.parent.parent / "examples" / "online_serving" / "qwen3_omni")
|
||||
# Create parameter combinations for model and stage config
|
||||
test_params = [(8091, model, stage_config) for model in models for stage_config in stage_configs]
|
||||
test_params = [
|
||||
OmniServerParams(model=model, port=8091, stage_config_path=stage_config)
|
||||
for model in models
|
||||
for stage_config in stage_configs
|
||||
]
|
||||
|
||||
|
||||
def run_cmd(command):
|
||||
|
||||
@@ -7,16 +7,20 @@ from types import SimpleNamespace
|
||||
import pytest
|
||||
import torch
|
||||
|
||||
from vllm_omni.model_executor.stage_input_processors.chunk_size_utils import (
|
||||
compute_dynamic_initial_chunk_size,
|
||||
max_ic_for_chunk_size,
|
||||
)
|
||||
from vllm_omni.model_executor.stage_input_processors.qwen3_tts import (
|
||||
talker2code2wav,
|
||||
talker2code2wav_async_chunk,
|
||||
)
|
||||
|
||||
_FRAME = [1, 2, 3, 4] # 4-codebook frame
|
||||
_Q = len(_FRAME) # num quantizers
|
||||
_FRAME = [1, 2, 3, 4]
|
||||
_Q = len(_FRAME)
|
||||
|
||||
|
||||
def _req(rid: str, *, finished: bool, initial_codec_chunk_frames: int | None = None):
|
||||
def _req(rid, *, finished, initial_codec_chunk_frames=None):
|
||||
ai = None
|
||||
if initial_codec_chunk_frames is not None:
|
||||
entry = SimpleNamespace(list_data=[initial_codec_chunk_frames])
|
||||
@@ -28,9 +32,10 @@ def _req(rid: str, *, finished: bool, initial_codec_chunk_frames: int | None = N
|
||||
)
|
||||
|
||||
|
||||
def _tm(*, chunk_frames=25, left_context=25, initial_chunk=0):
|
||||
def _tm(*, chunk_frames=25, left_context=25, max_num_seqs=1):
|
||||
return SimpleNamespace(
|
||||
code_prompt_token_ids=defaultdict(list),
|
||||
scheduler_max_num_seqs=max_num_seqs,
|
||||
put_req_chunk=defaultdict(int),
|
||||
request_payload={},
|
||||
connector=SimpleNamespace(
|
||||
@@ -38,16 +43,14 @@ def _tm(*, chunk_frames=25, left_context=25, initial_chunk=0):
|
||||
"extra": {
|
||||
"codec_chunk_frames": chunk_frames,
|
||||
"codec_left_context_frames": left_context,
|
||||
"initial_codec_chunk_frames": initial_chunk,
|
||||
}
|
||||
}
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def _call(tm, rid, *, n_frames, put_req=0, finished=False, req_ic=None):
|
||||
def _call(tm, rid, *, n_frames, finished=False, req_ic=None):
|
||||
tm.code_prompt_token_ids[rid] = [_FRAME[:] for _ in range(n_frames)]
|
||||
tm.put_req_chunk[rid] = put_req
|
||||
return talker2code2wav_async_chunk(
|
||||
transfer_manager=tm,
|
||||
pooling_output={"audio_codes": torch.zeros((0,))},
|
||||
@@ -56,84 +59,68 @@ def _call(tm, rid, *, n_frames, put_req=0, finished=False, req_ic=None):
|
||||
)
|
||||
|
||||
|
||||
def test_does_not_emit_empty_chunk_when_not_finished():
|
||||
def test_empty_returns_none():
|
||||
tm = _tm()
|
||||
request = _req("rid-empty", finished=False)
|
||||
|
||||
payload = talker2code2wav_async_chunk(
|
||||
p = talker2code2wav_async_chunk(
|
||||
transfer_manager=tm,
|
||||
pooling_output={"audio_codes": torch.zeros((0,))},
|
||||
request=request,
|
||||
request=_req("r", finished=False),
|
||||
)
|
||||
|
||||
assert payload is None
|
||||
assert p is None
|
||||
|
||||
|
||||
def test_flushes_tail_when_finished_without_pooler_output():
|
||||
def test_eof_marker_when_finished_empty():
|
||||
tm = _tm()
|
||||
rid = "rid-tail"
|
||||
tm.code_prompt_token_ids[rid] = [_FRAME[:] for _ in range(24)]
|
||||
request = _req(rid, finished=True)
|
||||
|
||||
payload = talker2code2wav_async_chunk(
|
||||
p = talker2code2wav_async_chunk(
|
||||
transfer_manager=tm,
|
||||
pooling_output=None,
|
||||
request=request,
|
||||
request=_req("r", finished=True),
|
||||
is_finished=True,
|
||||
)
|
||||
|
||||
assert payload is not None
|
||||
assert payload["finished"].item() is True
|
||||
assert len(payload["code_predictor_codes"]) == _Q * 24
|
||||
assert p == {"code_predictor_codes": [], "finished": torch.tensor(True, dtype=torch.bool)}
|
||||
|
||||
|
||||
def test_emits_eof_marker_when_finished_with_no_frames():
|
||||
def test_flush_on_finish():
|
||||
tm = _tm()
|
||||
request = _req("rid-eof", finished=True)
|
||||
|
||||
payload = talker2code2wav_async_chunk(
|
||||
tm.code_prompt_token_ids["r"] = [_FRAME[:] for _ in range(24)]
|
||||
p = talker2code2wav_async_chunk(
|
||||
transfer_manager=tm,
|
||||
pooling_output=None,
|
||||
request=request,
|
||||
request=_req("r", finished=True),
|
||||
is_finished=True,
|
||||
)
|
||||
|
||||
assert payload == {
|
||||
"code_predictor_codes": [],
|
||||
"finished": torch.tensor(True, dtype=torch.bool),
|
||||
}
|
||||
assert p is not None
|
||||
assert p["finished"].item() is True
|
||||
assert len(p["code_predictor_codes"]) == _Q * 24
|
||||
|
||||
|
||||
_CASES = [
|
||||
# Normal path (initial=0): emit at chunk_size boundaries
|
||||
((25, 25, 0), (24, 0, False), None),
|
||||
((25, 25, 0), (25, 0, False), (0, 25)),
|
||||
# Initial-chunk phase: hold, first emit, second emit
|
||||
((25, 25, 10), (9, 0, False), None),
|
||||
((25, 25, 10), (10, 0, False), (0, 10)),
|
||||
((25, 25, 10), (20, 1, False), (10, 20)),
|
||||
# Non-divisible: holds at chunk boundary
|
||||
((25, 25, 12), (25, 2, False), None),
|
||||
# Normal phase: offset by initial_coverage (chunk//initial * initial)
|
||||
((25, 25, 10), (45, 2, False), (20, 45)),
|
||||
# Second normal emit (offset must stay stable)
|
||||
((25, 25, 10), (70, 3, False), (25, 50)),
|
||||
# initial >= chunk clamps to chunk_size (behaves as normal)
|
||||
((25, 25, 30), (25, 0, False), (0, 25)),
|
||||
# finished=True flushes IC tail
|
||||
((25, 25, 10), (5, 0, True), (0, 5)),
|
||||
# finished=True flushes non-divisible IC residual
|
||||
((25, 25, 12), (25, 2, True), (24, 25)),
|
||||
# finished=True flushes normal phase tail
|
||||
((25, 25, 10), (30, 2, True), (20, 30)),
|
||||
# Dynamic IC=16, cs=25, initial_coverage=16
|
||||
((25, 25, 0), 24, False, None), # IC phase: 24%16!=0 -> hold
|
||||
((25, 25, 0), 25, False, None), # transition: adjusted=9, hold (no replay)
|
||||
((25, 25, 0), 41, False, (16, 41)), # first normal emit, lc=16
|
||||
# Per-request IC=10, cs=25, initial_coverage=20
|
||||
((25, 25, 10), 9, False, None), # IC: hold
|
||||
((25, 25, 10), 10, False, (0, 10)), # IC: emit at boundary
|
||||
((25, 25, 10), 25, False, None), # transition: hold (no replay)
|
||||
((25, 25, 10), 45, False, (20, 45)), # first normal emit, lc=20
|
||||
((25, 25, 10), 5, True, (0, 5)), # finished flushes IC tail
|
||||
((25, 25, 10), 33, True, (20, 33)), # finished flushes normal tail
|
||||
# IC=8, cs=16: IC divides chunk_size evenly (edge case)
|
||||
((16, 25, 8), 8, False, (0, 8)), # IC: emit
|
||||
((16, 25, 8), 16, False, None), # transition: hold (no replay)
|
||||
((16, 25, 8), 24, False, (8, 24)), # first normal emit, lc=8
|
||||
# Per-request override: IC=15 at n_frames=10 -> 10%15!=0 -> hold
|
||||
((25, 25, 15), 10, False, None),
|
||||
]
|
||||
|
||||
|
||||
@pytest.mark.parametrize("config, state, expected", _CASES)
|
||||
def test_streaming_decoding_with_variable_initial(config, state, expected):
|
||||
chunk_frames, left_context, initial_chunk = config
|
||||
n_frames, put_req, finished = state
|
||||
|
||||
tm = _tm(chunk_frames=chunk_frames, left_context=left_context, initial_chunk=initial_chunk)
|
||||
payload = _call(tm, "r", n_frames=n_frames, put_req=put_req, finished=finished)
|
||||
@pytest.mark.parametrize("config, n_frames, finished, expected", _CASES)
|
||||
def test_streaming_phases(config, n_frames, finished, expected):
|
||||
chunk_frames, left_context, req_ic_val = config
|
||||
tm = _tm(chunk_frames=chunk_frames, left_context=left_context)
|
||||
req_ic = req_ic_val if req_ic_val > 0 else None
|
||||
payload = _call(tm, "r", n_frames=n_frames, finished=finished, req_ic=req_ic)
|
||||
|
||||
if expected is None:
|
||||
assert payload is None
|
||||
@@ -144,22 +131,86 @@ def test_streaming_decoding_with_variable_initial(config, state, expected):
|
||||
assert len(payload["code_predictor_codes"]) == _Q * exp_window
|
||||
|
||||
|
||||
def test_per_request_override_activates_initial_phase():
|
||||
tm = _tm(initial_chunk=0)
|
||||
payload = _call(tm, "r-override", n_frames=10, req_ic=10)
|
||||
assert payload is not None
|
||||
assert payload["left_context_size"] == 0
|
||||
assert len(payload["code_predictor_codes"]) == _Q * 10
|
||||
def test_dynamic_ic_adapts_to_load():
|
||||
# chunk_size=25 -> max_ic=16, steps=[2,4,8,16]
|
||||
tm = _tm(max_num_seqs=8)
|
||||
|
||||
# Low load (1/8) -> IC=2 -> emit at 2
|
||||
p1 = _call(tm, "r", n_frames=2)
|
||||
assert p1 is not None
|
||||
assert len(p1["code_predictor_codes"]) == _Q * 2
|
||||
|
||||
# High load: add 4 others -> active=5/8 -> IC=8 -> emit at 8
|
||||
for i in range(4):
|
||||
tm.code_prompt_token_ids[f"other-{i}"] = [[0]]
|
||||
p2 = _call(tm, "r", n_frames=8)
|
||||
assert p2 is not None
|
||||
assert len(p2["code_predictor_codes"]) == _Q * 8
|
||||
|
||||
# Requests past initial phase still count in load factor
|
||||
tm2 = _tm(max_num_seqs=4)
|
||||
for i in range(3):
|
||||
tm2.code_prompt_token_ids[f"long-{i}"] = [[0]] * 50 # well past cs=25
|
||||
# active=4/4=1.0 -> IC=16
|
||||
p3 = _call(tm2, "new", n_frames=16)
|
||||
assert p3 is not None
|
||||
assert len(p3["code_predictor_codes"]) == _Q * 16
|
||||
|
||||
|
||||
def test_per_request_override_wins_over_stage_config():
|
||||
tm = _tm(initial_chunk=5)
|
||||
payload = _call(tm, "r-override2", n_frames=10, put_req=0, req_ic=15)
|
||||
assert payload is None
|
||||
def test_ic_load_change_mid_request():
|
||||
"""IC stateless: load spike mid-request shifts initial_coverage."""
|
||||
tm = _tm(chunk_frames=25, left_context=25, max_num_seqs=8)
|
||||
|
||||
# Low load -> IC=2 -> emit at frame 2
|
||||
p1 = _call(tm, "r", n_frames=2)
|
||||
assert p1 is not None
|
||||
|
||||
# Spike load: 6 others -> IC=16 -> initial_coverage=16
|
||||
for i in range(6):
|
||||
tm.code_prompt_token_ids[f"other-{i}"] = [[0]] * 10
|
||||
|
||||
# adjusted=25-16=9, 9%25!=0 -> hold
|
||||
assert _call(tm, "r", n_frames=25) is None
|
||||
|
||||
# First normal emit at 16+25=41
|
||||
p3 = _call(tm, "r", n_frames=41)
|
||||
assert p3 is not None
|
||||
assert p3["left_context_size"] == 16
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"active,max_bs,max_ic,expected",
|
||||
[
|
||||
(0, 4, 32, 2), # zero load -> min step
|
||||
(2, 4, 32, 8), # mid load
|
||||
(4, 4, 32, 32), # full load
|
||||
(10, 4, 16, 16), # over capacity, capped
|
||||
(0, 4, 1, 1), # max_ic below min step
|
||||
(0, 0, 16, 2), # zero capacity edge case
|
||||
],
|
||||
)
|
||||
def test_compute_dynamic_initial_chunk_size(active, max_bs, max_ic, expected):
|
||||
assert compute_dynamic_initial_chunk_size(active, max_bs, max_ic) == expected
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"chunk_size,expected",
|
||||
[
|
||||
(25, 16),
|
||||
(50, 32),
|
||||
(70, 64),
|
||||
(8, 4),
|
||||
(4, 2),
|
||||
(2, 1),
|
||||
(1, 1),
|
||||
],
|
||||
)
|
||||
def test_max_ic_for_chunk_size(chunk_size, expected):
|
||||
assert max_ic_for_chunk_size(chunk_size) == expected
|
||||
|
||||
|
||||
def test_first_streaming_chunk_prepends_ref_code_context():
|
||||
tm = _tm(initial_chunk=10)
|
||||
tm = _tm()
|
||||
rid = "r-ref"
|
||||
tm.code_prompt_token_ids[rid] = [_FRAME[:] for _ in range(10)]
|
||||
ref_code = torch.tensor([[9, 9, 9, 9], [8, 8, 8, 8]], dtype=torch.long)
|
||||
@@ -177,7 +228,7 @@ def test_first_streaming_chunk_prepends_ref_code_context():
|
||||
|
||||
|
||||
def test_ref_code_context_only_applies_to_first_streaming_chunk():
|
||||
tm = _tm(initial_chunk=10)
|
||||
tm = _tm()
|
||||
rid = "r-ref2"
|
||||
tm.code_prompt_token_ids[rid] = [_FRAME[:] for _ in range(20)]
|
||||
tm.put_req_chunk[rid] = 1
|
||||
@@ -196,7 +247,7 @@ def test_ref_code_context_only_applies_to_first_streaming_chunk():
|
||||
|
||||
|
||||
def test_ref_code_context_can_be_buffered_before_first_emit():
|
||||
tm = _tm(initial_chunk=10)
|
||||
tm = _tm()
|
||||
rid = "r-ref-buffered"
|
||||
ref_code = torch.tensor([[9, 9, 9, 9], [8, 8, 8, 8]], dtype=torch.long)
|
||||
|
||||
|
||||
@@ -107,8 +107,14 @@ def omni_server(request):
|
||||
print("OmniServer stopped")
|
||||
|
||||
|
||||
def run_benchmark(args: list, test_name: str, flow, dataset_name: str, num_prompt) -> Any:
|
||||
"""Generate synthetic image with random values."""
|
||||
def run_benchmark(
|
||||
args: list,
|
||||
test_name: str,
|
||||
flow,
|
||||
dataset_name: str,
|
||||
num_prompt,
|
||||
) -> Any:
|
||||
"""Run a single benchmark iteration and return the parsed result JSON."""
|
||||
current_dt = datetime.now().strftime("%Y%m%d-%H%M%S")
|
||||
result_filename = f"result_{test_name}_{dataset_name}_{flow}_{num_prompt}_{current_dt}.json"
|
||||
if "--result-filename" in args:
|
||||
@@ -117,10 +123,6 @@ def run_benchmark(args: list, test_name: str, flow, dataset_name: str, num_promp
|
||||
["vllm", "bench", "serve", "--omni"]
|
||||
+ args
|
||||
+ [
|
||||
"--backend",
|
||||
"openai-chat-omni",
|
||||
"--endpoint",
|
||||
"/v1/chat/completions",
|
||||
"--save-result",
|
||||
"--result-dir",
|
||||
os.environ.get("BENCHMARK_DIR", "tests"),
|
||||
@@ -196,7 +198,10 @@ def benchmark_params(request, omni_server):
|
||||
total = len(all_params)
|
||||
print(f"\n Running benchmark {current}/{total} for {test_name}")
|
||||
|
||||
return {"test_name": test_name, "params": all_params[param_index]}
|
||||
return {
|
||||
"test_name": test_name,
|
||||
"params": all_params[param_index],
|
||||
}
|
||||
|
||||
|
||||
def assert_result(result, params, num_prompt):
|
||||
@@ -266,7 +271,11 @@ def test_performance_benchmark(omni_server, benchmark_params):
|
||||
for qps, num_prompt in zip(qps_list, num_prompt_list):
|
||||
args = args + ["--request-rate", str(qps), "--num-prompts", str(num_prompt)]
|
||||
result = run_benchmark(
|
||||
args=args, test_name=test_name, flow=qps, dataset_name=dataset_name, num_prompt=num_prompt
|
||||
args=args,
|
||||
test_name=test_name,
|
||||
flow=qps,
|
||||
dataset_name=dataset_name,
|
||||
num_prompt=num_prompt,
|
||||
)
|
||||
assert_result(result, params, num_prompt=num_prompt)
|
||||
|
||||
@@ -274,6 +283,10 @@ def test_performance_benchmark(omni_server, benchmark_params):
|
||||
for concurrency, num_prompt in zip(max_concurrency_list, num_prompt_list):
|
||||
args = args + ["--max-concurrency", str(concurrency), "--num-prompts", str(num_prompt), "--request-rate", "inf"]
|
||||
result = run_benchmark(
|
||||
args=args, test_name=test_name, flow=concurrency, dataset_name=dataset_name, num_prompt=num_prompt
|
||||
args=args,
|
||||
test_name=test_name,
|
||||
flow=concurrency,
|
||||
dataset_name=dataset_name,
|
||||
num_prompt=num_prompt,
|
||||
)
|
||||
assert_result(result, params, num_prompt=num_prompt)
|
||||
@@ -0,0 +1,99 @@
|
||||
# Stage config for running Qwen3-TTS with 2-stage architecture
|
||||
# Stage 0: Talker (text -> 8-layer RVQ codec codes)
|
||||
# Stage 1: Code2Wav (codec codes -> audio waveform)
|
||||
#
|
||||
# The following config has been verified on 1x H100-80G GPU.
|
||||
async_chunk: true
|
||||
stage_args:
|
||||
- stage_id: 0
|
||||
stage_type: llm
|
||||
runtime:
|
||||
devices: "0"
|
||||
max_batch_size: 4
|
||||
engine_args:
|
||||
model_stage: qwen3_tts
|
||||
model_arch: Qwen3TTSTalkerForConditionalGeneration
|
||||
hf_overrides:
|
||||
architectures: [Qwen3TTSTalkerForConditionalGeneration]
|
||||
worker_type: ar
|
||||
scheduler_cls: vllm_omni.core.sched.omni_ar_scheduler.OmniARScheduler
|
||||
enforce_eager: false
|
||||
trust_remote_code: true
|
||||
async_scheduling: false
|
||||
enable_prefix_caching: false
|
||||
engine_output_type: latent
|
||||
gpu_memory_utilization: 0.3
|
||||
distributed_executor_backend: "mp"
|
||||
max_num_batched_tokens: 512
|
||||
max_model_len: 4096
|
||||
custom_process_next_stage_input_func: vllm_omni.model_executor.stage_input_processors.qwen3_tts.talker2code2wav_async_chunk
|
||||
output_connectors:
|
||||
to_stage_1: connector_of_shared_memory
|
||||
default_sampling_params:
|
||||
temperature: 0.9
|
||||
top_k: 50
|
||||
max_tokens: 4096
|
||||
seed: 42
|
||||
detokenize: false
|
||||
repetition_penalty: 1.05
|
||||
stop_token_ids: [2150]
|
||||
|
||||
- stage_id: 1
|
||||
stage_type: llm
|
||||
runtime:
|
||||
devices: "0"
|
||||
max_batch_size: 4
|
||||
engine_args:
|
||||
model_stage: code2wav
|
||||
model_arch: Qwen3TTSCode2Wav
|
||||
hf_overrides:
|
||||
architectures: [Qwen3TTSCode2Wav]
|
||||
worker_type: generation
|
||||
scheduler_cls: vllm_omni.core.sched.omni_generation_scheduler.OmniGenerationScheduler
|
||||
enforce_eager: true
|
||||
trust_remote_code: true
|
||||
async_scheduling: false
|
||||
enable_prefix_caching: false
|
||||
engine_output_type: audio
|
||||
gpu_memory_utilization: 0.2
|
||||
distributed_executor_backend: "mp"
|
||||
max_num_batched_tokens: 8192
|
||||
max_model_len: 32768
|
||||
engine_input_source: [0]
|
||||
final_output: true
|
||||
final_output_type: audio
|
||||
input_connectors:
|
||||
from_stage_0: connector_of_shared_memory
|
||||
tts_args:
|
||||
max_instructions_length: 500
|
||||
default_sampling_params:
|
||||
temperature: 0.0
|
||||
top_p: 1.0
|
||||
top_k: -1
|
||||
max_tokens: 65536
|
||||
seed: 42
|
||||
detokenize: true
|
||||
repetition_penalty: 1.0
|
||||
|
||||
runtime:
|
||||
enabled: true
|
||||
defaults:
|
||||
window_size: -1
|
||||
max_inflight: 4
|
||||
|
||||
connectors:
|
||||
connector_of_shared_memory:
|
||||
name: SharedMemoryConnector
|
||||
extra:
|
||||
shm_threshold_bytes: 65536
|
||||
codec_streaming: true
|
||||
connector_get_sleep_s: 0.01
|
||||
connector_get_max_wait_first_chunk: 3000
|
||||
connector_get_max_wait: 300
|
||||
codec_chunk_frames: 25
|
||||
codec_left_context_frames: 25
|
||||
|
||||
edges:
|
||||
- from: 0
|
||||
to: 1
|
||||
window_size: -1
|
||||
@@ -8,6 +8,8 @@
|
||||
"benchmark_params": [
|
||||
{
|
||||
"dataset_name": "random",
|
||||
"backend": "openai-chat-omni",
|
||||
"endpoint": "/v1/chat/completions",
|
||||
"num_prompts": [
|
||||
10,
|
||||
40
|
||||
@@ -28,6 +30,8 @@
|
||||
},
|
||||
{
|
||||
"dataset_name": "random-mm",
|
||||
"backend": "openai-chat-omni",
|
||||
"endpoint": "/v1/chat/completions",
|
||||
"num_prompts": [
|
||||
10,
|
||||
40
|
||||
@@ -88,6 +92,8 @@
|
||||
"benchmark_params": [
|
||||
{
|
||||
"dataset_name": "random",
|
||||
"backend": "openai-chat-omni",
|
||||
"endpoint": "/v1/chat/completions",
|
||||
"num_prompts": [
|
||||
10,
|
||||
40
|
||||
@@ -108,6 +114,8 @@
|
||||
},
|
||||
{
|
||||
"dataset_name": "random-mm",
|
||||
"backend": "openai-chat-omni",
|
||||
"endpoint": "/v1/chat/completions",
|
||||
"num_prompts": [
|
||||
10,
|
||||
40
|
||||
@@ -140,5 +148,38 @@
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"test_name": "test_qwen3_tts",
|
||||
"server_params": {
|
||||
"model": "Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice",
|
||||
"stage_config_name": "qwen3_tts.yaml"
|
||||
},
|
||||
"benchmark_params": [
|
||||
{
|
||||
"dataset_name": "random",
|
||||
"backend": "openai-audio-speech",
|
||||
"endpoint": "/v1/audio/speech",
|
||||
"num_prompts": [
|
||||
10,
|
||||
40
|
||||
],
|
||||
"max_concurrency": [
|
||||
1,
|
||||
4
|
||||
],
|
||||
"random_input_len": 100,
|
||||
"random_output_len": 100,
|
||||
"extra_body": {
|
||||
"voice": "Vivian",
|
||||
"language": "English"
|
||||
},
|
||||
"percentile-metrics": "ttft,e2el,audio_rtf,audio_ttfp,audio_duration",
|
||||
"baseline": {
|
||||
"mean_audio_ttfp_ms": 100000,
|
||||
"mean_audio_rtf": 100000
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
@@ -20,6 +20,8 @@ from typing_extensions import ParamSpec
|
||||
from vllm.platforms import current_platform
|
||||
from vllm.utils.torch_utils import cuda_device_count_stateless
|
||||
|
||||
from vllm_omni.platforms import current_omni_platform
|
||||
|
||||
_P = ParamSpec("_P")
|
||||
|
||||
if current_platform.is_rocm():
|
||||
@@ -441,27 +443,11 @@ def npu_marks(*, res: str, num_cards: int):
|
||||
return [mark for mark in [test_platform, test_resource, test_distributed] if mark is not None]
|
||||
|
||||
|
||||
def hardware_test(*, res: dict[str, str], num_cards: int | dict[str, int] = 1):
|
||||
def hardware_marks(*, res: dict[str, str], num_cards: int | dict[str, int] = 1):
|
||||
"""
|
||||
Decorate a test for multiple hardware platforms with a single call.
|
||||
Automatically wraps the test with @create_new_process_for_each_test() for distributed tests.
|
||||
|
||||
Args:
|
||||
res: Mapping from platform to resource type. Supported platforms/resources:
|
||||
- cuda: L4, H100
|
||||
- rocm: MI325
|
||||
- npu: A2, A3
|
||||
num_cards: Number of cards required. Can be:
|
||||
- int: same card count for all platforms (default: 1)
|
||||
- dict: per-platform card count, e.g., {"cuda": 2, "rocm": 2}
|
||||
|
||||
Example:
|
||||
@hardware_test(
|
||||
res={"cuda": "L4", "rocm": "MI325", "npu": "A2"},
|
||||
num_cards={"cuda": 2, "rocm": 2, "npu": 2},
|
||||
)
|
||||
def test_multi_platform():
|
||||
...
|
||||
Get a collection of pytest marks to apply for `@hardware_test`,
|
||||
including CUDA, ROCm, and NPU,
|
||||
based on the specified platforms and resources.
|
||||
"""
|
||||
# Validate platforms
|
||||
# Don't validate platform details in this decorator
|
||||
@@ -484,7 +470,7 @@ def hardware_test(*, res: dict[str, str], num_cards: int | dict[str, int] = 1):
|
||||
num_cards_dict[platform] = 1
|
||||
|
||||
# Collect marks from all platforms
|
||||
all_marks: list[Callable[[Callable[_P, None]], Callable[_P, None]]] = []
|
||||
all_marks: list[pytest.MarkDecorator] = []
|
||||
for platform, resource in res.items():
|
||||
cards = num_cards_dict[platform]
|
||||
if platform == "cuda" or platform == "rocm":
|
||||
@@ -494,6 +480,32 @@ def hardware_test(*, res: dict[str, str], num_cards: int | dict[str, int] = 1):
|
||||
else:
|
||||
raise ValueError(f"Unsupported platform: {platform}")
|
||||
all_marks.extend(marks)
|
||||
return all_marks
|
||||
|
||||
|
||||
def hardware_test(*, res: dict[str, str], num_cards: int | dict[str, int] = 1):
|
||||
"""
|
||||
Decorate a test for multiple hardware platforms with a single call.
|
||||
Automatically wraps the test with @create_new_process_for_each_test() for distributed tests.
|
||||
|
||||
Args:
|
||||
res: Mapping from platform to resource type. Supported platforms/resources:
|
||||
- cuda: L4, H100
|
||||
- rocm: MI325
|
||||
- npu: A2, A3
|
||||
num_cards: Number of cards required. Can be:
|
||||
- int: same card count for all platforms (default: 1)
|
||||
- dict: per-platform card count, e.g., {"cuda": 2, "rocm": 2}
|
||||
|
||||
Example:
|
||||
@hardware_test(
|
||||
res={"cuda": "L4", "rocm": "MI325", "npu": "A2"},
|
||||
num_cards={"cuda": 2, "rocm": 2, "npu": 2},
|
||||
)
|
||||
def test_multi_platform():
|
||||
...
|
||||
"""
|
||||
all_marks = hardware_marks(res=res, num_cards=num_cards)
|
||||
|
||||
def wrapper(f: Callable[_P, None]) -> Callable[_P, None]:
|
||||
func = f
|
||||
@@ -504,8 +516,8 @@ def hardware_test(*, res: dict[str, str], num_cards: int | dict[str, int] = 1):
|
||||
return wrapper
|
||||
|
||||
|
||||
class GPUMemoryMonitor:
|
||||
"""Poll global device memory usage via CUDA APIs."""
|
||||
class DeviceMemoryMonitor:
|
||||
"""Poll global device memory usage."""
|
||||
|
||||
def __init__(self, device_index: int, interval: float = 0.05):
|
||||
self.device_index = device_index
|
||||
@@ -518,8 +530,8 @@ class GPUMemoryMonitor:
|
||||
def monitor_loop() -> None:
|
||||
while not self._stop_event.is_set():
|
||||
try:
|
||||
with torch.cuda.device(self.device_index):
|
||||
free_bytes, total_bytes = torch.cuda.mem_get_info()
|
||||
with current_omni_platform.device(self.device_index):
|
||||
free_bytes, total_bytes = current_omni_platform.mem_get_info()
|
||||
used_mb = (total_bytes - free_bytes) / (1024**2)
|
||||
self._peak_used_mb = max(self._peak_used_mb, used_mb)
|
||||
except Exception:
|
||||
@@ -537,8 +549,8 @@ class GPUMemoryMonitor:
|
||||
|
||||
@property
|
||||
def peak_used_mb(self) -> float:
|
||||
fallback_alloc = torch.cuda.max_memory_allocated(device=self.device_index) / (1024**2)
|
||||
fallback_reserved = torch.cuda.max_memory_reserved(device=self.device_index) / (1024**2)
|
||||
fallback_alloc = current_omni_platform.max_memory_allocated(device=self.device_index) / (1024**2)
|
||||
fallback_reserved = current_omni_platform.max_memory_reserved(device=self.device_index) / (1024**2)
|
||||
return max(self._peak_used_mb, fallback_alloc, fallback_reserved)
|
||||
|
||||
def __del__(self):
|
||||
|
||||
@@ -208,10 +208,11 @@ async def async_request_openai_chat_omni_completions(
|
||||
async def async_request_openai_audio_speech(
|
||||
request_func_input: RequestFuncInput, session: aiohttp.ClientSession, pbar: tqdm | None = None
|
||||
) -> MixRequestFuncOutput:
|
||||
"""Non-streaming request to /v1/audio/speech endpoint.
|
||||
"""Streaming request to /v1/audio/speech endpoint.
|
||||
|
||||
The endpoint returns raw audio bytes (e.g. WAV). Pass voice, instructions,
|
||||
and other TTS-specific fields via ``extra_body``.
|
||||
Sends ``stream=true`` with ``response_format=pcm`` so the server returns
|
||||
raw PCM chunks as they are decoded. This allows measuring TTFP (time to
|
||||
first audio packet) separately from E2EL.
|
||||
"""
|
||||
api_url = request_func_input.api_url
|
||||
_validate_api_url(api_url, "OpenAI Audio Speech API", "audio/speech")
|
||||
@@ -219,16 +220,11 @@ async def async_request_openai_audio_speech(
|
||||
payload = {
|
||||
"model": request_func_input.model_name if request_func_input.model_name else request_func_input.model,
|
||||
"input": request_func_input.prompt,
|
||||
"stream": True,
|
||||
"response_format": "pcm",
|
||||
}
|
||||
_update_payload_common(payload, request_func_input)
|
||||
|
||||
response_format = payload.get("response_format", "wav")
|
||||
if response_format == "pcm":
|
||||
raise ValueError(
|
||||
"pcm response format is not supported yet. \
|
||||
Please use other formats like wav, mp3, etc. instead."
|
||||
)
|
||||
|
||||
headers = {
|
||||
"Content-Type": "application/json",
|
||||
"Authorization": f"Bearer {os.environ.get('OPENAI_API_KEY')}",
|
||||
@@ -238,41 +234,38 @@ async def async_request_openai_audio_speech(
|
||||
output = MixRequestFuncOutput()
|
||||
output.prompt_len = request_func_input.prompt_len
|
||||
|
||||
# PCM format: 16-bit signed, 24 kHz, mono
|
||||
sample_rate = 24000
|
||||
sample_width = 2 # 16-bit = 2 bytes
|
||||
channels = 1
|
||||
|
||||
st = time.perf_counter()
|
||||
output.start_time = st
|
||||
total_pcm_bytes = 0
|
||||
try:
|
||||
async with session.post(url=api_url, json=payload, headers=headers) as response:
|
||||
if response.status == 200:
|
||||
audio_bytes = await response.read()
|
||||
async for chunk in response.content.iter_any():
|
||||
if not chunk:
|
||||
continue
|
||||
timestamp = time.perf_counter()
|
||||
if output.audio_ttfp == 0.0:
|
||||
output.audio_ttfp = timestamp - st
|
||||
output.ttft = output.audio_ttfp
|
||||
total_pcm_bytes += len(chunk)
|
||||
|
||||
end_time = time.perf_counter()
|
||||
output.latency = end_time - st
|
||||
# ttft = latency since this is a non-streaming request
|
||||
# hence there is no distinction between first and last token/audio
|
||||
output.ttft = output.latency
|
||||
output.audio_ttfp = output.latency
|
||||
|
||||
try:
|
||||
audio_segment = AudioSegment.from_file(io.BytesIO(audio_bytes))
|
||||
output.audio_duration = len(audio_segment) / 1000.0
|
||||
frame_width = audio_segment.frame_width
|
||||
if frame_width > 0:
|
||||
output.audio_frames = len(audio_segment.raw_data) // frame_width
|
||||
else:
|
||||
output.audio_frames = 0
|
||||
logger.warning("Audio frame width is zero")
|
||||
if output.audio_duration > 0:
|
||||
# rtf = audio_generate_time / audio_duration and
|
||||
# audio_generate_time = latency since this is a non-streaming request
|
||||
# so the time to receive last portion of audio is the latency
|
||||
output.audio_rtf = output.latency / output.audio_duration
|
||||
else:
|
||||
output.audio_rtf = 0
|
||||
logger.warning("Audio duration is zero")
|
||||
output.success = True
|
||||
except Exception as e:
|
||||
output.success = False
|
||||
output.error = f"Failed to parse audio response: {e}"
|
||||
logger.error(f"ERROR: Failed to parse audio response: {e}")
|
||||
total_samples = total_pcm_bytes // (sample_width * channels)
|
||||
output.audio_duration = total_samples / sample_rate
|
||||
output.audio_frames = total_samples
|
||||
if output.audio_duration > 0:
|
||||
output.audio_rtf = output.latency / output.audio_duration
|
||||
else:
|
||||
output.audio_rtf = 0
|
||||
logger.warning("Audio duration is zero")
|
||||
output.success = True
|
||||
else:
|
||||
output.error = response.reason or ""
|
||||
output.success = False
|
||||
|
||||
@@ -43,6 +43,9 @@ class DiffusionParallelConfig:
|
||||
tensor_parallel_size: int = 1
|
||||
"""Number of tensor parallel groups."""
|
||||
|
||||
enable_expert_parallel: bool = False
|
||||
"""Enable expert parallelism for MoE layers (TP is still used for non-MoE layers)."""
|
||||
|
||||
sequence_parallel_size: int | None = None
|
||||
"""Number of sequence parallel groups. sequence_parallel_size = ring_degree * ulysses_degree"""
|
||||
|
||||
@@ -457,6 +460,17 @@ class OmniDiffusionConfig:
|
||||
quantization: str | None = None
|
||||
quantization_config: "DiffusionQuantizationConfig | dict[str, Any] | None" = None
|
||||
|
||||
@property
|
||||
def is_moe(self) -> bool:
|
||||
num_experts = self.tf_model_config.get("num_experts", None)
|
||||
if isinstance(num_experts, int):
|
||||
return num_experts > 1
|
||||
|
||||
if isinstance(num_experts, (list, tuple)):
|
||||
return any(isinstance(n, int) and n > 1 for n in num_experts)
|
||||
|
||||
return False
|
||||
|
||||
def settle_port(self, port: int, port_inc: int = 42, max_attempts: int = 100) -> int:
|
||||
"""
|
||||
Find an available port with retry logic.
|
||||
|
||||
@@ -35,6 +35,8 @@ from vllm.distributed.parallel_state import get_tensor_model_parallel_world_size
|
||||
from vllm.logger import init_logger
|
||||
|
||||
from vllm_omni.diffusion import envs
|
||||
from vllm_omni.diffusion.data import OmniDiffusionConfig
|
||||
from vllm_omni.diffusion.forward_context import get_forward_context
|
||||
from vllm_omni.platforms import current_omni_platform
|
||||
|
||||
from .group_coordinator import (
|
||||
@@ -51,7 +53,7 @@ logger = init_logger(__name__)
|
||||
|
||||
|
||||
_WORLD: GroupCoordinator | None = None
|
||||
# get _TP from vllm.distributed.parallel_state
|
||||
# get _TP&_EP from vllm.distributed.parallel_state
|
||||
_SP: SequenceParallelGroupCoordinator | None = None
|
||||
_PP: PipelineGroupCoordinator | None = None
|
||||
_CFG: GroupCoordinator | None = None
|
||||
@@ -188,6 +190,7 @@ class RankGenerator:
|
||||
self.fs = fs
|
||||
self.rank_offset = rank_offset
|
||||
self.world_size = tp * sp * pp * cfg * dp
|
||||
self.ep = tp * sp * cfg * dp # EP level exclude PP
|
||||
|
||||
self.name_to_size = {
|
||||
"tp": self.tp,
|
||||
@@ -250,6 +253,15 @@ class RankGenerator:
|
||||
ranks.append(group)
|
||||
return ranks
|
||||
|
||||
if token == "ep":
|
||||
ranks = []
|
||||
num_pp_stages = self.pp
|
||||
for i in range(num_pp_stages):
|
||||
start = i * self.ep + self.rank_offset
|
||||
end = start + self.ep
|
||||
ranks.append(list(range(start, end)))
|
||||
return ranks
|
||||
|
||||
mask = self.get_mask(self.order, token)
|
||||
ranks = generate_masked_orthogonal_rank_groups(self.world_size, self.ordered_size, mask)
|
||||
if self.rank_offset > 0:
|
||||
@@ -474,6 +486,7 @@ def init_model_parallel_group(
|
||||
"data",
|
||||
"pipeline",
|
||||
"tensor",
|
||||
"expert",
|
||||
"sequence",
|
||||
"classifier_free_guidance",
|
||||
"fully_shard",
|
||||
@@ -670,6 +683,7 @@ def initialize_model_parallel(
|
||||
pipeline_parallel_size: int = 1,
|
||||
fully_shard_degree: int = 1,
|
||||
hsdp_replicate_size: int = 1,
|
||||
enable_expert_parallel: bool = False,
|
||||
backend: str | None = None,
|
||||
) -> None:
|
||||
if backend is None:
|
||||
@@ -837,6 +851,18 @@ def initialize_model_parallel(
|
||||
parallel_mode="fully_shard",
|
||||
)
|
||||
|
||||
if enable_expert_parallel:
|
||||
od_config: OmniDiffusionConfig | None = get_forward_context().omni_diffusion_config
|
||||
if od_config and od_config.is_moe:
|
||||
vllm_parallel_state._EP = init_model_parallel_group(
|
||||
group_ranks=rank_generator.get_ranks("ep"),
|
||||
local_rank=get_world_group().local_rank,
|
||||
backend=backend,
|
||||
parallel_mode="expert",
|
||||
)
|
||||
else:
|
||||
raise RuntimeError("Expert parallelism enabled for a non-MoE model ")
|
||||
|
||||
init_dit_group(dit_parallel_size, backend)
|
||||
|
||||
|
||||
@@ -861,6 +887,10 @@ def destroy_model_parallel():
|
||||
vllm_parallel_state._TP.destroy()
|
||||
vllm_parallel_state._TP = None
|
||||
|
||||
if vllm_parallel_state._EP:
|
||||
vllm_parallel_state._EP.destroy()
|
||||
vllm_parallel_state._EP = None
|
||||
|
||||
global _PP
|
||||
if _PP:
|
||||
_PP.destroy()
|
||||
|
||||
@@ -31,15 +31,35 @@ class ForwardContext:
|
||||
# Original sequence length before padding (for removing padding in gather)
|
||||
sp_original_seq_len: int | None = None
|
||||
|
||||
# SP active scope tracking
|
||||
# Set by registry when _sp_plan hooks are applied.
|
||||
# When True, sp_active is determined by _sp_shard_depth (for _sp_plan hooks)
|
||||
# When False, sp_active defaults to True when sequence_parallel_size > 1 (for manual SP, standalone tests, etc.)
|
||||
sp_plan_hooks_applied: bool = False
|
||||
# SP active scope tracking within the _sp_plan hook mechanism.
|
||||
# Tracks the depth of SP sharding - incremented on shard, decremented on gather
|
||||
# Used by attention layers to determine if SP communication should be enabled
|
||||
_sp_shard_depth: int = 0
|
||||
|
||||
@property
|
||||
def sp_active(self) -> bool:
|
||||
"""Returns True when inside an SP sharded region (between shard and gather)."""
|
||||
return self._sp_shard_depth > 0
|
||||
"""Returns True when SP attention parallelism should be enabled.
|
||||
|
||||
- If _sp_plan hooks are applied: use _sp_shard_depth (0 = outside sharded region).
|
||||
- If _sp_plan hooks are NOT applied: default to True when sequence_parallel_size > 1,
|
||||
since _sp_shard_depth is only meaningful within the _sp_plan hook mechanism.
|
||||
"""
|
||||
if self.sp_plan_hooks_applied:
|
||||
return self._sp_shard_depth > 0
|
||||
# No _sp_plan: assume SP active when configured (manual SP, standalone tests)
|
||||
if self.omni_diffusion_config is None:
|
||||
raise ValueError(
|
||||
"omni_diffusion_config is not set when checking sp_active! "
|
||||
"This usually means set_forward_context() was not called. "
|
||||
"Please call with set_forward_context(omni_diffusion_config=...)."
|
||||
)
|
||||
|
||||
sp_size = self.omni_diffusion_config.parallel_config.sequence_parallel_size
|
||||
return sp_size is not None and sp_size > 1
|
||||
|
||||
def __post_init__(self):
|
||||
pass
|
||||
|
||||
@@ -0,0 +1,17 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||
"""Flux2 diffusion model components."""
|
||||
|
||||
from vllm_omni.diffusion.models.flux2.flux2_transformer import (
|
||||
Flux2Transformer2DModel,
|
||||
)
|
||||
from vllm_omni.diffusion.models.flux2.pipeline_flux2 import (
|
||||
Flux2Pipeline,
|
||||
get_flux2_post_process_func,
|
||||
)
|
||||
|
||||
__all__ = [
|
||||
"Flux2Pipeline",
|
||||
"Flux2Transformer2DModel",
|
||||
"get_flux2_post_process_func",
|
||||
]
|
||||
@@ -0,0 +1,770 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||
from collections.abc import Iterable
|
||||
from types import SimpleNamespace
|
||||
from typing import Any
|
||||
|
||||
import torch
|
||||
from diffusers.models.embeddings import (
|
||||
TimestepEmbedding,
|
||||
Timesteps,
|
||||
get_1d_rotary_pos_embed,
|
||||
)
|
||||
from diffusers.models.modeling_outputs import Transformer2DModelOutput
|
||||
from diffusers.models.normalization import AdaLayerNormContinuous
|
||||
from diffusers.utils import is_torch_npu_available
|
||||
from torch import nn
|
||||
from vllm.model_executor.layers.layernorm import RMSNorm
|
||||
from vllm.model_executor.layers.linear import (
|
||||
ColumnParallelLinear,
|
||||
MergedColumnParallelLinear,
|
||||
QKVParallelLinear,
|
||||
RowParallelLinear,
|
||||
)
|
||||
from vllm.model_executor.model_loader.weight_utils import default_weight_loader
|
||||
|
||||
from vllm_omni.diffusion.attention.backends.abstract import AttentionMetadata
|
||||
from vllm_omni.diffusion.attention.layer import Attention
|
||||
from vllm_omni.diffusion.layers.rope import RotaryEmbedding
|
||||
|
||||
|
||||
class Flux2SwiGLU(nn.Module):
|
||||
"""SwiGLU activation used by Flux2."""
|
||||
|
||||
def __init__(self):
|
||||
super().__init__()
|
||||
self.gate_fn = nn.SiLU()
|
||||
|
||||
def forward(self, x: torch.Tensor) -> torch.Tensor:
|
||||
x1, x2 = x.chunk(2, dim=-1)
|
||||
return self.gate_fn(x1) * x2
|
||||
|
||||
|
||||
class Flux2FeedForward(nn.Module):
|
||||
def __init__(
|
||||
self,
|
||||
dim: int,
|
||||
dim_out: int | None = None,
|
||||
mult: float = 3.0,
|
||||
inner_dim: int | None = None,
|
||||
bias: bool = False,
|
||||
):
|
||||
super().__init__()
|
||||
if inner_dim is None:
|
||||
inner_dim = int(dim * mult)
|
||||
dim_out = dim_out or dim
|
||||
|
||||
self.linear_in = MergedColumnParallelLinear(
|
||||
dim,
|
||||
[inner_dim, inner_dim],
|
||||
bias=bias,
|
||||
return_bias=False,
|
||||
)
|
||||
self.act_fn = Flux2SwiGLU()
|
||||
self.linear_out = RowParallelLinear(
|
||||
inner_dim,
|
||||
dim_out,
|
||||
bias=bias,
|
||||
input_is_parallel=True,
|
||||
return_bias=False,
|
||||
)
|
||||
|
||||
def forward(self, x: torch.Tensor) -> torch.Tensor:
|
||||
x = self.linear_in(x)
|
||||
x = self.act_fn(x)
|
||||
return self.linear_out(x)
|
||||
|
||||
|
||||
class Flux2Attention(nn.Module):
|
||||
def __init__(
|
||||
self,
|
||||
query_dim: int,
|
||||
heads: int = 8,
|
||||
dim_head: int = 64,
|
||||
dropout: float = 0.0,
|
||||
bias: bool = False,
|
||||
added_kv_proj_dim: int | None = None,
|
||||
added_proj_bias: bool | None = True,
|
||||
out_bias: bool = True,
|
||||
eps: float = 1e-5,
|
||||
out_dim: int = None,
|
||||
elementwise_affine: bool = True,
|
||||
):
|
||||
super().__init__()
|
||||
self.head_dim = dim_head
|
||||
self.inner_dim = out_dim if out_dim is not None else dim_head * heads
|
||||
self.query_dim = query_dim
|
||||
self.out_dim = out_dim if out_dim is not None else query_dim
|
||||
self.heads = out_dim // dim_head if out_dim is not None else heads
|
||||
self.dropout = dropout
|
||||
self.added_kv_proj_dim = added_kv_proj_dim
|
||||
|
||||
self.to_qkv = QKVParallelLinear(
|
||||
hidden_size=query_dim,
|
||||
head_size=self.head_dim,
|
||||
total_num_heads=self.heads,
|
||||
bias=bias,
|
||||
)
|
||||
self.query_num_heads = self.to_qkv.num_heads
|
||||
self.kv_num_heads = self.to_qkv.num_kv_heads
|
||||
|
||||
self.norm_q = RMSNorm(dim_head, eps=eps)
|
||||
self.norm_k = RMSNorm(dim_head, eps=eps)
|
||||
|
||||
self.to_out = nn.ModuleList(
|
||||
[
|
||||
RowParallelLinear(
|
||||
self.inner_dim,
|
||||
self.out_dim,
|
||||
bias=out_bias,
|
||||
input_is_parallel=True,
|
||||
return_bias=False,
|
||||
),
|
||||
nn.Dropout(dropout),
|
||||
]
|
||||
)
|
||||
|
||||
if added_kv_proj_dim is not None:
|
||||
self.norm_added_q = RMSNorm(dim_head, eps=eps)
|
||||
self.norm_added_k = RMSNorm(dim_head, eps=eps)
|
||||
self.add_kv_proj = QKVParallelLinear(
|
||||
hidden_size=added_kv_proj_dim,
|
||||
head_size=self.head_dim,
|
||||
total_num_heads=self.heads,
|
||||
bias=added_proj_bias,
|
||||
)
|
||||
self.add_query_num_heads = self.add_kv_proj.num_heads
|
||||
self.add_kv_num_heads = self.add_kv_proj.num_kv_heads
|
||||
self.to_add_out = RowParallelLinear(
|
||||
self.inner_dim,
|
||||
query_dim,
|
||||
bias=out_bias,
|
||||
input_is_parallel=True,
|
||||
return_bias=False,
|
||||
)
|
||||
|
||||
self.rope = RotaryEmbedding(is_neox_style=False)
|
||||
self.attn = Attention(
|
||||
num_heads=self.query_num_heads,
|
||||
head_size=self.head_dim,
|
||||
softmax_scale=1.0 / (self.head_dim**0.5),
|
||||
causal=False,
|
||||
num_kv_heads=self.kv_num_heads,
|
||||
)
|
||||
|
||||
def forward(
|
||||
self,
|
||||
hidden_states: torch.Tensor,
|
||||
encoder_hidden_states: torch.Tensor | None = None,
|
||||
attention_mask: torch.Tensor | None = None,
|
||||
image_rotary_emb: tuple[torch.Tensor, torch.Tensor] | None = None,
|
||||
**kwargs,
|
||||
) -> torch.Tensor | tuple[torch.Tensor, torch.Tensor]:
|
||||
qkv, _ = self.to_qkv(hidden_states)
|
||||
query, key, value = qkv.chunk(3, dim=-1)
|
||||
|
||||
encoder_query = encoder_key = encoder_value = None
|
||||
if encoder_hidden_states is not None and self.added_kv_proj_dim is not None:
|
||||
encoder_qkv, _ = self.add_kv_proj(encoder_hidden_states)
|
||||
encoder_query, encoder_key, encoder_value = encoder_qkv.chunk(3, dim=-1)
|
||||
|
||||
query = query.unflatten(-1, (self.query_num_heads, -1))
|
||||
key = key.unflatten(-1, (self.kv_num_heads, -1))
|
||||
value = value.unflatten(-1, (self.kv_num_heads, -1))
|
||||
|
||||
query = self.norm_q(query)
|
||||
key = self.norm_k(key)
|
||||
|
||||
if encoder_hidden_states is not None and self.added_kv_proj_dim is not None:
|
||||
encoder_query = encoder_query.unflatten(-1, (self.add_query_num_heads, -1))
|
||||
encoder_key = encoder_key.unflatten(-1, (self.add_kv_num_heads, -1))
|
||||
encoder_value = encoder_value.unflatten(-1, (self.add_kv_num_heads, -1))
|
||||
|
||||
encoder_query = self.norm_added_q(encoder_query)
|
||||
encoder_key = self.norm_added_k(encoder_key)
|
||||
|
||||
query = torch.cat([encoder_query, query], dim=1)
|
||||
key = torch.cat([encoder_key, key], dim=1)
|
||||
value = torch.cat([encoder_value, value], dim=1)
|
||||
|
||||
if image_rotary_emb is not None:
|
||||
cos, sin = image_rotary_emb
|
||||
cos = cos.to(query.dtype)
|
||||
sin = sin.to(query.dtype)
|
||||
query = self.rope(query, cos, sin)
|
||||
key = self.rope(key, cos, sin)
|
||||
|
||||
attn_metadata = None
|
||||
if attention_mask is not None:
|
||||
if attention_mask.dim() == 3:
|
||||
attention_mask = attention_mask.unsqueeze(1)
|
||||
attn_metadata = AttentionMetadata(attn_mask=attention_mask)
|
||||
|
||||
hidden_states = self.attn(query, key, value, attn_metadata)
|
||||
hidden_states = hidden_states.flatten(2, 3).to(query.dtype)
|
||||
|
||||
if encoder_hidden_states is not None:
|
||||
context_len = encoder_hidden_states.shape[1]
|
||||
encoder_hidden_states, hidden_states = hidden_states.split_with_sizes(
|
||||
[context_len, hidden_states.shape[1] - context_len],
|
||||
dim=1,
|
||||
)
|
||||
encoder_hidden_states = self.to_add_out(encoder_hidden_states)
|
||||
|
||||
hidden_states = self.to_out[0](hidden_states)
|
||||
hidden_states = self.to_out[1](hidden_states)
|
||||
|
||||
if encoder_hidden_states is not None:
|
||||
return hidden_states, encoder_hidden_states
|
||||
return hidden_states
|
||||
|
||||
|
||||
class Flux2ParallelSelfAttention(nn.Module):
|
||||
"""
|
||||
Parallel attention block that fuses QKV projections with MLP input projections.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
query_dim: int,
|
||||
heads: int = 8,
|
||||
dim_head: int = 64,
|
||||
dropout: float = 0.0,
|
||||
bias: bool = False,
|
||||
out_bias: bool = True,
|
||||
eps: float = 1e-5,
|
||||
out_dim: int = None,
|
||||
elementwise_affine: bool = True,
|
||||
mlp_ratio: float = 4.0,
|
||||
mlp_mult_factor: int = 2,
|
||||
):
|
||||
super().__init__()
|
||||
self.head_dim = dim_head
|
||||
self.inner_dim = out_dim if out_dim is not None else dim_head * heads
|
||||
self.query_dim = query_dim
|
||||
self.out_dim = out_dim if out_dim is not None else query_dim
|
||||
self.heads = out_dim // dim_head if out_dim is not None else heads
|
||||
self.dropout = dropout
|
||||
|
||||
self.mlp_ratio = mlp_ratio
|
||||
self.mlp_hidden_dim = int(query_dim * self.mlp_ratio)
|
||||
self.mlp_mult_factor = mlp_mult_factor
|
||||
|
||||
self.to_qkv_mlp_proj = ColumnParallelLinear(
|
||||
self.query_dim,
|
||||
self.inner_dim * 3 + self.mlp_hidden_dim * self.mlp_mult_factor,
|
||||
bias=bias,
|
||||
gather_output=True,
|
||||
)
|
||||
self.mlp_act_fn = Flux2SwiGLU()
|
||||
|
||||
self.norm_q = RMSNorm(dim_head, eps=eps)
|
||||
self.norm_k = RMSNorm(dim_head, eps=eps)
|
||||
|
||||
self.to_out = ColumnParallelLinear(
|
||||
self.inner_dim + self.mlp_hidden_dim,
|
||||
self.out_dim,
|
||||
bias=out_bias,
|
||||
gather_output=True,
|
||||
)
|
||||
self.rope = RotaryEmbedding(is_neox_style=False)
|
||||
self.attn = Attention(
|
||||
num_heads=self.heads,
|
||||
head_size=self.head_dim,
|
||||
softmax_scale=1.0 / (self.head_dim**0.5),
|
||||
causal=False,
|
||||
)
|
||||
|
||||
def forward(
|
||||
self,
|
||||
hidden_states: torch.Tensor,
|
||||
attention_mask: torch.Tensor | None = None,
|
||||
image_rotary_emb: tuple[torch.Tensor, torch.Tensor] | None = None,
|
||||
**kwargs,
|
||||
) -> torch.Tensor:
|
||||
hidden_states, _ = self.to_qkv_mlp_proj(hidden_states)
|
||||
qkv, mlp_hidden_states = torch.split(
|
||||
hidden_states,
|
||||
[3 * self.inner_dim, self.mlp_hidden_dim * self.mlp_mult_factor],
|
||||
dim=-1,
|
||||
)
|
||||
|
||||
query, key, value = qkv.chunk(3, dim=-1)
|
||||
query = query.unflatten(-1, (self.heads, -1))
|
||||
key = key.unflatten(-1, (self.heads, -1))
|
||||
value = value.unflatten(-1, (self.heads, -1))
|
||||
|
||||
query = self.norm_q(query)
|
||||
key = self.norm_k(key)
|
||||
|
||||
if image_rotary_emb is not None:
|
||||
cos, sin = image_rotary_emb
|
||||
cos = cos.to(query.dtype)
|
||||
sin = sin.to(query.dtype)
|
||||
query = self.rope(query, cos, sin)
|
||||
key = self.rope(key, cos, sin)
|
||||
|
||||
attn_metadata = None
|
||||
if attention_mask is not None:
|
||||
if attention_mask.dim() == 3:
|
||||
attention_mask = attention_mask.unsqueeze(1)
|
||||
attn_metadata = AttentionMetadata(attn_mask=attention_mask)
|
||||
|
||||
attn_output = self.attn(query, key, value, attn_metadata)
|
||||
attn_output = attn_output.flatten(2, 3).to(query.dtype)
|
||||
|
||||
mlp_hidden_states = self.mlp_act_fn(mlp_hidden_states)
|
||||
hidden_states = torch.cat([attn_output, mlp_hidden_states], dim=-1)
|
||||
hidden_states, _ = self.to_out(hidden_states)
|
||||
return hidden_states
|
||||
|
||||
|
||||
class Flux2SingleTransformerBlock(nn.Module):
|
||||
def __init__(
|
||||
self,
|
||||
dim: int,
|
||||
num_attention_heads: int,
|
||||
attention_head_dim: int,
|
||||
mlp_ratio: float = 3.0,
|
||||
eps: float = 1e-6,
|
||||
bias: bool = False,
|
||||
):
|
||||
super().__init__()
|
||||
self.norm = nn.LayerNorm(dim, elementwise_affine=False, eps=eps)
|
||||
self.attn = Flux2ParallelSelfAttention(
|
||||
query_dim=dim,
|
||||
dim_head=attention_head_dim,
|
||||
heads=num_attention_heads,
|
||||
out_dim=dim,
|
||||
bias=bias,
|
||||
out_bias=bias,
|
||||
eps=eps,
|
||||
mlp_ratio=mlp_ratio,
|
||||
mlp_mult_factor=2,
|
||||
)
|
||||
|
||||
def forward(
|
||||
self,
|
||||
hidden_states: torch.Tensor,
|
||||
encoder_hidden_states: torch.Tensor | None,
|
||||
temb_mod_params: tuple[torch.Tensor, torch.Tensor, torch.Tensor],
|
||||
image_rotary_emb: tuple[torch.Tensor, torch.Tensor] | None = None,
|
||||
joint_attention_kwargs: dict[str, Any] | None = None,
|
||||
split_hidden_states: bool = False,
|
||||
text_seq_len: int | None = None,
|
||||
) -> torch.Tensor | tuple[torch.Tensor, torch.Tensor]:
|
||||
if encoder_hidden_states is not None:
|
||||
text_seq_len = encoder_hidden_states.shape[1]
|
||||
hidden_states = torch.cat([encoder_hidden_states, hidden_states], dim=1)
|
||||
|
||||
mod_shift, mod_scale, mod_gate = temb_mod_params
|
||||
|
||||
norm_hidden_states = self.norm(hidden_states)
|
||||
norm_hidden_states = (1 + mod_scale) * norm_hidden_states + mod_shift
|
||||
|
||||
joint_attention_kwargs = joint_attention_kwargs or {}
|
||||
attn_output = self.attn(
|
||||
hidden_states=norm_hidden_states,
|
||||
image_rotary_emb=image_rotary_emb,
|
||||
**joint_attention_kwargs,
|
||||
)
|
||||
|
||||
hidden_states = hidden_states + mod_gate * attn_output
|
||||
if hidden_states.dtype == torch.float16:
|
||||
hidden_states = hidden_states.clip(-65504, 65504)
|
||||
|
||||
if split_hidden_states:
|
||||
encoder_hidden_states, hidden_states = hidden_states[:, :text_seq_len], hidden_states[:, text_seq_len:]
|
||||
return encoder_hidden_states, hidden_states
|
||||
return hidden_states
|
||||
|
||||
|
||||
class Flux2TransformerBlock(nn.Module):
|
||||
def __init__(
|
||||
self,
|
||||
dim: int,
|
||||
num_attention_heads: int,
|
||||
attention_head_dim: int,
|
||||
mlp_ratio: float = 3.0,
|
||||
eps: float = 1e-6,
|
||||
bias: bool = False,
|
||||
):
|
||||
super().__init__()
|
||||
self.mlp_hidden_dim = int(dim * mlp_ratio)
|
||||
|
||||
self.norm1 = nn.LayerNorm(dim, elementwise_affine=False, eps=eps)
|
||||
self.norm1_context = nn.LayerNorm(dim, elementwise_affine=False, eps=eps)
|
||||
|
||||
self.attn = Flux2Attention(
|
||||
query_dim=dim,
|
||||
added_kv_proj_dim=dim,
|
||||
dim_head=attention_head_dim,
|
||||
heads=num_attention_heads,
|
||||
out_dim=dim,
|
||||
bias=bias,
|
||||
added_proj_bias=bias,
|
||||
out_bias=bias,
|
||||
eps=eps,
|
||||
)
|
||||
|
||||
self.norm2 = nn.LayerNorm(dim, elementwise_affine=False, eps=eps)
|
||||
self.ff = Flux2FeedForward(dim=dim, dim_out=dim, mult=mlp_ratio, bias=bias)
|
||||
|
||||
self.norm2_context = nn.LayerNorm(dim, elementwise_affine=False, eps=eps)
|
||||
self.ff_context = Flux2FeedForward(dim=dim, dim_out=dim, mult=mlp_ratio, bias=bias)
|
||||
|
||||
def forward(
|
||||
self,
|
||||
hidden_states: torch.Tensor,
|
||||
encoder_hidden_states: torch.Tensor,
|
||||
temb_mod_params_img: tuple[tuple[torch.Tensor, torch.Tensor, torch.Tensor], ...],
|
||||
temb_mod_params_txt: tuple[tuple[torch.Tensor, torch.Tensor, torch.Tensor], ...],
|
||||
image_rotary_emb: tuple[torch.Tensor, torch.Tensor] | None = None,
|
||||
joint_attention_kwargs: dict[str, Any] | None = None,
|
||||
) -> tuple[torch.Tensor, torch.Tensor]:
|
||||
joint_attention_kwargs = joint_attention_kwargs or {}
|
||||
|
||||
(shift_msa, scale_msa, gate_msa), (shift_mlp, scale_mlp, gate_mlp) = temb_mod_params_img
|
||||
(c_shift_msa, c_scale_msa, c_gate_msa), (c_shift_mlp, c_scale_mlp, c_gate_mlp) = temb_mod_params_txt
|
||||
|
||||
norm_hidden_states = self.norm1(hidden_states)
|
||||
norm_hidden_states = (1 + scale_msa) * norm_hidden_states + shift_msa
|
||||
|
||||
norm_encoder_hidden_states = self.norm1_context(encoder_hidden_states)
|
||||
norm_encoder_hidden_states = (1 + c_scale_msa) * norm_encoder_hidden_states + c_shift_msa
|
||||
|
||||
attn_output, context_attn_output = self.attn(
|
||||
hidden_states=norm_hidden_states,
|
||||
encoder_hidden_states=norm_encoder_hidden_states,
|
||||
image_rotary_emb=image_rotary_emb,
|
||||
**joint_attention_kwargs,
|
||||
)
|
||||
|
||||
attn_output = gate_msa * attn_output
|
||||
hidden_states = hidden_states + attn_output
|
||||
|
||||
norm_hidden_states = self.norm2(hidden_states)
|
||||
norm_hidden_states = norm_hidden_states * (1 + scale_mlp) + shift_mlp
|
||||
ff_output = self.ff(norm_hidden_states)
|
||||
hidden_states = hidden_states + gate_mlp * ff_output
|
||||
|
||||
context_attn_output = c_gate_msa * context_attn_output
|
||||
encoder_hidden_states = encoder_hidden_states + context_attn_output
|
||||
|
||||
norm_encoder_hidden_states = self.norm2_context(encoder_hidden_states)
|
||||
norm_encoder_hidden_states = norm_encoder_hidden_states * (1 + c_scale_mlp) + c_shift_mlp
|
||||
context_ff_output = self.ff_context(norm_encoder_hidden_states)
|
||||
encoder_hidden_states = encoder_hidden_states + c_gate_mlp * context_ff_output
|
||||
if encoder_hidden_states.dtype == torch.float16:
|
||||
encoder_hidden_states = encoder_hidden_states.clip(-65504, 65504)
|
||||
|
||||
return encoder_hidden_states, hidden_states
|
||||
|
||||
|
||||
class Flux2PosEmbed(nn.Module):
|
||||
def __init__(self, theta: int, axes_dim: list[int]):
|
||||
super().__init__()
|
||||
self.theta = theta
|
||||
self.axes_dim = axes_dim
|
||||
|
||||
def forward(self, ids: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor]:
|
||||
# Expected ids shape: [S, len(self.axes_dim)]
|
||||
cos_out = []
|
||||
sin_out = []
|
||||
pos = ids.float()
|
||||
is_mps = ids.device.type == "mps"
|
||||
is_npu = ids.device.type == "npu"
|
||||
freqs_dtype = torch.float32 if (is_mps or is_npu) else torch.float64
|
||||
# Unlike Flux 1, loop over len(self.axes_dim) rather than ids.shape[-1]
|
||||
for i in range(len(self.axes_dim)):
|
||||
freqs_cis = get_1d_rotary_pos_embed(
|
||||
self.axes_dim[i],
|
||||
pos[..., i],
|
||||
theta=self.theta,
|
||||
use_real=False,
|
||||
freqs_dtype=freqs_dtype,
|
||||
)
|
||||
cos_out.append(freqs_cis.real)
|
||||
sin_out.append(freqs_cis.imag)
|
||||
freqs_cos = torch.cat(cos_out, dim=-1).to(ids.device)
|
||||
freqs_sin = torch.cat(sin_out, dim=-1).to(ids.device)
|
||||
return freqs_cos, freqs_sin
|
||||
|
||||
|
||||
class Flux2TimestepGuidanceEmbeddings(nn.Module):
|
||||
def __init__(
|
||||
self,
|
||||
in_channels: int = 256,
|
||||
embedding_dim: int = 6144,
|
||||
bias: bool = False,
|
||||
guidance_embeds: bool = True,
|
||||
):
|
||||
super().__init__()
|
||||
self.time_proj = Timesteps(num_channels=in_channels, flip_sin_to_cos=True, downscale_freq_shift=0)
|
||||
self.timestep_embedder = TimestepEmbedding(
|
||||
in_channels=in_channels,
|
||||
time_embed_dim=embedding_dim,
|
||||
sample_proj_bias=bias,
|
||||
)
|
||||
|
||||
if guidance_embeds:
|
||||
self.guidance_embedder = TimestepEmbedding(
|
||||
in_channels=in_channels,
|
||||
time_embed_dim=embedding_dim,
|
||||
sample_proj_bias=bias,
|
||||
)
|
||||
else:
|
||||
self.guidance_embedder = None
|
||||
|
||||
def forward(self, timestep: torch.Tensor, guidance: torch.Tensor | None) -> torch.Tensor:
|
||||
timesteps_proj = self.time_proj(timestep)
|
||||
timesteps_emb = self.timestep_embedder(timesteps_proj.to(timestep.dtype))
|
||||
|
||||
if guidance is not None and self.guidance_embedder is not None:
|
||||
guidance_proj = self.time_proj(guidance)
|
||||
guidance_emb = self.guidance_embedder(guidance_proj.to(guidance.dtype))
|
||||
return timesteps_emb + guidance_emb
|
||||
return timesteps_emb
|
||||
|
||||
|
||||
class Flux2Modulation(nn.Module):
|
||||
def __init__(self, dim: int, mod_param_sets: int = 2, bias: bool = False):
|
||||
super().__init__()
|
||||
self.mod_param_sets = mod_param_sets
|
||||
|
||||
self.linear = nn.Linear(dim, dim * 3 * self.mod_param_sets, bias=bias)
|
||||
self.act_fn = nn.SiLU()
|
||||
|
||||
def forward(self, temb: torch.Tensor) -> tuple[tuple[torch.Tensor, torch.Tensor, torch.Tensor], ...]:
|
||||
mod = self.act_fn(temb)
|
||||
mod = self.linear(mod)
|
||||
|
||||
if mod.ndim == 2:
|
||||
mod = mod.unsqueeze(1)
|
||||
mod_params = torch.chunk(mod, 3 * self.mod_param_sets, dim=-1)
|
||||
# Return tuple of 3-tuples of modulation params shift/scale/gate
|
||||
return tuple(mod_params[3 * i : 3 * (i + 1)] for i in range(self.mod_param_sets))
|
||||
|
||||
|
||||
class Flux2Transformer2DModel(nn.Module):
|
||||
"""
|
||||
The Transformer model introduced in Flux 2.
|
||||
"""
|
||||
|
||||
_repeated_blocks = ["Flux2TransformerBlock", "Flux2SingleTransformerBlock"]
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
patch_size: int = 1,
|
||||
in_channels: int = 128,
|
||||
out_channels: int | None = None,
|
||||
num_layers: int = 8,
|
||||
num_single_layers: int = 48,
|
||||
attention_head_dim: int = 128,
|
||||
num_attention_heads: int = 48,
|
||||
joint_attention_dim: int = 15360,
|
||||
timestep_guidance_channels: int = 256,
|
||||
mlp_ratio: float = 3.0,
|
||||
axes_dims_rope: tuple[int, ...] = (32, 32, 32, 32),
|
||||
rope_theta: int = 2000,
|
||||
eps: float = 1e-6,
|
||||
guidance_embeds: bool = True,
|
||||
):
|
||||
super().__init__()
|
||||
self.stacked_params_mapping = None
|
||||
self.out_channels = out_channels or in_channels
|
||||
self.inner_dim = num_attention_heads * attention_head_dim
|
||||
self.config = SimpleNamespace(
|
||||
patch_size=patch_size,
|
||||
in_channels=in_channels,
|
||||
out_channels=self.out_channels,
|
||||
num_layers=num_layers,
|
||||
num_single_layers=num_single_layers,
|
||||
attention_head_dim=attention_head_dim,
|
||||
num_attention_heads=num_attention_heads,
|
||||
joint_attention_dim=joint_attention_dim,
|
||||
timestep_guidance_channels=timestep_guidance_channels,
|
||||
mlp_ratio=mlp_ratio,
|
||||
axes_dims_rope=axes_dims_rope,
|
||||
rope_theta=rope_theta,
|
||||
eps=eps,
|
||||
guidance_embeds=guidance_embeds,
|
||||
)
|
||||
|
||||
self.pos_embed = Flux2PosEmbed(theta=rope_theta, axes_dim=axes_dims_rope)
|
||||
self.time_guidance_embed = Flux2TimestepGuidanceEmbeddings(
|
||||
in_channels=timestep_guidance_channels,
|
||||
embedding_dim=self.inner_dim,
|
||||
bias=False,
|
||||
guidance_embeds=guidance_embeds,
|
||||
)
|
||||
|
||||
self.double_stream_modulation_img = Flux2Modulation(self.inner_dim, mod_param_sets=2, bias=False)
|
||||
self.double_stream_modulation_txt = Flux2Modulation(self.inner_dim, mod_param_sets=2, bias=False)
|
||||
self.single_stream_modulation = Flux2Modulation(self.inner_dim, mod_param_sets=1, bias=False)
|
||||
|
||||
self.x_embedder = nn.Linear(in_channels, self.inner_dim, bias=False)
|
||||
self.context_embedder = nn.Linear(joint_attention_dim, self.inner_dim, bias=False)
|
||||
|
||||
self.transformer_blocks = nn.ModuleList(
|
||||
[
|
||||
Flux2TransformerBlock(
|
||||
dim=self.inner_dim,
|
||||
num_attention_heads=num_attention_heads,
|
||||
attention_head_dim=attention_head_dim,
|
||||
mlp_ratio=mlp_ratio,
|
||||
eps=eps,
|
||||
bias=False,
|
||||
)
|
||||
for _ in range(num_layers)
|
||||
]
|
||||
)
|
||||
|
||||
self.single_transformer_blocks = nn.ModuleList(
|
||||
[
|
||||
Flux2SingleTransformerBlock(
|
||||
dim=self.inner_dim,
|
||||
num_attention_heads=num_attention_heads,
|
||||
attention_head_dim=attention_head_dim,
|
||||
mlp_ratio=mlp_ratio,
|
||||
eps=eps,
|
||||
bias=False,
|
||||
)
|
||||
for _ in range(num_single_layers)
|
||||
]
|
||||
)
|
||||
|
||||
self.norm_out = AdaLayerNormContinuous(
|
||||
self.inner_dim, self.inner_dim, elementwise_affine=False, eps=eps, bias=False
|
||||
)
|
||||
self.proj_out = nn.Linear(self.inner_dim, patch_size * patch_size * self.out_channels, bias=False)
|
||||
|
||||
@property
|
||||
def dtype(self) -> torch.dtype:
|
||||
return next(self.parameters()).dtype
|
||||
|
||||
def forward(
|
||||
self,
|
||||
hidden_states: torch.Tensor,
|
||||
encoder_hidden_states: torch.Tensor = None,
|
||||
timestep: torch.LongTensor = None,
|
||||
img_ids: torch.Tensor = None,
|
||||
txt_ids: torch.Tensor = None,
|
||||
guidance: torch.Tensor | None = None,
|
||||
joint_attention_kwargs: dict[str, Any] | None = None,
|
||||
return_dict: bool = True,
|
||||
) -> torch.Tensor | Transformer2DModelOutput:
|
||||
joint_attention_kwargs = joint_attention_kwargs or {}
|
||||
|
||||
num_txt_tokens = encoder_hidden_states.shape[1]
|
||||
|
||||
timestep = timestep.to(hidden_states.dtype) * 1000
|
||||
if guidance is not None:
|
||||
guidance = guidance.to(hidden_states.dtype) * 1000
|
||||
|
||||
temb = self.time_guidance_embed(timestep, guidance)
|
||||
|
||||
double_stream_mod_img = self.double_stream_modulation_img(temb)
|
||||
double_stream_mod_txt = self.double_stream_modulation_txt(temb)
|
||||
single_stream_mod = self.single_stream_modulation(temb)[0]
|
||||
|
||||
hidden_states = self.x_embedder(hidden_states)
|
||||
encoder_hidden_states = self.context_embedder(encoder_hidden_states)
|
||||
|
||||
if img_ids.ndim == 3:
|
||||
img_ids = img_ids[0]
|
||||
if txt_ids.ndim == 3:
|
||||
txt_ids = txt_ids[0]
|
||||
|
||||
if is_torch_npu_available():
|
||||
freqs_cos_image, freqs_sin_image = self.pos_embed(img_ids.cpu())
|
||||
image_rotary_emb = (freqs_cos_image.npu(), freqs_sin_image.npu())
|
||||
freqs_cos_text, freqs_sin_text = self.pos_embed(txt_ids.cpu())
|
||||
text_rotary_emb = (freqs_cos_text.npu(), freqs_sin_text.npu())
|
||||
else:
|
||||
image_rotary_emb = self.pos_embed(img_ids)
|
||||
text_rotary_emb = self.pos_embed(txt_ids)
|
||||
concat_rotary_emb = (
|
||||
torch.cat([text_rotary_emb[0], image_rotary_emb[0]], dim=0),
|
||||
torch.cat([text_rotary_emb[1], image_rotary_emb[1]], dim=0),
|
||||
)
|
||||
|
||||
for index_block, block in enumerate(self.transformer_blocks):
|
||||
encoder_hidden_states, hidden_states = block(
|
||||
hidden_states=hidden_states,
|
||||
encoder_hidden_states=encoder_hidden_states,
|
||||
temb_mod_params_img=double_stream_mod_img,
|
||||
temb_mod_params_txt=double_stream_mod_txt,
|
||||
image_rotary_emb=concat_rotary_emb,
|
||||
joint_attention_kwargs=joint_attention_kwargs,
|
||||
)
|
||||
|
||||
hidden_states = torch.cat([encoder_hidden_states, hidden_states], dim=1)
|
||||
|
||||
for index_block, block in enumerate(self.single_transformer_blocks):
|
||||
hidden_states = block(
|
||||
hidden_states=hidden_states,
|
||||
encoder_hidden_states=None,
|
||||
temb_mod_params=single_stream_mod,
|
||||
image_rotary_emb=concat_rotary_emb,
|
||||
joint_attention_kwargs=joint_attention_kwargs,
|
||||
)
|
||||
|
||||
hidden_states = hidden_states[:, num_txt_tokens:, ...]
|
||||
hidden_states = self.norm_out(hidden_states, temb)
|
||||
output = self.proj_out(hidden_states)
|
||||
|
||||
if not return_dict:
|
||||
return (output,)
|
||||
|
||||
return Transformer2DModelOutput(sample=output)
|
||||
|
||||
def load_weights(self, weights: Iterable[tuple[str, torch.Tensor]]) -> set[str]:
|
||||
stacked_params_mapping = [
|
||||
(".to_qkv.", ".to_q.", "q"),
|
||||
(".to_qkv.", ".to_k.", "k"),
|
||||
(".to_qkv.", ".to_v.", "v"),
|
||||
(".add_kv_proj", ".add_q_proj", "q"),
|
||||
(".add_kv_proj", ".add_k_proj", "k"),
|
||||
(".add_kv_proj", ".add_v_proj", "v"),
|
||||
]
|
||||
# Expose packed shard mappings for LoRA handling of fused projections.
|
||||
self.stacked_params_mapping = stacked_params_mapping
|
||||
|
||||
params_dict = dict(self.named_parameters())
|
||||
|
||||
for name, buffer in self.named_buffers():
|
||||
if name.endswith(".beta") or name.endswith(".eps"):
|
||||
params_dict[name] = buffer
|
||||
|
||||
loaded_params: set[str] = set()
|
||||
for name, loaded_weight in weights:
|
||||
original_name = name
|
||||
mapped = False
|
||||
for param_name, weight_name, shard_id in stacked_params_mapping:
|
||||
if weight_name not in original_name:
|
||||
continue
|
||||
name = original_name.replace(weight_name, param_name)
|
||||
param = params_dict.get(name)
|
||||
if param is None:
|
||||
break
|
||||
weight_loader = param.weight_loader
|
||||
weight_loader(param, loaded_weight, shard_id)
|
||||
loaded_params.add(name)
|
||||
mapped = True
|
||||
break
|
||||
if mapped:
|
||||
continue
|
||||
|
||||
name = original_name
|
||||
if name not in params_dict and ".to_out.0." in name:
|
||||
name = name.replace(".to_out.0.", ".to_out.")
|
||||
# Some GGUF checkpoints include quantized tensors for modules that
|
||||
# are intentionally left unquantized in this model.
|
||||
param = params_dict.get(name)
|
||||
if param is None:
|
||||
continue
|
||||
weight_loader = getattr(param, "weight_loader", default_weight_loader)
|
||||
weight_loader(param, loaded_weight)
|
||||
loaded_params.add(name)
|
||||
return loaded_params
|
||||
@@ -10,6 +10,7 @@ from vllm.model_executor.models.registry import _LazyRegisteredModel, _ModelRegi
|
||||
from vllm_omni.diffusion.data import OmniDiffusionConfig
|
||||
from vllm_omni.diffusion.distributed.autoencoders.distributed_vae_executor import DistributedVaeMixin
|
||||
from vllm_omni.diffusion.distributed.sp_plan import SequenceParallelConfig, get_sp_plan_from_model
|
||||
from vllm_omni.diffusion.forward_context import get_forward_context
|
||||
from vllm_omni.diffusion.hooks.sequence_parallel import apply_sequence_parallel
|
||||
|
||||
logger = init_logger(__name__)
|
||||
@@ -131,6 +132,11 @@ _DIFFUSION_MODELS = {
|
||||
"pipeline_helios",
|
||||
"HeliosPipeline",
|
||||
),
|
||||
"Flux2Pipeline": (
|
||||
"flux2",
|
||||
"pipeline_flux2",
|
||||
"Flux2Pipeline",
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
@@ -264,6 +270,11 @@ def _apply_sequence_parallel_if_enabled(model, od_config: OmniDiffusionConfig) -
|
||||
apply_sequence_parallel(transformer, sp_config, plan)
|
||||
applied_count += 1
|
||||
|
||||
# update forward context sp_plan_hooks_applied
|
||||
ctx = get_forward_context()
|
||||
ctx.sp_plan_hooks_applied = applied_count > 0
|
||||
logger.debug(f"Setting sp_plan_hooks_applied={ctx.sp_plan_hooks_applied} in ``ForwardContext``!")
|
||||
|
||||
if applied_count == 0:
|
||||
logger.warning(
|
||||
f"Sequence parallelism is enabled (sp_size={sp_size}) but no transformer with _sp_plan found. "
|
||||
@@ -299,6 +310,7 @@ _DIFFUSION_POST_PROCESS_FUNCS = {
|
||||
"OmniGen2Pipeline": "get_omnigen2_post_process_func",
|
||||
"HeliosPipeline": "get_helios_post_process_func",
|
||||
"HeliosPyramidPipeline": "get_helios_post_process_func",
|
||||
"Flux2Pipeline": "get_flux2_post_process_func",
|
||||
}
|
||||
|
||||
_DIFFUSION_PRE_PROCESS_FUNCS = {
|
||||
|
||||
@@ -106,6 +106,7 @@ class DiffusionWorker:
|
||||
vllm_config = VllmConfig(compilation_config=CompilationConfig())
|
||||
vllm_config.parallel_config.tensor_parallel_size = self.od_config.parallel_config.tensor_parallel_size
|
||||
vllm_config.parallel_config.data_parallel_size = self.od_config.parallel_config.data_parallel_size
|
||||
vllm_config.parallel_config.enable_expert_parallel = self.od_config.parallel_config.enable_expert_parallel
|
||||
self.vllm_config = vllm_config
|
||||
|
||||
# Initialize distributed environment
|
||||
@@ -127,6 +128,7 @@ class DiffusionWorker:
|
||||
pipeline_parallel_size=parallel_config.pipeline_parallel_size,
|
||||
fully_shard_degree=parallel_config.hsdp_shard_size if parallel_config.use_hsdp else 1,
|
||||
hsdp_replicate_size=parallel_config.hsdp_replicate_size if parallel_config.use_hsdp else 1,
|
||||
enable_expert_parallel=parallel_config.enable_expert_parallel,
|
||||
)
|
||||
init_workspace_manager(self.device)
|
||||
|
||||
|
||||
@@ -236,6 +236,9 @@ class OmniChunkTransferAdapter(OmniTransferAdapterBase):
|
||||
self.put_req_chunk[request_id] += 1
|
||||
logger.debug(f"[Stage-{stage_id}] Sent {connector_put_key}")
|
||||
|
||||
if is_finished:
|
||||
self.code_prompt_token_ids.pop(request_id, None)
|
||||
|
||||
########################################################################
|
||||
# Cleanup
|
||||
########################################################################
|
||||
|
||||
@@ -0,0 +1,27 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||
|
||||
from .load_balancer import (
|
||||
LoadBalancer,
|
||||
LoadBalancingPolicy,
|
||||
RandomBalancer,
|
||||
Task,
|
||||
)
|
||||
from .messages import InstanceEvent, InstanceInfo, InstanceList, StageStatus
|
||||
from .omni_coord_client_for_hub import OmniCoordClientForHub
|
||||
from .omni_coord_client_for_stage import OmniCoordClientForStage
|
||||
from .omni_coordinator import OmniCoordinator
|
||||
|
||||
__all__ = [
|
||||
"OmniCoordinator",
|
||||
"StageStatus",
|
||||
"InstanceEvent",
|
||||
"InstanceInfo",
|
||||
"InstanceList",
|
||||
"OmniCoordClientForStage",
|
||||
"OmniCoordClientForHub",
|
||||
"Task",
|
||||
"LoadBalancer",
|
||||
"LoadBalancingPolicy",
|
||||
"RandomBalancer",
|
||||
]
|
||||
@@ -0,0 +1,82 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import random
|
||||
from abc import ABC, abstractmethod
|
||||
from enum import Enum
|
||||
from typing import Any, TypedDict
|
||||
|
||||
from .messages import InstanceInfo
|
||||
|
||||
|
||||
class Task(TypedDict, total=False):
|
||||
"""Task structure passed from async_omni (stage.submit(task)).
|
||||
|
||||
Mirrors the dict built in AsyncOmni with request_id, engine_inputs,
|
||||
sampling_params. Future load-balancing policies may use these fields.
|
||||
"""
|
||||
|
||||
request_id: str
|
||||
engine_inputs: Any
|
||||
sampling_params: Any
|
||||
|
||||
|
||||
class LoadBalancingPolicy(str, Enum):
|
||||
"""Enumeration for load balancing policies.
|
||||
|
||||
Only ``RANDOM`` is implemented. Additional policies (e.g. round-robin,
|
||||
least-connections) can be added in the future.
|
||||
"""
|
||||
|
||||
RANDOM = "random"
|
||||
|
||||
|
||||
class LoadBalancer(ABC):
|
||||
"""Abstract base class for load balancers.
|
||||
|
||||
Subclasses implement :meth:`select` to choose an instance for a given task.
|
||||
"""
|
||||
|
||||
@abstractmethod
|
||||
def select(self, task: Task, instances: list[InstanceInfo]) -> int:
|
||||
"""Route a task to one of the available instances.
|
||||
|
||||
Args:
|
||||
task: The task to route. Not used by the random policy but reserved
|
||||
for future strategies that may inspect task metadata.
|
||||
instances: List of available instances to choose from.
|
||||
|
||||
Returns:
|
||||
Index of the selected instance in ``instances``.
|
||||
|
||||
Raises:
|
||||
ValueError: If ``instances`` is empty.
|
||||
"""
|
||||
|
||||
raise NotImplementedError
|
||||
|
||||
|
||||
class RandomBalancer(LoadBalancer):
|
||||
"""Load balancer that selects an instance uniformly at random.
|
||||
|
||||
This is the initial and only policy supported. It intentionally ignores
|
||||
the task payload and chooses a random index from the provided instance
|
||||
list. More sophisticated policies (e.g. round-robin, least-connections)
|
||||
can be implemented as additional subclasses of :class:`LoadBalancer`.
|
||||
"""
|
||||
|
||||
def select(self, task: Task, instances: list[InstanceInfo]) -> int: # noqa: ARG002
|
||||
if not instances:
|
||||
raise ValueError("instances must not be empty")
|
||||
|
||||
return random.randrange(len(instances))
|
||||
|
||||
|
||||
__all__ = [
|
||||
"Task",
|
||||
"LoadBalancingPolicy",
|
||||
"LoadBalancer",
|
||||
"RandomBalancer",
|
||||
]
|
||||
@@ -0,0 +1,61 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
from enum import Enum
|
||||
|
||||
|
||||
class StageStatus(str, Enum):
|
||||
"""Enumeration for stage instance status."""
|
||||
|
||||
UP = "up" # Instance is ready and available
|
||||
DOWN = "down" # Instance is shutdown gracefully
|
||||
ERROR = "error" # Instance encountered an error or timeout
|
||||
|
||||
|
||||
@dataclass
|
||||
class InstanceEvent:
|
||||
"""Wire payload from OmniCoordClientForStage to OmniCoordinator.
|
||||
|
||||
Schema for Stage → Coordinator events over ZMQ:
|
||||
input_addr, output_addr, stage_id, status, queue_length, event_type.
|
||||
"""
|
||||
|
||||
input_addr: str # Stage instance input ZMQ address (e.g., "tcp://host:port")
|
||||
output_addr: str # Stage instance output ZMQ address (e.g., "tcp://host:port")
|
||||
stage_id: int # Stage ID
|
||||
event_type: str # "update" | "heartbeat"
|
||||
status: StageStatus # Current status
|
||||
queue_length: int # Current queue length
|
||||
|
||||
|
||||
@dataclass
|
||||
class InstanceInfo:
|
||||
"""Metadata for a single stage instance.
|
||||
|
||||
This type is stored in OmniCoordinator's internal registry and is also
|
||||
published to hubs via :class:`InstanceList`.
|
||||
"""
|
||||
|
||||
input_addr: str # Stage instance input ZMQ address (e.g., "tcp://host:port")
|
||||
output_addr: str # Stage instance output ZMQ address (e.g., "tcp://host:port")
|
||||
stage_id: int # Stage ID of this instance
|
||||
status: StageStatus # Current status of the instance
|
||||
queue_length: int # Current queue length of this instance
|
||||
last_heartbeat: float # Timestamp of the last heartbeat received (seconds)
|
||||
registered_at: float # Timestamp when the instance was registered (seconds)
|
||||
|
||||
|
||||
@dataclass
|
||||
class InstanceList:
|
||||
"""Container for instance list updates.
|
||||
|
||||
OmniCoordinator publishes an :class:`InstanceList` whenever its view of
|
||||
active instances changes. OmniCoordClientForHub caches the latest value
|
||||
and exposes it to AsyncOmni and the load balancer.
|
||||
"""
|
||||
|
||||
instances: list[InstanceInfo]
|
||||
timestamp: float # Time when the list was last updated (seconds)
|
||||
@@ -0,0 +1,164 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||
|
||||
import json
|
||||
import logging
|
||||
import threading
|
||||
from time import sleep, time
|
||||
from typing import Any
|
||||
|
||||
import zmq
|
||||
|
||||
from .messages import InstanceInfo, InstanceList, StageStatus
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class OmniCoordClientForHub:
|
||||
"""Client for AsyncOmni side to receive instance list updates.
|
||||
|
||||
This client maintains a SUB socket connected to OmniCoordinator's PUB
|
||||
endpoint and caches the latest :class:`InstanceList` in memory for use by
|
||||
the load balancer and routing logic.
|
||||
"""
|
||||
|
||||
def __init__(self, coord_zmq_addr: str) -> None:
|
||||
"""Initialize client and start receive thread (socket created in thread)."""
|
||||
self._coord_zmq_addr = coord_zmq_addr
|
||||
|
||||
self._ctx = zmq.Context()
|
||||
self._lock = threading.Lock()
|
||||
self._instance_list: InstanceList | None = None
|
||||
self._closed = False
|
||||
self._stop_event = threading.Event()
|
||||
self._init_done = threading.Event()
|
||||
self._init_error: list[BaseException] = []
|
||||
|
||||
self._thread = threading.Thread(target=self._recv_loop, daemon=True)
|
||||
self._thread.start()
|
||||
|
||||
self._init_done.wait(timeout=5.0)
|
||||
if self._init_error:
|
||||
raise RuntimeError(f"Failed to connect to coordinator at {self._coord_zmq_addr}") from self._init_error[0]
|
||||
|
||||
def _decode_instance_list(self, payload: dict[str, Any]) -> InstanceList:
|
||||
"""Convert a JSON-decoded dict into an :class:`InstanceList`."""
|
||||
instances_payload = payload.get("instances", [])
|
||||
instances: list[InstanceInfo] = []
|
||||
for inst in instances_payload:
|
||||
instances.append(
|
||||
InstanceInfo(
|
||||
input_addr=inst["input_addr"],
|
||||
output_addr=inst["output_addr"],
|
||||
stage_id=int(inst["stage_id"]),
|
||||
status=StageStatus(inst["status"]),
|
||||
queue_length=int(inst["queue_length"]),
|
||||
last_heartbeat=float(inst["last_heartbeat"]),
|
||||
registered_at=float(inst["registered_at"]),
|
||||
)
|
||||
)
|
||||
|
||||
timestamp = float(payload.get("timestamp", time()))
|
||||
return InstanceList(instances=instances, timestamp=timestamp)
|
||||
|
||||
def _recv_loop(self) -> None:
|
||||
"""Background loop that receives and caches instance lists."""
|
||||
sub = None
|
||||
try:
|
||||
sub = self._ctx.socket(zmq.SUB)
|
||||
sub.setsockopt(zmq.SUBSCRIBE, b"")
|
||||
sub.setsockopt(zmq.RCVTIMEO, 100) # 100ms timeout, avoids busy-wait
|
||||
sub.connect(self._coord_zmq_addr)
|
||||
except (zmq.ZMQError, OSError) as e:
|
||||
self._init_error.append(e)
|
||||
sub = None
|
||||
finally:
|
||||
self._init_done.set()
|
||||
|
||||
try:
|
||||
while not self._stop_event.is_set():
|
||||
# (Re)create and connect SUB socket if needed.
|
||||
if sub is None:
|
||||
try:
|
||||
sub = self._ctx.socket(zmq.SUB)
|
||||
sub.setsockopt(zmq.SUBSCRIBE, b"")
|
||||
sub.setsockopt(zmq.RCVTIMEO, 100) # 100ms timeout, avoids busy-wait
|
||||
sub.connect(self._coord_zmq_addr)
|
||||
except (zmq.ZMQError, OSError) as e:
|
||||
logger.error(
|
||||
"Hub client failed to connect to coordinator at %s, will retry",
|
||||
self._coord_zmq_addr,
|
||||
exc_info=e,
|
||||
)
|
||||
if sub is not None:
|
||||
try:
|
||||
sub.close()
|
||||
except zmq.ZMQError:
|
||||
pass
|
||||
sub = None
|
||||
sleep(1.0)
|
||||
continue
|
||||
|
||||
try:
|
||||
data = sub.recv()
|
||||
except zmq.Again:
|
||||
continue
|
||||
except zmq.ZMQError as e:
|
||||
logger.error("Hub client recv failed, will reconnect", exc_info=e)
|
||||
try:
|
||||
sub.close()
|
||||
except zmq.ZMQError:
|
||||
pass
|
||||
sub = None
|
||||
sleep(1.0)
|
||||
continue
|
||||
|
||||
try:
|
||||
payload = json.loads(data.decode("utf-8"))
|
||||
inst_list = self._decode_instance_list(payload)
|
||||
with self._lock:
|
||||
self._instance_list = inst_list
|
||||
except (
|
||||
json.JSONDecodeError,
|
||||
KeyError,
|
||||
ValueError,
|
||||
TypeError,
|
||||
AttributeError,
|
||||
) as e:
|
||||
logger.warning("Invalid instance list message, skipping: %s", e)
|
||||
finally:
|
||||
try:
|
||||
if sub is not None:
|
||||
sub.close()
|
||||
except zmq.ZMQError:
|
||||
pass
|
||||
try:
|
||||
self._ctx.term()
|
||||
except zmq.ZMQError:
|
||||
pass
|
||||
|
||||
def get_instance_list(self) -> InstanceList:
|
||||
"""Return the latest cached :class:`InstanceList`.
|
||||
|
||||
If no update has been received yet, returns an empty list with
|
||||
``timestamp=0.0``.
|
||||
"""
|
||||
with self._lock:
|
||||
if self._instance_list is None:
|
||||
return InstanceList(instances=[], timestamp=0.0)
|
||||
return self._instance_list
|
||||
|
||||
def get_instances_for_stage(self, stage_id: int) -> InstanceList:
|
||||
"""Return instances filtered by ``stage_id``."""
|
||||
base = self.get_instance_list()
|
||||
filtered = [inst for inst in base.instances if inst.stage_id == stage_id]
|
||||
return InstanceList(instances=filtered, timestamp=base.timestamp)
|
||||
|
||||
def close(self) -> None:
|
||||
"""Close the SUB socket and stop the background thread."""
|
||||
if self._closed:
|
||||
raise RuntimeError("Client already closed")
|
||||
|
||||
self._closed = True
|
||||
self._stop_event.set()
|
||||
self._thread.join(timeout=1.0)
|
||||
@@ -0,0 +1,193 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||
|
||||
import json
|
||||
import logging
|
||||
import threading
|
||||
import time
|
||||
from dataclasses import asdict
|
||||
|
||||
import zmq
|
||||
|
||||
from .messages import InstanceEvent, StageStatus
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class OmniCoordClientForStage:
|
||||
"""Client used by stage instances to send events to OmniCoordinator.
|
||||
|
||||
This client maintains a DEALER socket connected to OmniCoordinator's
|
||||
ROUTER endpoint and sends JSON-encoded events describing instance status.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
coord_zmq_addr: str,
|
||||
input_addr: str,
|
||||
output_addr: str,
|
||||
stage_id: int,
|
||||
) -> None:
|
||||
"""Initialize client and send initial registration / status-up event."""
|
||||
self._coord_zmq_addr = coord_zmq_addr
|
||||
self._input_addr = input_addr
|
||||
self._output_addr = output_addr
|
||||
self._stage_id = stage_id
|
||||
|
||||
self._ctx = zmq.Context()
|
||||
self._socket = self._ctx.socket(zmq.DEALER)
|
||||
try:
|
||||
self._socket.connect(self._coord_zmq_addr)
|
||||
except zmq.ZMQError as e:
|
||||
self._socket.close()
|
||||
raise RuntimeError(f"Failed to connect to coordinator at {self._coord_zmq_addr}: {e}") from e
|
||||
|
||||
self._status = StageStatus.UP
|
||||
self._queue_length = 0
|
||||
self._closed = False
|
||||
self._heartbeat_interval = 5.0
|
||||
self._stop_event = threading.Event()
|
||||
self._send_lock = threading.Lock()
|
||||
|
||||
self._send_event("update")
|
||||
|
||||
self._heartbeat_thread = threading.Thread(
|
||||
target=self._heartbeat_loop,
|
||||
daemon=True,
|
||||
)
|
||||
self._heartbeat_thread.start()
|
||||
|
||||
def _reconnect(self) -> bool:
|
||||
"""Best-effort reconnect with up to ``max_retries`` attempts.
|
||||
|
||||
Each attempt closes the current socket/context, sleeps 5 seconds,
|
||||
then creates a new DEALER socket and reconnects to the coordinator.
|
||||
Caller must hold ``_send_lock``.
|
||||
Returns True on success, False if all attempts fail.
|
||||
"""
|
||||
while not self._stop_event.is_set() and not self._closed:
|
||||
try:
|
||||
self._socket.close(0)
|
||||
except zmq.ZMQError:
|
||||
pass
|
||||
try:
|
||||
self._ctx.term()
|
||||
except zmq.ZMQError:
|
||||
pass
|
||||
|
||||
time.sleep(5.0)
|
||||
|
||||
try:
|
||||
self._ctx = zmq.Context()
|
||||
self._socket = self._ctx.socket(zmq.DEALER)
|
||||
self._socket.connect(self._coord_zmq_addr)
|
||||
return True
|
||||
except zmq.ZMQError as e:
|
||||
logger.error(
|
||||
"Stage client reconnect failed, will retry in 5s (coord=%s)",
|
||||
self._coord_zmq_addr,
|
||||
exc_info=e,
|
||||
)
|
||||
continue
|
||||
return False
|
||||
|
||||
def _send_event(self, event_type: str) -> None:
|
||||
"""Send an InstanceEvent to OmniCoordinator.
|
||||
|
||||
Wire format: input_addr, output_addr, stage_id, status, queue_length, event_type.
|
||||
For "update": includes status and queue_length from instance state.
|
||||
For "heartbeat": status and queue_length are null.
|
||||
|
||||
On send failure (ZMQError / RuntimeError), attempts to reconnect up
|
||||
to 3 times (5s sleep each) and retries the send once after a
|
||||
successful reconnect. Raises if reconnect or the retry send fails.
|
||||
"""
|
||||
if self._closed:
|
||||
raise RuntimeError("Client already closed")
|
||||
|
||||
event = InstanceEvent(
|
||||
input_addr=self._input_addr,
|
||||
output_addr=self._output_addr,
|
||||
stage_id=self._stage_id,
|
||||
event_type=event_type,
|
||||
status=self._status,
|
||||
queue_length=self._queue_length,
|
||||
)
|
||||
data = json.dumps(asdict(event)).encode("utf-8")
|
||||
|
||||
with self._send_lock:
|
||||
try:
|
||||
self._socket.send(data, flags=zmq.NOBLOCK)
|
||||
return
|
||||
except zmq.Again:
|
||||
logger.debug("Send buffer full, dropping event")
|
||||
return
|
||||
except (RuntimeError, zmq.ZMQError) as e:
|
||||
# First send failed; try reconnecting a few times.
|
||||
if not self._reconnect:
|
||||
logger.error("Failed to send event and reconnect to coordinator", exc_info=e)
|
||||
raise
|
||||
|
||||
# Reconnected successfully; try sending once more.
|
||||
try:
|
||||
self._socket.send(data, flags=zmq.NOBLOCK)
|
||||
except zmq.Again:
|
||||
logger.debug("Send buffer full after reconnect, dropping event")
|
||||
except (RuntimeError, zmq.ZMQError) as e2:
|
||||
logger.error("Failed to send event after successful reconnect", exc_info=e2)
|
||||
raise
|
||||
|
||||
def update_info(
|
||||
self,
|
||||
status: StageStatus | None = None,
|
||||
queue_length: int | None = None,
|
||||
) -> None:
|
||||
"""Update instance information and notify OmniCoordinator.
|
||||
|
||||
At least one of ``status`` or ``queue_length`` must be provided.
|
||||
"""
|
||||
if status is None and queue_length is None:
|
||||
raise ValueError("At least one of status or queue_length must be provided")
|
||||
|
||||
if status is not None:
|
||||
self._status = status
|
||||
if queue_length is not None:
|
||||
self._queue_length = queue_length
|
||||
|
||||
self._send_event("update")
|
||||
|
||||
def _heartbeat_loop(self) -> None:
|
||||
"""Periodically send heartbeat events while the client is alive."""
|
||||
while not self._stop_event.wait(timeout=self._heartbeat_interval):
|
||||
if self._closed:
|
||||
break
|
||||
|
||||
try:
|
||||
self._send_event("heartbeat")
|
||||
except (RuntimeError, zmq.ZMQError):
|
||||
break
|
||||
|
||||
def close(self) -> None:
|
||||
"""Send a final down event and close the underlying socket."""
|
||||
if self._closed:
|
||||
raise RuntimeError("Client already closed")
|
||||
|
||||
# Stop heartbeat thread first to avoid concurrent sends during shutdown.
|
||||
self._stop_event.set()
|
||||
if hasattr(self, "_heartbeat_thread"):
|
||||
self._heartbeat_thread.join(timeout=1.0)
|
||||
|
||||
# Mark status as DOWN and send one last update.
|
||||
self._status = StageStatus.DOWN
|
||||
try:
|
||||
self._send_event("update")
|
||||
except zmq.ZMQError:
|
||||
pass # Socket may already be broken, proceed with close
|
||||
|
||||
# Close DEALER socket and terminate this client's context.
|
||||
self._socket.close(0)
|
||||
try:
|
||||
self._ctx.term()
|
||||
except zmq.ZMQError:
|
||||
pass
|
||||
self._closed = True
|
||||
@@ -0,0 +1,330 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||
|
||||
|
||||
import json
|
||||
import logging
|
||||
import threading
|
||||
from dataclasses import asdict
|
||||
from time import time
|
||||
from typing import Any
|
||||
|
||||
import zmq
|
||||
|
||||
from .messages import InstanceEvent, InstanceInfo, InstanceList, StageStatus
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class OmniCoordinator:
|
||||
"""Coordinator for stage instances and hub clients.
|
||||
|
||||
This service receives instance events from :class:`OmniCoordClientForStage`
|
||||
via a ZMQ ROUTER socket and publishes active instance lists to
|
||||
:class:`OmniCoordClientForHub` via a PUB socket.
|
||||
|
||||
The coordinator maintains an in-memory registry of all known instances,
|
||||
including their status, queue length, and heartbeat timestamps. A
|
||||
background thread periodically checks for heartbeat timeouts and marks
|
||||
unhealthy instances as ``StageStatus.ERROR``.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
router_zmq_addr: str,
|
||||
pub_zmq_addr: str,
|
||||
heartbeat_timeout: float = 30.0,
|
||||
) -> None:
|
||||
"""Initialize coordinator and start background service loops.
|
||||
|
||||
Args:
|
||||
router_zmq_addr: ZMQ address to bind the ROUTER socket.
|
||||
pub_zmq_addr: ZMQ address to bind the PUB socket.
|
||||
heartbeat_timeout: Seconds before an instance is considered
|
||||
unhealthy if no heartbeat / update is received.
|
||||
"""
|
||||
self._router_zmq_addr = router_zmq_addr
|
||||
self._pub_zmq_addr = pub_zmq_addr
|
||||
self._heartbeat_timeout = heartbeat_timeout
|
||||
|
||||
# Dedicated ZMQ context for this coordinator instance.
|
||||
self._ctx = zmq.Context()
|
||||
self._router = self._ctx.socket(zmq.ROUTER)
|
||||
self._router.bind(self._router_zmq_addr)
|
||||
|
||||
self._pub = self._ctx.socket(zmq.PUB)
|
||||
self._pub.bind(self._pub_zmq_addr)
|
||||
|
||||
self._instances: dict[str, InstanceInfo] = {}
|
||||
self._lock = threading.Lock()
|
||||
self._pub_lock = threading.Lock()
|
||||
|
||||
self._publish_min_interval: float = 0.1 # seconds
|
||||
self._pending_broadcast: bool = False
|
||||
|
||||
self._running = True
|
||||
self._closed = False
|
||||
self._stop_event = threading.Event()
|
||||
|
||||
self._router.setsockopt(zmq.RCVTIMEO, 100)
|
||||
|
||||
self._recv_thread = threading.Thread(target=self._recv_loop, daemon=True)
|
||||
self._recv_thread.start()
|
||||
|
||||
self._periodic_thread = threading.Thread(target=self._periodic_loop, daemon=True)
|
||||
self._periodic_thread.start()
|
||||
|
||||
def get_active_instances(self) -> InstanceList:
|
||||
"""Return an :class:`InstanceList` of active (UP) instances only."""
|
||||
with self._lock:
|
||||
active = [inst for inst in self._instances.values() if inst.status == StageStatus.UP]
|
||||
return InstanceList(instances=active, timestamp=time())
|
||||
|
||||
def add_new_instance(self, event: InstanceEvent) -> None:
|
||||
"""Add a new instance based on an incoming event."""
|
||||
with self._lock:
|
||||
self._add_new_instance_locked(event)
|
||||
self.publish_instance_list_update()
|
||||
|
||||
def update_instance_info(self, event: InstanceEvent) -> None:
|
||||
"""Update an existing instance based on an incoming event."""
|
||||
with self._lock:
|
||||
self._update_instance_info_locked(event)
|
||||
self.publish_instance_list_update()
|
||||
|
||||
def remove_instance(self, event: InstanceEvent) -> None:
|
||||
"""Mark an instance as removed / down based on an incoming event.
|
||||
|
||||
This marks the instance's status as DOWN or ERROR (depending on the
|
||||
event) but keeps it in the internal registry. It is removed from the
|
||||
*active* instance list published to hubs.
|
||||
"""
|
||||
with self._lock:
|
||||
self._remove_instance_locked(event)
|
||||
self.publish_instance_list_update()
|
||||
|
||||
def publish_instance_list_update(self) -> None:
|
||||
"""Publish the current active instance list to all subscribers."""
|
||||
active_list = self.get_active_instances()
|
||||
payload = asdict(active_list)
|
||||
data = json.dumps(payload).encode("utf-8")
|
||||
|
||||
with self._pub_lock:
|
||||
try:
|
||||
# PUB socket is best-effort; drop update if not ready.
|
||||
self._pub.send(data, flags=zmq.NOBLOCK)
|
||||
except (zmq.Again, zmq.ZMQError):
|
||||
# Silently ignore send failures; next update will catch up.
|
||||
return
|
||||
|
||||
def _schedule_broadcast(self, force: bool) -> None:
|
||||
"""Schedule a broadcast, optionally bypassing throttling.
|
||||
|
||||
When ``force`` is True, publish immediately. Otherwise, mark a pending
|
||||
broadcast that will be flushed by the periodic loop at most once per
|
||||
``_publish_min_interval``.
|
||||
"""
|
||||
if force:
|
||||
self.publish_instance_list_update()
|
||||
else:
|
||||
self._pending_broadcast = True
|
||||
|
||||
def _mark_instance_error_locked(self, info: InstanceInfo) -> None:
|
||||
"""Mark instance as ERROR (e.g. after heartbeat timeout)."""
|
||||
info.status = StageStatus.ERROR
|
||||
|
||||
def _check_heartbeat_timeouts(self) -> None:
|
||||
"""Mark instances as ERROR if their heartbeat has timed out."""
|
||||
now = time()
|
||||
timed_out = False
|
||||
gc_ttl = 600.0 # 10 minutes
|
||||
|
||||
with self._lock:
|
||||
to_delete: list[str] = []
|
||||
|
||||
for input_addr, info in self._instances.items():
|
||||
if info.status == StageStatus.UP and now - info.last_heartbeat > self._heartbeat_timeout:
|
||||
self._mark_instance_error_locked(info)
|
||||
timed_out = True
|
||||
elif info.status in (StageStatus.DOWN, StageStatus.ERROR) and now - info.last_heartbeat > gc_ttl:
|
||||
to_delete.append(input_addr)
|
||||
|
||||
for input_addr in to_delete:
|
||||
del self._instances[input_addr]
|
||||
if timed_out:
|
||||
# Instance liveness changed; force immediate broadcast.
|
||||
self._schedule_broadcast(force=True)
|
||||
|
||||
def close(self) -> None:
|
||||
"""Shut down background threads and close all ZMQ sockets."""
|
||||
if self._closed:
|
||||
raise RuntimeError("Coordinator already closed")
|
||||
|
||||
self._closed = True
|
||||
self._running = False
|
||||
self._stop_event.set()
|
||||
|
||||
# Wait for threads to exit before closing sockets.
|
||||
for thread in (self._recv_thread, self._periodic_thread):
|
||||
thread.join(timeout=1.0)
|
||||
|
||||
try:
|
||||
self._router.close(0)
|
||||
except zmq.ZMQError:
|
||||
pass
|
||||
|
||||
try:
|
||||
self._pub.close(0)
|
||||
except zmq.ZMQError:
|
||||
pass
|
||||
|
||||
try:
|
||||
self._ctx.term()
|
||||
except zmq.ZMQError:
|
||||
pass
|
||||
|
||||
def _parse_instance_event(self, data: dict[str, Any]) -> InstanceEvent | None:
|
||||
"""Parse wire payload dict into InstanceEvent. Returns None if invalid."""
|
||||
try:
|
||||
return InstanceEvent(
|
||||
input_addr=str(data["input_addr"]),
|
||||
output_addr=str(data["output_addr"]),
|
||||
stage_id=int(data["stage_id"]),
|
||||
event_type=str(data["event_type"]),
|
||||
status=StageStatus(data.get("status")),
|
||||
queue_length=data.get("queue_length"),
|
||||
)
|
||||
except (KeyError, ValueError, TypeError):
|
||||
return None
|
||||
|
||||
def _recv_loop(self) -> None:
|
||||
"""Background loop that receives and processes instance events."""
|
||||
while self._running:
|
||||
try:
|
||||
frames = self._router.recv_multipart()
|
||||
except zmq.Again:
|
||||
# RCVTIMEO expired, loop to recheck _running.
|
||||
continue
|
||||
except zmq.ZMQError:
|
||||
# Socket likely closed or context terminated.
|
||||
break
|
||||
|
||||
if not frames:
|
||||
continue
|
||||
|
||||
payload = frames[-1]
|
||||
try:
|
||||
data = json.loads(payload.decode("utf-8"))
|
||||
event = self._parse_instance_event(data)
|
||||
except json.JSONDecodeError as e:
|
||||
logger.warning("Invalid JSON in instance event, dropping: %s", e)
|
||||
continue
|
||||
if event is None:
|
||||
logger.warning("Malformed instance event, dropping")
|
||||
continue
|
||||
|
||||
self._handle_event(event)
|
||||
|
||||
def _periodic_loop(self) -> None:
|
||||
"""Periodic loop to check heartbeat timeouts and flush broadcasts.
|
||||
|
||||
Heartbeat timeouts are checked on their original cadence, while
|
||||
queue_length / non-liveness updates are coalesced and flushed at
|
||||
most once per ``_publish_min_interval``.
|
||||
"""
|
||||
heartbeat_interval = max(1.0, min(self._heartbeat_timeout / 2.0, 5.0))
|
||||
loop_interval = self._publish_min_interval
|
||||
|
||||
last_heartbeat_check = 0.0
|
||||
while self._running:
|
||||
now = time()
|
||||
|
||||
if now - last_heartbeat_check >= heartbeat_interval:
|
||||
self._check_heartbeat_timeouts()
|
||||
last_heartbeat_check = now
|
||||
|
||||
if self._pending_broadcast:
|
||||
self.publish_instance_list_update()
|
||||
self._pending_broadcast = False
|
||||
|
||||
if self._stop_event.wait(timeout=loop_interval):
|
||||
break
|
||||
|
||||
def _handle_event(self, event: InstanceEvent) -> None:
|
||||
"""Dispatch an incoming event to the appropriate handler."""
|
||||
try:
|
||||
input_addr = event.input_addr
|
||||
|
||||
# Heartbeat: only update last_heartbeat; if previously ERROR,
|
||||
# promote back to UP and broadcast once.
|
||||
if event.event_type == "heartbeat":
|
||||
promote = False
|
||||
with self._lock:
|
||||
info = self._instances.get(input_addr)
|
||||
if info is not None:
|
||||
info.last_heartbeat = time()
|
||||
if info.status == StageStatus.ERROR:
|
||||
info.status = StageStatus.UP
|
||||
promote = True
|
||||
if promote:
|
||||
self._schedule_broadcast(force=True)
|
||||
return
|
||||
|
||||
# Check-and-act under single lock to avoid TOCTOU race (duplicate
|
||||
# registration when concurrent events arrive for the same instance).
|
||||
with self._lock:
|
||||
force_broadcast = False
|
||||
if input_addr not in self._instances:
|
||||
self._add_new_instance_locked(event)
|
||||
force_broadcast = True
|
||||
else:
|
||||
if event.status == StageStatus.DOWN:
|
||||
self._remove_instance_locked(event)
|
||||
force_broadcast = True
|
||||
else:
|
||||
self._update_instance_info_locked(event)
|
||||
|
||||
# New instances / DOWN events are broadcast immediately; other
|
||||
# updates (e.g. queue_length changes) are throttled via the
|
||||
# periodic loop.
|
||||
self._schedule_broadcast(force=force_broadcast)
|
||||
except (KeyError, ValueError, TypeError) as e:
|
||||
logger.warning("Dropping malformed event: %s", e)
|
||||
|
||||
def _add_new_instance_locked(self, event: InstanceEvent) -> None:
|
||||
input_addr = event.input_addr
|
||||
if not input_addr:
|
||||
raise KeyError("input_addr required")
|
||||
stage_id = event.stage_id
|
||||
if stage_id < 0:
|
||||
raise KeyError("stage_id required and must be non-negative")
|
||||
|
||||
now = time()
|
||||
info = InstanceInfo(
|
||||
input_addr=input_addr,
|
||||
output_addr=event.output_addr,
|
||||
stage_id=stage_id,
|
||||
status=event.status,
|
||||
queue_length=event.queue_length,
|
||||
last_heartbeat=now,
|
||||
registered_at=now,
|
||||
)
|
||||
self._instances[input_addr] = info
|
||||
|
||||
def _update_instance_info_locked(self, event: InstanceEvent) -> None:
|
||||
input_addr = event.input_addr
|
||||
info = self._instances[input_addr]
|
||||
|
||||
if event.status is not None:
|
||||
info.status = event.status
|
||||
|
||||
if event.queue_length is not None:
|
||||
info.queue_length = event.queue_length
|
||||
|
||||
def _remove_instance_locked(self, event: InstanceEvent) -> None:
|
||||
input_addr = event.input_addr
|
||||
info = self._instances.get(input_addr)
|
||||
if info is None:
|
||||
return
|
||||
|
||||
info.status = StageStatus.DOWN
|
||||
@@ -140,6 +140,13 @@ class AsyncOmni(OmniBase):
|
||||
getattr(self, "_inline_engine", None),
|
||||
)
|
||||
|
||||
async def get_supported_tasks(self) -> set[str]:
|
||||
"""Return supported tasks based on the configured stage output modalities."""
|
||||
tasks: set[str] = set()
|
||||
if "text" in self.output_modalities:
|
||||
tasks.add("generate")
|
||||
return tasks
|
||||
|
||||
def _create_default_diffusion_stage_cfg(self, kwargs: dict[str, Any]) -> dict[str, Any]:
|
||||
"""Create default diffusion stage configuration."""
|
||||
# TODO: here is different from the Omni class. We should merge the two in the future.
|
||||
@@ -157,6 +164,7 @@ class AsyncOmni(OmniBase):
|
||||
ring_degree = kwargs.get("ring_degree") or 1
|
||||
sequence_parallel_size = kwargs.get("sequence_parallel_size")
|
||||
tensor_parallel_size = kwargs.get("tensor_parallel_size") or 1
|
||||
enable_expert_parallel = kwargs.get("enable_expert_parallel") or False
|
||||
cfg_parallel_size = kwargs.get("cfg_parallel_size") or 1
|
||||
vae_patch_parallel_size = kwargs.get("vae_patch_parallel_size") or 1
|
||||
use_hsdp = kwargs.get("use_hsdp", False)
|
||||
@@ -179,6 +187,7 @@ class AsyncOmni(OmniBase):
|
||||
pipeline_parallel_size=1,
|
||||
data_parallel_size=1,
|
||||
tensor_parallel_size=tensor_parallel_size,
|
||||
enable_expert_parallel=enable_expert_parallel,
|
||||
sequence_parallel_size=sequence_parallel_size,
|
||||
ulysses_degree=ulysses_degree,
|
||||
ring_degree=ring_degree,
|
||||
|
||||
@@ -100,6 +100,7 @@ class AsyncOmniDiffusion:
|
||||
if cfg is None:
|
||||
raise ValueError(f"Could not find config.json or model_index.json for model {od_config.model}")
|
||||
|
||||
od_config.tf_model_config = TransformerConfig.from_dict(cfg)
|
||||
model_type = cfg.get("model_type")
|
||||
architectures = cfg.get("architectures") or []
|
||||
# Bagel/NextStep models don't have a model_index.json, so we set the pipeline class name manually
|
||||
|
||||
@@ -98,7 +98,7 @@ class OmniDiffusion:
|
||||
|
||||
if od_config.model_class_name is None:
|
||||
od_config.model_class_name = pipeline_class
|
||||
od_config.tf_model_config = TransformerConfig()
|
||||
od_config.tf_model_config = TransformerConfig().from_dict(cfg)
|
||||
od_config.update_multimodal_support()
|
||||
|
||||
if cfg_kv_collect_func is not None:
|
||||
|
||||
@@ -178,6 +178,10 @@ class _DiffusionServingModels:
|
||||
def __init__(self, base_model_paths: list[BaseModelPath]) -> None:
|
||||
self._base_model_paths = base_model_paths
|
||||
|
||||
@property
|
||||
def base_model_paths(self) -> list[BaseModelPath]:
|
||||
return self._base_model_paths
|
||||
|
||||
async def show_available_models(self) -> ModelList:
|
||||
return ModelList(
|
||||
data=[
|
||||
@@ -241,7 +245,10 @@ async def omni_run_server_worker(listen_address, sock, args, client_config=None,
|
||||
else:
|
||||
supported_tasks = ("generate",)
|
||||
if not supported_tasks:
|
||||
supported_tasks = ("generate",)
|
||||
# Only default to "generate" when get_supported_tasks is not implemented;
|
||||
# TTS-only models intentionally return an empty set.
|
||||
if not hasattr(engine_client, "get_supported_tasks"):
|
||||
supported_tasks = ("generate",)
|
||||
|
||||
# OMNI: Pass supported_tasks to build_app (required by upstream vLLM)
|
||||
app = build_openai_app(args, supported_tasks)
|
||||
|
||||
@@ -58,7 +58,7 @@ class OpenAICreateSpeechRequest(BaseModel):
|
||||
initial_codec_chunk_frames: int | None = Field(
|
||||
default=None,
|
||||
ge=0,
|
||||
description="Initial chunk size for reduced TTFA. Overrides stage config for this request.",
|
||||
description="Per-request initial chunk size override. If null, computed dynamically based on server load.",
|
||||
)
|
||||
|
||||
@field_validator("stream_format")
|
||||
|
||||
@@ -255,7 +255,7 @@ class MiMoAudioLLMDummyInputsBuilder(BaseDummyInputsBuilder[MiMoAudioLLMProcessi
|
||||
|
||||
dummy_processor_inputs = ProcessorInputs(
|
||||
prompt=dummy_text,
|
||||
mm_items=dummy_mm_items,
|
||||
mm_data_items=dummy_mm_items,
|
||||
)
|
||||
|
||||
return dummy_processor_inputs
|
||||
@@ -309,10 +309,6 @@ class MiMoAudioDataParser(MultiModalDataParser):
|
||||
if data is None:
|
||||
return AudioProcessorItems(None)
|
||||
|
||||
# also check single audio item with sampling rate
|
||||
if self._is_empty(data) or (isinstance(data, tuple) and self._is_empty(data[0])):
|
||||
return None
|
||||
|
||||
if (
|
||||
is_list_of(data, float)
|
||||
or isinstance(data, (np.ndarray, torch.Tensor))
|
||||
|
||||
@@ -159,6 +159,12 @@ class Qwen3OmniMoeForConditionalGeneration(
|
||||
# suppress tokens by setting their probability to ~1e-9 (finite very small)
|
||||
self.suppressed_tokens = self._get_talker_suppressed_tokens()
|
||||
self.requires_raw_input_tokens = True
|
||||
# Keys that should stay on GPU in model_intermediate_buffer to avoid CPU↔GPU round-trips
|
||||
self.gpu_resident_buffer_keys: set[str] = {
|
||||
"last_talker_hidden",
|
||||
"trailing_text_hidden",
|
||||
"tts_pad_embed_projected",
|
||||
}
|
||||
|
||||
elif self.model_stage == "code2wav":
|
||||
self.enable_update_additional_information = True
|
||||
@@ -223,14 +229,16 @@ class Qwen3OmniMoeForConditionalGeneration(
|
||||
|
||||
# ==================== Forward Pass ====================
|
||||
def _get_talker_suppressed_tokens(self):
|
||||
return [
|
||||
i
|
||||
for i in range(
|
||||
self.config.talker_config.text_config.vocab_size - 1024,
|
||||
self.config.talker_config.text_config.vocab_size,
|
||||
)
|
||||
if i != self.config.talker_config.codec_eos_token_id
|
||||
]
|
||||
"""Return a boolean mask on GPU for suppressed token positions."""
|
||||
vocab_size = self.config.talker_config.text_config.vocab_size
|
||||
mask = torch.zeros(vocab_size, dtype=torch.bool)
|
||||
start = vocab_size - 1024
|
||||
eos_id = self.config.talker_config.codec_eos_token_id
|
||||
for i in range(start, vocab_size):
|
||||
if i != eos_id:
|
||||
mask[i] = True
|
||||
# Will be moved to the correct device on first use
|
||||
return mask
|
||||
|
||||
def get_mrope_input_positions(
|
||||
self,
|
||||
@@ -578,7 +586,7 @@ class Qwen3OmniMoeForConditionalGeneration(
|
||||
Postprocess the talker hidden states.
|
||||
"""
|
||||
update_dict = {}
|
||||
update_dict["last_talker_hidden"] = hidden_states[-1, :].detach().to("cpu").contiguous()
|
||||
update_dict["last_talker_hidden"] = hidden_states[-1, :].detach()
|
||||
return update_dict
|
||||
|
||||
def talker_preprocess(self, input_ids: torch.Tensor, input_embeds: torch.Tensor, **info_dict: dict):
|
||||
@@ -632,9 +640,9 @@ class Qwen3OmniMoeForConditionalGeneration(
|
||||
if inputs_embeds.shape[-1] == 2048:
|
||||
inputs_embeds = self.text_projection(inputs_embeds)
|
||||
code_predictor_codes, summed_embeddings = self.talker.code_predictor_forward(
|
||||
input_ids, inputs_embeds.clone(), last_talker_hidden=last_talker_hidden
|
||||
input_ids, inputs_embeds, last_talker_hidden=last_talker_hidden
|
||||
)
|
||||
inputs_embeds = summed_embeddings.clone()
|
||||
inputs_embeds = summed_embeddings
|
||||
inputs_embeds = (inputs_embeds + text_step).reshape(-1, self.talker_config.text_config.hidden_size)
|
||||
return inputs_embeds, code_predictor_codes.squeeze(-1)
|
||||
|
||||
@@ -758,7 +766,7 @@ class Qwen3OmniMoeForConditionalGeneration(
|
||||
# compatible with old shape [1,S,D]
|
||||
rem_tail = trailing_text_hidden.squeeze(0)
|
||||
if rem_tail.shape[0] > 0:
|
||||
update_dict["trailing_text_hidden"] = rem_tail.detach().to("cpu").contiguous()
|
||||
update_dict["trailing_text_hidden"] = rem_tail.detach()
|
||||
# Also persist projected tts_pad for decode fallback if needed
|
||||
if isinstance(tts_pad_thinker, torch.Tensor):
|
||||
pad_in = tts_pad_thinker
|
||||
@@ -767,7 +775,7 @@ class Qwen3OmniMoeForConditionalGeneration(
|
||||
if pad_in.ndim == 1:
|
||||
pad_in = pad_in.view(1, 1, -1)
|
||||
pad_proj = self.talker.text_projection(pad_in.to(self._module_device(self.talker)))
|
||||
update_dict["tts_pad_embed_projected"] = pad_proj.detach().to("cpu").contiguous()
|
||||
update_dict["tts_pad_embed_projected"] = pad_proj.detach()
|
||||
except Exception:
|
||||
pass
|
||||
self._talker_cache_thinker_decode_embeds(info_dict, update_dict)
|
||||
@@ -926,7 +934,7 @@ class Qwen3OmniMoeForConditionalGeneration(
|
||||
if isinstance(q_tail, torch.Tensor) and q_tail.numel() > 0:
|
||||
use_vec = q_tail[0:1, :]
|
||||
new_q_tail = (
|
||||
q_tail[1:, :].detach().to("cpu").contiguous()
|
||||
q_tail[1:, :].detach()
|
||||
if q_tail.shape[0] > 1
|
||||
else self.tts_pad_embed.to(input_embeds.device, dtype=input_embeds.dtype)
|
||||
)
|
||||
@@ -1122,16 +1130,10 @@ class Qwen3OmniMoeForConditionalGeneration(
|
||||
# implemented by assigning their logits to log(1e-9).
|
||||
|
||||
if getattr(self, "model_stage", None) == "talker" and isinstance(logits, torch.Tensor):
|
||||
try:
|
||||
logits_cpu = logits.cpu()
|
||||
logits_cpu[:, self.suppressed_tokens] = -1e9
|
||||
logits = logits_cpu.to(logits.device)
|
||||
except Exception as e:
|
||||
print(f"Error in logits suppression: {e}")
|
||||
print(f"logits.shape: {logits.shape}")
|
||||
print(f"self.suppressed_tokens: {self.suppressed_tokens}")
|
||||
raise e
|
||||
logits[:, self.suppressed_tokens] = -1e9
|
||||
# Move mask to device once (lazy), then reuse every step
|
||||
if self.suppressed_tokens.device != logits.device:
|
||||
self.suppressed_tokens = self.suppressed_tokens.to(logits.device)
|
||||
logits.masked_fill_(self.suppressed_tokens.unsqueeze(0), -1e9)
|
||||
return logits
|
||||
|
||||
def sample(
|
||||
|
||||
@@ -1,27 +1,16 @@
|
||||
"""Qwen3-Omni Code Predictor with MTP (Multi-Token Prediction) support.
|
||||
"""Qwen3-Omni Code Predictor -- optimized re-prefill, no KV cache.
|
||||
|
||||
This module implements the code predictor component for Qwen3-Omni talker models.
|
||||
|
||||
The code predictor generates residual RVQ (Residual Vector Quantization) codes
|
||||
autoregressively, predicting layers 1 to N based on layer-0 codes from the talker.
|
||||
* SDPA attention (F.scaled_dot_product_attention) -- no HF backend fallback
|
||||
* Persistent pre-allocated buffers (_proj_buf, _pos_ids) -- zero per-call alloc
|
||||
* Inline top-k sampling -- no LogitsProcessorList / custom-op overhead
|
||||
* torch.compile on inner transformer by default
|
||||
* No @support_torch_compile / static_forward_context / namedtuple
|
||||
"""
|
||||
|
||||
from collections import namedtuple
|
||||
from typing import Any
|
||||
|
||||
import torch
|
||||
import torch.nn as nn
|
||||
import torch.nn.functional as F
|
||||
from transformers import Cache, PretrainedConfig
|
||||
from transformers.generation.logits_process import (
|
||||
LogitsProcessorList,
|
||||
TopKLogitsWarper,
|
||||
TopPLogitsWarper,
|
||||
)
|
||||
from transformers.modeling_utils import ALL_ATTENTION_FUNCTIONS
|
||||
from vllm.compilation.decorators import support_torch_compile
|
||||
from vllm.config import CacheConfig, ModelConfig, VllmConfig, get_current_vllm_config
|
||||
from vllm.forward_context import get_forward_context
|
||||
from vllm.config import VllmConfig
|
||||
from vllm.logger import init_logger
|
||||
from vllm.model_executor.layers.layernorm import RMSNorm
|
||||
from vllm.model_executor.layers.linear import (
|
||||
@@ -32,47 +21,51 @@ from vllm.model_executor.layers.linear import (
|
||||
from vllm.model_executor.layers.quantization import QuantizationConfig
|
||||
from vllm.model_executor.layers.rotary_embedding import get_rope
|
||||
from vllm.model_executor.layers.vocab_parallel_embedding import VocabParallelEmbedding
|
||||
from vllm.utils.torch_utils import direct_register_custom_op
|
||||
|
||||
logger = init_logger(__name__)
|
||||
|
||||
# ============================================================================
|
||||
# Code Predictor Attention Layer
|
||||
# ============================================================================
|
||||
|
||||
# ===================================================================
|
||||
# Standalone Attention (SDPA, no KV cache, no HF backend fallback)
|
||||
# ===================================================================
|
||||
|
||||
|
||||
class Qwen3OmniCodePredictorAttention(nn.Module):
|
||||
"""Multi-head self-attention for code predictor with vLLM optimization."""
|
||||
"""Multi-head self-attention for code predictor.
|
||||
|
||||
Uses ``F.scaled_dot_product_attention`` directly. No KV cache -- the code
|
||||
predictor always re-prefills the full (short) sequence each AR step.
|
||||
|
||||
Input : [B, seq_len, hidden_size]
|
||||
Output: [B, seq_len, hidden_size]
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
config,
|
||||
layer_idx: int,
|
||||
vllm_config: VllmConfig = None,
|
||||
quant_config: QuantizationConfig | None = None,
|
||||
prefix: str = "",
|
||||
):
|
||||
super().__init__()
|
||||
|
||||
self.num_heads = config.code_predictor_config.num_attention_heads
|
||||
self.num_key_value_heads = config.code_predictor_config.num_key_value_heads
|
||||
cp_cfg = config.code_predictor_config
|
||||
self.num_heads = cp_cfg.num_attention_heads
|
||||
self.num_kv_heads = cp_cfg.num_key_value_heads
|
||||
self.head_dim = getattr(
|
||||
config.code_predictor_config,
|
||||
cp_cfg,
|
||||
"head_dim",
|
||||
config.code_predictor_config.hidden_size // config.code_predictor_config.num_attention_heads,
|
||||
cp_cfg.hidden_size // cp_cfg.num_attention_heads,
|
||||
)
|
||||
self.hidden_size = config.code_predictor_config.hidden_size
|
||||
|
||||
if self.num_heads % self.num_key_value_heads != 0:
|
||||
raise ValueError("num_attention_heads must be divisible by num_key_value_heads")
|
||||
|
||||
self.num_key_value_groups = self.num_heads // self.num_key_value_heads
|
||||
self.hidden_size = cp_cfg.hidden_size
|
||||
self.q_size = self.num_heads * self.head_dim
|
||||
self.kv_size = self.num_kv_heads * self.head_dim
|
||||
self.scaling = self.head_dim**-0.5
|
||||
self._use_gqa = self.num_kv_heads != self.num_heads
|
||||
|
||||
self.qkv_proj = QKVParallelLinear(
|
||||
hidden_size=self.hidden_size,
|
||||
head_size=self.head_dim,
|
||||
total_num_heads=self.num_heads,
|
||||
total_num_kv_heads=self.num_key_value_heads,
|
||||
total_num_kv_heads=self.num_kv_heads,
|
||||
bias=False,
|
||||
quant_config=quant_config,
|
||||
prefix=f"{prefix}.qkv_proj",
|
||||
@@ -88,120 +81,55 @@ class Qwen3OmniCodePredictorAttention(nn.Module):
|
||||
)
|
||||
self.rotary_emb = get_rope(
|
||||
self.head_dim,
|
||||
max_position=config.code_predictor_config.max_position_embeddings,
|
||||
max_position=cp_cfg.max_position_embeddings,
|
||||
rope_parameters=None,
|
||||
dual_chunk_attention_config=None,
|
||||
)
|
||||
|
||||
self.q_size = self.num_heads * self.head_dim
|
||||
self.kv_size = self.num_key_value_heads * self.head_dim
|
||||
|
||||
# Query/Key normalization
|
||||
self.q_norm = RMSNorm(self.head_dim, eps=config.code_predictor_config.rms_norm_eps)
|
||||
self.k_norm = RMSNorm(self.head_dim, eps=config.code_predictor_config.rms_norm_eps)
|
||||
self.is_causal = True
|
||||
self.config = config
|
||||
|
||||
self.attention_backends = ["flash_attention_2", "xformers", "eager", "sdpa"]
|
||||
cudagraph_mode = get_current_vllm_config().compilation_config.cudagraph_mode
|
||||
if "flash_attention_2" in ALL_ATTENTION_FUNCTIONS and cudagraph_mode.has_full_cudagraphs():
|
||||
logger.warning(
|
||||
f"CUDAGraphMode.{cudagraph_mode.name} is currently not supported "
|
||||
f"with flash attention for Qwen3-Omni talker MTP."
|
||||
f"removing flash attention from attention_backends"
|
||||
)
|
||||
self.attention_backends.remove("flash_attention_2")
|
||||
self.q_norm = RMSNorm(self.head_dim, eps=cp_cfg.rms_norm_eps)
|
||||
self.k_norm = RMSNorm(self.head_dim, eps=cp_cfg.rms_norm_eps)
|
||||
|
||||
def forward(
|
||||
self,
|
||||
hidden_states: torch.Tensor,
|
||||
past_key_values: Cache | None = None,
|
||||
cache_position: torch.LongTensor | None = None,
|
||||
use_cache: bool = False,
|
||||
position_ids: torch.LongTensor | None = None,
|
||||
position_ids: torch.Tensor,
|
||||
) -> torch.Tensor:
|
||||
bsz, seq_len, _ = hidden_states.shape
|
||||
|
||||
qkv, _ = self.qkv_proj(hidden_states)
|
||||
# Flatten to 2-D so vLLM rotary_emb gets [num_tokens, size]
|
||||
qkv, _ = self.qkv_proj(hidden_states.reshape(bsz * seq_len, -1))
|
||||
q, k, v = qkv.split([self.q_size, self.kv_size, self.kv_size], dim=-1)
|
||||
|
||||
# Reshape for attention
|
||||
q = q.reshape(bsz, seq_len, self.num_heads, self.head_dim)
|
||||
k = k.reshape(bsz, seq_len, self.num_key_value_heads, self.head_dim)
|
||||
v = v.reshape(bsz, seq_len, self.num_key_value_heads, self.head_dim)
|
||||
|
||||
# Apply normalization
|
||||
q = self.q_norm(q).contiguous()
|
||||
k = self.k_norm(k).contiguous()
|
||||
q = q.reshape(-1, self.q_size)
|
||||
k = k.reshape(-1, self.kv_size)
|
||||
|
||||
# Apply RoPE
|
||||
# QK-norm -> RoPE (both 2-D)
|
||||
q = self.q_norm(q.view(-1, self.num_heads, self.head_dim)).view(q.shape)
|
||||
k = self.k_norm(k.view(-1, self.num_kv_heads, self.head_dim)).view(k.shape)
|
||||
q, k = self.rotary_emb(position_ids, q, k)
|
||||
|
||||
# Reshape for attention
|
||||
q = q.reshape(bsz, seq_len, self.num_heads, self.head_dim)
|
||||
k = k.reshape(bsz, seq_len, self.num_key_value_heads, self.head_dim)
|
||||
# [B, heads, seq, head_dim] for SDPA
|
||||
q = q.view(bsz, seq_len, self.num_heads, self.head_dim).transpose(1, 2)
|
||||
k = k.view(bsz, seq_len, self.num_kv_heads, self.head_dim).transpose(1, 2)
|
||||
v = v.view(bsz, seq_len, self.num_kv_heads, self.head_dim).transpose(1, 2)
|
||||
|
||||
v_heads = v.transpose(1, 2).contiguous()
|
||||
q_heads = q.transpose(1, 2).contiguous()
|
||||
k_heads = k.transpose(1, 2).contiguous()
|
||||
attn_out = F.scaled_dot_product_attention(
|
||||
q,
|
||||
k,
|
||||
v,
|
||||
scale=self.scaling,
|
||||
is_causal=True,
|
||||
enable_gqa=self._use_gqa,
|
||||
)
|
||||
|
||||
if past_key_values is not None:
|
||||
sin, cos = self.rotary_emb.get_cos_sin(seq_len)
|
||||
# sin and cos are specific to RoPE models; cache_position needed for the static cache
|
||||
cache_kwargs = {"sin": sin, "cos": cos, "cache_position": cache_position}
|
||||
k_heads, v_heads = past_key_values.update(k_heads, v_heads, self.layer_idx, cache_kwargs)
|
||||
|
||||
# Try attention backends in order of preference, with runtime error handling
|
||||
# This handles cases where the backend is registered but not actually available
|
||||
attn_output = None
|
||||
last_error = None
|
||||
|
||||
for backend_name in self.attention_backends:
|
||||
if backend_name not in ALL_ATTENTION_FUNCTIONS:
|
||||
continue
|
||||
|
||||
try:
|
||||
attention_interface = ALL_ATTENTION_FUNCTIONS[backend_name]
|
||||
attn_output, _ = attention_interface(
|
||||
self,
|
||||
q_heads,
|
||||
k_heads,
|
||||
v_heads,
|
||||
None,
|
||||
dropout=0.0 if not self.training else getattr(self, "attention_dropout", 0.0),
|
||||
scaling=self.head_dim**-0.5,
|
||||
sliding_window=None,
|
||||
use_cache=use_cache,
|
||||
position_ids=position_ids[:seq_len].unsqueeze(0),
|
||||
output_hidden_states=True,
|
||||
output_attentions=False,
|
||||
)
|
||||
break
|
||||
except (ValueError, ImportError, RuntimeError, AttributeError) as e:
|
||||
# Store error and try next backend
|
||||
last_error = e
|
||||
continue
|
||||
|
||||
if attn_output is None:
|
||||
raise RuntimeError(
|
||||
f"All attention backends failed. Last error: {last_error}. "
|
||||
"Please install flash-attn, or ensure PyTorch's scaled_dot_product_attention is available."
|
||||
)
|
||||
attn_output = attn_output.reshape(*(hidden_states.shape[:-1]), -1).contiguous()
|
||||
|
||||
attn_output, _ = self.o_proj(attn_output)
|
||||
return attn_output
|
||||
attn_out = attn_out.transpose(1, 2).reshape(bsz * seq_len, -1)
|
||||
output, _ = self.o_proj(attn_out)
|
||||
return output.view(bsz, seq_len, -1)
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Code Predictor MLP Layer
|
||||
# ============================================================================
|
||||
# ===================================================================
|
||||
# MLP
|
||||
# ===================================================================
|
||||
|
||||
|
||||
class Qwen3OmniCodePredictorMLP(nn.Module):
|
||||
"""Feed-forward network for code predictor with fused gate/up projection."""
|
||||
"""SiLU-gated MLP for code predictor."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
@@ -221,7 +149,6 @@ class Qwen3OmniCodePredictorMLP(nn.Module):
|
||||
prefix=f"{prefix}.gate_up_proj",
|
||||
disable_tp=True,
|
||||
)
|
||||
|
||||
self.down_proj = RowParallelLinear(
|
||||
input_size=intermediate_size,
|
||||
output_size=hidden_size,
|
||||
@@ -238,35 +165,23 @@ class Qwen3OmniCodePredictorMLP(nn.Module):
|
||||
return down
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# MTP Layer (Multi-Token Prediction Layer)
|
||||
# ============================================================================
|
||||
# ===================================================================
|
||||
# Decoder Layer
|
||||
# ===================================================================
|
||||
|
||||
|
||||
class Qwen3OmniCodePredictorMTPLayer(nn.Module):
|
||||
"""MTP layer for speculative decoding - predicts next residual code layer."""
|
||||
class Qwen3OmniCodePredictorDecoderLayer(nn.Module):
|
||||
"""Transformer decoder layer (SDPA, no KV cache)."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
config: PretrainedConfig,
|
||||
prefix: str,
|
||||
model_config: ModelConfig,
|
||||
layer_idx: int,
|
||||
cache_config: CacheConfig | None = None,
|
||||
config,
|
||||
quant_config: QuantizationConfig | None = None,
|
||||
prefix: str = "",
|
||||
) -> None:
|
||||
super().__init__()
|
||||
self.layer_idx = layer_idx
|
||||
self.config = config
|
||||
|
||||
self.self_attn = Qwen3OmniCodePredictorAttention(
|
||||
config,
|
||||
layer_idx,
|
||||
vllm_config=type(
|
||||
"VllmConfig",
|
||||
(),
|
||||
{"cache_config": cache_config, "quant_config": quant_config, "model_config": model_config},
|
||||
)(),
|
||||
quant_config=quant_config,
|
||||
prefix=f"{prefix}.self_attn",
|
||||
)
|
||||
@@ -275,260 +190,243 @@ class Qwen3OmniCodePredictorMTPLayer(nn.Module):
|
||||
quant_config=quant_config,
|
||||
prefix=f"{prefix}.mlp",
|
||||
)
|
||||
self.input_layernorm = RMSNorm(
|
||||
config.code_predictor_config.hidden_size, eps=config.code_predictor_config.rms_norm_eps
|
||||
)
|
||||
self.post_attention_layernorm = RMSNorm(
|
||||
config.code_predictor_config.hidden_size, eps=config.code_predictor_config.rms_norm_eps
|
||||
)
|
||||
cp_cfg = config.code_predictor_config
|
||||
self.input_layernorm = RMSNorm(cp_cfg.hidden_size, eps=cp_cfg.rms_norm_eps)
|
||||
self.post_attention_layernorm = RMSNorm(cp_cfg.hidden_size, eps=cp_cfg.rms_norm_eps)
|
||||
|
||||
def mtp_block(
|
||||
def forward(
|
||||
self,
|
||||
hidden_states: torch.Tensor,
|
||||
past_key_values: Cache | None = None,
|
||||
cache_position: torch.LongTensor | None = None,
|
||||
use_cache: bool = False,
|
||||
position_ids: torch.LongTensor | None = None,
|
||||
position_ids: torch.Tensor,
|
||||
) -> torch.Tensor:
|
||||
# Self-attention with residual
|
||||
residual = hidden_states
|
||||
hidden_states = self.input_layernorm(hidden_states)
|
||||
hidden_states = self.self_attn(hidden_states, past_key_values, cache_position, use_cache, position_ids)
|
||||
hidden_states = self.self_attn(hidden_states, position_ids)
|
||||
hidden_states = residual + hidden_states
|
||||
|
||||
# MLP with residual
|
||||
residual = hidden_states
|
||||
hidden_states = self.post_attention_layernorm(hidden_states)
|
||||
hidden_states = self.mlp(hidden_states)
|
||||
hidden_states = residual + hidden_states
|
||||
|
||||
return hidden_states
|
||||
|
||||
|
||||
class Qwen3OmniCodePredictorBaseModel(nn.Module):
|
||||
"""
|
||||
Base model for code predictor - matches HF Qwen3OmniMoeTalkerCodePredictorModel structure.
|
||||
# ===================================================================
|
||||
# Base Transformer Model (re-prefill, no KV cache)
|
||||
# ===================================================================
|
||||
|
||||
This is a simple transformer that processes inputs_embeds and outputs hidden states.
|
||||
|
||||
class Qwen3OmniCodePredictorBaseModel(nn.Module):
|
||||
"""Inner transformer for code predictor.
|
||||
|
||||
Signature: ``forward(inputs_embeds, position_ids) -> hidden_states``
|
||||
-- plain Tensor in, plain Tensor out (no namedtuple).
|
||||
"""
|
||||
|
||||
def __init__(self, *, vllm_config: VllmConfig, prefix: str = ""):
|
||||
super().__init__()
|
||||
config = vllm_config.model_config.hf_config.code_predictor_config
|
||||
|
||||
self.config = config
|
||||
self.vocab_size = config.vocab_size
|
||||
self.num_code_groups = config.num_code_groups
|
||||
|
||||
# Codec embeddings (for layers 1-num_code_groups-1)
|
||||
self.codec_embedding = nn.ModuleList(
|
||||
[
|
||||
VocabParallelEmbedding(
|
||||
config.vocab_size,
|
||||
config.hidden_size,
|
||||
)
|
||||
for _ in range(config.num_code_groups - 1)
|
||||
]
|
||||
[VocabParallelEmbedding(config.vocab_size, config.hidden_size) for _ in range(config.num_code_groups - 1)]
|
||||
)
|
||||
|
||||
# Decoder layers
|
||||
self.layers = nn.ModuleList(
|
||||
[
|
||||
Qwen3OmniCodePredictorMTPLayer(
|
||||
Qwen3OmniCodePredictorDecoderLayer(
|
||||
vllm_config.model_config.hf_config,
|
||||
f"{prefix}.layers.{idx}",
|
||||
model_config=vllm_config.model_config,
|
||||
layer_idx=idx,
|
||||
cache_config=vllm_config.cache_config,
|
||||
quant_config=vllm_config.quant_config,
|
||||
prefix=f"{prefix}.layers.{idx}",
|
||||
)
|
||||
for idx in range(config.num_hidden_layers)
|
||||
]
|
||||
)
|
||||
|
||||
self.norm = RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
|
||||
|
||||
def forward(
|
||||
self,
|
||||
inputs_embeds: torch.Tensor,
|
||||
attention_mask: torch.Tensor | None = None,
|
||||
position_ids: torch.LongTensor | None = None,
|
||||
past_key_values: Any | None = None,
|
||||
use_cache: bool | None = False,
|
||||
cache_position: torch.LongTensor | None = None,
|
||||
**kwargs: Any,
|
||||
) -> Any:
|
||||
"""
|
||||
Forward pass matching HF structure.
|
||||
|
||||
Args:
|
||||
inputs_embeds: [batch, seq_len, hidden_size]
|
||||
position_ids: Optional position IDs tensor
|
||||
past_key_values: Optional cached key-value pairs
|
||||
use_cache: Whether to use cache
|
||||
cache_position: Optional cache position tensor
|
||||
**kwargs: Additional keyword arguments
|
||||
|
||||
Returns:
|
||||
Named tuple with .last_hidden_state and .past_key_values attributes
|
||||
"""
|
||||
batch_size, seq_len, _ = inputs_embeds.shape
|
||||
# Forward through decoder layers
|
||||
position_ids: torch.Tensor,
|
||||
) -> torch.Tensor:
|
||||
hidden_states = inputs_embeds
|
||||
|
||||
for layer in self.layers:
|
||||
hidden_states = layer.mtp_block(hidden_states, past_key_values, cache_position, use_cache, position_ids)
|
||||
|
||||
# Final norm
|
||||
hidden_states = layer(hidden_states, position_ids)
|
||||
hidden_states = self.norm(hidden_states)
|
||||
|
||||
# Return in HF-compatible format
|
||||
Output = namedtuple("Output", ["last_hidden_state", "past_key_values"])
|
||||
return Output(last_hidden_state=hidden_states, past_key_values=None) # [batch, num_code_groups-1, hidden_size]
|
||||
|
||||
def get_input_embeddings(self):
|
||||
"""Return codec embeddings for HF compatibility."""
|
||||
return self.codec_embedding
|
||||
return hidden_states
|
||||
|
||||
|
||||
def code_predictor_sample(
|
||||
logits: torch.Tensor,
|
||||
layer_name: str,
|
||||
) -> torch.Tensor:
|
||||
forward_context = get_forward_context()
|
||||
self = forward_context.no_compile_layers[layer_name]
|
||||
logits = self.logits_processors(None, logits[:, -1])
|
||||
probs = F.softmax(logits, dim=-1)
|
||||
code = torch.multinomial(probs.squeeze(1), num_samples=1) # [batch, 1]
|
||||
return code
|
||||
# ===================================================================
|
||||
# Code Predictor Wrapper (optimized re-prefill, persistent buffers)
|
||||
# ===================================================================
|
||||
|
||||
|
||||
def code_predictor_sample_fake(
|
||||
logits: torch.Tensor,
|
||||
layer_name: str,
|
||||
) -> torch.Tensor:
|
||||
return torch.empty((logits.shape[0], 1), dtype=torch.int64, device=logits.device)
|
||||
|
||||
|
||||
direct_register_custom_op(
|
||||
op_name="qwen3_omni_code_predictor_sample",
|
||||
op_func=code_predictor_sample,
|
||||
fake_impl=code_predictor_sample_fake,
|
||||
)
|
||||
|
||||
|
||||
@support_torch_compile
|
||||
class Qwen3OmniMoeTalkerCodePredictor(nn.Module):
|
||||
"""
|
||||
Code predictor wrapper matching HF structure.
|
||||
"""Optimized code predictor -- re-prefill approach, no KV cache.
|
||||
|
||||
Structure:
|
||||
- self.model: Qwen3OmniCodePredictorBaseModel (transformer)
|
||||
- self.lm_head: ModuleList of output heads
|
||||
Each AR step forwards the full growing sequence (len 2 -> num_code_groups+1)
|
||||
through the transformer. The extra O(T^2) FLOPs are negligible for
|
||||
short sequences, and this avoids all KV-cache management overhead.
|
||||
|
||||
Optimizations:
|
||||
1. Pre-allocated embedding buffer -- no torch.cat per step.
|
||||
2. Pre-allocated position_ids -- no torch.arange per step.
|
||||
3. Inline top-k sampling -- no LogitsProcessorList / custom op.
|
||||
4. Cached module references -- bypass ModuleList indexing.
|
||||
5. torch.compile on inner transformer.
|
||||
"""
|
||||
|
||||
def __init__(self, *, vllm_config: VllmConfig, prefix: str = ""):
|
||||
super().__init__()
|
||||
|
||||
talker_code_predictor_config = vllm_config.model_config.hf_config
|
||||
config = vllm_config.model_config.hf_config
|
||||
self.config = config
|
||||
self.quant_config = vllm_config.quant_config
|
||||
self.prefix = prefix
|
||||
|
||||
self.config = talker_code_predictor_config
|
||||
self.vocab_size = self.config.code_predictor_config.vocab_size
|
||||
self.num_code_groups = self.config.code_predictor_config.num_code_groups
|
||||
self.num_code_groups = config.code_predictor_config.num_code_groups
|
||||
self._hidden_size = config.code_predictor_config.hidden_size
|
||||
|
||||
# Base transformer model (matches HF structure)
|
||||
self.model = Qwen3OmniCodePredictorBaseModel(vllm_config=vllm_config, prefix=prefix)
|
||||
self.model = Qwen3OmniCodePredictorBaseModel(
|
||||
vllm_config=vllm_config,
|
||||
prefix=prefix,
|
||||
)
|
||||
|
||||
# Output heads for each residual layer (1-num_layers-1)
|
||||
# One lm_head per residual layer (layers 1 .. G-1)
|
||||
self.lm_head = nn.ModuleList(
|
||||
[
|
||||
nn.Linear(
|
||||
self.config.code_predictor_config.hidden_size,
|
||||
self.config.code_predictor_config.vocab_size,
|
||||
config.code_predictor_config.hidden_size,
|
||||
config.code_predictor_config.vocab_size,
|
||||
bias=False,
|
||||
)
|
||||
for _ in range(self.num_code_groups - 1)
|
||||
]
|
||||
)
|
||||
self.logits_processors = LogitsProcessorList(
|
||||
[
|
||||
TopKLogitsWarper(top_k=50),
|
||||
TopPLogitsWarper(top_p=0.8),
|
||||
]
|
||||
|
||||
# Sampling hyperparams (inlined)
|
||||
self._top_k = 50
|
||||
|
||||
# Persistent buffers (lazily initialised on first forward)
|
||||
self._proj_buf: torch.Tensor | None = None
|
||||
self._pos_ids: torch.Tensor | None = None
|
||||
|
||||
# Cached plain-list refs (set once)
|
||||
self._lm_heads: list | None = None
|
||||
self._codec_embeds: list | None = None
|
||||
|
||||
# Model forward (optionally compiled)
|
||||
self._model_fwd: object | None = None
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Lazy-init helpers
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def _ensure_buffers(self, bsz: int, device: torch.device, dtype: torch.dtype) -> None:
|
||||
max_seq = self.num_code_groups + 1
|
||||
if (
|
||||
self._proj_buf is not None
|
||||
and self._proj_buf.shape[0] >= bsz
|
||||
and self._proj_buf.device == device
|
||||
and self._proj_buf.dtype == dtype
|
||||
):
|
||||
return
|
||||
self._proj_buf = torch.zeros(bsz, max_seq, self._hidden_size, dtype=dtype, device=device)
|
||||
self._pos_ids = torch.arange(max_seq, dtype=torch.long, device=device)
|
||||
|
||||
def _ensure_cached_refs(self) -> None:
|
||||
if self._lm_heads is not None:
|
||||
return
|
||||
self._lm_heads = list(self.lm_head)
|
||||
self._codec_embeds = list(self.model.codec_embedding)
|
||||
|
||||
def _ensure_model_fwd(self) -> None:
|
||||
if self._model_fwd is not None:
|
||||
return
|
||||
self._model_fwd = torch.compile(
|
||||
self.model.forward,
|
||||
mode="default",
|
||||
dynamic=True,
|
||||
)
|
||||
logger.info("code_predictor: torch.compile enabled")
|
||||
|
||||
compilation_config = get_current_vllm_config().compilation_config
|
||||
if prefix in compilation_config.static_forward_context:
|
||||
raise ValueError(f"Duplicate layer name: {prefix}")
|
||||
compilation_config.static_forward_context[prefix] = self
|
||||
self.layer_name = prefix
|
||||
# ------------------------------------------------------------------
|
||||
# Forward -- re-prefill + persistent buffers + inline sampling
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
@torch.inference_mode()
|
||||
def forward(
|
||||
self,
|
||||
layer0_code: torch.Tensor,
|
||||
layer0_embed: torch.Tensor,
|
||||
last_talker_hidden: torch.Tensor,
|
||||
) -> tuple[torch.Tensor, torch.Tensor]:
|
||||
"""
|
||||
Forward pass for code predictor.
|
||||
"""Predict residual codebooks 1..G-1 autoregressively via re-prefill.
|
||||
|
||||
Args:
|
||||
layer0_code:
|
||||
Code index for code-group (layer) 0.
|
||||
Shape: [batch_size, 1], dtype typically int64.
|
||||
|
||||
last_talker_hidden:
|
||||
|
||||
Shape: [batch_size, hidden_size].
|
||||
layer0_code: [bsz, 1] int64
|
||||
layer0_embed: [bsz, 1, hidden_size]
|
||||
last_talker_hidden: [bsz, 1, hidden_size]
|
||||
|
||||
Returns:
|
||||
pos_all_layers:
|
||||
Predicted codes for all code groups, including `layer0_code`.
|
||||
Shape: [batch_size, num_code_groups, 1].
|
||||
|
||||
current_input:
|
||||
The final input embedding sequence after appending embeddings of all
|
||||
predicted codes (one token per predicted layer).
|
||||
Shape: [batch_size, num_code_groups + 2, hidden_size].
|
||||
all_codes: [bsz, num_code_groups, 1]
|
||||
proj_buf: [bsz, num_code_groups + 1, hidden_size]
|
||||
pos 0 = last_talker_hidden (NOT a codec embed)
|
||||
pos 1 = layer0_embed
|
||||
pos 2.. = codec_embedding[i](predicted_code_i)
|
||||
"""
|
||||
pos_codes = [layer0_code] # Start with layer 0: [batch, 1]
|
||||
try:
|
||||
current_input = torch.cat([last_talker_hidden, layer0_embed], dim=1) # [batch, 2, hidden_size]
|
||||
except Exception as e:
|
||||
print(f"Error in current_input: {e}")
|
||||
print(f"last_talker_hidden shape: {last_talker_hidden.shape}")
|
||||
print(f"prev_embed shape: {layer0_embed.shape}")
|
||||
raise e
|
||||
batch_size = current_input.shape[0]
|
||||
bsz = int(layer0_code.shape[0])
|
||||
device = layer0_code.device
|
||||
dtype = last_talker_hidden.dtype
|
||||
num_groups = self.num_code_groups
|
||||
top_k = self._top_k
|
||||
|
||||
# Predict all residual layers (layers 1 to num_code_groups-1) autoregressively
|
||||
for layer_idx in range(self.num_code_groups - 1):
|
||||
seq_len = layer_idx + 2
|
||||
# Compute position_ids dynamically to avoid torch.compile specializing batch_size
|
||||
position_ids = torch.arange(seq_len, device=current_input.device, dtype=torch.int64).repeat(batch_size)
|
||||
# Forward through code_predictor model
|
||||
outputs = self.model(
|
||||
inputs_embeds=current_input,
|
||||
attention_mask=None,
|
||||
position_ids=position_ids,
|
||||
past_key_values=None,
|
||||
use_cache=False,
|
||||
cache_position=None,
|
||||
)
|
||||
hidden_state = outputs.last_hidden_state # [batch, 2, hidden_size]
|
||||
# Lazy init
|
||||
self._ensure_buffers(bsz, device, dtype)
|
||||
self._ensure_model_fwd()
|
||||
self._ensure_cached_refs()
|
||||
|
||||
# Use the corresponding lm_head for this layer
|
||||
logits = self.lm_head[layer_idx](hidden_state[:, -1:, :])
|
||||
code = torch.ops.vllm.qwen3_omni_code_predictor_sample(logits, self.layer_name)
|
||||
pos_codes.append(code)
|
||||
# Update prev_embed for next layer (if not last layer)
|
||||
# layer_idx=0 predicts layer 1, embed with codec_embedding[1]
|
||||
new_embed = self.model.codec_embedding[layer_idx](code) # [batch, 1, hidden_size]
|
||||
current_input = torch.cat([current_input, new_embed], dim=1) # [batch, 3~n, hidden_size]
|
||||
pos_all_layers = torch.stack(pos_codes, dim=1) # [batch, num_code_groups, 1]
|
||||
return pos_all_layers, current_input
|
||||
proj_buf = self._proj_buf
|
||||
pos_ids = self._pos_ids
|
||||
model_fwd = self._model_fwd
|
||||
lm_heads = self._lm_heads
|
||||
codec_embeds = self._codec_embeds
|
||||
|
||||
# Output codes
|
||||
all_codes = torch.empty(bsz, num_groups, 1, dtype=torch.int64, device=device)
|
||||
all_codes[:, 0] = layer0_code
|
||||
|
||||
# Fill buffer positions 0 & 1
|
||||
proj_buf[:bsz, 0:1, :] = last_talker_hidden
|
||||
proj_buf[:bsz, 1:2, :] = layer0_embed
|
||||
|
||||
# Autoregressive loop: predict layers 1..G-1
|
||||
for step in range(1, num_groups):
|
||||
seq_len = step + 1
|
||||
projected = proj_buf[:bsz, :seq_len, :]
|
||||
step_pos_ids = pos_ids[:seq_len] if bsz == 1 else pos_ids[:seq_len].repeat(bsz)
|
||||
|
||||
hidden_out = model_fwd(projected, step_pos_ids)
|
||||
|
||||
# Inline top-k sampling
|
||||
logits = lm_heads[step - 1](hidden_out[:, -1, :])
|
||||
if top_k > 0:
|
||||
topk_vals, _ = logits.topk(top_k, dim=-1)
|
||||
logits = logits.masked_fill(logits < topk_vals[:, -1:], float("-inf"))
|
||||
probs = F.softmax(logits, dim=-1)
|
||||
code = torch.multinomial(probs, num_samples=1) # [bsz, 1]
|
||||
|
||||
all_codes[:, step] = code
|
||||
|
||||
# Embed predicted code -> next buffer position
|
||||
new_embed = codec_embeds[step - 1](code)
|
||||
proj_buf[:bsz, step + 1 : step + 2, :] = new_embed
|
||||
|
||||
return all_codes, proj_buf[:bsz]
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Weight loading
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def load_weights(self, weights: list[tuple[str, torch.Tensor]]) -> set[str]:
|
||||
"""Load weights with mapping for fused QKV and gate_up projections.
|
||||
|
||||
@@ -129,111 +129,70 @@ class Qwen3OmniMoeTalkerForConditionalGeneration(
|
||||
self.code_predictor = Qwen3OmniMoeTalkerCodePredictor(
|
||||
vllm_config=vllm_config, prefix=maybe_prefix(prefix, "code_predictor")
|
||||
)
|
||||
max_batch_size = max(
|
||||
vllm_config.scheduler_config.max_num_seqs, vllm_config.compilation_config.max_cudagraph_capture_size
|
||||
)
|
||||
self.layer0_embed_buffer = torch.zeros(
|
||||
(max_batch_size, 1, self.config.text_config.hidden_size),
|
||||
dtype=vllm_config.model_config.dtype,
|
||||
)
|
||||
|
||||
def code_predictor_forward(
|
||||
self,
|
||||
input_ids: torch.Tensor,
|
||||
inputs_embeds: torch.Tensor | None = None,
|
||||
*,
|
||||
temperature: float = 1.0,
|
||||
top_k: int = 50, # Match transformers default
|
||||
top_p: float = 0.8, # Match transformers default
|
||||
generation_steps: int | None = None,
|
||||
last_talker_hidden: torch.Tensor | None = None,
|
||||
**_: object,
|
||||
) -> tuple[torch.Tensor, torch.Tensor]:
|
||||
"""
|
||||
Generate full RVQ codec codes for the provided sequence.
|
||||
"""Generate full RVQ codes + summed embeddings (single-loop, no KV cache).
|
||||
|
||||
The code predictor consumes the layer-0 codec codes produced by the talker
|
||||
alongside the talker's hidden states, and autoregressively predicts the remaining
|
||||
residual layers (to num_codec_groups).
|
||||
The code predictor uses re-prefill: each AR step re-forwards the full
|
||||
(short) sequence through the transformer. The returned ``proj_buf``
|
||||
already contains all codec embeddings at positions 1..G,
|
||||
so summed_embeddings = proj_buf[:, 1:, :].sum(dim=1) — no second
|
||||
loop or re-embedding needed.
|
||||
|
||||
Returns:
|
||||
tuple containing:
|
||||
- residual_codes: A tensor of shape [batch, num_code_groups, seq_len] containing
|
||||
the complete set of codec codes
|
||||
- summed_embeddings: A tensor of shape [batch, seq_len, hidden_size]
|
||||
Sum of all layer embeddings at each position (like Transformers)
|
||||
result_codes: [batch, num_code_groups, seq_len]
|
||||
summed_embeddings: [batch, seq_len, hidden_size]
|
||||
"""
|
||||
if input_ids is None:
|
||||
raise ValueError("`input_ids` containing layer-0 codec codes must be provided.")
|
||||
raise ValueError("`input_ids` (layer-0 codes) must be provided.")
|
||||
if inputs_embeds is None:
|
||||
raise ValueError("`inputs_embeds` containing talker hidden states must be provided.")
|
||||
raise ValueError("`inputs_embeds` (talker hidden states) must be provided.")
|
||||
|
||||
if inputs_embeds.ndim == 2:
|
||||
inputs_embeds = inputs_embeds.unsqueeze(0)
|
||||
if input_ids.ndim == 1:
|
||||
input_ids = input_ids.unsqueeze(0)
|
||||
|
||||
# Ensure the tensors are contiguous for the autoregressive sampling loop
|
||||
inputs_embeds = inputs_embeds.contiguous()
|
||||
input_ids = input_ids.contiguous()
|
||||
|
||||
# Generate full codec codes using MTP
|
||||
# This will be the parallel prediction implementation
|
||||
batch_size, seq_len = input_ids.shape
|
||||
device = input_ids.device
|
||||
embed_fn = self.language_model.model.codec_embedding
|
||||
hidden_size = self.config.code_predictor_config.hidden_size
|
||||
|
||||
# For now, use sequential generation (TODO: implement parallel)
|
||||
# Result will be [batch, num_code_groups, seq_len]
|
||||
# - all_codes_per_position will collect [batch, num_code_groups, 1] for each position
|
||||
all_codes_per_position = []
|
||||
middle_hidden_states = [] # Collect hidden states for each position
|
||||
|
||||
# Generate residual layers for each position
|
||||
for pos in range(seq_len):
|
||||
layer0_code = input_ids[:, pos : pos + 1] # [batch, 1]
|
||||
|
||||
# Initial input: [last_talker_hidden, layer0_embed]
|
||||
layer0_embed = self.embed_input_ids(layer0_code)
|
||||
self.layer0_embed_buffer[:batch_size].copy_(layer0_embed)
|
||||
pos_all_layers, current_input = self.code_predictor(
|
||||
layer0_code, self.layer0_embed_buffer[:batch_size], last_talker_hidden
|
||||
)
|
||||
|
||||
# Stack all layers for this position: [batch, num_code_groups, 1]
|
||||
all_codes_per_position.append(pos_all_layers)
|
||||
middle_hidden_states.append(current_input[:, 2:-1, :])
|
||||
|
||||
# Concatenate across positions: [batch, num_code_groups, seq_len]
|
||||
result_codes = torch.cat(all_codes_per_position, dim=2)
|
||||
|
||||
# Build summed embeddings for each position (like Transformers)
|
||||
# This combines layer-0 embed, mid layers hidden states, and last layer embed
|
||||
all_summed_embeddings = []
|
||||
result_codes = torch.empty(
|
||||
batch_size,
|
||||
self.num_code_groups,
|
||||
seq_len,
|
||||
dtype=torch.int64,
|
||||
device=device,
|
||||
)
|
||||
summed_embeddings = torch.empty(
|
||||
batch_size,
|
||||
seq_len,
|
||||
hidden_size,
|
||||
dtype=inputs_embeds.dtype,
|
||||
device=device,
|
||||
)
|
||||
|
||||
for pos in range(seq_len):
|
||||
# Layer 0 embedding
|
||||
layer0_code = result_codes[:, 0, pos : pos + 1] # [batch, 1]
|
||||
layer0_embed = self.embed_input_ids(layer0_code) # [batch, 1, hidden_size]
|
||||
layer0_code = input_ids[:, pos : pos + 1]
|
||||
layer0_embed = embed_fn(layer0_code)
|
||||
|
||||
# mid layers hidden states (from CodePredictor)
|
||||
mid_residual_hiddens = middle_hidden_states[pos] # [batch, num_code_groups-2, hidden_size]
|
||||
mid_list = list(mid_residual_hiddens.split(1, dim=1))
|
||||
|
||||
# last layer embedding
|
||||
last_layer_code = result_codes[:, -1, pos : pos + 1] # [batch, 1]
|
||||
last_residual_hidden = self.code_predictor.model.codec_embedding[-1](last_layer_code)
|
||||
|
||||
# Concatenate all layers: [batch, num_code_groups, hidden_size]
|
||||
pos_codec_hiddens = torch.cat(
|
||||
[layer0_embed] + mid_list + [last_residual_hidden],
|
||||
dim=1,
|
||||
pos_all_layers, proj_buf = self.code_predictor(
|
||||
layer0_code,
|
||||
layer0_embed,
|
||||
last_talker_hidden,
|
||||
)
|
||||
|
||||
# Sum across layers: [batch, 1, hidden_size] (like Transformers)
|
||||
pos_summed = pos_codec_hiddens.sum(dim=1, keepdim=True)
|
||||
all_summed_embeddings.append(pos_summed)
|
||||
|
||||
# Concatenate across positions: [batch, seq_len, hidden_size]
|
||||
summed_embeddings = torch.cat(all_summed_embeddings, dim=1).squeeze(1)
|
||||
result_codes[:, :, pos : pos + 1] = pos_all_layers
|
||||
# proj_buf layout: [0]=talker_hidden, [1..G]=codec embeds
|
||||
summed_embeddings[:, pos, :] = proj_buf[:, 1:, :].sum(dim=1)
|
||||
|
||||
return result_codes, summed_embeddings
|
||||
|
||||
|
||||
@@ -29,7 +29,7 @@ class CUDAGraphDecoderWrapper:
|
||||
output = wrapper.decode(codes) # Automatically uses CUDA graph if possible
|
||||
"""
|
||||
|
||||
DEFAULT_CAPTURE_SIZES = [25, 50, 100, 150, 200, 250, 300]
|
||||
DEFAULT_CAPTURE_SIZES = [2, 4, 8, 16, 25, 32, 50, 100, 150, 200, 250, 300]
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
|
||||
@@ -96,10 +96,6 @@ runtime:
|
||||
# Align with Omni: small chunks with sufficient context overlap.
|
||||
codec_chunk_frames: 25
|
||||
codec_left_context_frames: 25
|
||||
# First chunk size for reduced TTFA (0 = disabled).
|
||||
# When > 0, emits small chunks every N frames during the initial phase,
|
||||
# then switches to codec_chunk_frames cadence.
|
||||
initial_codec_chunk_frames: 0
|
||||
|
||||
edges:
|
||||
- from: 0
|
||||
|
||||