227 lines
6.6 KiB
Markdown
227 lines
6.6 KiB
Markdown
# vLLM-Omni Helm Chart
|
|
|
|
Source <https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/chart-helm>.
|
|
|
|
|
|
Helm chart for deploying [vLLM-Omni](https://github.com/vllm-project/vllm-omni) on Kubernetes. vLLM-Omni extends vLLM with omni-modality model serving, supporting text-to-image, multimodal chat, text-to-speech, and more.
|
|
|
|
## Prerequisites
|
|
|
|
- Kubernetes 1.24+
|
|
- Helm 3.x
|
|
- NVIDIA GPU nodes with [NVIDIA Device Plugin](https://github.com/NVIDIA/k8s-device-plugin)
|
|
|
|
## Quick Start
|
|
|
|
```bash
|
|
helm install my-release ./chart-helm \
|
|
--set model=Tongyi-MAI/Z-Image-Turbo
|
|
```
|
|
|
|
## Configuration
|
|
|
|
### Model Selection
|
|
|
|
Set the `model` value to any supported HuggingFace model ID:
|
|
|
|
| Model | Type | GPUs | Notes |
|
|
|-------|------|------|-------|
|
|
| `Tongyi-MAI/Z-Image-Turbo` | text-to-image | 1 | Small, fast (default) |
|
|
| `stabilityai/stable-diffusion-3.5-medium` | text-to-image | 1 | ~6GB VRAM |
|
|
| `Qwen/Qwen-Image` | text-to-image | 1 | Large, ~40GB+ VRAM |
|
|
| `Qwen/Qwen2.5-Omni-7B` | multimodal | 2 | Text + audio + image + video |
|
|
| `Qwen/Qwen3-Omni-7B-Chat` | multimodal | 2 | Latest omni model |
|
|
| `Qwen/Qwen3-TTS` | text-to-speech | 1 | TTS |
|
|
|
|
### HuggingFace Token
|
|
|
|
For gated models that require authentication:
|
|
|
|
```bash
|
|
helm install my-release ./chart-helm \
|
|
--set model=Qwen/Qwen2.5-Omni-7B \
|
|
--set hfToken=hf_xxxxx \
|
|
--set resources.requests."nvidia\.com/gpu"=2 \
|
|
--set resources.limits."nvidia\.com/gpu"=2
|
|
```
|
|
|
|
### Omni-Specific Flags
|
|
|
|
Enable VAE memory optimizations for diffusion models:
|
|
|
|
```bash
|
|
helm install my-release ./chart-helm \
|
|
--set model=Qwen/Qwen-Image \
|
|
--set omniArgs.vaeUseSlicing=true \
|
|
--set omniArgs.vaeUseTiling=true
|
|
```
|
|
|
|
Enable CPU offloading:
|
|
|
|
```bash
|
|
helm install my-release ./chart-helm \
|
|
--set model=Qwen/Qwen-Image \
|
|
--set omniArgs.enableCpuOffload=true
|
|
```
|
|
|
|
Pass additional raw CLI flags:
|
|
|
|
```bash
|
|
helm install my-release ./chart-helm \
|
|
--set model=Qwen/Qwen-Image \
|
|
--set omniArgs.extraArgs[0]="--enable-layerwise-offload"
|
|
```
|
|
|
|
### Model Cache
|
|
|
|
By default, a PersistentVolumeClaim is created for the HuggingFace model cache to avoid re-downloading models on pod restarts:
|
|
|
|
```yaml
|
|
modelCache:
|
|
enabled: true
|
|
storageSize: "50Gi"
|
|
storageClassName: ""
|
|
```
|
|
|
|
To use an ephemeral volume instead:
|
|
|
|
```bash
|
|
helm install my-release ./chart-helm \
|
|
--set modelCache.enabled=false
|
|
```
|
|
|
|
### Custom Command Override
|
|
|
|
To fully override the container command:
|
|
|
|
```bash
|
|
helm install my-release ./chart-helm \
|
|
--set image.command[0]=vllm \
|
|
--set image.command[1]=serve \
|
|
--set image.command[2]=my-model \
|
|
--set image.command[3]=--omni \
|
|
--set image.command[4]=--host \
|
|
--set image.command[5]=0.0.0.0
|
|
```
|
|
|
|
## API Endpoints
|
|
|
|
Once deployed, vLLM-Omni exposes the following OpenAI-compatible endpoints:
|
|
|
|
| Endpoint | Method | Description |
|
|
|----------|--------|-------------|
|
|
| `/health` | GET | Health check |
|
|
| `/v1/models` | GET | List available models |
|
|
| `/v1/chat/completions` | POST | Chat completions (text/multimodal) |
|
|
| `/v1/images/generations` | POST | Image generation |
|
|
| `/v1/images/edits` | POST | Image editing |
|
|
| `/v1/audio/speech` | POST | Text-to-speech |
|
|
|
|
## Files
|
|
|
|
| File | Description |
|
|
|------|-------------|
|
|
| `Chart.yaml` | Chart metadata (name, version, maintainers) |
|
|
| `values.yaml` | Default configuration values |
|
|
| `values.schema.json` | JSON schema for validating values |
|
|
| `templates/_helpers.tpl` | Helper templates for common configurations |
|
|
| `templates/deployment.yaml` | Kubernetes Deployment |
|
|
| `templates/service.yaml` | Kubernetes Service (ClusterIP) |
|
|
| `templates/secrets.yaml` | Secrets (generic + HuggingFace token) |
|
|
| `templates/pvc.yaml` | PersistentVolumeClaim for model cache |
|
|
| `templates/configmap.yaml` | Optional ConfigMap |
|
|
| `templates/hpa.yaml` | HorizontalPodAutoscaler |
|
|
| `templates/poddisruptionbudget.yaml` | PodDisruptionBudget |
|
|
| `templates/custom-objects.yaml` | Custom Kubernetes objects |
|
|
|
|
## Running Tests
|
|
|
|
This chart includes unit tests using [helm-unittest](https://github.com/helm-unittest/helm-unittest). Install the plugin and run tests:
|
|
|
|
```bash
|
|
# Install plugin
|
|
helm plugin install https://github.com/helm-unittest/helm-unittest
|
|
|
|
# Run tests
|
|
helm unittest .
|
|
```
|
|
|
|
## Example materials
|
|
|
|
??? abstract ".helmignore"
|
|
``````
|
|
--8<-- "examples/online_serving/chart-helm/.helmignore"
|
|
``````
|
|
??? abstract "Chart.yaml"
|
|
``````yaml
|
|
--8<-- "examples/online_serving/chart-helm/Chart.yaml"
|
|
``````
|
|
??? abstract "ct.yaml"
|
|
``````yaml
|
|
--8<-- "examples/online_serving/chart-helm/ct.yaml"
|
|
``````
|
|
??? abstract "lintconf.yaml"
|
|
``````yaml
|
|
--8<-- "examples/online_serving/chart-helm/lintconf.yaml"
|
|
``````
|
|
??? abstract "templates/_helpers.tpl"
|
|
``````tpl
|
|
--8<-- "examples/online_serving/chart-helm/templates/_helpers.tpl"
|
|
``````
|
|
??? abstract "templates/configmap.yaml"
|
|
``````yaml
|
|
--8<-- "examples/online_serving/chart-helm/templates/configmap.yaml"
|
|
``````
|
|
??? abstract "templates/custom-objects.yaml"
|
|
``````yaml
|
|
--8<-- "examples/online_serving/chart-helm/templates/custom-objects.yaml"
|
|
``````
|
|
??? abstract "templates/deployment.yaml"
|
|
``````yaml
|
|
--8<-- "examples/online_serving/chart-helm/templates/deployment.yaml"
|
|
``````
|
|
??? abstract "templates/hpa.yaml"
|
|
``````yaml
|
|
--8<-- "examples/online_serving/chart-helm/templates/hpa.yaml"
|
|
``````
|
|
??? abstract "templates/ingress.yaml"
|
|
``````yaml
|
|
--8<-- "examples/online_serving/chart-helm/templates/ingress.yaml"
|
|
``````
|
|
??? abstract "templates/poddisruptionbudget.yaml"
|
|
``````yaml
|
|
--8<-- "examples/online_serving/chart-helm/templates/poddisruptionbudget.yaml"
|
|
``````
|
|
??? abstract "templates/pvc.yaml"
|
|
``````yaml
|
|
--8<-- "examples/online_serving/chart-helm/templates/pvc.yaml"
|
|
``````
|
|
??? abstract "templates/secrets.yaml"
|
|
``````yaml
|
|
--8<-- "examples/online_serving/chart-helm/templates/secrets.yaml"
|
|
``````
|
|
??? abstract "templates/service.yaml"
|
|
``````yaml
|
|
--8<-- "examples/online_serving/chart-helm/templates/service.yaml"
|
|
``````
|
|
??? abstract "tests/deployment_test.yaml"
|
|
``````yaml
|
|
--8<-- "examples/online_serving/chart-helm/tests/deployment_test.yaml"
|
|
``````
|
|
??? abstract "tests/ingress_test.yaml"
|
|
``````yaml
|
|
--8<-- "examples/online_serving/chart-helm/tests/ingress_test.yaml"
|
|
``````
|
|
??? abstract "tests/pvc_test.yaml"
|
|
``````yaml
|
|
--8<-- "examples/online_serving/chart-helm/tests/pvc_test.yaml"
|
|
``````
|
|
??? abstract "tests/secrets_test.yaml"
|
|
``````yaml
|
|
--8<-- "examples/online_serving/chart-helm/tests/secrets_test.yaml"
|
|
``````
|
|
??? abstract "values.yaml"
|
|
``````yaml
|
|
--8<-- "examples/online_serving/chart-helm/values.yaml"
|
|
``````
|