From 0e07eb6ff920937e003399cd9a92b6055c37796d Mon Sep 17 00:00:00 2001 From: Ding Zuhao Date: Fri, 30 Jan 2026 07:56:10 +0800 Subject: [PATCH] [Doc][Bagel] Add BAGEL-7B-MoT documentation and edit the default stage configuration (#987) Signed-off-by: Ding Zuhao Signed-off-by: jzz --- .../examples/offline_inference/bagel.md | 179 ++++++++++++++ .../examples/online_serving/bagel.md | 232 ++++++++++++++++++ examples/offline_inference/bagel/README.md | 177 +++++++++++++ .../bagel/run_single_prompt.sh | 6 - examples/online_serving/bagel/README.md | 230 +++++++++++++++++ .../model_executor/stage_configs/bagel.yaml | 4 +- .../stage_configs/bagel_multiconnector.yaml | 2 +- 7 files changed, 821 insertions(+), 9 deletions(-) create mode 100644 docs/user_guide/examples/offline_inference/bagel.md create mode 100644 docs/user_guide/examples/online_serving/bagel.md create mode 100644 examples/offline_inference/bagel/README.md delete mode 100644 examples/offline_inference/bagel/run_single_prompt.sh create mode 100644 examples/online_serving/bagel/README.md diff --git a/docs/user_guide/examples/offline_inference/bagel.md b/docs/user_guide/examples/offline_inference/bagel.md new file mode 100644 index 000000000..b2ee5d241 --- /dev/null +++ b/docs/user_guide/examples/offline_inference/bagel.md @@ -0,0 +1,179 @@ +# BAGEL-7B-MoT + +Source . + +## Set up + +Please refer to the [stage configuration documentation](https://docs.vllm.ai/projects/vllm-omni/en/latest/configuration/stage_configs/) to configure memory allocation appropriately for your hardware setup. + +## Run examples + +**Note**: These examples work with the default configuration on an **NVIDIA A100 (80GB)**. We also tested on dual **NVIDIA RTX 5000 Ada (32GB each)**. For dual-GPU setups, please modify the stage configuration to distribute the model across devices. + +Get into the bagel folder + +```bash +cd examples/offline_inference/bagel +``` + +### Modality Control + +BAGEL-7B-MoT supports multiple modality modes. You can control the mode using the `--modality` argument: + +#### Text to Image (text2img) + +- **Pipeline**: Text → Thinker → DiT → VAE Decode → Image +- **Stages Used**: Stage 0 (Thinker) + Stage 1 (DiT) +- **KV Transfer**: Thinker sends KV cache to DiT for conditioned generation + +Generate images from text prompts: + +```bash +python end2end.py --model ByteDance-Seed/BAGEL-7B-MoT \ + --modality text2img \ + --prompts "A cute cat" +``` + +#### Image to Image (img2img) + +- **Pipeline**: Image → VAE Encode → DiT → VAE Decode → New Image +- **Stages Used**: Stage 1 (DiT) only +- **Special**: Bypasses the Thinker stage, direct image-to-image transformation + +Transform images based on text prompts: + +```bash +python end2end.py --model ByteDance-Seed/BAGEL-7B-MoT \ + --modality img2img \ + --image-path /path/to/image.jpg \ + --prompts "Let the woman wear a blue dress" +``` + +#### Image to Text (img2text) + +- **Pipeline**: Image → ViT + VAE Encode → Thinker → Text Output +- **Stages Used**: Stage 0 (Thinker) only +- **Special**: Uses both VAE latent encoding AND ViT semantic encoding for comprehensive image understanding + +Generate text descriptions from images: + +```bash +python end2end.py --model ByteDance-Seed/BAGEL-7B-MoT \ + --modality img2text \ + --image-path /path/to/image.jpg \ + --prompts "Describe this image in detail" +``` + +#### Text to Text (text2text) + +- **Pipeline**: Text → Thinker → Text Output +- **Stages Used**: Stage 0 (Thinker) only +- **Special**: No visual components involved, operates as pure language model + +Pure text generation: + +```bash +python end2end.py --model ByteDance-Seed/BAGEL-7B-MoT \ + --modality text2text \ + --prompts "What is the capital of France?" + +# You can load prompts from a text file (one prompt per line): +python end2end.py --model ByteDance-Seed/BAGEL-7B-MoT \ + --modality text2text \ + --txt-prompts /path/to/prompts.txt +``` + +### Inference Steps + +Control the number of inference steps for image generation: + +```bash +# You can adjust steps to 100 to improve image quality +python end2end.py --model ByteDance-Seed/BAGEL-7B-MoT \ + --modality text2img \ + --steps 50 \ + --prompts "A cute cat" +``` + +### Key arguments + +BAGEL-7B-MoT supports **multiple modality modes** for different use cases. + +The default yaml configuration deploys Thinker and DiT on the same GPU. You can use the default configuration file: [`bagel.yaml`](../../../vllm_omni/model_executor/stage_configs/bagel.yaml) + +#### 📌 Command Line Arguments (end2end.py) + +| Argument | Type | Default | Description | +| :--------------------- | :----- | :---------------------------- | :----------------------------------------------------------- | +| `--model` | string | `ByteDance-Seed/BAGEL-7B-MoT` | Model path or name | +| `--modality` | choice | `text2img` | Modality mode: `text2img`, `img2img`, `img2text`, `text2text` | +| `--prompts` | list | `None` | Input text prompts directly | +| `--txt-prompts` | string | `None` | Path to txt file with one prompt per line | +| `--image-path` | string | `None` | Input image path (for `img2img`/`img2text`) | +| `--steps` | int | `50` | Number of inference steps | +| `--stage-configs-path` | string | `None` | Custom stage config file path | +| `--worker-backend` | choice | `process` | Worker backend: `process` or `ray` | +| `--ray-address` | string | `None` | Ray cluster address | +| `--enable-stats` | flag | `False` | Enable statistics logging | +| `--init-sleep-seconds` | int | `20` | Initialization sleep time | +| `--batch-timeout` | int | `5` | Batch timeout | +| `--init-timeout` | int | `300` | Initialization timeout | + +------ + +#### ⚙️ Stage Configuration Parameters (bagel.yaml) + + **Stage 0 - Thinker (LLM Stage)** + +| Parameter | Value | Description | +| :------------------------------- | :------------------------------ | :----------------------- | +| `stage_type` | `llm` | Stage type | +| `devices` | `"0"` | GPU device ID | +| `max_batch_size` | `1` | Maximum batch size | +| `model_stage` | `thinker` | Model stage identifier | +| `model_arch` | `BagelForConditionalGeneration` | Model architecture | +| `gpu_memory_utilization` | `0.4` | GPU memory utilization | +| `tensor_parallel_size` | `1` | Tensor parallel size | +| `max_num_batched_tokens` | `32768` | Maximum batched tokens | +| `omni_kv_config.need_send_cache` | `true` | Whether to send KV cache | + +------ + +**Stage 1 - DiT (Diffusion Stage)** + +| Parameter | Value | Description | +| :------------------------------- | :---------- | :-------------------------- | +| `stage_type` | `diffusion` | Stage type | +| `devices` | `"0"` | GPU device ID | +| `max_batch_size` | `1` | Maximum batch size | +| `model_stage` | `dit` | Model stage identifier | +| `gpu_memory_utilization` | `0.4` | GPU memory utilization | +| `omni_kv_config.need_recv_cache` | `true` | Whether to receive KV cache | +| `engine_input_source` | `[0]` | Input source from Stage 0 | + +------ + +#### 🔗 Runtime Configuration + +| Parameter | Value | Description | +| :-------------------- | :------ | :------------------------------- | +| `window_size` | `-1` | Window size (-1 means unlimited) | +| `max_inflight` | `1` | Maximum inflight requests | +| `shm_threshold_bytes` | `65536` | Shared memory threshold (64KB) | + +## FAQ + +- If you encounter an error about the backend of librosa, try to install ffmpeg with the command below. + +```bash +sudo apt update +sudo apt install ffmpeg +``` + +- If you don’t know how much VRAM is needed for the model or encounter the OOM error, you can try to decrease the max_model_len. + +| Stage | VRAM | +| :------------------ | :--------------------------- | +| Stage-0 (Thinker) | **15.04 GiB** **+ KV Cache** | +| Stage-1 (DiT) | **26.50 GiB** | +| Total | **~42 GiB + KV Cache** | diff --git a/docs/user_guide/examples/online_serving/bagel.md b/docs/user_guide/examples/online_serving/bagel.md new file mode 100644 index 000000000..107e42449 --- /dev/null +++ b/docs/user_guide/examples/online_serving/bagel.md @@ -0,0 +1,232 @@ +# BAGEL-7B-MoT + +Source . + +## 🛠️ Installation + +Please refer to [README.md](../../../README.md) + +## Run examples (BAGEL-7B-MoT) + +**Note**: These examples work with the default configuration on an **NVIDIA A100 (80GB)**. We also tested on dual **NVIDIA RTX 5000 Ada (32GB each)**. For dual-GPU setups, please modify the stage configuration to distribute the model across devices. + +### Launch the Server + +```bash +# Use default configuration +vllm serve ByteDance-Seed/BAGEL-7B-MoT --omni --port 8091 +``` + +Or use the convenience script: + +```bash +cd /workspace/vllm-omni/examples/online_serving/bagel +bash run_server.sh +``` + +If you have a custom stage configs file, launch the server with the command below: + +```bash +vllm serve ByteDance-Seed/BAGEL-7B-MoT --omni --port 8091 --stage-configs-path /path/to/stage_configs_file +``` + +### Send Multi-modal Request + +Get into the bagel folder: + +```bash +cd examples/online_serving/bagel +``` + +Send request via Python + +```bash +python openai_chat_client.py --prompt "A cute cat" --modality text2img +``` + +The Python client supports the following command-line arguments: + +- `--prompt` (or `-p`): Text prompt for generation (default: `A cute cat`) +- `--output` (or `-o`): Output file path for image results (default: `bagel_output.png`) +- `--server` (or `-s`): Server URL (default: `http://localhost:8091`) +- `--image-url` (or `-i`): Input image URL or local file path (for img2img/img2text modes) +- `--modality` (or `-m`): Task modality (default: `text2img`). Options: `text2img`, `img2img`, `img2text`, `text2text` +- `--height`: Image height in pixels (default: 512) +- `--width`: Image width in pixels (default: 512) +- `--steps`: Number of inference steps (default: 25) +- `--seed`: Random seed (default: 42) +- `--negative`: Negative prompt for image generation + +Example with custom parameters: + +```bash +python openai_chat_client.py \ + --prompt "A futuristic city" \ + --modality text2img \ + --height 768 \ + --width 768 \ + --steps 50 \ + --seed 42 \ + --negative "blurry, low quality" +``` + +## Modality Control + +BAGEL-7B-MoT supports **multiple modality modes** for different use cases. + +The default yaml configuration deploys Thinker and DiT on the same GPU. You can use the default configuration file: [`bagel.yaml`](../../../vllm_omni/model_executor/stage_configs/bagel.yaml) + +| Modality | Input | Output | Description | +| ----------- | ------------ | ------ | -------------------------------------- | +| `text2img` | Text | Image | Generate images from text prompts | +| `img2img` | Image + Text | Image | Transform images using text guidance | +| `img2text` | Image + Text | Text | Generate text descriptions from images | +| `text2text` | Text | Text | Pure text generation | + +### Text to Image (text2img) + +Generate images from text prompts: + +**Using Python client** + +```bash +python openai_chat_client.py \ + --prompt "A beautiful sunset over mountains" \ + --modality text2img \ + --output sunset.png \ + --steps 50 +``` + +**Using curl** + +```bash +curl http://localhost:8091/v1/chat/completions \ + -H "Content-Type: application/json" \ + -d '{ + "messages": [{"role": "user", "content": [{"type": "text", "text": "<|im_start|>A beautiful sunset over mountains<|im_end|>"}]}], + "modalities": ["image"], + "height": 512, + "width": 512, + "num_inference_steps": 50, + "seed": 42 + }' +``` + + +### Image to Image (img2img) + +Transform images based on text prompts: + +**Using Python client** + +```bash +python openai_chat_client.py \ + --prompt "Make the cat stand up" \ + --modality img2img \ + --image-url /path/to/input.jpg \ + --output transformed.png +``` + +**Using curl** + +```bash +IMAGE_BASE64=$(base64 -w 0 cat.jpg) + +cat < payload.json +{ + "messages": [{ + "role": "user", + "content": [ + {"type": "text", "text": "<|im_start|>Make the cat stand up<|im_end|>"}, + {"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,${IMAGE_BASE64}"}} + ] + }], + "modalities": ["image"], + "height": 512, + "width": 512, + "num_inference_steps": 50, + "seed": 42 +} +EOF + +curl http://localhost:8091/v1/chat/completions \ + -H "Content-Type: application/json" \ + -d @payload.json + +``` + +### Image to Text (img2text) + +Generate text descriptions from images: + +**Using Python client** + +```bash +python openai_chat_client.py \ + --prompt "Describe this image in detail" \ + --modality img2text \ + --image-url /path/to/image.jpg +``` + +**Using curl** + +```bash +IMAGE_BASE64=$(base64 -w 0 cat.jpg) + +cat < payload.json +{ + "messages": [{ + "role": "user", + "content": [ + {"type": "text", "text": "<|im_start|>user\n<|image_pad|>\nDescribe this image in detail<|im_end|>\n<|im_start|>assistant\n"}, + {"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,${IMAGE_BASE64}"}} + ] + }], + "modalities": ["text"] +} +EOF + +curl http://localhost:8091/v1/chat/completions \ + -H "Content-Type: application/json" \ + -d @payload.json +``` + +### Text to Text (text2text) + +Pure text generation: + +**Using Python client** + +```bash +python openai_chat_client.py \ + --prompt "What is the capital of France?" \ + --modality text2text +``` + +**Using curl** + +```bash +curl http://localhost:8091/v1/chat/completions \ + -H "Content-Type: application/json" \ + -d '{ + "messages": [{"role": "user", "content": [{"type": "text", "text": "<|im_start|>user\nWhat is the capital of France?<|im_end|>\n<|im_start|>assistant\n"}]}] + "modalities": ["text"] + }' +``` + +## FAQ + +- If you encounter an error about the backend of librosa, try to install ffmpeg with the command below. + +```bash +sudo apt update +sudo apt install ffmpeg +``` + +- If you don’t know how much VRAM is needed for the model or encounter the OOM error, you can try to decrease the max_model_len. + +| Stage | VRAM | +| :------------------ | :--------------------------- | +| Stage-0 (Thinker) | **15.04 GiB** **+ KV Cache** | +| Stage-1 (DiT) | **26.50 GiB** | +| Total | **~42 GiB + KV Cache** | diff --git a/examples/offline_inference/bagel/README.md b/examples/offline_inference/bagel/README.md new file mode 100644 index 000000000..7bcfb42fa --- /dev/null +++ b/examples/offline_inference/bagel/README.md @@ -0,0 +1,177 @@ +# BAGEL-7B-MoT + +## Set up + +Please refer to the [stage configuration documentation](https://docs.vllm.ai/projects/vllm-omni/en/latest/configuration/stage_configs/) to configure memory allocation appropriately for your hardware setup. + +## Run examples + +**Note**: These examples work with the default configuration on an **NVIDIA A100 (80GB)**. We also tested on dual **NVIDIA RTX 5000 Ada (32GB each)**. For dual-GPU setups, please modify the stage configuration to distribute the model across devices. + +Get into the bagel folder + +```bash +cd examples/offline_inference/bagel +``` + +### Modality Control + +BAGEL-7B-MoT supports multiple modality modes. You can control the mode using the `--modality` argument: + +#### Text to Image (text2img) + +- **Pipeline**: Text → Thinker → DiT → VAE Decode → Image +- **Stages Used**: Stage 0 (Thinker) + Stage 1 (DiT) +- **KV Transfer**: Thinker sends KV cache to DiT for conditioned generation + +Generate images from text prompts: + +```bash +python end2end.py --model ByteDance-Seed/BAGEL-7B-MoT \ + --modality text2img \ + --prompts "A cute cat" +``` + +#### Image to Image (img2img) + +- **Pipeline**: Image → VAE Encode → DiT → VAE Decode → New Image +- **Stages Used**: Stage 1 (DiT) only +- **Special**: Bypasses the Thinker stage, direct image-to-image transformation + +Transform images based on text prompts: + +```bash +python end2end.py --model ByteDance-Seed/BAGEL-7B-MoT \ + --modality img2img \ + --image-path /path/to/image.jpg \ + --prompts "Let the woman wear a blue dress" +``` + +#### Image to Text (img2text) + +- **Pipeline**: Image → ViT + VAE Encode → Thinker → Text Output +- **Stages Used**: Stage 0 (Thinker) only +- **Special**: Uses both VAE latent encoding AND ViT semantic encoding for comprehensive image understanding + +Generate text descriptions from images: + +```bash +python end2end.py --model ByteDance-Seed/BAGEL-7B-MoT \ + --modality img2text \ + --image-path /path/to/image.jpg \ + --prompts "Describe this image in detail" +``` + +#### Text to Text (text2text) + +- **Pipeline**: Text → Thinker → Text Output +- **Stages Used**: Stage 0 (Thinker) only +- **Special**: No visual components involved, operates as pure language model + +Pure text generation: + +```bash +python end2end.py --model ByteDance-Seed/BAGEL-7B-MoT \ + --modality text2text \ + --prompts "What is the capital of France?" + +# You can load prompts from a text file (one prompt per line): +python end2end.py --model ByteDance-Seed/BAGEL-7B-MoT \ + --modality text2text \ + --txt-prompts /path/to/prompts.txt +``` + +### Inference Steps + +Control the number of inference steps for image generation: + +```bash +# You can adjust steps to 100 to improve image quality +python end2end.py --model ByteDance-Seed/BAGEL-7B-MoT \ + --modality text2img \ + --steps 50 \ + --prompts "A cute cat" +``` + +### Key arguments + +BAGEL-7B-MoT supports **multiple modality modes** for different use cases. + +The default yaml configuration deploys Thinker and DiT on the same GPU. You can use the default configuration file: [`bagel.yaml`](../../../vllm_omni/model_executor/stage_configs/bagel.yaml) + +#### 📌 Command Line Arguments (end2end.py) + +| Argument | Type | Default | Description | +| :--------------------- | :----- | :---------------------------- | :----------------------------------------------------------- | +| `--model` | string | `ByteDance-Seed/BAGEL-7B-MoT` | Model path or name | +| `--modality` | choice | `text2img` | Modality mode: `text2img`, `img2img`, `img2text`, `text2text` | +| `--prompts` | list | `None` | Input text prompts directly | +| `--txt-prompts` | string | `None` | Path to txt file with one prompt per line | +| `--image-path` | string | `None` | Input image path (for `img2img`/`img2text`) | +| `--steps` | int | `50` | Number of inference steps | +| `--stage-configs-path` | string | `None` | Custom stage config file path | +| `--worker-backend` | choice | `process` | Worker backend: `process` or `ray` | +| `--ray-address` | string | `None` | Ray cluster address | +| `--enable-stats` | flag | `False` | Enable statistics logging | +| `--init-sleep-seconds` | int | `20` | Initialization sleep time | +| `--batch-timeout` | int | `5` | Batch timeout | +| `--init-timeout` | int | `300` | Initialization timeout | + +------ + +#### ⚙️ Stage Configuration Parameters (bagel.yaml) + + **Stage 0 - Thinker (LLM Stage)** + +| Parameter | Value | Description | +| :------------------------------- | :------------------------------ | :----------------------- | +| `stage_type` | `llm` | Stage type | +| `devices` | `"0"` | GPU device ID | +| `max_batch_size` | `1` | Maximum batch size | +| `model_stage` | `thinker` | Model stage identifier | +| `model_arch` | `BagelForConditionalGeneration` | Model architecture | +| `gpu_memory_utilization` | `0.4` | GPU memory utilization | +| `tensor_parallel_size` | `1` | Tensor parallel size | +| `max_num_batched_tokens` | `32768` | Maximum batched tokens | +| `omni_kv_config.need_send_cache` | `true` | Whether to send KV cache | + +------ + +**Stage 1 - DiT (Diffusion Stage)** + +| Parameter | Value | Description | +| :------------------------------- | :---------- | :-------------------------- | +| `stage_type` | `diffusion` | Stage type | +| `devices` | `"0"` | GPU device ID | +| `max_batch_size` | `1` | Maximum batch size | +| `model_stage` | `dit` | Model stage identifier | +| `gpu_memory_utilization` | `0.4` | GPU memory utilization | +| `omni_kv_config.need_recv_cache` | `true` | Whether to receive KV cache | +| `engine_input_source` | `[0]` | Input source from Stage 0 | + +------ + +#### 🔗 Runtime Configuration + +| Parameter | Value | Description | +| :-------------------- | :------ | :------------------------------- | +| `window_size` | `-1` | Window size (-1 means unlimited) | +| `max_inflight` | `1` | Maximum inflight requests | +| `shm_threshold_bytes` | `65536` | Shared memory threshold (64KB) | + +## FAQ + +- If you encounter an error about the backend of librosa, try to install ffmpeg with the command below. + +```bash +sudo apt update +sudo apt install ffmpeg +``` + +- If you don’t know how much VRAM is needed for the model or encounter the OOM error, you can try to decrease the max_model_len. + +| Stage | VRAM | +| :------------------ | :--------------------------- | +| Stage-0 (Thinker) | **15.04 GiB** **+ KV Cache** | +| Stage-1 (DiT) | **26.50 GiB** | +| Total | **~42 GiB + KV Cache** | diff --git a/examples/offline_inference/bagel/run_single_prompt.sh b/examples/offline_inference/bagel/run_single_prompt.sh deleted file mode 100644 index 2099620e3..000000000 --- a/examples/offline_inference/bagel/run_single_prompt.sh +++ /dev/null @@ -1,6 +0,0 @@ -prompt="<|im_start|>user\nWhat is the capital of France?<|im_end|>\n<|im_start|>assistant\n" - -python end2end.py --model ByteDance-Seed/BAGEL-7B-MoT \ - --prompt_type text \ - --init-sleep-seconds 0 \ - --prompts ${prompt} diff --git a/examples/online_serving/bagel/README.md b/examples/online_serving/bagel/README.md new file mode 100644 index 000000000..3fbea0550 --- /dev/null +++ b/examples/online_serving/bagel/README.md @@ -0,0 +1,230 @@ +# BAGEL-7B-MoT + +## 🛠️ Installation + +Please refer to [README.md](../../../README.md) + +## Run examples (BAGEL-7B-MoT) + +**Note**: These examples work with the default configuration on an **NVIDIA A100 (80GB)**. We also tested on dual **NVIDIA RTX 5000 Ada (32GB each)**. For dual-GPU setups, please modify the stage configuration to distribute the model across devices. + +### Launch the Server + +```bash +# Use default configuration +vllm serve ByteDance-Seed/BAGEL-7B-MoT --omni --port 8091 +``` + +Or use the convenience script: + +```bash +cd /workspace/vllm-omni/examples/online_serving/bagel +bash run_server.sh +``` + +If you have a custom stage configs file, launch the server with the command below: + +```bash +vllm serve ByteDance-Seed/BAGEL-7B-MoT --omni --port 8091 --stage-configs-path /path/to/stage_configs_file +``` + +### Send Multi-modal Request + +Get into the bagel folder: + +```bash +cd examples/online_serving/bagel +``` + +Send request via Python + +```bash +python openai_chat_client.py --prompt "A cute cat" --modality text2img +``` + +The Python client supports the following command-line arguments: + +- `--prompt` (or `-p`): Text prompt for generation (default: `A cute cat`) +- `--output` (or `-o`): Output file path for image results (default: `bagel_output.png`) +- `--server` (or `-s`): Server URL (default: `http://localhost:8091`) +- `--image-url` (or `-i`): Input image URL or local file path (for img2img/img2text modes) +- `--modality` (or `-m`): Task modality (default: `text2img`). Options: `text2img`, `img2img`, `img2text`, `text2text` +- `--height`: Image height in pixels (default: 512) +- `--width`: Image width in pixels (default: 512) +- `--steps`: Number of inference steps (default: 25) +- `--seed`: Random seed (default: 42) +- `--negative`: Negative prompt for image generation + +Example with custom parameters: + +```bash +python openai_chat_client.py \ + --prompt "A futuristic city" \ + --modality text2img \ + --height 768 \ + --width 768 \ + --steps 50 \ + --seed 42 \ + --negative "blurry, low quality" +``` + +## Modality Control + +BAGEL-7B-MoT supports **multiple modality modes** for different use cases. + +The default yaml configuration deploys Thinker and DiT on the same GPU. You can use the default configuration file: [`bagel.yaml`](../../../vllm_omni/model_executor/stage_configs/bagel.yaml) + +| Modality | Input | Output | Description | +| ----------- | ------------ | ------ | -------------------------------------- | +| `text2img` | Text | Image | Generate images from text prompts | +| `img2img` | Image + Text | Image | Transform images using text guidance | +| `img2text` | Image + Text | Text | Generate text descriptions from images | +| `text2text` | Text | Text | Pure text generation | + +### Text to Image (text2img) + +Generate images from text prompts: + +**Using Python client** + +```bash +python openai_chat_client.py \ + --prompt "A beautiful sunset over mountains" \ + --modality text2img \ + --output sunset.png \ + --steps 50 +``` + +**Using curl** + +```bash +curl http://localhost:8091/v1/chat/completions \ + -H "Content-Type: application/json" \ + -d '{ + "messages": [{"role": "user", "content": [{"type": "text", "text": "<|im_start|>A beautiful sunset over mountains<|im_end|>"}]}], + "modalities": ["image"], + "height": 512, + "width": 512, + "num_inference_steps": 50, + "seed": 42 + }' +``` + + +### Image to Image (img2img) + +Transform images based on text prompts: + +**Using Python client** + +```bash +python openai_chat_client.py \ + --prompt "Make the cat stand up" \ + --modality img2img \ + --image-url /path/to/input.jpg \ + --output transformed.png +``` + +**Using curl** + +```bash +IMAGE_BASE64=$(base64 -w 0 cat.jpg) + +cat < payload.json +{ + "messages": [{ + "role": "user", + "content": [ + {"type": "text", "text": "<|im_start|>Make the cat stand up<|im_end|>"}, + {"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,${IMAGE_BASE64}"}} + ] + }], + "modalities": ["image"], + "height": 512, + "width": 512, + "num_inference_steps": 50, + "seed": 42 +} +EOF + +curl http://localhost:8091/v1/chat/completions \ + -H "Content-Type: application/json" \ + -d @payload.json + +``` + +### Image to Text (img2text) + +Generate text descriptions from images: + +**Using Python client** + +```bash +python openai_chat_client.py \ + --prompt "Describe this image in detail" \ + --modality img2text \ + --image-url /path/to/image.jpg +``` + +**Using curl** + +```bash +IMAGE_BASE64=$(base64 -w 0 cat.jpg) + +cat < payload.json +{ + "messages": [{ + "role": "user", + "content": [ + {"type": "text", "text": "<|im_start|>user\n<|image_pad|>\nDescribe this image in detail<|im_end|>\n<|im_start|>assistant\n"}, + {"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,${IMAGE_BASE64}"}} + ] + }], + "modalities": ["text"] +} +EOF + +curl http://localhost:8091/v1/chat/completions \ + -H "Content-Type: application/json" \ + -d @payload.json +``` + +### Text to Text (text2text) + +Pure text generation: + +**Using Python client** + +```bash +python openai_chat_client.py \ + --prompt "What is the capital of France?" \ + --modality text2text +``` + +**Using curl** + +```bash +curl http://localhost:8091/v1/chat/completions \ + -H "Content-Type: application/json" \ + -d '{ + "messages": [{"role": "user", "content": [{"type": "text", "text": "<|im_start|>user\nWhat is the capital of France?<|im_end|>\n<|im_start|>assistant\n"}]}] + "modalities": ["text"] + }' +``` + +## FAQ + +- If you encounter an error about the backend of librosa, try to install ffmpeg with the command below. + +```bash +sudo apt update +sudo apt install ffmpeg +``` + +- If you don’t know how much VRAM is needed for the model or encounter the OOM error, you can try to decrease the max_model_len. + +| Stage | VRAM | +| :------------------ | :--------------------------- | +| Stage-0 (Thinker) | **15.04 GiB** **+ KV Cache** | +| Stage-1 (DiT) | **26.50 GiB** | +| Total | **~42 GiB + KV Cache** | diff --git a/vllm_omni/model_executor/stage_configs/bagel.yaml b/vllm_omni/model_executor/stage_configs/bagel.yaml index 93d2b611a..ee2b17729 100644 --- a/vllm_omni/model_executor/stage_configs/bagel.yaml +++ b/vllm_omni/model_executor/stage_configs/bagel.yaml @@ -4,7 +4,7 @@ stage_args: - stage_id: 0 stage_type: llm runtime: - devices: "0" # Adjust as needed + devices: "0" max_batch_size: 1 engine_args: model_stage: thinker @@ -38,7 +38,7 @@ stage_args: - stage_id: 1 stage_type: diffusion runtime: - devices: "1" + devices: "0" max_batch_size: 1 engine_args: model_stage: dit diff --git a/vllm_omni/model_executor/stage_configs/bagel_multiconnector.yaml b/vllm_omni/model_executor/stage_configs/bagel_multiconnector.yaml index e378fe675..eddc844d5 100644 --- a/vllm_omni/model_executor/stage_configs/bagel_multiconnector.yaml +++ b/vllm_omni/model_executor/stage_configs/bagel_multiconnector.yaml @@ -4,7 +4,7 @@ stage_args: - stage_id: 0 stage_type: llm runtime: - devices: "0" # Adjust as needed + devices: "0" max_batch_size: 1 engine_args: model_stage: thinker