hsliu
afe5c2692c
- Improve API server with better error handling and response formatting - Enhance CLI with additional options for DiT stages and configuration - Add comprehensive examples in examples/basic/ including: - API client with health checks and text generation - Docker setup and usage examples - Simple usage patterns for different scenarios - Add utility scripts for model downloading and Docker setup - Update documentation with implementation details and testing guidelines - Fix configuration validation issues in OmniLLM - Improve stage configuration handling for AR and DiT stages - Add proper error handling and fallback mechanisms Tested with Qwen3-0.6B model: - Server starts successfully on port 8000 - Health and info endpoints working correctly - Text generation with various parameters functioning - API client examples working as expected - CLI help and configuration options working properly
77 lines
1.5 KiB
Markdown
77 lines
1.5 KiB
Markdown
# Basic vLLM-omni Examples
|
|
|
|
This directory contains simple examples demonstrating how to use vLLM-omni for text generation.
|
|
|
|
## Examples
|
|
|
|
### 1. `simple_usage.py` - Direct Library Usage
|
|
|
|
Demonstrates how to use vLLM-omni directly in your Python code:
|
|
|
|
```bash
|
|
# Run the example
|
|
python simple_usage.py
|
|
```
|
|
|
|
**Features:**
|
|
- Synchronous text generation
|
|
- Asynchronous text generation
|
|
- Multi-prompt processing
|
|
- Shows both text output and token information
|
|
|
|
### 2. `api_client.py` - API Server Usage
|
|
|
|
Demonstrates how to interact with the vLLM-omni API server:
|
|
|
|
```bash
|
|
# First, start the server
|
|
vllm serve Qwen/Qwen3-0.6B --omni --port 8000
|
|
|
|
# Then run the client (in another terminal)
|
|
python api_client.py
|
|
```
|
|
|
|
**Features:**
|
|
- Health check
|
|
- Text generation via HTTP API
|
|
- Multiple prompt types
|
|
- Full response structure inspection
|
|
|
|
## Quick Start
|
|
|
|
1. **Install vLLM-omni:**
|
|
```bash
|
|
pip install -e .
|
|
```
|
|
|
|
2. **Run direct usage example:**
|
|
```bash
|
|
cd examples/basic
|
|
python simple_usage.py
|
|
```
|
|
|
|
3. **Run API server example:**
|
|
```bash
|
|
# Terminal 1: Start server
|
|
vllm serve Qwen/Qwen3-0.6B --omni --port 8000
|
|
|
|
# Terminal 2: Run client
|
|
cd examples/basic
|
|
python api_client.py
|
|
```
|
|
|
|
## Expected Output
|
|
|
|
The examples will show:
|
|
- ✅ Generated text responses
|
|
- ✅ Token counts and completion status
|
|
- ✅ Stage processing information
|
|
- ✅ Error handling
|
|
|
|
## Requirements
|
|
|
|
- Python 3.8+
|
|
- vLLM-omni installed
|
|
- For API examples: HTTP server running
|
|
|