hsliu
afe5c2692c
- Improve API server with better error handling and response formatting - Enhance CLI with additional options for DiT stages and configuration - Add comprehensive examples in examples/basic/ including: - API client with health checks and text generation - Docker setup and usage examples - Simple usage patterns for different scenarios - Add utility scripts for model downloading and Docker setup - Update documentation with implementation details and testing guidelines - Fix configuration validation issues in OmniLLM - Improve stage configuration handling for AR and DiT stages - Add proper error handling and fallback mechanisms Tested with Qwen3-0.6B model: - Server starts successfully on port 8000 - Health and info endpoints working correctly - Text generation with various parameters functioning - API client examples working as expected - CLI help and configuration options working properly
Basic vLLM-omni Examples
This directory contains simple examples demonstrating how to use vLLM-omni for text generation.
Examples
1. simple_usage.py - Direct Library Usage
Demonstrates how to use vLLM-omni directly in your Python code:
# Run the example
python simple_usage.py
Features:
- Synchronous text generation
- Asynchronous text generation
- Multi-prompt processing
- Shows both text output and token information
2. api_client.py - API Server Usage
Demonstrates how to interact with the vLLM-omni API server:
# First, start the server
vllm serve Qwen/Qwen3-0.6B --omni --port 8000
# Then run the client (in another terminal)
python api_client.py
Features:
- Health check
- Text generation via HTTP API
- Multiple prompt types
- Full response structure inspection
Quick Start
-
Install vLLM-omni:
pip install -e . -
Run direct usage example:
cd examples/basic python simple_usage.py -
Run API server example:
# Terminal 1: Start server vllm serve Qwen/Qwen3-0.6B --omni --port 8000 # Terminal 2: Run client cd examples/basic python api_client.py
Expected Output
The examples will show:
- ✅ Generated text responses
- ✅ Token counts and completion status
- ✅ Stage processing information
- ✅ Error handling
Requirements
- Python 3.8+
- vLLM-omni installed
- For API examples: HTTP server running