feat: enhance serving functionality and add comprehensive examples

- Improve API server with better error handling and response formatting
- Enhance CLI with additional options for DiT stages and configuration
- Add comprehensive examples in examples/basic/ including:
  - API client with health checks and text generation
  - Docker setup and usage examples
  - Simple usage patterns for different scenarios
- Add utility scripts for model downloading and Docker setup
- Update documentation with implementation details and testing guidelines
- Fix configuration validation issues in OmniLLM
- Improve stage configuration handling for AR and DiT stages
- Add proper error handling and fallback mechanisms

Tested with Qwen3-0.6B model:
- Server starts successfully on port 8000
- Health and info endpoints working correctly
- Text generation with various parameters functioning
- API client examples working as expected
- CLI help and configuration options working properly
This commit is contained in:
hsliu committed 2025-09-26 15:47:55 +08:00
1 parent 51d2570464
commit afe5c2692c
16 files changed
+1051 -134

No files matched your search

+12 -9
View File
@@ -53,19 +53,22 @@ async def generate(request: GenerateRequest):
# Prepare stage arguments
if request.stage_args is None:
# Create default stage arguments
stage_args = []
for i, prompt in enumerate(request.prompts):
stage_args.append({
"prompt": prompt,
"max_tokens": request.max_tokens,
"temperature": request.temperature
})
# Create default stage arguments - one per stage config
# For now, we'll process all prompts in the first stage
stage_args = [{
"prompt": " ".join(request.prompts) if request.prompts else "",
"max_tokens": request.max_tokens,
"temperature": request.temperature
}]
else:
stage_args = request.stage_args
# Generate using omni_llm
outputs = await omni_llm.generate_async(stage_args)
try:
outputs = await omni_llm.generate_async(stage_args)
except Exception as e:
# Fallback to synchronous generation
outputs = omni_llm.generate(stage_args)
# Convert outputs to response format
response_outputs = []