Roger WangandGao Han
a5800cd2c2
Signed-off-by: Roger Wang <hey@rogerw.io> Signed-off-by: Gao Han <gaohan19@huawei.com> Co-authored-by: Gao Han <gaohan19@huawei.com>
535 B
535 B
Quickstart
This guide will help you quickly get started with vLLM-Omni to perform:
- Offline batched inference
- Online serving using OpenAI-compatible server
Prerequisites
- OS: Linux
- Python: 3.12
Installation
Please refer to installation
Offline Inference
Please refer to offline inference
Online Serving with OpenAI-Completions API
Please refer to online serving