[New Model]: XiaomiMiMo/MiMo-Audio-7B-Instruct support (#750)
Signed-off-by: wangyu31577 <wangyu31577@hundsun.com> Signed-off-by: 齐保元 <qibaoyuan@xiaomi.com> Signed-off-by: hsliu <liuhongsheng4@huawei.com> Signed-off-by: Hongsheng Liu <liuhongsheng4@huawei.com> Signed-off-by: GG-li <3226868735@qq.com> Signed-off-by: Sihao Li <111170255+GG-li@users.noreply.github.com> Signed-off-by: XU Mingshi <91017482+mxuax@users.noreply.github.com> Signed-off-by: mxuax <mxuax@connect.ust.hk> Signed-off-by: Baoyuan Qi <qibaoyuan@126.com> Signed-off-by: gcanlin <canlinguosdu@gmail.com> Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn> Signed-off-by: wuzhongjian <wuzhongjian_yewu@cmss.chinamobile.com> Signed-off-by: dongbo910220 <1275604947@qq.com> Signed-off-by: dongbo910220 <32610838+dongbo910220@users.noreply.github.com> Signed-off-by: Jiangyun Zhu <riverclouds.zhu@qq.com> Signed-off-by: Junhong Liu <98734602+LJH-LBJ@users.noreply.github.com> Signed-off-by: tjtanaa <tunjian.tan@embeddedllm.com> Signed-off-by: baoyuan qi <qibaoyuan@126.com> Signed-off-by: tzhouam <tzhouam@connect.ust.hk> Signed-off-by: zjy0516 <riverclouds.zhu@qq.com> Signed-off-by: Prajwal A <prajwalanagani@gmail.com> Signed-off-by: Shijin Zhang <75300765+Dovis01@users.noreply.github.com> Signed-off-by: 丁宁 <nndding@gmail.com> Signed-off-by: SHIJIN ZHANG <75300765+Dovis01@users.noreply.github.com> Signed-off-by: dingning<dingning7@xiaomi.com> Signed-off-by: dingning <dingning7@xiaomi.com> Signed-off-by: dingning <dingning@xiaomi.com> Co-authored-by: wangyu <53896905+yenuo26@users.noreply.github.com> Co-authored-by: wangyu31577 <wangyu31577@hundsun.com> Co-authored-by: Zhang Shijin <zhangshijin@xiaomi.com> Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com> Co-authored-by: Sihao Li <111170255+GG-li@users.noreply.github.com> Co-authored-by: XU Mingshi <91017482+mxuax@users.noreply.github.com> Co-authored-by: Canlin Guo <canlinguosdu@gmail.com> Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn> Co-authored-by: JohnJan <wuzhongjian_yewu@cmss.chinamobile.com> Co-authored-by: WeiQing Chen <40507679+david6666666@users.noreply.github.com> Co-authored-by: dongbo910220 <32610838+dongbo910220@users.noreply.github.com> Co-authored-by: Jiangyun Zhu <riverclouds.zhu@qq.com> Co-authored-by: Junhong Liu <ljh_lbj@163.com> Co-authored-by: TJian <tunjian.tan@embeddedllm.com> Co-authored-by: shijin zhang <zsj1364226740@gmail.com> Co-authored-by: Zhou Taichang <tzhouam@connect.ust.hk> Co-authored-by: root <root@hk01dgx028.cm.cluster> Co-authored-by: Prajwal A <34590600+LawJarp-A@users.noreply.github.com> Co-authored-by: Shijin Zhang <75300765+Dovis01@users.noreply.github.com> Co-authored-by: dingning <dingning7@xiaomi.com> Co-authored-by: ning ding <nndding@gmail.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
This commit is contained in:
23 files changed
+8041
No files matched your search
@@ -0,0 +1,232 @@
|
||||
# MiMo-Audio Offline Inference
|
||||
|
||||
This directory contains an offline demo for running MiMo-Audio models with vLLM Omni. It builds task-specific inputs and generates WAV files or text outputs locally.
|
||||
|
||||
## Model Overview
|
||||
|
||||
MiMo-Audio provides multiple task variants for audio understanding and generation:
|
||||
|
||||
- **tts_sft**: Basic text-to-speech generation from text input.
|
||||
- **tts_sft_with_instruct**: TTS generation with explicit voice style instructions.
|
||||
- **tts_sft_with_audio**: TTS generation with audio reference for voice cloning.
|
||||
- **tts_sft_with_natural_instruction**: TTS generation from natural language descriptions embedded in text.
|
||||
- **audio_trancribing_sft**: Transcribe audio to text (speech-to-text).
|
||||
- **audio_understanding_sft**: Understand and analyze audio content with text queries.
|
||||
- **audio_understanding_sft_with_thinking**: Audio understanding with reasoning chain.
|
||||
- **spoken_dialogue_sft_multiturn**: Multi-turn spoken dialogue with audio input/output.
|
||||
- **speech2text_dialogue_sft_multiturn**: Multi-turn dialogue converting speech to text.
|
||||
- **text_dialogue_sft_multiturn**: Multi-turn text-only dialogue.
|
||||
|
||||
## Setup
|
||||
|
||||
Please refer to the [stage configuration documentation](https://docs.vllm.ai/projects/vllm-omni/en/latest/configuration/stage_configs/) to configure memory allocation appropriately for your hardware setup.
|
||||
|
||||
### Environment Variables
|
||||
|
||||
The `MIMO_AUDIO_TOKENIZER_PATH` environment variable is mandatory due to the specialized architecture:
|
||||
|
||||
```bash
|
||||
export MIMO_AUDIO_TOKENIZER_PATH="XiaomiMiMo/MiMo-Audio-Tokenizer"
|
||||
```
|
||||
|
||||
## Quick Start
|
||||
|
||||
Run a single sample for basic TTS:
|
||||
|
||||
```bash
|
||||
python3 -u end2end.py \
|
||||
--stage-configs-path vllm_omni/model_executor/stage_configs/mimo_audio.yaml \
|
||||
--model-name XiaomiMiMo/MiMo-Audio-7B-Instruct \
|
||||
--query-type tts_sft
|
||||
```
|
||||
|
||||
Run batch samples for basic TTS:
|
||||
|
||||
```bash
|
||||
python3 -u end2end.py \
|
||||
--stage-configs-path vllm_omni/model_executor/stage_configs/mimo_audio.yaml \
|
||||
--model-name XiaomiMiMo/MiMo-Audio-7B-Instruct \
|
||||
--query-type tts_sft \
|
||||
--num-prompts {batch_size}
|
||||
```
|
||||
|
||||
When enabling multi-batch processing, if the total number of tokens passed to the next stage exceeds the `max_model_len` value in the `mimo_audio.yaml` configuration file, you must also synchronously update the `max_position_embeddings` value in `MiMo-Audio-7B-Instruct/config.json` to match the modified value.
|
||||
|
||||
Generated audio files are saved to `output_audio/` by default. `--num-prompts` also can be used to all tasks below.
|
||||
|
||||
## Task Usage
|
||||
|
||||
### tts_sft (Basic Text-to-Speech)
|
||||
|
||||
Generate speech from text input:
|
||||
|
||||
```bash
|
||||
python3 -u end2end.py \
|
||||
--stage-configs-path vllm_omni/model_executor/stage_configs/mimo_audio.yaml \
|
||||
--model-name XiaomiMiMo/MiMo-Audio-7B-Instruct \
|
||||
--query-type tts_sft \
|
||||
--text "The weather is so nice today."
|
||||
```
|
||||
|
||||
### tts_sft_with_instruct (TTS with Voice Instructions)
|
||||
|
||||
Generate speech with explicit voice style instructions:
|
||||
|
||||
```bash
|
||||
python3 -u end2end.py \
|
||||
--stage-configs-path vllm_omni/model_executor/stage_configs/mimo_audio.yaml \
|
||||
--model-name XiaomiMiMo/MiMo-Audio-7B-Instruct \
|
||||
--query-type tts_sft_with_instruct \
|
||||
--text "The weather is so nice today." \
|
||||
--instruct "Speak happily in a child's voice"
|
||||
```
|
||||
|
||||
### tts_sft_with_audio (TTS with Audio Reference)
|
||||
|
||||
Generate speech using an audio reference for voice cloning:
|
||||
|
||||
```bash
|
||||
python3 -u end2end.py \
|
||||
--stage-configs-path vllm_omni/model_executor/stage_configs/mimo_audio.yaml \
|
||||
--model-name XiaomiMiMo/MiMo-Audio-7B-Instruct \
|
||||
--query-type tts_sft_with_audio \
|
||||
--text "The weather is so nice today." \
|
||||
--audio-path "./spoken_dialogue_assistant_turn_1.wav"
|
||||
```
|
||||
|
||||
### tts_sft_with_natural_instruction (Natural Language TTS)
|
||||
|
||||
Generate speech from text containing natural voice descriptions:
|
||||
|
||||
```bash
|
||||
python3 -u end2end.py \
|
||||
--stage-configs-path vllm_omni/model_executor/stage_configs/mimo_audio.yaml \
|
||||
--model-name XiaomiMiMo/MiMo-Audio-7B-Instruct \
|
||||
--query-type tts_sft_with_natural_instruction \
|
||||
--text "In a panting young male voice, he said: I can't run anymore, wait for me!"
|
||||
```
|
||||
|
||||
### audio_trancribing_sft (Speech-to-Text)
|
||||
|
||||
Transcribe audio to text:
|
||||
|
||||
```bash
|
||||
python3 -u end2end.py \
|
||||
--stage-configs-path vllm_omni/model_executor/stage_configs/mimo_audio.yaml \
|
||||
--model-name XiaomiMiMo/MiMo-Audio-7B-Instruct \
|
||||
--query-type audio_trancribing_sft \
|
||||
--audio-path "./spoken_dialogue_assistant_turn_1.wav"
|
||||
```
|
||||
|
||||
### audio_understanding_sft (Audio Understanding)
|
||||
|
||||
Understand and analyze audio content with text queries:
|
||||
|
||||
```bash
|
||||
python3 -u end2end.py \
|
||||
--stage-configs-path vllm_omni/model_executor/stage_configs/mimo_audio.yaml \
|
||||
--model-name XiaomiMiMo/MiMo-Audio-7B-Instruct \
|
||||
--query-type audio_understanding_sft \
|
||||
--text "Summarize the audio." \
|
||||
--audio-path "./spoken_dialogue_assistant_turn_1.wav"
|
||||
```
|
||||
|
||||
### audio_understanding_sft_with_thinking (Audio Understanding with Reasoning)
|
||||
|
||||
Audio understanding with reasoning chain:
|
||||
|
||||
```bash
|
||||
python3 -u end2end.py \
|
||||
--stage-configs-path vllm_omni/model_executor/stage_configs/mimo_audio.yaml \
|
||||
--model-name XiaomiMiMo/MiMo-Audio-7B-Instruct \
|
||||
--query-type audio_understanding_sft_with_thinking \
|
||||
--text "Summarize the audio." \
|
||||
--audio-path "./spoken_dialogue_assistant_turn_1.wav"
|
||||
```
|
||||
|
||||
### spoken_dialogue_sft_multiturn (Multi-turn Spoken Dialogue)
|
||||
|
||||
Multi-turn dialogue with audio input and output:
|
||||
|
||||
```bash
|
||||
python3 -u end2end.py \
|
||||
--stage-configs-path vllm_omni/model_executor/stage_configs/mimo_audio.yaml \
|
||||
--model-name XiaomiMiMo/MiMo-Audio-7B-Instruct \
|
||||
--query-type spoken_dialogue_sft_multiturn \
|
||||
--audio-path "./prompt_speech_zh_m.wav"
|
||||
```
|
||||
|
||||
Note: This task uses hardcoded audio files in the script. The audio files used in examples are available at: https://github.com/XiaomiMiMo/MiMo-Audio/tree/main/examples
|
||||
|
||||
### speech2text_dialogue_sft_multiturn (Speech-to-Text Dialogue)
|
||||
|
||||
Multi-turn dialogue converting speech to text:
|
||||
|
||||
```bash
|
||||
python3 -u end2end.py \
|
||||
--stage-configs-path vllm_omni/model_executor/stage_configs/mimo_audio.yaml \
|
||||
--model-name XiaomiMiMo/MiMo-Audio-7B-Instruct \
|
||||
--query-type speech2text_dialogue_sft_multiturn
|
||||
```
|
||||
|
||||
Note: This task uses hardcoded audio files and message lists in the script.
|
||||
|
||||
### text_dialogue_sft_multiturn (Text Dialogue)
|
||||
|
||||
Multi-turn text-only dialogue:
|
||||
|
||||
```bash
|
||||
python3 -u end2end.py \
|
||||
--stage-configs-path vllm_omni/model_executor/stage_configs/mimo_audio.yaml \
|
||||
--model-name XiaomiMiMo/MiMo-Audio-7B-Instruct \
|
||||
--query-type text_dialogue_sft_multiturn
|
||||
```
|
||||
|
||||
Note: This task uses hardcoded message lists in the script.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Audio dependencies (soundfile, librosa)
|
||||
|
||||
This example depends on **soundfile** (read/write WAV) and **librosa** (load audio including MP3). Install the project requirements first:
|
||||
|
||||
```bash
|
||||
pip install -r requirements/common.txt
|
||||
# or at least: pip install soundfile>=0.13.1 librosa>=0.11.0
|
||||
```
|
||||
|
||||
- **`soundfile` / libsndfile not found**
|
||||
`soundfile` uses the C library **libsndfile**. On Linux, install the system package before pip:
|
||||
- Debian/Ubuntu: `sudo apt-get install libsndfile1`
|
||||
- For development builds: `sudo apt-get install libsndfile1-dev`
|
||||
- Then: `pip install soundfile`
|
||||
|
||||
- **`librosa` fails to load MP3 or reports "No backend available"**
|
||||
Loading MP3 (e.g. in `spoken_dialogue_sft_multiturn` with `.mp3` files) uses **ffmpeg** as the backend. Install ffmpeg:
|
||||
- Debian/Ubuntu: `sudo apt-get install ffmpeg`
|
||||
- macOS: `brew install ffmpeg`
|
||||
|
||||
- **`ImportError: No module named 'soundfile'` or `ModuleNotFoundError: ... librosa`**
|
||||
Ensure you are in the same Python environment where vLLM Omni and the example dependencies are installed, and that `requirements/common.txt` (or the packages above) are installed.
|
||||
|
||||
### Tokenizer path
|
||||
|
||||
- **`MIMO_AUDIO_TOKENIZER_PATH` not set or model fails to find tokenizer**
|
||||
Export the tokenizer path before running:
|
||||
```bash
|
||||
export MIMO_AUDIO_TOKENIZER_PATH="XiaomiMiMo/MiMo-Audio-Tokenizer"
|
||||
```
|
||||
See [Environment Variables](#environment-variables) in Setup.
|
||||
|
||||
### Other
|
||||
|
||||
- If the model or stage config fails to load, check [stage configuration documentation](https://docs.vllm.ai/projects/vllm-omni/en/latest/configuration/stage_configs/) for memory and GPU settings.
|
||||
- For errors when reading/writing WAV (e.g. unsupported format), ensure input files are standard WAV/MP3 and that `soundfile` is linked to a working libsndfile (see above).
|
||||
|
||||
## Notes
|
||||
|
||||
- The script uses default model paths and audio files embedded in `end2end.py`. Update them if your local cache path differs.
|
||||
- Use `--output-dir` to change the output folder (default: `./output_audio`).
|
||||
- Use `--num-prompts` to generate multiple prompts in one run (default: 1).
|
||||
- Audio files used in multi-turn dialogue examples are available at: https://github.com/XiaomiMiMo/MiMo-Audio/tree/main/examples
|
||||
- The script supports various configuration options for initialization timeouts, batch timeouts, and shared memory thresholds. See `--help` for details.
|
||||
@@ -0,0 +1,443 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||
"""
|
||||
This example shows how to use vLLM for running offline inference
|
||||
with the correct prompt format on MiMo-Audio-Omni.
|
||||
"""
|
||||
|
||||
import copy
|
||||
import json
|
||||
import os
|
||||
from typing import NamedTuple
|
||||
|
||||
import soundfile as sf
|
||||
from message_convert import (
|
||||
get_audio_data,
|
||||
get_audio_understanding_sft_prompt,
|
||||
get_s2t_dialogue_sft_multiturn_prompt,
|
||||
get_spoken_dialogue_sft_multiturn_prompt,
|
||||
get_text_dialogue_sft_multiturn_prompt,
|
||||
get_tts_sft_prompt,
|
||||
to_prompt,
|
||||
)
|
||||
from vllm import SamplingParams
|
||||
from vllm.utils.argparse_utils import FlexibleArgumentParser
|
||||
|
||||
from vllm_omni.entrypoints.omni import Omni
|
||||
from vllm_omni.inputs.data import OmniTokensPrompt
|
||||
|
||||
SEED = 42
|
||||
|
||||
MAX_CODE2WAV_TOKENS = 18192 # Maximum tokens supported by code2wav model
|
||||
|
||||
|
||||
class QueryResult(NamedTuple):
|
||||
inputs: dict
|
||||
limit_mm_per_prompt: dict[str, int]
|
||||
|
||||
|
||||
def get_codes_query_from_json(codes_path: str) -> QueryResult:
|
||||
with open(codes_path, encoding="utf-8") as f:
|
||||
data = json.load(f)
|
||||
|
||||
if isinstance(data, list):
|
||||
code_final = data
|
||||
elif isinstance(data, dict) and "code_final" in data:
|
||||
code_final = data["code_final"]
|
||||
else:
|
||||
raise ValueError(
|
||||
f"Unsupported codes json format in {codes_path}.\n"
|
||||
"Expect a JSON list[int] or {{'code_final': list[int]}}."
|
||||
)
|
||||
|
||||
if not isinstance(code_final, list) or not all(isinstance(x, int) for x in code_final):
|
||||
raise ValueError("code_final must be a list[int].")
|
||||
|
||||
if len(code_final) > MAX_CODE2WAV_TOKENS:
|
||||
print(f"[Warn] code_final len={len(code_final)} > {MAX_CODE2WAV_TOKENS}, truncating.")
|
||||
code_final = code_final[:MAX_CODE2WAV_TOKENS]
|
||||
|
||||
return QueryResult(
|
||||
inputs=OmniTokensPrompt(
|
||||
prompt_token_ids=code_final,
|
||||
multi_modal_data=None,
|
||||
mm_processor_kwargs=None,
|
||||
),
|
||||
limit_mm_per_prompt={},
|
||||
)
|
||||
|
||||
|
||||
def get_tts_sft(
|
||||
text="The weather is so nice today.",
|
||||
instruct=None,
|
||||
read_text_only=True,
|
||||
prompt_speech=None,
|
||||
audio_list=None,
|
||||
):
|
||||
res = get_tts_sft_prompt(
|
||||
text,
|
||||
instruct=instruct,
|
||||
read_text_only=read_text_only,
|
||||
prompt_speech=prompt_speech,
|
||||
)
|
||||
|
||||
prompt = to_prompt(res)
|
||||
final_prompt = {
|
||||
"prompt": prompt,
|
||||
}
|
||||
if audio_list is not None:
|
||||
final_prompt.update(
|
||||
{
|
||||
"multi_modal_data": {
|
||||
"audio": audio_list,
|
||||
},
|
||||
}
|
||||
)
|
||||
return final_prompt
|
||||
|
||||
|
||||
def get_audio_understanding_sft(audio_path, text="", thinking=False, use_sostm=False):
|
||||
audio_list = []
|
||||
audio_list.append(get_audio_data(audio_path))
|
||||
res = get_audio_understanding_sft_prompt(
|
||||
input_speech=audio_path, input_text=text, thinking=thinking, use_sostm=use_sostm
|
||||
)
|
||||
prompt = to_prompt(res)
|
||||
final_prompt = {
|
||||
"prompt": prompt,
|
||||
"multi_modal_data": {
|
||||
"audio": audio_list,
|
||||
},
|
||||
}
|
||||
return final_prompt
|
||||
|
||||
|
||||
def get_spoken_dialogue_sft_multiturn(message_list, system_prompt=None, ref_audio_path=None, audio_list=None):
|
||||
res = get_spoken_dialogue_sft_multiturn_prompt(
|
||||
message_list, system_prompt=system_prompt, prompt_speech=ref_audio_path
|
||||
)
|
||||
prompt = to_prompt(res)
|
||||
final_prompt = {
|
||||
"prompt": prompt,
|
||||
"multi_modal_data": {
|
||||
"audio": audio_list,
|
||||
},
|
||||
}
|
||||
return final_prompt
|
||||
|
||||
|
||||
def get_speech2text_dialogue_sft_multiturn(message_list, thinking=False, audio_list=None):
|
||||
res = get_s2t_dialogue_sft_multiturn_prompt(
|
||||
message_list,
|
||||
thinking=thinking,
|
||||
)
|
||||
prompt = to_prompt(res)
|
||||
final_prompt = {
|
||||
"prompt": prompt,
|
||||
"multi_modal_data": {
|
||||
"audio": audio_list,
|
||||
},
|
||||
}
|
||||
return final_prompt
|
||||
|
||||
|
||||
def get_text_dialogue_sft_multiturn(
|
||||
message_list,
|
||||
):
|
||||
res = get_text_dialogue_sft_multiturn_prompt(
|
||||
message_list,
|
||||
)
|
||||
prompt = to_prompt(res)
|
||||
final_prompt = {
|
||||
"prompt": prompt,
|
||||
}
|
||||
return final_prompt
|
||||
|
||||
|
||||
query_map = {
|
||||
"tts_sft": get_tts_sft,
|
||||
"tts_sft_with_instruct": get_tts_sft,
|
||||
"tts_sft_with_audio": get_tts_sft,
|
||||
"tts_sft_with_natural_instruction": get_tts_sft,
|
||||
"audio_trancribing_sft": get_audio_understanding_sft,
|
||||
"audio_understanding_sft": get_audio_understanding_sft,
|
||||
"audio_understanding_sft_with_thinking": get_audio_understanding_sft,
|
||||
"spoken_dialogue_sft_multiturn": get_spoken_dialogue_sft_multiturn,
|
||||
"speech2text_dialogue_sft_multiturn": get_speech2text_dialogue_sft_multiturn,
|
||||
"text_dialogue_sft_multiturn": get_text_dialogue_sft_multiturn,
|
||||
}
|
||||
|
||||
|
||||
def main(args):
|
||||
model_name = args.model_name
|
||||
|
||||
# Get paths from args
|
||||
text = getattr(args, "text", None)
|
||||
audio_path = getattr(args, "audio_path", None)
|
||||
|
||||
instruct = getattr(args, "instruct", None)
|
||||
|
||||
# Get the query function and call it with appropriate parameters
|
||||
query_func = query_map[args.query_type]
|
||||
|
||||
omni_llm = Omni(
|
||||
model=model_name,
|
||||
stage_configs_path=args.stage_configs_path,
|
||||
log_stats=args.enable_stats,
|
||||
log_file=("omni_llm_pipeline.log" if args.enable_stats else None),
|
||||
init_sleep_seconds=args.init_sleep_seconds,
|
||||
batch_timeout=args.batch_timeout,
|
||||
init_timeout=args.init_timeout,
|
||||
shm_threshold_bytes=args.shm_threshold_bytes,
|
||||
)
|
||||
|
||||
thinker_sampling_params = SamplingParams(
|
||||
temperature=0.0,
|
||||
top_p=1.0,
|
||||
top_k=-1,
|
||||
max_tokens=2048,
|
||||
seed=SEED,
|
||||
logit_bias={},
|
||||
repetition_penalty=1.1,
|
||||
)
|
||||
|
||||
code2wav_sampling_params = SamplingParams(
|
||||
temperature=0.0,
|
||||
top_p=1.0,
|
||||
top_k=-1,
|
||||
max_tokens=4096 * 16,
|
||||
seed=SEED,
|
||||
detokenize=True,
|
||||
repetition_penalty=1.1,
|
||||
)
|
||||
|
||||
sampling_params_list = [
|
||||
thinker_sampling_params,
|
||||
code2wav_sampling_params,
|
||||
]
|
||||
|
||||
# Build query result based on query type
|
||||
# Notice: The audio files used in this example are available at: https://github.com/XiaomiMiMo/MiMo-Audio/tree/main/examples
|
||||
if args.query_type == "tts_sft":
|
||||
# python3 -u end2end.py --stage-configs-path ${config_file} --model ${MODEL_PATH} --query-type tts_sft
|
||||
query_result = query_func(text=text, read_text_only=True)
|
||||
elif args.query_type == "tts_sft_with_instruct":
|
||||
# python3 -u end2end.py --stage-configs-path ${config_file} --model ${MODEL_PATH} --query-type tts_sft_with_instruct --instruct "Speak happily in a child's voice"
|
||||
query_result = query_func(text=text, instruct=instruct, read_text_only=True)
|
||||
elif args.query_type == "tts_sft_with_audio":
|
||||
# python3 -u end2end.py --stage-configs-path ${config_file} --model ${MODEL_PATH} --query-type tts_sft_with_audio --audio_path "./spoken_dialogue_assistant_turn_1.wav"
|
||||
audio_list = [get_audio_data(audio_path)]
|
||||
query_result = query_func(text=text, read_text_only=True, prompt_speech=audio_path, audio_list=audio_list)
|
||||
elif args.query_type == "tts_sft_with_natural_instruction":
|
||||
# python3 -u end2end.py --stage-configs-path ${config_file} --model ${MODEL_PATH} --query-type tts_sft_with_natural_instruction --text "In a panting young male voice, he said: I can't run anymore, wait for me!"
|
||||
query_result = query_func(text=text, read_text_only=False)
|
||||
elif args.query_type == "audio_trancribing_sft":
|
||||
# python3 -u end2end.py --stage-configs-path ${config_file} --model ${MODEL_PATH} --query-type audio_trancribing_sft --audio_path "./spoken_dialogue_assistant_turn_1.wav"
|
||||
audio_path = "spoken_dialogue_assistant_turn_1.wav"
|
||||
text = "Please transcribe this audio and repeat it once."
|
||||
query_result = query_func(text=text, audio_path=audio_path, use_sostm=True)
|
||||
elif args.query_type == "audio_understanding_sft":
|
||||
# python3 -u end2end.py --stage-configs-path ${config_file} --model ${MODEL_PATH} --query-type audio_understanding_sft --text "Summarize the audio." --audio_path "./spoken_dialogue_assistant_turn_1.wav"
|
||||
query_result = query_func(text=text, audio_path=audio_path)
|
||||
elif args.query_type == "audio_understanding_sft_with_thinking":
|
||||
# python3 -u end2end.py --stage-configs-path ${config_file} --model ${MODEL_PATH} --query-type audio_understanding_sft_with_thinking --text "Summarize the audio." --audio_path "./spoken_dialogue_assistant_turn_1.wav"
|
||||
query_result = query_func(text=text, audio_path=audio_path, thinking=True)
|
||||
elif args.query_type == "spoken_dialogue_sft_multiturn":
|
||||
# python3 -u end2end.py --stage-configs-path ${config_file} --model ${MODEL_PATH} --query-type spoken_dialogue_sft_multiturn --audio_path "./prompt_speech_zh_m.wav"
|
||||
first_turn_text_response = "我没办法获取实时的天气信息。不过呢,你可以试试几个方法来查看今天的天气。首先,你可以用手机自带的天气功能,比如苹果手机的天气应用,或者直接在系统设置里查看。其次,你也可以用一些专业的天气服务,像是国外的AccuWeather、Weather.com,或者国内的中国天气网、墨迹天气等等。再有就是,你还可以在谷歌或者百度里直接搜索你所在的城市加上天气这两个字。如果你能告诉我你所在的城市,我也可以帮你分析一下历史天气趋势,不过最新的数据还是需要你通过官方渠道去获取哦。"
|
||||
audio_list = []
|
||||
s1_audio_path = "weather_of_today.mp3"
|
||||
s2_audio_path = "spoken_dialogue_assistant_turn_1.wav"
|
||||
s3_audio_path = "beijing.mp3"
|
||||
audio_list.append(get_audio_data(audio_path))
|
||||
audio_list.append(get_audio_data(s1_audio_path))
|
||||
audio_list.append(get_audio_data(s2_audio_path))
|
||||
audio_list.append(get_audio_data(s3_audio_path))
|
||||
|
||||
message_list = [
|
||||
{"role": "user", "content": s1_audio_path},
|
||||
{"role": "assistant", "content": {"text": first_turn_text_response, "audio": s2_audio_path}},
|
||||
{"role": "user", "content": s3_audio_path},
|
||||
]
|
||||
query_result = query_func(message_list, system_prompt=None, ref_audio_path=audio_path, audio_list=audio_list)
|
||||
elif args.query_type == "speech2text_dialogue_sft_multiturn":
|
||||
# python3 -u end2end.py --stage-configs-path ${config_file} --model ${MODEL_PATH} --query-type speech2text_dialogue_sft_multiturn
|
||||
s1_audio_path = "weather_of_today.mp3"
|
||||
s2_audio_path = "beijing.mp3"
|
||||
audio_list = []
|
||||
audio_list.append(get_audio_data(s1_audio_path))
|
||||
audio_list.append(get_audio_data(s2_audio_path))
|
||||
message_list = [
|
||||
{"role": "user", "content": s1_audio_path},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": "Hello, I can't get real-time weather information. If you can tell me your city, I can help you analyze historical weather trends, but for the latest data, you'll need to get it through official channels.",
|
||||
},
|
||||
{"role": "user", "content": s2_audio_path},
|
||||
]
|
||||
query_result = query_func(message_list, thinking=True, audio_list=audio_list)
|
||||
elif args.query_type == "text_dialogue_sft_multiturn":
|
||||
# python3 -u end2end.py --stage-configs-path ${config_file} --model ${MODEL_PATH} --query-type text_dialogue_sft_multiturn
|
||||
message_list = [
|
||||
{"role": "user", "content": "Could you recommend some tourist attractions in China?"},
|
||||
{"role": "assistant", "content": "Hello, which city would you like to travel to?"},
|
||||
{"role": "user", "content": "Beijing"},
|
||||
]
|
||||
query_result = query_func(message_list=message_list)
|
||||
else:
|
||||
raise ValueError(f"Invalid query type: {args.query_type}")
|
||||
|
||||
prompts = [copy.deepcopy(query_result) for _ in range(args.num_prompts)]
|
||||
|
||||
print("prompts", prompts)
|
||||
omni_outputs = omni_llm.generate(prompts, sampling_params_list)
|
||||
|
||||
output_dir = args.output_dir if getattr(args, "output_dir", None) else args.output_wav
|
||||
if args.query_type is not None:
|
||||
output_dir = os.path.join(output_dir, args.query_type)
|
||||
os.makedirs(output_dir, exist_ok=True)
|
||||
|
||||
for stage_outputs in omni_outputs:
|
||||
if stage_outputs.final_output_type == "text":
|
||||
for output in stage_outputs.request_output:
|
||||
request_id = output.request_id
|
||||
text_output = output.outputs[0].text
|
||||
# Save aligned text file per request
|
||||
prompt_text = output.prompt
|
||||
out_txt = os.path.join(output_dir, f"{request_id}.txt")
|
||||
lines = []
|
||||
lines.append("Prompt:\n")
|
||||
lines.append(str(prompt_text) + "\n")
|
||||
lines.append("vllm_text_output:\n")
|
||||
lines.append(str(text_output).strip() + "\n")
|
||||
try:
|
||||
with open(out_txt, "w", encoding="utf-8") as f:
|
||||
print("lines", lines)
|
||||
f.writelines(lines)
|
||||
except Exception as e:
|
||||
print(f"[Warn] Failed writing text file {out_txt}: {e}")
|
||||
print(f"Request ID: {request_id}, Text saved to {out_txt}\n")
|
||||
elif stage_outputs.final_output_type == "audio":
|
||||
for output in stage_outputs.request_output:
|
||||
request_id = output.request_id
|
||||
audio_tensor = output.outputs[0].multimodal_output.get("audio")
|
||||
|
||||
if audio_tensor is None:
|
||||
continue
|
||||
|
||||
output_wav = os.path.join(output_dir, f"{request_id}.wav")
|
||||
|
||||
# Convert to numpy array and ensure correct format
|
||||
audio_numpy = audio_tensor.float().detach().cpu().numpy()
|
||||
|
||||
# Ensure audio is 1D (flatten if needed)
|
||||
if audio_numpy.ndim > 1:
|
||||
audio_numpy = audio_numpy.flatten()
|
||||
|
||||
# Save audio file with explicit WAV format
|
||||
sf.write(output_wav, audio_numpy, samplerate=24000, format="WAV")
|
||||
print(f"Request ID: {request_id}, Audio saved to {output_wav}")
|
||||
|
||||
|
||||
def parse_args():
|
||||
parser = FlexibleArgumentParser(description="Demo on using vLLM for offline inference with audio language models")
|
||||
parser.add_argument(
|
||||
"--model-name",
|
||||
"-m",
|
||||
type=str,
|
||||
default="XiaomiMiMo/MiMo-Audio-7B-Instruct",
|
||||
help="Backbone LLM path.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--text",
|
||||
"-t",
|
||||
type=str,
|
||||
default="The weather is so nice today.",
|
||||
help="input text",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--query-type",
|
||||
"-q",
|
||||
type=str,
|
||||
default="tts_sft",
|
||||
choices=query_map.keys(),
|
||||
help="Query type.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--audio-path",
|
||||
"-a",
|
||||
type=str,
|
||||
default=None,
|
||||
help="Path to local audio file. If not provided, uses default audio asset.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--instruct",
|
||||
type=str,
|
||||
default=None,
|
||||
help="instruct",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--enable-stats",
|
||||
action="store_true",
|
||||
help="Enable writing detailed statistics (default: disabled)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--init-sleep-seconds",
|
||||
type=int,
|
||||
default=20,
|
||||
help="Sleep seconds after starting each stage process to allow initialization (default: 20)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--batch-timeout",
|
||||
type=int,
|
||||
default=5,
|
||||
help="Timeout for batching in seconds (default: 5)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--init-timeout",
|
||||
type=int,
|
||||
default=5000,
|
||||
help="Timeout for initializing stages in seconds (default: 5000)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--shm-threshold-bytes",
|
||||
type=int,
|
||||
default=65536,
|
||||
help="Threshold for using shared memory in bytes (default: 65536)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--output-dir",
|
||||
default="./output_audio",
|
||||
help="Output audio wav directory.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--output-wav",
|
||||
default="output_audio",
|
||||
help="[Deprecated] Output wav directory (use --output-dir).",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--num-prompts",
|
||||
type=int,
|
||||
default=1,
|
||||
help="Number of prompts to generate.",
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
"--sampling-rate",
|
||||
type=int,
|
||||
default=24000,
|
||||
help="Sampling rate for audio.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--stage-configs-path",
|
||||
type=str,
|
||||
default="../../../model_executor/stage_configs/mimo_audio.yaml",
|
||||
help="Path to a stage configs file.",
|
||||
)
|
||||
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
args = parse_args()
|
||||
main(args)
|
||||
@@ -0,0 +1,56 @@
|
||||
{
|
||||
"messages": [
|
||||
{
|
||||
"role": "system",
|
||||
"content": [
|
||||
{
|
||||
"type": "text",
|
||||
"text": "Your voice should be:"
|
||||
},
|
||||
{
|
||||
"type": "audio_url",
|
||||
"audio_url": {
|
||||
"url": "data:audio/wav;base64,UklGRiRKAwBXQVZFZm10IBAAAAABAAEAwF0AAIC7AAACABAAZGF0YQBKAwAFAP3/CAD7/wkAAAD9/wgA+v8JAP//AQADAP//CAD+/wEABAABAP7/BgD8//7/BgD0/wYA/f/8/wMA9f8GAP///P8DAPz/AAAFAP3//v8CAP7////+////+/8IAP//AQAAAAMABgD6/wUA+v8DAAcA+/8AAAMAAAAAAP3//v8DAAMACgD8/wQACQD9/wUA/f8BAAcABQADAAAABAADAAEA+v8AAAIAAAAFAPz//f8DAAAA/f8AAAMA/v8FAP7/9v8GAP7//f8EAAAAAQAIAPv/+/8AAP3////+//3/AAAHAPr//v8BAPz/AwD///n/AAACAPr//f/8//7/AAD//wAA+f8GAPz/9v8CAPX/BgACAPv/AQD//wQA///7/wAAAAAHAAQA/f8FAAAAAgAAAPv/BAAGAAAABgACAP7/DQD7//3/BAD//wkA//8DAPz/AwAHAPf/AwAAAAAABAD7//f/AwD8/wAA/f/5/wQA/P8AAPr//P8AAAAA//8AAP////8DAPL/AQD9//r/CwD4/wEAAgD9//7//P/9//7/AAD//wAA/v8DAPr/+v8CAPr///8CAPz///8GAPr/AAAEAP//AgADAP//AAAHAPz/BgD//wUACwD//wYAAgABAAQABQD8/wgABgACAAUA//8CAAQAAAADAP7/AQAIAPr/AwD+//v/AgACAPv/AQAJAPf/AQD+//n/AgAAAP///f8CAAAA/P8DAPf/+/8HAPv///////7//f8AAP3//P8BAPr/AAD6//z/AAD5/wIAAAD7/wIAAgD4/wAA/v/6/wQAAQAAAAAABwAAAAAAAwD+/wIABQADAAEABAADAAAAAwABAAAABAAGAAEAAgAGAAAABQD///7/BQAAAAMAAQAAAAIABAD8/wMA/////wQA/f8AAAEAAQD//wMA+/8FAPr///////P/CwDz/wQA/P/6/wIA8/8AAPH//P/+//f//v/+//n//P/8//L//v/7//r////7//3//v/5//v/AAD5/wMAAAD8/wYA/P///////////wAABwD6/wwAAQD+/wUA/P8GAP7/AwD+/wUABwD+/wUA//8FAAMAAgD//wUAAAAEAAcA+/8KAAUAAgAEAAAA//8HAAAA+v8EAAIAAQAEAPz/+/8EAP7//////wAAAQABAP//+///////AAD9/wEA/f8DAAAA+P8AAPj/AQD+//v/AQD8/wIA/v/6/wAA/////wIA/P///wMA+/8BAPj///8CAAAAAwD+/wcAAAAFAP3/AAAFAP7/AwABAP7/AAACAPv/AwD+/wMABAD9/wUA/v8AAAEA/v8AAAIAAQAAAAAAAAD8/wIA/v/8/wMAAAD///3/AAD5//7/AwD2/wgAAAD8/wQA+f8CAP7/AAAAAP//AwD//wAA/v//////AAAAAAAAAwD9/wIA+//9/wQA+/8DAAAA//8AAP///f/+/wAAAAABAP//BgD8/wIAAgD3/wwA//8AAAkA/f8EAAAA//8BAAAAAwADAP//BQD///z/AwD8/wEAAwAAAAEAAQAAAAAA/v8BAAAAAQAGAP7/AAD///3//v8AAAAAAgAGAP//AgAAAP3////9/wAAAQACAAAAAQD///3/AQD9/wAABwD//wMABAD8/wEAAwD+/wIABQD+/wEAAwD6////BAD5/wYAAAD+/wgA+f8CAP3///8DAP7/AgAAAAAA/v8EAPv/AAAHAPv/CQACAP7/BgD//wIAAAADAAAAAgAEAPz/BQD8/wQA/P/+/wMA+f8EAP3//P8AAP////8AAPv/AgD8/wAAAgDz/wgA/f/7/wQA+P8BAAAA+v8BAP7/AAACAP3/AAACAP7/AQAEAPj/BwAAAAAABgD7/wcA+v8CAP7/+/8IAP3/AQACAAAAAQACAAAA/v8CAAcA/f8HAAIA/v8IAPv/AQAAAAAABAD+/wQAAgD//wMA/P8AAAAA//8BAPr/BQD9/wAA///4/wIA+/8AAP3//v8EAPr/AAD9//r/BQD3/wEA+v/6/wYA8/8BAPz/9v8CAPn/+f/+//3//v/+//3//v///wAA/P/7/wAAAAD//wAA//8AAP7/AgD4////BAD7/wYAAAADAAAAAwAAAPr/BQD6/wUABAD//wcA/f8EAPz//P8FAPv/BQADAP//BgAAAP//AQAAAAAAAgAAAAEAAwAAAAAA//8AAAAAAAAAAP7/BAD///7/AgD8/wAAAQD8////AQD9/wQA/v///wIA//8BAPv/AQD9/wMA/v/8/wQA+f8DAP3/9/8EAAAA/P8GAPz/AwAEAP//AAD8/wQAAAAGAAAAAgAFAAEABAD8/wEAAQADAAAAAgADAAAACAD7//7/BQD9/wUAAAD+/wcAAAABAAAA/f8DAAMAAAAAAAEA/v8AAAAA+f8AAAEA//8AAP//AAD7/wUA9//6/wQA9/8BAP3//f8AAP7/AAD7//3/AwD+//v//v/7//3/BAD3//r/CAD8/wIA/v/6//3/AQD+//j/AAAAAAgAAAD///v//f8BAP7//v/9/wEA//8EAAAA//8BAAEAAAD+/////f8AAP7/AgD9/wQABQD4/wIA/f/5/wAA/P/9/wEAAQABAPf/AAD8//7/AQD5/wAA/f8BAPf/+f/7//b/AwADAPz///8BAPn//P/4/////P8AAAAA+P8JAAAA/f/8//3/AAABAAAA8v8EAAMAAAACAPn/BAAOAAcA+//t////DwAEAAIA/f8HAAYADAD7//f/BgD//wwA//8BAAEADwAEAPv/BQD1/xEA//8DAP//9/8LAPn/AgABAAMABAAEAPz//f8FAPb/BAD+//r/CwAAAP7/BQD4////BAD4/wEA/v8AAAAA/v/+//z/BAABAP//BQD+//z/AQD7/wAA/P8DAAUAAAAHAP//AQD9////AAABAP//AAACAAQACAD+/wYA//8DAAcA/v8CAAEABQADAAIAAgACAAUABQAAAAAAAgAAAAAAAwABAAAABQAAAAYABgAAAAEA/v8EAAAA/P8GAP7/BAAGAPn/AQABAAIA/f8EAPn///8GAPT/BgD5/wAAAgD8/wQA+v8BAPv/9/8AAP3/AAAAAPz/AQD9//r////6////AAD9/////f8AAPz/AAD9//z/CQD5/wUA+v/7/wIA9P8FAPn/AQADAP3//v8AAP3/+/8DAPr/AQABAAAA/f8BAAAA9/8EAP7///8GAPr///8BAPz/AQD9/wAABQD//wEAAAD9/wMAAQD7/wAAAwAAAAMAAAD+/wMAAQAAAP3/AAABAP//AgD+/wEAAQAHAPv///8EAPb/CQD//wAAAAAAAAUA9v8JAPf///8FAPr/AwD9/wIA+v8DAP//+P8GAP////8EAPv/+/8EAPv/AAD+//7/AQD//wEA+/8AAAAA/v8CAP///P8GAP////8DAPv/BgAAAAAAAgD//wEA//8CAP7/AgAAAP//BgAAAAEAAgAAAP//BwD6/wIABAD6/woA+/8EAAAAAAAEAPv/BgD//wIAAAABAAMAAAAHAPf/BwAAAPr/BwD4/wQAAAAAAAAAAAD//wAAAAD7/wcA/v8AAAAA///+/wIA///7/wkA+/8GAAAA/f8BAP//AgD9/wYA//8FAAQA//8CAPz/AQAFAPr/AwAAAP//BgD5//7//v8CAPv/AAD9//b/CAD4//z//v/6/wAAAAD+//z/AAABAP///v////3/BwABAAAAAAAAAAUA/f8DAPb/AAAIAPv/AwD9////BAAAAPf/+/8AAPv/AQD7//v/AAABAP7/+v8BAPn/AQACAPr/AQACAAEA/f8FAPv/AAAFAPj/AAABAPv//P/+//n//v////r//f/+//r/AAD0//b//f/0/wMA9//+//7//f8BAPP/BAD6////AwD8/wEAAAAKAPf/BwACAPz/DAD9//3/AwAGAPn/BwD+//3/DgD7/wAAAAD7/wYA/f/5/wQA+v8GAAgA+P8IAAMAAwAKAP//AQAFAAMABQAAAAUABQAGAAcA//8IAPr/AgAAAPX/BAD+//3/BAAAAPX/AwD5//T/AwD3//n/BgD6//z/BAD3/wAAAAD+/wAA//8AAAAA/v/+//7///8BAAAAAAABAP7//v/+//b/AAD8/wAAAAD+/wEA/f8CAPr/AQD7/wAABQD7/wcA//8FAAcAAgAIAAUABAAGAAcAAwAGAAQABAAKAAkABQAIAAUABwADAAMAAAACAAMAAgAFAP7/DAABAAgA/v/+/wYA/P8KAPj/BAAGAP7/CQAAAAEACAAEAAUA/P///wIA+/8DAP7///8FAP//AAD7/wAA/f/2/wIA9P/9/wMA8v///wAA+/8DAAEA/P8CAP7//v/5//r/CAAAAAkABgAAAAUABwD+//3/CwD//w4ABwAAAAkAAgAJAPz/BwALAAEAEgD//wAACgABAAcAAgAFAAcABwAGAAAABAAEAAoAAwAIAAsABQAMAAAAAQACAAkABwAGAA4ACwAKABMAAgAAAAoA/f8JAAAABAADAAsACQD9/wQA+/8AAPf//v/z//r/AQDv//3/8f/w//r/8f/0/+//9P/u/+j/8v/g/+7/8f/r//X/8f/x/+7/7v/m/+v/7v/3//D/8//4/+3//v/z//b/9//7//n/9f8BAPT//P8AAPj/AAADAAAAAAD+/wIA+f/8/wQA+v8DAAcA/v8AAA8AAAAFABEA//8NAA4ABwAHABAAEgAKABgAFAARABYAFAARABIAHAAXABIAIAAYABQAIgAUABEAIQAXABUAGgAVABMAFwAVAAgADQAJAAkABAAJAAkAAAAKAPz/+f/3//z/8P/2/wAA8/8CAP7/8//0//j/7v/t/Line truncated
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{
|
||||
"type": "audio_url",
|
||||
"audio_url": {
|
||||
"url": "data:audio/wav;base64,UklGRgrVAABXQVZFZm10IBAAAAABAAEAwF0AAIC7AAACABAAZGF0YebUAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA//8AAAAAAAAAAAAAAAAAAAAAAAAAAP////8AAAAAAAAAAAAAAAD//wAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAP////8AAAAAAAAAAAAA///p/9T/xv/G/8b/yf/K/8r/zP/M/83/zf/O/83/zf/P/8//z//O/87/z//P/8//z//P/8//z//P/8//z//P/8//z//Q/9D/0P/Q/9D/0P/R/9H/0f/R/9H/0f/R/9H/0f/R/9L/0v/S/9L/0v/S/9P/0//T/9P/0//T/9P/0//U/9T/1P/U/9T/1P/V/9T/1f/V/9X/1f/W/9b/1v/W/9b/1v/W/9b/1//X/9f/1//Y/9j/2P/Y/9j/2P/Z/9n/2f/Z/9n/2f/a/9r/2v/a/9v/2//b/9v/2//b/9v/3P/c/9z/3f/c/93/3f/d/93/3f/e/97/3v/e/9//3//f/9//4P/g/+D/4P/g/+D/4P/h/+H/4f/h/+H/4f/i/+L/4v/j/+P/4//j/+P/4//k/+P/5P/k/+T/5P/l/+X/5f/l/+X/5f/l/+X/5f/l/+X/5v/n/+f/5//o/+j/6P/o/+j/5v/n/+f/5//o/+j/6P/o/+n/6f/p/+j/6P/o/+j/6P/p/+n/6f/p/+n/6v/q/+r/6v/q/+r/6v/q/+r/6v/q/+r/6//r/+r/6v/r/+r/6v/p/+r/6v/r/+r/6v/r/+z/7P/s/+v/7P/r/+v/6//s/+v/6//r/+v/6//t/+z/7f/t/+3/7f/r/+z/6//s/+v/7P/s/+z/7P/s/+z/7P/t/+3/7f/s/+z/6//s/+r/6//q/+z/7P/t/+3/7v/u/+3/7f/t/+z/7P/s/+z/7P/t/+3/7f/t/+3/7f/t/+3/7f/t/+z/7P/s/+z/7f/t/+3/7f/u/+3/7f/t/+3/7f/t/+3/7f/u/+3/7f/t/+3/7f/t/+3/7f/t/+3/7P/r/+z/6//t/+z/7f/t/+7/7f/u/+7/7f/t/+z/7P/t/+z/7P/t/+3/7P/t/+7/7v/v/+7/7v/t/+z/7P/t/+3/7v/u/+7/7v/v/+//7v/u/+3/7P/u/+z/7P/s/+z/7f/u/+7/7v/v/+7/7v/s/+z/7P/t/+3/7f/u/+7/7f/u/+3/7v/t/+3/7f/t/+3/7P/t/+z/7f/u/+3/7//u/+7/7f/t/+3/7f/u/+//7v/v/+7/7v/t/+7/7f/v/+7/7v/t/+7/7f/t/+3/7v/u/+7/7v/v/+3/7v/t/+z/7P/r/+z/6//t/+r/7P/t/+7/7v/u/+7/7f/s/+z/6//s/+v/7P/t/+3/7//w//D/7v/w/+7/7v/s/+3/6//t/+v/7f/u/+//7//u/+//7v/u/+r/6//q/+v/6//s/+7/7v/w/+//7//v/+3/7f/s/+v/7P/s/+3/7f/u/+7/7v/u/+3/7P/r/+z/7f/s/+z/7P/t/+3/7P/t/+3/7v/s/+3/7f/t/+z/6//r/+z/7P/s/+z/7P/t/+z/7f/t/+z/7v/v/+7/7P/s/+z/6//u/+3/7v/u/+7/7f/u/+3/7f/t/+v/7f/r/+v/6//s/+3/7P/u/+7/7v/v/+7/7v/u/+z/7P/r/+3/6//t/+3/7v/t/+7/7v/v/+7/7v/t/+3/7P/t/+7/7v/v/+3/7v/t/+3/7f/s/+z/7f/s/+3/7f/t/+7/7v/u/+3/7//u/+7/7f/s/+z/7P/s/+v/7P/t/+3/7f/v/+7/7v/u/+3/7f/t/+3/7f/u/+7/7v/u/+7/7//u/+7/7v/t/+3/7f/t/+3/7f/u/+//7//v/+//7v/u/+7/7v/u/+7/7v/u/+7/7v/t/+7/7v/u/+7/7v/u/+7/7f/t/+3/7f/t/+7/7f/t/+3/7P/u/+3/7v/t/+7/7v/u/+7/7v/v/+7/7v/v/+7/7v/u/+7/7v/u/+3/7f/u/+3/7v/u/+7/7f/u/+3/7v/u/+7/7f/t/+7/7v/v/+7/7v/u/+3/7v/t/+3/7f/t/+3/7f/u/+7/7v/u/+3/7v/t/+3/7P/t/+3/7f/u/+3/7v/u/+//7v/u/+7/7v/u/+7/7v/u/+7/7f/u/+3/7f/u/+7/7f/u/+3/7f/t/+3/7f/t/+3/7f/s/+z/7f/t/+3/7f/t/+3/7P/t/+3/7f/t/+z/7P/s/+3/7P/t/+3/7f/u/+3/7f/t/+3/7f/t/+7/7P/t/+3/7f/t/+3/7f/t/+3/7f/t/+3/7f/t/+3/7f/t/+3/7f/s/+z/7f/t/+3/7f/t/+3/7f/t/+z/7P/t/+3/7f/t/+3/7f/t/+3/7f/t/+3/7f/t/+z/7P/s/+3/7f/t/+3/7f/t/+3/7P/s/+z/7P/t/+z/7P/s/+z/7P/s/+3/7P/t/+3/7f/t/+3/7f/t/+z/7P/t/+z/7P/s/+z/7P/s/+z/7P/t/+z/7P/t/+z/7P/s/+z/7P/s/+z/7P/t/+z/7P/s/+z/7P/s/+z/7P/s/+z/7P/s/+z/7P/s/+z/7P/s/+z/7P/s/+v/6//s/+z/7P/s/+z/7P/s/+z/7P/s/+z/7P/s/+z/7P/s/+z/7f/s/+z/7P/s/+z/7P/s/+v/6//r/+z/6//s/+z/7P/s/+z/7P/s/+z/7P/s/+z/6//s/+z/7P/s/+v/6//s/+z/7P/s/+z/7P/s/+z/7P/s/+z/7P/s/+z/7P/s/+z/7P/s/+z/7P/s/+z/7P/s/+z/7P/s/+z/7P/t/+z/7P/s/+z/7P/s/+z/7P/s/+z/7P/s/+3/7P/s/+z/7P/s/+z/7P/s/+z/7P/s/+z/7P/s/+z/7P/t/+z/7P/s/+z/7P/t/+z/7P/s/+z/7P/t/+z/7P/s/+z/7P/s/+z/7P/s/+z/7f/t/+3/7f/t/+3/7f/t/+3/7f/t/+3/7f/t/+3/7f/s/+3/7P/t/+3/7P/s/+z/7P/t/+z/7f/t/+3/7f/t/+3/7f/u/+z/7f/t/+3/7f/t/+3/7f/u/+3/7f/t/+z/7P/s/+z/7P/t/+z/7v/t/+7/7f/t/+7/7f/v/+z/7f/r/+3/7P/t/+3/7f/u/+7/7v/v/+7/7v/u/+z/7f/s/+7/7f/u/+3/7f/v/+7/7//u/+7/7//t/+//7v/u/+z/7P/s/+7/7f/w/+3/7f/u/+//7f/u/+7/7//s/+//6//w/+3/7//u/+3/7v/s/+7/7P/u/+//7v/x/+z/7//t/+3/7f/u/+//7//u/+7/6//w/+3/8f/t/+//7v/v/+7/7v/u/+//7v/w/+3/7v/t/+//7//t/+7/7v/w/+3/7P/v/+3/7//t/+//7v/w/+7/7//y/+3/7v/v/+3/7//t/+7/7P/v/+3/9P/w/+3/6//r//D/7//w/+3/8v/t/+7/7f/v/+3/7f/v//D/7f/t//D/7v/u/+3/8P/v/+7/7v/w/+z/7//r//L/7v/x/+3/7v/u/+7/7//t/+//7f/v/+7/7f/v/+7/7f/u/+7/7f/v/+//8P/u/+3/7//u/+//7f/v/+z/7f/v/+3/7v/p//D/7f/x/+7/7v/s/+3/8P/v/+//7f/r/+7/7P/t/+3/7v/t/+3/7f/u/+7/7//t/+7/7f/u/+3/6//s/+z/7f/t/+3/7//s/+//7f/y/+z/7v/s/+3/7P/v/+3/6//r/+3/7P/u/+//7f/t/+7/7v/v/+z/7f/r/+//7f/w/+z/7f/v/+v/7f/s/+7/7v/v/+z/7P/u/+z/7//s/+3/7f/u/+7/7f/w/+z/7v/t/+3/7v/t/+3/7P/u/+7/7v/s/+z/7f/v/+3/7v/t/+3/7f/t/+7/7P/t/+3/7v/v/+z/7f/u/+3/7v/v/+z/7v/t/+//7f/u/Line truncated
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": [
|
||||
{
|
||||
"type": "text",
|
||||
"text": "我没办法获取实时的天气信息。不过呢,你可以试试几个方法来查看今天的天气。首先,你可以用手机自带的天气功能,比如苹果手机的天气应用,或者直接在系统设置里查看。其次,你也可以用一些专业的天气服务,像是国外的AccuWeather、Weather.com,或者国内的中国天气网、墨迹天气等等。再有就是,你还可以在谷歌或者百度里直接搜索你所在的城市加上天气这两个字。如果你能告诉我你所在的城市,我也可以帮你分析一下历史天气趋势,不过最新的数据还是需要你通过官方渠道去获取哦。"
|
||||
},
|
||||
{
|
||||
"type": "audio_url",
|
||||
"audio_url": {
|
||||
"url": "data:audio/wav;base64,UklGRiQiKQBXQVZFZm10IBAAAAABAAEAwF0AAIC7AAACABAAZGF0YQAiKQAGAAcABwAIAAkACQAJAAoACwAMAAwADQAOAA4ADwAPABAAEAAQABAAEQAQABAAEQARABAAEQARABEAEAAQABEAEQAQABAAEAAPAA4ADwAOAA0ADQANAAwACwAJAAYABQAEAAIAAQAAAP7//f/7//r/+v/5//b/9v/0//X/9f/0//X/9P/0//L/9P/y//D/9P/z//T/8v/x//L/8v/x//H/8P/v/+//7v/u/+3/7P/t/+3/7P/t/+7/7v/u//D/8f/z//X/9f/3//n/+//7//3///8AAAEABAAGAAgACgAMAA4AEAASABMAFAATABQAFAAUABUAFQATABIAEgARABEAEAAOAAwACwAJAAcABgAEAAMAAgABAAEAAAAAAAAA//////7//f/8//v/+f/4//j/9//3//f/9v/2//b/9f/1//b/9f/1//b/9v/2//b/9f/1//X/9P/z//P/9P/z//P/8//0//T/9P/0//X/9v/2//f/+P/4//n/+v/7//z//f/+//7///8AAAEAAgACAAMABAAFAAYABwAJAAoADAANAA4ADwAPAA8AEAAQABEAEgASABMAFAAVABYAFwAXABYAFgAWABYAFgAVABQAEwATABMAEgASABIAEQARABAADwAOAA4ADQANAAsACgAIAAcABAACAAEA///9//v/+f/4//b/9f/z//H/8f/v/+3/7P/r/+n/5//m/+X/5f/k/+T/5P/l/+X/5//o/+r/7P/u//D/8v/1//f/+f/7//z//v8AAAEAAgAEAAQABQAHAAUABQAGAAQAAwACAAEAAgAAAAEAAQD+/wAA/v8AAP7//P////7//f/9/wAA/v/9//7//v///wAA//////////8AAAEAAwACAAIAAwADAAMAAgACAAIAAwACAAEAAAABAAEAAAD////////////////+/////v/9//3//P/8//z//f/9//3//v/+//7///8AAP////////////////////////7//v/9//3//f/8//r/+P/3//j/9//3//b/9f/2//b/9f/1//b/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//4//j/+P/5//n/+f/5//n/+f/5//n/+f/6//r/+//7//z//P/+////AAACAAMABQAGAAcACAAIAAkACQAJAAkACQAKAAoACQAJAAcABgAEAAMAAQD///3/+//6//n/+P/3//b/9v/1//b/9v/2//b/9v/2//b/9//4//n/+v/8//7//v///wEAAwAEAAUABwAJAAoACgALAA4ADAANAA0ADwAOAA0ADwAPAA8ADgARABEAEQARABEAEAAQABEAEQARABAAEAAQABEAEQAQABAADwAPAA4ADQAOAAwADQANAAwADQANAAsADAALAAsACgAJAAkACAAHAAcABQAFAAUAAgABAAEAAAD///7//P/6//n/9v/z//P/8v/w/+//7f/r/+v/6v/p/+r/6f/o/+j/6P/o/+j/6f/p/+n/6v/r/+z/7P/t/+7/7//w//H/8//1//b/+P/5//r//P/9////AAABAAIAAwAFAAYABwAIAAkACwALAAwADQAOAA4ADwAQABEAEgASABMAFAAVABUAFQAVABYAFQAVABUAFAATABIAEQAPAA4ADQAMAAsACgAJAAcABQADAAAA///8//r/+P/2//X/8//x/+//7v/t/+z/6//r/+r/6v/q/+v/6//s/+z/7f/t/+7/7//w//H/8v/z//T/9f/2//f/9//4//j/+f/6//r/+//7//z//P/9//7//v///wAAAAAAAAEAAQACAAMABAAGAAcACQAKAAwADQAPABAAEQASABIAEgATABUAFwAYABkAGgAbAB0AHgAfAB4AHgAdABwAGwAaABkAGAAWABQAEgAQAA0ACwAJAAcABQADAAEA/v/8//r/+P/2//T/8//x//D/7//t/+z/6//r/+v/6//r/+v/6//s/+3/7f/t/+7/7v/v/+//8P/w//D/8f/w//D/7//w//H/8f/w//D/8f/y//L/8v/y//P/8//1//f/+P/5//z//f/8/wAA//8DAAEAAAAEAAIAAQADAAMAAQAAAAEAAAABAAEA/////////v/9//7////9//z//P/9//3//P/8//z//f/+//7//v////////8AAAAAAQACAAIAAgACAAIAAgABAAEAAQABAAIAAQAAAAAAAAD///7//v/9//3/+//6//r/+v/6//r/+f/6//r/+v/5//n/+v/7//v/+v/6//v//f/+//7//v//////AAD///////////7//f/8//v/+//6//r/+f/5//n/+v/6//r/+v/6//z//P/9//3///8AAAEAAgADAAUABQAGAAYABwAIAAkACQAKAAsADAAMAAwADAAMAA0ADAAMAAsACwAKAAkACAAHAAUAAwACAAEAAAD+//3/+//6//j/9//1//T/8//y//L/8//z//T/9f/1//b/9v/2//f/+P/4//n/+f/5//r//P/9////AAACAAMABAAFAAYACAAJAAoACwANAA4ADQAOABEAEAARABEAFAATABMAFQAVABYAFQAXABYAFgAWABUAFAATABQAEwASABIAEQAQABAADwAPAA4ADQAMAAsACwAMAAoACwAKAAkACgAJAAgABwAFAAUAAwACAAEA///9//3/+v/5//j/9P/y//D/7v/s/+v/6P/m/+X/4v/h/+H/4f/g/+H/4P/g/+H/4v/j/+T/5f/m/+f/6P/q/+v/7f/u/+//8f/y//T/9f/3//j/+v/7//z//v///wEAAgADAAQABQAGAAgACQAKAAsADAAOAA8AEAASABMAFAAVABYAFgAXABgAGAAZABkAGgAaABoAGQAZABgAFgAVABMAEgAQAA8ADgAMAAoACQAHAAUABAACAAAA///9//v/+f/3//X/8//x//D/7//v/+7/7v/t/+3/7f/t/+7/7v/v//D/8P/x//L/8//0//X/9f/2//f/9//4//j/+f/5//r/+v/6//r/+v/6//r/+f/6//r/+v/7//v//P/8//3//v///wEAAgADAAUABgAIAAsADQAPABEAEwAUABYAFwAYABgAGQAZABkAGAAZABkAGQAZABgAFwAVABQAEgARAA4ACwAIAAUAAwABAP///v/8//v/+v/4//b/9f/0//P/8v/y//H/8P/w/+//7v/u/+3/7f/t/+3/7f/s/+z/7P/t/+3/7v/u/+7/7v/u/+//8P/v//D/8P/x//H/8f/y//P/9P/0//X/9f/1//f/+P/5//n/+//8//z//f/9//7//f///wAAAAAAAAEAAQD//wAA/f8AAPz/+//+//v/+v/8//z/+//7//z//f/+/////v///wEAAQAAAAIABAADAAIAAwADAAQAAwADAAMAAwADAAIAAgACAAEAAAAAAAAAAAD//////v/+//7//v/9//7//v/+//////////////////////8AAAAA////////AAAAAAAAAAAAAAEAAAD///7//////////v/9//3//////////v/+//7//v/+//7//v////////8AAAAAAQACAAQABQAGAAcACAAKAAoACgALAAwADAAMAAwADAAMAAwADAAMAAwACwAKAAkACQAJAAgACAAIAAgABwAHAAYABQAEAAMAAgAAAP///v/9//z/+v/5//f/9f/0//P/8v/x//D/7//u/+3/7f/s/+z/7P/s/+7/7//w//L/8//0//X/9v/3//j/+v/8//3///8BAAMABgAIAAoADAAOABAAEQARABIAEwAUABQAFQAVABYAFAAUABYAFAAUABQAFgAVABMAFQAVABUAFAAVABUAFQAVABQAFAATABQAFAAUABQAFAATABQAFAAUABMAEgASABAAEAAQAA4ADgAMAAkACAAGAAIAAAD8//r/9//0//L/7//t/+v/6f/n/+b/4//h/+D/3v/e/9//3v/e/97/3v/e/+H/4v/j/+b/5v/n/+n/6v/s/+7/7//w//L/8//0//b/9//4//n/+v/7//z//f/+////AAABAAIABAAFAAcACAAJAAoACwANAA4ADwARABIAEwAUABUAFgAXABcAFwAXABcAFgAWABUAFAATABIAEAAPAA0ACwAJAAYABAABAP///f/7//n/9//1//T/8v/w/+//7f/s/+r/6f/o/+f/5v/l/+T/5P/j/+T/5P/l/+X/5v/n/+j/6P/p/+r/6//s/+z/7f/u/+7/7v/v/+//7//v/+//8P/w//H/8f/y//P/9P/1//f/+P/5//v//P/+/wEAAwAFAAcACgALAA4AEAATABUAFwAZABsAHQAfACEAIwAkACUAJgAmACUAJQAkACMAIQAgAB0AGwAaABkAGAAWABQAEgAPAA4ADAALAAgABgAEAAIAAQAAAAAA///+//7//f/8//v/+f/5//j/+P/4//f/9//2//X/9f/0//T/9P/0//X/9f/1//X/9f/2//f/9//4//j/+f/6//v//P/8//3//v///wAAAAABAAIAAwAEAAUABQAGAAcACAAIAAcACAAJAAgACAAIAAgABwAIAAkACQAIAAoACgAIAAkABwAKAAcABgAKAAgABwAKAAoACQAJAAoACwALAAwACgAKAAsACQAIAAgACAAGAAQAAwACAAEA///+//3//P/7//r/+f/4//j/9//3//f/9//3//f/9//3//j/+P/4//j/+P/5//n/+f/5//n/+f/5//j/+f/5//n/+f/4//n/+f/6//r/+f/6//v/+//6//r/+//8//3//P/8//7/AAABAAIAAgADAAQABQAGAAcACAAJALine truncated
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{
|
||||
"type": "audio_url",
|
||||
"audio_url": {
|
||||
"url": "data:audio/wav;base64,UklGRkqYAABXQVZFZm10IBAAAAABAAEAwF0AAIC7AAACABAAZGF0YSaYAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABAAAAAAAAAAAAAAAAAAAAAAAAAP//AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD//wAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAP//AAAAAP//AAAAAAAA///o/9T/xv/G/8b/yP/K/8r/y//M/8z/zf/O/83/zf/P/87/zv/O/87/zv/P/8//z//P/8//z//P/8//z//P/8//z//P/8//0P/Q/8//z//Q/9D/0f/R/9D/0f/R/9H/0f/R/9H/0f/R/9H/0f/S/9L/0v/S/9L/0v/S/9L/0v/T/9P/0//T/9P/0//T/9P/0//T/9P/0//T/9P/0//U/9T/1f/V/9T/1P/U/9X/1P/V/9X/1f/V/9b/1v/W/9b/1v/W/9b/1v/W/9f/1//X/9f/1//X/9f/1//X/9j/2P/Y/9j/2P/Y/9j/2P/Y/9j/2f/Z/9r/2v/a/9r/2v/b/9v/2//b/9v/2//c/9v/3P/c/9z/3P/d/9z/3P/c/93/3f/e/93/3f/e/97/3v/e/97/3//f/+D/4P/g/9//3//g/+D/4P/g/+H/4f/g/+H/4P/h/+H/4f/h/+L/4v/i/+H/4f/i/+L/4v/i/+L/4v/j/+L/4//j/+P/4//j/+P/4//j/+P/5P/k/+T/4//l/+T/5P/k/+T/5P/l/+T/5f/l/+X/5f/m/+X/5v/m/+b/5v/m/+b/5v/m/+b/5v/n/+b/5//n/+b/5v/o/+f/5//n/+j/5//p/+f/5//o/+j/6P/n/+j/6P/p/+n/6f/o/+j/6P/p/+j/6P/o/+n/6P/p/+j/6P/p/+n/6f/q/+n/6v/p/+n/6f/q/+n/6f/p/+n/6f/p/+n/6f/p/+n/6v/p/+n/6f/p/+n/6f/p/+n/6f/r/+r/6v/q/+r/6v/r/+n/6v/p/+r/6v/r/+n/6v/q/+r/6v/q/+r/6v/q/+r/6v/q/+r/6//q/+v/6//r/+r/6//r/+r/6v/q/+r/6v/q/+r/6v/q/+r/6//q/+v/6//s/+r/6//q/+r/6v/s/+n/6v/q/+v/6v/r/+v/6//s/+v/6//s/+v/7P/r/+r/6//s/+z/6//r/+v/7P/r/+z/6//r/+v/6//r/+r/6//r/+v/6//r/+v/6//s/+v/6//r/+z/7P/r/+v/6//s/+v/7P/r/+v/6//r/+v/6//s/+v/6//r/+z/7P/s/+z/7P/r/+z/6//s/+v/6//r/+v/6//s/+v/7P/r/+z/7P/s/+z/6//t/+v/7P/s/+v/7P/r/+z/6//s/+z/7P/r/+z/6//t/+v/7P/s/+v/7P/s/+z/6//s/+z/7P/s/+z/7P/s/+v/6//s/+z/7P/r/+z/7P/r/+z/7P/t/+z/7P/s/+z/7P/s/+z/7P/s/+z/7P/s/+z/7P/s/+z/7P/t/+z/7P/s/+z/7P/s/+z/7P/s/+v/6//s/+z/7P/s/+z/6//r/+z/7P/s/+v/7P/s/+v/7P/s/+v/6//s/+v/7P/s/+z/6//s/+v/7P/q/+v/6//s/+v/6//r/+z/6//r/+z/7P/r/+z/6//s/+z/7P/r/+z/7P/s/+v/6//t/+r/7P/r/+v/6//s/+v/7P/s/+v/7P/r/+z/6//s/+v/6//r/+z/6//r/+v/6//r/+v/6//t/+v/7P/r/+z/6//r/+z/7P/s/+v/6//r/+v/6//s/+v/7P/r/+z/7P/r/+v/6//s/+v/6//r/+z/6//s/+v/6//s/+v/7P/s/+z/7P/r/+v/6//s/+v/7P/r/+z/6//r/+v/6//s/+v/6//r/+z/7P/r/+z/6//s/+v/6//r/+v/6//s/+v/6//s/+r/6//r/+v/6//r/+v/6//r/+v/7P/r/+v/6//s/+v/6//r/+z/6//s/+v/6//q/+v/6//s/+v/6//s/+v/6//s/+v/6//r/+v/6//s/+v/6//r/+z/6//s/+v/6//s/+v/7P/q/+z/6//s/+v/6//s/+v/6//r/+v/6//r/+v/6//r/+v/6//s/+v/6//r/+v/6//s/+v/7P/r/+v/6//r/+v/6//r/+v/6//r/+v/6//s/+v/6//r/+v/6//r/+v/6//s/+r/6//r/+v/6//r/+r/6//r/+v/6//s/+v/6//q/+v/6v/s/+r/6//r/+v/6//r/+v/6//s/+v/6//r/+v/6//r/+v/6//r/+v/6//s/+v/6//r/+r/6//q/+v/6//r/+v/6//r/+v/6//r/+r/6//r/+v/6//r/+v/6//r/+v/6//r/+v/6//r/+v/6//r/+v/6//r/+v/6//r/+v/6//r/+v/6//r/+v/6//r/+v/6//r/+v/6//r/+v/6//r/+v/6//r/+v/6//r/+v/6//r/+v/6//r/+v/6//r/+v/6//r/+v/6//r/+v/6//r/+v/6//r/+v/6//s/+v/6//r/+v/6//r/+v/6//s/+v/6//r/+v/6//r/+v/6//r/+v/6//r/+v/6//r/+v/6//r/+v/6//r/+v/6//r/+v/6//s/+v/6//r/+v/6//r/+v/6//s/+v/6//r/+v/6//s/+v/6//r/+z/6//r/+v/6//r/+v/6//r/+v/6//r/+v/6//s/+v/6//r/+v/6//r/+v/6//r/+v/6//r/+v/6//r/+v/6//r/+v/6//r/+v/6//s/+v/6//r/+v/6//s/+v/6//r/+v/6//r/+v/6//r/+v/6//r/+v/6//r/+v/6//r/+v/6//r/+v/6//s/+v/6//s/+v/6//r/+v/6//s/+v/6//r/+v/6//s/+v/6//s/+v/6//r/+z/6//r/+v/7P/r/+v/6//s/+v/6//q/+v/6//s/+r/6//r/+v/6//s/+v/6//r/+v/6//r/+v/6//r/+v/6//r/+v/6//s/+v/6//r/+v/6//s/+v/6//s/+v/7P/r/+z/6//s/+v/6//r/+v/6//r/+v/6//r/+v/6//s/+v/6//r/+v/6//r/+v/6//r/+v/6//r/+z/6//r/+v/6//s/+v/6//r/+v/6//r/+v/6//s/+v/6//s/+v/7P/r/+v/6//r/+v/6//s/+v/6//r/+v/6//r/+v/6v/s/+v/6//r/+v/6//r/+v/6//s/+v/6//r/+v/6//s/+v/6//s/+z/7P/r/+v/7P/s/+v/6//s/+v/6//r/+v/6//r/+z/7P/s/+v/6//r/+z/6//s/+v/7P/r/+z/6//s/+v/6//s/+v/6//r/+z/6//s/+v/6//s/+v/7P/r/+v/6//s/+v/7P/r/+v/6//s/+v/7P/s/+v/7P/r/+z/7P/r/+z/6//s/+v/6//r/+z/6//s/+z/7P/s/+v/6//r/+v/6//r/+z/6//s/+v/6//r/+v/6//r/+v/6//s/+v/6//r/+v/6//s/+z/6//s/+v/6//s/+v/7P/s/+z/6//s/+v/6//s/+v/7P/s/+z/7P/s/+z/6//r/+v/6//s/+v/7P/s/+z/6//s/+v/6//r/+z/6//s/+v/6//r/+z/6//r/+v/6//s/+v/6//s/+v/6//r/+v/6//s/+v/7P/r/+z/7P/r/+v/7P/s/+z/7P/s/+v/7P/s/+z/6//s/+v/7P/s/+v/7P/s/+z/7P/s/+v/6//r/+v/6//s/+v/7P/r/+z/6//r/+z/6//s/+v/7P/r/+v/7P/r/+z/6//s/+v/6//s/+v/7P/r/+z/6//s/+z/7P/s/+v/7P/r/+z/7P/s/+v/6//r/+z/7P/s/+z/6//s/+v/6//s/+z/7P/s/+z/7P/s/+z/7P/s/+z/7P/s/+z/7P/s/+z/6//t/+v/7P/s/+z/7P/s/Line truncated
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,714 @@
|
||||
import base64
|
||||
import io
|
||||
import os
|
||||
import random
|
||||
import re
|
||||
from collections.abc import Callable
|
||||
|
||||
import librosa
|
||||
import numpy as np
|
||||
import torch
|
||||
import torchaudio
|
||||
from process_speechdata import InputSegment, StreamingInputSegment
|
||||
from torchaudio.transforms import MelSpectrogram
|
||||
|
||||
speech_zeroemb_idx = 151667
|
||||
empty_token = "<|empty|>"
|
||||
mimo_audio_tokenizer = None
|
||||
device = "cpu"
|
||||
|
||||
asr_zh_templates = [
|
||||
"请将这段语音转换为文字",
|
||||
"帮我识别这个音频文件中的内容",
|
||||
"把这段录音转成文本",
|
||||
"请转录这段语音",
|
||||
"将音频内容转换成文字格式",
|
||||
"识别并转写这段语音",
|
||||
"把语音内容写成文字",
|
||||
"转录这个音频片段",
|
||||
"将这段对话转换为文本",
|
||||
"麻烦帮我把这段录音整理成详细的文字记录",
|
||||
]
|
||||
|
||||
asr_en_templates = [
|
||||
"Please transcribe this audio file",
|
||||
"Convert this speech recording to text",
|
||||
"Transcribe the following voice message",
|
||||
"Turn this audio into readable text",
|
||||
"Please convert the recording to written format",
|
||||
"Transcribe what you hear in this audio",
|
||||
"Convert this spoken content to text",
|
||||
"Please write down what is said in this recording",
|
||||
"Transcribe this voice recording",
|
||||
"Could you please help me transcribe this important recording?",
|
||||
"Would you mind converting this voice message into a readable text format?",
|
||||
"I'd really appreciate it if you could turn this audio file into a written document",
|
||||
]
|
||||
|
||||
tts_zh_templates = [
|
||||
"请将这段文字转换为语音",
|
||||
"帮我把这个文本读出来",
|
||||
"将这些文字生成音频",
|
||||
"请朗读这段内容",
|
||||
"把这段话转换成语音文件",
|
||||
"生成这段文字的语音版本",
|
||||
"请用语音播报这些内容",
|
||||
"将文本转换为可听的音频",
|
||||
"帮我朗读这段文字",
|
||||
"把这些内容念出来",
|
||||
]
|
||||
|
||||
tts_en_templates = [
|
||||
"Please convert this text to speech",
|
||||
"Turn this writing into audio",
|
||||
"Generate speech from this text",
|
||||
"Read this content out loud",
|
||||
"Convert these words to voice",
|
||||
"Create an audio version of this text",
|
||||
"Please vocalize this content",
|
||||
"Turn this text into audible format",
|
||||
"Help me convert this writing to speech",
|
||||
"Make this text into spoken audio",
|
||||
]
|
||||
|
||||
|
||||
def detect_language(text):
|
||||
if re.search(r"[\u4e00-\u9fff]", text):
|
||||
return "zh"
|
||||
else:
|
||||
return "en"
|
||||
|
||||
|
||||
# ============================================
|
||||
# Common Helper Functions - InputSegment Creation
|
||||
# ============================================
|
||||
|
||||
|
||||
def create_segment(text: str = "", audio=None) -> InputSegment:
|
||||
"""Create a standard InputSegment with default zeroemb parameters"""
|
||||
return InputSegment(
|
||||
text=text,
|
||||
audio=audio,
|
||||
speech_zeroemb_idx=speech_zeroemb_idx,
|
||||
text_zeroemb_idx=empty_token,
|
||||
)
|
||||
|
||||
|
||||
def create_streaming_segment(
|
||||
text: str, audio, tokenizer, group_size: int, audio_channels: int
|
||||
) -> StreamingInputSegment:
|
||||
"""Create a StreamingInputSegment"""
|
||||
return StreamingInputSegment(
|
||||
text=text,
|
||||
audio=audio,
|
||||
tokenizer=tokenizer,
|
||||
group_size=group_size,
|
||||
audio_channels=audio_channels,
|
||||
speech_zeroemb_idx=speech_zeroemb_idx,
|
||||
text_zeroemb_idx=empty_token,
|
||||
)
|
||||
|
||||
|
||||
# ============================================
|
||||
# Common Helper Functions - Common Token Segment Creation
|
||||
# ============================================
|
||||
|
||||
|
||||
def create_user_start() -> InputSegment:
|
||||
"""Create user role start token"""
|
||||
return create_segment(text="<|im_start|>user\n")
|
||||
|
||||
|
||||
def create_user_end() -> InputSegment:
|
||||
"""Create role end token"""
|
||||
return create_segment(text="<|im_end|>\n")
|
||||
|
||||
|
||||
def create_assistant_start() -> InputSegment:
|
||||
"""Create assistant role start token"""
|
||||
return create_segment(text="<|im_start|>assistant\n")
|
||||
|
||||
|
||||
def create_system_start() -> InputSegment:
|
||||
"""Create system role start token"""
|
||||
return create_segment(text="<|im_start|>system\n")
|
||||
|
||||
|
||||
def create_thinking_segment(thinking: bool = False) -> InputSegment:
|
||||
"""Create thinking token, closed or open based on thinking parameter"""
|
||||
if thinking:
|
||||
return create_segment(text="<think>\n")
|
||||
else:
|
||||
return create_segment(text="<think>\n\n</think>\n")
|
||||
|
||||
|
||||
def create_sostm_segment() -> InputSegment:
|
||||
"""Create streaming output start token"""
|
||||
return create_segment(text="<|sostm|>")
|
||||
|
||||
|
||||
def create_assistant_start_with_sostm() -> InputSegment:
|
||||
"""Create assistant start token with sostm"""
|
||||
return create_segment(text="<|im_start|>assistant\n<|sostm|>")
|
||||
|
||||
|
||||
def create_assistant_start_with_think() -> InputSegment:
|
||||
"""Create assistant start token with think opening"""
|
||||
return create_segment(text="<|im_start|>assistant\n<think>\n")
|
||||
|
||||
|
||||
# ============================================
|
||||
# Common Helper Functions - Composite Segment Creation
|
||||
# ============================================
|
||||
|
||||
|
||||
def create_user_turn_with_audio(audio_tokenized, extra_text: str = None) -> list[InputSegment]:
|
||||
"""Create a user turn containing audio"""
|
||||
segments = [
|
||||
create_user_start(),
|
||||
create_segment(audio=audio_tokenized),
|
||||
]
|
||||
if extra_text:
|
||||
segments.append(create_segment(text=extra_text))
|
||||
segments.append(create_user_end())
|
||||
return segments
|
||||
|
||||
|
||||
def create_user_turn_with_text(text: str) -> list[InputSegment]:
|
||||
"""Create a text-only user turn"""
|
||||
return [
|
||||
create_user_start(),
|
||||
create_segment(text=text),
|
||||
create_user_end(),
|
||||
]
|
||||
|
||||
|
||||
def create_system_turn_with_voice_prompt(prompt_text: str, audio_token) -> list[InputSegment]:
|
||||
"""Create a system turn with voice prompt"""
|
||||
return [
|
||||
create_system_start(),
|
||||
create_segment(text=prompt_text),
|
||||
create_segment(text="", audio=audio_token),
|
||||
create_user_end(),
|
||||
]
|
||||
|
||||
|
||||
def create_system_turn_text_only(system_text: str) -> list[InputSegment]:
|
||||
"""Create a text-only system turn"""
|
||||
return [
|
||||
create_system_start(),
|
||||
create_segment(text=system_text),
|
||||
create_user_end(),
|
||||
]
|
||||
|
||||
|
||||
# ============================================
|
||||
# Common Helper Functions - Multi-turn Dialogue Processing
|
||||
# ============================================
|
||||
|
||||
|
||||
def process_multiturn_messages(
|
||||
message_list: list[dict],
|
||||
user_processor: Callable[[dict], list[InputSegment]],
|
||||
assistant_processor: Callable[[dict], list[InputSegment]],
|
||||
) -> list[InputSegment]:
|
||||
"""
|
||||
Generic multi-turn dialogue message processing function
|
||||
|
||||
Args:
|
||||
message_list: List of messages, each containing 'role' and 'content'
|
||||
user_processor: Function to process user messages
|
||||
assistant_processor: Function to process assistant messages
|
||||
|
||||
Returns:
|
||||
Processed list of InputSegments
|
||||
"""
|
||||
lm_prompt = []
|
||||
for message in message_list:
|
||||
role = message["role"]
|
||||
if role == "user":
|
||||
lm_prompt.extend(user_processor(message))
|
||||
elif role == "assistant":
|
||||
lm_prompt.extend(assistant_processor(message))
|
||||
else:
|
||||
raise ValueError(f"Invalid role: {role}")
|
||||
return lm_prompt
|
||||
|
||||
|
||||
def create_text_user_message(message: dict) -> list[InputSegment]:
|
||||
"""Process a text-only user message"""
|
||||
return [
|
||||
create_user_start(),
|
||||
create_segment(text=message["content"]),
|
||||
create_user_end(),
|
||||
]
|
||||
|
||||
|
||||
def create_text_assistant_message(message: dict) -> list[InputSegment]:
|
||||
"""Process a text-only assistant message"""
|
||||
return [
|
||||
create_assistant_start(),
|
||||
create_segment(text=message["content"]),
|
||||
create_user_end(),
|
||||
]
|
||||
|
||||
|
||||
def create_audio_user_message(message: dict) -> list[InputSegment]:
|
||||
"""Process an audio user message"""
|
||||
return [
|
||||
create_user_start(),
|
||||
create_segment(audio=preprocess_input(message["content"])),
|
||||
create_user_end(),
|
||||
]
|
||||
|
||||
|
||||
def append_assistant_ending(
|
||||
lm_prompt: list[InputSegment], thinking: bool = False, use_sostm: bool = False
|
||||
) -> list[InputSegment]:
|
||||
"""
|
||||
Append assistant ending to the prompt
|
||||
|
||||
Args:
|
||||
lm_prompt: Existing prompt list
|
||||
thinking: Whether to use open thinking token
|
||||
use_sostm: Whether to use sostm token (for speech output)
|
||||
"""
|
||||
if use_sostm:
|
||||
lm_prompt.append(create_assistant_start_with_sostm())
|
||||
else:
|
||||
lm_prompt.append(create_assistant_start())
|
||||
lm_prompt.append(create_thinking_segment(thinking))
|
||||
return lm_prompt
|
||||
|
||||
|
||||
def get_asr_sft_prompt(
|
||||
input: None | str = None,
|
||||
):
|
||||
"""Build prompt for ASR (Automatic Speech Recognition) task"""
|
||||
audio_tokenized = preprocess_input(input)
|
||||
template = random.choice(asr_zh_templates + asr_en_templates)
|
||||
|
||||
lm_prompt = create_user_turn_with_audio(audio_tokenized, extra_text=template)
|
||||
lm_prompt = append_assistant_ending(lm_prompt, thinking=False)
|
||||
return lm_prompt
|
||||
|
||||
|
||||
def resample_audio_if_needed(wav_tensor: torch.Tensor, original_sr: int):
|
||||
target_sr = 24000
|
||||
if original_sr != target_sr:
|
||||
wav_tensor = torchaudio.functional.resample(wav_tensor, original_sr, target_sr)
|
||||
return wav_tensor
|
||||
|
||||
|
||||
def wav2mel(wav, device="cpu"):
|
||||
mel_transform = MelSpectrogram(
|
||||
sample_rate=mimo_audio_tokenizer.config.sampling_rate,
|
||||
n_fft=mimo_audio_tokenizer.config.nfft,
|
||||
hop_length=mimo_audio_tokenizer.config.hop_length,
|
||||
win_length=mimo_audio_tokenizer.config.window_size,
|
||||
f_min=mimo_audio_tokenizer.config.fmin,
|
||||
f_max=mimo_audio_tokenizer.config.fmax,
|
||||
n_mels=mimo_audio_tokenizer.config.n_mels,
|
||||
power=1.0,
|
||||
center=True,
|
||||
).to(device)
|
||||
spec = mel_transform(wav[None, :])
|
||||
return torch.log(torch.clip(spec, min=1e-7)).squeeze()
|
||||
|
||||
|
||||
def group_by_length(features: torch.Tensor, lengths: torch.Tensor, max_length: int):
|
||||
if features.size(0) != lengths.sum().item():
|
||||
raise ValueError(f"Feature size mismatch: {features.size(0)} vs {lengths.sum().item()}")
|
||||
|
||||
split_points = []
|
||||
current_sum = 0
|
||||
|
||||
for i, seq_len in enumerate(lengths):
|
||||
if current_sum + seq_len > max_length and current_sum > 0:
|
||||
split_points.append(i)
|
||||
current_sum = seq_len.item()
|
||||
else:
|
||||
current_sum += seq_len.item()
|
||||
|
||||
# Convert split points to group sizes
|
||||
group_sizes = []
|
||||
prev = 0
|
||||
for point in split_points:
|
||||
group_sizes.append(point - prev)
|
||||
prev = point
|
||||
if prev < len(lengths):
|
||||
group_sizes.append(len(lengths) - prev)
|
||||
|
||||
len_groups = torch.split(lengths, group_sizes)
|
||||
feature_sizes = [group.sum().item() for group in len_groups]
|
||||
feature_groups = torch.split(features, feature_sizes)
|
||||
|
||||
return feature_groups, len_groups
|
||||
|
||||
|
||||
def encode_batch(input_features: torch.Tensor, input_lens: torch.Tensor, max_length: int = 256000):
|
||||
feature_groups, len_groups = group_by_length(input_features, input_lens, max_length)
|
||||
|
||||
encoded_parts = []
|
||||
for features, lengths in zip(feature_groups, len_groups):
|
||||
with torch.no_grad():
|
||||
codes, _ = mimo_audio_tokenizer.encoder.encode(
|
||||
input_features=features.to(device), input_lens=lengths.to(device), return_codes_only=True
|
||||
)
|
||||
encoded_parts.append(codes)
|
||||
|
||||
return torch.cat(encoded_parts, dim=-1)
|
||||
|
||||
|
||||
def preprocess_input(input: None | str | torch.Tensor = None, device="cpu", audio_channels=4, group_size=8):
|
||||
if isinstance(input, torch.Tensor) or (isinstance(input, str) and os.path.isfile(input)):
|
||||
return "<|sosp|><|empty|><|eosp|>"
|
||||
|
||||
else:
|
||||
text = input
|
||||
if (
|
||||
text.isupper() or text.islower()
|
||||
): # If the text only contains upper-case or lower-case letters, capitalize it.
|
||||
text = text.capitalize()
|
||||
return text
|
||||
|
||||
|
||||
def _build_tts_system_prompt(has_voice_prompt: bool, voice_audio_token=None) -> list[InputSegment]:
|
||||
"""Build system prompt for TTS task"""
|
||||
if has_voice_prompt and voice_audio_token is not None:
|
||||
return [
|
||||
create_system_start(),
|
||||
create_segment(
|
||||
# text="You need to generate a speech with the same timbre as the speech prompt, based on the specified style instructions and text content. Your timbre should be: "
|
||||
text="你需要根据指定的风格指令和文本内容来生成和语音prompt具有相同音色的语音。你的音色应该是:"
|
||||
),
|
||||
create_segment(text="", audio=voice_audio_token),
|
||||
create_user_end(),
|
||||
]
|
||||
else:
|
||||
return create_system_turn_text_only("你需要根据指定的风格指令和文本内容来生成语音。")
|
||||
# return create_system_turn_text_only(
|
||||
# "You need to generate speech based on the specified style instructions and text content."
|
||||
# )
|
||||
|
||||
|
||||
def _build_tts_system_prompt_no_instruct(has_voice_prompt: bool, voice_audio_token=None) -> list[InputSegment]:
|
||||
"""Build system prompt for TTS task"""
|
||||
if has_voice_prompt and voice_audio_token is not None:
|
||||
return [
|
||||
create_system_start(),
|
||||
create_segment(
|
||||
# text="You need to generate a speech with the same timbre as the speech prompt, based on the specified style instructions and text content. Your timbre should be:"
|
||||
text="你需要根据指定的风格指令和文本内容来生成和语音prompt具有相同音色的语音。你的音色应该是:"
|
||||
),
|
||||
create_segment(text="", audio=voice_audio_token),
|
||||
create_user_end(),
|
||||
]
|
||||
else:
|
||||
return []
|
||||
|
||||
|
||||
def get_tts_sft_prompt(
|
||||
input: None | str = None,
|
||||
instruct=None,
|
||||
read_text_only=True,
|
||||
prompt_speech=None,
|
||||
):
|
||||
"""
|
||||
Build prompt for TTS (Text-to-Speech) task
|
||||
|
||||
Args:
|
||||
input: Input text
|
||||
instruct: Style instruction (e.g., "speak happily in a child's voice")
|
||||
read_text_only: Whether to read only plain text (False means text contains template)
|
||||
prompt_speech: Reference audio (for voice cloning)
|
||||
"""
|
||||
assistant_prompt_audio_token = preprocess_input(prompt_speech) if prompt_speech is not None else None
|
||||
|
||||
if not read_text_only:
|
||||
# Not just reading text, text contains template (template:text format)
|
||||
text = preprocess_input(input)
|
||||
lm_prompt = _build_tts_system_prompt(
|
||||
has_voice_prompt=assistant_prompt_audio_token is not None,
|
||||
voice_audio_token=assistant_prompt_audio_token,
|
||||
)
|
||||
lm_prompt.append(create_segment(text=f"<|im_start|>user\n{text}<|im_end|>\n"))
|
||||
lm_prompt.append(create_assistant_start_with_think())
|
||||
else:
|
||||
# Plain text (no instruction inside)
|
||||
language = detect_language(input)
|
||||
template = random.choice(tts_zh_templates if language == "zh" else tts_en_templates)
|
||||
text = preprocess_input(input)
|
||||
|
||||
if instruct is None:
|
||||
# No instruct instruction
|
||||
lm_prompt = _build_tts_system_prompt_no_instruct(
|
||||
has_voice_prompt=assistant_prompt_audio_token is not None,
|
||||
voice_audio_token=assistant_prompt_audio_token,
|
||||
)
|
||||
lm_prompt.extend(
|
||||
[
|
||||
create_segment(text=f"<|im_start|>user\n{template}: {text}<|im_end|>\n"),
|
||||
create_assistant_start_with_sostm(),
|
||||
]
|
||||
)
|
||||
else:
|
||||
# Has instruct instruction
|
||||
lm_prompt = _build_tts_system_prompt(
|
||||
has_voice_prompt=assistant_prompt_audio_token is not None,
|
||||
voice_audio_token=assistant_prompt_audio_token,
|
||||
)
|
||||
lm_prompt.append(create_segment(text=f"<|im_start|>user\n{template}: {text}({instruct})<|im_end|>\n"))
|
||||
lm_prompt.append(create_assistant_start_with_think())
|
||||
|
||||
return lm_prompt
|
||||
|
||||
|
||||
def get_audio_understanding_sft_prompt(
|
||||
input_speech,
|
||||
input_text,
|
||||
thinking=False,
|
||||
use_sostm=False,
|
||||
):
|
||||
"""Build prompt for audio understanding task"""
|
||||
audio_tokenized = preprocess_input(input_speech)
|
||||
|
||||
lm_prompt = create_user_turn_with_audio(audio_tokenized, extra_text=input_text)
|
||||
lm_prompt = append_assistant_ending(lm_prompt, thinking=thinking, use_sostm=use_sostm)
|
||||
return lm_prompt
|
||||
|
||||
|
||||
def _build_voice_prompt_system(prompt_speech) -> list[InputSegment]:
|
||||
"""Build system prompt with voice prompt"""
|
||||
return create_system_turn_with_voice_prompt(
|
||||
prompt_text="Your voice should be:", audio_token=preprocess_input(prompt_speech)
|
||||
)
|
||||
|
||||
|
||||
def get_spoken_dialogue_sft_prompt(
|
||||
input_speech,
|
||||
system_prompt=None,
|
||||
prompt_speech=None,
|
||||
add_history=False,
|
||||
):
|
||||
"""
|
||||
Build prompt for spoken dialogue task
|
||||
|
||||
Args:
|
||||
input_speech: Input speech
|
||||
system_prompt: System prompt text
|
||||
prompt_speech: Reference audio (for voice cloning)
|
||||
add_history: Whether to add history (Note: history variable is undefined in original code)
|
||||
"""
|
||||
audio_tokenized = preprocess_input(input_speech)
|
||||
lm_prompt = []
|
||||
|
||||
# Note: history variable is undefined in original code, this branch may never execute
|
||||
# To use history feature, history should be passed as a parameter
|
||||
if add_history:
|
||||
# Simplified form of adding history
|
||||
lm_prompt = create_user_turn_with_audio(audio_tokenized)
|
||||
lm_prompt.append(create_assistant_start_with_sostm())
|
||||
else:
|
||||
# Add voice prompt (if available)
|
||||
if prompt_speech:
|
||||
lm_prompt.extend(_build_voice_prompt_system(prompt_speech))
|
||||
|
||||
# Add user turn
|
||||
lm_prompt.append(create_user_start())
|
||||
if system_prompt:
|
||||
lm_prompt.append(create_segment(text=system_prompt))
|
||||
lm_prompt.append(create_segment(audio=audio_tokenized))
|
||||
lm_prompt.append(create_user_end())
|
||||
lm_prompt.append(create_assistant_start_with_sostm())
|
||||
|
||||
return lm_prompt
|
||||
|
||||
|
||||
def get_spoken_dialogue_sft_multiturn_prompt(
|
||||
message_list,
|
||||
system_prompt=None,
|
||||
prompt_speech=None,
|
||||
tokenizer=None,
|
||||
group_size=8,
|
||||
audio_channels=4,
|
||||
):
|
||||
"""
|
||||
Build prompt for multi-turn spoken dialogue task
|
||||
|
||||
Args:
|
||||
message_list: List of messages containing role and content
|
||||
system_prompt: System prompt text
|
||||
prompt_speech: Reference audio (for voice cloning)
|
||||
tokenizer: Tokenizer
|
||||
group_size: Group size
|
||||
audio_channels: Number of audio channels
|
||||
"""
|
||||
lm_prompt = []
|
||||
|
||||
# Add voice prompt (if available)
|
||||
if prompt_speech:
|
||||
lm_prompt.extend(
|
||||
create_system_turn_with_voice_prompt(
|
||||
prompt_text="Your voice should be:", audio_token=preprocess_input(prompt_speech)
|
||||
)
|
||||
)
|
||||
|
||||
# Define message processors
|
||||
def user_processor(msg):
|
||||
segments = [create_user_start()]
|
||||
if system_prompt:
|
||||
segments.append(create_segment(text=system_prompt))
|
||||
segments.append(create_segment(audio=preprocess_input(msg["content"])))
|
||||
segments.append(create_user_end())
|
||||
return segments
|
||||
|
||||
def assistant_processor(msg):
|
||||
return [
|
||||
create_assistant_start(),
|
||||
create_streaming_segment(
|
||||
text=msg["content"]["text"],
|
||||
audio=preprocess_input(msg["content"]["audio"]),
|
||||
tokenizer=tokenizer,
|
||||
group_size=group_size,
|
||||
audio_channels=audio_channels,
|
||||
),
|
||||
create_user_end(),
|
||||
]
|
||||
|
||||
# Process message list
|
||||
lm_prompt.extend(process_multiturn_messages(message_list, user_processor, assistant_processor))
|
||||
lm_prompt.append(create_assistant_start_with_sostm())
|
||||
|
||||
return lm_prompt
|
||||
|
||||
|
||||
def get_s2t_dialogue_sft_prompt(
|
||||
input_speech,
|
||||
thinking=False,
|
||||
):
|
||||
"""Build prompt for speech-to-text dialogue task"""
|
||||
audio_tokenized = preprocess_input(input_speech)
|
||||
|
||||
lm_prompt = create_user_turn_with_audio(audio_tokenized)
|
||||
lm_prompt = append_assistant_ending(lm_prompt, thinking=thinking)
|
||||
return lm_prompt
|
||||
|
||||
|
||||
def get_s2t_dialogue_sft_multiturn_prompt(message_list, thinking=False):
|
||||
"""Build prompt for multi-turn speech-to-text dialogue task"""
|
||||
lm_prompt = process_multiturn_messages(
|
||||
message_list, user_processor=create_audio_user_message, assistant_processor=create_text_assistant_message
|
||||
)
|
||||
lm_prompt = append_assistant_ending(lm_prompt, thinking=thinking)
|
||||
return lm_prompt
|
||||
|
||||
|
||||
def get_text_dialogue_sft_prompt(
|
||||
input_text,
|
||||
thinking=False,
|
||||
):
|
||||
"""Build prompt for text-only dialogue task"""
|
||||
lm_prompt = create_user_turn_with_text(input_text)
|
||||
lm_prompt = append_assistant_ending(lm_prompt, thinking=thinking)
|
||||
return lm_prompt
|
||||
|
||||
|
||||
def get_text_dialogue_sft_multiturn_prompt(
|
||||
message_list,
|
||||
thinking=False,
|
||||
):
|
||||
"""Build prompt for multi-turn text-only dialogue task"""
|
||||
lm_prompt = process_multiturn_messages(
|
||||
message_list, user_processor=create_text_user_message, assistant_processor=create_text_assistant_message
|
||||
)
|
||||
lm_prompt = append_assistant_ending(lm_prompt, thinking=thinking)
|
||||
return lm_prompt
|
||||
|
||||
|
||||
def get_in_context_learning_s2s_prompt(
|
||||
instruction,
|
||||
prompt_examples,
|
||||
audio,
|
||||
tokenizer=None,
|
||||
group_size=8,
|
||||
audio_channels=4,
|
||||
):
|
||||
"""
|
||||
Build prompt for In-Context Learning speech-to-speech task
|
||||
|
||||
Args:
|
||||
instruction: Instruction text
|
||||
prompt_examples: List of examples, each containing input_audio, output_transcription, output_audio
|
||||
audio: Input audio to be processed
|
||||
tokenizer: Tokenizer
|
||||
group_size: Group size
|
||||
audio_channels: Number of audio channels
|
||||
"""
|
||||
prompt = [create_segment(text=f"[Int]:{instruction}\n")]
|
||||
|
||||
# Add examples
|
||||
for example in prompt_examples:
|
||||
prompt.extend(
|
||||
[
|
||||
create_segment(audio=preprocess_input(example["input_audio"])),
|
||||
create_segment(text="\n"),
|
||||
create_streaming_segment(
|
||||
text=example["output_transcription"],
|
||||
audio=preprocess_input(example["output_audio"]),
|
||||
tokenizer=tokenizer,
|
||||
group_size=group_size,
|
||||
audio_channels=audio_channels,
|
||||
),
|
||||
create_segment(text=" \n\n"),
|
||||
]
|
||||
)
|
||||
|
||||
# Add input audio to be processed
|
||||
prompt.extend(
|
||||
[
|
||||
create_segment(audio=preprocess_input(audio)),
|
||||
create_segment(text="\n"),
|
||||
create_sostm_segment(),
|
||||
]
|
||||
)
|
||||
|
||||
return prompt
|
||||
|
||||
|
||||
def get_audio_data(audio_url):
|
||||
if audio_url.startswith("data:"):
|
||||
header, b64_data = audio_url.split(",", 1)
|
||||
audio_bytes = base64.b64decode(b64_data.strip())
|
||||
audio_file = io.BytesIO(audio_bytes)
|
||||
else:
|
||||
# File path
|
||||
audio_file = audio_url
|
||||
|
||||
audio_signal, sr = librosa.load(audio_file, sr=24000)
|
||||
audio_data = (audio_signal.astype(np.float32), sr)
|
||||
return audio_data
|
||||
|
||||
|
||||
def to_prompt(input_segs):
|
||||
out_put = []
|
||||
|
||||
for input_seg in input_segs:
|
||||
if isinstance(input_seg, StreamingInputSegment) and input_seg.text:
|
||||
out_put.append("<|sostm|>")
|
||||
if input_seg.audio is not None and isinstance(input_seg.audio, str):
|
||||
out_put.append(input_seg.text)
|
||||
out_put.append("<|eot|>")
|
||||
out_put.append("<|empty|>")
|
||||
else:
|
||||
out_put.append(input_seg.text)
|
||||
out_put.append("<|eot|>")
|
||||
out_put.append("<|eostm|>")
|
||||
|
||||
else:
|
||||
out_put.append(input_seg.text)
|
||||
if input_seg.audio is not None:
|
||||
out_put.append(input_seg.audio)
|
||||
|
||||
prompt = "".join(out_put)
|
||||
return prompt
|
||||
@@ -0,0 +1,261 @@
|
||||
#!/usr/bin/env python3
|
||||
# Copyright 2025 Xiaomi Corporation.
|
||||
# Copyright (c) 2023, NVIDIA CORPORATION. All rights reserved.
|
||||
|
||||
|
||||
import torch
|
||||
|
||||
|
||||
class InputSegment:
|
||||
def __init__(
|
||||
self,
|
||||
text: str = "",
|
||||
audio: torch.Tensor = None,
|
||||
tokenized_text: torch.Tensor = None,
|
||||
speech_zeroemb_idx: int | list[int] = 1024,
|
||||
text_zeroemb_idx: int = 152067,
|
||||
add_sosp_eosp=True,
|
||||
) -> None:
|
||||
has_text = text is not None
|
||||
has_tokenized_text = tokenized_text is not None
|
||||
assert has_text or has_tokenized_text, "Text or tokenized text must be provided"
|
||||
|
||||
self.audio = audio
|
||||
self.text = text
|
||||
self.tokenized_text = tokenized_text
|
||||
self.speech_zeroemb_idx = speech_zeroemb_idx
|
||||
self.text_zeroemb_idx = text_zeroemb_idx
|
||||
self.add_sosp_eosp = add_sosp_eosp
|
||||
|
||||
@staticmethod
|
||||
def insert_between(tensor, i, value=-1):
|
||||
return torch.scatter(
|
||||
torch.full(
|
||||
(1, tensor.shape[1] + (tensor.shape[1] - 1) * i + i),
|
||||
value,
|
||||
dtype=tensor.dtype,
|
||||
),
|
||||
1,
|
||||
torch.arange(0, tensor.shape[1], dtype=torch.int64)[None] * (i + 1),
|
||||
tensor,
|
||||
)
|
||||
|
||||
def to_input_id(
|
||||
self,
|
||||
tokenizer,
|
||||
group_size: int,
|
||||
audio_channels: int = 8,
|
||||
) -> tuple[torch.Tensor, torch.Tensor]:
|
||||
if (
|
||||
self.audio is None
|
||||
): ## If no audio is provided, tokenize the text with (group_size-1) tokens inserted between each, fill with -100; and set audio to empty indices
|
||||
if self.tokenized_text is None:
|
||||
tokenized_text = tokenizer(
|
||||
self.text,
|
||||
return_tensors="pt",
|
||||
truncation=True,
|
||||
max_length=999999,
|
||||
padding=False,
|
||||
add_special_tokens=False,
|
||||
)["input_ids"].int()
|
||||
else:
|
||||
tokenized_text = self.tokenized_text.unsqueeze(0)
|
||||
|
||||
if group_size > 1:
|
||||
tokenized_text = self.insert_between(tokenized_text, group_size - 1, value=-100)
|
||||
|
||||
if isinstance(self.speech_zeroemb_idx, list):
|
||||
audio_part_input_id = torch.zeros((audio_channels, tokenized_text.shape[1]), dtype=torch.int)
|
||||
for i, idx in enumerate(self.speech_zeroemb_idx):
|
||||
audio_part_input_id[i, :] = idx
|
||||
else:
|
||||
audio_part_input_id = torch.full(
|
||||
(audio_channels, tokenized_text.shape[1]), self.speech_zeroemb_idx, dtype=torch.int
|
||||
)
|
||||
|
||||
else: # If audio is provided, add sosp/eosp markers and convert audio: fill with empty tokens based on audio length
|
||||
sosp_token = tokenizer.convert_tokens_to_ids("<|sosp|>") if self.add_sosp_eosp else None
|
||||
eosp_token = tokenizer.convert_tokens_to_ids("<|eosp|>") if self.add_sosp_eosp else None
|
||||
audio_part = self.audio.reshape(-1, audio_channels).T # [audio_channels, seqlen]
|
||||
|
||||
assert audio_part.shape[1] % group_size == 0, (
|
||||
f"Audio shape {audio_part.shape} is not divisible by group_size {group_size}"
|
||||
)
|
||||
|
||||
text_len = audio_part.shape[1] // group_size
|
||||
empty_token = self.text_zeroemb_idx
|
||||
if empty_token is None:
|
||||
empty_token = tokenizer.eod
|
||||
tokenized_text = torch.full((1, text_len), empty_token, dtype=torch.int)
|
||||
|
||||
tokenized_text = (
|
||||
torch.cat(
|
||||
[
|
||||
torch.tensor([[sosp_token]], dtype=torch.int),
|
||||
tokenized_text,
|
||||
torch.tensor([[eosp_token]], dtype=torch.int),
|
||||
],
|
||||
dim=1,
|
||||
)
|
||||
if self.add_sosp_eosp
|
||||
else tokenized_text
|
||||
)
|
||||
tokenized_text = self.insert_between(tokenized_text, group_size - 1, value=-100)
|
||||
|
||||
if self.add_sosp_eosp:
|
||||
if isinstance(self.speech_zeroemb_idx, list):
|
||||
sosp_part = torch.zeros((audio_channels, group_size), dtype=torch.int)
|
||||
eosp_part = torch.zeros((audio_channels, group_size), dtype=torch.int)
|
||||
for i, idx in enumerate(self.speech_zeroemb_idx):
|
||||
sosp_part[i, :] = idx
|
||||
eosp_part[i, :] = idx
|
||||
audio_part_input_id = torch.cat([sosp_part, audio_part, eosp_part], dim=1)
|
||||
else:
|
||||
audio_part_input_id = torch.cat(
|
||||
[
|
||||
torch.full((audio_channels, group_size), self.speech_zeroemb_idx, dtype=torch.int),
|
||||
audio_part,
|
||||
torch.full((audio_channels, group_size), self.speech_zeroemb_idx, dtype=torch.int),
|
||||
],
|
||||
dim=1,
|
||||
)
|
||||
else:
|
||||
audio_part_input_id = audio_part
|
||||
|
||||
input_ids = torch.cat([tokenized_text, audio_part_input_id], dim=0) # [n_rvq + 1, seqlen]
|
||||
|
||||
return input_ids
|
||||
|
||||
|
||||
class StreamingInputSegment:
|
||||
def __init__(
|
||||
self,
|
||||
text: str = "",
|
||||
audio: torch.Tensor = None,
|
||||
tokenized_text: torch.Tensor = None,
|
||||
speech_zeroemb_idx: int | list[int] = 1024,
|
||||
text_zeroemb_idx: int = 152067,
|
||||
text_segment_size: int = 5,
|
||||
audio_segment_size: int = 5,
|
||||
tokenizer=None,
|
||||
group_size=None,
|
||||
audio_channels=None,
|
||||
) -> None:
|
||||
has_text = text is not None
|
||||
has_tokenized_text = tokenized_text is not None
|
||||
assert has_text or has_tokenized_text, "Text or tokenized text must be provided"
|
||||
|
||||
self.audio = audio
|
||||
self.text = text
|
||||
self.tokenized_text = tokenized_text
|
||||
self.speech_zeroemb_idx = speech_zeroemb_idx
|
||||
self.text_zeroemb_idx = text_zeroemb_idx
|
||||
self.text_segment_size = text_segment_size
|
||||
self.audio_segment_size = audio_segment_size
|
||||
self.tokenizer = tokenizer
|
||||
self.group_size = group_size
|
||||
self.audio_channels = audio_channels
|
||||
|
||||
def to_input_id(
|
||||
self,
|
||||
tokenizer,
|
||||
group_size: int,
|
||||
audio_channels: int = 8,
|
||||
):
|
||||
if self.tokenized_text is None:
|
||||
tokenized_text = tokenizer(
|
||||
self.text,
|
||||
return_tensors="pt",
|
||||
truncation=True,
|
||||
max_length=999999,
|
||||
padding=False,
|
||||
add_special_tokens=False,
|
||||
)["input_ids"].int() # [1, seqlen]
|
||||
else:
|
||||
tokenized_text = self.tokenized_text.unsqueeze(0)
|
||||
|
||||
tokenized_text = tokenized_text.squeeze(0)
|
||||
|
||||
text_segments = tokenized_text.split(self.text_segment_size, dim=0)
|
||||
audio_segments = self.audio.split(self.audio_segment_size * group_size * audio_channels, dim=0)
|
||||
|
||||
tokenized_segments = []
|
||||
tokenized_segments.append(
|
||||
InputSegment(
|
||||
text="<|sostm|>",
|
||||
speech_zeroemb_idx=self.speech_zeroemb_idx,
|
||||
text_zeroemb_idx=self.text_zeroemb_idx,
|
||||
),
|
||||
)
|
||||
|
||||
eot_tokens = tokenizer(
|
||||
"<|eot|>",
|
||||
return_tensors="pt",
|
||||
truncation=True,
|
||||
max_length=999999,
|
||||
padding=False,
|
||||
add_special_tokens=False,
|
||||
)["input_ids"][0].to(text_segments[-1])
|
||||
|
||||
text_segments = text_segments[:-1] + (torch.cat([text_segments[-1], eot_tokens], dim=0),)
|
||||
|
||||
length = min(len(text_segments), len(audio_segments)) # The number of empty tokens here may vary
|
||||
for i in range(length):
|
||||
text_segment = text_segments[i]
|
||||
audio_segment = audio_segments[i]
|
||||
|
||||
tokenized_segments.append(
|
||||
InputSegment(
|
||||
tokenized_text=text_segment,
|
||||
speech_zeroemb_idx=self.speech_zeroemb_idx,
|
||||
text_zeroemb_idx=self.text_zeroemb_idx,
|
||||
),
|
||||
)
|
||||
tokenized_segments.append(
|
||||
InputSegment(
|
||||
audio=audio_segment,
|
||||
add_sosp_eosp=False,
|
||||
speech_zeroemb_idx=self.speech_zeroemb_idx,
|
||||
text_zeroemb_idx=self.text_zeroemb_idx,
|
||||
),
|
||||
)
|
||||
|
||||
for j in range(length, len(text_segments)):
|
||||
tokenized_segments.append(
|
||||
InputSegment(
|
||||
tokenized_text=text_segments[j],
|
||||
speech_zeroemb_idx=self.speech_zeroemb_idx,
|
||||
text_zeroemb_idx=self.text_zeroemb_idx,
|
||||
),
|
||||
)
|
||||
|
||||
for j in range(length, len(audio_segments)):
|
||||
tokenized_segments.append(
|
||||
InputSegment(
|
||||
audio=audio_segments[j],
|
||||
add_sosp_eosp=False,
|
||||
speech_zeroemb_idx=self.speech_zeroemb_idx,
|
||||
text_zeroemb_idx=self.text_zeroemb_idx,
|
||||
),
|
||||
)
|
||||
|
||||
tokenized_segments.append(
|
||||
InputSegment(
|
||||
text="<|eostm|>",
|
||||
speech_zeroemb_idx=self.speech_zeroemb_idx,
|
||||
text_zeroemb_idx=self.text_zeroemb_idx,
|
||||
),
|
||||
)
|
||||
|
||||
input_ids = [
|
||||
seg.to_input_id(
|
||||
self.tokenizer,
|
||||
self.group_size,
|
||||
self.audio_channels,
|
||||
)
|
||||
for seg in tokenized_segments
|
||||
]
|
||||
|
||||
input_ids = torch.cat(input_ids, dim=1).type(torch.int64) # [n_rvq + 1, seqlen]
|
||||
|
||||
return input_ids
|
||||
@@ -0,0 +1,58 @@
|
||||
# Online serving Example of vLLM-Omni for MiMo-Audio
|
||||
|
||||
## 🛠️ Installation
|
||||
|
||||
Please refer to [README.md](../../../README.md)
|
||||
|
||||
## Run examples (MiMo-Audio)
|
||||
|
||||
### Launch the Server
|
||||
```bash
|
||||
export MIMO_AUDIO_TOKENIZER_PATH="XiaomiMiMo/MiMo-Audio-Tokenizer"
|
||||
|
||||
vllm-omni serve XiaomiMiMo/MiMo-Audio-7B-Instruct --omni \
|
||||
--served-model-name "MiMo-Audio-7B-Instruct" \
|
||||
--port 18091 --stage-configs-path ./vllm_omni/model_executor/stage_configs/mimo_audio.yaml \
|
||||
--chat-template ./examples/online_serving/mimo_audio/chat_template.jinja
|
||||
```
|
||||
> ⚠️ **Important**
|
||||
> **MiMo-Audio is not compatible with the default chat template.**
|
||||
> The provided `chat_template.jinja` implements MiMo-specific role, audio token, and instruction formatting and **must be used for all inference**.
|
||||
|
||||
|
||||
### Send Multi-modal Request
|
||||
|
||||
Get into the example folder
|
||||
```bash
|
||||
cd examples/online_serving/mimo_audio
|
||||
```
|
||||
|
||||
#### Send request via python
|
||||
|
||||
```bash
|
||||
# Audio dialogue task
|
||||
python openai_chat_completion_client_for_multimodal_generation.py \
|
||||
--query-type multi_audios \
|
||||
--message-json ../../offline_inference/mimo_audio/message_base64_wav.json
|
||||
```
|
||||
|
||||
The Python client supports the following command-line arguments:
|
||||
|
||||
- `--query-type` (or `-q`): Query type (default: `multi_audios`)
|
||||
- Options: `multi_audios`, `text`
|
||||
- `--message-json` (or `-m`): Path to `base64` multi rounds audio messages json file
|
||||
- Do not pass any value for "text" query type
|
||||
- Supports local file paths (automatically encoded to base64) or HTTP/HTTPS URLs, only for "Are these two audio clips the same?" task
|
||||
- Example: `---message-json ./examples/offline_inference/mimo_audio/message_base64_wav.json`
|
||||
- `--prompt` (or `-p`): Custom text prompt/question, only for query type is "text"(TTS task)
|
||||
- Attention! Do not pass any value for "multi_audios" query type
|
||||
- Example: `--prompt "What are the main activities shown in this video?"`
|
||||
|
||||
|
||||
For example, to use multi rounds audios with local files:
|
||||
|
||||
```bash
|
||||
python openai_chat_completion_client_for_multimodal_generation.py \
|
||||
--query-type multi_audios \
|
||||
--message-json ../../offline_inference/mimo_audio/message_base64_wav.json
|
||||
```
|
||||
@@ -0,0 +1,86 @@
|
||||
{%- if tools %}
|
||||
{{- '<|im_start|>system\n' }}
|
||||
{%- if messages[0]['role'] == 'system' %}
|
||||
{{- messages[0]['content'] }}
|
||||
{%- else %}
|
||||
{{- 'You are a helpful assistant.' }}
|
||||
{%- endif %}
|
||||
{{- "\n\n# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>" }}
|
||||
{%- for tool in tools %}
|
||||
{{- "\n" }}
|
||||
{{- tool | tojson }}
|
||||
{%- endfor %}
|
||||
{{- "\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call><|im_end|>\n" }}
|
||||
{%- else %}
|
||||
{%- if messages[0]['role'] == 'system' %}
|
||||
{%- set _m = '<|sosp|><|empty|><|eosp|>' -%}
|
||||
{%- set _raw0 = messages[0]['content'] if messages[0]['content'] is string else '' -%}
|
||||
{%- if _m in _raw0 %}
|
||||
{%- set _t0 = (_raw0 | replace(_m ~ '\n', '') | replace(_m, '') | trim) -%}
|
||||
{{- '<|im_start|>system\n' + (_t0 ~ _m if _t0 else _m) + '<|im_end|>\n' }}
|
||||
{%- else %}
|
||||
{{- '<|im_start|>system\n' + _raw0 + '<|im_end|>\n' }}
|
||||
{%- endif %}
|
||||
{%- else %}
|
||||
{{- '<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n' }}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
|
||||
{%- for message in messages %}
|
||||
{%- if message['role'] == 'assistant' %}
|
||||
{{- '<|im_start|>assistant' }}
|
||||
{%- set _sosp = '<|sosp|><|empty|><|eosp|>' -%}
|
||||
{%- set _text = message['content'] if message['content'] is string else '' -%}
|
||||
{%- if _sosp in _text %}
|
||||
{%- set _clean = _text | replace(_sosp, '') -%}
|
||||
{%- set _body = _clean[1:] if (_clean and _clean[0] == '\n') else _clean -%}
|
||||
{{- '\n<|sostm|>' + _body + '<|eot|><|empty|><|eostm|>' }}
|
||||
{%- else %}
|
||||
{%- set _body = _text[1:] if (_text and _text[0] == '\n') else _text -%}
|
||||
{{- '\n<|sostm|>' + _body + '<|eot|><|eostm|>' }}
|
||||
{%- endif %}
|
||||
{{- '<|im_end|>\n' }}
|
||||
|
||||
{%- elif message['role'] == 'user' %}
|
||||
{%- set _m = '<|sosp|><|empty|><|eosp|>' -%}
|
||||
{%- set _raw = message['content'] if message['content'] is string else '' -%}
|
||||
{%- if _m in _raw %}
|
||||
{%- set _t = (_raw | replace(_m ~ '\n', '') | replace(_m, '') | trim) -%}
|
||||
{{- '<|im_start|>user\n' + (_t ~ _m if _t else _m) + '<|im_end|>\n' }}
|
||||
{%- else %}
|
||||
{{- '<|im_start|>user\n' + _raw + '<|im_end|>\n' }}
|
||||
{%- endif %}
|
||||
|
||||
{%- elif message['role'] == 'system' %}
|
||||
{%- if not loop.first %}
|
||||
{%- set _m = '<|sosp|><|empty|><|eosp|>' -%}
|
||||
{%- set _raw = message['content'] if message['content'] is string else '' -%}
|
||||
{%- if _m in _raw %}
|
||||
{%- set _t = (_raw | replace(_m ~ '\n', '') | replace(_m, '') | trim) -%}
|
||||
{{- '<|im_start|>system\n' + (_t ~ _m if _t else _m) + '<|im_end|>\n' }}
|
||||
{%- else %}
|
||||
{{- '<|im_start|>system\n' + _raw + '<|im_end|>\n' }}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
|
||||
{%- elif message['role'] == 'tool' %}
|
||||
{%- if (loop.index0 == 0) or (messages[loop.index0 - 1]['role'] != 'tool') %}
|
||||
{{- '<|im_start|>user' }}
|
||||
{%- endif %}
|
||||
{%- set _m = '<|sosp|><|empty|><|eosp|>' -%}
|
||||
{%- set _raw = message['content'] if message['content'] is string else '' -%}
|
||||
{%- if _m in _raw %}
|
||||
{%- set _t = (_raw | replace(_m ~ '\n', '') | replace(_m, '') | trim) -%}
|
||||
{{- '\n<tool_response>\n' + (_t ~ _m if _t else _m) + '\n</tool_response>' }}
|
||||
{%- else %}
|
||||
{{- '\n<tool_response>\n' + _raw + '\n</tool_response>' }}
|
||||
{%- endif %}
|
||||
{%- if loop.last or (messages[loop.index0 + 1]['role'] != 'tool') %}
|
||||
{{- '<|im_end|>\n' }}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{%- endfor %}
|
||||
|
||||
{%- if add_generation_prompt %}
|
||||
{{- '<|im_start|>assistant\n<|sostm|>' }}
|
||||
{%- endif %}
|
||||
+545
@@ -0,0 +1,545 @@
|
||||
import base64
|
||||
import concurrent.futures
|
||||
import datetime
|
||||
import json
|
||||
import os
|
||||
import queue
|
||||
import sys
|
||||
import threading
|
||||
import time
|
||||
from typing import Any
|
||||
|
||||
import requests
|
||||
from openai import OpenAI
|
||||
from vllm.assets.audio import AudioAsset
|
||||
from vllm.utils.argparse_utils import FlexibleArgumentParser
|
||||
|
||||
# Modify OpenAI's API key and API base to use vLLM's API server.
|
||||
openai_api_key = "EMPTY"
|
||||
openai_api_base = "http://localhost:18091/v1"
|
||||
|
||||
|
||||
# Modify OpenAI's API key and API base to use vLLM's API server.
|
||||
|
||||
|
||||
client = OpenAI(
|
||||
# defaults to os.environ.get("OPENAI_API_KEY")
|
||||
api_key=openai_api_key,
|
||||
base_url=openai_api_base,
|
||||
)
|
||||
|
||||
SEED = 42
|
||||
|
||||
|
||||
def encode_base64_content_from_url(content_url: str) -> str:
|
||||
"""Encode a content retrieved from a remote url to base64 format."""
|
||||
|
||||
with requests.get(content_url) as response:
|
||||
response.raise_for_status()
|
||||
result = base64.b64encode(response.content).decode("utf-8")
|
||||
|
||||
return result
|
||||
|
||||
|
||||
def encode_base64_content_from_file(file_path: str) -> str:
|
||||
"""Encode a local file to base64 format."""
|
||||
with open(file_path, "rb") as f:
|
||||
content = f.read()
|
||||
result = base64.b64encode(content).decode("utf-8")
|
||||
return result
|
||||
|
||||
|
||||
def get_video_url_from_path(video_path: str | None) -> str:
|
||||
"""Convert a video path (local file or URL) to a video URL format for the API.
|
||||
|
||||
If video_path is None or empty, returns the default URL.
|
||||
If video_path is a local file path, encodes it to base64 data URL.
|
||||
If video_path is a URL, returns it as-is.
|
||||
"""
|
||||
if not video_path:
|
||||
# Default video URL
|
||||
return "https://huggingface.co/datasets/raushan-testing-hf/videos-test/resolve/main/sample_demo_1.mp4"
|
||||
|
||||
# Check if it's a URL (starts with http:// or https://)
|
||||
if video_path.startswith(("http://", "https://")):
|
||||
return video_path
|
||||
|
||||
# Otherwise, treat it as a local file path
|
||||
if not os.path.exists(video_path):
|
||||
raise FileNotFoundError(f"Video file not found: {video_path}")
|
||||
|
||||
# Detect video MIME type from file extension
|
||||
video_path_lower = video_path.lower()
|
||||
if video_path_lower.endswith(".mp4"):
|
||||
mime_type = "video/mp4"
|
||||
elif video_path_lower.endswith(".webm"):
|
||||
mime_type = "video/webm"
|
||||
elif video_path_lower.endswith(".mov"):
|
||||
mime_type = "video/quicktime"
|
||||
elif video_path_lower.endswith(".avi"):
|
||||
mime_type = "video/x-msvideo"
|
||||
elif video_path_lower.endswith(".mkv"):
|
||||
mime_type = "video/x-matroska"
|
||||
else:
|
||||
# Default to mp4 if extension is unknown
|
||||
mime_type = "video/mp4"
|
||||
|
||||
video_base64 = encode_base64_content_from_file(video_path)
|
||||
return f"data:{mime_type};base64,{video_base64}"
|
||||
|
||||
|
||||
def get_image_url_from_path(image_path: str | None) -> str:
|
||||
"""Convert an image path (local file or URL) to an image URL format for the API.
|
||||
|
||||
If image_path is None or empty, returns the default URL.
|
||||
If image_path is a local file path, encodes it to base64 data URL.
|
||||
If image_path is a URL, returns it as-is.
|
||||
"""
|
||||
if not image_path:
|
||||
# Default image URL
|
||||
return "https://vllm-public-assets.s3.us-west-2.amazonaws.com/vision_model_images/cherry_blossom.jpg"
|
||||
|
||||
# Check if it's a URL (starts with http:// or https://)
|
||||
if image_path.startswith(("http://", "https://")):
|
||||
return image_path
|
||||
|
||||
# Otherwise, treat it as a local file path
|
||||
if not os.path.exists(image_path):
|
||||
raise FileNotFoundError(f"Image file not found: {image_path}")
|
||||
|
||||
# Detect image MIME type from file extension
|
||||
image_path_lower = image_path.lower()
|
||||
if image_path_lower.endswith((".jpg", ".jpeg")):
|
||||
mime_type = "image/jpeg"
|
||||
elif image_path_lower.endswith(".png"):
|
||||
mime_type = "image/png"
|
||||
elif image_path_lower.endswith(".gif"):
|
||||
mime_type = "image/gif"
|
||||
elif image_path_lower.endswith(".webp"):
|
||||
mime_type = "image/webp"
|
||||
else:
|
||||
# Default to jpeg if extension is unknown
|
||||
mime_type = "image/jpeg"
|
||||
|
||||
image_base64 = encode_base64_content_from_file(image_path)
|
||||
return f"data:{mime_type};base64,{image_base64}"
|
||||
|
||||
|
||||
def get_audio_url_from_path(audio_path: str | None) -> str:
|
||||
"""Convert an audio path (local file or URL) to an audio URL format for the API.
|
||||
|
||||
If audio_path is None or empty, returns the default URL.
|
||||
If audio_path is already a base64 data URL (starts with "data:"), returns it as-is.
|
||||
If audio_path is a URL (starts with http:// or https://), returns it as-is.
|
||||
If audio_path is a local file path, encodes it to base64 data URL.
|
||||
"""
|
||||
if not audio_path:
|
||||
# Default audio URL
|
||||
return AudioAsset("mary_had_lamb").url
|
||||
|
||||
# Check if it's already a base64 data URL
|
||||
if audio_path.startswith("data:"):
|
||||
return audio_path
|
||||
|
||||
# Check if it's a URL (starts with http:// or https://)
|
||||
if audio_path.startswith(("http://", "https://")):
|
||||
return audio_path
|
||||
|
||||
# Otherwise, treat it as a local file path
|
||||
if not os.path.exists(audio_path):
|
||||
raise FileNotFoundError(f"Audio file not found: {audio_path}")
|
||||
|
||||
# Detect audio MIME type from file extension
|
||||
audio_path_lower = audio_path.lower()
|
||||
if audio_path_lower.endswith((".mp3", ".mpeg")):
|
||||
mime_type = "audio/mpeg"
|
||||
elif audio_path_lower.endswith(".wav"):
|
||||
mime_type = "audio/wav"
|
||||
elif audio_path_lower.endswith(".ogg"):
|
||||
mime_type = "audio/ogg"
|
||||
elif audio_path_lower.endswith(".flac"):
|
||||
mime_type = "audio/flac"
|
||||
elif audio_path_lower.endswith(".m4a"):
|
||||
mime_type = "audio/mp4"
|
||||
else:
|
||||
# Default to wav if extension is unknown
|
||||
mime_type = "audio/wav"
|
||||
|
||||
audio_base64 = encode_base64_content_from_file(audio_path)
|
||||
return f"data:{mime_type};base64,{audio_base64}"
|
||||
|
||||
|
||||
def load_messages_from_json(message_json_path: str) -> dict[str, Any]:
|
||||
"""Load messages from a JSON file."""
|
||||
with open(message_json_path, encoding="utf-8") as f:
|
||||
data = json.load(f)
|
||||
return data
|
||||
|
||||
|
||||
def process_audio_url_in_content(content: dict[str, Any]) -> dict[str, Any]:
|
||||
"""Process audio_url in content, handling both file paths and base64 URLs."""
|
||||
if content.get("type") == "audio_url":
|
||||
audio_url = content.get("audio_url", {}).get("url")
|
||||
if audio_url:
|
||||
# Process the audio URL (handles base64, file paths, and URLs)
|
||||
processed_url = get_audio_url_from_path(audio_url)
|
||||
content = content.copy()
|
||||
content["audio_url"] = {"url": processed_url}
|
||||
return content
|
||||
|
||||
|
||||
def get_system_prompt(message_json_path: str | None = None):
|
||||
"""Get system prompt, optionally from message.json file."""
|
||||
if message_json_path and os.path.exists(message_json_path):
|
||||
data = load_messages_from_json(message_json_path)
|
||||
messages = data.get("messages", [])
|
||||
# Find the first system message
|
||||
for msg in messages:
|
||||
if msg.get("role") == "system":
|
||||
# Process audio URLs in the content
|
||||
processed_content = []
|
||||
for content_item in msg.get("content", []):
|
||||
processed_item = process_audio_url_in_content(content_item)
|
||||
processed_content.append(processed_item)
|
||||
return {
|
||||
"role": "system",
|
||||
"content": processed_content,
|
||||
}
|
||||
|
||||
|
||||
def get_text_query(custom_prompt: str | None = None):
|
||||
question = f"Please convert this text to speech: {custom_prompt}"
|
||||
prompt = {
|
||||
"role": "user",
|
||||
"content": [
|
||||
{
|
||||
"type": "text",
|
||||
"text": f"{question}",
|
||||
}
|
||||
],
|
||||
}
|
||||
return prompt
|
||||
|
||||
|
||||
def get_multi_audios_query(
|
||||
audio_path: str | None = None,
|
||||
custom_prompt: str | None = None,
|
||||
message_json_path: str | None = None,
|
||||
) -> list[dict[str, Any]] | dict[str, Any]:
|
||||
"""Get multi-audios query, optionally from message.json file."""
|
||||
if message_json_path and os.path.exists(message_json_path):
|
||||
data = load_messages_from_json(message_json_path)
|
||||
messages = data.get("messages", [])
|
||||
|
||||
# Skip the first system role and return all other messages
|
||||
rebuild_prompt_messages = []
|
||||
skipped_first_system = False
|
||||
|
||||
for msg in messages:
|
||||
if msg.get("role") == "system" and not skipped_first_system:
|
||||
skipped_first_system = True
|
||||
continue
|
||||
|
||||
# Process all content items in the message
|
||||
processed_content = []
|
||||
for content_item in msg.get("content", []):
|
||||
processed_item = process_audio_url_in_content(content_item)
|
||||
processed_content.append(processed_item)
|
||||
|
||||
rebuild_prompt_messages.append(
|
||||
{
|
||||
"role": msg.get("role"),
|
||||
"content": processed_content,
|
||||
}
|
||||
)
|
||||
|
||||
return rebuild_prompt_messages
|
||||
|
||||
# Original behavior when message_json_path is not provided
|
||||
question = custom_prompt or "Are these two audio clips the same?"
|
||||
audio_url = get_audio_url_from_path(audio_path)
|
||||
prompt = {
|
||||
"role": "user",
|
||||
"content": [
|
||||
{
|
||||
"type": "audio_url",
|
||||
"audio_url": {"url": audio_url},
|
||||
},
|
||||
{
|
||||
"type": "audio_url",
|
||||
"audio_url": {"url": AudioAsset("winning_call").url},
|
||||
},
|
||||
{
|
||||
"type": "text",
|
||||
"text": f"{question}",
|
||||
},
|
||||
],
|
||||
}
|
||||
return prompt
|
||||
|
||||
|
||||
query_map = {
|
||||
"multi_audios": get_multi_audios_query,
|
||||
"text": get_text_query,
|
||||
}
|
||||
|
||||
|
||||
def run_multimodal_generation(args) -> None:
|
||||
model_name = "MiMo-Audio-7B-Instruct"
|
||||
thinker_sampling_params = {
|
||||
"temperature": 0.0, # Deterministic - no randomness
|
||||
"top_p": 1.0, # Disable nucleus sampling
|
||||
"top_k": -1, # Disable top-k sampling
|
||||
"max_tokens": 3200,
|
||||
"seed": SEED, # Fixed seed for sampling
|
||||
"detokenize": True,
|
||||
"repetition_penalty": 1.1,
|
||||
}
|
||||
code2wav_sampling_params = {
|
||||
"temperature": 0.0, # Deterministic - no randomness
|
||||
"top_p": 1.0, # Disable nucleus sampling
|
||||
"top_k": -1, # Disable top-k sampling
|
||||
"max_tokens": 4096 * 16,
|
||||
"seed": SEED, # Fixed seed for sampling
|
||||
"detokenize": True,
|
||||
"repetition_penalty": 1.1,
|
||||
}
|
||||
|
||||
sampling_params_list = [
|
||||
thinker_sampling_params,
|
||||
code2wav_sampling_params,
|
||||
]
|
||||
|
||||
# Get paths and custom prompt from args
|
||||
audio_path = getattr(args, "audio_path", None)
|
||||
custom_prompt = getattr(args, "prompt", None)
|
||||
message_json_path = getattr(args, "message_json", None)
|
||||
output_audio_path = getattr(args, "output_audio_path", None)
|
||||
|
||||
# Get the query function and call it with appropriate parameters
|
||||
query_func = query_map[args.query_type]
|
||||
if args.query_type == "multi_audios":
|
||||
prompt = query_func(audio_path=audio_path, custom_prompt=custom_prompt, message_json_path=message_json_path)
|
||||
elif args.query_type == "text":
|
||||
prompt = query_func(custom_prompt=custom_prompt)
|
||||
else:
|
||||
prompt = query_func()
|
||||
|
||||
extra_body = {
|
||||
"sampling_params_list": sampling_params_list
|
||||
# Optional, it has a default setting in stage_configs of the corresponding model.
|
||||
}
|
||||
|
||||
# Build messages list (filter None so concurrent tasks get valid structure)
|
||||
system = get_system_prompt(message_json_path=message_json_path)
|
||||
if args.query_type == "multi_audios" and isinstance(prompt, list):
|
||||
messages = ([system] if system else []) + prompt
|
||||
elif args.query_type == "text":
|
||||
messages = [prompt]
|
||||
|
||||
num_concurrent_requests = args.num_concurrent_requests
|
||||
start_time = time.perf_counter()
|
||||
|
||||
def run_one_request(req_id: int):
|
||||
"""Submit one request and return (req_id, response) to preserve request order."""
|
||||
return req_id, client.chat.completions.create(
|
||||
messages=messages,
|
||||
model=model_name,
|
||||
extra_body=extra_body,
|
||||
stream=args.stream,
|
||||
)
|
||||
|
||||
with concurrent.futures.ThreadPoolExecutor(max_workers=num_concurrent_requests) as executor:
|
||||
futures = [executor.submit(run_one_request, req_id) for req_id in range(num_concurrent_requests)]
|
||||
|
||||
if not args.stream:
|
||||
# Collect by req_id so chat_completions[i] = response for request i
|
||||
results_by_req = [None] * num_concurrent_requests
|
||||
for future in concurrent.futures.as_completed(futures):
|
||||
req_id, completion = future.result()
|
||||
results_by_req[req_id] = completion
|
||||
chat_completions = results_by_req
|
||||
|
||||
assert len(chat_completions) == num_concurrent_requests
|
||||
count = 0
|
||||
for req_id, chat_completion in enumerate(chat_completions):
|
||||
chat_completion_id = getattr(chat_completion, "id", "")
|
||||
for choice in chat_completion.choices:
|
||||
if choice.message.audio:
|
||||
audio_data = base64.b64decode(choice.message.audio.data)
|
||||
audio_file_path = f"{output_audio_path}/{args.query_type}/audio_{count}.wav"
|
||||
os.makedirs(os.path.dirname(audio_file_path), exist_ok=True)
|
||||
with open(audio_file_path, "wb") as f:
|
||||
f.write(audio_data)
|
||||
print(f"[req {req_id}_{chat_completion_id}] Audio saved to {audio_file_path}")
|
||||
count += 1
|
||||
elif choice.message.content:
|
||||
print(
|
||||
f"[req {req_id}_{chat_completion_id}] Chat completion output from text:",
|
||||
choice.message.content,
|
||||
)
|
||||
else:
|
||||
# Stream mode: process chunks from all requests in real-time,
|
||||
# displaying them in the order they arrive.
|
||||
chunk_queue = queue.Queue()
|
||||
audio_counters = {}
|
||||
chat_completion_id_by_req = {}
|
||||
chat_completion_id_lock = threading.Lock()
|
||||
last_text_req_id = None
|
||||
|
||||
def _stream_reader(req_id: int, stream):
|
||||
"""Read one stream and relay every chunk to the shared queue."""
|
||||
try:
|
||||
for chunk in stream:
|
||||
chunk_queue.put((req_id, time.perf_counter(), chunk))
|
||||
chunk_queue.put((req_id, time.perf_counter(), None))
|
||||
except Exception as e:
|
||||
print(f"\n[Request {req_id}] stream error: {e}", file=sys.stderr, flush=True)
|
||||
chunk_queue.put((req_id, time.time(), None))
|
||||
|
||||
# Kick off a reader thread per request
|
||||
reader_threads = []
|
||||
for req_id, future in enumerate(futures):
|
||||
req_id_from_future, stream = future.result()
|
||||
assert req_id == req_id_from_future, f"Request ID mismatch: {req_id} != {req_id_from_future}"
|
||||
audio_counters[req_id] = 0
|
||||
t = threading.Thread(target=_stream_reader, args=(req_id, stream), daemon=True)
|
||||
t.start()
|
||||
reader_threads.append(t)
|
||||
|
||||
# Main loop – consume chunks in arrival order
|
||||
active_streams = set(range(num_concurrent_requests))
|
||||
while active_streams:
|
||||
try:
|
||||
request_id, ts, chunk = chunk_queue.get(timeout=2.0)
|
||||
except queue.Empty:
|
||||
if all(not t.is_alive() for t in reader_threads):
|
||||
break
|
||||
continue
|
||||
|
||||
if chunk is None:
|
||||
elapsed = ts - start_time
|
||||
print(f" ({elapsed:.2f}s)", flush=True)
|
||||
with chat_completion_id_lock:
|
||||
chat_completion_id = chat_completion_id_by_req.get(request_id, "")
|
||||
print(
|
||||
f" [req {request_id}_{chat_completion_id}] Time finished for streaming: ",
|
||||
datetime.datetime.now(),
|
||||
file=sys.stderr,
|
||||
flush=True,
|
||||
)
|
||||
active_streams.discard(request_id)
|
||||
continue
|
||||
|
||||
with chat_completion_id_lock:
|
||||
if request_id not in chat_completion_id_by_req:
|
||||
chat_completion_id_by_req[request_id] = getattr(chunk, "id", "")
|
||||
chat_completion_id = chat_completion_id_by_req[request_id]
|
||||
|
||||
modality = getattr(chunk, "modality", None)
|
||||
elapsed = ts - start_time
|
||||
for choice in chunk.choices:
|
||||
content = getattr(choice.delta, "content", None) if hasattr(choice, "delta") else None
|
||||
|
||||
if modality == "audio" and content:
|
||||
audio_data = base64.b64decode(content)
|
||||
audio_dir = (
|
||||
f"{output_audio_path}/{args.query_type}/{num_concurrent_requests}/{chat_completion_id}"
|
||||
)
|
||||
os.makedirs(audio_dir, exist_ok=True)
|
||||
audio_file_path = f"{audio_dir}/audio_{audio_counters[request_id]}.wav"
|
||||
with open(audio_file_path, "wb") as f:
|
||||
f.write(audio_data)
|
||||
print(
|
||||
f"\n[{elapsed:7.2f}s][req {request_id}_{chat_completion_id}] Audio saved to {audio_file_path}",
|
||||
file=sys.stderr,
|
||||
flush=True,
|
||||
)
|
||||
audio_counters[request_id] += 1
|
||||
|
||||
elif modality == "text" and content:
|
||||
if last_text_req_id != request_id:
|
||||
if last_text_req_id is not None:
|
||||
print(flush=True)
|
||||
print(
|
||||
f"\n[{elapsed:7.2f}s][req {request_id}_{chat_completion_id}] content:",
|
||||
end="",
|
||||
flush=True,
|
||||
)
|
||||
last_text_req_id = request_id
|
||||
print(
|
||||
f"\n[{elapsed:7.2f}s][req {request_id}_{chat_completion_id}] {content}", end="", flush=True
|
||||
)
|
||||
|
||||
# Final newline if the last output was streaming text
|
||||
if last_text_req_id is not None:
|
||||
print(flush=True)
|
||||
|
||||
for t in reader_threads:
|
||||
t.join(timeout=1.0)
|
||||
|
||||
elapsed = time.perf_counter() - start_time
|
||||
print(
|
||||
f"num_concurrent_requests_{num_concurrent_requests}>>>>>>>>>Time finished for streaming<<<<<<<<: ",
|
||||
elapsed,
|
||||
file=sys.stderr,
|
||||
flush=True,
|
||||
)
|
||||
timing_audio_folder = f"{output_audio_path}/{args.query_type}/{num_concurrent_requests}"
|
||||
os.makedirs(timing_audio_folder, exist_ok=True)
|
||||
timing_file = os.path.join(timing_audio_folder, "streaming_finish_time.txt")
|
||||
with open(timing_file, "w") as f:
|
||||
f.write(f"num_concurrent_requests_{num_concurrent_requests} elapsed_seconds: {elapsed}\n")
|
||||
|
||||
|
||||
def parse_args():
|
||||
parser = FlexibleArgumentParser(description="Demo on using vLLM for offline inference with audio language models")
|
||||
parser.add_argument(
|
||||
"--query-type",
|
||||
"-q",
|
||||
type=str,
|
||||
default="multi_audios",
|
||||
choices=query_map.keys(),
|
||||
help="Query type.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--prompt",
|
||||
"-p",
|
||||
type=str,
|
||||
default="Who are you?",
|
||||
help="Custom text prompt/question to use instead of the default prompt for the selected query type.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--message-json",
|
||||
"-m",
|
||||
type=str,
|
||||
# default="../../offline_inference/mimo_audio/message_base64_wav.json",
|
||||
default="../../offline_inference/mimo_audio/message_base64_wav_tts.json",
|
||||
help="Path to message.json file containing conversation history. When provided, "
|
||||
"system prompt and multi_audios query will be loaded from this file.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--output-audio-path",
|
||||
"-o",
|
||||
type=str,
|
||||
default="./",
|
||||
help="Path to save the generated audio files.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--stream",
|
||||
action="store_true",
|
||||
help="Stream the response.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--num-concurrent-requests",
|
||||
type=int,
|
||||
default=1,
|
||||
help="Number of concurrent requests to send. Default is 1.",
|
||||
)
|
||||
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
args = parse_args()
|
||||
run_multimodal_generation(args)
|
||||
Reference in new issue
Block a user