[New Model]: XiaomiMiMo/MiMo-Audio-7B-Instruct support (#750)

Signed-off-by: wangyu31577 <wangyu31577@hundsun.com>
Signed-off-by: 齐保元 <qibaoyuan@xiaomi.com>
Signed-off-by: hsliu <liuhongsheng4@huawei.com>
Signed-off-by: Hongsheng Liu <liuhongsheng4@huawei.com>
Signed-off-by: GG-li <3226868735@qq.com>
Signed-off-by: Sihao Li <111170255+GG-li@users.noreply.github.com>
Signed-off-by: XU Mingshi <91017482+mxuax@users.noreply.github.com>
Signed-off-by: mxuax <mxuax@connect.ust.hk>
Signed-off-by: Baoyuan Qi <qibaoyuan@126.com>
Signed-off-by: gcanlin <canlinguosdu@gmail.com>
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>
Signed-off-by: wuzhongjian <wuzhongjian_yewu@cmss.chinamobile.com>
Signed-off-by: dongbo910220 <1275604947@qq.com>
Signed-off-by: dongbo910220 <32610838+dongbo910220@users.noreply.github.com>
Signed-off-by: Jiangyun Zhu <riverclouds.zhu@qq.com>
Signed-off-by: Junhong Liu <98734602+LJH-LBJ@users.noreply.github.com>
Signed-off-by: tjtanaa <tunjian.tan@embeddedllm.com>
Signed-off-by: baoyuan qi <qibaoyuan@126.com>
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com>
Signed-off-by: Prajwal A <prajwalanagani@gmail.com>
Signed-off-by: Shijin Zhang <75300765+Dovis01@users.noreply.github.com>
Signed-off-by: 丁宁 <nndding@gmail.com>
Signed-off-by: SHIJIN ZHANG <75300765+Dovis01@users.noreply.github.com>
Signed-off-by: dingning<dingning7@xiaomi.com>
Signed-off-by: dingning <dingning7@xiaomi.com>
Signed-off-by: dingning <dingning@xiaomi.com>
Co-authored-by: wangyu <53896905+yenuo26@users.noreply.github.com>
Co-authored-by: wangyu31577 <wangyu31577@hundsun.com>
Co-authored-by: Zhang Shijin <zhangshijin@xiaomi.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
Co-authored-by: Sihao Li <111170255+GG-li@users.noreply.github.com>
Co-authored-by: XU Mingshi <91017482+mxuax@users.noreply.github.com>
Co-authored-by: Canlin Guo <canlinguosdu@gmail.com>
Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn>
Co-authored-by: JohnJan <wuzhongjian_yewu@cmss.chinamobile.com>
Co-authored-by: WeiQing Chen <40507679+david6666666@users.noreply.github.com>
Co-authored-by: dongbo910220 <32610838+dongbo910220@users.noreply.github.com>
Co-authored-by: Jiangyun Zhu <riverclouds.zhu@qq.com>
Co-authored-by: Junhong Liu <ljh_lbj@163.com>
Co-authored-by: TJian <tunjian.tan@embeddedllm.com>
Co-authored-by: shijin zhang <zsj1364226740@gmail.com>
Co-authored-by: Zhou Taichang <tzhouam@connect.ust.hk>
Co-authored-by: root <root@hk01dgx028.cm.cluster>
Co-authored-by: Prajwal A <34590600+LawJarp-A@users.noreply.github.com>
Co-authored-by: Shijin Zhang <75300765+Dovis01@users.noreply.github.com>
Co-authored-by: dingning <dingning7@xiaomi.com>
Co-authored-by: ning ding <nndding@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
This commit is contained in:
23 files changed
+8041

No files matched your search

@@ -0,0 +1,232 @@
# MiMo-Audio Offline Inference
This directory contains an offline demo for running MiMo-Audio models with vLLM Omni. It builds task-specific inputs and generates WAV files or text outputs locally.
## Model Overview
MiMo-Audio provides multiple task variants for audio understanding and generation:
- **tts_sft**: Basic text-to-speech generation from text input.
- **tts_sft_with_instruct**: TTS generation with explicit voice style instructions.
- **tts_sft_with_audio**: TTS generation with audio reference for voice cloning.
- **tts_sft_with_natural_instruction**: TTS generation from natural language descriptions embedded in text.
- **audio_trancribing_sft**: Transcribe audio to text (speech-to-text).
- **audio_understanding_sft**: Understand and analyze audio content with text queries.
- **audio_understanding_sft_with_thinking**: Audio understanding with reasoning chain.
- **spoken_dialogue_sft_multiturn**: Multi-turn spoken dialogue with audio input/output.
- **speech2text_dialogue_sft_multiturn**: Multi-turn dialogue converting speech to text.
- **text_dialogue_sft_multiturn**: Multi-turn text-only dialogue.
## Setup
Please refer to the [stage configuration documentation](https://docs.vllm.ai/projects/vllm-omni/en/latest/configuration/stage_configs/) to configure memory allocation appropriately for your hardware setup.
### Environment Variables
The `MIMO_AUDIO_TOKENIZER_PATH` environment variable is mandatory due to the specialized architecture:
```bash
export MIMO_AUDIO_TOKENIZER_PATH="XiaomiMiMo/MiMo-Audio-Tokenizer"
```
## Quick Start
Run a single sample for basic TTS:
```bash
python3 -u end2end.py \
--stage-configs-path vllm_omni/model_executor/stage_configs/mimo_audio.yaml \
--model-name XiaomiMiMo/MiMo-Audio-7B-Instruct \
--query-type tts_sft
```
Run batch samples for basic TTS:
```bash
python3 -u end2end.py \
--stage-configs-path vllm_omni/model_executor/stage_configs/mimo_audio.yaml \
--model-name XiaomiMiMo/MiMo-Audio-7B-Instruct \
--query-type tts_sft \
--num-prompts {batch_size}
```
When enabling multi-batch processing, if the total number of tokens passed to the next stage exceeds the `max_model_len` value in the `mimo_audio.yaml` configuration file, you must also synchronously update the `max_position_embeddings` value in `MiMo-Audio-7B-Instruct/config.json` to match the modified value.
Generated audio files are saved to `output_audio/` by default. `--num-prompts` also can be used to all tasks below.
## Task Usage
### tts_sft (Basic Text-to-Speech)
Generate speech from text input:
```bash
python3 -u end2end.py \
--stage-configs-path vllm_omni/model_executor/stage_configs/mimo_audio.yaml \
--model-name XiaomiMiMo/MiMo-Audio-7B-Instruct \
--query-type tts_sft \
--text "The weather is so nice today."
```
### tts_sft_with_instruct (TTS with Voice Instructions)
Generate speech with explicit voice style instructions:
```bash
python3 -u end2end.py \
--stage-configs-path vllm_omni/model_executor/stage_configs/mimo_audio.yaml \
--model-name XiaomiMiMo/MiMo-Audio-7B-Instruct \
--query-type tts_sft_with_instruct \
--text "The weather is so nice today." \
--instruct "Speak happily in a child's voice"
```
### tts_sft_with_audio (TTS with Audio Reference)
Generate speech using an audio reference for voice cloning:
```bash
python3 -u end2end.py \
--stage-configs-path vllm_omni/model_executor/stage_configs/mimo_audio.yaml \
--model-name XiaomiMiMo/MiMo-Audio-7B-Instruct \
--query-type tts_sft_with_audio \
--text "The weather is so nice today." \
--audio-path "./spoken_dialogue_assistant_turn_1.wav"
```
### tts_sft_with_natural_instruction (Natural Language TTS)
Generate speech from text containing natural voice descriptions:
```bash
python3 -u end2end.py \
--stage-configs-path vllm_omni/model_executor/stage_configs/mimo_audio.yaml \
--model-name XiaomiMiMo/MiMo-Audio-7B-Instruct \
--query-type tts_sft_with_natural_instruction \
--text "In a panting young male voice, he said: I can't run anymore, wait for me!"
```
### audio_trancribing_sft (Speech-to-Text)
Transcribe audio to text:
```bash
python3 -u end2end.py \
--stage-configs-path vllm_omni/model_executor/stage_configs/mimo_audio.yaml \
--model-name XiaomiMiMo/MiMo-Audio-7B-Instruct \
--query-type audio_trancribing_sft \
--audio-path "./spoken_dialogue_assistant_turn_1.wav"
```
### audio_understanding_sft (Audio Understanding)
Understand and analyze audio content with text queries:
```bash
python3 -u end2end.py \
--stage-configs-path vllm_omni/model_executor/stage_configs/mimo_audio.yaml \
--model-name XiaomiMiMo/MiMo-Audio-7B-Instruct \
--query-type audio_understanding_sft \
--text "Summarize the audio." \
--audio-path "./spoken_dialogue_assistant_turn_1.wav"
```
### audio_understanding_sft_with_thinking (Audio Understanding with Reasoning)
Audio understanding with reasoning chain:
```bash
python3 -u end2end.py \
--stage-configs-path vllm_omni/model_executor/stage_configs/mimo_audio.yaml \
--model-name XiaomiMiMo/MiMo-Audio-7B-Instruct \
--query-type audio_understanding_sft_with_thinking \
--text "Summarize the audio." \
--audio-path "./spoken_dialogue_assistant_turn_1.wav"
```
### spoken_dialogue_sft_multiturn (Multi-turn Spoken Dialogue)
Multi-turn dialogue with audio input and output:
```bash
python3 -u end2end.py \
--stage-configs-path vllm_omni/model_executor/stage_configs/mimo_audio.yaml \
--model-name XiaomiMiMo/MiMo-Audio-7B-Instruct \
--query-type spoken_dialogue_sft_multiturn \
--audio-path "./prompt_speech_zh_m.wav"
```
Note: This task uses hardcoded audio files in the script. The audio files used in examples are available at: https://github.com/XiaomiMiMo/MiMo-Audio/tree/main/examples
### speech2text_dialogue_sft_multiturn (Speech-to-Text Dialogue)
Multi-turn dialogue converting speech to text:
```bash
python3 -u end2end.py \
--stage-configs-path vllm_omni/model_executor/stage_configs/mimo_audio.yaml \
--model-name XiaomiMiMo/MiMo-Audio-7B-Instruct \
--query-type speech2text_dialogue_sft_multiturn
```
Note: This task uses hardcoded audio files and message lists in the script.
### text_dialogue_sft_multiturn (Text Dialogue)
Multi-turn text-only dialogue:
```bash
python3 -u end2end.py \
--stage-configs-path vllm_omni/model_executor/stage_configs/mimo_audio.yaml \
--model-name XiaomiMiMo/MiMo-Audio-7B-Instruct \
--query-type text_dialogue_sft_multiturn
```
Note: This task uses hardcoded message lists in the script.
## Troubleshooting
### Audio dependencies (soundfile, librosa)
This example depends on **soundfile** (read/write WAV) and **librosa** (load audio including MP3). Install the project requirements first:
```bash
pip install -r requirements/common.txt
# or at least: pip install soundfile>=0.13.1 librosa>=0.11.0
```
- **`soundfile` / libsndfile not found**
`soundfile` uses the C library **libsndfile**. On Linux, install the system package before pip:
- Debian/Ubuntu: `sudo apt-get install libsndfile1`
- For development builds: `sudo apt-get install libsndfile1-dev`
- Then: `pip install soundfile`
- **`librosa` fails to load MP3 or reports "No backend available"**
Loading MP3 (e.g. in `spoken_dialogue_sft_multiturn` with `.mp3` files) uses **ffmpeg** as the backend. Install ffmpeg:
- Debian/Ubuntu: `sudo apt-get install ffmpeg`
- macOS: `brew install ffmpeg`
- **`ImportError: No module named 'soundfile'` or `ModuleNotFoundError: ... librosa`**
Ensure you are in the same Python environment where vLLM Omni and the example dependencies are installed, and that `requirements/common.txt` (or the packages above) are installed.
### Tokenizer path
- **`MIMO_AUDIO_TOKENIZER_PATH` not set or model fails to find tokenizer**
Export the tokenizer path before running:
```bash
export MIMO_AUDIO_TOKENIZER_PATH="XiaomiMiMo/MiMo-Audio-Tokenizer"
```
See [Environment Variables](#environment-variables) in Setup.
### Other
- If the model or stage config fails to load, check [stage configuration documentation](https://docs.vllm.ai/projects/vllm-omni/en/latest/configuration/stage_configs/) for memory and GPU settings.
- For errors when reading/writing WAV (e.g. unsupported format), ensure input files are standard WAV/MP3 and that `soundfile` is linked to a working libsndfile (see above).
## Notes
- The script uses default model paths and audio files embedded in `end2end.py`. Update them if your local cache path differs.
- Use `--output-dir` to change the output folder (default: `./output_audio`).
- Use `--num-prompts` to generate multiple prompts in one run (default: 1).
- Audio files used in multi-turn dialogue examples are available at: https://github.com/XiaomiMiMo/MiMo-Audio/tree/main/examples
- The script supports various configuration options for initialization timeouts, batch timeouts, and shared memory thresholds. See `--help` for details.
@@ -0,0 +1,443 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
"""
This example shows how to use vLLM for running offline inference
with the correct prompt format on MiMo-Audio-Omni.
"""
import copy
import json
import os
from typing import NamedTuple
import soundfile as sf
from message_convert import (
get_audio_data,
get_audio_understanding_sft_prompt,
get_s2t_dialogue_sft_multiturn_prompt,
get_spoken_dialogue_sft_multiturn_prompt,
get_text_dialogue_sft_multiturn_prompt,
get_tts_sft_prompt,
to_prompt,
)
from vllm import SamplingParams
from vllm.utils.argparse_utils import FlexibleArgumentParser
from vllm_omni.entrypoints.omni import Omni
from vllm_omni.inputs.data import OmniTokensPrompt
SEED = 42
MAX_CODE2WAV_TOKENS = 18192 # Maximum tokens supported by code2wav model
class QueryResult(NamedTuple):
inputs: dict
limit_mm_per_prompt: dict[str, int]
def get_codes_query_from_json(codes_path: str) -> QueryResult:
with open(codes_path, encoding="utf-8") as f:
data = json.load(f)
if isinstance(data, list):
code_final = data
elif isinstance(data, dict) and "code_final" in data:
code_final = data["code_final"]
else:
raise ValueError(
f"Unsupported codes json format in {codes_path}.\n"
"Expect a JSON list[int] or {{'code_final': list[int]}}."
)
if not isinstance(code_final, list) or not all(isinstance(x, int) for x in code_final):
raise ValueError("code_final must be a list[int].")
if len(code_final) > MAX_CODE2WAV_TOKENS:
print(f"[Warn] code_final len={len(code_final)} > {MAX_CODE2WAV_TOKENS}, truncating.")
code_final = code_final[:MAX_CODE2WAV_TOKENS]
return QueryResult(
inputs=OmniTokensPrompt(
prompt_token_ids=code_final,
multi_modal_data=None,
mm_processor_kwargs=None,
),
limit_mm_per_prompt={},
)
def get_tts_sft(
text="The weather is so nice today.",
instruct=None,
read_text_only=True,
prompt_speech=None,
audio_list=None,
):
res = get_tts_sft_prompt(
text,
instruct=instruct,
read_text_only=read_text_only,
prompt_speech=prompt_speech,
)
prompt = to_prompt(res)
final_prompt = {
"prompt": prompt,
}
if audio_list is not None:
final_prompt.update(
{
"multi_modal_data": {
"audio": audio_list,
},
}
)
return final_prompt
def get_audio_understanding_sft(audio_path, text="", thinking=False, use_sostm=False):
audio_list = []
audio_list.append(get_audio_data(audio_path))
res = get_audio_understanding_sft_prompt(
input_speech=audio_path, input_text=text, thinking=thinking, use_sostm=use_sostm
)
prompt = to_prompt(res)
final_prompt = {
"prompt": prompt,
"multi_modal_data": {
"audio": audio_list,
},
}
return final_prompt
def get_spoken_dialogue_sft_multiturn(message_list, system_prompt=None, ref_audio_path=None, audio_list=None):
res = get_spoken_dialogue_sft_multiturn_prompt(
message_list, system_prompt=system_prompt, prompt_speech=ref_audio_path
)
prompt = to_prompt(res)
final_prompt = {
"prompt": prompt,
"multi_modal_data": {
"audio": audio_list,
},
}
return final_prompt
def get_speech2text_dialogue_sft_multiturn(message_list, thinking=False, audio_list=None):
res = get_s2t_dialogue_sft_multiturn_prompt(
message_list,
thinking=thinking,
)
prompt = to_prompt(res)
final_prompt = {
"prompt": prompt,
"multi_modal_data": {
"audio": audio_list,
},
}
return final_prompt
def get_text_dialogue_sft_multiturn(
message_list,
):
res = get_text_dialogue_sft_multiturn_prompt(
message_list,
)
prompt = to_prompt(res)
final_prompt = {
"prompt": prompt,
}
return final_prompt
query_map = {
"tts_sft": get_tts_sft,
"tts_sft_with_instruct": get_tts_sft,
"tts_sft_with_audio": get_tts_sft,
"tts_sft_with_natural_instruction": get_tts_sft,
"audio_trancribing_sft": get_audio_understanding_sft,
"audio_understanding_sft": get_audio_understanding_sft,
"audio_understanding_sft_with_thinking": get_audio_understanding_sft,
"spoken_dialogue_sft_multiturn": get_spoken_dialogue_sft_multiturn,
"speech2text_dialogue_sft_multiturn": get_speech2text_dialogue_sft_multiturn,
"text_dialogue_sft_multiturn": get_text_dialogue_sft_multiturn,
}
def main(args):
model_name = args.model_name
# Get paths from args
text = getattr(args, "text", None)
audio_path = getattr(args, "audio_path", None)
instruct = getattr(args, "instruct", None)
# Get the query function and call it with appropriate parameters
query_func = query_map[args.query_type]
omni_llm = Omni(
model=model_name,
stage_configs_path=args.stage_configs_path,
log_stats=args.enable_stats,
log_file=("omni_llm_pipeline.log" if args.enable_stats else None),
init_sleep_seconds=args.init_sleep_seconds,
batch_timeout=args.batch_timeout,
init_timeout=args.init_timeout,
shm_threshold_bytes=args.shm_threshold_bytes,
)
thinker_sampling_params = SamplingParams(
temperature=0.0,
top_p=1.0,
top_k=-1,
max_tokens=2048,
seed=SEED,
logit_bias={},
repetition_penalty=1.1,
)
code2wav_sampling_params = SamplingParams(
temperature=0.0,
top_p=1.0,
top_k=-1,
max_tokens=4096 * 16,
seed=SEED,
detokenize=True,
repetition_penalty=1.1,
)
sampling_params_list = [
thinker_sampling_params,
code2wav_sampling_params,
]
# Build query result based on query type
# Notice: The audio files used in this example are available at: https://github.com/XiaomiMiMo/MiMo-Audio/tree/main/examples
if args.query_type == "tts_sft":
# python3 -u end2end.py --stage-configs-path ${config_file} --model ${MODEL_PATH} --query-type tts_sft
query_result = query_func(text=text, read_text_only=True)
elif args.query_type == "tts_sft_with_instruct":
# python3 -u end2end.py --stage-configs-path ${config_file} --model ${MODEL_PATH} --query-type tts_sft_with_instruct --instruct "Speak happily in a child's voice"
query_result = query_func(text=text, instruct=instruct, read_text_only=True)
elif args.query_type == "tts_sft_with_audio":
# python3 -u end2end.py --stage-configs-path ${config_file} --model ${MODEL_PATH} --query-type tts_sft_with_audio --audio_path "./spoken_dialogue_assistant_turn_1.wav"
audio_list = [get_audio_data(audio_path)]
query_result = query_func(text=text, read_text_only=True, prompt_speech=audio_path, audio_list=audio_list)
elif args.query_type == "tts_sft_with_natural_instruction":
# python3 -u end2end.py --stage-configs-path ${config_file} --model ${MODEL_PATH} --query-type tts_sft_with_natural_instruction --text "In a panting young male voice, he said: I can't run anymore, wait for me!"
query_result = query_func(text=text, read_text_only=False)
elif args.query_type == "audio_trancribing_sft":
# python3 -u end2end.py --stage-configs-path ${config_file} --model ${MODEL_PATH} --query-type audio_trancribing_sft --audio_path "./spoken_dialogue_assistant_turn_1.wav"
audio_path = "spoken_dialogue_assistant_turn_1.wav"
text = "Please transcribe this audio and repeat it once."
query_result = query_func(text=text, audio_path=audio_path, use_sostm=True)
elif args.query_type == "audio_understanding_sft":
# python3 -u end2end.py --stage-configs-path ${config_file} --model ${MODEL_PATH} --query-type audio_understanding_sft --text "Summarize the audio." --audio_path "./spoken_dialogue_assistant_turn_1.wav"
query_result = query_func(text=text, audio_path=audio_path)
elif args.query_type == "audio_understanding_sft_with_thinking":
# python3 -u end2end.py --stage-configs-path ${config_file} --model ${MODEL_PATH} --query-type audio_understanding_sft_with_thinking --text "Summarize the audio." --audio_path "./spoken_dialogue_assistant_turn_1.wav"
query_result = query_func(text=text, audio_path=audio_path, thinking=True)
elif args.query_type == "spoken_dialogue_sft_multiturn":
# python3 -u end2end.py --stage-configs-path ${config_file} --model ${MODEL_PATH} --query-type spoken_dialogue_sft_multiturn --audio_path "./prompt_speech_zh_m.wav"
first_turn_text_response = "我没办法获取实时的天气信息。不过呢,你可以试试几个方法来查看今天的天气。首先,你可以用手机自带的天气功能,比如苹果手机的天气应用,或者直接在系统设置里查看。其次,你也可以用一些专业的天气服务,像是国外的AccuWeather、Weather.com,或者国内的中国天气网、墨迹天气等等。再有就是,你还可以在谷歌或者百度里直接搜索你所在的城市加上天气这两个字。如果你能告诉我你所在的城市,我也可以帮你分析一下历史天气趋势,不过最新的数据还是需要你通过官方渠道去获取哦。"
audio_list = []
s1_audio_path = "weather_of_today.mp3"
s2_audio_path = "spoken_dialogue_assistant_turn_1.wav"
s3_audio_path = "beijing.mp3"
audio_list.append(get_audio_data(audio_path))
audio_list.append(get_audio_data(s1_audio_path))
audio_list.append(get_audio_data(s2_audio_path))
audio_list.append(get_audio_data(s3_audio_path))
message_list = [
{"role": "user", "content": s1_audio_path},
{"role": "assistant", "content": {"text": first_turn_text_response, "audio": s2_audio_path}},
{"role": "user", "content": s3_audio_path},
]
query_result = query_func(message_list, system_prompt=None, ref_audio_path=audio_path, audio_list=audio_list)
elif args.query_type == "speech2text_dialogue_sft_multiturn":
# python3 -u end2end.py --stage-configs-path ${config_file} --model ${MODEL_PATH} --query-type speech2text_dialogue_sft_multiturn
s1_audio_path = "weather_of_today.mp3"
s2_audio_path = "beijing.mp3"
audio_list = []
audio_list.append(get_audio_data(s1_audio_path))
audio_list.append(get_audio_data(s2_audio_path))
message_list = [
{"role": "user", "content": s1_audio_path},
{
"role": "assistant",
"content": "Hello, I can't get real-time weather information. If you can tell me your city, I can help you analyze historical weather trends, but for the latest data, you'll need to get it through official channels.",
},
{"role": "user", "content": s2_audio_path},
]
query_result = query_func(message_list, thinking=True, audio_list=audio_list)
elif args.query_type == "text_dialogue_sft_multiturn":
# python3 -u end2end.py --stage-configs-path ${config_file} --model ${MODEL_PATH} --query-type text_dialogue_sft_multiturn
message_list = [
{"role": "user", "content": "Could you recommend some tourist attractions in China?"},
{"role": "assistant", "content": "Hello, which city would you like to travel to?"},
{"role": "user", "content": "Beijing"},
]
query_result = query_func(message_list=message_list)
else:
raise ValueError(f"Invalid query type: {args.query_type}")
prompts = [copy.deepcopy(query_result) for _ in range(args.num_prompts)]
print("prompts", prompts)
omni_outputs = omni_llm.generate(prompts, sampling_params_list)
output_dir = args.output_dir if getattr(args, "output_dir", None) else args.output_wav
if args.query_type is not None:
output_dir = os.path.join(output_dir, args.query_type)
os.makedirs(output_dir, exist_ok=True)
for stage_outputs in omni_outputs:
if stage_outputs.final_output_type == "text":
for output in stage_outputs.request_output:
request_id = output.request_id
text_output = output.outputs[0].text
# Save aligned text file per request
prompt_text = output.prompt
out_txt = os.path.join(output_dir, f"{request_id}.txt")
lines = []
lines.append("Prompt:\n")
lines.append(str(prompt_text) + "\n")
lines.append("vllm_text_output:\n")
lines.append(str(text_output).strip() + "\n")
try:
with open(out_txt, "w", encoding="utf-8") as f:
print("lines", lines)
f.writelines(lines)
except Exception as e:
print(f"[Warn] Failed writing text file {out_txt}: {e}")
print(f"Request ID: {request_id}, Text saved to {out_txt}\n")
elif stage_outputs.final_output_type == "audio":
for output in stage_outputs.request_output:
request_id = output.request_id
audio_tensor = output.outputs[0].multimodal_output.get("audio")
if audio_tensor is None:
continue
output_wav = os.path.join(output_dir, f"{request_id}.wav")
# Convert to numpy array and ensure correct format
audio_numpy = audio_tensor.float().detach().cpu().numpy()
# Ensure audio is 1D (flatten if needed)
if audio_numpy.ndim > 1:
audio_numpy = audio_numpy.flatten()
# Save audio file with explicit WAV format
sf.write(output_wav, audio_numpy, samplerate=24000, format="WAV")
print(f"Request ID: {request_id}, Audio saved to {output_wav}")
def parse_args():
parser = FlexibleArgumentParser(description="Demo on using vLLM for offline inference with audio language models")
parser.add_argument(
"--model-name",
"-m",
type=str,
default="XiaomiMiMo/MiMo-Audio-7B-Instruct",
help="Backbone LLM path.",
)
parser.add_argument(
"--text",
"-t",
type=str,
default="The weather is so nice today.",
help="input text",
)
parser.add_argument(
"--query-type",
"-q",
type=str,
default="tts_sft",
choices=query_map.keys(),
help="Query type.",
)
parser.add_argument(
"--audio-path",
"-a",
type=str,
default=None,
help="Path to local audio file. If not provided, uses default audio asset.",
)
parser.add_argument(
"--instruct",
type=str,
default=None,
help="instruct",
)
parser.add_argument(
"--enable-stats",
action="store_true",
help="Enable writing detailed statistics (default: disabled)",
)
parser.add_argument(
"--init-sleep-seconds",
type=int,
default=20,
help="Sleep seconds after starting each stage process to allow initialization (default: 20)",
)
parser.add_argument(
"--batch-timeout",
type=int,
default=5,
help="Timeout for batching in seconds (default: 5)",
)
parser.add_argument(
"--init-timeout",
type=int,
default=5000,
help="Timeout for initializing stages in seconds (default: 5000)",
)
parser.add_argument(
"--shm-threshold-bytes",
type=int,
default=65536,
help="Threshold for using shared memory in bytes (default: 65536)",
)
parser.add_argument(
"--output-dir",
default="./output_audio",
help="Output audio wav directory.",
)
parser.add_argument(
"--output-wav",
default="output_audio",
help="[Deprecated] Output wav directory (use --output-dir).",
)
parser.add_argument(
"--num-prompts",
type=int,
default=1,
help="Number of prompts to generate.",
)
parser.add_argument(
"--sampling-rate",
type=int,
default=24000,
help="Sampling rate for audio.",
)
parser.add_argument(
"--stage-configs-path",
type=str,
default="../../../model_executor/stage_configs/mimo_audio.yaml",
help="Path to a stage configs file.",
)
return parser.parse_args()
if __name__ == "__main__":
args = parse_args()
main(args)
@@ -0,0 +1,56 @@
{
"messages": [
{
"role": "system",
"content": [
{
"type": "text",
"text": "Your voice should be:"
},
{
"type": "audio_url",
"audio_url": {
"url": "data:audio/wav;base64,UklGRiRKAwBXQVZFZm10IBAAAAABAAEAwF0AAIC7AAACABAAZGF0YQBKAwAFAP3/CAD7/wkAAAD9/wgA+v8JAP//AQADAP//CAD+/wEABAABAP7/BgD8//7/BgD0/wYA/f/8/wMA9f8GAP///P8DAPz/AAAFAP3//v8CAP7////+////+/8IAP//AQAAAAMABgD6/wUA+v8DAAcA+/8AAAMAAAAAAP3//v8DAAMACgD8/wQACQD9/wUA/f8BAAcABQADAAAABAADAAEA+v8AAAIAAAAFAPz//f8DAAAA/f8AAAMA/v8FAP7/9v8GAP7//f8EAAAAAQAIAPv/+/8AAP3////+//3/AAAHAPr//v8BAPz/AwD///n/AAACAPr//f/8//7/AAD//wAA+f8GAPz/9v8CAPX/BgACAPv/AQD//wQA///7/wAAAAAHAAQA/f8FAAAAAgAAAPv/BAAGAAAABgACAP7/DQD7//3/BAD//wkA//8DAPz/AwAHAPf/AwAAAAAABAD7//f/AwD8/wAA/f/5/wQA/P8AAPr//P8AAAAA//8AAP////8DAPL/AQD9//r/CwD4/wEAAgD9//7//P/9//7/AAD//wAA/v8DAPr/+v8CAPr///8CAPz///8GAPr/AAAEAP//AgADAP//AAAHAPz/BgD//wUACwD//wYAAgABAAQABQD8/wgABgACAAUA//8CAAQAAAADAP7/AQAIAPr/AwD+//v/AgACAPv/AQAJAPf/AQD+//n/AgAAAP///f8CAAAA/P8DAPf/+/8HAPv///////7//f8AAP3//P8BAPr/AAD6//z/AAD5/wIAAAD7/wIAAgD4/wAA/v/6/wQAAQAAAAAABwAAAAAAAwD+/wIABQADAAEABAADAAAAAwABAAAABAAGAAEAAgAGAAAABQD///7/BQAAAAMAAQAAAAIABAD8/wMA/////wQA/f8AAAEAAQD//wMA+/8FAPr///////P/CwDz/wQA/P/6/wIA8/8AAPH//P/+//f//v/+//n//P/8//L//v/7//r////7//3//v/5//v/AAD5/wMAAAD8/wYA/P///////////wAABwD6/wwAAQD+/wUA/P8GAP7/AwD+/wUABwD+/wUA//8FAAMAAgD//wUAAAAEAAcA+/8KAAUAAgAEAAAA//8HAAAA+v8EAAIAAQAEAPz/+/8EAP7//////wAAAQABAP//+///////AAD9/wEA/f8DAAAA+P8AAPj/AQD+//v/AQD8/wIA/v/6/wAA/////wIA/P///wMA+/8BAPj///8CAAAAAwD+/wcAAAAFAP3/AAAFAP7/AwABAP7/AAACAPv/AwD+/wMABAD9/wUA/v8AAAEA/v8AAAIAAQAAAAAAAAD8/wIA/v/8/wMAAAD///3/AAD5//7/AwD2/wgAAAD8/wQA+f8CAP7/AAAAAP//AwD//wAA/v//////AAAAAAAAAwD9/wIA+//9/wQA+/8DAAAA//8AAP///f/+/wAAAAABAP//BgD8/wIAAgD3/wwA//8AAAkA/f8EAAAA//8BAAAAAwADAP//BQD///z/AwD8/wEAAwAAAAEAAQAAAAAA/v8BAAAAAQAGAP7/AAD///3//v8AAAAAAgAGAP//AgAAAP3////9/wAAAQACAAAAAQD///3/AQD9/wAABwD//wMABAD8/wEAAwD+/wIABQD+/wEAAwD6////BAD5/wYAAAD+/wgA+f8CAP3///8DAP7/AgAAAAAA/v8EAPv/AAAHAPv/CQACAP7/BgD//wIAAAADAAAAAgAEAPz/BQD8/wQA/P/+/wMA+f8EAP3//P8AAP////8AAPv/AgD8/wAAAgDz/wgA/f/7/wQA+P8BAAAA+v8BAP7/AAACAP3/AAACAP7/AQAEAPj/BwAAAAAABgD7/wcA+v8CAP7/+/8IAP3/AQACAAAAAQACAAAA/v8CAAcA/f8HAAIA/v8IAPv/AQAAAAAABAD+/wQAAgD//wMA/P8AAAAA//8BAPr/BQD9/wAA///4/wIA+/8AAP3//v8EAPr/AAD9//r/BQD3/wEA+v/6/wYA8/8BAPz/9v8CAPn/+f/+//3//v/+//3//v///wAA/P/7/wAAAAD//wAA//8AAP7/AgD4////BAD7/wYAAAADAAAAAwAAAPr/BQD6/wUABAD//wcA/f8EAPz//P8FAPv/BQADAP//BgAAAP//AQAAAAAAAgAAAAEAAwAAAAAA//8AAAAAAAAAAP7/BAD///7/AgD8/wAAAQD8////AQD9/wQA/v///wIA//8BAPv/AQD9/wMA/v/8/wQA+f8DAP3/9/8EAAAA/P8GAPz/AwAEAP//AAD8/wQAAAAGAAAAAgAFAAEABAD8/wEAAQADAAAAAgADAAAACAD7//7/BQD9/wUAAAD+/wcAAAABAAAA/f8DAAMAAAAAAAEA/v8AAAAA+f8AAAEA//8AAP//AAD7/wUA9//6/wQA9/8BAP3//f8AAP7/AAD7//3/AwD+//v//v/7//3/BAD3//r/CAD8/wIA/v/6//3/AQD+//j/AAAAAAgAAAD///v//f8BAP7//v/9/wEA//8EAAAA//8BAAEAAAD+/////f8AAP7/AgD9/wQABQD4/wIA/f/5/wAA/P/9/wEAAQABAPf/AAD8//7/AQD5/wAA/f8BAPf/+f/7//b/AwADAPz///8BAPn//P/4/////P8AAAAA+P8JAAAA/f/8//3/AAABAAAA8v8EAAMAAAACAPn/BAAOAAcA+//t////DwAEAAIA/f8HAAYADAD7//f/BgD//wwA//8BAAEADwAEAPv/BQD1/xEA//8DAP//9/8LAPn/AgABAAMABAAEAPz//f8FAPb/BAD+//r/CwAAAP7/BQD4////BAD4/wEA/v8AAAAA/v/+//z/BAABAP//BQD+//z/AQD7/wAA/P8DAAUAAAAHAP//AQD9////AAABAP//AAACAAQACAD+/wYA//8DAAcA/v8CAAEABQADAAIAAgACAAUABQAAAAAAAgAAAAAAAwABAAAABQAAAAYABgAAAAEA/v8EAAAA/P8GAP7/BAAGAPn/AQABAAIA/f8EAPn///8GAPT/BgD5/wAAAgD8/wQA+v8BAPv/9/8AAP3/AAAAAPz/AQD9//r////6////AAD9/////f8AAPz/AAD9//z/CQD5/wUA+v/7/wIA9P8FAPn/AQADAP3//v8AAP3/+/8DAPr/AQABAAAA/f8BAAAA9/8EAP7///8GAPr///8BAPz/AQD9/wAABQD//wEAAAD9/wMAAQD7/wAAAwAAAAMAAAD+/wMAAQAAAP3/AAABAP//AgD+/wEAAQAHAPv///8EAPb/CQD//wAAAAAAAAUA9v8JAPf///8FAPr/AwD9/wIA+v8DAP//+P8GAP////8EAPv/+/8EAPv/AAD+//7/AQD//wEA+/8AAAAA/v8CAP///P8GAP////8DAPv/BgAAAAAAAgD//wEA//8CAP7/AgAAAP//BgAAAAEAAgAAAP//BwD6/wIABAD6/woA+/8EAAAAAAAEAPv/BgD//wIAAAABAAMAAAAHAPf/BwAAAPr/BwD4/wQAAAAAAAAAAAD//wAAAAD7/wcA/v8AAAAA///+/wIA///7/wkA+/8GAAAA/f8BAP//AgD9/wYA//8FAAQA//8CAPz/AQAFAPr/AwAAAP//BgD5//7//v8CAPv/AAD9//b/CAD4//z//v/6/wAAAAD+//z/AAABAP///v////3/BwABAAAAAAAAAAUA/f8DAPb/AAAIAPv/AwD9////BAAAAPf/+/8AAPv/AQD7//v/AAABAP7/+v8BAPn/AQACAPr/AQACAAEA/f8FAPv/AAAFAPj/AAABAPv//P/+//n//v////r//f/+//r/AAD0//b//f/0/wMA9//+//7//f8BAPP/BAD6////AwD8/wEAAAAKAPf/BwACAPz/DAD9//3/AwAGAPn/BwD+//3/DgD7/wAAAAD7/wYA/f/5/wQA+v8GAAgA+P8IAAMAAwAKAP//AQAFAAMABQAAAAUABQAGAAcA//8IAPr/AgAAAPX/BAD+//3/BAAAAPX/AwD5//T/AwD3//n/BgD6//z/BAD3/wAAAAD+/wAA//8AAAAA/v/+//7///8BAAAAAAABAP7//v/+//b/AAD8/wAAAAD+/wEA/f8CAPr/AQD7/wAABQD7/wcA//8FAAcAAgAIAAUABAAGAAcAAwAGAAQABAAKAAkABQAIAAUABwADAAMAAAACAAMAAgAFAP7/DAABAAgA/v/+/wYA/P8KAPj/BAAGAP7/CQAAAAEACAAEAAUA/P///wIA+/8DAP7///8FAP//AAD7/wAA/f/2/wIA9P/9/wMA8v///wAA+/8DAAEA/P8CAP7//v/5//r/CAAAAAkABgAAAAUABwD+//3/CwD//w4ABwAAAAkAAgAJAPz/BwALAAEAEgD//wAACgABAAcAAgAFAAcABwAGAAAABAAEAAoAAwAIAAsABQAMAAAAAQACAAkABwAGAA4ACwAKABMAAgAAAAoA/f8JAAAABAADAAsACQD9/wQA+/8AAPf//v/z//r/AQDv//3/8f/w//r/8f/0/+//9P/u/+j/8v/g/+7/8f/r//X/8f/x/+7/7v/m/+v/7v/3//D/8//4/+3//v/z//b/9//7//n/9f8BAPT//P8AAPj/AAADAAAAAAD+/wIA+f/8/wQA+v8DAAcA/v8AAA8AAAAFABEA//8NAA4ABwAHABAAEgAKABgAFAARABYAFAARABIAHAAXABIAIAAYABQAIgAUABEAIQAXABUAGgAVABMAFwAVAAgADQAJAAkABAAJAAkAAAAKAPz/+f/3//z/8P/2/wAA8/8CAP7/8//0//j/7v/t/Line truncated
}
}
]
},
{
"role": "user",
"content": [
{
"type": "audio_url",
"audio_url": {
"url": "data:audio/wav;base64,UklGRgrVAABXQVZFZm10IBAAAAABAAEAwF0AAIC7AAACABAAZGF0YebUAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA//8AAAAAAAAAAAAAAAAAAAAAAAAAAP////8AAAAAAAAAAAAAAAD//wAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAP////8AAAAAAAAAAAAA///p/9T/xv/G/8b/yf/K/8r/zP/M/83/zf/O/83/zf/P/8//z//O/87/z//P/8//z//P/8//z//P/8//z//P/8//z//Q/9D/0P/Q/9D/0P/R/9H/0f/R/9H/0f/R/9H/0f/R/9L/0v/S/9L/0v/S/9P/0//T/9P/0//T/9P/0//U/9T/1P/U/9T/1P/V/9T/1f/V/9X/1f/W/9b/1v/W/9b/1v/W/9b/1//X/9f/1//Y/9j/2P/Y/9j/2P/Z/9n/2f/Z/9n/2f/a/9r/2v/a/9v/2//b/9v/2//b/9v/3P/c/9z/3f/c/93/3f/d/93/3f/e/97/3v/e/9//3//f/9//4P/g/+D/4P/g/+D/4P/h/+H/4f/h/+H/4f/i/+L/4v/j/+P/4//j/+P/4//k/+P/5P/k/+T/5P/l/+X/5f/l/+X/5f/l/+X/5f/l/+X/5v/n/+f/5//o/+j/6P/o/+j/5v/n/+f/5//o/+j/6P/o/+n/6f/p/+j/6P/o/+j/6P/p/+n/6f/p/+n/6v/q/+r/6v/q/+r/6v/q/+r/6v/q/+r/6//r/+r/6v/r/+r/6v/p/+r/6v/r/+r/6v/r/+z/7P/s/+v/7P/r/+v/6//s/+v/6//r/+v/6//t/+z/7f/t/+3/7f/r/+z/6//s/+v/7P/s/+z/7P/s/+z/7P/t/+3/7f/s/+z/6//s/+r/6//q/+z/7P/t/+3/7v/u/+3/7f/t/+z/7P/s/+z/7P/t/+3/7f/t/+3/7f/t/+3/7f/t/+z/7P/s/+z/7f/t/+3/7f/u/+3/7f/t/+3/7f/t/+3/7f/u/+3/7f/t/+3/7f/t/+3/7f/t/+3/7P/r/+z/6//t/+z/7f/t/+7/7f/u/+7/7f/t/+z/7P/t/+z/7P/t/+3/7P/t/+7/7v/v/+7/7v/t/+z/7P/t/+3/7v/u/+7/7v/v/+//7v/u/+3/7P/u/+z/7P/s/+z/7f/u/+7/7v/v/+7/7v/s/+z/7P/t/+3/7f/u/+7/7f/u/+3/7v/t/+3/7f/t/+3/7P/t/+z/7f/u/+3/7//u/+7/7f/t/+3/7f/u/+//7v/v/+7/7v/t/+7/7f/v/+7/7v/t/+7/7f/t/+3/7v/u/+7/7v/v/+3/7v/t/+z/7P/r/+z/6//t/+r/7P/t/+7/7v/u/+7/7f/s/+z/6//s/+v/7P/t/+3/7//w//D/7v/w/+7/7v/s/+3/6//t/+v/7f/u/+//7//u/+//7v/u/+r/6//q/+v/6//s/+7/7v/w/+//7//v/+3/7f/s/+v/7P/s/+3/7f/u/+7/7v/u/+3/7P/r/+z/7f/s/+z/7P/t/+3/7P/t/+3/7v/s/+3/7f/t/+z/6//r/+z/7P/s/+z/7P/t/+z/7f/t/+z/7v/v/+7/7P/s/+z/6//u/+3/7v/u/+7/7f/u/+3/7f/t/+v/7f/r/+v/6//s/+3/7P/u/+7/7v/v/+7/7v/u/+z/7P/r/+3/6//t/+3/7v/t/+7/7v/v/+7/7v/t/+3/7P/t/+7/7v/v/+3/7v/t/+3/7f/s/+z/7f/s/+3/7f/t/+7/7v/u/+3/7//u/+7/7f/s/+z/7P/s/+v/7P/t/+3/7f/v/+7/7v/u/+3/7f/t/+3/7f/u/+7/7v/u/+7/7//u/+7/7v/t/+3/7f/t/+3/7f/u/+//7//v/+//7v/u/+7/7v/u/+7/7v/u/+7/7v/t/+7/7v/u/+7/7v/u/+7/7f/t/+3/7f/t/+7/7f/t/+3/7P/u/+3/7v/t/+7/7v/u/+7/7v/v/+7/7v/v/+7/7v/u/+7/7v/u/+3/7f/u/+3/7v/u/+7/7f/u/+3/7v/u/+7/7f/t/+7/7v/v/+7/7v/u/+3/7v/t/+3/7f/t/+3/7f/u/+7/7v/u/+3/7v/t/+3/7P/t/+3/7f/u/+3/7v/u/+//7v/u/+7/7v/u/+7/7v/u/+7/7f/u/+3/7f/u/+7/7f/u/+3/7f/t/+3/7f/t/+3/7f/s/+z/7f/t/+3/7f/t/+3/7P/t/+3/7f/t/+z/7P/s/+3/7P/t/+3/7f/u/+3/7f/t/+3/7f/t/+7/7P/t/+3/7f/t/+3/7f/t/+3/7f/t/+3/7f/t/+3/7f/t/+3/7f/s/+z/7f/t/+3/7f/t/+3/7f/t/+z/7P/t/+3/7f/t/+3/7f/t/+3/7f/t/+3/7f/t/+z/7P/s/+3/7f/t/+3/7f/t/+3/7P/s/+z/7P/t/+z/7P/s/+z/7P/s/+3/7P/t/+3/7f/t/+3/7f/t/+z/7P/t/+z/7P/s/+z/7P/s/+z/7P/t/+z/7P/t/+z/7P/s/+z/7P/s/+z/7P/t/+z/7P/s/+z/7P/s/+z/7P/s/+z/7P/s/+z/7P/s/+z/7P/s/+z/7P/s/+v/6//s/+z/7P/s/+z/7P/s/+z/7P/s/+z/7P/s/+z/7P/s/+z/7f/s/+z/7P/s/+z/7P/s/+v/6//r/+z/6//s/+z/7P/s/+z/7P/s/+z/7P/s/+z/6//s/+z/7P/s/+v/6//s/+z/7P/s/+z/7P/s/+z/7P/s/+z/7P/s/+z/7P/s/+z/7P/s/+z/7P/s/+z/7P/s/+z/7P/s/+z/7P/t/+z/7P/s/+z/7P/s/+z/7P/s/+z/7P/s/+3/7P/s/+z/7P/s/+z/7P/s/+z/7P/s/+z/7P/s/+z/7P/t/+z/7P/s/+z/7P/t/+z/7P/s/+z/7P/t/+z/7P/s/+z/7P/s/+z/7P/s/+z/7f/t/+3/7f/t/+3/7f/t/+3/7f/t/+3/7f/t/+3/7f/s/+3/7P/t/+3/7P/s/+z/7P/t/+z/7f/t/+3/7f/t/+3/7f/u/+z/7f/t/+3/7f/t/+3/7f/u/+3/7f/t/+z/7P/s/+z/7P/t/+z/7v/t/+7/7f/t/+7/7f/v/+z/7f/r/+3/7P/t/+3/7f/u/+7/7v/v/+7/7v/u/+z/7f/s/+7/7f/u/+3/7f/v/+7/7//u/+7/7//t/+//7v/u/+z/7P/s/+7/7f/w/+3/7f/u/+//7f/u/+7/7//s/+//6//w/+3/7//u/+3/7v/s/+7/7P/u/+//7v/x/+z/7//t/+3/7f/u/+//7//u/+7/6//w/+3/8f/t/+//7v/v/+7/7v/u/+//7v/w/+3/7v/t/+//7//t/+7/7v/w/+3/7P/v/+3/7//t/+//7v/w/+7/7//y/+3/7v/v/+3/7//t/+7/7P/v/+3/9P/w/+3/6//r//D/7//w/+3/8v/t/+7/7f/v/+3/7f/v//D/7f/t//D/7v/u/+3/8P/v/+7/7v/w/+z/7//r//L/7v/x/+3/7v/u/+7/7//t/+//7f/v/+7/7f/v/+7/7f/u/+7/7f/v/+//8P/u/+3/7//u/+//7f/v/+z/7f/v/+3/7v/p//D/7f/x/+7/7v/s/+3/8P/v/+//7f/r/+7/7P/t/+3/7v/t/+3/7f/u/+7/7//t/+7/7f/u/+3/6//s/+z/7f/t/+3/7//s/+//7f/y/+z/7v/s/+3/7P/v/+3/6//r/+3/7P/u/+//7f/t/+7/7v/v/+z/7f/r/+//7f/w/+z/7f/v/+v/7f/s/+7/7v/v/+z/7P/u/+z/7//s/+3/7f/u/+7/7f/w/+z/7v/t/+3/7v/t/+3/7P/u/+7/7v/s/+z/7f/v/+3/7v/t/+3/7f/t/+7/7P/t/+3/7v/v/+z/7f/u/+3/7v/v/+z/7v/t/+//7f/u/Line truncated
}
}
]
},
{
"role": "assistant",
"content": [
{
"type": "text",
"text": "我没办法获取实时的天气信息。不过呢,你可以试试几个方法来查看今天的天气。首先,你可以用手机自带的天气功能,比如苹果手机的天气应用,或者直接在系统设置里查看。其次,你也可以用一些专业的天气服务,像是国外的AccuWeather、Weather.com,或者国内的中国天气网、墨迹天气等等。再有就是,你还可以在谷歌或者百度里直接搜索你所在的城市加上天气这两个字。如果你能告诉我你所在的城市,我也可以帮你分析一下历史天气趋势,不过最新的数据还是需要你通过官方渠道去获取哦。"
},
{
"type": "audio_url",
"audio_url": {
"url": "data:audio/wav;base64,UklGRiQiKQBXQVZFZm10IBAAAAABAAEAwF0AAIC7AAACABAAZGF0YQAiKQAGAAcABwAIAAkACQAJAAoACwAMAAwADQAOAA4ADwAPABAAEAAQABAAEQAQABAAEQARABAAEQARABEAEAAQABEAEQAQABAAEAAPAA4ADwAOAA0ADQANAAwACwAJAAYABQAEAAIAAQAAAP7//f/7//r/+v/5//b/9v/0//X/9f/0//X/9P/0//L/9P/y//D/9P/z//T/8v/x//L/8v/x//H/8P/v/+//7v/u/+3/7P/t/+3/7P/t/+7/7v/u//D/8f/z//X/9f/3//n/+//7//3///8AAAEABAAGAAgACgAMAA4AEAASABMAFAATABQAFAAUABUAFQATABIAEgARABEAEAAOAAwACwAJAAcABgAEAAMAAgABAAEAAAAAAAAA//////7//f/8//v/+f/4//j/9//3//f/9v/2//b/9f/1//b/9f/1//b/9v/2//b/9f/1//X/9P/z//P/9P/z//P/8//0//T/9P/0//X/9v/2//f/+P/4//n/+v/7//z//f/+//7///8AAAEAAgACAAMABAAFAAYABwAJAAoADAANAA4ADwAPAA8AEAAQABEAEgASABMAFAAVABYAFwAXABYAFgAWABYAFgAVABQAEwATABMAEgASABIAEQARABAADwAOAA4ADQANAAsACgAIAAcABAACAAEA///9//v/+f/4//b/9f/z//H/8f/v/+3/7P/r/+n/5//m/+X/5f/k/+T/5P/l/+X/5//o/+r/7P/u//D/8v/1//f/+f/7//z//v8AAAEAAgAEAAQABQAHAAUABQAGAAQAAwACAAEAAgAAAAEAAQD+/wAA/v8AAP7//P////7//f/9/wAA/v/9//7//v///wAA//////////8AAAEAAwACAAIAAwADAAMAAgACAAIAAwACAAEAAAABAAEAAAD////////////////+/////v/9//3//P/8//z//f/9//3//v/+//7///8AAP////////////////////////7//v/9//3//f/8//r/+P/3//j/9//3//b/9f/2//b/9f/1//b/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//4//j/+P/5//n/+f/5//n/+f/5//n/+f/6//r/+//7//z//P/+////AAACAAMABQAGAAcACAAIAAkACQAJAAkACQAKAAoACQAJAAcABgAEAAMAAQD///3/+//6//n/+P/3//b/9v/1//b/9v/2//b/9v/2//b/9//4//n/+v/8//7//v///wEAAwAEAAUABwAJAAoACgALAA4ADAANAA0ADwAOAA0ADwAPAA8ADgARABEAEQARABEAEAAQABEAEQARABAAEAAQABEAEQAQABAADwAPAA4ADQAOAAwADQANAAwADQANAAsADAALAAsACgAJAAkACAAHAAcABQAFAAUAAgABAAEAAAD///7//P/6//n/9v/z//P/8v/w/+//7f/r/+v/6v/p/+r/6f/o/+j/6P/o/+j/6f/p/+n/6v/r/+z/7P/t/+7/7//w//H/8//1//b/+P/5//r//P/9////AAABAAIAAwAFAAYABwAIAAkACwALAAwADQAOAA4ADwAQABEAEgASABMAFAAVABUAFQAVABYAFQAVABUAFAATABIAEQAPAA4ADQAMAAsACgAJAAcABQADAAAA///8//r/+P/2//X/8//x/+//7v/t/+z/6//r/+r/6v/q/+v/6//s/+z/7f/t/+7/7//w//H/8v/z//T/9f/2//f/9//4//j/+f/6//r/+//7//z//P/9//7//v///wAAAAAAAAEAAQACAAMABAAGAAcACQAKAAwADQAPABAAEQASABIAEgATABUAFwAYABkAGgAbAB0AHgAfAB4AHgAdABwAGwAaABkAGAAWABQAEgAQAA0ACwAJAAcABQADAAEA/v/8//r/+P/2//T/8//x//D/7//t/+z/6//r/+v/6//r/+v/6//s/+3/7f/t/+7/7v/v/+//8P/w//D/8f/w//D/7//w//H/8f/w//D/8f/y//L/8v/y//P/8//1//f/+P/5//z//f/8/wAA//8DAAEAAAAEAAIAAQADAAMAAQAAAAEAAAABAAEA/////////v/9//7////9//z//P/9//3//P/8//z//f/+//7//v////////8AAAAAAQACAAIAAgACAAIAAgABAAEAAQABAAIAAQAAAAAAAAD///7//v/9//3/+//6//r/+v/6//r/+f/6//r/+v/5//n/+v/7//v/+v/6//v//f/+//7//v//////AAD///////////7//f/8//v/+//6//r/+f/5//n/+v/6//r/+v/6//z//P/9//3///8AAAEAAgADAAUABQAGAAYABwAIAAkACQAKAAsADAAMAAwADAAMAA0ADAAMAAsACwAKAAkACAAHAAUAAwACAAEAAAD+//3/+//6//j/9//1//T/8//y//L/8//z//T/9f/1//b/9v/2//f/+P/4//n/+f/5//r//P/9////AAACAAMABAAFAAYACAAJAAoACwANAA4ADQAOABEAEAARABEAFAATABMAFQAVABYAFQAXABYAFgAWABUAFAATABQAEwASABIAEQAQABAADwAPAA4ADQAMAAsACwAMAAoACwAKAAkACgAJAAgABwAFAAUAAwACAAEA///9//3/+v/5//j/9P/y//D/7v/s/+v/6P/m/+X/4v/h/+H/4f/g/+H/4P/g/+H/4v/j/+T/5f/m/+f/6P/q/+v/7f/u/+//8f/y//T/9f/3//j/+v/7//z//v///wEAAgADAAQABQAGAAgACQAKAAsADAAOAA8AEAASABMAFAAVABYAFgAXABgAGAAZABkAGgAaABoAGQAZABgAFgAVABMAEgAQAA8ADgAMAAoACQAHAAUABAACAAAA///9//v/+f/3//X/8//x//D/7//v/+7/7v/t/+3/7f/t/+7/7v/v//D/8P/x//L/8//0//X/9f/2//f/9//4//j/+f/5//r/+v/6//r/+v/6//r/+f/6//r/+v/7//v//P/8//3//v///wEAAgADAAUABgAIAAsADQAPABEAEwAUABYAFwAYABgAGQAZABkAGAAZABkAGQAZABgAFwAVABQAEgARAA4ACwAIAAUAAwABAP///v/8//v/+v/4//b/9f/0//P/8v/y//H/8P/w/+//7v/u/+3/7f/t/+3/7f/s/+z/7P/t/+3/7v/u/+7/7v/u/+//8P/v//D/8P/x//H/8f/y//P/9P/0//X/9f/1//f/+P/5//n/+//8//z//f/9//7//f///wAAAAAAAAEAAQD//wAA/f8AAPz/+//+//v/+v/8//z/+//7//z//f/+/////v///wEAAQAAAAIABAADAAIAAwADAAQAAwADAAMAAwADAAIAAgACAAEAAAAAAAAAAAD//////v/+//7//v/9//7//v/+//////////////////////8AAAAA////////AAAAAAAAAAAAAAEAAAD///7//////////v/9//3//////////v/+//7//v/+//7//v////////8AAAAAAQACAAQABQAGAAcACAAKAAoACgALAAwADAAMAAwADAAMAAwADAAMAAwACwAKAAkACQAJAAgACAAIAAgABwAHAAYABQAEAAMAAgAAAP///v/9//z/+v/5//f/9f/0//P/8v/x//D/7//u/+3/7f/s/+z/7P/s/+7/7//w//L/8//0//X/9v/3//j/+v/8//3///8BAAMABgAIAAoADAAOABAAEQARABIAEwAUABQAFQAVABYAFAAUABYAFAAUABQAFgAVABMAFQAVABUAFAAVABUAFQAVABQAFAATABQAFAAUABQAFAATABQAFAAUABMAEgASABAAEAAQAA4ADgAMAAkACAAGAAIAAAD8//r/9//0//L/7//t/+v/6f/n/+b/4//h/+D/3v/e/9//3v/e/97/3v/e/+H/4v/j/+b/5v/n/+n/6v/s/+7/7//w//L/8//0//b/9//4//n/+v/7//z//f/+////AAABAAIABAAFAAcACAAJAAoACwANAA4ADwARABIAEwAUABUAFgAXABcAFwAXABcAFgAWABUAFAATABIAEAAPAA0ACwAJAAYABAABAP///f/7//n/9//1//T/8v/w/+//7f/s/+r/6f/o/+f/5v/l/+T/5P/j/+T/5P/l/+X/5v/n/+j/6P/p/+r/6//s/+z/7f/u/+7/7v/v/+//7//v/+//8P/w//H/8f/y//P/9P/1//f/+P/5//v//P/+/wEAAwAFAAcACgALAA4AEAATABUAFwAZABsAHQAfACEAIwAkACUAJgAmACUAJQAkACMAIQAgAB0AGwAaABkAGAAWABQAEgAPAA4ADAALAAgABgAEAAIAAQAAAAAA///+//7//f/8//v/+f/5//j/+P/4//f/9//2//X/9f/0//T/9P/0//X/9f/1//X/9f/2//f/9//4//j/+f/6//v//P/8//3//v///wAAAAABAAIAAwAEAAUABQAGAAcACAAIAAcACAAJAAgACAAIAAgABwAIAAkACQAIAAoACgAIAAkABwAKAAcABgAKAAgABwAKAAoACQAJAAoACwALAAwACgAKAAsACQAIAAgACAAGAAQAAwACAAEA///+//3//P/7//r/+f/4//j/9//3//f/9//3//f/9//3//j/+P/4//j/+P/5//n/+f/5//n/+f/5//j/+f/5//n/+f/4//n/+f/6//r/+f/6//v/+//6//r/+//8//3//P/8//7/AAABAAIAAgADAAQABQAGAAcACAAJALine truncated
}
}
]
},
{
"role": "user",
"content": [
{
"type": "audio_url",
"audio_url": {
"url": "data:audio/wav;base64,UklGRkqYAABXQVZFZm10IBAAAAABAAEAwF0AAIC7AAACABAAZGF0YSaYAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABAAAAAAAAAAAAAAAAAAAAAAAAAP//AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD//wAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAP//AAAAAP//AAAAAAAA///o/9T/xv/G/8b/yP/K/8r/y//M/8z/zf/O/83/zf/P/87/zv/O/87/zv/P/8//z//P/8//z//P/8//z//P/8//z//P/8//0P/Q/8//z//Q/9D/0f/R/9D/0f/R/9H/0f/R/9H/0f/R/9H/0f/S/9L/0v/S/9L/0v/S/9L/0v/T/9P/0//T/9P/0//T/9P/0//T/9P/0//T/9P/0//U/9T/1f/V/9T/1P/U/9X/1P/V/9X/1f/V/9b/1v/W/9b/1v/W/9b/1v/W/9f/1//X/9f/1//X/9f/1//X/9j/2P/Y/9j/2P/Y/9j/2P/Y/9j/2f/Z/9r/2v/a/9r/2v/b/9v/2//b/9v/2//c/9v/3P/c/9z/3P/d/9z/3P/c/93/3f/e/93/3f/e/97/3v/e/97/3//f/+D/4P/g/9//3//g/+D/4P/g/+H/4f/g/+H/4P/h/+H/4f/h/+L/4v/i/+H/4f/i/+L/4v/i/+L/4v/j/+L/4//j/+P/4//j/+P/4//j/+P/5P/k/+T/4//l/+T/5P/k/+T/5P/l/+T/5f/l/+X/5f/m/+X/5v/m/+b/5v/m/+b/5v/m/+b/5v/n/+b/5//n/+b/5v/o/+f/5//n/+j/5//p/+f/5//o/+j/6P/n/+j/6P/p/+n/6f/o/+j/6P/p/+j/6P/o/+n/6P/p/+j/6P/p/+n/6f/q/+n/6v/p/+n/6f/q/+n/6f/p/+n/6f/p/+n/6f/p/+n/6v/p/+n/6f/p/+n/6f/p/+n/6f/r/+r/6v/q/+r/6v/r/+n/6v/p/+r/6v/r/+n/6v/q/+r/6v/q/+r/6v/q/+r/6v/q/+r/6//q/+v/6//r/+r/6//r/+r/6v/q/+r/6v/q/+r/6v/q/+r/6//q/+v/6//s/+r/6//q/+r/6v/s/+n/6v/q/+v/6v/r/+v/6//s/+v/6//s/+v/7P/r/+r/6//s/+z/6//r/+v/7P/r/+z/6//r/+v/6//r/+r/6//r/+v/6//r/+v/6//s/+v/6//r/+z/7P/r/+v/6//s/+v/7P/r/+v/6//r/+v/6//s/+v/6//r/+z/7P/s/+z/7P/r/+z/6//s/+v/6//r/+v/6//s/+v/7P/r/+z/7P/s/+z/6//t/+v/7P/s/+v/7P/r/+z/6//s/+z/7P/r/+z/6//t/+v/7P/s/+v/7P/s/+z/6//s/+z/7P/s/+z/7P/s/+v/6//s/+z/7P/r/+z/7P/r/+z/7P/t/+z/7P/s/+z/7P/s/+z/7P/s/+z/7P/s/+z/7P/s/+z/7P/t/+z/7P/s/+z/7P/s/+z/7P/s/+v/6//s/+z/7P/s/+z/6//r/+z/7P/s/+v/7P/s/+v/7P/s/+v/6//s/+v/7P/s/+z/6//s/+v/7P/q/+v/6//s/+v/6//r/+z/6//r/+z/7P/r/+z/6//s/+z/7P/r/+z/7P/s/+v/6//t/+r/7P/r/+v/6//s/+v/7P/s/+v/7P/r/+z/6//s/+v/6//r/+z/6//r/+v/6//r/+v/6//t/+v/7P/r/+z/6//r/+z/7P/s/+v/6//r/+v/6//s/+v/7P/r/+z/7P/r/+v/6//s/+v/6//r/+z/6//s/+v/6//s/+v/7P/s/+z/7P/r/+v/6//s/+v/7P/r/+z/6//r/+v/6//s/+v/6//r/+z/7P/r/+z/6//s/+v/6//r/+v/6//s/+v/6//s/+r/6//r/+v/6//r/+v/6//r/+v/7P/r/+v/6//s/+v/6//r/+z/6//s/+v/6//q/+v/6//s/+v/6//s/+v/6//s/+v/6//r/+v/6//s/+v/6//r/+z/6//s/+v/6//s/+v/7P/q/+z/6//s/+v/6//s/+v/6//r/+v/6//r/+v/6//r/+v/6//s/+v/6//r/+v/6//s/+v/7P/r/+v/6//r/+v/6//r/+v/6//r/+v/6//s/+v/6//r/+v/6//r/+v/6//s/+r/6//r/+v/6//r/+r/6//r/+v/6//s/+v/6//q/+v/6v/s/+r/6//r/+v/6//r/+v/6//s/+v/6//r/+v/6//r/+v/6//r/+v/6//s/+v/6//r/+r/6//q/+v/6//r/+v/6//r/+v/6//r/+r/6//r/+v/6//r/+v/6//r/+v/6//r/+v/6//r/+v/6//r/+v/6//r/+v/6//r/+v/6//r/+v/6//r/+v/6//r/+v/6//r/+v/6//r/+v/6//r/+v/6//r/+v/6//r/+v/6//r/+v/6//r/+v/6//r/+v/6//r/+v/6//r/+v/6//r/+v/6//r/+v/6//s/+v/6//r/+v/6//r/+v/6//s/+v/6//r/+v/6//r/+v/6//r/+v/6//r/+v/6//r/+v/6//r/+v/6//r/+v/6//r/+v/6//s/+v/6//r/+v/6//r/+v/6//s/+v/6//r/+v/6//s/+v/6//r/+z/6//r/+v/6//r/+v/6//r/+v/6//r/+v/6//s/+v/6//r/+v/6//r/+v/6//r/+v/6//r/+v/6//r/+v/6//r/+v/6//r/+v/6//s/+v/6//r/+v/6//s/+v/6//r/+v/6//r/+v/6//r/+v/6//r/+v/6//r/+v/6//r/+v/6//r/+v/6//s/+v/6//s/+v/6//r/+v/6//s/+v/6//r/+v/6//s/+v/6//s/+v/6//r/+z/6//r/+v/7P/r/+v/6//s/+v/6//q/+v/6//s/+r/6//r/+v/6//s/+v/6//r/+v/6//r/+v/6//r/+v/6//r/+v/6//s/+v/6//r/+v/6//s/+v/6//s/+v/7P/r/+z/6//s/+v/6//r/+v/6//r/+v/6//r/+v/6//s/+v/6//r/+v/6//r/+v/6//r/+v/6//r/+z/6//r/+v/6//s/+v/6//r/+v/6//r/+v/6//s/+v/6//s/+v/7P/r/+v/6//r/+v/6//s/+v/6//r/+v/6//r/+v/6v/s/+v/6//r/+v/6//r/+v/6//s/+v/6//r/+v/6//s/+v/6//s/+z/7P/r/+v/7P/s/+v/6//s/+v/6//r/+v/6//r/+z/7P/s/+v/6//r/+z/6//s/+v/7P/r/+z/6//s/+v/6//s/+v/6//r/+z/6//s/+v/6//s/+v/7P/r/+v/6//s/+v/7P/r/+v/6//s/+v/7P/s/+v/7P/r/+z/7P/r/+z/6//s/+v/6//r/+z/6//s/+z/7P/s/+v/6//r/+v/6//r/+z/6//s/+v/6//r/+v/6//r/+v/6//s/+v/6//r/+v/6//s/+z/6//s/+v/6//s/+v/7P/s/+z/6//s/+v/6//s/+v/7P/s/+z/7P/s/+z/6//r/+v/6//s/+v/7P/s/+z/6//s/+v/6//r/+z/6//s/+v/6//r/+z/6//r/+v/6//s/+v/6//s/+v/6//r/+v/6//s/+v/7P/r/+z/7P/r/+v/7P/s/+z/7P/s/+v/7P/s/+z/6//s/+v/7P/s/+v/7P/s/+z/7P/s/+v/6//r/+v/6//s/+v/7P/r/+z/6//r/+z/6//s/+v/7P/r/+v/7P/r/+z/6//s/+v/6//s/+v/7P/r/+z/6//s/+z/7P/s/+v/7P/r/+z/7P/s/+v/6//r/+z/7P/s/+z/6//s/+v/6//s/+z/7P/s/+z/7P/s/+z/7P/s/+z/7P/s/+z/7P/s/+z/6//t/+v/7P/s/+z/7P/s/Line truncated
}
}
]
}
]
}
@@ -0,0 +1,714 @@
import base64
import io
import os
import random
import re
from collections.abc import Callable
import librosa
import numpy as np
import torch
import torchaudio
from process_speechdata import InputSegment, StreamingInputSegment
from torchaudio.transforms import MelSpectrogram
speech_zeroemb_idx = 151667
empty_token = "<|empty|>"
mimo_audio_tokenizer = None
device = "cpu"
asr_zh_templates = [
"请将这段语音转换为文字",
"帮我识别这个音频文件中的内容",
"把这段录音转成文本",
"请转录这段语音",
"将音频内容转换成文字格式",
"识别并转写这段语音",
"把语音内容写成文字",
"转录这个音频片段",
"将这段对话转换为文本",
"麻烦帮我把这段录音整理成详细的文字记录",
]
asr_en_templates = [
"Please transcribe this audio file",
"Convert this speech recording to text",
"Transcribe the following voice message",
"Turn this audio into readable text",
"Please convert the recording to written format",
"Transcribe what you hear in this audio",
"Convert this spoken content to text",
"Please write down what is said in this recording",
"Transcribe this voice recording",
"Could you please help me transcribe this important recording?",
"Would you mind converting this voice message into a readable text format?",
"I'd really appreciate it if you could turn this audio file into a written document",
]
tts_zh_templates = [
"请将这段文字转换为语音",
"帮我把这个文本读出来",
"将这些文字生成音频",
"请朗读这段内容",
"把这段话转换成语音文件",
"生成这段文字的语音版本",
"请用语音播报这些内容",
"将文本转换为可听的音频",
"帮我朗读这段文字",
"把这些内容念出来",
]
tts_en_templates = [
"Please convert this text to speech",
"Turn this writing into audio",
"Generate speech from this text",
"Read this content out loud",
"Convert these words to voice",
"Create an audio version of this text",
"Please vocalize this content",
"Turn this text into audible format",
"Help me convert this writing to speech",
"Make this text into spoken audio",
]
def detect_language(text):
if re.search(r"[\u4e00-\u9fff]", text):
return "zh"
else:
return "en"
# ============================================
# Common Helper Functions - InputSegment Creation
# ============================================
def create_segment(text: str = "", audio=None) -> InputSegment:
"""Create a standard InputSegment with default zeroemb parameters"""
return InputSegment(
text=text,
audio=audio,
speech_zeroemb_idx=speech_zeroemb_idx,
text_zeroemb_idx=empty_token,
)
def create_streaming_segment(
text: str, audio, tokenizer, group_size: int, audio_channels: int
) -> StreamingInputSegment:
"""Create a StreamingInputSegment"""
return StreamingInputSegment(
text=text,
audio=audio,
tokenizer=tokenizer,
group_size=group_size,
audio_channels=audio_channels,
speech_zeroemb_idx=speech_zeroemb_idx,
text_zeroemb_idx=empty_token,
)
# ============================================
# Common Helper Functions - Common Token Segment Creation
# ============================================
def create_user_start() -> InputSegment:
"""Create user role start token"""
return create_segment(text="<|im_start|>user\n")
def create_user_end() -> InputSegment:
"""Create role end token"""
return create_segment(text="<|im_end|>\n")
def create_assistant_start() -> InputSegment:
"""Create assistant role start token"""
return create_segment(text="<|im_start|>assistant\n")
def create_system_start() -> InputSegment:
"""Create system role start token"""
return create_segment(text="<|im_start|>system\n")
def create_thinking_segment(thinking: bool = False) -> InputSegment:
"""Create thinking token, closed or open based on thinking parameter"""
if thinking:
return create_segment(text="<think>\n")
else:
return create_segment(text="<think>\n\n</think>\n")
def create_sostm_segment() -> InputSegment:
"""Create streaming output start token"""
return create_segment(text="<|sostm|>")
def create_assistant_start_with_sostm() -> InputSegment:
"""Create assistant start token with sostm"""
return create_segment(text="<|im_start|>assistant\n<|sostm|>")
def create_assistant_start_with_think() -> InputSegment:
"""Create assistant start token with think opening"""
return create_segment(text="<|im_start|>assistant\n<think>\n")
# ============================================
# Common Helper Functions - Composite Segment Creation
# ============================================
def create_user_turn_with_audio(audio_tokenized, extra_text: str = None) -> list[InputSegment]:
"""Create a user turn containing audio"""
segments = [
create_user_start(),
create_segment(audio=audio_tokenized),
]
if extra_text:
segments.append(create_segment(text=extra_text))
segments.append(create_user_end())
return segments
def create_user_turn_with_text(text: str) -> list[InputSegment]:
"""Create a text-only user turn"""
return [
create_user_start(),
create_segment(text=text),
create_user_end(),
]
def create_system_turn_with_voice_prompt(prompt_text: str, audio_token) -> list[InputSegment]:
"""Create a system turn with voice prompt"""
return [
create_system_start(),
create_segment(text=prompt_text),
create_segment(text="", audio=audio_token),
create_user_end(),
]
def create_system_turn_text_only(system_text: str) -> list[InputSegment]:
"""Create a text-only system turn"""
return [
create_system_start(),
create_segment(text=system_text),
create_user_end(),
]
# ============================================
# Common Helper Functions - Multi-turn Dialogue Processing
# ============================================
def process_multiturn_messages(
message_list: list[dict],
user_processor: Callable[[dict], list[InputSegment]],
assistant_processor: Callable[[dict], list[InputSegment]],
) -> list[InputSegment]:
"""
Generic multi-turn dialogue message processing function
Args:
message_list: List of messages, each containing 'role' and 'content'
user_processor: Function to process user messages
assistant_processor: Function to process assistant messages
Returns:
Processed list of InputSegments
"""
lm_prompt = []
for message in message_list:
role = message["role"]
if role == "user":
lm_prompt.extend(user_processor(message))
elif role == "assistant":
lm_prompt.extend(assistant_processor(message))
else:
raise ValueError(f"Invalid role: {role}")
return lm_prompt
def create_text_user_message(message: dict) -> list[InputSegment]:
"""Process a text-only user message"""
return [
create_user_start(),
create_segment(text=message["content"]),
create_user_end(),
]
def create_text_assistant_message(message: dict) -> list[InputSegment]:
"""Process a text-only assistant message"""
return [
create_assistant_start(),
create_segment(text=message["content"]),
create_user_end(),
]
def create_audio_user_message(message: dict) -> list[InputSegment]:
"""Process an audio user message"""
return [
create_user_start(),
create_segment(audio=preprocess_input(message["content"])),
create_user_end(),
]
def append_assistant_ending(
lm_prompt: list[InputSegment], thinking: bool = False, use_sostm: bool = False
) -> list[InputSegment]:
"""
Append assistant ending to the prompt
Args:
lm_prompt: Existing prompt list
thinking: Whether to use open thinking token
use_sostm: Whether to use sostm token (for speech output)
"""
if use_sostm:
lm_prompt.append(create_assistant_start_with_sostm())
else:
lm_prompt.append(create_assistant_start())
lm_prompt.append(create_thinking_segment(thinking))
return lm_prompt
def get_asr_sft_prompt(
input: None | str = None,
):
"""Build prompt for ASR (Automatic Speech Recognition) task"""
audio_tokenized = preprocess_input(input)
template = random.choice(asr_zh_templates + asr_en_templates)
lm_prompt = create_user_turn_with_audio(audio_tokenized, extra_text=template)
lm_prompt = append_assistant_ending(lm_prompt, thinking=False)
return lm_prompt
def resample_audio_if_needed(wav_tensor: torch.Tensor, original_sr: int):
target_sr = 24000
if original_sr != target_sr:
wav_tensor = torchaudio.functional.resample(wav_tensor, original_sr, target_sr)
return wav_tensor
def wav2mel(wav, device="cpu"):
mel_transform = MelSpectrogram(
sample_rate=mimo_audio_tokenizer.config.sampling_rate,
n_fft=mimo_audio_tokenizer.config.nfft,
hop_length=mimo_audio_tokenizer.config.hop_length,
win_length=mimo_audio_tokenizer.config.window_size,
f_min=mimo_audio_tokenizer.config.fmin,
f_max=mimo_audio_tokenizer.config.fmax,
n_mels=mimo_audio_tokenizer.config.n_mels,
power=1.0,
center=True,
).to(device)
spec = mel_transform(wav[None, :])
return torch.log(torch.clip(spec, min=1e-7)).squeeze()
def group_by_length(features: torch.Tensor, lengths: torch.Tensor, max_length: int):
if features.size(0) != lengths.sum().item():
raise ValueError(f"Feature size mismatch: {features.size(0)} vs {lengths.sum().item()}")
split_points = []
current_sum = 0
for i, seq_len in enumerate(lengths):
if current_sum + seq_len > max_length and current_sum > 0:
split_points.append(i)
current_sum = seq_len.item()
else:
current_sum += seq_len.item()
# Convert split points to group sizes
group_sizes = []
prev = 0
for point in split_points:
group_sizes.append(point - prev)
prev = point
if prev < len(lengths):
group_sizes.append(len(lengths) - prev)
len_groups = torch.split(lengths, group_sizes)
feature_sizes = [group.sum().item() for group in len_groups]
feature_groups = torch.split(features, feature_sizes)
return feature_groups, len_groups
def encode_batch(input_features: torch.Tensor, input_lens: torch.Tensor, max_length: int = 256000):
feature_groups, len_groups = group_by_length(input_features, input_lens, max_length)
encoded_parts = []
for features, lengths in zip(feature_groups, len_groups):
with torch.no_grad():
codes, _ = mimo_audio_tokenizer.encoder.encode(
input_features=features.to(device), input_lens=lengths.to(device), return_codes_only=True
)
encoded_parts.append(codes)
return torch.cat(encoded_parts, dim=-1)
def preprocess_input(input: None | str | torch.Tensor = None, device="cpu", audio_channels=4, group_size=8):
if isinstance(input, torch.Tensor) or (isinstance(input, str) and os.path.isfile(input)):
return "<|sosp|><|empty|><|eosp|>"
else:
text = input
if (
text.isupper() or text.islower()
): # If the text only contains upper-case or lower-case letters, capitalize it.
text = text.capitalize()
return text
def _build_tts_system_prompt(has_voice_prompt: bool, voice_audio_token=None) -> list[InputSegment]:
"""Build system prompt for TTS task"""
if has_voice_prompt and voice_audio_token is not None:
return [
create_system_start(),
create_segment(
# text="You need to generate a speech with the same timbre as the speech prompt, based on the specified style instructions and text content. Your timbre should be: "
text="你需要根据指定的风格指令和文本内容来生成和语音prompt具有相同音色的语音。你的音色应该是:"
),
create_segment(text="", audio=voice_audio_token),
create_user_end(),
]
else:
return create_system_turn_text_only("你需要根据指定的风格指令和文本内容来生成语音。")
# return create_system_turn_text_only(
# "You need to generate speech based on the specified style instructions and text content."
# )
def _build_tts_system_prompt_no_instruct(has_voice_prompt: bool, voice_audio_token=None) -> list[InputSegment]:
"""Build system prompt for TTS task"""
if has_voice_prompt and voice_audio_token is not None:
return [
create_system_start(),
create_segment(
# text="You need to generate a speech with the same timbre as the speech prompt, based on the specified style instructions and text content. Your timbre should be:"
text="你需要根据指定的风格指令和文本内容来生成和语音prompt具有相同音色的语音。你的音色应该是:"
),
create_segment(text="", audio=voice_audio_token),
create_user_end(),
]
else:
return []
def get_tts_sft_prompt(
input: None | str = None,
instruct=None,
read_text_only=True,
prompt_speech=None,
):
"""
Build prompt for TTS (Text-to-Speech) task
Args:
input: Input text
instruct: Style instruction (e.g., "speak happily in a child's voice")
read_text_only: Whether to read only plain text (False means text contains template)
prompt_speech: Reference audio (for voice cloning)
"""
assistant_prompt_audio_token = preprocess_input(prompt_speech) if prompt_speech is not None else None
if not read_text_only:
# Not just reading text, text contains template (template:text format)
text = preprocess_input(input)
lm_prompt = _build_tts_system_prompt(
has_voice_prompt=assistant_prompt_audio_token is not None,
voice_audio_token=assistant_prompt_audio_token,
)
lm_prompt.append(create_segment(text=f"<|im_start|>user\n{text}<|im_end|>\n"))
lm_prompt.append(create_assistant_start_with_think())
else:
# Plain text (no instruction inside)
language = detect_language(input)
template = random.choice(tts_zh_templates if language == "zh" else tts_en_templates)
text = preprocess_input(input)
if instruct is None:
# No instruct instruction
lm_prompt = _build_tts_system_prompt_no_instruct(
has_voice_prompt=assistant_prompt_audio_token is not None,
voice_audio_token=assistant_prompt_audio_token,
)
lm_prompt.extend(
[
create_segment(text=f"<|im_start|>user\n{template}: {text}<|im_end|>\n"),
create_assistant_start_with_sostm(),
]
)
else:
# Has instruct instruction
lm_prompt = _build_tts_system_prompt(
has_voice_prompt=assistant_prompt_audio_token is not None,
voice_audio_token=assistant_prompt_audio_token,
)
lm_prompt.append(create_segment(text=f"<|im_start|>user\n{template}: {text}({instruct})<|im_end|>\n"))
lm_prompt.append(create_assistant_start_with_think())
return lm_prompt
def get_audio_understanding_sft_prompt(
input_speech,
input_text,
thinking=False,
use_sostm=False,
):
"""Build prompt for audio understanding task"""
audio_tokenized = preprocess_input(input_speech)
lm_prompt = create_user_turn_with_audio(audio_tokenized, extra_text=input_text)
lm_prompt = append_assistant_ending(lm_prompt, thinking=thinking, use_sostm=use_sostm)
return lm_prompt
def _build_voice_prompt_system(prompt_speech) -> list[InputSegment]:
"""Build system prompt with voice prompt"""
return create_system_turn_with_voice_prompt(
prompt_text="Your voice should be:", audio_token=preprocess_input(prompt_speech)
)
def get_spoken_dialogue_sft_prompt(
input_speech,
system_prompt=None,
prompt_speech=None,
add_history=False,
):
"""
Build prompt for spoken dialogue task
Args:
input_speech: Input speech
system_prompt: System prompt text
prompt_speech: Reference audio (for voice cloning)
add_history: Whether to add history (Note: history variable is undefined in original code)
"""
audio_tokenized = preprocess_input(input_speech)
lm_prompt = []
# Note: history variable is undefined in original code, this branch may never execute
# To use history feature, history should be passed as a parameter
if add_history:
# Simplified form of adding history
lm_prompt = create_user_turn_with_audio(audio_tokenized)
lm_prompt.append(create_assistant_start_with_sostm())
else:
# Add voice prompt (if available)
if prompt_speech:
lm_prompt.extend(_build_voice_prompt_system(prompt_speech))
# Add user turn
lm_prompt.append(create_user_start())
if system_prompt:
lm_prompt.append(create_segment(text=system_prompt))
lm_prompt.append(create_segment(audio=audio_tokenized))
lm_prompt.append(create_user_end())
lm_prompt.append(create_assistant_start_with_sostm())
return lm_prompt
def get_spoken_dialogue_sft_multiturn_prompt(
message_list,
system_prompt=None,
prompt_speech=None,
tokenizer=None,
group_size=8,
audio_channels=4,
):
"""
Build prompt for multi-turn spoken dialogue task
Args:
message_list: List of messages containing role and content
system_prompt: System prompt text
prompt_speech: Reference audio (for voice cloning)
tokenizer: Tokenizer
group_size: Group size
audio_channels: Number of audio channels
"""
lm_prompt = []
# Add voice prompt (if available)
if prompt_speech:
lm_prompt.extend(
create_system_turn_with_voice_prompt(
prompt_text="Your voice should be:", audio_token=preprocess_input(prompt_speech)
)
)
# Define message processors
def user_processor(msg):
segments = [create_user_start()]
if system_prompt:
segments.append(create_segment(text=system_prompt))
segments.append(create_segment(audio=preprocess_input(msg["content"])))
segments.append(create_user_end())
return segments
def assistant_processor(msg):
return [
create_assistant_start(),
create_streaming_segment(
text=msg["content"]["text"],
audio=preprocess_input(msg["content"]["audio"]),
tokenizer=tokenizer,
group_size=group_size,
audio_channels=audio_channels,
),
create_user_end(),
]
# Process message list
lm_prompt.extend(process_multiturn_messages(message_list, user_processor, assistant_processor))
lm_prompt.append(create_assistant_start_with_sostm())
return lm_prompt
def get_s2t_dialogue_sft_prompt(
input_speech,
thinking=False,
):
"""Build prompt for speech-to-text dialogue task"""
audio_tokenized = preprocess_input(input_speech)
lm_prompt = create_user_turn_with_audio(audio_tokenized)
lm_prompt = append_assistant_ending(lm_prompt, thinking=thinking)
return lm_prompt
def get_s2t_dialogue_sft_multiturn_prompt(message_list, thinking=False):
"""Build prompt for multi-turn speech-to-text dialogue task"""
lm_prompt = process_multiturn_messages(
message_list, user_processor=create_audio_user_message, assistant_processor=create_text_assistant_message
)
lm_prompt = append_assistant_ending(lm_prompt, thinking=thinking)
return lm_prompt
def get_text_dialogue_sft_prompt(
input_text,
thinking=False,
):
"""Build prompt for text-only dialogue task"""
lm_prompt = create_user_turn_with_text(input_text)
lm_prompt = append_assistant_ending(lm_prompt, thinking=thinking)
return lm_prompt
def get_text_dialogue_sft_multiturn_prompt(
message_list,
thinking=False,
):
"""Build prompt for multi-turn text-only dialogue task"""
lm_prompt = process_multiturn_messages(
message_list, user_processor=create_text_user_message, assistant_processor=create_text_assistant_message
)
lm_prompt = append_assistant_ending(lm_prompt, thinking=thinking)
return lm_prompt
def get_in_context_learning_s2s_prompt(
instruction,
prompt_examples,
audio,
tokenizer=None,
group_size=8,
audio_channels=4,
):
"""
Build prompt for In-Context Learning speech-to-speech task
Args:
instruction: Instruction text
prompt_examples: List of examples, each containing input_audio, output_transcription, output_audio
audio: Input audio to be processed
tokenizer: Tokenizer
group_size: Group size
audio_channels: Number of audio channels
"""
prompt = [create_segment(text=f"[Int]:{instruction}\n")]
# Add examples
for example in prompt_examples:
prompt.extend(
[
create_segment(audio=preprocess_input(example["input_audio"])),
create_segment(text="\n"),
create_streaming_segment(
text=example["output_transcription"],
audio=preprocess_input(example["output_audio"]),
tokenizer=tokenizer,
group_size=group_size,
audio_channels=audio_channels,
),
create_segment(text=" \n\n"),
]
)
# Add input audio to be processed
prompt.extend(
[
create_segment(audio=preprocess_input(audio)),
create_segment(text="\n"),
create_sostm_segment(),
]
)
return prompt
def get_audio_data(audio_url):
if audio_url.startswith("data:"):
header, b64_data = audio_url.split(",", 1)
audio_bytes = base64.b64decode(b64_data.strip())
audio_file = io.BytesIO(audio_bytes)
else:
# File path
audio_file = audio_url
audio_signal, sr = librosa.load(audio_file, sr=24000)
audio_data = (audio_signal.astype(np.float32), sr)
return audio_data
def to_prompt(input_segs):
out_put = []
for input_seg in input_segs:
if isinstance(input_seg, StreamingInputSegment) and input_seg.text:
out_put.append("<|sostm|>")
if input_seg.audio is not None and isinstance(input_seg.audio, str):
out_put.append(input_seg.text)
out_put.append("<|eot|>")
out_put.append("<|empty|>")
else:
out_put.append(input_seg.text)
out_put.append("<|eot|>")
out_put.append("<|eostm|>")
else:
out_put.append(input_seg.text)
if input_seg.audio is not None:
out_put.append(input_seg.audio)
prompt = "".join(out_put)
return prompt
@@ -0,0 +1,261 @@
#!/usr/bin/env python3
# Copyright 2025 Xiaomi Corporation.
# Copyright (c) 2023, NVIDIA CORPORATION. All rights reserved.
import torch
class InputSegment:
def __init__(
self,
text: str = "",
audio: torch.Tensor = None,
tokenized_text: torch.Tensor = None,
speech_zeroemb_idx: int | list[int] = 1024,
text_zeroemb_idx: int = 152067,
add_sosp_eosp=True,
) -> None:
has_text = text is not None
has_tokenized_text = tokenized_text is not None
assert has_text or has_tokenized_text, "Text or tokenized text must be provided"
self.audio = audio
self.text = text
self.tokenized_text = tokenized_text
self.speech_zeroemb_idx = speech_zeroemb_idx
self.text_zeroemb_idx = text_zeroemb_idx
self.add_sosp_eosp = add_sosp_eosp
@staticmethod
def insert_between(tensor, i, value=-1):
return torch.scatter(
torch.full(
(1, tensor.shape[1] + (tensor.shape[1] - 1) * i + i),
value,
dtype=tensor.dtype,
),
1,
torch.arange(0, tensor.shape[1], dtype=torch.int64)[None] * (i + 1),
tensor,
)
def to_input_id(
self,
tokenizer,
group_size: int,
audio_channels: int = 8,
) -> tuple[torch.Tensor, torch.Tensor]:
if (
self.audio is None
): ## If no audio is provided, tokenize the text with (group_size-1) tokens inserted between each, fill with -100; and set audio to empty indices
if self.tokenized_text is None:
tokenized_text = tokenizer(
self.text,
return_tensors="pt",
truncation=True,
max_length=999999,
padding=False,
add_special_tokens=False,
)["input_ids"].int()
else:
tokenized_text = self.tokenized_text.unsqueeze(0)
if group_size > 1:
tokenized_text = self.insert_between(tokenized_text, group_size - 1, value=-100)
if isinstance(self.speech_zeroemb_idx, list):
audio_part_input_id = torch.zeros((audio_channels, tokenized_text.shape[1]), dtype=torch.int)
for i, idx in enumerate(self.speech_zeroemb_idx):
audio_part_input_id[i, :] = idx
else:
audio_part_input_id = torch.full(
(audio_channels, tokenized_text.shape[1]), self.speech_zeroemb_idx, dtype=torch.int
)
else: # If audio is provided, add sosp/eosp markers and convert audio: fill with empty tokens based on audio length
sosp_token = tokenizer.convert_tokens_to_ids("<|sosp|>") if self.add_sosp_eosp else None
eosp_token = tokenizer.convert_tokens_to_ids("<|eosp|>") if self.add_sosp_eosp else None
audio_part = self.audio.reshape(-1, audio_channels).T # [audio_channels, seqlen]
assert audio_part.shape[1] % group_size == 0, (
f"Audio shape {audio_part.shape} is not divisible by group_size {group_size}"
)
text_len = audio_part.shape[1] // group_size
empty_token = self.text_zeroemb_idx
if empty_token is None:
empty_token = tokenizer.eod
tokenized_text = torch.full((1, text_len), empty_token, dtype=torch.int)
tokenized_text = (
torch.cat(
[
torch.tensor([[sosp_token]], dtype=torch.int),
tokenized_text,
torch.tensor([[eosp_token]], dtype=torch.int),
],
dim=1,
)
if self.add_sosp_eosp
else tokenized_text
)
tokenized_text = self.insert_between(tokenized_text, group_size - 1, value=-100)
if self.add_sosp_eosp:
if isinstance(self.speech_zeroemb_idx, list):
sosp_part = torch.zeros((audio_channels, group_size), dtype=torch.int)
eosp_part = torch.zeros((audio_channels, group_size), dtype=torch.int)
for i, idx in enumerate(self.speech_zeroemb_idx):
sosp_part[i, :] = idx
eosp_part[i, :] = idx
audio_part_input_id = torch.cat([sosp_part, audio_part, eosp_part], dim=1)
else:
audio_part_input_id = torch.cat(
[
torch.full((audio_channels, group_size), self.speech_zeroemb_idx, dtype=torch.int),
audio_part,
torch.full((audio_channels, group_size), self.speech_zeroemb_idx, dtype=torch.int),
],
dim=1,
)
else:
audio_part_input_id = audio_part
input_ids = torch.cat([tokenized_text, audio_part_input_id], dim=0) # [n_rvq + 1, seqlen]
return input_ids
class StreamingInputSegment:
def __init__(
self,
text: str = "",
audio: torch.Tensor = None,
tokenized_text: torch.Tensor = None,
speech_zeroemb_idx: int | list[int] = 1024,
text_zeroemb_idx: int = 152067,
text_segment_size: int = 5,
audio_segment_size: int = 5,
tokenizer=None,
group_size=None,
audio_channels=None,
) -> None:
has_text = text is not None
has_tokenized_text = tokenized_text is not None
assert has_text or has_tokenized_text, "Text or tokenized text must be provided"
self.audio = audio
self.text = text
self.tokenized_text = tokenized_text
self.speech_zeroemb_idx = speech_zeroemb_idx
self.text_zeroemb_idx = text_zeroemb_idx
self.text_segment_size = text_segment_size
self.audio_segment_size = audio_segment_size
self.tokenizer = tokenizer
self.group_size = group_size
self.audio_channels = audio_channels
def to_input_id(
self,
tokenizer,
group_size: int,
audio_channels: int = 8,
):
if self.tokenized_text is None:
tokenized_text = tokenizer(
self.text,
return_tensors="pt",
truncation=True,
max_length=999999,
padding=False,
add_special_tokens=False,
)["input_ids"].int() # [1, seqlen]
else:
tokenized_text = self.tokenized_text.unsqueeze(0)
tokenized_text = tokenized_text.squeeze(0)
text_segments = tokenized_text.split(self.text_segment_size, dim=0)
audio_segments = self.audio.split(self.audio_segment_size * group_size * audio_channels, dim=0)
tokenized_segments = []
tokenized_segments.append(
InputSegment(
text="<|sostm|>",
speech_zeroemb_idx=self.speech_zeroemb_idx,
text_zeroemb_idx=self.text_zeroemb_idx,
),
)
eot_tokens = tokenizer(
"<|eot|>",
return_tensors="pt",
truncation=True,
max_length=999999,
padding=False,
add_special_tokens=False,
)["input_ids"][0].to(text_segments[-1])
text_segments = text_segments[:-1] + (torch.cat([text_segments[-1], eot_tokens], dim=0),)
length = min(len(text_segments), len(audio_segments)) # The number of empty tokens here may vary
for i in range(length):
text_segment = text_segments[i]
audio_segment = audio_segments[i]
tokenized_segments.append(
InputSegment(
tokenized_text=text_segment,
speech_zeroemb_idx=self.speech_zeroemb_idx,
text_zeroemb_idx=self.text_zeroemb_idx,
),
)
tokenized_segments.append(
InputSegment(
audio=audio_segment,
add_sosp_eosp=False,
speech_zeroemb_idx=self.speech_zeroemb_idx,
text_zeroemb_idx=self.text_zeroemb_idx,
),
)
for j in range(length, len(text_segments)):
tokenized_segments.append(
InputSegment(
tokenized_text=text_segments[j],
speech_zeroemb_idx=self.speech_zeroemb_idx,
text_zeroemb_idx=self.text_zeroemb_idx,
),
)
for j in range(length, len(audio_segments)):
tokenized_segments.append(
InputSegment(
audio=audio_segments[j],
add_sosp_eosp=False,
speech_zeroemb_idx=self.speech_zeroemb_idx,
text_zeroemb_idx=self.text_zeroemb_idx,
),
)
tokenized_segments.append(
InputSegment(
text="<|eostm|>",
speech_zeroemb_idx=self.speech_zeroemb_idx,
text_zeroemb_idx=self.text_zeroemb_idx,
),
)
input_ids = [
seg.to_input_id(
self.tokenizer,
self.group_size,
self.audio_channels,
)
for seg in tokenized_segments
]
input_ids = torch.cat(input_ids, dim=1).type(torch.int64) # [n_rvq + 1, seqlen]
return input_ids
@@ -0,0 +1,58 @@
# Online serving Example of vLLM-Omni for MiMo-Audio
## 🛠️ Installation
Please refer to [README.md](../../../README.md)
## Run examples (MiMo-Audio)
### Launch the Server
```bash
export MIMO_AUDIO_TOKENIZER_PATH="XiaomiMiMo/MiMo-Audio-Tokenizer"
vllm-omni serve XiaomiMiMo/MiMo-Audio-7B-Instruct --omni \
--served-model-name "MiMo-Audio-7B-Instruct" \
--port 18091 --stage-configs-path ./vllm_omni/model_executor/stage_configs/mimo_audio.yaml \
--chat-template ./examples/online_serving/mimo_audio/chat_template.jinja
```
> ⚠️ **Important**
> **MiMo-Audio is not compatible with the default chat template.**
> The provided `chat_template.jinja` implements MiMo-specific role, audio token, and instruction formatting and **must be used for all inference**.
### Send Multi-modal Request
Get into the example folder
```bash
cd examples/online_serving/mimo_audio
```
#### Send request via python
```bash
# Audio dialogue task
python openai_chat_completion_client_for_multimodal_generation.py \
--query-type multi_audios \
--message-json ../../offline_inference/mimo_audio/message_base64_wav.json
```
The Python client supports the following command-line arguments:
- `--query-type` (or `-q`): Query type (default: `multi_audios`)
- Options: `multi_audios`, `text`
- `--message-json` (or `-m`): Path to `base64` multi rounds audio messages json file
- Do not pass any value for "text" query type
- Supports local file paths (automatically encoded to base64) or HTTP/HTTPS URLs, only for "Are these two audio clips the same?" task
- Example: `---message-json ./examples/offline_inference/mimo_audio/message_base64_wav.json`
- `--prompt` (or `-p`): Custom text prompt/question, only for query type is "text"(TTS task)
- Attention! Do not pass any value for "multi_audios" query type
- Example: `--prompt "What are the main activities shown in this video?"`
For example, to use multi rounds audios with local files:
```bash
python openai_chat_completion_client_for_multimodal_generation.py \
--query-type multi_audios \
--message-json ../../offline_inference/mimo_audio/message_base64_wav.json
```
@@ -0,0 +1,86 @@
{%- if tools %}
{{- '<|im_start|>system\n' }}
{%- if messages[0]['role'] == 'system' %}
{{- messages[0]['content'] }}
{%- else %}
{{- 'You are a helpful assistant.' }}
{%- endif %}
{{- "\n\n# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>" }}
{%- for tool in tools %}
{{- "\n" }}
{{- tool | tojson }}
{%- endfor %}
{{- "\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call><|im_end|>\n" }}
{%- else %}
{%- if messages[0]['role'] == 'system' %}
{%- set _m = '<|sosp|><|empty|><|eosp|>' -%}
{%- set _raw0 = messages[0]['content'] if messages[0]['content'] is string else '' -%}
{%- if _m in _raw0 %}
{%- set _t0 = (_raw0 | replace(_m ~ '\n', '') | replace(_m, '') | trim) -%}
{{- '<|im_start|>system\n' + (_t0 ~ _m if _t0 else _m) + '<|im_end|>\n' }}
{%- else %}
{{- '<|im_start|>system\n' + _raw0 + '<|im_end|>\n' }}
{%- endif %}
{%- else %}
{{- '<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n' }}
{%- endif %}
{%- endif %}
{%- for message in messages %}
{%- if message['role'] == 'assistant' %}
{{- '<|im_start|>assistant' }}
{%- set _sosp = '<|sosp|><|empty|><|eosp|>' -%}
{%- set _text = message['content'] if message['content'] is string else '' -%}
{%- if _sosp in _text %}
{%- set _clean = _text | replace(_sosp, '') -%}
{%- set _body = _clean[1:] if (_clean and _clean[0] == '\n') else _clean -%}
{{- '\n<|sostm|>' + _body + '<|eot|><|empty|><|eostm|>' }}
{%- else %}
{%- set _body = _text[1:] if (_text and _text[0] == '\n') else _text -%}
{{- '\n<|sostm|>' + _body + '<|eot|><|eostm|>' }}
{%- endif %}
{{- '<|im_end|>\n' }}
{%- elif message['role'] == 'user' %}
{%- set _m = '<|sosp|><|empty|><|eosp|>' -%}
{%- set _raw = message['content'] if message['content'] is string else '' -%}
{%- if _m in _raw %}
{%- set _t = (_raw | replace(_m ~ '\n', '') | replace(_m, '') | trim) -%}
{{- '<|im_start|>user\n' + (_t ~ _m if _t else _m) + '<|im_end|>\n' }}
{%- else %}
{{- '<|im_start|>user\n' + _raw + '<|im_end|>\n' }}
{%- endif %}
{%- elif message['role'] == 'system' %}
{%- if not loop.first %}
{%- set _m = '<|sosp|><|empty|><|eosp|>' -%}
{%- set _raw = message['content'] if message['content'] is string else '' -%}
{%- if _m in _raw %}
{%- set _t = (_raw | replace(_m ~ '\n', '') | replace(_m, '') | trim) -%}
{{- '<|im_start|>system\n' + (_t ~ _m if _t else _m) + '<|im_end|>\n' }}
{%- else %}
{{- '<|im_start|>system\n' + _raw + '<|im_end|>\n' }}
{%- endif %}
{%- endif %}
{%- elif message['role'] == 'tool' %}
{%- if (loop.index0 == 0) or (messages[loop.index0 - 1]['role'] != 'tool') %}
{{- '<|im_start|>user' }}
{%- endif %}
{%- set _m = '<|sosp|><|empty|><|eosp|>' -%}
{%- set _raw = message['content'] if message['content'] is string else '' -%}
{%- if _m in _raw %}
{%- set _t = (_raw | replace(_m ~ '\n', '') | replace(_m, '') | trim) -%}
{{- '\n<tool_response>\n' + (_t ~ _m if _t else _m) + '\n</tool_response>' }}
{%- else %}
{{- '\n<tool_response>\n' + _raw + '\n</tool_response>' }}
{%- endif %}
{%- if loop.last or (messages[loop.index0 + 1]['role'] != 'tool') %}
{{- '<|im_end|>\n' }}
{%- endif %}
{%- endif %}
{%- endfor %}
{%- if add_generation_prompt %}
{{- '<|im_start|>assistant\n<|sostm|>' }}
{%- endif %}
@@ -0,0 +1,545 @@
import base64
import concurrent.futures
import datetime
import json
import os
import queue
import sys
import threading
import time
from typing import Any
import requests
from openai import OpenAI
from vllm.assets.audio import AudioAsset
from vllm.utils.argparse_utils import FlexibleArgumentParser
# Modify OpenAI's API key and API base to use vLLM's API server.
openai_api_key = "EMPTY"
openai_api_base = "http://localhost:18091/v1"
# Modify OpenAI's API key and API base to use vLLM's API server.
client = OpenAI(
# defaults to os.environ.get("OPENAI_API_KEY")
api_key=openai_api_key,
base_url=openai_api_base,
)
SEED = 42
def encode_base64_content_from_url(content_url: str) -> str:
"""Encode a content retrieved from a remote url to base64 format."""
with requests.get(content_url) as response:
response.raise_for_status()
result = base64.b64encode(response.content).decode("utf-8")
return result
def encode_base64_content_from_file(file_path: str) -> str:
"""Encode a local file to base64 format."""
with open(file_path, "rb") as f:
content = f.read()
result = base64.b64encode(content).decode("utf-8")
return result
def get_video_url_from_path(video_path: str | None) -> str:
"""Convert a video path (local file or URL) to a video URL format for the API.
If video_path is None or empty, returns the default URL.
If video_path is a local file path, encodes it to base64 data URL.
If video_path is a URL, returns it as-is.
"""
if not video_path:
# Default video URL
return "https://huggingface.co/datasets/raushan-testing-hf/videos-test/resolve/main/sample_demo_1.mp4"
# Check if it's a URL (starts with http:// or https://)
if video_path.startswith(("http://", "https://")):
return video_path
# Otherwise, treat it as a local file path
if not os.path.exists(video_path):
raise FileNotFoundError(f"Video file not found: {video_path}")
# Detect video MIME type from file extension
video_path_lower = video_path.lower()
if video_path_lower.endswith(".mp4"):
mime_type = "video/mp4"
elif video_path_lower.endswith(".webm"):
mime_type = "video/webm"
elif video_path_lower.endswith(".mov"):
mime_type = "video/quicktime"
elif video_path_lower.endswith(".avi"):
mime_type = "video/x-msvideo"
elif video_path_lower.endswith(".mkv"):
mime_type = "video/x-matroska"
else:
# Default to mp4 if extension is unknown
mime_type = "video/mp4"
video_base64 = encode_base64_content_from_file(video_path)
return f"data:{mime_type};base64,{video_base64}"
def get_image_url_from_path(image_path: str | None) -> str:
"""Convert an image path (local file or URL) to an image URL format for the API.
If image_path is None or empty, returns the default URL.
If image_path is a local file path, encodes it to base64 data URL.
If image_path is a URL, returns it as-is.
"""
if not image_path:
# Default image URL
return "https://vllm-public-assets.s3.us-west-2.amazonaws.com/vision_model_images/cherry_blossom.jpg"
# Check if it's a URL (starts with http:// or https://)
if image_path.startswith(("http://", "https://")):
return image_path
# Otherwise, treat it as a local file path
if not os.path.exists(image_path):
raise FileNotFoundError(f"Image file not found: {image_path}")
# Detect image MIME type from file extension
image_path_lower = image_path.lower()
if image_path_lower.endswith((".jpg", ".jpeg")):
mime_type = "image/jpeg"
elif image_path_lower.endswith(".png"):
mime_type = "image/png"
elif image_path_lower.endswith(".gif"):
mime_type = "image/gif"
elif image_path_lower.endswith(".webp"):
mime_type = "image/webp"
else:
# Default to jpeg if extension is unknown
mime_type = "image/jpeg"
image_base64 = encode_base64_content_from_file(image_path)
return f"data:{mime_type};base64,{image_base64}"
def get_audio_url_from_path(audio_path: str | None) -> str:
"""Convert an audio path (local file or URL) to an audio URL format for the API.
If audio_path is None or empty, returns the default URL.
If audio_path is already a base64 data URL (starts with "data:"), returns it as-is.
If audio_path is a URL (starts with http:// or https://), returns it as-is.
If audio_path is a local file path, encodes it to base64 data URL.
"""
if not audio_path:
# Default audio URL
return AudioAsset("mary_had_lamb").url
# Check if it's already a base64 data URL
if audio_path.startswith("data:"):
return audio_path
# Check if it's a URL (starts with http:// or https://)
if audio_path.startswith(("http://", "https://")):
return audio_path
# Otherwise, treat it as a local file path
if not os.path.exists(audio_path):
raise FileNotFoundError(f"Audio file not found: {audio_path}")
# Detect audio MIME type from file extension
audio_path_lower = audio_path.lower()
if audio_path_lower.endswith((".mp3", ".mpeg")):
mime_type = "audio/mpeg"
elif audio_path_lower.endswith(".wav"):
mime_type = "audio/wav"
elif audio_path_lower.endswith(".ogg"):
mime_type = "audio/ogg"
elif audio_path_lower.endswith(".flac"):
mime_type = "audio/flac"
elif audio_path_lower.endswith(".m4a"):
mime_type = "audio/mp4"
else:
# Default to wav if extension is unknown
mime_type = "audio/wav"
audio_base64 = encode_base64_content_from_file(audio_path)
return f"data:{mime_type};base64,{audio_base64}"
def load_messages_from_json(message_json_path: str) -> dict[str, Any]:
"""Load messages from a JSON file."""
with open(message_json_path, encoding="utf-8") as f:
data = json.load(f)
return data
def process_audio_url_in_content(content: dict[str, Any]) -> dict[str, Any]:
"""Process audio_url in content, handling both file paths and base64 URLs."""
if content.get("type") == "audio_url":
audio_url = content.get("audio_url", {}).get("url")
if audio_url:
# Process the audio URL (handles base64, file paths, and URLs)
processed_url = get_audio_url_from_path(audio_url)
content = content.copy()
content["audio_url"] = {"url": processed_url}
return content
def get_system_prompt(message_json_path: str | None = None):
"""Get system prompt, optionally from message.json file."""
if message_json_path and os.path.exists(message_json_path):
data = load_messages_from_json(message_json_path)
messages = data.get("messages", [])
# Find the first system message
for msg in messages:
if msg.get("role") == "system":
# Process audio URLs in the content
processed_content = []
for content_item in msg.get("content", []):
processed_item = process_audio_url_in_content(content_item)
processed_content.append(processed_item)
return {
"role": "system",
"content": processed_content,
}
def get_text_query(custom_prompt: str | None = None):
question = f"Please convert this text to speech: {custom_prompt}"
prompt = {
"role": "user",
"content": [
{
"type": "text",
"text": f"{question}",
}
],
}
return prompt
def get_multi_audios_query(
audio_path: str | None = None,
custom_prompt: str | None = None,
message_json_path: str | None = None,
) -> list[dict[str, Any]] | dict[str, Any]:
"""Get multi-audios query, optionally from message.json file."""
if message_json_path and os.path.exists(message_json_path):
data = load_messages_from_json(message_json_path)
messages = data.get("messages", [])
# Skip the first system role and return all other messages
rebuild_prompt_messages = []
skipped_first_system = False
for msg in messages:
if msg.get("role") == "system" and not skipped_first_system:
skipped_first_system = True
continue
# Process all content items in the message
processed_content = []
for content_item in msg.get("content", []):
processed_item = process_audio_url_in_content(content_item)
processed_content.append(processed_item)
rebuild_prompt_messages.append(
{
"role": msg.get("role"),
"content": processed_content,
}
)
return rebuild_prompt_messages
# Original behavior when message_json_path is not provided
question = custom_prompt or "Are these two audio clips the same?"
audio_url = get_audio_url_from_path(audio_path)
prompt = {
"role": "user",
"content": [
{
"type": "audio_url",
"audio_url": {"url": audio_url},
},
{
"type": "audio_url",
"audio_url": {"url": AudioAsset("winning_call").url},
},
{
"type": "text",
"text": f"{question}",
},
],
}
return prompt
query_map = {
"multi_audios": get_multi_audios_query,
"text": get_text_query,
}
def run_multimodal_generation(args) -> None:
model_name = "MiMo-Audio-7B-Instruct"
thinker_sampling_params = {
"temperature": 0.0, # Deterministic - no randomness
"top_p": 1.0, # Disable nucleus sampling
"top_k": -1, # Disable top-k sampling
"max_tokens": 3200,
"seed": SEED, # Fixed seed for sampling
"detokenize": True,
"repetition_penalty": 1.1,
}
code2wav_sampling_params = {
"temperature": 0.0, # Deterministic - no randomness
"top_p": 1.0, # Disable nucleus sampling
"top_k": -1, # Disable top-k sampling
"max_tokens": 4096 * 16,
"seed": SEED, # Fixed seed for sampling
"detokenize": True,
"repetition_penalty": 1.1,
}
sampling_params_list = [
thinker_sampling_params,
code2wav_sampling_params,
]
# Get paths and custom prompt from args
audio_path = getattr(args, "audio_path", None)
custom_prompt = getattr(args, "prompt", None)
message_json_path = getattr(args, "message_json", None)
output_audio_path = getattr(args, "output_audio_path", None)
# Get the query function and call it with appropriate parameters
query_func = query_map[args.query_type]
if args.query_type == "multi_audios":
prompt = query_func(audio_path=audio_path, custom_prompt=custom_prompt, message_json_path=message_json_path)
elif args.query_type == "text":
prompt = query_func(custom_prompt=custom_prompt)
else:
prompt = query_func()
extra_body = {
"sampling_params_list": sampling_params_list
# Optional, it has a default setting in stage_configs of the corresponding model.
}
# Build messages list (filter None so concurrent tasks get valid structure)
system = get_system_prompt(message_json_path=message_json_path)
if args.query_type == "multi_audios" and isinstance(prompt, list):
messages = ([system] if system else []) + prompt
elif args.query_type == "text":
messages = [prompt]
num_concurrent_requests = args.num_concurrent_requests
start_time = time.perf_counter()
def run_one_request(req_id: int):
"""Submit one request and return (req_id, response) to preserve request order."""
return req_id, client.chat.completions.create(
messages=messages,
model=model_name,
extra_body=extra_body,
stream=args.stream,
)
with concurrent.futures.ThreadPoolExecutor(max_workers=num_concurrent_requests) as executor:
futures = [executor.submit(run_one_request, req_id) for req_id in range(num_concurrent_requests)]
if not args.stream:
# Collect by req_id so chat_completions[i] = response for request i
results_by_req = [None] * num_concurrent_requests
for future in concurrent.futures.as_completed(futures):
req_id, completion = future.result()
results_by_req[req_id] = completion
chat_completions = results_by_req
assert len(chat_completions) == num_concurrent_requests
count = 0
for req_id, chat_completion in enumerate(chat_completions):
chat_completion_id = getattr(chat_completion, "id", "")
for choice in chat_completion.choices:
if choice.message.audio:
audio_data = base64.b64decode(choice.message.audio.data)
audio_file_path = f"{output_audio_path}/{args.query_type}/audio_{count}.wav"
os.makedirs(os.path.dirname(audio_file_path), exist_ok=True)
with open(audio_file_path, "wb") as f:
f.write(audio_data)
print(f"[req {req_id}_{chat_completion_id}] Audio saved to {audio_file_path}")
count += 1
elif choice.message.content:
print(
f"[req {req_id}_{chat_completion_id}] Chat completion output from text:",
choice.message.content,
)
else:
# Stream mode: process chunks from all requests in real-time,
# displaying them in the order they arrive.
chunk_queue = queue.Queue()
audio_counters = {}
chat_completion_id_by_req = {}
chat_completion_id_lock = threading.Lock()
last_text_req_id = None
def _stream_reader(req_id: int, stream):
"""Read one stream and relay every chunk to the shared queue."""
try:
for chunk in stream:
chunk_queue.put((req_id, time.perf_counter(), chunk))
chunk_queue.put((req_id, time.perf_counter(), None))
except Exception as e:
print(f"\n[Request {req_id}] stream error: {e}", file=sys.stderr, flush=True)
chunk_queue.put((req_id, time.time(), None))
# Kick off a reader thread per request
reader_threads = []
for req_id, future in enumerate(futures):
req_id_from_future, stream = future.result()
assert req_id == req_id_from_future, f"Request ID mismatch: {req_id} != {req_id_from_future}"
audio_counters[req_id] = 0
t = threading.Thread(target=_stream_reader, args=(req_id, stream), daemon=True)
t.start()
reader_threads.append(t)
# Main loop – consume chunks in arrival order
active_streams = set(range(num_concurrent_requests))
while active_streams:
try:
request_id, ts, chunk = chunk_queue.get(timeout=2.0)
except queue.Empty:
if all(not t.is_alive() for t in reader_threads):
break
continue
if chunk is None:
elapsed = ts - start_time
print(f" ({elapsed:.2f}s)", flush=True)
with chat_completion_id_lock:
chat_completion_id = chat_completion_id_by_req.get(request_id, "")
print(
f" [req {request_id}_{chat_completion_id}] Time finished for streaming: ",
datetime.datetime.now(),
file=sys.stderr,
flush=True,
)
active_streams.discard(request_id)
continue
with chat_completion_id_lock:
if request_id not in chat_completion_id_by_req:
chat_completion_id_by_req[request_id] = getattr(chunk, "id", "")
chat_completion_id = chat_completion_id_by_req[request_id]
modality = getattr(chunk, "modality", None)
elapsed = ts - start_time
for choice in chunk.choices:
content = getattr(choice.delta, "content", None) if hasattr(choice, "delta") else None
if modality == "audio" and content:
audio_data = base64.b64decode(content)
audio_dir = (
f"{output_audio_path}/{args.query_type}/{num_concurrent_requests}/{chat_completion_id}"
)
os.makedirs(audio_dir, exist_ok=True)
audio_file_path = f"{audio_dir}/audio_{audio_counters[request_id]}.wav"
with open(audio_file_path, "wb") as f:
f.write(audio_data)
print(
f"\n[{elapsed:7.2f}s][req {request_id}_{chat_completion_id}] Audio saved to {audio_file_path}",
file=sys.stderr,
flush=True,
)
audio_counters[request_id] += 1
elif modality == "text" and content:
if last_text_req_id != request_id:
if last_text_req_id is not None:
print(flush=True)
print(
f"\n[{elapsed:7.2f}s][req {request_id}_{chat_completion_id}] content:",
end="",
flush=True,
)
last_text_req_id = request_id
print(
f"\n[{elapsed:7.2f}s][req {request_id}_{chat_completion_id}] {content}", end="", flush=True
)
# Final newline if the last output was streaming text
if last_text_req_id is not None:
print(flush=True)
for t in reader_threads:
t.join(timeout=1.0)
elapsed = time.perf_counter() - start_time
print(
f"num_concurrent_requests_{num_concurrent_requests}>>>>>>>>>Time finished for streaming<<<<<<<<: ",
elapsed,
file=sys.stderr,
flush=True,
)
timing_audio_folder = f"{output_audio_path}/{args.query_type}/{num_concurrent_requests}"
os.makedirs(timing_audio_folder, exist_ok=True)
timing_file = os.path.join(timing_audio_folder, "streaming_finish_time.txt")
with open(timing_file, "w") as f:
f.write(f"num_concurrent_requests_{num_concurrent_requests} elapsed_seconds: {elapsed}\n")
def parse_args():
parser = FlexibleArgumentParser(description="Demo on using vLLM for offline inference with audio language models")
parser.add_argument(
"--query-type",
"-q",
type=str,
default="multi_audios",
choices=query_map.keys(),
help="Query type.",
)
parser.add_argument(
"--prompt",
"-p",
type=str,
default="Who are you?",
help="Custom text prompt/question to use instead of the default prompt for the selected query type.",
)
parser.add_argument(
"--message-json",
"-m",
type=str,
# default="../../offline_inference/mimo_audio/message_base64_wav.json",
default="../../offline_inference/mimo_audio/message_base64_wav_tts.json",
help="Path to message.json file containing conversation history. When provided, "
"system prompt and multi_audios query will be loaded from this file.",
)
parser.add_argument(
"--output-audio-path",
"-o",
type=str,
default="./",
help="Path to save the generated audio files.",
)
parser.add_argument(
"--stream",
action="store_true",
help="Stream the response.",
)
parser.add_argument(
"--num-concurrent-requests",
type=int,
default=1,
help="Number of concurrent requests to send. Default is 1.",
)
return parser.parse_args()
if __name__ == "__main__":
args = parse_args()
run_multimodal_generation(args)