12 KiB
SheetSage2 audio-to-score preprocessing — H200
Summary
- Vendor: m-a-p
- Model:
m-a-p/SheetSage2with them-a-p/MERT-v2-FullSongparent - Task: recording → ABC score, MIDI and timed music annotations
- Mode: standalone official Transformers inference; optional YuE2 request export
- Hardware: one NVIDIA H200 (141 GB)
- Maintainer: @princepride
When to use this recipe
Use a recording as the musical score for a YuE2 cover, or inspect its melodies,
chords and structure. The tool calls SheetSage2's official transcribe() API.
It does not register SheetSage2 as a native vLLM model or start an Omni server.
The transcription tool works independently of YuE2. For score-conditioned
generation, use an Omni checkout containing the merged
PR #7886
(423f34326ed420e5acf0b1fb862a1b5ffb7e0fa7 or later). See the
YuE2 recipe for the generation model's runtime contract.
Supported model contract
| Item | Contract |
|---|---|
| Input | One local recording decodable by FFmpeg; upstream converts it to mono, 24 kHz |
| Duration | Entire recording by default; --max-seconds limits preprocessing to a prefix |
| Full score | Default: melody voices and chords; exported request uses cot=full |
| Melody score | --melody-only: vocal and instrumental melodies, without score/playback chords; request uses cot=melody |
| Outputs | score.abc, transcription.mid, upstream annotations and manifest.json |
| YuE2 handoff | --lyrics-file and --style together also write yue2_request.json |
| Output directory | Must be new or empty; failed exports produce no YuE2 request |
| Deployment profile | One process, one H200, FP32 weight loading and BF16 inference autocast |
--melody-only does not isolate or clone a singer. The upstream model still
decodes its task annotations and retains those raw annotation files. YuE2
uses the exported score to condition newly generated audio; a transcription
does not guarantee an exact reproduction of the recording.
References
- Official SheetSage2 model card
- Pinned SheetSage2 source
- Official YuE2 model card and cover documentation
- Tool:
tools/sheetsage2_transcribe.py - Offline generation:
examples/offline_inference/yue2/end2end.py - Supported generation models
Hardware
- Accelerator: one NVIDIA H200, 141 GB; no multi-device interconnect required.
- Qualification: local M4A transcription, melody-only prefix, full-score whole-song export and one offline YuE2 cover. Run transcription and generation sequentially in separate processes. Other devices, CPU inference and performance scaling are not qualified by this recipe.
Software environment
- Ubuntu 22.04, Python 3.10, NVIDIA driver 590.48.01.
- PyTorch/torchaudio 2.8.0 (CUDA 12.8 wheels), Transformers 4.45.2.
- FFmpeg 4.4.2 was used for the local M4A check; upstream recommends FFmpeg 6.1.
- vLLM: not required in the preprocessing environment.
- vLLM-Omni: tool developed against main
a038b38179e9c788d3af6a8e73f94652afb247e4. - YuE2 qualification: vLLM 0.30.0, PyTorch 2.13.0+cu130, Transformers 5.14.1;
#7886 at
5cc8acc942e76fa6f12f8b2f2558c2663abc2aa5with the FP32 VAE-loading and scalartruncatedfixes subsequently included in the merged integration.
Command
Set up transcription
Run from this repository's root. Create a separate environment: the
official transcriber's Transformers version differs from Omni's serving
dependencies. FFmpeg must be on PATH.
uv venv --python 3.10 .venv-sheetsage2
uv pip install --python .venv-sheetsage2/bin/python \
-r tools/requirements/sheetsage2.txt
.venv-sheetsage2/bin/python tools/sheetsage2_transcribe.py reference.wav \
--trust-remote-code --device cuda:0 --melody-only \
--output-dir outputs/reference-score
The default Hub checkpoint and its custom Python code are pinned to
cafc0df1021e14f49e928c4b345f5959d414ef64. Its configuration pins the MERT-v2
parent to d8ba1c745e733b3908ce6ad16ebeb17ac7600a42. Review that code before
passing --trust-remote-code. --revision can override the default pin;
custom Hub IDs use their default revision unless one is supplied.
Local snapshots must preserve this pinned source layout: modeling_sheetsage2.py
and its transitive relative-import dependencies are sibling .py files in the
snapshot root. The local module-cache workaround targets that flat layout;
reorganized snapshots with nested Python packages are not supported by it.
For offline use, download both model snapshots, including their Python, JSON and safetensors files, before disconnecting:
.venv-sheetsage2/bin/hf download m-a-p/SheetSage2 \
--revision cafc0df1021e14f49e928c4b345f5959d414ef64 \
--local-dir models/SheetSage2
.venv-sheetsage2/bin/hf download m-a-p/MERT-v2-FullSong \
--revision d8ba1c745e733b3908ce6ad16ebeb17ac7600a42 \
--local-dir models/MERT-v2-FullSong
Transcribe a full recording for a cover
Supply target lyrics in lyrics.txt, including YuE2 section tags such as
[Verse] and [Chorus]. Match the lyric phrasing to the reference melody.
This example retains the melody and chords of the whole recording:
.venv-sheetsage2/bin/python tools/sheetsage2_transcribe.py reference.wav \
--model models/SheetSage2 --base-model-path models/MERT-v2-FullSong \
--local-files-only --trust-remote-code --device cuda:0 \
--lyrics-file lyrics.txt --style 'Chinese folk, guzheng, gentle vocals' \
--seed 831001 --output-dir outputs/cover-score
Add --melody-only for a new arrangement without the original chords;
this also changes the exported request from cot=full to cot=melody.
For a short transcription smoke test, add --max-seconds 45 and use lyrics
appropriate to that excerpt. Use a fresh output directory for each run.
Generate offline
Switch to your Omni environment for generation. Download m-a-p/YuE2-3B
and m-a-p/YuE2-Vae to models/YuE2-3B and models/YuE2-Vae, respectively.
The offline entrypoint needs a local YuE2 directory containing qwen.tiktoken.
From the repository root, pass the exported ABC to the existing entrypoint:
CUDA_VISIBLE_DEVICES=0 python examples/offline_inference/yue2/end2end.py \
--model models/YuE2-3B --vae models/YuE2-Vae \
--lyrics "$(cat lyrics.txt)" \
--style 'Chinese folk, guzheng, gentle vocals' \
--abc-file outputs/cover-score/score.abc --cot full \
--seed 831001 --max-frames 9000 --gpu-memory-utilization 0.25 \
--output cover.wav
Use --cot melody if the score was transcribed with --melody-only.
Changing --cot alone does not remove chords from an existing score.
The 9,000-frame budget permits up to 360 seconds at 25 frames/s; it does
not force that duration. The entrypoint's default 200-frame budget caps
audio at 8 seconds. Check that the final report says truncated=False.
The memory fraction above was used on one H200; it is not a measured
minimum memory requirement.
Submit over HTTP (configuration-only)
The exported JSON can also be submitted to YuE2's speech endpoint. These commands have not been qualified with an HTTP synthesis run in this recipe. Start YuE2 in its Omni environment:
vllm serve m-a-p/YuE2-3B --omni --port 8091
Submit the exported file:
curl --fail-with-body http://localhost:8091/v1/audio/speech \
-H 'Content-Type: application/json' \
--data-binary @outputs/cover-score/yue2_request.json \
--output cover.wav
The JSON contains the score itself, not a server-local file path. It sets
stream=false, requests WAV, and preserves an optional generation seed.
Use --yue2-model to match a custom served model name. The tool does not
submit the request or set a generation token budget; the server's own
duration and context limits apply.
Verification
After a successful transcription, set output below to its output directory
(outputs/reference-score for the transcription-only example). The YuE2
request is checked when exported with --lyrics-file and --style:
.venv-sheetsage2/bin/python - <<'PY'
import json
from pathlib import Path
import mido
output = Path("outputs/cover-score")
abc = (output / "score.abc").read_text(encoding="utf-8")
manifest = json.loads((output / "manifest.json").read_text(encoding="utf-8"))
assert abc.strip()
request_path = output / "yue2_request.json"
if request_path.exists():
request = json.loads(request_path.read_text(encoding="utf-8"))
assert request["extra_params"]["abc"] == abc
assert request["extra_params"]["cot"] == ("melody" if manifest["melody_only"] else "full")
print("YuE2 request verified")
midi = mido.MidiFile(output / "transcription.mid")
assert any(msg.type == "note_on" and msg.velocity for track in midi.tracks for msg in track)
print("ABC, MIDI and manifest verified")
PY
Run the CPU CLI contract tests in the repository's normal test environment (with pytest and pytest-mock installed):
python -m pytest tests/tools/test_sheetsage2_transcribe.py -m 'core_model and cpu' -q
These tests cover request construction, loading options, validation and failure handling with a mocked backend. They do not measure transcription accuracy or generated cover similarity.
Local H200 checks with the pinned backend:
| Input / mode | Result |
|---|---|
| 45 s M4A prefix, melody-only | 579 ABC characters; parseable MIDI with 118 note-on events |
| 242.219 s M4A, full score | 3,033 ABC characters; parseable MIDI with 1,087 note-on events |
| Default Hub ID, 10 s prefix | Successful ABC and MIDI export |
| Local snapshots, fresh Transformers module cache | Successful offline full-score export |
Both generated JSON requests, #7886 adapter at 5cc8acc942e76fa6f12f8b2f2558c2663abc2aa5 |
Validation and prompt construction passed (919 / 2,821 prompt tokens) |
Full score + new Mandarin lyrics, offline YuE2, cot=full, seed 831001, 9,000-frame budget |
241.319 s, 48 kHz stereo WAV; 6,034 generated tokens; truncated=False |
The offline cover used a 108 BPM Chinese traditional pop style and one H200
with --gpu-memory-utilization 0.25. The saved waveform contained finite
samples, peak amplitude 0.9501, RMS 0.1232 and no full-scale samples. This is
one functional run, not an accuracy or performance benchmark. It does not
establish perceptual equivalence to the recording. HTTP synthesis and
recordings longer than 300 seconds were not exercised. The local recording,
lyrics, generated scores and cover audio are not included in the repository.
Notes
- Weights load in FP32 because upstream merges MERT adapters before inference
autocast.
--dtype fp32disables BF16 autocast. - For local snapshots the tool prepares transitive Python dependencies in Transformers' module cache. This avoids the pinned Transformers version's direct-import-only copy behavior without editing upstream files.
- SheetSage2 manages long recordings with its own overlapping windows. This tool leaves windowing and notation generation to the official implementation.
- Failed runs may retain upstream diagnostic annotations. Choose a new output directory when retrying; no incomplete request is published for synthesis.
- Model weights have upstream license terms (CC BY-NC 4.0); this repository does not bundle weights or recordings.
Supported features
| Feature | Status |
|---|---|
| Audio-to-score preprocessing | Official backend; local file input |
| YuE2 score-conditioned generation | Offline ABC handoff qualified on H200; HTTP request export validated, HTTP synthesis configuration-only |
| Native Omni inference, continuous batching, TP/PP/SP | Not implemented for SheetSage2 |
| Streaming transcription or cover audio | Not supported by this tool |