vllm-omni/recipes/NVIDIA/PersonaPlex.md

PersonaPlex

Live full-duplex speech-to-speech (the model listens while it speaks) with persona and voice control, on the native vLLM-Omni engine.

Summary

• Vendor: NVIDIA
• Model: nvidia/personaplex-7b-v1 (gated; a Moshi finetune)
• Task: Full-duplex speech-to-speech. 24 kHz mic audio in, agent speech + inner-monologue text out, in 80 ms lockstep. Persona (role text) and voice (zero-shot clone) are set per session.
• Mode: Live duplex WebSocket serving through the unified duplex stack (/v1/duplex and /v1/realtime?duplex=1)
• Maintainer: @linyueqian

When to use this recipe

Use this recipe to run nvidia/personaplex-7b-v1 as a live voice agent on one GPU. The integration is moshi-free: the temporal transformer, depformer, and streaming Mimi codec are all native vLLM-Omni modules (Mimi runs on transformers.MimiModel), so no vendored fork is installed. Decoding is greedy, matching the reference implementation frame for frame on the golden replays used as the acceptance gate.

PersonaPlex is the first Moshi-class (pure-lockstep) model on the vLLM-Omni unified full-duplex framework: its pipeline.py declares one duplex_plugin (PersonaPlexDuplexPlugin), and the model-specific package at vllm_omni/model_executor/models/personaplex/duplex/ holds the plugin, the worker-side lockstep Stage 0 runtime and the 80 ms input framing.

References

• Duplex serving package: vllm_omni/model_executor/models/personaplex/duplex/
• Native model modules (temporal / depformer / streaming Mimi): vllm_omni/model_executor/models/personaplex/
• Online serving example: examples/online_serving/personaplex/
• Staged pipeline (talker -> Mimi code2wav, async-chunk streaming): vllm_omni/deploy/personaplex.yaml
• Integration PR: vllm-project/vllm-omni#4771
• Framework context: RFC #3745 (duplex adapter patterns), #1335 (full-duplex target), PR #3907 (fullduplex core contracts this adapter conforms to)
• Upstream model card: nvidia/personaplex-7b-v1

Why it is different from MiniCPM-o / JoyVL duplex

	MiniCPM-o 4.5 (#3907)	JoyVL	PersonaPlex (this)
Cadence	1 s chunk groups	~1 fps frames	80 ms lockstep (12.5 Hz)
Turn control	learned ⟨listen⟩/⟨speak⟩	</silence>/</response>	none, pure lockstep
Per step	variable-length token group	text decision	1 user frame in -> 1 agent frame + 1 text token
Barge-in	at chunk boundary	n/a	native (model always hears the user)
Session state	chunk-group KV	per-tick HTTP	persistent ring KV + streaming Mimi state

This is the lockstep ("parallel-frame joint") shape of full duplex: audio flows continuously in both directions with no client commits or external turn signals. On the unified stack this is expressed purely through the capability payload (supports_client_commit=false, supports_external_turn_signal=false, supports_barge_in=false); every session on a PersonaPlex deployment is native duplex (is_enabled() is unconditionally true). Turn-based models (MiniCPM-o 4.5) are unaffected.

Architecture (Moshi RQ-Transformer, all native)

• Mimi codec 24 kHz, 12.5 Hz, 1920 samples/frame, 8 active codebooks (card 2048). Streaming encode + decode via transformers.MimiModel with the PersonaPlex reference checkpoint weights.
• Temporal transformer (Helium backbone) 4096-d / 32 layers / 32 heads, sliding window 3000 frames (a 4-minute rolling context) held in a ring KV cache with per-slot recycle.
• Depformer 1024-d / 6 layers / 16 heads, autoregressive over 16 codebooks per frame (8 vocoded), reset each frame.
• Per-frame token column [B, 17, 1]: row 0 = inner-monologue text, rows 1-8 = agent audio, rows 9-16 = user audio.
• Persona + voice are injected once at session open through the same lockstep step: voice clone forces the agent stream from reference-audio Mimi codes; persona forces the text stream from <system> ... <system> tokens.

Hardware Support

Verified on GPU (Hopper-class). The serving path is plain PyTorch eager plus the native modules, so other CUDA GPUs with enough memory are expected to work.

GPU

1x 141 GB Hopper-class GPU (H20-class, verified)

Environment

• OS: Linux
• Python: 3.10+ (CI covers 3.11 / 3.12)
• vLLM-Omni: current main
• export HF_TOKEN=... with access to the gated nvidia/personaplex-7b-v1 repo (accept the license on the model page)

First run auto-downloads from the model repo: model.safetensors, tokenizer_spm_32k_3.model, voices.tgz, and the Mimi reference checkpoint; kyutai/mimi is fetched via transformers for the codec module graph.

Command

Live duplex serving through the unified stack:

HF_TOKEN=... CUDA_VISIBLE_DEVICES=0 python -m vllm_omni.entrypoints.cli.main serve \
  /path/to/personaplex-7b-v1 \
  --omni \
  --deploy-config vllm_omni/deploy/personaplex.yaml

This exposes WS /v1/realtime?duplex=1 (alias WS /v1/duplex): the OpenAI Realtime session protocol, client API and wire vocabulary in docs/serving/realtime_duplex_api.md. Voice (voice, a bundled .pt basename) and persona (instructions) are set in session.update; the model takes no client commits and serves no /v1/chat/completions route.

Verification

# GPU-free contract tests (plugin, stage0 runtime, runner scenario)
pytest tests/model_executor/models/personaplex/duplex/ tests/engine/duplex/test_session_runner_personaplex.py -q

# GPU e2e: paced 24 kHz PCM over /v1/realtime?duplex=1, two concurrent
# sessions, overflow admission, slot recycling, non-silent output
python tests/e2e/online_serving/personaplex_realtime_duplex.py \
    --model /path/to/personaplex-7b-v1 \
    --input-wav /path/to/speech.wav \
    --output-dir /tmp/personaplex-realtime-duplex

Notes

• Realtime budget is 80 ms/frame. Verified eager per-tick latency at four concurrent sessions is ~70-74 ms on this hardware class, i.e. all four conversations stay realtime; single-session has comfortable headroom.
• Decoding is greedy only (temperature/top-k knobs are intentionally not exposed): greedy is what the parity gates pin against the reference implementation, and sampling amplifies sub-bit numeric drift into divergence.
• Session concurrency is governed by the deploy config (vllm_omni/deploy/personaplex.yaml); connections beyond the limit are rejected until a slot frees. A recycled slot is bit-exact with a fresh engine for the same inputs.
• Voice prompts: bundled voices.tgz names (NATF*/NATM* natural, VARF*/VARM* varied) or a path to your own .pt/.wav.
• Run the client near the server. The 80 ms cadence is sensitive to network jitter; over a high-latency link playback can stutter regardless of engine speed. On localhost it is smooth.
• Sliding window is 3000 frames (~4 min). The ring KV recycles beyond that; very long sessions keep running with a rolling context.