Files
vllm-omni/vllm_omni/deploy/qwen3_tts_cuda_ipc.yaml
T
tzhouamandClaude Fable 5.1 74acd66307 [Edge][WP7] Add CudaIpcConnector (CUDA IPC for device tensor leaves over shm control plane)
Wraps SharedMemoryConnector: CUDA tensor leaves are exported with
torch.multiprocessing reduce_tensor + an inter-process CUDA event, rebuilt and
cloned by the consumer, released by the producer after an ack file appears.
CPU leaves and metadata keep the shm path. Registered as "CudaIpcConnector";
deploy/qwen3_tts_cuda_ipc.yaml selects it for the talker->code2wav hop.
Adoption in the edge profile is measurement-gated (hop latency bench).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-09 14:41:33 +08:00

21 lines
826 B
YAML

# Qwen3-TTS with the CUDA-IPC connector between talker and code2wav
# (co-located CUDA stages, same GPU). GPU tensor leaves in the stage payload
# are shared through CUDA IPC instead of D2H + msgpack; CPU leaves and the
# control-plane metadata still use shared memory. Measurement gate: adopt in
# the edge profile only if the per-chunk hop latency improves (see
# benchmarks/tts/edge_results/stream_latency and WP7 notes).
base_config: qwen3_tts.yaml
async_chunk: true
connectors:
connector_of_shared_memory:
name: CudaIpcConnector
extra:
codec_streaming: true
connector_get_sleep_s: 0.01
connector_get_max_wait_first_chunk: 3000
connector_get_max_wait: 300
codec_chunk_frames: 25
codec_left_context_frames: 72
initial_codec_chunk_frames: 1
decode_batch_max_size: 4