pipette
ESEspañol

VOSSA: Voiceprint Optimization for Streaming Speech Architectures

Mu-Ruei Tseng, Waris Quamer, Ghady Nasrallah, Ricardo Gutierrez-Osuna

Preprint with a published versionReal-world use

In the authors' words

Real-time voice conversion (VC) systems commonly rely on pretrained speaker embeddings from automatic speaker verification (ASV) models. While effective for speaker discrimination, these embeddings are trained to remain stable across phonetic and prosodic variations within-speaker, which may conflict with frame-level acoustic generation in streaming constraints. To address this issue, we propose VOSSA (Voiceprint Optimization for Streaming Speech Architectures), a speaker representation framework that extracts speaker information from intermediate content encoder layers and aggregates using attentive statistics pooling. The embedding is trained jointly with VC objectives, removing the need for a separate speaker encoder. Across six datasets, VOSSA improves F0 dynamics and vowel-discriminative acoustic cues while maintaining comparable NISQA-MOS, WER, and speaker similarity. Perceptual tests further indicate improvements in naturalness, speaker similarity, intelligibility, and vibrancy.

Main resultThe abstract does not state a limitation.

Appeared: Thursday, October 1. arXiv. Preprint with a published version.

DOI: 10.21437/Interspeech.2026-2763

Published version: Tseng, M.-R., Quamer, W., Nasrallah, G., Gutierrez-Osuna, R. (2026) VOSSA: Voiceprint Optimization for Streaming Speech Architectures

Authors' comment: Published in Proceedings of Interspeech 2026