pipette
ENEnglish

Capability-Guided Iterative Online Optimization of Cascaded Spoken-Dialogue Modules

Yuhang Qiu, Bohan Li, Wenbin Huang, Zhihan Li, Yiwei Guo, Hankun Wang, Xie Chen, Kai Yu

PreprintUso en el mundo real

En palabras de los autores

Cascaded spoken dialogue systems comprise perception-and-comprehension, decision-making, and response-generation modules. We study iterative optimization of the first and last modules. Their reciprocal speech-content interfaces enable one-sided data to provide cross-module supervision: acoustic observations, such as waveforms or feature sequences, can be pseudo-labeled to train generation, whereas content-only samples can be rendered into speech to train comprehension. Because the reliable teacher-student direction varies across domains, we propose a two-stage, capability-guided pipeline for iterative online optimization. Stage 1 compares both modules through a shared intelligibility probe and obtains a domain-wise capability prior. Stage 2 activates the compatible one-sided-data route, constructs and filters supervision with the current teacher, updates the selected student, and refreshes the relevant checkpoints across cycles. We instantiate comprehension with Whisper large-v3 and generation with CosyVoice2. After only 200 steps for Whisper and 600 for CosyVoice2 of our iterative online optimization, we observe that, on the Chinese MAGICDATA-RAMC dataset, Whisper reduces CER from 14.18% to 10.77%, while CosyVoice2 reduces it from 4.78% to 4.04%. On the Japanese Common Voice dataset, Whisper reduces CER from 15.23% to 15.10%, while CosyVoice2 reduces it from 14.85% to 11.18%. These results demonstrate the feasibility of capability-guided iterative optimization of complementary spoken-dialogue modules.

Resultado principalEl resumen no menciona limitaciones.

Apareció: martes, 10 de noviembre. arXiv. Preprint, todavía sin revisión por pares.

Comentario de los autores: 15 pages, 4 figures. Pre-review version of a paper accepted at NCMMSC 2026