pipette
ESEspañol

Capability-Guided Iterative Online Optimization of Cascaded Spoken-Dialogue Modules

Yuhang Qiu, Bohan Li, Wenbin Huang, Zhihan Li, Yiwei Guo, Hankun Wang, Xie Chen, Kai Yu

PreprintReal-world use

In the authors' words

Cascaded spoken dialogue systems comprise perception-and-comprehension, decision-making, and response-generation modules. We study iterative optimization of the first and last modules. Their reciprocal speech-content interfaces enable one-sided data to provide cross-module supervision: acoustic observations, such as waveforms or feature sequences, can be pseudo-labeled to train generation, whereas content-only samples can be rendered into speech to train comprehension. Because the reliable teacher-student direction varies across domains, we propose a two-stage, capability-guided pipeline for iterative online optimization. Stage 1 compares both modules through a shared intelligibility probe and obtains a domain-wise capability prior. Stage 2 activates the compatible one-sided-data route, constructs and filters supervision with the current teacher, updates the selected student, and refreshes the relevant checkpoints across cycles. We instantiate comprehension with Whisper large-v3 and generation with CosyVoice2. After only 200 steps for Whisper and 600 for CosyVoice2 of our iterative online optimization, we observe that, on the Chinese MAGICDATA-RAMC dataset, Whisper reduces CER from 14.18% to 10.77%, while CosyVoice2 reduces it from 4.78% to 4.04%. On the Japanese Common Voice dataset, Whisper reduces CER from 15.23% to 15.10%, while CosyVoice2 reduces it from 14.85% to 11.18%. These results demonstrate the feasibility of capability-guided iterative optimization of complementary spoken-dialogue modules.

Main resultThe abstract does not state a limitation.

Appeared: Tuesday, November 10. arXiv. Preprint, not yet peer-reviewed.

Authors' comment: 15 pages, 4 figures. Pre-review version of a paper accepted at NCMMSC 2026