pipette
ESEspañol

SteerablePlex: Can We Steer Full-Duplex Models?

Haolong Zheng, Maike Z\"ufle, Dominik Mach\'a\v{c}ek, Peter Pol\'ak, Xulin Fan, Xavier Sumba, Siyin Wang, Ond\v{r}ej Klejch, Mark Hasegawa-Johnson

Preprint

In the authors' words

Full-duplex speech models can listen and speak simultaneously, enabling natural interaction, but become increasingly difficult to control as the conversation history grows. When used as user simulators, this lack of control can cause them to deviate from prescribed scenarios and produce unreliable evaluation outcomes. We introduce SimIF-Bench (Simulator Instruction-Following Benchmark), which evaluates whether a conversational model stays within a prescribed scenario and completes multiple goals in the required order. The benchmark reveals that current open-source full-duplex models struggle to follow such constraints. We then introduce a Group Reward-Decoupled Normalization Policy Optimization (GDPO)-based training recipe that enables a full-duplex model to follow textual instructions during an ongoing conversation while maintaining its turn-taking ability. By connecting the resulting SteerablePlex to an asynchronous backend language model that monitors the conversation and provides instructions when needed, we build a more controllable full-duplex user simulator that follows multi-stage constraints more reliably than existing open-source models and GPT-Realtime.

Main resultThe abstract does not state a limitation.

Appeared: Friday, October 9. arXiv. Preprint, not yet peer-reviewed.

Authors' comment: 5 pages, 2 figures