pipette
ENEnglish

Analyzing Repetition and Interference Effects in Memory-Constrained User Simulators

Cara Su-Yi Leong, Nicholas Tomlin, and Tal Linzen

Preprint

En palabras de los autores

User simulators can help evaluate and train AI systems to interact with humans at scale. To do so, evaluation of user simulators must measure behavioral parity with people. We evaluate whether memory-constrained user simulators are sensitive to two factors known to affect human information retention: repetition of a fact and interference from competing information. We find that some simulators achieve human-like overall accuracy, and replicate the magnitude of repetition and interference effects, but none match people's accuracy on particular questions: while humans misremember facts, models forget them, leading to different error patterns. We release this data as a challenge benchmark for future work on user simulators.

Resultado principalLimitación que admiten los autores

Apareció: martes, 20 de octubre. arXiv. Preprint, todavía sin revisión por pares.