Empirically Evaluating an Adaptable Spoken Dialogue System
Diane J. Litman, Shimei Pan
Abstract
Recent technological advances have made it possible to build real-time, interactive spoken dialogue systems for a wide variety of applications. However, when users do not respect the limitations of such systems, performance typically degrades. Although users differ with respect to their knowledge of system limitations, and although different dialogue strategies make system limitations more apparent to users, most current systems do not try to improve performance by adapting dialogue behavior to individual users. This paper presents an empirical evaluation of TOOT, an adaptable spoken dialogue system for retrieving train schedules on the web. We conduct an experiment in which 20 users carry out 4 tasks with both adaptable and non-adaptable versions of TOOT, resulting in a corpus of 80 dialogues. The values for a wide range of evaluation measures are then extracted from this corpus. Our results show that adaptable TOOT generally outperforms non-adaptable TOOT, and that the utility of adaptation depends on TOOT's initial dialogue strategies.
Create a lesson
Related papers
CritICL: Inference-Time Weak-to-Strong Generalization from Small Language Model Failure Modes
Yufan Wu, Yinghui He, Zhengyi Hu et al.
TTPO: Test-Time Policy Optimization
Aozhe Wang, Zhengxi Lu, Jianze Wang et al.
Stochastic Estimation of Transduced Language Models
Vésteinn Snæbjarnarson, Samuel Kiegeland, Manuel de Prada Corral et al.
Boosting LLM Exploration via Weak-Model Guidance in RLVR
Xingyu Shen, Huishuai Zhang, Peng Li et al.
Consolidating RLVR Capabilities Across Domains: A Deep Dive into Fusion Paradigms
Siye Wu, Kai Yang, Yuchen Cai et al.
How Language Models Organize and Structure Moral Knowledge
Orion Reblitz-Richardson