Verbal Interactions in Virtual Worlds
Pierre Nugues
Abstract
We first discuss respective advantages of language interaction in virtual worlds and of using 3D images in dialogue systems. Then, we describe an example of a verbal interaction system in virtual reality: Ulysse. Ulysse is a conversational agent that helps a user navigate in virtual worlds. It has been designed to be embedded in the representation of a participant of a virtual conference and it responds positively to motion orders. Ulysse navigates the user's viewpoint on his/her behalf in the virtual world. On tests we carried out, we discovered that users, novices as well as experienced ones have difficulties moving in a 3D environment. Agents such as Ulysse enable a user to carry out navigation motions that would have been impossible with classical interaction devices. From the whole Ulysse system, we have stripped off a skeleton architecture that we have ported to VRML, Java, and Prolog. We hope this skeleton helps the design of language applications in virtual worlds.
Create a lesson
Related papers
RSI-Anything: Unlocking Recursive Self-Improvement with World Models
Taoyong Cui, Pinxian Zeng, Wanghan Xu et al.
When VLMs Destroy What They Know: Spatial Representations Fall Below Lexical Priors
Shubh Garg, Chen Shani
Structural Automata for Neural Semantic Parsing
Xiao Zhang
Task Context Shapes Component Roles in Language Models
Sovesh Mohapatra, Dani S Bassett
Can Language Models Learn When to Abstain? Policy Learning and Calibrated Answer Selection
Juan Terven, Diana-Margarita Córdova-Esparza, Ruben Gomez et al.
Error Rate is Not Enough: Costs of Augmentation in Low-Resource ASR
Katsumi Ibaraki, David Chiang