Virtual environments, avatars and agents in networked environments, game development for education and training, natural human-computer interaction, embodied conversational agents, social and linguistic behavior modeling, real-time graphical visualization
Avatars are graphical representations of users in online virtual environments, often in the form of articulated characters. Many online environments require their users to animate their avatars through buttons and menus. However if people were meeting face-to-face, they would not have to make any conscious effort to produce the right nonverbal behavior. Since face-to-face conversations often rely on these subtle nonverbal cues such as posture, gaze and gesture, we must try to preserve their spontaneous creation online as well. We can do this by automatically generating communicative behavior in users' avatars based on models of expected human nonverbal behavior,a real-time analysis of the online social situation and a few user supplied high-level parameters. I created the first demonstration of this approach in 1996, followed by BodyChat in 1997. An in-depth examination of augmentation of online collaboration through automated avatars was completed as part of my dissertation work in 2003. This is still a very active area of research and an open question how far we can take the idea of an automated personal conversational representative.
Much work has been done on text-to-speech technology, but it only deals with a synthetic voice. When you give the voice to a graphical character, you also have the problem of generating a convincing animated performance. Most research in the area of embodied speech has focused on phoneme to viseme mappings or on ways to randomly generate movements that appear life-like. Fewer have looked at what is being said to generate relevant behavior, but it tends to focus on isolated keywords that map onto predefined moods or attitudes. In collaboration with Tim Bickmore and Justine Cassell, I developed an approach that took this generation a level deeper by representing the linguistic and social context and then applying behavior generation rules that were backed up by empirical results that linked behaviors to context. The first demonstration of this approach was the BEAT toolkit in 2001. The approach and modular toolkit continues to grow as better methods to track context become available and more behavior rules are discovered.
Game environments provide an engaging context for interaction and when they are populated with embodied conversational agents that behave according to social rules, they can simulate complex social circumstances where players have to rely on social interaction skills to progress. Accurate simulation of social behavior, in particular patterns of nonverbal interaction, becomes a key factor in adequate training in these environments. As one of the principal inventors and technical director of the Tactical Language and Culture Training system at ISI, I explored the modularization of social behavior generation within a game engine and focused on developing levels of abstraction that allow us to carry visual social simulations to a variety of game platforms. The idea is to create something akin to a physics engine for games, but have it be a social engine that produces the correct social nonverbal behavior given a situation. This is very much an ongoing effort, both through my research at Reykjavík University and through my involvement with the ISI spin-off company Alelo Inc.
I draw my inspiration from the social world around me. This world may seem chaotic but it is remarkably structured. Interactions are governed by a protocol of behavior that greases the gears of human society's clockwork. But we are naturally oblivious to the mechanisms that steer our everyday social conduct, much in the same way we don't see what guides our feet when walking.
The rigorous methods of science help reveal the underlying regularities and dependencies, bringing to light how order rises from apparent chaos. Once we have a model of how things work, we can apply methods of engineering to build systems that simulate, participate in or assist with social interaction. To render such systems engaging and exciting, or to embrace innovation through larger-than-life concepts, we should value the perspective of art and design.