Mood Detection and Gaze Recognition
From EQUIS Lab Wiki
Contents |
Overview
Mood Detection
Exciting research has advanced our ability to automatically detect emotions. Various sensors can be used to detect a player's affect, including camera, microphone, and heart rate, blood pressure and skin conductance sensors. Once a player's emotion is known, it can be used to control the behaviour and appearance of that player's avatar.
Speech analysis techniques can be extended to detect emotion. As described by Polzin and Waibel, voice streams can be analyzed to determine juxtaposition of words, which through statistical analysis can reveal emotional state. Non-verbal analysis is also effective, using properties of the speaker's voice such as frequency, jitter, intensity and tremor. A neural network is trained to use these parameters to recognize emotional state from voice, similarly to how neural networks can be trained for speech or handwriting recognition. Thus, as players are speaking, their emotional state can be analyzed, and used within the game.
Another approach is to use [http://affect.media.mit.edu/projects.php?id=546 computer vision]. While this requires the availability of a video camera, it has the advantage of constantly providing feedback, as opposed to voice where emotion can be detected only when the player is speaking.
Also, physiological sensors are an effective way to detect a user's mood. However these sensors are typically somewhat invasive, requiring the user to wear electrodes touching the skin.
Better yet is to use a [http://aos2.uniba.it:8080/sonthofen/conati-f.pdf probabilistic model] which takes information from a number of sources (such as voice and video) and combines it to determine the best hypothesis of the player's mood.
Facial Expression Analysis
- In Authentic Emotion Detection in Real-Time Video, work has been put into assembling a database of facial expressions in order for the software to properly associate said expressions with emotional states.
- It might be worth finding a copy of "Computer Vision in Human-Computer Interaction: Proceedings of the ECCV 2004 Workshop on HCI", it might have other potentially appropriate papers
- The Robotics Institute at CMU seem to have a group working on facial expression problems
Voice Analysis
- Devillers, Vasilescu, and Lamel have looked into the Annotation and Detection of Emotion in a Task-oriented Human-Human Dialog Corpus. This seems to be particularly relevant as it examines emotional recogniton with colaboration in mind.
Gaze Recognition
Imagine that we could translate players' attention into actions of that player's avatar. [http://www.tobii.se/ Eye tracking technology] makes this possible, by sensing where the player is looking via lasers or cameras mounted on the user's display. It is likely that in time, eye tracking will be the biggest single future contributor to immersion in virtual worlds. Eye trackers are currently expensive, but so were video cards before they became widely popular amongst gamers.
Applications in Collaborative Virtual Environments
Unread
- Maia Garau has a collection of publications the topic of avatars and quality social interations in a virtual setting. Many of which involve gaze recognition. In particular "The Impact of Avatar Realism and Eye Gaze Control on Perceived Quality of Communication in a Shared Immersive Virtual Environment" (pdf) seems relevant.
- Designing a non-verbal language for expressive avatars by Salem and Earle (pdf) offers some insight into how an avatar might benefit from emotional and gaze input.
Read
Visual deitic reference (Duchowski)
Duchowski et al. published "Visual deictic reference in a collaborative virtual environment" in the Proceedings of the Eye tracking research & applications symposium on Eye tracking research & applications. This experiment does a good job of showing that certain collaborative tasks (the task isolated here was pointing or referring to something) can benefit from head and eye tracking. One participant wears an apparatus that tracks head and eye rotation and then in a CVE tries to convey what they are looking at to a second participant. In the cases where a pointer ray was cast from the head, either slaved to the head or the eye rotation, projecting a spot on wall, the effectiveness of their referral increased. For our purposes, this shows that games could benefit from improved gaze recognition in avatar collaboration.
