“Multimodal affect bursts in social interaction with robots during Smalltalk: automatic detection, adaptation, generation, and evaluation”. Primitive multimodal affect bursts may have been a major precursor of the evolution of speech.
LIMSI-CNRS and ENSTA-ParisTech - Projet DIGITEO SOCRATE : The SOCRATE project will provide generic technologies for use with various platforms, such as the different robots: Meka (ENSTA) and NAO (LIMSI). The Meka robot has been designed to work in human-centered environments. Experiences with NAO at LIMSI, Team: affective and social dimensions in spoken interaction (excerpt of interview filmed at LIMSI 31:52) http://www.dailymotion.com/video/xsrjhg_vivre-avec-les-robots_tech
Abstract
The notion of affect burst refers to sudden, full-blown displays of emotion in which facial, vocal,and gestural components are highly synchronized. Affect bursts (ex: laughter) are defined as short emotional non-speech expressions that convey a clearly identifiable emotional meaning in response to highly affective charged and often unexpected events. In order to design
social and affective interactive systems with robots, experimental grounding is required to study expressions of affect bursts during interaction. This subject, although theoretically described in detail, has been very little studied experimentally. We will focus on understanding and generating affect bursts in audio, facial, and gestural expressions. This thesis will address 3 main challenges: (1) how to detect affect bursts in Smalltalk and mimic; (2) how to automatically learn when it is appropriate to generate them and to adapt the affect bursts in social interaction; and (3) how to define a new measure of engagemen
Context
Social interaction studies with companion-machine (Robots, ECAs): Human conversation is an amalgam of task-based dialogue and informal chat, where the more serious task-based elements are strong on linguistic propositional content and have been widely studied and implemented in dialog systems; while the social or chat elements function not to transmit linguistic messages but rather to build and maintain interpersonal relationships. The notion of “affect burst” has been introduced by (Scherer, 1994) to refer to “very brief, discrete, nonverbal expressions of affect in both face and voice as triggered by clearly identifiable events”.
Objectives
The work aims addressing 3 challenging research questions:
(1) How to detect multimodal affect bursts in Smalltalk and mimic. A study of affective emblems and raw affect bursts (Scherer, 1994) will be also carried out,
(2) How to automatically learn when it is appropriate to generate them and how to adapt them (e.g., duration, expressivity, etc.),
(3) How to define a new evaluation measure of social engagement.
The long-term interaction of the human peers interacting with the robots is also one of the sub-challenges of this project.
Work program
At the beginning of the project, we will use a Wizard of OZ protocol to collect a multimodal database with both Meka and NAO robots. The
conversations will be script-based, consisting largely of social chat with some task-based elements.
(1) For answering to the first challenge, the database will be annotated with affect bursts. The annotation will help understand the position of apparition for generation and analyze them for automatic multimodal affect burst detection using machine learning techniques (SVM, NN) and also for generating them.
(2) The second challenge concerns the automatic learning of position of affect burst information during the interaction (automatic adaptation of the models).
(3) Then, for answering to the third challenge, which is to define a new evaluation measure of social engagement by using the type and number of affect bursts generated by the human peer, real tests will be done.
Extra information
Prerequisite
Détails
Expected funding
DIGITEO AAP2013
Status of funding
Expected
Candidates
Utilisateur
laurence.devillers
Créé
Vendredi 01 mars 2013 17:54:56 CET
dernière modif.
Vendredi 01 mars 2013 18:07:27 CET
Fichiers joints
filename
créé
hits
filesize
Aucun fichier joint à cette fiche
Connexion
Ecole Doctorale Informatique Paris-Sud
Directrice
Nicole Bidoit Assistante
Stéphanie Druetta Conseiller aux thèses
Dominique Gouyou-Beauchamps
ED 427 - Université Paris-Sud
UFR Sciences Orsay
Bat 650 - aile nord - 417
Tel : 01 69 15 63 19
Fax : 01 69 15 63 87
courriel: ed-info à lri.fr