Emotion Recognition for Human-Robot Interaction

UMBC Ph.D. student Farshad Safavi presented his dissertation proposal on recognizing emotions for better human-robot interaction on February 9, 2024.

Emotion Recognition via Multimodal Fusion for Human-Robot Interaction Using Deep Learning

One of the primary challenges in Human-Robot Interaction (HRI) is enabling robots to effectively understand and respond to human emotions. Humans express emotions through verbal and non-verbal cues, while robots typically rely on pre-programmed algorithms and physical gestures. Our research aims to develop HRI that bridges this gap by leveraging multimodal emotion detection. Emotions play a crucial role in human communication and decision-making, significantly influencing human-robot interactions. We aim for robots to understand and respond to human emotions by integrating neurophysiological and behavioral channels. Initially, we examine unimodal facial expression recognition using Convolutional Neural Networks (CNN) and Vision Transformers (ViT). Next, we enhance the model with a Mixture of Transformers (MiT). Using this enhanced model, we have developed a human-robot interaction perception system. Subsequently, we investigate multimodal emotion recognition in conveying emotions in Human-Robot Interaction (HRI). While unimodal techniques have been used to recognize emotions from various sources, research indicates that emotion recognition is inherently multimodal. Fusion representations provide a more comprehensive view of the emotional state, thereby enhancing emotion recognition accuracy. Therefore, exploring the role of multimodal fusion through computational models and neurophysiological experiments is essential. Our framework uses machine learning and deep learning to interpret complex physiological and facial expression data, enabling nuanced human-robot interactions. We focus on the offline fusion of multimodal methods, combining brain and behavior models, and exploring real-time fusion solutions. These human-robot interactions, based on emotions, will be validated through neurophysiological experiments, aiming for seamless and intuitive interactions based on a thorough understanding of human emotions.

UMBC Professor Cynthia Matuszek is the PI on a new award from the National Science Foundation to support the 2024 joint Principal Investigator Meeting for NSF's two research programs on robotics and related areas, National Robotics Initiative (NRI) and Foundational Research in Robotics (FRR). Professor Jie Yin from North Carolina State University is the grant Co-Principal investigator. Dr. Matuszek will chair the meeting, which will take place in downtown Baltimore, Maryland, on 28-30 April 2024.

The meeting will serve as a conference bringing together a community of robotics researchers whose work is specifically relevant to the NRI and FRR programs. The agenda includes talks from selected projects, poster sessions, workshops, panel discussions, multiple keynote talks, a diversity-focused event, and Federal agency program updates from leaders in the research community. This meeting will also support the participation of a cohort of potential future proposers to the FRR program.

The annual PI Meeting for the NSF FRR and NRI research communities enables investigators to meet and share their research and best practices, discuss new research opportunities, explore new ideas and partnerships, and interact with Federal agency representatives, industry, and other stakeholders interested in NSF robotics research.