Dissertation
Facial expressions of emotion as a measure of emotional responses and listening difficulty
University of Iowa
Doctor of Philosophy (PhD), University of Iowa
Summer 2022
DOI: 10.25820/etd.006717
Abstract
Research in emotional responses is in its infancy in Audiology. Past studies have shown a reduced range of emotional responses to non-speech sounds in individuals with hearing loss, and a relationship between emotional responses and feelings of social disconnectedness and isolation. Hence, it is important to study emotional responses. Objective measures of emotional responses or emotional responses to speech stimuli have not been explored. In this study we use a novel method- automatic facial expression recognition - to measure objective, unbiased and unconscious measures of emotional responses. Further, we measure emotional responses to speech stimuli as this is important for communication. We measure emotional responses using facial expressions associated with confusion and frustration in adverse listening situation, i.e., to speech in noise stimuli in normal hearing, and in simulated hearing loss conditions. Our primary aim is to determine the effect of hearing loss on objective and subjective measures of emotional responses. We also determine the relationship between emotional responses, and measures of listening effort. Our secondary aim is also to determine whether objective emotional responses can be obtained remotely. We measured emotional responses in 33 participants with normal hearing in the normal hearing and in the simulated hearing loss condition in the lab, while simultaneously measuring pupil responses. We also measured emotional responses remotely. Our main outcome variables were objective and subjective measures of emotional responses (obtained in the laboratory and remotely) and listening effort. We found that objective and subjective measures confusion and frustration were greater in the simulated hearing loss condition. This finding could have consequences for individuals with hearing loss. For example, experiencing increased emotions of confusion and frustration may discourage individuals with hearing loss from participating in communication situations. We also found a weak positive correlation between subjective and objective measures of emotional response. This could mean that it is important to complement objective measures with subjective ratings of emotional response to get a measure of perceived and unconscious emotional responses. Further, we did not find a correlation between our objective measures of emotional response and objective measures of listening effort, though there was a negative correlation between the subjective measures. Though we see confusion and frustration, as well as listening effort increase with increasing listening difficulty, they are not statistically correlated. Hence, these constructs may be related, but measure different components of the communication and speech perception process. When comparing the responses for lab vs remote condition, we found that greater confusion and frustration was seen in the remote condition. This indicated that some stimulus and hardware considerations need to be taken when measuring objective emotional responses to speech stimuli in remote conditions. Overall, we found that this new method of measuring emotional responses using automatic facial expressions detection algorithm is feasible for use in measuring emotional responses.
Details
- Title: Subtitle
- Facial expressions of emotion as a measure of emotional responses and listening difficulty
- Creators
- Soumya Venkitakrishnan
- Contributors
- Yu-Hsiang Wu (Advisor)Inyong Choi (Committee Member)Carolyn J Brown (Committee Member)Elizabeth Walker (Committee Member)Erin M Picou (Committee Member)
- Resource Type
- Dissertation
- Degree Awarded
- Doctor of Philosophy (PhD), University of Iowa
- Degree in
- Speech Pathology and Audiology
- Date degree season
- Summer 2022
- DOI
- 10.25820/etd.006717
- Publisher
- University of Iowa
- Number of pages
- xiii, 127 pages
- Copyright
- Copyright 2022 Soumya Venkitakrishnan
- Language
- English
- Description illustrations
- Illustrations, charts, graphs, tables
- Description bibliographic
- Includes bibliographical references (pages 112-121).
- Public Abstract (ETD)
Emotions felt by individuals can influence their behavior, attention, participation in communication and overall quality of life. Emotions may affect individuals with hearing loss even more strongly than individuals with normal hearing because they are already at increased risk of social isolation, depression, dementia, etc. Measuring the impact that emotions have on individuals (with and without hearing loss) who are placed in difficult listening situations may help determine their participation in social situations. In this study, we measure emotions in the difficult listening situation of understanding speech in noise. Along with asking participants to rate their emotions, we also use a novel method of measuring emotions using automatic facial expressions recognition algorithm. This provides us with a measure of their conscious or perceived emotions as well as a measure of unconscious emotions. We measure without and with the presence of a simulated hearing loss, and in the lab as well as remotely. The study describes how this measure of emotional response is influenced by the presence of noise, hearing loss, and by its remote measurement. We also describe how emotional responses may be related to another construct known as listening effort, or the cognitive resources required to attend to and understand speech.
- Academic Unit
- Communication Sciences and Disorders
- Record Identifier
- 9984285452902771
Metrics
1 File views/ downloads
23 Record Views