The overall aim of this current research was to investigate effects of individual differences and task demand on co-speech gestures in communication. Specifically, we examined whether gesture use affected speakers' information content, and whether individual differences in working memory (WM) profiles and lexical retrieval, and task demand could account for variability in gesture use.
Forty-four speaker-listener pairs of Mandarin-speaking adults participated in a video description task. The speaker watched and described motion event videos to the listener, who had two options to choose from. The speaker's descriptions were transcribed and coded for motion element type (manner, path, source, goal, and trajectory), modality use (speech vs. gesture), gesture type (deictic vs. iconic), gestures' relation to speech (complementary vs. supplementary), and information type carried by gesture (spatial vs. semantic). A WM profile/discrepancy was measured by a difference between visuo-spatial and verbal working memory using Automated Working Memory Asessment (Alloway, 2007). Lexical retrieval was measured using a semantic fluency task (naming `animals' or `foods' in a one-minute interval). Task demand was manipulated by changing number of motion elements to be described in each video, ranging from two to four.
The results of an ANOVA showed that speakers did not include more information when they chose to gesture, although they sometimes used supplementary gestures that carried information absent from speech. However, a series of mixed model regression analyses showed that spatial complementary gestures decreased with task demand, whereas spatial supplementary gestures increased with task demand. Also, Individual differences in WM discrepancy and spatial WM capacity, not lexical retrieval, predicted production of semantic supplementary gestures. The interaction between task demand and WM discrepancy predicted spatial complementary gestures. Also, the interaction between task demand and WM discrepancy predicted semantic supplementary gestures.
Most importantly, we found that verbal dominant speakers produced fewer spatial complementary gestures when task demand was high, whereas spatial dominant speakers used these gestures similarly across task demands. Also, spatial dominant speakers tended to use more semantic supplementary gestures than verbal dominant speakers when task demand was low, but no such differences were found when task demand was high.
Taken together, our findings reveal that individuals' gesture production is a complex process, in which speaker-internal factors, such as WM, and speaker-external factors, such as task demand, and even interactions between the two factors could play a role. Given that communication is dynamic and complex, instead of restricting to one factor at a time, we may need to expand our scope to more influencing factors and their interactions to fully understand the underlying mechanism of multi-modal communication.