Augmenting Social Science Research with Multimodal Data Collection: The EZ-MMLA Toolkit

Bertrand Schneider; Javaria Hassan; Gahyun Sung

doi:10.3390/s22020568

Back

Augmenting Social Science Research with Multimodal Data Collection: The EZ-MMLA Toolkit

Journal article

Open access

Peer reviewed

Augmenting Social Science Research with Multimodal Data Collection: The EZ-MMLA Toolkit

Bertrand Schneider, Javaria Hassan and Gahyun Sung

Sensors (Basel, Switzerland), Vol.22(2), p.568

01/01/2022

DOI: 10.3390/s22020568

PMCID: PMC8780387

PMID: 35062528

Files and links (1)

url

https://doi.org/10.3390/s22020568View

Published (Version of record) Open Access

Abstract

While the majority of social scientists still rely on traditional research instruments (e.g., surveys, self-reports, qualitative observations), multimodal sensing is becoming an emerging methodology for capturing human behaviors. Sensing technology has the potential to complement and enrich traditional measures by providing high frequency data on people's behavior, cognition and affects. However, there is currently no easy-to-use toolkit for recording multimodal data streams. Existing methodologies rely on the use of physical sensors and custom-written code for accessing sensor data. In this paper, we present the EZ-MMLA toolkit. This toolkit was implemented as a website and provides easy access to multimodal data collection algorithms. One can collect a variety of data modalities: data on users' attention (eye-tracking), physiological states (heart rate), body posture (skeletal data), gestures (from hand motion), emotions (from facial expressions and speech) and lower-level computer vision algorithms (e.g., fiducial/color tracking). This toolkit can run from any browser and does not require dedicated hardware or programming experience. We compare this toolkit with traditional methods and describe a case study where the EZ-MMLA toolkit was used by aspiring educational researchers in a classroom context. We conclude by discussing future work and other applications of this toolkit, potential limitations and implications.

Chemistry

Chemistry, Analytical

Engineering

Engineering, Electrical & Electronic

Instruments & Instrumentation

Physical Sciences

Science & Technology

Technology

Details

Title: Subtitle: Augmenting Social Science Research with Multimodal Data Collection: The EZ-MMLA Toolkit
Creators: Bertrand Schneider - Harvard University
Javaria Hassan - Harvard University
Gahyun Sung - Harvard University
Resource Type: Journal article
Publication Details: Sensors (Basel, Switzerland), Vol.22(2), p.568
Publisher: MDPI AG
DOI: 10.3390/s22020568
PMID: 35062528
PMCID: PMC8780387
ISSN: 1424-8220
eISSN: 1424-8220
Number of pages: 18
Language: English
Date published: 01/01/2022
Academic Unit: Psychological and Quantitative Foundations
Record Identifier: 9984701826502771

Metrics

12 Record Views

5 Times Cited - Web of Science

See more details