A Sociotechnical Lens for Evaluating Computer Vision Models: A Case Study on Detecting and Reasoning about Gender and Emotion

Sha Luo; Sang Jung Kim; Zening Duan; Kaiping Chen

doi:10.48550/arxiv.2406.08222

Back

A Sociotechnical Lens for Evaluating Computer Vision Models: A Case Study on Detecting and Reasoning about Gender and Emotion

Preprint

Open access

A Sociotechnical Lens for Evaluating Computer Vision Models: A Case Study on Detecting and Reasoning about Gender and Emotion

Sha Luo, Sang Jung Kim, Zening Duan and Kaiping Chen

arXiv.org

Cornell University

06/12/2024

DOI: 10.48550/arxiv.2406.08222

Files and links (1)

url

https://doi.org/10.48550/arxiv.2406.08222View

Preprint (Author's original)This preprint has not been evaluated by subject experts through peer review. Preprints may undergo extensive changes and/or become peer-reviewed journal articles. Open Access

Abstract

In the evolving landscape of computer vision (CV) technologies, the automatic detection and interpretation of gender and emotion in images is a critical area of study. This paper investigates social biases in CV models, emphasizing the limitations of traditional evaluation metrics such as precision, recall, and accuracy. These metrics often fall short in capturing the complexities of gender and emotion, which are fluid and culturally nuanced constructs. Our study proposes a sociotechnical framework for evaluating CV models, incorporating both technical performance measures and considerations of social fairness. Using a dataset of 5,570 images related to vaccination and climate change, we empirically compared the performance of various CV models, including traditional models like DeepFace and FER, and generative models like GPT-4 Vision. Our analysis involved manually validating the gender and emotional expressions in a subset of images to serve as benchmarks. Our findings reveal that while GPT-4 Vision outperforms other models in technical accuracy for gender classification, it exhibits discriminatory biases, particularly in response to transgender and non-binary personas. Furthermore, the model's emotion detection skew heavily towards positive emotions, with a notable bias towards associating female images with happiness, especially when prompted by male personas. These findings underscore the necessity of developing more comprehensive evaluation criteria that address both validity and discriminatory biases in CV models. Our proposed framework provides guidelines for researchers to critically assess CV tools, ensuring their application in communication research is both ethical and effective. The significant contribution of this study lies in its emphasis on a sociotechnical approach, advocating for CV technologies that support social good and mitigate biases rather than perpetuate them.

Computer Science - Artificial Intelligence

Computer Science - Computer Vision and Pattern Recognition

Computer Science - Computers and Society

Computer Science - Human-Computer Interaction

Details

Title: Subtitle: A Sociotechnical Lens for Evaluating Computer Vision Models: A Case Study on Detecting and Reasoning about Gender and Emotion
Creators: Sha Luo - University of Wisconsin–Madison
Sang Jung Kim - University of Iowa
Zening Duan - University of Wisconsin–Madison
Kaiping Chen - University of Wisconsin–Madison
Resource Type: Preprint
Publication Details: arXiv.org
DOI: 10.48550/arxiv.2406.08222
eISSN: 2331-8422
Publisher: Cornell University; Ithaca, New York
Language: English
Date posted: 06/12/2024
Academic Unit: Center for Social Science Innovation; School of Journalism and Mass Communication
Record Identifier: 9984641960302771

Metrics

15 Record Views