Logo image
Back
Quality through flow and immersion: gamifying crowdsourced relevance assessments
Conference proceeding

Quality through flow and immersion: gamifying crowdsourced relevance assessments

Carsten Eickhoff, Christopher Harris, Arjen de Vries and Padmini Srinivasan
Proceedings of the 35th international ACM SIGIR conference on research and development in information retrieval, pp.871-880
SIGIR '12
08/12/2012
DOI: 10.1145/2348283.2348400

View Online

Abstract

Crowdsourcing is a market of steadily-growing importance upon which both academia and industry increasingly rely. However, this market appears to be inherently infested with a significant share of malicious workers who try to maximise their profits through cheating or sloppiness. This serves to undermine the very merits crowdsourcing has come to represent. Based on previous experience as well as psychological insights, we propose the use of a game in order to attract and retain a larger share of reliable workers to frequently-requested crowdsourcing tasks such as relevance assessments and clustering. In a large-scale comparative study conducted using recent TREC data, we investigate the performance of traditional HIT designs and a game-based alternative that is able to achieve high quality at significantly lower pay rates, facing fewer malicious submissions.
 gamification clustering serious games crowdsourcing relevance assessments

Details

Metrics

21 Record Views
Logo image