Journal article
Bayesian Decision Process for Cost-Efficient Dynamic Ranking via Crowdsourcing
Journal of machine learning research, Vol.17, pp.1-40
11/01/2016
Abstract
Rank aggregation based on pairwise comparisons over a set of items has a wide range of applications. Although considerable research has been devoted to the development of rank aggregation algorithms, one basic question is how to efficiently collect a large amount of high-quality pairwise comparisons for the ranking purpose. Because of the advent of many crowdsourcing services, a crowd of workers are often hired to conduct pairwise comparisons with a small monetary reward for each pair they compare. Since different workers have different levels of reliability and different pairs have different levels of ambiguity, it is desirable to wisely allocate the limited budget for comparisons among the pairs of items and workers so that the global ranking can be accurately inferred from the comparison results. To this end, we model the active sampling problem in crowdsourced ranking as a Bayesian Markov decision process, which dynamically selects item pairs and workers to improve the ranking accuracy under a budget constraint. We further develop a computationally efficient sampling policy based on knowledge gradient as well as a moment matching technique for posterior approximation. Experimental evaluations on both synthetic and real data show that the proposed policy achieves high ranking accuracy with a lower labeling cost.
Details
- Title: Subtitle
- Bayesian Decision Process for Cost-Efficient Dynamic Ranking via Crowdsourcing
- Creators
- Xi Chen - NYU, Stern Sch Business, 550 1St Ave, New York, NY 10012 USAKevin Jiao - NYU, Stern Sch Business, 550 1St Ave, New York, NY 10012 USAQihang Lin - Univ Iowa, Tippie Coll Business, Iowa City, IA 52242 USA
- Resource Type
- Journal article
- Publication Details
- Journal of machine learning research, Vol.17, pp.1-40
- Publisher
- Microtome Publ
- ISSN
- 1532-4435
- eISSN
- 1533-7928
- Number of pages
- 40
- Grant note
- Google Faculty Research Award; Google Incorporated
- Language
- English
- Date published
- 11/01/2016
- Academic Unit
- Business Analytics
- Record Identifier
- 9984380516502771
Metrics
1 Record Views