The DOI matches its Crossref record; 32 of the 32 references it cites were found in a registry or an index.
Pairwise ranking aggregation in a crowdsourced setting
Abstract
Inferring rankings over elements of a set of objects, such as documents or images, is a key learning problem for such important applications as Web search and recommender systems. Crowdsourcing services provide an inexpensive and efficient means to acquire preferences over objects via labeling by sets of annotators. We propose a new model to predict a gold-standard ranking that hinges on combining pairwise comparisons via crowdsourcing. In contrast to traditional ranking aggregation methods, the approach learns about and folds into consideration the quality of contributions of each annotator. In addition, we minimize the cost of assessment by introducing a generalization of the traditional active learning scenario to jointly select the annotator and pair to assess while taking into account the annotator quality, the uncertainty over ordering of the pair, and the current model uncertainty. We formalize this as an active learning strategy that incorporates an exploration-exploitation tradeoff and implement it using an efficient online Bayesian updating scheme. Using simulated and real-world data, we demonstrate that the active learning strategy achieves significant reductions in labeling cost while maintaining accuracy.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
(2021) · A First Course in Statistical Programming with R · doi:10.1017/9781108993456.011 · Cited by 13,724
(1999) · Springer Series in Operations Research and Financial Engineering · doi:10.1007/b98874 · Cited by 9,482
(2009) · Minds at UW (University of Wisconsin) · Cited by 4,791
Individual Choice Behavior: A Theoretical Analysis.
Cane, V., & Luce, R. D. (1960) · Journal of the Royal Statistical Society. Series A (General) · doi:10.2307/2343282 · Cited by 2,504
Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons
Bradley, R. A., & Terry, M. E. (1952) · Biometrika · doi:10.2307/2334029 · Cited by 2,412
Learning to rank for information retrieval
Liu, T.-Y. (2010) · Proceedings of the 33rd international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/1835449.1835676 · Cited by 1,856
Maximum Likelihood Estimation of Observer Error-Rates Using the EM Algorithm
Dawid, A. P., & Skene, A. M. (1979) · Applied Statistics · doi:10.2307/2346806 · Cited by 1,692 · cited 2×
“(11), (12), (13), (14), (16) and (17). end for Output: Rank objects by sorting the obtained {μi}.”
BackgroundLearning to Rank for Information Retrieval
Liu, T.-Y. (2009) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000016 · Cited by 1,568
RANK ANALYSIS OF INCOMPLETE BLOCK DESIGNS: THE METHOD OF PAIRED COMPARISONS
BRADLEY, R. A., & TERRY, M. E. (1952) · Biometrika · doi:10.1093/biomet/39.3-4.324 · Cited by 1,434
The Method of Paired Comparisons.
Curnow, R. N., & David, H. A. (1964) · Journal of the Royal Statistical Society. Series A (General) · doi:10.2307/2344014 · Cited by 1,271
Individual choice behavior: A theoretical analysis.
Luce, R. D. (2005) · doi:10.1037/14396-000 · Cited by 1,157 · cited 2×
“Copyright 2013 ACM 978-1-4503-1869-3/13/02 ...$15.00.”
MethodInfluentialRaykar, V. C., Yu, S., Zhao, L. H., Valadez, G. H., Florin, C., Bogoni, L., & Moy, L. P. (2010) · Journal of Machine Learning Research · doi:10.5555/1756006.1859894 · Cited by 951
MALLOWS, C. L. (1957) · Biometrika · doi:10.1093/biomet/44.1-2.114 · Cited by 820
The multidimensional wisdom of crowds
Welinder, P., Branson, S., Belongie, S., & Perona, P. (2010) · CaltechAUTHORS (California Institute of Technology) · Cited by 742
Plackett, R. L. (1975) · Applied Statistics · doi:10.2307/2346567 · Cited by 731 · cited 2×
“Copyright 2013 ACM 978-1-4503-1869-3/13/02 ...$15.00.”
MethodInfluentialBRADLEY, R. A. (1954) · Biometrika · doi:10.1093/biomet/41.3-4.502 · Cited by 716
TrueSkill™: A Bayesian Skill Rating System
Herbrich, R., Minka, T., & Graepel, T. (2007) · Advances in Neural Information Processing Systems 19 · doi:10.7551/mitpress/7503.003.0076 · Cited by 639
Iterative Learning for Reliable Crowdsourcing Systems
Karger, D. R., Oh, S., & Shah, D. (2011) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 468
The method of paired comparisons for social values.
Thurstone, L. L. (1927) · The Journal of Abnormal and Social Psychology · doi:10.1037/h0065439 · Cited by 467 · cited 2×
“Copyright 2013 ACM 978-1-4503-1869-3/13/02 ...$15.00.”
MethodInfluentialNewton's Method for Large Bound-Constrained Optimization Problems
Lin, C., & Morè, J. J. (1999) · SIAM Journal on Optimization · doi:10.1137/s1052623498345075 · Cited by 322
Combining human and machine intelligence in large-scale crowdsourcing
Kamar, E. S., Hacker, S., & Horvitz, E. (2012) · doi:10.5555/2343576.2343643 · Cited by 318
Self-citationA Language Modeling Approach to Predicting Reading Difficulty.
Collins‐Thompson, K., & Callan, J. P. (2004) · North American Chapter of the Association for Computational Linguistics · Cited by 303
Self-citationCOMPUTING “ELO RATINGS” OF MOVE PATTERNS IN THE GAME OF GO1
Coulom, R. (2007) · ICGA Journal · doi:10.3233/icg-2007-30403 · Cited by 298
Yan, Y., Rosales, R., Fung, G., & Dy, J. (2011) · Cited by 289
Solution of a Ranking Problem from Binary Comparisons
Ford, L. R. (1957) · The American Mathematical Monthly · doi:10.1080/00029890.1957.11989117 · Cited by 229
Generalized Bradley-Terry Models and Multi-Class Probability Estimates
Huang, T.-K., Weng, R. C., & Lin, C. (2006) · Journal of Machine Learning Research · doi:10.5555/1248547.1248551 · Cited by 192
Here or there: preference judgments for relevance
Carterette, B., Bennett, P. N., Chickering, D. M., & Dumais, S. (2008) · ScholarWorks@UMassAmherst (University of Massachusetts Amherst) · doi:10.5555/1793274.1793281 · Cited by 156
Self-citationHow to Analyze Paired Comparison Data
Tsukida, K., & Gupta, M. R. (2011) · Cited by 121
A New Probabilistic Model for Rank Aggregation
Qin, T., Geng, X., & Liu, T. (2010) · Cited by 80
Adaptive Polling for Information Aggregation
Pfeiffer, T., Gao, X. A., Chen, Y., Mao, A., & Rand, D. G. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v26i1.8099 · Cited by 63
A Bayesian Approximation Method for Online Ranking
Weng, R. C., & Lin, C. (2011) · doi:10.5555/1953048.1953057 · Cited by 60
A flexible generative model for preference aggregation
Volkovs, M., & Zemel, R. S. (2012) · International World Wide Web Conference (WWW) · doi:10.1145/2187836.2187902 · Cited by 48
Cited by
Crowdsourced Data Management: A Survey
Li, G., Wang, J., Zheng, Y., & Franklin, M. J. (2016) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2016.2535242 · Cited by 283
Toward a Perspectivist Turn in Ground Truthing for Predictive Computing
Cabitza, F., Campagner, A., & Basile, V. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i6.25840 · Cited by 114
“We emphasize the distinction between raters and annotations because, as simply as it can be, raters could express more than one label for a given object to classify [19,32] (also as a way to express their indecision in case of strictly alternative labels), or they could expressly be asked to rank available labels in t…”
BackgroundKimball, A. B., Alora-Palli, M. B., Tamura, M., Mullins, L. A., Soh, C., Binder, R. L., Houston, N. A., Conley, E. D., Tung, J. Y., Annunziata, N. E., et al. (2018) · Journal of the American Academy of Dermatology · doi:10.1016/j.jaad.2017.09.012 · Cited by 62
MAPLE: Multi-Aspect Panels of LLM Evaluators for Open-Ended Questions
Jinji, M., Atarashi, K., Takeuchi, K., & Kashima, H. (2026) · Findings of the Association for Computational Linguistics: ACL 2026 · doi:10.18653/v1/2026.findings-acl.1351
“Step 3: Aggregation We integrate the comparison results collected in Step 2 using an extension of the Crowd-BT method.”
Influential
Metadata
- Authors
- Chen, X., Bennett, P. N., Collins-Thompson, K., & Horvitz, E.
- Venue
- Proceedings of the sixth ACM international conference on Web search and data mining
- Publisher
- ACM
- Published
- 2013-02-04
- Pages
- 193-202
- OpenAlex
- W2164545125
- Cited by
- 290
- References
- 32
- FWCI
- 18.79
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.