The DOI matches its Crossref record; 32 of the 32 references it cites were found in a registry or an index.
Cicero: Multi-Turn, Contextual Argumentation for Accurate Crowdsourcing
Open accessAbstract
Traditional approaches for ensuring high quality crowdwork have failed to achieve high-accuracy on difficult problems. Aggregating redundant answers often fails on the hardest problems when the majority is confused. Argumentation has been shown to be effective in mitigating these drawbacks. However, existing argumentation systems only support limited interactions and show workers general justifications, not context-specific arguments targeted to their reasoning. This paper presents Cicero, a new workflow that improves crowd accuracy on difficult tasks by engaging workers in multi-turn, contextual discussions through real-time, synchronous argumentation. Our experiments show that compared to previous argumentation systems which only improve the average individual worker accuracy by 6.8 percentage points on the Relation Extraction domain, our workflow achieves 16.7 percentage point improvement. Furthermore, previous argumentation approaches don't apply to tasks with many possible answers; in contrast, Cicero works well in these cases, raising accuracy from 66.7% to 98.8% on the Codenames domain.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Kincaid, J. P., Fishburne, Jr., Robert P., R., Richard L., C., & Brad, S. (1975) · doi:10.21236/ada006655 · Cited by 3,497
Maximum Likelihood Estimation of Observer Error-Rates Using the EM Algorithm
Dawid, A. P., & Skene, A. M. (1979) · Applied Statistics · doi:10.2307/2346806 · Cited by 1,623
The multidimensional wisdom of crowds
Welinder, P., Branson, S., Belongie, S., & Perona, P. (2010) · CaltechAUTHORS (California Institute of Technology) · Cited by 742
Soylent: a word processor with a crowd inside
Bernstein, M. S., Little, G., Miller, R. C., Hartmann, B., Ackerman, M. S., Karger, D. R., Crowell, D., & Panovich, K. (2010) · Proceedings of the 23nd annual ACM symposium on User interface software and technology · doi:10.1145/1866029.1866078 · Cited by 686
VizWiz: nearly real-time answers to visual questions
Bigham, J. P., Jayant, C., Ji, H., Little, G., Miller, A., Miller, R. C., Miller, R., Tatarowicz, A., White, B., White, S., et al. (2010) · Proceedings of the 23nd annual ACM symposium on User interface software and technology · doi:10.1145/1866029.1866080 · Cited by 601
Bigham, J. P., Jayant, C., Ji, H., Little, G., Miller, A. E., Miller, R. C., Tatarowicz, A., White, B., White, S., & YEH, T. P. (2010) · International Cross-Disciplinary Conference on Web Accessibility (W4A) · doi:10.1145/1805986.1806020 · Cited by 536
Development and use of a gold-standard data set for subjectivity classifications
Wiebe, J., Bruce, R. F., & O'Hara, T. P. (1999) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.3115/1034678.1034721 · Cited by 525
Demartini, G., Difallah, D. E., & Cudré-Mauroux, P. (2012) · International World Wide Web Conference (WWW) · doi:10.1145/2187836.2187900 · Cited by 448
Shepherding the crowd yields better work
Dow, S., Kulkarni, A., Klemmer, S., & Hartmann, B. (2012) · Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work · doi:10.1145/2145204.2145355 · Cited by 334
Collaboratively crowdsourcing workflows with turkomatic
Kulkarni, A. P., Can, M., & Hartmann, B. (2012) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/2145204.2145354 · Cited by 288
Noronha, J., Hysen, E., Zhang, H., & Gajos, K. Z. (2011) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/2047196.2047198 · Cited by 247
Little, G., Chilton, L. B., Goldman, M., & Miller, R. C. (2009) · doi:10.1145/1600150.1600159 · Cited by 237
Self-citationChilton, L. B., Little, G., Edge, D., Weld, D. S., & Landay, J. A. (2013) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2470654.2466265 · Cited by 234
Self-citationExpert crowdsourcing with flash teams
Retelny, D., Robaszkiewicz, S., To, A., Lasecki, W. S., Patel, J., Rahmati, N., Doshi, T. S., Valentine, M., & Bernstein, M. S. (2014) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/2642918.2647409 · Cited by 232
Budget-optimal crowdsourcing using low-rank matrix approximations
Karger, D. R., Oh, S., & Shah, D. (2011) · Allerton Conference on Communication, Control, and Computing · doi:10.1109/allerton.2011.6120180 · Cited by 150
Combining Distant and Partial Supervision for Relation Extraction
Angeli, G., Tibshirani, J., Wu, J. Y., & Manning, C. D. (2014) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1164 · Cited by 145
Information Extraction A Multidisciplinary Approach to an Emerging Information Technology
Pazienza, M. T. (1997) · Lecture notes in computer science · doi:10.1007/3-540-63438-x · Cited by 138
Making Sense of Group Chat through Collaborative Tagging and Summarization
Zhang, A. X., & Cranshaw, J. (2018) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3274465 · Cited by 138
2011 International Conference on Computer Vision, ICCV 2011
Donahue, J., & Grauman, K. (2011) · CiteSeer X (The Pennsylvania State University) · Cited by 120
POMDP-based control of workflows for crowdsourcing
Dai, P., Lin, C. H., Mausam, M., & Weld, D. S. (2013) · Artificial Intelligence · doi:10.1016/j.artint.2013.06.002 · Cited by 117
Self-citationAnnotator rationales for visual recognition
Donahue, J., & Grauman, K. L. (2011) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2011.6126394 · Cited by 66
Measuring Crowdsourcing Effort with Error-Time Curves
Cheng, J., Teevan, J. B., & Bernstein, M. S. (2015) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2702123.2702145 · Cited by 64
Zhu, H., Dow, S. P., Kraut, R. E., & Kittur, A. (2014) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/2531602.2531718 · Cited by 55
Resolvable vs. Irresolvable Disagreement
Schaekermann, M., Goh, J., Larson, K. S., & Law, E. (2018) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3274423 · Cited by 54
Infusion of Labeled Data into Distant Supervision for Relation Extraction
Pershina, M., Min, B., Xu, W. H., & Grishman, R. (2014) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.3115/v1/p14-2119 · Cited by 52
Salehi, N., & Bernstein, M. S. (2018) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3274420 · Cited by 52
Zhou, S., Valentine, M., & Bernstein, M. S. (2018) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173682 · Cited by 44
Bragg, J., Mausam, M., & Weld, D. S. (2018) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3242587.3242598 · Cited by 38
Self-citationLi, T., Luther, K., & North, C. L. (2018) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3274374 · Cited by 30
Kriplean, T., Morgan, J. T., Freelon, D., Borning, A. H., & Bennett, L. (2011) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/1979742.1979869 · Cited by 17
An Empirical Study on Short- and Long-Term Effects of Self-Correction in Crowdsourced Microtasks
Kobayashi, M., Morita, H., Matsubara, M., Shimizu, N., & Morishima, A. (2018) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v6i1.13324 · Cited by 16
Working in Pairs: Understanding the Effects of Worker Interactions in CrowdworkPreprint
Ho, C.-J., & Yin, M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.09634
Flagged first, then by citations
Research works printed but not linked: 20
- Ryan Drapeau, Lydia B. Chilton, Jonathan Bragg, and Daniel S. Weld. 2016. MicroTalk: Using Argumentation to Improve Crowdsourcing Accuracy.
- 10.5555/645856.669801 · DOI
- Ting-Hao Kenneth Huang and Jeffrey P Bigham. 2017. A 10-MonthLong Deployment Study of On-Demand Recruiting for Low-Latency Crowdsourcing. In Proceedings of The fifth AAAI Conference on Human Computation and Crowdsourcing (HCOMP 2017).
- Ting-Hao Kenneth Huang, Walter S Lasecki, Amos Azaria, and Jeffrey P Bigham. 2016. " Is There Anything Else I Can Help You With?" Challenges in Deploying an On-Demand Crowd-Powered Conversational Agent. In Fourth AAAI Conference on Human Computation and Crowdsourcing.
- Ece Kamar, Severin Hacker, and Eric Horvitz. 2012. Combining human and machine intelligence in large-scale crowdsourcing. In Proceedings of the 11th International Conference on Autonomous Agents and Multiagent Systems-Volume 1. International Foundation for Autonomous Agents and Multiagent Systems, 467--474.
- David R. Karger, Sewoong Oh, and Devavrat Shah. 201 Budgetoptimal Crowdsourcing using Low-rank Matrix Approximations. In Conference on Communication, Control, and Computing.
- Chinmay Kulkarni, Koh Pang Wei, Huy Le, Daniel Chia, Kathryn Papadopoulos, Justin Cheng, Daphne Koller, and Scott R Klemmer. 2015. Peer and self assessment in massive online classes. In Design thinking research. Springer, 131--168.
- Christopher H. Lin, Mausam, and Daniel S. Weld. 2014. To Re(label), or Not To Re(label). In HCOMP.
- Angli Liu, Stephen Soderland, Jonathan Bragg, Christopher H. Lin, Xiao Ling, and Daniel S. Weld. 2016. Effective Crowd Annotation for Relation Extraction. In Proceedings of NAACL and HLT 2016.
- Angli Liu, Stephen Soderland, Jonathan Bragg, Christopher H. Lin, Xiao Ling, and Daniel S. Weld. 2016. Effective Crowd Annotation for Relation Extraction. In Proceedings of NAACL and HLT 2016.
- Andrew Mao, Yiling Chen, Krzysztof Z. Gajos, David C. Parkes, Ariel D. Procaccia, and Haoqi Zhang. 2012. TurkServer: Enabling Synchronous and Longitudinal Online Experiments. In The Fourth Workshop on Human Computation (HCOMP 2012).
- 10.1609/hcomp.v4i1.13287 · DOI
- Sarah Michaels, Catherine O'Connor, and Lauren B Resnick. 2008. Deliberative discourse idealized and realized: Accountable talk in the classroom and in civic life. Studies in philosophy and education 27, 4 (2008), 283--297.
- David Oleson, Alexander Sorokin, Greg P Laughlin, Vaughn Hester, John Le, and Lukas Biewald. 2011. Programmatic Gold: Targeted and Scalable Quality Assurance in Crowdsourcing.. In Human Computation Workshop. 11.
- Rion Snow, Brendan O'Connor, Daniel Jurafsky, and A. Ng. 2008. Cheap and Fast - But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks. In EMNLP'08.
- Yu-An Sun, Christopher R Dance, Shourya Roy, and Greg Little. 2011. How to assure the quality of human computation tasks when majority voting fails. In Workshop on Computational Social Science and the Wisdom of Crowds, NIPS.
- Mihai Surdeanu. 20 Overview of the TAC2013 Knowledge Base Population Evaluation: English Slot Filling and Temporal Slot Filling. In TAC 2013.
- Jacob Whitehill, Paul Ruvolo, Tingfan Wu, Jacob Bergsma, and Javier Movellan. 2009. Whose Vote Should Count More: Optimal Integration of Labels from Laberlers of Unknown Expertise. In In Proc. of NIPS. 2035--2043.
- Omar F Zaidan, Jason Eisner, and Christine D Piatko. 200 Using "annotator rationales" to improve machine learning for text categorization. In Proceedings of NAACL and HLT 2007.
- Ce Zhang, Feng Niu, Christopher Ré, and Jude Shavlik. 2012. Big data versus the crowd: Looking for relationships in all the right places. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers-Volume 1. Association for Computational Linguistics, 825--834.
Cited by
Sorted by citations
Explanations Can Reduce Overreliance on AI Systems During Decision-Making
Vasconcelos, H., Jörke, M., Grunde-McLaughlin, M., Gerstenberg, T., Bernstein, M. S., & Krishna, R. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3579605 · Cited by 402
Digital Juries: A Civics-Oriented Approach to Platform Governance
Fan, J., & Zhang, A. X. (2020) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376293 · Cited by 90
Moderator Chatbot for Deliberative Discussion
Kim, S., Eun, J., Seering, J., & Lee, J. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3449161 · Cited by 88
A Survey on Task Assignment in Crowdsourcing
Hettiachchi, D., Kostakos, V., & Gonçalves, J. (2022) · ACM Computing Surveys · doi:10.1145/3494522 · Cited by 85
Zhang, A., Walker, O., Nguyen, K., Dai, J., Chen, A., & Lee, M. K. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3579601 · Cited by 95
Ambiguity-aware AI Assistants for Medical Data Analysis
Schaekermann, M., Beaton, G., Sanoubari, E., Lim, A., Larson, K., & Law, E. (2020) · International Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376506 · Cited by 61
Understanding Expert Disagreement in Medical Data Analysis through Structured Adjudication
Schaekermann, M., Beaton, G., Habib, M., Lim, A. S. P., Larson, K. S., & Law, E. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359178 · Cited by 58
Ma, S., Chen, Q., Wang, X., Zheng, C., Peng, Z., Yin, M., & Ma, X. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713423 · Cited by 48
Self-citationEfficient Elicitation Approaches to Estimate Collective Crowd Answers
Chung, J. J. Y., Song, J. Y., Kutty, S., Hong, S.-S., Kim, J., & Lasecki, W. S. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359164 · Cited by 46
Annotator Rationales for Labeling Tasks in Crowdsourcing
Kutlu, M., McDonnell, T., Lease, M., & Elsayed, T. (2020) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.12012 · Cited by 36
Expert Discussions Improve Comprehension of Difficult Cases in Medical Image Assessment
Schaekermann, M., Cai, C. J., Huang, A. E., & Sayres, R. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376290 · Cited by 28
Paying Crowd Workers for Collaborative Work
d'Eon, G., Goh, J., Larson, K. S., & Law, E. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359227 · Cited by 28
Finding Generalizable Evidence by Learning to Convince Q&A Models
Perez, E., Karamcheti, S., Fergus, R., Weston, J., Kiela, D., & Cho, K. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1244 · Cited by 27
The Effects of Feedback and Goal on the Quality of Crowdsourcing Tasks
Lim, J., Lee, J., & Kim, D. (2021) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2021.1876355 · Cited by 19
Reliance and Automation for Human-AI Collaborative Data Labeling Conflict Resolution
Brachman, M., Ashktorab, Z., Desmond, M. J., Duesterwald, E., Dugan, C., Joshi, N. N., Pan, Q., & Sharma, A. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3555212 · Cited by 18
Galileo: Citizen-led Experimentation Using a Social Computing System
Pandey, V., Koul, T., Yang, C., McDonald, D. T., Ball, M. P., Tzovaras, B. G., Knight, R. D., & Klemmer, S. (2021) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445668 · Cited by 13
Li, C., Zhang, Z., Saugstad, M., Safranchik, E., Kulkarni, M., Huang, X., Patel, S., Iyer, V., Althoff, T., & Froehlich, J. E. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642089 · Cited by 15
Goldilocks: Consistent Crowdsourced Scalar Annotations with Relative Uncertainty
Chen, Q. Z., Weld, D. S., & Zhang, A. X. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3476076 · Cited by 13
Self-citationEliciting Confidence for Improving Crowdsourced Audio Annotations
Méndez, A. E. M., Cartwright, M., Bello, J. P., & Nov, O. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3512935 · Cited by 10
Chen, Q. Z., & Zhang, A. X. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3610074 · Cited by 14
Self-citationStathopoulos, E. A., Vassiliades, A., Diplaris, S., Vrochidis, S., & Kompatsiaris, I. Y. (2024) · The Knowledge Engineering Review · doi:10.1017/s0269888924000043 · Cited by 8
Aitamurto, T., Royal, P. G., & Saldivar, J. (2023) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3544549.3585708 · Cited by 8
Pradhan, V. K., Schaekermann, M., & Lease, M. (2022) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2022.828187 · Cited by 7
Bertran, F. A., Kim, S., Chang, M., Dagan, E., Duval, J., Isbister, K., & Vidal, L. T. (2021) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3411763.3441311 · Cited by 5
Crowdsourcing Beyond Annotation: Case Studies in Benchmark Data Collection
Suhr, A., Vania, C., Nangia, N., Sap, M., Yatskar, M., Bowman, S. R., & Artzi, Y. (2021) · doi:10.18653/v1/2021.emnlp-tutorials.1 · Cited by 6
Mizrahi, E., Danzig, N., & Gordon, G. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3555573 · Cited by 8
Anjum, S., Lin, C., & Gurari, D. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3434175 · Cited by 5
Making Sense of Online Discussions: Can Automated Reports help?
Anastasiou, L., & Liddo, A. D. (2021) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3411763.3451815 · Cited by 5
Meals for Monsters: a Mobile Application for the Feasibility of Gaming and Social Mechanisms
Olinsky, S., Desai, P. M., Türkay, S., Heitkemper, E., Mitchell, E. G., Mamykina, L., & Hwang, M. L. (2021) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3411763.3451789 · Cited by 5
Finding Generalizable Evidence by Learning to Convince Q&A ModelsPreprint
Perez, E., Karamcheti, S., Fergus, R., Weston, J., Kiela, D., & Cho, K. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.05863 · Cited by 13
Case Law Grounding: Using Precedents to Align Decision-Making for Humans and AI
Chen, Q. Z., & Zhang, A. X. (2025) · ACM Collective Intelligence Conference (CI) · doi:10.1145/3715928.3737487 · Cited by 17
Self-citationWhat Quality Control Mechanisms do We Need for High-Quality Crowd Work?
Hall, M., Afzali, F. M., Krause, M., & Caton, S. J. (2022) · IEEE Access · doi:10.1109/access.2022.3207292 · Cited by 5
Evaluating medical AI systems in dermatology under uncertain ground truth
Stutz, D., Cemgil, A. T., Roy, A. G., Matejovicova, T., Barsbey, M., Strachan, P., Schaekermann, M., Freyberg, J., Rikhye, R. V., Freeman, B., et al. (2025) · Medical Image Analysis · doi:10.1016/j.media.2025.103556 · Cited by 15
Harmon, S., Heitkemper, E., Mamykina, L., & Hwang, M. L. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3555593 · Cited by 3
Vassiliades, A., Bassiliades, N., Patkos, T., & Vrakas, D. (2022) · International Journal on Semantic Web and Information Systems · doi:10.4018/ijswis.309421 · Cited by 3
Khadar, M., Runningen, D., Tang, J., Chancellor, S., & Kaur, H. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3757707 · Cited by 2
Investigation of Group Dynamics in Video-conferencing Activity Facilitated by a Virtual AgentPreprint
Gordon, G., Mizrahi, E., & Danzig, N. (2022) · Research Square · doi:10.21203/rs.3.rs-2223058/v1 · Cited by 3
CrowdMOT: Crowdsourcing Strategies for Tracking Multiple Objects in VideosPreprint
Anjum, S., Lin, C., & Gurari, D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.14265 · Cited by 1
Rechkemmer, A., & Yin, M. (2021) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2021/658
Examining Discursive Features in Crowdsourced Deliberation: Idea Generation through Elaboration
Aitamurto, T., Royal, P. G., & Saldivar, J. (2026) · Computer Supported Cooperative Work (CSCW) · doi:10.1007/s10606-025-09535-z
Pluralistic-Alignment Urbanism: Operationalizing a Right to AI for Inclusive Public Space
Mushkani, R. A. (2026) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3805689.3812244
Metadata
- Authors
- Chen, Q., Bragg, J., Chilton, L. B., & Weld, D. S.
- Affiliations
- University of Washington, Seattle, WA, USA
- University of Washington (US)
- Columbia University (US)
- Venue
- Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2019-05-02
- Pages
- 1-14
- OpenAlex
- W2898349875
- Cited by
- 41
- References
- 30
- Citations by year
- 2026: 2 · 2025: 4 · 2024: 2 · 2023: 4 · 2022: 9 · 2021: 10 · 2020: 5 · 2019: 5
- FWCI
- 8.39
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion)
- Funders
- National Science Foundation (IIS-1420667)
- Office of Naval Research (N00014-15-1-2774,N00014-18-1-2193)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
- Physical Sciences › Computer Science › Artificial Intelligence › Multi-Agent Systems and Negotiation
- Physical Sciences › Computer Science › Information Systems › Expert finding and Q&A systems
- Keywords
- argumentation, relation extraction, Codenames
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
