Fast, cheap, and creative: evaluating translation quality using Amazon's Mechanical Turk
Open accessAbstract
Manual evaluation of translation quality is generally thought to be excessively time consuming and expensive.We explore a fast and inexpensive way of doing it using Amazon's Mechanical Turk to pay small sums to a large number of non-expert annotators.For $10 we redundantly recreate judgments from a WMT08 translation task.We find that when combined non-expert judgments have a high-level of agreement with the existing gold-standard judgments of machine translation quality, and correlate more strongly with expert judgments than Bleu does.We go on to show that Mechanical Turk can be used to calculate human-mediated translation edit rate (HTER), to conduct reading comprehension experiments with machine translation, and to create high quality reference translations.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Traditional human annotation of texts is overly demanding in time and cost.
“Traditional methodologies predominantly rely on human annotation of texts (Callison-Burch, 2009), an approach considered overly demanding in terms of time and cost.”
From ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate (2023) · cites this paper as Callison-Burch, 2009
The Abstract and Introduction say manual evaluation of machine translation is generally thought to be time-consuming and expensive. But the Introduction says researchers routinely use automatic metrics rather than predominantly relying on human annotation, and the paper demonstrates that manual evaluation can be cheaper and faster than commonly thought. Its findings concern translation evaluation, not text annotation generally. (LLM, conf=0.86)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
The citation lists haven’t been fetched yet
References 15 · Cited by 412
Research works printed but not linked: 12
- Bogdan Babych and Anthony Hartley. 2004. Extend- ing the Bleu MT evaluation method with frequency weightings. In Proceedings of ACL.
- Chris Callison-Burch, Miles Osborne, and Philipp Koehn. 2006. Re-evaluating the role of Bleu in ma- chine translation research. In Proceedings of EACL.
- Chris Callison-Burch, Cameron Fordyce, Philipp Koehn, Christof Monz, and Josh Schroeder. 2008. Further meta-evaluation of machine translation. In Proceedings of the Third Workshop on Statistical Machine Translation (WMT08).
- Chris Callison-Burch, Philipp Koehn, Christof Monz, and Josh Schroeder. 2009. Findings of the 2009 Workshop on Statistical Machine Translation. In Proceedings of the Fourth Workshop on Statistical Machine Translation (WMT09), March.
- David Chiang, Steve DeNeefe, Yee Seng Chan, and Hwee Tou Ng. 2008. Decomposability of trans- lation metrics for improved evaluation and efficient algorithms. In Proceedings of EMNLP.
- Douglas Jones, Wade Shen, Neil Granoien, Martha Herzog, and Clifford Weinstein. 2005. Measuring translation quality by testing English speakers with a new defense language proficiency test for Arabic. In Proceedings of the 2005 International Conference on Intelligence Analysis.
- LDC. 2005. Linguistic data annotation specification: Assessment of fluency and adequacy in translations. Revision 1.5. Evgeny Matusov, Nicola Ueffing, and Hermann Ney. 2006. Computing consensus translation for multiple machine translation systems using enhanced hypoth- esis alignment. In Proceedings of EACL.
- Michael Paul. 2006. Overview of the IWSLT 2006 evaluation campaign. In Proceedings of Interna- tional Workshop on Spoken Language Translation.
- Antti-Veikko Rosti, Necip Fazil Ayan, Bing Xiang, Spyros Matsoukas, Richard Schwartz, and Bonnie Dorr. 2007. Combining outputs from multiple machine translation systems. In Proceedings of HLT/NAACL.
- Matthew Snover, Bonnie Dorr, Richard Schwartz, Lin- nea Micciulla, and John Makhoul. 2006. A study of translation edit rate with targeted human annotation. In Proceedings of AMTA.
- Rion Snow, Brendan O'Connor, Daniel Jurafsky, and Andrew Y. Ng. 2008. Cheap and fast -but is it good? Evaluating non-expert annotations for natural language tasks. In Proceedings of EMNLP.
- Omar F. Zaidan and Chris Callison-Burch. 2009. Fea- sibility of human-in-the-loop minimum error rate training. In Proceedings of EMNLP.
- Authors
- Callison-Burch, C.
- Affiliations
- Johns Hopkins University (US)
- Corresponding author
- Callison-Burch, Chris
- Venue
- Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing Volume 1 - EMNLP '09
- Published
- 2009
- Volume
- 1
- Pages
- 286
- OpenAlex
- W2143539737
- Cited by
- 412
- References
- 15
- Citations by year
- 2025: 4 · 2024: 4 · 2023: 7 · 2022: 14 · 2021: 20 · 2020: 13 · 2019: 14 · 2018: 17 · 2017: 28 · 2016: 25 · 2015: 33 · 2014: 52 · 2013: 47 · 2012: 43
- FWCI
- 31.44
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion)
- Funders
- National Science Foundation (0713448)
- European Commission
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Social Sciences › Arts and Humanities › Language and Linguistics › Translation Studies and Practices
- Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
- Keywords
- translation quality assessment, Mechanical Turk, non-expert annotators, crowdsourcing, reading comprehension
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
