Can machine translation systems be evaluated by the crowd alone
Open accessThe DOI matches its Crossref record; 1 of the 1 references it cites were found in a registry or an index.
Abstract
Crowd-sourced assessments of machine translation quality allow evaluations to be carried out cheaply and on a large scale. It is essential, however, that the crowd's work be filtered to avoid contamination of results through the inclusion of false assessments. One method is to filter via agreement with experts, but even amongst experts agreement levels may not be high. In this paper, we present a new methodology for crowd-sourcing human assessments of translation quality, which allows individual workers to develop their own individual assessment strategy. Agreement with experts is no longer required, and a worker is deemed reliable if they are consistent relative to their own previous work. Individual translations are assessed in isolation from all others in the form of direct estimates of translation quality. This allows more meaningful statistics to be computed for systems and enables significance to be determined on smaller sets of assessments. We demonstrate the methodology's feasibility in large-scale human evaluation through replication of the human evaluation component of Workshop on Statistical Machine Translation shared translation task for two language pairs, Spanish-to-English and English-to-Spanish. Results for measurement based solely on crowd-sourced assessments show system rankings in line with those of the original evaluation. Comparison of results produced by the relative preference approach and the direct estimate method described here demonstrate that the direct estimate method has a substantially increased ability to identify significant differences between translation systems.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Inter-Coder Agreement for Computational Linguistics
Artstein, R., & Poesio, M. (2008) · Computational Linguistics · doi:10.1162/coli.07-034-r2 · Cited by 1,665
Research works printed but not linked: 62
- Graham, 2015, Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics and Human Language Technologies, 1183
- Dreyer, 2012, Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 162
- Culy, 2003, Proceedings of the 9th Machine Translation Summit, 71
- Mathet, 2012, Proceedings of the 24th International Conference on Computational Linguistics: Posters, 809
- 10.3115/v1/W14-3333 · DOI · Paper page
- 10.1162/COLI_a_00057 · DOI · Paper page
- Koehn, 2012, Proceedings of the 9th International Workshop on Spoken Language Translation, 179
- Smith, 2005, Proceedings of the International Joint Conference on Artificial Intelligence Workshop on Grammatical Inference Applications, 73
- Schwartz L. , Aikawa T. , and Quirk C. 2003. Disambiguation of English PP attachment using multilingual aligned data. In Proceedings of the 9th Machine Translation Summit, New Orleans, LA.
- Berger, 1994, Proceedings of the 1994 Human Language Technology Workshop, 157
- Callison-Burch, 2010, Proceedings of the Joint 5th Workshop on Statistical Machine Translation and Metrics MATR, 17
- 10.1145/1380584.1380586 · DOI · Paper page
- 10.1177/001316446002000104 · DOI · Paper page
- Yuan, 2013, Proceedings of the 7th Conference on Computational Natural Language Learning, 52
- Lopez, 2012, Proceedings of the 7th Workshop on Statistical Machine Translation, 1
- Kumar, 2004, Proceedings of the Human Language Technology Conference/North American Chapter of the Association for Computational Linguistics Annual Meeting, 169
- Hopkins, 2013, Proceedings of the Fifty-First Annual Meeting of the Association for Computational Linguistics, 1416
- Denkowski M. , and Lavie A. 2010. Choosing the right evaluation for machine translation: an examination of annotator and automatic metric performance on human judgement tasks. In Proceedings of the 12th Conference of the Association of Machine Translation in the Americas, Denver, CO.
- Izumi, 2003, Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics, 145
- Callison-Burch, 2006, Proceedings of the 11th European Chapter of the Association for Computational Linguistics, 249
- 10.3115/v1/W14-3302 · DOI · Paper page
- Turian, 2003, Proceedings of the 9th Machine Translation Summit, 386
- Sakaguchi, 2014, Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, 1
- 10.3115/1654650.1654666 · DOI · Paper page
- 10.1007/s10590-009-9065-6 · DOI
- Callison-Burch, 2008, Proceedings of the 3rd Workshop on Statistical Machine Translation, 70, 10.3115/1626394.1626403 · DOI · Paper page
- Callison-Burch, 2009, Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, 286
- Sjöbergh J. , and Knutsson O. 2005. Faking errors to avoid making errors: very weakly supervised learning for error detection in writing. In Proceedings of Recent Advances in Natural Language Processing, Borovets, Bulgaria.
- Wagner, 2009, Judging grammaticality: experiments in sentence classification, Computer-Assisted Language Instruction Consortium Journal, 26, 474
- Wagner, 2007, Proceedings of the Joint Meeting of the Conference on Empirical Methods in Natural Language Processing and the Conference on Computational Natural Language Learning, 112
- Bojar, 2013, Proceedings of the Eighth Workshop on Statistical Machine Translation, 1
- 10.1162/089120102317341756 · DOI · Paper page
- Yamron, 1994, Proceedings of the 1994 Human Language Technology Workshop, 163
- Callison-Burch, 2012, Proceedings of the 7th Workshop on Statistical Machine Translation, 10
- Bond, 1995, Proceedings of the Second Paclic Association for Computational Linguistics Conference: PACLING-95, 32
- Lo, 2013, Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), 375
- Rozovskaya, 2010, Proceedings of Human Language Technologies: The Eleventh Annual Conference of the North American Chapter of the Association for Computational Linguistics, 154
- Lee, 2008, Proceedings of the 46th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, 174
- Bigert, 2004, Proceedings of the 4th International Conference on Language Resources and Evaluation, 1633
- Bojar, 2011, Proceedings of the 6th Workshop on Statistical Machine Translation, 1
- Och, 2003, Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics, 160
- 10.3115/1626431 · doi:10.3115/1626431
- Graham, 2012, Proceedings of the Australasian Language Technology Workshop, 70
- Dickinson, 2010, Proceedings of the 23rd International Conference on Computational Linguistics, 259
- Graham, 2013, Proceedings Seventh Linguistic Annotation Workshop and Interoperability with Discourse, 33
- Graham Y. , Baldwin T. , Moffat A. , and Zobel J. 2014. Is machine translation getting better over time? In Proceedings of the 14th European Chapter of the Association for Computational Linguistics, Gothenburg, Sweden, pp. 443–51. · DOI
- Poesio, 2005, Proceedings of Frontiers in Corpus Annotations II: Pie in the Sky, 76
- Gupta, 2014, Proceedings of the 18th International Conference on Supporting Group Work, 1
- Okanohara, 2007, Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics, 73
- White, 1994, Proceedings of the First Conference of the Association of Machine Translation in the Americas, 193
- 10.3115/v1/D14-1020 · DOI · Paper page
- Ikehara, 1994, Criteria for evaluating the linguistic quality of Japanese to English machine translations, Journal of the Japanese Society for Artificial Intelligence, 9, 569
- Foster J. 2007. Treebanks gone bad: International Journal of Document Analysis and Recognition (IJDAR) 10(3–4): 129–45. · doi:10.1007/s10032-007-0059-8
- 10.3115/v1/W14-3336 · DOI · Paper page
- Brockett, 2006, Proceedings of the Joint Conference of the International Committee on Computational Linguistics and the Association for Computational Linguistics, 249
- Bigert J. , Sjöbergh J. , Knutsson O. , and Sahlgren M. 2005. Unsupervised evaluation of parser robustness. In Proceedings of the 6th International Conference on Computational Linguistics and Intelligent Text Processing, pp. 142–54. Mexico City, Mexico: Springer. · DOI
- Frederking, 1994, Proceedings of the 1st Conference of the Association of Machine Translation in the Americas, 73
- 10.3115/1626355.1626373 · DOI · Paper page
- Madnani N. , Resnik P. , Dorr B. J. , and Schwartz R. 2008. Are multiple reference translations necessary? Investigating the value of paraphrased reference translations in parameter optimization. In Proceedings of the 10th Conference of the Association of Machine Translation in the Americas, Waikiki, HI.
- Smith, 2005, Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics, 354
- Foster, 2009, Proceedings of the 4th Workshop on Innovative Use of Natural Language Processing for Building Educational Applications, 82
- Callison-Burch, 2011, Proceedings of the 6th Workshop on Statistical Machine Translation, 22
Cited by
COMET: A Neural Framework for MT Evaluation
Rei, R., Stewart, C., Farinha, A. C., & Lavie, A. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.213 · Cited by 617
Findings of the 2016 Conference on Machine Translation
Bojar, O., Chatterjee, R., Federmann, C., Graham, Y., Haddow, B., Huck, M., Jimeno Yepes, A., Koehn, P., Logacheva, V., Monz, C., et al. (2016) · Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers · doi:10.18653/v1/w16-2301 · Cited by 539
“In addition to the standard relative ranking (RR) manual evaluation, this year a new method of human evaluation was also trialed in the main translation task: monolingual direct assessment (DA) of translation fluency (Graham et al., 2013) and adequacy (Graham et al., 2014, 2016).”
MethodSelf-citationAchieving Human Parity on Automatic Chinese to English News Translation
Hassan, H., Aue, A., Chen, C., Chowdhary, V., Clark, J. H., Federmann, C., Huang, X., Junczys-Dowmunt, M., Lewis, W. D., Li, M., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.05567 · Cited by 521
Findings of the 2018 Conference on Machine Translation (WMT18)
Bojar, O., Federmann, C., Fishel, M., Graham, Y., Haddow, B., Koehn, P., & Monz, C. (2018) · Proceedings of the Third Conference on Machine Translation: Shared Task Papers · doi:10.18653/v1/w18-6401 · Cited by 520
Self-citationFindings of the 2019 Conference on Machine Translation (WMT19)
Barrault, L., Bojar, O., Costa-jussà, M. R., Federmann, C., Fishel, M., Graham, Y., Haddow, B., Huck, M., Koehn, P., Malmasi, S., et al. (2019) · Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1) · doi:10.18653/v1/w19-5301 · Cited by 461
Self-citation
Metadata
- Authors
- GRAHAM, Y., BALDWIN, T., MOFFAT, A., & ZOBEL, J.
- Venue
- Natural Language Engineering
- Publisher
- Cambridge University Press (CUP)
- Published
- 2015-09-16
- Volume
- 23
- Issue
- 1
- Pages
- 3-30
- ISSN
- 1351-3249 (ISSN-L)
- 1469-8110
- OpenAlex
- W2260677151
- Cited by
- 140
- References
- 1
- FWCI
- 11.57
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Can machine translation systems be evaluated by the crowd alone. https://app.cochat.ai/papers/10.1017/s1351324915000339
- The paper
- GRAHAM, Y., BALDWIN, T., MOFFAT, A., & ZOBEL, J. (2017). Can machine translation systems be evaluated by the crowd alone. Natural Language Engineering. https://doi.org/10.1017/s1351324915000339
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.