CoChat

Can machine translation systems be evaluated by the crowd alone

Open access
GRAHAM, YVETTE, BALDWIN, TIMOTHY, MOFFAT, ALISTAIR, ZOBEL, JUSTIN · 2017 · Natural Language Engineering
Cited by 140

The DOI matches its Crossref record; 1 of the 1 references it cites were found in a registry or an index.

Abstract

Crowd-sourced assessments of machine translation quality allow evaluations to be carried out cheaply and on a large scale. It is essential, however, that the crowd's work be filtered to avoid contamination of results through the inclusion of false assessments. One method is to filter via agreement with experts, but even amongst experts agreement levels may not be high. In this paper, we present a new methodology for crowd-sourcing human assessments of translation quality, which allows individual workers to develop their own individual assessment strategy. Agreement with experts is no longer required, and a worker is deemed reliable if they are consistent relative to their own previous work. Individual translations are assessed in isolation from all others in the form of direct estimates of translation quality. This allows more meaningful statistics to be computed for systems and enables significance to be determined on smaller sets of assessments. We demonstrate the methodology's feasibility in large-scale human evaluation through replication of the human evaluation component of Workshop on Statistical Machine Translation shared translation task for two language pairs, Spanish-to-English and English-to-Spanish. Results for measurement based solely on crowd-sourced assessments show system rankings in line with those of the original evaluation. Comparison of results produced by the relative preference approach and the direct estimate method described here demonstrate that the direct estimate method has a substantially increased ability to identify significant differences between translation systems.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1017/s1351324915000339

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 1 references

Research works printed but not linked: 62

  1. Graham, 2015, Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics and Human Language Technologies, 1183
  2. Dreyer, 2012, Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 162
  3. Culy, 2003, Proceedings of the 9th Machine Translation Summit, 71
  4. Mathet, 2012, Proceedings of the 24th International Conference on Computational Linguistics: Posters, 809
  5. 10.3115/v1/W14-3333 · DOI · Paper page
  6. 10.1162/COLI_a_00057 · DOI · Paper page
  7. Koehn, 2012, Proceedings of the 9th International Workshop on Spoken Language Translation, 179
  8. Smith, 2005, Proceedings of the International Joint Conference on Artificial Intelligence Workshop on Grammatical Inference Applications, 73
  9. Schwartz L. , Aikawa T. , and Quirk C. 2003. Disambiguation of English PP attachment using multilingual aligned data. In Proceedings of the 9th Machine Translation Summit, New Orleans, LA.
  10. Berger, 1994, Proceedings of the 1994 Human Language Technology Workshop, 157
  11. Callison-Burch, 2010, Proceedings of the Joint 5th Workshop on Statistical Machine Translation and Metrics MATR, 17
  12. 10.1145/1380584.1380586 · DOI · Paper page
  13. 10.1177/001316446002000104 · DOI · Paper page
  14. Yuan, 2013, Proceedings of the 7th Conference on Computational Natural Language Learning, 52
  15. Lopez, 2012, Proceedings of the 7th Workshop on Statistical Machine Translation, 1
  16. Kumar, 2004, Proceedings of the Human Language Technology Conference/North American Chapter of the Association for Computational Linguistics Annual Meeting, 169
  17. Hopkins, 2013, Proceedings of the Fifty-First Annual Meeting of the Association for Computational Linguistics, 1416
  18. Denkowski M. , and Lavie A. 2010. Choosing the right evaluation for machine translation: an examination of annotator and automatic metric performance on human judgement tasks. In Proceedings of the 12th Conference of the Association of Machine Translation in the Americas, Denver, CO.
  19. Izumi, 2003, Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics, 145
  20. Callison-Burch, 2006, Proceedings of the 11th European Chapter of the Association for Computational Linguistics, 249
  21. 10.3115/v1/W14-3302 · DOI · Paper page
  22. Turian, 2003, Proceedings of the 9th Machine Translation Summit, 386
  23. Sakaguchi, 2014, Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, 1
  24. 10.3115/1654650.1654666 · DOI · Paper page
  25. 10.1007/s10590-009-9065-6 · DOI
  26. Callison-Burch, 2008, Proceedings of the 3rd Workshop on Statistical Machine Translation, 70, 10.3115/1626394.1626403 · DOI · Paper page
  27. Callison-Burch, 2009, Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, 286
  28. Sjöbergh J. , and Knutsson O. 2005. Faking errors to avoid making errors: very weakly supervised learning for error detection in writing. In Proceedings of Recent Advances in Natural Language Processing, Borovets, Bulgaria.
  29. Wagner, 2009, Judging grammaticality: experiments in sentence classification, Computer-Assisted Language Instruction Consortium Journal, 26, 474
  30. Wagner, 2007, Proceedings of the Joint Meeting of the Conference on Empirical Methods in Natural Language Processing and the Conference on Computational Natural Language Learning, 112
  31. Bojar, 2013, Proceedings of the Eighth Workshop on Statistical Machine Translation, 1
  32. 10.1162/089120102317341756 · DOI · Paper page
  33. Yamron, 1994, Proceedings of the 1994 Human Language Technology Workshop, 163
  34. Callison-Burch, 2012, Proceedings of the 7th Workshop on Statistical Machine Translation, 10
  35. Bond, 1995, Proceedings of the Second Paclic Association for Computational Linguistics Conference: PACLING-95, 32
  36. Lo, 2013, Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), 375
  37. Rozovskaya, 2010, Proceedings of Human Language Technologies: The Eleventh Annual Conference of the North American Chapter of the Association for Computational Linguistics, 154
  38. Lee, 2008, Proceedings of the 46th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, 174
  39. Bigert, 2004, Proceedings of the 4th International Conference on Language Resources and Evaluation, 1633
  40. Bojar, 2011, Proceedings of the 6th Workshop on Statistical Machine Translation, 1
  41. Och, 2003, Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics, 160
  42. 10.3115/1626431 · doi:10.3115/1626431
  43. Graham, 2012, Proceedings of the Australasian Language Technology Workshop, 70
  44. Dickinson, 2010, Proceedings of the 23rd International Conference on Computational Linguistics, 259
  45. Graham, 2013, Proceedings Seventh Linguistic Annotation Workshop and Interoperability with Discourse, 33
  46. Graham Y. , Baldwin T. , Moffat A. , and Zobel J. 2014. Is machine translation getting better over time? In Proceedings of the 14th European Chapter of the Association for Computational Linguistics, Gothenburg, Sweden, pp. 443–51. · DOI
  47. Poesio, 2005, Proceedings of Frontiers in Corpus Annotations II: Pie in the Sky, 76
  48. Gupta, 2014, Proceedings of the 18th International Conference on Supporting Group Work, 1
  49. Okanohara, 2007, Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics, 73
  50. White, 1994, Proceedings of the First Conference of the Association of Machine Translation in the Americas, 193
  51. 10.3115/v1/D14-1020 · DOI · Paper page
  52. Ikehara, 1994, Criteria for evaluating the linguistic quality of Japanese to English machine translations, Journal of the Japanese Society for Artificial Intelligence, 9, 569
  53. Foster J. 2007. Treebanks gone bad: International Journal of Document Analysis and Recognition (IJDAR) 10(3–4): 129–45. · doi:10.1007/s10032-007-0059-8
  54. 10.3115/v1/W14-3336 · DOI · Paper page
  55. Brockett, 2006, Proceedings of the Joint Conference of the International Committee on Computational Linguistics and the Association for Computational Linguistics, 249
  56. Bigert J. , Sjöbergh J. , Knutsson O. , and Sahlgren M. 2005. Unsupervised evaluation of parser robustness. In Proceedings of the 6th International Conference on Computational Linguistics and Intelligent Text Processing, pp. 142–54. Mexico City, Mexico: Springer. · DOI
  57. Frederking, 1994, Proceedings of the 1st Conference of the Association of Machine Translation in the Americas, 73
  58. 10.3115/1626355.1626373 · DOI · Paper page
  59. Madnani N. , Resnik P. , Dorr B. J. , and Schwartz R. 2008. Are multiple reference translations necessary? Investigating the value of paraphrased reference translations in parameter optimization. In Proceedings of the 10th Conference of the Association of Machine Translation in the Americas, Waikiki, HI.
  60. Smith, 2005, Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics, 354
  61. Foster, 2009, Proceedings of the 4th Workshop on Innovative Use of Natural Language Processing for Building Educational Applications, 82
  62. Callison-Burch, 2011, Proceedings of the 6th Workshop on Statistical Machine Translation, 22

Cited by

See all 140

Metadata

Authors
GRAHAM, Y., BALDWIN, T., MOFFAT, A., & ZOBEL, J.
Venue
Natural Language Engineering
Published
2015-09-16
Volume
23
Issue
1
Pages
3-30
ISSN
  • 1351-3249 (ISSN-L)
  • 1469-8110
OpenAlex
W2260677151
Cited by
140
References
1
FWCI
11.57
Type
journal article
Language
en
Open access
bronze

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Can machine translation systems be evaluated by the crowd alone. https://app.cochat.ai/papers/10.1017/s1351324915000339
The paper
GRAHAM, Y., BALDWIN, T., MOFFAT, A., & ZOBEL, J. (2017). Can machine translation systems be evaluated by the crowd alone. Natural Language Engineering. https://doi.org/10.1017/s1351324915000339

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.