The DOI matches its Crossref record; 32 of the 32 references it cites were found in a registry or an index.
Further meta-evaluation of machine translation
Open accessAbstract
This paper analyzes the translation quality of machine translation systems for 10 language pairs translating between Czech, English, French, German, Hungarian, and Spanish. We report the translation quality of over 30 diverse translation systems based on a large-scale manual evaluation involving hundreds of hours of effort. We use the human judgments of the systems to analyze automatic evaluation metrics for translation quality, and we report the strength of the correlation with human judgments at both the system-level and at the sentence-level. We validate our manual evaluation methodology by measuring intra- and inter-annotator agreement, and collecting timing information.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Measurement of Observer Agreement for Categorical Data
Landis, J. R., & Koch, G. G. (1977) · Biometrics · doi:10.2307/2529310 · Cited by 81,148
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344
A Systematic Comparison of Various Statistical Alignment Models
Och, F. J., & Ney, H. (2003) · Computational Linguistics · doi:10.1162/089120103321337421 · Cited by 3,854
Statistical phrase-based translation
Koehn, P., Och, F. J., & Marcu, D. (2003) · Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology - NAACL '03 · doi:10.3115/1073445.1073462 · Cited by 3,197
Self-citationMinimum error rate training in statistical machine translation
Och, F. J. (2003) · Proceedings of the 41st Annual Meeting on Association for Computational Linguistics - ACL '03 · doi:10.3115/1075096.1075117 · Cited by 2,725
Assessing Agreement on Classification Tasks: The Kappa Statistic
Carletta, J. (1996) · arXiv · doi:10.48550/arxiv.cmp-lg/9602004 · Cited by 2,088 · cited 1×
“We measured pairwise agreement among annotators using the kappa coefficient (K) which is widely used in computational linguistics for measuring agreement in category judgments (Carletta, 1996).”
MethodStatistical Phrase-Based Translation
Koehn, P., Och, F. J., & Marcu, D. (2003) · doi:10.21236/ada461156 · Cited by 965
Self-citationRe-evaluating the Role of Bleu in Machine Translation Research
Callison-Burch, C., Osborne, M., & Koehn, P. (2006) · Edinburgh Research Explorer (University of Edinburgh) · Cited by 632
Self-citation(Meta-) evaluation of machine translation
Callison-Burch, C., Fordyce, C., Koehn, P., Monz, C., & Schroeder, J. (2007) · Proceedings of the Second Workshop on Statistical Machine Translation - StatMT '07 · doi:10.3115/1626355.1626373 · Cited by 379
Self-citationLattice-based minimum error rate training for statistical machine translation
Macherey, W., Och, F. J., Thayer, I., & Uszkoreit, J. (2008) · Proceedings of the Conference on Empirical Methods in Natural Language Processing - EMNLP '08 · doi:10.3115/1613715.1613807 · Cited by 346
Experiments in domain adaptation for statistical machine translation
Koehn, P., & Schroeder, J. (2007) · Proceedings of the Second Workshop on Statistical Machine Translation - StatMT '07 · doi:10.3115/1626355.1626388 · Cited by 320
Self-citationManual and automatic evaluation of machine translation between European languages
Koehn, P., & Monz, C. (2006) · Proceedings of the Workshop on Statistical Machine Translation - StatMT '06 · doi:10.3115/1654650.1654666 · Cited by 269
Self-citationDesign of a multi-lingual, parallel-processing statistical parsing engine
Bikel, D. M. (2002) · International Conference on Human Language Technology Research · doi:10.3115/1289189.1289191 · Cited by 135
Word error rates: decomposition over Pos classes and applications for error analysis
Popović, M., & Ney, H. (2007) · Conference on Machine Translation (WMT) · doi:10.3115/1626355.1626362 · Cited by 119
Improved Word-Level System Combination for Machine Translation
Rosti, A.-V. I., Matsoukas, S., & Schwartz, R. M. (2007) · Cited by 114
Agarwal, A., & Lavie, A. (2008) · Conference on Machine Translation (WMT) · doi:10.3115/1626394.1626406 · Cited by 110
A Learning Approach to Improving Sentence-Level MT Evaluation
Kulesza, A., & Shieber, S. M. (2004) · Cited by 89
Žabokrtský, Z., Ptáček, J., & Pajas, P. (2008) · Conference on Machine Translation (WMT) · doi:10.3115/1626394.1626419 · Cited by 86
Regression for Sentence-Level MT Evaluation with Pseudo References
Albrecht, J. S., & Hwa, R. (2007) · Cited by 81
A Re-examination of Machine Learning Approaches for Sentence-Level MT Evaluation
Albrecht, J. S., & Hwa, R. (2007) · Cited by 67
A smorgasbord of features for automatic MT evaluation
Giménez, J., & Màrquez, L. (2008) · Conference on Machine Translation (WMT) · doi:10.3115/1626394.1626426 · Cited by 63
Using Moses to integrate multiple rule-based machine translation engines into a hybrid system
Eisele, A., Federmann, C., Saint-Amand, H., Jellinghaus, M., Herrmann, T., & Chen, Y. (2008) · Conference on Machine Translation (WMT) · doi:10.3115/1626394.1626422 · Cited by 54
Ranking vs. regression in machine translation evaluation
Duh, K. (2008) · Conference on Machine Translation (WMT) · doi:10.3115/1626394.1626425 · Cited by 54
Rosti, A.-V. I., Zhang, B., Matsoukas, S., & Schwartz, R. M. (2008) · Conference on Machine Translation (WMT) · doi:10.3115/1626394.1626423 · Cited by 51
Rich source-side context for statistical machine translation
Gimpel, K., & Smith, N. A. (2008) · Conference on Machine Translation (WMT) · doi:10.3115/1626394.1626396 · Cited by 50
MATREX: The DCU MT System for WMT 2010
Penkale, S., Haque, R., Dandapat, S., Banerjee, P., Srivastava, A. K., Du, J., Pecina, P., Naskar, S. K., Forcada, M. L., & Way, A. (2010) · Arrow@dit (Dublin Institute of Technology) · Cited by 41
Towards better machine translation quality for the German--English language pairs
Koehn, P., Arun, A., & Hoang, H. (2008) · Conference on Machine Translation (WMT) · doi:10.3115/1626394.1626412 · Cited by 31
Self-citationPhrase-based and deep syntactic English-to-Czech statistical machine translation
Bojar, O., & Hajič, J. (2008) · Conference on Machine Translation (WMT) · doi:10.3115/1626394.1626413 · Cited by 28
The role of pseudo references in MT evaluation
Albrecht, J. S., & Hwa, R. (2008) · Conference on Machine Translation (WMT) · doi:10.3115/1626394.1626424 · Cited by 28
Effects of morphological analysis in translation between German and English
Stymne, S., Holmqvist, M., & Ahrenberg, L. (2008) · Conference on Machine Translation (WMT) · doi:10.3115/1626394.1626411 · Cited by 27
LIMSI's statistical translation systems for WMT'08
Déchelotte, D., Adda, G., Allauzen, A., Bonneau-Maynard, H., Galibert, O., Gauvain, J., Langlais, P., & Yvon, F. (2008) · Conference on Machine Translation (WMT) · doi:10.3115/1626394.1626404 · Cited by 25
Statistical transfer systems for French--English and German--English machine translation
Hanneman, G., Huber, E., Agarwal, A., Ambati, V., Parlikar, A., Peterson, E. J., & Lavie, A. (2008) · Conference on Machine Translation (WMT) · doi:10.3115/1626394.1626418 · Cited by 8
Cited by
Gatt, A., & Krahmer, E. (2018) · Journal of Artificial Intelligence Research · doi:10.1613/jair.5477 · Cited by 776
Findings of the 2016 Conference on Machine Translation
Bojar, O., Chatterjee, R., Federmann, C., Graham, Y., Haddow, B., Huck, M., Jimeno Yepes, A., Koehn, P., Logacheva, V., Monz, C., et al. (2016) · Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers · doi:10.18653/v1/w16-2301 · Cited by 539
“This conference builds on nine previous WMT workshops (Koehn and Monz, 2006; Callison-Burch et al., 2007, 2008, 2009, 2010, 2011, 2012; Bojar et al., 2013, 2014, 2015).”
BackgroundSelf-citationFindings of the 2014 Workshop on Statistical Machine Translation
Bojar, O., Buck, C., Federmann, C., Haddow, B., Koehn, P., Leveling, J., Monz, C., Pecina, P., Post, M., Saint-Amand, H., et al. (2014) · Proceedings of the Ninth Workshop on Statistical Machine Translation · doi:10.3115/v1/w14-3302 · Cited by 529
Self-citationFindings of the 2018 Conference on Machine Translation (WMT18)
Bojar, O., Federmann, C., Fishel, M., Graham, Y., Haddow, B., Koehn, P., & Monz, C. (2018) · Proceedings of the Third Conference on Machine Translation: Shared Task Papers · doi:10.18653/v1/w18-6401 · Cited by 520
Self-citationFindings of the 2019 Conference on Machine Translation (WMT19)
Barrault, L., Bojar, O., Costa-jussà, M. R., Federmann, C., Fishel, M., Graham, Y., Haddow, B., Huck, M., Koehn, P., Malmasi, S., et al. (2019) · Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1) · doi:10.18653/v1/w19-5301 · Cited by 461
Self-citation
Metadata
- Authors
- Callison-Burch, C., Fordyce, C., Koehn, P., Monz, C., & Schroeder, J.
- Venue
- Proceedings of the Third Workshop on Statistical Machine Translation - StatMT '08
- Published
- 2008
- Pages
- 70-106
- OpenAlex
- W2169279899
- Cited by
- 267
- References
- 32
- FWCI
- 28.53
- Type
- conference paper
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.