The DOI matches its Crossref record; 10 of the 10 references it cites were found in a registry or an index.
Meteor: an automatic metric for MT evaluation with high levels of correlation with human judgments
Abstract
Meteor is an automatic metric for Machine Translation evaluation which has been demonstrated to have high levels of correlation with human judgments of translation quality, significantly outperforming the more commonly used Bleu metric. It is one of several automatic metrics used in this year's shared task within the ACL WMT-07 workshop. This paper recaps the technical details underlying the metric and describes recent improvements in the metric. The latest release includes improved metric parameters and extends the metric to support evaluation of MT output in Spanish, French and German, in addition to English.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Popular evaluation methods for natural language generation such as BLEU-460, ROUGE-L61, and METEOR62 primarily compare n-grams between two documents and may not effectively capture semantic similarities.”
From Generating dermatopathology reports from gigapixel whole slide images with HistoGPT (2025) · cites this paper as [62]
In the citing paper
“BLEU computes n-gram overlap and is representative for the family of text overlap-based natural language generation metrics such as CIDEr,31 METEOR,30 and ROUGE.32”
From Evaluating progress in automatic chest X-ray radiology report generation (2023) · cites this paper as [30]
Citations
Cites
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344
(2005) · Cited by 4,152
Self-citationMinimum error rate training in statistical machine translation
Och, F. J. (2003) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.3115/1075096.1075117 · Cited by 2,728
A Study of Translation Edit Rate with Targeted Human Annotation
Snover, M., Dorr, B. J., Schwartz, R. M., Micciulla, L., & Makhoul, J. I. (2006) · Cited by 2,380
Confidence estimation for machine translation
Blatz, J., Fitzgerald, E., Foster, G. F., Gandrabur, S., Goutte, C., Kulesza, A., Sanchís, A., & Ueffing, N. (2004) · International Conference on Computational Linguistics · doi:10.3115/1220355.1220401 · Cited by 368
Lattice-based minimum error rate training for statistical machine translation
Macherey, W., Och, F. J., Thayer, I., & Uszkoreit, J. (2008) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/1613715.1613807 · Cited by 347
Precision and recall of machine translation
Melamed, I. D., Green, R., & Turian, J. P. (2003) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.3115/1073483.1073504 · Cited by 228
The Significance of Recall in Automatic Metrics for MT Evaluation
Lavie, A., Sagae, K., & Jayaraman, S. (2004) · Lecture notes in computer science · doi:10.1007/978-3-540-30194-3_16 · Cited by 123
Self-citationCDER: Efficient MT Evaluation Using Block Movements.
Leusch, G., Ueffing, N., & Ney, H. (2006) · RWTH Publications (RWTH Aachen) · Cited by 109
(2024) · ACM eBooks · doi:10.1145/3674127 · Cited by 86
Cited by
Sorted by citations
Prefix-Tuning: Optimizing Continuous Prompts for Generation
Li, X. L., & Liang, P. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.353 · Cited by 2,473
Generation and Comprehension of Unambiguous Object Descriptions
Mao, J., Huang, J., Toshev, A., Camburu, O., Yuille, A., & Murphy, K. P. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.9 · Cited by 1,281
DIALOGPT : Large-Scale Generative Pre-training for Conversational Response Generation
Zhang, Y., Sun, S., Galley, M., Chen, Y.-C., Brockett, C., Gao, X., Gao, J., Liu, J., & Dolan, B. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations · doi:10.18653/v1/2020.acl-demos.30 · Cited by 1,076
Gatt, A., & Krahmer, E. (2018) · Journal of Artificial Intelligence Research · doi:10.1613/jair.5477 · Cited by 776
Variational Autoencoder for Deep Learning of Images, Labels and CaptionsPreprint
Pu, Y., Gan, Z., Henao, R., Yuan, X., Li, C., Stevens, A., & Carin, L. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1609.08976 · Cited by 669
Metadata
- Authors
- Lavie, A., & Agarwal, A.
- Affiliations
- Carnegie Mellon University (US)
- ORCID
- Lavie, Alon
- Venue
- Proceedings of the Second Workshop on Statistical Machine Translation - StatMT '07
- Published
- 2007
- Pages
- 228-231
- OpenAlex
- W2133512280
- Cited by
- 1,047
- References
- 10
- Citations by year
- 2026: 50 · 2025: 104 · 2024: 91 · 2023: 80 · 2022: 78 · 2021: 124 · 2020: 98 · 2019: 84 · 2018: 43 · 2017: 24 · 2016: 29 · 2015: 32 · 2014: 23 · 2013: 23 · 2012: 30
- FWCI
- 21.44
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- Conference on Machine Translation (WMT) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Social Sciences › Arts and Humanities › Language and Linguistics › Translation Studies and Practices
- Keywords
- meteors, machine translation evaluation, automatic evaluation metrics, translation quality, BLEU, human judgment correlation, multilingual evaluation, Spanish language, French language, German
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
