Meteor: an automatic metric for MT evaluation with high levels of correlation with human judgments
Abstract
Meteor is an automatic metric for Machine Translation evaluation which has been demonstrated to have high levels of correlation with human judgments of translation quality, significantly outperforming the more commonly used Bleu metric. It is one of several automatic metrics used in this year's shared task within the ACL WMT-07 workshop. This paper recaps the technical details underlying the metric and describes recent improvements in the metric. The latest release includes improved metric parameters and extends the metric to support evaluation of MT output in Spanish, French and German, in addition to English.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Popular evaluation methods for natural language generation such as BLEU-4, ROUGE-L, and METEOR primarily compare n-grams between two documents and may not effectively capture semantic similarities.
“Popular evaluation methods for natural language generation such as BLEU-460, ROUGE-L61, and METEOR62 primarily compare n-grams between two documents and may not effectively capture semantic similarities.”
From Generating dermatopathology reports from gigapixel whole slide images with HistoGPT (2025) · cites this paper as [62]
BLEU computes n-gram overlap and is representative for the family of text overlap-based natural language generation metrics such as CIDEr, METEOR, and ROUGE.
“BLEU computes n-gram overlap and is representative for the family of text overlap-based natural language generation metrics such as CIDEr,31 METEOR,30 and ROUGE.32”
From Evaluating progress in automatic chest X-ray radiology report generation (2023) · cites this paper as [30]
Cites
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344 · see citing works
(2005) · Cited by 4,152 · see citing works
Self-citationMinimum error rate training in statistical machine translation
Och, F. J. (2003) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.3115/1075096.1075117 · Cited by 2,728 · see citing works
A Study of Translation Edit Rate with Targeted Human Annotation
Snover, M., Dorr, B. J., Schwartz, R. M., Micciulla, L., & Makhoul, J. I. (2006) · Cited by 2,380 · see citing works
Confidence estimation for machine translation
Blatz, J., Fitzgerald, E., Foster, G. F., Gandrabur, S., Goutte, C., Kulesza, A., Sanchís, A., & Ueffing, N. (2004) · International Conference on Computational Linguistics · doi:10.3115/1220355.1220401 · Cited by 356 · see citing works
Lattice-based minimum error rate training for statistical machine translation
Macherey, W., Och, F. J., Thayer, I., & Uszkoreit, J. (2008) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/1613715.1613807 · Cited by 347 · see citing works
Precision and recall of machine translation
Melamed, I. D., Green, R., & Turian, J. P. (2003) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.3115/1073483.1073504 · Cited by 228 · see citing works
The Significance of Recall in Automatic Metrics for MT Evaluation
Lavie, A., Sagae, K., & Jayaraman, S. (2004) · Lecture notes in computer science · doi:10.1007/978-3-540-30194-3_16 · Cited by 123 · see citing works
Self-citationCDER: Efficient MT Evaluation Using Block Movements.
Leusch, G., Ueffing, N., & Ney, H. (2006) · RWTH Publications (RWTH Aachen) · Cited by 109 · see citing works
(2024) · ACM eBooks · doi:10.1145/3674127 · Cited by 82 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Prefix-Tuning: Optimizing Continuous Prompts for Generation
Li, X. L., & Liang, P. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.353 · Cited by 2,473 · see citing works
Generation and Comprehension of Unambiguous Object Descriptions
Mao, J., Huang, J., Toshev, A., Camburu, O., Yuille, A., & Murphy, K. P. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.9 · Cited by 1,211 · see citing works
DIALOGPT : Large-Scale Generative Pre-training for Conversational Response Generation
Zhang, Y., Sun, S., Galley, M., Chen, Y.-C., Brockett, C., Gao, X., Gao, J., Liu, J., & Dolan, B. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations · doi:10.18653/v1/2020.acl-demos.30 · Cited by 1,076 · see citing works
Gatt, A., & Krahmer, E. (2018) · Journal of Artificial Intelligence Research · doi:10.1613/jair.5477 · Cited by 776 · see citing works
Variational Autoencoder for Deep Learning of Images, Labels and CaptionsPreprint
Pu, Y., Gan, Z., Henao, R., Yuan, X., Li, C., Stevens, A., & Carin, L. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1609.08976 · Cited by 662 · see citing works
SummEval: Re-evaluating Summarization Evaluation
Fabbri, A. R., Kryściński, W., McCann, B., Xiong, C., Socher, R., & Radev, D. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00373 · Cited by 587 · see citing works
ECO: Efficient Convolutional Network for Online Video Understanding
Zolfaghari, M., Singh, K., & Brox, T. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01216-8_43 · Cited by 541 · see citing works
MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance
Zhao, W., Peyrard, M., Liu, F., Gao, Y., Meyer, C. M., & Eger, S. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1053 · Cited by 490 · see citing works
Guiding the Long-Short Term Memory Model for Image Caption Generation
Jia, X., Gavves, E., Fernando, B., & Tuytelaars, T. (2015) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.277 · Cited by 469 · see citing works
Are You Talking to a Machine? Dataset and Methods for Multilingual Image Question AnsweringPreprint
Gao, H., Mao, J., Zhou, J., Huang, Z., Wang, L., & Xu, W. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1505.05612 · Cited by 437 · see citing works
Why We Need New Evaluation Metrics for NLG
Novikova, J., Dušek, O., Cercas Curry, A., & Rieser, V. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1238 · Cited by 422 · see citing works
Improving automatic source code summarization via deep reinforcement learning
Wan, Y., Zhao, Z., Yang, M., Xu, G., Ying, H., Wu, J., & Yu, P. S. (2018) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3238147.3238206 · Cited by 392 · see citing works
nocaps: novel object captioning at scale
Agrawal, H., Desai, K., Wang, Y., Chen, X., Jain, R., Johnson, M., Batra, D., Parikh, D., Lee, S., & Anderson, P. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00904 · Cited by 386 · see citing works
(Meta-) evaluation of machine translation
Callison-Burch, C., Fordyce, C. S., Koehn, P., Monz, C., & Schroeder, J. (2007) · Conference on Machine Translation (WMT) · doi:10.3115/1626355.1626373 · Cited by 381 · see citing works
In Defense of Grid Features for Visual Question Answering
Jiang, H., Misra, I., Rohrbach, M., Miller, E. G., & Chen, X. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01028 · Cited by 379 · see citing works
(2020) · doi:10.18653/v1/2020.acl-main.450 · Cited by 356 · see citing works
The Meteor metric for automatic evaluation of machine translation
Lavie, A., & Denkowski, M. (2009) · Machine Translation · doi:10.1007/s10590-009-9059-4 · Cited by 351 · see citing works
Self-citationFindings of the 2015 Workshop on Statistical Machine Translation
Bojar, O., Chatterjee, R., Federmann, C., Haddow, B., Huck, M., Hokamp, C., Koehn, P., Logacheva, V., Monz, C., Negri, M., et al. (2015) · Conference on Machine Translation (WMT) · doi:10.18653/v1/w15-3001 · Cited by 287 · see citing works
The E2E Dataset: New Challenges For End-to-End Generation
Novikova, J., Dušek, O., & Rieser, V. (2017) · Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/w17-5525 · Cited by 275 · see citing works
How Much Can CLIP Benefit Vision-and-Language Tasks?Preprint
Shen, S., Li, L. H., Tan, H. X., Bansal, M., Rohrbach, A., Chang, K.-W., Yao, Z., & Keutzer, K. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.06383 · Cited by 272 · see citing works
DialoGPT: Large-Scale Generative Pre-training for Conversational Response GenerationPreprint
Zhang, Y., Sun, S., Galley, M., Chen, Y.-C., Brockett, C., Gao, X., Gao, J., Liu, J., & Dolan, B. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.00536 · Cited by 258 · see citing works
Neural Paraphrase Generation with Stacked Residual LSTM NetworksPreprint
Prakash, A., Hasan, S. A., Lee, K., Datla, V. V., Qadir, A., Liu, J., & Farri, O. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.03098 · Cited by 245 · see citing works
Adversarial text-to-image synthesis: A review
Frolov, S., Hinz, T., Raue, F., Hees, J., & Dengel, A. R. (2021) · Neural Networks · doi:10.1016/j.neunet.2021.07.019 · Cited by 214 · see citing works
Paraphrase Generation with Deep Reinforcement Learning
Li, Z., Jiang, X., Shang, L., & Li, H. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1421 · Cited by 211 · see citing works
Evaluation of Text Generation: A Survey
Celikyilmaz, A., Clark, E., & Gao, J. (2020) · arXiv · doi:10.48550/arxiv.2006.14799 · Cited by 207 · see citing works
A Review on Explainability in Multimodal Deep Neural Nets
Joshi, G., Walambe, R., & Kotecha, K. (2021) · IEEE Access · doi:10.1109/access.2021.3070212 · Cited by 204 · see citing works
Evaluating progress in automatic chest X-ray radiology report generation
Yu, F., Endo, M., Krishnan, R., Pan, I., Tsai, A., Reis, E. P., Fonseca, E. K. U. N., Lee, H. M. H., Abad, Z. S. H., Ng, A. Y., et al. (2023) · Patterns · doi:10.1016/j.patter.2023.100802 · Cited by 200 · see citing works
Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metrics
Pagnoni, A., Balachandran, V., & Tsvetkov, Y. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.383 · Cited by 197 · see citing works
A Deep Generative Framework for Paraphrase Generation
Gupta, A., Agarwal, A. K., Singh, P., & Rai, P. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11956 · Cited by 196 · see citing works
Self-citationEvaluating the state-of-the-art of End-to-End Natural Language Generation: The E2E NLG challenge
Dušek, O., Novikova, J., & Rieser, V. (2019) · Computer Speech & Language · doi:10.1016/j.csl.2019.06.009 · Cited by 194 · see citing works
Explainable Deep Learning Methods in Medical Image Classification: A Survey
Patrício, C., Neves, J. C., & Teixeira, L. F. (2023) · ACM Computing Surveys · doi:10.1145/3625287 · Cited by 190 · see citing works
Liu, C., Zhao, R., Chen, H., Zou, Z., & Shi, Z. (2022) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2022.3218921 · Cited by 182 · see citing works
Know More Say Less: Image Captioning Based on Scene Graphs
Li, X., & Jiang, S. (2019) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2019.2896516 · Cited by 181 · see citing works
MedDialog: Large-scale Medical Dialogue Datasets
Zeng, G., Yang, W., Ju, Z., Yang, Y., Wang, S., Zhang, R., Zhou, M., Zeng, J., Dong, X., Zhang, R., et al. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.743 · Cited by 178 · see citing works
Transformer models used for text-based question answering systems
Nassiri, K., & Akhloufi, M. A. (2022) · Applied Intelligence · doi:10.1007/s10489-022-04052-8 · Cited by 177 · see citing works
ILDC for CJPE: Indian Legal Documents Corpus for Court Judgment Prediction and Explanation
Malik, V., Sanjay, R., Nigam, S. K., Ghosh, K., Guha, S. K., Bhattacharya, A., & Modi, A. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.313 · Cited by 176 · see citing works
QuestEval: Summarization Asks for Fact-based Evaluation
Scialom, T., Dray, P.-A., Lamprier, S., Piwowarski, B., Staiano, J., Wang, A., & Gallinari, P. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.529 · Cited by 175 · see citing works
Visual Question Generation as Dual Task of Visual Question Answering
Li, Y., Duan, N., Zhou, B., Chu, X., Ouyang, W., Wang, X., & Zhou, M. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00640 · Cited by 171 · see citing works
Selective Attention for Context-aware Neural Machine Translation
Maruf, S., Martins, A. F. T., & Haffari, G. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1313 · Cited by 168 · see citing works
TER-Plus: paraphrase, semantic, and alignment enhancements to Translation Edit Rate
Snover, M., Madnani, N., Dorr, B. J., & Schwartz, R. M. (2009) · Machine Translation · doi:10.1007/s10590-009-9062-9 · Cited by 167 · see citing works
Machine translation evaluation versus quality estimation
Specia, L., Raj, D., & Turchi, M. (2010) · Machine Translation · doi:10.1007/s10590-010-9077-2 · Cited by 165 · see citing works
Assessing The Factual Accuracy of Generated Text
Goodrich, B., Rao, V., Liu, P. J., & Saleh, M. (2019) · Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3292500.3330955 · Cited by 162 · see citing works
Ground Truth for Grammaticality Correction Metrics
Napoles, C., Sakaguchi, K., Post, M., & Tetreault, J. (2015) · doi:10.3115/v1/p15-2097 · Cited by 151 · see citing works
Active Learning and Crowd-Sourcing for Machine Translation
Ambati, V., Vogel, S., & Carbonell, J. (2010) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/5dqoxv96xk95 · Cited by 150 · see citing works
Document Context Neural Machine Translation with Memory Networks
Maruf, S., & Haffari, G. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-1118 · Cited by 150 · see citing works
A Survey on Document-level Neural Machine Translation
Maruf, S., Saleh, F., & Haffari, G. (2021) · ACM Computing Surveys · doi:10.1145/3441691 · Cited by 148 · see citing works
Learning Like a Child: Fast Novel Visual Concept Learning from Sentence Descriptions of Images
Mao, J., Xu, W., Yang, Y., Wang, J., Huang, Z., & Yuille, A. (2015) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.291 · Cited by 147 · see citing works
Approaches to Human and Machine Translation Quality Assessment
Castilho, S., Doherty, S., Gaspari, F., & Moorkens, J. (2018) · Machine translation · doi:10.1007/978-3-319-91241-7_2 · Cited by 145 · see citing works
Text-to-Text Pre-Training for Data-to-Text Tasks
Kale, M., & Rastogi, A. (2020) · International Conference on Natural Language Generation · doi:10.18653/v1/2020.inlg-1.14 · Cited by 145 · see citing works
CharacTer: Translation Edit Rate on Character Level
Wang, W., Peter, J.-T., Rosendahl, H., & Ney, H. (2016) · Conference on Machine Translation (WMT) · doi:10.18653/v1/w16-2342 · Cited by 134 · see citing works
Region-Object Relation-Aware Dense Captioning via Transformer
Shao, Z., Han, J., Marnerides, D., & Debattista, K. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3152990 · Cited by 133 · see citing works
Ramanishka, V., Das, A., Park, D. H., Venugopalan, S., Hendricks, L. A., Rohrbach, M., & Saenko, K. (2016) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/2964284.2984066 · Cited by 132 · see citing works
Grounded Conversation Generation as Guided Traverses in Commonsense Knowledge Graphs
Zhang, H., Liu, Z., Xiong, C., & Liu, Z. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.184 · Cited by 131 · see citing works
Sezgın, E., Sirrianni, J., & Linwood, S. L. (2022) · JMIR Medical Informatics · doi:10.2196/32875 · Cited by 129 · see citing works
A Neural Architecture for Generating Natural Language Descriptions from Source Code Changes
Loyola, P., Marrese-Taylor, E., & Matsuo, Y. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/p17-2045 · Cited by 128 · see citing works
A long video caption generation algorithm for big video data retrieval
Ding, S., Qu, S., Xi, Y., & Wan, S. (2018) · Future Generation Computer Systems · doi:10.1016/j.future.2018.10.054 · Cited by 128 · see citing works
Lattice Minimum Bayes-Risk decoding for statistical machine translation
Tromble, R. W., Kumar, S., Och, F. J., & Macherey, W. (2008) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/1613715.1613792 · Cited by 127 · see citing works
Discriminative corpus weight estimation for machine translation
Matsoukas, S., Rosti, A.-V. I., & Zhang, B. (2009) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/1699571.1699605 · Cited by 123 · see citing works
Large Language Model Assisted Software Engineering: Prospects, Challenges, and a Case Study
Belzner, L., Gabor, T., & Wirsing, M. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-46002-9_23 · Cited by 118 · see citing works
FINDINGS OF THE IWSLT 2020 EVALUATION CAMPAIGN
Ansari, E., Axelrod, A. E., Bách, N., Bojar, O., Cattoni, R., Dalvi, F., Durrani, N., Federico, M., Federmann, C., Gu, J., et al. (2020) · International Conference on Spoken Language Translation (IWSLT) · doi:10.18653/v1/2020.iwslt-1.1 · Cited by 111 · see citing works
Changes to Captions: An Attentive Network for Remote Sensing Change Captioning
Chang, S., & Ghamisi, P. (2023) · IEEE Transactions on Image Processing · doi:10.1109/tip.2023.3328224 · Cited by 112 · see citing works
High-Resolution Remote Sensing Image Captioning Based on Structured Attention
Zhao, R., Shi, Z., & Zou, Z. (2021) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2021.3070383 · Cited by 109 · see citing works
Agarwal, A., & Lavie, A. (2008) · Conference on Machine Translation (WMT) · doi:10.3115/1626394.1626406 · Cited by 109 · see citing works
Self-citationA Survey on Deep Learning and Explainability for Automatic Report Generation from Medical Images
Messina, P., Pino, P., Parra, D., Soto, Á., Besa, C., Uribe, S. A., Andía, M. E., Cristián Tejos, N., Prieto, C., & Daniel Capurro, N. (2022) · ACM Computing Surveys · doi:10.1145/3522747 · Cited by 103 · see citing works
Automated audio captioning with recurrent neural networks
Drossos, K., Adavanne, S., & Virtanen, T. I. (2017) · IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA) · doi:10.1109/waspaa.2017.8170058 · Cited by 102 · see citing works
deltaBLEU: A Discriminative Metric for Generation Tasks with Intrinsically Diverse Targets
Galley, M., Brockett, C., Sordoni, A., Ji, Y., Auli, M., Quirk, C., Mitchell, M., Gao, J., & Dolan, B. (2015) · doi:10.3115/v1/p15-2073 · Cited by 101 · see citing works
Image Captioning: A Comprehensive Survey
Sharma, H., Agrahari, M., Singh, S. K., Firoj, M., & Mishra, R. K. (2020) · International Conference on Power Electronics & IoT Applications in Renewable Energy and its Control (PARC) · doi:10.1109/parc49193.2020.236619 · Cited by 100 · see citing works
Learning to rank with partially-labeled data
Duh, K., & Kirchhoff, K. (2008) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/1390334.1390379 · Cited by 100 · see citing works
Toward Subgraph-Guided Knowledge Graph Question Generation With Graph Neural Networks
Chen, Y., Wu, L., & Zaki, M. J. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3264519 · Cited by 98 · see citing works
Conversing by Reading: Contentful Neural Conversation with On-demand Machine Reading
Qin, L., Galley, M., Brockett, C., Liu, X., Gao, X., Dolan, B., Choi, Y., & Gao, J. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1539 · Cited by 97 · see citing works
Neural data-to-text generation: A comparison between pipeline and end-to-end architectures
Ferreira, T. C., van der Lee, C., van Miltenburg, E., & Krahmer, E. (2019) · Tilburg University Research Portal · Cited by 97 · see citing works
Automatic Source Code Summarization with Extended Tree-LSTM
Shido, Y., Kobayashi, Y., Yamamoto, A., Miyamoto, A., & Matsumura, T. (2019) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2019.8851751 · Cited by 97 · see citing works
Sampling alignment structure under a Bayesian translation model
DeNero, J., Bouchard‐Côté, A., & Klein, D. (2008) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/1613715.1613758 · Cited by 97 · see citing works
TrustRank: Inducing Trust in Automatic Translations via Ranking
Soricut, R., & Echihabi, A. (2010) · Cited by 95 · see citing works
Text2Scene: Generating Compositional Scenes From Textual Descriptions
Tan, F., Feng, S., & Ordóñez, V. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00687 · Cited by 95 · see citing works
High-Order Interaction Learning for Image Captioning
Wang, Y., Xu, N., Liu, A.-A., Li, W., & Zhang, Y. (2021) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2021.3121062 · Cited by 95 · see citing works
POINTER: Constrained Progressive Text Generation via Insertion-based Generative Pre-training
Zhang, Y., Wang, G., Li, C., Gan, Z., Brockett, C., & Dolan, B. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.698 · Cited by 94 · see citing works
Large Language Models in Mental Health Care: A Scoping Review
Hua, Y., Liu, F., Yang, K., Li, Z., Na, H., Sheu, Y.-H., Zhou, P., Moran, L. V., Ananiadou, S., Clifton, D. A., et al. (2025) · Current Treatment Options in Psychiatry · doi:10.1007/s40501-025-00363-y · Cited by 92 · see citing works
Global Visual Feature and Linguistic State Guided Attention for Remote Sensing Image Captioning
Zhang, Z., Zhang, W., Yan, M., Gao, X., Fu, K., & Sun, X. (2021) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2021.3132095 · Cited by 92 · see citing works
Multimodal machine translation through visuals and speech
Sulubacak, U., Çağlayan, O., Grönroos, S.-A., Rouhe, A., Elliott, D., Specia, L., & Tiedemann, J. (2020) · Machine Translation · doi:10.1007/s10590-020-09250-0 · Cited by 91 · see citing works
A Decoupling Paradigm With Prompt Learning for Remote Sensing Image Change Captioning
Liu, C., Zhao, R., Chen, J., Qi, Z., Zou, Z., & Shi, Z. (2023) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2023.3321752 · Cited by 92 · see citing works
BEER: BEtter Evaluation as Ranking
Stanojević, M., & Sima’an, K. (2014) · Conference on Machine Translation (WMT) · doi:10.3115/v1/w14-3354 · Cited by 91 · see citing works
Inducing Sentence Structure from Parallel Corpora for Reordering
DeNero, J., & Uszkoreit, J. (2011) · Cited by 90 · see citing works
deltaBLEU: A Discriminative Metric for Generation Tasks with Intrinsically Diverse TargetsPreprint
Galley, M., Brockett, C., Sordoni, A., Ji, Y., Auli, M., Quirk, C., Mitchell, M., Gao, J., & Dolan, B. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.06863 · Cited by 90 · see citing works
Very Deep Transformers for Neural Machine TranslationPreprint
Liu, X., Duh, K., Liu, L., & Gao, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.07772 · Cited by 88 · see citing works
Semantic Noise Matters for Neural Natural Language Generation
Dušek, O., Howcroft, D. M., & Rieser, V. (2019) · Proceedings of the 12th International Conference on Natural Language Generation · doi:10.18653/v1/w19-8652 · Cited by 85 · see citing works
Semantic Graphs for Generating Deep Questions
Pan, L., Xie, Y., Feng, Y., Chua, T., & Kan, M. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.135 · Cited by 85 · see citing works
A survey of types of text noise and techniques to handle noisy text
Subramaniam, L. V., Roy, S., Faruquie, T. A., & Negi, S. (2009) · Analytics for Noisy Unstructured Text Data · doi:10.1145/1568296.1568315 · Cited by 85 · see citing works
Large-Scale Transfer Learning for Natural Language Generation
Golovanov, S., Kurbanov, R., Nikolenko, S., Truskovskyi, K., Tselousov, A., & Wolf, T. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1608 · Cited by 84 · see citing works
Ensemble-based deep reinforcement learning for chatbots
Cuayáhuitl, H., Lee, D., Ryu, S., Cho, Y. J., Choi, S., Indurthi, S., Yu, S., Choi, H., Hwang, I.-C., & Kim, J. (2019) · Neurocomputing · doi:10.1016/j.neucom.2019.08.007 · Cited by 84 · see citing works
Change-Agent: Toward Interactive Comprehensive Remote Sensing Change Interpretation and Analysis
Liu, C., Chen, K., Zhang, H., Qi, Z., Zou, Z., & Shi, Z. (2024) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2024.3425815 · Cited by 80 · see citing works
Automatic medical image interpretation: State of the art and future directions
Ayesha, H., Iqbal, S., Tariq, M., Abrar, M., Sanaullah, M., Abbas, I., Rehman, A. U., Niazi, M. F. K., & Hussain, S. (2021) · Pattern Recognition · doi:10.1016/j.patcog.2021.107856 · Cited by 79 · see citing works
A Review on the Use of Large Language Models as Virtual Tutors
García-Méndez, S., de Arriba-Pérez, F., & Somoza-López, M. D. C. (2024) · Science & Education · doi:10.1007/s11191-024-00530-2 · Cited by 78 · see citing works
Diagnosing meaning errors in short answers to reading comprehension questions
Bailey, S. M., & Meurers, D. (2008) · doi:10.3115/1631836.1631849 · Cited by 78 · see citing works
Multimodal Attention for Neural Machine TranslationPreprint
Çağlayan, O., Barrault, L., & Bougares, F. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1609.03976 · Cited by 78 · see citing works
Non-Monotonic Sequential Text GenerationPreprint
Welleck, S., Brantley, K., Daumé, H., & Cho, K. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.02192 · Cited by 77 · see citing works
Scheduled Multi-Task Learning: From Syntax to Translation
Kiperwasser, E., & Ballesteros, M. (2018) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00017 · Cited by 76 · see citing works
Deep learning for ultrasound image caption generation based on object detection
Zeng, X., Wen, L., Liu, B., & Qi, X. (2019) · Neurocomputing · doi:10.1016/j.neucom.2018.11.114 · Cited by 76 · see citing works
Spatio-Temporal Memory Attention for Image Captioning
Ji, J., Xu, C., Zhang, X., Wang, B., & Song, X. (2020) · IEEE Transactions on Image Processing · doi:10.1109/tip.2020.3004729 · Cited by 76 · see citing works
MAXSIM: A Maximum Similarity Metric for Machine Translation Evaluation
Chan, Y. S., & Ng, H. T. (2008) · Cited by 76 · see citing works
- Authors
- Lavie, A., & Agarwal, A.
- Affiliations
- Carnegie Mellon University (US)
- ORCID
- Lavie, Alon
- Venue
- Proceedings of the Second Workshop on Statistical Machine Translation - StatMT '07
- Published
- 2007
- Pages
- 228-231
- OpenAlex
- W2133512280
- Cited by
- 1,047
- References
- 10
- Citations by year
- 2026: 50 · 2025: 104 · 2024: 91 · 2023: 80 · 2022: 78 · 2021: 124 · 2020: 98 · 2019: 84 · 2018: 43 · 2017: 24 · 2016: 29 · 2015: 32 · 2014: 23 · 2013: 23 · 2012: 30
- FWCI
- 21.44
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- Conference on Machine Translation (WMT) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Social Sciences › Arts and Humanities › Language and Linguistics › Translation Studies and Practices
- Keywords
- meteors, machine translation evaluation, automatic evaluation metrics, translation quality, BLEU, human judgment correlation, multilingual evaluation, Spanish language, French language, German
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review