Automatic evaluation of machine translation quality using longest common subsequence and skip-bigram statistics
Open accessAbstract
In this paper we describe two new objective automatic evaluation methods for machine translation. The first method is based on longest common subsequence between a candidate translation and a set of reference translations. Longest common subsequence takes into account sentence level structure similarity naturally and identifies longest co-occurring in-sequence n-grams automatically. The second method relaxes strict n-gram matching to skip-bigram matching. Skip-bigram is any pair of words in their sentence order. Skip-bigram cooccurrence statistics measure the overlap of skip-bigrams between a candidate translation and a set of reference translations. The empirical results show that both methods correlate with human judgments very well in both adequacy and fluency.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344 · see citing works
Binary Codes capable of currecting deletions, insertions, and reversals
Levenshtein, V. I. (1965) · Soviet physics. Doklady · Cited by 10,387 · see citing works
(2004) · Cited by 9,718 · see citing works
An algorithm for suffix stripping
Porter, M. F. (1980) · Program · doi:10.1108/eb046814 · Cited by 8,096 · see citing works
Automatic evaluation of machine translation quality using n-gram co-occurrence statistics
Doddington, G. R. (2002) · International Conference on Human Language Technology Research · doi:10.3115/1289189.1289273 · Cited by 1,603 · see citing works
Automatic evaluation of summaries using N-gram co-occurrence statistics
Lin, C.-Y., & Hovy, E. H. (2003) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.3115/1073445.1073465 · Cited by 1,591 · see citing works
Self-citationDiscovering word senses from text
Pantel, P., & Lin, D. (2002) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/775047.775138 · Cited by 604 · see citing works
Barzilay, R., & Lee, L. (2003) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.3115/1073445.1073448 · Cited by 432 · see citing works
(2004) · doi:10.3115/1220355.1220427 · Cited by 294 · see citing works
An Evaluation Tool for Machine Translation: Fast Evaluation for MT Research
Nießen, S., Och, F. J., Leusch, G., et al. (2000) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/56t9t4rchhar · Cited by 284 · see citing works
Self-citationEvaluation of Machine Translation and its Evaluation
Turian, J. P., Shea, L., & Melamed, I. D. (2006) · doi:10.21236/ada453509 · Cited by 250 · see citing works
Precision and recall of machine translation
Melamed, I. D., Green, R., & Turian, J. P. (2003) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.3115/1073483.1073504 · Cited by 228 · see citing works
A New Quantitative Quality Measure for Machine Translation Systems
Su, K.-Y., Wu, M.-W., & Chang, J.-S. (1992) · International Conference on Computational Linguistics · doi:10.3115/992133.992137 · Cited by 112 · see citing works
A novel string-to-string distance measure with applications to machine translation evaluation
Leusch, G., Ueffing, N., & Ney, H. (2003) · Cited by 86 · see citing works
(2024) · ACM eBooks · doi:10.1145/3674127 · Cited by 82 · see citing works
Using multiple edit distances to automatically rank machine translation output
Akiba, Y., Imamura, K., & Sumita, E. (2001) · Cited by 67 · see citing works
Automatic Evaluation and Uniform Filter Cascades for Inducing N-Best Translation LexiconsPreprint
Melamed, I. D. (1995) · arXiv (Cornell University) · doi:10.48550/arxiv.cmp-lg/9505044 · Cited by 61 · see citing works
Meta-evaluation of summaries in a cross-lingual environment using content-based metrics
Saggion, H., Teufel, S., Radev, D., et al. (2002) · International Conference on Computational Linguistics · doi:10.3115/1072228.1072301 · Cited by 55 · see citing works
Flagged first, then by citations
Research works printed but not linked: 3
- Barzilay, R. and L. Lee. 2003. Learning to Para- phrase: An Unsupervised Approach Using Mul- tiple-Sequence Alignmen. In Proceeding of NAACL-HLT 2003, Edmonton, Canada.
- Miller, G. 1990. WordNet: An Online Lexical Da- tabase. International Journal of Lexicography, 3(4).
- Thompson, H. S. 1991. Automatic Evaluation of Translation Quality: Outline of Methodology and Report on Pilot Experiment. In Proceedings of the Evaluator's Forum, ISSCO, Geneva, Switzerland.
Cited by
Sorted by citations
Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning
Sharma, P., Ding, N., Goodman, S., & Soricut, R. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1238 · Cited by 1,879 · see citing works
A Comprehensive Survey of Deep Learning for Image Captioning
Hossain, M. Z., Sohel, F., Shiratuddin, M. F., & Laga, H. (2019) · ACM Computing Surveys · doi:10.1145/3295748 · Cited by 879 · see citing works
“There are some cases where an increase in BLEU score does not mean that the quality of the generated text is good [82].”
BackgroundGatt, A., & Krahmer, E. (2018) · Journal of Artificial Intelligence Research · doi:10.1613/jair.5477 · Cited by 776 · see citing works
A Corpus and Cloze Evaluation for Deeper Understanding of Commonsense Stories
Mostafazadeh, N., Chambers, N., He, X., Parikh, D., Batra, D., Vanderwende, L., Kohli, P., & Allen, J. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/n16-1098 · Cited by 598 · see citing works
IntelliCode compose: code generation using transformer
Svyatkovskiy, A., Deng, S. K., Fu, S., & Sundaresan, N. (2020) · Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3368089.3417058 · Cited by 474 · see citing works
Generating Natural Questions About an Image
Mostafazadeh, N., Misra, I., Devlin, J., Mitchell, M., He, X., & Vanderwende, L. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1170 · Cited by 270 · see citing works
A Systematic Comparison of Smoothing Techniques for Sentence-Level BLEU
Chen, B., & Cherry, C. (2014) · Proceedings of the Ninth Workshop on Statistical Machine Translation · doi:10.3115/v1/w14-3346 · Cited by 191 · see citing works
Improving chest X-ray report generation by leveraging warm starting
Nicolson, A., Dowling, J., & Koopman, B. (2023) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2023.102633 · Cited by 161 · see citing works
RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs
Chaudhari, S., Aggarwal, P., Murahari, V., Rajpurohit, T., Kalyan, A., Narasimhan, K., Deshpande, A., & Castro da Silva, B. (2026) · ACM Computing Surveys · doi:10.1145/3743127 · Cited by 92 · see citing works
“Automated metrics like BLEU and ROUGE [99] often do not correlate with human judgment [103, 164, 166, 177], making them unreliable for evaluation and training.”
e-ViL: A Dataset and Benchmark for Natural Language Explanations in Vision-Language Tasks
Kayser, M., Camburu, O.-M., Salewski, L., Emde, C., Do, V., Akata, Z., & Lukasiewicz, T. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00128 · Cited by 68 · see citing works
Dong, J., Lou, Y., Zhu, Q., Sun, Z., Li, Z., Zhang, W., & Hao, D. (2022) · Proceedings of the 44th International Conference on Software Engineering · doi:10.1145/3510003.3510069 · Cited by 61 · see citing works
Kuhn, L., Gal, Y., & Farquhar, S. (2023) · arXiv · doi:10.48550/arxiv.2302.09664 · Cited by 52 · see citing works
“Normalised entropy reimplements single-model variant of Malinin & Gales (2020), lexical similarity measures the average Rouge-L in a sampled set of answers for a given question analogously to Fomicheva et al. (2020), entropy and p ( True ) reimplement Kadavath et al. (2022).”
BackgroundInfluentialMetaWriter: Exploring the Potential and Perils of AI Writing Support in Scientific Peer Review
Sun, L., Tao, S., Hu, J., & Dow, S. P. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3637371 · Cited by 37 · see citing works
Adaptation with Self-Evaluation to Improve Selective Prediction in LLMs
Chen, J., Yoon, J., Ebrahimi, S., Arik, S., Pfister, T., & Jha, S. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.345 · Cited by 13 · see citing works
Transfer learning for multimodal dialog
Palaskar, S., Sanabria, R., & Metze, F. (2020) · Computer Speech & Language · doi:10.1016/j.csl.2020.101093 · Cited by 6 · see citing works
Recursively Summarizing Books with Human Feedback
Wu, J., Ouyang, L., Ziegler, D. M., Stiennon, N., Lowe, R., Leike, J., & Christiano, P. (2021) · arXiv · doi:10.48550/arxiv.2109.10862 · Cited by 76 · see citing works
- Authors
- Lin, C.-Y., & Och, F. J.
- Affiliations
- University of Southern California (US)
- ORCID
- Lin, Chin-Yew
- Venue
- Proceedings of the 42nd Annual Meeting on Association for Computational Linguistics - ACL '04
- Published
- 2004
- Pages
- 605-es
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W2108325777
- Cited by
- 737
- References
- 18
- Citations by year
- 2026: 27 · 2025: 70 · 2024: 70 · 2023: 69 · 2022: 57 · 2021: 63 · 2020: 62 · 2019: 52 · 2018: 35 · 2017: 31 · 2016: 21 · 2015: 16 · 2014: 19 · 2013: 20 · 2012: 20
- FWCI
- 9.71
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Social Sciences › Arts and Humanities › Language and Linguistics › Translation Studies and Practices
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Keywords
- machine translation quality assessment, automated assessment, longest common subsequence, reference translations
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
