Automatic evaluation of summaries using N-gram co-occurrence statistics
The DOI matches its Crossref record; 7 of the 7 references it cites were found in a registry or an index.
Abstract
Following the recent adoption by the machine translation community of automatic evaluation using the BLEU/NIST scoring process, we conduct an in-depth study of a similar idea for evaluating summaries. The results show that automatic evaluation using unigram co-occurrences between summary pairs correlates surprising well with human evaluations, based on various statistical metrics; while direct application of the BLEU evaluation procedure does not always give good results.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344
An algorithm for suffix stripping
Porter, M. F. (1980) · Program · doi:10.1108/eb046814 · Cited by 8,096
Automatic evaluation of machine translation quality using n-gram co-occurrence statistics
Doddington, G. (2002) · Proceedings of the second international conference on Human Language Technology Research - · doi:10.3115/1289189.1289273 · Cited by 1,584
(1996) · Medical Entomology and Zoology · Cited by 276
Tracking and summarizing news on a daily basis with Columbia's Newsblaster
McKeown, K. R., Barzilay, R., Evans, D., Hatzivassiloglou, V., Klavans, J. L., Nenkova, A., Sable, C., Schiffman, B., & Sigelman, S. (2002) · International Conference on Human Language Technology Research · doi:10.3115/1289189.1289212 · Cited by 269
Manual and automatic evaluation of summaries
Lin, C.-Y., & Hovy, E. (2002) · Proceedings of the ACL-02 Workshop on Automatic Summarization - · doi:10.3115/1118162.1118168 · Cited by 265
Self-citationA comparison of rankings produced by summarization evaluation measures
Donaway, R. L., Drummey, K. W., & Mather, L. A. (2000) · NAACL-ANLP Workshop on Automatic Summarization · doi:10.3115/1117575.1117583 · Cited by 119
Cited by
(2004) · Cited by 9,718
Multimodal Machine Learning: A Survey and Taxonomy
Baltrusaitis, T., Ahuja, C., & Morency, L.-P. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2798607 · Cited by 5,018
Framing Image Description as a Ranking Task: Data, Models and Evaluation Metrics
Hodosh, M., Young, P., & Hockenmaier, J. (2013) · Journal of Artificial Intelligence Research · doi:10.1613/jair.3994 · Cited by 1,314
Narayan, S., Cohen, S. B., & Lapata, M. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1206 · Cited by 1,207
“We evaluated summarization quality using F1 ROUGE (Lin and Hovy, 2003).”
BackgroundMethodOn Faithfulness and Factuality in Abstractive Summarization
Maynez, J., Narayan, S., Bohnet, B., & McDonald, R. (2020) · arXiv · doi:10.48550/arxiv.2005.00661 · Cited by 1,185
“Interestingly, all summaries are topical and fluent, and perform well in terms of ROUGE scores (Lin and Hovy, 2003).”
BackgroundInfluentialOn Faithfulness and Factuality in Abstractive Summarization
Maynez, J., Narayan, S., Bohnet, B., & McDonald, R. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.173 · Cited by 1,185
RACE: Large-scale ReAding Comprehension Dataset From Examinations
Lai, G., Xie, Q., Liu, H., Yang, Y., & Hovy, E. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1082 · Cited by 1,044
“, 2002) or ROUGE (Lin and Hovy, 2003) as metrics, which measure the overlap between the prediction and ground truth answers since the”
BackgroundSelf-citationBabyTalk: Understanding and Generating Simple Image Descriptions
Kulkarni, G., Premraj, V., Ordonez, V., Dhar, S., Li, S., Choi, Y., Berg, A. C., & Berg, T. L. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2012.162 · Cited by 839
Neural Summarization by Extracting Sentences and Words
Cheng, J., & Lapata, M. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1046 · Cited by 790
Gatt, A., & Krahmer, E. (2018) · Journal of Artificial Intelligence Research · doi:10.1613/jair.5477 · Cited by 776
Lin, C.-Y., & Och, F. J. (2004) · Proceedings of the 42nd Annual Meeting on Association for Computational Linguistics - ACL '04 · doi:10.3115/1218955.1219032 · Cited by 737
Self-citationModeling Local Coherence: An Entity-Based Approach
Barzilay, R., & Lapata, M. (2008) · Computational Linguistics · doi:10.1162/coli.2008.34.1.1 · Cited by 671
A comprehensive survey on pretrained foundation models: a history from BERT to ChatGPT
Zhou, C., Li, Q., Li, C., Yu, J., Liu, Y., Wang, G., Zhang, K., Ji, C., Yan, Q., He, L., et al. (2025) · International Journal of Machine Learning and Cybernetics · doi:10.1007/s13042-024-02443-6 · Cited by 557
Evaluating Content Selection in Summarization: The Pyramid Method
Nenkova, A., & Passonneau, R. (2004) · Columbia University · doi:10.7916/d80r9xvd · Cited by 527
Baby talk: Understanding and generating simple image descriptions
Kulkarni, G., Premraj, V., Dhar, S., Li, S., Choi, Y., Berg, A. C., & Berg, T. L. (2011) · CVPR 2011 · doi:10.1109/cvpr.2011.5995466 · Cited by 521
Semantic text similarity using corpus-based word similarity and string similarity
Islam, A., & Inkpen, D. (2008) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/1376815.1376819 · Cited by 492
A Hierarchical Neural Autoencoder for Paragraphs and Documents
Li, J., Luong, T., & Jurafsky, D. (2015) · Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.3115/v1/p15-1107 · Cited by 490
Graph-based ranking algorithms for sentence extraction, applied to text summarization
Mihalcea, R. (2004) · Proceedings of the ACL 2004 on Interactive poster and demonstration sessions - · doi:10.3115/1219044.1219064 · Cited by 469
Search-based structured prediction
Daumé, H., Langford, J., & Marcu, D. (2009) · Machine Learning · doi:10.1007/s10994-009-5106-x · Cited by 465
Improved Image Captioning via Policy Gradient optimization of SPIDEr
Liu, S., Zhu, Z., Ye, N., Guadarrama, S., & Murphy, K. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.100 · Cited by 430
“Common choices include BLEU, CIDEr, METEOR and ROUGE.”
MethodSentence Fusion for Multidocument News Summarization
Barzilay, R., & McKeown, K. R. (2005) · Computational Linguistics · doi:10.1162/089120105774321091 · Cited by 429
An Actor-Critic Algorithm for Sequence Prediction
Bahdanau, D., Brakel, P., Xu, K., Goyal, A., Lowe, R., Pineau, J., Courville, A., & Bengio, Y. (2016) · arXiv · doi:10.48550/arxiv.1607.07086 · Cited by 423
Extractive Summarization as Text Matching
Zhong, M., Liu, P., Chen, Y., Wang, D., Qiu, X., & Huang, X. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.552 · Cited by 413
Leveraging Pre-trained Checkpoints for Sequence Generation Tasks
Rothe, S., Narayan, S., & Severyn, A. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00313 · Cited by 391
Measuring the semantic similarity of texts
Corley, C., & Mihalcea, R. (2005) · Proceedings of the ACL Workshop on Empirical Modeling of Semantic Equivalence and Entailment - EMSEE '05 · doi:10.3115/1631862.1631865 · Cited by 352
Automatic Description Generation from Images: A Survey of Models, Datasets, and Evaluation Measures
Bernardi, R., Cakici, R., Elliott, D., Erdem, A., Erdem, E., Ikizler-Cinbis, N., Keller, F., Muscat, A., & Plank, B. (2016) · Journal of Artificial Intelligence Research · doi:10.1613/jair.4900 · Cited by 349
Exploring and Distilling Posterior and Prior Knowledge for Radiology Report Generation
Liu, F., Wu, X., Ge, S., Fan, W., & Zou, Y. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01354 · Cited by 348
Abstractive text summarization using LSTM-CNN based deep learning
Song, S., Huang, H., & Ruan, T. (2019) · Multimedia Tools and Applications · doi:10.1007/s11042-018-5749-3 · Cited by 342
Computational Generation of Referring Expressions: A Survey
Krahmer, E., & van Deemter, K. (2012) · Computational Linguistics · doi:10.1162/coli_a_00088 · Cited by 325
LCSTS: A Large Scale Chinese Short Text Summarization Dataset
Hu, B., Chen, Q., & Zhu, F. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1229 · Cited by 316
Heterogeneous Graph Neural Networks for Extractive Document Summarization
Wang, D., Liu, P., Zheng, Y., Qiu, X., & Huang, X. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.553 · Cited by 285
Das, P., Xu, C., Doell, R. F., & Corso, J. J. (2013) · 2013 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2013.340 · Cited by 282
Aliguliyev, R. M. (2009) · Expert Systems with Applications · doi:10.1016/j.eswa.2008.11.022 · Cited by 265
BIGPATENT: A Large-Scale Dataset for Abstractive and Coherent Summarization
Sharma, E., Li, C., & Wang, L. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1212 · Cited by 173
Towards Effective Event Detection, Tracking and Summarization on Microblog Data
Long, R., Wang, H., Chen, Y., Jin, O., & Yu, Y. (2011) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-23535-1_55 · Cited by 109
The limits of automatic summarisation according to ROUGE
Schluter, N. (2017) · Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers · doi:10.18653/v1/e17-2007 · Cited by 108
“We know that 100% perfect scores are impossible, so what is a perfect score according to ROUGE? How do we know when no improvement is possible? Previous research on evaluation metrics for automatic summarisation has tried to empirically show a correlation between human judgments and system output quality (Lin, 2004; L…”
BackgroundWill Pyramids Built of Nuggets Topple Over?
Lin, J., & Demner-Fushman, D. (2006) · Proceedings of the main conference on Human Language Technology Conference of the North American Chapter of the Association of Computational Linguistics - · doi:10.3115/1220835.1220884 · Cited by 77
CrystalBLEU: Precisely and Efficiently Measuring the Similarity of Code
Eghbali, A., & Pradel, M. (2022) · Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering · doi:10.1145/3551349.3556903 · Cited by 42
Benchmarking Large Language Models on CMExam - A Comprehensive Chinese Medical Exam Dataset
Liu, J., Zhou, P., Hua, Y., Chong, D., Tian, Z., Liu, A., Wang, H., You, C., Guo, Z., Zhu, L., et al. (2023) · arXiv · doi:10.48550/arxiv.2306.03030 · Cited by 36
(2024) · doi:10.1145/3613904.3642939 · Cited by 31
Moirangthem, D. S., & Lee, M. (2020) · Neural Networks · doi:10.1016/j.neunet.2019.12.022 · Cited by 30
Yi, S., Goel, R., Khatri, C., Cervone, A., Chung, T., Hedayatnia, B., Venkatesh, A., Gabriel, R., & Hakkani-Tur, D. (2019) · Proceedings of the 12th International Conference on Natural Language Generation · doi:10.18653/v1/w19-8608 · Cited by 26
Conditional Generation with a Question-Answering Blueprint
Narayan, S., Maynez, J., Amplayo, R. K., Ganchev, K., Louis, A., Huot, F., Sandholm, A., Das, D., & Lapata, M. (2022) · arXiv · doi:10.48550/arxiv.2207.00397 · Cited by 24
Talk2Me: Automated linguistic data collection for personal assessment
Komeili, M., Pou-Prom, C., Liaqat, D., Fraser, K. C., Yancheva, M., & Rudzicz, F. (2019) · PLOS ONE · doi:10.1371/journal.pone.0212342 · Cited by 23
MOPRD: A multidisciplinary open peer review dataset
Lin, J., Song, J., Zhou, Z., Chen, Y., & Shi, X. (2022) · arXiv · doi:10.48550/arxiv.2212.04972 · Cited by 11
“Then, we use ROUGE [21, 22] and BARTScore [47] as the automatic metrics to evaluate the performance of different generation methods.”
MethodScaling Laws for Forgetting When Fine-Tuning Large Language Models
Kalajdzievski, D. (2024) · arXiv · doi:10.48550/arxiv.2401.05605 · Cited by 2
“The points 1 and 2 are also directly applicable as arguments against any other metric which compares the evaluation of M, M ′ in terms of a ground truth target in D , including for example the ROUGE or BLEU score on D (Lin & Hovy, 2003; Papineni et al., 2002).”
Metadata
- Authors
- Lin, C.-Y., & Hovy, E.
- Venue
- Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology - NAACL '03
- Published
- 2003
- Volume
- 1
- Pages
- 71-78
- OpenAlex
- W2150824314
- Cited by
- 1,578
- References
- 7
- FWCI
- 36.82
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.