The DOI matches its Crossref record; 31 of the 32 references it cites were found in a registry or an index.
A Dataset and Evaluation Metrics for Abstractive Compression of Sentences and Short Paragraphs
Open accessAbstract
We introduce a manually-created, multireference dataset for abstractive sentence and short paragraph compression.First, we examine the impact of single-and multi-sentence level editing operations on human compression quality as found in this corpus.We observe that substitution and rephrasing operations are more meaning preserving than other operations, and that compressing in context improves quality.Second, we systematically explore the correlations between automatic evaluation metrics and human judgments of meaning preservation and grammaticality in the compression task, and analyze the impact of the linguistic units used and precision versus recall measures on the quality of the metrics.Multi-reference evaluation metrics are shown to offer significant advantage over single reference-based metrics.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Most text compression datasets are abstractive, treating prompt compression as a generative task where original prompts are rephrased into condensed ones.
“most text compression datasets are abstractive (Toutanova et al., 2016; Koupaee and Wang, 2018; Kim et al., 2019), meaning that they treat prompt compression as a generative task where the original prompts are rephrased into condensed ones.”
From LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression (2024) · cites this paper as Toutanova et al., 2016
The Introduction describes this paper’s dataset as abstractive, with source texts rewritten into shorter versions. But Related Work (§2) describes several deletion-based corpora and does not establish that most text-compression datasets are abstractive. The paper studies sentences and short paragraphs, not prompt compression. (LLM, conf=0.88)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Draper, N. R., & Smith, H. (2005) · Technometrics · doi:10.1198/tech.2005.s303 · Cited by 17,993
(2004) · Cited by 9,718
Weighted kappa: nominal scale agreement provision for scaled disagreement or partial credit
Cohen, J. (1968) · Psychological Bulletin · doi:10.1037/h0026256 · Cited by 8,736
Accurate unlexicalized parsing
Klein, D., & Manning, C. D. (2003) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.3115/1075096.1075150 · Cited by 3,055
A Neural Attention Model for Abstractive Sentence Summarization
Rush, A. M., Chopra, S., & Weston, J. (2015) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1044 · Cited by 2,879
Optimizing Statistical Machine Translation for Text Simplification
Xu, W. H., Napoles, C., Pavlick, E., Chen, Q. Z., & Callison-Burch, C. (2016) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00107 · Cited by 524
Summarization beyond sentence extraction: A probabilistic approach to sentence compression
Knight, K. K., & Marcu, D. (2002) · Artificial Intelligence · doi:10.1016/s0004-3702(02)00222-9 · Cited by 491
A New Dataset and Method for Automatically Grading ESOL Texts
Yannakoudakis, H., Briscoe, T., & Medlock, B. (2011) · Cited by 482
Beyond SumBasic: Task-focused summarization with sentence simplification and lexical expansion
Vanderwende, L., Suzuki, H., Brockett, C., & Nenkova, A. (2007) · Information Processing & Management · doi:10.1016/j.ipm.2007.01.023 · Cited by 310
Self-citationGlobal Inference for Sentence Compression: An Integer Linear Programming Approach
Clarke, J., & Lapata, M. (2008) · Journal of Artificial Intelligence Research · doi:10.1613/jair.2433 · Cited by 306
Sentence reduction for automatic text summarization
Jing, H. (2000) · Conference on Applied Natural Language Processing · doi:10.3115/974147.974190 · Cited by 290
Robust Accurate Statistical Annotation of General Text
Briscoe, T., & Carroll, J. A. (2002) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/2aswu2tg29np · Cited by 249
Sentence Compression by Deletion with LSTMs
Filippova, K., Alfonseca, E., Colmenares, C. A., Kaiser, Ł., & Vinyals, O. (2015) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d15-1042 · Cited by 246
Jointly Learning to Extract and Compress
Berg-Kirkpatrick, T., Gillick, D., & Klein, D. (2011) · Cited by 209
Sentence compression beyond word deletion
Cohn, T., & Lapata, M. (2008) · International Conference on Computational Linguistics · doi:10.3115/1599081.1599099 · Cited by 171
The decomposition of human-written summary sentences
Jing, H., & McKeown, K. R. (1999) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/312624.312666 · Cited by 164
Re-evaluating Automatic Summarization with BLEU and 192 Shades of ROUGE
Graham, Y. (2015) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d15-1013 · Cited by 143
An Empirical Analysis of Formality in Online Communication
Pavlick, E., & Tetreault, J. (2016) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00083 · Cited by 140
Overcoming the Lack of Parallel Data in Sentence Compression
Filippova, K., & Altun, Y. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1155 · Cited by 132
Evaluation metrics for generation
Bangalore, S., Rambow, O., & Whittaker, S. (2000) · doi:10.3115/1118253.1118255 · Cited by 127
Models for sentence compression
Clarke, J., & Lapata, M. (2006) · International Conference on Computational Linguistics · doi:10.3115/1220175.1220223 · Cited by 103
deltaBLEU: A Discriminative Metric for Generation Tasks with Intrinsically Diverse Targets
Galley, M., Brockett, C., Sordoni, A., Ji, Y., Auli, M., Quirk, C., Mitchell, M., Gao, J., & Dolan, B. (2015) · doi:10.3115/v1/p15-2073 · Cited by 101
Self-citationThe Manually Annotated Sub-Corpus: A Community Resource for and by the People
Ide, N., Baker, C. F., Fellbaum, C., & Passonneau, R. J. (2010) · Cited by 84
Riezler, S., King, T. H., Crouch, R., & Zaenen, A. (2003) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.3115/1073445.1073471 · Cited by 80
MASC: the Manually Annotated Sub-Corpus of American English
Ide, N., Baker, C. F., Fellbaum, C., Fillmore, C. J., & Passonneau, R. J. (2008) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/2s5uigy45uwu · Cited by 73
Learning Sentential Paraphrases from Bilingual Parallel Corpora for Text-to-Text Generation
Ganitkevitch, J., Callison-Burch, C., Napoles, C., & Durme, B. V. (2011) · Cited by 59
Speech Summarization: An Approach through Word Extraction and a Method for Evaluation ∗
Hori, C., & Furui, S. (2004) · Institutional Repositories DataBase (IRDB) · Cited by 55
Evaluating sentence compression: Pitfalls and suggested remedies
Napoles, C., Durme, B. V., & Callison-Burch, C. (2011) · Cited by 50
A Lightweight and High Performance Monolingual Word Aligner
Yao, X., Durme, B. V., Callison-Burch, C., & Clark, P. E. (2013) · Cited by 35
Trimming CFG parse trees for sentence compression using machine learning approaches
Unno, Y., Ninomiya, T., Miyao, Y., & Tsujii, J. (2006) · International Conference on Computational Linguistics and Annual Meeting of the Association for Computational Linguistics (COLING/ACL) · doi:10.3115/1273073.1273182 · Cited by 22
Cost Optimization in Crowdsourcing Translation: Low cost translations made even cheaper
Gao, M., Xu, W. H., & Callison-Burch, C. (2015) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.3115/v1/n15-1072 · Cited by 13
Flagged first, then by citations
1 of the 32 references couldn’t be matched to a known paper.
Research works printed but not linked: 4
- James Clarke and Mirella Lapata. 2008. Global infer- ence for sentence compression: An integer linear pro- gramming approach. Journal of Artificial Intelligence Research, pages 399-429.
- Kevin Knight and Daniel Marcu. 2002. Summariza- tion beyond sentence extraction: A probabilistic ap- proach to sentence compression. Artificial Intelli- gence, 139(1):91-107.
- Courtney Napoles, Matthew Gormley, and Benjamin Van Durme. 2012. Annotated gigaword. In Proceed- ings of the Joint Workshop on Automatic Knowledge Base Construction and Web-scale Knowledge Extrac- tion.
- Evan James Williams. 1959. Regression analysis. Wiley, New York.
Cited by
Sorted by citations
BERTScore: Evaluating Text Generation with BERT
Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., & Artzi, Y. (2019) · arXiv · doi:10.48550/arxiv.1904.09675 · Cited by 2,865
Selective Encoding for Abstractive Sentence Summarization
Zhou, Q. S., Yang, N., Wei, F., & Zhou, M. (2017) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p17-1101 · Cited by 276
A Continuously Growing Dataset of Sentential Paraphrases
Lan, W., Qiu, S., He, H., & Xu, W. H. (2017) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d17-1126 · Cited by 175
Encode, Tag, Realize: High-Precision Text Editing
Malmi, E., Krause, S., Rothe, S., Mirylenka, D., & Severyn, A. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1510 · Cited by 164
The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics
Gehrmann, S., Adewumi, T., Aggarwal, K., Ammanamanchi, P. S., Aremu, A., Bosselut, A., Chandu, K. R., Clinciu, M.-A., Das, D., Dhole, K., et al. (2021) · Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics (GEM 2021) · doi:10.18653/v1/2021.gem-1.10 · Cited by 159
Recent advances in document summarization
Yao, J.-G., Wan, X., & Xiao, J. (2017) · Knowledge and Information Systems · doi:10.1007/s10115-017-1042-4 · Cited by 149
Narayan, S., Gardent, C., Cohen, S. B., & Shimorina, A. (2017) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d17-1064 · Cited by 89
Semantic Structural Evaluation for Text Simplification
Sulem, E., Abend, O., & Rappoport, A. (2018) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n18-1063 · Cited by 85
Learning To Split and Rephrase From Wikipedia Edit History
Botha, J. A., Faruqui, M., Alex, J., Baldridge, J., & Das, D. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1080 · Cited by 80
FELIX: Flexible Text Editing Through Tagging and Insertion
Mallinson, J., Severyn, A., Malmi, E., & Garrido, G. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.111 · Cited by 76
A Task in a Suit and a Tie: Paraphrase Generation with Semantic Augmentation
Wang, S., Gupta, R., Chang, N., & Baldridge, J. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33017176 · Cited by 64
SemSUM: Semantic Dependency Guided Neural Abstractive Summarization
Jin, H., Wang, T., & Wan, X. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6312 · Cited by 60
Sentence-Level Fluency Evaluation: References Help, But Can Be Spared!
Kann, K., Rothe, S., & Filippova, K. (2018) · Conference on Computational Natural Language Learning · doi:10.18653/v1/k18-1031 · Cited by 56
GRUEN for Evaluating Linguistic Quality of Generated Text
Zhu, W., & Bhat, S. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.9 · Cited by 56
LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression
Pan, Z., Wu, Q., Jiang, H., Xia, M., Luo, X., Zhang, J., Lin, Q., Rühle, V., Yang, Y., Lin, C.-Y., et al. (2024) · Findings of the Association for Computational Linguistics: ACL 2024 · doi:10.18653/v1/2024.findings-acl.57 · Cited by 58
Automatic Text Evaluation through the Lens of Wasserstein Barycenters
Colombo, P., Staerman, G., Clavel, C., & Piantanida, P. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.817 · Cited by 35
Encode, Tag, Realize: High-Precision Text EditingPreprint
Malmi, E., Krause, S., Rothe, S., Mirylenka, D., & Severyn, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.01187 · Cited by 33
An Imitation Learning Curriculum for Text Editing with Non-Autoregressive Models
Agrawal, S., & Carpuat, M. J. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.520 · Cited by 22
Automatic Text Evaluation through the Lens of Wasserstein BarycentersPreprint
Colombo, P., Staerman, G., Clavel, C., & Piantanida, P. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.12463 · Cited by 20
From Shakespeare to Twitter: What are Language Styles all about?
Xu, W. H. (2017) · doi:10.18653/v1/w17-4901 · Cited by 18
Neural sentence fusion for diversity driven abstractive multi-document summarization
Fuad, T. A., Nayeem, M. T., Mahmud, A., & Chali, Y. (2019) · Computer Speech & Language · doi:10.1016/j.csl.2019.04.006 · Cited by 18
Sentence Compression for Arbitrary Languages via Multilingual Pivoting
Mallinson, J., Sennrich, R., & Lapata, M. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1267 · Cited by 17
An eye-tracking attention based model for abstractive text headline
Xie, J., Wang, X., Pang, G., Pang, G., & Qin, X. (2019) · Cognitive Systems Research · doi:10.1016/j.cogsys.2019.07.003 · Cited by 9
Semantic Structural Evaluation for Text SimplificationPreprint
Sulem, E., Abend, O., & Rappoport, A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.05022 · Cited by 9
An Operation Network for Abstractive Sentence Compression.
Yu, N., Zhang, J., Huang, M., & Zhu, X. (2018) · International Conference on Computational Linguistics · Cited by 8
Neural Diverse Abstractive Sentence Compression Generation
Nayeem, M. T., Fuad, T. A., & Chali, Y. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-15719-7_14 · Cited by 8
Claim Optimization in Computational Argumentation
Skitalinskaya, G., Spliethöver, M., & Wachsmuth, H. (2023) · International Conference on Natural Language Generation · doi:10.18653/v1/2023.inlg-main.10 · Cited by 8
Deep Learning Approaches to Text Production
Gardent, C., & Narayan, S. (2018) · doi:10.18653/v1/n18-6002 · Cited by 6
Unsupervised Token-wise Alignment to Improve Interpretation of Encoder-Decoder Models
Kiyono, S., Takase, S., Suzuki, J., Okazaki, N., Inui, K., & Nagata, M. (2018) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/w18-5410 · Cited by 5
Data-efficient Neural Text Compression with Interactive Learning
Avinesh, P. V. S., & Meyer, C. M. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1262 · Cited by 5
Understanding the Behaviour of Neural Abstractive Summarizers using Contrastive Examples
Kumar, K., & Cheung, J. C. K. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1396 · Cited by 5
Shichel, Y., Kalech, M., & Tsur, O. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.139 · Cited by 4
Revision for Concision: A Constrained Paraphrase Generation Task
Mu, W., & Lim, K. H. (2022) · Workshop on Text Simplification, Accessibility and Readability (TSAR) · doi:10.18653/v1/2022.tsar-1.6 · Cited by 4
Fiction Sentence Expansion and Enhancement via Focused Objective and Novelty Curve Sampling
Safovich, Y., & Azaria, A. (2020) · IEEE International Conference on Tools with Artificial Intelligence (ICTAI) · doi:10.1109/ictai50040.2020.00132 · Cited by 3
LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression
Pan, Z., Wu, Q., Jiang, H., Xia, M., Luo, X., Zhang, J., Lin, Q., Rühle, V., Yang, Y., Lin, C.-Y., et al. (2024) · arXiv · Cited by 2
“For Q1, most text compression datasets are ab-stractive (Toutanova et al., 2016; Koupaee and Wang, 2018; Kim et al., 2019), meaning that they treat prompt compression as a generative task where the original prompts are rephrased into condensed ones.”
BackgroundStrong natural language query generation
Liu, B., Lu, X., & Culpepper, J. S. (2021) · Information Retrieval · doi:10.1007/s10791-021-09395-3 · Cited by 2
Improving a Language Model Evaluator for Sentence Compression Without Reinforcement Learning
Kuvshinova, T., & Khritankov, A. (2019) · International Symposium on Information and Communication Technology · doi:10.1145/3368926.3369713 · Cited by 2
Abstractive Sentence Compression with Event Attention
Choi, S. J., Jung, I., Park, S., & Park, S.-B. (2019) · Applied Sciences · doi:10.3390/app9193949 · Cited by 2
An Efficient Compression Scheme for Natural Language Text by Hashing
Mahmood, M. A., & Hasan, K. M. A. (2022) · SN Computer Science · doi:10.1007/s42979-022-01210-0 · Cited by 1
From Lengthy to Lucid: A Systematic Literature Review on NLP Techniques for Taming Long Sentences
Passali, T., Chatzikyriakidis, E., Andreadis, S. T., Stavropoulos, T. G., Matonaki, A., Fachantidis, A., & Tsoumakas, G. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2312.05172 · Cited by 1
Towards Context-free Information Importance Estimation
Zopf, M. (2019) · TUbilio (Technical University of Darmstadt) · doi:10.26083/tuprints-00008976 · Cited by 1
Similarity between Units of Natural Language: The Transition from Coarse to Fine EstimationPreprint
Mu, W. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2210.14275 · Cited by 1
Anemone: a Visual Semantic Graph
Vila, J. (2019)
Learning to Summarize Passages: Mining Passage-Summary Pairs from Wikipedia Revision HistoriesPreprint
Zhou, Q. S., Wei, F., & Zhou, M. Q. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.02592
Nong, Q., Zhang, X., Jin, Z. G., Gao, C., Shang, W., Du, K., Ding, Z., & Ma, Y. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4600240
Assessing and Scoring Difficulty of Hard-to-Solve Data in Summarization Tasks
Jung, J., Seo, H., Namgoong, H., & Jung, S. (2024) · IEEE Access · doi:10.1109/access.2024.3519548
Meta Feature Classification on a Syntetic Dataset Generator
Lemos, J. C. F. P. D. (2022) · Portuguese National Funding Agency for Science, Research and Technology (RCAAP Project by FCT) · doi:10.34626/98ns-y517
Metadata
- Authors
- Toutanova, K., Brockett, C., Tran, K. M., & Amershi, S.
- Affiliations
- Microsoft (United States) (US)
- Venue
- Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing
- Published
- 2016
- Pages
- 340-350
- OpenAlex
- W2511538013
- Cited by
- 42
- References
- 32
- Citations by year
- 2024: 1 · 2023: 2 · 2022: 3 · 2021: 5 · 2020: 5 · 2019: 13 · 2018: 8 · 2017: 5
- FWCI
- 4.54
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- UvA-DARE (University of Amsterdam) (submittedVersion, other-oa)
- Data Archiving and Networked Services (DANS) (submittedVersion, other-oa)
- Funders
- Microsoft Research
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Advanced Text Analysis Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- text compression, sentence reduction, automatic evaluation metrics, human judgment, meaning preservation, grammaticality, substitution, paraphrase, precision and recall
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
