Efficient Long-Text Understanding with Short-Text Models
Open accessThe DOI matches its Crossref record; 51 of the 51 references it cites were found in a registry or an index.
Abstract
Transformer-based pretrained language models (LMs) are ubiquitous across natural language understanding, but cannot be applied to long sequences such as stories, scientific articles, and long documents due to their quadratic complexity. While a myriad of efficient transformer variants have been proposed, they are typically based on custom implementations that require expensive pretraining from scratch. In this work, we propose SLED: SLiding-Encoder and Decoder, a simple approach for processing long sequences that re-uses and leverages battle-tested short-text pretrained LMs. Specifically, we partition the input into overlapping chunks, encode each with a short-text LM encoder and use the pretrained decoder to fuse information across chunks (fusion-in-decoder). We illustrate through controlled experiments that SLED offers a viable strategy for long text understanding and evaluate our approach on SCROLLS, a benchmark with seven datasets across a wide range of language understanding tasks. We find that SLED is competitive with specialized models that are up to 50x larger and require a dedicated and expensive pretraining step.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Our experimental setup is similar to the needle-in-a-haystack experiments of Ivgi et al. (2023), who compare question answering performance when the relevant paragraph is placed (i) at the beginning of the input or (ii) a random position within the input.They find that encoder-decoder models have significantly higher performance when relevant information is placed at the start of the input context.”
From Lost in the Middle: How Language Models Use Long Contexts (2024) · cites this paper as Ivgi et al. (2023)
In the citing paper
“They find that encoder-decoder models have significantly higher performance when relevant information is placed at the start of the input context.”
From Lost in the Middle: How Language Models Use Long Contexts (2024) · cites this paper as Ivgi et al. (2023)
In the citing paper
“This category is a real-world adaptation (Ivgi et al., 2023) of NIAH”
From RULER: What's the Real Context Size of Your Long-Context Language Models? (2024) · cites this paper as Ivgi et al., 2023
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 3×
“Recent advances in natural language processing have been by and large fueled by the transformer architecture (Vaswani et al., 2017).”
BackgroundMethodBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515 · cited 1×
“Our approach can be readily applied to any pretrained encoder-decoder LM such as T5 (Raffel et al., 2020b) and BART (Lewis et al., 2020) (but is not applicable to decoder-only (Brown et al., 2020) or encoder-only models (Liu et al., 2019; Conneau et al., 2020)).”
MethodAdvances in neural information processing systems 7
(1996) · Computers & Mathematics with Applications · doi:10.1016/s0898-1221(96)90463-0 · Cited by 14,281
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · cited 5×
“, 2020) or T5 (Raffel et al., 2020b), and embedding it into SLED’s framework results in dramatic improvement in performance (6 points on average across models).”
BackgroundMethodInfluential(2004) · Cited by 9,718
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821
Transformers: State-of-the-Art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2020.emnlp-demos.6 · Cited by 8,741
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208 · cited 2×
“To check if SLED can ignore irrelevant text and locate a single piece of information, we cast SQuAD 1.1 (Rajpurkar et al., 2016) as a sequence-to-sequence task with long input.”
MethodInfluentialUnsupervised Cross-lingual Representation Learning at Scale
Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F., Grave, E., Ott, M., Zettlemoyer, L., & Stoyanov, V. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.747 · Cited by 5,637
Longformer: The Long-Document Transformer
Beltagy, I., Peters, M. E., & Cohan, A. (2020) · arXiv · doi:10.48550/arxiv.2004.05150 · Cited by 2,281 · cited 12×
“In contrast, The attention mechanism in Longformer, and subsequently LED, was implemented by Beltagy et al. (2020) with a specialized CUDA kernel that is coupled to the architecture and implementation of BART.”
BackgroundMethodInfluentialHotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering
Yang, Z., Qi, P., Zhang, S., Bengio, Y., Cohen, W., Salakhutdinov, R., & Manning, C. D. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1259 · Cited by 1,804
Reformer: The Efficient Transformer
Kitaev, N., Kaiser, Ł., & Levskaya, A. (2020) · arXiv · doi:10.48550/arxiv.2001.04451 · Cited by 1,435 · cited 2×
“To bridge this gap, recent work has developed more efficient transformer variants (Kitaev et al., 2020a; Beltagy et al., 2020; Zaheer et al., 2020a; Guo et al., 2022) and applied them over 100M 1B 10B 200M 400M 2B 4B 20B # of parameters 20 25 30 35 40 45”
BackgroundMethodBig Bird: Transformers for Longer Sequences
Zaheer, M., Guruganesh, G., Dubey, A., Ainslie, J., Alberti, C., Ontanon, S., Pham, P., Ravula, A., Wang, Q., Yang, L., et al. (2020) · arXiv · doi:10.48550/arxiv.2007.14062 · Cited by 1,349
Efficient Transformers: A Survey
Tay, Y., Dehghani, M., Bahri, D., & Metzler, D. (2020) · arXiv · doi:10.48550/arxiv.2009.06732 · Cited by 1,267 · cited 4×
“For a survey on the families of efficient transformers, see Tay et al. (2020).”
BackgroundInfluentialEfficient transformers: A survey
Tay, Y., Dehghani, M., Bahri, D., & Metzler, D. (2023) · ACM Computing Surveys · doi:10.1145/3530811 · Cited by 1,267 · cited 4×
“For a survey on the families of efficient transformers, see Tay et al. (2020).”
BackgroundInfluentialLeveraging Passage Retrieval with Generative Models for Open Domain Question Answering
Izacard, G., & Grave, E. (2021) · Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume · doi:10.18653/v1/2021.eacl-main.74 · Cited by 1,109
Linformer: Self-Attention with Linear Complexity
Wang, S., Li, B. Z., Khabsa, M., Fang, H., & Ma, H. (2020) · arXiv · doi:10.48550/arxiv.2006.04768 · Cited by 867 · cited 1×
“, 2021), and using low-rank factorizations (Wang et al., 2020).”
Efficiently Modeling Long Sequences with Structured State Spaces
Gu, A., Goel, K., & Ré, C. (2021) · arXiv · doi:10.48550/arxiv.2111.00396 · Cited by 805
Rethinking Attention with Performers
Choromanski, K., Likhosherstov, V., Dohan, D., Song, X., Gane, A., Sarlos, T., Hawkins, P., Davis, J., Mohiuddin, A., Kaiser, L., et al. (2020) · arXiv · doi:10.48550/arxiv.2009.14794 · Cited by 695
The NarrativeQA Reading Comprehension Challenge
Kočiský, T., Schwarz, J., Blunsom, P., Dyer, C., Hermann, K. M., Melis, G., & Grefenstette, E. (2018) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00023 · Cited by 596 · cited 3×
“, 2021) is a QA benchmark that contains questions over NLP papers; NarrativeQA (Kočiský et al., 2018) contains questions over entire books and movie scripts; QuALITY (Pang et al.”
BackgroundInfluentialEfficient Content-Based Sparse Attention with Routing Transformers
Roy, A., Saffar, M., Vaswani, A., & Grangier, D. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00353 · Cited by 509 · cited 1×
“, 2020; Gupta and Berant, 2020) or to tokens that are semantically similar (Kitaev et al., 2020b; Roy et al., 2021).”
BackgroundLong Range Language Modeling via Gated State SpacesPreprint
Mehta, H., Gupta, A., Cutkosky, A., & Neyshabur, B. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2206.13947 · Cited by 379
ETC: Encoding Long and Structured Inputs in Transformers
Ainslie, J., Ontañón, S., Alberti, C., Cvicek, V., Fisher, Z., Pham, P., Ravula, A., Sanghai, S., Wang, Q., & Yang, L. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.19 · Cited by 268
Long Range Arena: A Benchmark for Efficient Transformers
Tay, Y., Dehghani, M., Abnar, S., Shen, Y., Bahri, D., Pham, P., Rao, J., Yang, L., Ruder, S., & Metzler, D. (2020) · arXiv · doi:10.48550/arxiv.2011.04006 · Cited by 241 · cited 3×
“(2021) proposed the Structured State Space (S4) architecture showing dramatic gains over transformers on the LRA benchmark (Tay et al., 2021).”
BackgroundMethodInfluentialLongT5: Efficient Text-To-Text Transformer for Long Sequences
Guo, M., Ainslie, J., Uthus, D., Ontanon, S., Ni, J., Sung, Y.-H., & Yang, Y. (2022) · Findings of the Association for Computational Linguistics: NAACL 2022 · doi:10.18653/v1/2022.findings-naacl.55 · Cited by 230
LongT5: Efficient Text-To-Text Transformer for Long Sequences
Guo, M., Ainslie, J., Uthus, D., Ontanon, S., Ni, J., Sung, Y.-H., & Yang, Y. (2021) · arXiv · doi:10.48550/arxiv.2112.07916 · Cited by 230
QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarization
Zhong, M., Yin, D., Yu, T., Zaidi, A., Mutuma, M., Jha, R., Awadallah, A. H., Celikyilmaz, A., Liu, Y., Qiu, X., et al. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.472 · Cited by 178
Efficient Attentions for Long Document Summarization
Huang, L., Cao, S., Parulian, N., Ji, H., & Wang, L. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.112 · Cited by 146
A Practical Survey on Faster and Lighter Transformers
Fournier, Q., Caron, G. M., & Aloise, D. (2023) · ACM Computing Surveys · doi:10.1145/3586074 · Cited by 145
A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers
Dasigi, P., Lo, K., Beltagy, I., Cohan, A., Smith, N. A., & Gardner, M. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.365 · Cited by 138
Multi-passage BERT: A Globally Normalized BERT Model for Open-domain Question Answering
Wang, Z., Ng, P., Ma, X., Nallapati, R., & Xiang, B. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1599 · Cited by 116
Peng, H., Pappas, N., Yogatama, D., Schwartz, R., Smith, N. A., & Kong, L. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.02143 · Cited by 110 · cited 1×
“, 2021), using random features to approximate the attention matrix (Choromanski et al., 2021; Peng et al., 2021), and using low-rank factorizations (Wang et al.”
MethodDiagonal State Spaces Are as Effective as Structured State Spaces
Gupta, A., Gu, A., & Berant, J. (2022) · neural information processing systems · doi:10.52202/068431-1670 · Cited by 95 · cited 5×
“In NarrativeQA and GovReport, which contain ∼71K and ∼19K samples respectively, LED is comparable to SLED and even slightly outperforms it on some metrics.”
MethodInfluentialSelf-citationJiang, Y., & Bansal, M. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1262 · Cited by 83 · cited 1×
“Finally, past work has shown that many examples in HotpotQA can be answered with access to the “second” gold para-graph only, which contains the answer (Jiang and Bansal, 2019).”
BackgroundContractNLI: A Dataset for Document-level Natural Language Inference for Contracts
Koreeda, Y., & Manning, C. (2021) · Findings of the Association for Computational Linguistics: EMNLP 2021 · doi:10.18653/v1/2021.findings-emnlp.164 · Cited by 76
Exploring Neural Models for Query-Focused Summarization
Vig, J., Fabbri, A. R., Kryściński, W., Wu, C.-S., & Liu, W. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.109 · Cited by 69
Diagonal State Spaces are as Effective as Structured State SpacesPreprint
Gupta, A., Gu, A., & Berant, J. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2203.14343 · Cited by 64 · cited 5×
“In NarrativeQA and GovReport, which contain ∼71K and ∼19K samples respectively, LED is comparable to SLED and even slightly outperforms it on some metrics.”
MethodInfluentialSelf-citationQuALITY: Question Answering with Long Input Texts, Yes!
Pang, R. Y., Parrish, A., Joshi, N., Nangia, N., Phang, J., Chen, A., Padmakumar, V., Ma, J., Thompson, J., He, H., et al. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.391 · Cited by 51
SummScreen: A Dataset for Abstractive Screenplay Summarization
Chen, M., Chu, Z., Wiseman, S., & Gimpel, K. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.589 · Cited by 46
Sliding Selector Network with Dynamic Memory for Extractive Summarization of Long Documents
Cui, P., & Hu, L. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.470 · Cited by 45
SCROLLS: Standardized CompaRison Over Long Language Sequences
Shaham, U., Segal, E., Ivgi, M., Efrat, A., Yoran, O., Haviv, A., Gupta, A., Xiong, W., Geva, M., Berant, J., et al. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.823 · Cited by 40
Self-citationScaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling?
Tay, Y., Dehghani, M., Abnar, S., Chung, H., Fedus, W., Rao, J., Narang, S., Tran, V., Yogatama, D., & Metzler, D. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.825 · Cited by 34
Modeling Multi-hop Question Answering as Single Sequence Prediction
Yavuz, S., Hashimoto, K., Zhou, Y., Keskar, N. S., & Xiong, C. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.69 · Cited by 28
GMAT: Global Memory Augmentation for Transformers
Gupta, A., & Berant, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.03274 · Cited by 20 · cited 2×
“Longformer (Beltagy et al., 2020), GMAT (Gupta and Berant, 2020), and ETC (Ainslie et al., 2020) use short windows of full attention, combined with full attention to a small number of predefined global input tokens.”
BackgroundSelf-citationSimple Local Attentions Remain Competitive for Long-Context Tasks
Xiong, W., Oğuz, B., Gupta, A., Chen, X., Liskovich, D., Levy, O., Yih, S., & Mehdad, Y. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.144 · Cited by 20
Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling?
Tay, Y., Dehghani, M., Abnar, S., Chung, H. W., Fedus, W., Rao, J., Narang, S., Tran, V. Q., Yogatama, D., & Metzler, D. (2022) · arXiv · doi:10.48550/arxiv.2207.10551 · Cited by 11
QuALITY: Question Answering with Long Input Texts, Yes!
Pang, R. Y., Parrish, A., Joshi, N., Nangia, N., Phang, J., Chen, A., Padmakumar, V., Ma, J., Thompson, J., He, H., et al. (2021) · arXiv · doi:10.48550/arxiv.2112.08608 · Cited by 10
Amouyal, S. J., Wolfson, T., Rubin, O., Yoran, O., Herzig, J., & Berant, J. (2022) · arXiv · doi:10.48550/arxiv.2205.12665 · cited 1×
“Despite encouraging empirical evidence (Amouyal et al., 2022; Yavuz et al., 2022), we are the first (to our knowledge) to analyze FiD’s feasibility and limitations in a controlled setting.”
BackgroundSelf-citationLuna: Linear Unified Nested Attention
Ma, X., Kong, X., Wang, S., Zhou, C., May, J., Ma, H., & Zettlemoyer, L. (2021) · arXiv · doi:10.48550/arxiv.2106.01540 · cited 1×
“, 2020a), attending only to a fixed number of hidden vectors (Ma et al., 2021), using random features to approximate the attention matrix (Choromanski et al.”
Background
Research works printed but not linked: 4
- Raffel, 2020, Exploring the limits of transfer learning with a unified text-to-text transformer, ArXiv
- Yi, 2022, Unifying language learning paradigms, ArXiv
- Xiong, 2022, Simple local attentions remain competitive for long-context tasks, 1975, 10.18653/v1/2022.naacl-main.144 · DOI
- Zaheer, 2020, Big bird: Transformers for longer sequences
Cited by
Lost in the Middle: How Language Models Use Long Contexts
Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00638 · Cited by 1,612
“Our experimental setup is similar to the needle-in-a-haystack experiments of Ivgi et al. (2023), who compare question answering performance when the relevant paragraph is placed (i) at the beginning of the input or (ii) a random position within the input.”
MethodLost in the Middle: How Language Models Use Long Contexts
Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2023) · arXiv · doi:10.48550/arxiv.2307.03172 · Cited by 1,599
“Our experimental setup is similar to the needle-in-a-haystack experiments of Ivgi et al. (2023), who compare question answering performance when the relevant paragraph is placed (i) at the beginning of the input or (ii) a random position within the input.”
MethodResource-efficient Algorithms and Systems of Foundation Models: A Survey
Xu, M., Cai, D., Yin, W., Wang, S., Jin, X., & Liu, X. (2024) · ACM Computing Surveys · doi:10.1145/3706418 · Cited by 276
“SLED [108], short for SLiding-Encoder and Decoder, repurposes and capitalizes on well-validated short-text pre-trained language models.”
An Empirical Study on Fine-Tuning Large Language Models of Code for Automated Program Repair
Huang, K., Meng, X., Zhang, J., Liu, Y., Wang, W., Li, S., & Zhang, Y. (2023) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00181 · Cited by 192
“For example, we can adopt a sliding-encoder and decoder (SLED) [26] on the pre-trained models to accept long and/or dependent methods.”
MethodParallel Context Windows for Large Language Models
Ratner, N., Levine, Y., Belinkov, Y., Ram, O., Magar, I., Abend, O., Karpas, E., Shashua, A., Leyton-Brown, K., & Shoham, Y. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.352 · Cited by 111
“Ivgi et al. (2022) suggest SLED, an encoder–decoder model for long texts, which encodes short overlapping chunks of the input text, and fuses the information in the decoder, a-la Fusion-in-Decoder (Izacard and Grave, 2020).”
BackgroundMethod
Metadata
- Authors
- Ivgi, M., Shaham, U., & Berant, J.
- Affiliations
- Tel Aviv University (IL)
- Corresponding author
- Ivgi, Maor
- Shaham, Uri
- Berant, Jonathan
- ORCID
- Shaham, Uri
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press
- Published
- 2023
- Volume
- 11
- Pages
- 284-299
- ISSN
- 2307-387X (ISSN-L)
- OpenAlex
- W4353004366
- Cited by
- 42
- References
- 50
- Citations by year
- 2026: 5 · 2025: 11 · 2024: 11 · 2023: 15
- FWCI
- 3.72
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- Locations
- Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Funders
- European Commission
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- efficient transformers, chunking, pre-trained language models, quadratic complexity, natural language understanding
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.