Transformers: State-of-the-Art Natural Language Processing
Open accessAbstract
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, Alexander Rush. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. 2020.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Almost all CLaMs have been made publicly accessible via online model repositories like HuggingFace.
“Almost all CLaMs have been made publicly accessible via online model repositories like HuggingFace57.”
From The shaky foundations of large language models and foundation models for electronic health records (2023) · cites this paper as [57]
Section 4, “Community Model Hub” (page not provided), says the Hugging Face hub lets users access and download pretrained models and held 2,097 user models. It does not establish that almost all CLaMs were publicly accessible, or give a proportion of CLaMs available through repositories. (LLM, conf=0.85)
Experimental — accuracy not yet measuredEmploying BERT, GPT-2, RoBERTa, and DistilBERT achieved robust results.
“Employing BERT, GPT-2, RoBERTa, and DistilBERT, we achieved robust results; BERT for deep contextual insights, GPT-2 for coherent text generation, RoBERTa for performance with large data, and DistilBERT for fast, cost-effective efficiency [39].”
From Generative AI-Powered Educational Alignment: A Framework for Matching Syllabus Course Topics with Web Description (2024) · cites this paper as [39]
The Library Design section (page not provided) lists BERT and RoBERTa as masked models, GPT-2 as autoregressive, and DistilBERT as efficient; the References also describe DistilBERT as faster and cheaper. These passages support several model-role descriptions, but do not establish RoBERTa’s performance with large data or GPT-2’s generation of coherent text. The citing authors’ statement that they achieved robust results is their own finding, not a result attributed to this source. (LLM, conf=0.84)
Experimental — accuracy not yet measuredOpen-source ecosystems such as Hugging Face and BLOOM reduce technical barriers but can reproduce extractive defaults without Indigenous authority over datasets, objectives, and deployment pathways.
“Open-source ecosystems such as Hugging Face and BLOOM reduce technical barriers but can reproduce extractive defaults without Indigenous authority over datasets, objectives, and deployment pathways (BigScience Workshop, 2022;Wolf et al., 2020).”
From Towards a decolonial AI sovereignty model for indigenous educational settings (2026) · cites this paper as Wolf et al., 2020
Section 4, “Community Model Hub” (page unavailable), says Hugging Face’s hub makes pretrained models easy to access and use, supporting the reduced-barriers portion. It does not address BLOOM or the claim that open-source ecosystems can reproduce extractive defaults without Indigenous authority over datasets, objectives, and deployment. (LLM, conf=0.87)
Experimental — accuracy not yet measuredBERT is a commonly used pre-trained machine learning model for NLP tasks and a ubiquitous baseline in text classification experiments of research in recent years.
“Before the emergence of ChatGPT, BERT is a commonly used pre-trained machine learning model for NLP tasks (Wolf et al., 2020), and a ubiquitous baseline in text classification experiments of research in recent years (Rogers et al., 2021).”
From Towards reliable generative AI-driven scaffolding: Reducing hallucinations and enhancing quality in self-regulated learning support (2026) · cites this paper as Wolf et al., 2020
Tamarin is implemented in roughly 9K lines of Python on top of HuggingFace Transformers; it uses no custom CUDA and does not modify model weights.
“Tamarin is implemented in roughly 9K lines of Python on top of HuggingFace Transformers (Wolf et al., 2020); it uses no custom CUDA and does not modify model weights.”
From Tamarin (HSQ): Reversible Hierarchical KV-Cache Compression for LLM Inference (2026) · cites this paper as Wolf et al., 2020
FlashAttention achieves 3x speedup on GPT-2 (seq length 1K) compared to baselines.
“3 speedup on GPT-2 (seq. length 1K)”
From FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (2022) · cites this paper as [87]
The Deployment section reports speedups from ONNX optimization of models including GPT-2, but does not discuss FlashAttention. Searches for FlashAttention and a 3x speedup at sequence length 1K found no passage addressing the claimed result; the ONNX result does not verify or contradict it. Page numbers were unavailable in our copy. (LLM, conf=0.76)
Experimental — accuracy not yet measuredGPT-2 small and medium using FlashAttention achieve up to 3x speedup compared to Huggingface and up to 1.7x compared to Megatron-LM.
“Table 2 shows up to 3 endto-end speedup compared to Huggingface and 1.7 speedup compared to Megatron-LM.”
From FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (2022) · cites this paper as [87]
The Deployment section reports a nearly 4× speedup using ONNX, not FlashAttention. Searches for the claimed 3× and 1.7× end-to-end comparisons found no such results in our copy; Related Work mentions Megatron-LM but does not benchmark against it. (LLM, conf=0.77)
Experimental — accuracy not yet measuredEarly explorations of interactive model cards do not explore how much or what kind of interactivity is beneficial.
“While these early explorations are promising, they do not explore how much or what kind of interactivity is beneficial.”
From Interactive Model Cards: A Human-Centered Approach to Model Documentation (2022) · cites this paper as [73]
High-quality open-source libraries have lowered the barrier to entry and make it possible to build conversational models that exhibit high generalization and coherence.
“along with high-quality open-source libraries [113] have lowered the barrier to entry and make it possible to build conversational models that exhibit high generalization and coherence [90].”
From Future directions for chatbot research: An interdisciplinary research agenda (2021) · cites this paper as [113]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 2× · see citing works
“The Transformer (Vaswani et al., 2017) has rapidly become the dominant architecture for natural language processing, surpassing alternative neural models such as convolutional and recurrent neural networks in performance for tasks in both natural language understanding and natural language…”
Jakubův, J., Chvalovský, K., Goertzel, Z., et al. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,648 · see citing works
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515 · cited 3× · see citing works
“…text corpora, leading to major gains in accuracy on downstream tasks including text classification (Yang et al., 2019), language understanding (Liu et al., 2019b; Wang et al., 2018, 2019), machine translation (Lample and Conneau, 2019a), coreference resolution (Joshi et al., 2019),…”
InfluentialDeep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · cited 1× · see citing works
“Model pretraining (McCann et al., 2017; Howard and Ruder, 2018; Peters et al., 2018; Devlin et al., 2018) allows models to be trained on generic corpora and subsequently be easily adapted to specific tasks with strong performance.”
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · cited 2× · see citing works
“, 2019) T5 (Raffel et al., 2019) MarianMT (J.”
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821 · see citing works
The Stanford CoreNLP Natural Language Processing Toolkit
Manning, C. D., Surdeanu, M., Bauer, J., Finkel, J. R., Bethard, S. J., & McClosky, D. (2014) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.3115/v1/p14-5010 · Cited by 7,171 · cited 1× · see citing works
“Two core libraries are NLTK (Loper and Bird, 2002) and Stanford CoreNLP (Manning et al., 2014), which collect a variety of different approaches to NLP in a single package.”
XLNet: Generalized Autoregressive Pretraining for Language Understanding
Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1906.08237 · Cited by 6,687 · cited 3× · see citing works
“The Transformer architecture is particularly conducive to pretraining on large text corpora, leading to major gains in accuracy on downstream tasks including text classification (Yang et al., 2019), language understanding (Liu et al.”
InfluentialDistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter
Sanh, V., Debut, L., Chaumond, J., & Wolf, T. (2019) · arXiv · doi:10.48550/arxiv.1910.01108 · Cited by 4,653 · cited 2× · see citing works
“While core models like BERT and GPT-2 continue to be popular, other specialized models including DistilBERT (Sanh et al., 2019), which was developed for the library, are now widely downloaded by the community.”
Self-citationALBERT: A Lite BERT for Self-supervised Learning of Language Representations
Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., & Soricut, R. (2019) · arXiv · doi:10.48550/arxiv.1909.11942 · Cited by 4,308 · cited 1× · see citing works
“ALBERT (Lan et al., 2019) Electra (Clark et al.”
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. (2018) · Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/w18-5446 · Cited by 4,222 · cited 1× · see citing works
“…leading to major gains in accuracy on downstream tasks including text classification (Yang et al., 2019), language understanding (Liu et al., 2019b; Wang et al., 2018, 2019), machine translation (Lample and Conneau, 2019a), coreference resolution (Joshi et al., 2019), commonsense inference…”
Universal Language Model Fine-tuning for Text Classification
Howard, J., & Ruder, S. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1031 · Cited by 4,100 · cited 1× · see citing works
“Model pretraining (McCann et al., 2017; Howard and Ruder, 2018; Peters et al., 2018; Devlin et al., 2018) allows models to be trained on generic corpora and subsequently be easily adapted to specific tasks with strong performance.”
fairseq: A Fast, Extensible Toolkit for Sequence Modeling
Ott, M., Edunov, S., Baevski, A., Fan, A., Gross, S., Ng, N., Grangier, D., & Auli, M. (2019) · North American Chapter of the Association for Computational Linguistics · doi:10.18653/v1/n19-4009 · Cited by 3,538 · cited 2× · see citing works
“The library is also closely related to neural translation and language modeling systems, such as Fairseq (Ott et al., 2019), OpenNMT (Klein et al.”
SciBERT: A Pretrained Language Model for Scientific Text
Beltagy, I., Lo, K., & Cohan, A. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1371 · Cited by 3,218 · cited 4× · see citing works
“We highlight specific systems developed by users with different goals following the architect, trainer, and end-user distinction of Strobelt et al. (2017): Case 1: Model Architects AllenAI, a major NLP research lab, developed a new pretrained model for improved extraction from biomedical texts called SciBERT (Beltagy…”
InfluentialTransformer-XL: Attentive language models beyond a fixed-length context
Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q., & Salakhutdinov, R. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1285 · Cited by 3,169 · cited 3× · see citing works
“, 2019) Trans-XL (Dai et al., 2019) XLNet (Yang et al.”
Model Cards for Model Reporting
Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., & Gebru, T. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287596 · Cited by 2,681 · cited 1× · see citing works
“Additional model-specific metadata can be provided via a model card (Mitchell et al., 2018) that describes properties of its training, a citation to the work, datasets used during pretraining, and any caveats about known biases in the model and its predictions.”
Longformer: The Long-Document Transformer
Beltagy, I., Peters, M. E., & Cohan, A. (2020) · arXiv · doi:10.48550/arxiv.2004.05150 · Cited by 2,281 · cited 1× · see citing works
“, 2020) Longformer (Beltagy et al., 2020)”
NLTK: The Natural Language ToolkitPreprint
Loper, E., & Bird, S. (2002) · arXiv (Cornell University) · doi:10.48550/arxiv.cs/0205028 · Cited by 1,893 · cited 2× · see citing works
“Two core libraries are NLTK (Loper and Bird, 2002) and Stanford CoreNLP (Manning et al., 2014), which collect a variety of different approaches to NLP in a single package.”
ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators
Clark, K., Luong, M.-T., Le, Q. V., & Manning, C. D. (2020) · arXiv · doi:10.48550/arxiv.2003.10555 · Cited by 1,844 · see citing works
SpanBERT: Improving Pre-training by Representing and Predicting Spans
Joshi, M., Chen, D., Liu, Y., Weld, D. S., Zettlemoyer, L., & Levy, O. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00300 · Cited by 1,827 · cited 2× · see citing works
“…classification (Yang et al., 2019), language understanding (Liu et al., 2019b; Wang et al., 2018, 2019), machine translation (Lample and Conneau, 2019a), coreference resolution (Joshi et al., 2019), commonsense inference (Bosselut et al., 2019), and summarization (Lewis et al., 2019) among others.”
OpenNMT: Open-Source Toolkit for Neural Machine Translation
Klein, G., Kim, Y., Deng, Y., Sénellart, J., & Rush, A. M. (2017) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.18653/v1/p17-4012 · Cited by 1,771 · see citing works
Self-citationCross-lingual Language Model Pretraining
Lample, G., & Conneau, A. (2019) · arXiv · doi:10.48550/arxiv.1901.07291 · Cited by 1,586 · cited 1× · see citing works
“…tasks including text classification (Yang et al., 2019), language understanding (Liu et al., 2019b; Wang et al., 2018, 2019), machine translation (Lample and Conneau, 2019a), coreference resolution (Joshi et al., 2019), commonsense inference (Bosselut et al., 2019), and summarization (Lewis et…”
Question Answering For Toxicological Information Extraction
Ferreira, B. C. L., Gonçalo Oliveira, H., Amaro, H., et al. (2022) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/oasics.slate.2022.3 · Cited by 1,535 · see citing works
Stanza: A Python Natural Language Processing Toolkit for Many Human Languages
Qi, P., Zhang, Y., Zhang, Y., Bolton, J., & Manning, C. D. (2020) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.18653/v1/2020.acl-demos.14 · Cited by 1,505 · see citing works
Reformer: The Efficient Transformer
Kitaev, N., Kaiser, Ł., & Levskaya, A. (2020) · arXiv · doi:10.48550/arxiv.2001.04451 · Cited by 1,435 · cited 1× · see citing works
“Reformer (Kitaev et al., 2020) Longformer (Beltagy et al.”
BERT Rediscovers the Classical NLP Pipeline
Tenney, I., Das, D., & Pavlick, E. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1452 · Cited by 1,307 · cited 1× · see citing works
“They used the Transformers API as a generic front-end and performed fine-tuning on a variety of different models, leading to research on the structure of BERT (Tenney et al., 2019).”
AllenNLP: A Deep Semantic Natural Language Processing Platform
Gardner, M., Grus, J., Neumann, M., Tafjord, O., Dasigi, P., Liu, N. F., Peters, M. E., Schmitz, M., & Zettlemoyer, L. (2018) · Workshop for NLP Open Source Software (NLP-OSS) · doi:10.18653/v1/w18-2501 · Cited by 1,273 · cited 4× · see citing works
“The concept of providing easy caching for pretrained models stemmed from AllenNLP (Gardner et al., 2018).”
InfluentialTVM: an automated end-to-end optimizing compiler for deep learning
(2018) · arXiv (Cornell University) · doi:10.5555/3291168.3291211 · Cited by 1,170 · see citing works
SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems
Wang, A., Pruksachatkun, Y., Nangia, N., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. R. (2019) · arXiv · doi:10.48550/arxiv.1905.00537 · Cited by 1,051 · cited 1× · see citing works
“…leading to major gains in accuracy on downstream tasks including text classification (Yang et al., 2019), language understanding (Liu et al., 2019b; Wang et al., 2018, 2019), machine translation (Lample and Conneau, 2019a), coreference resolution (Joshi et al., 2019), commonsense inference…”
COMET: Commonsense Transformers for Automatic Knowledge Graph Construction
Bosselut, A., Rashkin, H., Sap, M., Malaviya, C., Celikyilmaz, A., & Choi, Y. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1470 · Cited by 904 · cited 2× · see citing works
“, 2019), commonsense inference (Bosselut et al., 2019), and summarization (Lewis et al.”
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., & Catanzaro, B. (2019) · arXiv · doi:10.48550/arxiv.1909.08053 · Cited by 893 · cited 1× · see citing works
“The library is also closely related to neural translation and language modeling systems, such as Fairseq (Ott et al., 2019), OpenNMT (Klein et al., 2017), Texar (Hu et al., 2018), Megatron-LM (Shoeybi et al., 2019), and Marian NMT (Junczys-Dowmunt et al., 2018).”
Learned in Translation: Contextualized Word Vectors
McCann, B., Bradbury, J., Xiong, C., & Socher, R. (2017) · arXiv · doi:10.48550/arxiv.1708.00107 · Cited by 572 · cited 1× · see citing works
“Model pretraining (McCann et al., 2017; Howard and Ruder, 2018; Peters et al., 2018; Devlin et al., 2018) allows models to be trained on generic corpora and subsequently be easily adapted to specific tasks with strong performance.”
Tensor2Tensor for Neural Machine TranslationPreprint
Vaswani, A., Bengio, S., Brevdo, E., Chollet, F., Gomez, A. N., Gouws, S., Jones, L., Kaiser, Ł., Kalchbrenner, N., Parmar, N., et al. (2018) · Conference of the Association for Machine Translation in the Americas · doi:10.48550/arxiv.1803.07416 · Cited by 556 · cited 1× · see citing works
“The structure of Transformers is inspired by the pioneering tensor2tensor library (Vaswani et al., 2018) and the original source code for BERT (Devlin et al., 2018), both from Google Research.”
LSTMVis: A Tool for Visual Analysis of Hidden State Dynamics in Recurrent Neural Networks
Strobelt, H., Gehrmann, S., Pfister, H., & Rush, A. M. (2017) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2017.2744158 · Cited by 399 · cited 2× · see citing works
“We highlight specific systems developed by users with different goals following the architect, trainer, and end-user distinction of Strobelt et al. (2017):”
Self-citationSupervised Multimodal Bitransformers for Classifying Images and TextPreprint
Kiela, D., Bhooshan, S., Firooz, H., Perez, E., & Testuggine, D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.02950 · Cited by 271 · cited 3× · see citing works
“Datasets Language Modeling x1:n−1 xn ∈ V Generation WikiText-103 Sequence Classification x1:N y ∈ C Classification, Sentiment Analysis GLUE, SST, MNLI Question Answering x1:M ,xM :N y span [1 : N ] QA, Reading Comprehension SQuAD, Natural Questions Token Classification x1:N y1:N ∈ CN NER, Tagging OntoNotes, WNUT Multi…”
FlauBERT: Unsupervised Language Model Pre-training for FrenchPreprint
Le, H., Vial, L., Frej, J., Segonne, V., Coavoux, M., Lecouteux, B., Allauzen, A., Crabbé, B., Besacier, L., & Schwab, D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.05372 · Cited by 271 · cited 3× · see citing works
“To load FlauBERT (Le et al., 2020), a BERT model pretrained on a French training corpus, the command is:”
Influentialjiant: A Software Toolkit for Research on General-Purpose Text Understanding Models
Pruksachatkun, Y., Yeres, P., Liu, H., Phang, J., Htut, P. M., Wang, A., Tenney, I., & Bowman, S. R. (2020) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.18653/v1/2020.acl-demos.15 · Cited by 67 · see citing works
exBERT: A Visual Analysis Tool to Explore Learned Representations in Transformers ModelsPreprint
Hoover, B., Strobelt, H., & Gehrmann, S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.05276 · Cited by 55 · cited 1× · see citing works
“For example, model pages can link to exBERT (Hoover et al., 2019), a Transformer visualization library.”
Texar: A Modularized, Versatile, and Extensible Toolkit for Text Generation
Hu, Z., Shi, H., Tan, B., Wang, W., Yang, Z., Zhao, T., He, J., Qin, L., Wang, D., Ma, X., et al. (2019) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.18653/v1/p19-3027 · Cited by 41 · cited 1× · see citing works
“The library is also closely related to neural translation and language modeling systems, such as Fairseq (Ott et al., 2019), OpenNMT (Klein et al., 2017), Texar (Hu et al., 2018), Megatron-LM (Shoeybi et al., 2019), and Marian NMT (Junczys-Dowmunt et al., 2018).”
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · cited 4×
“, 2018) and the original source code for BERT (Devlin et al., 2018), both from Google Research.”
InfluentialStanza: A Python Natural Language Processing Toolkit for Many Human Languages
Qi, P., Zhang, Y., Zhang, Y., Bolton, J., & Manning, C. D. (2020) · arXiv · doi:10.48550/arxiv.2003.07082 · cited 2×
“More recently, general-purpose, open-source libraries have focused primarily on machine learning for a variety of NLP tasks, these include Spacy (Honnibal and Montani, 2017), AllenNLP (Gardner et al., 2018), flair (Akbik et al., 2019), and Stanza (Qi et al., 2020).”
Flagged first, then by citations
Research works printed but not linked: 2
- Alan Akbik, Tanja Bergmann, Duncan Blythe, Kashif Rasul, Stefan Schweter, and Roland Vollgraf. 2019. Flair: An easy-to-use framework for state-of-the-art nlp. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Com- putational Linguistics (Demonstrations), pages 54- 59. aclweb.org.
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8):9.
Cited by
Sorted by citations
High-Resolution Image Synthesis with Latent Diffusion Models
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01042 · Cited by 15,880 · see citing works
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988 · see citing works
DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter
Sanh, V., Debut, L., Chaumond, J., & Wolf, T. (2019) · arXiv · doi:10.48550/arxiv.1910.01108 · Cited by 4,653 · see citing works
Self-citationOn the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676 · see citing works
SimCSE: Simple Contrastive Learning of Sentence Embeddings
Gao, T., Yao, X., & Chen, D. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.552 · Cited by 2,851 · see citing works
LoRA: Low-Rank Adaptation of Large Language Models
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2021) · arXiv · doi:10.48550/arxiv.2106.09685 · Cited by 2,523 · see citing works
Prefix-Tuning: Optimizing Continuous Prompts for Generation
Li, X. L., & Liang, P. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.353 · Cited by 2,473 · see citing works
Efficient Memory Management for Large Language Model Serving with PagedAttention
Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., & Stoica, I. (2023) · arXiv · doi:10.48550/arxiv.2309.06180 · Cited by 1,802 · see citing works
RoFormer: Enhanced transformer with rotary position embedding
Su, J., Ahmed, M., Lu, Y., Pan, S., Bo, W., & Liu, Y. (2024) · Neurocomputing · doi:10.1016/j.neucom.2023.127063 · Cited by 1,957 · see citing works
Efficient Memory Management for Large Language Model Serving with PagedAttention
Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J., Zhang, H., & Stoica, I. (2023) · Proceedings of the 29th Symposium on Operating Systems Principles · doi:10.1145/3600006.3613165 · Cited by 1,817 · see citing works
Heterogeneous Graph Transformer
Hu, Z., Dong, Y., Wang, K., & Sun, Y. (2020) · Proceedings of The Web Conference 2020 · doi:10.1145/3366423.3380027 · Cited by 1,646 · see citing works
A Comprehensive Overview of Large Language Models
Naveed, H., Khan, A. U., Qiu, S., Saqib, M., Anwar, S., Usman, M., Akhtar, N., Barnes, N., & Mian, A. (2025) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3744746 · Cited by 1,257 · see citing works
BioGPT: generative pre-trained transformer for biomedical text generation and mining
Luo, R., Sun, L., Xia, Y., Qin, T., Zhang, S., Poon, H., & Liu, T.-Y. (2022) · Briefings in Bioinformatics · doi:10.1093/bib/bbac409 · Cited by 1,150 · see citing works
Fastai: A Layered API for Deep Learning
Howard, J., & Gugger, S. (2020) · Information · doi:10.3390/info11020108 · Cited by 992 · see citing works
Self-citationAlzubaidi, L., Bai, J., Al-Sabaawi, A., Santamaría, J., Albahri, A. S., Al-dabbagh, B. S. N., Fadhel, M. A., Manoufali, M., Zhang, J., Al-Timemy, A. H., et al. (2023) · Journal Of Big Data · doi:10.1186/s40537-023-00727-2 · Cited by 992 · see citing works
Rasmy, L., Xiang, Y., Xie, Z., Tao, C., & Zhi, D. (2021) · npj Digital Medicine · doi:10.1038/s41746-021-00455-y · Cited by 886 · see citing works
From ChatGPT to ThreatGPT: Impact of Generative AI in Cybersecurity and Privacy
Gupta, M., Akiri, C., Aryal, K., Parker, E., & Praharaj, L. (2023) · IEEE Access · doi:10.1109/access.2023.3300381 · Cited by 714 · see citing works
Adapted large language models can outperform medical experts in clinical text summarization
Van Veen, D., Van Uden, C., Blankemeier, L., Delbrouck, J.-B., Aali, A., Bluethgen, C., Pareek, A., Polacin, M., Reis, E. P., Seehofnerová, A., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-02855-5 · Cited by 891 · see citing works
BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models
Ben Zaken, E., Goldberg, Y., & Ravfogel, S. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/2022.acl-short.1 · Cited by 765 · see citing works
MDETR - Modulated Detection for End-to-End Multi-Modal Understanding
Kamath, A., Singh, M., LeCun, Y., Synnaeve, G., Misra, I., & Carion, N. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00180 · Cited by 761 · see citing works
TruthfulQA: Measuring How Models Mimic Human Falsehoods
Lin, S., Hilton, J., & Evans, O. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.229 · Cited by 904 · see citing works
An introduction to Deep Learning in Natural Language Processing: Models, techniques, and tools
Lauriola, I., Lavelli, A., & Aiolli, F. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.05.103 · Cited by 721 · see citing works
Harnessing the power of llms in practice: a survey on chatGPT and beyond
Yang, J., Jin, H., Tang, R., Han, X., Feng, Q., Jiang, H., Zhong, S., Yin, B., & Hu, X. (2024) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3649506 · Cited by 644 · see citing works
Less is More: CLIPBERT for Video-and-Language Learning via Sparse Sampling
Lei, J., Li, L., Zhou, L., Gan, Z., Berg, T. L., Bansal, M., & Liu, J. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00725 · Cited by 606 · see citing works
Swin Transformer Embedding UNet for Remote Sensing Image Semantic Segmentation
He, X. J., Zhou, Y. Z., Zhao, J., Zhang, D., Yao, R., & Xue, Y. (2022) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2022.3144165 · Cited by 561 · see citing works
Machine Learning Methods for Small Data Challenges in Molecular Science
Dou, B., Zhu, Z., Merkurjev, E., Ke, L., Chen, L., Jian, J., Zhu, Y., Liu, J., Zhang, B., & Wei, G. (2023) · Chemical Reviews · doi:10.1021/acs.chemrev.3c00189 · Cited by 522 · see citing works
LayoutLMv2: Multi-modal Pre-training for Visually-rich Document Understanding
Xu, Y., Xu, Y., Lv, T., Cui, L., Wei, F., Wang, G., Lu, Y., Florencio, D. A. F., Zhang, C., Che, W., et al. (2021) · doi:10.18653/v1/2021.acl-long.201 · Cited by 511 · see citing works
PTR: Prompt Tuning with Rules for Text Classification
Han, X., Zhao, W., Ding, N., Liu, Z., & Sun, M. (2022) · AI Open · doi:10.1016/j.aiopen.2022.11.003 · Cited by 503 · see citing works
Holistic Evaluation of Language Models
Bommasani, R., Liang, P., & Lee, T. (2023) · Annals of the New York Academy of Sciences · doi:10.1111/nyas.15007 · Cited by 480 · see citing works
Machine learning and deep learning—A review for ecologists
Pichler, M., & Hartig, F. (2023) · Methods in Ecology and Evolution · doi:10.1111/2041-210x.14061 · Cited by 504 · see citing works
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
Dao, T., Fu, D. Y., Ermon, S., Rudra, A., & Ré, C. (2022) · arXiv · doi:10.48550/arxiv.2205.14135 · Cited by 1,214 · see citing works
CLAP Learning Audio Concepts from Natural Language Supervision
Elizalde, B., Deshmukh, S., Ismail, M. A., & Wang, H. (2023) · ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp49357.2023.10095889 · Cited by 449 · see citing works
Hu, Z., Dong, Y., Wang, K., Chang, K.-W., & Sun, Y. (2020) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3394486.3403237 · Cited by 517 · see citing works
MTEB: Massive Text Embedding Benchmark
Muennighoff, N., Tazi, N., Magne, L., & Reimers, N. (2023) · Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.148 · Cited by 548 · see citing works
LAVT: Language-Aware Vision Transformer for Referring Image Segmentation
Yang, Z., Wang, J., Tang, Y., Chen, K., Zhao, H., & Torr, P. H. S. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01762 · Cited by 439 · see citing works
Efficient Acceleration of Deep Learning Inference on Resource-Constrained Edge Devices: A Review
Shuvo, M. M. H., Islam, S. K., Cheng, J., & Morshed, B. I. (2023) · Proceedings of the IEEE · doi:10.1109/jproc.2022.3226481 · Cited by 436 · see citing works
In-Context Retrieval-Augmented Language Models
Ram, O., Levine, Y., Dalmedigos, I., Muhlgay, D., Shashua, A., Leyton-Brown, K., & Shoham, Y. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00605 · Cited by 449 · see citing works
A Survey on Large Language Models: Applications, Challenges, Limitations, and Practical UsagePreprint
Hadi, M. U., Al-Tashi, Q., Qureshi, R., Shah, A., Muneer, A., Irfan, M., Zafar, A., Shaikh, M. B., Akhtar, N., Hassan, S. Z., et al. (2023) · doi:10.36227/techrxiv.23589741.v1 · Cited by 429 · see citing works
Hossain, E., Rana, R., Higgins, N., Soar, J., Barua, P. D., Pisani, A. R., & Turner, K. (2023) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2023.106649 · Cited by 407 · see citing works
LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Models
Song, C. H., Sadler, B. M., Wu, J., Chao, W.-L., Washington, C., & Su, Y. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00280 · Cited by 435 · see citing works
Deep entity matching with pre-trained language models
Li, Y., Li, J., Suhara, Y., Doan, A., & Tan, W.-C. (2020) · Proceedings of the VLDB Endowment · doi:10.14778/3421424.3421431 · Cited by 446 · see citing works
Semantic reconstruction of continuous language from non-invasive brain recordings
Tang, J., LeBel, A., Jain, S., & Huth, A. G. (2023) · Nature Neuroscience · doi:10.1038/s41593-023-01304-9 · Cited by 442 · see citing works
Fu, M. C., & Tantithamthavorn, C. K. (2022) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/3524842.3528452 · Cited by 424 · see citing works
RWKV: Reinventing RNNs for the Transformer Era
Peng, B., Alcaide, E., Anthony, Q., Albalak, A., Arcadinho, S., Biderman, S., Cao, H., Cheng, X., Chung, M., Derczynski, L., et al. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.936 · Cited by 390 · see citing works
DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations
Giorgi, J., Nitski, O., Wang, B., & Bader, G. D. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.72 · Cited by 432 · see citing works
Hadi Mogavi, R., Deng, C., Juho Kim, J., Zhou, P., D. Kwon, Y., Hosny Saleh Metwally, A., Tlili, A., Bassanelli, S., Bucchiarone, A., Gujar, S., et al. (2024) · Computers in Human Behavior: Artificial Humans · doi:10.1016/j.chbah.2023.100027 · Cited by 369 · see citing works
Mapping the space of chemical reactions using attention-based neural networks
Schwaller, P., Probst, D., Vaucher, A. C., Nair, V. H., Kreutter, D., Laino, T., & Reymond, J. (2021) · Nature Machine Intelligence · doi:10.1038/s42256-020-00284-w · Cited by 360 · see citing works
(2020) · doi:10.18653/v1/2020.acl-main.450 · Cited by 356 · see citing works
Lin, J., Ma, X., Lin, S.-C., Yang, J.-H., Pradeep, R., & Nogueira, R. (2021) · Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3404835.3463238 · Cited by 360 · see citing works
ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction
Santhanam, K., Khattab, O., Saad-Falcon, J., Potts, C., & Zaharia, M. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.272 · Cited by 353 · see citing works
A Simple Language Model for Task-Oriented DialoguePreprint
Hosseini-Asl, E., McCann, B., Wu, C.-S., Yavuz, S., & Socher, R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.00796 · Cited by 368 · see citing works
Siriwardhana, S., Weerasekera, R., Wen, E., Kaluarachchi, T., Rana, R., & Nanayakkara, S. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00530 · Cited by 337 · see citing works
Generative AI Literacy: Twelve Defining Competencies
Annapureddy, R., Fornaroli, A., & Gatica-Perez, D. (2025) · Digital Government: Research and Practice · doi:10.1145/3685680 · Cited by 333 · see citing works
Can Large Language Models Be an Alternative to Human Evaluations?
Chiang, C.-H., & Lee, H.-Y. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.870 · Cited by 331 · see citing works
The shaky foundations of large language models and foundation models for electronic health records
Wornow, M., Xu, Y., Thapa, R., Patel, B., Steinberg, E., Fleming, S., Pfeffer, M. A., Fries, J., & Shah, N. H. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00879-8 · Cited by 324 · see citing works
Evidence of a predictive coding hierarchy in the human brain listening to speech
Caucheteux, C., Gramfort, A., & King, J.-R. (2023) · Nature Human Behaviour · doi:10.1038/s41562-022-01516-2 · Cited by 308 · see citing works
SwinBERT: End-to-End Transformers with Sparse Attention for Video Captioning
Lin, K., Li, L., Lin, C.-C., Ahmed, F., Gan, Z., Liu, Z., Lu, Y., & Wang, L. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01742 · Cited by 321 · see citing works
Split Computing and Early Exiting for Deep Learning Applications: Survey and Research Challenges
Matsubara, Y., Levorato, M., & Restuccia, F. (2022) · ACM Computing Surveys · doi:10.1145/3527155 · Cited by 327 · see citing works
PromptCast: A New Prompt-Based Learning Paradigm for Time Series Forecasting
Xue, H., & Salim, F. D. (2023) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2023.3342137 · Cited by 332 · see citing works
Wortsman, M., Ilharco, G., Gadre, S. Y., Roelofs, R., Gontijo-Lopes, R., Morcos, A. S., Namkoong, H., Farhadi, A., Carmon, Y., Kornblith, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.05482 · Cited by 307 · see citing works
Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus
Dodge, J., Sap, M., Marasović, A., Agnew, W., Ilharco, G., Groeneveld, D., Mitchell, M., & Gardner, M. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.98 · Cited by 306 · see citing works
Extraction of organic chemistry grammar from unsupervised learning of chemical reactions
Schwaller, P., Hoover, B., Reymond, J., Strobelt, H., & Laino, T. (2021) · Science Advances · doi:10.1126/sciadv.abe4166 · Cited by 301 · see citing works
Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Sampling
Hofstätter, S., Lin, S.-C., Yang, J.-H., Lin, J., & Hanbury, A. (2021) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3404835.3462891 · Cited by 318 · see citing works
Autoregressive Image Generation using Residual Quantization
Lee, D., Kim, C., Kim, S., Cho, M., & Han, W.-S. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01123 · Cited by 305 · see citing works
Splitwise: Efficient Generative LLM Inference Using Phase Splitting
Patel, P., Choukse, E., Zhang, C., Shah, A., Goiri, Í., Maleki, S., & Bianchini, R. (2024) · 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA) · doi:10.1109/isca59077.2024.00019 · Cited by 298 · see citing works
Prediction of chemical reaction yields using deep learning
Schwaller, P., Vaucher, A. C., Laino, T., & Reymond, J.-L. (2021) · Machine Learning: Science and Technology · doi:10.1088/2632-2153/abc81d · Cited by 291 · see citing works
♫ MuSiQue: Multihop Questions via Single-hop Question Composition
Trivedi, H., Balasubramanian, N., Khot, T., & Sabharwal, A. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00475 · Cited by 283 · see citing works
polyBERT: a chemical language model to enable fully machine-driven ultrafast polymer informatics
Kuenneth, C., & Ramprasad, R. (2023) · Nature Communications · doi:10.1038/s41467-023-39868-6 · Cited by 283 · see citing works
A systematic review of hate speech automatic detection using natural language processing
Jahan, M. S., & Oussalah, M. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.126232 · Cited by 345 · see citing works
Sun, L., Lian, Z., Liu, B., & Tao, J. (2023) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2023.3274829 · Cited by 283 · see citing works
Large Language Models for Mental Health Applications: Systematic Review
Guo, Z., Lai, A., Thygesen, J. H., Farrington, J., Keen, T., & Li, K. (2024) · JMIR Mental Health · doi:10.2196/57400 · Cited by 269 · see citing works
CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X
Zheng, Q., Xia, X., Zou, X., Dong, Y., Wang, S., Xue, Y., Shen, L., Wang, Z., Wang, A., Li, Y., et al. (2023) · Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining · doi:10.1145/3580305.3599790 · Cited by 263 · see citing works
TimeLMs: Diachronic Language Models from Twitter
Loureiro, D., Barbieri, F., Neves, L., Espinosa Anke, L., & Camacho-collados, J. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations · doi:10.18653/v1/2022.acl-demo.25 · Cited by 263 · see citing works
Bulathwela, S., Pérez-Ortiz, M., Holloway, C., Cukurova, M., & Shawe-Taylor, J. (2024) · Sustainability · doi:10.3390/su16020781 · Cited by 269 · see citing works
A Survey on Text Classification Algorithms: From Text to Predictions
Gasparetto, A., Marcuzzo, M., Zangari, A., & Albarelli, A. (2022) · Information · doi:10.3390/info13020083 · Cited by 278 · see citing works
Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks
Wang, Y., Mishra, S., Alipoormolabashi, P., Kordi, Y., Mirzaei, A., Naik, A., Ashok, A., Dhanasekaran, A. S., Arunkumar, A., Stap, D., et al. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.340 · Cited by 247 · see citing works
Future directions for chatbot research: An interdisciplinary research agenda
Følstad, A., Araujo, T., Law, E. L.-C., Brandtzaeg, P. B., Papadopoulos, S., Reis, L., Baez, M., Laban, G., McAllister, P., Ischen, C., et al. (2021) · Computing · doi:10.1007/s00607-021-01016-7 · Cited by 247 · see citing works
From Zero to Hero: On the Limitations of Zero-Shot Language Transfer with Multilingual Transformers
Lauscher, A., Ravishankar, V., Vulić, I., & Glavaš, G. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.363 · Cited by 250 · see citing works
AI deception: A survey of examples, risks, and potential solutions
Park, P. S., Goldstein, S., O’Gara, A., Chen, M., & Hendrycks, D. (2024) · Patterns · doi:10.1016/j.patter.2024.100988 · Cited by 243 · see citing works
AI Deception: A Survey of Examples, Risks, and Potential Solutions
Park, P. S., Goldstein, S., O'Gara, A., Chen, M., & Hendrycks, D. (2023) · arXiv · doi:10.48550/arxiv.2308.14752 · Cited by 243 · see citing works
Mao, R., Liu, Q., He, K., Li, W., & Cambria, E. (2022) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2022.3204972 · Cited by 247 · see citing works
Generative Models for De Novo Drug Design
Tong, X., Liu, X., Tan, X., Li, X., Jiang, J., Xiong, Z., Xu, T., Jiang, H., Qiao, N., & Zheng, M. (2021) · Journal of Medicinal Chemistry · doi:10.1021/acs.jmedchem.1c00927 · Cited by 242 · see citing works
<scp>SummaC</scp>: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
Laban, P., Schnabel, T., Bennett, P. N., & Hearst, M. A. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00453 · Cited by 238 · see citing works
Xie, T., Wu, C. H., Shi, P., Zhong, R., Scholak, T., Yasunaga, M., Wu, C.-S., Zhong, M., Yin, P., Wang, S. I., et al. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.39 · Cited by 271 · see citing works
Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality
Thrush, T., Jiang, R., Bartolo, M., Singh, A., Williams, A., Kiela, D., & Ross, C. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00517 · Cited by 240 · see citing works
Bad Actor, Good Advisor: Exploring the Role of Large Language Models in Fake News Detection
Hu, B., Sheng, Q., Cao, J., Shi, Y., Li, Y., Wang, D., & Qi, P. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i20.30214 · Cited by 252 · see citing works
What does a platypus look like? Generating customized prompts for zero-shot image classification
Pratt, S., Covert, I., Liu, R., & Farhadi, A. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01438 · Cited by 240 · see citing works
Sanger: A Co-Design Framework for Enabling Sparse Attention using Reconfigurable Architecture
Lu, L., Jin, Y., Bi, H., Luo, Z., Li, P., Wang, T., & Liang, Y. E. (2021) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1145/3466752.3480125 · Cited by 232 · see citing works
XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languages
Hasan, T., Bhattacharjee, A., Islam, M. S., Mubasshir, K. S., Li, Y.-F., Kang, Y., Rahman, M. S., & Shahriyar, R. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.413 · Cited by 240 · see citing works
Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLP
Schick, T., Udupa, S., & Schütze, H. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00434 · Cited by 227 · see citing works
Towards Emotional Support Dialog Systems
Liu, S., Zheng, C., DeMasi, O., Sabour, S., Yu, L., Yu, Z., Jiang, Y., & Huang, M. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.269 · Cited by 286 · see citing works
Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Models
Xie, X., Zhou, P., Li, H., Lin, Z., & Yan, S. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3423382 · Cited by 247 · see citing works
Few-NERD: A Few-shot Named Entity Recognition Dataset
Ding, N., Xu, G., Chen, Y., Wang, X., Han, X., Xie, P., Zheng, H.-T., & Liu, Z. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.248 · Cited by 245 · see citing works
Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations
Davani, A. M., Díaz, M., & Prabhakaran, V. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00449 · Cited by 237 · see citing works
BRIO: Bringing Order to Abstractive Summarization
Liu, Y., Liu, P., Radev, D., & Neubig, G. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.207 · Cited by 244 · see citing works
Measuring and improving consistency in pretrained language models
Elazar, Y., Kassner, N., Ravfogel, S., Ravichander, A., Hovy, E., Schütze, H., & Goldberg, Y. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00410 · Cited by 220 · see citing works
SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarization
Liu, Y., & Liu, P. (2021) · doi:10.18653/v1/2021.acl-short.135 · Cited by 236 · see citing works
LexGLUE: A Benchmark Dataset for Legal Language Understanding in English
Chalkidis, I., Jana, A., Hartung, D., Bommarito, M. J., Androutsopoulos, I., Katz, D. M., & Αλέτρας, Ν. (2021) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.297 · Cited by 376 · see citing works
ChatGPT: Fundamentals, Applications and Social Impacts
Abdullah, M., Madain, A., & Jararweh, Y. (2022) · 2022 Ninth International Conference on Social Networks Analysis, Management and Security (SNAMS) · doi:10.1109/snams58071.2022.10062688 · Cited by 310 · see citing works
- Authors
- Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., Davison, J., Shleifer, S., von Platen, P., Ma, C., Jernite, Y., Plu, J., Xu, C., Le Scao, T., Gugger, S., Drame, M., Lhoest, Q., & Rush, A.
- Affiliations
- Hugging Face (US)
- ORCID
- Venue
- Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations
- Published
- 2020
- Pages
- 38-45
- OpenAlex
- W2979826702
- Cited by
- 8,741
- References
- 39
- Citations by year
- 2026: 851 · 2025: 1481 · 2024: 1405 · 2023: 2347 · 2022: 1686 · 2021: 872 · 2020: 94 · 2019: 2 · 2013: 1 · 2012: 2
- FWCI
- 477.50
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- transformer models, natural language processing, Hugging Face
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
