DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter
Sanh, Victor, Debut, Lysandre, Chaumond, Julien, Wolf, Thomas·2019·arXiv
Cited by (4,692)
Items 301–400 of 4,692
- Hate Speech Detection in Twitter using Transformer MethodsRaymond T Mutanga, Nalindren Naicker, O Oludayo · 2020 · International Journal of Advanced Computer Science and Applications
- Discriminative Nearest Neighbor Few-Shot Intent Detection by Transferring Natural Language InferenceJianguo Zhang, Kazuma Hashimoto, Wenhao Liu, Chien-Sheng Wu, Yao Wan, Philip S. Yu, Richard Socher, Caiming Xiong · 2020 · Conference on Empirical Methods in Natural Language Processing (EMNLP)
- Toward Text Data Augmentation for Sentiment AnalysisHugo Abonizio, Emerson Cabrera Paraíso, Sylvio Barbon · 2021 · IEEE Transactions on Artificial Intelligence
- Language-Driven Representation Learning for RoboticsSiddharth Karamcheti, Suraj Nair, Annie Chen, Thomas Kollar, Chelsea Finn, Dorsa Sadigh, Percy Liang · 2023 · Robotics: Science and Systems
- Structured Pruning of a BERT-based Question Answering ModelMcCarley, J. S., Rishav Chakravarti, Avirup Sil · 2019 · arXiv (Cornell University)
- Applying and Understanding an Advanced, Novel Deep Learning Approach: A Covid 19, Text Based, Emotions Analysis StudyJyoti Choudrie, Shruti Patil, Ketan V. Kotecha, Nikhil Matta, Ilias O. Pappas · 2021 · Information Systems Frontiers
- The AI‐IP: Minimizing the guesswork of personality scale item development through artificial intelligenceIvan Hernandez, Weiwen Nie · 2022 · Personnel Psychology
- Investigating Evasive Techniques in SMS Spam Filtering: A Comparative Analysis of Machine Learning ModelsMuhammad Salman, Muhammad Ikram, Mohamed Ali Kâafar · 2024 · IEEE Access
- Antisocial online behavior detection using deep learningElizaveta Zinovyeva, Wolfgang Karl Härdle, Stefan Lessmann · 2020 · Decision Support Systems
- Stance detection with BERT embeddings for credibility analysis of information on social mediaHema Karande, Rahee A. Walambe, Victor Benjamin, Ketan V. Kotecha, T. S. Raghu · 2021 · PeerJ Computer Science
- Towards Interpreting and Mitigating Shortcut Learning Behavior of NLU modelsMengnan Du, Varun Manjunatha, Rajiv Jain, Ruchi Deshpande, Franck Dernoncourt, Jiuxiang Gu, Tong Sun, Xia Hu · 2021 · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT)
- Towards Lightweight Transformer Via Group-Wise Transformation for Vision-and-Language TasksGen Luo, Yiyi Zhou, Xiaoshuai Sun, Yan Wang, Liujuan Cao, Yongjian Wu, Feiyue Huang, Rongrong Ji · 2022 · IEEE Transactions on Image Processing
- Prompt–RSVQA: Prompting visual context to a language model for Remote Sensing Visual Question AnsweringChristel Chappuis, Valérie Zermatten, Sylvain Lobry, Bertrand Le Saux, Devis Tuia · 2022 · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)
- CySecBERT : A Domain-Adapted Language Model for the Cybersecurity DomainMarkus Bayer, Philipp Kuehn, Ramin Shanehsaz, Christian Reuter · 2024 · ACM Transactions on Privacy and Security
- RenAIssance: A Survey Into AI Text-to-Image Generation in the Era of Large ModelFengxiang Bie, Yibo Yang, Zhongzhu Zhou, Adam Ghanem, Minjia Zhang, Zhewei Yao, Xiaoxia Wu, Connor Holmes, Pareesa Ameneh Golnari, David Andrew Clifton et al. · 2024 · IEEE Transactions on Pattern Analysis and Machine Intelligence
- Question answering method for infrastructure damage information retrieval from textual data using bidirectional encoder representations from transformersYohan Kim, Seongdeok Bang, Jiu Sohn, Hyoungkwan Kim · 2021 · Automation in Construction
- Fantastic Questions and Where to Find Them: FairytaleQA – An Authentic Dataset for Narrative ComprehensionYing Xu, Dakuo Wang, Mo Yu, Daniel Ritchie, Bingsheng Yao, Tongshuang Wu, Zheng Zhang, Toby Jia-Jun Li, Nora Bradford, Branda Sun et al. · 2022 · Proceedings of the conference - Association for Computational Linguistics. Meeting
- Socially Aware Synthetic Data Generation for Suicidal Ideation Detection Using Large Language ModelsHamideh Ghanadian, Isar Nejadgholi, Hussein Al Osman · 2024 · IEEE Access
- Generating Plausible Counterfactual Explanations for Deep Transformers in Financial Text ClassificationLinyi Yang, Eoin M. Kenny, Tin Lok James Ng, Yi Yang, Barry Smyth, Ruihai Dong · 2020 · International Conference on Computational Linguistics
- Unmasking Contextual Stereotypes: Measuring and Mitigating BERT's Gender BiasBartl, Marion, Nissim, Malvina, Gatt, Albert · 2020 · arXiv
- Exploring Transformers in Natural Language Generation: GPT, BERT, and XLNetM. Onat Topal, Anil Bas, Imke van Heerden · 2021 · arXiv (Cornell University)
- Learning an Unreferenced Metric for Online Dialogue EvaluationKoustuv Sinha, Prasanna Parthasarathi, Jasmine Wang, Ryan Lowe, William L. Hamilton, Joëlle Pineau · 2020 · Proceedings of the conference - Association for Computational Linguistics. Meeting
- Transformers: “The End of History” for Natural Language Processing?Anton Chernyavskiy, Dmitry Ilvovsky, Preslav Nakov · 2021 · Lecture notes in computer science
- Rethinking attention mechanism in time series classificationBowen Zhao, Huanlai Xing, Xinhan Wang, Fuhong Song, Zhiwen Xiao · 2023 · Information Sciences
- Enhancing large language model capabilities for rumor detection with Knowledge-Powered PromptingYeqing Yan, Peng Zheng, Yongjun Wang · 2024 · Engineering Applications of Artificial Intelligence
- Improving BERT Fine-Tuning via Self-Ensemble and Self-DistillationYige Xu, Xipeng Qiu, Li-Gao Zhou, Xuanjing Huang · 2023 · Journal of Computer Science and Technology
- YolactEdge: Real-time Instance Segmentation on the EdgeLiu, Haotian, Rivera Soto, Rafael A., Xiao, Fanyi, Jae Lee, Yong · 2021 · 2021 IEEE International Conference on Robotics and Automation (ICRA)
- Masking as an Efficient Alternative to Finetuning for Pretrained Language ModelsMengjie Zhao, Tao Lin, Fei Mi, Martin Jaggi, Hinrich Schütze · 2020 · Conference on Empirical Methods in Natural Language Processing (EMNLP)
- Automatic classification of scanned electronic health record documentsHeath Goodrum, Kirk E Roberts, Elmer Victor Bernstam · 2020 · International Journal of Medical Informatics
- Deep Entity MatchingYuliang Li, Jinfeng Li, Yoshihiko Suhara, Jin Wang, Wataru Hirota, Wang-Chiew Tan · 2021 · Journal of Data and Information Quality
- RotomZhengjie Miao, Yuliang Li, Xiaolan Wang · 2021 · ACM SIGMOD International Conference on Management of Data (SIGMOD)
- Aspect-Controllable Opinion SummarizationReinald Kim Amplayo, Stefanos Angelidis, Mirella Lapata · 2021 · Conference on Empirical Methods in Natural Language Processing (EMNLP)
- Beyond word embeddings: A surveyFrancesca Incitti, Federico Urli, Lauro Snidaro · 2022 · Information Fusion
- Machine Learning Driven Mental Stress Detection on Reddit Posts Using Natural Language ProcessingShaunak Inamdar, Rishikesh Chapekar, Shilpa Gite, Biswajeet Pradhan · 2023 · Human-Centric Intelligent Systems
- DeepJoin: Joinable Table Discovery with Pre-Trained Language ModelsYuyang Dong, Chuan Xiao, Takuma Nozawa, Masafumi Enomoto, Masafumi Oyamada · 2023 · Proceedings of the VLDB Endowment
- PointDistiller: Structured Knowledge Distillation Towards Efficient and Compact 3D DetectionLinfeng Zhang, Runpei Dong, Hung-Shuo Tai, Kaisheng Ma · 2023 · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings
- Harnessing the Power of Hugging Face Transformers for Predicting Mental Health Disorders in Social NetworksAlireza Pourkeyvan, Ramin Safa, Ali Sorourkhah · 2024 · IEEE Access
- A hybrid machine learning model for classifying gene mutations in cancer using LSTM, BiLSTM, CNN, GRU, and GloVeSanad Aburass, Osama M. Dorgham, Jamil Al Shaqsi · 2024 · Systems and Soft Computing
- Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender BiasJesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Qian, Sharon, Daniel Nevo, Sakenis, Simas, Huang, Jason, Singer, Yaron, Stuart M. Shieber · 2020 · arXiv (Cornell University)
- Assessing Phrasal Representation and Composition in TransformersYu, Lang, Ettinger, Allyson · 2020 · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)
- A review of LLMs and their applications in the architecture, engineering and construction industryKampelopoulos, Dimitrios, Tsanousa, Athina, Vrochidis, Stefanos, Kompatsiaris, Ioannis · 2025 · Artificial Intelligence Review
- UNQOVERing Stereotyping Biases via Underspecified QuestionsTao Li, Daniel Khashabi, Tushar Khot, Ashish Sabharwal, Vivek Srikumar · 2020 · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP)
- Contrastive Distillation on Intermediate Representations for Language Model CompressionSiqi Sun, Zhe Gan, Yuwei Fang, Yu Cheng, Shuohang Wang, Jingjing Liu · 2020 · Conference on Empirical Methods in Natural Language Processing (EMNLP)
- Findings of the Association for Computational Linguistics: EMNLP 2020Stefan Dumitrescu, Andrei-Marius Avram, Sampo Pyysalo · 2022 · UTUPub (University of Turku)
- Great Power, Great Responsibility: Recommendations for Reducing Energy for Training Language ModelsMcDonald, Joseph, Li, Baolin, Frey, Nathan, Tiwari, Devesh, Gadepally, Vijay, Samsi, Siddharth · 2022 · Findings of the Association for Computational Linguistics: NAACL 2022
- Deep Unlearning via Randomized Conditionally Independent HessiansRonak Mehta, Sourav Pal, Vikas Pratap Singh, Sathya N. Ravi · 2022 · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings
- A Survey on Model Compression and Acceleration for Pretrained Language ModelsCanwen Xu, Julian McAuley · 2023 · Proceedings of the AAAI Conference on Artificial Intelligence
- Hate-CLIPper: Multimodal Hateful Meme Classification based on Cross-modal Interaction of CLIP FeaturesGokul Karthik Kumar, Karthik Nandakumar · 2022 · Workshop on NLP for Positive Impact (NLP4PI)
- ENVQA: Improving Visual Question Answering model by enriching the visual featureSouvik Chowdhury, Badal Soni · 2025 · Engineering Applications of Artificial Intelligence
- FairFil: Contrastive Neural Debiasing Method for Pretrained Text EncodersPengyu Cheng, Weituo Hao, Siyang Yuan, Shijing Si, Lawrence Carin · 2021 · arXiv (Cornell University)
- On the Language Neutrality of Pre-trained Multilingual RepresentationsJindřich Libovický, Rudolf Rosa, Alexander Fraser · 2020 · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP)
- Real-Time Text Steganalysis Based on Multi-Stage Transfer LearningWanli Peng, Jinyu Zhang, Yiming Xue, Zhenghong Yang · 2021 · IEEE Signal Processing Letters
- Analyzing Social Media to Explore the Attitudes and Behaviors Following the Announcement of Successful COVID-19 Vaccine Trials: Infodemiology StudyJean-Christophe Boucher, Kirsten Cornelson, Jamie Laura Benham, Madison M. Fullerton, Theresa Tang, Cora Constantinescu, Mehdi Mourali, Robert J. Oxoby, Deborah Anne Marshall, Hadi Hemmati et al. · 2021 · JMIR Infodemiology
- Rationalization for explainable NLP: a surveySai Gurrapu, Ajay Kulkarni, Lifu Huang, Ismini Lourentzou, Feras A. Batarseh · 2023 · Frontiers in Artificial Intelligence
- Exploring transformer models for sentiment classification: A comparison of BERT , RoBERTa , ALBERT , DistilBERT , and XLNetAli Areshey, Hassan I. Mathkour · 2024 · Expert Systems
- Analyzing Redundancy in Pretrained Transformer ModelsFahim Dalvi, Hassan Sajjad, Nadir Durrani, Yonatan Belinkov · 2020 · Conference on Empirical Methods in Natural Language Processing (EMNLP)
- Pre-trained Summarization DistillationSam Shleifer, Alexander M. Rush · 2020 · arXiv (Cornell University)
- Building a Bridge: A Method for Image-Text Sarcasm Detection Without Pretraining on Image-Text DataXinyu Wang, Xiaowen Sun, Tan Yang, Hongbo Wang · 2020
- Pre-trained transformers: an empirical comparisonSilvia Casola, Ivano Lauriola, Alberto Lavelli · 2022 · Machine Learning with Applications
- Constructing and analyzing domain-specific language model for financial text miningMasahiro Suzuki, Hiroki Sakaji, Masanori Hirano, Kiyoshi Izumi · 2022 · Information Processing & Management
- A Comparative Survey of Instance Selection Methods applied to Non-Neural and Transformer-Based Text ClassificationWashington Luiz Miranda da Cunha, Felipe Augusto Resende Viegas, Celso França, Thierson Couto Rosa, Leonardo Chaves Dutra da Rocha, Marcos André Gonçalves · 2023 · ACM Computing Surveys
- A 95.6-TOPS/W Deep Learning Inference Accelerator With Per-Vector Scaled 4-bit Quantization in 5 nmBen Keller, Rangharajan Venkatesan, Steve X. Dai, Stephen G. Tell, Brian Zimmer, Charbel Sakr, W.J. Dally, C. Thomas Gray, Brucek Khailany · 2023 · IEEE Journal of Solid-State Circuits
- Analyzing the Performance of Sentiment Analysis using BERT, DistilBERT, and RoBERTaArcha Joshy, Sumod Sundar · 2022 · IEEE International Power and Renewable Energy Conference (IPRECON)
- Discop: Provably Secure Steganography in Practice Based on "Distribution Copies"Jinyang Ding, Kejiang Chen, Yaofei Wang, Na Zhao, Weiming Zhang, Nenghai Yu · 2023 · IEEE Symposium on Security and Privacy (S&P)
- Current limitations in cyberbullying detection: On evaluation criteria, reproducibility, and data scarcityChris Emmery, Ben Verhoeven, Guy De Pauw, Gilles Jacobs, Cynthia Van Hee, Els Lefever, Bart Desmet, Véronique Hoste, Walter M. P. Daelemans · 2020 · Language Resources and Evaluation
- NewsMTSC: A Dataset for (Multi-)Target-dependent Sentiment Classification in Political News ArticlesFelix Hamborg, Karsten Donnay · 2021 · Conference of the European Chapter of the Association for Computational Linguistics
- Automatically Matching Bug Reports With Related App ReviewsMarlo Häring, Christoph Stanik, Walid Maalej · 2021 · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion)
- Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with SearchKim, Gyuwan, Cho, Kyunghyun · 2021 · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)
- TAG: Gradient Attack on Transformer-based Language ModelsJieren Deng, Yijue Wang, Li Ji · 2021 · Conference on Empirical Methods in Natural Language Processing
- Structured Knowledge Distillation for Accurate and Efficient Object DetectionLinfeng Zhang, Kaisheng Ma · 2023 · IEEE Transactions on Pattern Analysis and Machine Intelligence
- A Survey of Active Learning for Text Classification using Deep Neural NetworksChristopher Schröder, Andreas Niekler · 2020 · arXiv (Cornell University)
- MalBERT: Malware Detection using Bidirectional Encoder Representations from TransformersAbir Rahali, Moulay A. Akhloufi · 2021 · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics
- Measuring Fairness with Biased Rulers: A Comparative Study on Bias Metrics for Pre-trained Language ModelsDelobelle, Pieter, Tokpo, Ewoenam, Calders, Toon, Berendt, Bettina · 2022 · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
- NLX-GPT: A Model for Natural Language Explanations in Vision and Vision-Language TasksFawaz Sammani, Tanmoy Mukherjee, Nikos Deligiannis · 2022 · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings
- TCM-GPT: Efficient pre-training of large language models for domain adaptation in Traditional Chinese MedicineGuoxing Yang, Xiaohong Liu, Jian‐Yu Shi, Zan Wang, Guangyu Wang · 2024 · Computer Methods and Programs in Biomedicine Update
- Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of TransformersZhuohan Li, Eric Wallace, Sheng Shen, Kevin Lin, Kurt Keutzer, Dan Klein, Joseph E. Gonzalez · 2020 · arXiv (Cornell University)
- Isotropy in the Contextual Embedding Space: Clusters and ManifoldsXingyu Cai, Jiaji Huang, Yuchen Bian, Kenneth Church · 2021 · International Conference on Learning Representations
- Comparative Analysis of Different Transformer Based Architectures Used in Sentiment AnalysisKeval Pipalia, Rahul Bhadja, Madhu Shukla · 2020
- A Review of Text Style Transfer Using Deep LearningMartina Toshevska, Sonja Gievska · 2021 · IEEE Transactions on Artificial Intelligence
- Revisiting Uncertainty-based Query Strategies for Active Learning with TransformersChristopher Schröder, Andreas Niekler, Martin Potthast · 2022 · Findings of the Association for Computational Linguistics: ACL (Findings of ACL)
- No, to the RightYuchen Cui, Siddharth Karamcheti, Raj Palleti, Nidhya Shivakumar, Percy Liang, Dorsa Sadigh · 2023 · ACM/IEEE International Conference on Human-Robot Interaction (HRI)
- Common vulnerability scoring system prediction based on open source intelligence information sourcesPhilipp Kühn, David N. Relke, Christian Reuter · 2023 · Computers & Security
- Challenges and Issues in Sentiment Analysis: A Comprehensive SurveyNilaa Raghunathan, Saravanakumar Kandasamy · 2023 · IEEE Access
- Improving measurement and prediction in personnel selection through the application of machine learningNick Koenig, Scott Tonidandel, Isaac Benjamin Thompson, Betsy H. Albritton, Farshad Koohifar, Georgi P. Yankov, Andrew B. Speer, Jay H. Hardy, Carter Gibson, Chris Frost et al. · 2023 · Personnel Psychology
- Fairness in Deep Learning: A Survey on Vision and Language ResearchOtávio Parraga, Martin D. Móre, Christian Mattjie, Nathan Schneider Gavenski, Lucas Silveira Kupssinskü, Adilson Medronha, Luís Vinícius de Moura, Gabriel S. Simões, Rodrigo Coelho Barros · 2023 · ACM Computing Surveys
- Revisiting Mahalanobis Distance for Transformer-Based Out-of-Domain DetectionAlexander Podolskiy, Dmitry Lipin, Andrey Bout, Ekaterina Artemova, Irina Piontkovskaya · 2021 · Proceedings of the AAAI Conference on Artificial Intelligence
- Transformer Based Language Identification for Malayalam-English Code-Mixed TextS Thara, Prabaharan Poornachandran · 2021 · IEEE Access
- aeroBERT-Classifier: Classification of Aerospace Requirements Using BERTArchana Tikayat Ray, Bjorn F. Cole, Olivia J. Pinon Fischer, Ryan T. White, Dimitri N. Mavris · 2023 · Aerospace
- ByteTransformer: A High-Performance Transformer Boosted for Variable-Length InputsYujia Zhai, Chengquan Jiang, Leyuan Wang, Xiaoying Jia, Shang Zhang, Zizhong Chen, Xin Liu, Yibo Zhu · 2023 · IEEE International Parallel and Distributed Processing Symposium (IPDPS)
- Large Scale Legal Text Classification Using Transformer ModelsZein Shaheen, Gerhard Wohlgenannt, Erwin Filtz · 2020 · arXiv (Cornell University)
- Session-based cyberbullying detection in social media: A surveyYi, Peiling, Zubiaga, Arkaitz · 2023 · Online Social Networks and Media
- Accelerating Framework of Transformer by Hardware Design and Model Compression Co-OptimizationPanjie Qi, Edwin H.‐M. Sha, Qingfeng Zhuge, Hongwu Peng, Shaoyi Huang, Zhenglun Kong, Yuhong Song, Bingbing Li · 2021 · International Conference on Computer Aided Design
- Using Bidirectional Encoder Representations from Transformers (BERT) to classify traffic crash severity typesAmir Hossein Oliaee, Subasish Das, Jinli Liu, M. Ashifur Rahman · 2023 · Natural Language Processing Journal
- LRC-BERT: Latent-representation Contrastive Knowledge Distillation for Natural Language UnderstandingHao Fu, Shaojun Zhou, Qihong Yang, Junjie Tang, Guiquan Liu, Kaikui Liu, Xiaolong Li · 2021 · Proceedings of the AAAI Conference on Artificial Intelligence
- Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced AnnotationSuraj Nair, Eric Mitchell, Kevin Chen, Brian Ichter, Silvio Savarese, Chelsea Finn · 2021 · arXiv (Cornell University)
- PADA: Example-based Prompt Learning for on-the-fly Adaptation to
Unseen DomainsBen-David, Eyal, Oved, Nadav, Reichart, Roi · 2022 · Transactions of the Association for Computational Linguistics
- Causal Modeling of Twitter Activity during COVID-19Oguzhan Gencoglu, Mathias Gruber · 2020 · Computation
- Don’t Patronize Me! An Annotated Dataset with Patronizing and Condescending Language towards Vulnerable CommunitiesCarla Perez-Almendros, Luis Espinosa-Anke, Steven Schockaert · 2020 · International Conference on Computational Linguistics
- Perspectives and Prospects on Transformer Architecture for Cross-Modal Tasks with Language and VisionAndrew Shin, Masato Ishii, Takuya Narihira · 2022 · International Journal of Computer Vision
- A BERT-based transfer learning approach to text classification on software requirements specificationsDerya Kıcı, Garima Malik, Mücahit Çevik, Devang Parikh, Ayşe Başar · 2021