RoBERTa: A Robustly Optimized BERT Pretraining Approach

Liu, Yinhan, Ott, Myle, Goyal, Naman, Du, Jingfei, Joshi, Mandar, Chen, Danqi, Levy, Omer, Lewis, Mike +2 more·2019·arXiv
DOI 10.48550/arxiv.1907.11692

Cited by (1,276)

Items 1,201–1,276 of 1,276

  1. Are Emergent Abilities in Large Language Models just In-Context Learning?
    Lu, Sheng, Bigoulaeva, Irina, Sachdeva, Rachneet, Madabushi, Harish Tayyar, Gurevych, Iryna · 2023 · arXiv
  2. Deanthropomorphising NLP: Can a language model be conscious?
    Shardlow, Matthew, Przybyła, Piotr · 2024 · PLoS ONE
  3. The Topic Confusion Task: A Novel Evaluation Scenario for Authorship Attribution
    Altakrori, Malik, Cheung, Jackie Chi Kit, Fung, Benjamin C. M. · 2021 · Findings of the Association for Computational Linguistics: EMNLP 2021
  4. Skill but not Effort Drive GPT Overperformance over Humans in Cognitive Reframing of Negative Scenarios
    Li, Joanna Z., Herderich, Alina, Goldenberg, Amit · 2024
  5. Universality of representation in biological and artificial neural networks
    Hosseini, Eghbal, Casto, Colton, Zaslavsky, Noga, Conwell, Colin, Richardson, Mark, Fedorenko, Evelina · 2024
  6. Adversarial NLI for Factual Correctness in Text Summarisation Models
    Barrantes, Mario, Herudek, Benedikt, Wang, Richard · 2020 · arXiv
  7. Learning Interpretable Style Embeddings via Prompting LLMs
    Patel, Ajay, Rao, Delip, Kothary, Ansh, McKeown, Kathleen, Callison-Burch, Chris · 2023 · arXiv
  8. GOBO: Quantizing Attention-Based NLP Models for Low Latency and Energy Efficient Inference
    Zadeh, Ali Hadi, Edo, Isak, Awad, Omar Mohamed, Moshovos, Andreas · 2020 · 2020 53rd Annual IEEE/ACM International Symposium on Microarchitecture (MICRO)
  9. Bigger is not always better: The importance of human-scale language modeling for psycholinguistics
    Wilcox, Ethan Gotlieb, Hu, Michael Y., Mueller, Aaron, Warstadt, Alex, Choshen, Leshem, Zhuang, Chengxu, Williams, Adina, Cotterell, Ryan, Linzen, Tal · 2025 · Journal of Memory and Language
  10. What BERT Sees: Cross-Modal Transfer for Visual Question Generation
    Scialom, Thomas, Bordes, Patrick, Dray, Paul-Alexis, Staiano, Jacopo, Gallinari, Patrick · 2020 · Proceedings of the 13th International Conference on Natural Language Generation
  11. On the Robustness of Dialogue History Representation in Conversational Question Answering: A Comprehensive Study and a New Prompt-based Method
    Gekhman, Zorik, Oved, Nadav, Keller, Orgad, Szpektor, Idan, Reichart, Roi · 2023 · Transactions of the Association for Computational Linguistics
  12. Enhancing the Robustness of AI-Generated Text Detectors: A Survey
    Liu, Xin, Li, Yang, Li, Kan · 2025 · Mathematics
  13. Effective Long-Context Scaling of Foundation Models
    Xiong, Wenhan, Liu, Jingyu, Molybog, Igor, Zhang, Hejia, Bhargava, Prajjwal, Hou, Rui, Martin, Louis, Rungta, Rashi, Sankararaman, Karthik Abinav, Oguz, Barlas et al. · 2023 · arXiv
  14. Are Small Language Models Ready to Compete with Large Language Models for Practical Applications?
    Sinha, Neelabh, Jain, Vinija, Chadha, Aman · 2025 · Proceedings of the 5th Workshop on Trustworthy NLP (TrustNLP 2025)
  15. Multimodal Multi-Speaker Merger & Acquisition Financial Modeling: A New Task, Dataset, and Neural Baselines
    Sawhney, Ramit, Goyal, Mihir, Goel, Prakhar, Mathur, Puneet, Shah, Rajiv Ratn · 2021 · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)
  16. Students' argumentation performance in online learning environments: Bridging culture and gender
    Ranjbaran, Fatemeh, Babaee, Maryam, Parvaneh Akhteh Khaneh, Marzieh, Gohari, Mehrnaz, Daneshvar Ghorbani, Babak, Taghizadeh Kerman, Nafiseh, Banihashem, Seyyed Kazem, Noroozi, Omid · 2023 · International Journal of Technology in Education
  17. Detecting AI-Generated Sentences in Human-AI Collaborative Hybrid Texts: Challenges, Strategies, and Insights
    Zeng, Zijie, Liu, Shiqi, Sha, Lele, Li, Zhuang, Yang, Kaixun, Liu, Sannyuya, Gasevic, Dragan, Chen, Guangliang · 2024 · Proceedings of the Thirty-ThirdInternational Joint Conference on Artificial Intelligence
  18. Deciphering Stereotypes in Pre-Trained Language Models
    Ma, Weicheng, Scheible, Henry, Wang, Brian, Veeramachaneni, Goutham, Chowdhary, Pratim, Sun, Alan, Koulogeorge, Andrew, Wang, Lili, Yang, Diyi, Vosoughi, Soroush · 2023 · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing
  19. A Transformer-Based Approach for Smart Invocation of Automatic Code Completion
    de Moor, Aral, van Deursen, Arie, Izadi, Maliheh · 2024 · Proceedings of the 1st ACM International Conference on AI-Powered Software
  20. AI vs. Human Writing: Developing a Novel Method for Text Authenticity Detection in Education
    Department of Computer Science and Engineering, Rajarambapu Institute of Technology, Rajaramnagar, Affiliated to Shivaji University, Kolhapur, Maharashtra – 415414, India, Kalmani, Vijay H., Adamuthe, Amol C., Premnath Gondil, Arati, Prashant Patil, Vaishnavi, Amar Kore, Riya, Mahadev Metkari, Vaishnavi · 2025 · International Journal of Modern Education and Computer Science
  21. Detection of artificial intelligence-generated essays for academic assessment integrity using large language models
    Qorich, Mohammed, El Ouazzani, Rajae · 2025 · Expert Systems with Applications
  22. Guiding Clinical Reasoning with Large Language Models via Knowledge Seeds
    Wu, Jiageng, Wu, Xian, Yang, Jie · 2024 · Proceedings of the Thirty-ThirdInternational Joint Conference on Artificial Intelligence
  23. Limitations in Employing Natural Language Supervision for Sensor-Based Human Activity Recognition - And Ways to Overcome Them
    Haresamudram, Harish, Beedu, Apoorva, Rabbi, Mashfiqui, Saha, Sankalita, Essa, Irfan, Ploetz, Thomas · 2025 · Proceedings of the AAAI Conference on Artificial Intelligence
  24. Review of Recent Trends in the Detection of Hate Speech and Offensive Language on Social Media
    Sokolová, Zuzana, Staš, Ján, Juhár, Jozef · 2022 · Acta Electrotechnica et Informatica
  25. Analyzing Cyberbullying Negative Content on Twitter Social Media with the RoBERTa Method
    Yani, Muh. Akib A., Maharani, Warih · 2023 · JINAV: Journal of Information and Visualization
  26. Do Fine-tuned Commonsense Language Models Really Generalize?
    Kejriwal, Mayank, Shen, Ke · 2020 · arXiv
  27. The power of specific emotion analysis in predicting donations: A comparative empirical study between sentiment and specific emotion analysis in social media
    Lee, Sanghyub John, Paas, Leo, Ahn, Ho Seok · 2024 · International Journal of Market Research
  28. Tutor CoPilot: A Human-AI Approach for Scaling Real-Time Expertise
    Wang, Rose E., Ribeiro, Ana T., Robinson, Carly D., Loeb, Susanna, Demszky, Dora · 2024 · arXiv
  29. ChatGPT and Deepseek: Can They Predict the Stock Market and Macroeconomy?
    Chen, Jian, Tang, Guohao, Zhou, Guofu, Zhu, Wu · 2025 · arXiv
  30. A Feasibility and Implementation Integrity Study of the Community Builder (CoBi): An AI-based Collaboration Support System in K-12 Classrooms
    Breideband, Thomas, Bush, Jeffrey B., Reitman, Jason G., Rose, Sierra, Weatherley, John, Penuel, William R., D’Mello, Sidney K. · 2025 · International Journal of Artificial Intelligence in Education
  31. How Context Affects Language Models' Factual Predictions
    Fabio Petroni, Patrick Lewis · 2020 · Automated Knowledge Base Construction (AKBC)
  32. SummQA at MEDIQA-Chat 2023:In-Context Learning with GPT-4 for Medical Summarization
    Mathur, Yash, Rangreji, Sanketh, Kapoor, Raghav, Palavalli, Medha, Bertsch, Amanda, Gormley, Matthew R. · 2023 · arXiv
  33. Safetywashing: Do AI Safety Benchmarks Actually Measure Safety Progress?
    Ren, Richard, Basart, Steven, Khoja, Adam, Gatti, Alice, Phan, Long, Yin, Xuwang, Mazeika, Mantas, Pan, Alexander, Mukobi, Gabriel, Kim, Ryan H. et al. · 2024 · arXiv
  34. Generative AI-Powered Educational Alignment: A Framework for Matching Syllabus Course Topics with Web Description
    Liu, Liyuan, Mendoza, Ruben A., Martin, Thomas R., Miori, Virginia M. · 2024 · Proceedings of the 2024 9th International Conference on Distance Education and Learning
  35. Dermacen Analytica: A novel methodology integrating multi-modal large language models with machine learning in tele-dermatology
    Panagoulias, Dimitrios P., Tsoureli-Nikita, Evridiki, Virvou, Maria, Tsihrintzis, George A. · 2024 · arXiv
  36. Human-inspired Perspectives: A Survey on AI Long-term Memory
    He, Zihong, Lin, Weizhe, Zheng, Hao, Zhang, Fan, Jones, Matt W., Aitchison, Laurence, Xu, Xuhai, Liu, Miao, Kristensson, Per Ola, Shen, Junxiao · 2024 · arXiv
  37. Cost-sensitive PLM-based Approach for Arabic and English Cyberbullying Classification
    Harbaoui, Azza, Benaissa, Azzeddine Rachid, ghezala, Henda Ben · 2023
  38. Large Language Models Are Better Logical Fallacy Reasoners with Counterargument, Explanation, and Goal-Aware Prompt Formulation
    Jeong, Jiwon, Jang, Hyeju, Park, Hogun · 2025 · Findings of the Association for Computational Linguistics: NAACL 2025
  39. PeerQA: A Scientific Question Answering Dataset from Peer Reviews
    Baumgärtner, Tim, Briscoe, Ted, Gurevych, Iryna · 2025 · Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)
  40. Adversarial Paraphrasing: A Universal Attack for Humanizing AI-Generated Text
    Cheng, Yize, Sadasivan, Vinu Sankar, Saberi, Mehrdad, Saha, Shoumik, Feizi, Soheil · 2025 · Advances in Neural Information Processing Systems 38
  41. A Fuzzy Logic Framework for Text-Based Incident Prioritization: Mathematical Modeling and Case Study Evaluation
    Peralta, Arturo, Olivas, José A., Navarro-Illana, Pedro · 2025 · Mathematics
  42. Automatic Prompt Augmentation and Selection with Chain-of-Thought from Labeled Data
    Shum, KaShun, Diao, Shizhe, Zhang, Tong · 2023 · arXiv
  43. Single Sequence Prediction over Reasoning Graphs for Multi-hop QA
    Ramesh, Gowtham, Sreedhar, Makesh Narsimhan, Hu, Junjie · 2023 · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
  44. Extracting the Sample Size From Randomized Controlled Trials in Explainable Fashion Using Natural Language Processing
    Windisch, Paul, Dennstädt, Fabio, Koechli, Carole, Förster, Robert, Schröder, Christina, Aebersold, Daniel M., Zwahlen, Daniel R. · 2024
  45. Emotion Classification With Visibility Graphs
    Şimşek, Ecem, Topcu, Atakan, Koç, Emirhan, Saritas, Emine Ulku, Koç, Aykut · 2025 · IEEE Signal Processing Letters
  46. CLIMB – Curriculum Learning for Infant-inspired Model Building
    Martinez, Richard Diehl, McGovern, Hope, Goriely, Zebulon, Davis, Christopher, Caines, Andrew, Buttery, Paula, Beinborn, Lisa · 2023 · Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning
  47. Programming Puzzles
    Schuster, Tal, Kalyan, Ashwin, Polozov, Oleksandr, Kalai, Adam Tauman · 2021 · arXiv
  48. Generative Agents Navigating Digital Libraries
    Zerhoudi, Saber, Granitzer, Michael · 2025 · Lecture notes in computer science
  49. Criticmate: Stagewise Human–AI Co-Critique in Single-Screen UI Evaluation
    Ko, Jisu, Choi, Jinyoung, Morales, Cielo, Kim, Dajung, Ko, Minsam · 2026 · Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems
  50. ToddlerBERTa: Exploiting BabyBERTa for Grammar Learning and Language Understanding
    Veysel Çağatan, Ömer · 2023 · Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning
  51. Too Much Information: Keeping Training Simple for BabyLMs
    Edman, Lukas, Bylinina, Lisa · 2023 · Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning
  52. Peerify: Benchmarking Peer-Review Claim Verification
    Daghighfarsoodeh, Alireza, Ebrahimi, Sajad, Ghorbanpour, Ali, Sadeghian, Soroush, Cheraghi, Radin, Arabzadeh, Negar, Bagheri, Ebrahim · 2026 · arXiv
  53. CoCoReviewBench: A Completeness- and Correctness-Oriented Benchmark for AI Reviewers
    Deng, Hexuan, Ke, Xiaopeng, Li, Yichen, Hu, Ruina, Huang, Dehao, Wong, Derek F., Wang, Yue, Liu, Xuebo, Zhang, Min · 2026 · arXiv
  54. PeerCheck: Enhancing LLM-Generated Academic Reviews Towards Human-Level Quality
    Chen, Zeyuan, Yang, Ziqing, Ma, Yihan, Backes, Michael, Zhang, Yang · 2026 · Findings of the Association for Computational Linguistics: ACL 2026
  55. PatchWorld: Gradient-Free Optimization of Executable World Models for Agent Environments
    Bai, Jiaxin, Guo, Yue, Dong, Yifei, Xiong, Jiaxuan, Zheng, Tianshi, Li, Yixia, Fang, Tianqing, Li, Yufei, Gao, Yisen, Huang, Haoyu et al. · 2026 · arXiv
  56. A Transformer-Based Approach for Smart Invocation of Automatic Code Completion
    de Moor, Aral, van Deursen, Arie, Izadi, Maliheh · 2024 · arXiv
  57. The Llama 3 Herd of Models
    Grattafiori, Aaron, Dubey, Abhimanyu, Jauhri, Abhinav, Pandey, Abhinav, Kadian, Abhishek, Al-Dahle, Ahmad, Letman, Aiesha, Mathur, Akhil, Schelten, Alan, Vaughan, Alex et al. · 2024 · arXiv
  58. Attacking Neural Text Detectors
    Wolff, Max, Wolff, Stuart · 2020 · arXiv
  59. Machine Generated Text: A Comprehensive Survey of Threat Models and Detection Methods
    Crothers, Evan, Japkowicz, Nathalie, Viktor, Herna · 2022 · arXiv
  60. Language Models for Code Completion: A Practical Evaluation
    Izadi, Maliheh, Katzy, Jonathan, van Dam, Tim, Otten, Marc, Popescu, Razvan Mihai, van Deursen, Arie · 2024 · arXiv
  61. Language Models are Few-Shot Learners
    Brown, Tom B., Mann, Benjamin, Ryder, Nick, Subbiah, Melanie, Kaplan, Jared, Dhariwal, Prafulla, Neelakantan, Arvind, Shyam, Pranav, Sastry, Girish, Askell, Amanda et al. · 2020 · arXiv
  62. HuggingFace's Transformers: State-of-the-art Natural Language Processing
    Wolf, Thomas, Debut, Lysandre, Sanh, Victor, Chaumond, Julien, Delangue, Clement, Moi, Anthony, Cistac, Pierric, Rault, Tim, Louf, Rémi, Funtowicz, Morgan et al. · 2019 · arXiv
  63. BEiT: BERT Pre-Training of Image Transformers
    Bao, Hangbo, Dong, Li, Piao, Songhao, Wei, Furu · 2021 · arXiv
  64. Big Bird: Transformers for Longer Sequences
    Zaheer, Manzil, Guruganesh, Guru, Dubey, Avinava, Ainslie, Joshua, Alberti, Chris, Ontanon, Santiago, Pham, Philip, Ravula, Anirudh, Wang, Qifan, Yang, Li et al. · 2020 · arXiv
  65. Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
    Shoeybi, Mohammad, Patwary, Mostofa, Puri, Raul, LeGresley, Patrick, Casper, Jared, Catanzaro, Bryan · 2019 · arXiv
  66. REALM: Retrieval-Augmented Language Model Pre-Training
    Guu, Kelvin, Lee, Kenton, Tung, Zora, Pasupat, Panupong, Chang, Ming-Wei · 2020 · arXiv
  67. GraphCodeBERT: Pre-training Code Representations with Data Flow
    Guo, Daya, Ren, Shuo, Lu, Shuai, Feng, Zhangyin, Tang, Duyu, Liu, Shujie, Zhou, Long, Duan, Nan, Svyatkovskiy, Alexey, Fu, Shengyu et al. · 2020 · arXiv
  68. The Pile: An 800GB Dataset of Diverse Text for Language Modeling
    Gao, Leo, Biderman, Stella, Black, Sid, Golding, Laurence, Hoppe, Travis, Foster, Charles, Phang, Jason, He, Horace, Thite, Anish, Nabeshima, Noa et al. · 2021 · arXiv
  69. Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping
    Dodge, Jesse, Ilharco, Gabriel, Schwartz, Roy, Farhadi, Ali, Hajishirzi, Hannaneh, Smith, Noah · 2020 · arXiv
  70. Prefix-Tuning: Optimizing Continuous Prompts for Generation
    Li, Xiang Lisa, Liang, Percy · 2021 · arXiv
  71. On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselines
    Mosbach, Marius, Andriushchenko, Maksym, Klakow, Dietrich · 2020 · arXiv
  72. Evaluation of Text Generation: A Survey
    Celikyilmaz, Asli, Clark, Elizabeth, Gao, Jianfeng · 2020 · arXiv
  73. I-BERT: Integer-only BERT Quantization
    Kim, Sehoon, Gholami, Amir, Yao, Zhewei, Mahoney, Michael W., Keutzer, Kurt · 2021 · arXiv
  74. Training with Quantization Noise for Extreme Model Compression
    Fan, Angela, Stock, Pierre, Graham, Benjamin, Grave, Edouard, Gribonval, Remi, Jegou, Herve, Joulin, Armand · 2020 · arXiv
  75. Mind the Gap: Assessing Temporal Generalization in Neural Language Models
    Lazaridou, Angeliki, Kuncoro, Adhiguna, Gribovskaya, Elena, Agrawal, Devang, Liska, Adam, Terzi, Tayfun, Gimenez, Mai, d'Autume, Cyprien de Masson, Kocisky, Tomas, Ruder, Sebastian et al. · 2021 · arXiv
  76. Improving Code Autocompletion with Transfer Learning
    Zhou, Wen, Kim, Seohyun, Murali, Vijayaraghavan, Aye, Gareth Ari · 2021 · arXiv