CoChat
Sign Up
Cited by 18Adversarial Robustness in Machine LearningTopic ModelingMachine Learning in Healthcare

Improving the robustness and accuracy of biomedical language models through adversarial training

Open access
Moradi, Milad, Samwald, Matthias·2022·Journal of Biomedical Informatics
DOI 10.1016/j.jbi.2022.104114
Open this paper in CoChat — free accountOpen PDF

Abstract

Deep transformer neural network models have improved the predictive accuracy of intelligent text processing systems in the biomedical domain. They have obtained state-of-the-art performance scores on a wide variety of biomedical and clinical Natural Language Processing (NLP) benchmarks. However, the robustness and reliability of these models has been less explored so far. Neural NLP models can be easily fooled by adversarial samples, i.e. minor changes to input that preserve the meaning and understandability of the text but force the NLP system to make erroneous decisions. This raises serious concerns about the security and trust-worthiness of biomedical NLP systems, especially when they are intended to be deployed in real-world use cases. We investigated the robustness of several transformer neural language models, i.e. BioBERT, SciBERT, BioMed-RoBERTa, and Bio-ClinicalBERT, on a wide range of biomedical and clinical text processing tasks. We implemented various adversarial attack methods to test the NLP systems in different attack scenarios. Experimental results showed that the biomedical NLP models are sensitive to adversarial samples; their performance dropped in average by 21 and 18.9 absolute percent on character-level and word-level adversarial noise, respectively, on Micro-F1, Pearson Correlation, and Accuracy measures. Conducting extensive adversarial training experiments, we fine-tuned the NLP models on a mixture of clean samples and adversarial inputs. Results showed that adversarial training is an effective defense mechanism against adversarial noise; the models' robustness improved in average by 11.3 absolute percent. In addition, the models' performance on clean data increased in average by 2.4 absolute percent, demonstrating that adversarial training can boost generalization abilities of biomedical NLP systems. This study takes an important step towards revealing vulnerabilities of deep neural language models in biomedical NLP applications. It also provides practical and effective strategies to develop secure, trust-worthy, and accurate intelligent text processing systems in the biomedical domain.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1016/j.jbi.2022.104114

Cites

5 of 5 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Moradi and Samwald (2022) proposed an adversarial training framework targeting both character-level and word-level perturbations.

    “Moradi and Samwald (2022) proposed an adversarial training framework targeting both character-level and word-level perturbations.”

    From Trustworthy Medical Question Answering: An Evaluation-Centric Survey (2025) · cites this paper as Moradi and Samwald, 2022

  • By systematically integrating adversarial samples into training, this approach improves robustness and generalization in biomedical NLP tasks, including medical QA.

    “By systematically integrating adversarial samples into training, this approach improves robustness and generalization in biomedical NLP tasks, including medical QA.”

    From Trustworthy Medical Question Answering: An Evaluation-Centric Survey (2025) · cites this paper as Moradi and Samwald, 2022

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 45

  1. Y. Peng, S. Yan, and Z. Lu, "Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasets," in Proceedings of the BioNLP 2019 workshop, Florence, Italy, 2019, pp. 58-65.
  2. D. Ravi, C. Wong, F. Deligianni, M. Berthelot, J. Andreu-Perez, B. Lo, G.-Z. Yang, Deep Learning for Health Informatics, IEEE J. Biomed. Health. Inf. 21 (1) (2017) 4-21.
  3. M. Moradi, G. Dorffner, M. Samwald, Deep contextualized embeddings for quantifying the informative content in biomedical text summarization, Comput. Methods Programs Biomed. 184 (2020), 105117.
  4. M. Moradi, M. Dashti, M. Samwald, Summarization of biomedical articles using domain-specific word embeddings and graph ranking, J. Biomed. Inform. 107 (2020), 103452.
  5. W.E. Zhang, Q.Z. Sheng, A. Alhazmi, C. Li, Adversarial Attacks on Deep-learning Models in Natural Language Processing: A Survey, ACM Trans. Intell. Syst. Technol. 11 (3) (2020) 1-41.
  6. J. Ebrahimi, A. Rao, D. Lowd, and D. Dou, "HotFlip: White-Box Adversarial Examples for Text Classification," in Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), Melbourne, Australia, 2018, pp. 31-36.
  7. I. J. Goodfellow, J. Shlens, and C. Szegedy, "Explaining and harnessing adversarial examples," in the 3rd International Conference on Learning Representations (ICLR 2015), 2015.
  8. I. J. Goodfellow, J. Shlens, and C. Szegedy, "Explaining and harnessing adversarial examples," in the 3rd International Conference on Learning Representations (ICLR 2015), 2015.
  9. R. Jia, P. Liang, Adversarial Examples for Evaluating Reading Comprehension Systems, Denmark, Copenhagen, 2017, pp. 2021-2031.
  10. B. Liang, H. Li, M. Su, P. Bian, X. Li, and W. Shi, "Deep text classification can be fooled," presented at the Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence (IJCAI-18), 2018.
  11. M. Cheng, J. Yi, P.-Y. Chen, H. Zhang, and C.-J. Hsieh, "Seq2Sick: Evaluating the Robustness of Sequence-to-Sequence Models with Adversarial Examples," Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, pp. 3601-3608, 04/03 2020.
  12. Y. Wang, M. Bansal, Robust Machine Comprehension Models via Adversarial Training, Louisiana, New Orleans, 2018, pp. 575-581.
  13. Y. Belinkov, Y. Bisk, "Synthetic and Natural Noise Both Break Neural Machine Translation," presented at the 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, 2018.
  14. W. Hu and Y. Tan, "Black-box attacks against RNN based malware detection algorithms," in Workshops at the Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
  15. A. Al-Dujaili, A. Huang, E. Hemberg, U. O'Reilly, Adversarial Deep Learning for Robust Detection of Binary Encoded Malware, IEEE Security and Privacy Workshops (SPW) 2018 (2018) 76-82.
  16. J. Ebrahimi, D. Lowd, D. Dou, On Adversarial Examples for Character-Level Neural Machine Translation, New Mexico, USA, Santa Fe, 2018, pp. 653-663.
  17. M. Blohm, G. Jagfeld, E. Sood, X. Yu, N.T. Vu, Comparing Attention-Based Convolutional and Recurrent Neural Networks: Success and Limitations in Machine Reading Comprehension, Belgium, Brussels, 2018, pp. 108-118.
  18. J. Gao, J. Lanchantin, M.L. Soffa, Y. Qi, "Black-Box Generation of Adversarial Text Sequences to Evade Deep Learning Classifiers," in, IEEE Security and Privacy Workshops (SPW) 2018 (2018) 50-56.
  19. J. Li, S. Ji, T. Du, B. Li, and T. Wang, "Textbugger: Generating adversarial text against real-world applications," presented at the Network and Distributed Systems Security (NDSS) Symposium, 2019.
  20. M. Iyyer, J. Wieting, K. Gimpel, L. Zettlemoyer, Adversarial Example Generation with Syntactically Controlled Paraphrase Networks, Louisiana, New Orleans, 2018, pp. 1875-1885.
  21. G. Rawlinson, The Significance of Letter Position in Word Recognition, IEEE Aerosp. Electron. Syst. Mag. 22 (1) (2007) 26-27.
  22. J. Xu and Q. Du, "TextTricker: Loss-based and gradient-based adversarial attacks on text classification models," Engineering Applications of Artificial Intelligence, vol. 92, p. 103641, 2020/06/01/ 2020.
  23. M. Alzantot, Y. Sharma, A. Elgohary, B.-J. Ho, M. Srivastava, and K.-W. Chang, "Generating natural language adversarial examples," arXiv preprint arXiv: 1804.07998, 2018. · arXiv
  24. P. Minervini, S. Riedel, Adversarially Regularising Neural NLI Models to Integrate Logical Background Knowledge, Belgium, Brussels, 2018, pp. 65-74.
  25. T. Miyato, A. M. Dai, and I. J. Goodfellow, "Adversarial Training Methods for Semi- Supervised Text Classification," presented at the 5th International Conference on Learning Representations, ICLR 2017, 2017.
  26. D. Kang, T. Khot, A. Sabharwal, E. Hovy, AdvEntuRe: Adversarial Training for Textual Entailment with Knowledge-Guided Examples, Australia, Melbourne, 2018, pp. 2418-2428.
  27. M. Sato, J. Suzuki, H. Shindo, and Y. Matsumoto, "Interpretable adversarial perturbation in input embedding space for text," in the Twenty-Seventh International Joint Conference on Artificial Intelligence (IJCAI), 2018.
  28. M. Sun, F. Tang, J. Yi, F. Wang, and J. Zhou, "Identify Susceptible Locations in Medical Records via Adversarial Attacks on Deep Predictive Models," presented at the Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, London, United Kingdom, 2018.
  29. B. Rosario and M. A. Hearst, "Classifying semantic relations in bioscience texts," in Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL-04), 2004, pp. 430-437.
  30. A. Romanov and C. Shivade, "Lessons from Natural Language Inference in the Clinical Domain," in Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, 2018, pp. 1586-1596.
  31. F. Dernoncourt and J. Y. Lee, "PubMed 200k RCT: a Dataset for Sequential Sentence Classification in Medical Abstracts," in Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 2: Short Papers), Taipei, Taiwan, 2017, pp. 308-313.
  32. Q. Jin, B. Dhingra, Z. Liu, W. Cohen, X. Lu, PubMedQA: A Dataset for Biomedical Research Question Answering, China, Hong Kong, 2019, pp. 2567-2577.
  33. M. Moradi, M. Samwald, Explaining Black-Box Models for Biomedical Text Classification, IEEE J. Biomed. Health. Inf. 25 (2021) 3112-3120.
  34. A. Agibetov, K. Blagec, H. Xu, and M. Samwald, "Fast and scalable neural embedding models for biomedical sentence classification," BMC Bioinformatics, vol. 19, p. 541, 2018/12/22 2018.
  35. T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, et al., "Transformers: State-of-the-Art Natural Language Processing," in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, Online, 2020, pp. 38-45.
  36. Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, et al., "Roberta: A robustly optimized bert pretraining approach," arXiv preprint arXiv:1907.11692, 2019. · arXiv
  37. E. Alsentzer, J. Murphy, W. Boag, W.-H. Weng, D. Jindi, T. Naumann, et al., "Publicly Available Clinical BERT Embeddings," in Proceedings of the 2nd Clinical Natural Language Processing Workshop, Minneapolis, Minnesota, USA, 2019, pp. 72- 78.
  38. G. Zeng, F. Qi, Q. Zhou, T. Zhang, B. Hou, Y. Zang, et al., "OpenAttack: An Open- source Textual Adversarial Attack Toolkit," arXiv preprint arXiv:2009.09191, 2020. · arXiv
  39. J. Pennington, R. Socher, and C. D. Manning, "Glove: Global vectors for word representation," in Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), 2014, pp. 1532-1543.
  40. D. Jin, Z. Jin, J. T. Zhou, and P. Szolovits, "Is bert really robust? natural language attack on text classification and entailment," presented at the AAAI, 2020.
  41. H. Zhang, H. Zhou, N. Miao, L. Li, Generating Fluent Adversarial Examples for Natural Languages, Italy, Florence, 2019, pp. 5564-5569.
  42. M. Moradi and M. Samwald, "Evaluating the Robustness of Neural Language Models to Input Perturbations," arXiv preprint arXiv:2108.12237, 2021. · arXiv
  43. M. Moradi, K. Blagec, and M. Samwald, "Deep learning models are not robust against noise in clinical text," arXiv preprint arXiv:2108.12242, 2021. · arXiv
  44. M. Moradi, M. Samwald, Post-hoc explanation of black-box classifiers using confident itemsets, Expert Syst. Appl. 165 (2021), 113941.
  45. M. T. Ribeiro, S. Singh, and C. Guestrin, ""Why Should I Trust You?": Explaining the Predictions of Any Classifier," presented at the Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, California, USA, 2016.

Cited by

Sorted by citations

Authors
Moradi, M., & Samwald, M.
Affiliations
Medical University of Vienna (AT)
Corresponding author
Moradi, Milad
ORCID
Moradi, Milad
Venue
Journal of Biomedical Informatics
Publisher
Elsevier BV
Published
2022-08
Volume
132
Pages
104114
ISSN
1532-0464 (ISSN-L)
DOI
10.1016/j.jbi.2022.104114
OpenAlex
W3212212549
Cited by
18
References
5
Citations by year
2026: 6 · 2025: 3 · 2024: 4 · 2023: 4 · 2022: 1
FWCI
0.94
Type
journal article
Language
en
Open access
Licence
Locations
  • Journal of Biomedical Informatics (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, other-oa)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
Keywords
adversarial training, adversarial robustness, adversarial attacks, biomedical language models, transformer models, BioBERT, SciBERT, clinical text processing, model generalization
MeSH terms
Language, Natural Language Processing, Reproducibility of Results, Neural Networks, Computer

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review