CoChat
Sign Up
Cited by 4Meta-analysis and systematic reviewsInformation Retrieval and Search BehaviorTopic Modeling

Automatic literature screening using the PAJO deep-learning model for clinical practice guidelines

Open access
Lin, Yucong, Li, Jia, Xiao, Huan, Zheng, Lujie, Xiao, Ying, Song, Hong, Fan, Jingfan, Xiao, Deqiang +5 more·2023·BMC Medical Informatics and Decision Making
DOI 10.1186/s12911-023-02328-8PMC PMC10625217
Open this paper in CoChat — free accountOpen PDF

Abstract

Background: Clinical practice guidelines (CPGs) are designed to assist doctors in clinical decision making. High-quality research articles are important for the development of good CPGs. Commonly used manual screening processes are time-consuming and labor-intensive. Artificial intelligence (AI)-based techniques have been widely used to analyze unstructured data, including texts and images. Currently, there are no effective/efficient AI-based systems for screening literature. Therefore, developing an effective method for automatic literature screening can provide significant advantages. Methods: Using advanced AI techniques, we propose the Paper title, Abstract, and Journal (PAJO) model, which treats article screening as a classification problem. For training, articles appearing in the current CPGs are treated as positive samples. The others are treated as negative samples. Then, the features of the texts (e.g., titles and abstracts) and journal characteristics are fully utilized by the PAJO model using the pretrained bidirectional-encoder-representations-from-transformers (BERT) model. The resulting text and journal encoders, along with the attention mechanism, are integrated in the PAJO model to complete the task. Results: We collected 89,940 articles from PubMed to construct a dataset related to neck pain. Extensive experiments show that the PAJO model surpasses the state-of-the-art baseline by 1.91% (F1 score) and 2.25% (area under the receiver operating characteristic curve). Its prediction performance was also evaluated with respect to subject-matter experts, proving that PAJO can successfully screen high-quality articles. Conclusions: The PAJO model provides an effective solution for automatic literature screening. It can screen high-quality articles on neck pain and significantly improve the efficiency of CPG development. The methodology of PAJO can also be easily extended to other diseases for literature screening.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1186/s12911-023-02328-8

Cites

3 of 3 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Literature screening is the most time-consuming process in evidence synthesis practice, particularly in the medical field.

    “Literature screening is the most time-consuming process in evidence synthesis practice, particularly in the medical field [9–11]”

    From Artificial intelligence for the science of evidence synthesis: how good are AI-powered tools for automatic literature screening? (2025) · cites this paper as [9]

    The Abstract’s Background and the Conclusions describe manual screening or curation for clinical practice guidelines as time-consuming and labor-intensive. They do not say literature screening is the most time-consuming process in evidence synthesis; the comparative claim is unsupported. Page numbers are unavailable in our copy. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

  • The AI tools demonstrated comparatively limited false negative fraction in literature screening, outperforming results reported in some prior research.

    “Through this research, the AI tools demonstrated comparatively limited FNF in literature screening, outperforming results reported in some prior research [9, 16, 28].”

    From Artificial intelligence for the science of evidence synthesis: how good are AI-powered tools for automatic literature screening? (2025) · cites this paper as [9]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 28

  1. [1]Chen Y Yang K Marušić A Qaseem A Meerpohl JJ Flottorp S A reporting tool for practice guidelines in health care: the RIGHT statement Ann Intern Med 2017 166 128 32 10.7326/M16-1565 27893062 · DOI
  2. [2]Shekelle PG Clinical practice guidelines: what’s Next? J Am Med Assoc 2018 320 757 8 10.1001/jama.2018.9660 30098168 · DOI
  3. [5]Feng Y Liang S Zhang Y Chen S Wang Q Huang T Automated medical literature screening using artificial intelligence: a systematic review and meta-analysis J Am Med Inform Assoc 2022 29 1425 32 10.1093/jamia/ocac066 35641139 PMC9277646 · DOI
  4. [7]Kumar V Recupero DR Riboni D Ensembling classical machine learning and deep learning approaches for morbidity identification from clinical notes IEEE Access 2020 9 7107 26 10.1109/ACCESS.2020.3043221 · DOI
  5. [8]Wu S Roberts K Datta S Du J Ji Z Si Y Deep learning in clinical natural language processing: a methodical review J Am Med Inform Assoc 2020 27 457 70 10.1093/jamia/ocz200 31794016 PMC7025365 · DOI
  6. [9]Mullenbach J, Wiegreffe S, Duke J, Sun J, Eisenstein J. Explainable prediction of medical codes from clinical text. 2018; DOI:10.18653/v1/N18-1100. · DOI · Paper page
  7. [10]Prabhakar SK Won DO Medical text classification using hybrid deep learning models with multihead attention Comput Intell Neurosci 2021 2021 9425655 10.1155/2021/9425655 34603437 PMC8486521 · DOI
  8. [11]Zhang Y Liang S Feng Y Wang Q Sun F Chen S Automation of literature screening using machine learning in medical evidence synthesis: a diagnostic test accuracy systematic review protocol Syst Rev 2022 11 11 10.1186/s13643-021-01881-5 35031074 PMC8760775 · DOI
  9. [12]Lan Z, Chen M, Goodman S, Gimpel K, Sharma P, Soricut R. ALBERT: A lite BERT for self-supervised learning of language representations. In International Conference on Learning Representations. 2020:1311–28.
  10. [13]Beltagy I, Lo K, Cohan A. SciBERT: A Pretrained Language Model for Scientific Text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing. 2019:3615–20.
  11. [14]Lee J Yoon W Kim S Kim D Kim S So CH BioBERT: a pre-trained biomedical language representation model for biomedical text mining Bioinformatics 2020 36 1234 40 10.1093/bioinformatics/btz682 31501885 PMC7703786 · DOI · Paper page
  12. [15]Moen H, Alhuwail D, Björne J, et al. Towards Automated Screening of Literature on Artificial Intelligence in Nursing. Stud Health Technol Inform. 2022;290:637–40. 35673094
  13. [16]Kumar V Recupero DR Helaoui R K-LM: knowledge augmenting in Language Models within the Scholarly Domain IEEE Access 2022 10 91802 15 10.1109/ACCESS.2022.3201542 · DOI
  14. [17]Lin TY Goyal P Girshick R He K Dollar P Focal loss for dense object detection IEEE Trans Pattern Anal Mach Intell 2020 42 318 27 10.1109/TPAMI.2018.2858826 30040631 · DOI · Paper page
  15. [18]Gu Y, Tinn R, Cheng H, et al. Domain-specific language model pretraining for biomedical natural language processing. ACM Trans Comput Healthc. 2021;3(1):1–23.
  16. [19]Garfield E The history and meaning of the journal impact factor J Am Med Assoc 2006 295 90 3 10.1001/jama.295.1.90 16391221 · DOI · Paper page
  17. [20]Van Noorden R Impact factor gets heavyweight rival J Cit Rep 2016 30 20 27974784
  18. [21]Falagas ME Kouranos VD Arencibia-Jorge R Karageorgopoulos DE Comparison of SCImago journal rank indicator with journal impact factor FASEB J 2008 22 2623 8 10.1096/fj.08-107938 18408168 · DOI
  19. [23]Roldan-Valadez E Salazar-Ruiz SY Ibarra-Contreras R Rios C Current concepts on bibliometrics: a brief review about impact factor, eigenfactor score, CiteScore, SCImago journal rank, source-normalised impact per paper, H-index, and alternative metrics Ir J Med Sci 2019 188 939 51 10.1007/s11845-018-1936-5 30511320 · DOI
  20. [24]Devlin J, Chang MW, Lee K, Toutanova K, Bert. Pre-training of deep bidirectional transformers for language understanding. In Annual Conference of the North American Chapter of the Association for Computational Linguistics. 2019:4171–86.
  21. [25]Sun Y, Li Y, Zeng Q, et al. Application research of text classification based on random forest algorithm. In 2020 3rd International Conference on Advanced Electronic Materials, Computers and Software Engineering (AEMCSE). 2020:370–4.
  22. [26]Aseervatham S Antoniadis A Gaussier E Burlet M Denneulin Y A sparse version of the ridge logistic regression for large-scale text categorization Pattern Recognit Lett 2011 32 101 6 10.1016/j.patrec.2010.09.023 · DOI
  23. [27]Qing L Linhong W Xuehai D A novel neural network-based method for medical text classification Future Internet 2019 11 255 10.3390/fi11120255 · DOI
  24. [28]Deng J Cheng L Wang Z Attention-based BiLSTM fused CNN with gating mechanism model for chinese long text classification Comput Speech Lang 2021 68 101182 10.1016/j.csl.2020.101182 · DOI
  25. [29]Kim Y. Convolutional neural networks for sentence classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2014:1746–51. PMC5300751 28191551
  26. [30]Lai S, Xu L, Liu K, et al. Recurrent convolutional neural networks for text classification. In The 29th AAAI Conference on Artificial Intelligence. 2015:2267–73.
  27. [31]Pan L Lim WH Gan Y A method of Sustainable Development for three Chinese short-text datasets based on BERT-CAM Electronics 2023 12 7 1531 10.3390/electronics12071531 · DOI
  28. [32]Mingyu J Jiawei Z Ning W AFR-BERT: attention-based mechanism feature relevance fusion multimodal sentiment analysis model PLoS ONE 2022 17 9 e0273936 10.1371/journal.pone.0273936 36084041 PMC9462790 · DOI

Cited by

Sorted by citations

Authors
Lin, Y., Li, J., Xiao, H., Zheng, L., Xiao, Y., Song, H., Fan, J., Xiao, D., Ai, D., Fu, T., Wang, F., Lv, H., & Yang, J.
Affiliations
  • Beijing Institute of Technology (CN)
  • Capital Medical University (CN)
  • Renmin University of China (CN)
Corresponding author
  • Wang, Feifei
  • Lv, Han
  • Yang, Jian
ORCID
Venue
BMC Medical Informatics and Decision Making
Publisher
Springer Science and Business Media LLC
Published
2023-11-03
Volume
23
Issue
1
Pages
247-247
ISSN
1472-6947 (ISSN-L)
DOI
10.1186/s12911-023-02328-8
PMC
PMC10625217
OpenAlex
W4388290402
Cited by
4
References
3
Citations by year
2026: 2 · 2025: 2
FWCI
0.74
Type
journal article
Language
en
Open access
Licence
Locations
  • BMC Medical Informatics and Decision Making (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
  • National Key R&D Program of China (2021ZD0113201)
  • The MOE Project of Key Research Institute of Humanities and Social Sciences (22JJD910001)
  • National Natural Science Foundation of China (72371241, 62171297)
  • National Science Foundation Program of China (62025104)
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
BERT, journal characteristics, neck pain, guideline development, text classification, attention mechanism
MeSH terms
Clinical Decision-Making, Deep Learning, Artificial Intelligence, Humans, Review Literature as Topic, Practice Guidelines as Topic, Neck Pain

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review