Matching patients to clinical trials using semantically enriched document representation
Open accessAbstract
Recruiting eligible patients for clinical trials is crucial for reliably answering specific questions about medical interventions and evaluation. However, clinical trial recruitment is a bottleneck in clinical research and drug development. Our goal is to provide an approach towards automating this manual and time-consuming patient recruitment task using natural language processing and machine learning techniques. Specifically, our approach extracts key information from series of narrative clinical documents in patient's records and collates helpful evidence to make decisions on eligibility of patients according to certain inclusion and exclusion criteria. Challenges in applying narrative clinical documents such as differences in reporting styles and sub-languages are addressed by enriching them with knowledge from domain ontologies in the form of semantic vector representations. We show that a machine learning model based on Multi-Layer Perceptron (MLP) is more effective for the task than five other neural networks and four conventional machine learning models. Our approach achieves overall micro-F1-Score of 84% for 13 different eligibility criteria. Our experiments also indicate that semantically enriched documents are more effective than using original documents for cohort selection. Our system provides an end-to-end machine learning-based solution that achieves comparable results with the state-of-the-art which relies on hand-crafted rules or data-centric engineered features.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
BERT-based tools have been used for subject matching across various diseases while struggling with generalizability.
“Bidirectional Encoder Representations from Transformers (BERT)-based tools have been used for subject matching across various diseases while struggling with generalizability [4–7].”
From Enhancing hepatopathy clinical trial efficiency: a secure, large language model-powered pre-screening pipeline (2025) · cites this paper as [6]
Section 2.3 mentions BERT as a contextual text representation, but does not describe a BERT-based patient-matching tool. Section 5.2 attributes poor generalisability to rule-based systems; it says deep-learning approaches are more transferable and generalisable, contrary to the claim’s generalisability point. No page numbers are available in our copy. (LLM, conf=0.83)
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126 · see citing works
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016 · see citing works
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · see citing works
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · see citing works
(2016) · Cited by 11,679 · see citing works
TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G. S., Davis, A., Dean, J., Devin, M., et al. (2016) · arXiv · doi:10.48550/arxiv.1603.04467 · Cited by 9,408 · see citing works
Savova, G. K., Masanz, J. J., Ogren, P. V., Zheng, J., Sohn, S., Kipper-Schuler, K. C., & Chute, C. G. (2010) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.2009.001560 · Cited by 1,967 · see citing works
Big Data In Health Care: Using Analytics To Identify And Manage High-Risk And High-Cost Patients
Bates, D. W., Saria, S., Ohno-Machado, L., Shah, A., & Escobar, G. (2014) · Health Affairs · doi:10.1377/hlthaff.2014.0041 · Cited by 1,235 · see citing works
Convolutional Neural Networks for Sentence ClassificationPreprint
Kim, Y. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1408.5882 · Cited by 1,112 · see citing works
Transfer learning using computational intelligence: A survey
Jie Lu, B., Behbood, V., Peng, H., Zuo, H., Xue, S., & Zhang, G. (2015) · Knowledge-Based Systems · doi:10.1016/j.knosys.2015.01.010 · Cited by 971 · see citing works
Clinical information extraction applications: A literature review
Wang, Y., Wang, L., Rastegar-Mojarad, M., Moon, S., Shen, F., Afzal, N., Liu, S., Zeng, Y., Mehrabi, S., Sohn, S., et al. (2018) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2017.11.011 · Cited by 910 · see citing works
What can natural language processing do for clinical decision support?
Demner‐Fushman, D., Chapman, W. W., & McDonald, C. J. (2009) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2009.08.007 · Cited by 690 · see citing works
Kreimeyer, K., Foster, M., Pandey, A., Arya, N., Halford, G., Jones, S. F., Forshee, R., Walderhaug, M., & Botsis, T. (2017) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2017.07.012 · Cited by 616 · see citing works
Key Concepts of Clinical Trials: A Narrative Review
Umscheid, C. A., Margolis, D. J., & Grossman, C. E. (2011) · Postgraduate Medicine · doi:10.3810/pgm.2011.09.2475 · Cited by 414 · see citing works
Data from clinical notes: a perspective on the tension between structure and flexible documentation
Rosenbloom, S. T., Denny, J. C., Xu, H., Lorenzi, N., Stead, W. W., & Johnson, K. B. (2011) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.2010.007237 · Cited by 404 · see citing works
Biomedical Ontologies in Action: Role in Knowledge Management, Data Integration and Decision Support
Bodenreider, O. (2008) · Yearbook of Medical Informatics · doi:10.1055/s-0038-1638585 · Cited by 368 · see citing works
Randomized controlled trials – a matter of design
Siepmann, T., Spieth, P. M., Kubasch, A. S., Penzlin, A. I., Illigens, B. M.-W., & Barlinn, K. (2016) · Neuropsychiatric Disease and Treatment · doi:10.2147/ndt.s101938 · Cited by 292 · see citing works
The open biomedical annotator.
Jonquet, C., Shah, N. H., & Musen, M. A. (2009) · PubMed · Cited by 253 · see citing works
Gehrmann, S., Dernoncourt, F., Li, Y., Carlson, E. T., Wu, J. T.-Y., Welt, J. A., Foote, J., Moseley, E. T., Grant, D. W., Tyler, P. D., et al. (2018) · PLoS ONE · doi:10.1371/journal.pone.0192360 · Cited by 243 · see citing works
Information extraction from multi-institutional radiology reports
Hassanpour, S., & Langlotz, C. P. (2016) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2015.09.007 · Cited by 182 · see citing works
Challenges in recruitment and retention of clinical trial subjects
Kadam, R., Borde, S., Madas, S., Salvi, S., & Limaye, S. (2016) · Perspectives in Clinical Research · doi:10.4103/2229-3485.184820 · Cited by 179 · see citing works
A ‘Green Button’ For Using Aggregate Patient Data At The Point Of Care
Longhurst, C., Harrington, R. A., & Shah, N. H. (2014) · Health Affairs · doi:10.1377/hlthaff.2014.0099 · Cited by 159 · see citing works
Ni, Y., Wright, J., Perentesis, J., Lingren, T., Deleger, L., Kaiser, M., Kohane, I., & Solti, I. (2015) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-015-0149-3 · Cited by 142 · see citing works
Drivers and barriers to patient participation in RCTs
Jenkins, V. A., Farewell, V. T., Farewell, D. M., et al. (2013) · British Journal of Cancer · doi:10.1038/bjc.2013.113 · Cited by 108 · see citing works
Machine Learning and Rule-based Approaches to Assertion Classification
Uzuner, Ö., Zhang, X., & Sibanda, T. (2008) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m2950 · Cited by 98 · see citing works
Roberts, K. E., Simpson, M. S., Demner‐Fushman, D., et al. (2015) · Information Retrieval · doi:10.1007/s10791-015-9259-x · Cited by 96 · see citing works
Ontoserver: a syndicated terminology server
Metke‐Jimenez, A., Steel, J. R. H., Hansen, D. T., et al. (2018) · Journal of Biomedical Semantics · doi:10.1186/s13326-018-0191-z · Cited by 86 · see citing works
Openness of patients' reporting with use of electronic records: psychiatric clinicians' views
Salomon, R. M., Blackford, J. U., Rosenbloom, S. T., Seidel, S., Clayton, E. W., Dilts, D. M., & Finder, S. G. (2010) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m3341 · Cited by 80 · see citing works
Cohort selection for clinical trials: n2c2 2018 shared task track 1
Stubbs, A., Filannino, M., Soysal, E., Henry, S., & Uzuner, Ö. (2019) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocz163 · Cited by 73 · see citing works
Karimi, S., Dai, X., Hassanzadeh, H., & Nguyen, A. N. (2017) · doi:10.18653/v1/w17-2342 · Cited by 59 · see citing works
Self-citationTowards achieving semantic interoperability of clinical study data with FHIR
Leroux, H., Metke‐Jimenez, A., & Lawley, M. (2017) · Journal of Biomedical Semantics · doi:10.1186/s13326-017-0148-7 · Cited by 59 · see citing works
Improving Patient Cohort Identification Using Natural Language Processing
Sarmiento, R. F., & Dernoncourt, F. (2016) · Springer eBooks · doi:10.1007/978-3-319-43742-2_28 · Cited by 39 · see citing works
Cohort selection for clinical trials using hierarchical neural network
Xiong, Y., Shi, X., Chen, S., Jiang, D., Tang, B., Wang, X., Chen, Q., & Yan, J. (2019) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocz099 · Cited by 30 · see citing works
Beyond word embeddings: learning entity and concept representations from large scale knowledge bases
Shalaby, W., Zadrozny, W., & Jin, H. (2018) · Information Retrieval · doi:10.1007/s10791-018-9340-3 · Cited by 29 · see citing works
Identifying Risk Factors For Heart Disease in Electronic Medical Records: A Deep Learning Approach
Chokwijitkul, T., Nguyen, A. N., Hassanzadeh, H., & Perez, S. (2018) · Workshop on Biomedical Language Processing (BioNLP) · doi:10.18653/v1/w18-2303 · Cited by 28 · see citing works
Self-citationCohort selection for clinical trials using deep learning models
Segura-Bédmar, I., & Raez, P. (2019) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocz139 · Cited by 28 · see citing works
Hassanzadeh, H., Nguyen, A. N., Karimi, S., & Chu, K. (2018) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2018.07.017 · Cited by 26 · see citing works
Self-citationHybrid bag of approaches to characterize selection criteria for cohort identification
Vydiswaran, V. G. V., Strayhorn, A., Zhao, X., Robinson, P., Agarwal, M., Bagazinski, E., Essiet, M., Iott, B. E., Joo, H., Ko, P., et al. (2019) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocz079 · Cited by 22 · see citing works
Abrahão, M. T. F., Nobre, M. R. C., & Gutierrez, M. A. (2017) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2017.03.015 · Cited by 21 · see citing works
Clinical Document Classification Using Labeled and Unlabeled Data Across Hospitals
Hassanzadeh, H., Kholghi, M., Nguyen, A. N., & Chu, K. (2018) · Europe PMC (PubMed Central) · doi:10.48550/arxiv.1812.00677 · Cited by 14 · see citing works
Self-citationLearning relevance models for patient cohort retrieval
Goodwin, T. R., & Harabagiu, S. M. (2018) · JAMIA Open · doi:10.1093/jamiaopen/ooy010 · Cited by 12 · see citing works
Hassanzadeh, H., Nguyen, A. N., & Verspoor, K. M. (2019) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2019.103321 · Cited by 11 · see citing works
Self-citationEvaluation of Medical Concept Annotation Systems on Clinical Records.
Hassanzadeh, H., Nguyen, A. N., & Koopman, B. (2016) · Cited by 9 · see citing works
Self-citationRule-based and Machine Learning Hybrid System for Patient Cohort Selection
Antunes, R., Silva, J., Pereira, A., & Matos, S. (2019) · International Joint Conference on Biomedical Engineering Systems and Technologies (BIOSTEC) · doi:10.5220/0007349300590067 · Cited by 9 · see citing works
Designing a Novel Framework for Precision Medicine Information Retrieval
Chen, H., Ding, J., Chen, J., et al. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-03649-2_16 · Cited by 2 · see citing works
Flagged first, then by citations
Research works printed but not linked: 8
- F.S. Kury, V. Huser, J.J. Cimino, Reproducing a prospective clinical study as a computational retrospective study in MIMIC-II, in: AMIA Annual Symposium Proceedings, American Medical Informatics Association, Washington, DC, 2015, pp. 804-813.
- K. Kreimeyer, M. Foster, A. Pandey, N. Arya, G. Halford, S.F. Jones, R. Forshee, M. Walderhaug, T. Botsis, Natural language processing systems for capturing and standardizing unstructured clinical information: a systematic review, J. Biomed. Informatics 73 (2017) 14-29.
- T. Mikolov, I. Sutskever, K. Chen, G.S. Corrado, J. Dean, Distributed representations of words and phrases and their compositionality, in: International Conference on Neural Information Processing Systems, Lake Tahoe, NV, 2013, pp. 3111-3119.
- J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, BERT: Pre-training of deep bidirec- tional transformers for language understanding, in: Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Minneapolis, Minnesota, 2019, pp. 4171-4186.
- J. Mitchell, M. Lapata, Vector-based models of semantic composition, in: The Annual Meeting of the Association for Computational Linguistics with the Human Language Technology Conference, Columbus, OH, 2008, pp. 236-244.
- Q. Le, T. Mikolov, Distributed representations of sentences and documents, in: 31st International Conference on Machine Learning, Beijing, China, 2014, pp. 1188-1196.
- R. Rehruvrek, P. Sojka, Software framework for topic modelling with large corpora, in: LREC Workshop on New Challenges for NLP Frameworks, Valletta, Malta, 2010, pp. 45-50.
- B. Neyshabur, S. Bhojanapalli, D. McAllester, N. Srebro, Exploring generalization in deep learning, in: International Conference on Neural Information Processing Systems, Long Beach, CA, 2017, pp. 5947-5956.
Cited by
Sorted by citations
Lu, X., Yang, C., Liang, L., Hu, G., Zhong, Z., & Jiang, Z. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae243 · Cited by 77 · see citing works
Chow, R., Midroni, J., Kaur, J., Boldt, G., Liu, G., Eng, L., Liu, F.-F., Haibe-Kains, B., Lock, M., & Raman, S. (2023) · JNCI: Journal of the National Cancer Institute · doi:10.1093/jnci/djad013 · Cited by 65 · see citing works
Towards evidence-based practice 2.0: leveraging artificial intelligence in healthcare
Nilsen, P., Sundemo, D., Heintz, F., Neher, M., Nygren, J., Svedberg, P., & Petersson, L. (2024) · Frontiers in Health Services · doi:10.3389/frhs.2024.1368030 · Cited by 39 · see citing works
Application of Information Technology to Clinical Trial Evaluation and Enrollment: A Review
von Itzstein, M. S., Hullings, M., Mayo, H., Beg, M. S., Williams, E. L., & Gerber, D. E. (2021) · JAMA Oncology · doi:10.1001/jamaoncol.2021.1165 · Cited by 36 · see citing works
Goel, I., Bhaskar, Y., Kumar, N., Singh, S., Amanullah, M., Dhar, R., & Karmakar, S. (2025) · Frontiers in Digital Health · doi:10.3389/fdgth.2025.1550407 · Cited by 31 · see citing works
Gui, X., Lv, H., Wang, X., Lv, L., Xiao, Y., & Wang, L. (2025) · BioData Mining · doi:10.1186/s13040-025-00458-5 · Cited by 7 · see citing works
- Authors
- Hassanzadeh, H., Karimi, S., & Nguyen, A.
- Affiliations
- Commonwealth Scientific and Industrial Research Organisation (AU)
- Data61 (AU)
- Corresponding author
- Hassanzadeh, Hamed
- Venue
- Journal of Biomedical Informatics
- Publisher
- Elsevier BV
- Published
- 2020-05
- Volume
- 105
- Pages
- 103406
- ISSN
- 1532-0464 (ISSN-L)
- OpenAlex
- W3011079674
- Cited by
- 52
- References
- 45
- Citations by year
- 2026: 7 · 2025: 12 · 2024: 11 · 2023: 8 · 2022: 7 · 2021: 6 · 2020: 1
- FWCI
- 1.79
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Licence
- Locations
- Journal of Biomedical Informatics (publishedVersion)
- PubMed (publishedVersion)
- Funders
- National Health and Medical Research Council Centre for Research Excellence in Digital Health (1134919)
- Topics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Biomedical Text Mining and Ontologies
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Keywords
- inclusion and exclusion criteria, patient recruitment, semantic enrichment, domain ontology, multilayer perceptron, eligibility criteria, cohort selection, natural language processing
- MeSH terms
- Machine Learning, Humans, Language, Natural Language Processing, Semantics, Neural Networks, Computer
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
