Natural Language Processing for Mimicking Clinical Trial Recruitment in Critical Care: A Semi-Automated Simulation Based on the LeoPARDS Trial
Open accessPublished version of a preprint
Abstract
Clinical trials often fail to recruit an adequate number of appropriate patients. Identifying eligible trial participants is resource-intensive when relying on manual review of clinical notes, particularly in critical care settings where the time window is short. Automated review of electronic health records (EHR) may help, but much of the information is in free text rather than a computable form. We applied natural language processing (NLP) to free text EHR data using the CogStack platform to simulate recruitment into the LeoPARDS study, a clinical trial aiming to reduce organ dysfunction in septic shock. We applied an algorithm to identify eligible patients using a moving 1-hour time window, and compared patients identified by our approach with those actually screened and recruited for the trial, for the time period that data were available. We manually reviewed records of a random sample of patients identified by the algorithm but not screened in the original trial. Our method identified 376 patients, including 34 patients with EHR data available who were actually recruited to LeoPARDS in our centre. The sensitivity of CogStack for identifying patients screened was 90% (95% CI 85%, 93%). Of the 203 patients identified by both manual screening and CogStack, the index date matched in 95 (47%) and CogStack was earlier in 94 (47%). In conclusion, analysis of EHR data using NLP could effectively replicate recruitment in a critical care trial, and identify some eligible patients at an earlier stage, potentially improving trial recruitment if implemented in real time.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
BERT-based tools have been used for subject matching across various diseases while struggling with generalizability.
“Bidirectional Encoder Representations from Transformers (BERT)-based tools have been used for subject matching across various diseases while struggling with generalizability [4–7].”
From Enhancing hepatopathy clinical trial efficiency: a secure, large language model-powered pre-screening pipeline (2025) · cites this paper as [7]
The Discussion (p. 10) reports that an NLP-based algorithm matched patients to clinical-trial selection criteria, and Strengths and limitations (p. 11) notes that the study was limited to one site. The paper describes CogStack and SemEHR rather than a BERT-based tool, and it tested recruitment for a sepsis trial—not subject matching across various diseases. Its suggestion that the method could be adapted for other studies does not establish the claimed generalizability problem for BERT-based tools. (LLM, conf=0.88)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Transection of the oesophagus for bleeding oesophageal varices
Pugh, R. N. H., Murray-Lyon, I. M., Dawson, J. L., Pietroni, M. C., & Williams, R. (1973) · Journal of British Surgery · doi:10.1002/bjs.1800600817 · Cited by 7,941 · see citing works
2001 SCCM/ESICM/ACCP/ATS/SIS International Sepsis Definitions Conference
Levy, M. M., Fink, M. P., Marshall, J. C., Abraham, E. H., Angus, D. C., Cook, D., Cohen, J., Opal, S. M., Vincent, J., & Ramsay, G. (2003) · Critical Care Medicine · doi:10.1097/01.ccm.0000050454.01978.3b · Cited by 5,976 · see citing works
2001 SCCM/ESICM/ACCP/ATS/SIS International Sepsis Definitions Conference
Conference, F. T. I. S. D., Levy, M. M., Fink, M. P., Marshall, J. C., Abraham, E. H., Angus, D. C., Cook, D., Cohen, J., Opal, S. M., Vincent, J., et al. (2003) · Intensive Care Medicine · doi:10.1007/s00134-003-1662-x · Cited by 2,877 · see citing works
Savova, G. K., Masanz, J. J., Ogren, P. V., Zheng, J., Sohn, S., Kipper-Schuler, K. C., & Chute, C. G. (2010) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.2009.001560 · Cited by 1,967 · see citing works
The PRECIS-2 tool: designing trials that are fit for purpose
Loudon, K., Treweek, S., Sullivan, F., Donnan, P., Thorpe, K. E., & Zwarenstein, M. (2015) · BMJ · doi:10.1136/bmj.h2147 · Cited by 1,846 · see citing works
Kennedy-Martin, T., Curtis, S., Faries, D., Robinson, S., & Johnston, J. (2015) · Trials · doi:10.1186/s13063-015-1023-4 · Cited by 878 · see citing works
Kreimeyer, K., Foster, M., Pandey, A., Arya, N., Halford, G., Jones, S. F., Forshee, R., Walderhaug, M., & Botsis, T. (2017) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2017.07.012 · Cited by 616 · see citing works
Walters, S. J., Bonacho dos Anjos Henriques-Cadby, I., Bortolami, O., Flight, L., Hind, D., Jacques, R. M., Knox, C., Nadin, B., Rothwell, J., Surtees, M., et al. (2017) · BMJ Open · doi:10.1136/bmjopen-2016-015276 · Cited by 560 · see citing works
The Randomized Registry Trial — The Next Disruptive Technology in Clinical Research?
Lauer, M. S., & D'Agostino, R. B. (2013) · New England Journal of Medicine · doi:10.1056/nejmp1310102 · Cited by 497 · see citing works
Harkema, H., Dowling, J., Thornblade, T., & Chapman, W. W. (2009) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2009.05.002 · Cited by 406 · see citing works
Levosimendan for the Prevention of Acute Organ Dysfunction in Sepsis
Gordon, A. C., Perkins, G. D., Singer, M., McAuley, D. F., Orme, R. M. L., Santhakumaran, S., Mason, A. J., Cross, M., Al-Beidh, F., Best-Lane, J., et al. (2016) · New England Journal of Medicine · doi:10.1056/nejmoa1609409 · Cited by 372 · see citing works
Self-citationGetting More Out of Biomedical Documents with GATE's Full Lifecycle Open Source Text Analytics
Cunningham, H., Tablan, V., Roberts, A., & Bontcheva, K. (2013) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1002854 · Cited by 353 · see citing works
Registry-based randomized clinical trials—a new clinical trial paradigm
James, S., Rao, S. V., & Granger, C. B. (2015) · Nature Reviews Cardiology · doi:10.1038/nrcardio.2015.33 · Cited by 331 · see citing works
Randomized controlled trials – a matter of design
Siepmann, T., Spieth, P. M., Kubasch, A. S., Penzlin, A. I., Illigens, B. M.-W., & Barlinn, K. (2016) · Neuropsychiatric Disease and Treatment · doi:10.2147/ndt.s101938 · Cited by 292 · see citing works
Prokosch, H. (2009) · doi:10.3414/me9132 · Cited by 262 · see citing works
Gehrmann, S., Dernoncourt, F., Li, Y., Carlson, E. T., Wu, J. T.-Y., Welt, J. A., Foote, J., Moseley, E. T., Grant, D. W., Tyler, P. D., et al. (2018) · PLoS ONE · doi:10.1371/journal.pone.0192360 · Cited by 243 · see citing works
Natural Language Processing and the Representation of Clinical Data
Sager, N., Lyman, M. S., Bucknall, C. E., Nhàn, N. T., & Tick, L. J. (1994) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.1994.95236145 · Cited by 217 · see citing works
van Staa, T. P., Dyson, L., McCann, G., Padmanabhan, S., Belatri, R., Goldacre, B. M., Cassell, J., Pirmohamed, M., Torgerson, D. J., Ronaldson, S. J., et al. (2014) · Health Technology Assessment · doi:10.3310/hta18430 · Cited by 156 · see citing works
A practical method for transforming free-text eligibility criteria into computable criteria
Tu, S. W., Mor Peleg, B., Carini, S., Bobak, M., Ross, J., Rubin, D. L., & Sim, I. (2010) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2010.09.007 · Cited by 144 · see citing works
Ni, Y., Wright, J., Perentesis, J., Lingren, T., Deleger, L., Kaiser, M., Kohane, I., & Solti, I. (2015) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-015-0149-3 · Cited by 142 · see citing works
Jackson, R., Kartoglu, I., Stringer, C., Gorrell, G., Roberts, A., Song, X., Wu, H., Agrawal, A., Lui, K., Groza, T., et al. (2018) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-018-0623-9 · Cited by 134 · see citing works
Self-citationWu, H., Toti, G., Morley, K. I., Ibrahim, Z. M., Folarin, A., Jackson, R., Kartoglu, I. E., Agrawal, A., Stringer, C., Gale, D., et al. (2018) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocx160 · Cited by 123 · see citing works
Self-citationBurns, K. E. A., Zubrinich, C. M., Tan, W., Raptis, S., Xiong, W., Smith, O. M., McDonald, E., Marshall, J. C., Saginur, R., Heslegrave, R. J., et al. (2013) · American Journal of Respiratory and Critical Care Medicine · doi:10.1164/rccm.201208-1537oc · Cited by 115 · see citing works
Ni, Y., Kennebeck, S., Dexheimer, J. W., McAneney, C. M., Tang, H., Lingren, T., Li, Q., Zhai, H., & Solti, I. (2014) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2014-002887 · Cited by 115 · see citing works
Wang, Z., Shah, A. D., Tate, A. R., et al. (2012) · PLoS ONE · doi:10.1371/journal.pone.0030412 · Cited by 112 · see citing works
Self-citationMatching Patient Records to Clinical Trials Using Ontologies
Patel, C. O., Cimino, J. J., Dolby, J., Fokoue, A., Kalyanpur, A. A., Kershenbaum, A., Ma, L., Schonberg, E., & Srinivas, K. (2007) · Lecture notes in computer science · doi:10.1007/978-3-540-76298-0_59 · Cited by 96 · see citing works
Comparing semi-automatic systems for recruitment of patients to clinical trials
Marc Cuggia, A., Besana, P., & Glasspool, D. W. (2011) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2011.02.003 · Cited by 91 · see citing works
Li, L., Chase, H. S., Patel, C. O., Friedman, C., & Weng, C. (2008) · PubMed Central · Cited by 81 · see citing works
Prospective recruitment of patients with congestive heart failure using an ad-hoc binary classifier
Pakhomov, S., Buntrock, J. D., & Chute, C. G. (2004) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2004.11.016 · Cited by 69 · see citing works
Commentary: External validity of results of randomized trials: disentangling a complex concept
Rothwell, P. M. (2009) · International Journal of Epidemiology · doi:10.1093/ije/dyp305 · Cited by 64 · see citing works
Jonnalagadda, S., Adupa, A. K., Garg, R., Corona-Cox, J., & Shah, S. J. (2017) · Journal of Cardiovascular Translational Research · doi:10.1007/s12265-017-9752-2 · Cited by 64 · see citing works
Harris, S. K., Shi, S., Brealey, D. A., MacCallum, N. S., Denaxas, S., Pérez–Suárez, D., Ercole, A., Watkinson, P., Jones, A., Ashworth, S., et al. (2018) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2018.01.006 · Cited by 56 · see citing works
Self-citationExploring obstacles to critical care trials in the UK: A qualitative investigation
Pattison, N. A., Arulkumaran, N., Humphreys, S. K., et al. (2016) · Journal of the Intensive Care Society · doi:10.1177/1751143716663749 · Cited by 36 · see citing works
Classifying clinical notes with pain assessment using machine learning
Fodeh, S. J., Finch, D. K., Bouayad, L., Luther, S. L., Ling, H., Kerns, R. D., & Brandt, C. (2017) · Medical & Biological Engineering & Computing · doi:10.1007/s11517-017-1772-1 · Cited by 35 · see citing works
Rowlands, C., Rooshenas, L., Fairhurst, K., Rees, J. R. E., Gamble, C. L., & Blazeby, J. M. (2018) · BMJ Open · doi:10.1136/bmjopen-2017-018581 · Cited by 34 · see citing works
Gulliford, M. C., van Staa, T. P., McDermott, L., McCann, G., Charlton, J., & Dregan, A. (2014) · Trials · doi:10.1186/1745-6215-15-220 · Cited by 32 · see citing works
Assessing clinical trial eligibility with logic expression queries
Lonsdale, D. W., Tustison, C., Parker, C. G., et al. (2007) · Data & Knowledge Engineering · doi:10.1016/j.datak.2007.07.005 · Cited by 26 · see citing works
Asselbergs, F. W., Visseren, F. L. J., Bots, M. L., de Borst, G. J., Buijsrogge, M. P., Dieleman, J. M., Dinther, B. G. F., Doevendans, P. A., Hoefer, I. E., Hollander, M., et al. (2017) · European Journal of Preventive Cardiology · doi:10.1177/2047487317690284 · Cited by 25 · see citing works
Self-citationNormalisation of imprecise temporal expressions extracted from text
Tissot, H. C., Fabro, M. D. D., Derczynski, L., et al. (2019) · Knowledge and Information Systems · doi:10.1007/s10115-019-01338-1 · Cited by 24 · see citing works
Self-citationXiao, L., Lv, N., Rosas, L. G., Au, D. H., & Ma, J. (2017) · Obesity · doi:10.1002/oby.21737 · Cited by 23 · see citing works
Information extraction from Italian medical reports: An ontology-driven approach
Viani, N., Larizza, C., Tibollo, V., Napolitano, C., Priori, S. G., Bellazzi, R., & Sacchi, L. (2017) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2017.12.013 · Cited by 19 · see citing works
Shah, A. D., Bailey, E., Williams, T. J., Denaxas, S., Dobson, R., & Hemingway, H. (2019) · Journal of Biomedical Semantics · doi:10.1186/s13326-019-0214-4 · Cited by 18 · see citing works
Self-citationOlasov, B., & Sim, I. (2006) · PubMed · Cited by 14 · see citing works
Analysis of Temporal Expressions Annotated in Clinical Notes
Tissot, H. C., Roberts, A., Derczynski, L., et al. (2015) · UCL Discovery (University College London) · Cited by 12 · see citing works
Self-citationTissot, H. C., & Dobson, R. (2019) · Journal of Biomedical Semantics · doi:10.1186/s13326-019-0216-2 · Cited by 12 · see citing works
Self-citationECRL: an eligibility criteria representation language based on the UMLS Semantic Network.
Patel, C. O., & Weng, C. (2008) · PubMed · Cited by 8 · see citing works
Gray, E., Norris, S., Schmitz, S., & O’Leary, A. C. (2016) · Journal of Comparative Effectiveness Research · doi:10.2217/cer-2016-0042 · Cited by 8 · see citing works
Observational Cohort Studies and the Challenges of In Silico Experiments
Walsh, C. G., & Johnson, K. B. (2016) · JAMA Oncology · doi:10.1001/jamaoncol.2016.3478 · Cited by 6 · see citing works
Bio-YODIE: A Named Entity Linking System for Biomedical TextPreprint
Gorrell, G., Song, X., & Roberts, A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.04860 · Cited by 4 · see citing works
Viani, N., Tissot, H. C., Bernardino, A., et al. (2019) · doi:10.18653/v1/w19-5021 · Cited by 2 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 3
- [18][18] J. Pustejovsky, K. Lee, H. Bunt, and L. Romary, 'ISO-TimeML: An international standard for semantic annotation,' in Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC'10) . ELRA, 2010. -
- [25][25] J. Kremidas, 'Recruitment roles,' Applied Clinical Trials , vol. 20, no. 9, pp. 32-33, Sep 2011. -
- [29][29] R. Shah, B. Steinberg, B. Bucher, A. Chapman, D. Lloyd-Jones, M. Rondina, and W. Chapman, '2182: Developing a corpus for natural language processing to identify bleeding complications among intensive care unit patients,' Journal of Clinical and Translational Science , vol. 1, no. S1, pp. 12-12, 2017. -
Cited by
Sorted by citations
A survey on clinical natural language processing in the United Kingdom from 2007 to 2022
Wu, H., Wang, M., Wu, J., Francis, F., Chang, Y.-H., Shavick, A., Dong, H., Poon, M. T. C., Fitzpatrick, N., Levine, A. P., et al. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00730-6 · Cited by 101 · see citing works
Self-citationLu, X., Yang, C., Liang, L., Hu, G., Zhong, Z., & Jiang, Z. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae243 · Cited by 77 · see citing works
“59 Transparency was also an important ethical consideration of AI.”
BackgroundMethodInfluentialApplication of Information Technology to Clinical Trial Evaluation and Enrollment: A Review
von Itzstein, M. S., Hullings, M., Mayo, H., Beg, M. S., Williams, E. L., & Gerber, D. E. (2021) · JAMA Oncology · doi:10.1001/jamaoncol.2021.1165 · Cited by 36 · see citing works
Gui, X., Lv, H., Wang, X., Lv, L., Xiao, Y., & Wang, L. (2025) · BioData Mining · doi:10.1186/s13040-025-00458-5 · Cited by 7 · see citing works
“Bidirectional Encoder Representations from Transformers (BERT)-based tools have been used for subject matching across various diseases while struggling with generalizability [4–7].”
- Authors
- Tissot, H. C., Shah, A. D., Brealey, D., Harris, S., Agbakoba, R., Folarin, A., Romao, L., Roguski, L., Dobson, R., & Asselbergs, F. W.
- Affiliations
- University College Hospital (GB)
- University of London (GB)
- Corresponding author
- Tissot, Hegler C.
- ORCID
- Venue
- IEEE Journal of Biomedical and Health Informatics
- Published
- 2020-10
- Volume
- 24
- Issue
- 10
- Pages
- 2950-2959
- ISSN
- 2168-2194 (ISSN-L)
- 2168-2208
- OpenAlex
- W3012073911
- Cited by
- 61
- References
- 50
- Citations by year
- 2026: 6 · 2025: 12 · 2024: 8 · 2023: 10 · 2022: 18 · 2021: 5 · 2020: 2
- FWCI
- 3.44
- Type
- journal article
- Language
- en
- Open access
- green
- doi.org
- Licence
- Locations
- IEEE Journal of Biomedical and Health Informatics (publishedVersion)
- medRxiv (acceptedVersion)
- PubMed (publishedVersion)
- Utrecht University Repository (Utrecht University) (submittedVersion, other-oa)
- UCL Discovery (University College London) (submittedVersion)
- Funders
- National Institute for Health Research
- NIHR Health Informatics Collaborative
- Institute of Health Informatics at University College London
- Health Data Research UK (LOND1)
- Medical Research Council
- Engineering and Physical Sciences Research Council
- Economic and Social Research Council
- Department of Health and Social Care
- Chief Scientist Office of the Scottish Government Health and Social Care Directorates
- Health and Social Care Research and Development Division
- Public Health Agency
- British Heart Foundation
- postdoctoral fellowship
- Division of Critical Care
- University College Hospital
- UCLH Biomedical Research Centre
- NIHR Biomedical Research Centre at South London
- South London and Maudsley NHS Foundation Trust
- Kings College London
- The BigData@Heart Consortium
- Innovative Medicines Initiative (116074)
- European Unions Horizon 2020
- European Federation of Pharmaceutical Industries and Associations
- National Institute for Health Research University College London Hospitals Biomedical Research Centre
- Health Research University College London Hospitals Biomedical Research Centre
- UCL Hospitals NIHR Biomedical Research Centre
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Biomedical Text Mining and Ontologies
- Keywords
- clinical trial recruitment, natural language processing, electronic health records, septic shock, organ dysfunction, clinical text, intensive care
- MeSH terms
- Adult, Clinical Trials as Topic, Computer Simulation, Critical Care, Female, Humans, Male, Natural Language Processing, Patient Selection, Data Mining, Electronic Health Records
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
