Randomized controlled trial of an automated problem list with improved sensitivity
The DOI matches its Crossref record.
Abstract
Purpose To improve the completeness and timeliness of an electronic problem list, we have developed a system using Natural Language Processing (NLP) to automatically extract potential medical problems from clinical, free-text documents; these problems are then proposed for inclusion in an electronic problem list management application. Methods A prospective randomized controlled evaluation of the Automatic Problem List (APL) system in an intensive care unit and in a cardiovascular surgery unit is reported here. A total of 247 patients were enrolled: 76 in an initial control phase and 171 in the randomized controlled trial that followed. During this latter phase, patients were randomly assigned to a control or an intervention group. All patients had their documents analyzed by the system, but the medical problems discovered were only proposed in the problem list for intervention patients. We measured the sensitivity, specificity, positive and negative predictive values, likelihood ratios and the timeliness of the problem lists. Results Our system significantly increased the sensitivity of the problem lists in the intensive care unit, from about 9% to 41%, and even 77% if problems automatically proposed but not acknowledged by users were also considered. Timeliness of addition of problems to the list was greatly improved, with a time between a problem's first mention in a clinical document and its addition to the problem list reduced from about 6 days to less than 2 days. No si…
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Significantly increased sensitivity of problem lists (from 9 to 41%)”
From Moving from bytes to bedside: a systematic review on the use of artificial intelligence in the intensive care unit (2021) · cites this paper as [31]
Citations
Cites
No references could be resolved yet.
Research works printed but not linked: 31
- Bayegan, 2002, The helpful patient record system: problem oriented and knowledge based, Proc. AMIA Symp., 36
- Clayton, 2003, Building a comprehensive clinical information system from components. The approach at Intermountain Health Care, Methods Inf. Med., 42, 1, 10.1055/s-0038-1634203 · DOI · Paper page
- Meystre, 2006, Natural language processing to extract medical problems from electronic clinical documents: performance evaluation, J Biomed. Inform., 39, 589, 10.1016/j.jbi.2005.11.004 · DOI · Paper page
- Meystre, 2005, Automation of a problem list using natural language processing, BMC Med. Inform. Decis. Mak., 5, 30, 10.1186/1472-6947-5-30 · DOI · Paper page
- Weed, 1968, Medical records that guide and teach, N. Engl. J. Med., 278, 593, 10.1056/NEJM196803142781105 · DOI · Paper page
- Weed, 1968, Medical records that guide and teach, N. Engl. J. Med., 278, 652, 10.1056/NEJM196803212781204 · DOI · Paper page
- Campbell, 1998, Strategies for problem list implementation in a complex clinical enterprise, Proc. AMIA Symp., 285
- Campbell, 1994, A comparison of four schemes for codification of problem lists, Proc. Annu. Symp. Comput. Appl. Med. Care, 201
- Donaldson, 1985, Improving the master problem list: a case study in changing clinician behavior, QRB Qual. Rev. Bull., 11, 327
- Elkin, 1997, Standardized problem list generation, utilizing the Mayo canonical vocabulary embedded within the Unified Medical Language System, Proc. AMIA Annu. Fall Symp., 500
- Goldberg, 1998, An evaluation of UMLS as a controlled terminology for the Problem List Toolkit, Medinfo, 9, 609
- Hales, 1998, Extracting medical knowledge for a coded problem list vocabulary from the UMLS Knowledge Sources, Proc. AMIA Symp., 275
- Starmer, 1998, Development of a structured problem list management system at vanderbilt, Proc. AMIA Annu. Fall Symp., 1083
- Simborg, 1976, Information factors affecting problem follow-up in ambulatory care, Med. Care, 14, 848, 10.1097/00005650-197610000-00005 · doi:10.1097/00005650-197610000-00005
- Starfield, 1979, Concordance between medical records and observations regarding information on coordination of care, Med. Care, 17, 758, 10.1097/00005650-197907000-00006 · doi:10.1097/00005650-197907000-00006
- Institute of Medicine (U.S.), Committee on Improving the Patient Record, R.S. Dick, E.B. Steen, D.E. Detmer, The Computer-based Patient Record: An Essential Technology for Health Care, Rev. ed., National Academy Press, Washington, DC, 1997.
- van Ginneken, 2002, The computerized patient record: balancing effort and benefit, Int. J. Med. Inform., 65, 97, 10.1016/S1386-5056(02)00007-2 · DOI
- A.A. Bui, R.K. Taira, S. El-Saden, A. Dordoni, D.R. Aberle, Automated medical problem list generation: towards a patient timeline, Medinfo, San Francisco, CA, 2004, pp. 587–591.
- Scherpbier, 1994, A simple approach to physician entry of patient problem list, Proc. Annu. Symp. Comput. Appl. Med. Care, 206
- Wasserman, 2003, An applied evaluation of SNOMED CT as a clinical vocabulary for the computerized diagnosis and problem list, Proc. AMIA Symp., 699
- Payne, 1993, How useful is the UMLS metathesaurus in developing a controlled vocabulary for an automated problem list?, Proc. Annu. Symp. Comput. Appl. Med. Care, 705
- Zelingher, 1995, Categorization of free-text problem lists: an effective method of capturing clinical data, Proc. Annu. Symp. Comput. Appl. Med. Care, 416
- Warren, 1998, Just-in-time coding of the problem list in a clinical environment, Proc. AMIA Symp., 280
- Wang, 2003, Automated coded ambulatory problem lists: evaluation of a vocabulary and a data entry tool, Int. J. Med. Inform., 72, 17, 10.1016/j.ijmedinf.2003.08.002 · DOI
- Pratt, 1973, Medicine, computers, and linguistics, Adv. Biomed. Eng., 3, 97, 10.1016/B978-0-12-004903-5.50007-8 · DOI
- Spyns, 1996, Natural language processing in medicine: an overview, Methods Inf. Med., 35, 285
- Reichert, 2002, Using LOINC to link an EMR to the pertinent paragraph in a structured reference knowledge base, Proc. AMIA Symp., 652
- Aronson, 2001, Effective mapping of biomedical text to the UMLS Metathesaurus: the MetaMap program, Proc. AMIA Symp., 17
- Aronson, 2000, The NLM indexing initiative, Proc. AMIA Symp., 17
- Ashton, 1999, An empirical assessment of the validity of explicit and implicit process-of-care criteria for quality assessment, Med. Care, 37, 798, 10.1097/00005650-199908000-00009 · DOI · Paper page
- Szeto, 2002, Accuracy of computerized outpatient diagnoses in a Veterans Affairs general medicine clinic, Am. J. Manage. Care, 8, 37
Cited by
van de Sande, D., van Genderen, M. E., Huiskens, J., Gommers, D., & van Bommel, J. (2021) · Intensive Care Medicine · doi:10.1007/s00134-021-06446-7 · Cited by 279
(2017) · PubMed Central · Cited by 12
Metadata
- Authors
- Meystre, S. M., & Haug, P. J.
- Venue
- International Journal of Medical Informatics
- Publisher
- Elsevier BV
- Published
- 2008-09
- Volume
- 77
- Issue
- 9
- Pages
- 602-612
- ISSN
- 1386-5056
- Cited by
- 33
- References
- 0
- Language
- en
- Licence
- www.elsevier.com/tdm/userlicense/1.0/ (tdm)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Randomized controlled trial of an automated problem list with improved sensitivity. https://app.cochat.ai/papers/10.1016/j.ijmedinf.2007.12.001
- The paper
- Meystre, S. M., & Haug, P. J. (2008). Randomized controlled trial of an automated problem list with improved sensitivity. International Journal of Medical Informatics. https://doi.org/10.1016/j.ijmedinf.2007.12.001
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.