Using electronic medical records to enable large-scale studies in psychiatry: treatment resistant depression as a model
Open accessAbstract
BACKGROUND: Electronic medical records (EMR) provide a unique opportunity for efficient, large-scale clinical investigation in psychiatry. However, such studies will require development of tools to define treatment outcome. METHOD: Natural language processing (NLP) was applied to classify notes from 127 504 patients with a billing diagnosis of major depressive disorder, drawn from out-patient psychiatry practices affiliated with multiple, large New England hospitals. Classifications were compared with results using billing data (ICD-9 codes) alone and to a clinical gold standard based on chart review by a panel of senior clinicians. These cross-sectional classifications were then used to define longitudinal treatment outcomes, which were compared with a clinician-rated gold standard. RESULTS: Models incorporating NLP were superior to those relying on billing data alone for classifying current mood state (area under receiver operating characteristic curve of 0.85-0.88 v. 0.54-0.55). When these cross-sectional visits were integrated to define longitudinal outcomes and incorporate treatment data, 15% of the cohort remitted with a single antidepressant treatment, while 13% were identified as failing to remit despite at least two antidepressant trials. Non-remitting patients were more likely to be non-Caucasian (p<0.001). CONCLUSIONS: The application of bioinformatics tools such as NLP should enable accurate and efficient determination of longitudinal outcomes, enabling existing EMR data to be applied to clinical research, including biomarker investigations. Continued development will be required to better address moderators of outcome such as adherence and co-morbidity.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The authors classified individual visits as depressed (v. well) with a positive predictive value of 78%.
“The authors classified individual visits as depressed (v. well) with a positive predictive value of 78%.”
From Irremediability in psychiatric euthanasia: examining the objective standard (2023) · cites this paper as Perlis et al., 2012
The Results describe classifiers for individual visits as depressed, well, or intermediate. The Method says precision was estimated, but the passages in our copy do not report a 78% positive predictive value; the Results instead report AUC and sensitivity figures. The visit-classification part is supported, but the claimed 78% value is not verified. (LLM, conf=0.85)
Experimental — accuracy not yet measuredA third set of two studies used patient records to predict treatment-resistance.
“A third set of two studies used patient records to predict treatment-resistance (Cepeda et al., 2018; Perlis et al., 2012).”
From Irremediability in psychiatric euthanasia: examining the objective standard (2023) · cites this paper as Perlis et al., 2012
The Methods (TRD longitudinal classification; page unavailable) describe an algorithm that used electronic patient records to classify patients as treatment-resistant or treatment-responsive. The study identified treatment resistance from recorded visits and prescriptions; it did not show that records predicted future treatment resistance. The other study named in the citing sentence is outside this verdict. (LLM, conf=0.84)
Experimental — accuracy not yet measuredAgreement between the model's predictions of treatment-resistance and the opinion of a board of expert clinicians was 76.4%.
“Agreement between the model's predictions of treatment-resistance and the opinion of a board of expert clinicians was 76.4%.”
From Irremediability in psychiatric euthanasia: examining the objective standard (2023) · cites this paper as Perlis et al., 2012
The Results report 0.764 concordance with a clinician gold standard, equivalent to 76.4%. But the Methods describe that standard as psychiatrists’ ratings of mood state at individual visits, not their opinions on treatment resistance; treatment data were not considered at that stage. The claim misidentifies what the clinicians rated. (LLM, conf=0.85)
Experimental — accuracy not yet measuredA second study used natural language processing of 5198 patient records to develop a model predicting whether patients were depressed on every given visit following an antidepressant prescription.
“A second study used natural language processing of 5198 patient records to develop a model predicting whether patients were depressed on every given visit following an antidepressant prescription (Perlis et al., 2012).”
From Irremediability in psychiatric euthanasia: examining the objective standard (2023) · cites this paper as Perlis et al., 2012
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
The citation lists haven’t been fetched yet
References 33 · Cited by 193
Research works printed but not linked: 23
- Bates DW Evans RS Murff H Stetson PD Pizziferri L Hripcsak G Detecting adverse events using information technology Journal of the American Medical Informatics Association 2003 10 115 128 12595401 PMC150365
- Bunea F She Y Ombao H Gongvatana A Devlin K Cohen R Penalized least squares regression methods and applications to neuroimaging Neuroimage 2011 55 1519 1527 21167288 PMC5485905
- Charlson ME Pompei P Ales KL MacKenzie CR A new method of classifying prognostic comorbidity in longitudinal studies: development and validation Journal of Chronic Diseases 1987 40 373 383 3558716
- Effler P Ching-Lee M Bogard A Ieong MC Nekomoto T Jernigan D Statewide system of electronic notifiable disease reporting from clinical laboratories: comparing automated reporting with conventional methods Journal of the American Medical Association 1999 282 1845 1850 10573276
- Fava M Rush AJ Current status of augmentation and combination treatments for major depressive disorder: a literature review and a proposal for a novel approach to improve practice Psychotherapy and Psychosomatics 2006 75 139 153 16636629
- Garfield DA Rapp C Evens M Natural language processing in psychiatry. Artificial intelligence technology and psychopathology Journal of Nervous and Mental Disease 1992 180 227 237 1556562
- Gibson TB Jing Y Smith Carls G Kim E Bagalman JE Burton WN Tran QV Pikalov A Goetzel RZ Cost burden of treatment resistance in patients with depression American Journal of Managed Care 2010 16 370 377 20469957
- Jakobsen K Hansen T Dam H Larsen E Gether U Werge T Reliability of clinical ICD-10 diagnoses among electroconvulsive therapy patients with chronic affective disorders European Journal of Psychiatry 2008 22 167 172
- Klompas M Haney G Church D Lazarus R Hou X Platt R Automated identification of acute hepatitis B using electronic medical record data to facilitate public health surveillance PLoS One 2008 3 e2626 18612462 PMC2440348
- Lazarus R Klompas M Campion FX McNabb SJ Hou X Daniel J Haney G DeMaria A Lenert L Platt R Electronic support for public health: validated case finding and reporting for notifiable diseases using electronic medical data Journal of the American Medical Informatics Association 2009 16 18 24 18952940 PMC2605594
- Levin MA Krol M Doshi AM Reich DL Extraction and mapping of drug names from free text to a standardized nomenclature 2007 438 442 AMIA Annual Symposium Proceedings PMC2655777 18693874
- Meystre S Haug P Improving the sensitivity of the problem list in an intensive care unit by using natural language processing 2006a 554 558 AMIA Annual Symposium Proceedings PMC1839473 17238402
- Meystre S Haug PJ Natural language processing to extract medical problems from electronic clinical documents: performance evaluation Journal of Biomedical Informatics 2006b 39 589 599 16359928
- Murphy SN Mendis ME Hackett K Kuttan R Pan W Phillips L Gainer VS Berkowicz D Glaser J Kohane IS Chueh H Architecture of the open-source clinical research chart from informatics for integrating biology and the bedside 2007 548 552 AMIA Annual Symposium Proceedings PMC2655844 18693896
- Papakostas GI Petersen T Pava J Masson E Worthington JJ 3rd Alpert JE Fava M Nierenberg AA Hopelessness and suicidal ideation in outpatients with treatment-resistant depression: prevalence and impact on treatment outcome Journal of Nervous and Mental Disease 2003 191 444 449 12891091
- Penz JF Wilcox AB Hurdle JF Automated identification of adverse events related to central venous catheters Journal of Biomedical Informatics 2007 40 174 182 16901760
- Pestian JP Matykiewicz P Grupp-Phelan J Lavanier SA Combs J Kowatch R Using natural language processing to classify suicide notes 2008 Annual Symposium Proceedings of the American Medical Informatics Association 6 November 2008 Abstract 1091 19006447
- Rush AJ Thase ME Dube S Research issues in the study of difficult-to-treat depression Biological Psychiatry 2003a 53 743 753 12706958
- Rush AJ Trivedi MH Ibrahim HM Carmody TJ Arnow B Klein DN Markowitz JC Ninan PT Kornstein S Manber R Thase ME Kocsis JH Keller MB The 16-Item Quick Inventory of Depressive Symptomatology (QIDS), clinician rating (QIDS-C), and self-report (QIDS-SR): a psychometric evaluation in patients with chronic major depression Biological Psychiatry 2003b 54 573 583 12946886
- Solti I Aaronson B Fletcher G Solti M Gennari JH Cooper M Payne T Building an automated problem list based on natural language processing: lessons learned in the early phase of development 2008 687 691 AMIA Annual Symposium Proceedings PMC2655946 18999050
- Trivedi MH Fava M Wisniewski SR Thase ME Quitkin F Warden D Ritz L Nierenberg AA Lebowitz BD Biggs MM Luther JF Shores-Wilson K Rush AJ Medication augmentation after the failure of SSRIs for depression New England Journal of Medicine 2006 354 1243 1252 16554526
- Turchin A Morin L Semere LG Kashyap V Palchuk MB Shubina M Chang F Li Q Comparative evaluation of accuracy of extraction of medication information from narrative physician notes by commercial and academic natural language processing software packages 2006 789 793 AMIA Annual Symposium Proceedings 17238449
- Zeng QT Goryachev S Weiss S Sordo M Murphy SN Lazarus R Extracting principal diagnosis, comorbidity and smoking status for asthma research: evaluation of a natural language processing system BMC Medical Informatics and Decision Making 2006 6 30 16872495 PMC1553439
- Authors
- Perlis, R. H., Iosifescu, D. V., Castro, V. M., Murphy, S. N., Gainer, V. S., Minnier, J., Cai, T., Goryachev, S., Zeng, Q., Gallagher, P. J., Fava, M., Weilburg, J. B., Churchill, S. E., Kohane, I. S., & Smoller, J. W.
- Affiliations
- Massachusetts General Hospital (US)
- Mount Sinai Hospital (US)
- Harvard University (US)
- Brigham and Women's Hospital (US)
- Corresponding author
- Perlis, R. H.
- ORCID
- Venue
- Psychological Medicine
- Publisher
- Cambridge University Press (CUP)
- Published
- 2011-06-20
- Volume
- 42
- Issue
- 1
- Pages
- 41-50
- ISSN
- 0033-2917 (ISSN-L)
- 1469-8978
- PMC
- PMC3837420
- OpenAlex
- W2108844216
- Cited by
- 193
- References
- 33
- Citations by year
- 2026: 8 · 2025: 13 · 2024: 11 · 2023: 11 · 2022: 8 · 2021: 13 · 2020: 14 · 2019: 13 · 2018: 17 · 2017: 22 · 2016: 15 · 2015: 13 · 2014: 16 · 2013: 13 · 2012: 6
- FWCI
- 8.20
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- Psychological Medicine (publishedVersion)
- PubMed (publishedVersion)
- Europe PMC (PubMed Central) (acceptedVersion)
- PubMed Central (acceptedVersion)
- Funders
- Bristol-Myers Squibb
- Eli Lilly and Company
- National Institute of Mental Health (R01MH085542, R01MH086026)
- U.S. National Library of Medicine (U54LM008748)
- Topics
- Health Sciences › Medicine › Pharmacology › Treatment of Major Depression
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Keywords
- electronic health records, natural language processing, treatment-resistant depression, major depressive disorder, antidepressant response, clinical notes, ICD-9 codes, antidepressant trials, racial differences in treatment response
- MeSH terms
- Adult, Algorithms, Ambulatory Care, Cross-Sectional Studies, Female, Humans, Longitudinal Studies, Male, Middle Aged, Models, Theoretical, Natural Language Processing, New England, Psychiatry, ROC Curve, Logistic Models, Outcome Assessment, Health Care, Biomedical Research, International Classification of Diseases, Electronic Health Records, Depressive Disorder, Treatment-Resistant
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
