CoChat
Sign Up
Cited by 61Machine Learning in HealthcareEpilepsy research and treatmentArtificial Intelligence in Healthcare and Education

Prospective validation of a machine learning model that uses provider notes to identify candidates for resective epilepsy surgery

Open access
Wissel, Benjamin D., Greiner, Hansel M., Glauser, Tracy A., Holland‐Bouley, Katherine D., Mangano, Francesco T., Santel, Daniel, Faist, Robert, Zhang, Nanhua +3 more·2020·Epilepsia
DOI 10.1111/epi.16398PMC PMC6980264
Open this paper in CoChat — free account

Abstract

Objective: Delay to resective epilepsy surgery results in avoidable disease burden and increased risk of mortality. The objective was to prospectively validate a natural language processing (NLP) application that uses provider notes to assign epilepsy surgery candidacy scores. Methods: The application was trained on notes from (1) patients with a diagnosis of epilepsy and a history of resective epilepsy surgery and (2) patients who were seizure‐free without surgery. The testing set included all patients with unknown surgical candidacy status and an upcoming neurology visit. Training and testing sets were updated weekly for 1 year. One‐ to three‐word phrases contained in patients’ notes were used as features. Patients prospectively identified by the application as candidates for surgery were manually reviewed by two epileptologists. Performance metrics were defined by comparing NLP‐derived surgical candidacy scores with surgical candidacy status from expert chart review. Results: The training set was updated weekly and included notes from a mean of 519 ± 67 patients. The area under the receiver operating characteristic curve (AUC) from 10‐fold cross‐validation was 0.90 ± 0.04 (range = 0.83‐0.96) and improved by 0.002 per week (P < .001) as new patients were added to the training set. Of the 6395 patients who visited the neurology clinic, 4211 (67%) were evaluated by the model. The prospective AUC on this test set was 0.79 (95% confidence interval [CI] = 0.62‐0.96). Using the optimal surgical candidacy score threshold, sensitivity was 0.80 (95% CI = 0.29‐0.99), specificity was 0.77 (95% CI = 0.64‐0.88), positive predictive value was 0.25 (95% CI = 0.07‐0.52), and negative predictive value was 0.98 (95% CI = 0.87‐1.00). The number needed to screen was 5.6. Significance: An electronic health record–integrated NLP application can accurately assign surgical candidacy scores to patients in a clinical setting.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1111/epi.16398

Cites

2 of 2 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 44

  1. [2]Begley CE, Famulari M, Annegers JF, The cost of epilepsy in the United States: an estimate from population-based clinical and survey data. Epilepsia 2000; 41: 342– 351. 10714408
  2. [3]Ryvlin P, Rheims S. Epilepsy surgery: eligibility criteria and presurgical evaluation. Dialogues Clin Neurosci 2008; 10: 91– 103. 18472487 PMC3181859
  3. [4]Wiebe S, Blume WT, Girvin JP, A Randomized, Controlled Trial of Surgery for Temporal-Lobe Epilepsy. N Engl J Med 2001; 345: 311– 318. 11484687
  4. [5]Wyllie E, Comair YG, Kotagal P, Seizure outcome after epilepsy surgery in children and adolescents. Ann Neurol 1998; 44: 740– 748. 9818929
  5. [6]Dwivedi R, Ramanujam B, Chandra PS, Surgery for drug-resistant epilepsy in children. N Engl J Med 2017; 377: 1639– 1647. 29069568
  6. [7]Moseley BD, Nickels K, Wirrell EC. Surgical outcomes for intractable epilepsy in children with epileptic spasms. J Child Neurol 2012; 27: 713– 720. 22123426
  7. [8]Paolicchi JM, Jayakar P, Dean P, Predictors of outcome in pediatric epilepsy surgery. Neurology 2000; 54: 642– 647. 10680797
  8. [9]Wyllie E, Comair YG, Kotagal P, Seizure outcome after epilepsy surgery in children and adolescents. Ann Neurol 1998; 44: 740– 748. 9818929
  9. [10]Engel J, McDermott MP, Wiebe S, Early surgical therapy for drug-resistant temporal lobe epilepsy: A randomized trial. JAMA 2012; 307: 922– 930. 22396514 PMC4821633
  10. [11]Skirrow C, Cross JH, Cormack F, Long-term intellectual outcome after temporal lobe surgery in childhood. Neurology 2011; 76: 1330– 1337. 21482948 PMC3090063
  11. [12]Jobst BC, Cascino GD. Resective epilepsy surgery for drug-resistant focal epilepsy: A review. JAMA 2015; 313: 285– 293. 25602999
  12. [13]Engel J Jr, Wiebe S, French J, Practice parameter: temporal lobe and localized neocortical resections for epilepsy: report of the Quality Standards Subcommittee of the American Academy of Neurology, in association with the American Epilepsy Society and the American Association of Neurological Surgeons. Epilepsia 2003; 44: 741– 751. 12601090
  13. [14]Kwan P, Arzimanoglou A, Berg AT, Definition of drug resistant epilepsy: consensus proposal by the ad hoc Task Force of the ILAE Commission on Therapeutic Strategies. Epilepsia 2010; 51: 1069– 1077. 19889013
  14. [15]Lukmanji S, Altura KC, Rydenhag B, Accuracy of an online tool to assess appropriateness for an epilepsy surgery evaluation-A population-based Swedish study. Epilepsy Res 2018; 145: 140– 144. 30007238
  15. [16]Haneef Z, Stern J, Dewar S, Referral pattern for epilepsy surgery after evidence-based recommendations: A retrospective study(Patient Page). Neurology 2010; 75: 699– 704. 20733145 PMC2931651
  16. [17]Roberts JI, Hrazdil C, Wiebe S, Neurologists’ knowledge of and attitudes toward epilepsy surgery: A national survey. Neurology 2015; 84: 159– 166. 25503624 PMC4336084
  17. [18]Steinbrenner M, Kowski AB, Holtkamp M. Referral to evaluation for epilepsy surgery: Reluctance by epileptologists and patients. Epilepsia 2019; 60: 211– 219. 30653659
  18. [19]Roberts JI, Hrazdil C, Wiebe S, Feasibility of using an online tool to assess appropriateness for an epilepsy surgery evaluation. Neurology 2014; 83: 913– 919. 25107882 PMC4153845
  19. [20]Jette N, Quan H, Tellez-Zenteno JF, Development of an online tool to determine appropriateness for an epilepsy surgery evaluation. Neurology 2012; 79: 1084– 1093. 22895589
  20. [21]Kawamoto K, Houlihan CA, Balas EA, Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success. BMJ 2005; 330: 765. 15767266 PMC555881
  21. [22]Garg AX, Adhikari NKJ, McDonald H, Effects of Computerized Clinical Decision Support Systems on Practitioner Performance and Patient OutcomesA Systematic Review. JAMA 2005; 293: 1223– 1238. 15755945
  22. [23]Pestian J, Deleger L, Savova G, Natural Language Processing - The Basics In: Pediatric biomedical informatics computer: applications in pediatric research. Springer: New York; 2012.
  23. [24]Cohen KB, Glass B, Greiner HM, Methodological Issues in Predicting Pediatric Epilepsy Surgery Candidates Through Natural Language Processing and Machine Learning. Biomed Inform Insights 2016; 8: 11– 18. 27257386 PMC4876984
  24. [25]Matykiewicz P, Cohen K, Holland KD, Earlier identification of epilepsy surgery candidates using natural language processing. Proceedings of the 2013 Workshop on Biomedical Natural Language Processing 2013; 1– 9.
  25. [26]Wissel BD, Greiner HM, Glauser TA, Investigation of bias in an epilepsy machine learning algorithm trained on physician notes. Epilepsia 2019; 60: e93– e98. 31441044 PMC6731998
  26. [27]Moons KG, Altman DG, Reitsma JB, Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): explanation and elaboration. Ann Intern Med 2015; 162: W1– 73. 25560730
  27. [28]Jetté N, Reid AY, Quan H, How accurate is ICD coding for epilepsy? Epilepsia 2010; 51: 62– 69. 19682027
  28. [29]Pedregosa F, Varoquaux G, Gramfort A, Scikit-learn: Machine learning in Python. Journal of machine learning research 2011; 12: 2825– 2830.
  29. [30]Rembold CM. Number needed to screen: development of a statistic for disease screening. BMJ 1998; 317: 307– 312. 9685274 PMC28622
  30. [32]McHugh ML. Interrater reliability: the kappa statistic. Biochemia Medica 2012; 22: 276– 282. 23092060 PMC3900052
  31. [34]Barbour K, Hesdorffer DC, Tian N, Automated detection of sudden unexpected death in epilepsy risk factors in electronic medical records using natural language processing. Epilepsia 2019; 60: 1209– 1220. 31111463 PMC11771062
  32. [35]Roberts JI, Hrazdil C, Wiebe S, Neurologists' knowledge of and attitudes toward epilepsy surgery. Neurology 2014. PMC4336084 25503624
  33. [36]Berg AT, Levy SR, Testa FM, Remission of epilepsy after two drug failures in children: a prospective study. Annals of Neurology: Official Journal of the American Neurological Association and the Child Neurology Society 2009; 65: 510– 519. PMC2818062 19475671
  34. [37]Callaghan B, Schlesinger M, Rodemer W, Remission and relapse in a drug‐resistant epilepsy population followed prospectively. Epilepsia 2011; 52: 619– 626. 21269287 PMC3147304
  35. [38]Berg A, Langfitt J, Shinnar S, How long does it take for partial epilepsy to become intractable? Neurology 2003; 60: 186– 190. 12552028
  36. [39]Garg AX, Adhikari NK, McDonald H, Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: a systematic review. JAMA 2005; 293: 1223– 1238. 15755945
  37. [40]Cross JH, Jayakar P, Nordli D, Proposed criteria for referral and evaluation of children for epilepsy surgery: recommendations of the Subcommission for Pediatric Epilepsy Surgery. Epilepsia 2006; 47: 952– 959. 16822241
  38. [41]Ryvlin P, Cross JH, Rheims S. Epilepsy surgery in children and adults. The Lancet Neurology 2014; 13: 1114– 1126. 25316018
  39. [42]Fois C, Kovac S, Khalil A, Predictors for being offered epilepsy surgery: 5-year experience of a tertiary referral centre. J Neurol Neurosurg Psychiatry 2016; 87: 209– 211. 25935890
  40. [43]Connolly B, Matykiewicz P, Bretonnel Cohen K, Assessing the similarity of surface linguistic features related to epilepsy across pediatric hospitals. J Am Med Inform Assoc 2014; 21: 866– 870. 24692393 PMC4147613
  41. [44]LeCun Y, Bengio Y, Hinton G. Deep learning. Nature 2015; 521: 436– 444. 26017442
  42. [45]Hirschberg J, Manning CD. Advances in natural language processing. Science 2015; 349: 261– 266. 26185244
  43. [46]Pennington J, Socher R, Manning C. Glove: Global vectors for word representation. Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP) 2014; 1532– 1543.
  44. [47]Mikolov T, Sutskever I, Chen K, Distributed representations of words and phrases and their compositionality. Adv Neural Inf Process Syst 2013; 3111– 3119.

Cited by

Sorted by citations

Authors
Wissel, B. D., Greiner, H. M., Glauser, T. A., Holland‐Bouley, K. D., Mangano, F. T., Santel, D., Faist, R., Zhang, N., Pestian, J. P., Szczesniak, R. D., & Dexheimer, J. W.
Affiliations
Cincinnati Children's Hospital Medical Center (US)
Corresponding author
Dexheimer, Judith W.
ORCID
Venue
Epilepsia
Publisher
Wiley
Published
2019-11-29
Volume
61
Issue
1
Pages
39-48
ISSN
  • 0013-9580 (ISSN-L)
  • 1528-1167
DOI
10.1111/epi.16398
PMC
PMC6980264
OpenAlex
W2989633658
Cited by
61
References
2
Citations by year
2026: 6 · 2025: 8 · 2024: 15 · 2023: 9 · 2022: 12 · 2021: 9 · 2020: 2
FWCI
2.62
Type
journal article
Language
en
Open access
Licence
onlinelibrary.wiley.com/termsAndConditions#vor (vor)
Locations
  • Epilepsia (publishedVersion)
  • PubMed (publishedVersion)
  • Europe PMC (PubMed Central) (acceptedVersion)
  • PubMed Central (acceptedVersion)
Funders
  • National Heart, Lung, and Blood Institute (K25 HL12595)
  • Agency for Healthcare Research and Quality (R21 HS024977)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Health Sciences › Medicine › Psychiatry and Mental health › Epilepsy research and treatment
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
Keywords
resective epilepsy surgery, patient selection for surgery, natural language processing, electronic health records, epilepsy, clinical decision support, model validation
MeSH terms
Machine Learning, Adolescent, Adult, Child, Child, Preschool, Epilepsy, Female, Humans, Infant, Infant, Newborn, Male, Middle Aged, Natural Language Processing, Prospective Studies, Patient Selection, Decision Support Systems, Clinical, Young Adult, Electronic Health Records

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review