CoChat

Evaluation of an artificial intelligence-based medical device for diagnosis of autism spectrum disorder

Open access
Megerian, Jonathan T., Dey, Sangeeta, Melmed, Raun D., Coury, Daniel L., Lerner, Marc, Nicholls, Christopher J., Sohl, Kristin, Rouhbakhsh, Rambod, Narasimhan, Anandhi, Romain, Jonathan, Golla, Sailaja, Shareef, Safiullah, Ostrovsky, Andrey, Shannon, Jennifer, Kraft, Colleen, Liu-Mayo, Stuart, Abbas, Halim, Gal-Szabo, Diana E., Wall, Dennis P., Taraman, Sharief · 2022 · npj Digital Medicine
Cited by 111

The DOI matches its Crossref record; 18 of the 18 references it cites were found in a registry or an index.

Abstract

Autism spectrum disorder (ASD) can be reliably diagnosed at 18 months, yet significant diagnostic delays persist in the United States. This double-blinded, multi-site, prospective, active comparator cohort study tested the accuracy of an artificial intelligence-based Software as a Medical Device designed to aid primary care healthcare providers (HCPs) in diagnosing ASD. The Device combines behavioral features from three distinct inputs (a caregiver questionnaire, analysis of two short home videos, and an HCP questionnaire) in a gradient boosted decision tree machine learning algorithm to produce either an ASD positive, ASD negative, or indeterminate output. This study compared Device outputs to diagnostic agreement by two or more independent specialists in a cohort of 18–72-month-olds with developmental delay concerns (425 study completers, 36% female, 29% ASD prevalence). Device output PPV for all study completers was 80.8% (95% confidence intervals (CI), 70.3%–88.8%) and NPV was 98.3% (90.6%–100%). For the 31.8% of participants who received a determinate output (ASD positive or negative) Device sensitivity was 98.4% (91.6%–100%) and specificity was 78.9% (67.6%–87.7%). The Device’s indeterminate output acts as a risk control measure when inputs are insufficiently granular to make a determinate recommendation with confidence. If this risk control measure were removed, the sensitivity for all study completers would fall to 51.6% (63/122) (95% CI 42.4%, 60.8%), and specificity would fall to 18.5% (56/303) (95% CI 14.3%, 23.3%). Among participants for whom the Device abstained from providing a result, specialists identified that 91% had one or more complex neurodevelopmental disorders. No significant differences in Device performance were found across participants’ sex, race/ethnicity, income, or education level. For nearly a third of this primary care sample, the Device enabled timely diagnostic evaluation with a high degree of accuracy. The Device shows promise to significantly increase the number of children able to be diagnosed with ASD in a primary care setting, potentially facilitating earlier intervention and more efficient use of specialist resources.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1038/s41746-022-00598-6

Cites

18 of 18 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 18 references

Research works printed but not linked: 35

  1. Mazurek, M. O., Kanne, S. M. & Miles, J. H. Predicting improvement in social–communication symptoms of autism spectrum disorders using retrospective treatment data. Res. Autism Spectr. Disord. 6, 535–545 (2012). · DOI
  2. MacDonald, R., Parry-Cruwys, D., Dupere, S. & Ahearn, W. Assessing progress and outcome of early intensive behavioral intervention for toddlers with autism. Res. Dev. Disabil. 35, 3632–3644 (2014). · DOI · Paper page
  3. Ben Itzchak, E. & Zachor, D. A. Who benefits from early intervention in autism spectrum disorders? Res. Autism Spectr. Disord. 5, 345–350 (2011). · DOI · Paper page
  4. Flanagan, H. E., Perry, A. & Freeman, N. L. Effectiveness of large-scale community-based intensive behavioral intervention: a waitlist comparison study exploring outcomes and predictors. Res. Autism Spectr. Disord. 6, 673–682 (2012). · DOI
  5. Smith, T., Klorman, R. & Mruzek, D. W. Predicting outcome of community-based early intensive behavioral intervention for children with autism. J. Abnorm. Child Psychol. 43, 1271–1282 (2015). · DOI
  6. van’t Hof, M. et al. Age at autism spectrum disorder diagnosis: a systematic review and meta-analysis from 2012 to 2019. Autism 25, 862–873 (2021). · DOI · Paper page
  7. Oswald, D. P., Haworth, S. M., Mackenzie, B. K. & Willis, J. H. Parental report of the diagnostic process and outcome: ASD compared with other developmental disabilities. Focus Autism Other Dev. Disabil. 32, 152–160 (2017). · DOI · Paper page
  8. Wiggins, L. D. et al. Disparities in documented diagnoses of autism spectrum disorder based on demographic, individual, and service factors. Autism Res. 13, 464–473 (2020). · DOI · Paper page
  9. Gordon-Lipkin, E., Foster, J. & Peacock, G. Whittling down the wait time: exploring models to minimize the delay from initial concern to diagnosis and treatment of autism spectrum disorder. Pediatr. Clin. 63, 851–859 (2016).
  10. Bridgemohan, C. et al. A workforce survey on developmental-behavioral pediatrics. Pediatrics 141, e20172164 (2018). · DOI
  11. Falkmer, T., Anderson, K., Falkmer, M. & Horlin, C. Diagnostic procedures in autism spectrum disorders: a systematic literature review. Eur. Child Adolesc. Psychiatry 22, 329–340 (2013). · DOI · Paper page
  12. Rhoades, R. A., Scarpa, A. & Salley, B. The importance of physician knowledge of autism spectrum disorder: results of a parent survey. BMC Pediatrics 7, 37 (2007). · DOI · Paper page
  13. Carbone, P. S., Norlin, C. & Young, P. C. Improving early identification and ongoing care of children with autism spectrum disorder. Pediatrics 137, e20151850 (2016). · DOI
  14. Fenikilé, T. S., Ellerbeck, K., Filippi, M. K. & Daley, C. M. Barriers to autism screening in family medicine practice: a qualitative study. Prim. Health Care Res. Dev. 16, 356–366 (2015). · DOI · Paper page
  15. Self, T. L., Parham, D. F. & Rajagopalan, J. Autism spectrum disorder early screening practices: a survey of physicians. Commun. Disord. Q. 36, 195–207 (2015). · DOI · Paper page
  16. Nazneen, N. et al. Supporting parents for in-home capture of problem behaviors of children with developmental disabilities. Personal. Ubiquitous Comput. 16, 193–207 (2012). · DOI
  17. Lord, C., Rutter, M. & Le Couteur, A. Autism diagnostic interview-revised: a revised version of a diagnostic interview for caregivers of individuals with possible pervasive developmental disorders. J. Autism Dev. Disord. 24, 659–685 (1994). · DOI · Paper page
  18. Cicchetti, D. V., Lord, C., Koenig, K., Klin, A. & Volkmar, F. R. Reliability of the ADI-R: Multiple examiners evaluate a single case. J. Autism Dev. Disord. 38, 764–770 (2008). · DOI
  19. Hill, A. et al. Stability and interpersonal agreement of the interview-based diagnosis of autism. Psychopathology 34, 187–191 (2001). · DOI · Paper page
  20. Randall, M. et al. Diagnostic tests for autism spectrum disorder (ASD) in preschool children. Cochrane Database Syst. Rev. (2018). · DOI · Paper page
  21. Kaufman, N. K. Rethinking “gold standards” and “best practices” in the assessment of autism. Appl. Neuropsychol. Child 1–12 https://doi.org/10.1080/21622965.2020.1809414 (2020). · DOI
  22. Hyde, K. K. et al. Applications of supervised machine learning in autism spectrum disorder research: a review. Rev. J. Autism Dev. Disord. 6, 128–146 (2019). · DOI · Paper page
  23. Abbas, H., Garberson, F., Liu-Mayo, S., Glover, E. & Wall, D. P. Multi-modular AI approach to streamline autism diagnosis in young children. Sci. Rep. 10, 1–8 (2020). · DOI
  24. Abbas, H., Garberson, F., Glover, E. & Wall, D. P. Machine learning approach for early detection of autism by combining questionnaire and home video screening. J. Am. Med. Inform. Assoc. 25, 1000–1007 (2018). · DOI · Paper page
  25. Levy, S., Duda, M., Haber, N. & Wall, D. P. Sparsifying machine learning models identify stable subsets of predictive features for behavioral detection of autism. Mol. Autism 8, 1–17 (2017). · DOI
  26. Tariq, Q. et al. Mobile detection of autism through machine learning on home video: a development and prospective validation study. PloS Med. 15, e1002705 (2018). · DOI · Paper page
  27. Wall, D. P., Dally, R., Luyster, R., Jung, J.-Y. & DeLuca, T. F. Use of artificial intelligence to shorten the behavioral diagnosis of autism. 7, e43855 (2012). · DOI · Paper page
  28. Kosmicki, J. A., Sochat, V., Duda, M. & Wall, D. P. Searching for a minimal set of behaviors for autism detection through feature selection-based machine learning. Transl. Psychiatry 5, e514 (2015). · DOI · Paper page
  29. Cortes, C., DeSalvo, G., Gentile, C., Mohri, M. & Yang, S. Online learning with abstention. International conference on machine learning 1059–1067 (2018).
  30. Yingling, M. E., Hock, R. M. & Bell, B. A. Time-lag between diagnosis of autism spectrum disorder and onset of publicly-funded early intensive behavioral intervention: do race–ethnicity and neighborhood matter? J. Autism Dev. Disord. 48, 561–571 (2018). · DOI
  31. Ros-Demarize, R. et al. ASD symptoms in toddlers and preschoolers: an examination of sex differences. Autism Res. 13, 157–166 (2020). · DOI
  32. Hus, Y. & Segal, O. Challenges surrounding the diagnosis of autism in children. Neuropsychiatr. Dis. Treat. 17, 3509 (2021). · DOI · Paper page
  33. Liao, X. & Li, Y. Economic burdens on parents of children with autism: a literature review. CNS Spectr. 25, 468–474 (2020). · DOI
  34. Westman Andersson, G., Miniscalco, C. & Gillberg, C. Autism in preschoolers: does individual clinician’s first visit diagnosis agree with final comprehensive diagnosis? Sci. World J. 2013, 716267 (2013). · DOI
  35. McDonnell, C. G. et al. When are we sure? predictors of clinician certainty in the diagnosis of autism spectrum disorder. J. Autism Dev. Disord. 49, 1391–1401 (2019). · DOI

Cited by

See all 111

Metadata

Authors
Megerian, J. T., Dey, S., Melmed, R. D., Coury, D. L., Lerner, M., Nicholls, C. J., Sohl, K., Rouhbakhsh, R., Narasimhan, A., Romain, J., Golla, S., Shareef, S., Ostrovsky, A., Shannon, J., Kraft, C., Liu-Mayo, S., Abbas, H., Gal-Szabo, D. E., Wall, D. P., & Taraman, S.
Venue
npj Digital Medicine
Published
2022-05-05
Volume
5
Issue
1
Pages
57-57
ISSN
2398-6352 (ISSN-L)
OpenAlex
W4229007573
Cited by
111
References
18
FWCI
11.02
Type
journal article
Language
en
Open access
gold
Licence

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Evaluation of an artificial intelligence-based medical device for diagnosis of autism spectrum disorder. https://app.cochat.ai/papers/10.1038/s41746-022-00598-6
The paper
Megerian, J. T., Dey, S., Melmed, R. D., Coury, D. L., Lerner, M., Nicholls, C. J., Sohl, K., Rouhbakhsh, R., Narasimhan, A., Romain, J., Golla, S., Shareef, S., Ostrovsky, A., Shannon, J., Kraft, C., Liu-Mayo, S., Abbas, H., Gal-Szabo, D. E., Wall, D. P., & Taraman, S. (2022). Evaluation of an artificial intelligence-based medical device for diagnosis of autism spectrum disorder. npj Digital Medicine. https://doi.org/10.1038/s41746-022-00598-6

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.