Evaluation of an artificial intelligence-based medical device for diagnosis of autism spectrum disorder
Open accessThe DOI matches its Crossref record; 18 of the 18 references it cites were found in a registry or an index.
Abstract
Autism spectrum disorder (ASD) can be reliably diagnosed at 18 months, yet significant diagnostic delays persist in the United States. This double-blinded, multi-site, prospective, active comparator cohort study tested the accuracy of an artificial intelligence-based Software as a Medical Device designed to aid primary care healthcare providers (HCPs) in diagnosing ASD. The Device combines behavioral features from three distinct inputs (a caregiver questionnaire, analysis of two short home videos, and an HCP questionnaire) in a gradient boosted decision tree machine learning algorithm to produce either an ASD positive, ASD negative, or indeterminate output. This study compared Device outputs to diagnostic agreement by two or more independent specialists in a cohort of 18–72-month-olds with developmental delay concerns (425 study completers, 36% female, 29% ASD prevalence). Device output PPV for all study completers was 80.8% (95% confidence intervals (CI), 70.3%–88.8%) and NPV was 98.3% (90.6%–100%). For the 31.8% of participants who received a determinate output (ASD positive or negative) Device sensitivity was 98.4% (91.6%–100%) and specificity was 78.9% (67.6%–87.7%). The Device’s indeterminate output acts as a risk control measure when inputs are insufficiently granular to make a determinate recommendation with confidence. If this risk control measure were removed, the sensitivity for all study completers would fall to 51.6% (63/122) (95% CI 42.4%, 60.8%), and specificity would fall to 18.5% (56/303) (95% CI 14.3%, 23.3%). Among participants for whom the Device abstained from providing a result, specialists identified that 91% had one or more complex neurodevelopmental disorders. No significant differences in Device performance were found across participants’ sex, race/ethnicity, income, or education level. For nearly a third of this primary care sample, the Device enabled timely diagnostic evaluation with a high degree of accuracy. The Device shows promise to significantly increase the number of children able to be diagnosed with ASD in a primary care setting, potentially facilitating earlier intervention and more efficient use of specialist resources.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Maenner, M. J., Shaw, K. A., Baio, J., EdS1, Washington, A., Patrick, M., DiRienzo, M., Christensen, D. L., Wiggins, L. D., Pettygrove, S., et al. (2020) · MMWR. Surveillance Summaries · doi:10.15585/mmwr.ss6904a1 · Cited by 3,780 · cited 4×
“Autism spectrum disorder (ASD) is one of the most common developmental disorders, with a US prevalence of 1.9% 1 .”
InfluentialDawson, G., Rogers, S., Munson, J., Smith, M., Winter, J., Greenson, J., Donaldson, A., & Varley, J. (2010) · Pediatrics · doi:10.1542/peds.2009-0958 · Cited by 2,835
Global Prevalence of Autism and Other Pervasive Developmental Disorders
Elsabbagh, M., Divan, G., Koh, Y., Kim, Y. S., Kauchali, S., Marcín, C., Montiel‐Nava, C., Patel, V., Paula, C. S., Wang, C., et al. (2012) · Autism Research · doi:10.1002/aur.239 · Cited by 2,636
Lai, M.-C., Kassee, C., Besney, R., Bonato, S., Hull, L., Mandy, W., Szatmari, P., & Ameis, S. H. (2019) · The Lancet Psychiatry · doi:10.1016/s2215-0366(19)30289-5 · Cited by 1,892
Maenner, M. J., Shaw, K. A., Bakian, A. V., Bilder, D. A., Durkin, M. S., Esler, A., Furnier, S. M., Hallas, L., Hall-Lande, J., Hudson, A., et al. (2021) · MMWR. Surveillance Summaries · doi:10.15585/mmwr.ss7011a1 · Cited by 1,818 · cited 1×
“Despite documented bene fi ts of early intervention, the mean age of ASD diagnosis in the US remains high at over 4 years 1,9 – 11 .”
Identification, Evaluation, and Management of Children With Autism Spectrum Disorder
Hyman, S. L., Levy, S. E., Myers, S. M., COUNCIL ON CHILDREN WITH DISABILITIES, S. O. D. A. B. P., Kuo, D. Z., Apkon, S., Davidson, L. F., Ellerbeck, K. A., Foster, J. E. A., Noritz, G. H., et al. (2020) · Pediatrics · doi:10.1542/peds.2019-3447 · Cited by 1,798
Shattuck, P. T., Durkin, M., Maenner, M., Newschaffer, C., Mandell, D. S., Wiggins, L., Lee, L.-C., Rice, C., Giarelli, E., Kirby, R., et al. (2009) · Journal of the American Academy of Child & Adolescent Psychiatry · doi:10.1097/chi.0b013e31819b3848 · Cited by 623 · cited 3×
“Roughly 27% of children with ASD remain undiagnosed at age 8 15 .”
Second opinion needed: Communicating uncertainty in medical machine learning
Kompa, B., Snoek, J., & Beam, A. L. (2021) · npj Digital Medicine · doi:10.1038/s41746-020-00367-3 · Cited by 514 · cited 3×
“The Device ’ s indeterminate output also referred to in the literature as an “ abstention ” or “ no result ” output, is a standard method of risk control in machine-learning algorithms 45,46 .”
Pierce, K., Gazestani, V. H., Bacon, E., Barnes, C. C., Cha, D., Nalabolu, S., Lopez, L., Moore, A., Pence-Stophaeros, S., & Courchesne, E. (2019) · JAMA Pediatrics · doi:10.1001/jamapediatrics.2019.0624 · Cited by 333 · cited 1×
“For example, the mean age of completers for whom the Device rendered an ASD positive result was 2.81 years, which is 1.5 years earlier than the current average age of diagnosis 1,9,10 .”
InfluentialTiming of the Diagnosis of Autism in African American Children
Constantino, J. N., Abbacchi, A. M., Saulnier, C., Klaiman, C., Mandell, D. S., Zhang, Y., Hawks, Z., Bates, J., Klin, A., Shattuck, P., et al. (2020) · Pediatrics · doi:10.1542/peds.2019-3629 · Cited by 315 · cited 1×
“This fi nding is important given that gender and racial/ethnic biases exist in the current standard of care, such that females, African American, and Hispanic children are less often diagnosed, misdiagnosed more often, and when diagnosed, are diagnosed later on average 47 – 50 .”
Research works printed but not linked: 35
- Mazurek, M. O., Kanne, S. M. & Miles, J. H. Predicting improvement in social–communication symptoms of autism spectrum disorders using retrospective treatment data. Res. Autism Spectr. Disord. 6, 535–545 (2012). · DOI
- MacDonald, R., Parry-Cruwys, D., Dupere, S. & Ahearn, W. Assessing progress and outcome of early intensive behavioral intervention for toddlers with autism. Res. Dev. Disabil. 35, 3632–3644 (2014). · DOI · Paper page
- Ben Itzchak, E. & Zachor, D. A. Who benefits from early intervention in autism spectrum disorders? Res. Autism Spectr. Disord. 5, 345–350 (2011). · DOI · Paper page
- Flanagan, H. E., Perry, A. & Freeman, N. L. Effectiveness of large-scale community-based intensive behavioral intervention: a waitlist comparison study exploring outcomes and predictors. Res. Autism Spectr. Disord. 6, 673–682 (2012). · DOI
- Smith, T., Klorman, R. & Mruzek, D. W. Predicting outcome of community-based early intensive behavioral intervention for children with autism. J. Abnorm. Child Psychol. 43, 1271–1282 (2015). · DOI
- van’t Hof, M. et al. Age at autism spectrum disorder diagnosis: a systematic review and meta-analysis from 2012 to 2019. Autism 25, 862–873 (2021). · DOI · Paper page
- Oswald, D. P., Haworth, S. M., Mackenzie, B. K. & Willis, J. H. Parental report of the diagnostic process and outcome: ASD compared with other developmental disabilities. Focus Autism Other Dev. Disabil. 32, 152–160 (2017). · DOI · Paper page
- Wiggins, L. D. et al. Disparities in documented diagnoses of autism spectrum disorder based on demographic, individual, and service factors. Autism Res. 13, 464–473 (2020). · DOI · Paper page
- Gordon-Lipkin, E., Foster, J. & Peacock, G. Whittling down the wait time: exploring models to minimize the delay from initial concern to diagnosis and treatment of autism spectrum disorder. Pediatr. Clin. 63, 851–859 (2016).
- Bridgemohan, C. et al. A workforce survey on developmental-behavioral pediatrics. Pediatrics 141, e20172164 (2018). · DOI
- Falkmer, T., Anderson, K., Falkmer, M. & Horlin, C. Diagnostic procedures in autism spectrum disorders: a systematic literature review. Eur. Child Adolesc. Psychiatry 22, 329–340 (2013). · DOI · Paper page
- Rhoades, R. A., Scarpa, A. & Salley, B. The importance of physician knowledge of autism spectrum disorder: results of a parent survey. BMC Pediatrics 7, 37 (2007). · DOI · Paper page
- Carbone, P. S., Norlin, C. & Young, P. C. Improving early identification and ongoing care of children with autism spectrum disorder. Pediatrics 137, e20151850 (2016). · DOI
- Fenikilé, T. S., Ellerbeck, K., Filippi, M. K. & Daley, C. M. Barriers to autism screening in family medicine practice: a qualitative study. Prim. Health Care Res. Dev. 16, 356–366 (2015). · DOI · Paper page
- Self, T. L., Parham, D. F. & Rajagopalan, J. Autism spectrum disorder early screening practices: a survey of physicians. Commun. Disord. Q. 36, 195–207 (2015). · DOI · Paper page
- Nazneen, N. et al. Supporting parents for in-home capture of problem behaviors of children with developmental disabilities. Personal. Ubiquitous Comput. 16, 193–207 (2012). · DOI
- Lord, C., Rutter, M. & Le Couteur, A. Autism diagnostic interview-revised: a revised version of a diagnostic interview for caregivers of individuals with possible pervasive developmental disorders. J. Autism Dev. Disord. 24, 659–685 (1994). · DOI · Paper page
- Cicchetti, D. V., Lord, C., Koenig, K., Klin, A. & Volkmar, F. R. Reliability of the ADI-R: Multiple examiners evaluate a single case. J. Autism Dev. Disord. 38, 764–770 (2008). · DOI
- Hill, A. et al. Stability and interpersonal agreement of the interview-based diagnosis of autism. Psychopathology 34, 187–191 (2001). · DOI · Paper page
- Randall, M. et al. Diagnostic tests for autism spectrum disorder (ASD) in preschool children. Cochrane Database Syst. Rev. (2018). · DOI · Paper page
- Kaufman, N. K. Rethinking “gold standards” and “best practices” in the assessment of autism. Appl. Neuropsychol. Child 1–12 https://doi.org/10.1080/21622965.2020.1809414 (2020). · DOI
- Hyde, K. K. et al. Applications of supervised machine learning in autism spectrum disorder research: a review. Rev. J. Autism Dev. Disord. 6, 128–146 (2019). · DOI · Paper page
- Abbas, H., Garberson, F., Liu-Mayo, S., Glover, E. & Wall, D. P. Multi-modular AI approach to streamline autism diagnosis in young children. Sci. Rep. 10, 1–8 (2020). · DOI
- Abbas, H., Garberson, F., Glover, E. & Wall, D. P. Machine learning approach for early detection of autism by combining questionnaire and home video screening. J. Am. Med. Inform. Assoc. 25, 1000–1007 (2018). · DOI · Paper page
- Levy, S., Duda, M., Haber, N. & Wall, D. P. Sparsifying machine learning models identify stable subsets of predictive features for behavioral detection of autism. Mol. Autism 8, 1–17 (2017). · DOI
- Tariq, Q. et al. Mobile detection of autism through machine learning on home video: a development and prospective validation study. PloS Med. 15, e1002705 (2018). · DOI · Paper page
- Wall, D. P., Dally, R., Luyster, R., Jung, J.-Y. & DeLuca, T. F. Use of artificial intelligence to shorten the behavioral diagnosis of autism. 7, e43855 (2012). · DOI · Paper page
- Kosmicki, J. A., Sochat, V., Duda, M. & Wall, D. P. Searching for a minimal set of behaviors for autism detection through feature selection-based machine learning. Transl. Psychiatry 5, e514 (2015). · DOI · Paper page
- Cortes, C., DeSalvo, G., Gentile, C., Mohri, M. & Yang, S. Online learning with abstention. International conference on machine learning 1059–1067 (2018).
- Yingling, M. E., Hock, R. M. & Bell, B. A. Time-lag between diagnosis of autism spectrum disorder and onset of publicly-funded early intensive behavioral intervention: do race–ethnicity and neighborhood matter? J. Autism Dev. Disord. 48, 561–571 (2018). · DOI
- Ros-Demarize, R. et al. ASD symptoms in toddlers and preschoolers: an examination of sex differences. Autism Res. 13, 157–166 (2020). · DOI
- Hus, Y. & Segal, O. Challenges surrounding the diagnosis of autism in children. Neuropsychiatr. Dis. Treat. 17, 3509 (2021). · DOI · Paper page
- Liao, X. & Li, Y. Economic burdens on parents of children with autism: a literature review. CNS Spectr. 25, 468–474 (2020). · DOI
- Westman Andersson, G., Miniscalco, C. & Gillberg, C. Autism in preschoolers: does individual clinician’s first visit diagnosis agree with final comprehensive diagnosis? Sci. World J. 2013, 716267 (2013). · DOI
- McDonnell, C. G. et al. When are we sure? predictors of clinician certainty in the diagnosis of autism spectrum disorder. J. Autism Dev. Disord. 49, 1391–1401 (2019). · DOI
Cited by
Federated learning for healthcare applications
Chaddad, A., Wu, Y., & Desrosiers, C. (2024) · IEEE Internet of Things Journal · doi:10.1109/jiot.2023.3325822 · Cited by 136
The use of AI in mental health services to support decision-making: scoping review
Auf, H., Svedberg, P., Nygren, J., Nair, M., & Lundgren, L. E. (2025) · Journal of Medical Internet Research · doi:10.2196/63548 · Cited by 56
“[45]; and (12) Tanguay-Sela et al [46].”
Child and adolescent psychiatry: challenges, solutions, opportunities, and future directions
Cortese, S., Arango, C., Aymerich, C., Catalan, A., Chetouani, M., Cohen, D., Coghill, D., Gabellone, A., Iniesta, R., Kadan, A., et al. (2026) · World Psychiatry · doi:10.1002/wps.70044 · Cited by 4
Metadata
- Authors
- Megerian, J. T., Dey, S., Melmed, R. D., Coury, D. L., Lerner, M., Nicholls, C. J., Sohl, K., Rouhbakhsh, R., Narasimhan, A., Romain, J., Golla, S., Shareef, S., Ostrovsky, A., Shannon, J., Kraft, C., Liu-Mayo, S., Abbas, H., Gal-Szabo, D. E., Wall, D. P., & Taraman, S.
- Venue
- npj Digital Medicine
- Published
- 2022-05-05
- Volume
- 5
- Issue
- 1
- Pages
- 57-57
- ISSN
- 2398-6352 (ISSN-L)
- OpenAlex
- W4229007573
- Cited by
- 111
- References
- 18
- FWCI
- 11.02
- Type
- journal article
- Language
- en
- Open access
- gold
- Licence
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Evaluation of an artificial intelligence-based medical device for diagnosis of autism spectrum disorder. https://app.cochat.ai/papers/10.1038/s41746-022-00598-6
- The paper
- Megerian, J. T., Dey, S., Melmed, R. D., Coury, D. L., Lerner, M., Nicholls, C. J., Sohl, K., Rouhbakhsh, R., Narasimhan, A., Romain, J., Golla, S., Shareef, S., Ostrovsky, A., Shannon, J., Kraft, C., Liu-Mayo, S., Abbas, H., Gal-Szabo, D. E., Wall, D. P., & Taraman, S. (2022). Evaluation of an artificial intelligence-based medical device for diagnosis of autism spectrum disorder. npj Digital Medicine. https://doi.org/10.1038/s41746-022-00598-6
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.