Applying Machine Learning to Facilitate Autism Diagnostics: Pitfalls and Promises
Open accessThe DOI matches its Crossref record; 12 of the 12 references it cites were found in a registry or an index.
Abstract
Machine learning has immense potential to enhance diagnostic and intervention research in the behavioral sciences, and may be especially useful in investigations involving the highly prevalent and heterogeneous syndrome of autism spectrum disorder. However, use of machine learning in the absence of clinical domain expertise can be tenuous and lead to misinformed conclusions. To illustrate this concern, the current paper critically evaluates and attempts to reproduce results from two studies (Wall et al. in Transl Psychiatry 2(4):e100, 2012a; PloS One 7(8), 2012b) that claim to drastically reduce time to diagnose autism using machine learning. Our failure to generate comparable findings to those reported by Wall and colleagues using larger and more balanced data underscores several conceptual and methodological problems associated with these studies. We conclude with proposed best-practices when using machine learning in autism research, and highlight some especially promising areas for collaborative work at the intersection of computational and behavioral science.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The WEKA data mining software: an update
Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P., & Witten, I. H. (2009) · ACM SIGKDD Explorations Newsletter · doi:10.1145/1656274.1656278 · Cited by 17,750
A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection
Kohavi, R. (1995) · Zenodo · doi:10.5281/zenodo.19712698 · Cited by 10,909
Lord, C., Rutter, M., & Le Couteur, A. (1994) · Journal of Autism and Developmental Disorders · doi:10.1007/bf02172145 · Cited by 9,065 · cited 3×
“…research is to create an efficient and robust diagnostic algorithm based on human coded behaviors from diagnostic instruments such as the Autism Diagnostic Interview-Revised (ADI-R; Lord et al., 1994) and the Autism Diagnostic Observation Schedule (ADOS; Gotham et al, 2007; Lord et al., 2000).”
MethodLord, C., Risi, S., Lambrecht, L., Cook, E. H., Leventhal, B. L., DiLavore, P. C., Pickles, A., & Rutter, M. (2000) · Journal of Autism and Developmental Disorders · doi:10.1023/a:1005592401947 · Cited by 7,520 · cited 12×
“Of the 29 Module 1 codes, 12 are utilized in the original algorithm (Lord et al. 2000) and 14 in the revised algorithms (Gotham et al.”
BackgroundMethodInfluentialPicard, R. W. (1997) · MIT Press · doi:10.7551/mitpress/1140.001.0001 · Cited by 7,382
The INTERSPEECH 2009 emotion challenge
Schuller, B., Steidl, S., & Batliner, A. (2009) · Interspeech 2009 · doi:10.21437/interspeech.2009-103 · Cited by 938 · cited 2×
“In our experiments we use a measure called unweighted average recall (UAR; Schuller et al., 2009), which is the mean of sensitivity (recall of positive instances) and specificity (recall of negative instances) 8 .”
MethodDefining the broader phenotype of autism: genetic, brain, and behavioral perspectives
DAWSON, G., WEBB, S., SCHELLENBERG, G. D., DAGER, S., FRIEDMAN, S., AYLWARD, E., & RICHARDS, T. (2002) · Development and Psychopathology · doi:10.1017/s0954579402003103 · Cited by 467
Subgrouping the Autism Spectrum: Reflections on DSM-5
Lai, M.-C., Lombardo, M. V., Chakrabarti, B., & Baron-Cohen, S. (2013) · PLoS Biology · doi:10.1371/journal.pbio.1001544 · Cited by 330 · cited 1×
“Lai et al. (2013) generated a list of desired behavioral specifiers toward identification of subgroups in autism, offering one potential direction for bringing computational methods to bear.”
BackgroundGeschwind, D. H., Sowinski, J., Lord, C., Iversen, P., Shestack, J., Jones, P., Ducat, L., & Spence, S. J. (2001) · The American Journal of Human Genetics · doi:10.1086/321292 · Cited by 302 · cited 1×
“(2012a) consisted primarily(4) of ADOS Module 1 administrations from the Autism Genetics Resource Exchange (AGRE; Geschwind et al. 2001) database.”
BackgroundAnnual research review: re-thinking the classification of autism spectrum disorders
Lord, C., & Jones, R. M. (2012) · Journal of Child Psychology and Psychiatry · doi:10.1111/j.1469-7610.2012.02547.x · Cited by 290 · cited 1×
“Researchers will need efficient, reliable, and valid quantitative, dimensional measures that can help stratify the disorder (Lord and Jones 2012).”
BackgroundLevitt, P., & Campbell, D. B. (2009) · Journal of Clinical Investigation · doi:10.1172/jci37934 · Cited by 234 · cited 2×
“The ADOS is a widely used, standardized assessment for diagnosing ASD that consists of 30–45 min of semistructured interaction with a trained administrator to elicit and code behaviors relating to social interaction, communication, play, and imaginative use of materials (Lord et al. 2000). The ADOS consists of four di…”
BackgroundMethodConnecting Genes to Brain in the Autism Spectrum Disorders
Abrahams, B. S., & Geschwind, D. H. (2010) · Archives of Neurology · doi:10.1001/archneurol.2010.47 · Cited by 193 · cited 2×
“More rapid diagnostic procedures could facilitate acquisition of the very large cohorts (over 10,000 subjects; Abrahams & Geschwind, 2010) needed to reliably uncover the complex neurogenic etiology of ASD.”
Background
Cited by
Machine learning algorithm validation with a limited sample size
Vabalas, A., Gowen, E., Poliakoff, E., & Casson, A. J. (2019) · PLOS ONE · doi:10.1371/journal.pone.0224365 · Cited by 1,901
“To help in this process, there are good guideline studies advising how to avoid pitfalls, including how to reliably validate the results [28, 29].”
BackgroundLeakage and the reproducibility crisis in machine-learning-based science
Kapoor, S., & Narayanan, A. (2023) · Patterns · doi:10.1016/j.patter.2023.100804 · Cited by 1,387
“For example, Bone et al.(61) highlight that in a study on predicting autism using ML models, excluding the data corresponding to borderline cases of autism leads to leakage because the test set is no longer representative of the general population about which claims are made.”
BackgroundLuo, W., Phung, D., Tran, T., Gupta, S., Rana, S., Karmakar, C., Shilton, A., Yearwood, J., Dimitrova, N., Ho, T. B., et al. (2016) · Journal of Medical Internet Research · doi:10.2196/jmir.5870 · Cited by 988
Analysis and Detection of Autism Spectrum Disorder Using Machine Learning Techniques
Raj, S., & Masood, S. (2020) · Procedia Computer Science · doi:10.1016/j.procs.2020.03.399 · Cited by 388
Machine learning in autistic spectrum disorder behavioral research: A review and ways forward
Thabtah, F. (2019) · Informatics for Health and Social Care · doi:10.1080/17538157.2017.1399132 · Cited by 292
Metadata
- Authors
- Bone, D., Goodwin, M. S., Black, M. P., Lee, C.-C., Audhkhasi, K., & Narayanan, S.
- Venue
- Journal of Autism and Developmental Disorders
- Published
- 2014-10-08
- Volume
- 45
- Issue
- 5
- Pages
- 1121-1136
- ISSN
- 0162-3257 (ISSN-L)
- 1573-3432
- OpenAlex
- W2092444470
- Cited by
- 236
- References
- 12
- FWCI
- 3.92
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.