Erroneous inference based on a lack of preference within one group: Autism, mice, and the social approach task
Open accessThe DOI matches its Crossref record; 13 of the 13 references it cites were found in a registry or an index.
Abstract
The Social Approach Task is commonly used to identify sociability deficits when modeling liability factors for autism spectrum disorder (ASD) in mice. It was developed to expand upon existing assays to examine distinct aspects of social behavior in rodents and has become a standard component of mouse ASD‐relevant phenotyping pipelines. However, there is variability in the statistical analysis and interpretation of results from this task. A common analytical approach is to conduct within‐group comparisons only, and then interpret a difference in significance levels as if it were a group difference, without any direct comparison. As an efficient shorthand, we named this approach EWOCs: Erroneous Within‐group Only Comparisons. Here, we examined the prevalence of EWOCs and used simulations to test whether this approach could produce misleading inferences. Our review of Social Approach studies of high‐confidence ASD genes revealed 45% of papers sampled used only this analytical approach. Through simulations, we then demonstrate how a lack of significant difference within one group often does not correspond to a significant difference between groups, and show this erroneous interpretation increases the rate of false positives up to 25%. Finally, we define a simple solution: use an index, like a social preference score, with direct statistical comparisons between groups to identify significant differences. We also provide power calculations to guide sample size in future studies. Overall, elimination of EWOCs and adoption of direct comparisons should result in more accurate, reliable, and reproducible data interpretations from the Social Approach Task across ASD liability models. Autism Res 2019, 12: 1171–1183 . © 2019 International Society for Autism Research, Wiley Periodicals, Inc. Lay Summary: The Social Approach Task is widely used to assess social behavior in mice and is frequently used in studies modeling autism. However, reviewing published studies showed nearly half do not use correct comparisons to interpret these data. Using simulated and original data, we argue the correct statistical approach is a direct comparison of scores between groups. This simple solution should reduce false positives and improve consistency of results across studies.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Moy, S. S., Nadler, J. J., Perez, A., Barbaro, R. P., Johns, J. M., Magnuson, T. R., Piven, J., & Crawley, J. N. (2004) · Genes, Brain and Behavior · doi:10.1111/j.1601-1848.2004.00076.x · Cited by 1,522 · cited 6×
“The typical version of this task comprises two test trials: the sociability trial and the preference for social novelty trial [Moy et al., 2004; Nadler et al., 2004], along with two preceding habituation trials.”
BackgroundMethodInfluentialErroneous analyses of interactions in neuroscience: A problem of significance
Nieuwenhuis, S., Forstmann, B. U., & Wagenmakers, E.-J. (2011) · Nature Neuroscience · doi:10.1038/nn.2886 · Cited by 864 · cited 1×
“…highlighted the occurrence of EWOCs with regards to this one assay, this flaw certainly has been seen in a variety of other experiments in the past (Nieuwenhuis et al., 2011), and the same erroneous logic could easily be applied to a variety of other experiments in behavior (e.g. lack of…”
BackgroundAutomated apparatus for quantitation of social approach behaviors in mice
Nadler, J. J., Moy, S. S., Dold, G., Trang, D., Simmons, N., Perez, A., Young, N. B., Barbaro, R. P., Piven, J., Magnuson, T. R., et al. (2004) · Genes, Brain and Behavior · doi:10.1111/j.1601-183x.2004.00071.x · Cited by 775 · cited 2×
“The motivation behind development of the Social Approach Task was to improve face validity of murine social behavioral assays with regards to specific social impairments that characterize ASD (Moy et al., 2004; Nadler et al., 2004).”
BackgroundMethodAutistic-like social behaviour in Shank2-mutant mice improved by restoring NMDA receptor function
Won, H., Lee, H.-R., Gee, H. Y., Mah, W., Kim, J.-I., Lee, J., Ha, S., Chung, C., Jung, E. S., Cho, Y. S., et al. (2012) · Nature · doi:10.1038/nature11208 · Cited by 695
Filipello, F., Morini, R., Corradini, I., Zerbi, V., Canzi, A., Michalski, B., Erreni, M., Markicevic, M., Starvaggi-Cucuzza, C., Otero, K., et al. (2018) · Immunity · doi:10.1016/j.immuni.2018.04.016 · Cited by 695
Exogenous and evoked oxytocin restores social behavior in the Cntnap2 mouse model of autism
Peñagarikano, O., Lázaro, M. T., Lu, X.-H., Gordon, A., Dong, H., Lam, H. A., Peles, E., Maidment, N. T., Murphy, N. P., Yang, X. W., et al. (2015) · Science Translational Medicine · doi:10.1126/scitranslmed.3010257 · Cited by 378 · cited 2×
“…et al., 2010; Grabrucker, Boeckers, & Grabrucker, 2016; Lugo, Swann, & Anderson, 2014; Maloney et al., 2018; Page, Kuti, Prestia, & Sur, 2009; Penagarikano et al., 2015; Samaco et al., 2012; Schwartzer et al., 2013; Stoppel et al., 2018; Won et al., 2012) The use of the Social Approach Task…”
MethodZhou, Y., Kaiser, T., Monteiro, P., Zhang, X., Van der Goes, M. S., Wang, D., Barak, B., Zeng, M., Li, C., Lu, C., et al. (2016) · Neuron · doi:10.1016/j.neuron.2015.11.023 · Cited by 325
Minimal aberrant behavioral phenotypes of neuroligin‐3 R451C knockin mice
Chadman, K. K., Gong, S., Scattoni, M. L., Boltuck, S. E., Gandhy, S. U., Heintz, N., & Crawley, J. N. (2008) · Autism Research · doi:10.1002/aur.22 · Cited by 292
5-HT release in nucleus accumbens rescues social deficits in mouse autism model
Walsh, J. J., Christoffel, D. J., Heifets, B. D., Ben-Dor, G. A., Selimbeyoglu, A., Hung, L. W., Deisseroth, K., & Malenka, R. C. (2018) · Nature · doi:10.1038/s41586-018-0416-4 · Cited by 256
Reproducibility and replicability of rodent phenotyping in preclinical studies
Kafkafi, N., Agassi, J., Chesler, E. J., Crabbe, J. C., Crusio, W. E., Eilam, D., Gerlai, R., Golani, I., Gomez-Marin, A., Heller, R., et al. (2018) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2018.01.003 · Cited by 215
Drd3 Signaling in the Lateral Septum Mediates Early Life Stress-Induced Social Dysfunction
Shin, S., Pribiag, H., Lilascharoen, V., Knowland, D., Wang, X.-Y., & Lim, B. K. (2018) · Neuron · doi:10.1016/j.neuron.2017.11.040 · Cited by 120
Oxytocin and same-sex social behavior in female meadow voles
Beery, A. K., & Zucker, I. (2010) · Neuroscience · doi:10.1016/j.neuroscience.2010.05.023 · Cited by 108
Stoppel, L. J., Kazdoba, T. M., Schaffler, M. D., Preza, A. R., Heynen, A., Crawley, J. N., & Bear, M. F. (2018) · Neuropsychopharmacology · doi:10.1038/npp.2017.236 · Cited by 92
Cited by
Dong, Z., Chen, W., Chen, C., Wang, H., Cui, W., Tan, Z., Robinson, H., Gao, N., Luo, B., Zhang, L., et al. (2020) · Neuron · doi:10.1016/j.neuron.2019.10.035 · Cited by 109
Tau Reduction Prevents Key Features of Autism in Mouse Models
Tai, C., Chang, C.-W., Yu, G.-Q., Lopez, I., Yu, X., Wang, X., Guo, W., & Mucke, L. (2020) · Neuron · doi:10.1016/j.neuron.2020.01.038 · Cited by 96
“Using the more rigorous but less sensitive approach to the statistical analysis of the social preference test recommended by (Nygaard et al., 2019), Shank3B−/− mice of all Mapt genotypes did not differ significantly from WT controls in this test (data not shown).”
MethodSociability development in mice with cell-specific deletion of the NMDA receptor NR1 subunit gene
Ferri, S. L., Pallathra, A. A., Kim, H., Dow, H. C., Raje, P., McMullen, M., Bilker, W. B., Siegel, S. J., Abel, T., & Brodkin, E. S. (2020) · Genes, Brain and Behavior · doi:10.1111/gbb.12624 · Cited by 15
Rais (2022) · doi:10.1101/2022.02.08.479618 · Cited by 2
Metadata
- Authors
- Nygaard, K. R., Maloney, S. E., & Dougherty, J. D.
- Venue
- Autism Research
- Publisher
- Wiley
- Published
- 2019-06-11
- Volume
- 12
- Issue
- 8
- Pages
- 1171-1183
- ISSN
- 1939-3792 (ISSN-L)
- 1939-3806
- OpenAlex
- W2950954773
- Cited by
- 30
- References
- 13
- FWCI
- 2.70
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.