CoChat
Sign Up
Cited by 49

The DOI matches its Crossref record; 10 of the 10 references it cites were found in a registry or an index.

Power and optimal study design in iPSC-based brain disease modelling

Brunner, Jessie W., Lammertse, Hanna C. A., van Berkel, Annemiek A., Koopmans, Frank, Li, Ka Wan, Smit, August B., Toonen, Ruud F., Verhage, Matthijs, van der Sluis, Sophie·2023·Molecular Psychiatry
DOI 10.1038/s41380-022-01866-3PMC PMC10208961
Open this paper in CoChat — free account

Abstract

Studies using induced pluripotent stem cells (iPSCs) are gaining momentum in brain disorder modelling, but optimal study designs are poorly defined. Here, we compare commonly used designs and statistical analysis for different research aims. Furthermore, we generated immunocytochemical, electrophysiological, and proteomic data from iPSC-derived neurons of five healthy subjects, analysed data variation and conducted power simulations. These analyses show that published case–control iPSC studies are generally underpowered. Designs using isogenic iPSC lines typically have higher power than case–control designs, but generalization of conclusions is limited. We show that, for the realistic settings used in this study, a multiple isogenic pair design increases absolute power up to 60% or requires up to 5-fold fewer lines. A free web tool is presented to explore the power of different study designs, using any (pilot) data.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1038/s41380-022-01866-3

Cites

10 of 10 references verified0 citation errors · 0 not checkable

Statistics

Figures

Fig. 1 iPSC study designs address different research questions. Design 1 (case–control) is characterized by including iPSC lines from multiple independent donors per condition. For design 2A (single isogenic pair), an iPSC line derived from a single donor is gene-edited (or treated), to create two iPSC lines that share the same genetic background and only differ at the gene locus of interest (or treatment). Design 2B is an extension of 2A, in which multiple variants (or treatments) are edited into the same genetic background. In Design 3, multiple isogenic pairs are created. The table represents the ability of each design to answer the research questions described on the left. Green: suitable; Orange: possible, but not optimal; Red: not possible.
Fig. 1 iPSC study designs address different research questions. Design 1 (case–control) is characterized by including iPSC lines from multiple independent donors per condition. For design 2A (single isogenic pair), an iPSC line derived from a single donor is gene-edited (or treated), to create two iPSC lines that share the same genetic background and only differ at the gene locus of interest (or treatment). Design 2B is an extension of 2A, in which multiple variants (or treatments) are edited into the same genetic background. In Design 3, multiple isogenic pairs are created. The table represents the ability of each design to answer the research questions described on the left. Green: suitable; Orange: possible, but not optimal; Red: not possible.CC BY · PubMed Central
Fig. 2 Culture batch and inter-individual variation contribute to the total variance. A Schematic overview of the study, indicating sources of variation: inter-individual variation (i.e., variation introduced by including different individuals; orange box) and culture batch variation (i.e., variation introduced by acquiring data from different culture batches; purple box). Note that for this study, five iPSC-derived lines were used for morphological characterization, four lines for electrophysiology, and three lines for proteomic analyses. For the proteomics analysis, neurons were either cultured on a glia feeder layer (‘co-culture’) or on coating without glia (‘neuron-only’). B The contribution of different sources of variance is assessed by plotting the proportion of variance explained by culture batch (purple) and line (orange) for each parameter, as calculated following Nakagawa 2017 [17] for details, see Supplementary Methods). Values per parameter are included in Supplementary Table 2. Together, the variance contributed by line and culture batch accounted for a median of 10% (IQR: 6.0–21%) of total variance. C The explained variance calculated for all proteins in the proteomics dataset from neuron-glia co-cultures (top violin plot) and neuron-only cultures (bottom violin plot). For co-cultures, median variance explained by culture batch is 14.8% and for line 9.4%. For neuron-only cultures, the median variance explained by culture batch is 5.5% and 34.3% for line.
Fig. 2 Culture batch and inter-individual variation contribute to the total variance. A Schematic overview of the study, indicating sources of variation: inter-individual variation (i.e., variation introduced by including different individuals; orange box) and culture batch variation (i.e., variation introduced by acquiring data from different culture batches; purple box). Note that for this study, five iPSC-derived lines were used for morphological characterization, four lines for electrophysiology, and three lines for proteomic analyses. For the proteomics analysis, neurons were either cultured on a glia feeder layer (‘co-culture’) or on coating without glia (‘neuron-only’). B The contribution of different sources of variance is assessed by plotting the proportion of variance explained by culture batch (purple) and line (orange) for each parameter, as calculated following Nakagawa 2017 [17] for details, see Supplementary Methods). Values per parameter are included in Supplementary Table 2. Together, the variance contributed by line and culture batch accounted for a median of 10% (IQR: 6.0–21%) of total variance. C The explained variance calculated for all proteins in the proteomics dataset from neuron-glia co-cultures (top violin plot) and neuron-only cultures (bottom violin plot). For co-cultures, median variance explained by culture batch is 14.8% and for line 9.4%. For neuron-only cultures, the median variance explained by culture batch is 5.5% and 34.3% for line.CC BY · PubMed Central
Fig. 3 Flow chart for a priori power analysis. Flow chart describing the steps for performing power analysis per study design. Based on pilot data, variance estimates can be obtained and used to calculate effect sizes. For Designs 1 and 3, intra-cluster correlation (ICC) values should be taken into account. Power analyses for a wide range of study design scenarios can be performed using the freely available online application.
Fig. 3 Flow chart for a priori power analysis. Flow chart describing the steps for performing power analysis per study design. Based on pilot data, variance estimates can be obtained and used to calculate effect sizes. For Designs 1 and 3, intra-cluster correlation (ICC) values should be taken into account. Power analyses for a wide range of study design scenarios can be performed using the freely available online application.CC BY · PubMed Central
Fig. 4 Power simulations to calculate statistical power of Design 1-type studies. A Schematic overview of the study design for the power analysis. In this hypothetical scenario, two conditions (control and case) are compared. Within each condition, multiple individuals are sampled. The statistical power is calculated by a simulation experiment (1000 simulations per scenario) varying the number of iPSC lines (either 2, 4, 6, 10, 20 or 50 per group, i.e., 4, 8, 12, 20, 40 or 100 lines in total) and the number of observations per individual. To model a series of scenarios, the simulations were performed for three representative ICC values. B ICC values (as shown in Fig. 3B) per parameter sorted in ascending order. Three representative ICC values (reflecting low, medium and high clustering) were selected for the power simulation. C–K Simulated power curves, showing the relationship between statistical power and the total number of observations (number of iPSC lines times the number of observations per iPSC line).For each plot, the grey dotted line represents the cut-off value of 80% power. To assess statistical power for a range of effect sizes, three scenarios were compared, in which the two groups showed a mean difference of 15% (small), 50% (medium) or 70% (large). Corresponding Cohen’s d values were calculated using these mean differences and measured variance of the morphology parameter ‘Synapse Density’ (SD: 0.038) from the data example: 15% mean difference: d = 0.32; 50% mean difference: d = 1.1; 70% mean difference: d = 1.54.
Fig. 4 Power simulations to calculate statistical power of Design 1-type studies. A Schematic overview of the study design for the power analysis. In this hypothetical scenario, two conditions (control and case) are compared. Within each condition, multiple individuals are sampled. The statistical power is calculated by a simulation experiment (1000 simulations per scenario) varying the number of iPSC lines (either 2, 4, 6, 10, 20 or 50 per group, i.e., 4, 8, 12, 20, 40 or 100 lines in total) and the number of observations per individual. To model a series of scenarios, the simulations were performed for three representative ICC values. B ICC values (as shown in Fig. 3B) per parameter sorted in ascending order. Three representative ICC values (reflecting low, medium and high clustering) were selected for the power simulation. C–K Simulated power curves, showing the relationship between statistical power and the total number of observations (number of iPSC lines times the number of observations per iPSC line).For each plot, the grey dotted line represents the cut-off value of 80% power. To assess statistical power for a range of effect sizes, three scenarios were compared, in which the two groups showed a mean difference of 15% (small), 50% (medium) or 70% (large). Corresponding Cohen’s d values were calculated using these mean differences and measured variance of the morphology parameter ‘Synapse Density’ (SD: 0.038) from the data example: 15% mean difference: d = 0.32; 50% mean difference: d = 1.1; 70% mean difference: d = 1.54.CC BY · PubMed Central
Fig. 5 Power simulations to calculate statistical power of Design 2-type studies. A–C Design 2A describes a comparison within a single isogenic pair. Simulated power curves are shown for three mean difference-scenarios. The corresponding Cohen’s d values were calculated for an iPSC-line showing low variability (SDC2 = 0.031; thick line) or high variability (SDC1 = 0.044; dashed). Corresponding Cohen’s d values: 15% mean difference: d = 0.29 (high) and 0.43 (low); 30% mean difference: d = 0.58 (high) and 0.86 (low); 50% mean difference; d = 0.96 (high) and 1.43 (low). D–G For Design 2B, a hypothetical study was simulated with three experimental groups for the high- and low-variability iPSC lines, as for Design 2A. Four scenarios were tested, comparing the impact of having a small (15%) or medium (30%) mean difference (D and E), and the impact of including two groups with the same (F) or different (G) effect sizes.
Fig. 5 Power simulations to calculate statistical power of Design 2-type studies. A–C Design 2A describes a comparison within a single isogenic pair. Simulated power curves are shown for three mean difference-scenarios. The corresponding Cohen’s d values were calculated for an iPSC-line showing low variability (SDC2 = 0.031; thick line) or high variability (SDC1 = 0.044; dashed). Corresponding Cohen’s d values: 15% mean difference: d = 0.29 (high) and 0.43 (low); 30% mean difference: d = 0.58 (high) and 0.86 (low); 50% mean difference; d = 0.96 (high) and 1.43 (low). D–G For Design 2B, a hypothetical study was simulated with three experimental groups for the high- and low-variability iPSC lines, as for Design 2A. Four scenarios were tested, comparing the impact of having a small (15%) or medium (30%) mean difference (D and E), and the impact of including two groups with the same (F) or different (G) effect sizes.CC BY · PubMed Central
Fig. 6 Power simulations to calculate statistical power of Design 3-type studies. A Schematic overview of the study design. Several independent iPSC lines are sampled and for each line, an isogenic ‘control’ is generated. Thus, two conditions (‘cases’ and ‘controls’) are compared, while accounting for clustering in the data that is due to the use of multiple individual iPSC lines B–I The statistical power is calculated by a simulation experiment (1000 simulations per scenario) as for Design 1 (Fig. 4), for the highest ICC value from Fig. 4 (0.35), the high (SDC1 = 0.044) and low (SDC2 = 0.031) variable lines as for Design 2 A, and two mean difference scenarios (50%: B–E; 15%: F–I). In addition, four different slope variance values are tested: 0.001 (negligible); 0.05 (‘medium’; Aarts et al. 2015); 0.15 (‘high’: Aarts et al. 2015); 0.5 (‘extreme’).
Fig. 6 Power simulations to calculate statistical power of Design 3-type studies. A Schematic overview of the study design. Several independent iPSC lines are sampled and for each line, an isogenic ‘control’ is generated. Thus, two conditions (‘cases’ and ‘controls’) are compared, while accounting for clustering in the data that is due to the use of multiple individual iPSC lines B–I The statistical power is calculated by a simulation experiment (1000 simulations per scenario) as for Design 1 (Fig. 4), for the highest ICC value from Fig. 4 (0.35), the high (SDC1 = 0.044) and low (SDC2 = 0.031) variable lines as for Design 2 A, and two mean difference scenarios (50%: B–E; 15%: F–I). In addition, four different slope variance values are tested: 0.001 (negligible); 0.05 (‘medium’; Aarts et al. 2015); 0.15 (‘high’: Aarts et al. 2015); 0.5 (‘extreme’).CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 10 references

Research works printed but not linked: 26

  1. Nord CL, Valton V, Wood J, Roiser JP. Power-up: a reanalysis of ‘power failure’ in neuroscience using mixture modelling. J Neurosci. 2017;37:8051–61. · DOI
  2. Galbraith S, Daniel JA, Vissel B. A study of clustered data and approaches to its analysis. J Neurosci. 2010;30:10601–8. · DOI
  3. Aarts E, Verhage M, Veenvliet JV, Dolan CV, Van Der Sluis S. A solution to dependency: using multilevel analysis to accommodate nested data. Nat Neurosci. 2014;17:491–6. · DOI
  4. Aarts E, Dolan CV, Verhage M, Van der Sluis S. Multilevel analysis quantifies variation in the experimental effect while optimizing power and preventing false positives. BMC Neurosci. 2015;16:1–15. · DOI
  5. Boisgontier MP, Cheval B. The anova to mixed model transition. Neurosci Biobehav Rev. 2016;68:1004–5. Available from https://doi.org/10.1016/j.neubiorev.2016.05.034. · DOI
  6. Yu Z, Guindani M, Grieco SF, Chen L, Holmes TC, Xu X. Beyond t test and ANOVA: applications of mixed-effects models for more rigorous statistical analysis in neuroscience research. Neuron. [Internet]. 2022;110:21–35. Available from https://doi.org/10.1016/j.neuron.2021.10.030. · DOI · Paper page
  7. Volpato V, Webber C Addressing variability in iPSC-derived models of human disease: Guidelines to promote reproducibility. DMM Dis Model Mech. 2020;13. · DOI · Paper page
  8. Nehme R, Zuccaro E, Ghosh SD, Li C, Sherwood JL, Pietilainen O, et al. Combining NGN2 programming with developmental patterning generates human excitatory neurons with NMDAR-mediated synaptic transmission. Cell Rep. [Internet]. 2018;23:2509–23. https://www.sciencedirect.com/science/article/pii/S2211124718306259. · DOI
  9. Meijer M, Rehbach K, Brunner JW, Classen JA, Lammertse HCA, van Linge LA, et al. A single-cell model for synaptic transmission and plasticity in human iPSC-derived neurons. Cell Rep. 2019;27:2199–2211.e6. · DOI
  10. Gonzalez-Lozano MA, Koopmans F. Data-independent acquisition (SWATH) mass spectrometry analysis of protein content in primary neuronal cultures. In: Li K, editor. Neuroproteomics. Humana, New York, NY; 2019. · DOI
  11. Cox J, Mann M. MaxQuant enables high peptide identification rates, individualized p.p.b.-range mass accuracies and proteome-wide protein quantification. Nat Biotechnol. 2008;26:1367–72. · DOI · Paper page
  12. Schmitz SK, Hjorth JJJ, Joemai RMS, Wijntjes R, Eijgenraam S, de Bruijn P, et al. Automated analysis of neuronal morphology, synapse number and synaptic recruitment. J Neurosci Methods. [Internet]. 2011;195:185–93. Available from https://doi.org/10.1016/j.jneumeth.2010.12.011. · DOI
  13. Nakagawa S, Johnson PCD, Schielzeth H The coefficient of determination R2 and intra-class correlation coefficient from generalized linear mixed-effects models revisited and expanded. J R Soc Interface. 2017;14. · DOI
  14. Hoffman GE, Schadt EE. variancePartition: Interpreting drivers of variation in complex gene expression studies. BMC Bioinforma [Internet]. 2016;17:17–22. Available from https://doi.org/10.1186/s12859-016-1323-z. · DOI · Paper page
  15. Dominicus A, Skrondal A, Gjessing HK, Pedersen NL, Palmgren J. Likelihood ratio tests in behavioral genetics: Problems and solutions. Behav Genet. 2006;36:331–40. · DOI
  16. Wickham H ggplot2: Elegant Graphics for Data Analysis [Internet]. Springer-Verlag New York; 2016 (189–201). Available from: https://ggplot2.tidyverse.org · DOI
  17. Koopmans F, van Nierop P, Andres-Alonso M, Byrnes A, Cijsouw T, Coba MP, et al. SynGO: an evidence-based, expert-curated knowledge base for the synapse. Neuron. 2019;103:217–34.e4. · DOI · Paper page
  18. Tang X, Zhou L, Wagner AM, Marchetto MCN, Muotri AR, Gage FH, et al. Astroglial cells regulate the developmental timeline of human neurons differentiated from induced pluripotent stem cells. Stem Cell Res. [Internet]. 2013;11:743–57. https://doi.org/10.1016/j.scr.2013.05.002. · DOI
  19. Klapper SD, Garg P, Dagar S, Lenk K, Gottmann K, Nieweg K. Astrocyte lineage cells are essential for functional neuronal differentiation and synapse maturation in human iPSC-derived neural networks. Glia 2019;67:1893–909. · DOI
  20. Burke EE, Chenoweth JG, Shin JH, Collado-Torres L, Kim SK, Micali N, et al. Dissecting transcriptomic signatures of neuronal differentiation and maturation using iPSCs. Nat Commun. [Internet]. 2020;11:1–14. Available from https://doi.org/10.1038/s41467-019-14266-z. · DOI
  21. Raudenbush SW, Liu X. Statistical power and optimal design for multisite randomized trials. Psychol Methods. 2000;5:199–213. · DOI · Paper page
  22. Kajiwara M, Aoi T, Okita K, Takahashi R, Inoue H, Takayama N, et al. Donor-dependent variations in hepatic differentiation from human-induced pluripotent stem cells. Proc Natl Acad Sci USA. 2012;109:12538–43. · DOI
  23. Burrows CK, Banovich NE, Pavlovic BJ, Patterson K, Gallego Romero I, Pritchard JK, et al. Genetic variation, not cell type of origin, underlies the majority of identifiable regulatory differences in iPSCs. PLoS Genet. 2016;12:1–18. · DOI
  24. Carcamo-Orive I, Hoffman GE, Cundiff P, Beckmann ND, D’Souza SL, Knowles JW, et al. Analysis of transcriptional variability in a large human iPSC library reveals genetic and non-genetic determinants of heterogeneity. Cell Stem Cell. 2017;20:518–32.e9. https://doi.org/10.1016/j.stem.2016.11.005. · DOI
  25. Bisogno LS, Yang J, Bennett BD, Ward JM, MacKey LC, Annab LA, et al. Ancestry-dependent gene expression correlates with reprogramming to pluripotency and multiple dynamic biological processes. Sci Adv. 2020;6:1–14. · DOI
  26. Hoffman GE, Schrode N, Flaherty E, Brennand KJ. New considerations for hiPSC-based models of neuropsychiatric disorders. Mol Psychiatry. [Internet]. 2019;24:49–66. https://doi.org/10.1038/s41380-018-0029-1. · DOI

Cited by

Sorted by citations

Metadata

Authors
Brunner, J. W., Lammertse, H. C. A., van Berkel, A. A., Koopmans, F., Li, K. W., Smit, A. B., Toonen, R. F., Verhage, M., & van der Sluis, S.
ORCID
Venue
Molecular Psychiatry
Publisher
Springer Science and Business Media LLC
Published
2022-11-16
Volume
28
Issue
4
Pages
1545-1556
ISSN
  • 1359-4184
  • 1476-5578
DOI
10.1038/s41380-022-01866-3
PMC
PMC10208961
Cited by
49
References
10
Language
en
Licence

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review