Multiple inference and gender differences in the effects of early intervention: A reevaluation of the Abecedarian, Perry Preschool, and Early Training Projects
Abstract
The view that the returns to educational investments are highest for early childhood interventions is widely held and stems primarily from several influential randomized trials—Abecedarian, Perry, and the Early Training Project—that point to super-normal returns to early interventions. This article presents a de novo analysis of these experiments, focusing on two core issues that have received limited attention in previous analyses: treatment effect heterogeneity by gender and overrejection of the null hypothesis due to multiple inference. To address the latter issue, a statistical framework that combines summary index tests with familywise error rate and false discovery rate corrections is implemented. The first technique reduces the number of tests conducted; the latter two techniques adjust the p values for multiple inference. The primary finding of the reanalysis is that girls garnered substantial short- and long-term benefits from the interventions, but there were no significant long-term benefits for boys. These conclusions, which have appeared ambiguous when using “naive” estimators that fail to adjust for multiple testing, contribute to a growing literature on the emerging female–male academic achievement gap. They also demonstrate that in complex studies where multiple questions are asked of the same data set, it can be important to declare the family of tests under consideration and to either consolidate measures or report adjusted and unadjusted p values.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the statistical analysis, the authors adjusted for multiple tests by implementing FDR corrections and report FDR q values, and also report bootstrapped P values to account for regressions with a small number of clusters.
“In the results presented in Extended Data Tables 6 and 7, we also adjusted for the fact that we conducted multiple tests on the same dataset by implementing FDR corrections and report the FDR q values50. We also report the bootstrapped P value51 to account for regressions with a small number of clusters.”
From Last-mile delivery increases vaccine uptake in Sierra Leone (2024) · cites this paper as [50]
The findings are robust to Type I error concerns: sharpened two-stage q-values show the observed impacts remain statistically significant at the 1 percent level.
“Finally, we note that our findings are robust to Type I error concerns. 10 We calculate sharpened two-stage q-values in Table 4 (Benjamini et al., 2006b; Anderson, 2008) and the observed impacts remain statistically significant at the 1 percent level.”
From Food Coma is Real: The Effect of Digestive Fatigue on Adolescents' Cognitive Performance (2024) · cites this paper as Anderson, 2008
The analysis performs False Discovery Rate (FDR) adjustment using the method described in (77), considering each experiment separately and grouping variables into primary, secondary, and tertiary outcomes.
“We perform False Discovery Rate (FDR) adjustment using the method described in (77), considering each experiment (Reduced and Increased Exposure) separately and grouping the variables as:”
From Reranking partisan animosity in algorithmic social media feeds alters affective polarization (2025) · cites this paper as [77]
Anderson's indices are difficult to interpret because they group diverse outcomes and their components change with age.
“The outcomes used to create each index are quite diverse and group a variety of very different outcomes (e.g., crime, employment, education). It is difficult to interpret his indices. Moreover, the components of his indices change with age.”
From Analyzing social experiments as implemented: A reexamination of the evidence from the HighScope Perry Preschool Program (2010) · cites this paper as Anderson (2008)
Cites
Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing
Benjamini, Y., & Hochberg, Y. (1995) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1995.tb02031.x · Cited by 109,482 · see citing works
Statistical power analysis for the behavioral sciences
Cohen, J. (1990) · Computers Environment and Urban Systems · doi:10.1016/0198-9715(90)90050-4 · Cited by 65,406 · see citing works
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1994) · doi:10.1201/9780429246593 · Cited by 40,500 · see citing works
A heteroskedasticity-consistent covariance matrix estimator and a direct test for heteroskedasticity
White, H. (1980) · Econometrica · doi:10.2307/1912934 · Cited by 25,861 · see citing works
The control of the false discovery rate in multiple testing under dependency
Benjamini, Y., & Yekutieli, D. (2001) · The Annals of Statistics · doi:10.1214/aos/1013699998 · Cited by 10,490 · see citing works
Fisher, R. A. (1936) · BMJ · doi:10.1136/bmj.1.3923.554-a · Cited by 4,980 · see citing works
A sharper Bonferroni procedure for multiple tests of significance
HOCHBERG, Y. (1988) · Biometrika · doi:10.1093/biomet/75.4.800 · Cited by 4,829 · see citing works
Adaptive linear step-up procedures that control the false discovery rate
Benjamini, Y., Krieger, A. M., & Yekutieli, D. (2006) · Biometrika · doi:10.1093/biomet/93.3.491 · Cited by 3,228 · see citing works
Experimental Analysis of Neighborhood Effects
Kling, J. R., Liebman, J. B., & Katz, L. F. (2007) · Econometrica · doi:10.1111/j.1468-0262.2007.00733.x · Cited by 2,719 · see citing works
(1986) · Journal of Economic Literature · Cited by 2,705 · see citing works
The importance of noncognitive skills: lessons from the GED testing program
Heckman, J. J., & Rubinstein, Y. (2001) · American Economic Review · doi:10.1257/aer.91.2.145 · Cited by 1,488 · see citing works
Carneiro, P. M., & Heckman, J. J. (2003) · doi:10.2139/ssrn.434544 · Cited by 1,325 · see citing works
(2005) · Medical Entomology and Zoology · Cited by 1,313 · see citing works
Significant Benefits: The High/Scope Perry Preschool Study Through Age 27
Schweinhart, L. J., Barnes, H., Weikart, D. P., & Ypsilanti, M. (1993) · Cited by 1,217 · see citing works
Procedures for Comparing Samples with Multiple Endpoints
O’Brien, P. C. (1984) · Biometrics · doi:10.2307/2531158 · Cited by 1,163 · see citing works
Early Childhood Education: Young Adult Outcomes From the Abecedarian Project
Campbell, F. A., Ramey, C. T., Pungello, E., Sparling, J., & Miller-Johnson, S. (2002) · Applied Developmental Science · doi:10.1207/s1532480xads0601_05 · Cited by 1,144 · see citing works
Stepwise Multiple Testing as Formalized Data Snooping
Romano, J. P., & Wolf, M. (2005) · Econometrica · doi:10.1111/j.1468-0262.2005.00615.x · Cited by 1,019 · see citing works
Early Childhood Education Programs
Currie, J. (2001) · The Journal of Economic Perspectives · doi:10.1257/jep.15.2.213 · Cited by 1,002 · see citing works
Empirical Benchmarks for Interpreting Effect Sizes in Research
Hill, C. J., Bloom, H. S., Black, A. R., & Lipsey, M. W. (2008) · Child Development Perspectives · doi:10.1111/j.1750-8606.2008.00061.x · Cited by 950 · see citing works
Does Head Start Make a Difference?
Currie, J., & Thomas, D. (1993) · National Bureau of Economic Research · doi:10.3386/w4406 · Cited by 927 · see citing works
Resampling-Based Multiple Testing.
Booth, J. G., Westfall, P. H., & Young, S. S. (1994) · Journal of the American Statistical Association · doi:10.2307/2291234 · Cited by 914 · see citing works
Abadie, A., Angrist, J. D., & Imbens, G. W. (2002) · Econometrica · doi:10.1111/1468-0262.00270 · Cited by 728 · see citing works
Exact and Approximate Stepdown Methods for Multiple Hypothesis Testing
Romano, J. P., & Wolf, M. (2005) · Journal of the American Statistical Association · doi:10.1198/016214504000000539 · Cited by 609 · see citing works
Campbell, F. A., & Ramey, C. T. (1995) · American Educational Research Journal · doi:10.3102/00028312032004743 · Cited by 383 · see citing works
Interference Between Units in Randomized Experiments
Rosenbaum, P. R. (2007) · Journal of the American Statistical Association · doi:10.1198/016214506000001112 · Cited by 371 · see citing works
Head Start Impact Study: First Year Findings.
Puma, M. J., Bell, S., Cook, R., Heid, C. A., & López, M. (2005) · Cited by 278 · see citing works
Resampling: The new statistics
Simon, J. L. (1992) · Cited by 234 · see citing works
Williams, V. S. L., Jones, L. V., & Tukey, J. W. (1999) · Journal of Educational and Behavioral Statistics · doi:10.2307/1165261 · Cited by 186 · see citing works
Stepwise Multiple Testing as Formalized Data SnoopingPreprint
Wolf, M., & Romano, J. P. (2004) · SSRN Electronic Journal · doi:10.2139/ssrn.563209 · Cited by 143 · see citing works
Inequality, Too Much of a Good Thing
Krueger, A. B. (2018) · Routledge eBooks · doi:10.4324/9780429494468-4 · Cited by 116 · see citing works
Early Training Project, 1962-1979
Gray, S. W. (1988) · Harvard Dataverse · doi:10.7910/dvn/kez9tw · Cited by 75 · see citing works
Can intervention early prevent crime later? The abecedarian project compared with other programs
Clarke, S. H., & Campbell, F. A. (1998) · Early Childhood Research Quarterly · doi:10.1016/s0885-2006(99)80042-8 · Cited by 73 · see citing works
Into Adulthood: A Study of the Effects of Head Start
Oden, S. L., Schweinhart, L. J., & Weikart, D. P. (2000) · Cited by 66 · see citing works
Mizell, C. A., & Steelman, L. C. (2000) · Journal of Family Issues · doi:10.1177/019251300021007003 · Cited by 44 · see citing works
Anderson, M. (2005) · SSRN Electronic Journal · doi:10.2139/ssrn.812426 · Cited by 28 · see citing works
Self-citationRelationship between Exposureto Class Size Reductionand Student Achievementin California
Stecher, B. M., McCaffrey, D. F., & Bugliari, D. (2003) · Education Policy Analysis Archives · doi:10.14507/epaa.v11n40.2003 · Cited by 6 · see citing works
Flagged first, then by citations
1 of the 37 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Mostly Harmless Econometrics: An Empiricist's Companion
Angrist, J. D., & Pischke, J.-S. (2009) · doi:10.1515/9781400829828 · Cited by 9,467 · see citing works
The Oregon Health Insurance Experiment: Evidence from the First Year*
Finkelstein, A. N., Taubman, S. L., Wright, B. J., Mira Bernstein, A., B, J. G. A., Newhouse, J. P., ALLEN, H. L., D, K. B. A., & Oregon Health Study Group (2012) · The Quarterly Journal of Economics · doi:10.1093/qje/qjs020 · Cited by 1,661 · see citing works
Heckman, J., Pinto, R., & Savelyev, P. (2013) · American Economic Review · doi:10.1257/aer.103.6.2052 · Cited by 1,628 · see citing works
The rate of return to the HighScope Perry Preschool Program
Heckman, J. J., Moon, S. H., Pinto, R., Savelyev, P. A., & Yavitz, A. (2010) · Journal of Public Economics · doi:10.1016/j.jpubeco.2009.11.001 · Cited by 1,341 · see citing works
Human capital development before age five
Currie, J., & Almond, D. V. (2011) · Handbook of labour economics · doi:10.1016/s0169-7218(11)02413-0 · Cited by 1,214 · see citing works
“Anderson (2008) conducts a re-analysis of the Perry Preschool and Abcedarian data (and a third intervention called the Earkly Training Project) and finds that like MTO public housing experiment, the significant effects of the intervention was largely concentrated among girls.”
A multifaceted program causes lasting progress for the very poor: Evidence from six countries
Banerjee, A., Duflo, E., Goldberg, N., Karlan, D., Osei, R., Parienté, W., Shapiro, J., Thuysbaert, B., & Udry, C. (2015) · Science · doi:10.1126/science.1260799 · Cited by 1,075 · see citing works
THE SHORT-TERM IMPACT OF UNCONDITIONAL CASH TRANSFERS TO THE POOR: EXPERIMENTAL EVIDENCE FROM KENYA
Haushofer, J., & Shapiro, J. P. (2016) · Quarterly Journal of Economics · doi:10.1093/qje/qjw025 · Cited by 947 · see citing works
Expanding Credit Access: Using Randomized Supply Decisions to Estimate the Impacts
Karlan, D., & Zinman, J. (2009) · Review of Financial Studies · doi:10.1093/rfs/hhp092 · Cited by 899 · see citing works
“http://www.econ.yale.edu/~egcenter/research.htm”
The welfare effects of social media
Allcott, H., Braghieri, L., Eichmeyer, S., & Gentzkow, M. (2020) · American Economic Review · doi:10.1257/aer.20190658 · Cited by 857 · see citing works
Long-Run Impacts of Childhood Access to the Safety Net
Hoynes, H. W., Schanzenbach, D. W., & Almond, D. V. (2016) · American Economic Review · doi:10.1257/aer.20130375 · Cited by 793 · see citing works
Vandell, D. L., Belsky, J., Burchinal, M. A., Steinberg, L. D., & Vandergrift, N. (2010) · Child Development · doi:10.1111/j.1467-8624.2010.01431.x · Cited by 798 · see citing works
“Although two recent long-term follow-ups of early educational interventions have reported gender-moderated effects (Anderson, 2008; Belfield et al., 2006), we found no evidence of differential child-care effects as a function of child gender in either cognitive–academic performance or problem…”
Investing in Preschool Programs
Duncan, G. J., & Magnuson, K. (2013) · The Journal of Economic Perspectives · doi:10.1257/jep.27.2.109 · Cited by 764 · see citing works
Information from Markets Near and Far: Mobile Phones and Agricultural Markets in Niger
Aker, J. C. (2010) · American Economic Journal Applied Economics · doi:10.1257/app.2.3.46 · Cited by 757 · see citing works
“13 I also use a variant of the nonparametric permutation test (Bradley Efron and Robert Tibshirani 1993; Michael L. Anderson 2008), which computes the null distribution of the test statistic under the assumptions of random assignment and no treatment effect.”
MethodWeiland, C., & Yoshikawa, H. (2013) · Child Development · doi:10.1111/cdev.12099 · Cited by 701 · see citing works
“A recent study pooling the Perry, Abecedarian, and Early Training Project data found stronger benefits for girls than boys (Anderson, 2008).”
BackgroundEarly Childhood Intervention and Life-Cycle Skill Development: Evidence from Head Start
Deming, D. (2009) · American Economic Journal: Applied Economics · doi:10.1257/app.1.3.111 · Cited by 695 · see citing works
“…is about one-third of the size of the outcome gap between the bottom quartile and the median permanent income in the CNLSY sample, and is about 80 percent as large as the gains from the Perry Preschool and Carolina Abecedarian model preschool programs ( Carneiro and Heckman 2003; Anderson 2008 .”
InfluentialYoung, A. (2018) · The Quarterly Journal of Economics · doi:10.1093/qje/qjy029 · Cited by 637 · see citing works
“Several recent papers (Anderson 2008, Heckman et al 2010, Lee & Shaikh 2014, List, Shaikh & Xu 2016) have explored the robustness of individually significant results to stepdown randomization multiple-testing procedures in a few experiments.”
BackgroundSocial Media, News Consumption, and Polarization: Evidence from a Field Experiment
Levy, R. (2021) · American Economic Review · doi:10.1257/aer.20191777 · Cited by 624 · see citing works
Lifestyle and mental health disruptions during COVID-19
Giuntella, O., Hyde, K., Saccardo, S., & Sadoff, S. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2016632118 · Cited by 592 · see citing works
Rural Roads and Local Economic Development
Asher, S., & Novosad, P. M. (2020) · American Economic Review · doi:10.1257/aer.20180268 · Cited by 586 · see citing works
Strengthening State Capabilities: The Role of Financial Incentives in the Call to Public Service*
Bó, E. D., Finan, F., & Rossi, M. A. (2013) · The Quarterly Journal of Economics · doi:10.1093/qje/qjt008 · Cited by 553 · see citing works
Cantoni, D., Chen, Y., Yang, D. Y., Yuchtman, N., & Zhang, Y. J. (2014) · Journal of Political Economy · doi:10.1086/690951 · Cited by 556 · see citing works
Social Media and Mental Health
Braghieri, L., Levy, R., & Makarin, A. (2022) · American Economic Review · doi:10.1257/aer.20211218 · Cited by 617 · see citing works
Duncan, G. J., Morris, P., & Rodrigues, C. A. G. (2011) · Developmental Psychology · doi:10.1037/a0023875 · Cited by 519 · see citing works
“As shown in the first column, averaging all available achievement reports for each child increased both coefficient estimates and standard errors modestly (Anderson, 2008).”
BackgroundEnvironmental Health Risks and Housing Values: Evidence from 1,600 Toxic Plant Openings and Closings
Currie, J., Davis, L. W., Greenstone, M., & Walker, R. (2015) · American Economic Review · doi:10.1257/aer.20121656 · Cited by 501 · see citing works
Thinking, fast and slow? Some field experiments to reduce crime and dropout in Chicago
Heller, S. B., Shah, A. K., Guryan, J., Ludwig, J., Mullainathan, S., & Pollack, H. A. (2017) · The Quarterly Journal of Economics · doi:10.1093/qje/qjw033 · Cited by 493 · see citing works
“A sociological discussion of code-switching with respect to other behaviors is in Anderson (1999).”
Human Capital Development Before Age Five
Almond, D. V., & Currie, J. (2010) · National Bureau of Economic Research · doi:10.3386/w15827 · Cited by 488 · see citing works
Multiple hypothesis testing in experimental economics
List, J. A., Shaikh, A. M., & Xu, Y. (2019) · Experimental Economics · doi:10.1007/s10683-018-09597-5 · Cited by 465 · see citing works
“(2007), who use a more restrictive resampling-based multiple testing procedure due to Westfall and Young (1993) and Anderson (2008), Heckman et al.”
Civil War and Social Cohesion: Lab‐in‐the‐Field Evidence from Nepal
Gilligan, M., Pasquale, B., & Samii, C. (2013) · American Journal of Political Science · doi:10.1111/ajps.12067 · Cited by 464 · see citing works
“By creating new differences in the collective action potential of communities—that is, by transforming the “social cohesion” landscape—wartime violence may fundamentally alter a country’s grassroots politics (Wood 2003, 2008).”
Reshaping Institutions: Evidence on Aid Impacts Using a Pre-Analysis Plan
Casey, K., Glennerster, R., & Miguel, E. (2012) · The Quarterly Journal of Economics · doi:10.1093/qje/qje027 · Cited by 524 · see citing works
Transparency, Reproducibility, and the Credibility of Economics Research
Christensen, G., & Miguel, E. A. (2018) · Journal of Economic Literature · doi:10.1257/jel.20171350 · Cited by 448 · see citing works
Aker, J. C., Boumnijel, R., McClelland, A., & Tierney, N. (2016) · Economic Development and Cultural Change · doi:10.1086/687578 · Cited by 430 · see citing works
Eggers, A. C., Fowler, A., Hainmueller, J., Hall, A. B., & Snyder, J. M. (2014) · American Journal of Political Science · doi:10.1111/ajps.12127 · Cited by 424 · see citing works
Financial Incentives and Student Achievement: Evidence from Randomized Trials *
Fryer, R. G. (2011) · The Quarterly Journal of Economics · doi:10.1093/qje/qjr045 · Cited by 416 · see citing works
Swedberg, R. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.002 · Cited by 408 · see citing works
The Effects of High Stakes High School Achievement Awards: Evidence from a Randomized Trial
Angrist, J. D., & Lavy, V. (2009) · American Economic Review · doi:10.1257/aer.99.4.1384 · Cited by 397 · see citing works
Ultra-processed food consumption among US adults from 2001 to 2018
Juul, F., Parekh, N., Steele, E. M., Monteiro, C. A., & Chang, V. W. (2021) · American Journal of Clinical Nutrition · doi:10.1093/ajcn/nqab305 · Cited by 413 · see citing works
Geographic Boundaries as Regression Discontinuities
Keele, L. J., & Titiunik, R. (2015) · Political Analysis · doi:10.1093/pan/mpu014 · Cited by 391 · see citing works
Milligan, K. S., & Stabile, M. (2011) · American Economic Journal Economic Policy · doi:10.1257/pol.3.3.175 · Cited by 384 · see citing works
Heckman, J., Pinto, R., & Savelyev, P. (2012) · doi:10.3386/w18581 · Cited by 415 · see citing works
Technological Innovations, Downside Risk, and the Modernization of Agriculture
Emerick, K., de Janvry, A., Sadoulet, E., & Dar, M. H. (2016) · American Economic Review · doi:10.1257/aer.20150474 · Cited by 375 · see citing works
What Does Human Capital Do? A Review of Goldin and Katz's The Race between Education and Technology
Acemoğlu, D., & Autor, D. (2012) · Journal of Economic Literature · doi:10.1257/jel.50.2.426 · Cited by 364 · see citing works
Chuang, Y., & Schechter, L. (2015) · Journal of Development Economics · doi:10.1016/j.jdeveco.2015.07.008 · Cited by 359 · see citing works
“We use the calculation designed by Benjamini and Hochberg (1995) and explored in detail by Anderson (2008).”
MethodReducing Crime and Violence: Experimental Evidence from Cognitive Behavioral Therapy in Liberia
Blattman, C., Jamison, J. C., & Sheridan, M. (2017) · American Economic Review · doi:10.1257/aer.20150503 · Cited by 355 · see citing works
“33Other papers taking this approach include Kling et al. (2007); Casey et al. (2012); Anderson (2008).”
How do social media feed algorithms affect attitudes and behavior in an election campaign?
Guess, A. M., Malhotra, N., Pan, J., Barberá, P., Allcott, H., Brown, T., Crespo-Tenorio, A., Dimmery, D., Freelon, D., Gentzkow, M., et al. (2023) · Science · doi:10.1126/science.abp9364 · Cited by 363 · see citing works
Becker, S. O., Egger, P. H., & Ehrlich, M. V. (2013) · American Economic Journal Economic Policy · doi:10.1257/pol.5.4.29 · Cited by 351 · see citing works
School Choice, School Quality, and Postsecondary Attainment
Deming, D. J., Hastings, J., Kane, T. J., & Staiger, D. O. (2014) · American Economic Review · doi:10.1257/aer.104.3.991 · Cited by 335 · see citing works
Moved to Opportunity: The Long-Run Effects of Public Housing Demolition on Children
Chyn, E. T. (2018) · American Economic Review · doi:10.1257/aer.20161352 · Cited by 332 · see citing works
Chernozhukov, V., Chetverikov, D., & Kato, K. (2013) · The Annals of Statistics · doi:10.1214/13-aos1161 · Cited by 325 · see citing works
“The latter possibility allows us to cover the case of testing multiple coefficients in multiple regressions, which is often required in empirical studies; see, for example, [2].”
BackgroundChernozhukov, V., Chetverikov, D., & Kato, K. (2013) · doi:10.1920/wp.cem.2013.7613 · Cited by 312 · see citing works
Field, E. M., Pande, R., Rigol, N., Schaner, S. G., & Moore, C. (2021) · The American Economic Review · doi:10.1257/aer.20200705 · Cited by 121 · see citing works
“As guidance for assessing multiple comparison concerns with our subsequent analysis, we estimate sharpened q -values that control the false discovery rate ( FDR ) ( Benjamini, Krieger, and Yekutieli 2006; Anderson 2008 .”
(2010) · Quantitative Economics · doi:10.3982/qe8 · Cited by 304 · see citing works
“Others express the concern that previous analyses selectively report statistically signifi-cant estimates, biasing the inference about the program (Anderson (2008)).”
InfluentialDobbie, W., & Fryer, R. (2009) · doi:10.3386/w15473 · Cited by 299 · see citing works
Angrist, J. D., Cohodes, S. R., Dynarski, S. M., Pathak, P. A., & Walters, C. R. (2016) · Journal of Labor Economics · doi:10.1086/683665 · Cited by 294 · see citing works
“In view of evidence that many educational interventions do not work well for males, this finding seems noteworthy ð see, e.g., Anderson 2008; Angrist, Lang, and Oreopoulos 2009; Legewie and DiPrete 2012; Deming et al. 2014 Þ .”
The Good, the Bad, and the Average: Evidence on Ability Peer Effects in Schools
Lavy, V., Silva, O., & Weinhardt, F. (2012) · Journal of Labor Economics · doi:10.1086/663592 · Cited by 297 · see citing works
“…is slightly negative for boys at 0.040 (SE 0.040), although not significant, but positive, significant, and sizable at 0.122 (SE 21 For example, Anderson (2008) shows that three well-known early childhood interventions (namely, Abecedarian, Perry, and the Early Training Project) had…”
BackgroundMethodInfluentialThe RAND Health Insurance Experiment, Three Decades Later
Aron-Dine, A., Liran Einav, B., & Finkelstein, A. N. (2013) · The Journal of Economic Perspectives · doi:10.1257/jep.27.1.197 · Cited by 292 · see citing works
Does Money Matter? The Effects of Cash Transfers on Child Development in Rural Ecuador
Paxson, C. H., & Schady, N. (2010) · Economic Development and Cultural Change · doi:10.1086/655458 · Cited by 281 · see citing works
Worms at Work: Long-run Impacts of a Child Health Investment*
Baird, S., Hicks, J. H., Kremer, M. R., & Miguel, E. A. (2016) · The Quarterly Journal of Economics · doi:10.1093/qje/qjw022 · Cited by 266 · see citing works
“Anderson (2008) presents multiple instances in which the FDR q-value is smaller than the per-comparison (naïve) P-value, and this occurs for several outcomes in our data.”
InfluentialThe Romano–Wolf multiple-hypothesis correction in Stata
Clarke, D., Romano, J. P., & Wolf, M. (2020) · The Stata Journal Promoting communications on statistics and Stata · doi:10.1177/1536867x20976314 · Cited by 269 · see citing works
The Effect of Classmate Characteristics on Post-Secondary Outcomes: Evidence from the Add Health
Bifulco, R., Fletcher, J. M., & Ross, S. L. (2011) · American Economic Journal Economic Policy · doi:10.1257/pol.3.1.25 · Cited by 264 · see citing works
Reshaping Adolescents' Gender Attitudes: Evidence from a School-Based Experiment in India
Dhar, D., Jain, T., & Jayachandran, S. (2022) · American Economic Review · doi:10.1257/aer.20201112 · Cited by 261 · see citing works
U.S. High School Graduation Rates: Patterns and Explanations
Murnane, R. J. (2013) · Journal of Economic Literature · doi:10.1257/jel.51.2.370 · Cited by 260 · see citing works
Does information change attitudes toward immigrants?
Grigorieff, A., Roth, C., & Ubfal, D. (2020) · Demography · doi:10.1007/s13524-020-00882-8 · Cited by 254 · see citing works
“17For each family of outcomes, we control for a false discovery rate of 5 percent (Anderson 2008).”
InfluentialDobbie, W., & Fryer, R. G. (2011) · American Economic Journal: Applied Economics · doi:10.1257/app.3.3.158 · Cited by 536 · see citing works
The Impact of Media Censorship: 1984 or Brave New World?
Chen, Y., & Yang, D. Y. (2019) · American Economic Review · doi:10.1257/aer.20171765 · Cited by 250 · see citing works
“E categories are summarized by a z -score index, weighting by the inverse covariance of the standardized variables, following Anderson ( 2008 ) .”
InfluentialExperimental Evidence on the Economics of Rural Electrification
Lee, K., Miguel, E. A., & Wolfram, C. (2019) · Journal of Political Economy · doi:10.1086/705417 · Cited by 249 · see citing works
“…summarized in table 3, panel E, which combines the primary economic outcomes (from panel C) into a mean-effect economic index and the primary noneconomic outcomes (from panel D) 32 As per our preanalysis plans, we follow the FDR approach of Anderson (2008) and Casey, Glennerster, and Miguel (2012).”
MethodBuilding Nations through Shared Experiences: Evidence from African Football
Depetris-Chauvin, E., Durante, R., & Campante, F. (2020) · The American Economic Review · doi:10.1257/aer.20180805 · Cited by 201 · see citing works
“Also, since we run our econometric model on multiple outcomes and we test multiple heterogeneous effects, we report False Discovery Rate ( FDR ) adjusted p -values following Anderson ( 2008 ) .”
Administrative Unit Proliferation
Grossman, G., & Lewis, J. I. (2014) · American Political Science Review · doi:10.1017/s0003055413000567 · Cited by 245 · see citing works
Clans, entrepreneurship, and development of the private sector in China
Zhang, C. (2016) · Journal of Comparative Economics · doi:10.1016/j.jce.2019.08.008 · Cited by 260 · see citing works
Allcott, H., Gentzkow, M. A., & Song, L. (2022) · American Economic Review · doi:10.1257/aer.20210867 · Cited by 243 · see citing works
The Political Legacies of Combat: Attitudes Toward War and Peace Among Israeli Ex-Combatants
Grossman, G., Manekin, D. S., & Miodownik, D. (2015) · International Organization · doi:10.1017/s002081831500020x · Cited by 238 · see citing works
Paler, L. (2013) · American Political Science Review · doi:10.1017/s0003055413000415 · Cited by 229 · see citing works
Keys, T. D., Farkas, G., Burchinal, M., Duncan, G. J., Vandell, D. L., Li, W., Ruzek, E. A., & Howes, C. H. (2013) · Child Development · doi:10.1111/cdev.12048 · Cited by 239 · see citing works
“Two studies reported that boys appear to benefit more than girls from child-care experiences (Brooks-Gunn, Han, & Waldfogel, 2002; Votruba-Drzal et al., 2004), while reanalysis of the Abecedarian and Perry Preschool Projects found the opposite (Anderson, 2008).”
Impact of Village Savings and Loan Associations: Evidence from a cluster randomized trial
Ksoll, C., Lilleør, H. B., Lønborg, J. H., & Rasmussen, O. D. (2015) · Journal of Development Economics · doi:10.1016/j.jdeveco.2015.12.003 · Cited by 235 · see citing works
“When multiple hypotheses are tested, the probability of rejecting at least one null hypothesis is higher than the signi fi cance level, even when a program has no effect on any of the outcomes (see, for example, Anderson, 2008).”
BackgroundMethodInfluentialWhat Happens When Extremists Win Primaries?
Hall, A. B. (2015) · American Political Science Review · doi:10.1017/s0003055414000641 · Cited by 231 · see citing works
Macours, K., Schady, N., & Vakis, R. (2012) · American Economic Journal Applied Economics · doi:10.1257/app.4.2.247 · Cited by 224 · see citing works
Dynarski, S., Hyman, J., & Schanzenbach, D. W. (2013) · Journal of Policy Analysis and Management · doi:10.1002/pam.21715 · Cited by 223 · see citing works
“The Perry Preschool Program had no statistically significant effect on postsecondary outcomes (Anderson, 2008).”
Cornwell, C., Mustard, D. B., & Van Parys, J. (2013) · Journal of Human Resources · doi:10.1353/jhr.2013.0002 · Cited by 222 · see citing works
Data-intensive Innovation and the State: Evidence from AI Firms in China
Beraja, M., Yang, D. Y., & Yuchtman, N. (2022) · The Review of Economic Studies · doi:10.1093/restud/rdac056 · Cited by 217 · see citing works
Long-Term Impacts of Compensatory Preschool on Health and Behavior: Evidence from Head Start
Carneiro, P., & Ginja, R. (2014) · American Economic Journal Economic Policy · doi:10.1257/pol.6.4.135 · Cited by 215 · see citing works
Protests as Strategic Games: Experimental Evidence from Hong Kong's Antiauthoritarian Movement*
Cantoni, D., Yang, D. Y., Yuchtman, N., & Zhang, Y. J. (2019) · The Quarterly Journal of Economics · doi:10.1093/qje/qjz002 · Cited by 209 · see citing works
The Rate of Return to the High/Scope Perry Preschool Program
Heckman, J. J., Moon, S. H., Pinto, R. R., Savelyev, P. A., & Yavitz, A. (2009) · National Bureau of Economic Research · doi:10.3386/w15471 · Cited by 212 · see citing works
Baranov, V., Bhalotra, S. R., Biroli, P., & Maselko, J. (2020) · American Economic Review · doi:10.1257/aer.20180511 · Cited by 202 · see citing works
Summer jobs reduce violence among disadvantaged youth
Heller, S. B. (2014) · Science · doi:10.1126/science.1257809 · Cited by 206 · see citing works
Milli, S., Carroll, M., Wang, Y., Pandey, S., & Dragan, A. D. (2023) · arXiv.org · doi:10.48550/arxiv.2305.16941 · Cited by 200 · see citing works
Synthetic Control Method: Inference, Sensitivity Analysis and Confidence Sets
FIRPO, S. P., & Possebom, V. (2018) · Journal of Causal Inference · doi:10.1515/jci-2016-0026 · Cited by 192 · see citing works
“After adapting the familywise error rate control methodology suggested by Anderson (2008) to our framework, we propose one way to test any sharp null hypothesis for a large number of outcome variables, preserving the correct test size for each variable of interest.”
BackgroundMethodInfluentialCampenhout, B. V., Spielman, D. J., & Lecoutere, E. (2020) · American Journal of Agricultural Economics · doi:10.1002/ajae.12089 · Cited by 196 · see citing works
“” To guard against over-rejection of the null hypothesis due to multiple inference, outcomes of the knowledge questions were combined into an index, constructed as the weighted mean of the individual standardized outcomes, using as weights the inverse of the co-variance matrix of the transformed outcomes (Anderson, 20…”
BackgroundMethodInfluentialTackling Youth Unemployment: Evidence From a Labor Market Experiment in Uganda
Alfonsi, L., Bandiera, O., Bassi, V., Burgess, R., Imran Rasul, A., Sulaiman, M., & Vitali, A. (2020) · Econometrica · doi:10.3982/ecta15959 · Cited by 196 · see citing works
Boy-Girl Differences in Parental Time Investments: Evidence from Three Countries
Baker, M., & Milligan, K. S. (2013) · Journal of Human Capital · doi:10.1086/688899 · Cited by 194 · see citing works
“, Angrist and Evans [1998] for the United States and McDougall, DeWit, and Ebanks [1999] for Canada).”
BackgroundSeeing is believing? Evidence from an extension network experiment
Kondylis, F., Mueller, V., & Zhu, J. (2017) · Journal of Development Economics · doi:10.1016/j.jdeveco.2016.10.004 · Cited by 191 · see citing works
“31 We also follow Anderson (2008) and address multiple inference in two additional ways (not reported): (1) using a free step-down resampling method to our p-values for familywise error rate, and (2) employing the false discovery rate control methodology proposed by Benjamini and Hochberg (1995).”
MethodWhen the money runs out: Do cash transfers have sustained effects on human capital accumulation?
Baird, S., McIntosh, C., & Özler, B. (2019) · Journal of Development Economics · doi:10.1016/j.jdeveco.2019.04.004 · Cited by 192 · see citing works
“In Appendix Table S12, we present q-values controlling for FDR, as described in Anderson (2008).”
The effect of the affordable care act Medicaid expansions on financial wellbeing
Hu, L., Kaestner, R. J., Bhashkar Mazumder, B., Miller, S., & Wong, A. (2018) · Journal of Public Economics · doi:10.1016/j.jpubeco.2018.04.009 · Cited by 191 · see citing works
“23Because of the computationally intensive way our p-values are calculated, it is not feasible for us to use more powerful family-wise error rate controls that rely on bootstrapping such as those described in Anderson (2008). Hu et al.”
MethodThe Role of Parallel Trends in Event Study Settings: An Application to Environmental Economics
Marcus, M., & Sant’Anna, P. H. C. (2021) · Journal of the Association of Environmental and Resource Economists · doi:10.1086/711509 · Cited by 192 · see citing works
“As a direct consequence, significant treatment effects may emerge simply by chance, even when all ATT (g, t)’s are equal to zero, see, e.g., Romano and Wolf (2005), Anderson (2008) and section 8 of Romano et al. (2010).”
The Power of Social Pensions: Evidence from China's New Rural Pension Scheme
Huang, W., & Zhang, C. (2021) · American Economic Journal Applied Economics · doi:10.1257/app.20170789 · Cited by 189 · see citing works
Multiple Hypothesis Testing in Experimental Economics
List, J. A., Shaikh, A. M., & Xu, Y. (2016) · National Bureau of Economic Research · doi:10.3386/w21875 · Cited by 188 · see citing works
Cohen, J. L., Pascaline Dupas, Y., & Schaner, S. G. (2015) · American Economic Review · doi:10.1257/aer.20130267 · Cited by 186 · see citing works
Auerbach, A. M. (2015) · World Politics · doi:10.1017/s0043887115000313 · Cited by 184 · see citing works
Liebman, J. B., & Luttmer, E. F. P. (2015) · American Economic Journal Economic Policy · doi:10.1257/pol.20120081 · Cited by 182 · see citing works
Haushofer, J., & Shapiro, J. (2013) · Cited by 181 · see citing works
Exhaustive or exhausting? Evidence on respondent fatigue in long surveys
Jeong, D., Aggarwal, S., Robinson, J., Kumar, N., Spearot, A., & Park, D. S. (2023) · Journal of Development Economics · doi:10.1016/j.jdeveco.2022.102992 · Cited by 175 · see citing works
The Effect of Macroeconomic Uncertainty on Household Spending
Coibion, O., Georgarakos, D., Gorodnichenko, Y., Kenny, G., & Weber, M. (2024) · American Economic Review · doi:10.1257/aer.20221167 · Cited by 168 · see citing works
- Authors
- Anderson, M. L.
- Affiliations
- University of California, Berkeley (US)
- Corresponding author
- Anderson, Michael L.
- ORCID
- Anderson, Michael L.
- Venue
- Journal of the American Statistical Association
- Publisher
- Informa UK Limited
- Published
- 2008-12-01
- Volume
- 103
- Issue
- 484
- Pages
- 1481-1495
- ISSN
- 0162-1459 (ISSN-L)
- 1537-274X
- OpenAlex
- W2099805722
- Cited by
- 2,410
- References
- 37
- Citations by year
- 2026: 210 · 2025: 258 · 2024: 262 · 2023: 266 · 2022: 258 · 2021: 240 · 2020: 187 · 2019: 147 · 2018: 117 · 2017: 85 · 2016: 93 · 2015: 65 · 2014: 47 · 2013: 68 · 2012: 37
- FWCI
- 99.90
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of the American Statistical Association (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- eScholarship (California Digital Library) (submittedVersion)
- Topics
- Social Sciences › Education › Early Childhood Education and Development
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Keywords
- gender differences, early childhood intervention, Abecedarian Project, treatment effect heterogeneity, family-wise error rate, false discovery rate, returns to education
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
