Cited by 194Advanced Causal Inference TechniquesEvaluation and Performance AssessmentPoverty, Education, and Child Welfare
Experimentation at Scale
Open accessMuralidharan, Karthik, Niehaus, Paul·2017·Journal of Economic Perspectives
Abstract
This paper makes the case for greater use of randomized experiments “at scale.” We review various critiques of experimental program evaluation in developing countries, and discuss how experimenting at scale along three specific dimensions—the size of the sampling frame, the number of units treated, and the size of the unit of randomization— can help alleviate the concerns raised. We find that program-evaluation randomized controlled trials published over the last 15 years have typically been “small” in these senses, but also identify a number of examples—including from our own work— demonstrating that experimentation at much larger scales is both feasible and valuable.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Crossref · 10.1257/jep.31.4.103
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Results from RCTs are not always scalable to larger, aggregated levels because market responses and spillovers are largely absent at the small scale of the experiments.
“the results from RCTs are not always scalable to larger, aggregated levels because market responses and spillovers are largely absent at the small scale of the experiments ( 44 , 45 ).”
From Environmental impacts of genetically modified crops (2024) · cites this paper as [44]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
The citation lists haven’t been fetched yet
References 24 · Cited by 194
Research works printed but not linked: 88
- Banerjee, Abhijit, Rema Hanna, Jordan Kyle, Benjamin A. Olken, and Sudarno Sumarto. Forthcoming. "Tangible Information and Citizen Empowerment: Identification Cards and Food Subsidy Programs in Indonesia." Journal of Political Economy.
- de Ree, Joppe, Karthik Muralidharan, Menno Pradhan, and Halsey Rogers. Forth- coming. "Double for Nothing? Experimental Evidence on an Unconditional Teacher Salary Increase in Indonesia." Quarterly Journal of Economics.
- Duflo, Esther, and Abhijit Banerjee, ed. 2017. Handbook of Field Experiments, vol. 1. Handbooks in Economics. Amsterdam: Elsevier.
- Gertler, Paul J., and Simone Boyce. 2003. "An Experiment in Incentive-Based Welfare: The Impact of PROGRESA on Health in Mexico." Paper presented at Royal Economic Society Annual Conference, Coventry, UK, April 7-9.
- Gueron, Judith M. 2017. "The Politics and Prac- tice of Social Experiments: Seeds of a Revolution." In Handbook of Field Experiments, vol. 1, edited by Abhijit Vinayak Banerjee and Esther Duflo, 27-69. Amsterdam: Elsevier.
- Levy, Santiago. 2006. Progress against Poverty: Sustaining Mexico's Progresa-Oportunidades Program. Washington, DC: Brookings Institution Press.
- Susan W Parker, Tom Vogl. 2023. Do Conditional Cash Transfers Improve Economic Outcomes in the Next Generation? Evidence from Mexico. The Economic Journal . [Crossref] · DOI
- Isaac Mbiti, Mauricio Romero, Youdi Schipper. 2023. Designing Effective Teacher Performance Pay Programs: Experimental Evidence from Tanzania*. The Economic Journal 133:653, 1968-2000. [Crossref] · DOI
- Francisco J. Buera, Joseph P. Kaboski, Robert M. Townsend. 2023. From Micro to Macro Development. Journal of Economic Literature 61:2, 471-503. [Abstract] [View PDF article] [PDF with links] · DOI
- Reetika Khera. 2023. Some questions of ethics in randomized controlled trials. Review of Development Economics 7. . [Crossref] · DOI
- Jean Drèze. 2023. The perils of embedded experiments. Review of Development Economics 127. . [Crossref] · DOI
- Laura Abramovsky, Britta Augsburg, Melanie Lührmann, Francisco Oteiza, Juan Pablo Rud. 2023. Community matters: Heterogeneous impacts of a sanitation intervention. World Development 165, 106197. [Crossref] · DOI
- David K. Evans. 2023. Towards improved and more transparent ethics in randomised controlled trials in development social science. Journal of Development Effectiveness 47, 1-11. [Crossref] · DOI
- Abhijit Banerjee, Rema Hanna, Benjamin A. Olken, Elan Satriawan, Sudarno Sumarto. 2023. Electronic Food Vouchers: Evidence from an At-Scale Experiment in Indonesia. American Economic Review 113:2, 514-547. [Abstract] [View PDF article] [PDF with links] · DOI
- Gaurav Khanna. 2023. Large-Scale Education Reform in General Equilibrium: Regression Discontinuity Evidence from India. Journal of Political Economy 131:2, 549-591. [Crossref] · DOI
- Erin Byrne, Caitlin Tulloch, Naeem Sohail, Silvia Diazgranados Ferráns. 2023. Harnessing cost data to improve early grade reading: cost evidence from a large-scale literacy initiative in Pakistan. Journal of Development Effectiveness 15:1, 43-76. [Crossref] · DOI
- Ashwin Nair, Burak Eskici. Digital Public Services: The Development of Biometric Authentication in India 533-561. [Crossref] · DOI
- Andrew Dustan, Juan Manuel Hernandez-Agramonte, Stanislao Maldonado. 2023. Motivating bureaucrats with behavioral insights when state capacity is weak: Evidence from large-scale field experiments in Peru. Journal of Development Economics 160, 102995. [Crossref] · DOI
- Matteo Bobba, Veronica Frisancho, Marco Pariguana. 2023. Perceived Ability and School Choices: Experimental Evidence and Scale-Up Effects. SSRN Electronic Journal 107. . [Crossref] · DOI
- Karthik Muralidharan, Paul Niehaus, Sandip Sukhtankar. 2023. General Equilibrium Effects of (Improving) Public Employment Programs: Experimental Evidence From India. Econometrica 91:4, 1261-1295. [Crossref] · DOI
- Guthrie Gray-Lobe, Parag A Pathak, Christopher R Walters. 2022. The Long-Term Effects of Universal Preschool in Boston. The Quarterly Journal of Economics 138:1, 363-411. [Crossref] · DOI
- Arthur Bragança, Ricardo Dahis. 2022. Cutting special interests by the roots: Evidence from the Brazilian Amazon. Journal of Public Economics 215, 104753. [Crossref] · DOI
- Siwan Anderson. 2022. Unbundling female empowerment. Canadian Journal of Economics/Revue canadienne d'économique 55:4, 1671-1701. [Crossref] · DOI
- Lorenzo Casaburi, Tristan Reed. 2022. Using Individual-Level Randomized Treatment to Learn about Market Structure. American Economic Journal: Applied Economics 14:4, 58-90. [Abstract] [View PDF article] [PDF with links] · DOI
- Ole-Andreas Elvik Naess. 2022. Increasing turnout with a text message: evidence from a large campaign from the government. Journal of Elections, Public Opinion and Parties 32, 1-19. [Crossref] · DOI
- Patrick Premand, Oumar Barry. 2022. Behavioral change promotion, cash transfers and early childhood development: Experimental evidence from a government program in a low-income setting. Journal of Development Economics 158, 102921. [Crossref] · DOI
- Daniel Rodriguez-Segura. 2022. EdTech in Developing Countries: A Review of the Evidence. The World Bank Research Observer 37:2, 171-203. [Crossref] · DOI
- Lauren H. Supplee, Robert T. Ammerman, Anne K. Duggan, John A. List, Dana Suskind. 2022. The Role of Open Science Practices in Scaling Evidence-Based Prevention Programs. Prevention Science 23:5, 799-808. [Crossref] · DOI
- Christopher Hoy, Mathias Sinning, Luke McKenzie. 2022. Improving Tax Compliance without Increasing Revenue: Evidence from Population-Wide Randomized Controlled Trials in Papua New Guinea. Economic Development and Cultural Change 91013. . [Crossref] · DOI
- Thomas Bossuroy, Markus Goldstein, Bassirou Karimou, Dean Karlan, Harounan Kazianga, William Parienté, Patrick Premand, Catherine C. Thomas, Christopher Udry, Julia Vaillant, Kelsey A. Wright. 2022. Tackling psychosocial and capital constraints to alleviate poverty. Nature 605:7909, 291-297. [Crossref] · DOI
- 2022. Addressing social, psychological and economic barriers helps people out of extreme poverty. Nature 370. . [Crossref] · DOI
- Wenjia Joyce Zhao, Aoife Coady, Sudeep Bhatia. 2022. Computational mechanisms for context-based behavioral interventions: A large-scale analysis. Proceedings of the National Academy of Sciences 119:15. . [Crossref] · DOI
- Helen X.H. Bao, Guy M. Robinson. 2022. Behavioural land use policy studies: Past, present, and future. Land Use Policy 115, 106013. [Crossref] · DOI
- Quentin Stoeffler, Gülce Opuz. 2022. Price, information and product quality: Explaining index insurance demand in Burkina Faso. Food Policy 108, 102213. [Crossref] · DOI
- Mariella Gonzales, Gianmarco León-Ciliotta, Luis R. Martínez. 2022. How Effective Are Monetary Incentives to Vote? Evidence from a Nationwide Policy. American Economic Journal: Applied Economics 14:1, 293-326. [Abstract] [View PDF article] [PDF with links] · DOI
- Manuel Oechslin, Elias Steiner. 2022. Statistical capacity and corrupt bureaucracies. The Review of International Organizations 17:1, 143-174. [Crossref] · DOI
- Stefano DellaVigna, Elizabeth Linos. 2022. RCTs to Scale: Comprehensive Evidence From Two Nudge Units. Econometrica 90:1, 81-116. [Crossref] · DOI
- Dennis Egger, Johannes Haushofer, Edward Miguel, Paul Niehaus, Michael Walker. 2022. General Equilibrium Effects of Cash Transfers: Experimental Evidence From Kenya. Econometrica 90:6, 2603-2643. [Crossref] · DOI
- Sebastian Gallegos, Benjamin Roseth, Ana Cuesta, Mario A. Sanchez. 2022. Increasing the Take-Up of Public Health Services: An At-Scale Experiment on Digital Government. SSRN Electronic Journal 105. . [Crossref] · DOI
- Jules Gazeaud, Claire Ricard. 2022. Conditional Cash Transfers at Scale: A Regression Discontinuity Analysis in Morocco. SSRN Electronic Journal 58. . [Crossref] · DOI
- Stefano DellaVigna, Woojin Kim, Elizabeth Linos. 2022. Bottlenecks for Evidence Adoption. SSRN Electronic Journal 130. . [Crossref] · DOI
- Lauren Bergquist, Benjamin Faber, Thibault Fally, Matthias Hoelzlein, Edward Miguel, Andrés Rodríguez-Clare. 2022. Scaling Agricultural Policy Interventions. SSRN Electronic Journal 82. . [Crossref] · DOI
- Sylvain Chassang, Samuel Kapon. 2022. Designing Randomized Controlled Trials with External Validity in Mind. SSRN Electronic Journal 107. . [Crossref] · DOI
- Katy Bergstrom, Berk Ozler. Improving the Well-Being of Adolescent Girls in Developing Countries 15, . [Crossref] · DOI
- M. Danish Shakeel, Kaitlin P. Anderson, Patrick J. Wolf. 2021. The participant effects of private school vouchers around the globe: a meta-analytic and systematic review. School Effectiveness and School Improvement 32:4, 509-542. [Crossref] · DOI
- Florence Kondylis, John Loeser. Intervention Size and Persistence . [Crossref] · DOI
- Sunghee Choi, Seok-Joon Hwang, Arthur T. Denzau. 2021. Do households conserve electricity when they receive signals of greater consumption than neighbours? The Korean case. Energy 225, 120292. [Crossref] · DOI
- Michael G. Findley, Kyosuke Kikuta, Michael Denly. 2021. External Validity. Annual Review of Political Science 24:1, 365-393. [Crossref] · DOI
- Francesco Agostinelli, Ciro Avitabile, Matteo Bobba. Enhancing Human Capital at Scale . [Crossref] · DOI
- Jan-Emmanuel De Neve, Clément Imbert, Johannes Spinnewijn, Teodora Tsankova, Maarten Luts. 2021. How to Improve Tax Compliance? Evidence from Population-Wide Experiments in Belgium. Journal of Political Economy 129:5, 1425-1463. [Crossref] · DOI
- David McKenzie, Susana Puerto. 2021. Growing Markets through Business Training for Female Entrepreneurs: A Market-Level Randomized Experiment in Kenya. American Economic Journal: Applied Economics 13:2, 297-332. [Abstract] [View PDF article] [PDF with links] · DOI
- Michael Carter, Rachid Laajaj, Dean Yang. 2021. Subsidies and the African Green Revolution: Direct Effects and Social Network Spillovers of Randomized Input Subsidies in Mozambique. American Economic Journal: Applied Economics 13:2, 206-229. [Abstract] [View PDF article] [PDF with links] · DOI
- Karthik Muralidharan, Paul Niehaus, Sandip Sukhtankar, Jeffrey Weaver. 2021. Improving Last-Mile Service Delivery Using Phone-Based Monitoring. American Economic Journal: Applied Economics 13:2, 52-82. [Abstract] [View PDF article] [PDF with links] · DOI
- Thomas Bossuroy, Markus Goldstein, Dean Karlan, Harounan Kazianga, William Parienté, Patrick Premand, Catherine Thomas, Christopher Udry, Julia Vaillant, Kelsey Wright. Pathways out of Extreme Poverty: Tackling Psychosocial and Capital Constraints with a Multi-Faceted Social Protection Program in Niger 10, . [Crossref] · DOI
- Christopher Hoy, Luke McKenzie, Mathias Sinning. Improving Tax Compliance without Increasing Revenue: Evidence from Population-Wide Randomized Controlled Trials in Papua New Guinea . [Crossref] · DOI
- OMAR AL-UBAYDLI, MIN SOK LEE, JOHN A. LIST, CLAIRE L. MACKEVICIUS, DANA SUSKIND. 2021. How can experiments play a greater role in public policy? Twelve proposals from an economic model of scaling. Behavioural Public Policy 5:1, 2-49. [Crossref] · DOI
- ANANDI MANI. 2021. Experimental evidence, scaling and public policy: a perspective from developing countries. Behavioural Public Policy 5:1, 103-111. [Crossref] · DOI
- Helen X. H. Bao, Guy M. Robinson. 2021. Behavioural Land Use Policy Studies: Past, Present, and Future. SSRN Electronic Journal 31. . [Crossref] · DOI
- Alec Brandon, Christopher Clapp, John A. List, Robert D. Metcalfe, Michael K. Price. 2021. Smart Tech, Dumb Humans: The Perils of Scaling Household Technologies. SSRN Electronic Journal 2. . [Crossref] · DOI
- Martin Mattsson. 2021. Information Systems, Service Delivery, and Corruption: Evidence from the Bangladesh Civil Service. SSRN Electronic Journal 95. . [Crossref] · DOI
- José Ramón Enríquez, Horacio Larreguy, Alberto Simpser. 2021. Confounding, Spillovers and Interactions Influence Estimates of Social Distancing Policy Effects. SSRN Electronic Journal 136. . [Crossref] · DOI
- Nur Cahyadi, Rema Hanna, Benjamin A. Olken, Rizal Adi Prima, Elan Satriawan, Ekki Syamsulhakim. 2020. Cumulative Impacts of Conditional Cash Transfer Programs: Experimental Evidence from Indonesia. American Economic Journal: Economic Policy 12:4, 88-110. [Abstract] [View PDF article] [PDF with links] · DOI
- Diane Alexander. 2020. How Do Doctors Respond to Incentives? Unintended Consequences of Paying Doctors to Reduce Costs. Journal of Political Economy 128:11, 4046-4096. [Crossref] · DOI
- Omar Al-Ubaydli, John A. List, Dana Suskind. 2020. 2017 KLEIN LECTURE: THE SCIENCE OF USING SCIENCE: TOWARD AN UNDERSTANDING OF THE THREATS TO SCALABILITY. International Economic Review 61:4, 1387-1409. [Crossref] · DOI
- Johannes Binswanger, Manuel Oechslin. 2020. Better statistics, better economic policies?. European Economic Review 130, 103588. [Crossref] · DOI
- Di Mo, Yu Bai, Yaojiang Shi, Cody Abbey, Linxiu Zhang, Scott Rozelle, Prashant Loyalka. 2020. Institutions, implementation, and program effectiveness: Evidence from a randomized evaluation of computer-assisted learning in rural China. Journal of Development Economics 146, 102487. [Crossref] · DOI
- Catalina Herrera-Almanza, Maria F. Rosales-Rueda. 2020. Reducing the Cost of Remoteness: Community-Based Health Interventions and Fertility Choices. Journal of Health Economics 73, 102365. [Crossref] · DOI
- Martin J Williams. 2020. External Validity and Policy Adaptation: From Impact Evaluation to Policy Design. The World Bank Research Observer 35:2, 158-191. [Crossref] · DOI
- Ingrid Harvold Kvangraven. 2020. Nobel Rebels in Disguise -Assessing the Rise and Rule of the Randomistas. Review of Political Economy 32:3, 305-341. [Crossref] · DOI
- Esther Duflo. 2020. Field Experiments and the Practice of Policy. American Economic Review 110:7, 1952-1973. [Citation] [View PDF article] [PDF with links] · DOI
- Benjamin A. Olken. 2020. Banerjee, Duflo, Kremer, and the Rise of Modern Development Economics*. The Scandinavian Journal of Economics 122:3, 853-878. [Crossref] · DOI
- James Berry, Saurabh Mehta, Priya Mukherjee, Hannah Ruebeck, Gauri Kartini Shastry. 2020. Implementation and effects of India's national school-based iron supplementation program. Journal of Development Economics 144, 102428. [Crossref] · DOI
- Gopal Naik, Chetan Chitre, Manaswini Bhalla, Jothsna Rajan. 2020. Impact of use of technology on student learning outcomes: Evidence from a large-scale experiment in India. World Development 127, 104736. [Crossref] · DOI
- Yana van der Meulen Rodgers, Anthony Bebbington, Catherine Boone, Jampel Dell'Angelo, Jean- Philippe Platteau, Arun Agrawal. 2020. Experimental approaches in development and poverty alleviation. World Development 127, 104807. [Crossref] · DOI
- C. Austin Davis, Ahmed Mushfiq Mobarak. 2020. The challenges of scaling effective interventions: A path forward for research and policy. World Development 127, 104817. [Crossref] · DOI · Paper page
- Vijayendra Rao. 2020. Evidence-based development needs a diversity of tools, with a bottom-up process of "embedded" dialogue. World Development 127, 104823. [Crossref] · DOI
- Abhijit Banerjee, Rema Hanna, Benjamin A. Olken, Sudarno Sumarto. 2020. The (lack of ) distortionary effects of proxy-means tests: Results from a nationwide experiment in Indonesia. Journal of Public Economics Plus 1, 100001. [Crossref] · DOI
- Alberto Abadie, Susan Athey, Guido W. Imbens, Jeffrey M. Wooldridge. 2020. Sampling-Based versus Design-Based Uncertainty in Regression Analysis. Econometrica 88:1, 265-296. [Crossref] · DOI
- Clément de Chaisemartin, Jaime Ramirez-Cuellar. 2020. At What Level Should One Cluster Standard Errors in Paired Experiments, and in Stratified Experiments with Small Strata?. SSRN Electronic Journal 88. . [Crossref] · DOI
- Eszter Czibor, David Jimenez-Gomez, John A. List. 2019. The Dozen Things Experimental Economists Should Do (More of ). Southern Economic Journal 86:2, 371-432. [Crossref] · DOI
- Isaac Mbiti, Karthik Muralidharan, Mauricio Romero, Youdi Schipper, Constantine Manda, Rakesh Rajani. 2019. Inputs, Incentives, and Complementarities in Education: Experimental Evidence from Tanzania*. The Quarterly Journal of Economics 134:3, 1627-1673. [Crossref] · DOI
- Lisa Cameron, Susan Olivia, Manisha Shah. 2019. Scaling up sanitation: Evidence from an RCT in Indonesia. Journal of Development Economics 138, 1-16. [Crossref] · DOI
- Sam Harper. 2019. Comment on the equity impact of women's community groups on inequalities in neonatal mortality. International Journal of Epidemiology 48:1, 182-185. [Crossref] · DOI
- Vinod Thomas, Namrata Chindarkar. The Spectrum of Impact Evaluations 25-61. [Crossref] · DOI
- Eszter Czibor, David Jimenez-Gomez, John A. List. 2019. The Dozen Things Experimental Economists Should Do (More Of ). SSRN Electronic Journal 23. . [Crossref] · DOI
- Kevin Boudreau, Nilam Kaushik. 2019. Gender, Tastes for Tech and for the Tech Work Environment? Large-Scale Field Experimental Evidence from an Internet of Things Platform. SSRN Electronic Journal . [Crossref] · DOI
- Lucy Page, Rohini Pande. 2018. Ending Global Poverty: Why Money Isn't Enough. Journal of Economic Perspectives 32:4, 173-200. [Abstract] [View PDF article] [PDF with links] · DOI
- Mariella Gonzales, Gianmarco León, Luis Martinez. 2018. Monetary Incentives to Vote: Evidence From a Nationwide Policy. SSRN Electronic Journal . [Crossref] · DOI
- Authors
- Muralidharan, K., & Niehaus, P.
- Affiliations
- National Bureau of Economic Research (US)
- Venue
- Journal of Economic Perspectives
- Publisher
- American Economic Association
- Published
- 2017-11-01
- Volume
- 31
- Issue
- 4
- Pages
- 103-124
- ISSN
- 0895-3309 (ISSN-L)
- OpenAlex
- W2766046948
- Cited by
- 194
- References
- 24
- Citations by year
- 2026: 15 · 2025: 17 · 2024: 27 · 2023: 24 · 2022: 23 · 2021: 28 · 2020: 31 · 2019: 18 · 2018: 8 · 2017: 3
- FWCI
- 11.79
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- The Journal of Economic Perspectives (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Social Sciences › Decision Sciences › Management Science and Operations Research › Evaluation and Performance Assessment
- Social Sciences › Safety Research › Poverty, Education, and Child Welfare
- Keywords
- randomized controlled trials, experimental program evaluation, developing countries
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
