Empirical audit and review and an assessment of evidentiary value in research on the psychological consequences of scarcity
Open accessThe DOI matches its Crossref record; 17 of the 17 references it cites were found in a registry or an index; 3 of the 11 sources its claims cite were checked for support.
Abstract
Empirical audit and review is an approach to assessing the evidentiary value of a research area. It involves identifying a topic and selecting a cross-section of studies for replication. We apply the method to research on the psychological consequences of scarcity. Starting with the papers citing a seminal publication in the field, we conducted replications of 20 studies that evaluate the role of scarcity priming in pain sensitivity, resource allocation, materialism, and many other domains. There was considerable variability in the replicability, with some strong successes and other undeniable failures. Empirical audit and review does not attempt to assign an overall replication rate for a heterogeneous field, but rather facilitates researchers seeking to incorporate strength of evidence as they refine theories and plan new investigations in the research area. This method allows for an integration of qualitative and quantitative approaches to review and enables the growth of a cumulative science.
Overview
CoChat Summary
## TL;DR
This paper conducts an empirical audit of 20 replications from research on the psychological consequences of scarcity. The replications were mixed, but overall suggest that scarcity primes have minimal influence on cognitive ability, product attitudes, or well being, with the clearest effects appearing for financial/consumer decision tasks.
## Key findings
- The authors replicated 20 studies on scarcity priming and found considerable variability: some “strong successes” and other “undeniable failures.”
- In 80% of the 20 studies, replication effect sizes were smaller than the original effect sizes; in 30%, the replication effects were directionally opposite.
- Of the 20 original studies that were significant, only 4 replication attempts were significant.
- The authors estimate that only 9 of the original studies had 33% power included within the upper bounds of the 95% CIs from the replication-based power estimates, suggesting many original effects were too small to be detectably studied.
## Methods
- The paper uses an empirical audit and review design: it defines a topic area, selects a cross-section of studies, and conducts close replications to assess evidentiary value.
- The sample began with 198 articles citing Shah et al. (2012), narrowed to 32, then 23, and finally 20 studies selected for replication; only studies that involved scarcity, cited the seminal scarcity paper, and could be run with an online sample were eligible.
- Replications used original materials when possible, were preregistered, had IRB approval and online consent, and used 2.5 times the original sample size; analyses focused on replication effect sizes, significance, and implied power of the original studies.
## Limitations
- The authors state that their project has “neither the goal nor the capacity to ‘accept the null’” for the tests.
- Their inclusion criteria are a strength for definitional clarity, but also a weakness because they limit generalization; the paper notes that online studies may be inherently smaller or less replicable, and that some larger field effects are harder to replicate.
- Evident from the text: the findings apply to a specific subset of scarcity research—studies that could be replicated online and fit the authors’ selection rules—so the results may not generalize to all scarcity research.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures

Verification
Claims the paper makes
3 Valid · 0 Not valid · 5 Not checked · 19 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“is indifferent to the quality of the original evidence (14)”
§ Empirical audit and review and an assessment of evidentiary value in research on the psychological consequences of scarcity > Assessing the Evidence of a Literature
What the sources say
- Partly supportedcites [14] sourceExperimental — accuracy not yet measured
The Conclusion (page unavailable in our copy) says systematic reviews cannot be expected to correct deficiencies in primary evidence, supporting the concern behind the claim. But it also says they can reveal how unreliable published evidence is, so describing meta-analysis as simply indifferent to evidence quality overstates the paper's position. (LLM, conf=0.82)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“But meta-analysis risks reifying selectively reported findings (12)”
§ Empirical audit and review and an assessment of evidentiary value in research on the psychological consequences of scarcity > Assessing the Evidence of a Literature
What the sources say
- Supportedcites [12] source
The Introduction says publication bias and selective outcome reporting can leave meta-analyses synthesising incomplete evidence and producing biased summary results. The Abstract warns against uncritically treating such meta-analyses as reliable, supporting the citing sentence's broader phrasing. (LLM, conf=0.90)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“a threat evident in large-scale replication efforts (10, 11).”
§ Empirical audit and review and an assessment of evidentiary value in research on the psychological consequences of scarcity > Assessing the Evidence of a Literature
What the sources say
- Partly supportedcites [10] sourceExperimental — accuracy not yet measured
In Insights on Reproducibility (p. 13), the paper says its large-scale replication results are consistent with publication, selection, and reporting biases in the original studies. That supports the selective-reporting concern, but the passage treats those biases as plausible explanations and does not establish p-hacking specifically. (LLM, conf=0.83)
- Not checked — no full text of the sourcecites [11] source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Three of the four successful replications involved priming participants with some type of financial constraint and then requiring them to engage in a financial or consumer decision task under these constraints.”
§ Empirical audit and review and an assessment of evidentiary value in research on the psychological consequences of scarcity > Results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“directionally opposite for 30% of these 20 studies”
§ Empirical audit and review and an assessment of evidentiary value in research on the psychological consequences of scarcity > Results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
10 more claims
Create a free account to see all 27 claims and every source verdict.
- It may state two separate claims as one. 7
- Probably not a substantial claim. 7
Citations
Cites
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Self-citationPoverty impedes cognitive function.
Mani, A., Mullainathan, S., Shafir, E., & Zhao, J. (2013) · Science · doi:10.1126/science.1238041 · Cited by 2,967 · cited 1×
“Over the past decade, behavioral scientists have investigated how scarcity, or a lack of resources, changes how people think and behave, and how those changes can perpetuate poverty and other negative consequences such as reduced life expectancy (1), reduced cognitive ability (2), impulsiveness (3), and pain (4).”
BackgroundThe Association Between Income and Life Expectancy in the United States, 2001–2014
Chetty, R., Stepner, M., Abraham, S., Lin, S., Scuderi, B., Turner, N., Bergeron, A., & Cutler, D. (2016) · JAMA · doi:10.1001/jama.2016.4226 · Cited by 2,812
Some Consequences of Having Too Little
Shah, A. K., Mullainathan, S., & Shafir, E. (2012) · Science · doi:10.1126/science.1222426 · Cited by 1,567
Camerer, C. F., Dreber, A., Holzmeister, F., Ho, T.-H., Huber, J., Johannesson, M., Kirchler, M., Nave, G., Nosek, B. A., Pfeiffer, T., et al. (2018) · Nature Human Behaviour · doi:10.1038/s41562-018-0399-z · Cited by 1,521 · cited 1×
“Selective reporting of measures, treatments, and analyses, and the p-hacking they enable (7) has been identified as an existential threat to behavioral science (8, 9), a threat evident in large-scale replication efforts (10, 11).”
BackgroundThe Mass Production of Redundant, Misleading, and Conflicted Systematic Reviews and Meta‐analyses
IOANNIDIS, J. P. A. (2016) · The Milbank Quarterly · doi:10.1111/1468-0009.12210 · Cited by 1,483
Small Telescopes: Detectability and the Evaluation of Replication Results
Simonsohn, U. (2015) · Psychological Science · doi:10.1177/0956797614567341 · Cited by 742
Shah, A. K., Shafir, E., & Mullainathan, S. (2015) · Psychological Science · doi:10.1177/0956797614563958 · Cited by 433
Ahmed, I., Sutton, A. J., & Riley, R. D. (2012) · BMJ · doi:10.1136/bmj.d7762 · Cited by 404
Cited by
How Effective Is (More) Money? Randomizing Unconditional Cash Transfer Amounts in the USPreprint
Jaroszewicz, A., Jachimowicz, J., Hauser, O., & Jamison, J. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.4154000 · Cited by 97
Hilbert, L. P., Noordewier, M. K., & DIJK, W. W. V. (2022) · Journal of Economic Psychology · doi:10.1016/j.joep.2022.102546 · Cited by 56
İşler, O., & Yılmaz, O. (2022) · Behavior Research Methods · doi:10.3758/s13428-022-01984-4 · Cited by 80
“Second, an increasing number of high-powered replication failures suggests that many of the previously identified effects could have been artifacts of methodological weaknesses such as small sample size (e.g., O’Donnell et al., 2021).”
BackgroundTully, S. M., & Sharma, E. (2021) · Consumer Psychology Review · doi:10.1002/arcp.1073 · Cited by 35
Taking the Full Measure: Integrating Replication into Research Practice to Assess Generalizability
Urminsky, O., & Dietvorst, B. J. (2024) · Journal of Consumer Research · doi:10.1093/jcr/ucae007 · Cited by 31
Metadata
- Authors
- O’Donnell, M., Dev, A. S., Antonoplis, S., Baum, S. M., Benedetti, A. H., Brown, N. D., Carrillo, B., Choi, A. L., Connor, P., Donnelly, K., Ellwood-Lowe, M. E., Foushee, R., Jansen, R., Jarvis, S. N., Lundell-Creagh, R., Ocampo, J. M., Okafor, G. N., Azad, Z. R., Rosenblum, M., Schatz, D., Stein, D. H., Wang, Y., Moore, D. A., & Nelson, L. D.
- Affiliations
- McDonough School of Business, Georgetown University, Washington, DC 20057;
- University of Miami (US)
- University of California, Berkeley (US)
- Airbnb (United States) (US)
- New York University (US)
- University of Chicago (US)
- University of Tübingen (DE)
- McKinsey & Company (United States) (US)
- Peking University (CN)
- ORCID
- O’Donnell, Michael
- Dev, Amelia S.
- Antonoplis, Stephen
- Benedetti, Arianna H.
- Brown, N. Derek
- Carrillo, Belinda
- Choi, Andrew L.
- Donnelly, Kristin
- Ellwood-Lowe, Monica E.
- Foushee, Ruthe
- Jansen, Rachel
- Jarvis, Shoshana N.
- Lundell-Creagh, Ryan
- Ocampo, Joseph M.
- Okafor, Gold N.
- Azad, Zahra Rahmani
- Rosenblum, Michael
- Stein, Daniel H.
- Wang, Yilu
- Moore, Don A.
- Nelson, Leif D.
- Venue
- Proceedings of the National Academy of Sciences
- Publisher
- National Academy of Sciences
- Published
- 2021-10-28
- Volume
- 118
- Issue
- 44
- ISSN
- 0027-8424 (ISSN-L)
- 1091-6490
- PMC
- PMC8612349
- OpenAlex
- W3208956304
- Cited by
- 56
- References
- 17
- Citations by year
- 2026: 7 · 2025: 5 · 2024: 13 · 2023: 16 · 2022: 13 · 2021: 2
- FWCI
- 10.62
- Type
- journal article
- Language
- en
- Update notices
- correction (2022-11-21)
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Proceedings of the National Academy of Sciences (publishedVersion, cc-by-nc-nd)
- PubMed Central (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- Knowledge@UChicago (University of Chicago) (submittedVersion, cc-by-nc-nd)
- eScholarship (California Digital Library) (submittedVersion, cc-by-nc-nd)
- Funders
- National Science Foundation (DGE 1752814)
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Decision Sciences › General Decision Sciences › Decision-Making and Behavioral Economics
- Keywords
- replication, evidentiary value, pain sensitivity, resource allocation, materialism, cumulative science
- MeSH terms
- Food Insecurity, Humans, Pain Measurement, Research Design, Reproducibility of Results, Empirical Research, Resource Allocation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-09, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-09 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Empirical audit and review and an assessment of evidentiary value in research on the psychological consequences of scarcity. https://app.cochat.ai/papers/10.1073/pnas.2103313118
- The paper
- O’Donnell, M., Dev, A. S., Antonoplis, S., Baum, S. M., Benedetti, A. H., Brown, N. D., Carrillo, B., Choi, A. L., Connor, P., Donnelly, K., Ellwood-Lowe, M. E., Foushee, R., Jansen, R., Jarvis, S. N., Lundell-Creagh, R., Ocampo, J. M., Okafor, G. N., Azad, Z. R., Rosenblum, M., . . . Nelson, L. D. (2021). Empirical audit and review and an assessment of evidentiary value in research on the psychological consequences of scarcity. Proceedings of the National Academy of Sciences. https://doi.org/10.1073/pnas.2103313118
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.