CoChat

Empirical audit and review and an assessment of evidentiary value in research on the psychological consequences of scarcity

Open access

The DOI matches its Crossref record; 17 of the 17 references it cites were found in a registry or an index; 3 of the 11 sources its claims cite were checked for support.

Abstract

Empirical audit and review is an approach to assessing the evidentiary value of a research area. It involves identifying a topic and selecting a cross-section of studies for replication. We apply the method to research on the psychological consequences of scarcity. Starting with the papers citing a seminal publication in the field, we conducted replications of 20 studies that evaluate the role of scarcity priming in pain sensitivity, resource allocation, materialism, and many other domains. There was considerable variability in the replicability, with some strong successes and other undeniable failures. Empirical audit and review does not attempt to assign an overall replication rate for a heterogeneous field, but rather facilitates researchers seeking to incorporate strength of evidence as they refine theories and plan new investigations in the research area. This method allows for an integration of qualitative and quantitative approaches to review and enables the growth of a cumulative science.

Overview

CoChat Summary

## TL;DR

This paper conducts an empirical audit of 20 replications from research on the psychological consequences of scarcity. The replications were mixed, but overall suggest that scarcity primes have minimal influence on cognitive ability, product attitudes, or well being, with the clearest effects appearing for financial/consumer decision tasks.

## Key findings

  • The authors replicated 20 studies on scarcity priming and found considerable variability: some “strong successes” and other “undeniable failures.”
  • In 80% of the 20 studies, replication effect sizes were smaller than the original effect sizes; in 30%, the replication effects were directionally opposite.
  • Of the 20 original studies that were significant, only 4 replication attempts were significant.
  • The authors estimate that only 9 of the original studies had 33% power included within the upper bounds of the 95% CIs from the replication-based power estimates, suggesting many original effects were too small to be detectably studied.

## Methods

  • The paper uses an empirical audit and review design: it defines a topic area, selects a cross-section of studies, and conducts close replications to assess evidentiary value.
  • The sample began with 198 articles citing Shah et al. (2012), narrowed to 32, then 23, and finally 20 studies selected for replication; only studies that involved scarcity, cited the seminal scarcity paper, and could be run with an online sample were eligible.
  • Replications used original materials when possible, were preregistered, had IRB approval and online consent, and used 2.5 times the original sample size; analyses focused on replication effect sizes, significance, and implied power of the original studies.

## Limitations

  • The authors state that their project has “neither the goal nor the capacity to ‘accept the null’” for the tests.
  • Their inclusion criteria are a strength for definitional clarity, but also a weakness because they limit generalization; the paper notes that online studies may be inherently smaller or less replicable, and that some larger field effects are harder to replicate.
  • Evident from the text: the findings apply to a specific subset of scarcity research—studies that could be replicated online and fit the authors’ selection rules—so the results may not generalize to all scarcity research.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1073/pnas.2103313118

Cites

17 of 17 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Fig. 1. The Leftmost columns indicate common features among the replicated studies and the Middle column depicts effect size (correlation coefficients) for the original and replication studies. Effect sizes are bounded by 95% CIs. The Right columns indicate the estimated power in the original studies (third column from Right), the upper bound of the 95% CI for estimated power in the original (second column from Right), and well as an estimated sample size required for 80% power, based on the replication effect (Rightmost column).
Fig. 1. The Leftmost columns indicate common features among the replicated studies and the Middle column depicts effect size (correlation coefficients) for the original and replication studies. Effect sizes are bounded by 95% CIs. The Right columns indicate the estimated power in the original studies (third column from Right), the upper bound of the 95% CI for estimated power in the original (second column from Right), and well as an estimated sample size required for 80% power, based on the replication effect (Rightmost column).CC BY-NC-ND · PubMed Central

Verification

Claims the paper makes

3 Valid · 0 Not valid · 5 Not checked · 19 Own results and interpretations

10 more claims

Create a free account to see all 27 claims and every source verdict.

See all claims — free account
  • It may state two separate claims as one. 7
  • Probably not a substantial claim. 7

Create a free account to see these claims.

Citations

Cites

Show all 17 references

Cited by

See all 56

Metadata

Authors
O’Donnell, M., Dev, A. S., Antonoplis, S., Baum, S. M., Benedetti, A. H., Brown, N. D., Carrillo, B., Choi, A. L., Connor, P., Donnelly, K., Ellwood-Lowe, M. E., Foushee, R., Jansen, R., Jarvis, S. N., Lundell-Creagh, R., Ocampo, J. M., Okafor, G. N., Azad, Z. R., Rosenblum, M., Schatz, D., Stein, D. H., Wang, Y., Moore, D. A., & Nelson, L. D.
Affiliations
  • McDonough School of Business, Georgetown University, Washington, DC 20057;
  • University of Miami (US)
  • University of California, Berkeley (US)
  • Airbnb (United States) (US)
  • New York University (US)
  • University of Chicago (US)
  • University of Tübingen (DE)
  • McKinsey & Company (United States) (US)
  • Peking University (CN)
Venue
Proceedings of the National Academy of Sciences
Published
2021-10-28
Volume
118
Issue
44
ISSN
  • 0027-8424 (ISSN-L)
  • 1091-6490
OpenAlex
W3208956304
Cited by
56
References
17
Citations by year
2026: 7 · 2025: 5 · 2024: 13 · 2023: 16 · 2022: 13 · 2021: 2
FWCI
10.62
Type
journal article
Language
en
Open access
Locations
  • Proceedings of the National Academy of Sciences (publishedVersion, cc-by-nc-nd)
  • PubMed Central (publishedVersion, cc-by-nc-nd)
  • PubMed (publishedVersion)
  • Knowledge@UChicago (University of Chicago) (submittedVersion, cc-by-nc-nd)
  • eScholarship (California Digital Library) (submittedVersion, cc-by-nc-nd)
Funders
National Science Foundation (DGE 1752814)
Topics
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Social Sciences › Decision Sciences › General Decision Sciences › Decision-Making and Behavioral Economics
Keywords
replication, evidentiary value, pain sensitivity, resource allocation, materialism, cumulative science
MeSH terms
Food Insecurity, Humans, Pain Measurement, Research Design, Reproducibility of Results, Empirical Research, Resource Allocation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-09, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-09 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Empirical audit and review and an assessment of evidentiary value in research on the psychological consequences of scarcity. https://app.cochat.ai/papers/10.1073/pnas.2103313118
The paper
O’Donnell, M., Dev, A. S., Antonoplis, S., Baum, S. M., Benedetti, A. H., Brown, N. D., Carrillo, B., Choi, A. L., Connor, P., Donnelly, K., Ellwood-Lowe, M. E., Foushee, R., Jansen, R., Jarvis, S. N., Lundell-Creagh, R., Ocampo, J. M., Okafor, G. N., Azad, Z. R., Rosenblum, M., . . . Nelson, L. D. (2021). Empirical audit and review and an assessment of evidentiary value in research on the psychological consequences of scarcity. Proceedings of the National Academy of Sciences. https://doi.org/10.1073/pnas.2103313118

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.