Generalizability is not optional: insights from a cross-cultural study of social discounting
Open accessThe DOI matches its Crossref record; 22 of the 22 references it cites were found in a registry or an index.
Abstract
Current scientific reforms focus more on solutions to the problem of reliability (e.g. direct replications) than generalizability. Here, we use a cross-cultural study of social discounting to illustrate the utility of a complementary focus on generalizability across diverse human populations. Social discounting is the tendency to sacrifice more for socially close individuals—a phenomenon replicated across countries and laboratories. Yet, when adapting a typical protocol to low-literacy, resource-scarce settings in Bangladesh and Indonesia, we find no independent effect of social distance on generosity, despite still documenting this effect among US participants. Several reliability and validity checks suggest that methodological issues alone cannot explain this finding. These results illustrate why we must complement replication efforts with investment in strong checks on generalizability. By failing to do so, we risk developing theories of human nature that reliably explain behaviour among only a thin slice of humanity.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Inclusion of Other in the Self Scale and the structure of interpersonal closeness.
Aron, A., Aron, E. N., & Smollan, D. (1992) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.63.4.596 · Cited by 5,157
A manifesto for reproducible science
Munafò, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers, C. D., Percie du Sert, N., Simonsohn, U., Wagenmakers, E.-J., Ware, J. J., & Ioannidis, J. P. A. (2017) · Nature Human Behaviour · doi:10.1038/s41562-016-0021 · Cited by 3,397
A practical solution to the pervasive problems ofp values
Wagenmakers, E.-J. (2007) · Psychonomic Bulletin & Review · doi:10.3758/bf03194105 · Cited by 2,490
SIMR : an R package for power analysis of generalized linear mixed models by simulation
Green, P., & MacLeod, C. J. (2016) · Methods in Ecology and Evolution · doi:10.1111/2041-210x.12504 · Cited by 2,467
Sears, D. O. (1986) · Journal of Personality and Social Psychology · doi:10.1037//0022-3514.51.3.515 · Cited by 2,119
“Economic man” in cross-cultural perspective: behavioral experiments in 15 small-scale societies
Henrich, J., Boyd, R., Bowles, S., Camerer, C., Fehr, E., Gintis, H., McElreath, R., Alvard, M., Barr, A., Ensminger, J., et al. (2005) · Behavioral and Brain Sciences · doi:10.1017/s0140525x05000142 · Cited by 1,898
Costly Punishment Across Human Societies
Henrich, J., McElreath, R., Barr, A., Ensminger, J., Barrett, C., Bolyanatz, A., Cardenas, J. C., Gurven, M., Gwako, E., Henrich, N., et al. (2006) · Science · doi:10.1126/science.1127333 · Cited by 1,771 · cited 1×
“Another is to invest in large-scale collaborative projects (e.g. AHRC Culture and the Mind [70]) that investigate the same phenomenon in diverse contexts [71–73].”
Camerer, C. F., Dreber, A., Holzmeister, F., Ho, T.-H., Huber, J., Johannesson, M., Kirchler, M., Nave, G., Nosek, B. A., Pfeiffer, T., et al. (2018) · Nature Human Behaviour · doi:10.1038/s41562-018-0399-z · Cited by 1,521
Evaluating replicability of laboratory experiments in economics
Camerer, C. F., Dreber, A., Forsell, E., Ho, T.-H., Huber, J., Johannesson, M., Kirchler, M., Almenberg, J., Altmejd, A., Chan, T., et al. (2016) · Science · doi:10.1126/science.aaf0918 · Cited by 1,314
Peterson, R. A. (2001) · Journal of Consumer Research · doi:10.1086/323732 · Cited by 1,258 · cited 1×
“Social scientists have long worried that convenience samples bias our inferences about human nature by limiting the range of humanity that we study [18,24–28].”
Constraints on Generality (COG): A Proposed Addition to All Empirical Papers
Simons, D. J., Shoda, Y., & Lindsay, D. S. (2017) · Perspectives on Psychological Science · doi:10.1177/1745691617708630 · Cited by 986
Many Labs 2: Investigating variation in replicability across samples and settings
Klein, R. A., Vianello, M., Hasselman, F., Adams, B. G., Adams, R. B., Alper, S., Aveyard, M., Axt, J. R., Babalola, M. T., Bahník, Š., et al. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245918810225 · Cited by 944 · cited 1×
“Another is to invest in large-scale collaborative projects (e.g. AHRC Culture and the Mind [70]) that investigate the same phenomenon in diverse contexts [71–73].”
Bem, D. J. (2011) · Journal of Personality and Social Psychology · doi:10.1037/a0021524 · Cited by 893
How to Make More Published Research True
Ioannidis, J. P. A. (2014) · PLoS Medicine · doi:10.1371/journal.pmed.1001747 · Cited by 760
Rad, M. S., Martingano, A. J., & Ginges, J. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1721165115 · Cited by 746
Robust research needs many lines of evidence
Munafò, M. R., & Davey Smith, G. (2018) · Nature · doi:10.1038/d41586-018-01023-3 · Cited by 666
The alleged crisis and the illusion of exact replication
Stroebe, W., & Strack, F. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691613514450 · Cited by 597
Nelson, L. D., Simmons, J., & Simonsohn, U. (2018) · Annual Review of Psychology · doi:10.1146/annurev-psych-122216-011836 · Cited by 548 · cited 1×
“Large-scale interdisciplinary replication teams are emerging across the globe, scientific journals are increasingly publishing direct replications, and federal governments are beginning to invest resources in replication efforts [11,12].”
Gifts as Economic Signals and Social Symbols
Camerer, C. (1988) · American Journal of Sociology · doi:10.1086/228946 · Cited by 372 · cited 1×
“One factor that may affect social discounting is that resource transfers have different meanings across societies [60–62].”
Crowdsourcing hypothesis tests: Making transparent how design choices shape research results.
Landy, J. F., Jia, M. (., Ding, I. L., Viganola, D., Tierney, W., Dreber, A., Johannesson, M., Pfeiffer, T., Ebersole, C. R., Gronau, Q. F., et al. (2020) · Psychological Bulletin · doi:10.1037/bul0000220 · Cited by 168
Cited by
Why Hypothesis Testers Should Spend Less Time Testing Hypotheses
Scheel, A. M., Tiokhin, L., Isager, P. M., & Lakens, D. (2021) · Perspectives on Psychological Science · doi:10.1177/1745691620966795 · Cited by 382
Self-citationCrowdsourcing hypothesis tests: Making transparent how design choices shape research results.
Landy, J. F., Jia, M. (., Ding, I. L., Viganola, D., Tierney, W., Dreber, A., Johannesson, M., Pfeiffer, T., Ebersole, C. R., Gronau, Q. F., et al. (2020) · Psychological Bulletin · doi:10.1037/bul0000220 · Cited by 168
Generalization Bias in Science
Peters, U., Krauss, A., & Braganza, O. (2022) · Cognitive Science · doi:10.1111/cogs.13188 · Cited by 63
“Several researchers have argued that the replication crisis is closely connected to a “generalizability crisis” in the sciences (Redish, Kummerfeld, Morris, & Love, 2018; Tiokhin et al., 2019; Yarkoni, 2020).”
BackgroundMcDanal, R., Rubin, A., Fox, K. R., & Schleider, J. L. (2022) · Behavior Therapy · doi:10.1016/j.beth.2021.10.004 · Cited by 43
Parochial altruism: What it is and why it varies
Pisor, A. C., & Ross, C. T. (2024) · Evolution and Human Behavior · doi:10.1016/j.evolhumbehav.2023.06.005 · Cited by 38
Metadata
- Authors
- Tiokhin, L., Hackman, J., Munira, S., Jesmin, K., & Hruschka, D.
- Venue
- Royal Society Open Science
- Publisher
- The Royal Society
- Published
- 2019-02
- Volume
- 6
- Issue
- 2
- Pages
- 181386
- ISSN
- 2054-5703 (ISSN-L)
- OpenAlex
- W2915497009
- Cited by
- 57
- References
- 22
- FWCI
- 11.34
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.