Creative destruction in science
Open accessThe DOI matches its Crossref record; 53 of the 53 references it cites were found in a registry or an index.
Abstract
Drawing on the concept of a gale of creative destruction in a capitalistic economy, we argue that initiatives to assess the robustness of findings in the organizational literature should aim to simultaneously test competing ideas operating in the same theoretical space. In other words, replication efforts should seek not just to support or question the original findings, but also to replace them with revised, stronger theories with greater explanatory power. Achieving this will typically require adding new measures, conditions, and subject populations to research designs, in order to carry out conceptual tests of multiple theories in addition to directly replicating the original findings. To illustrate the value of the creative destruction approach for theory pruning in organizational scholarship, we describe recent replication initiatives re-examining culture and work morality, working parents’ reasoning about day care options, and gender discrimination in hiring decisions. It is becoming increasingly clear that many, if not most, published research findings across scientific fields are not readily replicable when the same method is repeated. Although extremely valuable, failed replications risk leaving a theoretical void— reducing confidence the original theoretical prediction is true, but not replacing it with positive evidence in favor of an alternative theory. We introduce the creative destruction approach to replication, which combines theory pruning methods from the field of management with emerging best practices from the open science movement, with the aim of making replications as generative as possible. In effect, we advocate for a Replication 2.0 movement in which the goal shifts from checking on the reliability of past findings to actively engaging in competitive theory testing and theory building. The materials, code, and data for this article are posted publicly on the Open Science Framework, with links provided in the article.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Falsification and the Methodology of Scientific Research Programmes
Lakatos, I. (1970) · Criticism and the Growth of Knowledge · doi:10.1017/cbo9781139171434.009 · Cited by 4,649
The Psychology of Self-Affirmation: Sustaining the Integrity of the Self
Steele, C. M. (1988) · Advances in Experimental Social Psychology · doi:10.1016/s0065-2601(08)60229-4 · Cited by 3,697 · cited 1×
“To further test the original theory that hiring criteria and a sense of personal objectivity are constructed and maintained in a motivated manner, we included a manipulation of self-affirmation vs. self-threat (Steele, 1988; Uhlmann & Nosek, 2012).”
MethodUnderstanding and using the Implicit Association Test: III. Meta-analysis of predictive validity.
Greenwald, A. G., Poehlman, T. A., Uhlmann, E. L., & Banaji, M. R. (2009) · Journal of Personality and Social Psychology · doi:10.1037/a0015575 · Cited by 3,257
Self-citationA Theory of Cultural Values and Some Implications for Work
Schwartz, S. H. (1999) · Applied Psychology · doi:10.1111/j.1464-0597.1999.tb00047.x · Cited by 3,102 · cited 1×
“Management scholars have long noted that work centrality and work values vary across countries, as a function of both differences in organizational forms (Parboteeah & Cullen, 2003), and deeply embedded cultural assumptions (Bond & Smith, 1996; Hofstede, 2001; Schwartz, 1999).”
BackgroundRaise standards for preclinical cancer research
Begley, C. G., & Ellis, L. M. (2012) · Nature · doi:10.1038/483531a · Cited by 2,979
Platt, J. R. (1964) · Science · doi:10.1126/science.146.3642.347 · Cited by 2,907 · cited 6×
“Platt (1964) used the term strong inference to describe at a high level how faster-moving sciences tend to pit theories against one another to accelerate progress (see also Albertini, 2017).”
BackgroundMethodInfluentialWhat Do Laboratory Experiments Measuring Social Preferences Reveal About the Real World?
Levitt, S. D., & List, J. A. (2007) · Journal of Economic Perspectives · doi:10.1257/jep.21.2.153 · Cited by 2,668
Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling
John, L. K., Loewenstein, G., & Prelec, D. (2012) · Psychological Science · doi:10.1177/0956797611430953 · Cited by 2,239
HARKing: Hypothesizing after the results are known
Kerr, N. L. (1998) · Personality and Social Psychology Review · doi:10.1207/s15327957pspr0203_4 · Cited by 1,989
Believe it or not: how much can we rely on published data on potential drug targets?
Prinz, F., Schlange, T., & Asadullah, K. (2011) · Nature Reviews Drug Discovery · doi:10.1038/nrd3439-c1 · Cited by 1,888
Camerer, C. F., Dreber, A., Holzmeister, F., Ho, T.-H., Huber, J., Johannesson, M., Kirchler, M., Nave, G., Nosek, B. A., Pfeiffer, T., et al. (2018) · Nature Human Behaviour · doi:10.1038/s41562-018-0399-z · Cited by 1,521
Self-citationKoriat, A., Lichtenstein, S., & Fischhoff, B. (1980) · Journal of Experimental Psychology: Human Learning and Memory · doi:10.1037/0278-7393.6.2.107 · Cited by 1,477
Large-Scale Psychological Differences Within China Explained by Rice Versus Wheat Agriculture
Talhelm, T., Zhang, X., Oishi, S., Shimin, C., Duan, D., Lan, X., & Kitayama, S. (2014) · Science · doi:10.1126/science.1246850 · Cited by 1,374
Evaluating replicability of laboratory experiments in economics
Camerer, C. F., Dreber, A., Forsell, E., Ho, T.-H., Huber, J., Johannesson, M., Kirchler, M., Almenberg, J., Altmejd, A., Chan, T., et al. (2016) · Science · doi:10.1126/science.aaf0918 · Cited by 1,314 · cited 2×
“Across many disciplines, including medicine (Begley & Ellis, 2012; Prinz, Schlange, & Asadullah, 2011), economics (Camerer et al., 2016; Chang & Li, 2017; McCullough, McGeary, & Harrison, 2006), psychology (Ebersole et al.”
BackgroundSelf-citationNosek (2012) · doi:10.1177/1745691612459058 · Cited by 1,301 · cited 1×
“To further test the original theory that hiring criteria and a sense of personal objectivity are constructed and maintained in a motivated manner, we included a manipulation of self-affirmation vs. self-threat (Steele, 1988; Uhlmann & Nosek, 2012).”
MethodExpert political judgment: how good is it? how can we know?
(2006) · Choice Reviews Online · doi:10.5860/choice.43-3634 · Cited by 1,260 · cited 1×
“In this politically charged context (Tetlock, 2005), we further examined whether scientists’ beliefs and values regarding gender moderate the accuracy of their predictions.”
BackgroundInvestigating Variation in Replicability: A “Many Labs” Replication Project
Klein, R. A., Ratliff, K. A., Vianello, M., Adams, R. B., Bahník, Š., Bernstein, M. J., Bocian, K., Brandt, M. J., Brooks, B., Brumbaugh, C. C., et al. (2014) · Social Psychology · doi:10.1027/1864-9335/a000178 · Cited by 1,208
The police officer's dilemma: Using ethnicity to disambiguate potentially threatening individuals.
Correll, J., Park, B., Judd, C. M., & Wittenbrink, B. (2002) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.83.6.1314 · Cited by 1,175
Eagly, A. H., Nater, C., Miller, D. I., Kaufmann, M., & Sczesny, S. (2020) · American Psychologist · doi:10.1037/amp0000494 · Cited by 1,136
Shall we Really do it Again? The Powerful Concept of Replication is Neglected in the Social Sciences
Schmidt, S. (2009) · Review of General Psychology · doi:10.1037/a0015108 · Cited by 1,125 · cited 6×
“Thus, while scholars continue to generate new theory at an accelerated pace, their propositions typically enjoy preliminary rather than definitive support, and are rarely subjected to attempts at direct replication (Schmidt, 2009; Simons, 2014) or placed in competition against adjacent (and sometimes contradictory) th…”
BackgroundMethodInfluentialThe Duke University Religion Index (DUREL): A Five-Item Measure for Use in Epidemological Studies
Koenig, H. G., & Büssing, A. (2010) · Religions · doi:10.3390/rel1010078 · Cited by 1,113
Ross, L., Lepper, M. R., & Hubbard, M. (1975) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.32.5.880 · Cited by 1,052
An Agenda for Purely Confirmatory Research
Wagenmakers, E.-J., Wetzels, R., Borsboom, D., van der Maas, H. L. J., & Kievit, R. A. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612463078 · Cited by 1,042
Publication prejudices: An experimental study of confirmatory bias in the peer review system
Mahoney, M. J. (1977) · Cognitive Therapy and Research · doi:10.1007/bf01173636 · Cited by 1,013 · cited 2×
“A host of related findings speak to the influence of prior convictions on information processing (Koriat, Lichtenstein, & Fischhoff, 1980; Mahoney, 1977; Pitz, 1969; Ross, Lepper, & Hubbard, 1975), which is arguably rationally defensible in Bayesian terms (Baron & Jost, 2019; Krueger & Funder, 2004).”
BackgroundMany Labs 2: Investigating variation in replicability across samples and settings
Klein, R. A., Vianello, M., Hasselman, F., Adams, B. G., Adams, R. B., Alper, S., Aveyard, M., Axt, J. R., Babalola, M. T., Bahník, Š., et al. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245918810225 · Cited by 944
Self-citationTransparency and replicability in qualitative research: the case of interviews with elite informants
Aguinis, H., & Solarino, A. M. (2019) · Strategic Management Journal · doi:10.1002/smj.3015 · Cited by 791
“Positive” results increase down the Hierarchy of the Sciences
Fanelli, D. (2010) · PLoS ONE · doi:10.1371/journal.pone.0010068 · Cited by 734 · cited 2×
“Moreover, because research finding support for the proposed hypotheses is far more likely to lead to a publication (i.e., publication bias; Fanelli, 2010; Kepes, Banks, McDanel, & Whetzel, 2012), available reports using such an approach are unlikely to result in the conclusion that a third…”
BackgroundDevine, P. G., Plant, E. A., Amodio, D. M., Harmon-Jones, E., & Vance, S. L. (2002) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.82.5.835 · Cited by 720
Dreber, A., Rand, D. G., Fudenberg, D., & Nowak, M. A. (2008) · Nature · doi:10.1038/nature06723 · Cited by 718
Self-citationTightness–looseness across the 50 united states
Harrington, J. R., & Gelfand, M. J. (2014) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1317937111 · Cited by 708
The Value of Direct Replication
Simons, D. J. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691613514755 · Cited by 630
You Can't Always Get What You Want: Educational Attainment, Agency, and Choice.
Snibbe, A. C., & Markus, H. R. (2005) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.88.4.703 · Cited by 614
The alleged crisis and the illusion of exact replication
Stroebe, W., & Strack, F. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691613514450 · Cited by 597
Do Student Samples Provide an Accurate Estimate of the General Public?
Hanel, P. H. P., & Vione, K. C. (2016) · PLOS ONE · doi:10.1371/journal.pone.0168354 · Cited by 510
Service with A Smile and Encounter Satisfaction: Emotional Contagion and Appraisal Mechanisms
Barger, P. B., & Grandey, A. A. (2006) · Academy of Management Journal · doi:10.5465/amj.2006.23478695 · Cited by 503
Replication, falsification, and the crisis of confidence in social psychology
Earp, B. D., & Trafimow, D. (2015) · Frontiers in Psychology · doi:10.3389/fpsyg.2015.00621 · Cited by 472
Many Labs 3: Evaluating participant pool quality across the academic semester via replication
Ebersole, C. R., Atherton, O. E., Belanger, A. L., Skulborstad, H. M., Allen, J. M., Banks, J. B., Baranski, E., Bernstein, M. J., Bonfiglio, D. B. V., Boucher, L., et al. (2016) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2015.10.012 · Cited by 468 · cited 1×
“, 2016; Chang & Li, 2017; McCullough, McGeary, & Harrison, 2006), psychology (Ebersole et al., 2016; Klein et al., 2014, 2018; Open Science Collaboration, 2015), and the social sciences, broadly defined (Camerer et al.”
Self-citationPratt, M. G., Kaplan, S., & Whittington, R. (2020) · Administrative Science Quarterly · doi:10.1177/0001839219887663 · Cited by 458
Social perception and social reality: A reflection-construction model.
Jussim, L. (1991) · Psychological Review · doi:10.1037/0033-295x.98.1.54 · Cited by 446 · cited 1×
“At the same time, other work supports the importance of accuracydriven reasoning (Devine, Hirt, & Gehrke, 1990; Funder, 1987; Jussim, 1991; Trope & Bassok, 1982).”
Thau, S., & Mitchell, M. S. (2010) · Journal of Applied Psychology · doi:10.1037/a0020540 · Cited by 367
Questionable research practices in ecology and evolution
Fraser, H., Parker, T., Nakagawa, S., Barnett, A., & Fidler, F. (2018) · PLOS ONE · doi:10.1371/journal.pone.0200303 · Cited by 346
Using prediction markets to estimate the reproducibility of scientific research
Dreber, A., Pfeiffer, T., Almenberg, J., Isaksson, S., Wilson, B., Chen, Y., Nosek, B. A., & Johannesson, M. (2015) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1516179112 · Cited by 331
Self-citationWhat meta-analyses reveal about the replicability of psychological research.
Stanley, T. D., Carter, E. C., & Doucouliagos, H. (2018) · Psychological Bulletin · doi:10.1037/bul0000169 · Cited by 307
Comparing meta-analyses and preregistered multiple-laboratory replication projects
Kvarven, A., Strømland, E., & Johannesson, M. (2019) · Nature Human Behaviour · doi:10.1038/s41562-019-0787-z · Cited by 271
Self-citationThe Chrysalis Effect: How Ugly Initial Results Metamorphosize Into Beautiful Articles
O’Boyle, E. H., Banks, G. C., & Gonzalez-Mulé, E. (2017) · Journal of Management · doi:10.1177/0149206314527133 · Cited by 246 · cited 1×
“, HARKing; Kerr, 1998) and publication bias may put forward only tests and patterns of control variables that support a conclusion (O’Boyle et al., 2017).”
Background“I think it, therefore it’s true”: Effects of self-perceived objectivity on hiring discrimination
Uhlmann, E. L., & Cohen, G. L. (2007) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2007.07.001 · Cited by 232
Self-citationRegistered Replication Report: Schooler and Engstler-Schooler (1990)
Alogna, V. K., Attaya, M. K., Aucoin, P., Bahník, Š., Birch, S., Birt, A. R., Bornstein, B. H., Bouwmeester, S., Brandimonte, M. A., Brown, C., et al. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691614545653 · Cited by 223 · cited 1×
“Fourth, creative destruction draws conclusions from especially large sample sizes, as per the lessons of recent replication initiatives (Alogna et al., 2014; Klein et al., 2018).”
BackgroundMcCullough, B. D., McGeary, K. A., & Harrison, T. D. (2006) · Journal of Money, Credit, and Banking · doi:10.1353/mcb.2006.0061 · Cited by 173
Crowdsourcing hypothesis tests: Making transparent how design choices shape research results.
Landy, J. F., Jia, M. (., Ding, I. L., Viganola, D., Tierney, W., Dreber, A., Johannesson, M., Pfeiffer, T., Ebersole, C. R., Gronau, Q. F., et al. (2020) · Psychological Bulletin · doi:10.1037/bul0000220 · Cited by 168 · cited 1×
“As others have noted, it is possible that theories are true only within specific measurements of key terms; that is, they are highly sensitive to the approach to measurement or conceptualization (Baribault et al., 2018; Landy et al., 2020).”
BackgroundSelf-citationSocial identity and achievement gaps: Evidence from an affirmation intervention
Dee, T. S. (2015) · Journal of Research on Educational Effectiveness · doi:10.1080/19345747.2014.906009 · Cited by 98 · cited 1×
“…effects observed in Uhlmann and Cohen (2005, 2007) are “hot” processes, they should be amplified under psychological threat and ameliorated when an unrelated but important identity has been affirmed (Sherman & Cohen, 2006, 2010; cf. Dee, 2015; Protzko & Aronson, 2016; Hanselman et al., in press).”
BackgroundContext Moderates Affirmation Effects on the Ethnic Achievement Gap
Protzko, J., & Aronson, J. (2016) · Social Psychological and Personality Science · doi:10.1177/1948550616646426 · Cited by 39
Research works printed but not linked: 96
- Albertini, D. F. (2017). On strong inferences and irreproducibility in reproductive medicine. Journal of Assisted Reproduction and Genetics, 34, 695–696. · doi:10.1007/s10815-017-0950-2
- Aguinis, 2009, First decade of organizational research methods: Trends in design, measurement, and data-analysis topics, Organizational Research Methods, 12, 9, 10.1177/1094428108322641 · DOI · Paper page
- Agnoli, 2017, Questionable research practices among Italian research psychologists, PLoS One, 12, e0172792, 10.1371/journal.pone.0172792 · DOI · Paper page
- Armor, 2006, Accuracy, error, and bias in predictions for real versus hypothetical events, Journal of Personality and Social Psychology, 91, 583, 10.1037/0022-3514.91.4.583 · DOI · Paper page
- Bamberger, 2019, On the replicability of abductive research in management and organizations: Internal replication and its alternatives, Academy of Management Discoveries, 5, 103, 10.5465/amd.2019.0121 · DOI
- Bamberger, 2008, Beyond contextualization: Using context theories to narrow the micro-macro gap in management research, Academy of Management Journal, 51, 839, 10.5465/amj.2008.34789630 · DOI · Paper page
- Baribault, 2018, Metastudies for robust tests of theory, Proceedings of the National Academy of Sciences, 115, 2607, 10.1073/pnas.1708285114 · DOI · Paper page
- Baron, 2019, False equivalence: Are liberals and conservatives in the United States equally biased?, Perspectives on Psychological Science, 14, 292, 10.1177/1745691618788876 · DOI · Paper page
- Barrick, 2005, Reducing voluntary, avoidable turnover through selection, Journal of Applied Psychology, 90, 159, 10.1037/0021-9010.90.1.159 · DOI · Paper page
- Bastardi, 2011, Wishful thinking: Belief, desire, and the motivated evaluation of scientific evidence, Psychological Science, 22, 731, 10.1177/0956797611406447 · DOI
- Bedeian, 2010, Management science on the credibility bubble: Cardinal sins and various misdemeanors, Academy of Management Learning & Education, 9, 715
- Bergh, 2017, Is there a credibility crisis in strategic management research? Evidence on the reproducibility of study findings, Strategic Organization, 15, 423, 10.1177/1476127017701076 · DOI · Paper page
- Bond, 1996, Cross-cultural social and organizational psychology, Annual Review of Psychology, 47, 205, 10.1146/annurev.psych.47.1.205 · DOI · Paper page
- Bosco, 2016, HARKing’s threat to organizational research: Evidence from primary and meta-analytic sources, Personnel Psychology, 69, 709, 10.1111/peps.12111 · DOI · Paper page
- Brandt, M. J., IJzerman, H., Dijksterhuis, A., Farach, F., Geller, J., Giner-Sorolla, R., … van‘t Veer, A. (2014). The replication recipe: What makes for a convincing replication? Journal of Experimental Social Psychology, 50, 217–224. · DOI · Paper page
- Brainerd, 2018, Replication, registration, and scientific creativity, Perspectives on Psychological Science, 13, 428, 10.1177/1745691617739421 · DOI
- Byington, 2017, Solutions to the credibility crisis in management science, Academy of Management Learning & Education, 16, 142, 10.5465/amle.2015.0035 · DOI
- Byrd, 2019, What we can (and can’t) infer about implicit bias from debiasing experiments, Synthese, 1
- Cairo, A.H., Green, J.D., Forsyth, D.R., Behler, A.M.C., & Raldiris, T.L. (in press). Gray (Literature) Matters: Evidence of Selective Hypothesis Reporting in Social Psychological Research. Personality and Social Psychology Bulletin.
- Carpenter, 2005, The effects of stakes in distribution experiments, Economics Letters, 86, 393, 10.1016/j.econlet.2004.08.007 · DOI · Paper page
- Cashen, 2004, Statistical power and the testing of null hypotheses: A review of contemporary management research and recommendations for future studies, Organizational Research Methods, 7, 151, 10.1177/1094428104263676 · DOI · Paper page
- Chang, 2017, A preanalysis plan to replicate sixty economics research papers that worked half of the time, American Economic Review, 107, 60, 10.1257/aer.p20171034 · DOI
- Crandall, 2016, On the scientific superiority of conceptual replications for scientific progress, Journal of Experimental Social Psychology, 66, 93, 10.1016/j.jesp.2015.10.002 · DOI · Paper page
- Davis, 2006, Mechanisms and the theory of organizations, Journal of Management Inquiry, 15, 114, 10.1177/1056492605280238 · doi:10.1177/1056492605280238
- Devine, 1990, Diagnostic and confirmation strategies in trait hypothesis testing, Journal of Personality and Social Psychology, 58, 952, 10.1037/0022-3514.58.6.952 · doi:10.1037/0022-3514.58.6.952 · Paper page
- DellaVigna, 2018, Predicting experimental results: Who knows what?, Journal of Political Economy, 126, 2410, 10.1086/699976 · DOI · Paper page
- Desai, 2014, The implications of marriage structure for men’s workplace attitudes, beliefs, and behaviors toward women, Administrative Science Quarterly, 59, 330, 10.1177/0001839214528704 · DOI
- Ditto, 2019, At least bias is bipartisan: A meta-analytic comparison of partisan bias in liberals and conservatives, Perspectives on Psychological Science, 14, 273, 10.1177/1745691617746796 · DOI · Paper page
- Ebersole, C. (2019). Pre-commitment and updating beliefs. Unpublished doctoral dissertation. · DOI
- Edwards, 2010, Reconsidering theoretical progress in organizational and management research, Organizational Research Methods, 13, 615, 10.1177/1094428110380468 · DOI · Paper page
- Evangelou, 2012, Perceived information gain from randomized trials correlates with publication in high–impact factor journals, Journal of Clinical Epidemiology, 65, 1274, 10.1016/j.jclinepi.2012.06.009 · DOI
- Forsell, 2019, Predicting replication outcomes in the Many Labs 2 study, Journal of Economic Psychology, 75, 102, 10.1016/j.joep.2018.10.009 · DOI · Paper page
- Funder, 1987, Errors and mistakes: Evaluating the accuracy of social judgment, Psychological Bulletin, 101, 75, 10.1037/0033-2909.101.1.75 · doi:10.1037/0033-2909.101.1.75 · Paper page
- Gelman, 2015, The connection between varying treatment effects and the crisis of unreplicable research: A Bayesian perspective, Journal of Management, 41, 632, 10.1177/0149206314525208 · DOI · Paper page
- Greenberg, J., & Jonas, E. (2003). Psychological motives and political orientation—The left, the right, and the rigid: Comment on Jost et al. (2003). Psychological Bulletin, 129(3), 376–382. · DOI · Paper page
- Goff, 1990, Employer supported child care, work/family conflict, and absenteeism: A field study, Personnel Psychology, 43, 793, 10.1111/j.1744-6570.1990.tb00683.x · DOI · Paper page
- Greenwald, 1995, Implicit social cognition: Attitudes, self-esteem, and stereotypes, Psychological Review, 102, 4, 10.1037/0033-295X.102.1.4 · DOI · Paper page
- Greenwald, 2002, A unified theory of implicit attitudes, beliefs, self-esteem and self-concept, Psychological Review, 109, 3, 10.1037/0033-295X.109.1.3 · DOI · Paper page
- Grossmann, 2011, Social class, culture, and cognition, Social Psychological and Personality Science, 2, 81, 10.1177/1948550610377119 · DOI · Paper page
- Hanselman, P., Rozek, C.S., Grigg, J., & Borman, G.D. (in press). New evidence on self-affirmation effects and theorized sources of heterogeneity from large-scale replications. Journal of Educational Psychology.
- Hambrick, 2007, The field of management’s devotion to theory: Too much of a good thing?, Academy of Management Journal, 50, 1346, 10.5465/amj.2007.28166119 · DOI · Paper page
- Harvie, 1998, Gender-based biases in occupational hiring decisions, Journal of Applied Social Psychology, 28, 1698, 10.1111/j.1559-1816.1998.tb01341.x · DOI
- He, 2019, Self-insight into emotional and cognitive abilities is not related to higher adjustment, Nature Human Behavior, 3, 867, 10.1038/s41562-019-0644-0 · DOI
- Jordan, 2016, Third-party punishment as a costly signal of trustworthiness, Nature, 530, 473, 10.1038/nature16981 · DOI · Paper page
- Katz, 1988, Racial ambivalence and American value conflict: Correlational and priming studies of dual cognitive structures, Journal of Personality and Social Psychology, 55, 893, 10.1037/0022-3514.55.6.893 · doi:10.1037/0022-3514.55.6.893 · Paper page
- Kepes, 2012, Publication bias in the organizational sciences, Organizational Research Methods, 15, 624, 10.1177/1094428112452760 · DOI · Paper page
- Kirkman, 2019, Is it just me, or was that sexist? The role of sexism type and perpetrator race in identifying sexism, The Journal of Social Psychology, 160, 1
- Kitayama, 2006, Voluntary settlement and the spirit of independence: Evidence from Japan’s “northern frontier”, Journal of Personality and Social Psychology, 91, 369, 10.1037/0022-3514.91.3.369 · DOI · Paper page
- Kluger, 2001, The error of accepting the “‘theoretical’” null hypothesis: The rise, fall, and resurrection of commonsense hypotheses in psychology, Psychological Bulletin, 127, 408, 10.1037/0033-2909.127.3.408 · doi:10.1037/0033-2909.127.3.408 · Paper page
- Köhler, T., & Cortina, J. M. (in press). Play it again, Sam! An analysis of constructive replication in the organizational sciences. Journal of Management.
- Krueger, 2004, Towards a balanced social psychology: Causes, consequences and cures for the problem-seeking approach to social behavior and cognition, Behavioral and Brain Sciences, 27, 313, 10.1017/S0140525X04000081 · DOI · Paper page
- Kunert, 2016, Internal conceptual replications do not increase independent replication success, Psychonomic Bulletin and Review, 23, 1631, 10.3758/s13423-016-1030-9 · DOI
- Kunst, 2019, Sexism, rape myths and feminist identification explain gender differences in attitudes toward the #metoo social media campaign in two countries, Media Psychology, 22, 818, 10.1080/15213269.2018.1532300 · DOI · Paper page
- Latham, 1988, Resolving scientific disputes by the joint design of crucial experiments by the antagonists – Application to the Erez-Latham dispute regarding participation in goal setting, Journal of Applied Psychology, 73, 753, 10.1037/0021-9010.73.4.753 · DOI · Paper page
- Leavitt, 2010, Theory pruning: Strategies for reducing our dense theoretical landscape, Organizational Research Methods, 13, 644, 10.1177/1094428109345156 · DOI · Paper page
- List, 2006, The behavioralist meets the market: Measuring social preferences and reputation effects in actual transactions, Journal of Political Economy, 114, 1, 10.1086/498587 · DOI · Paper page
- Lord, 1979, Biased assimilation and attitude polarization: The effects of prior theories on subsequently considered evidence, Journal of Personality and Social Psychology, 37, 2098, 10.1037/0022-3514.37.11.2098 · DOI · Paper page
- Luttrell, 2017, Replicating and fixing failed replications: The case of need for cognition and argument quality, Journal of Experimental Social Psychology, 69, 178, 10.1016/j.jesp.2016.09.006 · DOI
- Lykken, 1968, Statistical significance in psychological research, Psychological Bulletin, 70, 151, 10.1037/h0026141 · DOI · Paper page
- Makel, M. C., Hodges, J., Cook, B. G., & Plucker, J. A. (2019). Questionable and open research practices in education research. Unpublished manuscript. Available at: https://edarxiv.org/f7srb/. · DOI
- Meehl, 1978, Theoretical risks and tabular asterisks: Sir Karl, Sir Ronald, and the slow progress of soft psychology, Journal of Consulting and Clinical Psychology, 46, 806, 10.1037/0022-006X.46.4.806 · DOI · Paper page
- Mitchell, 2001, Building better theory: Time and the specification of when things happen, Academy of Management Review, 26, 530, 10.2307/3560240 · DOI · Paper page
- Murphy, 2019, HARKing: How badly can cherry-picking and question trolling produce bias in published results?, Journal of Business and Psychology, 34, 1, 10.1007/s10869-017-9524-7 · DOI
- Olian, 1988, The impact of applicant gender compared to qualifications on hiring recommendations: A meta-analysis of experimental studies, Organizational Behavior and Human Decision Processes, 41, 180, 10.1016/0749-5978(88)90025-8 · DOI · Paper page
- Parboteeah, 2003, Social institutions and work centrality: Explorations beyond national culture, Organization Science, 14, 137, 10.1287/orsc.14.2.137.14989 · DOI · Paper page
- Payne, 2019, Historical roots of implicit bias in slavery, Proceedings of the National Academy of Sciences, 116, 11693, 10.1073/pnas.1818816116 · DOI · Paper page
- Perry, 1994, Explaining gender-based selection decisions: A synthesis of contextual and cognitive approaches, Academy of Management Review, 19, 786, 10.2307/258745 · DOI · Paper page
- Petty, R. E., & Cacioppo, J. T. (2016). Methodological choices have predictable consequences in replicating studies on motivation to think: Commentary on Ebersole et al. (2016). Journal of Experimental Social Psychology, 67, 86–87. · DOI
- Peteraf, 2013, The elephant in the room of dynamic capabilities: Bringing two diverging conversations together, Strategic Management Journal, 34, 1389, 10.1002/smj.2078 · DOI · Paper page
- Pitz, 1969, An inertia effect (resistance to change) in the revision of opinion, Canadian Journal of Psychology, 23, 24, 10.1037/h0082790 · DOI
- Porter, 1996, Forty years of organization studies: Reflections from a micro perspective, Administrative Science Quarterly, 41, 262, 10.2307/2393718 · DOI
- Ramagopalan, 2014, Prevalence of primary outcome changes in clinical trials registered on ClinicalTrials.gov: A cross-sectional study, F1000Research, 3, 77, 10.12688/f1000research.3784.1 · DOI · Paper page
- Reynolds, 2014, The role of moral knowledge in everyday immorality: What does it matter if I know what is right?, Organizational Behavior and Human Decision Processes, 123, 124, 10.1016/j.obhdp.2013.10.008 · DOI
- Scherbaum, 2009, Estimating statistical power and required sample sizes for organizational research using multilevel modeling, Organizational Research Methods, 12, 347, 10.1177/1094428107308906 · DOI · Paper page
- Schlaegel, 2014, Determinants of entrepreneurial intent: A meta-analytic test and integration of competing models, Entrepreneurship Theory and Practice, 38, 291, 10.1111/etap.12087 · DOI · Paper page
- Schumpeter, J. A. (1942/1994). Capitalism, socialism and democracy. London: Routledge. pp. 82–83.
- Schweinsberg, 2016, The pipeline project: Pre-publication independent replications of a single laboratory’s research pipeline, Journal of Experimental Social Psychology, 66, 55, 10.1016/j.jesp.2015.10.001 · DOI · Paper page
- Sherman, 2006, The psychology of self-defense: Self-affirmation theory, Vol. 38, 183
- Schwarz, 2014, Does merely going through the same moves make for a “direct” replication? Concepts, contexts, and operationalizations, Social Psychology, 45, 305
- Soklaridis, 2018, Men’s fear of mentoring in the #MeToo era – What’s at stake for academic medicine?, New England Journal of Medicine, 379, 2270, 10.1056/NEJMms1805743 · DOI
- Stewart, 1996, Fathers' work experiences effect children's behaviors via job-related affect and parenting behaviors, Journal of Organizational Behavior, 17, 221, 10.1002/(SICI)1099-1379(199605)17:3<221::AID-JOB741>3.0.CO;2-G · DOI
- Stephens, 2011, When choice does not equal freedom: A sociocultural analysis of agency in working-class American contexts, Social Psychological and Personality Science, 2, 33, 10.1177/1948550610378757 · DOI · Paper page
- Strack, 2016, Reflection on the smiling registered replication report, Perspectives on Psychological Science, 11, 929, 10.1177/1745691616674460 · DOI
- Tetlock, 1982, Attribution bias: On the inconclusiveness of the cognition-motivation debate, Journal of Experimental Social Psychology, 18, 68, 10.1016/0022-1031(82)90082-8 · DOI · Paper page
- Tetlock, 1985, Impression management versus intrapsychic explanations in social psychology: A useful dichotomy?, Psychological Review, 92, 59, 10.1037/0033-295X.92.1.59 · doi:10.1037/0033-295x.92.1.59 · Paper page
- Tierney, W., Hardy, J. H., III., Ebersole, C., Viganola, D., Clemente, E., Gordon, M., et al. (2020). A creative destruction approach to replication: Implicit work and sex morality across cultures. Registered Report proposal accepted in principle at the Journal of Experimental Social Psychology. · DOI · Paper page
- Trope, 1982, Confirmatory and diagnosing strategies in social information gathering, Journal of Personality and Social Psychology, 43, 22, 10.1037/0022-3514.43.1.22 · doi:10.1037/0022-3514.43.1.22 · Paper page
- Tsang, 1999, Replication and theory development in organizational science: A critical realist perspective, Academy of Management Review, 24, 759, 10.2307/259353 · DOI · Paper page
- Uhlmann, 2005, Constructed criteria: Redefining merit to justify discrimination, Psychological Science, 16, 474, 10.1111/j.0956-7976.2005.01559.x · DOI · Paper page
- Uhlmann, 2012, My culture made me do it: Lay theories of responsibility for automatic prejudice, Social Psychology, 43, 108, 10.1027/1864-9335/a000089 · DOI
- Uhlmann, 2011, Implicit Puritanism in American moral cognition, Journal of Experimental Social Psychology, 47, 312, 10.1016/j.jesp.2010.10.013 · DOI
- Van de Ven, 2006, Knowledge for theory and practice, Academy of Management Review, 31, 802, 10.5465/amr.2006.22527385 · DOI · Paper page
- Vancouver, 2018, Translating informal theories into formal theories: The case of the dynamic computational model of the integrated model of work motivation, Organizational Research Methods, 23, 238, 10.1177/1094428118780308 · DOI
- Van den Akker, O., Weston, S. J., Campbell, L., Chopik, W. J., Damian, R. I., Davis-Kean, P., et al., (2019). Preregistration of secondary data analysis: A template and tutorial. Unpublished manuscript. · DOI
- Vandello, 1999, Patterns of individualism and collectivism across the United States, Journal of Personality and Social Psychology, 77, 279, 10.1037/0022-3514.77.2.279 · DOI · Paper page
- Zwaan, 2017, Making replication mainstream, Behavioral & Brain Sciences., 41, e120, 10.1017/S0140525X17001972 · DOI · Paper page
Cited by
Replicability, Robustness, and Reproducibility in Psychological Science
Nosek, B. A., Hardwicke, T. E., Moshontz, H., Allard, A., Corker, K. S., Dreber, A., Fidler, F., Hilgard, J., Kline Struhl, M., Nuijten, M. B., et al. (2022) · Annual Review of Psychology · doi:10.1146/annurev-psych-020821-114157 · Cited by 770
Self-citationThe Replication Crisis Has Led to Positive Structural, Procedural, and Community Changes
Korbmacher, M., Azevedo, F., Pennington, C. R., Hartmann, H., Pownall, M., Schmidt, K., Elsherif, M., Breznau, N., Robertson, O., Kalandadze, T., et al. (2023) · Communications Psychology · doi:10.1038/s44271-023-00003-2 · Cited by 137
Elsherif, M. M., Middleton, S. L., Phan, J. M., Azevedo, F., Iley, B. J., Grose-Hodge, M., Tyler, S. L., Kapp, S. K., Gourdon-Kanhukamwe, A., Grafton-Clarke, D., et al. (2022) · doi:10.31222/osf.io/k7a9p · Cited by 91
Schaerer, M., du Plessis, C., Nguyen, M. H. B., van Aert, R. C. M., Tiokhin, L., Lakens, D., Giulia Clemente, E., Pfeiffer, T., Dreber, A., Johannesson, M., et al. (2023) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2023.104280 · Cited by 87
“This and other recent cases of mis-prediction regarding the outcomes of pre-registered tests of gender bias (Tierney et al, 2020; 2021) could result from ideological blind spots reducing forecasting accuracy in this domain.”
InfluentialSelf-citation
Metadata
- Authors
- Tierney, W., Hardy, J. H., Ebersole, C. R., Leavitt, K., Viganola, D., Clemente, E. G., Gordon, M., Dreber, A., Johannesson, M., Pfeiffer, T., & Uhlmann, E. L.
- Venue
- Organizational Behavior and Human Decision Processes
- Publisher
- Elsevier BV
- Published
- 2020-11
- Volume
- 161
- Pages
- 291-309
- ISSN
- 0749-5978 (ISSN-L)
- OpenAlex
- W3091293017
- Cited by
- 66
- References
- 53
- FWCI
- 7.33
- Type
- journal article
- Language
- en
- Open access
- hybrid
- Licence
- www.elsevier.com/tdm/userlicense/1.0/ (tdm)
- www.elsevier.com/legal/tdmrep-license (tdm)
- creativecommons.org/licenses/by/4.0/ (vor)
- Funders
- Jan Wallanders och Tom Hedelius Stiftelse samt Tore Browaldhs Stiftelse
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.