Why Hypothesis Testers Should Spend Less Time Testing Hypotheses

“For example, natural selection favors strategies that maximize inclusive fitness (Hamilton, 1964), which is often not accomplished by self-preservation (e.”

“For example, when Zahavi (1975) proposed the idea that the costliness of signals ensures their reliability (i.e., the handicap principle), many biologists found the idea implausible.”

“For example, natural selection favors strategies that maximize inclusive fitness (Hamilton, 1964), which is often not accomplished by self-preservation (e.g., people investing less in their future health when extrinsic mortality risks are high; Nettle, 2010).”

“For example, terror-management theory (TMT) assumes that humans have an instinct for self-preservation that led to the evolution of an incapacitating fear of death with which humans cope via an anxiety-reducing “terrormanagement” system (Greenberg et al., 1986).”

“The claim that many psychological theories are critically immature has been leveled against the field so often that psychological scientists may well have grown tired of it (e.g., Fiedler, 2004; Gigerenzer, 1998; Meehl, 1967, 1978, 1990; Muthukrishna & Henrich, 2019).”

“Problems with a lack of differentiation have been noted regarding the concept grit, which may be redundant given its high correlation with conscientiousness (Credé et al., 2017).”

“In medical science, feasibility studies are used to assess recruitment and retention rates, adherence to procedures, rates of unusable responses, and the reliability and validity of measures and to estimate the standard deviation of dependent measures (Eldridge et al., 2016; Lancaster, 2015).”

“Third, if the Reproducibility Project: Psychology (Open Science Collaboration, 2015) taught us one thing about the state of the field, it is that psychologists have difficulty agreeing on whether findings have been successfully replicated (Maxwell et al., 2015).”

“As an example, Fried and Nesse (2015) used a vast array of observational research on symptoms of depression to show that different symptoms interact in complex but reliable ways that are not captured by the sum-score estimation of major depressive disorder.”

“One influential proposal was to separate confirmatory (hypothesis-testing) and exploratory (hypothesis-generating) research using preregistration (Wagenmakers et al., 2012).”

“The hypothesized causal relationships between target variables should be formalized and take relevant third variables into account, allowing others to judge whether the predicted effect is causally identified (e.g., Rohrer, 2018).”

“…transparency about boundary conditions makes it difficult to interpret empirical discrepancies (e.g., why an effect was not successfully replicated; Simons et al., 2017) and can lead to degenerative research lines (in which modifications are made to accommodate failed predictions without…”

“The claim that many psychological theories are critically immature has been leveled against the field so often that psychological scientists may well have grown tired of it (e.g., Fiedler, 2004; Gigerenzer, 1998; Meehl, 1967, 1978, 1990; Muthukrishna & Henrich, 2019).”

“Meehl (1990) termed this the “derivation chain,” a conjunction of theoretical and auxiliary premises that are necessary to predict observable outcomes.”

“The insight that psychologists struggle to define their hypotheses will not surprise those who have criticized psychologists’ practice of null-hypothesis significance testing (NHST) as “the null ritual” (Gigerenzer, 2004).”

“This problem is not solved with mere practice—forcing researchers to specify what would falsify their hypotheses when they have no theoretical basis for doing so can lead to testing against arbitrary values (Kruschke, 2018) and runs the risk of replacing one mindless ritual with another.”

“Scales are used without evidence of their validity or are simply created on the fly (Flake et al., 2017).”

“Further, measures with low reliability compromise the inferences drawn from hypothesis tests because noise factors obscure causal effects on the dependent variable (Loken & Gelman, 2017; Shadish et al., 2001).”

“Two criteria for good concepts are coherence and differentiation (Gerring, 1999): Concepts need to describe a class of entities with shared attributes and differentiate this class from other concepts.”

“Despite the importance of reliable and valid measures, measurement practices in psychology are suboptimal (Borsboom, 2006).”

“For example, researchers might assume that different measures are equivalent (e.g., using stated preferences vs. behavioral tasks to measure risk preference; Frey et al., 2017) without realizing that they capture different constructs.”

“Research that is “merely” descriptive is often considered less valuable despite being crucial for forming concepts, developing measures, and establishing phenomena that need explaining (Dubin, 1969; Gerring, 2012a; Rai & Fiske, 2010; Rozin, 2001).”

“Using this paradigm, researchers have established that the functional form of the relationship between social distance and willingness to sacrifice is hyperbolic ( Jones & Rachlin, 2006; but for issues with generalizability, see Tiokhin et al., 2019).”

“…urged psychological scientists to specify more precise hypotheses (e.g., by defining a smallest effect size of interest, a region of practical equivalence (ROPE) in Bayesian estimation, or Bayesian priors; Harms & Lakens, 2018) and test them with higher statistical power (Fraley & Vazire, 2014).”

“By tightening the screws on the HD machinery and incentivizing rigorous confirmatory research, psychology’s reform movement may have inadvertently exacerbated the notion of nonconfirmatory research as a “second-class citizen” (Klahr & Simon, 1999, p. 526).”

“Research that is “merely” descriptive is often considered less valuable despite being crucial for forming concepts, developing measures, and establishing phenomena that need explaining (Dubin, 1969; Gerring, 2012a; Rai & Fiske, 2010; Rozin, 2001).”

“In practice, however, researchers have substantial difficulties incorporating these recommendations in their research, and even preregistration’s most ardent proponents acknowledge that “Preregistration Is Hard” (Nosek et al., 2019).”

“Although scientists often think of experiments in the context of confirmation, philosophers of science have emphasized the role of exploratory experiments in theory development (Franklin, 2005; Steinle, 1997, 2002).”

“One approach is to formalize hypotheses via mathematical or computational modeling (Lewandowsky & Farrell, 2010; Smaldino, 2017).”

“This can result in perpetual disagreement and inconclusive debates (Loehle, 1987).”

“In medical science, feasibility studies are used to assess recruitment and retention rates, adherence to procedures, rates of unusable responses, and the reliability and validity of measures and to estimate the standard deviation of dependent measures (Eldridge et al., 2016; Lancaster, 2015).”

“One approach is to formalize hypotheses via mathematical or computational modeling (Lewandowsky & Farrell, 2010; Smaldino, 2017).”

“Subfields of psychology and neighboring disciplines in which nonconfirmatory research activities are common practice have already begun to tackle these issues (see, e.g., Crüwell et al., 2019; Jacobs, 2020; Moravcsik, 2014).”

“Reforms proposed in reaction to the crisis tried to repair the HD machinery by making methods more rigorous (Spellman, 2015).”

“The claim that many psychological theories are critically immature has been leveled against the field so often that psychological scientists may well have grown tired of it (e.g., Fiedler, 2004; Gigerenzer, 1998; Meehl, 1967, 1978, 1990; Muthukrishna & Henrich, 2019).”

“Research that is “merely” descriptive is often considered less valuable despite being crucial for forming concepts, developing measures, and establishing phenomena that need explaining (Dubin, 1969; Gerring, 2012a; Rai & Fiske, 2010; Rozin, 2001).”

“Exploratory experimentation is widely used in psychophysics to establish law-like relationships (for a discussion of this method in research on face perception, see Jack & Schyns, 2017).”

“Steinle (2002) discusses the vital role of exploratory experiments for concept formation in the history of research on electricity.”

“Although scientists often think of experiments in the context of confirmation, philosophers of science have emphasized the role of exploratory experiments in theory development (Franklin, 2005; Steinle, 1997, 2002).”

“For example, natural selection favors strategies that maximize inclusive fitness (Hamilton, 1964), which is often not accomplished by self-preservation (e.g., people investing less in their future health when extrinsic mortality risks are high; Nettle, 2010).”

“The authors found that Tsimane personality variation was better explained by two principal factors, not five, which inspired new theoretical models to explain why the covariance structure among human personality characteristics varies across populations (Smaldino et al., 2019).”

“In a time-series analysis of participants’ experiences while watching videos that induced kama muta, the KML documented a strong temporal connection between feeling moved, perceived closeness between the video protagonists, and expert ratings of communal sharing (Schubert et al., 2018).”

“Many scientists invoke Popperian hypothetico-deductivism when describing aspects of their research, and the HD method’s prominent role in textbooks suggests that it shapes scientific discourse in many fields, including psychology (Mulkay & Gilbert, 1981; Riesch, 2008; Rozin, 2009).”

“However, some scholars have pointed out that TMT’s assumptions appear to contradict basic tenets of evolutionary theory (Kirkpatrick & Navarrete, 2006).”

“In an effort “to support and promote open-ended, open science, providing a high-status specialized format for its publication” (McIntosh, 2017, p. A2), Cortex— the journal that first introduced Registered Reports in 2013—recently launched the new, complementary format Exploratory Reports.”

“Only then did researchers empirically test the theory in those conditions (for a review, see Grose, 2011).”

“Second, although the KAMMUS Two provides a valid measure of kama muta, the validity of the current operationalization of communal sharing relationships—a scale measuring “closeness” (Seibt et al., 2018)—may require further investigation.”

“The causal model of kama muta—its proximal causes and consequences as well as its evolved function—was inspired by relational models theory (Fiske, 2004).”

“First, even preregistered hypothesis tests are rarely specified in a way that eliminates flexibility in data analysis, with unambiguous criteria for concluding that a prediction is corroborated or falsified (Lakens & DeBruine, 2021; Bakker et al., 2018).”

“The HD method played a key part in psychology’s recent replication crisis (Derksen, 2019).”

“Research that is “merely” descriptive is often considered less valuable despite being crucial for forming concepts, developing measures, and establishing phenomena that need explaining (Dubin, 1969; Gerring, 2012a; Rai & Fiske, 2010; Rozin, 2001).”

“Thus, pilot studies are crucial for minimizing the risk that untested auxiliaries and “manipulation-check neglect” (Fiedler, 2018, p. 435) render a study uninformative.”

“Without formalization, the theory might have been rejected outright, and the conditions in which it was logically coherent might not have been discovered (for similar issues with prominent verbal theories in social psychology, see Harris, 1976).”

“…modification to the HD method was Popper’s critical rationalism (Popper, 1959): Although empirical data never allow us to infer that a theory is true, theories that survive repeated tests with a high capacity to falsify their predictions are more strongly “corroborated” (Fidler et al., 2018).”

“However, there were other situations in which people appeared to experience kama muta without the intensification of a communal sharing relationship (e.g., performing and listening to certain types of music without the physical presence of others; Fiske et al., 2017).”

“The claim that many psychological theories are critically immature has been leveled against the field so often that psychological scientists may well have grown tired of it (e.g., Fiedler, 2004; Gigerenzer, 1998; Meehl, 1967, 1978, 1990; Muthukrishna & Henrich, 2019).”

“Subfields of psychology and neighboring disciplines in which nonconfirmatory research activities are common practice have already begun to tackle these issues (see, e.g., Crüwell et al., 2019; Jacobs, 2020; Moravcsik, 2014).”

“For example, we may notice that open-ended exploration, description (Rozin, 2001), or prediction (Yarkoni & Westfall, 2017) are more suitable endeavors for the matter at hand or that more basic questions about measurement need to be settled first (Scheel et al., 2021).”

“Although the lack of conceptual clarity has been observed to be a widespread and fundamental problem in psychology (Antonakis, 2017; Eronen & Bringmann, 2021; Flake, 2021; MacKenzie, 2003; Podsakoff et al., 2016; Scheel et al., 2021), conceptual clarification plays a mostly marginal role in psychological research (see…”