Why Hypothesis Testers Should Spend Less Time Testing Hypotheses

“For example, natural selection favors strategies that maximize inclusive fitness (Hamilton, 1964), which is often not accomplished by self-preservation (e.”

“For example, when Zahavi (1975) proposed the idea that the costliness of signals ensures their reliability (i.e., the handicap principle), many biologists found the idea implausible.”

“For example, natural selection favors strategies that maximize inclusive fitness (Hamilton, 1964), which is often not accomplished by self-preservation (e.g., people investing less in their future health when extrinsic mortality risks are high; Nettle, 2010).”

“For example, terror-management theory (TMT) assumes that humans have an instinct for self-preservation that led to the evolution of an incapacitating fear of death with which humans cope via an anxiety-reducing “terrormanagement” system (Greenberg et al., 1986).”

“The claim that many psychological theories are critically immature has been leveled against the field so often that psychological scientists may well have grown tired of it (e.g., Fiedler, 2004; Gigerenzer, 1998; Meehl, 1967, 1978, 1990; Muthukrishna & Henrich, 2019).”

“Problems with a lack of differentiation have been noted regarding the concept grit, which may be redundant given its high correlation with conscientiousness (Credé et al., 2017).”

“In medical science, feasibility studies are used to assess recruitment and retention rates, adherence to procedures, rates of unusable responses, and the reliability and validity of measures and to estimate the standard deviation of dependent measures (Eldridge et al., 2016; Lancaster, 2015).”

“As an example, Fried and Nesse (2015) used a vast array of observational research on symptoms of depression to show that different symptoms interact in complex but reliable ways that are not captured by the sum-score estimation of major depressive disorder.”

“One influential proposal was to separate confirmatory (hypothesis-testing) and exploratory (hypothesis-generating) research using preregistration (Wagenmakers et al., 2012).”

“Third, if the Reproducibility Project: Psychology (Open Science Collaboration, 2015) taught us one thing about the state of the field, it is that psychologists have difficulty agreeing on whether findings have been successfully replicated (Maxwell et al., 2015).”

“The hypothesized causal relationships between target variables should be formalized and take relevant third variables into account, allowing others to judge whether the predicted effect is causally identified (e.g., Rohrer, 2018).”

“…transparency about boundary conditions makes it difficult to interpret empirical discrepancies (e.g., why an effect was not successfully replicated; Simons et al., 2017) and can lead to degenerative research lines (in which modifications are made to accommodate failed predictions without…”

“The claim that many psychological theories are critically immature has been leveled against the field so often that psychological scientists may well have grown tired of it (e.g., Fiedler, 2004; Gigerenzer, 1998; Meehl, 1967, 1978, 1990; Muthukrishna & Henrich, 2019).”

“Meehl (1990) termed this the “derivation chain,” a conjunction of theoretical and auxiliary premises that are necessary to predict observable outcomes.”

“The insight that psychologists struggle to define their hypotheses will not surprise those who have criticized psychologists’ practice of null-hypothesis significance testing (NHST) as “the null ritual” (Gigerenzer, 2004).”

“This problem is not solved with mere practice—forcing researchers to specify what would falsify their hypotheses when they have no theoretical basis for doing so can lead to testing against arbitrary values (Kruschke, 2018) and runs the risk of replacing one mindless ritual with another.”

“Scales are used without evidence of their validity or are simply created on the fly (Flake et al., 2017).”

“Further, measures with low reliability compromise the inferences drawn from hypothesis tests because noise factors obscure causal effects on the dependent variable (Loken & Gelman, 2017; Shadish et al., 2001).”

“Two criteria for good concepts are coherence and differentiation (Gerring, 1999): Concepts need to describe a class of entities with shared attributes and differentiate this class from other concepts.”

“Despite the importance of reliable and valid measures, measurement practices in psychology are suboptimal (Borsboom, 2006).”

“For example, researchers might assume that different measures are equivalent (e.g., using stated preferences vs. behavioral tasks to measure risk preference; Frey et al., 2017) without realizing that they capture different constructs.”

“Research that is “merely” descriptive is often considered less valuable despite being crucial for forming concepts, developing measures, and establishing phenomena that need explaining (Dubin, 1969; Gerring, 2012a; Rai & Fiske, 2010; Rozin, 2001).”

“Using this paradigm, researchers have established that the functional form of the relationship between social distance and willingness to sacrifice is hyperbolic ( Jones & Rachlin, 2006; but for issues with generalizability, see Tiokhin et al., 2019).”

“…urged psychological scientists to specify more precise hypotheses (e.g., by defining a smallest effect size of interest, a region of practical equivalence (ROPE) in Bayesian estimation, or Bayesian priors; Harms & Lakens, 2018) and test them with higher statistical power (Fraley & Vazire, 2014).”

“By tightening the screws on the HD machinery and incentivizing rigorous confirmatory research, psychology’s reform movement may have inadvertently exacerbated the notion of nonconfirmatory research as a “second-class citizen” (Klahr & Simon, 1999, p. 526).”

“Research that is “merely” descriptive is often considered less valuable despite being crucial for forming concepts, developing measures, and establishing phenomena that need explaining (Dubin, 1969; Gerring, 2012a; Rai & Fiske, 2010; Rozin, 2001).”

“In practice, however, researchers have substantial difficulties incorporating these recommendations in their research, and even preregistration’s most ardent proponents acknowledge that “Preregistration Is Hard” (Nosek et al., 2019).”

“Although scientists often think of experiments in the context of confirmation, philosophers of science have emphasized the role of exploratory experiments in theory development (Franklin, 2005; Steinle, 1997, 2002).”

“One approach is to formalize hypotheses via mathematical or computational modeling (Lewandowsky & Farrell, 2010; Smaldino, 2017).”

“This can result in perpetual disagreement and inconclusive debates (Loehle, 1987).”

“In medical science, feasibility studies are used to assess recruitment and retention rates, adherence to procedures, rates of unusable responses, and the reliability and validity of measures and to estimate the standard deviation of dependent measures (Eldridge et al., 2016; Lancaster, 2015).”

“One approach is to formalize hypotheses via mathematical or computational modeling (Lewandowsky & Farrell, 2010; Smaldino, 2017).”

“Reforms proposed in reaction to the crisis tried to repair the HD machinery by making methods more rigorous (Spellman, 2015).”

“The claim that many psychological theories are critically immature has been leveled against the field so often that psychological scientists may well have grown tired of it (e.g., Fiedler, 2004; Gigerenzer, 1998; Meehl, 1967, 1978, 1990; Muthukrishna & Henrich, 2019).”

“Subfields of psychology and neighboring disciplines in which nonconfirmatory research activities are common practice have already begun to tackle these issues (see, e.g., Crüwell et al., 2019; Jacobs, 2020; Moravcsik, 2014).”

“Research that is “merely” descriptive is often considered less valuable despite being crucial for forming concepts, developing measures, and establishing phenomena that need explaining (Dubin, 1969; Gerring, 2012a; Rai & Fiske, 2010; Rozin, 2001).”

“Exploratory experimentation is widely used in psychophysics to establish law-like relationships (for a discussion of this method in research on face perception, see Jack & Schyns, 2017).”

“Steinle (2002) discusses the vital role of exploratory experiments for concept formation in the history of research on electricity.”

“Although scientists often think of experiments in the context of confirmation, philosophers of science have emphasized the role of exploratory experiments in theory development (Franklin, 2005; Steinle, 1997, 2002).”

“For example, natural selection favors strategies that maximize inclusive fitness (Hamilton, 1964), which is often not accomplished by self-preservation (e.g., people investing less in their future health when extrinsic mortality risks are high; Nettle, 2010).”

“The authors found that Tsimane personality variation was better explained by two principal factors, not five, which inspired new theoretical models to explain why the covariance structure among human personality characteristics varies across populations (Smaldino et al., 2019).”

“Many scientists invoke Popperian hypothetico-deductivism when describing aspects of their research, and the HD method’s prominent role in textbooks suggests that it shapes scientific discourse in many fields, including psychology (Mulkay & Gilbert, 1981; Riesch, 2008; Rozin, 2009).”

“In a time-series analysis of participants’ experiences while watching videos that induced kama muta, the KML documented a strong temporal connection between feeling moved, perceived closeness between the video protagonists, and expert ratings of communal sharing (Schubert et al., 2018).”

“However, some scholars have pointed out that TMT’s assumptions appear to contradict basic tenets of evolutionary theory (Kirkpatrick & Navarrete, 2006).”

“In an effort “to support and promote open-ended, open science, providing a high-status specialized format for its publication” (McIntosh, 2017, p. A2), Cortex— the journal that first introduced Registered Reports in 2013—recently launched the new, complementary format Exploratory Reports.”

“Only then did researchers empirically test the theory in those conditions (for a review, see Grose, 2011).”

“The causal model of kama muta—its proximal causes and consequences as well as its evolved function—was inspired by relational models theory (Fiske, 2004).”

“First, even preregistered hypothesis tests are rarely specified in a way that eliminates flexibility in data analysis, with unambiguous criteria for concluding that a prediction is corroborated or falsified (Lakens & DeBruine, 2021; Bakker et al., 2018).”

“Second, although the KAMMUS Two provides a valid measure of kama muta, the validity of the current operationalization of communal sharing relationships—a scale measuring “closeness” (Seibt et al., 2018)—may require further investigation.”

“Thus, pilot studies are crucial for minimizing the risk that untested auxiliaries and “manipulation-check neglect” (Fiedler, 2018, p. 435) render a study uninformative.”

“The HD method played a key part in psychology’s recent replication crisis (Derksen, 2019).”

“Research that is “merely” descriptive is often considered less valuable despite being crucial for forming concepts, developing measures, and establishing phenomena that need explaining (Dubin, 1969; Gerring, 2012a; Rai & Fiske, 2010; Rozin, 2001).”

“Without formalization, the theory might have been rejected outright, and the conditions in which it was logically coherent might not have been discovered (for similar issues with prominent verbal theories in social psychology, see Harris, 1976).”

“The claim that many psychological theories are critically immature has been leveled against the field so often that psychological scientists may well have grown tired of it (e.g., Fiedler, 2004; Gigerenzer, 1998; Meehl, 1967, 1978, 1990; Muthukrishna & Henrich, 2019).”

“Subfields of psychology and neighboring disciplines in which nonconfirmatory research activities are common practice have already begun to tackle these issues (see, e.g., Crüwell et al., 2019; Jacobs, 2020; Moravcsik, 2014).”

“However, there were other situations in which people appeared to experience kama muta without the intensification of a communal sharing relationship (e.g., performing and listening to certain types of music without the physical presence of others; Fiske et al., 2017).”

“…modification to the HD method was Popper’s critical rationalism (Popper, 1959): Although empirical data never allow us to infer that a theory is true, theories that survive repeated tests with a high capacity to falsify their predictions are more strongly “corroborated” (Fidler et al., 2018).”

“For example, we may notice that open-ended exploration, description (Rozin, 2001), or prediction (Yarkoni & Westfall, 2017) are more suitable endeavors for the matter at hand or that more basic questions about measurement need to be settled first (Scheel et al., 2021).”

“Although the lack of conceptual clarity has been observed to be a widespread and fundamental problem in psychology (Antonakis, 2017; Eronen & Bringmann, 2021; Flake, 2021; MacKenzie, 2003; Podsakoff et al., 2016; Scheel et al., 2021), conceptual clarification plays a mostly marginal role in psychological research (see…”

“The coherence and plausibility of a theory depends on its degree of fit with existing knowledge or other, well-corroborated theory (Scheel et al., 2021).”

“Furthermore, without a strong theoretical background, hypotheses will be arbitrary and could lead to unhelpful inferences [32, 33], and so should be avoided in novel areas of research.”

“This tendency comes at the detriment of generating cumulative behavioural knowledge on which experimental research should be built 100 .”

“Taking these steps toward diversifying our samples and conducting culturally sensitive research will not just benefit our understanding of underrepresented groups, it will strengthen relationship science as a whole by establishing the foundational knowledge necessary to develop strong, testable theories that will bett…”

“Indeed, many researchers may not even be at a stage where they are ready or able to test hypotheses 175 .”

“More recently a growing number of scientists are making their voice heard about the importance of exploratory research (Aczel, 2024; Höfler et al., 2022; Kimmelman et al., 2014; Scheel et al., 2021; Schwab & Held, 2020; Tong, 2019).”

“Currently, the incentive structures in academic psychology tend to reward primary research that attempts to test hypotheses over validation of measures even when the ability to test those hypotheses relies on valid measurement (Scheel, Tiokhin, et al., 2021).”

“The general problem of developing rigorous measurement tools for theory building in the social sciences has been pointed out many times before (Eronen & Bringmann, 2021; Flake & Fried, 2020; Guest & Martin, 2021; Kellen, Davis-Stober, Dunn, & Kalish, 2021; Luce & Krumhansl, 1988; Meehl, 1967; Oberauer & Lewandowsky, 2…”

“These approaches enable us to focus more on data-driven research instead of hypothesis driven research (Scheel et al., 2020).”

“the value of exploratory analyses and urge reviewers and funders to take this into account more in the future (Scheel et al., 2021).”

“The key advance in resolving these apparent differences was systematically exploring variability in the phenomenon and boundary conditions (see also Scheel et al., 2021).”

“Importantly, if the prerequisites and assumptions of formal inferential methods (‘hypothesis testing’) are not met, rather using descriptive statistics (Scheel et al. 2020) and/or data visualizations (Buja et al.”

“answering big questions requires space for descriptive and exploratory research [10].”

“These activities are important elements in the “ derivation chain ” , are valuable and should not just be treated as a necessity for hypothesis testing (Scheel et al., 2021b).”

“descriptive approach, and this aim of psychology has been suggested for at least half a century [20, 42, 43].”

“If researchers realize that they are actually not yet ready to make a falsifiable statistical prediction when they are creating a machine-readable hypothesis test, we would consider this a benefit as well (Scheel et al., 2020).”

“They may face uncertainty about how to categorize or even conceptualize these new phenomena—and yet develop concepts and explanations they must (Scheel et al., 2021).”

“We expect that these data will provide a baseline as to how individuals with TBI are responding to the pandemic and will begin a derivation chain informing future hypothesis-driven research in this area.(28)”

“A key principle of NHST is the statements of the null hypothesis and alternative hypothesis, where the alternative hypothesis is sufficiently detailed to make it statistically falsifiable rather than just a complement of the null hypothesis [81].”

“Our study combines confirmatory testing with theory-informed exploratory analysis, consistent with recent recommendations concerning good research practice [84, 107].”

“This improves scientific efficiency by weeding out logically incoherent ideas, determining the conditions under which an idea applies, and making transparent which observations must be made to test an idea’s empirical validity 75 .”

“However, recent recommendations suggest that applying a confirmatory frame to research when there is insufficient precedent and gaps in the “derivation chain” [36] is unlikely to be successful.”

“Thus, it becomes evident that more specific theorizing enables more precise operationalizations and more adequate empirical testing (Scheel et al., 2021).”

“Developing and refining concepts are important parts of research because these processes define the objects to be studied and included in theories (Scheel et al., 2021).”

“There is a chain of assumptions that allow researchers to derive predictions of behavioral data from a theory that is rarely couched in behavioral terms (Meehl, 1990; Scheel et al., 2021).”

“Scheel et al. (2021) suggested that gaining profound field knowledge is crucial for theory and hypothesis building.”

“Rather, it must be preceded by studies that increase the validity of the theoretical and methodological assumptions to be made in the hypothesis test, and thereby strengthen the derivational chain (Scheel et al., 2020).”

“Possible tools to strengthen theoretical development relate to mathematically formalizing verbal expectations or using computational models (e.g., Devezer et al., 2021; Guest & Martin, 2021; Scheel et al., 2021; van Rooij & Blokpoel, 2020).”

“…is clear the field could benefit from a concerted constructive effort in establishing a coordinated framework from which to structure well-determined specifications from theory and/or model to hypotheses about observed empirical phenomena (Borsboom et al., 2021; Maatman, 2021; Scheel et al., 2021).”

“Theory construction in this sense builds on a range of non-confirmatory activities, including descriptive research and exploratory experimentation, which are crucial to reach consensus on concept definition and measures’ validity, among the elements that are necessary for theory building (Scheel et al. 2021).”

“For instance, when researchers rely on benchmarks because no other information is available, it may be more of an indication that the research is simply not ready yet to test hypotheses and that more exploratory research is necessary (see Scheel et al., 2021).”

“As this is an exploratory study, we do not draw any specific hypotheses in terms of their relationship with mental health and wellbeing (Scheel et al., 2020; Schmueli, 2010).”

“This often derives from a lack of understanding of the derivation chain (Meehl, 1990; Scheel et al., 2021) — the chain of the assumption that links an underlying theory to observed behavior.”

“This question is important in light of the call for a better conceptualization of (psychological) constructs to advance theory and measurement [30-32].”

“…of descriptive approaches to new and complex questions, especially prior to statistical prediction, explanation, and intervention (Cooper, 2016; Scheel et al., 2021), we engaged in an empirical effort to describe the content and contexts of intergenerational storytelling about…”

“Other psychologists have also pointed to the conceptual vagueness of hypotheses and insufficient definitions of key terms (Scheel, 2021; Scheel et al., 2021) Issues of validity are relevant for assessing the replication crisis.”

“research instead of hypothesis driven research (Scheel et al., 2020).”

“Nonetheless, both types of scientists would (hopefully) agree that we need proper conceptual precision, logical reasoning, valid measures, and good data quality to be at all able to properly ‘ test ’ any hypotheses (see Scheel, Tiokhin, Isager, & Lakens, 2020) or solve any problem, respectively.”

“Before studying sound processing in everyday life in a hypothesis-driven manner with this system, each component has to be carefully tested and validated (Scheel et al., 2021).”

“While such exploratory approaches play an important role, and flexibility can occur even after registration (Scheel et al., 2020), we suggest the content of measures should be attended to, particularly when combined.”

“Here, we present descriptive data that may form the foundation for future hypothesis-driven intervention research in this area [24].”

“[125] provide a useful overview of inputs to the derivation chain, as well as research activities that can bolster it.”

“Educational psychologists are investing too much time in applying and testing existing theories and not enough in articulating and developing theories (Borsboom et al., 2021; Scheel et al., 2021).”

“In reality, however, most work — descriptive included — falls on a continuum between confirmatory and exploratory (Scheel et al., 2021).”

“Of the many factors which play a part in the replication crisis, a consistent theme is that researchers take liberty with several steps in proceeding from concept formation to deriving statistical predictions (Scheel et al., 2020).”