Measurement Schmeasurement: Questionable Measurement Practices and How to Avoid Them
Open accessThe DOI matches its Crossref record; 52 of the 52 references it cites were found in a registry or an index.
Abstract
In this article, we define questionable measurement practices (QMPs) as decisions researchers make that raise doubts about the validity of the measures, and ultimately the validity of study conclusions. Doubts arise for a host of reasons, including a lack of transparency, ignorance, negligence, or misrepresentation of the evidence. We describe the scope of the problem and focus on how transparency is a part of the solution. A lack of measurement transparency makes it impossible to evaluate potential threats to internal, external, statistical-conclusion, and construct validity. We demonstrate that psychology is plagued by a measurement schmeasurement attitude: QMPs are common, hide a stunning source of researcher degrees of freedom, and pose a serious threat to cumulative psychological science, but are largely ignored. We address these challenges by providing a set of questions that researchers and consumers of scientific research can consider to identify and avoid QMPs. Transparent answers to these measurement questions promote rigorous research, allow for thorough evaluations of a study’s inferences, and are necessary for meaningful replication studies.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“In contrast, the conceptual clarity of constructs commonly used in education and educational psychology has been recently criticized (Bringmann et al., 2022; Flake & Fried, 2020; Marsh et al., 2019).”
From Screening Smarter, Not Harder: A Comparative Analysis of Machine Learning Screening Algorithms and Heuristic Stopping Criteria for Systematic Reviews in Educational Research (2024) · cites this paper as Flake & Fried, 2020
In the citing paper
“To continue to use the AAQ-II as a measure of flexibility is to engage in a questionable measurement practice (Flake & Fried, 2020) and contributes to the replication crisis in psychology (Lilienfeld & Strother, 2020)”
From Defining and measuring “psychological flexibility”: A narrative scoping review of diverse flexibility and rigidity constructs and perspectives (2021) · cites this paper as Flake & Fried, 2020
Citations
Cites
R: A Language and Environment for Statistical Computing
R Core Team (2025) · doi:10.32614/r.manuals · Cited by 352,181
(2002) · Cited by 13,211
Construct validity in psychological tests.
Cronbach, L. J., & Meehl, P. E. (1955) · Psychological Bulletin · doi:10.1037/h0040957 · Cited by 11,195 · cited 1×
“Construct validation— collecting evidence that the instruments scientists build actually measure the constructs scientists claim they measure—is a difficult and necessary part of the research process (Cronbach & Meehl, 1955).”
BackgroundSimmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling
John, L. K., Loewenstein, G., & Prelec, D. (2012) · Psychological Science · doi:10.1177/0956797611430953 · Cited by 2,239
Dawes, J. (2008) · International Journal of Market Research · doi:10.1177/147078530805000106 · Cited by 2,107
CONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials
the CONSORT Group, Schulz, K. F., Altman, D. G., & Moher, D. (2010) · Trials · doi:10.1186/1745-6215-11-32 · Cited by 1,871
Chan, A.-W., Hróbjartsson, A., Haahr, M. T., Gøtzsche, P. C., & Altman, D. G. (2004) · JAMA · doi:10.1001/jama.291.20.2457 · Cited by 1,845
A guide to constructs of control.
Skinner, E. A. (1996) · Journal of Personality and Social Psychology · doi:10.1037//0022-3514.71.3.549 · Cited by 1,800 · cited 1×
“Reviews on a number of constructs, including giftedness (Dai & Chen, 2013), charisma (Antonakis, Bastardoz, Jacquart, & Shamir, 2016), and control (Skinner, 1996), have demonstrated that constructs are often ill defined, which leads to confusion and invalid inferences in the literature.”
BackgroundMuch ado about grit: A meta-analytic synthesis of the grit literature
Credé, M., Tynan, M. C., & Harms, P. D. (2017) · Journal of Personality and Social Psychology · doi:10.1037/pspp0000102 · Cited by 1,653
Cited by
Use Omega Rather than Cronbach’s Alpha for Estimating Reliability. But…
Hayes, A. F., & Coutts, J. J. (2020) · Communication Methods and Measures · doi:10.1080/19312458.2020.1718629 · Cited by 2,675
“…the subsets option in the OMEGA macro can assist in reducing the number of items on ameasurement instrument being developed or already in existence, it is important to remember that good measurement is about far more than establishing high reliability (see Flake & Fried, 2019; Fried & Flake, 2018).”
BackgroundMethodParry, D. A., Davidson, B. I., Sewall, C. J. R., Fisher, J. T., Mieczkowski, H., & Quintana, D. S. (2021) · Nature Human Behaviour · doi:10.1038/s41562-021-01117-5 · Cited by 793
“confident that the measures we use accurately reflect the behaviour that they are designed to assess 5,20 .”
BackgroundMethodThinking twice about sum scores
McNeish, D., & Wolf, M. G. (2020) · Behavior Research Methods · doi:10.3758/s13428-020-01398-0 · Cited by 592
“For this reason, decisions about scoring have been considered an underemphasized source of replicability issues (Flake & Fried, 2019; Fried & Flake, 2018).”
BackgroundMajor Depression and Its Recurrences: Life Course Matters
Monroe, S. M., & Harkness, K. L. (2022) · Annual Review of Clinical Psychology · doi:10.1146/annurev-clinpsy-072220-021440 · Cited by 608
Evaluating Model Fit of Measurement Models in Confirmatory Factor Analysis
Goretzko, D., Siemund, K., & Sterner, P. (2024) · Educational and Psychological Measurement · doi:10.1177/00131644231163813 · Cited by 468
“…low item relia-bilites and high measurement errors in psychological questionnaire data (Gnambs, 2015) and questionable measurement practices (Flake & Fried, 2020), poorly fitting measurement models may severely distort empirical findings and hamper the progress of certain research areas.”
Background
Metadata
- Authors
- Flake, J. K., & Fried, E. I.
- Affiliations
- McGill University (CA)
- Leiden University (NL)
- Corresponding author
- Flake, Jessica Kay
- Venue
- Advances in Methods and Practices in Psychological Science
- Publisher
- SAGE Publications
- Published
- 2020-12
- Volume
- 3
- Issue
- 4
- Pages
- 456-465
- ISSN
- 2515-2459 (ISSN-L)
- 2515-2467
- OpenAlex
- W3108041100
- Cited by
- 969
- References
- 51
- Citations by year
- 2026: 165 · 2025: 170 · 2024: 163 · 2023: 144 · 2022: 124 · 2021: 101 · 2020: 71
- FWCI
- 66.38
- Type
- journal article
- Language
- en
- Open access
- green
- doi.org
- Licence
- cc-by
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
- Locations
- Advances in Methods and Practices in Psychological Science (publishedVersion)
- Leiden Repository (Leiden University) (submittedVersion, cc-by)
- Leiden Repository (Leiden University) (submittedVersion)
- PsyArXiv (OSF Preprints) (submittedVersion, cc-by)
- PsyArXiv (OSF Preprints) (submittedVersion)
- Funders
- Association for Psychological Science
- Topics
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Keywords
- construct validity, internal validity, generalizability, statistical conclusion validity, researcher degrees of freedom, cumulative science, replication, measurement validity, research rigor
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Measurement Schmeasurement: Questionable Measurement Practices and How to Avoid Them. https://app.cochat.ai/papers/10.1177/2515245920952393
- The paper
- Flake, J. K., & Fried, E. I. (2020). Measurement Schmeasurement: Questionable Measurement Practices and How to Avoid Them. Advances in Methods and Practices in Psychological Science. https://doi.org/10.1177/2515245920952393
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.