Improving the analysis of routine outcome measurement data: what a Bayesian approach can do for you.
Open accessAbstract
Since recent decades, clinicians offering interventions against mental problems must systematically collect data on how clients change over time. Since these data typically contain measurement error, statistical tests have been developed which should disentangle true changes from random error. These statistical tests can be subdivided into two types: classical tests and Bayesian tests. Over the past, there has been much confusion among analysts regarding the questions that are answered by each of these tests. In this paper we discuss each type of test in detail and explain which questions are, and which are not, answered by each of the types of tests. We then apply a test of each type on an empirical data set and compare the results. Copyright © 2015 John Wiley & Sons, Ltd.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The Bayesian framework for SCED suggested by de Vries et al. (2016) is stricter than the Jacobson and Truax (1991) method and does not rely on the conceptual problems of p-values.
“This framework is stricter than the most used method advocated by Jacobson and Truax (1991) and does not rely on the conceptual problems of p-values.”
From A Multiple-Baseline Evaluation of Acceptance and Commitment Therapy Focused on Repetitive Negative Thinking for Comorbid Generalized Anxiety Disorder and Depression (2020) · cites this paper as de Vries et al., 2016
The Introduction (page not available) identifies the Jacobson and Truax method as the most often used and says classical tests such as its reliable change index (RCI) are not measures of evidence. The Empirical example (page not available) finds cases where the RCI indicates reliable change but the Bayes factor shows little evidence for change. That supports the concern about p-values, but the paper says the methods have different aims; it does not establish that the Bayesian framework is generally stricter. (LLM, conf=0.86)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Cohen, J. (1992) · Psychological Bulletin · doi:10.1037//0033-2909.112.1.155 · Cited by 8,771 · see citing works
Jacobson, N. S., & Truax, P. (1991) · Journal of Consulting and Clinical Psychology · doi:10.1037/0022-006x.59.1.12 · Cited by 7,770 · see citing works
Toward Evidence-Based Medical Statistics. 1: The P Value Fallacy
Goodman, S. N. (1999) · Annals of Internal Medicine · doi:10.7326/0003-4819-130-12-199906150-00008 · Cited by 1,188 · see citing works
Toward Evidence-Based Medical Statistics. 2: The Bayes Factor
Goodman, S. N. (1999) · Annals of Internal Medicine · doi:10.7326/0003-4819-130-12-199906150-00019 · Cited by 913 · see citing works
Revised standards for statistical evidence
Johnson, V. E. (2013) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1313476110 · Cited by 808 · see citing works
Bayes factor approaches for testing interval null hypotheses.
Morey, R. D., & Rouder, J. N. (2011) · Psychological Methods · doi:10.1037/a0024377 · Cited by 584 · see citing works
Buonaccorsi, J. P. (2010) · Chapman & Hall/CRC eBooks · doi:10.1201/9781420066586 · Cited by 426 · see citing works
Clinical Significance Methods: A Comparison of Statistical Techniques
Bauer, S., Lambert, M. J., & Nielsen, S. L. (2004) · Journal of Personality Assessment · doi:10.1207/s15327752jpa8201_11 · Cited by 258 · see citing works
Ogles, B. M., Lunnen, K. M., & Bonesteel, K. (2001) · Clinical Psychology Review · doi:10.1016/s0272-7358(99)00058-6 · Cited by 255 · see citing works
Stiles, W. B., Barkham, M., Twigg, E., Mellor‐Clark, J., & Cooper, M. (2006) · Psychological Medicine · doi:10.1017/s0033291706007136 · Cited by 179 · see citing works
Finch, A. E., Lambert, M. J., & Schaalje, B. G. (2001) · Clinical Psychology & Psychotherapy · doi:10.1002/cpp.286 · Cited by 143 · see citing works
Lambert, M. J., Whipple, J. L., Bishop, M. J., Vermeersch, D. A., Gray, G. V., & Finch, A. E. (2002) · Clinical Psychology & Psychotherapy · doi:10.1002/cpp.333 · Cited by 128 · see citing works
Routine outcome assessment in mental health services
Slade, M. (2002) · Psychological Medicine · doi:10.1017/s0033291701004974 · Cited by 111 · see citing works
Slade, M. (2002) · Australian & New Zealand Journal of Psychiatry · doi:10.1046/j.1440-1614.2002.01099.x · Cited by 102 · see citing works
Simpson, S. A., Corney, R., Fitzgerald, P. B., & Beecham, J. K. (2003) · Psychological Medicine · doi:10.1017/s0033291702006517 · Cited by 65 · see citing works
A comparison of rational versus empirical methods in the prediction of psychotherapy outcome
Spielmans, G. I., Masters, K. S., & Lambert, M. J. (2006) · Clinical Psychology & Psychotherapy · doi:10.1002/cpp.491 · Cited by 49 · see citing works
Modelling Candidate Effectiveness Indicators for Mental Health Services
Burgess, P. M., Pirkis, J., & Coombs, T. (2009) · Australian & New Zealand Journal of Psychiatry · doi:10.1080/00048670902873656 · Cited by 41 · see citing works
Bayesian hypothesis testing for single-subject designs.
de Vries, R. M., & Morey, R. D. (2013) · Psychological Methods · doi:10.1037/a0031037 · Cited by 37 · see citing works
[Behavioural standard or coercive measure? Some considerations regarding the special issue on ROM].
van Os, J., Kahn, R. S., Denys, D. A. J. P., Schoevers, R. A., Beekman, A. T. F., Hoogendijk, W. J. G., van Hemert, A. M., Hodiamont, P. P. G., Scheepers, F. E., Delespaul, P., et al. (2012) · PubMed · Cited by 18 · see citing works
A Tutorial on Computing Bayes Factors for Single-Subject Designs
de Vries, R. M., Hartogs, B. M. A., & Morey, R. D. (2014) · Behavior Therapy · doi:10.1016/j.beth.2014.09.013 · Cited by 16 · see citing works
Flagged first, then by citations
2 of the 22 references couldn’t be matched to a known paper.
Research works printed but not linked: 4
- De Beurs E. (2010) De Genormaliseerde T-score. Een "euro" voor testuitslagen [The normalized T-score. A "euro" for test outcomes]. Maandblad Geestelijke volksgezondheid, 65(9), 684-695.
- Pearson, Inc.
- In Bernardo J.M. (eds) Bayesian Statistics 2, pp. 249-270, Amsterdam, North-Holland: Elsevier Science Publishers.
- Van Os J., Kahn R., Denys D., Schoevers R.A., Beekman A.T.F., Hoogendijk W.J.G., van Hemert A.M., Hodiamont P.P.G., Scheepers F., Delespaul Ph. A.E.G., Leentjens A.F.G. (2012) ROM: gedragsnorm of dwangmaatregel? Tijdschrift voor Psychiatrie, 54(3), 245-253.
Cited by
Sorted by citations
Ruiz, F. J., Flórez, C. L., García-Martín, M. B., Monroy-Cifuentes, A., Barreto-Montero, K., García-Beltrán, D. M., Riaño-Hernández, D., Sierra, M. A., Suárez-Falcón, J. C., Cardona-Betancourt, V., et al. (2018) · Journal of Contextual Behavioral Science · doi:10.1016/j.jcbs.2018.04.004 · Cited by 65 · see citing works
Between-group minimally important change versus individual treatment responders
Hays, R. D., & Peipert, J. D. (2021) · Quality of Life Research · doi:10.1007/s11136-021-02897-z · Cited by 63 · see citing works
Ruiz, F. J., Luciano, C., Flórez, C. L., Suárez-Falcón, J. C., & Cardona-Betancourt, V. (2020) · Frontiers in Psychology · doi:10.3389/fpsyg.2020.00356 · Cited by 49 · see citing works
“To analyze clinically significant changes produced by the intervention, the Bayesian framework for SCED suggested by de Vries et al. (2016) was adopted.”
Hays, R. D., Slaughter, M. E., Spritzer, K. L., & Herman, P. M. (2021) · Journal of Manipulative and Physiological Therapeutics · doi:10.1016/j.jmpt.2022.03.002 · Cited by 5 · see citing works
Fellinghauer, C., Hodel, J., Moreira, B., Kim, J.-M., & Sabariego, C. (2025) · PLoS ONE · doi:10.1371/journal.pone.0320068 · Cited by 4 · see citing works
Conijn, J. M., Spinhoven, P., Meijer, R. R., & Lamers, F. (2016) · International Journal of Methods in Psychiatric Research · doi:10.1002/mpr.1548 · Cited by 2 · see citing works
Redmayne, K. A., Reese, R. J., Zetzer, H. A., Conoley, C. W., & Owen, J. J. (2023) · Psychotherapy Research · doi:10.1080/10503307.2023.2181113 · Cited by 2 · see citing works
Cifuentes, C. E. G., Ruiz, F. J., Manrique, K. N. B., Hernández-Bernal, X., Morales, D. M. O., Peña-Vargas, A., Toquica-Orjuela, D. J., & Pérez-Nieves, K. T. (2025) · The Psychological Record · doi:10.1007/s40732-025-00634-3 · Cited by 1 · see citing works
- Authors
- de Vries, R. M., Meijer, R. R., van Bruggen, V., & Morey, R. D.
- Affiliations
- University of Groningen, Faculty of Behavioral and Social Sciences Department of Psychometrics and Statistics the Netherlands
- University of Groningen (NL)
- Dimence Almelo The Netherlands
- Cardiff UK
- Venue
- International Journal of Methods in Psychiatric Research
- Publisher
- Wiley
- Published
- 2015-10-08
- Volume
- 25
- Issue
- 3
- Pages
- 155-167
- ISSN
- 1049-8931 (ISSN-L)
- 1557-0657
- PMC
- PMC6877109
- OpenAlex
- W2144859260
- Cited by
- 8
- References
- 22
- Citations by year
- 2025: 2 · 2023: 1 · 2021: 2 · 2020: 1 · 2018: 1 · 2016: 1
- FWCI
- 0.59
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- International Journal of Methods in Psychiatric Research (publishedVersion)
- Europe PMC (PubMed Central) (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- University of Groningen research database (University of Groningen / Centre for Information Technology) (publishedVersion, other-oa)
- University of Twente Research Information (publishedVersion)
- University of Groningen research database (University of Groningen / Centre for Information Technology) (submittedVersion, other-oa)
- University of Twente Research Information (submittedVersion)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Mental Health Research Topics
- Keywords
- routine outcome measurement, measurement error, mental health interventions, comparison of statistical tests
- MeSH terms
- Bayes Theorem, Mental Disorders, Data Interpretation, Statistical, Humans, Outcome Assessment, Health Care
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
