Measuring Clinically Meaningful Change Following Mental Health Treatment
The DOI matches its Crossref record; 59 of the 63 references it cites were found in a registry or an index.
Abstract
Assessment of clinically meaningful change is useful for treatment planning, monitoring progress, and evaluating treatment response. Outcome studies often assess statistically significant change, which may not be clinically meaningful. Study objectives are to: (1) evaluate responsiveness of the BASIS-24 using three methods for determining clinically meaningful change: reliable change index (RCI), effect size (ES), and standard error of measurement (SEM); and (2) determine which method provides an estimate of clinically meaningful change most concordant with other change measures. BASIS-24 assessments were obtained at two time points for 1,397 inpatients and 850 outpatients. The proportion showing clinically meaningful change using each method was compared to the proportion showing change in global mental health, retrospectively reported change, and clinician-assessed change. BASIS-24 demonstrated responsiveness at both aggregate and individual levels. Regarding clinically meaningful improvement and decline, SEM was most concordant with all three outcome measures; regarding no change, RCI was most concordant with all three measures.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“As the Reliable Change Index can be considered a stringent criterion for clinically meaningful change (Eisen, Ranganathan, Seal, & Spiro III, 2007), we performed additional analyses using absolute cut-offs for clinically meaningful change available for all three measures.”
From Both positive mental health and psychopathology should be monitored in psychotherapy: Confirmation for the dual-factor model in acceptance and commitment therapy (2017) · cites this paper as Eisen, Ranganathan, Seal, & Spiro III, 2007
In the citing paper
“Clinically, a medium effect size (d = 0.5) corresponds to an amount of change that is noticeable to a careful observer.[28][29][30]”
From Effectiveness of Virtual Reality in Children With Cerebral Palsy: A Systematic Review and Meta-Analysis of Randomized Controlled Trials (2018) · cites this paper as [29]
Citations
Cites
Diagnostic and Statistical Manual of Mental Disorders
Williams, J. B. W., & First, M. (2013) · Encyclopedia of Social Work · doi:10.1093/acrefore/9780199975839.013.104 · Cited by 109,519
Statistical Power Analysis for the Behavioral Sciences
Muller, K. (1989) · Technometrics · doi:10.1080/00401706.1989.10488618 · Cited by 99,167
Statistical power analysis for the behavioral sciences
(1990) · Computers, Environment and Urban Systems · doi:10.1016/0198-9715(90)90050-4 · Cited by 65,269
A Coefficient of Agreement for Nominal Scales
Cohen, J. (1960) · Educational and Psychological Measurement · doi:10.1177/001316446002000104 · Cited by 42,398 · cited 1×
“The weighted kappa statistic was used to determine agreement in the proportion of participants categorized as declined, stable, or improved based on each of the three methods for evaluating individual change.(37) To determine which of the three methods is most concordant with clinically meaningful change on the BASIS…”
Statistical Power Analysis for the Behavioral Sciences (2nd ed.).
Lachenbruch, P. A., & Cohen, J. (1989) · Journal of the American Statistical Association · doi:10.2307/2290095 · Cited by 37,395
Health Measurement Scales: A Practical Guide to Their Development and Use
Streiner, D. L., Norman, G. R., & Cairney, J. (1989) · Cited by 9,595
Jacobson, N. S., & Truax, P. (1991) · Journal of Consulting and Clinical Psychology · doi:10.1037/0022-006x.59.1.12 · Cited by 7,770 · cited 2×
“Jacobson and Truax developed the reliable change index (RCI) as the first step of a two-step process for determining clinical significance of change.(24) The RCI is calculated to determine whether the magnitude of change is statistically reliable.”
Jaeschke, R., Singer, J., & Guyatt, G. H. (1989) · Controlled Clinical Trials · doi:10.1016/0197-2456(89)90005-6 · Cited by 4,982
Norman, G. R., Sloan, J. A., & Wyrwich, K. W. (2003) · Medical Care · doi:10.1097/01.mlr.0000062554.74615.4c · Cited by 4,700 · cited 3×
“80 are considered large.(20,23) Jacobson and Truax developed the reliable change index (RCI) as the first step of a two-step process for determining clinical significance of change.”
MethodInfluential(1997) · Obesity Surgery · doi:10.1381/096089297765555359 · Cited by 3,916
Streiner, D. L., & Norman, G. R. (2008) · doi:10.1093/acprof:oso/9780199231881.001.0001 · Cited by 2,631
Effect Sizes for Interpreting Changes in Health Status
Kazis, L. E., Anderson, J. J., & Meenan, R. F. (1989) · Medical Care · doi:10.1097/00005650-198903001-00015 · Cited by 2,401
Individual-patient monitoring in clinical practice: are available health status surveys adequate?
McHorney, C. A., & Tarlov, A. R. (1995) · Quality of Life Research · doi:10.1007/bf01593882 · Cited by 2,206 · cited 1×
“^ However, there is still no clear consensus regarding standards for determining clinically meaningful change.(3,6,18,19) Effect size was first developed by Cohen to assess the magnitude of a treatment effect, originally using aggregate data, but later to assess change in individuals as well.”
From hospital to community psychiatry
Jones, M. (1970) · Community Mental Health Journal · doi:10.1007/bf01435915 · Cited by 2,043
Measuring change over time: Assessing the usefulness of evaluative instruments
Guyatt, G., Walter, S., & Norman, G. (1987) · Journal of Chronic Diseases · doi:10.1016/0021-9681(87)90069-5 · Cited by 1,880 · cited 1×
“An important criterion for instruments designed to assess treatment effectiveness is that they be sensitive to change in the construct they were designed to measure.(1,2) Sensitivity, or responsiveness to change, should be considered a distinct criterion in assessing the psychometric properties of health status instru…”
The Use of Psychological Testing for Treatment Planning and Outcome Assessment
Maruish, M. E., & Leahy, R. L. (2000) · Journal of Cognitive Psychotherapy · doi:10.1891/0889-8391.14.2.205 · Cited by 1,634
The Use of Psychological Testing for Treatment Planning and Outcomes Assessment
(2004) · doi:10.4324/9781410610614 · Cited by 1,503
Methods to Explain the Clinical Significance of Health Status Measures
Guyatt, G. H., Osoba, D., Wu, A. W., Wyrwich, K. W., & Norman, G. R. (2002) · Mayo Clinic Proceedings · doi:10.4065/77.4.371 · Cited by 1,500 · cited 1×
“were self-report measures; consequently, they were not external Banchor^ measures, although these types of measures (global evaluations and transition ratings) have been used in previous research.(5,16,42) The third measure (GAF) was an external measure because it is determined by the clinician.”
Health Measurement Scales: A Practical Guide To Their Development And Use
Streiner, D. L., & Norman, G. R. (2003) · doi:10.1093/oso/9780198528470.001.0001 · Cited by 1,460
Jacobson, N. S., Follette, W. C., & Revenstorf, D. (1984) · Behavior Therapy · doi:10.1016/s0005-7894(84)80002-7 · Cited by 1,294 · cited 1×
“Second, because normative data for an untreated population are not available for BASIS-24, it was not possible to determine the proportion of individuals whose mental health status moved into the range of a Bnormal^ population.(24,25,41) Although three other mental health outcome measures were used to determine which…”
Assessing the Clinical or Applied Importance of Behavior Change through Social Validation
Kazdin, A. E. (1977) · Behavior Modification · doi:10.1177/014544557714001 · Cited by 1,117
Jones, S. H., Thornicroft, G., Coffey, M., & Dunn, G. (1995) · British Journal of Psychiatry · doi:10.1192/bjp.166.5.654 · Cited by 1,015
Diagnostic and Statistical Manual of Mental Disorders, 4th ed. (DSM-IV)
GUZE, S. B. (1995) · American Journal of Psychiatry · doi:10.1176/ajp.152.8.1228 · Cited by 890
Wyrwich, K. W., Tierney, W. M., & Wolinsky, F. D. (1999) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(99)00071-2 · Cited by 862 · cited 3×
“Although varying statistical thresholds have been used to determine clinically meaningful change using the SEM, recent research has reported that one SEM consistently corresponded to a minimal clinically important intra-individual change.(5,30,31) Consequently, in this paper, one SEM will be used as the criterion for…”
MethodJacobson, N. S., Roberts, L. J., Berns, S. B., & McGlinchey, J. B. (1999) · Journal of Consulting and Clinical Psychology · doi:10.1037/0022-006x.67.3.300 · Cited by 853
Handbook of Psychiatric Measures
Rush, A. J., First, M. B., & Blacker, D. (2000) · Medical Entomology and Zoology · Cited by 846
Assessing the clinical significance of change scores recorded on subjective outcome measures
Hurst, H., & Bolton, J. (2004) · Journal of Manipulative and Physiological Therapeutics · doi:10.1016/j.jmpt.2003.11.003 · Cited by 702 · cited 1×
“Consistent with previous findings reported in the literature, both the SEM and ES methods identified a higher proportion of individuals as meaningfully improved than did the RCI method.(7,10,17) Second, change varied as a function of both program type and symptom/problem domain.”
Wyrwich, K. W., Nienaber, N. A., Tierney, W. M., & Wolinsky, F. D. (1999) · Medical Care · doi:10.1097/00005650-199905000-00006 · Cited by 569 · cited 3×
“Although varying statistical thresholds have been used to determine clinically meaningful change using the SEM, recent research has reported that one SEM consistently corresponded to a minimal clinically important intra-individual change.(5,30,31) Consequently, in this paper, one SEM will be used as the criterion for…”
MethodHealth measurement scales. A practical guide to their development and use, 3rd ed
Rajmil, L. (2005) · Europe PMC (PubMed Central) · Cited by 567
Psychological testing: By Marianne Cederblad and Allis Danielson
(1968) · Acta Psychiatrica Scandinavica · doi:10.1111/j.1600-0447.1968.tb07939.x · Cited by 557
Psychometric considerations in evaluating health-related quality of life measures
Hays, R. D., Anderson, R., & Revicki, D. (1993) · Quality of Life Research · doi:10.1007/bf00422218 · Cited by 513 · cited 1×
“Statistical significance of change is often assessed at the aggregate level using paired t tests between T1 and T2, repeated measures analysis of variance or covariance, or generalized estimating equations.(4,5) However, with reasonably large samples, it is possible for small differences that may not be clinically mea…”
Estimating clinically significant differences in quality of life outcomes
Wyrwich, K. W., Bullinger, M., Aaronson, N., Hays, R. D., Patrick, D. L., & Symonds, T. (2005) · Quality of Life Research · doi:10.1007/s11136-004-0705-2 · Cited by 471 · cited 9×
“Clinically meaningful change can be defined as a noticeable, appreciable difference that is of value to the patient or the health professional, and that exceeds variation attributable to chance.(3,8) Minimally important change has been defined as Bthe difference in score on a health-related .”
InfluentialThe concurrent validity of the Global Assessment of Functioning (GAF)
Startup, M., Jackson, M. C., & Bendix, S. (2002) · British Journal of Clinical Psychology · doi:10.1348/014466502760387533 · Cited by 366
Outcomes Research: Measuring the End Results of Health Care
Clancy, C. M., & Eisenberg, J. M. (1998) · Science · doi:10.1126/science.282.5387.245 · Cited by 324 · cited 1×
“^ However, there is still no clear consensus regarding standards for determining clinically meaningful change.(3,6,18,19) Effect size was first developed by Cohen to assess the magnitude of a treatment effect, originally using aggregate data, but later to assess change in individuals as well.”
Liang, M. H. (2000) · PubMed · Cited by 321
Self-report quality of life measure for people with schizophrenia: The SQLS
Wilkinson, G., Hesdon, B., Wild, D., Cookson, R., Farina, C., Sharma, V., Fitzpatrick, R., & Jenkinson, C. (2000) · British Journal of Psychiatry · doi:10.1192/bjp.177.1.42 · Cited by 318
A critical look at transition ratings
Guyatt, G. H., Norman, G. R., Juniper, E. F., & Griffith, L. E. (2002) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(02)00435-3 · Cited by 298 · cited 1×
“This rating corresponds to transition ratings of change, which have also been used to determine clinically meaningful change in health status.(12) Third, the clinician-rated GAF, extracted from medical records or administrative databases, provided an external measure of level of impairment.”
Clinicians and outcome measurement: What's the use?
Garland, A. F., Kruse, M., & Aarons, G. A. (2003) · The Journal of Behavioral Health Services & Research · doi:10.1007/bf02287427 · Cited by 298 · cited 2×
“However, routine outcomes assessment can be both costly and of questionable value to clinical treatment providers.(50,51) Timely knowledge about progress and treatment outcomes for individuals can provide opportunities for clinicians to improve care for individuals, utilize individual outcome data to guide future trea…”
Clinically significant change: Jacobson and Truax (1991) revisited.
Speer, D. C. (1992) · Journal of Consulting and Clinical Psychology · doi:10.1037//0022-006x.60.3.402 · Cited by 270
The Revised Behavior and Symptom Identification Scale (BASIS-R): Reliability and Validity
Eisen, S. V., Normand, S.-L., Belanger, A. J., Spiro, A., & Esch, D. (2004) · Medical Care · doi:10.1097/00005650-200412000-00010 · Cited by 262 · cited 3×
“76 for outpatients between this rating and the BASIS-24 overall mental health score.(11) Second, a 5-point rating of perceived change (0=much worse, 1=somewhat worse, 2=about the same, 3=somewhat better and 4=much better) was used.”
Self-citationNorman, G. R., Gwadry Sridhar, F., Guyatt, G. H., & Walter, S. D. (2001) · Medical Care · doi:10.1097/00005650-200110000-00002 · Cited by 262 · cited 1×
“However, results of a simulation study exploring the relationship between an anchor-based approach and effect size found a near-linear relationship, suggesting that the proportion of individuals showing meaningful improvement can be directly estimated from ES, a distribution-based method.(48) Another concern in evalua…”
Reliability and Validity of a Brief Patient-Report Instrument for Psychiatric Outcome Evaluation
Eisen, S. V., Dill, D. L., & Grob, M. C. (1994) · Psychiatric Services · doi:10.1176/ps.45.3.242 · Cited by 241 · cited 1×
“and substance abuse.(11,32) In addition, an overall mental health score is computed.”
Self-citationIdentifying meaningful intra‐individual change standards for health‐related quality of life measures
Wyrwich, K. W., & Wolinsky, F. D. (2000) · Journal of Evaluation in Clinical Practice · doi:10.1046/j.1365-2753.2000.00238.x · Cited by 227 · cited 6×
“Although varying statistical thresholds have been used to determine clinically meaningful change using the SEM, recent research has reported that one SEM consistently corresponded to a minimal clinically important intra-individual change.(5,30,31) Consequently, in this paper, one SEM will be used as the criterion for…”
MethodInfluentialAssessing clinical significance: Does it matter which method we use?
Atkins, D. C., Bedics, J. D., Mcglinchey, J. B., & Beauchaine, T. P. (2005) · Journal of Consulting and Clinical Psychology · doi:10.1037/0022-006x.73.5.982 · Cited by 161 · cited 3×
“5 Recent work by Hays et al., 7 and by Atkins et al. 10 discussed and evaluated multiple methods for determining clinically meaningful change.”
Use of the Reliable Change Index to evaluate clinical significance in SF-36 outcomes
Ferguson, R. J., Robinson, A. B., & Splaine, M. E. (2002) · Quality of Life Research · doi:10.1023/a:1016350431190 · Cited by 152 · cited 2×
“In contrast, the SEM does not vary from sample to sample, providing a more stable method for determining clinically meaningful change.(5,18) These results are also consistent with the suggestion that the RCI may be a too stringent criterion for determining clinically meaningful change, particularly for individuals wit…”
Ogles, B. M., Lambert, M. J., & Sawyer, J. D. (1995) · Journal of Consulting and Clinical Psychology · doi:10.1037//0022-006x.63.2.321 · Cited by 135
Hays, R. D., Brodsky, M., Johnston, M. F., Spritzer, K. L., & Hui, K.-K. (2005) · Evaluation & the Health Professions · doi:10.1177/0163278705275339 · Cited by 110 · cited 3×
“Consistent with previous findings reported in the literature, both the SEM and ES methods identified a higher proportion of individuals as meaningfully improved than did the RCI method.(7,10,17) Second, change varied as a function of both program type and symptom/problem domain.”
Eisen, S. V., Gerena, M., Ranganathan, G., Esch, D., & Idiculla, T. (2006) · The Journal of Behavioral Health Services & Research · doi:10.1007/s11414-006-9025-3 · Cited by 107
Self-citationGILBODY, S. M., HOUSE, A. O., & SHELDON, T. (2002) · Psychological Medicine · doi:10.1017/s0033291702006001 · Cited by 103 · cited 1×
“However, routine outcomes assessment can be both costly and of questionable value to clinical treatment providers.(50,51) Timely knowledge about progress and treatment outcomes for individuals can provide opportunities for clinicians to improve care for individuals, utilize individual outcome data to guide future trea…”
Hageman, W. J., & Arrindell, W. (1993) · Behaviour Research and Therapy · doi:10.1016/0005-7967(93)90122-b · Cited by 92
Psychotherapy Outcome Research
Behar, E. S., & Borkovec, T. D. (2003) · Handbook of Psychology · doi:10.1002/0471264385.wei0209 · Cited by 64
Burlingame, G. M., Dunn, T. W., Chen, S., Lehman, A., Axman, R., Earnshaw, D., & Rees, F. M. (2005) · Psychiatric Services · doi:10.1176/appi.ps.56.4.444 · Cited by 57
Greenberg, G. A., & Rosenheck, R. A. (2005) · Psychiatric Services · doi:10.1176/appi.ps.56.4.420 · Cited by 56
Development of outcome indicators for monitoring the quality of public mental health care
Srebnik, D. S., HENDRYX, M. S., Stevenson, J. F., Caverly, S. E., Dyck, D. G., & Cauce, A. M. (1997) · Psychiatric Services · doi:10.1176/ps.48.7.903 · Cited by 52
Testa, M. A. (1987) · PubMed · Cited by 52
New Methodological Developments in FTA
Scapolo, F., & Porter, A. L. (2008) · Springer eBooks · doi:10.1007/978-3-540-68811-2_11 · Cited by 43
Guidelines for selecting psychological instruments for treatment planning and outcome assessment.
Newman, F. L., Ciarlo, J. A., & Carpenter, D. (1999) · Cited by 26
Behavior and symptom identification scale 32
Jerrell, J. (2005) · Journal of Behavioral Health Services & Research · doi:10.1007/bf02291833 · Cited by 7
(2000) · Medical Care · doi:10.1097/00005650-200009002-00004 · Cited by 4
Research works printed but not linked: 3
- Anastasi A, Urbina S.Psychological Testing (7th ed.). Upper Saddle River, NJ: Prentice-Hall; 1997.
- Jacobson NS, Roberts LJ, Berns SB, et al. Methods for defining and determining the clinical significance of treatment effects. Journal of Consulting and Clinical Psychology. 1999;67:300–307. · DOI · Paper page
- Nunnally JC.Psychometric Theory. New York: McGraw-Hill; 1994.
Cited by
Single item measures of self-rated mental health: a scoping review
Ahmad, F., Jhajj, A. K., Stewart, D. E., Burghardt, M., & Bierman, A. S. (2014) · BMC Health Services Research · doi:10.1186/1472-6963-14-398 · Cited by 554
Reflections on Physical Activity and Health: What Should We Recommend?
Warburton, D. E. R., & Bredin, S. S. D. (2016) · Canadian Journal of Cardiology · doi:10.1016/j.cjca.2016.01.024 · Cited by 544
“There is no clear consensus on the best method of determining a clinically relevant change.(71) However, levels of minimal clinical improvement are often defined according to the patient’s perception of what is important (consistent with patient-centred care).”
MethodChen, Y., Fanchiang, H. D., & Howard, A. (2018) · Physical Therapy · doi:10.1093/ptj/pzx107 · Cited by 257
Schambra, H. M., Ogden, R. T., Martínez-Hernández, I. E., Lin, X., Chang, Y. B., Rahman, A., Edwards, D. J., & Krakauer, J. W. (2015) · Frontiers in Cellular Neuroscience · doi:10.3389/fncel.2015.00335 · Cited by 142
Trompetter, H. R., Lamers, S. M. A., Westerhof, G. J., Fledderus, M., & Bohlmeijer, E. T. (2017) · Behaviour Research and Therapy · doi:10.1016/j.brat.2017.01.008 · Cited by 126
Metadata
- Authors
- Eisen, S. V., Ranganathan, G., Seal, P., & Spiro, A.
- Affiliations
- Boston University (US)
- Edith Nourse Rogers Memorial Veterans Hospital (US)
- Venue
- The Journal of Behavioral Health Services & Research
- Published
- 2007-05-30
- Volume
- 34
- Issue
- 3
- Pages
- 272-289
- ISSN
- 1094-3412 (ISSN-L)
- 1556-3308
- OpenAlex
- W1979948629
- Semantic Scholar
- 50837b7835fcc77ed33a090de8872be2e5a57451
- Cited by
- 126
- References
- 63
- Citations by year
- 2026: 3 · 2025: 8 · 2024: 6 · 2023: 4 · 2022: 7 · 2021: 11 · 2020: 4 · 2019: 5 · 2018: 12 · 2017: 11 · 2016: 8 · 2015: 10 · 2014: 17 · 2013: 6 · 2012: 5
- FWCI
- 2.86
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- www.springer.com/tdm (tdm)
- Locations
- The Journal of Behavioral Health Services & Research (publishedVersion)
- PubMed (publishedVersion)
- Funders
- U.S. Department of Veterans Affairs
- National Institutes of Health (5r01mh058240-07)
- National Institute of Mental Health (R01 MH58240)
- Topics
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Psychology › Clinical Psychology › Psychotherapy Techniques and Applications
- Social Sciences › Psychology › Applied Psychology › Psychological Testing and Assessment
- Keywords
- clinically meaningful change, reliable change index, standard error of measurement, effect size, responsiveness, mental health treatment outcomes, global mental health
- MeSH terms
- Adolescent, Adult, Mental Disorders, Behavioral Medicine, Female, Humans, Male, Middle Aged, Surveys and Questionnaires, United States, Outcome Assessment, Health Care
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.