Horizontal equity and mental health care: a study of priority ratings by clinicians and teams at outpatient clinics
Open accessAbstract
Background: In Norway, admission teams at Community Mental Health Centres (CMHCs) assess referrals from General Practitioners (GPs), and classify the referrals into priority groups according to treatment needs, as defined in the Act of Patient Rights. In this study, we analyzed classification of similar referrals to determine the reliability of classification into priority groups (i.e., horizontal equity). Methods: Twenty anonymous case vignettes based on representative referrals were classified by 42 admission team members at 16 CMHCs in the South-East Health Region of Norway. All clinicians were experienced, and were responsible for priority setting at their centres. The classifications were first performed independently by the 42 clinicians (i.e., individual rating), and then evaluated utilizing team consensus within each CMHC (i.e., team rating). Interrater reliability was estimated using intraclass correlation coefficients (ICCs) while the reliability of rating across raters and units (generalizability) were estimated using generalizability analysis. Results: The ICCs (2.1 single measure, absolute agreement) varied between 0.40 and 0.51 using individual ratings and between 0.39 and 0.58 using team ratings. Our findings suggest a fair (low) degree of interrater reliability, and no improvement of team ratings was observed when compared to individual ratings. The generalizability analysis, for one rater within each unit, yields a generalizability coefficient of 0.50 and a dependability coefficient of 0.53 (D study). These findings confirm that the reliability of ratings across raters and across units is low. Finally, the degree of inconsistency, for an average measurement, appears to be higher within units than between units (G study). Conclusion: The low interrater reliability and generalizability found in our study suggests that horizontal equity to mental health services is not ensured with respect to priority. Priority -setting in teams provides no significant improvement compared to individual rating, and the additional use of these resources may be questionable. Improved guidelines, tutorials, training and calibration of clinicians may be utilized to improve the reliability of priority-setting.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
A study found low agreement between teams when comparing triage decisions based on written referrals, pointing to possible weaknesses in the prevailing Norwegian model.
“One study points to possible weaknesses in the prevailing Norwegian model, as it found low agreement between the teams when comparing triage decisions based on written referrals [17]”
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Intraclass correlations: Uses in assessing rater reliability.
Shrout, P. E., & Fleiss, J. L. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.2.420 · Cited by 23,433 · see citing works
Generalizability theory: a primer
(1992) · Choice Reviews Online · doi:10.5860/choice.29-4192 · Cited by 1,294 · see citing works
Handbook of marketing scales: multi-item measures for marketing and consumer behavior research
Bearden, W. O., Netemeyer, R. G., & Association for Consumer Research (U.S.) (1993) · Choice Reviews Online · doi:10.5860/choice.31-1833 · Cited by 636 · see citing works
Scales for the measurement of attitudes
Shaw, M. E., & Wright, J. M. (1967) · Medical Entomology and Zoology · Cited by 551 · see citing works
Bearden, W. O., & Netemeyer, R. G. (1999) · Sage eBooks · doi:10.4135/9781412984379 · Cited by 483 · see citing works
Scales for the Measurement of Attitudes
Buttram, J. B. (1968) · Journal of Music Therapy · doi:10.1093/jmt/5.1.27-a · Cited by 458 · see citing works
Generalizability studies of the Global Assessment of Functioning–Split version
Pedersen, G., Hagtvet, K. A., & Karterud, S. (2006) · Comprehensive Psychiatry · doi:10.1016/j.comppsych.2006.03.008 · Cited by 446 · see citing works
Shavelson, R. J., Webb, N. M., & ROWLEY, G. L. (1989) · American Psychologist · doi:10.1037/0003-066x.44.6.922 · Cited by 372 · see citing works
Priority setting in health care: Lessons from the experiences of eight countries
Sabik, L. M., & Lie, R. K. (2008) · International Journal for Equity in Health · doi:10.1186/1475-9276-7-4 · Cited by 312 · cited 1× · see citing works
“Priority-setting is a complex and difficult challenge faced by decision-makers at all levels of a health care system, and a wide range of approaches and priority-setting guidelines are observed across various countries [2].”
Cicchetti, D. V., Shoinralter, D., & Tyrer, P. J. (1985) · Applied Psychological Measurement · doi:10.1177/014662168500900103 · Cited by 290 · see citing works
Comparing Rating Scales of Different Lengths: Equivalence of Scores from 5-Point and 7-Point Scales
Colman, A. M., Norris, C. E., & Preston, C. C. (1997) · Psychological Reports · doi:10.2466/pr0.1997.80.2.355 · Cited by 280 · see citing works
Priority setting in health care: learning from international experience
Ham, C. (1997) · Health Policy · doi:10.1016/s0168-8510(97)00054-7 · Cited by 214 · see citing works
Söderberg, P., Tungström, S., & Armelius, B. (2005) · Psychiatric Services · doi:10.1176/appi.ps.56.4.434 · Cited by 203 · see citing works
The Use of a Classification Approach to Identify Subgroups of Patients With Acute Low Back Pain
Fritz, J. M., & George, S. Z. (2000) · Spine · doi:10.1097/00007632-200001010-00018 · Cited by 194 · see citing works
Karterud, S., Pedersen, G., Bjordal, E., Brabrand, J., Friis, S., Haaseth, Ø., Haavaldsen, G., Irion, T., Leirvåg, H., Tørum, E., et al. (2003) · Journal of Personality Disorders · doi:10.1521/pedi.17.3.243.22151 · Cited by 178 · see citing works
Waiting for scheduled services in Canada: development of priority‐setting scoring systems
Noseworthy, T., McGurran, J. J., & Hadorn, D. C. (2003) · Journal of Evaluation in Clinical Practice · doi:10.1046/j.1365-2753.2003.00377.x · Cited by 156 · see citing works
Inter-rater agreement in the scoring of abstracts submitted to a primary care research conference
Montgomery, A. A., Graham, A., Evans, P. H., & Fahey, T. P. (2002) · BMC Health Services Research · doi:10.1186/1472-6963-2-8 · Cited by 148 · see citing works
Are GAF scores reliable in routine clinical use?
Vatnaland, T., Vatnaland, J., Friis, S., & Opjordsmoen, S. (2007) · Acta Psychiatrica Scandinavica · doi:10.1111/j.1600-0447.2006.00925.x · Cited by 137 · see citing works
Inter‐Rater Reliability of Global Assessment of Functioning in a Clinical Setting
Rey, J. M., Starling, J., Wever, C., Dossetor, D. R., & Plapp, J. M. (1995) · Journal of Child Psychology and Psychiatry · doi:10.1111/j.1469-7610.1995.tb01329.x · Cited by 127 · see citing works
Kane, M. T. (1996) · Applied Measurement in Education · doi:10.1207/s15324818ame0904_4 · Cited by 106 · see citing works
Morrison, K. A. (1997) · Psychological Reports · doi:10.2466/pr0.1997.80.1.255 · Cited by 89 · see citing works
Bjørkly, S. K., Hartvig, P., Heggen, F.-A., Brauer, H., & Moger, T. A. (2009) · European Psychiatry · doi:10.1016/j.eurpsy.2009.07.004 · Cited by 62 · see citing works
Routine evaluation of mental health: reliable information or worthless‘guesstimates’?
Loevdahl, H., & Friis, S. (1996) · Acta Psychiatrica Scandinavica · doi:10.1111/j.1600-0447.1996.tb09813.x · Cited by 46 · see citing works
Coster, C. D., McMillan, S., Brant, R., McGurran, J. J., Noseworthy, T., & Primary Care Panel of the Western Canada Waiting List Project (2006) · Journal of Evaluation in Clinical Practice · doi:10.1111/j.1365-2753.2006.00671.x · Cited by 43 · see citing works
Identifying occupational therapists' referral priorities in community health
Harries, P., & Gilhooly, K. J. (2003) · Occupational Therapy International · doi:10.1002/oti.182 · Cited by 30 · see citing works
The routine assessment of severity amongst people with mental illness
Phelan, M., Seller, J., & Leese, M. (2001) · Social Psychiatry and Psychiatric Epidemiology · doi:10.1007/s001270170064 · Cited by 17 · see citing works
helsedirektoratet, S. O., Larsen, B.-I., & Bjørneboe, G. A. (2009) · BIBSYS Brage (BIBSYS (Norway)) · Cited by 10 · see citing works
The Bed Requirement Inventory: A Simple Measure To Estimate The Need For A Psychiatric Bed
Tyrer, P. J., Suryanarayan, G., Rao, B., Cicchetti, D. V., Fulop, N., Roberts, F., & Slaughter, J. (2006) · International Journal of Social Psychiatry · doi:10.1177/0020764006067221 · Cited by 7 · see citing works
Bjørkly, S. K., & Moger, T. A. (2007) · Psychological Reports · doi:10.2466/pr0.101.4.1145-1161 · Cited by 5 · see citing works
Holman, P. A., Grepperud, S., & Tanum, L. (2011) · PubMed · Cited by 2 · see citing works
Self-citation
Flagged first, then by citations
1 of the 31 references couldn’t be matched to a known paper.
Research works printed but not linked: 1
- Kolstad A Hjort H Olson RP, Olson RP Mental health service in Norway, Chapter 3 in Mental health Systems Compared 2006 Charles C Thomas - Publisher, ltd, Illinois U.S.A 81 137
Cited by
Sorted by citations
Thorsen, O., Hartveit, M., & Bærheim, A. (2013) · BMC Family Practice · doi:10.1186/1471-2296-14-153 · Cited by 32 · see citing works
Hartveit, M., Thorsen, O., Biringer, E., Vanhaecht, K., Carlsen, B., & Aslaksen, A. B. (2013) · BMC Health Services Research · doi:10.1186/1472-6963-13-329 · Cited by 16 · see citing works
Januleviciute, J., Heggestad, T., & Norheim, O. F. (2016) · International Journal of Health Policy and Management · doi:10.15171/ijhpm.2016.23 · Cited by 12 · see citing works
Nymoen, M., Biringer, E., Hetlevik, Ø., Thorsen, O., Assmus, J., & Hartveit, M. (2022) · BMC Health Services Research · doi:10.1186/s12913-022-08139-3 · Cited by 11 · see citing works
Defining when specialised mental health care is needed: a focus group study
Nymoen, M., Biringer, E., Helgeland, J., Hellesen, H. B., Alsaker Sande, L., & Hartveit, M. (2020) · BJGP Open · doi:10.3399/bjgpopen20x101004 · Cited by 9 · see citing works
Grepperud, S., Holman, P. A., & Wangen, K. R. (2014) · BMC Health Services Research · doi:10.1186/s12913-014-0620-3 · Cited by 9 · see citing works
Self-citationYang, F. B., Wangen, K. R., Victor, M., Solbakken, O. A., & Holman, P. A. (2022) · BMC Health Services Research · doi:10.1186/s12913-022-08877-4 · Cited by 8 · see citing works
Self-citationLindberg, M. S., Brattmyr, M., Lundqvist, J., Solem, S., Hjemdal, O., Roos, E., Fjeldsæter, A. B., Björgvinsson, Þ., Cornish, P. A., & Havnen, A. (2024) · Psychotherapy Research · doi:10.1080/10503307.2024.2378017 · Cited by 8 · see citing works
Tveit, O. G., Ruud, T., Hanssen‐Bauer, K., Haavet, O. R., & Hussain, A. (2021) · BMC Health Services Research · doi:10.1186/s12913-021-06982-4 · Cited by 6 · see citing works
Self-citationFolkehelse og forebygging. Målgrupper og strategier i kommuner og fylkeskommuner
Helgesen, M. K., Hofstad, H., Risan, L. C., Stang, I., Goth, U. S., Rønningen, G. E., & Lorentzen, C. L. (2014) · NIBR-rapport · doi:10.7577/nibr/rapport/2014/3 · Cited by 6 · see citing works
Kvestad, C. A., Holte, I. R., Reitan, S. K., Chiappa, C. S., Helle, G. K., Skjervold, A. E., Rosenlund, A. M. A., Watne, Ø., Brattland, H., Helle, J., et al. (2024) · Trials · doi:10.1186/s13063-024-08028-6 · Cited by 5 · see citing works
“One study points to possible weaknesses in the prevailing Norwegian model, as it found low agreement between the teams when comparing triage decisions based on written referrals [17].”
ResultRights assessment and quality of referrals – rate of agreement between four pain management centres
Stedenfeldt, M., Kvarstein, G., Schjödt, B., Johansen, A., Berntsen, A. G. S., Brautaset, A., & Halsteinli, V. (2019) · Tidsskrift for Den norske legeforening · doi:10.4045/tidsskr.18.0396 · Cited by 2 · see citing works
Vi trenger en ny type henvisninger
Thorsen, O., Hartveit, M., Kristoffersen, J. E., & Holman, P. A. (2017) · Tidsskrift for Den norske legeforening · doi:10.4045/tidsskr.17.0201 · Cited by 1 · see citing works
Self-citationFra henvisningsbrev til elektronisk dialog
Kristoffersen, J. E. (2014)
How area health management leads to improved equity in health: a scoping review
Intarates, M., Dhippayom, T., Upakdee, N., Leesmidt, V., Weraphong, J., & Bangpan, M. (2022) · Journal of Health Research · doi:10.56808/2586-940x.1010
Bedre og enklere tilgang til riktig psykisk helsehjelp
Kvestad, C. A., & Reitan, S. K. (2024) · Tidsskrift for Den norske legeforening · doi:10.4045/tidsskr.24.0384
Kvestad, C. A., Holte, I. R., Reitan, S. K., Chiappa, C. S., Helle, G. K., Skjervold, A. E., Rosenlund, A. M. A., Watne, Ø., Brattland, H., Helle, J., et al. (2024) · Figshare · doi:10.6084/m9.figshare.25389458
- Authors
- Holman, P. A., Ruud, T., & Grepperud, S.
- Affiliations
- Lovisenberg Diakonale Sykehus (NO)
- University of Oslo (NO)
- Corresponding author
- Grepperud, Sverre
- Venue
- BMC Health Services Research
- Published
- 2012-06-15
- Volume
- 12
- Issue
- 1
- Pages
- 162-162
- ISSN
- 1472-6963 (ISSN-L)
- PMC
- PMC3430583
- OpenAlex
- W2059769999
- Cited by
- 17
- References
- 31
- Citations by year
- 2024: 4 · 2022: 3 · 2021: 1 · 2020: 1 · 2019: 1 · 2017: 1 · 2016: 1 · 2014: 3 · 2013: 2
- FWCI
- 1.81
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- BMC Health Services Research (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- BIBSYS Brage (BIBSYS (Norway)) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- NORA - Norwegian Open Research Archives (submittedVersion, other-oa)
- NORA - Norwegian Open Research Archives (submittedVersion, cc-by)
- Topics
- Social Sciences › Psychology › Clinical Psychology › Psychiatric care and mental health services
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › Psychology › Social Psychology › Mental Health Treatment and Access
- Keywords
- horizontal equity, mental health services, priority setting, inter-rater reliability, team consensus, generalizability analysis, community mental health centers, outpatient clinics, case vignettes
- MeSH terms
- Workforce, Administrative Personnel, Ambulatory Care, Mental Disorders, Catchment Area, Health, Community Mental Health Centers, Cost-Benefit Analysis, Diagnosis-Related Groups, Humans, Health Knowledge, Attitudes, Practice, Mental Health Services, Norway, Outcome and Process Assessment, Health Care, Patient Admission, Patient Care Team, Referral and Consultation, Severity of Illness Index, Health Care Rationing, Comorbidity, Treatment Refusal, Needs Assessment, Qualitative Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
