CoChat

Visualizing Uncertainty to Promote Clinicians’ Understanding of Measurement Error

Open access

The DOI matches its Crossref record; 47 of the 47 references it cites were found in a registry or an index; 10 of the 40 sources its claims cite were checked for support.

Abstract

Measurement error is an inherent part of any test score. This uncertainty is generally communicated in ways that can be difficult to understand for clinical practitioners. In this empirical study, we evaluate the impact of several communication formats on the interpretation of measurement accuracy and its influence on the decision-making process in clinical practice. We provided 230 clinical practitioners with score reports in five formats: textual, error bar, violin plot, diamond plot, and quantile dot plot. We found that quantile dot plots significantly increased accuracy in the assessment of measurement uncertainty compared with other formats. However, a direct relation between visualization format and decision quality could not be found. Although traditional confidence intervals and error bars were favored by many participants due to their familiarity, responses revealed several misconceptions that make the suitability of these formats for communicating uncertainty questionable. Our results indicate that new visualization formats can successfully reduce errors in interpretation.

Overview

CoChat Summary

TL;DR

This empirical study compared five formats for communicating measurement uncertainty to clinical practitioners and found that quantile dot plots improved probability-assessment accuracy relative to text and other formats. However, visualization format was not directly related to decision quality, and familiar formats such as text and error bars were often preferred despite misconceptions.

Key findings

  • Among 230 clinical practitioners, quantile dot plots produced the smallest median overestimation of the probability above cutoff (Mdn = .01, MAD = .08), compared with error bars/violin plots (Mdn = .07, MAD = .09) and diamond plots/text (Mdn = .08, MAD = .12).
  • In multilevel beta regression, quantile dot plots significantly improved absolute probability-assessment accuracy versus text (OR = 0.65, p < .001), as did diamond plots (OR = 0.84, p = .043); quantile dot plots were 1.4 times more accurate than violin plots (p < .001) and 1.3 times more accurate than diamond plots (p = .004) and error bars (p = .002).
  • Decision quality was high overall (72.2% correct referrals) and moderately correlated with absolute inaccuracy (r = –.27), but there was no relation between visualization format and decision quality.
  • Familiar formats were rated most understandable: 61% rated text or error bars as most understandable, while the quantile dot plot divided participants (20% ranked it most understandable; 26% ranked it least understandable).

Methods

  • Cross-sectional repeated measures design with 230 Dutch and Norwegian mental health practitioners who viewed five score-report formats: textual, error bar, violin plot, diamond plot, and quantile dot plot.
  • Participants made probability assessments for a score exceeding a cutoff of 60, chose referral to Department PD or Department G under one of three waiting-time scenarios, rated decision confidence, and ranked formats by understandability.
  • Analyses included mixed-effect beta regression for absolute probability-assessment inaccuracy, generalized multilevel modeling for decision quality, correlations for confidence, and qualitative content analysis of open-ended statements.

Limitations

  • The authors state that selective sampling may have affected representativeness: the Dutch sample overrepresented younger participants and remedial educationalists, while the Norwegian sample included many psychologists with research experience and/or a PhD; it is also unknown whether all participants currently worked in mental health contexts.
  • The authors state that viewing conditions were not standardized, with screen sizes ranging from 6.8-inch to 38.8-inch monitors, though screen size showed no significant relation with inaccuracy or decision quality.
  • Evident from the text: 30 participants did not view all formats, 22 viewed only one format, and dropout occurred more often after the textual format or error bar; understandability rankings were available for only 194 participants.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1177/10731911221147042

Cites

47 of 47 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 Five Formats Used in This Study: Text, Error Bar, Violin Plot, Diamond Plot, Quantile Dot Plot.
Figure 1 Five Formats Used in This Study: Text, Error Bar, Violin Plot, Diamond Plot, Quantile Dot Plot.CC BY · PubMed Central
Figure 2. Inaccuracy of Probability Assessments Iij=P^ij−Pi by Format. Note. The horizontal line indicates a perfectly accurate probability assessment by the participant (i.e., P^ij=Pi).
Figure 2. Inaccuracy of Probability Assessments Iij=P^ij−Pi by Format. Note. The horizontal line indicates a perfectly accurate probability assessment by the participant (i.e., P^ij=Pi).CC BY · PubMed Central
Figure 3. Difference in Expected Waiting Time for Decisions (Y-Axis) Plotted Against Different Formats. Note. Y-values lower than 0 indicate an incorrect decision.
Figure 3. Difference in Expected Waiting Time for Decisions (Y-Axis) Plotted Against Different Formats. Note. Y-values lower than 0 indicate an incorrect decision.CC BY · PubMed Central

Verification

Claims the paper makes

10 Valid · 0 Not valid · 18 Not checked · 95 Own results and interpretations

50 more claims

Create a free account to see all 123 claims and to verify the 11 references still open.

See all claims — free account
  • Probably not a substantial claim. 37
  • It may state two separate claims as one. 46

Create a free account to see these claims.

Statistics recomputed

Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.

  • ConsistentF(4, 188) = 8reported p < .001 · recomputed p < .001

    “F(4, 188) = 8.0, p < .001”

1 of 1

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

1 of 1

Citations

Cites

Show all 47 references

Research works printed but not linked: 1

  1. Ben-Porath Y. S., 2008, MMPI-2-RF: Manual for administration, scoring and interpretation

Cited by

See all 5

Metadata

Authors
Frans, N., Hummelen, B., Albers, C. J., & Paap, M. C. S.
Affiliations
  • Oslo University Hospital (NO)
  • University of Groningen (NL)
Corresponding author
Frans, Niek
Venue
Assessment
Published
2023-02-01
Volume
30
Issue
8
Pages
2449-2460
ISSN
  • 1073-1911 (ISSN-L)
  • 1552-3489
OpenAlex
W4318931757
Cited by
5
References
47
Citations by year
2026: 1 · 2025: 3 · 2024: 1
FWCI
0.45
Type
journal article
Language
en
Open access
Locations
  • Assessment (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • University of Groningen research database (University of Groningen / Centre for Information Technology) (submittedVersion, cc-by)
Funders
Norges Forskningsråd (NFR 286893)
Topics
  • Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › Data Visualization and Analytics
  • Health Sciences › Medicine › Family Practice › Clinical Reasoning and Diagnostic Skills
  • Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
Keywords
measurement error, uncertainty visualization, score reports, violin plots, confidence intervals, clinical decision-making, test score interpretation, uncertainty communication
MeSH terms
Communication, Humans, Uncertainty

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol and qwen3.8-omni-flash).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Visualizing Uncertainty to Promote Clinicians’ Understanding of Measurement Error. https://app.cochat.ai/papers/10.1177/10731911221147042
The paper
Frans, N., Hummelen, B., Albers, C. J., & Paap, M. C. S. (2023). Visualizing Uncertainty to Promote Clinicians’ Understanding of Measurement Error. Assessment. https://doi.org/10.1177/10731911221147042

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.