Visualizing Uncertainty to Promote Clinicians’ Understanding of Measurement Error
Open accessThe DOI matches its Crossref record; 47 of the 47 references it cites were found in a registry or an index; 10 of the 40 sources its claims cite were checked for support.
Abstract
Measurement error is an inherent part of any test score. This uncertainty is generally communicated in ways that can be difficult to understand for clinical practitioners. In this empirical study, we evaluate the impact of several communication formats on the interpretation of measurement accuracy and its influence on the decision-making process in clinical practice. We provided 230 clinical practitioners with score reports in five formats: textual, error bar, violin plot, diamond plot, and quantile dot plot. We found that quantile dot plots significantly increased accuracy in the assessment of measurement uncertainty compared with other formats. However, a direct relation between visualization format and decision quality could not be found. Although traditional confidence intervals and error bars were favored by many participants due to their familiarity, responses revealed several misconceptions that make the suitability of these formats for communicating uncertainty questionable. Our results indicate that new visualization formats can successfully reduce errors in interpretation.
Overview
CoChat Summary
TL;DR
This empirical study compared five formats for communicating measurement uncertainty to clinical practitioners and found that quantile dot plots improved probability-assessment accuracy relative to text and other formats. However, visualization format was not directly related to decision quality, and familiar formats such as text and error bars were often preferred despite misconceptions.
Key findings
- Among 230 clinical practitioners, quantile dot plots produced the smallest median overestimation of the probability above cutoff (Mdn = .01, MAD = .08), compared with error bars/violin plots (Mdn = .07, MAD = .09) and diamond plots/text (Mdn = .08, MAD = .12).
- In multilevel beta regression, quantile dot plots significantly improved absolute probability-assessment accuracy versus text (OR = 0.65, p < .001), as did diamond plots (OR = 0.84, p = .043); quantile dot plots were 1.4 times more accurate than violin plots (p < .001) and 1.3 times more accurate than diamond plots (p = .004) and error bars (p = .002).
- Decision quality was high overall (72.2% correct referrals) and moderately correlated with absolute inaccuracy (r = –.27), but there was no relation between visualization format and decision quality.
- Familiar formats were rated most understandable: 61% rated text or error bars as most understandable, while the quantile dot plot divided participants (20% ranked it most understandable; 26% ranked it least understandable).
Methods
- Cross-sectional repeated measures design with 230 Dutch and Norwegian mental health practitioners who viewed five score-report formats: textual, error bar, violin plot, diamond plot, and quantile dot plot.
- Participants made probability assessments for a score exceeding a cutoff of 60, chose referral to Department PD or Department G under one of three waiting-time scenarios, rated decision confidence, and ranked formats by understandability.
- Analyses included mixed-effect beta regression for absolute probability-assessment inaccuracy, generalized multilevel modeling for decision quality, correlations for confidence, and qualitative content analysis of open-ended statements.
Limitations
- The authors state that selective sampling may have affected representativeness: the Dutch sample overrepresented younger participants and remedial educationalists, while the Norwegian sample included many psychologists with research experience and/or a PhD; it is also unknown whether all participants currently worked in mental health contexts.
- The authors state that viewing conditions were not standardized, with screen sizes ranging from 6.8-inch to 38.8-inch monitors, though screen size showed no significant relation with inaccuracy or decision quality.
- Evident from the text: 30 participants did not view all formats, 22 viewed only one format, and dropout occurred more often after the textual format or error bar; understandability rankings were available for only 194 participants.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures



Verification
Claims the paper makes
10 Valid · 0 Not valid · 18 Not checked · 95 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Contrary to Kale et al., our findings did show a significant relation between inaccuracy and decision making.”
§ Visualizing Uncertainty to Promote Clinicians’ Understanding of Measurement Error > Discussion
What the sources say
- Partly supportedcites Kale et al. (2021) sourceExperimental — accuracy not yet measured
Section 4.4 reports that many participants who performed poorly at magnitude estimation did well on the decision task, supporting the contrast the citing sentence draws. But it does not establish that there was no statistically significant relation between inaccuracy and decision-making; the same section reports that greater sensitivity to effect size predicted more utility-optimal decisions. No page number is available in our copy. (LLM, conf=0.82)
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“While studies generally demonstrate a positive influence of visualizations on the user’s interpretation of uncertainty, the literature on uncertainty communication so far shows that what “works” is highly dependent on (a) the type of uncertainty being depicted; (b) the type of judgment users need to make; and (c) characteristics of the user (Hullman et al., 2015; Levontin et al., 2020).”
§ Visualizing Uncertainty to Promote Clinicians’ Understanding of Measurement Error > Abstract
What the sources say
- Partly supportedcites Hullman et al., 2015 sourceExperimental — accuracy not yet measured
The Discussion reports that HOPs performed differently across judgment tasks and distribution conditions, supporting the claim that effectiveness depends on what users must judge and on features of the uncertainty depicted. But the Limitations section says the authors do not know whether particular user subpopulations perform better or worse with different representations; this paper does not establish the claim about user characteristics. Page numbers were not available in our copy. (LLM, conf=0.87)
- Not matchedcites Levontin et al., 2020
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
boostedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Considering how characteristics such as education, graph reading ability (i.e., graph literacy), and statistical training have been shown to impact uncertainty understanding (Hopster-den Otter et al., 2019; Shah & Hoeffner, 2002; van der Bles et al., 2019; Zapata-Rivera et al., 2016), it is important to examine how to best communicate measurement accuracy in a population of clinical practitioners.”
§ Visualizing Uncertainty to Promote Clinicians’ Understanding of Measurement Error > Abstract
What the sources say
- Not matchedcites Hopster-den Otter et al., 2019
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not checked — no full text of the sourcecites Shah & Hoeffner, 2002 source
- Partly supportedcites van der Bles et al., 2019 sourceExperimental — accuracy not yet measured
In “The effect of communicating uncertainty on trust and credibility” (page unavailable), the paper reports that confidence intervals confused women in less-educated groups, while women in higher-educated groups were more accepting of scientific uncertainty. Its discussion of graphs concerns how presentation formats affect interpretation, not whether graph-reading ability or statistical training affects understanding. Those two characteristics are not supported by the passages found. (LLM, conf=0.84)
boostedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“Several studies suggest that there is ample room for improvement regarding the understanding of abstract statistical concepts by medical students and medical professionals (Hoffrage et al., 2000; Rutledge et al., 2004).”
§ Visualizing Uncertainty to Promote Clinicians’ Understanding of Measurement Error > Abstract
What the sources say
- Supportedcites Hoffrage et al., 2000 source
The 'Natural Frequencies' section of our copy reports that 'Only 1 out of 24 physicians gave the correct answer when the statistical information was expressed in probabilities,' and the 'Applications in Medicine' section documents the same deficit in trainees (96 advanced medical students inferring far less accurately with probabilities than with natural frequencies) and in HIV counselors, 15 of 20 of whom told a low-risk client that a positive test meant 99.9–100% certainty. The paper explicitly proposes improving 'both communication of statistical information and medical education,' which is the 'ample room for improvement' the citing sentence attributes to it. The only qualifier the source adds—that the deficit is representation-dependent and largely disappears under natural frequencies—is not contradicted by the hedged citing sentence ('several studies suggest'). (LLM, conf=0.85)
- Not checked — no full text of the sourcecites Rutledge et al., 2004 source
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“After accounting for authorship, none of the other characteristics (i.e., experience with standardized tests, having a PhD, subjective graph literacy, age, gender, profession, or country) were significantly related to difference in participants’ probability assessment inaccuracy.”
§ Visualizing Uncertainty to Promote Clinicians’ Understanding of Measurement Error > Results > Inaccuracy of Participants’ Probability Assessments
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on 3 assumptions not shown here.
50 more claims
Create a free account to see all 123 claims and to verify the 11 references still open.
- Probably not a substantial claim. 37
- It may state two separate claims as one. 46
Statistics recomputed
Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.
- ConsistentF(4, 188) = 8reported p < .001 · recomputed p < .001
“F(4, 188) = 8.0, p < .001”
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“Table 1 shows the expected waiting time for both referral decisions in the 18 score and cost scenarios formed by the nine score and uncertainty combinations and the three different waiting times.”
§ Visualizing Uncertainty to Promote Clinicians’ Understanding of Measurement Error > Method > Procedure
Reported 18 · recomputed 27
Citations
Cites
R: A Language and Environment for Statistical Computing
R Core Team (2025) · doi:10.32614/r.manuals · Cited by 352,181
null (2016) · Philosophy Study · doi:10.17265/2159-5313/2016.09.003 · Cited by 28,317
Brooks, M., Kristensen, K., Benthem, K., Magnusson, A., Berg, C., Nielsen, A., Skaug, H., Mächler, M., & Bolker, B. (2017) · The R Journal · doi:10.32614/rj-2017-066 · Cited by 12,915 · cited 2×
“After a descriptive analysis of the sample characteristics, inaccuracy of probability assessments, and decision quality, mixed-effect beta regression models (Brooks et al., 2017; Ferrari & Cribari-Neto, 2004) were used to predict absolute assessment inaccuracy across different formats.”
MethodSufficient Sample Sizes for Multilevel Modeling
Maas, C. J. M., & Hox, J. J. (2005) · Methodology · doi:10.1027/1614-2241.1.3.86 · Cited by 3,582 · cited 1×
“Based on a simulation study on sufficient sample sizes for multilevel modeling by Maas and Hox (2005), the target sample size was set at 100 participants from both countries.”
MethodIntegrative Guide for the 1991 CBCL/4-18, Ysr, and Trf Profiles
Achenbach, T. M. (1991) · Medical Entomology and Zoology · Cited by 3,122
Beta regression for modelling rates and proportions
Ferrari, S., & Cribari-Neto, F. (2004) · Journal of Applied Statistics · doi:10.1080/0266476042000214501 · Cited by 3,055 · cited 3×
“After a descriptive analysis of the sample characteristics, inaccuracy of probability assessments, and decision quality, mixed-effect beta regression models (Brooks et al., 2017; Ferrari & Cribari-Neto, 2004) were used to predict absolute assessment inaccuracy across different formats.”
MethodWechsler Adult Intelligence Scale
(2007) · Encyclopedia of Measurement and Statistics · doi:10.4135/9781412952644.n478 · Cited by 2,008
A better lemon squeezer? maximum-likelihood regression with beta-distributed dependent variables.
Smithson, M., & Verkuilen, J. (2006) · Psychological Methods · doi:10.1037/1082-989x.11.1.54 · Cited by 1,788 · cited 1×
“Since these models assume an outcome that is larger than 0 and smaller than 1, absolute inaccuracy was compressed slightly using a transformation by Smithson and Verkuilen (2006): , where N indicates the sample size.”
MethodA Theory and Procedure of Scale Analysis: With Applications in Political Research
Mokken, R. J. (1971) · doi:10.1515/9783110813203 · Cited by 1,120
Psychological Testing: History, Principles, and Applications
Gregory, R. J. (1991) · Medical Entomology and Zoology · Cited by 941
Research works printed but not linked: 1
- Ben-Porath Y. S., 2008, MMPI-2-RF: Manual for administration, scoring and interpretation
Cited by
Chen, W., Wang, H., Ten, X.-L., Fu, M., Lin, M., Xu, X., & Ma, Y. (2025) · BMC Infectious Diseases · doi:10.1186/s12879-025-11279-6 · Cited by 8
“…knowledge for model selection in policy evaluation [30]: the intuitive parameters of beta regression models are advantageous when focusing on short-term immediate effects [31, 32]; while the flexibility of GAM models may be more appropriate when exploring long-term complex dynamic trends [33].”
Zimmermann, J., Witthöft, M., Cicero, D. C., Forbes, M. K., Hopwood, C. J., Kotov, R., Krueger, R. F., Sellbom, M., Simms, L. J., Wright, A. G. C., et al. (2024) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/kp8yx · Cited by 4
Evaluating visual feedback of experience sampling data for mental health practitioners
Piot, M., Dejonckheere, E., Tuinstra, A., Tijs, I., Kuppens, P., & Verdonck, S. (2025) · Internet Interventions · doi:10.1016/j.invent.2025.100873 · Cited by 1
“Previous research has compared various methods for visualizing uncertainty in graphs, such as gradient plots, error bars or violin plots (Correll and Gleicher, 2014; Frans et al., 2023).”
Marchiori, M., Heidler, J., Segantin, G., Eckmann, H. R., Chinapaw, M. J. M., Kjærgaard, M., & Schipperijn, J. (2025) · Journal of Activity Sedentary and Sleep Behaviors · doi:10.1186/s44167-025-00088-6 · Cited by 1
Advancing the vision of “reliability metadata”: From conceptual refinement to clinical validation
Yu, Z., & Cheng, W. (2026) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2026.109267
Metadata
- Authors
- Frans, N., Hummelen, B., Albers, C. J., & Paap, M. C. S.
- Affiliations
- Oslo University Hospital (NO)
- University of Groningen (NL)
- Corresponding author
- Frans, Niek
- Venue
- Assessment
- Publisher
- SAGE Publications
- Published
- 2023-02-01
- Volume
- 30
- Issue
- 8
- Pages
- 2449-2460
- ISSN
- 1073-1911 (ISSN-L)
- 1552-3489
- PMC
- PMC10623599
- OpenAlex
- W4318931757
- Cited by
- 5
- References
- 47
- Citations by year
- 2026: 1 · 2025: 3 · 2024: 1
- FWCI
- 0.45
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- Assessment (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- University of Groningen research database (University of Groningen / Centre for Information Technology) (submittedVersion, cc-by)
- Funders
- Norges Forskningsråd (NFR 286893)
- Topics
- Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › Data Visualization and Analytics
- Health Sciences › Medicine › Family Practice › Clinical Reasoning and Diagnostic Skills
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Keywords
- measurement error, uncertainty visualization, score reports, violin plots, confidence intervals, clinical decision-making, test score interpretation, uncertainty communication
- MeSH terms
- Communication, Humans, Uncertainty
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol and qwen3.8-omni-flash).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Visualizing Uncertainty to Promote Clinicians’ Understanding of Measurement Error. https://app.cochat.ai/papers/10.1177/10731911221147042
- The paper
- Frans, N., Hummelen, B., Albers, C. J., & Paap, M. C. S. (2023). Visualizing Uncertainty to Promote Clinicians’ Understanding of Measurement Error. Assessment. https://doi.org/10.1177/10731911221147042
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.