Visualizing Uncertainty to Promote Clinicians’ Understanding of Measurement Error
Open accessThe DOI matches its Crossref record; 47 of the 47 references it cites were found in a registry or an index; 10 of the 40 sources its claims cite were checked for support.
Abstract
Measurement error is an inherent part of any test score. This uncertainty is generally communicated in ways that can be difficult to understand for clinical practitioners. In this empirical study, we evaluate the impact of several communication formats on the interpretation of measurement accuracy and its influence on the decision-making process in clinical practice. We provided 230 clinical practitioners with score reports in five formats: textual, error bar, violin plot, diamond plot, and quantile dot plot. We found that quantile dot plots significantly increased accuracy in the assessment of measurement uncertainty compared with other formats. However, a direct relation between visualization format and decision quality could not be found. Although traditional confidence intervals and error bars were favored by many participants due to their familiarity, responses revealed several misconceptions that make the suitability of these formats for communicating uncertainty questionable. Our results indicate that new visualization formats can successfully reduce errors in interpretation.
Overview
CoChat Summary
TL;DR
This empirical study compared five formats for communicating measurement uncertainty to clinical practitioners and found that quantile dot plots improved probability-assessment accuracy relative to text and other formats. However, visualization format was not directly related to decision quality, and familiar formats such as text and error bars were often preferred despite misconceptions.
Key findings
- Among 230 clinical practitioners, quantile dot plots produced the smallest median overestimation of the probability above cutoff (Mdn = .01, MAD = .08), compared with error bars/violin plots (Mdn = .07, MAD = .09) and diamond plots/text (Mdn = .08, MAD = .12).
- In multilevel beta regression, quantile dot plots significantly improved absolute probability-assessment accuracy versus text (OR = 0.65, p < .001), as did diamond plots (OR = 0.84, p = .043); quantile dot plots were 1.4 times more accurate than violin plots (p < .001) and 1.3 times more accurate than diamond plots (p = .004) and error bars (p = .002).
- Decision quality was high overall (72.2% correct referrals) and moderately correlated with absolute inaccuracy (r = –.27), but there was no relation between visualization format and decision quality.
- Familiar formats were rated most understandable: 61% rated text or error bars as most understandable, while the quantile dot plot divided participants (20% ranked it most understandable; 26% ranked it least understandable).
Methods
- Cross-sectional repeated measures design with 230 Dutch and Norwegian mental health practitioners who viewed five score-report formats: textual, error bar, violin plot, diamond plot, and quantile dot plot.
- Participants made probability assessments for a score exceeding a cutoff of 60, chose referral to Department PD or Department G under one of three waiting-time scenarios, rated decision confidence, and ranked formats by understandability.
- Analyses included mixed-effect beta regression for absolute probability-assessment inaccuracy, generalized multilevel modeling for decision quality, correlations for confidence, and qualitative content analysis of open-ended statements.
Limitations
- The authors state that selective sampling may have affected representativeness: the Dutch sample overrepresented younger participants and remedial educationalists, while the Norwegian sample included many psychologists with research experience and/or a PhD; it is also unknown whether all participants currently worked in mental health contexts.
- The authors state that viewing conditions were not standardized, with screen sizes ranging from 6.8-inch to 38.8-inch monitors, though screen size showed no significant relation with inaccuracy or decision quality.
- Evident from the text: 30 participants did not view all formats, 22 viewed only one format, and dropout occurred more often after the textual format or error bar; understandability rankings were available for only 194 participants.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures



Verification
Claims the paper makes
10 Valid · 0 Not valid · 18 Not checked · 95 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Contrary to Kale et al., our findings did show a significant relation between inaccuracy and decision making.”
§ Visualizing Uncertainty to Promote Clinicians’ Understanding of Measurement Error > Discussion
What the sources say
- Partly supportedcites Kale et al. (2021) matched to “Visual Reasoning Strategies for Effect Size Judgments and Decisions” (2021) sourceExperimental — accuracy not yet measured
Section 4.4 reports that many participants who performed poorly at magnitude estimation did well on the decision task, supporting the contrast the citing sentence draws. But it does not establish that there was no statistically significant relation between inaccuracy and decision-making; the same section reports that greater sensitivity to effect size predicted more utility-optimal decisions. No page number is available in our copy. (LLM, conf=0.82)
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“While studies generally demonstrate a positive influence of visualizations on the user’s interpretation of uncertainty, the literature on uncertainty communication so far shows that what “works” is highly dependent on (a) the type of uncertainty being depicted; (b) the type of judgment users need to make; and (c) characteristics of the user (Hullman et al., 2015; Levontin et al., 2020).”
§ Visualizing Uncertainty to Promote Clinicians’ Understanding of Measurement Error > Abstract
What the sources say
- Partly supportedcites Hullman et al., 2015 matched to “Hypothetical Outcome Plots Outperform Error Bars and Violin Plots for Inferences about Reliability of Variable Ordering” (2015) sourceExperimental — accuracy not yet measured
The Discussion reports that HOPs performed differently across judgment tasks and distribution conditions, supporting the claim that effectiveness depends on what users must judge and on features of the uncertainty depicted. But the Limitations section says the authors do not know whether particular user subpopulations perform better or worse with different representations; this paper does not establish the claim about user characteristics. Page numbers were not available in our copy. (LLM, conf=0.87)
- Not matchedcites Levontin et al., 2020
An author–year citation the paper’s reference list doesn’t settle. We don’t search these by name.
boostedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Considering how characteristics such as education, graph reading ability (i.e., graph literacy), and statistical training have been shown to impact uncertainty understanding (Hopster-den Otter et al., 2019; Shah & Hoeffner, 2002; van der Bles et al., 2019; Zapata-Rivera et al., 2016), it is important to examine how to best communicate measurement accuracy in a population of clinical practitioners.”
§ Visualizing Uncertainty to Promote Clinicians’ Understanding of Measurement Error > Abstract
What the sources say
- Not matchedcites Hopster-den Otter et al., 2019
An author–year citation the paper’s reference list doesn’t settle. We don’t search these by name.
- Not checked — no full text of the sourcecites Shah & Hoeffner, 2002 matched to “Review of Graph Comprehension Research: Implications for Instruction” (2002) source
- Partly supportedcites van der Bles et al., 2019 matched to “Communicating uncertainty about facts, numbers and science” (2019) sourceExperimental — accuracy not yet measured
In “The effect of communicating uncertainty on trust and credibility” (page unavailable), the paper reports that confidence intervals confused women in less-educated groups, while women in higher-educated groups were more accepting of scientific uncertainty. Its discussion of graphs concerns how presentation formats affect interpretation, not whether graph-reading ability or statistical training affects understanding. Those two characteristics are not supported by the passages found. (LLM, conf=0.84)
boostedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“Several studies suggest that there is ample room for improvement regarding the understanding of abstract statistical concepts by medical students and medical professionals (Hoffrage et al., 2000; Rutledge et al., 2004).”
§ Visualizing Uncertainty to Promote Clinicians’ Understanding of Measurement Error > Abstract
What the sources say
- Supportedcites Hoffrage et al., 2000 matched to “Communicating Statistical Information” (2000) source
The 'Natural Frequencies' section of our copy reports that 'Only 1 out of 24 physicians gave the correct answer when the statistical information was expressed in probabilities,' and the 'Applications in Medicine' section documents the same deficit in trainees (96 advanced medical students inferring far less accurately with probabilities than with natural frequencies) and in HIV counselors, 15 of 20 of whom told a low-risk client that a positive test meant 99.9–100% certainty. The paper explicitly proposes improving 'both communication of statistical information and medical education,' which is the 'ample room for improvement' the citing sentence attributes to it. The only qualifier the source adds—that the deficit is representation-dependent and largely disappears under natural frequencies—is not contradicted by the hedged citing sentence ('several studies suggest'). (LLM, conf=0.85)
- Not checked — no full text of the sourcecites Rutledge et al., 2004 matched to “Effect sizes and statistical testing in the determination of clinical significance in behavioral medicine research” (2004) source
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“After accounting for authorship, none of the other characteristics (i.e., experience with standardized tests, having a PhD, subjective graph literacy, age, gender, profession, or country) were significantly related to difference in participants’ probability assessment inaccuracy.”
§ Visualizing Uncertainty to Promote Clinicians’ Understanding of Measurement Error > Results > Inaccuracy of Participants’ Probability Assessments
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on 3 assumptions not shown here.
50 more claims
Create a free account to see all 123 claims and to verify the 11 references still open.
- Probably not a substantial claim. 37
- It may state two separate claims as one. 46
Statistics recomputed
Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.
- ConsistentF(4, 188) = 8reported p < .001 · recomputed p < .001
“F(4, 188) = 8.0, p < .001”
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“Table 1 shows the expected waiting time for both referral decisions in the 18 score and cost scenarios formed by the nine score and uncertainty combinations and the three different waiting times.”
§ Visualizing Uncertainty to Promote Clinicians’ Understanding of Measurement Error > Method > Procedure
Reported 18 · recomputed 27
Citations
Cites
R: A Language and Environment for Statistical Computing
R Core Team (2025) · doi:10.32614/r.manuals · Cited by 352,181
null (2016) · Philosophy Study · doi:10.17265/2159-5313/2016.09.003 · Cited by 28,317
Brooks, M., Kristensen, K., Benthem, K., Magnusson, A., Berg, C., Nielsen, A., Skaug, H., Mächler, M., & Bolker, B. (2017) · The R Journal · doi:10.32614/rj-2017-066 · Cited by 12,915 · cited 2×
“After a descriptive analysis of the sample characteristics, inaccuracy of probability assessments, and decision quality, mixed-effect beta regression models (Brooks et al., 2017; Ferrari & Cribari-Neto, 2004) were used to predict absolute assessment inaccuracy across different formats.”
MethodSufficient Sample Sizes for Multilevel Modeling
Maas, C. J. M., & Hox, J. J. (2005) · Methodology · doi:10.1027/1614-2241.1.3.86 · Cited by 3,582 · cited 1×
“Based on a simulation study on sufficient sample sizes for multilevel modeling by Maas and Hox (2005), the target sample size was set at 100 participants from both countries.”
MethodIntegrative Guide for the 1991 CBCL/4-18, Ysr, and Trf Profiles
Achenbach, T. M. (1991) · Medical Entomology and Zoology · Cited by 3,198
Beta regression for modelling rates and proportions
Ferrari, S., & Cribari-Neto, F. (2004) · Journal of Applied Statistics · doi:10.1080/0266476042000214501 · Cited by 3,055 · cited 3×
“After a descriptive analysis of the sample characteristics, inaccuracy of probability assessments, and decision quality, mixed-effect beta regression models (Brooks et al., 2017; Ferrari & Cribari-Neto, 2004) were used to predict absolute assessment inaccuracy across different formats.”
MethodWechsler Adult Intelligence Scale
(2007) · Encyclopedia of Measurement and Statistics · doi:10.4135/9781412952644.n478 · Cited by 2,008
A better lemon squeezer? maximum-likelihood regression with beta-distributed dependent variables.
Smithson, M., & Verkuilen, J. (2006) · Psychological Methods · doi:10.1037/1082-989x.11.1.54 · Cited by 1,788 · cited 1×
“Since these models assume an outcome that is larger than 0 and smaller than 1, absolute inaccuracy was compressed slightly using a transformation by Smithson and Verkuilen (2006): , where N indicates the sample size.”
MethodA Theory and Procedure of Scale Analysis: With Applications in Political Research
Mokken, R. J. (1971) · doi:10.1515/9783110813203 · Cited by 1,120
Psychological Testing: History, Principles, and Applications
Gregory, R. J. (1991) · Medical Entomology and Zoology · Cited by 941
Forman, J., & Damschroder, L. (2007) · Advances in Bioethics · doi:10.1016/s1479-3709(07)11003-7 · Cited by 808
Wechsler Intelligence Scale for Children
National Cancer Institute (2020) · Definitions · doi:10.32388/1lsxek · Cited by 693
Classification, assessment, prevalence, and effect of personality disorder
Tyrer, P., Reed, G. M., & Crawford, M. J. (2015) · The Lancet · doi:10.1016/s0140-6736(14)61995-4 · Cited by 656 · cited 1×
“We chose to focus on PD due to its prevalence in clinical practice and the wide range of available standardized instruments (Tyrer et al., 2015).”
BackgroundCommunicating Statistical Information
Hoffrage, U., Lindsey, S., Hertwig, R., & Gigerenzer, G. (2000) · Science · doi:10.1126/science.290.5500.2261 · Cited by 651 · cited 1×
“Several studies suggest that there is ample room for improvement regarding the understanding of abstract statistical concepts by medical students and medical professionals (Hoffrage et al., 2000; Rutledge et al., 2004).”
BackgroundCommunicating uncertainty about facts, numbers and science
van der Bles, A. M., van der Linden, S., Freeman, A. L. J., Mitchell, J., Galvao, A. B., Zaval, L., & Spiegelhalter, D. J. (2019) · Royal Society Open Science · doi:10.1098/rsos.181870 · Cited by 456 · cited 1×
“…literacy), and statistical training have been shown to impact uncertainty understanding (Hopster-den Otter et al., 2019; Shah & Hoeffner, 2002; van der Bles et al., 2019; Zapata-Rivera et al., 2016), it is important to examine how to best communicate measurement accuracy in a population of…”
BackgroundReview of Graph Comprehension Research: Implications for Instruction
Shah, P., & Hoeffner, J. (2002) · Educational Psychology Review · doi:10.1023/a:1013180410169 · Cited by 431 · cited 2×
“Extensive research experience may influence participants’ familiarity with standard errors, confidence intervals, and certain visualizations, which in turn may influence their performance (Shah & Hoeffner, 2002).”
BackgroundResearchers Misunderstand Confidence Intervals and Standard Error Bars.
Belia, S., Fidler, F., Williams, J., & Cumming, G. (2005) · Psychological Methods · doi:10.1037/1082-989x.10.4.389 · Cited by 390 · cited 1×
“Several studies show that such abstract metrics can be difficult to interpret (Charter & Feldt, 2002; Hildon et al., 2012; McManus, 2012; Plebani et al., 2018; Simpkin & Armstrong, 2019), even for people who have received rigorous training (Belia et al., 2005; Kalinowski et al., 2018).”
BackgroundGarcia-Retamero, R., & Cokely, E. T. (2017) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1177/0018720817690634 · Cited by 298 · cited 1×
“For example, a systematic review by Garcia-Retamero and Cokely (2017) on risk communication in health care showed that the use of visual aids is strikingly beneficial for a diverse audience of test users, including patients, physicians, and highly educated individuals.”
BackgroundError Bars Considered Harmful: Exploring Alternate Encodings for Mean and Error
Correll, M., & Gleicher, M. (2014) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2014.2346298 · Cited by 258 · cited 1×
“…this respect, visualizations that place less emphasis on confidence range limits might better represent the continuous nature of measurement uncertainty, and may help reduce categorical reasoning about uncertainty in users, while increasing accuracy (Correll & Gleicher, 2014; Helske et al., 2021).”
BackgroundCharter, R. A. (2003) · The Journal of General Psychology · doi:10.1080/00221300309601160 · Cited by 254 · cited 1×
“…sound standardized tests provide an estimate of the construct we are trying to measure, this estimate has a degree of uncertainty which differs widely across instruments and possibly populations, which needs to be taken into consideration to make an informed decision (Charter, 2003).”
BackgroundKay, M., Kola, T., Hullman, J. R., & Munson, S. A. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858558 · Cited by 239 · cited 1×
“Dividing the surface area into smaller meaningful areas or small countable quantities may aid the user’s interpretation (e.g., Kay et al., 2016).”
BackgroundJensen-Doss, A., & Hawley, K. M. (2010) · Journal of Clinical Child & Adolescent Psychology · doi:10.1080/15374416.2010.517169 · Cited by 238 · cited 1×
“In addition, there is some evidence that links clinical orientation to attitudes toward use of standardized assessment (Jensen-Doss & Hawley, 2010).”
BackgroundHullman, J., Resnick, P., & Adar, E. (2015) · PLOS ONE · doi:10.1371/journal.pone.0142444 · Cited by 194 · cited 1×
“…of uncertainty, the literature on uncertainty communication so far shows that what “works” is highly dependent on (a) the type of uncertainty being depicted; (b) the type of judgment users need to make; and (c) characteristics of the user (Hullman et al., 2015; Levontin et al., 2020).”
BackgroundCommunicating Uncertainty: a Narrative Review and Framework for Future Research
Simpkin, A. L., & Armstrong, K. A. (2019) · Journal of General Internal Medicine · doi:10.1007/s11606-019-04860-8 · Cited by 181 · cited 2×
“Several studies show that such abstract metrics can be difficult to interpret (Charter & Feldt, 2002; Hildon et al., 2012; McManus, 2012; Plebani et al., 2018; Simpkin & Armstrong, 2019), even for people who have received rigorous training (Belia et al., 2005; Kalinowski et al., 2018).”
BackgroundPizzas: π or Square? Psychophysical Biases in Area Comparisons
Krider, R. E., Raghubir, P., & Krishna, A. (2001) · Marketing Science · doi:10.1287/mksc.20.4.405.9756 · Cited by 180 · cited 1×
“However, some researchers (e.g., Krider et al., 2001) suggest that estimating the surface area of a figure may be challenging for most people.”
BackgroundRutledge, T., & Loh, C. (2004) · Annals of Behavioral Medicine · doi:10.1207/s15324796abm2702_9 · Cited by 144 · cited 1×
“Several studies suggest that there is ample room for improvement regarding the understanding of abstract statistical concepts by medical students and medical professionals (Hoffrage et al., 2000; Rutledge et al., 2004).”
BackgroundHildon, Z., Allwood, D., & Black, N. (2012) · International Journal for Quality in Health Care · doi:10.1093/intqhc/mzr072 · Cited by 119 · cited 1×
“Several studies show that such abstract metrics can be difficult to interpret (Charter & Feldt, 2002; Hildon et al., 2012; McManus, 2012; Plebani et al., 2018; Simpkin & Armstrong, 2019), even for people who have received rigorous training (Belia et al., 2005; Kalinowski et al., 2018).”
BackgroundVisual Reasoning Strategies for Effect Size Judgments and Decisions
Kale, A., Kay, M., & Hullman, J. (2021) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2020.3030335 · Cited by 109 · cited 1×
“Kale et al. (2021) similarly found that quantile dot plots reduce inaccuracies in participants’ probability assessments relative to error bars.”
BackgroundProceedings of the 2016 CHI Conference on Human Factors in Computing Systems
(2016) · doi:10.1145/2858036 · Cited by 107
Bar graphs depicting averages are perceptually misinterpreted: The within-the-bar bias
Newman, G. E., & Scholl, B. J. (2012) · Psychonomic Bulletin & Review · doi:10.3758/s13423-012-0247-5 · Cited by 101 · cited 1×
“…with a wide range of interpretation errors, including reinforcing categorical reasoning about probabilities (Helske et al., 2021; Levontin et al., 2020; Padilla et al., 2022), and misinterpreting the probability of values within the error bar (Levontin et al., 2020; Newman & Scholl, 2012).”
BackgroundCharter, R. A., & Feldt, L. S. (2001) · Journal of Clinical and Experimental Neuropsychology · doi:10.1076/jcen.23.4.530.1227 · Cited by 64 · cited 2×
“…estimates that express the degree of uncertainty associated with test scores (Charter & Feldt, 2001b; Gregory, 2015), and generally supplement this information with standard errors that can be used to calculate an uncertainty interval around an individual test score (Charter & Feldt, 2001a).”
BackgroundMeasuring Graph Literacy without a Test: A Brief Subjective Assessment
Garcia-Retamero, R., Cokely, E. T., Ghazal, S., & Joeris, A. (2016) · Medical Decision Making · doi:10.1177/0272989x16655334 · Cited by 64 · cited 2×
“The Subjective Graph Literacy scale (SGL; Garcia-Retamero et al., 2016) was administered to measure the participant’s graph literacy skills.”
MethodMeloncon, L., & Warner, E. (2017) · 2017 IEEE International Professional Communication Conference (ProComm) · doi:10.1109/ipcc.2017.8013960 · Cited by 50
Reporting test scores in more meaningful ways: A research-based approach to score report design.
Hambleton, R. K., & Zenisky, A. L. (2013) · American Psychological Association eBooks · doi:10.1037/14049-023 · Cited by 42 · cited 1×
“Consequently, information on measurement uncertainty is often not considered when interpreting test scores (Hambleton & Zenisky, 2013; Plebani et al., 2018), which may create a false sense of certainty and diminish trust in test outcomes, when repeated score estimates do not align (Simpkin &…”
BackgroundConfidence Intervals for True Scores: Is There a Correct Approach?
Charter, R. A., & Feldt, L. S. (2001) · Journal of Psychoeducational Assessment · doi:10.1177/073428290101900404 · Cited by 38 · cited 2×
“…estimates that express the degree of uncertainty associated with test scores (Charter & Feldt, 2001b; Gregory, 2015), and generally supplement this information with standard errors that can be used to calculate an uncertainty interval around an individual test score (Charter & Feldt, 2001a).”
BackgroundHelske, J., Helske, S., Cooper, M., Ynnerman, A., & Besancon, L. (2021) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2021.3073466 · Cited by 38 · cited 2×
“…this respect, visualizations that place less emphasis on confidence range limits might better represent the continuous nature of measurement uncertainty, and may help reduce categorical reasoning about uncertainty in users, while increasing accuracy (Correll & Gleicher, 2014; Helske et al., 2021).”
BackgroundWhat information on measurement uncertainty should be communicated to clinicians, and how?
Plebani, M., Sciacovelli, L., Bernardi, D., Aita, A., Antonelli, G., & Padoan, A. (2018) · Clinical Biochemistry · doi:10.1016/j.clinbiochem.2018.01.017 · Cited by 37 · cited 2×
“Several studies show that such abstract metrics can be difficult to interpret (Charter & Feldt, 2002; Hildon et al., 2012; McManus, 2012; Plebani et al., 2018; Simpkin & Armstrong, 2019), even for people who have received rigorous training (Belia et al., 2005; Kalinowski et al., 2018).”
BackgroundUncertainty VisualizationPreprint
Padilla, L., Kay, M., & Hullman, J. R. (2020) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/ebd6r · Cited by 37 · cited 3×
“…with a wide range of interpretation errors, including reinforcing categorical reasoning about probabilities (Helske et al., 2021; Levontin et al., 2020; Padilla et al., 2022), and misinterpreting the probability of values within the error bar (Levontin et al., 2020; Newman & Scholl, 2012).”
BackgroundMethodMcManus, I. C. (2012) · Medical Teacher · doi:10.3109/0142159x.2012.670318 · Cited by 32 · cited 2×
“Several studies show that such abstract metrics can be difficult to interpret (Charter & Feldt, 2002; Hildon et al., 2012; McManus, 2012; Plebani et al., 2018; Simpkin & Armstrong, 2019), even for people who have received rigorous training (Belia et al., 2005; Kalinowski et al., 2018).”
BackgroundThe Importance of Reliability as It Relates to True Score Confidence Intervals
Charter, R. A., & Feldt, L. S. (2002) · Measurement and Evaluation in Counseling and Development · doi:10.1080/07481756.2002.12069053 · Cited by 30 · cited 1×
“Several studies show that such abstract metrics can be difficult to interpret (Charter & Feldt, 2002; Hildon et al., 2012; McManus, 2012; Plebani et al., 2018; Simpkin & Armstrong, 2019), even for people who have received rigorous training (Belia et al., 2005; Kalinowski et al., 2018).”
BackgroundPractical, Evidence-Based Clinical Decision Making: Introduction to the Special Series
Jensen‐Doss, A. (2014) · Cognitive and Behavioral Practice · doi:10.1016/j.cbpra.2014.08.001 · Cited by 27 · cited 1×
“Psychological measurement plays an important supporting role in clinical decision making, by informing clinicians’ decisions regarding treatment options, effectiveness, and duration (Jensen-Doss, 2015).”
BackgroundThe effect of communication design and recipients’ numeracy on responses to UXO risk
Bruine de Bruin, W., Stone, E. R., Gibson, J. M., Fischbeck, P. S., & Shoraka, M. B. (2013) · Journal of Risk Research · doi:10.1080/13669877.2013.788055 · Cited by 24 · cited 1×
“Similar to the findings of studies by de Bruin et al. (2013) and Levontin et al. (2020), subjective understanding was participants rated familiar textual and error bar formats as more understandable, despite their probability assessments being less accurate with these formats.”
ResultZapata-Rivera, D., Zwick, R., & Vezzu, M. (2016) · Educational Assessment · doi:10.1080/10627197.2016.1202110 · Cited by 19 · cited 2×
“…statistical training have been shown to impact uncertainty understanding (Hopster-den Otter et al., 2019; Shah & Hoeffner, 2002; van der Bles et al., 2019; Zapata-Rivera et al., 2016), it is important to examine how to best communicate measurement accuracy in a population of clinical practitioners.”
BackgroundA Cross-Sectional Analysis of Students’ Intuitions When Interpreting CIs
Kalinowski, P., Lai, J., & Cumming, G. (2018) · Frontiers in Psychology · doi:10.3389/fpsyg.2018.00112 · Cited by 14 · cited 1×
“Several studies show that such abstract metrics can be difficult to interpret (Charter & Feldt, 2002; Hildon et al., 2012; McManus, 2012; Plebani et al., 2018; Simpkin & Armstrong, 2019), even for people who have received rigorous training (Belia et al., 2005; Kalinowski et al., 2018).”
Background(2020) · Spiral (Imperial College London) · Cited by 12
Otter, D. H., van den Boom-Muilenburg, S. N., Wools, S., Veldkamp, B. P., & Eggen, T. J. H. M. (2018) · Assessment in Education Principles Policy and Practice · doi:10.1080/0969594x.2018.1447908 · Cited by 8 · cited 3×
“Moreover, insight regarding measurement accuracy may have practical implications for decision making (Hopster-den Otter et al., 2019).”
BackgroundInfluentialHeltne, A., Frans, N., Hummelen, B., Falkum, E., Germans Selvik, S., & Paap, M. C. S. (2023) · Scandinavian Journal of Psychology · doi:10.1111/sjop.12918 · Cited by 5 · cited 7×
“Furthermore, studies included in the review by Heltne et al. (2023) suggest that participants are sensitive to information overload and often made more optimal decisions when presented with simple, less detailed, visualizations.”
BackgroundMethodInfluentialSelf-citation
Research works printed but not linked: 2
- Ben-Porath Y. S. Tellegen A. (2008). MMPI-2-RF: Manual for administration, scoring and interpretation. University of Minnesota Press.
- Wickham H. (2016). ggplot2: Elegant graphics for data analysis. Springer-Verlag. https://ggplot2.tidyverse.org
Cited by
Chen, W., Wang, H., Ten, X.-L., Fu, M., Lin, M., Xu, X., & Ma, Y. (2025) · BMC Infectious Diseases · doi:10.1186/s12879-025-11279-6 · Cited by 8
“…knowledge for model selection in policy evaluation [30]: the intuitive parameters of beta regression models are advantageous when focusing on short-term immediate effects [31, 32]; while the flexibility of GAM models may be more appropriate when exploring long-term complex dynamic trends [33].”
Zimmermann, J., Witthöft, M., Cicero, D. C., Forbes, M. K., Hopwood, C. J., Kotov, R., Krueger, R. F., Sellbom, M., Simms, L. J., Wright, A. G. C., et al. (2024) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/kp8yx · Cited by 5
Evaluating visual feedback of experience sampling data for mental health practitioners
Piot, M., Dejonckheere, E., Tuinstra, A., Tijs, I., Kuppens, P., & Verdonck, S. (2025) · Internet Interventions · doi:10.1016/j.invent.2025.100873 · Cited by 1
“Previous research has compared various methods for visualizing uncertainty in graphs, such as gradient plots, error bars or violin plots (Correll and Gleicher, 2014; Frans et al., 2023).”
Marchiori, M., Heidler, J., Segantin, G., Eckmann, H. R., Chinapaw, M. J. M., Kjærgaard, M., & Schipperijn, J. (2025) · Journal of Activity Sedentary and Sleep Behaviors · doi:10.1186/s44167-025-00088-6 · Cited by 1
Advancing the vision of “reliability metadata”: From conceptual refinement to clinical validation
Yu, Z., & Cheng, W. (2026) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2026.109267
Metadata
- Authors
- Frans, N., Hummelen, B., Albers, C. J., & Paap, M. C. S.
- Affiliations
- Oslo University Hospital (NO)
- University of Groningen (NL)
- Corresponding author
- Frans, Niek
- Venue
- Assessment
- Publisher
- SAGE Publications
- Published
- 2023-02-01
- Volume
- 30
- Issue
- 8
- Pages
- 2449-2460
- ISSN
- 1073-1911 (ISSN-L)
- 1552-3489
- PMC
- PMC10623599
- OpenAlex
- W4318931757
- Cited by
- 5
- References
- 47
- Citations by year
- 2026: 1 · 2025: 3 · 2024: 1
- FWCI
- 0.45
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- Assessment (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- University of Groningen research database (University of Groningen / Centre for Information Technology) (submittedVersion, cc-by)
- Funders
- Norges Forskningsråd (NFR 286893)
- Topics
- Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › Data Visualization and Analytics
- Health Sciences › Medicine › Family Practice › Clinical Reasoning and Diagnostic Skills
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Keywords
- measurement error, uncertainty visualization, score reports, violin plots, confidence intervals, clinical decision-making, test score interpretation, uncertainty communication
- MeSH terms
- Communication, Humans, Uncertainty
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol and qwen3.8-omni-flash).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.