CoChat
Sign Up
Cited by 73Human-Automation Interaction and SafetyExplainable Artificial Intelligence (XAI)Information Systems Theories and Implementation

Assessing the Impact of Automated Suggestions on Decision Making: Domain Experts Mediate Model Errors but Take Less Initiative

Open access
Levy, Ariel, Agrawal, Monica, Satyanarayan, Arvind, Sontag, David·2021·Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems
DOI 10.1145/3411764.3445522
Open this paper in CoChat — free accountOpen PDF

Abstract

Automated decision support can accelerate tedious tasks as users can focus their attention where it is needed most. However, a key concern is whether users overly trust or cede agency to automation. In this paper, we investigate the effects of introducing automation to annotating clinical texts — a multi-step, error-prone task of identifying clinical concepts (e.g., procedures) in medical notes, and mapping them to labels in a large ontology. We consider two forms of decision aid: recommending which labels to map concepts to, and pre-populating annotation suggestions. Through laboratory studies, we find that 18 clinicians generally build intuition of when to rely on automation and when to exercise their own judgement. However, when presented with fully pre-populated suggestions, these expert users exhibit less agency: accepting improper mentions, and taking less initiative in creating additional annotations. Our findings inform how systems and algorithms should be designed to mitigate the observed issues.

AI snapshot

Written from the paper’s full text.

TL;DR

In a two-stage study with 18 clinicians annotating clinical text, automated label recommendations sped labeling and did not reduce recall, but fully pre-populated suggestions led experts to accept some incorrect spans/labels and to create fewer new annotations. The authors conclude that domain experts can mediate model errors but may unknowingly cede agency when automation pre-populates the task.

Key findings

  • Label recommendations improved efficiency: median label time was 3 seconds with recommendations vs. 6 seconds without; when a correct answer was among the ten displayed recommendations, median time was 2 seconds. Standard-mode users created an average of 12% more annotations than None-mode users (375 vs. 337, p<0.02).
  • Experts generally searched more when recommendations lacked a correct label: on “easy” examples they searched further only 17% of the time, but on “difficult” examples Standard users searched 85% and Weakened users 82%. When the correct label was synthetically removed in Weakened mode, however, they searched only 73% of the time on those “weakened” examples and had significantly worse total recall (average 53% vs. 76% for the other two modes, p<0.02).
  • With pre-populated suggestions, users accepted correct span-and-label suggestions over 99% of the time, but accepted incorrect labels without modification a median of 17% of the time. In Weakened mode, participants kept 33% of the 32 suggestions provided with incorrect spans (the most conscientious user kept 16%); trust in incorrect spans correlated with trust in incorrect labels (ρ=0.70, Spearman) but not with Stage 1 span accuracy (ρ=-0.29).
  • Pre-populated suggestions reduced initiative: users with suggestions annotated fewer nontrivial spans (median 58%) than users without suggestions (median 74%), and users with suggestions dropped on average 12% in nontrivial recall from Stage 1 while users without suggestions increased 4%. On compound terms, users without suggestions had median 64% nontrivial span recall vs. 31% with suggestions; confidence flags had no noticeable impact.

Methods

  • Two-stage laboratory user study with 18 clinicians (8 men, 10 women; 4 medical school graduates, 6 fourth years, 8 third years) from 9 U.S. medical institutions, recruited via Twitter and email lists, compensated $20/hour, conducted virtually and ruled IRB exempt.
  • Stage 1 randomly assigned users to None (5), Standard recommendation (6), or Weakened recommendation (7) modes; users annotated 3 sections from de-identified MIMIC-III clinical notes, compared against a gold standard of 335 annotations, with metrics including span recall, total recall, label accuracy, and search actions, analyzed with Aligned Rank Transform and Mann-Whitney U tests.
  • Stage 2 reassigned the same 18 users to No suggestion (3), Standard suggestion (4), Augmented suggestion (5), or Weakened suggestion (5) modes; all received Standard label recommendations, suggestions covered 76% of gold-standard annotations with all spans and 85% of labels correct, Augmented added confidence flags, and Weakened added incorrect-span suggestions from cTAKES and MetaMap; users annotated 3 notes containing 449 gold-standard spans.

Limitations

  • Stated: findings may not extend to simpler or binary decision settings; the study was low-stakes and not deployed live, so users may not have been as careful as under real pressure; confidence indicators had little impact and users admitted not using them; amount of training may affect outcomes.
  • Evident from the reported design: per-condition sample sizes were small (Stage 1: 5/6/7 users; Stage 2: 3/4/5/5 users), limiting statistical power and generalizability.
  • Evident from the reported sample and task: most participants were medical students or recent graduates rather than practicing clinicians, and the task used MIMIC-III sections with artificially injected errors, which may limit ecological validity.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1145/3411764.3445522

Cites

51 of 51 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 25 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    In laboratory studies, 18 clinicians generally build intuition of when to rely on automation and when to exercise their own judgement.

    “Through laboratory studies, we fnd that 18 clinicians generally build intuition of when to rely on automation and when to exercise their own judgement.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Clinical text annotation can only be conducted by domain experts.
  • Findingown data

    When presented with fully pre-populated suggestions, expert users exhibit less agency by accepting improper mentions.

    “However, when presented with fully pre-populated suggestions, these expert users exhibit less agency: accepting improper mentions, and taking less initiative in creating additional annotations.”
    Quote in the paperSubstantial claim
  • Findingown data

    When presented with fully pre-populated suggestions, expert users take less initiative in creating additional annotations.

    “However, when presented with fully pre-populated suggestions, these expert users exhibit less agency: accepting improper mentions, and taking less initiative in creating additional annotations.”
    Quote in the paperSubstantial claim
  • Findingown data

    Domain experts demonstrate less initiative in the creation of additional spans.

    “While they do change incorrect labels, they are slightly more hesitant to intervene with incorrect spans, and they demonstrate less initiative in the creation of additional spans.”
    Quote in the paperSubstantial claim
  • Findingown data

    Human error patterns differ from algorithmic error patterns.

    “We also fnd that human error patterns difer from algorithmic error patterns, indicating the utility of combining the two.”
    Quote in the paperSubstantial claim
  • Classified as probably not a substantial claim. 41
  • May combine two claims. 53
  • The quote only partly states this claim. 1

  • Classified as probably not a substantial claim. May combine two claims.

    Findingown data

    In Stage 1, a correct label was presented in one of the ten displayed recommendations 83% of the time in Standard mode and 73% in Weakened mode.

    “In all, a correct label was presented in one of the ten displayed recommendations 83% of the time in the Standard mode and 73% in the Weakened mode.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Across all annotations, users in both recommendation modes found labels far quicker (median 3 seconds) than users without recommendations (median 6 seconds), and the difference was statistically significant.

    “Across all annotations, users in both recommendation modes are able to fnd labels far quicker (median of 3 seconds) than users without recommendations (median of 6 seconds) as displayed in Figure 4(b); the median time for both recommendation modes is statistically signifcant faster than users without (p<0.05across both, adjusted two-sided Mann-Whitney U test).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Humans achieve about 70% accuracy on procedures, compared to approximately 80% overall.

    “Meanwhile, humans achieve about 70% accuracy on procedures, compared to approximately 80% overall.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Users in Standard mode create an average of 12% more annotations than those in None mode (375 vs 337), a statistically significant increase.

    “Further, the users in Standard mode create an average of 12% more annotations than those in None mode (375 vs 337), a statistically signifcant increase (p<0.02,two-sided Mann-Whitney U test).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S1 (asserted), S3 (load-bearing), S4 (based). May combine two claims.

    Findingnone given

    On easy examples, users in both Standard and Weakened modes only searched further 17% of the time and ultimately chose a recommended label 96% of the time.

    “On”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Users in Weakened mode created an average of 353 annotations, not significantly different from other modes.

    “Users in Weakened mode were in the middle of both, with an average of 353 (not signifcantly diferent).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Across modes, searches are rarely initiated when the true label is provided in the recommendations (Easy), and are often initiated when the true label isn't provided (Difficult).

    “Across modes, searches are rarely initiated when the true label is provided in the recommendations (”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Both mode and annotation difficulty, and their interaction, significantly affect how often users search on the platform.

    “Both factors and their interaction additionally signifcantly afect how often users search on the platform (p<1e-8 across all three).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    On difficult examples, Standard users searched further 85% of the time and Weakened users 82% of the time.

    “However, on”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    When spans were correct and labels were incorrect, the median user accepted suggestions 17% of the time without modifications, with no observable difference between modes (p=0.72).

    “When spans were correct and labels were incorrect, the median user accepted suggestions 17% of the time without modifcations, and there were no observable diference between error rates between the modes (p=0.72,Kruskal-Wallis test).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The median user without suggestions had 89% span recall and 86% total recall on this set of annotations.

    “The median user without suggestions had a 89% span recall and 86% total recall on this set of annotations, confrming that users without suggestions do miss some examples.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Common failure modes included assuming there must be no matching label and either choosing that no label is present or settling for a related but suboptimal label.

    “Common failure modes include assuming there must be no matching label and (i) choosing that there is no label present (e.g. a user for Diastolic CHF), or (ii) settling for a related but suboptimal label (e.g.ultrasonography for echo instead of searching for echocardiogram).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Total recall was not significantly different between modes for Easy or Difficult examples, but the Weakened mode was significantly worse at the Weakened examples than the other two modes (p<0.02 for both).

    “A pairwise Mann-Whitney U test shows that while total recall was not signifcantly diferent between modes for”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    The Standard suggestion mode displays suggestions for examples where there was an exact match between the example text and one of the synonyms in the medical vocabulary (76% of the annotations in the gold standard).

    “The Standard suggestion mode displays suggestions for the examples where there was an exact match between the example text and one of the synonyms in the medical vocabulary (76% of the annotations in the gold standard).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The presence of confidence indicators in Augmented mode did not make any noticeable impact on users' rate of modifying incorrectly suggested labels or their accuracy on the low-confidence subset.

    “Further, the presence of the confdence indicators in the Augmented mode did not make any noticeable impact on user's rate of modifying incorrectly suggested labels or their accuracy on the low-confdence subset.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The user who accepted 58% did not appear less competent at the task; they achieved above-average accuracy for their mode in Stage 1.

    “Similarly, the user who accepted 58% did not appear less competent at the task; they achieved above-average accuracy for their mode in Stage 1.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The vast majority of users accepted between 10 and 20% of incorrect suggestions, one user accepted 8%, and another accepted 58%.

    “While the vast majority of users accepted between 10 and 20% of incorrect suggestions, one user only accepted 8%, and another accepted 58%.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The user who accepted 8% did not appear any better at the task; they had below-average total recall for their mode in Stage 1.

    “The user who accepted 8% did not appear any better at the task; they had below-average total recall for their mode in Stage 1.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Approximately one-third of the suggested labels are considered low confidence; high-confidence labels are 90% correct, and low-confidence labels are 70% correct.

    “Approximately one-third of the suggested labels are considered low confdence; the high-confdence labels are 90% correct, and the low-confdence labels are 70% correct.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    Of the Standard suggestions, all spans and 85% of the labels are correct.

    “Of these suggestions, all spans and 85% of the labels are correct.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingnone given

    Accuracy differed widely across users, but was not a function of skill.

    “Accuracy difered widely across users, but was not a function of skill.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    In Stage 2, users labeled three notes each, which contained a total of 449 spans in the gold standard.

    “As in the previous stage, they again labeled three notes each, which contained a total of 449 spans in the gold standard.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    On weakened examples, Weakened mode users conducted a further search 73% of the time and achieved significantly worse total recall (average 53%) than the other two modes (76%).

    “On these”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    In an Aligned Rank transform test, there was a statistically significant effect of stage (p<0.005) and of the interaction between stage and mode (p=0.01).

    “In an Aligned Rank transform test with factors of (i) stage and (ii) whether users received pre-annotations, there was a statistically signifcant efect of stage (p<0.005)and of the interaction between stage and mode (p=0.01).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Users without suggestions increased on average 4% between the two stages, while users with suggestions dropped on average 12%.

    “Users without suggestions increased on average 4% between the two stages, indicating Stage 2 may have been slightly easier, but users with suggestions dropped on average 12%.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The user who only accepted 16% of incorrect spans had below average accuracy for their mode in Stage 1.

    “For example, the user who only accepted 16% of incorrect spans had below average accuracy for their mode in Stage 1.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The difference was most pronounced in compound terms, where users without suggestions had a median of 64% nontrivial span recall on compound terms, and users with suggestions only had a median of 31%.

    “The diference was most pronounced in compound terms, where users without suggestions had a median of 64% nontrivial span recall on compound terms, and users with suggestions only had a median of 31%.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingnone given

    Other incorrect spans indicated concrete concepts, but were not clinical in nature, e.g. sister, and are therefore not supposed to be tagged.

    “Other incorrect spans indicated concrete concepts, but were not clinical in nature, e.g.sister, and are therefore not supposed to be tagged.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    For a patient with a persistent cough, the system provided the incorrect label of persistence, defined as mental perseverance; this incorrect label was accepted by 10 users.

    “For example, for a patient with a persistent cough, our system provided the incorrect label of persistence, defned as mental perseverance; this incorrect label was accepted by 10 users.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The majority of user errors were one-off, but errors were not randomly distributed, and they indicated snap judgements based off concept name.

    “Though the majority of user errors were one-of, errors were not randomly distributed, and they indicated snap judgements based of concept name.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    On average, participants in Weakened mode kept 33% of the 32 suggestions provided with incorrect spans, though the most conscientious user kept only 16%.

    “On average, participants in the Weakened mode kept 33% of the 32 suggestions provided with incorrect spans, though the most conscientious user kept only 16%.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    All users without suggestions tagged RV thrombus (compared to 46% of users with suggestions) and decompensated CHF (compared to 50% of users with suggestions).

    “For example, all users without suggestions tagged RV thrombus (compared to 46% of users with suggestions) and decompensated CHF (compared to 50% of users with suggestions).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    Some provided incorrect spans included obviously incorrect selections such as medical conditions. Exam which spanned two sentences (kept by 2/5 users), or of the superior segment branch, which contained unnecessary prepositions (kept by 3/5 users).

    “Some of the provided incorrect spans included obviously incorrect selections such as medical conditions.Exam which spanned two sentences (kept by 2/5 users), or of the superior segment branch, which contained unnecessary prepositions (kept by 3/5 users).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Users with pre-populated suggestions overall had slightly higher total recall than users without, but they initiated the creation of fewer additional annotations.

    “While users with pre-populated suggestions overall had slightly higher total recall than users without, they initiated the creation of fewer additional annotations.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    When labels are incorrect, users with suggestions were on average 5 seconds slower than users annotating from scratch; users also spent an average of 5 seconds on suggestions over incorrect spans.

    “However, when labels are incorrect, users with suggestions were on average 5 seconds slower than users annotating from scratch; users also spent an average of 5 seconds on suggestions over incorrect spans.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Users were faster on annotations that came with pre-populated suggestions, but when labels were incorrect, they were slower than users annotating from scratch.

    “Users were faster on annotations that came with pre-populated suggestions, but when labels were incorrect, they were slower than the users annotating from scratch.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The negative impacts of automation were user-specific and correlated across aid modalities.

    “Second, we found that the negative impacts of automation were user-specifc and correlated across aid modalities.”
    • Also stated in: § 7 DISCUSSION · hedged
      “Further, we found that automation had widely varying impacts on users and ceded agency tended to correlate across modalities.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    The user with the second lowest rate of accepting erroneous label suggestions was also one of the slowest, taking almost twice as long as the median.

    “In contrast, the user with the second lowest rate of accepting erroneous label suggestions was also one of the slowest, taking almost twice as long as the median.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The presence of pre-populated suggestions leads users to lose critical engagement in the task and to add fewer new annotations than each had previously, without their realizing it.

    “Without them realizing it, the presence of pre-populated suggestions leads them to lose critical engagement in the task and add fewer new annotations than each had previously.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The domain experts remained appropriately skeptical of label recommendations and formed an intuition for when further searching was required.

    “On the whole, we found that our domain experts remained appropriately skeptical of label recommendations, and they formed an intuition for when further searching was required.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    In cases where users do not properly learn when to trust automation, they can become overly reliant on systems.

    “However, in cases where users do not properly learn when to trust automation, they can become overly reliant on systems [25].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [25] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Effective group work requires an accurate mental model of one's teammates-humans or AI.

    “The idea that efective group work requires an accurate mental model of one's teammates-humans or AI-goes back decades [7,16].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [7]

      Not checked in the free run.

    • cites [16] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Knowledge of the error boundary enables the user to know when to trust automation and when to override it.

    “It has been empirically shown that users with better mental models of their AI teammates are more successful; in particular, knowledge of the error boundary enables the user to know when to trust automation and when to override it [3,12].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [3] source

      Not checked in the free run.

    • cites [12] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    On image recognition tasks, people will trust an incorrect machine decision, even if they would have made the correct decision on their own.

    “On image recognition tasks, Suresh et al. found that people will trust an incorrect machine decision, even if they would have made the correct decision on their own [48].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [48] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Users with better mental models of their AI teammates are more successful.

    “It has been empirically shown that users with better mental models of their AI teammates are more successful; in particular, knowledge of the error boundary enables the user to know when to trust automation and when to override it [3,12].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [3] source

      Not checked in the free run.

    • cites [12] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The highest accuracy yet achieved on Luo et al.'s data set is 85%.

    “On their clinical text corpus, Luo et al. showed that when provided with a correct text span, simple heuristic methods can achieve 77% label accuracy; the highest accuracy yet achieved on their data set is 85% [28].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [28] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S4 (based).

    Study design and conductreasoning

    How often users chose to search is an objective proxy for trust in the recommendations.

    “Due to the size of the label space, we were able to measure how often they chose to search, an objective proxy for trust in the recommendations.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    To isolate the effect of recommendations, no pre-populated annotations were included in Stage 1.

    “To isolate the efect of recommendations, we did not include any pre-populated annotations in this stage.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The order of notes was randomized between users.

    “The order of notes was randomized between users.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    In a study of radiologists, decision aid helped lower-performing radiologists but hindered the performance of the best ones.

    “Similarly, a study of radiologists showed that while decision aid helped lower-performing radiologists, it actually hindered the performance of the best ones [39].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [39] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data

    The study analyzes each annotator's behavior and outcomes over approximately eight hours of annotation each, allowing accounting for effects over long time scales like tedium.

    “We analyze each annotator's behavior and outcomes over approximately eight hours of annotation each, allowing us to account for efects over long time scales like tedium.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Clinical text annotation is tedious.

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    The tutorial modes had approximately the same proportions of correct and incorrect recommendations as the full notes.

    “We ensured the tutorial modes had approximately the same proportions of correct and incorrect recommendations as the full notes to allow users to build appropriate mental models.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Matching the tutorial modes' proportions of correct and incorrect recommendations to the full notes allows users to build appropriate mental models.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Exposure to examples increases conformity of users' drawings.

    “Kulkarni et al. showed in an experiment on sketching, exposure to examples increases conformity of users' drawings [24].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [24] source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Study design and conductown data

    In Stage 1, users were randomly assigned to one of three recommendation modes: None, Standard, or Weakened.

    “First, users were assigned randomly to one of three modes: the None mode, with no recommendations (5 users), the Standard recommendation mode (6 users), or the Weakened recommendation mode (7 users);”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Users exposed to regularly-spaced examples produced the fewest ideas.

    “Similarly, Siangliulue et al. investigated the efect of when examples were shown; users exposed to regularly-spaced examples produced the fewest ideas [44].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [44] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Users' subjective evaluations of their trust in and relationship to decision support systems do not always align with actual outcomes.

    “Further work has shown that users' subjective evaluations of their trust in and relationship to decision support systems does not always align with actual outcomes, highlighting the importance of quantitative studies [5].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [5] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The gold standard released by Agrawal et al. contained 335 annotations over the sections used in Stage 1.

    “For evaluation, we compared to the gold standard released by Agrawal et al., which contained 335 annotations over the sections used in this stage [1].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [1]

      Not checked in the free run.

  • May combine two claims.

    Interpretationown data

    Human errors don't follow the same error patterns as algorithms, and therefore are adding valuable signal.

    “This indicates that human errors don't follow the same error patterns as algorithms, and therefore are adding valuable signal.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Spans where the label was considered ambiguous in the gold standard were excluded.

    “Further, we excluded all spans where the label was considered ambiguous in the gold standard.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    Users' search patterns are closely aligned, and they generally learn to search further when necessary.

    “Therefore, we see that their search patterns are closely aligned, and they generally learn to search further when necessary.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    Algorithmic accuracy of one existing extraction system is around 48% overall, but only 24% for procedures.

    “For example, algorithmic accuracy of one existing extraction system is around 48% overall, but only 24% for procedures [1].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [1]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductreasoning

    Additional correct user labels were manually checked and added to the gold standard, with the process conducted blind to the user and their mode.

    “Further, since there may be multiple correct labels and the gold standard may not have contained them all, we manually checked whether any additional user labels were correct, and if so, added them to our gold standard.This process was conducted blind to the user and their mode.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Tutorial sentences were designed to contain span and label errors in the same proportion as the full study.

    “The tutorial sentences were designed to contain span and label errors in the same proportion as the full study, so that users could understand the role of automated decision aid.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductown data

    The same 18 clinicians were re-assigned randomly to one of four suggestion modes (No suggestion, Standard suggestion, Augmented suggestion, Weakened suggestion) in Stage 2.

    “They were now re-assigned randomly to one of four modes: the No suggestion mode (3 users), the Standard suggestion mode (4 users), the Augmented suggested mode (5 users), and the Weakened suggestion mode (5 users).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    Users were not solely searching further based on whether they thought a label was missing, but also based on whether they thought the recommendation algorithm should have been able to surface the correct label.

    “This diference indicates that users were not solely searching further based on whether they thought a label was missing, but also based on whether they thought the recommendation algorithm should have been able to surface the correct label.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationown data

    This indicates a decreased engagement.

    “This indicates a decreased engagement.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    Domain experts reacted differently to automation, but this reaction is not directly a function of skill.

    “This suggests that our domain experts reacted diferently to automation, but this reaction is not directly a function of skill.”
    Quote in the paperSubstantial claim

  • May combine two claims. The quote only partly states this claim.

    Interpretationown data

    Users with pre-populated suggestions exercised less agency in creating new annotations, but they noted the opposite in their exit surveys.

    “Users with pre-populated suggestions exercised less agency in creating new annotations, but they noted the opposite in their exit surveys.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationown data

    The results suggest implications for the appropriate amount of automation to include in our and similar platforms.

    “The results of our study suggest implications for the appropriate amount of automation to include in our and similar platforms.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    The presence of pre-populated annotations redirected users' attention and caused a loss of agency, and such techniques should be employed with caution.

    “Additionally, we found the presence of pre-populated annotations redirected users' attention and caused a loss of agency, and such techniques should be employed with caution.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationown data

    The set of pre-populated annotations would need to prioritize high precision over high recall.

    “This would indicate that the set of pre-populated annotations would need to prioritize high precision (surfaced annotations correspond to true spans) over high recall (most gold standard annotations are surfaced).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    Domain experts were relatively adept at dealing with incorrect labels: both searching for additional labels when recommendations failed, and modifying incorrect labels in prepopulated annotations.

    “We found that our domain experts were relatively adept at dealing with incorrect labels: both searching for additional labels when recommendations failed, and modifying incorrect labels in prepopulated annotations.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    In our context, the efficiency gains are slim and depend greatly on underlying accuracy.

    “We fnd that in our context the efciency gains are slim in our context and depend greatly on their underlying accuracy.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    Because automated systems and humans often have complementary strengths, joint systems can outperform either alone.

    “Since automated systems and humans often have complementary strengths, joint systems can outperform either alone[49].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [49]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    The optimal decision regarding how to integrate decision support into a system may depend on the downstream use case.

    “The optimal decision regarding how to integrate decision support into a system may depend on the downstream use case.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationown data

    Issues of algorithmic trust and agency extend far past user confidence and expertise to a more intrinsic behavior.

    “As we found with varied susceptibility among users, issues of algorithmic trust and agency extend far past user confdence and expertise to a more intrinsic behavior.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    The phenomenon of misplaced trust and loss of agency has been less studied for domain experts, who may be more confident in their own abilities and more skeptical of automation.

    “However, this phenomenon has been less studied for domain experts, who may be more confdent in their own abilities and more skeptical of automation[26].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [26] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationnone given

    The platform enables efficient annotation of text documents and could help scale data set creation in a domain where annotated data set sizes have been historically small.

    “More broadly, our platform enables efcient annotation of text documents and could help scale data set creation in a domain where annotated data set sizes have been historically small.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Premisecitation

    For optimal results, the human needs to understand when to listen to the computer and when to exercise their own agency.

    “For optimal results, the human needs to understand when to listen to the computer and when to exercise their own agency[3].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [3] source

      Not checked in the free run.

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    Humans may develop a misplaced trust in automation, which can degrade their output.

    “If not, the human may develop a misplaced trust in the automation, which could have the adverse efect of degrading their output, as has been demonstrated in prior work[48].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [48] source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Trust in automation may cause humans to lose critical engagement with the task, so they lack the attention to intervene when necessary.

    “Further, this trust may cause them to lose critical engagement with the task, so they do not have the attention to intervene when necessary, as has been shown in self-driving car examples[19].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [19] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    In the presence of noise, binary machine learning classifiers can still be successfully trained if the patterns of noise are well-characterized and below random.

    “Natarajan et al. showed that in the presence of noise, binary machine learning classifers can still be successfully trained, if the patterns of noise are well-characterized and below random (e.g.there is an estimate of what fraction of the time a specifc outcome may be incorrect) [30].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [30]

      Not checked in the free run.

82 of 82

  • The paper relies on this without support.

    Clinical text annotation is tedious.

    “the task is ripe for decision aid since it is tedious, can only be conducted by domain experts, and existing systems can act as partial solutions.”

  • The paper relies on this without support.

    Clinical text annotation can only be conducted by domain experts.

    “the task is ripe for decision aid since it is tedious, can only be conducted by domain experts, and existing systems can act as partial solutions.”

  • The paper relies on this without support.

    A note contains over a hundred annotations.

    “On the other hand, a note contains over a hundred annotations, and a well-designed suggestion system could decrease the burden on users, enabling them to focus their attention where it is needed most, instead of replicating what is known.”

  • The paper relies on this without support.

    The tutorial allows users to form mental models of the accuracy of the automated features.

    “The tutorial serves two purposes: (i) to familiarize users with the annotation task and the platform and (ii) to allow them to form mental models of the accuracy of the automated features.”

  • The paper relies on this without support.

    Existing algorithms often fail to extract concepts that can be described in a multitude of ways, like procedures, or rarer concepts.

    “However, existing algorithms often fail to extract concepts that can be described in a multitude of ways, like procedures, or rarer concepts, despite these being critical for many of the downstream applications.”

  • The paper relies on this without support.

    The large label space intensifies the dynamics around trust and mental models.

    “Further, our work difers due to (i) the large label space we are mapping to and suggesting over, which intensifes the dynamics around trust and mental models, (ii) the wider task defnition (e.g. compared to part-of-speech tagging or identifcation of a few specifc symptoms), and (iii) our purposeful introduction of certain modes of errors.”

  • The paper relies on this without support.

    The clinical text annotation task is a good sandbox for investigating trust and agency in the presence of automated decision aid.

    “The task acts as a good sandbox since users' decisions of what to label enable us to study how decision support afects their autonomy and agency, and the large label space enables us to study how users build trust in recommendations.”

  • The paper relies on this without support.

    Matching the tutorial modes' proportions of correct and incorrect recommendations to the full notes allows users to build appropriate mental models.

    “We ensured the tutorial modes had approximately the same proportions of correct and incorrect recommendations as the full notes to allow users to build appropriate mental models.”

  • The paper relies on this without support.

    The synthetic removal of correct labels in the Weakened mode creates a violation of intuition because users expect a correct label to appear.

    “However, we observe that performance breaks down if there is a”

  • The paper relies on this without support.

    New annotations (those the machine missed) provide the most new signal for training models.

    “Given that these new annotations are the ones that provide us the most new signal for training models, we would be in danger of models being hampered in their training process.”
10 of 10

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • When individuals are faced with tasks that have lower uncertainty, such as annotation and classification tasks, they tend to rely less on AI advice and rely more on their expertise and judgment.

    “On the other hand, when individuals are faced with tasks that have lower uncertainty, such as annotation and classifcation task [4, 77, 117], they tend to rely less on the AI advice and rely more on their expertise and judgment.”

    From Dealing with Uncertainty: Understanding the Impact of Prognostic Versus Diagnostic Tasks on Trust and Reliance in Human-AI Decision Making (2024) · cites this paper as [77]

    The Introduction reports that clinicians given fully pre-populated annotations exercised less agency, not more: they were more hesitant to correct spans and less likely to add annotations. The Discussion likewise says pre-populated suggestions caused a loss of agency and cautions that the findings may not extend to simpler classification tasks. The paper does not characterize its annotation task as lower-uncertainty. (LLM, conf=0.84)

    Checked by a reasoning model — may be wrong.

    Experimental — accuracy not yet measured

  • The expert's performance is significantly impacted by seeing the Machine Learning model's score along with the rest of the information pertaining to each instance.

    “It has been shown that the expert’s performance is significantly impacted by seeing the ML model’s score along with the rest of the information pertaining to each instance1,21,22.”

    From A benchmarking framework and dataset for learning to defer in human-AI decision-making (2025) · cites this paper as [22]

    The paper’s Results (§6.2; page unavailable) say that displaying model-confidence indicators had no noticeable effect on users’ accuracy or their correction of incorrect suggestions. The Discussion (§7; page unavailable) likewise says presenting model confidence had little-to-no impact. Although Results (§5.2) finds a significant effect of annotation mode on recall, that is not a finding that seeing the model’s score significantly affected performance. (LLM, conf=0.87)

    Checked by a reasoning model — may be wrong.

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Conflicting counts of incorrect-span suggestions

    Major

    The experimental design says the Weakened mode added 21 suggestions over incorrect spans, but the results say participants received 32 such suggestions.

  • Conflicting accuracy rates for pre-populated suggestions

    Major

    The design states that all suggested spans and 85% of labels were correct, implying 85% had both correct. The results instead report 81% with both a correct span and label among the suggested gold-standard spans.

  • Stage 2 mode counts account for only 17 clinicians

    Minor

    The four reported mode counts sum to 17, although the paper says the same 18 clinicians participated in Stage 2.

    Reported 18 · recomputed 17

3 of 3

Cites

Flagged first, then by citations

Research works printed but not linked: 4

  1. Monica Agrawal, Chloe O’Connell, Yasmin Fatemi, Ariel Levy, and David Sontag. 2020. Robust Benchmarking for Clinical Entity Recognition and Normalization. Conference on Machine Learning for Healthcare (MLHC) (2020).
  2. Carol Friedman. 1995. Medlee-a medical language extraction and encoding system. Columbia University, and Queens College of CUNY (1995).
  3. Inc. Grammarly. 2018. Grammarly: Free Writing Assistant.
  4. Pontus Stenetorp, Sampo Pyysalo, Goran Topíc, Tomoko Ohta, Sophia Ananiadou, and Jun’ichi Tsujii. 2012. BRAT: AWeb-based tool for NLP-Assisted text annotation. In EACL 2012 - Proceedings of the Demonstrations at the 13th Conference of the European Chapter of the Association for Computational Linguistics.

Cited by

Sorted by citations

Related

Authors
Levy, A., Agrawal, M., Satyanarayan, A., & Sontag, D.
Affiliations
MIT Computer Science and Artificial Intelligence Laboratory (US)
ORCID
Venue
Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems
Publisher
ACM
Published
2021-05-06
Pages
1-13
DOI
10.1145/3411764.3445522
OpenAlex
W3135357229
Cited by
73
References
49
Citations by year
2026: 6 · 2025: 14 · 2024: 21 · 2023: 20 · 2022: 12
FWCI
22.60
Type
conference paper
Language
en
Open access
Licence
Locations
  • CHI Conference on Human Factors in Computing Systems (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • DSpace@MIT (Massachusetts Institute of Technology) (submittedVersion, cc-by)
  • arXiv (Cornell University)
Topics
  • Social Sciences › Psychology › Social Psychology › Human-Automation Interaction and Safety
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Social Sciences › Sociology and Political Science › Information Systems Theories and Implementation
Keywords
automated decision support, clinical text annotation, ontology mapping, automated recommendation, user agency, trust in automation, domain experts

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-10.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-10.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review