CoChat
Sign Up
PreprintCited by 1Medical Image Segmentation TechniquesArtificial Intelligence in Healthcare and EducationDomain Adaptation and Few-Shot Learning

Beyond Manual Annotation: A Human-AI Collaborative Framework for Medical Image Segmentation Using Only "Better or Worse" Expert Feedback

Open access
Zhang, Yizhe·2025·arXiv
DOI 10.48550/arxiv.2507.05815arXiv 2507.05815
Open this paper in CoChat — free account

Abstract

Manual annotation of medical images is a labor-intensive and time-consuming process, posing a significant bottleneck in the development and deployment of robust medical imaging AI systems. This paper introduces a novel hands-free Human-AI collaborative framework for medical image segmentation that substantially reduces the annotation burden by eliminating the need for explicit manual pixel-level labeling. The core innovation lies in a preference learning paradigm, where human experts provide minimal, intuitive feedback -- simply indicating whether an AI-generated segmentation is better or worse than a previous version. The framework comprises four key components: (1) an adaptable foundation model (FM) for feature extraction, (2) label propagation based on feature similarity, (3) a clicking agent that learns from human better-or-worse feedback to decide where to click and with which label, and (4) a multi-round segmentation learning procedure that trains a state-of-the-art segmentation network using pseudo-labels generated by the clicking agent and FM-based label propagation. Experiments on three public datasets demonstrate that the proposed approach achieves competitive segmentation performance using only binary preference feedback, without requiring experts to directly manually annotate the images.

AI snapshot

Written from the paper’s full text.

TL;DR

The paper proposes a human-AI collaborative framework for medical image segmentation that trains from only binary “better or worse” expert preference feedback, using a clicking agent, DINOv2-based label propagation, and multi-round pseudo-label training. Experiments on three public datasets show progressive Dice improvement and decreasing HD95 without manual pixel-level annotation.

Key findings

  • Across BUSI (780 ultrasound scans), ISIC 2018 (2,594 dermoscopy images), and combined Kvasir-SEG/CVC-ClinicDB (1,450 colonoscopy images), segmentation quality improved over 5 annotation rounds, with increasing Dice scores and decreasing HD95.
  • Dice distributions shifted toward higher scores and became more concentrated from round 2 onward, often exceeding 0.8; the initial round showed wide and often bimodal distributions.
  • Ultrasound performance was consistently lower than other modalities, which the authors attribute to DINOv2 being a general-purpose foundation model primarily trained on natural images rather than ultrasound-specific features.
  • The framework achieved competitive segmentation performance using only binary preference feedback, without requiring experts to directly manually annotate images.

Methods

  • Iterative multi-round process with R = 5 rounds; per image, T = 5 clicks from a reinforcement-learning clicking agent. State is the image concatenated with the current predicted mask; action is a pixel coordinate; reward is +1 if Dice improves versus the oracle and −1 otherwise; policy is updated with REINFORCE.
  • DINOv2 ViT features are adapted with LoRA; a click label is propagated to patches whose cosine similarity to the clicked feature exceeds τ = 0.8, producing dense pseudo-masks. Optional top-K pseudo-label filtering is used.
  • A PVT-based HSNet segmentation model is fine-tuned on pseudo-labels each round. Ground-truth masks are used only as an oracle to simulate expert “better or worse” feedback, not as supervised training masks.

Limitations

  • The paper states that using a simulated oracle comparing Dice scores oversimplifies real-world expert judgments, which are often subjective and prioritize clinical relevance over global metrics; human feedback variability and cognitive bias are not captured.
  • The paper states that REINFORCE-based training suffers from high variance and sparse binary rewards, which may delay convergence and require many interactions before the clicking agent becomes effective.
  • Evident from the text: evaluation relies on a ground-truth Dice oracle rather than real human feedback, and the paper reports trends and Dice scores often exceeding 0.8 but does not provide exact final Dice/HD95 values or statistical comparisons.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2507.05815

Cites

0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

7 of 9 cited sources checked · 3 supported · 0 flagged · accuracy not yet measured

  • Premisecitation
    § 1 Introduction · p. 2

    Existing interactive segmentation approaches, including foundation-model-based ones, still require the user to manually provide explicit and spatially precise inputs.

    “Despite these advancements, existing interactive approaches, including those based on foundation models, still require the user to manually provide explicit and spatially precise inputs.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [12] 12. Liao, X., Li, W., Xu, Q., Wang, X., Jin, B., Zhang, X., Wang, Y., Zhang, Y.: Iteratively-refined interactive 3d medical image segmentation with multi-agent reinforcement learning. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 9394-9402 (2020) source

      The Introduction (page unavailable) describes interactive segmentation using user-provided points, scribbles, or bounding boxes, and the State section describes clicks placed on error regions. These passages support the claim about explicit, spatially located inputs for the methods discussed, but the paper does not address foundation-model-based approaches, so it cannot support that extension. (LLM, conf=0.88)

      Experimental — accuracy not yet measured
  • Findingown data
    § 3.2 Results and Analyses · p. 6

    As annotation rounds progress, segmentation quality consistently improves across all three datasets, evidenced by increasing Dice scores and decreasing HD95.

    “As the annotation rounds progress, we observe a consistent improvement in segmentation quality across all datasets, evidenced by increasing Dice scores and decreasing Hausdorff Distance at 95th percentile (HD95).”
    Quote in the paperSubstantial claim
    • Rests on an assumption: An increase in Dice Similarity Coefficient between the oracle ground truth and the segmentation mask is a valid proxy for an expert judging the new segmentation as better.
  • Findingreasoning
    § 3.2 Results and Analyses · p. 6

    Segmentation performance on ultrasound images was consistently lower than on other modalities.

    “Interestingly, we observed that segmentation performance on ultrasound images was consistently lower compared to other modalities.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: An increase in Dice Similarity Coefficient between the oracle ground truth and the segmentation mask is a valid proxy for an expert judging the new segmentation as better.
  • Findingown data
    § Beyond Manual Annotation: A Human-AI Collaborative Framework for Medical Image Segmentation Using Only 'Better or Worse' Expert Feedback · p. 1

    The proposed framework achieves competitive segmentation performance on three public medical datasets using only binary preference feedback, without requiring experts to directly manually annotate images.

    “Experiments on three public datasets demonstrate that the proposed approach achieves competitive segmentation performance using only binary preference feedback-without requiring experts to directly manually annotate the images.”
    • Also stated in: § 5 Conclusion · p. 8 · boosted
      “Our experiments across polyp, skin lesion, and breast ultrasound datasets validate that this minimal form of feedback is sufficient to achieve competitive segmentation accuracy.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: An increase in Dice Similarity Coefficient between the oracle ground truth and the segmentation mask is a valid proxy for an expert judging the new segmentation as better.
  • Findingown data
    § 3.2 Results and Analyses · p. 7

    The initial annotation round exhibits a wide and often bimodal distribution of Dice scores, reflecting substantial variability in interaction effectiveness.

    “Notably, the initial annotation round exhibits a wide and often bimodal distribution, reflecting substantial variability in interaction effectiveness.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: An increase in Dice Similarity Coefficient between the oracle ground truth and the segmentation mask is a valid proxy for an expert judging the new segmentation as better.
  • Classified as probably not a substantial claim. 8
  • May combine two claims. 7

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 1 Introduction · p. 2

    Adaptations of foundation models for the medical domain demonstrate potential to reduce annotation time.

    “Adaptations of these models for the medical domain, such as MedSAM [15] and SAM-Med2D [4], have further demonstrated their potential to reduce annotation time.”
    Quote in the paperNot a substantial claim2 of 2 sources checked
    • cites [15] 15. Ma, J., He, Y., Li, F., Han, L., You, C., Wang, B.: Segment anything in medical images. Nature Communications 15 (1), 654 (2024) source
    • cites [4] 4. Cheng, J., Ye, J., Deng, Z., Chen, J., Li, T., Wang, H., Su, Y., Huang, Z., Chen, J., Jiang, L., et al.: Sam-med2d. arXiv preprint arXiv:2308.16184 (2023) source

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation
    § 2.2 Clicking Agent and Label Propagation · p. 4

    The REINFORCE algorithm is a policy gradient method that adjusts parameters to favor actions that yield higher rewards and improve segmentation.

    “The agent's policy is updated using the REINFORCE algorithm [20], a policy gradient method that adjusts its parameters to favor actions that yield higher rewards and improve segmentation.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [20] 20. Williams, R.J.: Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning 8 , 229-256 (1992) source

      Section 4 defines REINFORCE as a parameter-update algorithm and shows that its average update follows a direction of increasing expected reward. The paper does not establish that REINFORCE improves segmentation; that is a claim about the citing paper’s application, not a result shown by this source. (LLM, conf=0.88)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 1 Introduction · p. 2

    Recent work has begun to explore preference learning for model optimization, for example using direct preference optimization to refine foundation models with simulated annotator rankings.

    “Recent work has begun to explore the use of preference learning for model optimization, for instance, by using direct preference optimization to refine foundation models with simulated annotator rankings [11].”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [11] 11. Konwer, A., Yang, Z., Bas, E., Xiao, C., Prasanna, P., Bhatia, P., Kass-Hout, T.: Enhancing sam with efficient prompting and preference optimization for semisupervised medical image segmentation. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 20990-21000 (2025) source

  • May combine two claims.

    Premisecitation
    § Beyond Manual Annotation: A Human-AI Collaborative Framework for Medical Image Segmentation Using Only 'Better or Worse' Expert Feedback · p. 1

    Manual annotation of medical images is labor-intensive, time-consuming, and a significant bottleneck in developing and deploying medical imaging AI, requiring extensive domain expertise.

    “Manual annotation of medical images is a labor-intensive and time-consuming process, posing a significant bottleneck in the development and deployment of robust medical imaging AI systems.”
    Quote in the paperSubstantial claim2 of 2 sources checked
    • cites [13] 13. Liao, Z., Hu, S., Xie, Y., Xia, Y.: Modeling annotator preference and stochastic annotation error for medical image segmentation. Medical Image Analysis 92 , 103028 (2024) source

      The Introduction says medical-image annotation has “tremendous cost,” relies on observers’ expertise, and makes the large volume of accurate labels needed for CNN training difficult to obtain. This supports a costly bottleneck in model development, but the paper does not specifically say annotation is time-consuming or identify it as a bottleneck to deployment. (LLM, conf=0.82)

      Experimental — accuracy not yet measured
    • cites [5] 5. Cheplygina, V., de Bruijne, M., Pluim, J.P.: Not-so-supervised: a survey of semisupervised, multi-instance, and transfer learning in medical image analysis. Medical image analysis 54 , 280-296 (2019) source

      The Introduction (p. 1) says manual labeling is expensive and/or time-consuming and that the lack of labeled data is a frequent problem when applying machine learning to medical images. Section 6.4 (p. 17) discusses expert labels, but the paper does not establish that manual annotation requires extensive domain expertise or specifically identify it as a bottleneck to deployment. (LLM, conf=0.84)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Findingown data
    § 3.2 Results and Analyses · p. 7

    Across all tasks, the distribution of segmentation quality shifts toward higher Dice scores as annotation rounds progress, indicating both average and consistency improve.

    “Across all tasks, we observe a clear shift in the distribution toward higher Dice scores as annotation rounds progress, indicating that both the average and consistency of segmentation quality improve with our method.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: An increase in Dice Similarity Coefficient between the oracle ground truth and the segmentation mask is a valid proxy for an expert judging the new segmentation as better.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning
    § 4 Limitations · p. 7

    Human feedback introduces variability and cognitive bias not captured by the simulated oracle.

    “Human feedback also introduces variability and cognitive bias not captured by our oracle.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § 1 Introduction · p. 2

    Foundation models trained on vast natural image datasets exhibit remarkable zero-shot and few-shot generalization, enabling segmentation based on minimal prompts like clicks or boxes.

    “These models, trained on vast datasets of natural images, exhibit remarkable zero-shot and few-shot generalization capabilities, enabling them to segment objects based on minimal user prompts like clicks or boxes.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [10] 10. Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., Lo, W.Y., et al.: Segment anything. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 3879-3893 (2023) source

  • May combine two claims.

    Interpretationreasoning
    § 2.1 Adaptable Foundation Model for Feature Extraction · p. 4

    The adaptation of DINOv2 with LoRA sharpens the feature space, improving the model's ability to distinguish between foreground and background tissue based on evolving pseudo-annotations.

    “This adaptation sharpens the feature space, improving the model's ability to distinguish between foreground and background tissue based on the evolving pseudo-annotations.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning
    § 4 Limitations · p. 7

    The use of a simulated oracle that compares Dice scores to generate better/worse feedback oversimplifies real-world expert judgments, which are often subjective and prioritize clinical relevance over global metrics.

    “First, we rely on a simulated oracle that compares Dice scores to generate 'better or worse' feedback. While this enables controlled evaluation, it oversimplifies real-world expert judgments, which are often subjective and prioritize clinical relevance over global metrics.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationreasoning
    § 4 Limitations · p. 7

    The REINFORCE-based training of the clicking agent suffers from high variance and sparse binary rewards, which may delay convergence and require many interactions before the agent becomes effective.

    “Second, the learning efficiency of the clicking agent remains a challenge. The REINFORCE-based training suffers from high variance and sparse binary rewards, which may delay convergence and require many interactions before the agent becomes effective.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning
    § 3.1 Experimental Setup · p. 6

    The binary signal from the simulated oracle is the sole feedback used to train the clicking agent.

    “This binary signal is the sole feedback used to train the clicking agent.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning
    § 1 Introduction · p. 3

    The proposed method entirely bypasses the need for manually providing pixel-level ground truth, learning to produce accurate segmentations from an expert's comparative judgments.

    “Our method entirely bypasses the need for manually providing pixel-level ground truth, instead learning to produce accurate segmentations from an expert's comparative judgments and feedback.”
    Quote in the paperSubstantial claim
12 of 12

  • The paper relies on this without support.

    A clinician can readily judge whether one segmentation output is better or worse than another even if they cannot pinpoint all errors.

    “Often, a clinician can readily judge whether one segmentation output is 'better' or 'worse' than another, even if they cannot, or do not have the time to, pinpoint the exact locations of all errors.”

  • The paper relies on this without support.

    Preference-based feedback is natural to human decision-making and can significantly streamline human-AI interaction.

    “This type of preference-based feedback is natural to human decision-making and has the potential to significantly streamline the human-AI interaction process.”

  • The paper relies on this without support.

    Existing interactive methods often necessitate multiple rounds of corrective interactions to refine segmentation to a clinically acceptable standard.

    “Furthermore, these methods often necessitate multiple rounds of corrective interactions to refine the segmentation to a clinically acceptable standard.”

  • The paper relies on this without support.

    An increase in Dice Similarity Coefficient between the oracle ground truth and the segmentation mask is a valid proxy for an expert judging the new segmentation as better.

    “We compute the Dice Similarity Coefficient (DSC) between the oracle and the segmentation mask both before and after the agent's click. If the DSC increases, the outcome is labeled 'better' (reward = +1).”

  • The paper relies on this without support.

    The success of supervised learning models is predicated on the availability of large-scale, high-quality, pixel-level annotations.

    “However, the success of these supervised learning models is predicated on the availability of large-scale, highquality, pixel-level annotations.”
5 of 5

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • The DINOv2 citation points to Segment Anything

    Minor

    The methods section cites reference [10] for DINOv2, but reference [10] is titled "Segment anything."

  • Fig. 2 reverses the ultrasound and polyp panel positions

    Minor

    The results text places ultrasound on the left and polyp on the right, while the Fig. 2 caption assigns those positions the other way around.

  • Fig. 3 reverses the ultrasound and polyp panel positions

    Minor

    The text describing Fig. 3 places ultrasound on the left and polyp on the right, but its caption states the reverse.

3 of 3

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 1

  1. [7]7. Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W., et al.: Lora: Low-rank adaptation of large language models. ICLR 1 (2), 3 (2022)

Cited by

Sorted by citations

Authors
Zhang, Y.
Corresponding author
Zhang, Yizhe
Venue
arXiv
Publisher
arXiv
Published
2025
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2507.05815
arXiv
2507.05815
OpenAlex
W6892128267
Semantic Scholar
2dbe33eeb7a7e4c5e457d79ead420b1479daa26b
Cited by
0
References
0
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Medical Image Segmentation Techniques
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
medical image segmentation, human-AI collaboration, preference learning, annotation burden reduction, label propagation, feature similarity, pseudo-labeling, foundation models

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-22.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-22.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review