Criticmate: Stagewise Human–AI Co-Critique in Single-Screen UI Evaluation

May combine two claims.

Each UI in the UICrit dataset was annotated by three professional designers, yielding 8,387 expert-authored heuristic critique comments grounded in bounding boxes.

“Each UI was annotated by three professional designers, yielding 8,387 expert-authored heuristic critique comments grounded in bounding boxes.”

• cites [16] [16] Peitong Duan, Chin-Yi Cheng, Gang Li, Bjoern Hartmann, and Yang Li. 2024. UICrit: Enhancing Automated Design Evaluation with a UI Critique Dataset. In Proceedings of the Annual ACM Symposium on User Interface Software and Technology . 1-17. - sourceDisputed

  The Data Collection section (p. 4) says only one worker was assigned per UI, not three. Results (p. 4) reports 3,059 critiques, not 8,387. The paper does support that the annotators had professional design experience, used design guidelines, and supplied bounding boxes for critiques. (LLM, conf=0.90)

  Checked by a reasoning model — may be wrong.

  Experimental — accuracy not yet measured

May combine two claims.

Post-hoc interaction with single-pass AI outputs is opaque and coarse-grained: experts must reverse-engineer the model's reasoning to correct errors, while novices risk automation bias and over-reliance on machine-generated authority.

“For humans, post-hoc interaction with such outputs is opaque and coarse-grained: experts must reverseengineer the model's reasoning to correct errors, while novices risk automation bias and over-reliance on machine-generated authority [32, 44, 87].”

• cites [32] [32] Kate Goddard, Abdul Roudsari, and Jeremy C Wyatt. 2012. Automation Bias: A Systematic Review of Frequency, Effect Mediators, and Mitigators. Journal of the American Medical Informatics Association 19, 1 (2012), 121-127. - sourcePartly supported

  The Experience section (page not available) reports that automation bias occurs more often among task-inexperienced users, while the Confidence and trust section links trust in decision-support systems to over-reliance. The paper does not establish that single-pass AI outputs are opaque or coarse-grained, or that experts must reverse-engineer model reasoning to correct errors. (LLM, conf=0.88)

  Experimental — accuracy not yet measured
• cites [44] [44] Maia Jacobs, Melanie F Pradier, Thomas H McCoy Jr, Roy H Perlis, Finale DoshiVelez, and Krzysztof Z Gajos. 2021. How Machine-Learning Recommendations Influence Clinician Treatment Selections: The Example of Antidepressant Selection. Translational Psychiatry 11 (2021). - sourcePartly supported

  The Discussion reports that incorrect ML recommendations, particularly with limited feature-based explanations, can reduce clinicians’ treatment-selection accuracy and create a risk of overreliance. It also reports that clinicians more familiar with ML were less likely to use a recommendation. The paper does not establish that experts must reverse-engineer model reasoning, that post-hoc interaction is opaque and coarse-grained, or that novices specifically exhibit automation bias. Page numbers were not provided in our copy. (LLM, conf=0.87)

  Experimental — accuracy not yet measured
• cites [87] [87] Helena Vasconcelos, Matthew Jörke, Madeleine Grunde-McLaughlin, Tobias Gerstenberg, Michael S Bernstein, and Ranjay Krishna. 2023. Explanations Can Reduce Overreliance on AI Systems During Decision-Making. Proceedings of the ACM on Human-Computer Interaction 7, CSCW1 (2023), 1-38. - sourcePartly supported

  The Introduction (page unavailable) says people often accept incorrect AI decisions without verifying them, supporting the over-reliance concern. The paper does not establish that this risk is specific to novices, or that experts must reverse-engineer opaque, coarse-grained, single-pass outputs to correct errors. (LLM, conf=0.85)

  Experimental — accuracy not yet measured

May combine two claims.

Heuristic evaluations remain human-intensive and expertise-dependent.

“Despite their advantages, heuristic evaluations remain humanintensive and expertise-dependent.”

• cites [52] [52] Fritz Lekschas, Spyridon Ampanavos, Pao Siangliulue, Hanspeter Pfister, and Krzysztof Z Gajos. 2021. Ask Me or Tell Me? Enhancing the Effectiveness of Crowdsourced Design Feedback. In Proceedings of the CHI Conference on Human Factors in Computing Systems . 1-12. - sourceCould not verify

  The Participants section (p. 7) describes using eight design experts to judge flyer improvements, but does not identify that work as a heuristic evaluation. Searches for heuristic evaluations and their labor or expertise requirements found no passage addressing the claim. (LLM, conf=0.73)

  Experimental — accuracy not yet measured

Classified as probably not a substantial claim. May combine two claims.

Recent generative models can offload parts of UI evaluation: metrics quantify visual properties, predictive models simulate user behaviors, and LLMs detect heuristic violations from screenshots.

“Metrics can quantify visual properties [71], predictive models can simulate user behaviors [17, 91], and Large Language Models (LLMs) can detect heuristic violations directly from screenshots [17, 85, 93].”

• cites [71] [71] Antti Oulasvirta, Samuli De Pascale, Janin Koch, Thomas Langerak, Jussi Jokinen, Kashyap Todi, Markku Laine, Manoj Kristhombuge, Yuxi Zhu, Aliaksei Miniukovich, et al. 2018. Aalto Interface Metrics (AIM): A Service and Codebase for Computational GUI Evaluation. In Adjunct Proceedings of the Annual ACM Symposium on User Interface Software and Technology . 16-19. - sourceSupported
• cites [17] [17] Peitong Duan, Jeremy Warner, Yang Li, and Bjoern Hartmann. 2024. Generating Automatic Feedback on UI Mockups with Large Language Models. In Proceedings of the CHI Conference on Human Factors in Computing Systems . 1-20. - sourcePartly supported

  The Introduction (p. 2) says GPT-4 can automatically evaluate some UI-design heuristics. But the Implementation section (p. 5) says the tool evaluates a JSON representation of the mockup, not a screenshot; it also reports that a multimodal model performed considerably worse for this task. The paper does not present its model as simulating user behavior. (LLM, conf=0.88)

  Experimental — accuracy not yet measured
• cites [91] [91] Ziming Wu, Yulun Jiang, Yiding Liu, and Xiaojuan Ma. 2020. Predicting and Diagnosing User Engagement with Mobile UI Animation via a Data-Driven Approach. In Proceedings of the CHI conference on human factors in computing systems . 1-13. -Not checked yet

  Not checked in the free run.

• cites [85] [85] Maryam Taeb, Amanda Swearngin, Eldon Schoop, Ruijia Cheng, Yue Jiang, and Jeffrey Nichols. 2024. AXNav: Replaying Accessibility Tests from Natural Language. In Proceedings of the CHI Conference on Human Factors in Computing Systems . 1-16. - sourcePartly supported

  The Introduction (p. 2) says AXNav replays tasks through accessibility services so testers can see and hear how a user might perform them, supporting the user-behavior portion. But Test Heuristics (p. 9) describes screenshot-based issue detection using a UI-element detection model, OCR, and rule-based matching—not an LLM detecting heuristic violations directly from screenshots. The paper does not support the sentence’s visual-metrics portion, which is cited to a different source. (LLM, conf=0.87)

  Experimental — accuracy not yet measured
• cites [93] [93] Bo Yang and Shanping Li. 2024. UISGPT: Automated Mobile UI Design Smell Detection with Large Language Models. Electronics 13, 16 (2024). - sourceNot checked — no full text of the source

May combine two claims.

Our sample of 26 exceeds the power analysis bound and aligns with prior HCI work on AI-supported evaluation and critique.

“Our sample of 26 exceeds this bound and aligns with prior HCI work on AI-supported evaluation and critique [11, 17, 88].”

• cites [11] [11] Ruijia Cheng, Ziwen Zeng, Maysnow Liu, and Steven Dow. 2020. Critique Me: Exploring How Creators Publicly Request Feedback in an Online Critique Community. Proceedings of the ACM on Human-Computer Interaction 4, CSCW2 (2020), 1-24. - sourcePartly supported

  Section 4.1 (page unavailable) describes an HCI study of online design critique, so the paper is relevant prior work on critique. It reports 12 interview participants, not 26, and the retrieved passages do not establish that its study was AI-supported or that a sample of 26 aligns with its sample size. The citing authors’ statement that 26 exceeds their own power-analysis bound is not attributed to this source. (LLM, conf=0.84)

  Experimental — accuracy not yet measured
• cites [17] [17] Peitong Duan, Jeremy Warner, Yang Li, and Bjoern Hartmann. 2024. Generating Automatic Feedback on UI Mockups with Large Language Models. In Proceedings of the CHI Conference on Human Factors in Computing Systems . 1-20. - sourceSupported
• cites [88] [88] Wen-Fan Wang, Chien-Ting Lu, Nil Ponsa i Campanyà, Bing-Yu Chen, and Mike Y Chen. 2025. AIdeation: Designing a Human-AI Collaborative Ideation System for Concept Designers. In Proceedings of the CHI Conference on Human Factors in Computing Systems . 1-28. - sourcePartly supported

  The Summative Study (p. 12) identifies this as HCI work evaluating an AI-supported ideation tool, so it supports the comparison to prior AI-related evaluation work. The Evaluation Approach (p. 13) describes participants assessing the tool, not AI-supported critique. The citing authors’ sample of 26 and power-analysis bound are their own claims, not findings attributed to this source. (LLM, conf=0.82)

  Experimental — accuracy not yet measured

May combine two claims.

Prior work reports frequent misrecognition of UI structure, limited sensitivity to design intent, and weakly actionable critiques in current AI UI evaluation systems.

“Prior work reports frequent misrecognition of UI structure [34, 94], limited sensitivity to design intent [17, 57], and weakly actionable critiques [48].”

• cites [34] [34] Boyu Gou, Ruohan Wang, Boyuan Zheng, Yanan Xie, Cheng Chang, Yiheng Shu, Huan Sun, and Yu Su. 2025. Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents. In The Thirteenth International Conference on Learning Representations . -Not checked yet

  Not checked in the free run.

• cites [94] [94] Keen You, Haotian Zhang, Eldon Schoop, Floris Weers, Amanda Swearngin, Jeffrey Nichols, Yinfei Yang, and Zhe Gan. 2024. Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs. In Proceedings of the European Conference on Computer Vision . Springer, 240-255. - sourceSupported
• cites [17] [17] Peitong Duan, Jeremy Warner, Yang Li, and Bjoern Hartmann. 2024. Generating Automatic Feedback on UI Mockups with Large Language Models. In Proceedings of the CHI Conference on Human Factors in Computing Systems . 1-20. - sourcePartly supported

  The comparison with human evaluators (p. 13) reports that GPT-4 missed more issues requiring understanding of a UI’s purpose and context, supporting limited sensitivity to design intent. The qualitative results (p. 12) report that five of twelve participants found its suggestions too vague about how to fix violations. The cited passages do not establish frequent misrecognition of UI structure, so that part of the sentence is unsupported by this paper. (LLM, conf=0.85)

  Experimental — accuracy not yet measured
• cites [57] [57] Yuwen Lu, Tiffany Knearem, Shona Dutta, Jamie Blass, Clara Kliman-Silver, and Frank Bentley. 2024. AI Is Not Enough: A Hybrid Technical Approach to AI Adoption in UI Linting With Heuristics. In Extended Abstracts of the CHI Conference on Human Factors in Computing Systems . -Not checked yet

  Not checked in the free run.

• cites [48] [48] Mehweesh Kadegaonkar and Kayvan Karim. 2025. AI-Driven Usability Testing: Integrating Eye-Tracking Data and Agentic Systems for Automated UI Evaluation. In Proceedings of the AAAI Symposium Series , Vol. 6. 244-254. - sourceSupported

Classified as probably not a substantial claim. May combine two claims.

Fan et al. found that explanations foster trust and synchronization improves engagement in AI-assisted UX evaluation, but their Wizard-of-Oz setup left unclear how such systems would operate in practice.

“Fan et al. [26] studied AI-assisted UX evaluation, fi nding that explanations foster trust and synchronization improves engagement, but their Wizard-of-Oz setup left it unclear how such systems would operate in practice.”

• cites [26] [26] Mingming Fan, Xianyou Yang, Tsz Tung Yu, Q Vera Liao, and Jian Zhao. 2022. Human-AI Collaboration for UX Evaluation: Effects of Explanations and Synchronization. Proceedings of the ACM on Human-Computer Interaction 6, CSCW1 (2022), 96:1-96:32. - sourcePartly supported

  The Conclusion reports only tentative evidence that explanations may improve trust; survey results on trust were not significant. Section 5.2.2 finds greater engagement with synchronous than asynchronous AI specifically when explanations were absent. Section 6.4 says the Wizard-of-Oz study leaves open how to implement an AI that generates explanations of comparable quality, rather than how such systems would operate in general. (LLM, conf=0.88)

  Experimental — accuracy not yet measured

May combine two claims.

Prior work has extensively documented what usability issues experts identify, but far less is known about how experts reason through these issues across stages of perception, interpretation, and projection.

“While prior work has extensively documented what usability issues experts identify [35, 52], far less is known about how experts reason through these issues across stages of perception, interpretation, and projection [43, 76].”

• cites [35] [35] Colin M Gray. 2013. Informal Peer Critique and the Negotiation of Habitus in a Design Studio. Art, Design & Communication in Higher Education 12, 2 (2013), 195-209. - sourceNot checked — no full text of the source
• cites [52] [52] Fritz Lekschas, Spyridon Ampanavos, Pao Siangliulue, Hanspeter Pfister, and Krzysztof Z Gajos. 2021. Ask Me or Tell Me? Enhancing the Effectiveness of Crowdsourced Design Feedback. In Proceedings of the CHI Conference on Human Factors in Computing Systems . 1-12. - sourceCould not verify

  The citation is attached to the claim about how experts reason through usability issues. The Conclusion and Future Work (p. 11) discusses differences in how designers with varying expertise make sense of feedback, but our copy does not address expert reasoning through usability issues or the stages of perception, interpretation, and projection. Searches for those stages and for expert reasoning about usability issues found no supporting passage. (LLM, conf=0.73)

  Experimental — accuracy not yet measured
• cites [43] [43] Dongwook Hwang, Byunghyun Choi, Soonho Bae, and Woojin Park. 2021. Mitigating Design Fixation: A Visualization Tool for Enhancing Situation Awareness. Journal of Mechanical Design 143, 6 (2021). - sourceNot checked — no full text of the source
• cites [76] [76] A Terry Purcell and John S Gero. 1996. Design and Other Types of Fixation. Design Studies 17, 4 (1996), 363-383. - sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing).

These findings relate to work on automation bias, algorithm aversion, and appropriate reliance.

“These fi ndings relate to work on automation bias, algorithm aversion, and 'appropriate reliance' [5, 32, 44, 87].”

• cites [5] [5] Gagan Bansal, Besmira Nushi, Ece Kamar, Daniel S Weld, Walter S Lasecki, and Eric Horvitz. 2019. Beyond Accuracy: The Role of Mental Models in Human-AI Team Performance. Proceedings of the AAAI Conference on Human Computation and Crowdsourcing 7, 1 (2019), 2-11. - sourceSupported
• cites [32] [32] Kate Goddard, Abdul Roudsari, and Jeremy C Wyatt. 2012. Automation Bias: A Systematic Review of Frequency, Effect Mediators, and Mitigators. Journal of the American Medical Informatics Association 19, 1 (2012), 121-127. - sourcePartly supported

  The paper's Background discusses automation bias, and its “Automation bias avoidance Implementation factors” section explicitly discusses appropriate reliance. The cited sentence also names algorithm aversion, which our copy does not address; the sentence does not specify which of its four citations supports that term. (LLM, conf=0.84)

  Experimental — accuracy not yet measured
• cites [44] [44] Maia Jacobs, Melanie F Pradier, Thomas H McCoy Jr, Roy H Perlis, Finale DoshiVelez, and Krzysztof Z Gajos. 2021. How Machine-Learning Recommendations Influence Clinician Treatment Selections: The Example of Antidepressant Selection. Translational Psychiatry 11 (2021). - sourceSupported
• cites [87] [87] Helena Vasconcelos, Matthew Jörke, Madeleine Grunde-McLaughlin, Tobias Gerstenberg, Michael S Bernstein, and Ranjay Krishna. 2023. Explanations Can Reduce Overreliance on AI Systems During Decision-Making. Proceedings of the ACM on Human-Computer Interaction 7, CSCW1 (2023), 1-38. - sourceSupported

May combine two claims.

Participants report higher engagement and trust with Criticmate than with a single-pass baseline, with comparable perceived autonomy and overall satisfaction.

“Both participants and independent experts rate Criticmate's critiques as more helpful, logically consistent, and concrete, while participants report higher engagement and trust, with comparable perceived autonomy and overall satisfaction.”

• AssumptionRests on an assumption: The single-pass baseline used in the user study is a fair comparison to Criticmate.

May combine two claims.

Both participants and independent experts rate Criticmate's critiques as more helpful, logically consistent, and concrete than a single-pass baseline.

“Both participants and independent experts rate Criticmate's critiques as more helpful, logically consistent, and concrete, while participants report higher engagement and trust, with comparable perceived autonomy and overall satisfaction.”

• AssumptionRests on an assumption: The single-pass baseline used in the user study is a fair comparison to Criticmate.

May combine two claims.

The stagewise pipeline produces critiques that more closely align with expert feedback and better balance global and local issues than single-pass baselines.

“These results show that the stagewise pipeline produces critiques that more closely align with expert feedback and better balance global and local issues.”

• Also stated in: § Abstract · p. 1 · boosted

  “Across offline benchmarks and a controlled user study, we show that stagewise co-critique yields more expert-like and better balanced critiques than single-pass approaches, while supporting higher trust and engagement without reducing perceived autonomy.”

• AssumptionRests on an assumption: Expert-authored heuristic critiques in the UICrit dataset are a valid ground truth for critique quality.

May combine two claims.

Participants described AI's comprehension feedback as generic or redundant, forcing them to supply both contextual and visual reasoning themselves.

“Participants described such feedback as generic or redundant, forcing them to supply both contextual and visual reasoning themselves.”

May combine two claims.

Across Recall@0.5, Precision@0.5, and F1@0.5, CM-Full performs best (F1@0.5 = 0.646), with CM-w/oRCA next (0.603) and other methods trailing.

“Across Recall@0.5, Precision@0.5, and F1@0.5, CM-Full performs best (F1@0.5 = 0.646), with CM-w/oRCA next (0.603) and other methods trailing.”

May combine two claims.

For Identified Gaps, CM-Full scores 6.07, nearly doubling CM-w/oRCA (3.41) and exceeding FS (4.04) and ZS (4.32).

“For Identified Gaps, it scores 6.07, nearly doubling CMw/oRCA (3.41) and exceeding FS (4.04) and ZS (4.32).”

May combine two claims.

On average, CM-Full generates 9.6 critiques per UI, compared to 6.1 for FS, 6.5 for ZS, and 5.3-6.4 for the ablations.

“On average, it generates 9.6 critiques per UI, compared to 6.1 for FS, 6.5 for ZS, and 5.3-6.4 for the ablations.”

May combine two claims.

CM-Full achieves Vendi Score 3.91 overall, surpassing ZS (3.15) and FS (2.96).

“CM-Full achieves 3.91 overall, surpassing ZS (3.15) and FS (2.96).”

May combine two claims.

CM-Full produces both more and more varied comments.

“CM-Full produces both more and more varied comments.”

May combine two claims.

For Proposed Fixes, CM-Full reaches 4.06, above all alternatives (best competitor: ZS at 3.16).

“For Proposed Fixes, it reaches 4.06, above all alternatives (best competitor: ZS at 3.16).”

May combine two claims.

Designers fluidly integrated global and local perspectives across stages, while AI outputs tended to fragment these views.

“Designers also fl uidly integrated global (section-level) and local (component-level) perspectives across stages, while AI outputs tended to fragment these views.”

May combine two claims.

AI rarely extended its analysis to overall structure or hierarchy, and sometimes omitted central components such as navigation or primary calls-to-action.

“However, AI rarely extended its analysis to overall structure or hierarchy, and sometimes omitted central components such as navigation or primary calls-to-action.”

May combine two claims.

AI lacked contextual understanding of project goals or organizational constraints, often repeating surface-level issues without linking them to broader usability goals.

“By contrast, AI lacked contextual understanding of project goals or organizational constraints, often repeating surface-level issues without linking them to broader usability goals.”

• AssumptionRests on an assumption: ChatGPT-4o-augmented critique is representative of current AI systems for UI evaluation.

May combine two claims.

Cluttered arrangements or ambiguous component placement obscured intended entry points and led Criticmate to misidentify sections or overlook critical elements.

“Cluttered arrangements or ambiguous component placement obscured intended entry points and led Criticmate to misidentify sections or overlook critical elements.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Unclassified comments are rare and do not materially affect these trends.

“Unclassified comments are rare and do not materially affect these trends.”

May combine two claims.

Participants emphasized that they had to supply contextual information explicitly and expressed concern about over-reliance if AI feedback were accepted uncritically.

“Participants emphasized that they had to supply such information explicitly and expressed concern about over-reliance if feedback were accepted uncritically.”

May combine two claims.

AI supported the Perception stage by enumerating visible elements and flagging low-level issues.

“AI supported this stage by enumerating visible elements and flagging low-level issues.”

May combine two claims.

We observed two recurring sources of failure: ambiguous task specifications and underspecified user interfaces.

“We observed two recurring sources of failure: ambiguous task specifications and underspecified user interfaces.”

May combine two claims.

Current AI systems produce single-pass outputs without exposing intermediate reasoning, making it hard for designers to intervene at the right moment.

“By contrast, current AI systems produce single-pass outputs without exposing intermediate reasoning, making it hard for designers to intervene at the right moment.”

• AssumptionRests on an assumption: ChatGPT-4o-augmented critique is representative of current AI systems for UI evaluation.

Probably not a substantial claim: may not pass S3 (load-bearing).

In Projection, designers anticipated how current designs would affect user experience in real contexts.

“In Projection, designers anticipated how current designs would affect user experience in real contexts.”

May combine two claims.

AI's comprehension was comparatively weaker and its analyses rarely integrated visual and functional aspects.

“AI's comprehension was comparatively weaker.”

• Also stated in: § 3.3 Findings · p. 5 · plain

  “While it sometimes made plausible guesses, its analyses rarely integrated visual and functional aspects.”

• AssumptionRests on an assumption: ChatGPT-4o-augmented critique is representative of current AI systems for UI evaluation.

May combine two claims.

Experts asked whether inferred functions made sense for the scenario and sometimes deleted interpretations that overreached beyond what the single screenshot could support.

“They asked whether the inferred functions actually made sense for the scenario and sometimes deleted interpretations that overreached beyond what the single screenshot could support.”

May combine two claims.

Participants described Criticmate as offering fundamental improvements for the UI and concrete guidelines they could directly apply, while still considering the baseline helpful enough for a quick pass.

“describing Criticmate as offering 'fundamental improvements for this UI' (I7) and 'concrete guidelines that I could directly apply' (E7), while still considering the baseline 'helpful enough' for a quick pass.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Experts described their work as adapting heuristics to the domain rather than using them as-is and noted that domain-specific evaluation criteria are required.

“They described this as 'adapting heuristics to the domain rather than using them as-is' (E7) and noted that 'domain-specific evaluation criteria are required' (E11).”

May combine two claims.

For consistency, both self- and external ratings favored Criticmate.

“For consistency, both self- and external ratings favored Criticmate.”

Probably not a substantial claim: may not pass S1 (asserted).

One expert summarized the constraint that looking at just one UI and evaluating its problems or generating a critique is inherently difficult.

“One expert summarized the constraint more broadly: 'Looking at just one UI and evaluating its problems or generating a critique is inherently difficult' (E12).”

May combine two claims.

Some participants, especially intermediates, found Criticmate relatively unfamiliar and complex, and experts noted that the full staged process was not always necessary for simple screens, even if specific stages were useful as complements to existing workflows.

“some participants, especially intermediates, found Criticmate 'relatively unfamiliar and complex' (I1, I3, I13), and experts noted that the full staged process was not always necessary for simple screens, even if specific stages were useful as complements to existing workflows.”

May combine two claims.

Perceived usability and aesthetic appeal showed no reliable differences; both interfaces were seen as usable and visually acceptable for this study's tasks.

“Perceived usability and aesthetic appeal showed no reliable differences; both interfaces were seen as usable and visually acceptable for this study's tasks.”

May combine two claims.

Most interventions targeted component-level details, and participants predominantly edited by adding and modifying content, with deletions playing a smaller but still meaningful role.

“Most interventions targeted component-level details, and participants predominantly edited by adding and modifying content, with deletions playing a smaller but still meaningful role.”

May combine two claims.

Expert evaluators naturally distribute their work across Perception, Comprehension, and Projection, weaving together global and local reasoning and gradually moving from surface checks toward root-cause diagnosis.

“expert evaluators naturally distribute their work across Perception , Comprehension , and Projection [21, 22], weaving together global and local reasoning and gradually moving from surface checks toward root-cause diagnosis.”

• cites [21] [21] Mica R Endsley. 1995. Measurement of Situation Awareness in Dynamic Systems. Human Factors 37, 1 (1995), 65-84. - sourceNot checked — no full text of the source
• cites [22] [22] Mica R Endsley. 2017. Toward a Theory of Situation Awareness in Dynamic Systems. In Situational Awareness . Routledge, 9-42. - sourceNot checked — no full text of the source

May combine two claims.

Perceived autonomy did not differ reliably; participants generally felt that both systems allowed them to maintain control and express their preferences in the evaluation process.

“Perceived autonomy likewise did not differ reliably; participants generally felt that both systems allowed them to maintain control and express their preferences in the evaluation process.”

May combine two claims.

Participants described Criticmate's output as reliable because it was based on clear evidence from the screen, whereas baseline critiques were seen as loosely described without evidence.

“Participants described Criticmate's output as 'reliable because it was based on clear evidence from the screen' (I12), whereas baseline critiques were seen as 'loosely described without evidence' (E4).”

Classified as probably not a substantial claim. May combine two claims.

In Perception, designers inspect and repair the structural parsing of the screen; in Comprehension, they align visual and functional interpretations with task and domain; in Projection, they adapt guidelines, prioritize issues, and reframe root causes.

“In Perception, designers inspect and repair the structural parsing of the screen; in Comprehension, they align visual and functional interpretations with task and domain; in Projection, they adapt guidelines, prioritize issues, and reframe root causes.”

May combine two claims.

In Comprehension, experts appreciated being able to externalize usually tacit reasoning, and intermediates valued having component functions articulated explicitly.

“In Comprehension , experts appreciated being able to externalize usually tacit reasoning (E4, E12), and intermediates valued having component functions articulated explicitly (I2).”

May combine two claims.

In ablations, the full stagewise pipeline (CM-Full) outperformed both single-pass baselines and flattened variants on expert similarity, global-local balance, and diversity, despite using the same underlying models and no in-context examples.

“In our ablations, the full stagewise pipeline (CM-Full) outperformed both single-pass baselines and fl attened variants on expert similarity, global-local balance, and diversity, despite using the same underlying models and no in-context examples.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Several participants noted that working through Perception and Comprehension nudged them to treat heuristics as principles instantiated in concrete examples, rather than as abstract checklists.

“Several participants noted that working through Perception and Comprehension nudged them to treat heuristics as principles instantiated in concrete examples, rather than as abstract checklists.”

Probably not a substantial claim: may not pass S1 (asserted), S3 (load-bearing). May combine two claims.

Another expert remarked that the system felt like it only considered the static UI state but reported that this limitation could be partly mitigated by revising and tightening design guidelines applied in this stage.

“Another expert similarly remarked that the system 'felt like it only considered the static UI state' but reported that this limitation could be partly mitigated by revising and tightening design guidelines applied in this stage (E13).”

May combine two claims.

Participants' self-ratings and external assessments both favored Criticmate for concreteness, with the largest mode effect among the four dimensions in the expert ratings.

“Participants' self-ratings and external assessments both favored Criticmate, with the largest mode effect among the four dimensions in the expert ratings.”

May combine two claims.

Participants perceived Criticmate's critiques as more logically structured, with a significant positive mode effect.

“Participants perceived Criticmate's critiques as more logically structured (significant positive mode effect)”

May combine two claims.

Across both groups, Perception was where participants most actively repaired AI misrecognition, treating the section-component map as a useful but imperfect scaffold that needed to be made fully explicit before subsequent stages.

“Across both groups, Perception was where participants most actively repaired AI misrecognition, treating the section-component map as a useful but imperfect scaffold that needed to be made 'fully explicit, ' as several participants described it, before subsequent stages.”

May combine two claims.

Satisfaction showed a positive but nonsignificant coefficient, indicating broadly similar overall satisfaction across systems.

“Satisfaction showed a positive but nonsignificant coefficient, indicating broadly similar overall satisfaction across systems.”

May combine two claims.

Overall engagement and the Focused Attention subscale were significantly higher for Criticmate.

“Overall engagement and the Focused Attention subscale were significantly higher for Criticmate.”

May combine two claims.

Experts emphasized that Criticmate more often made what was wrong and why it mattered explicit in the same passage, while baseline critiques sometimes mixed screen-specific observations with generic advice, making the core message harder to follow.

“They emphasized that Criticmate more often made what was wrong and why it mattered explicit in the same passage (J2), while baseline critiques sometimes mixed screenspecific observations with generic advice, making the core message harder to follow.”

May combine two claims.

External experts rated Criticmate's critiques significantly clearer, with a smaller effect than for helpfulness or concreteness.

“External experts, however, rated Criticmate's critiques significantly clearer, albeit with a smaller effect than for helpfulness or concreteness.”

May combine two claims.

In Perception, experts likened Criticmate's detailed parsing to reviewing a PRD line by line, while intermediates said the structured descriptions made the screen easier to understand.

“In Perception , experts likened Criticmate's detailed parsing to 'reviewing a PRD line by line' (E10), while intermediates said the structured descriptions made the screen 'easier to understand' (I4).”

May combine two claims.

In Projection, experts emphasized domain-specific adaptation, while intermediates benefited from concrete, heuristic-based guidance.

“In Projection , experts emphasized domain-specific adaptation'being able to adjust guidelines for our own domain was impressive' (E7, E12)-while intermediates benefited from concrete, heuristic-based guidance (I7, I8).”

May combine two claims.

Intermediates enriched generic explanations and softened overconfident claims when behavior could not be directly observed.

“They enriched generic explanations-for example, updating 'this text block describes the podcast' to 'this text block describes the currently playing podcast episode, highlighted in a green background so users can easily identify what is playing' -and softened overconfident claims when behavior could not be directly observed.”

May combine two claims.

Participants frequently added missing components and corrected positional or shape descriptions.

“Participants frequently added missing components (e.g., a back arrow in the header of the podcast screen) and corrected positional or shape descriptions.”

May combine two claims.

Experts were active in editing guidelines and pruning or modifying proposed fixes, using Projection to enforce strategic coherence and to compensate for the lack of multi-screen context.

“Log data indicate that experts were active in editing guidelines and pruning or modifying proposed fi xes, using Projection to enforce strategic coherence and to compensate for the lack of multiscreen context.”

Probably not a substantial claim: may not pass S3 (load-bearing).

On the podcast screen, one expert reframed a surface-level comment about menu visibility into a structural critique of hierarchy and placement.

“On the podcast screen, one expert reframed a surfacelevel comment about menu visibility into a structural critique of hierarchy and placement: 'The color hierarchy for the menu icon is too low, and its position on the far left diverges from what users have learned to expect' (E3).”

May combine two claims.

For many intermediates, Comprehension served to calibrate the AI's level of certainty and to anchor functional interpretations tightly to visible cues.

“For many intermediates, Comprehension thus served to calibrate the AI's level of certainty and to anchor functional interpretations tightly to visible cues.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Some participants emphasized Criticmate's potential as a learning tool for heuristic evaluation, especially for less experienced designers.

“Some participants also emphasized Criticmate's potential as a learning tool for heuristic evaluation, especially for less experienced designers: 'for those with less design knowledge, it would be very helpful as a learning resource' (I7, I8).”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Baseline summaries were described as fast, convenient, and familiar.

“Baseline summaries were described as 'fast, convenient, and familiar' (I4, I11).”

May combine two claims.

Compared with Perception, edit rates decreased but remained substantial, with experts more likely to modify or delete entire interpretations and intermediates more often adding clarifications.

“Compared with Perception, edit rates decreased but remained substantial, with experts more likely to modify or delete entire interpretations and intermediates more often adding clarifications.”

May combine two claims.

Participants reported being more absorbed and described the staged evaluation as something they could sink into, working through the screen step by step.

“Participants reported being more absorbed and described the staged evaluation as something they could 'sink into, ' working through the screen step by step.”

May combine two claims.

Participants added missing issues, pruned redundant or low-priority points, and rewrote vague fixes into more actionable, screen-specific recommendations.

“They added missing issues (e.g., touch accessibility or icon interpretability), pruned redundant or low-priority points, and rewrote vague fi xes into more actionable, screen-specific recommendations.”

Classified as probably not a substantial claim.

Experts observed that Criticmate's suggestions often reflected only the current UI state without regard to prior or subsequent steps in the flow.

“They observed that Criticmate's suggestions, like the task framing itself, often reflected only the current UI state without regard to prior or subsequent steps in the fl ow: 'the AI seemed to consider only the static state and not the existing UI fl ow; even for humans, that would be tiring, and it did not feel like it understood the service' (E10).”

Probably not a substantial claim: may not pass S3 (load-bearing).

Another expert revised a suggestion to add a color change effect on hover into add a color change effect on press so the recommendation matched how users interact with mobile UIs in practice.

“Another expert (E8) revised a suggestion to 'add a color change effect on hover' into 'add a color change effect on press' so the recommendation matched how users interact with mobile UIs in practice.”

May combine two claims.

Four participants, including both experts and intermediates, reported being more satisfied with the baseline, citing its speed and simplicity, and several characterized it as fast, convenient, and familiar.

“four participants, including both experts and intermediates, reported being more satisfied with the baseline, citing its speed and simplicity, and several characterized it as 'fast, convenient, and familiar' (I4, I11).”

May combine two claims.

In self-ratings of helpfulness, Criticmate's mean rating exceeded the baseline's, but the difference was not significant after correction.

“Criticmate's mean rating exceeded the baseline's, but the difference was not significant after correction”

May combine two claims.

Experts rewarded critiques that referenced identifiable UI components, tied issues to the given task, and proposed context-specific remedies, while generic advice that could apply to almost any screen received low scores.

“Experts rewarded critiques that referenced identifiable UI components, tied issues to the given task, and proposed context-specific remedies (J1-J3); generic advice that could apply to almost any screen received low scores.”

May combine two claims.

Experts made more edits at section level, whereas intermediates concentrated on component rows.

“Log data show that experts made more edits at section level, whereas intermediates concentrated on component rows.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Several participants noted that both systems were easy enough to read, and that usefulness and specificity mattered more.

“Several noted that both systems were 'easy enough to read,' and that usefulness and specificity mattered more.”

May combine two claims.

Intermediates treated Criticmate as a scaffold for learning structured heuristic evaluation; experts used it to free attention for higher-level reasoning.

“Intermediates treated this as a scaffold for learning structured heuristic evaluation; experts used it to free attention for higher-level reasoning.”

May combine two claims.

Participants reported higher trust and engagement with the stagewise pipeline.

“Participants reported higher trust and engagement, attributing these gains not to anthropomorphic cues but to the transparency and editability of intermediate reasoning.”

May combine two claims.

Experts and intermediates used Criticmate differently: experts concentrated edits in structural recognition, while intermediates intervened most in Projection.

“Experts and intermediates used Criticmate differently: experts concentrated edits in structural recognition, while intermediates intervened most in Projection.”

May combine two claims.

Log data show that edits are front-loaded in Perception and Comprehension and that experts and intermediates engage in different ways.

“Log data show that edits are front-loaded in Perception and Comprehension and that experts and intermediates engage in different ways (Table 5).”

May combine two claims.

SA-inspired stages aligned with many designers' mental models and reduced cognitive burden, but four participants-including one expert-found the workflow mismatched with their habits.

“SA-inspired stages aligned with many designers' mental models and reduced cognitive burden [69], but four participants-including one expert-found the workflow mismatched with their habits.”

• cites [69] [69] Don Norman. 2013. The Design of Everyday Things: Revised and Expanded Edition . Basic books. -Not checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

Many participants noted that real projects rarely involve isolated screens.

“Our current scope is single-screen heuristic evaluation, yet many participants noted that real projects rarely involve isolated screens.”

Classified as probably not a substantial claim.

Editable guideline sets-the 'expected standards' in the triplets-were key for aligning evaluation with brand, accessibility, and domain conventions.

“Editable guideline sets-the 'expected standards' in our triplets-were key for aligning evaluation with brand, accessibility, and domain conventions.”

May combine two claims.

Participants reported higher engagement and trust with Criticmate.

“Participants reported higher engagement and trust (Table 4, Figure 13).”

May combine two claims.

The system reliably handled surface-level checks and coverage, while designers contributed task context, domain and brand knowledge, and long-term strategy.

“The system reliably handled surface-level checks and coverage (e.g., spacing, contrast, basic hierarchy), while designers contributed task context, domain and brand knowledge, and long-term strategy.”

May combine two claims.

Compared to single-pass and flattened variants, the stagewise pipeline produced heuristic critiques that were closer to expert references, more diverse, and better balanced between global and local concerns.

“Compared to single-pass and fl attened variants, the stagewise pipeline produced heuristic critiques that were closer to expert references, more diverse, and better balanced between global and local concerns.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Criticmate's interface does not visualize the diffs of logged YAML snapshots or the impact of individual edits for end users.

“the interface does not visualize these diffs or the impact of individual edits for end users”

May combine two claims.

Performance remains imperfect: UI-element overlap is not perfect for all methods, and failures still arise when tasks are ambiguous or UIs are visually underspecified.

“Yet performance remains imperfect: UI-element overlap is not perfect for all methods, and failures still arise when tasks are ambiguous or UIs are visually underspecified.”

May combine two claims.

Offline, Criticmate's stagewise pipeline produces heuristic critiques closer to expert references than zero-shot and few-shot baselines, achieving higher semantic similarity and coverage, a more balanced mix of global and local issues, and more diverse comments.

“Offline, Criticmate's stagewise pipeline produces heuristic critiques that are closer to expert references than zero-shot and fewshot baselines: it achieves higher semantic similarity and coverage, a more balanced mix of global and local issues, and more diverse comments (Figures 9, 10, Tables 1, 2).”

May combine two claims.

With Criticmate, both participants and external experts rated single-screen heuristic critiques as more consistent and concrete than those from a single-pass baseline, and experts also judged them more helpful and clearer.

“With Criticmate, both participants and external experts rated single-screen heuristic critiques as more consistent and concrete than those from a single-pass baseline, and experts also judged them more helpful and clearer (Table 3, Figure 12).”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Criticmate produces text-only critiques and does not integrate visual annotations, overlays, or direct connections to existing design tools.

“Criticmate produces text-only critiques; it does not yet integrate visual annotations, overlays, or direct connections to existing design tools.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Several experts noted that a full staged workflow can feel heavy for trivial screens, highlighting a depth-speed trade-off and suggesting a need for adaptive staging.

“At the same time, several experts noted that a full staged workflow can feel heavy for trivial screens, highlighting a depthspeed trade-off and suggesting a need for adaptive staging.”

May combine two claims.

When evaluation is treated as a single output, neither the AI nor the human is well supported: the model lacks intermediate scaffolding and the human lacks clear entry points for understanding and intervention.

“When evaluation is treated as a single output, neither the AI nor the human is well supported: the model lacks intermediate scaffolding to reason effectively, and the human lacks clear entry points for understanding and intervention.”

May combine two claims.

By giving designers leverage over intermediate representations, Criticmate scaffolds the AI's reasoning, supports transparent and contestable feedback, and helps mitigate automation bias.

“By giving designers leverage over these intermediate representations, Criticmate scaffolds the AI's reasoning, supports transparent and contestable feedback, and helps mitigate automation bias.”

Probably not a substantial claim: may not pass S4 (based).

Practitioners frequently face a trade-off between shallow, checklist-style reviews of many screens and deep inspections of only a few, leaving many design decisions under-examined.

“Practitioners frequently face a trade-off between shallow, checklist-style reviews of many screens and deep inspections of only a few, leaving many design decisions under-examined.”

May combine two claims.

Current AI UI evaluation tools remain limited to single-pass evaluations or iterative prompting, producing static heuristic feedback, cumbersome revisions, and risks of automation bias.

“Current tools remain limited to single-pass evaluations or iterative prompting, producing static heuristic feedback, cumbersome revisions, and risks of automation bias.”

Classified as probably not a substantial claim.

The resulting section-component map serves as the shared perceptual grounding for the Comprehension stage.

“The resulting section-component map serves as the shared perceptual grounding for the Comprehension stage.”

• AssumptionRests on an assumption: Designers can accurately validate and correct the AI's intermediate structural representations.

Classified as probably not a substantial claim. May combine two claims.

The Perception stage mirrors the global-to-local scanning process observed in the formative study and exposes the model's recognition results as explicit, editable artifacts that designers can validate before higher-level reasoning begins.

“This stage mirrors the global-to-local scanning process observed in our formative study and exposes the model's recognition results as explicit, editable artifacts that designers can validate before higher-level reasoning begins.”

• AssumptionRests on an assumption: Designers can accurately validate and correct the AI's intermediate structural representations.

Classified as probably not a substantial claim.

These cases illustrate residual grounding errors that motivate keeping humans in the loop.

“These cases illustrate residual grounding errors that motivate keeping humans in the loop.”

• AssumptionRests on an assumption: The LLM judge for UI-element overlap validly determines whether element sets overlap.

Probably not a substantial claim: may not pass S3 (load-bearing).

Order of collaboration modes was counterbalanced within each expertise group.

“Order of collaboration modes was counterbalanced within each expertise group.”

May combine two claims.

CM-Full achieves a more balanced mix of global and local critique within this single-screen setting, supporting both structural and element-level reasoning in downstream human-AI co-critique.

“CM-Full therefore achieves a more balanced mix of global and local critique within this single-screen setting, supporting both structural (layout, hierarchy) and element-level (labels, affordances) reasoning in downstream human-AI co-critique.”

Classified as probably not a substantial claim.

Situation Awareness theory conceptualizes reasoning as a progression through three stages: Perception, Comprehension, and Projection.

“Situation Awareness (SA) theory conceptualizes reasoning as a progression through three stages: Perception (noticing elements), Comprehension (interpreting their meaning), and Projection (anticipating consequences) [21, 22].”

• cites [21] [21] Mica R Endsley. 1995. Measurement of Situation Awareness in Dynamic Systems. Human Factors 37, 1 (1995), 65-84. - sourceNot checked — no full text of the source
• cites [22] [22] Mica R Endsley. 2017. Toward a Theory of Situation Awareness in Dynamic Systems. In Situational Awareness . Routledge, 9-42. - sourceNot checked — no full text of the source

May combine two claims.

Criticmate's design is grounded in the formative study, which showed that designers naturally follow SA stages, integrate global and local perspectives, and inject domain knowledge more heavily in later phases, while current AI tools rarely structure support around these processes.

“Its design is grounded in our formative study, which showed that designers naturally follow Situation Awareness (SA) stages, integrate global and local perspectives, and inject domain knowledge more heavily in later phases, while current AI tools rarely structure support around these processes.”

• AssumptionRests on an assumption: ChatGPT-4o-augmented critique is representative of current AI systems for UI evaluation.

May combine two claims.

When SA stages are disrupted, people tend to fixate and explore less effectively; conversely, scaffolding the cycle from perception through comprehension to projection can broaden search and improve outcomes.

“Prior research shows that when these stages are disrupted, people tend to fi xate and explore less effectively; conversely, scaffolding the cycle from perception through comprehension to projection can broaden search and improve outcomes [43, 46, 76].”

• cites [43] [43] Dongwook Hwang, Byunghyun Choi, Soonho Bae, and Woojin Park. 2021. Mitigating Design Fixation: A Visualization Tool for Enhancing Situation Awareness. Journal of Mechanical Design 143, 6 (2021). - sourceNot checked — no full text of the source
• cites [46] [46] David G Jansson and Steven M Smith. 1991. Design Fixation. Design Studies 12, 1 (1991), 3-11. - sourceNot checked — no full text of the source
• cites [76] [76] A Terry Purcell and John S Gero. 1996. Design and Other Types of Fixation. Design Studies 17, 4 (1996), 363-383. - sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

These cases are evidence of the inherent difficulty of reasoning about poorly specified or visually underspecified interfaces in a single-screen setting, and such screens are likely challenging even for human experts.

“We interpret these cases as evidence of the inherent difficulty of reasoning about poorly specified or visually underspecified interfaces in a single-screen setting, and note that such screens are likely challenging even for human experts.”

May combine two claims.

Models generate more comments than experts and thus incur more unmatched outputs.

“Recall@0.5 is consistently higher than Precision@0.5 (e.g., 0.800 vs. 0.542 for CM-Full), reflecting that models generate more comments than experts and thus incur more unmatched outputs.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The mapping between UIs and modes was counterbalanced so that every system-UI pairing was observed across participants.

“Each participant evaluated two UIs, and the mapping between UIs and modes was counterbalanced so that every system-UI pairing was observed across participants.”

Classified as probably not a substantial claim. May combine two claims.

Three researchers independently coded transcripts and iteratively refined the coding scheme until reaching consensus.

“Following thematic analysis guidelines [9], three researchers independently coded transcripts and iteratively refined the scheme until reaching consensus.”

• cites [9] [9] Virginia Braun and Victoria Clarke. 2006. Using Thematic Analysis in Psychology. Qualitative Research in Psychology 3, 2 (2006), 77-101. - sourceNot checked — no full text of the source

May combine two claims.

Most AI UI evaluation systems expose only static outputs, requiring post-hoc reprompting for refinement, which restricts fine-grained intervention and increases the risk of automation bias.

“From the human side, most systems expose only static outputs, requiring post-hoc reprompting for refinement. This restricts fi ne-grained intervention and increases the risk of automation bias.”

May combine two claims.

The lack of significant difference in self-rated helpfulness suggests a ceiling effect: participants felt that either system provided usable feedback.

“suggesting a ceiling effect: participants felt that either system provided usable feedback”

Classified as probably not a substantial claim. May combine two claims.

All models were estimated via restricted maximum likelihood (REML), with p-values and denominator degrees of freedom using Satterthwaite's approximation and Holm-Bonferroni correction within each family of related outcomes.

“All models were estimated via restricted maximum likelihood (REML). We obtained 𝑝 -values and denominator degrees of freedom using Satterthwaite's approximation and applied Holm-Bonferroni correction within each family of related outcomes.”

May combine two claims.

Aligning the decomposition of reasoning with human cognitive stages can be as impactful as adding more data, and process-oriented prompting can make models more expert-like in their standalone critiques.

“This suggests that aligning the decomposition of reasoning with human cognitive stages can be as impactful as adding more data: process-oriented prompting can make models more 'expert-like' in their standalone critiques.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Residual diagnostics did not indicate substantial assumption violations.

“Residual diagnostics did not indicate substantial assumption violations.”

May combine two claims.

Effective collaboration depends not only on the quality of AI outputs, but also on how reasoning is organized and exposed.

“Effective collaboration depends not only on the quality of AI outputs, but also on how reasoning is organized and exposed.”

Classified as probably not a substantial claim. May combine two claims.

For external assessment, three professional designers rated each critique independently and blindly; artifacts were anonymized and randomized, and system labels were removed.

“For external assessment, three professional designers ( J ) rated each critique independently and blindly: artifacts were anonymized and randomized, and system labels were removed.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

This interaction pattern mirrors current use of general-purpose LLMs.

“This interaction pattern mirrors current use of general-purpose LLMs, but it often does not align with the cognitive structure we observed in our formative study”

Probably not a substantial claim: may not pass S3 (load-bearing).

Qualitative data were analyzed with thematic analysis and iterative codebook refinement by three researchers.

“Qualitative data (open-ended responses and logs) were analyzed with thematic analysis and iterative codebook refinement by three researchers.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The single-pass plus revision pattern is common in practice.

“This 'single-pass plus revision' pattern is common in practice, but it is not the only plausible comparison.”

• AssumptionRests on an assumption: The single-pass plus revision pattern is common in practice.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The study procedures involved minimal risk (primarily time and potential fatigue), and session length was capped to mitigate burden.

“The procedures involved minimal risk (primarily time and potential fatigue), and session length was capped to mitigate burden.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

Situation Awareness (SA) provided a useful scaffold for structuring Criticmate, but it is ultimately only one lens on critique.

“While Situation Awareness (SA) provided a useful scaffold for structuring Criticmate, it is ultimately only one lens on critique.”

Probably not a substantial claim: may not pass S1 (asserted), S2 (checkable), S3 (load-bearing), S4 (based).

A larger and more diverse panel could surface additional perspectives on what constitutes good critique.

“yet a larger and more diverse panel (e.g., educators, accessibility specialists, product stakeholders) could surface additional perspectives on what constitutes 'good' critique.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Generalization to other layouts, domains, device types, and types of critique remains open.

“generalization to other layouts, domains, device types, and types of critique (e.g., conceptual, socio-technical, or longitudinal critique) remains open.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Because Projection uses a nonzero sampling temperature, re-running the same stage after a different sequence of edits may not be strictly deterministic.

“Projection currently uses a nonzero sampling temperature to support ideation, so re-running the same stage after a different sequence of edits may not be strictly deterministic.”

May combine two claims.

The contribution is not only a stronger automated evaluator, but a concrete instantiation of human-AI co-critique for UI evaluation.

“Our contribution is therefore not only a stronger automated evaluator, but a concrete instantiation of human-AI co-critique for UI evaluation.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The study did not examine how stagewise co-critique affects iteration processes, final product quality, or alignment with usability, creativity, and organizational goals.

“we did not examine how stagewise co-critique affects iteration processes, fi nal product quality, or alignment with usability, creativity, and organizational goals.”

Classified as probably not a substantial claim.

The study deliberately scoped its paradigm and implementation to single-screen, heuristic-style evaluation of mobile UIs.

“We deliberately scoped our paradigm and implementation to single-screen, heuristic-style evaluation of mobile UIs.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Other perspectives-distributed cognition, activity theory, reflection-in-action, and systems thinking-highlight collaboration across actors and artifacts, cultural and organizational constraints, and ecosystem-level effects.

“Other perspectives-distributed cognition [42], activity theory [50, 66], reflection-in-action [81], and systems thinking [62, 82]-highlight collaboration across actors and artifacts, cultural and organizational constraints, and ecosystem-level effects.”

• cites [42] [42] Edwin Hutchins. 1996. Cognition in the wild . The MIT Press. - sourceNot checked — no full text of the source
• cites [50] [50] Victor Kaptelinin and Bonnie A Nardi. 2006. Acting with Technology: Activity Theory and Interaction Design . The MIT Press. - sourceNot checked — no full text of the source
• cites [66] [66] Bonnie A Nardi. 1996. Context and Consciousness: Activity Theory and HumanComputer Interaction . The MIT Press. -Not checked yet

  Not checked in the free run.

• cites [81] [81] Donald A Schön. 1985. The Design Studio: An Exploration of Its Traditions and Potential . RIBA Publications. -Not checked yet

  Not checked in the free run.

May combine two claims.

Rather than asking whether designers should trust an automated judge, Criticmate makes intermediate reasoning visible and editable, allowing users to decide when to rely on or override AI suggestions.

“Rather than asking whether designers should trust an automated judge, Criticmate makes intermediate reasoning visible and editable, allowing users to decide when to rely on or override AI suggestions.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Errors introduced in the Perception and Comprehension stages of Criticmate's pipeline can propagate into the Projection stage.

“errors introduced in Perception and Comprehension can propagate into Projection”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

No personally identifiable information was collected; all sessions were conducted remotely, recordings were stored securely, and transcripts were anonymized prior to analysis.

“No personally identifiable information was collected; all sessions were conducted remotely, recordings were stored securely, and transcripts were anonymized prior to analysis.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Criticmate's text-only output limits how easily feedback can be inspected, discussed, and acted upon in professional workflows.

“This limits how easily feedback can be inspected, discussed, and acted upon in professional workflows”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Participants were recruited on a voluntary basis through professional and academic channels, provided informed consent, and received monetary compensation.

“participants were recruited on a voluntary basis through professional and academic channels, provided informed consent, and received monetary compensation.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

SA foregrounds stepwise perception and reasoning, but gives less attention to social negotiation, historical context, and iterative reframing in design work.

“It foregrounds stepwise perception and reasoning, but gives less attention to social negotiation, historical context, and iterative reframing in design work.”

Probably not a substantial claim: may not pass S4 (based).

The baseline condition approximates how designers currently use general-purpose LLMs for critique: request a one-shot list of issues for a screen, then refine it with natural-language revision prompts.

“Our baseline condition approximates how designers currently use general-purpose LLMs for critique: request a one-shot list of issues for a screen, then refine it with natural-language revision prompts.”

• AssumptionRests on an assumption: The baseline condition approximates how designers currently use general-purpose LLMs for critique.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The study did not test baselines that expose only an editable issue list without SA-style staging, or that provide different granularities of intermediate representations.

“We did not, for example, test baselines that expose only an editable issue list without SA-style staging, or that provide different granularities of intermediate representations.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The participant pool included only professional designers and intermediates; complete novices, non-design stakeholders, and classroom settings were not studied.

“Our participant pool also included only professional designers and intermediates; complete novices, non-design stakeholders, and classroom settings were not studied, even though several participants explicitly highlighted Criticmate's potential as a learning tool.”