ABScribe: Rapid Exploration & Organization of Multiple Writing Variations in Human-AI Co-Writing Tasks using Large Language Models
Open accessThe DOI matches its Crossref record; 76 of the 76 references it cites were found in a registry or an index; 8 of the 99 sources its claims cite were checked for support.
Abstract
Exploring alternative ideas by rewriting text is integral to the writing process. State-of-the-art Large Language Models (LLMs) can simplify writing variation generation. However, current interfaces pose challenges for simultaneous consideration of multiple variations: creating new variations without overwriting text can be difficult, and pasting them sequentially can clutter documents, increasing workload and disrupting writers’ flow. To tackle this, we present ABScribe, an interface that supports rapid, yet visually structured, exploration and organization of writing variations in human-AI co-writing tasks. With ABScribe, users can swiftly modify variations using LLM prompts, which are auto-converted into reusable buttons. Variations are stored adjacently within text fields for rapid in-place comparisons using mouse-over interactions on a popup toolbar. Our user study with 12 writers shows that ABScribe significantly reduces task workload (d = 1.20, p < 0.001), enhances user perceptions of the revision process (d = 2.41, p < 0.001) compared to a popular baseline workflow, and provides insights into how writers explore variations using LLMs.
Overview
CoChat Summary
TL;DR
ABScribe is an LLM-based writing interface that stores multiple text variations in Variation Fields with a Popup Toolbar, Variation Sidebar, AI Modifiers, and AI Drafter. In a within-subjects study with 12 writers, it reduced subjective task workload and improved perceptions of the revision process compared with a chat-based baseline.
Key findings
- ABScribe significantly reduced NASA-TLX subjective task workload versus the baseline (𝑑 = 1.20, 𝑝 < 0.001).
- ABScribe significantly increased summed Likert agreement on the efficacy of the revision process versus the baseline (𝑑 = 2.41, 𝑝 < 0.001).
- Qualitative themes: non-linear variation storage reduced clutter and pressure to commit early; the Popup Toolbar supported in-context comparison; AI Modifiers made prompts reusable and encouraged more imperative, generalizable prompt writing.
- Trade-offs: two participants perceived clutter as unproblematic or beneficial, and some preferred linear side-by-side comparison for large segments, though the Variation Sidebar was seen as mitigating this.
Methods
- Within-subjects controlled evaluation with 12 writers (5 women, 7 men, aged 18–34), compensated 30 CAD; each completed two guided tasks (LinkedIn post and email to a professor) randomly paired with counterbalanced ABScribe versus baseline.
- Participants used AI to explore eight variations of three text segments (24 variations total) per task; the baseline was a GPT-4 rich-text editor with a chat-based assistant and direct AI text insertion.
- Measures included weighted NASA-TLX and 7-point Likert scales; analysis used pairwise one-sided t-tests and reflexive thematic analysis of interview transcripts and task observation notes.
Limitations
- Stated: the baseline was constructed rather than an existing system, which may have affected ecological validity; future studies could compare ABScribe with real-world AI-assisted writing workflows.
- Stated: external validity is limited to English writers and a single 1.5-hour session with two guided tasks; AI Drafter and AI Modifiers carry hallucination risk; no logging metrics were collected.
- Evident: the small sample (N = 12) and lab-based tasks limit generalizability, and the reported trade-offs from two participants suggest individual differences in preference for clutter and linear editing.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures









Verification
Claims the paper makes
5 Valid · 3 Not valid · 43 Not checked · 57 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Advanced Large Language Models (LLMs) such as ChatGPT [68], GPT-4 [69], PaLM 2 [32], LLaMA 2 [90], and Gemini [73] can enable writers to generate multiple variations of text via natural language prompting [8,15,26,66], potentially reducing the workload of generating text variations.”
§ ABScribe: Rapid Exploration & Organization of Multiple Writing Variations in Human-AI Co-Writing Tasks using Large Language Models > 1 INTRODUCTION
What the sources say
- Not checked yetcites [8]
- Not checked yetcites [15]
- Not checked yetcites [26] matched to “Sparks: Inspiration for Science Writing using Language Models” (2022) source
- Partly supportedcites [66] matched to “Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals” (2023) sourceExperimental — accuracy not yet measured
Our copy clearly supports the first half of the attributed point: LLMs prompted in natural language yield multiple text variations for a writer to draw on — §3.2 (p. ~3) states that 'the current state-of-the-art method for longer text generation incorporates a human-in-the-loop who selects from various generations sampled from the model', §3.6 lists 'Generate a new suggestion (i.e., run the LLM again with the same prompt)' as a standard writer operation, and Figure 2's caption describes several title outputs generated for different random seeds; §1 frames LLMs as 'an augmentative tool for human writers'. What the paper does not state is the workload-reduction consequence attached to this citation: the closest material is qualitative (§5.2.4, p8: 'getting words on the page. This helps with that step'; §5.2.2 on generating alternative choices), and §5.1.2 (p1) records that running multiple generations gave 'lots of material' to 'cherrypick' but also 'requires more massaging on my end'. The paper also studies Chinchilla 70B via Dramatron rather than the chat systems the citing sentence names, so the specific model list is not this source's. (LLM, conf=0.68)
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“Insights from HCI and traditional design practice suggest that simultaneous consideration of multiple (at least 5) variations can lead to better ideas [89] and avoid fixation [10,34,44,89].”
§ ABScribe: Rapid Exploration & Organization of Multiple Writing Variations in Human-AI Co-Writing Tasks using Large Language Models > 1 INTRODUCTION
What the sources say
- Not checked yetcites [10]
- Not checked yetcites [34]
- Could not verifycites [44] matched to “Survey of Hallucination in Natural Language Generation” (2023) sourceExperimental — accuracy not yet measured
Our copy of Ji et al. (2023) is a survey of hallucination in NLG and contains no passage on design/HCI fixation or on considering multiple variations simultaneously to avoid it (searched: 'design fixation / multiple variations', 'simultaneous consideration', 'generation diversity avoiding convergence'). The closest material concerns model-side diversity — Sec. 3.2 ('Erroneous decoding') notes that decoding strategies improving generation diversity, e.g. top-k sampling, are positively correlated with increased hallucination, and Sec. 5.2.2 ('Planning/Sketching') speaks of striking a balance between faithfulness and diversity — but neither addresses the human ideation claim the citing sentence attributes here. The claim's point is simply not covered by this source, so it cannot be confirmed or contradicted from our copy. (LLM, conf=0.70)
- Not checked yetcites [89]
hedgedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“HCI and traditional design practices encourage the parallel exploration of multiple variations to help avoid fixation on a singular idea [30,44], to reduce the chances of eliminating rough but innovative ideas due to premature evaluation [10,34], and to make us less prone to inflated subjective appraisals by giving us an opportunity to critically assess ideas in relation to each other [10,89].”
§ ABScribe: Rapid Exploration & Organization of Multiple Writing Variations in Human-AI Co-Writing Tasks using Large Language Models > 2.1 Exploring Multiple Variations
What the sources say
- Not checked yetcites [30] matched to “Avoiding Design Fixation: Transformation and Abstraction in Mapping from Source to Target” (2011) source
- Could not verifycites [44] matched to “Survey of Hallucination in Natural Language Generation” (2023) sourceExperimental — accuracy not yet measured
Our copy of Ji et al.'s 'Survey of Hallucination in Natural Language Generation' is a survey of hallucination definitions, metrics and mitigation across summarization, dialogue, GQA, data-to-text, NMT and LLMs; it contains no discussion of HCI or traditional design practice, parallel exploration of variations, or fixation on a single idea. Searched: 'HCI design practice parallel exploration of multiple variations avoid fixation', 'human-computer interaction design disciplines brainstorming divergent thinking', 'design practice fixation premature evaluation creative ideas multiple prototypes', and 'human factors interaction design diversity of generated candidates comparing alternatives' — the closest hits are only metaphorical neighbours (e.g. Planning/Sketching in 5.2.2, ensemble/debate and contrast-candidate selection in 13.3.2 and 7.3), none of which attribute anything to HCI or design methodology. Since no passage addresses the claim's point, it can be neither confirmed nor contradicted from this source. (LLM, conf=0.72)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“Advanced Large Language Models (LLMs) such as ChatGPT [68], GPT-4 [69], PaLM 2 [32], LLaMA 2 [90], and Gemini [73] can enable writers to generate multiple variations of text via natural language prompting [8,15,26,66], potentially reducing the workload of generating text variations.”
§ ABScribe: Rapid Exploration & Organization of Multiple Writing Variations in Human-AI Co-Writing Tasks using Large Language Models > 1 INTRODUCTION
What the sources say
- Not checked yetcites [8]
- Not checked yetcites [15]
- Not checked yetcites [26] matched to “Sparks: Inspiration for Science Writing using Language Models” (2022) source
- Supportedcites [66] matched to “Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals” (2023) source
Our copy of Mirowski et al. (2023) treats LLM-assisted writers generating repeated text variants as a core mechanic: the Introduction's contribution list states authors 'can generate multiple responses, edit, and then regenerate at various levels of the story hierarchy', and Section 3.6 (Interactive Writing with Dramatron) lists 'Generate a new suggestion (i.e., run the LLM again with the same prompt)' as an operation the writer may perform as many times as desired. Section 3.1 explains the mechanism (probabilistic sampling, different random seeds yielding different samples; Fig. 2 shows several title outputs for different seeds), and Section 5.1.2 reports participant p1 curating outputs from 3-4 generations for 'lots of material' and 'more interpretations'. The natural-language-prompting aspect is covered by Section 3.5, which describes prompt prefixes combined with user-supplied inputs (e.g., the log line) as the way users control the LLM. (LLM, conf=0.85)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“F3: AI Integration-ABScribe nudges writers toward an imperative LLM prompt writing style in contrast to the conversational style in Baseline.”
§ ABScribe: Rapid Exploration & Organization of Multiple Writing Variations in Human-AI Co-Writing Tasks using Large Language Models > 5.1 RQ1: User Perceptions on the AI-Assisted Revision Process
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
35 more claims
Create a free account to see all 108 claims and to verify the 91 references still open.
- Probably not a substantial claim. 60
- It may state two separate claims as one. 21
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Researchers in HCI have worked extensively to investigate these dimensions, producing studies characterizing user values, social dynamics, and professional contexts [9,39,53,63,77,105] that influence ownership preferences and how they shape users' attitudes toward AI assistance.”
From Co-Writing with AI, on Human Terms: Aligning Research with User Demands Across the Writing Process (2025) · cites this paper as [105]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Rezwana et al. 's [105] work highlighted that ownership perceptions depend on both contribution levels and leadership in the writing process, suggesting that interaction designs that maximize user agency can enhance ownership.”
From Co-Writing with AI, on Human Terms: Aligning Research with User Demands Across the Writing Process (2025) · cites this paper as [105]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ ABScribe: Rapid Exploration & Organization of Multiple Writing Variations in Human-AI Co-Writing Tasks using Large Language Models > 2.2 Working with Multiple Variations from Large Language Models
“We also incorporate Kim et al. 's concepts of object-oriented interactions with LLMs [37]”
§ ABScribe: Rapid Exploration & Organization of Multiple Writing Variations in Human-AI Co-Writing Tasks using Large Language Models > References
“37. Han L. Han, Miguel A. Renom, Wendy E. Mackay, and Michel Beaudouin-Lafon. 2020. Textlets: Supporting Constraints and Consistency in Text Documents.”
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173 · cited 1×
“…31, 47, 48, 50, 56, 67, 82, 84, 97] on interfaces for writing with LLMs in tandem with rapid advancements in AI writing capabilities using LLMs [8, 18, 66, 76, 91], examining various aspects of the AI-assisted writing process including prompting [28, 94], text suggestions [9, 16], creative…”
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Survey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023) · ACM Computing Surveys · doi:10.1145/3571730 · Cited by 5,025
Survey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y., Chen, D., Dai, W., et al. (2022) · arXiv · doi:10.48550/arxiv.2202.03629 · Cited by 4,981
NASA-Task Load Index (NASA-TLX); 20 Years Later
Hart, S. G. (2006) · Proceedings of the Human Factors and Ergonomics Society Annual Meeting · doi:10.1177/154193120605000909 · Cited by 4,297 · cited 1×
“To assess subjective task-workload, we used the widely used NASA-TLX [39] procedure with weighting.”
Recent Advances in Natural Language Processing via Large Pre-trained Language Models: A Survey
Min, B., Ross, H., Sulem, E., Veyseh, A. P. B., Nguyen, T. H., Sainz, O., Agirre, E., Heintz, I., & Roth, D. (2024) · ACM Computing Surveys · doi:10.1145/3605943 · Cited by 1,387 · cited 2×
“Advanced Large Language Models (LLMs) such as ChatGPT [68], GPT-4 [69], PaLM 2 [32], LLaMA 2 [90], and Gemini[73] can enable writers to generate multiple variations of text via natural language prompting [8, 15, 26, 66], potentially reducing the workload of generating text variations.”
Why Johnny Can’t Prompt: How Non-AI Experts Try (and Fail) to Design LLM Prompts
Zamfirescu-Pereira, J. D., Wong, R. Y., Hartmann, B., & Yang, Q. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581388 · Cited by 1,274 · cited 2×
“Recent HCI studies on LLM-based tool design have mainly focused on prompt engineering [6, 16, 46, 55, 94, 98] and exploring the generative capabilities of LLMs [50, 58, 63, 67, 97].”
A cognitive process theory of writing
Flower, L., & Hayes, J. R. (1981) · College Composition & Communication · doi:10.58680/ccc198115885 · Cited by 1,025 · cited 1×
“…of facilitating the swift exploration of multiple writing variations using LLMs: the role and nuances of revision within the writing process [23, 25, 38, 57, 83, 87]; HCI design philosophies that emphasize the consideration of multiple ideas before evaluation [19, 34, 89]; principles on…”
Prompt Engineering with ChatGPT: A Guide for Academic Writers
Giray, L. (2023) · Annals of Biomedical Engineering · doi:10.1007/s10439-023-03272-4 · Cited by 776 · cited 2×
“There has been an explosion of recent work [4, 9, 14, 16, 26–28, 31, 47, 48, 50, 56, 67, 82, 84, 97] on interfaces for writing with LLMs in tandem with rapid advancements in AI writing capabilities using LLMs [8, 18, 66, 76, 91], examining various aspects of the AI-assisted writing process…”
Revision Strategies of Student Writers and Experienced Adult Writers
Sommers, N. (1980) · College Composition & Communication · doi:10.2307/356588 · Cited by 749 · cited 9×
“This is problematic because when we turn to research on the revision and writing process, we see that revision goes beyond surface level edits [23], encompassing deeper writing subprocesses such as revising and evaluating ideas [24] and meaning discovery [87].”
InfluentialDesign Guidelines for Prompt Engineering Text-to-Image Generative Models
Liu, V., & Chilton, L. B. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3501825 · Cited by 639 · cited 2×
“Recent HCI studies on LLM-based tool design have mainly focused on prompt engineering [6, 16, 46, 55, 94, 98] and exploring the generative capabilities of LLMs [50, 58, 63, 67, 97].”
Design Guidelines for Prompt Engineering Text-to-Image Generative Models
Liu, V., & Chilton, L. B. (2021) · arXiv · doi:10.48550/arxiv.2109.06977 · Cited by 635 · cited 2×
“Recent HCI studies on LLM-based tool design have mainly focused on prompt engineering [6, 16, 46, 55, 94, 98] and exploring the generative capabilities of LLMs [50, 58, 63, 67, 97].”
Usability evaluation considered harmful (some of the time)
Greenberg, S., & Buxton, B. (2008) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1357054.1357074 · Cited by 598 · cited 4×
“Insights from HCI and traditional design practice suggest that simultaneous consideration of multiple (at least 5) variations can lead to better ideas [89] and avoid fixation [10, 34, 44, 89].”
InfluentialViewpoint: when will AI exceed human performance? evidence from AI experts
Grace, K., Salvatier, J., Dafoe, A., Zhang, B., & Evans, O. (2018) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.11222 · Cited by 579
Faigley, L., & Witte, S. (1981) · College Composition & Communication · doi:10.2307/356602 · Cited by 481 · cited 1×
“…in repeated cycles, with 2 Link to GiHub Repository: https://github.com/mohireza/abscribe/ multiple objectives including finding the form or shape of an ar-gument [87], experimenting with vocabulary and style [43], and going back and forth between multiple composing activities during revision [22].”
Paper Prototyping: The Fast and Easy Way to Design and Refine User Interfaces
Lancaster, A. (2004) · IEEE Transactions on Professional Communication · doi:10.1109/tpc.2004.837973 · Cited by 410
Wordcraft: Story Writing With Large Language Models
Yuan, A., Coenen, A., Reif, E., & Ippolito, D. (2022) · 27th International Conference on Intelligent User Interfaces · doi:10.1145/3490099.3511105 · Cited by 374
Parallel prototyping leads to better design results, more divergence, and increased self-efficacy
Dow, S. P., Glassco, A., Kass, J., Schwarz, M., Schwartz, D. L., & Klemmer, S. R. (2010) · ACM Transactions on Computer-Human Interaction · doi:10.1145/1879831.1879836 · Cited by 367 · cited 1×
“…within the writing process [23, 25, 38, 57, 83, 87]; HCI design philosophies that emphasize the consideration of multiple ideas before evaluation [19, 34, 89]; principles on reification and reuse for designing visual interfaces [3, 37]; and the latest research on utilizing LLMs in writing…”
Lee, M., Liang, P., & Yang, Q. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3502030 · Cited by 345 · cited 6×
“There has been an explosion of recent work [4, 9, 14, 16, 26–28, 31, 47, 48, 50, 56, 67, 82, 84, 97] on interfaces for writing with LLMs in tandem with rapid advancements in AI writing capabilities using LLMs [8, 18, 66, 76, 91], examining various aspects of the AI-assisted writing process…”
InfluentialLarge Language Models Are Human-Level Prompt Engineers
Zhou, Y., Muresanu, A. I., Han, Z., Paster, K., Pitis, S., Chan, H., & Ba, J. (2022) · arXiv · doi:10.48550/arxiv.2211.01910 · Cited by 330 · cited 1×
“As LLMs increasingly blur the boundaries between human and AI writing [33, 100], the HCI community needs to explore interfaces that improve writers’ interactions with AI.”
How We Write: Writing as Creative Design
Sharples, M. (2002) · doi:10.4324/9780203019900 · Cited by 303
Design, implementation, and evaluation of a Revision Control System
Tichy, W. F. (1982) · Purdue e-Pubs (Purdue University System) · doi:10.5555/800254.807748 · Cited by 300
The Creativity of Text-to-Image Generation
Oppenlaender, J. (2022) · Proceedings of the 25th International Academic Mindtrek Conference · doi:10.1145/3569219.3569352 · Cited by 290 · cited 1×
“For example, an application for editing AI-generated images could implement an AI Modifier which accepts textual instructions and generates corresponding image variations, which are then organized within Variation Fields, contributing to HCI research on text-to-image AI editors [6, 54, 55, 70, 93].”
When Will AI Exceed Human Performance? Evidence from AI Experts
Grace, K., Salvatier, J., Dafoe, A., Zhang, B., & Evans, O. (2017) · arXiv · doi:10.48550/arxiv.1705.08807 · Cited by 285 · cited 1×
“As LLMs increasingly blur the boundaries between human and AI writing [33, 100], the HCI community needs to explore interfaces that improve writers’ interactions with AI.”
Writing Down the Bones: Freeing the Writer Within
Goldberg, N. (1998) · Medical Entomology and Zoology · Cited by 274
ChatGPT: these are not hallucinations – they’re fabrications and falsifications
Emsley, R. (2023) · Schizophrenia · doi:10.1038/s41537-023-00379-4 · Cited by 258 · cited 1×
“A potential risk with any generative AI tool is that the AI may produce hallucinated or confabulated content [21, 45, 79, 85].”
Mirowski, P., Mathewson, K. W., Pittman, J., & Evans, R. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581225 · Cited by 240 · cited 2×
“Recent HCI studies on LLM-based tool design have mainly focused on prompt engineering [6, 16, 46, 55, 94, 98] and exploring the generative capabilities of LLMs [50, 58, 63, 67, 97].”
High-Fidelity or Low-Fidelity, Paper or Computer? Choosing Attributes when Testing Web Prototypes
Walker, M., Takayama, L., & Landay, J. A. (2002) · Proceedings of the Human Factors and Ergonomics Society Annual Meeting · doi:10.1177/154193120204600513 · Cited by 235 · cited 1×
“…Cognitive Walkthroughs [53, 59]: We used informal cognitive walkthroughs to rapidly iter-ate on our sketches and identify which potential solutions were intuitive; (iii) Web-based Prototyping [92]: We then developed functional versions of our intuitive design solutions in a high-fidelity prototype.”
Five-Point Likert Items: t test versus Mann-Whitney-Wilcoxon (Addendum added October 2012)
de Winter, J. F. C., & Dodou, D. (2010) · University of Massachusetts Amherst · doi:10.7275/bj1p-ts64 · Cited by 231 · cited 1×
“T-test was shown to be robust for aggregated data of this kind [17].”
Getting the right design and the design right
Tohidi, M., Buxton, W., Baecker, R., & Sellen, A. (2006) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1124772.1124960 · Cited by 211 · cited 7×
“To get the right design [89] the lead author iterated through multiple versions of each interface element, and tested them with five pilot users during in-depth brainstorming and design-review sessions over six months.”
InfluentialSefelin, R., Tscheligi, M., & Giller, V. (2003) · CHI '03 extended abstracts on Human factors in computing systems - CHI '03 · doi:10.1145/765891.765986 · Cited by 209 · cited 1×
“The design process involved transitioning from low to high-fidelity prototyping via three overlapping processes: (i) Paper-Prototyping [80, 86]: We began by considering an in-place editing interface as a foundational concept, and sketched various design elements that could be integrated into this…”
The Writing Process and Process Writing
Seow, A. (2002) · Methodology in Language Teaching · doi:10.1017/cbo9780511667190.044 · Cited by 208
PromptChainer: Chaining Large Language Model Prompts through Visual Programming
Wu, T., Jiang, E., Donsbach, A., Gray, J., Molina, A., Terry, M., & Cai, C. J. (2022) · CHI Conference on Human Factors in Computing Systems Extended Abstracts · doi:10.1145/3491101.3519729 · Cited by 198 · cited 2×
“Recent HCI studies on LLM-based tool design have mainly focused on prompt engineering [6, 16, 46, 55, 94, 98] and exploring the generative capabilities of LLMs [50, 58, 63, 67, 97].”
Lewis, C., & Wharton, C. (1997) · Elsevier eBooks · doi:10.1016/b978-044481862-1.50096-0 · Cited by 188 · cited 2×
“The design process involved transitioning from low to high-fidelity prototyping via three overlapping processes: (i) Paper-Prototyping [80, 86]: We began by considering an in-place editing interface as a foundational concept, and sketched various design elements that could be integrated into this interface to enhance…”
Handbook of Research on Writing
(2009) · Routledge eBooks · doi:10.4324/9781410616470 · Cited by 176
Managing Messes in Computational Notebooks
Head, A., Hohman, F., Barik, T., Drucker, S. M., & DeLine, R. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300500 · Cited by 165 · cited 2×
“In our work, we contributed an ensemble of five interface elements that significantly enhance user perceptions on AI-assisted revision (RQ1) by supporting rapid exploration and organization of multiple writing variations without overwhelming users (RQ2).”
Sparks: Inspiration for Science Writing using Language Models
Gero, K. I., Liu, V., & Chilton, L. (2022) · Designing Interactive Systems Conference · doi:10.1145/3532106.3533533 · Cited by 161
PromptMaker: Prompt-based Prototyping with Large Language Models
Jiang, E., Olson, K., Toh, E., Molina, A., Donsbach, A., Terry, M., & Cai, C. J. (2022) · CHI Conference on Human Factors in Computing Systems Extended Abstracts · doi:10.1145/3491101.3503564 · Cited by 157
Reification, polymorphism and reuse: three principles for designing visual interfaces
Beaudouin-Lafon, M., & Mackay, W. E. (2000) · Proceedings of the working conference on Advanced visual interfaces · doi:10.1145/345513.345267 · Cited by 145
Artificial intelligence hallucinations
Salvagno, M., Taccone, F. S., & Gerli, A. G. (2023) · Critical Care · doi:10.1186/s13054-023-04473-y · Cited by 136
Rewriting: how to do things with texts
(2007) · Choice Reviews Online · doi:10.5860/choice.44-4300 · Cited by 136 · cited 1×
“…facilitating the swift exploration of multiple writing variations using LLMs: the role and nuances of revision within the writing process [23, 25, 38, 57, 83, 87]; HCI design philosophies that emphasize the consideration of multiple ideas before evaluation [19, 34, 89]; principles on reification…”
Social Dynamics of AI Support in Creative Writing
Gero, K. I., Long, T., & Chilton, L. B. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580782 · Cited by 129 · cited 1×
“…interfaces for writing with LLMs in tandem with rapid advancements in AI writing capabilities using LLMs [8, 18, 66, 76, 91], examining various aspects of the AI-assisted writing process including prompting [28, 94], text suggestions [9, 16], creative ideation [14, 26], and social dynamics [4, 27].”
Guidelines for designing writing prompts: Clarifications, caveats, and cautions
Kroll, B., & Reid, J. M. (1994) · Journal of Second Language Writing · doi:10.1016/1060-3743(94)90018-3 · Cited by 128 · cited 1×
“These interfaces are highly intuitive, and mimic conversational interactions between humans, but lack scaffolding for crafting prompts, which can be difficult for novice AI users [49, 98].”
Design Thinking and the Wicked Problem of Teaching Writing
Leverenz, C. S. (2014) · Computers and Composition · doi:10.1016/j.compcom.2014.07.001 · Cited by 120 · cited 1×
“In addition to the rich-body of work underscoring the important and complex role of revision in writing, researchers have explored the benefits of adopting design language in writing pedagogy, such as characterizing writing pedagogy as a wicked [78] design thinking problem [52, 75].”
RePrompt: Automatic Prompt Editing to Refine AI-Generative Art Towards Precise Expressions
Wang, Y., Shen, S., & Lim, B. Y. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581402 · Cited by 120 · cited 1×
“For example, an application for editing AI-generated images could implement an AI Modifier which accepts textual instructions and generates corresponding image variations, which are then organized within Variation Fields, contributing to HCI research on text-to-image AI editors [6, 54, 55, 70, 93].”
Where to Hide a Stolen Elephant: Leaps in Creative Writing with Multimodal Machine Intelligence
Singh, N., Bernal, G., Savchenko, D., & Glassman, E. L. (2023) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3511599 · Cited by 119 · cited 1×
“There has been an explosion of recent work [4, 9, 14, 16, 26–28, 31, 47, 48, 50, 56, 67, 82, 84, 97] on interfaces for writing with LLMs in tandem with rapid advancements in AI writing capabilities using LLMs [8, 18, 66, 76, 91], examining various aspects of the AI-assisted writing process…”
Biermann, O. C., Ma, N. F., & Yoon, D. (2022) · Designing Interactive Systems Conference · doi:10.1145/3532106.3533506 · Cited by 116
Writing Quality, Coherence, and Cohesion
McCulley, G. A. (1985) · Research in the Teaching of English · doi:10.58680/rte198515640 · Cited by 111 · cited 1×
“Maintaining text flow is crucial since writers need to engage with information processing tasks such as ensuring the document maintains cohesion, which requires matching segments to the surrounding text [60].”
Buschek, D., Zürn, M., & Eiband, M. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445372 · Cited by 108
Barrett, W. A., & Cheney, A. S. (2002) · ACM SIGGRAPH Conference (SIGGRAPH) · doi:10.1145/566570.566651 · Cited by 108 · cited 1×
“The Popup Toolbar could be adapted to non-linear layer management in image editors [1, 11], or applied in presentation software, allowing users to experiment with different headings and bullet points on a slide.”
Hallucination or Confabulation? Neuroanatomy as metaphor in Large Language Models
Smith, A. L., Greaves, F., & Panch, T. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000388 · Cited by 92
Understanding Iterative Revision from Human-Written Text
Du, W., Raheja, V., Kumar, D., Kim, Z. M., Lopez, M., & Kang, D. (2022) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2022.acl-long.250 · Cited by 92 · cited 1×
“The revision process is iterative [20, 23]: happens in repeated cycles, throughout the writing process, granular [62, 81]: happens at the word, sentence, or paragraph-level, and non-linear [13, 87]: requires constant reconsideration of potential variations of existing text throughout the passage.”
Dang, H., Goller, S., Lehmann, F., & Buschek, D. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580969 · Cited by 83 · cited 11×
“There has been an explosion of recent work [4, 9, 14, 16, 26–28, 31, 47, 48, 50, 56, 67, 82, 84, 97] on interfaces for writing with LLMs in tandem with rapid advancements in AI writing capabilities using LLMs [8, 18, 66, 76, 91], examining various aspects of the AI-assisted writing process…”
InfluentialLaMPost: Design and Evaluation of an AI-assisted Email Writing Prototype for Adults with Dyslexia
Goodman, S. M., Buehler, E., Clary, P., Coenen, A., Donsbach, A., Horne, T. N., Lahav, M., MacDonald, R., Michaels, R. B., Narayanan, A., et al. (2022) · Proceedings of the 24th International ACM SIGACCESS Conference on Computers and Accessibility · doi:10.1145/3517428.3544819 · Cited by 71 · cited 2×
“…for each participant. timing constraints of our study and served as realistic use-cases for LLMs, in alignment with recent HCI studies on human-AI co-writing [9, 31, 47], and commonly advertised applications of commercial AI-assisted writing tools, such as Grammarly Go, Respond-able, and Copy.ai.”
SAGA: Collaborative Storytelling with GPT-3
Shakeri, H., Neustaedter, C., & DiPaola, S. (2021) · Companion Publication of the 2021 Conference on Computer Supported Cooperative Work and Social Computing · doi:10.1145/3462204.3481771 · Cited by 66 · cited 1×
“There has been an explosion of recent work [4, 9, 14, 16, 26–28, 31, 47, 48, 50, 56, 67, 82, 84, 97] on interfaces for writing with LLMs in tandem with rapid advancements in AI writing capabilities using LLMs [8, 18, 66, 76, 91], examining various aspects of the AI-assisted writing process…”
Avoiding Design Fixation: Transformation and Abstraction in Mapping from Source to Target
GOLDSCHMIDT, G. (2011) · The Journal of Creative Behavior · doi:10.1002/j.2162-6057.2011.tb01088.x · Cited by 64 · cited 1×
“HCI and traditional design practices encourage the parallel exploration of multiple variations to help avoid fixation on a singular idea [30, 44], to reduce the chances of eliminating rough but innovative ideas due to premature evaluation [10, 34], and to make us less prone to inflated subjective…”
The Black Box, Unlocked: Predictability and Understandability in Military AI
United Nations Institute for Disarmament Research, & Holland Michel, A. (2020) · doi:10.37559/sectec/20/ai1 · Cited by 64 · cited 1×
“Unlike traditional menu buttons which have predictable outcomes, AI Modifiers can yield unexpected results due to the inherent unpredictability of AI models [40, 64, 95, 96].”
The Structure of Writing Processes as Revealed by Secondary Task Demands
Ransdell, S., Levy, C. M., & Kellogg, R. T. (2002) · L1 Educational Studies in Language and Literature · doi:10.1023/a:1020851300668 · Cited by 61 · cited 1×
“We also considered that during the writing process writers frequently struggle with cognitive over-load [61] and that even small demands on working memory can lead to decreased fluency [77].”
Making sense of revision-control systems
O'Sullivan, B. (2009) · Communications of the ACM · doi:10.1145/1562164.1562183 · Cited by 61 · cited 1×
“Together, D1 and D2 contribute to existing discussions surrounding linear versus non-linear representations of versions history across various domains [11, 41, 71, 74, 88, 99].”
A new approach to version control
Plaice, J., & Wadge, W. W. (1993) · IEEE Transactions on Software Engineering · doi:10.1109/32.221137 · Cited by 58 · cited 1×
“Together, D1 and D2 contribute to existing discussions surrounding linear versus non-linear representations of versions history across various domains [11, 41, 71, 74, 88, 99].”
Kim, T. S., Lee, Y., Chang, M., & Kim, J. (2023) · Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3586183.3606833 · Cited by 56
Nonlinear revision control for images
Chen, H., Wei, L., & Chang, C. (2011) · ACM Transactions on Graphics · doi:10.1145/2010324.1965000 · Cited by 49
Cheung, Y. L. (2016) · English Language Education · doi:10.1007/978-3-319-38834-2_13 · Cited by 49
Automatically Generating CS Learning Materials with Large Language Models
MacNeil, S., Tran, A., Leinonen, J., Denny, P., Kim, J., Hellas, A., Bernstein, S., & Sarsa, S. (2022) · Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 2 · doi:10.1145/3545947.3569630 · Cited by 44
Unpredictability of AI: On the Impossibility of Accurately Predicting All Actions of a Smarter Agent
Yampolskiy, R. V. (2020) · Journal of Artificial Intelligence and Consciousness · doi:10.1142/s2705078520500034 · Cited by 42
Beyond Text-to-Image: Multimodal Prompts to Explore Generative AI
Liu, V. (2023) · Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544549.3577043 · Cited by 38 · cited 1×
“For example, an application for editing AI-generated images could implement an AI Modifier which accepts textual instructions and generates corresponding image variations, which are then organized within Variation Fields, contributing to HCI research on text-to-image AI editors [6, 54, 55, 70, 93].”
Textlets: Supporting Constraints and Consistency in Text Documents
Han, H. L., Renom, M. A., Mackay, W. E., & Beaudouin-Lafon, M. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376804 · Cited by 36 · cited 2×
“…that emphasize the consideration of multiple ideas before evaluation [19, 34, 89]; principles on reification and reuse for designing visual interfaces [3, 37]; and the latest research on utilizing LLMs in writing interfaces to foster experimentation and creativity [9, 14, 16, 26, 47, 50, 97].”
VRGit: A Version Control System for Collaborative Content Creation in Virtual Reality
Zhang, L., Agrawal, A., Oney, S., & Guo, A. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581136 · Cited by 36 · cited 1×
“Together, D1 and D2 contribute to existing discussions surrounding linear versus non-linear representations of versions history across various domains [11, 41, 71, 74, 88, 99].”
Chenoweth, N. A. (1987) · ELT Journal · doi:10.1093/elt/41.1.25 · Cited by 35 · cited 2×
“” W2, 3, and 12 specifically commented that the lack of interface clutter due to the way the Variation Fields , Popup Toolbar , and Variation Sidebar handle variation storage meant that they could create, revise and evaluate a larger number of variations with ABScribe.”
Do We Still Need Human Assessors? Prompt-Based GPT-3 User Simulation in Conversational AI
Meyer, S., Elsweiler, D., Ludwig, B., Fernandez-Pichel, M., & Losada, D. E. (2022) · Proceedings of the 4th Conference on Conversational User Interfaces · doi:10.1145/3543829.3544529 · Cited by 34 · cited 1×
“Recent HCI studies on LLM-based tool design have mainly focused on prompt engineering [6, 16, 46, 55, 94, 98] and exploring the generative capabilities of LLMs [50, 58, 63, 67, 97].”
INTELLIGENT FAILURES: CLIPPY MEMES AND THE LIMITS OF DIGITAL ASSISTANTS
Baym, N. K., Shifman, L., Persaud, C. J., & Wagman, K. B. (2019) · AoIR Selected Papers of Internet Research · doi:10.5210/spir.v2019i0.10923 · Cited by 22
Tweetorial Hooks: Generative AI Tools to Motivate Science on Social MediaPreprint
Long, T., Zhang, D., Li, G., Taraif, B., Menon, S., Smith, K. S., Wang, S., Gero, K. I., & Chilton, L. B. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.12265 · Cited by 21 · cited 1×
“There has been an explosion of recent work [4, 9, 14, 16, 26–28, 31, 47, 48, 50, 56, 67, 82, 84, 97] on interfaces for writing with LLMs in tandem with rapid advancements in AI writing capabilities using LLMs [8, 18, 66, 76, 91], examining various aspects of the AI-assisted writing process…”
Teaching Revision: A Model of the Drafting Process
Huff, R. K. (1983) · College English · doi:10.2307/376699 · Cited by 7 · cited 1×
“…the text in repeated cycles, with 2 Link to GiHub Repository: https://github.com/mohireza/abscribe/ multiple objectives including finding the form or shape of an ar-gument [87], experimenting with vocabulary and style [43], and going back and forth between multiple composing activities during…”
Brade, S., Wang, B. K., Sousa, M., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2304.09337 · Cited by 5
Fundamentals in HCI: Learning the Value of Consistency and User Models
Bly, S. A. (2007) · The MIT Press eBooks · doi:10.7551/mitpress/7455.003.0012
OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.08774
Research works printed but not linked: 19
- 10.7551/mitpress · doi:10.7551/mitpress
- Virginia Braun and Victoria Clarke. 2019. Reflecting on reflexive thematic analysis. Qualitative research in sport, exercise and health 11, 4 (2019), 589-597.
- Bill Buxton. 2010. Sketching user experiences: getting the design right and the right design. Morgan kaufmann, Boston.
- John Joon Young Chung, Wooseok Kim, Kang Min Yoo, Hwaran Lee, Eytan Adar, and Minsuk Chang. 2022. TaleBrush: Sketching stories with generative pretrained language models. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA, 1-19.
- Robert Dale. 2021. GPT-3: What's it good for? Natural Language Engineering 27, 1 (2021), 113-118.
- 10.18653/v1 · DOI
- Jill Fitzgerald. 1987. Research on revision in writing. Review of educational research 57, 4 (1987), 481-506.
- Linda Flower and John R. Hayes. 1981. A Cognitive Process Theory of Writing. College Composition and Communication 32, 4 (1981), 365-387. http://www. jstor.org/stable/356600 · DOI · Paper page
- Han L Han. 2020. Designing Representations for Digital Documents. In Adjunct Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology. Association for Computing Machinery, New York, NY, USA, 174- 178.
- Han L Han, Miguel A Renom, Wendy E Mackay, and Michel Beaudouin-Lafon. 2020. Textlets: Supporting constraints and consistency in text documents. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA, 1-13. · DOI · Paper page
- Iikka Hauhio, Anna Kantosalo, Simo Linkola, and Hannu Toivonen. 2023. The Spectrum of Unpredictability and its Relation to Creative Autonomy. In Proceedings of the 14th International Conference on Computational Creativ- ity. Association for Computational Creativity (ACC), Spain, 148-152. https: //computationalcreativity.net/iccc23/ International Conference on Computa- tional Creativity, ICCC ; Conference date: 19-06-2023 Through 23-06-2023.
- Ernest Hemingway. 2014. Moveable feast: the restored edition. Simon and Schuster, New York, NY, USA.
- David G Jansson and Steven M Smith. 1991. Design fixation. Design studies 12, 1 (1991), 3-11.
- Charles A MacArthur. 2018. Evaluation and revision. Best practices in writing instruction 287 (2018), 23 pages.
- Thomas Mahatody, Mouldi Sagar, and Christophe Kolski. 2010. State of the art on the cognitive walkthrough method, its variants and evolutions. Intl. Journal of Human-Computer Interaction 26, 8 (2010), 741-785.
- James P. Purdy. 2014. What Can Design Thinking Offer Writing Studies? College Composition and Communication 65, 4 (2014), 612-641. http://www.jstor.org/ stable/43490875
- Horst Rittel. 1967. Wicked problems. Management Science,(December 1967) 4, 14 (1967), B141-B142.
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. In Advances in Neural Information Processing Systems, I.
- William Knowlton Zinsser. 2001. On writing well: The classic guide to writing nonfiction. Quill/A Harper Collins Books, 195 Broadway New York, NY, USA.
Cited by
AI makes you smarter but none the wiser: The disconnect between performance and metacognition
Fernandes, D., Villa, S., Nicholls, S., Haavisto, O., Buschek, D., Schmidt, A., Kosch, T., Shen, C., & Welsch, R. (2025) · Computers in Human Behavior · doi:10.1016/j.chb.2025.108779 · Cited by 188
Chakrabarty, T., Laban, P., & Wu, C.-S. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713559 · Cited by 92
Performance and Metacognition Disconnect when Reasoning in Human-AI Interaction
Fernandes, D., Villa, S., Nicholls, S., Haavisto, O., Buschek, D., Schmidt, A., Kosch, T., Shen, C., & Welsch, R. (2024) · arXiv · doi:10.48550/arxiv.2409.16708 · Cited by 71
AI as Extraherics: Fostering Higher-order Thinking Skills in Human-AI InteractionPreprint
Yatani, K., Sramek, Z., & Yang, C.-L. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2409.09218 · Cited by 64
Designing Interactions with Generative AI for Art and Creativity: A Systematic Review and Taxonomy
Hu, X., Xing, Y., Cai, X., Zhao, Y., Cook, M., Borgo, R., & Neate, T. (2025) · Designing Interactive Systems Conference · doi:10.1145/3715336.3735843 · Cited by 61
Metadata
- Authors
- Reza, M., Laundry, N. M., Musabirov, I., Dushniku, P., Yu, Z. Y. “., Mittal, K., Grossman, T., Liut, M., Kuzminykh, A., & Williams, J. J.
- Affiliations
- University of Toronto (CA)
- Computer Science, University of Toronto, Canada
- ORCID
- Venue
- Proceedings of the CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2024-05-11
- Pages
- 1-18
- OpenAlex
- W4396832282
- Cited by
- 52
- References
- 67
- Citations by year
- 2026: 16 · 2025: 34 · 2024: 2
- FWCI
- 16.76
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- NSF (National Science Foundation) (2209819)
- Natural Sciences and Engineering Research Council of Canada (RGPIN-2019-06968)
- Office of Naval Research (N00014-21-1-2576)
- Topics
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Physical Sciences › Computer Science › Human-Computer Interaction › Innovative Human-Technology Interaction
- Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › Data Visualization and Analytics
- Keywords
- rewriting, large language models, textual revision, LLM prompts, text structure
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-04, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-04 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (qwen3.8-omni-flash).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.