CoChat

The HaLLMark Effect: Supporting Provenance and Transparent Use of Large Language Models in Writing with Interactive Visualization

Open access
Hoque, Md Naimul, Mashiat, Tasfia, Ghai, Bhavya, Shelton, Cecilia D., Chevalier, Fanny, Kraus, Kari, Elmqvist, Niklas · 2024 · Proceedings of the CHI Conference on Human Factors in Computing Systems

The DOI matches its Crossref record; 65 of the 65 references it cites were found in a registry or an index.

Abstract

The use of Large Language Models (LLMs) for writing has sparked controversy both among readers and writers. On one hand, writers are concerned that LLMs will deprive them of agency and ownership, and readers are concerned about spending their time on text generated by soulless machines. On the other hand, AI-assistance can improve writing as long as writers can conform to publisher policies, and as long as readers can be assured that a text has been verified by a human. We argue that a system that captures the provenance of interaction with an LLM can help writers retain their agency, conform to policies, and communicate their use of AI to publishers and readers transparently. Thus we propose HaLLMark, a tool for visualizing the writer’s interaction with the LLM. We evaluated HaLLMark with 13 creative writers, and found that it helped them retain a sense of control and ownership of the text.

Overview

Integrity signals

Registry

Crossref · 10.1145/3613904.3641895

Cites

65 of 65 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

4 more claims about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Citations

Cites

Show all 65 references

Research works printed but not linked: 6

  1. Richard Brath. 2021. Visualizing with Text. CRC Press, Boca Raton, FL, USA.
  2. 10.18653/v1 · doi:10.18653/v1
  3. 10.18653/v1 · doi:10.18653/v1
  4. Association for Computing Machinery. 2023. ACM Policy on Authorship. https://www.acm.org/publications/policies/new-acm-policy-on-authorship Accessed: 2023-08-04.
  5. Springer Nature. 2023. Nature Portfolio. https://www.nature.com/nature-portfolio/editorial-policies/ai
  6. QuillJS. 2021. QuillJS: a Rich Text Editor. https://quilljs.com Accessed: 2022-02-04.

Cited by

See all 54

Metadata

Authors
Hoque, M. N., Mashiat, T., Ghai, B., Shelton, C. D., Chevalier, F., Kraus, K., & Elmqvist, N.
Affiliations
  • University of Maryland, College Park (US)
  • George Mason University (US)
  • Amazon (United States) (US)
  • Department of English, University of Maryland, United States
  • University of Toronto (CA)
  • Aarhus University (DK)
Venue
Proceedings of the CHI Conference on Human Factors in Computing Systems
Publisher
ACM
Published
2024-05-11
Pages
1-15
OpenAlex
W4396827151
Cited by
48
References
61
Citations by year
2026: 22 · 2025: 24 · 2024: 2
FWCI
21.90
Type
conference paper
Language
en
Open access
Locations
  • CHI Conference on Human Factors in Computing Systems (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • Scopus (Elsevier) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
NSF (IIS-2211628)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
  • Social Sciences › Arts and Humanities › Visual Arts and Performing Arts › Art, Technology, and Culture
  • Social Sciences › Arts and Humanities › Literature and Literary Theory › Digital Humanities and Scholarship
Keywords
large language models, AI-assisted writing, interactive visualization, writer agency, publisher policies, hallmarks, creative writing

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: The HaLLMark Effect: Supporting Provenance and Transparent Use of Large Language Models in Writing with Interactive Visualization. https://app.cochat.ai/papers/10.1145/3613904.3641895
The paper
Hoque, M. N., Mashiat, T., Ghai, B., Shelton, C. D., Chevalier, F., Kraus, K., & Elmqvist, N. (2024). The HaLLMark Effect: Supporting Provenance and Transparent Use of Large Language Models in Writing with Interactive Visualization. Proceedings of the CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3613904.3641895

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.