Readers’ perception of computer-generated news: Credibility, expertise, and readability
Open accessAbstract
We conducted an online experiment to study people’s perception of automated computer-written news. Using a 2 × 2 × 2 design, we varied the article topic (sports, finance; within-subjects) and both the articles’ actual and declared source (human-written, computer-written; between-subjects). Nine hundred eighty-six subjects rated two articles on credibility, readability, and journalistic expertise. Varying the declared source had small but consistent effects: subjects rated articles declared as human written always more favorably, regardless of the actual source. Varying the actual source had larger effects: subjects rated computer-written articles as more credible and higher in journalistic expertise but less readable. Across topics, subjects’ perceptions did not differ. The results provide conservative estimates for the favorability of computer-written news, which will further increase over time and endorse prior calls for establishing ethics of computer-written news.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
For algorithmically generated texts, participants were usually unable to reliably distinguish between human- and AI-generated texts.
“For algorithmically generated texts, participants were usually unable to reliably distinguish between human- and AI-generated texts [6,21].”
From Value attributed to text-based archives generated by artificial intelligence (2023) · cites this paper as [21]
Participants favoured articles labelled as human-written compared with those labelled as written by an algorithm, although both were created in the same way.
“participants favoured articles that were labelled as human-written compared with those labelled as written by an algorithm (although both were created in the same way; [21]).”
From Value attributed to text-based archives generated by artificial intelligence (2023) · cites this paper as [21]
Our results are in line with previous research in the domain of automated journalism and empirical aesthetics that suggest a bias against AI-generated productions.
“Our results are in line with previous research in the domain of automated journalism and empirical aesthetics that suggest a bias against AI-generated productions—we found a trend for individuals to assign lower values to AI-generated archives compared with original archives, and individuals assigned lower values to archives they themselves categorized as AI-generated [6,12–14,21,26,45–47].”
From Value attributed to text-based archives generated by artificial intelligence (2023) · cites this paper as [21]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
(2011) · Cited by 4,430 · see citing works
The Parable of Google Flu: Traps in Big Data Analysis
Lazer, D., Kennedy, R., King, G., & Vespignani, A. (2014) · Science · doi:10.1126/science.1248506 · Cited by 2,523 · cited 2× · see citing works
“For more information on the SoSci Panel, see Leiner (2014). A total of 1107 subjects participated in the study in December 2014.”
BackgroundCommunication and Persuasion: Psychological Studies of Opinion Change.
Riley, M. W., Hovland, C. I., Janis, I. L., & Kelley, H. H. (1954) · American Sociological Review · doi:10.2307/2087772 · Cited by 2,039 · see citing works
Social and Heuristic Approaches to Credibility Evaluation Online
Metzger, M. J., Flanagin, A. J., & Medders, R. B. (2010) · Journal of Communication · doi:10.1111/j.1460-2466.2010.01488.x · Cited by 1,394 · see citing works
Digital Media, Youth, and Credibility
Metzger, M. J., & Flanagin, A. J. (2007) · Cited by 1,036 · see citing works
Algorithmic Accountability: Journalistic investigation of computational power structures
Diakopoulos, N. (2015) · Digital Journalism · doi:10.1080/21670811.2014.976411 · Cited by 733 · see citing works
Johnson, T. J., & Kaye, B. K. (2004) · Journalism & Mass Communication Quarterly · doi:10.1177/107769900408100310 · Cited by 719 · see citing works
Social Media as Information Source: Recency of Updates and Credibility of Information
Westerman, D., Spence, P. R., & Van Der Heide, B. (2014) · Journal of Computer-Mediated Communication · doi:10.1111/jcc4.12041 · Cited by 708 · see citing works
Defining and Measuring Credibility of Newspapers: Developing an Index
Meyer, P. (1988) · Journalism Quarterly · doi:10.1177/107769908806500301 · Cited by 654 · see citing works
Carlson, M. (2015) · Digital Journalism · doi:10.1080/21670811.2014.976412 · Cited by 542 · see citing works
Kohring, M., & Matthes, J. (2007) · Communication Research · doi:10.1177/0093650206298071 · Cited by 413 · see citing works
Automated Media: An Institutional Theory Perspective on Algorithmic Media Production and Consumption
Napoli, P. M. (2014) · Communication Theory · doi:10.1111/comt.12039 · Cited by 405 · see citing works
Enter the Robot Journalist: Users' perceptions of automated content
Clerwall, C. (2014) · Journalism Practice · doi:10.1080/17512786.2014.883116 · Cited by 377 · see citing works
Actors, Actants, Audiences, and Activities in Cross-Media News Work
Lewis, S. C., & Westlund, O. (2014) · Digital Journalism · doi:10.1080/21670811.2014.927986 · Cited by 361 · see citing works
Towards a sociology of computational and algorithmic journalism
Anderson, C. W. N. (2012) · New Media & Society · doi:10.1177/1461444812465137 · Cited by 342 · see citing works
THE ALGORITHMS BEHIND THE HEADLINES
van Dalen, A. (2012) · Journalism Practice · doi:10.1080/17512786.2012.667268 · Cited by 330 · see citing works
Exploring Receivers' Criteria for Perception of Print and Online News
Sundar, S. S. (1999) · Journalism & Mass Communication Quarterly · doi:10.1177/107769909907600213 · Cited by 267 · see citing works
Graefe, A. (2016) · Columbia Academic Commons (Columbia University) · doi:10.7916/d80g3xdj · Cited by 250 · see citing works
Self-citationDigital media and youth: Unparalleled opportunity and unprecedented responsibility
Flanagin, A. J., & Metzger, M. J. (2008) · Issue Lab (Candid) · Cited by 220 · see citing works
Cohen, S., Hamilton, J. T., & Turner, F. (2011) · Communications of the ACM · doi:10.1145/2001269.2001288 · Cited by 208 · see citing works
Validating a Scale for the Measurement of Credibility: A Covariance Structure Modeling Approach
West, M. D. (1994) · Journalism Quarterly · doi:10.1177/107769909407100115 · Cited by 159 · see citing works
News Quality from the Recipients' Perspective
Urban, J., & Schweiger, W. (2013) · Journalism Studies · doi:10.1080/1461670x.2013.856670 · Cited by 155 · see citing works
From Mr. and Mrs. Outlier To Central Tendencies
Young, M. L., & Hermida, A. (2014) · Digital Journalism · doi:10.1080/21670811.2014.976409 · Cited by 111 · see citing works
Journalist versus news consumer : The perceived credibility of machine written news
van der Kaa, H. A. J., & Krahmer, E. (2014) · Data Archiving and Networked Services (DANS) · Cited by 88 · see citing works
The Robot Journalist in the Age of Social Physics: The End of Human Journalism?
Latar, N. L. (2014) · The economics of information, communication and entertainment · doi:10.1007/978-3-319-09009-2_6 · Cited by 88 · see citing works
New Perspectives on Media Credibility Research
Golan, G. J. (2010) · American Behavioral Scientist · doi:10.1177/0002764210376307 · Cited by 47 · see citing works
Poetry Generation with PoeTryMe
Oliveira, H. G., & Cardoso, A. (2014) · Atlantis thinking machines · doi:10.2991/978-94-6239-085-0_12 · Cited by 22 · see citing works
Implicit Force Control Of A Position Controlled Robot – A Comparison With Explicit Algorithms
Winkler, A., & Suchý, J. (2015) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.1106936 · Cited by 6 · see citing works
Natural Language News Generation From Big Data
Haarmann, B., & Sikorski, L. (2015) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.1107399 · Cited by 3 · see citing works
Self-citationGraefe, A., Haim, M., Haarmann, B., et al. (2016) · Harvard Dataverse · doi:10.7910/dvn/whueza · Cited by 2 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 1
- [17]17. Leiner D (2014) Convenience samples from online respondent pools: A case study of the SoSci Panel. (Working paper). Germany: Ludwig-Maximilians-Universität Munich. Available at: https://www.researchgate.net/publication/259669050_Convenience_Samples_from_Online_ Respondent_Pools_A_case_study_of_the_SoSci_Panel
Cited by
Sorted by citations
In AI we trust? Perceptions about automated decision-making by artificial intelligence
Araujo, T., Helberger, N., Kruikemeier, S., & de Vreese, C. H. (2020) · AI & SOCIETY · doi:10.1007/s00146-019-00931-w · Cited by 1,038 · see citing works
Köbis, N., & Mossink, L. D. (2021) · Computers in Human Behavior · doi:10.1016/j.chb.2020.106553 · Cited by 406 · see citing works
Jakesch, M., French, M., Ma, X., Hancock, J. T., & Naaman, M. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300469 · Cited by 190 · see citing works
Graefe, A., & Bohlken, N. (2020) · Media and Communication · doi:10.17645/mac.v8i3.3019 · Cited by 154 · see citing works
Self-citationDo teachers spot AI? Evaluating the detectability of AI-generated texts among student essays
Fleckenstein, J., Meyer, J., Jansen, T., Keller, S. D., Köller, O., & Möller, J. (2024) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2024.100209 · Cited by 126 · see citing works
Biermann, O. C., Ma, N. F., & Yoon, D. (2022) · Designing Interactive Systems Conference · doi:10.1145/3532106.3533506 · Cited by 116 · see citing works
Altay, S., & Gilardi, F. (2024) · PNAS Nexus · doi:10.1093/pnasnexus/pgae403 · Cited by 119 · see citing works
AI ‘Everywhere and Nowhere’: Addressing the AI Intelligibility Problem in Public Service Journalism
Jones, B., Jones, R., & Luger, E. (2022) · Digital Journalism · doi:10.1080/21670811.2022.2145328 · Cited by 84 · see citing works
Moravec, V., Hynek, N., Skare, M., Gavurova, B., & Kubak, M. (2024) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2023.123162 · Cited by 76 · see citing works
Value attributed to text-based archives generated by artificial intelligence
Darda, K., Carre, M., & Cross, E. (2023) · Royal Society Open Science · doi:10.1098/rsos.220915 · Cited by 37 · see citing works
- Authors
- Graefe, A., Haim, M., Haarmann, B., & Brosius, H.-B.
- Affiliations
- Hochschule Macromedia (DE)
- Ludwig-Maximilians-Universität München (DE)
- Fraunhofer Institute for Communication, Information Processing and Ergonomics (DE)
- Corresponding author
- Graefe, Andreas
- Venue
- Journalism
- Publisher
- SAGE Publications
- Published
- 2016-04-17
- Volume
- 19
- Issue
- 5
- Pages
- 595-610
- ISSN
- 1464-8849 (ISSN-L)
- 1741-3001
- OpenAlex
- W2337983557
- Cited by
- 315
- References
- 30
- Citations by year
- 2026: 51 · 2025: 60 · 2024: 41 · 2023: 32 · 2022: 28 · 2021: 25 · 2020: 29 · 2019: 28 · 2018: 10 · 2017: 8 · 2016: 3
- FWCI
- 46.00
- Type
- journal article
- Language
- en
- Open access
- Locations
- Journalism (publishedVersion)
- Open access LMU (Ludwid Maxmilian's Universitat Munchen) (submittedVersion)
- Publikationsdatenbank der Fraunhofer-Gesellschaft (Fraunhofer-Gesellschaft) (submittedVersion)
- Fraunhofer-Publica (Fraunhofer-Gesellschaft) (submittedVersion)
- Topics
- Social Sciences › Communication › Journalism and Media Studies
- Social Sciences › Communication › Media Studies and Communication
- Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
- Keywords
- automated reporting, news credibility, readability, source perception, sports news, business news
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
