Automated journalism: The effects of AI authorship and evaluative information on the perception of a science journalism article
Open accessPublished version of a preprint
The DOI matches its Crossref record; 20 of the 20 references it cites were found in a registry or an index.
Abstract
Texts produced by artificial intelligence (AI) are becoming increasingly prevalent in digital journalism. Research suggests that these texts do not differ from human-written texts in their perceived credibility or trustworthiness where simple and short text types are concerned. However, it is unclear how AI-written texts beyond simple fact reporting are perceived. Therefore, this research aimed to expand upon the existing literature on automated journalism by investigating the influence of AI authorship (vs. human authorship) and evaluative information presentation (vs. neutral information presentation). The results of three preregistered experimental studies revealed no differences in perceived credibility and trustworthiness between AI-written and human-written texts. However, presenting information in an evaluative way decreased the perception of credibility and trustworthiness. Moreover, the AI was perceived as less anthropomorphic than the human author. The belief in the machine heuristic was stronger for an AI than for a human author, particularly when participants had actually read an article allegedly written by an AI. A pooled analysis across the data of all three studies underpinned the main effect of information presentation. Concluding, we discuss the findings against the background of AI perception theory and suggest implications for future research.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Motivated Skepticism in the Evaluation of Political Beliefs
Taber, C. S., & Lodge, M. (2006) · American Journal of Political Science · doi:10.1111/j.1540-5907.2006.00214.x · Cited by 4,344
On seeing human: A three-factor theory of anthropomorphism
Epley, N., Waytz, A., & Cacioppo, J. T. (2007) · Psychological Review · doi:10.1037/0033-295x.114.4.864 · Cited by 4,069
Artificial intelligence in service
Huang, M.-H., & Rust, R. T. (2018) · Journal of Service Research · doi:10.1177/1094670517752459 · Cited by 3,978
Algorithm aversion: People erroneously avoid algorithms after seeing them err.
Dietvorst, B. J., Simmons, J. P., & Massey, C. (2015) · Journal of Experimental Psychology: General · doi:10.1037/xge0000033 · Cited by 3,232
Bartneck, C., Kulić, D., Croft, E., & Zoghbi, S. (2009) · International Journal of Social Robotics · doi:10.1007/s12369-008-0001-3 · Cited by 3,193
Resistance to Medical Artificial Intelligence
Longoni, C., Bonezzi, A., & Morewedge, C. K. (2019) · Journal of Consumer Research · doi:10.1093/jcr/ucz013 · Cited by 2,015
Task-Dependent Algorithm Aversion
Castelo, N., Bos, M. W., & Lehmann, D. R. (2019) · Journal of Marketing Research · doi:10.1177/0022243719851788 · Cited by 1,500
Luo, X., Tong, S., Fang, Z., & Qu, Z. (2019) · Marketing Science · doi:10.1287/mksc.2019.1192 · Cited by 1,262
Feeling robots and human zombies: Mind perception and the uncanny valley
Gray, K., & Wegner, D. M. (2012) · Cognition · doi:10.1016/j.cognition.2012.06.007 · Cited by 941
Artificial Intelligence in Utilitarian vs. Hedonic Contexts: The “Word-of-Machine” Effect
Longoni, C., & Cian, L. (2022) · Journal of Marketing · doi:10.1177/0022242920957347 · Cited by 831
Almost human: Anthropomorphism increases trust resilience in cognitive agents.
de Visser, E. J., Monfort, S. S., McKendrick, R., Smith, M. A. B., McKnight, P. E., Krueger, F., & Parasuraman, R. (2016) · Journal of Experimental Psychology: Applied · doi:10.1037/xap0000092 · Cited by 614
Carlson, M. (2015) · Digital Journalism · doi:10.1080/21670811.2014.976412 · Cited by 542
Myside Bias, Rational Thinking, and Intelligence
Stanovich, K. E., West, R. F., & Toplak, M. E. (2013) · Current Directions in Psychological Science · doi:10.1177/0963721413480174 · Cited by 427
Köbis, N., & Mossink, L. D. (2021) · Computers in Human Behavior · doi:10.1016/j.chb.2020.106553 · Cited by 406
Enter the Robot Journalist: Users' perceptions of automated content
Clerwall, C. (2014) · Journalism Practice · doi:10.1080/17512786.2014.883116 · Cited by 377
Eyssel, F., Kuchenbrandt, D., Bobinger, S., de Ruiter, L., & Hegel, F. (2012) · Proceedings of the seventh annual ACM/IEEE international conference on Human-Robot Interaction · doi:10.1145/2157689.2157717 · Cited by 307
Longitudinal Analysis of Attitudes Toward Wolves
TREVES, A., NAUGHTON‐TREVES, L., & SHELLEY, V. (2013) · Conservation Biology · doi:10.1111/cobi.12009 · Cited by 263
Graefe, A. (2016) · Columbia Academic Commons (Columbia University) · doi:10.7916/d80g3xdj · Cited by 186
Graefe, A., & Bohlken, N. (2020) · Media and Communication · doi:10.17645/mac.v8i3.3019 · Cited by 154
Research works printed but not linked: 32
- Appelman, 2016, Measuring message credibility: Construction and validation of an exclusive scale, Journalism & Mass Communication Quarterly, 93, 59, 10.1177/1077699015606057 · DOI · Paper page
- Bartneck, 2009, Does the design of a robot influence its animacy and perceived intelligence?, International Journal of Social Robotics, 1, 195, 10.1007/s12369-009-0013-7 · DOI · Paper page
- Ciechanowski, 2019, In the shades of the uncanny valley: An experimental study of human-chatbot interaction, Future Generation Computer Systems, 92, 539, 10.1016/j.future.2018.01.055 · DOI · Paper page
- Danzon-Chambaud, 2021, A systematic review of automated journalism scholarship: Guidelines and suggestions for future research, Open Research Europe, 1, 4, 10.12688/openreseurope.13096.1 · DOI
- Dörr, 2016, Mapping the field of algorithmic journalism [article], Digital Journalism, 4, 700, 10.1080/21670811.2015.1096748 · DOI · Paper page
- Festinger, 1957, Vol. 2
- Flanagin, 2000, Perceptions of internet information credibility, Journalism & Mass Communication Quarterly, 77, 515, 10.1177/107769900007700304 · DOI · Paper page
- Flemming, 2020, Research is tentative, but that's okay: Overcoming misconceptions about scientific tentativeness through refutation texts [article], Discourse Processes, 57, 17, 10.1080/0163853X.2019.1629805 · DOI
- Graefe, 2018, Readers' perception of computer-generated news: Credibility, expertise, and readability, Journalism, 19, 595, 10.1177/1464884916641269 · DOI · Paper page
- Haim, 2017, Automated news: Better than expected?, Digital Journalism, 5, 1044, 10.1080/21670811.2017.1345643 · DOI
- Hamilton, 1998, Message variables that mediate and moderate the effect of equivocal language on source credibility, Journal of Language and Social Psychology, 17, 109, 10.1177/0261927X980171006 · DOI
- Jang, 2021, The effects of anthropomorphism on how people evaluate algorithm-written news [article; early access], Digital Journalism, 22
- Jia, 2021, Source credibility matters: Does automated journalism inspire selective exposure?, International Journal of Communication, 15, 22
- Jung, 2017, Intrusion of software robots into journalism: The public's and journalists' perceptions of news written by algorithms and human journalists, Computers in Human Behavior, 71, 291, 10.1016/j.chb.2017.02.022 · DOI · Paper page
- Jussupow, 2020, Why are we averse towards algorithms? A comprehensive literature review on algorithm aversion, Proceedings of the 28th European Conference on Information Systems, Online AIS Conference
- Kimmerle, 2015, How laypeople understand the tentativeness of medical research news in the media: An experimental study on the perception of information about deep brain stimulation [article], Science Communication, 37, 173, 10.1177/1075547014556541 · DOI
- Kohring, 2004, Revision und Validierung einer Skala zur Erfassung von Vertrauen in Journalismus, Medien and Kommunikationswissenschaft : M and K, 52, 377, 10.5771/1615-634x-2004-3-377 · doi:10.5771/1615-634x-2004-3-377
- Kohring, 2007, Trust in news media: Development and validation of a multidimensional scale, Communication Research, 34, 231, 10.1177/0093650206298071 · DOI · Paper page
- Liu, 2019, Machine authorship in situ: Effect of news organization and news genre on news credibility [article], Digital Journalism, 7, 635, 10.1080/21670811.2018.1510740 · DOI · Paper page
- Meyer, 2010, The journalist behind the news: Credibility of straight, collaborative, opinionated, and blogged "news, American Behavioral Scientist, 54, 100, 10.1177/0002764210376313 · DOI
- Nickell, 1986, The computer attitude scale, Computers in Human Behavior, 2, 301, 10.1016/0747-5632(86)90010-5 · DOI · Paper page
- Nickerson, 1998, Confirmation bias: A ubiquitous phenomenon in many guises, Review of General Psychology, 2, 175, 10.1037/1089-2680.2.2.175 · DOI · Paper page
- Nowak, 2005, The influence of the avatar on online perceptions of anthropomorphism, androgyny, credibility, homophily, and attraction [article], Journal of Computer-Mediated Communication, 11, 26, 10.1111/j.1083-6101.2006.tb00308.x · DOI · Paper page
- Sundar, 1999, Exploring receivers' criteria for perception of print and online news, Journalism & Mass Communication Quarterly, 76, 373, 10.1177/107769909907600213 · DOI · Paper page
- Sundar, 2019, Machine heuristic: When we trust computers more than humans with our personal information, Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, 1
- Sundar, 2001, Conceptualizing sources in online news, Journal of Communication, 51, 52, 10.1111/j.1460-2466.2001.tb02872.x · doi:10.1111/j.1460-2466.2001.tb02872.x · Paper page
- Tandoc, 2020, Man vs. Machine? The impact of algorithm authorship on news credibility, Digital Journalism, 8, 548, 10.1080/21670811.2020.1762102 · DOI · Paper page
- Waddell, 2018, A robot wrote this? How perceived machine authorship affects news credibility, Digital Journalism, 6, 236, 10.1080/21670811.2017.1384319 · DOI · Paper page
- Waddell, 2019, Can an algorithm reduce the perceived bias of news? Testing the effect of machine attribution on news readers' evaluations of bias, anthropomorphism, and credibility, Journalism & Mass Communication Quarterly, 96, 82, 10.1177/1077699018815891 · DOI · Paper page
- Wölker, 2021, Algorithms in the newsroom? News readers' perceived credibility and selection of automated journalism, Journalism, 22, 86, 10.1177/1464884918757072 · DOI · Paper page
- Wu, 2020, Is automated journalistic writing less biased? An experimental test of auto-written and human-written news stories, Journalism Practice, 14, 1008, 10.1080/17512786.2019.1682940 · DOI
- Zheng, 2018, When algorithms meet journalism: The user perception to automated news in a cross-cultural context, Computers in Human Behavior, 86, 266, 10.1016/j.chb.2018.04.046 · DOI · Paper page
Cited by
Draxler, F., Werner, A., Lehmann, F., Hoppe, M., Schmidt, A., Buschek, D., & Welsch, R. (2023) · arXiv · doi:10.48550/arxiv.2303.03283 · Cited by 284
Dalalah, D., & Dalalah, O. M. A. (2023) · The International Journal of Management Education · doi:10.1016/j.ijme.2023.100822 · Cited by 218
Draxler, F., Werner, A., Lehmann, F., Hoppe, M., Schmidt, A., Buschek, D., & Welsch, R. (2024) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3637875 · Cited by 186
“• Questions on relevance, usefulness, creativity of AI and participants’ AI competence as in Study 2, Part 1 • Godspeed questionnaires on anthropomorphism and intelligence [6], as used in [72] The AI Ghostwriter Effect”
MethodMoravec, V., Hynek, N., Skare, M., Gavurova, B., & Kubak, M. (2024) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2023.123162 · Cited by 76
The Effects of Assumed AI vs. Human Authorship on the Perception of a GPT-Generated Text
Lermann Henestrosa, A., & Kimmerle, J. (2024) · Journalism and Media · doi:10.3390/journalmedia5030069 · Cited by 53
Self-citation
Metadata
- Authors
- Lermann Henestrosa, A., Greving, H., & Kimmerle, J.
- Venue
- Computers in Human Behavior
- Publisher
- Elsevier BV
- Published
- 2023-01
- Volume
- 138
- Pages
- 107445
- ISSN
- 0747-5632 (ISSN-L)
- OpenAlex
- W4293859197
- Cited by
- 113
- References
- 20
- FWCI
- 31.79
- Type
- journal article
- Language
- en
- Open access
- gold
- Licence
- www.elsevier.com/tdm/userlicense/1.0/ (tdm)
- www.elsevier.com/legal/tdmrep-license (tdm)
- doi.org/10.15223/policy-017 (stm-asf)
- doi.org/10.15223/policy-037 (stm-asf)
- doi.org/10.15223/policy-012 (stm-asf)
- doi.org/10.15223/policy-029 (stm-asf)
- doi.org/10.15223/policy-004 (stm-asf)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.