Deanthropomorphising NLP: Can a language model be conscious?
Open accessThe DOI matches its Crossref record; 81 of the 81 references it cites were found in a registry or an index; 30 of the 53 sources its claims cite were checked for support.
Abstract
This work is intended as a voice in the discussion over previous claims that a pretrained large language model (LLM) based on the Transformer model architecture can be sentient. Such claims have been made concerning the LaMDA model and also concerning the current wave of LLM-powered chatbots, such as ChatGPT. This claim, if confirmed, would have serious ramifications in the Natural Language Processing (NLP) community due to wide-spread use of similar models. However, here we take the position that such a large language model cannot be conscious, and that LaMDA in particular exhibits no advances over other similar models that would qualify it. We justify this by analysing the Transformer architecture through Integrated Information Theory of consciousness. We see the claims of sentience as part of a wider tendency to use anthropomorphic language in NLP reporting. Regardless of the veracity of the claims, we consider this an opportune moment to take stock of progress in language modelling and consider the ethical implications of the task. In order to make this work helpful for readers outside the NLP community, we also present the necessary background in language modelling.
Overview
CoChat Summary
TL;DR
The paper argues that Transformer-based LLMs such as LaMDA cannot be conscious under Integrated Information Theory, and that claims of sentience arise from anthropomorphic language in NLP and media reporting. It also documents anthropomorphic terminology in ACL abstracts and offers recommendations for developers, users, and reporters.
Key findings
- Applying IIT, LaMDA and other Transformer-based LLMs cannot possess consciousness for three reasons: they execute as a sequence of Transformer blocks with extremely limited inter-block information exchange; the blocks are feed-forward networks with no recurrent connections; and they are implemented on modular, low-connectivity computer hardware.
- In LaMDA, the link between modules corresponds to a single word represented by 8192 numbers, while within-module links are at least the number of non-embedding weights, 137 billion in the most advanced LaMDA; within-module links outnumber between-module links by at least 8 orders of magnitude.
- LaMDA has 64 layers, 128 attention heads, 137 billion parameters, and was trained on 1.56 trillion words; the paper argues there is no breakthrough innovation in such models that would give rise to consciousness, and that apparent consciousness is a simulated language style produced by prompting.
- In an analysis of the first 100 abstracts from the 2022 ACL Annual Meeting, 90 referred to language modelling technology and 36 used anthropomorphic terminology; media reporting of LaMDA was found to use explicit anthropomorphism.
Methods
- Conceptual and argumentative analysis: the paper reviews language modelling, Transformer-based LLMs, and LaMDA, then assesses them against Integrated Information Theory as presented by Tononi and Koch, including recent LLM developments such as larger models, larger context windows, non-Transformer architectures, and retrieval-augmented generation.
- Qualitative audit: the authors analysed the first 100 abstracts in order of appearance from the proceedings of the 2022 ACL Annual Meeting, identified anthropomorphic terminology, and proposed a three-level framework—non-anthropomorphic, ambiguous anthropomorphism, and explicit anthropomorphism—which they applied to the LaMDA paper and selected media reporting.
- No empirical consciousness test or direct experimental manipulation of LaMDA is reported; the study is an analysis of architecture, theory, and language use.
Limitations
- Stated: the identification of anthropomorphic language relies on the subjective application of the framework by one author, and others may reach different conclusions; the authors acknowledge their inherent bias, namely non-belief in the model’s consciousness, and cannot claim to have fully overcome it.
- Stated: the study presupposes that distinguishing consciousness from non-consciousness in LLMs is valuable; it uses IIT because it appears most applicable to neural networks, but other approaches are possible; conclusions are limited to English-language models and examples, and generalisability is argued for Transformer-based models rather than all possible architectures.
- Evident: the media analysis is limited to selected sources around LaMDA, and the anthropomorphism audit is not presented with inter-rater reliability or a full quantitative coding protocol; the paper does not directly test LaMDA or other LLMs for consciousness.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
22 Valid · 13 Not valid · 12 Not checked · 68 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“weakly connected aggregates are not conscious [9].”
§ Deanthropomorphising NLP: Can a language model be conscious? > Can an LLM be conscious? > Transformer-based LLMs in IIT
What the sources say
- Supportedcites [9] G Tononi, 2015, Consciousness: here, there and everywhere?, Philosophical Transactions of the Royal Society B: Biological Sciences source
In “(f) Aggregates are not conscious,” the paper says that interacting smaller complexes can form a larger system without forming a superordinate conscious entity. It explicitly concludes that aggregates have no consciousness; this does not mean their constituent complexes cannot be conscious. (LLM, conf=0.93)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“And computer hardware is unlikely to be conscious itself because of the way it is designed: as a collection of modules, which are tightly integrated internally, but with limited connection between them [9].”
§ Deanthropomorphising NLP: Can a language model be conscious? > Can an LLM be conscious? > Transformer-based LLMs in IIT
What the sources say
- Partly supportedcites [9] G Tononi, 2015, Consciousness: here, there and everywhere?, Philosophical Transactions of the Royal Society B: Biological Sciences sourceExperimental — accuracy not yet measured
The section ‘Simulations of conscious neural systems can be unconscious’ says a digital computer would likely break into many low-integration mini-complexes because its circuitry has limited fan-in and fan-out. That supports the claim’s central point, but the paper does not describe the components as tightly integrated internally, and its Endnotes allow that differently designed neuromorphic hardware could potentially approach human consciousness. (LLM, conf=0.84)
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Models of consciousness from neurology rely on stochastic variation [65] arising from noise caused by varied biological sources [66].”
§ Deanthropomorphising NLP: Can a language model be conscious? > Can an LLM be conscious? > Challenges for consciousness beyond IIT
What the sources say
- Supportedcites [65] HA Braun, 2021, Stochasticity Versus Determinacy in Neurobiology: From Ion Channels to the Question of the “Free Will”, Frontiers in Systems Neuroscience source
This source is cited for the noise clause [66], not the preceding claim about models of consciousness [65]. In “The Need of Appropriately Adjusted System Dynamics” (page not specified), the paper describes noise associated with synaptic connections, ion-channel stochasticity, and irregular neuronal firing patterns. (LLM, conf=0.88)
- Supportedcites [66] P Kraikivski, 2021, Implications of Noise on Neural Correlates of Consciousness: A Computational Analysis of Stochastic Systems of Mutually Connected Processes, Entropy source
The citation [66] is attached to the claim that neuronal noise arises from varied biological sources. The Introduction identifies voltage-gated channel, synaptic, and sensory-motor noise as sources in neuronal systems. (LLM, conf=0.91)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“According to IIT, this transition does not preserve consciousness, even if a simulated system can perform the same functions as the original one.”
§ Deanthropomorphising NLP: Can a language model be conscious? > Can an LLM be conscious? > Transformer-based LLMs in IIT
What the sources say
- Supportedcites [9] G Tononi, 2015, Consciousness: here, there and everywhere?, Philosophical Transactions of the Royal Society B: Biological Sciences source
The section “Simulations of conscious neural systems can be unconscious” says that faithfully replicating a brain’s functions in software does not justify granting the simulation the brain’s consciousness. It states that a simulation of a conscious brain will not have consciousness because IIT requires the relevant physical cause-and-effect power. (LLM, conf=0.94)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Tononi and Koch [9] show why such a system cannot be conscious.”
- Also stated in: § Transformer-based LLMs in IIT · plain
“Following this procedure recursively eliminates the whole network, showing that a feed-forward network cannot be conscious according to IIT.”
§ Deanthropomorphising NLP: Can a language model be conscious? > Can an LLM be conscious? > Transformer-based LLMs in IIT
What the sources say
- Supportedcites [9] G Tononi, 2015, Consciousness: here, there and everywhere?, Philosophical Transactions of the Royal Society B: Biological Sciences source
In “Multiple consciousnesses,” Tononi and Koch explain that a purely feed-forward network has no recurrent connections and, according to IIT, carries out tasks unconsciously. The paper also explicitly predicts that feed-forward networks are not conscious. (LLM, conf=0.95)
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § Transformer-based LLMs in IIT · plain
50 more claims · 9 with their cited sources checked
Create a free account to see all 115 claims and to verify the 3 references still open.
- Probably not a substantial claim. 52
- It may state two separate claims as one. 21
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Deanthropomorphising NLP: Can a language model be conscious? > Can an LLM be conscious? > Challenges for consciousness beyond IIT
“The susceptibility to prompting has been also observed in ChatGPT, which started to provide abusive and threatening responses after being asked about its ‘shadow self’ [67].”
§ Deanthropomorphising NLP: Can a language model be conscious? > References
“67. Roose K. Why a Conversation with Bing’s Chatbot Left Me Deeply Unsettled; 2023.”
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · cited 1×
“This means that our conclusions, especially regarding consciousness, apply to other models such as BERT (Devlin et al, 2019), GPT-3 (Brown et al, 2020a), T5 (Raffel et al, 2020) and future models using the same, or similar, architectures.”
I.—COMPUTING MACHINERY AND INTELLIGENCE
TURING, A. M. (1950) · Mind · doi:10.1093/mind/lix.236.433 · Cited by 10,290
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821 · cited 1×
“Among the most popular are XLNet (Yang et al, 2019), RoBERTa (Liu et al, 2019), BART (Lewis et al, 2020) and ELEC-TRA (Clark et al, 2020).”
Nagel, T. (1974) · The Philosophical Review · doi:10.2307/2183914 · Cited by 7,440
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988
Searle, J. R. (1980) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00005756 · Cited by 6,954
International Conference on Learning Representations (ICLR 2013)
松尾, 豊. (2013) · 人工知能学会誌 = Journal of Japanese Society for Artificial Intelligence · Cited by 6,286
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208
Introduction to WordNet: An On-line Lexical Database*
Miller, G. A., Beckwith, R., Fellbaum, C., Gross, D., & Miller, K. J. (1990) · International Journal of Lexicography · doi:10.1093/ijl/3.4.235 · Cited by 4,197
On seeing human: A three-factor theory of anthropomorphism
Epley, N., Waytz, A., & Cacioppo, J. T. (2007) · Psychological Review · doi:10.1037/0033-295x.114.4.864 · Cited by 4,069 · cited 2×
“In fact, we find that the claims of sentience are the results of leading the model into responding as a human agent and incorrectly inferring anthropomorphic properties on the resulting generated sequences.”
Energy and Policy Considerations for Deep Learning in NLP
Strubell, E., Ganesh, A., & McCallum, A. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1355 · Cited by 3,684 · cited 1×
“…resources, not only limiting the ability to investigate them to a few powerful research centers (Luitse and Denkena, 2021), but also resulting in significant emissions outputs (Strubell et al, 2019), as does the conference system associated with reporting on these (Przybyła and Shardlow, 2022).”
Speech and language processing
(2010) · Day-to-Day Dyslexia in the Classroom · doi:10.4324/9780203461891_chapter_3 · Cited by 3,544
Training Language Models to Follow Instructions with Human Feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2011 · Cited by 3,117
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-T., Rocktäschel, T., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.11401 · Cited by 3,101
Model Cards for Model Reporting
Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., & Gebru, T. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287596 · Cited by 2,681 · cited 2×
“Importantly, work has focussed on the reproducibility of results in the LLM world, with a push towards the use of model cards to properly describe the underlying technology (Mitchell et al, 2019).”
Baker, C. F., Fillmore, C. J., & Lowe, J. B. (1998) · Proceedings of the 36th annual meeting on Association for Computational Linguistics - · doi:10.3115/980845.980860 · Cited by 2,518
Zero-shot text-to-image generation
Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., & Sutskever, I. (2021) · arXiv · doi:10.48550/arxiv.2102.12092 · Cited by 2,417
Facing up to the problem of consciousness
Chalmers, D. J. (1996) · Toward a Science of Consciousness · doi:10.7551/mitpress/6860.003.0003 · Cited by 1,966 · cited 1×
“This problem has later been explored by Chalmers (1995), who distinguishes the easy problems and the hard problem of consciousness.”
An information integration theory of consciousness
Tononi, G. (2004) · BMC Neuroscience · doi:10.1186/1471-2202-5-42 · Cited by 1,958 · cited 1×
“Integrated Information Theory (IIT), proposed by Tononi (2004), is a popular framework of this kind.”
Word sense disambiguation: A survey
Navigli, R. (2009) · ACM Computing Surveys · doi:10.1145/1459352.1459355 · Cited by 1,819 · cited 1×
“Fine-grained (more senses) and coarse-grained (fewer senses) representations of meaning were developed to allow researchers to develop systems to interact with lexical meaning (Navigli, 2009).”
A Synopsis of Linguistic Theory, 1930-1955
Firth, J. R. (1957) · Medical Entomology and Zoology · Cited by 1,675
Emergent Abilities of Large Language Models
Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.07682 · Cited by 1,264
Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data
Bender, E. M., & Koller, A. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.463 · Cited by 1,082 · cited 2×
“For other discussions of the ability of machines to interpret language and the degree to which a machine may ‘understand’ a text, we refer the reader to the recent work of Bender and Koller (2020), Bender et al (2021) and Mitchell and Krakauer (2022).”
(1994) · CERN Document Server (European Organization for Nuclear Research) · Cited by 1,062
Continuous speech recognition by statistical methods
Jelinek, F. (1976) · Proceedings of the IEEE · doi:10.1109/proc.1976.10159 · Cited by 920 · cited 2×
“What do we need language modelling for? Primarly, knowing what word is most likely in a given context has found direct applications in scenarios of recovering text from noisy or unreliable sources, for example in OCR (Smith, 2011), speech recognition (Jelinek, 1976) or machine translation (Koehn, 2009).”
Consciousness: here, there and everywhere?
Tononi, G., & Koch, C. (2015) · Philosophical Transactions of the Royal Society B: Biological Sciences · doi:10.1098/rstb.2014.0167 · Cited by 811 · cited 1×
“And computer hardware is unlikely to be conscious itself because of the way it is designed: as a collection of modules, which are tightly integrated internally, but with limited connection between them (Tononi and Koch, 2015).”
Open artificial intelligence platforms in nursing education: Tools for academic progress or abuse?
O’Connor, S., & ChatGPT (2023) · Nurse Education in Practice · doi:10.1016/j.nepr.2022.103537 · Cited by 585 · cited 1×
“Articles that have gone through peer review are less likely to include exaggerated claims than those appearing as preprints or those driven by commercial pressure.”
Baker, C. F., Fillmore, C. J., & Lowe, J. B. (1998) · Proceedings of the 17th international conference on Computational linguistics - · doi:10.3115/980451.980860 · Cited by 522
D’Angiulli, A., Runge, M., Faulkner, A., Zakizadeh, J., Chan, A., & Morcos, S. (2013) · Frontiers in Psychology · doi:10.3389/fpsyg.2013.00001 · Cited by 509
RWKV: Reinventing RNNs for the Transformer Era
Peng, B., Alcaide, E., Anthony, Q., Albalak, A., Arcadinho, S., Biderman, S., Cao, H., Cheng, X., Chung, M., Derczynski, L., et al. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.936 · Cited by 390
Opinion Mining with Deep Recurrent Neural Networks
Irsoy, O., & Cardie, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1080 · Cited by 384 · cited 2×
“In addition to this, reinforcement learning has been applied to generate a reward process that prioritises responses from a LLM that follow a set of guidelines determined by the developer (Kwon et al, 2023).”
Improving language models by retrieving from trillions of tokens
Borgeaud, S., Mensch, A., Hoffmann, J., Cai, T., Rutherford, E., Millican, K., Driessche, G. V. D., Lespiau, J.-B., Damoc, B., Clark, A., et al. (2021) · arXiv · doi:10.48550/arxiv.2112.04426 · Cited by 382
The sciences of science communication
Fischhoff, B. (2013) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1213273110 · Cited by 362
The debate over understanding in AI’s large language models
Mitchell, M., & Krakauer, D. C. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2215907120 · Cited by 340 · cited 1×
“For other discussions of the ability of machines to interpret language and the degree to which a machine may ‘understand’ a text, we refer the reader to the recent work of Bender and Koller (2020), Bender et al (2021) and Mitchell and Krakauer (2022).”
Salles, A., Evers, K., & Farisco, M. (2020) · AJOB Neuroscience · doi:10.1080/21507740.2020.1740350 · Cited by 335
Consciousness in humans and non-human animals: recent advances and future directions
Boly, M., Seth, A. K., Wilke, M., Ingmundson, P., Baars, B., Laureys, S., Edelman, D. B., & Tsuchiya, N. (2013) · Frontiers in Psychology · doi:10.3389/fpsyg.2013.00625 · Cited by 291 · cited 1×
“This is different to biological consciousness, where animals similar to humans exhibit multiple neural characteristics associated with our consciousness (Boly et al, 2013).”
Beam Search Strategies for Neural Machine Translation
Freitag, M., & Al-Onaizan, Y. (2017) · Proceedings of the First Workshop on Neural Machine Translation · doi:10.18653/v1/w17-3207 · Cited by 286 · cited 2×
“Selecting the most probable token at each step (greedy search), or the most probable future list of tokens (beam search) (Freitag and Al-Onaizan, 2017) leads to text which is highly probable, but does not reflect typical language patterns (Jacobs et al, 2022).”
Lexicon Infused Phrase Embeddings for Named Entity Resolution
Passos, A., Kumar, V., & McCallum, A. (2015) · Cited by 277
Lipton, Z. C., & Steinhardt, J. (2019) · Queue · doi:10.1145/3317287.3328534 · Cited by 231 · cited 1×
“Regarding the anthropomorphic language describing AI, its usage has been recognised as potentially detrimental in other AI sub-fields, including machine learning (Lipton and Steinhardt, 2019), brain-inspired AI (Salles et al, 2020) or art-generating AI (Epstein et al, 2020).”
(1994) · Journal of Consciousness Studies · Cited by 216
Rapamycin in the context of Pascal's Wager: generative pre-trained transformer perspective
Generative Pre-trained Transformer, C., & Zhavoronkov, A. (2022) · Oncoscience · doi:10.18632/oncoscience.571 · Cited by 203 · cited 1×
“Articles that have gone through peer review are less likely to include exaggerated claims than those appearing as preprints or those driven by commercial pressure.”
Do Large Language Models Understand Us?
y Arcas, B. A. (2022) · Daedalus · doi:10.1162/daed_a_01909 · Cited by 197
The great Transformer: Examining the role of large language models in the political economy of AI
Luitse, D., & Denkena, W. (2021) · Big Data & Society · doi:10.1177/20539517211047734 · Cited by 190 · cited 2×
“But their proliferation has been accompanied with a growing recognition of their potential for harm (Bender et al, 2021; Okerlund et al, 2022; Luitse and Denkena, 2021).”
Human Language Understanding & Reasoning
Manning, C. D. (2022) · Daedalus · doi:10.1162/daed_a_01905 · Cited by 162 · cited 2×
“A more sympathetic treatment of machine understanding is given in alternative literature, where researchers have claimed that LLMs are paving the way for General AI (Manning, 2022), or considered”
Who Gets Credit for AI-Generated Art?
Epstein, Z., Levine, S., Rand, D. G., & Rahwan, I. (2020) · iScience · doi:10.1016/j.isci.2020.101515 · Cited by 157 · cited 2×
“Regarding the anthropomorphic language describing AI, its usage has been recognised as potentially detrimental in other AI sub-fields, including machine learning (Lipton and Steinhardt, 2019), brain-inspired AI (Salles et al, 2020) or art-generating AI (Epstein et al, 2020).”
The Rhetoric and Reality of Anthropomorphism in Artificial Intelligence
Watson, D. (2019) · Minds and Machines · doi:10.1007/s11023-019-09506-6 · Cited by 148 · cited 1×
“Even for the AI techniques that were explicitly modelled to resemble human neurons, the similarity to actual human cognition has been described as overstated, possibly leading to dangerous misunderstandings (Watson, 2019).”
Medical Semantic Similarity with a Neural Language Model
De Vine, L., Zuccon, G., Koopman, B., Sitbon, L., & Bruza, P. (2014) · Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management · doi:10.1145/2661829.2661974 · Cited by 145
Word meaning in minds and machines.
Lake, B. M., & Murphy, G. L. (2023) · Psychological Review · doi:10.1037/rev0000297 · Cited by 131 · cited 2×
“A more sympathetic treatment of machine understanding is given in alternative literature, where researchers have claimed that Large Language models are paving the way for General AI (Manning, 2022), or considered LMs from the perspective of the human mind (Lake and Murphy, 2021).”
Lexicon Infused Phrase Embeddings for Named Entity Resolution
Passos, A., Kumar, V., & McCallum, A. (2014) · Conference on Computational Natural Language Learning · doi:10.3115/v1/w14-1609 · Cited by 130 · cited 2×
“In addition to this, reinforcement learning has been applied to generate a reward process that prioritises responses from a LLM that follow a set of guidelines determined by the developer (Kwon et al, 2023).”
Law, A. K., Gupta, D., Levy, S., Wallace, D. C., McKeon, R. J., & Buck, C. R. (2004) · BMC Neuroscience · doi:10.1186/1471-2202-5-1 · Cited by 124
Markov, A. A. (2006) · Science in Context · doi:10.1017/s0269889706001074 · Cited by 121 · cited 2×
“When the task was first explored by Andrey Andreyevich Markov in 1913 (Markov, 2006), he simply calculated how many times a token t c followed another token t ∗ in a long text 3 .”
On the Origin of Hallucinations in Conversational Models: Is it the Datasets or the Models?
Dziri, N., Milton, S., Yu, M., Zaiane, O., & Reddy, S. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.387 · Cited by 121
Science, sentience, and animal welfare
Jones, R. C. (2013) · Biology & Philosophy · doi:10.1007/s10539-012-9351-1 · Cited by 119
PyPhi: A toolbox for integrated information theory
Mayner, W. G. P., Marshall, W., Albantakis, L., Findlay, G., Marchman, R., & Tononi, G. (2018) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1006343 · Cited by 118 · cited 1×
“While the necessary software is available (Mayner et al, 2018), it is limited due to processing time depending on the number of nodes ( n ) as O ( n 53 n ) .”
Kuzovkin, I., Vicente, R., Petton, M., Lachaux, J.-P., Baciu, M., Kahane, P., Rheims, S., Vidal, J. R., & Aru, J. (2018) · Communications Biology · doi:10.1038/s42003-018-0110-y · Cited by 104 · cited 1×
“Similarly, the visual cortex in the human brain, which has been shown to work similarly to a feed-forward neural network (Kuzovkin et al, 2018), does not possess its own consciousness.”
Hyena Hierarchy: Towards Larger Convolutional Language Models
Poli, M., Massaroli, S., Nguyen, E., Fu, D. Y., Dao, T., Baccus, S., Bengio, Y., Ermon, S., & Ré, C. (2023) · arXiv · doi:10.48550/arxiv.2302.10866 · Cited by 96
25 Continuous speech recognition: Statistical methods
Jelinek, F., Mercer, R. L., & Bahl, L. R. (1982) · Handbook of statistics · doi:10.1016/s0169-7161(82)02028-8 · Cited by 80
Braun, H. A. (2021) · Frontiers in Systems Neuroscience · doi:10.3389/fnsys.2021.629436 · Cited by 64 · cited 1×
“Examples of consciousness from neurology rely on stochastic variation (Braun, 2021) arising from noise caused by varied biological sources (Kraikivski, 2021).”
Document classification with distributions of word vectors
Xing, C., Wang, D., Zhang, X., et al. (2014) · doi:10.1109/apsipa.2014.7041633 · Cited by 53 · cited 2×
“In addition to this, reinforcement learning has been applied to generate a reward process that prioritises responses from a LLM that follow a set of guidelines determined by the developer (Kwon et al, 2023).”
Reward Design with Language Models
Kwon, M., Xie, S. M., Bullard, K., & Sadigh, D. (2023) · arXiv · doi:10.48550/arxiv.2303.00001 · Cited by 45
Limits on the Application of Frequency-Based Language Models to OCR
Smith, R. W. (2011) · Proceedings of the International Conference on Document Analysis and Recognition · doi:10.1109/icdar.2011.114 · Cited by 39 · cited 1×
“Pri-marly, knowing what word is most likely in a given context has found direct applications in scenarios of recovering text from noisy or unreliable sources, for example in OCR (Smith, 2011), speech recognition (Jelinek, 1976) or machine translation (Koehn, 2009).”
WHAT’S IN THE CHATTERBOX? LARGE LANGUAGE MODELS, WHY THEY MATTER, AND WHAT WE SHOULD DO ABOUT THEM
Okerlund, J., Klasky, E., Middha, A., Kim, S., Rosenfeld, H., Kleinman, M., & Parthasarathy, S. (2022) · Deep Blue (University of Michigan) · doi:10.7302/21898 · Cited by 37
Church, K. W., & Kordoni, V. (2022) · Natural Language Engineering · doi:10.1017/s1351324922000043 · Cited by 28 · cited 2×
“The large companies behind them were able to invest ever-increasing amounts of time, money and carbon emissions to obtain the stateof-the-art results in standard benchmarks – the process known as SOTA-chasing (Church and Kordoni, 2022).”
Generalizing frameworks for sentience beyond natural species
Levin, M. (2022) · Animal Sentience · doi:10.51291/2377-7478.1733 · Cited by 28
The Role of Sentience in the Theory of Consciousness and Medical Practice
Pereira, A. (2020) · Unpublished · doi:10.13140/rg.2.2.22668.74885 · Cited by 14
On the Origin of Hallucinations in Conversational Models: Is it the Datasets or the Models?
Dziri, N., Milton, S., Yu, M., Zaiane, O., & Reddy, S. (2022) · arXiv · doi:10.48550/arxiv.2204.07931 · Cited by 13
Does Machine Understanding Require Consciousness?
Pepperell, R. (2022) · Frontiers in Systems Neuroscience · doi:10.3389/fnsys.2022.788486 · Cited by 11 · cited 1×
“More generally, other researchers have sought to apply cognitive frameworks to the problem of language modelling (Pepperell, 2022; Levin, 2022; Chalmers, 2023), reaching similar conclusions to ours — that a LLM cannot be considered conscious, or indeed sentient, based on the current evidence.”
Pustejovsky, J. D. (2006) · Elsevier eBooks · doi:10.1016/b0-08-044854-2/01095-6 · Cited by 9 · cited 2×
“However, herein we only take into account the narrower point of view of lexical semantics, which focuses on the relationship between the linguistic forms and the concepts they refer to (Pustejovsky, 2005).”
BackgroundBrooker, P., Dutton, W., & Mair, M. (2019) · Zentral- und Hochschulbibliothek Luzern · doi:10.5281/zenodo.3459326 · Cited by 8
Koehn, P. (2009) · Cambridge University Press eBooks · doi:10.1017/cbo9780511815829.008 · Cited by 8
Using NLP to quantify the environmental cost and diversity benefits of in-person NLP conferences
Przybyła, P., & Shardlow, M. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.304 · Cited by 6 · cited 1×
“…resources, not only limiting the ability to investigate them to a few powerful research centers (Luitse and Denkena, 2021), but also resulting in significant emissions outputs (Strubell et al, 2019), as does the conference system associated with reporting on these (Przybyła and Shardlow, 2022).”
Self-citationKraikivski, P. (2021) · Entropy · doi:10.3390/e23050583 · Cited by 4 · cited 1×
“Examples of consciousness from neurology rely on stochastic variation (Braun, 2021) arising from noise caused by varied biological sources (Kraikivski, 2021).”
Masked language models directly encode linguistic uncertainty
Jacobs, C., Hubbard, R. J., & Federmeier, K. D. (2022) · ScholarWorks@UMassAmherst (University of Massachusetts Amherst) · doi:10.7275/znzq-3m28 · Cited by 3
Brooker, P., Dutton, W., & Mair, M. (2019) · Zentral- und Hochschulbibliothek Luzern · doi:10.5281/zenodo.3459327
Kuzovkin, I., Vicente, R., Petton, M., Lachaux, J.-P., Baciu, M., Kahane, P., Rheims, S., Vidal, J. R., & Aru, J. (2017) · doi:10.1101/133694
Word meaning in minds and machines
Lake, B. M., & Murphy, G. L. (2020) · arXiv · doi:10.48550/arxiv.2008.01766
Research works printed but not linked: 13
- Brown T, Mann B, Ryder N, Subbiah M, Kaplan JD, Dhariwal P, et al. Language Models are Few-Shot Learners. In: Larochelle H, Ranzato M, Hadsell R, Balcan MF, Lin H, editors. Advances in Neural Information Processing Systems. vol. 33. Curran Associates, Inc.; 2020. p. 1877–1901. Available from: https://proceedings.neurips.cc/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf.
- Thoppilan R, De Freitas D, Hall J, Shazeer N, Kulshreshtha A, Cheng HT, et al. Lamda: Language models for dialog applications. arXiv preprint arXiv:220108239. 2022;.
- J Austen, 1813, Pride and Prejudice
- Mikolov T, Sutskever I, Chen K, Corrado GS, Dean J. Distributed Representations of Words and Phrases and their Compositionality. In: Burges CJ, Bottou L, Welling M, Ghahramani Z, Weinberger KQ, editors. Advances in Neural Information Processing Systems. vol. 26. Curran Associates, Inc.; 2013.Available from: https://proceedings.neurips.cc/paper/2013/file/9aa42b31882ec039965f3c4923ce901b-Paper.pdf.
- Yang Z, Dai Z, Yang Y, Carbonell J, Salakhutdinov R, Le QV. XLNet: Generalized Autoregressive Pretraining for Language Understanding. arXiv: 190608237. 2019;.
- Clark K, Luong T, Le QV, Manning C. ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators. In: Proceddings of ICLR; 2020.Available from: https://openreview.net/pdf?id=r1xMH1BtvB.
- Zhu F, Lei W, Wang C, Zheng J, Poria S, Chua TS. Retrieving and reading: A comprehensive survey on open-domain question answering. arXiv preprint arXiv:210100774. 2021;.
- Chowdhery A, Narang S, Devlin J, Bosma M, Mishra G, Roberts A, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:220402311. 2022;.
- Zhang S, Roller S, Goyal N, Artetxe M, Chen M, Chen S, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:220501068. 2022;.
- Beltagy I, Peters ME, Cohan A. Longformer: The long-document transformer. arXiv preprint arXiv:200405150. 2020;.
- Khandelwal U, Levy O, Jurafsky D, Zettlemoyer L, Lewis M. Generalization through Memorization: Nearest Neighbor Language Models. In: 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net; 2020.Available from: https://openreview.net/forum?id=HklBjCEKvH.
- Fu DY, Dao T, Saab KK, Thomas AW, Rudra A, Re C. Hungry Hungry Hippos: Towards Language Modeling with State Space Models. In: The Eleventh International Conference on Learning Representations; 2022.
- Liang PP, Wu C, Morency LP, Salakhutdinov R. Towards Understanding and Mitigating Social Biases in Language Models. In: Meila M, Zhang T, editors. Proceedings of the 38th International Conference on Machine Learning. vol. 139 of Proceedings of Machine Learning Research. PMLR; 2021. p. 6565–6576. Available from: https://proceedings.mlr.press/v139/liang21a.html.
Cited by
Language Model Behavior: A Comprehensive Survey
Chang, T. A., & Bergen, B. K. (2024) · Computational Linguistics · doi:10.1162/coli_a_00492 · Cited by 105
Folk Psychological Attributions of Consciousness to Large Language Models
Colombatto, C., & Fleming, S. M. (2023) · doi:10.31234/osf.io/5cnrv · Cited by 96
Folk psychological attributions of consciousness to large language models
Colombatto, C., & Fleming, S. M. (2024) · Neuroscience of Consciousness · doi:10.1093/nc/niae013 · Cited by 96
The influence of mental state attributions on trust in large language models
Colombatto, C., Birch, J., & Fleming, S. M. (2025) · Communications Psychology · doi:10.1038/s44271-025-00262-1 · Cited by 34
“Thetendency toassignmentalstatestoAIsystemsisindependentfromwhetherthese systemstrulypossessthem — and indeed, the extent to which current AI systems possess consciousness remains contentious in scienti fi c analysis 1 – 5 .”
Six Fallacies in Substituting Large Language Models for Human Participants
Lin, Z. (2025) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459251357566 · Cited by 64
“Explicitly acknowledge in research designs and reports that current LLMs do not possess mental states or consciousness (Shardlow & Przybyla, 2024).”
Metadata
- Authors
- Shardlow, M., & Przybyła, P.
- Affiliations
- Manchester Metropolitan University (GB)
- Universitat Pompeu Fabra (ES)
- Corresponding author
- Shardlow, Matthew
- Venue
- PLoS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2024-12-04
- Volume
- 19
- Issue
- 12
- Pages
- e0307521
- ISSN
- 1932-6203 (ISSN-L)
- PMC
- PMC11617003
- OpenAlex
- W4405017998
- Cited by
- 16
- References
- 80
- Citations by year
- 2026: 7 · 2025: 6 · 2024: 2
- FWCI
- 6.11
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS ONE (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Repositori digital de la UPF (Universitat Pompeu Fabra) (publishedVersion, cc-by)
- e-space (Manchester Metropolitan University) (acceptedVersion)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Funders
- HORIZON EUROPE Framework Programme (101060930)
- Topics
- Social Sciences › Arts and Humanities › Philosophy › Philosophy and Theoretical Science
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Social Sciences › Arts and Humanities › History and Philosophy of Science › Cybernetics and Technology in Society
- Keywords
- large language models, sentience, consciousness, LaMDA, transformer models, integrated information theory, anthropomorphism, chatbots, ChatGPT, AI ethics, language modeling
- MeSH terms
- Consciousness, Humans, Language, Models, Theoretical, Natural Language Processing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
