CoChat
Sign Up
Cited by 15Cybernetics and Technology in SocietyWittgensteinian philosophy and applicationsEthics and Social Impacts of AI

The bewitching AI: The Illusion of Communication with Large Language Models

Open access
Bottazzi Grifoni, Emanuele, Ferrario, Roberta·2025·Philosophy & Technology
DOI 10.1007/s13347-025-00893-6
Open this paper in CoChat — free accountOpen PDF

Abstract

Abstract We investigate the ‘bewitchment’ of understanding interactions between humans and systems based on large language models (LLMs) inspired by Wittgenstein’s later view on language. This framework is particularly apt for analyzing human-LLM interaction as it treats understanding as a public phenomenon manifested in observable communicative practices, rather than as a mental or computational state—an approach especially valuable given LLMs’ inherent opacity. Drawing on this perspective, we show that successful communication requires not merely regularity in language use, but constancy in maintaining reference points through agreement in both definitions and judgments. Crucially, LLMs lack the constancy needed to track negations and contradictions throughout a dialogue, thereby disrupting the reference points necessary for genuine communication. The apparent understanding in human-LLM interactions arises from what we characterize as a ‘bewitchment’: the interaction between LLMs’ statistical adherence to linguistic patterns and humans’ tendency to blindly follow familiar language games. Moreover, when interaction with LLMs is based on stereotyped contexts in which the system seems capable of identifying reference points, we humans automatically apply the practical principle that there is understanding until proven otherwise. The bewitchment becomes thus more profound as LLMs improve in modeling stereotypical aspects of human interaction. This improvement, far from addressing the highlighted limitations, can only deepen the illusion of understanding, raising significant concerns for meaningful control over such systems.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1007/s13347-025-00893-6

Cites

112 of 112 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The conversational nature of large language models further amplifies the effect of fluency by fostering a sense of dialogue and understanding.

    “The conversational nature of large language models further amplifies this effect by fostering a sense of dialogue and understanding [31].”

    From Fluency Illusion: A Review on Influence of ChatGPT in Classroom Settings (2026) · cites this paper as [31]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

1 of the 113 references couldn’t be matched to a known paper.

Research works printed but not linked: 30

  1. Atil, B., Aykent, S., Chittams, A., Fu, L., Passonneau, R. J., Radcliffe, E., Rajagopal, G. R., Sloan, A., Tudrej, T., Ture, F., Wu, Z., Xu, L., & Baldwin, B. (2025). Non-determinism of "deterministic" LLM settings. arXiv. https:// arxiv. org/ pdf/ 2408. 04667
  2. Augustine. (1992). Sermons III/5 (148-183) on the New Testament (E. Hill, Trans.). New City Press.
  3. Bengio, Y., Cohen, M., Fornasiere, D., Ghosn, J., Greiner, P., MacDermott, M., Mindermann, S., Oberman, A., Richardson, J., Richardson, O., Rondeau, M.-A., St-Charles, P.-L., & Williams- King, D. (2025). Superintelligent agents pose catastrophic risks: Can scientist AI offer a safer path? arXiv. https:// arxiv. org/ abs/ 2502. 15657. Accessed 16 Mar 2025.
  4. Chalmers, D. J. (2023). Could a Large Language Model be Conscious? arXiv. https:// arxiv. org/ abs/ 2303. 07103. Accessed 10 Dec 2024.
  5. de Balzac, H. (1976). Avant-propos à La Comédie humaine. In P. -G. Castex (Ed.). Gallimard. (Origi- nal work published 1842).
  6. Douglas, M. R. (2023). Large language models. arXiv. https:// arxiv. org/ abs/ 2307. 05782. Accessed 10
  7. Du, Y., Li, S., Torralba, A., Tenenbaum, J. B., & Mordatch, I. (2023). Improving Factuality and Reasoning in Language Models through Multiagent Debate. arXiv. https:// arxiv. org/ abs/ 2305. 14325. Accessed 10 Dec 2024.
  8. Eagle, A. (2021). Chance versus Randomness. In E. N. Zalta (Ed.), The Stanford Encyclopedia of Philos- ophy (Spring 2021). Metaphysics research lab, Stanford University. Retrieved December 10, 2024, from https:// plato. stanf ord. edu/ archi ves/ spr20 21/ entri es/ chance-rando mness/
  9. Eriksson, M., Purificato, E., Noroozian, A., Vinagre, J., Chaslot, G., Gomez, E., & Fernandez-Llorca, D. (2025). Can we trust AI benchmarks? An interdisciplinary review of current issues in AI evalua- tion. arXiv. https:// arxiv. org/ abs/ 2502. 06559. Accessed 16 Mar 2025.
  10. ch15 · DOI
  11. Goldstein, L. (2004). The Barber, Russell's Paradox, Catch-22, God, Contradiction, and More. In G. Priest, J. Beall, & B. P. Armour-Garb (Eds.), The law of non-contradiction: New philosophical essays (pp. 295-313). Oxford University Press.
  12. Haimes, J., Wenner, C., Thaman, K., Tashev, V., Neo, C., Kran, E., & Schreiber, J. (2024). Benchmark inflation: Revealing LLM performance gaps using retro-holdouts. arXiv. https:// arxiv. org/ abs/ 2410. 09247. Accessed 28 Apr 2025.
  13. Han, S., Schoelkopf, H., Zhao, Y., Qi, Z., Riddell, M., Zhou, W., Coady, J., Peng, D., Qiao, Y., Benson, L., Sun, L., Wardle-Solano, A., Szabo, H., Zubova, E., Burtell, M., Fan, J., Liu, Y., Wong, B., Sailor, M., … Radev, D. (2024). FOLIO: Natural language reasoning with first-order logic. arXiv. https:// arxiv. org/ abs/ 2209. 00840. Accessed 16 Mar 2025.
  14. Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., & Steinhardt, J. (2020). Measuring massive multitask language understanding. arXiv. https:// arxiv. org/ abs/ 2009. 03300. Accessed 28
  15. Horn, L. R., & Wansing, H. (2024). Negation. In E. N. Zalta & U. Nodelman (Eds.), The Stanford Ency- clopedia of Philosophy (Spring 2024). Metaphysics Research Lab, Stanford University. Retrieved March 16, 2025, from https:// plato. stanf ord. edu/ archi ves/ spr20 24/ entri es/ negat ion/
  16. Ji, J., Qiu, T., Chen, B., Zhang, B., Lou, H., Wang, K., Duan, Y., He, Z., Zhou, J., Zhang, Z., Zeng, F., Ng, K. Y., Dai, J., Pan, X., O'Gara, A., Lei, Y., Xu, H., Tse, B., Fu, J., McAleer, S., Yang, Y., Wang, Y., Zhu, S.-C., Guo, Y., & Gao, W. (2023a). AI alignment: A comprehensive survey. arXiv. https:// arxiv. org/ abs/ 2310. 19852. Accessed 10 Dec 2024.
  17. Jurafsky, D., & Martin, J. H. (2025). Speech and language processing: An introduction to natural lan- guage processing, computational linguistics, and speech recognition with language models (3rd ed.). Retrieved March 16, 2025, from https:// web. stanf ord. edu/ ~juraf sky/ slp3/
  18. Kettelhoit, F. (2022). Wittgenstein on the diagonal argument (Doctoral dissertation, Universität Kassel). Universität Kassel. https:// kobra. uni-kassel. de/ bitst ream/ handle/ 12345 6789/ 14054/ Disse rtati onFre deric Kette lhoit. pdf
  19. Liu, H., Ning, R., Teng, Z., Liu, J., Zhou, Q., & Zhang, Y. (2023). Evaluating the logical reasoning abil- ity of chatgpt and gpt-4. arXiv. https:// arxiv. org/ abs/ 2304. 03439. Accessed 28 Apr 2025.
  20. Macmillan-Scott, O., & Musolesi, M. (2025). (Ir)rationality in AI: State of the art, research challenges and open questions. arXiv. https:// arxiv. org/ abs/ 2311. 17165. Accessed 16 Mar 2025.
  21. McCoy, R. T., Yao, S., Friedman, D., Hardy, M., & Griffiths, T. L. (2023). Embers of Autoregression: Understanding Large Language Models Through the Problem They are Trained to Solve. arXiv. https:// arxiv. org/ abs/ 2309. 13638. Accessed 10 Dec 2024.
  22. Mousavi, S. M., Roccabruna, G., Alghisi, S., Rizzoli, M., Ravanelli, M., & Riccardi, G. (2024). Are LLMs Robust for Spoken Dialogues? arXiv. https:// arxiv. org/ abs/ 2401. 02297. Accessed 16 Mar 2025.
  23. Mulligan, K. (2002). Getting Geist-Certainty, Rules and Us. In M. Ouelbani (Ed.), Cinquantenaire Lud- wig Wittgenstein, Proceedings of the 2001 Tunis Wittgenstein conference (pp. 35-62). University of Tunis.
  24. Pascal, B. (1995). Pensées. In A. Levi (Ed.), Pensées and other writings (pp. 1-181). Oxford Univer- sity Press. (Original work published 1670)
  25. Peng, B., Narayanan, S., & Papadimitriou, C. (2024). On Limitations of the Transformer Architecture. arXiv. https:// arxiv. org/ abs/ 2402. 08164. Accessed 16 Mar 2025.
  26. P. S. Churchland, Foreword; D. Føllesdal, Pref- ace). MIT Press. (Original work published 1960)
  27. Tennant, N. (1987). Anti-Realism and Logic. Oxford University Press.
  28. Zhang, Y., Chen, X., Jin, B., Wang, S., Ji, S., Wang, W., & Han, J. (2024). A comprehensive survey of scientific large language models and their applications in scientific discovery. arXiv. https:// arxiv. org/ abs/ 2406. 10833. Accessed 28 Apr 2025.
  29. Zhang, C., Dai, X., Wu, Y., Yang, Q., Wang, Y., Tang, R., & Liu, Y. (2025). A survey on multi-turn inter- action capabilities of large language models. arXiv. https:// arxiv. org/ abs/ 2501. 09959. Accessed 28
  30. Zverev, E., Abdelnabi, S., Fritz, M., & Lampert, C. H. (2024). Can LLMs separate instructions from data? And what do we even mean by that? In ICLR 2024 Workshop on Secure and Trustworthy Large Language Models. Retrieved March 16, 2025, from https:// openr eview. net/ forum? id= 32eyt C1Nt1

Cited by

Sorted by citations

Related

Authors
Bottazzi Grifoni, E., & Ferrario, R.
Affiliations
National Research Council (IT)
Corresponding author
Ferrario, Roberta
ORCID
Venue
Philosophy & Technology
Publisher
Springer Science and Business Media LLC
Published
2025-05-05
Volume
38
Issue
2
ISSN
  • 2210-5433 (ISSN-L)
  • 2210-5441
DOI
10.1007/s13347-025-00893-6
OpenAlex
W4410092719
Cited by
15
References
113
Citations by year
2026: 12 · 2025: 3
FWCI
57.06
Type
journal article
Language
en
Open access
Licence
Locations
Philosophy & Technology (publishedVersion, cc-by)
Funders
  • Ministero dell'Università e della Ricerca (2020SSKZ7R, 202223E8Y4X, PE 00000013)
  • Consiglio Nazionale Delle Ricerche
Topics
  • Social Sciences › Arts and Humanities › History and Philosophy of Science › Cybernetics and Technology in Society
  • Social Sciences › Arts and Humanities › Philosophy › Wittgensteinian philosophy and applications
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
Keywords
large language models, human-LLM interaction, Wittgenstein, later Wittgenstein, language games, understanding, bewitchment, constancy, reference points, illusion of understanding

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review