The DOI matches its Crossref record; 15 of the 15 references it cites were found in a registry or an index.
Quasi-Metacognitive Machines: Why We Don’t Need Morally Trustworthy AI and Communicating Reliability Is Enough
Abstract
Many policies and ethical guidelines recommend developing “trustworthy AI”. We argue that developing morally trustworthy AI is not only unethical, as it promotes trust in an entity that cannot be trustworthy, but it is also unnecessary for optimal calibration. Instead, we show that reliability, exclusive of moral trust, entails the appropriate normative constraints that enable optimal calibration and mitigate the vulnerability that arises in high-stakes hybrid decision-making environments, without also demanding, as moral trust would, the anthropomorphization of AI and thus epistemically dubious behavior. The normative demands of reliability for inter-agential action are argued to be met by an analogue to procedural metacognitive competence (i.e., the ability to evaluate the quality of one’s own informational states to regulate subsequent action). Drawing on recent empirical findings that suggest providing reliability scores (e.g., F1-scores) to human decision-makers improves calibration in the AI system, we argue that reliability scores provide a good index of competence and enable humans to determine how much they wish to rely on the system.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Philosophical work questions whether artificial systems are appropriate objects of trust in the full interpersonal sense at all, suggesting that what users extend is often better described as reliance (Ryan 2020;Bryson 2018;Dorsch and Deroy 2024).”
From Epistemic Trust in Generative AI as an Information Source: Development and Validation of the Trust in AI-Epistemic Scale (TAI-E) (2026) · cites this paper as Dorsch and Deroy 2024
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Trust in Automation: Designing for Appropriate Reliance
Lee, J. D., & See, K. A. (2004) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/hfes.46.1.50_30392 · Cited by 6,605 · cited 2×
“The relevant vulnerability arises because of the need, on the part of the human agent, to calibrate her reliance on the AI’s performance (see Lee and See (2004), who refer to this as “trust calibration”).”
Human trust in artificial intelligence: Review of empirical research
Glikson, E., & Woolley, A. W. (2020) · Academy of Management Annals · doi:10.5465/annals.2018.0057 · Cited by 2,921
Baier, A. (1986) · Ethics · doi:10.1086/292745 · Cited by 2,567
Explaining Behavior: Reasons in a World of Causes
Dretske, F. (1988) · doi:10.7551/mitpress/2927.001.0001 · Cited by 1,512
Fleming, S. M., & Lau, H. C. (2014) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2014.00443 · Cited by 1,408 · cited 1×
“In the context of experimental psychology, metacognitive competence refers to an individual’s ability to accurately assess and evaluate their own cognitive processes, such as their memory, perception, or problem-solving abilities (Fleming & Lau, 2014).”
On the Morality of Artificial Agents
Floridi, L., & Sanders, J. W. (2004) · Minds and Machines · doi:10.1023/b:mind.0000035461.63578.9d · Cited by 1,218
Zhang, Y., Liao, Q. V., & Bellamy, R. K. E. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372852 · Cited by 895
In AI We Trust: Ethics, Artificial Intelligence, and Reliability
Ryan, M. (2020) · Science and Engineering Ethics · doi:10.1007/s11948-020-00228-y · Cited by 600 · cited 2×
“Though reliability has been previously proposed as the correct target for ethical AI (Ryan, 2020), this paper aims to complement this research by conducting a normative analysis of reliability (Section 2), defending the claim that morally trustworthy AI is unnecessary (Section 3 ), and ultimately…”
The Principles and Limits of Algorithm-in-the-Loop Decision Making
Green, B., & Chen, Y. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359152 · Cited by 392 · cited 1×
“The core program is that humans and AI ought to build upon each other’s strengths and compensate for their weaknesses (Bansal et al., 2021; cf. Green & Chen, 2019).”
Knowing Ourselves Together: The Cultural Origins of Metacognition
Heyes, C., Bang, D., Shea, N., Frith, C. D., & Fleming, S. M. (2020) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2020.02.007 · Cited by 232 · cited 2×
“…the capacity to metarepresent is much more sophisticated than the capacity to explicitly represent (for representing rules explicitly in machines, see Sharkey, 2020), since metarepresentation is often argued to enable reflective self-knowledge (see Heyes et al., 2020 for an account of this).”
Research works printed but not linked: 41
- Bahrami, B., Olsen, K., Bang, D., Roepstorff, A., Rees, G., & Frith, C. (2012). Together, slowly but surely: The role of social interaction and feedback on the build-up of benefit in collective decision-making. Journal of Experimental Psychology: Human Perception and Performance, 38(1), 3–8. https://doi.org/10.1037/a0025708. · DOI
- Bansal, G., Nushi, B., Kamar, E., Horvitz, E., & Weld, D. S. (2021). Is the most accurate AI the best teammate? Optimizing AI for Teamwork. Proceedings of the AAAI Conference on Artificial Intelligence, 35(13), 11405–11414. https://doi.org/10.1609/aaai.v35i13.17359. · DOI
- Bayne, T., Brainard, D., Byrne, R. W., Chittka, L., Clayton, N., Heyes, C., Mather, J., Ölveczky, B., Shadlen, M., Suddendorf, T., & Webb, B. (2019). What is cognition? Current Biology, 29(13), R608–r615. https://doi.org/10.1016/j.cub.2019.05.044. · DOI · Paper page
- Buechner, J., & Tavani, H. T. (2011). Trust and multi-agent systems: Applying the diffuse, default model of trust to experiments involving artificial agents. Ethics and Information Technology, 13(1), 39–51. https://doi.org/10.1007/s10676-010-9249-z. · DOI
- Cherkowski, S., Walker, K. D., & Kutsyuruba, B. (2015). Principals’ Moral Agency and ethical Decision-Making: Toward a transformational Ethics. International Journal of Education Policy and Leadership, 10(5), n5. https://eric.ed.gov/?id=EJ1138586. · DOI
- Coeckelbergh, M. (2012). Can we trust robots? Ethics and Information Technology, 14(1), 53–60. https://doi.org/10.1007/s10676-011-9279-1. · DOI · Paper page
- Davidson, D. (1963). Actions, reasons and causes. Journal of Philosophy, 60, 685–670. · DOI · Paper page
- Deroy, O. (2023). The Ethics of Terminology: Can we use human terms to describe AI? Topoi, 42(3), 881–889. https://doi.org/10.1007/s11245-023-09934-1. · DOI
- Deutsch, M. (1977). The resolution of conflict: Constructive and destructive processes. Yale University Press. https://doi.org/10.12987/9780300159356. · DOI
- Erasmus, A., Brunet, T. D. P., & Fisher, E. (2021). What is Interpretability? Philosophy & Technology, 34(4), 833–862. https://doi.org/10.1007/s13347-020-00435-2. · DOI
- Evans, G. (1982). The Varieties of Reference. Oxford: Oxford University Press.
- Fodor, J. (2008). LOT2: The language of thought revisited. Oxford University Press. · DOI · Paper page
- Gibbard, A. (1990). Wise choices, apt feelings: A theory of normative Judgment. Harvard University Press. · DOI · Paper page
- Goldman, A. I. (1976). Discrimination and perceptual knowledge. The Journal of Philosophy, 73(20), 771–791. https://doi.org/10.2307/2025679. · DOI · Paper page
- Haugeland, J. (1990). The Intentionality All-Stars. Philosophical Perspectives, 4, 383–427. https://doi.org/10.2307/2214199. · DOI
- Hieronymi, P. (2004). The Force and Fairness of blame. Philosophical Perspectives, 18(1), 115–148. https://doi.org/10.1111/j.1520-8583.2004.00023.x. · DOI · Paper page
- Johnson, D. (2006). Computer systems: Moral entities but not moral agents. Ethics of Information Technology, 8, 195–204. https://doi.org/10.1007/s10676-006-9111-5. · DOI · Paper page
- Kamar, E. (2016). Directions in Hybrid Intelligence: Complementing AI Systems with Human Intelligence. Proceedings of the 25th International Joint Conference on Artificial Intelligence, New York, 9–15 July 2016, 4070–4073.
- Lahno, B. (2001). On the emotional character of Trust. Ethical Theory and Moral Practice, 4(2), 171–189. https://doi.org/10.1023/A:1011425102875. · DOI
- Le, T., Miller, T., Singh, R., & Sonenberg, L. (2022). Improving model understanding and trust with counterfactual explanations of Model confidence. arXiv Preprint arXiv:220602790.
- Lebreton, M., Bacily, K., Palminteri, S., & Engelmann, J. B. (2019). Contextual influence on confidence judgments in human reinforcement learning. PLOS Computational Biology, 15(4), e1006973. https://doi.org/10.1371/journal.pcbi.1006973. · DOI
- Longin, L., Bahrami, B., & Deroy, O. (2023). Intelligence brings responsibility - even smart AI-assistants are held responsible. iScience, 107494. https://doi.org/10.1016/j.isci.2023.107494. · DOI
- Luhmann, N. (1979). Trust and Power. Wiley.
- Mainz, J. T. (2023). Medical AI: Is trust really the issue? Journal of Medical Ethics. https://doi.org/10.1136/jme-2023-109414. · DOI
- Mayer, R. C., Davis, J. H., & Schoorman, F. D. (1995). An Integrative Model of Organizational Trust. The Academy of Management Review, 20(3), 709–734. https://doi.org/10.2307/258792. · DOI · Paper page
- McDowell, J. (1994). Mind and world. Harvard University Press.
- Moor, J. H. (2006). The Nature, Importance, and Difficulty of Machine Ethics. IEEE Intelligent Systems, 21(4), 18–21. https://doi.org/10.1109/MIS.2006.80. · DOI · Paper page
- National Academies of Sciences. (2017). Communicating Science effectively: A Research Agenda. National Academies Press. https://doi.org/10.17226/23674. · DOI · Paper page
- O’Connor, C., & Weatherall, J. O. (2019). The misinformation age: How false beliefs spread. Yale University Press. · DOI · Paper page
- Peacocke, C. (1983). Sense and Content: Experience, Thought and Their Relations. Oxford: Oxford University Press.
- Peacocke, C. (1992). A study of concepts. The MIT. · DOI · Paper page
- Porsdam Mann, S., Earp, B. D., Nyholm, S., Danaher, J., Møller, N., Bowman-Smart, H., Hatherley, J., Koplin, J., Plozza, M., Rodger, D., Treit, P. V., Renard, G., McMillan, J., & Savulescu, J. (2023). Generative AI entails a credit–blame asymmetry. Nature Machine Intelligence, 5(5), 472–475. https://doi.org/10.1038/s42256-023-00653-1. · DOI
- Proust, J. (2013). The philosophy of Metacognition: Mental Agency and Self-Awareness. Oxford University Press. · DOI · Paper page
- Schmidt, P., & Loidolt, S. (2023). Interacting with machines: Can an Artificially Intelligent Agent be a Partner? Philosophy & Technology, 36(3), 55. https://doi.org/10.1007/s13347-023-00656-1. · DOI
- Sharkey, A. (2020). Can we Program or Train Robots to be good? Ethics and Information Technology, 22(4), 283–295. https://doi.org/10.1007/s10676-017-9425-5. · DOI
- Sheridan, T. B., & Hennessy, R. T. (1984). Research and modeling of supervisory control behavior. Report of a workshop. National Research Council Washington DC Committee on Human Factors. · DOI
- Simon, J. (Ed.). (2020). The Routledge Handbook of Trust and Philosophy (1st ed.). Routledge. https://doi.org/10.4324/9781315542294. · DOI
- Strawson, P. (1962). Freedom and Resentment. Proceedings of the British Academy, 48, 187–211.
- Tsakiris, M., & De Preester, H. (Eds.). (2018). The interoceptive mind: From homeostasis to awareness. Oxford University Press. · DOI
- Wu, W. (2013). Mental Action and the threat of Automaticity. In A. Clark, J. Kiverstein, & T. Vierkant (Eds.), Decomposing the Will (pp. 244–261). Oxford University Press. · DOI
- Zawidzki, T. W. (2013). Mindshaping: A new framework for understanding human social cognition. MIT Press. · DOI · Paper page
Cited by
On the Need to Rethink Trust in AI Assistants for Software Development: A Critical Review
Baltes, S., Speith, T., Chiteri, B., Mohsenimofidi, S., Chakraborty, S., & Buschek, D. (2026) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2026.3659804 · Cited by 5
Balázs, L. (2026) · Social Sciences · doi:10.3390/socsci15070433 · Cited by 2
Metadata
- Authors
- Dorsch, J., & Deroy, O.
- Venue
- Philosophy & Technology
- Published
- 2024-05-03
- Volume
- 37
- Issue
- 2
- ISSN
- 2210-5433
- 2210-5441
- Cited by
- 26
- References
- 15
- Language
- en
- Licence
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Funders
- Ludwig-Maximilians-Universität München
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Quasi-Metacognitive Machines: Why We Don’t Need Morally Trustworthy AI and Communicating Reliability Is Enough. https://app.cochat.ai/papers/10.1007/s13347-024-00752-w
- The paper
- Dorsch, J., & Deroy, O. (2024). Quasi-Metacognitive Machines: Why We Don’t Need Morally Trustworthy AI and Communicating Reliability Is Enough. Philosophy & Technology. https://doi.org/10.1007/s13347-024-00752-w
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.