Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat
Cited by 26

The DOI matches its Crossref record; 15 of the 15 references it cites were found in a registry or an index.

Quasi-Metacognitive Machines: Why We Don’t Need Morally Trustworthy AI and Communicating Reliability Is Enough

Dorsch, John, Deroy, Ophelia · 2024 · Philosophy & Technology

Abstract

Many policies and ethical guidelines recommend developing “trustworthy AI”. We argue that developing morally trustworthy AI is not only unethical, as it promotes trust in an entity that cannot be trustworthy, but it is also unnecessary for optimal calibration. Instead, we show that reliability, exclusive of moral trust, entails the appropriate normative constraints that enable optimal calibration and mitigate the vulnerability that arises in high-stakes hybrid decision-making environments, without also demanding, as moral trust would, the anthropomorphization of AI and thus epistemically dubious behavior. The normative demands of reliability for inter-agential action are argued to be met by an analogue to procedural metacognitive competence (i.e., the ability to evaluate the quality of one’s own informational states to regulate subsequent action). Drawing on recent empirical findings that suggest providing reliability scores (e.g., F1-scores) to human decision-makers improves calibration in the AI system, we argue that reliability scores provide a good index of competence and enable humans to determine how much they wish to rely on the system.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1007/s13347-024-00752-w

Cites

15 of 15 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 15 references

Research works printed but not linked: 41

  1. Bahrami, B., Olsen, K., Bang, D., Roepstorff, A., Rees, G., & Frith, C. (2012). Together, slowly but surely: The role of social interaction and feedback on the build-up of benefit in collective decision-making. Journal of Experimental Psychology: Human Perception and Performance, 38(1), 3–8. https://doi.org/10.1037/a0025708. · DOI
  2. Bansal, G., Nushi, B., Kamar, E., Horvitz, E., & Weld, D. S. (2021). Is the most accurate AI the best teammate? Optimizing AI for Teamwork. Proceedings of the AAAI Conference on Artificial Intelligence, 35(13), 11405–11414. https://doi.org/10.1609/aaai.v35i13.17359. · DOI
  3. Bayne, T., Brainard, D., Byrne, R. W., Chittka, L., Clayton, N., Heyes, C., Mather, J., Ölveczky, B., Shadlen, M., Suddendorf, T., & Webb, B. (2019). What is cognition? Current Biology, 29(13), R608–r615. https://doi.org/10.1016/j.cub.2019.05.044. · DOI · Paper page
  4. Buechner, J., & Tavani, H. T. (2011). Trust and multi-agent systems: Applying the diffuse, default model of trust to experiments involving artificial agents. Ethics and Information Technology, 13(1), 39–51. https://doi.org/10.1007/s10676-010-9249-z. · DOI
  5. Cherkowski, S., Walker, K. D., & Kutsyuruba, B. (2015). Principals’ Moral Agency and ethical Decision-Making: Toward a transformational Ethics. International Journal of Education Policy and Leadership, 10(5), n5. https://eric.ed.gov/?id=EJ1138586. · DOI
  6. Coeckelbergh, M. (2012). Can we trust robots? Ethics and Information Technology, 14(1), 53–60. https://doi.org/10.1007/s10676-011-9279-1. · DOI · Paper page
  7. Davidson, D. (1963). Actions, reasons and causes. Journal of Philosophy, 60, 685–670. · DOI · Paper page
  8. Deroy, O. (2023). The Ethics of Terminology: Can we use human terms to describe AI? Topoi, 42(3), 881–889. https://doi.org/10.1007/s11245-023-09934-1. · DOI
  9. Deutsch, M. (1977). The resolution of conflict: Constructive and destructive processes. Yale University Press. https://doi.org/10.12987/9780300159356. · DOI
  10. Erasmus, A., Brunet, T. D. P., & Fisher, E. (2021). What is Interpretability? Philosophy & Technology, 34(4), 833–862. https://doi.org/10.1007/s13347-020-00435-2. · DOI
  11. Evans, G. (1982). The Varieties of Reference. Oxford: Oxford University Press.
  12. Fodor, J. (2008). LOT2: The language of thought revisited. Oxford University Press. · DOI · Paper page
  13. Gibbard, A. (1990). Wise choices, apt feelings: A theory of normative Judgment. Harvard University Press. · DOI · Paper page
  14. Goldman, A. I. (1976). Discrimination and perceptual knowledge. The Journal of Philosophy, 73(20), 771–791. https://doi.org/10.2307/2025679. · DOI · Paper page
  15. Haugeland, J. (1990). The Intentionality All-Stars. Philosophical Perspectives, 4, 383–427. https://doi.org/10.2307/2214199. · DOI
  16. Hieronymi, P. (2004). The Force and Fairness of blame. Philosophical Perspectives, 18(1), 115–148. https://doi.org/10.1111/j.1520-8583.2004.00023.x. · DOI · Paper page
  17. Johnson, D. (2006). Computer systems: Moral entities but not moral agents. Ethics of Information Technology, 8, 195–204. https://doi.org/10.1007/s10676-006-9111-5. · DOI · Paper page
  18. Kamar, E. (2016). Directions in Hybrid Intelligence: Complementing AI Systems with Human Intelligence. Proceedings of the 25th International Joint Conference on Artificial Intelligence, New York, 9–15 July 2016, 4070–4073.
  19. Lahno, B. (2001). On the emotional character of Trust. Ethical Theory and Moral Practice, 4(2), 171–189. https://doi.org/10.1023/A:1011425102875. · DOI
  20. Le, T., Miller, T., Singh, R., & Sonenberg, L. (2022). Improving model understanding and trust with counterfactual explanations of Model confidence. arXiv Preprint arXiv:220602790.
  21. Lebreton, M., Bacily, K., Palminteri, S., & Engelmann, J. B. (2019). Contextual influence on confidence judgments in human reinforcement learning. PLOS Computational Biology, 15(4), e1006973. https://doi.org/10.1371/journal.pcbi.1006973. · DOI
  22. Longin, L., Bahrami, B., & Deroy, O. (2023). Intelligence brings responsibility - even smart AI-assistants are held responsible. iScience, 107494. https://doi.org/10.1016/j.isci.2023.107494. · DOI
  23. Luhmann, N. (1979). Trust and Power. Wiley.
  24. Mainz, J. T. (2023). Medical AI: Is trust really the issue? Journal of Medical Ethics. https://doi.org/10.1136/jme-2023-109414. · DOI
  25. Mayer, R. C., Davis, J. H., & Schoorman, F. D. (1995). An Integrative Model of Organizational Trust. The Academy of Management Review, 20(3), 709–734. https://doi.org/10.2307/258792. · DOI · Paper page
  26. McDowell, J. (1994). Mind and world. Harvard University Press.
  27. Moor, J. H. (2006). The Nature, Importance, and Difficulty of Machine Ethics. IEEE Intelligent Systems, 21(4), 18–21. https://doi.org/10.1109/MIS.2006.80. · DOI · Paper page
  28. National Academies of Sciences. (2017). Communicating Science effectively: A Research Agenda. National Academies Press. https://doi.org/10.17226/23674. · DOI · Paper page
  29. O’Connor, C., & Weatherall, J. O. (2019). The misinformation age: How false beliefs spread. Yale University Press. · DOI · Paper page
  30. Peacocke, C. (1983). Sense and Content: Experience, Thought and Their Relations. Oxford: Oxford University Press.
  31. Peacocke, C. (1992). A study of concepts. The MIT. · DOI · Paper page
  32. Porsdam Mann, S., Earp, B. D., Nyholm, S., Danaher, J., Møller, N., Bowman-Smart, H., Hatherley, J., Koplin, J., Plozza, M., Rodger, D., Treit, P. V., Renard, G., McMillan, J., & Savulescu, J. (2023). Generative AI entails a credit–blame asymmetry. Nature Machine Intelligence, 5(5), 472–475. https://doi.org/10.1038/s42256-023-00653-1. · DOI
  33. Proust, J. (2013). The philosophy of Metacognition: Mental Agency and Self-Awareness. Oxford University Press. · DOI · Paper page
  34. Schmidt, P., & Loidolt, S. (2023). Interacting with machines: Can an Artificially Intelligent Agent be a Partner? Philosophy & Technology, 36(3), 55. https://doi.org/10.1007/s13347-023-00656-1. · DOI
  35. Sharkey, A. (2020). Can we Program or Train Robots to be good? Ethics and Information Technology, 22(4), 283–295. https://doi.org/10.1007/s10676-017-9425-5. · DOI
  36. Sheridan, T. B., & Hennessy, R. T. (1984). Research and modeling of supervisory control behavior. Report of a workshop. National Research Council Washington DC Committee on Human Factors. · DOI
  37. Simon, J. (Ed.). (2020). The Routledge Handbook of Trust and Philosophy (1st ed.). Routledge. https://doi.org/10.4324/9781315542294. · DOI
  38. Strawson, P. (1962). Freedom and Resentment. Proceedings of the British Academy, 48, 187–211.
  39. Tsakiris, M., & De Preester, H. (Eds.). (2018). The interoceptive mind: From homeostasis to awareness. Oxford University Press. · DOI
  40. Wu, W. (2013). Mental Action and the threat of Automaticity. In A. Clark, J. Kiverstein, & T. Vierkant (Eds.), Decomposing the Will (pp. 244–261). Oxford University Press. · DOI
  41. Zawidzki, T. W. (2013). Mindshaping: A new framework for understanding human social cognition. MIT Press. · DOI · Paper page

Cited by

See all 26

Metadata

Authors
Dorsch, J., & Deroy, O.
Venue
Philosophy & Technology
Published
2024-05-03
Volume
37
Issue
2
ISSN
  • 2210-5433
  • 2210-5441
Cited by
26
References
15
Language
en
Licence
Funders
Ludwig-Maximilians-Universität München

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Quasi-Metacognitive Machines: Why We Don’t Need Morally Trustworthy AI and Communicating Reliability Is Enough. https://app.cochat.ai/papers/10.1007/s13347-024-00752-w
The paper
Dorsch, J., & Deroy, O. (2024). Quasi-Metacognitive Machines: Why We Don’t Need Morally Trustworthy AI and Communicating Reliability Is Enough. Philosophy & Technology. https://doi.org/10.1007/s13347-024-00752-w

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.