Hidden Risks: Artificial Intelligence and Hermeneutic Harm

Cases where a person lacks technical understanding of an AI decision more plausibly involve distributive epistemic injustice, where someone receives less of an epistemic good-in this case information-than they are due.

“More plausibly, such cases involve 'distributive' epistemic injustice, where someone receives less of an epistemic good-in this case information-than they are due (Fricker, 2017, p. 53)”

• cites Fricker, 2017, p. 53 sourceSupported

Some harms caused by AI systems are due to the 'black box' problem: certain AI systems, especially those based on advanced machine learning models such as deep neural networks, are opaque and extremely difficult to fully interpret or explain.

“Some harms caused by AI systems are due to the 'black box' problem, namely that certain AI systems-especially those based on advanced machine learning (ML) models such as deep neural networks (DNNs)-are opaque: extremely difficult to fully interpret or explain.”

• cites Ali et al., 2023 sourceCould not verify

  Split into 2 atoms — 0 confirmed, 0 partially supported, 0 disputed, 0 misattributed, 2 unverifiable. Expand the claim to see per-atom verdicts.

  Experimental — accuracy not yet measured
• cites Barredo Arrieta et al., 2020 sourceSupported
• cites Gunning et al., 2019 sourceNot checked — no full text of the source

• AssumptionRests on an assumption: Advanced ML models identify patterns in complex datasets based on distributed, non-linear transformations across potentially billions of parameters, whereas traditional software executes clearly defined, rule-based instructions.

XAI techniques available today are not sufficient to resolve all explainability concerns, even if several tools are used.

“Are the XAI techniques available today sufficient to resolve all the explainability concerns, even if several tools are used?The answer is NO.”

• cites Ali et al., 2023 sourcePartly supported

  Section 1.1, Problem Description (page unavailable), identifies unresolved standards for assessing explanations, and Section 1.2, Challenges and Limitations (page unavailable), describes shortcomings in current XAI evaluations. These support the broad point that explainability concerns remain unresolved, but the passages do not establish that using several tools still cannot resolve them. (LLM, conf=0.82)

  Experimental — accuracy not yet measured

The price of uncovering relationships through advanced ML models is a lack of transparency as to how it is achieved on any occasion.

“But the price of uncovering relationships in this way is a lack of transparency as to how it is achieved on any occasion (Ali et al., 2023;Barredo Arrieta et al., 2020;Gunning et al., 2019).”

• cites Ali et al., 2023 sourceCould not verify

  The passages in §1.1 (Problem Description) and §1.2 (Challenges and Limitations; page numbers unavailable) discuss ML explainability and users’ ability to understand decisions. They do not say that uncovering relationships with advanced ML comes at the price of lacking transparency about how a particular result is achieved. Searches for that trade-off and for a lack of transparency found no decisive passage in our copy. (LLM, conf=0.73)

  Experimental — accuracy not yet measured
• cites Barredo Arrieta et al., 2020 sourceSupported
• cites Gunning et al., 2019 sourceNot checked — no full text of the source

• AssumptionRests on an assumption: Advanced ML models identify patterns in complex datasets based on distributed, non-linear transformations across potentially billions of parameters, whereas traditional software executes clearly defined, rule-based instructions.

There is generally reckoned to be a trade-off between the interpretability and accuracy of ML models.

“There is generally reckoned to be a trade-off between the interpretability and accuracy of ML models (Ali et al., 2023;Barredo Arrieta et al., 2020;Gunning et al., 2019;Langer et al., 2021).”

• cites Ali et al., 2023 sourceCould not verify

  Section 1.1 discusses interpretable-by-design models and mentions accuracy as an example of an evaluation measure, but does not relate model accuracy to interpretability. Searches for an interpretability–accuracy trade-off and for accuracy in black-box models found no passage addressing the claimed trade-off. (LLM, conf=0.70)

  Experimental — accuracy not yet measured
• cites Barredo Arrieta et al., 2020 sourceSupported
• cites Gunning et al., 2019 sourceNot checked — no full text of the source

If explainability in a full sense requires maintaining pre-existing procedures and protocols, enforcing it would undermine one of the main attractions of deploying AI systems, namely their capacity to function independently.

“But, as Maclure (2021, p. 14) sees, if this is what explainability, in a full sense, requires, enforcing it would undermine one of the main attractions of deploying AI systems, namely their capacity to function independently.”

• cites Maclure, 2021 sourceSupported

Further complications are introduced when AI systems produce outputs in interaction with humans or other AI or complex sociotechnical systems.

“Moreover, further complications are introduced when AI systems produce outputs in interaction with humans or other AI or complex sociotechnical systems (Borrego-Díaz & Galán-Páez, 2022;Miller, 2019).”

• cites Borrego-Díaz & Galán-Páez, 2022 sourcePartly supported

  The Introduction (§1; page not provided) says AI operates within broader sociotechnical systems that include humans, and that explaining its outcomes can become an interactive, sociotechnical challenge. Our copy does not clearly address the added claim that outputs produced through interaction with other AI introduce complications. (LLM, conf=0.83)

  Experimental — accuracy not yet measured
• cites Miller, 2019 sourcePartly supported

  Section 5 describes explanation as an interaction between an explainer and an explainee, and §5.3.6 identifies social interaction as a challenge for explainable AI. These passages support the human-interaction part of the claim, but do not address AI systems producing outputs in interaction with other AI or complex sociotechnical systems. The passages provide no page numbers. (LLM, conf=0.84)

  Experimental — accuracy not yet measured

Explainability tools can (at least currently) provide only an approximation of the decision-making functions in black-box systems.

“Explainability tools can (at least currently) provide only an approximation of the decision-making functions in black-box systems (Mittelstadt et al., 2019)”

• cites Mittelstadt et al., 2019 sourcePartly supported

  Section 3 (p. 3) says many current explainability methods use simplified models that approximate a system’s true decision criteria. But the claim’s ‘only’ generalizes this to all explainability tools; section 3.3 (p. 5) discusses other methods of producing explanations, and section 3 (p. 2) lists several approaches besides approximations. (LLM, conf=0.86)

  Experimental — accuracy not yet measured

Many applications need the modelling complexity provided by black-box systems.

“Many applications need the modelling complexity provided by black-box systems.”

• cites Ali et al., 2023 sourceCould not verify

  Section 1.1 discusses explainability methods, and Section 1.2 discusses how task complexity affects the evaluation of explanations. Neither says that many applications require the modelling complexity of black-box systems. Searches for that claim and for a complexity–interpretability trade-off found no supporting passage in our copy; page numbers were not provided. (LLM, conf=0.73)

  Experimental — accuracy not yet measured

Providing an explanation in the wrong way (explaining badly) might exacerbate the problem or make things worse.

“This is problematic because providing an explanation in the wrong way might exacerbate the problem-for instance by demonstrating to people how very little sense they can make of certain decisions affecting them (Keil, 2006, p. 229).”

• Also stated in: § 4 Transparency Expectations II: Incomprehensibility · hedged

  “Explaining badly might make things worse, as the following case suggests.”

• cites Keil, 2006 sourcePartly supported

  The Introduction (page unavailable in our copy) says explanations can fail when they omit essential details or provide too much, and that trying to explain can expose gaps in understanding. It does not establish the cited sentence’s specific example: that a poorly given explanation worsens a problem by showing people how little sense they can make of decisions affecting them. (LLM, conf=0.82)

  Experimental — accuracy not yet measured

Transparent models cannot handle sophisticated real-world applications.

“Transparent models cannot handle sophisticated real-world applications.”

• cites Ali et al., 2023 sourceCould not verify

  The 1.1 Problem Description passage mentions interpretable-by-design models but does not assess what applications they can handle. Section 1.2 discusses how task complexity affects evaluation of explanations, not whether transparent models can handle sophisticated real-world applications. Searches for the claim and paraphrases found no passage addressing that point. (LLM, conf=0.72)

  Experimental — accuracy not yet measured

Too great an emphasis on making explanations psychologically fitting risks making it easier for bad actors to deceive and manipulate end-users.

“Too great an emphasis on making explanations psychologically fitting risks making it easier for bad actors to deceive and manipulate end-users (Baron, 2023;Lipton, 2018, pp. 21-22).”

• cites Baron, 2023 sourcePartly supported

  Section 2, “Causal Understanding” (page not supplied), warns that an explanation users find satisfying may give them only perceived understanding rather than true beliefs about causes. This supports the concern about prioritizing psychological fit, but the passages do not say that doing so makes it easier for bad actors to deceive or manipulate end-users. (LLM, conf=0.84)

  Experimental — accuracy not yet measured
• cites Lipton, 2018 sourceSupported

Hermeneutic harm can arise quite independently of whether responsibility gaps are real or not.

“But as Rebera (2024) argues, hermeneutic harm can arise quite independently of whether responsibility gaps are real or not.”

• cites Rebera (2024) sourceSupported

Unexpected or jarring events show up discrepancies between global and situational meaning.

“Unexpected or jarring events show up discrepancies between global and situational meaning.”

All AI techniques can give rise to hermeneutic harm.

“we single out no particular techniques of AI (transformer-based large language models (LLMs), Reinforcement Learning from Human Feedback, neural networks, etc.), because all can give rise to hermeneutic harm”

Stretesky et al. (2010) report that co-victims of unsolved homicides felt frustration when the justice system limited their access to information about events surrounding the murder of their loved-ones.

“Stretesky et al. (2010) report the frustration felt by co-victims of unsolved homicides when the justice system limited their access to information about events surrounding the murder of their loved-ones.”

• cites Stretesky et al., 2010 sourceNot checked — no full text of the source

An event need not be objectively harmful to provoke hermeneutic harm.

“an event need not be objectively harmful to provoke hermeneutic harm”

XAI could plausibly reduce the risk of hermeneutic harm in some cases.

“Explainable AI (XAI) could plausibly reduce the risk of hermeneutic harm in some cases.”

XAI cannot fully mitigate the risk of hermeneutic harm.

“Yet XAI cannot fully mitigate the risk of hermeneutic harm”

It is not strictly necessary that an event be experienced as harmful at all to provoke hermeneutic harm.

“it is not strictly necessary that an event be experienced as harmful at all”

One form of harm caused, perpetuated or exacerbated by AI has been overlooked in the AI Ethics literature.

“One, however, has been overlooked.”

• Also stated in: § 1 Introduction · plain

  “One, however, has been overlooked.”

• AssumptionRests on an assumption: The AI Ethics literature has identified many forms of harm caused, perpetuated or exacerbated by AI.

AI systems can cause hermeneutic harm even if they do not objectively cause primary harm.

“AI systems can cause hermeneutic harm even if they do not (objectively) cause primary harm”

The increasing use of AI heightens the risk of hermeneutic harm.

“we argue that the increasing use of AI heightens the risk of 'hermeneutic harm'”

• Also stated in: § 1 Introduction · plain

  “We argue that the increasing use of AI heightens the risk of hermeneutic harm”

• Also stated in: § Hermeneutic harm. · plain

  “their proliferation increases the risk of hermeneutic harm”

AI systems can cause hermeneutic harm even if they function entirely as intended.

“AI systems can cause hermeneutic harm even if they do not (objectively) cause primary harm and even if they function entirely as intended;”

The risk of hermeneutic harm would persist even if all 'black-box' problems of system opacity were to be solved.

“which (as our choice of examples shows) would persist even if all 'black-box' problems of system opacity were to be solved.”

Hermeneutic harm can occur regardless of flawed design, biased training data, or misuse.

“our argument makes no assumption of flawed design, biased training data, or misuse: hermeneutic harm can occur regardless.”

• Also stated in: § Hermeneutic harm. · plain

  “moreover, our argument makes no assumption of flawed design, biased training data, or misuse: hermeneutic harm can occur regardless.”

In both cases, this obstruction plausibly constituted additional harm.

“and in both cases, this obstruction plausibly constituted additional harm.”

• cites Stretesky et al., 2010 sourceNot checked — no full text of the source
• cites Allais, 2008 sourceNot checked — no full text of the source

A lack of information, or the lack of an adequate explanation, obstructs the possibility of sense-making.

“These are cases where a lack of information, or the lack of an adequate explanation, obstructs the possibility of sense-making.”

• cites Stretesky et al., 2010 sourceNot checked — no full text of the source
• cites Allais, 2008 sourceNot checked — no full text of the source

In scenario (1), hermeneutic harm arises from the unresolved mismatch between the applicant's expectation of good reasons for rejection and his experience that no reasons are given.

“In scenario (1), hermeneutic harm arises from the unresolved mismatch between the applicant's expectation of there being good reasons why his very strong application was rejected and his experience that no reasons (good or otherwise) are given.”

Hermeneutic harm is plausibly not a form of hermeneutic injustice because it arises, at least in part, in response to ongoing epistemic injustice.

“More fundamentally, in our view hermeneutic harm is plausibly not a form of hermeneutic injustice because it arises, at least in part, in response to ongoing epistemic injustice.”

In cases such as (1), where decisions are relatively momentous, XAI techniques enable the development of systems based on white- or grey-box models or which, if based on black-box models, incorporate additional tools to facilitate interpretation and/or explanation.

“So, in cases such as (1), where decisions are relatively momentous, XAI techniques enable the development of systems based on white-or grey-box models or which, if based on black-box models, incorporate additional tools to facili-tate interpretation and/or explanation (Holland Michel, 2020; Wachter et al., 2018).”

• cites Holland Michel, 2020Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Wachter et al., 2018Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

In the second case, Mhlauli was denied the closure forgiveness may have allowed because the fitting subjects of her forgiveness remained unknown.

“in the second, Mhlauli was denied the closure forgiveness may have allowed because the fitting subjects of her forgiveness remained unknown.”

• cites Allais, 2008 sourceNot checked — no full text of the source

In some cases, a form of hermeneutic injustice arises or is exacerbated because victims lack the resources to put a 'perspective' on events, a way of thinking about them that captures their character in an illuminating way.

“we might further, in some cases, argue that a form of hermeneutic injustice arises or is exacerbated because victims lack the resources to put a 'perspective' on events, a way of thinking about them that captures their character in an illuminating way (Sliwa, 2023).”

• cites Sliwa, 2023 sourceNot checked — no full text of the source

Once interpreted by developers, white- and grey-box models can, in principle, be explained to non-expert users.

“Once interpreted by developers, white-and grey-box models can, in principle, be explained to (nonexpert) users.”

People often seek to understand decisions in order to then act upon them, for example to contest them.

“Wachter et al. (2018) stress that people often seek to understand decisions in order to then act upon them (e.g. to contest them).”

• cites Wachter et al. (2018)Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Explanation supports processes of understanding that enable people to move on from decisions that affect them, whether by challenging them or by reflecting on how they could have gone differently.

“Miller and Wachter et al. both emphasise the role of explanation as a support to processes of understanding that enable people to move on from decisions that affect them (whether by challenging them or by reflecting on how, in a practical sense, they could have gone differently).”

• cites MillerNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Wachter et al.Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Explanations of these kinds are well-suited to mitigating risks of hermeneutic harm as posed in (1) and (2).

“Explanations of these kinds are well-suited to mitigating risks of hermeneutic harm as posed in ( 1) and ( 2).”

AI systems are not all irremediably opaque.

“AI systems are not all irremediably opaque.”

In the first case, the lack of information extended co-victims' bereavement and grieving.

“In the first case, the lack of information extended co-victims' bereavement and grieving (Stretesky et al., 2010, p. 887)”

• cites Stretesky et al., 2010 sourceNot checked — no full text of the source

To suffer hermeneutic harm does not necessarily undermine one's status or capacity to function in one's epistemic community.

“For one thing, to suffer hermeneutic harm does not necessarily undermine one's status or capacity to function in one's epistemic community (as per Fricker's original statement of the problem).”

Primary harms caused by AI systems can give rise to secondary harms that are worthy of more consideration than they have so far received in the AI and AI Ethics literatures.

“In the cases to be discussed below, we argue that primary harms caused by AI systems can give rise to secondary harms that are worthy of more consideration than they have so far received in the AI and AI Ethics literatures.”

In scenario (1), system opacity obstructs access to information or explanations that people plausibly need in order to make sense of the AI's decisions.

“Moreover, in a straightforwardly procedural sense, system opacity obstructs access to information or explanations that people plausibly need in order to make sense of the AI's decisions.”

In scenario (2), the AI Decision Report was uninterpretable to the business owner.

“Yet the Report was uninterpretable to the business owner.”

Hermeneutic harm arises from the prolonged inability to resolve situations of epistemic injustice.

“That is, hermeneutic harm arises from the prolonged inability to resolve situations of epistemic injustice; it may be, for instance, the prolonged inability to terminate a process of what Sliwa (2023) calls 'hermeneutical inquiry'.”

• cites Sliwa (2023) sourceNot checked — no full text of the source

If successfully implemented, these measures allow systems or their operators to provide explanations that support the sense-making of people affected by those systems.

“If successfully implemented, these measures allow systems or their operators to provide explanations that support the sense-making of people affected by those systems.”

The concepts of hermeneutic harm and epistemic injustice are clearly related.

“From these observations, the concepts of hermeneutic harm and epistemic injustice are clearly related.”

In both cases, the co-victims' ability to make sense of what happened was obstructed.

“In both cases, the co-victims' ability to make sense of what happened was obstructed”

• cites Stretesky et al., 2010 sourceNot checked — no full text of the source
• cites Allais, 2008 sourceNot checked — no full text of the source

Satisfaction of all expectations is no prerequisite of sense-making.

“This shows that satisfaction of all expectations is no prerequisite of sense-making.”

Hermeneutic harm arises fundamentally from the misalignment of expectation and experience (of global and situational meaning), not from value misalignment as such.

“Value misalignment is the misalignment of human and AI values; but fundamentally, hermeneutic harm arises from the misalignment of expectation and experience, of global and situational meaning.”

AI-based decision support systems are being used for targeting in current and recent conflicts.

“and AI-based decision support systems are being used for targeting in current and recent conflicts.”

• cites Nadibaidze et al. (2024)Not checked — no full text of the source

A positive experience with the bank manager helps to mitigate the risk of hermeneutic harm.

“A positive experience with the bank manager helps to mitigate the risk of hermeneutic harm.”

The relationship between value alignment and the misalignment of expectation and experience is not one of identity but of causation.

“the relationship between them is not one of identity, but causation.”

Hermeneutic harm does not result only from poorly designed AI systems or systems trained on biased or otherwise problematic datasets.

“and as our choice of examples shows, hermeneutic harm does not result only from poorly designed AI systems or systems trained on biased or otherwise problematic datasets.”

These AI systems function as agents.

“These AI systems function as agents.”

Given the complexity and diversity of human social interaction, it remains unlikely that explanations delivered by AI systems will satisfy all our normative requirements.

“Nonetheless, given the complexity and diversity of human social interaction, it remains unlikely that explanations delivered by AI systems will satisfy all our normative requirements”

• AssumptionRests on an assumption: Humans are, in general, brought up and socialised to satisfy the myriad unspoken normative expectations which we each have of each other.

Scenarios like (4) and (5) present the possibility of a fundamental mismatch between people's deep-seated normative expectations and their actual experience (i.e. of global and situational meaning).

“Whatever the source of the deep-seated normative expectations that one's loved ones should not be instrumentally exploited, scenarios like ( 4) and ( 5) present the possibility of a fundamental mismatch of these people's normative expectations and actual experience (i.e. of global and situational meaning).”

• AssumptionRests on an assumption: People have normative expectations that neither they nor their loved ones should be deliberately harmed, and in particular that they will not be run down on a crossing and that their loved ones will not be massacred in the street.

Even if XAI could provide epistemically suitable explanations, hermeneutic harm may occur.

“But what if-as is likely in at least some cases-XAI could provide epistemically suitable explanations?Even here, we suggest, hermeneutic harm may occur.”

AI need not be faulty to cause hermeneutic harm.

“AI need not be faulty to cause hermeneutic harm”

Even if the actions of the car or the LAWS can be justified all things considered, there may remain a fundamental clash between our normative expectations of how things should go and our understanding of how they actually go.

“Yet even if the actions of the car or the LAWS can be justified all things considered, there may remain a fundamental clash between our normative expectations of how things should go and our understanding of how they actually go.”

In scenarios (4) and (5), an AI system acts with agency and autonomy.

“In scenarios ( 4) and ( 5), an AI system acts with agency and autonomy.”

The limited information end-users require-such as whether the system uses some intuitively relevant factor and how-may be hard to come by.

“even the limited information they do require-for instance, whether the system makes use of some factor that is intuitively relevant (and if so how)-may be hard to come by (for discussion see Lam, 2022).”

• cites Lam, 2022 sourceNot checked — no full text of the source

• AssumptionRests on an assumption: End-users do not necessarily need comprehensive explanations of decisions.

In these cases, hermeneutic harm results, in some degree, from misalignment.

“In these cases, hermeneutic harm results, in some degree, from misalignment.”

This increases the risk of hermeneutic harm.

“This increases the risk of hermeneutic harm.”

The proliferation of misaligned AI systems is also the proliferation of risks of hermeneutic harm.

“and thus the proliferation of misaligned AI systems is also the proliferation of risks of hermeneutic harm.”

There are conditions under which an agent can perform an otherwise impermissible action, which confounds normative expectations, but not be fully culpable.

“Yet there are conditions under which an agent can perform an otherwise impermissible action, which confounds normative expectations, but not be (fully) culpable.”

The agent-harms in (4) and (5) result from actions that violate deep-seated norms and expectations.

“More pertinently, the agent-harms result from actions that violate deep-seated norms and expectations.”

AI systems can cause hermeneutic harm (though not inevitably).

“First, we have not claimed that AI systems inevitably cause hermeneutic harm: our claim is that AI systems can cause hermeneutic harm (and to this end we have presented scenarios in which they do).”

XAI has not been sufficiently alive to the need to mitigate the risk of hermeneutic harm.

“But XAI has not been sufficiently alive to the need to mitigate the risk of hermeneutic harm.”

In (4) and (5), agent-harms result from actions that would, under straightforward descriptions, be considered impermissible.

“In ( 4) and (5), agent-harms result from actions that would, under straightforward descriptions ('driving into a pedestrian on a crossing', 'firing missiles into a crowd of non-combatants') be considered impermissible.”

Self-driving cars, LAWS and a variety of other technologies can initiate or end courses of action independently of human control, purposely achieving real-world outcomes.

“Self-driving cars, LAWS-and a variety of other technologies-can initiate (or end) courses of action independently of human control, purposely achieving real-world outcomes.”

The problem in scenarios (4) and (5) is not that the person cannot understand why the car hit them or why the LAWS attacked.

“It is important to be clear that the problem is not that the person in ( 4) cannot understand why the car chose to hit them, or that those who grieve the dead in (5) do not understand why the LAWS attacked.”

The proliferation of AI systems increases the risk of hermeneutic harm.

“For all the value they bring, the proliferation of AI systems increases the risk of hermeneutic harm.”

AI systems are unable to recognise or reliably respond to our deepest (and in some sense simplest) normative expectations.

“and yet they are unable to recognise or reliably respond to our deepest-and in some sense also our simplest-normative expectations.”

In some cases, not only an explanation but an attribution of responsibility is required to support sense-making.

“Moreover, in some cases, not only an explanation, but an attribution of responsibility is required to support sense-making.”

The confounding of normative expectations is a more fundamental risk-factor for hermeneutic harm than the confounding of epistemic ones.

“Indeed, in a certain sense, the confounding of normative expectations is a more fundamental risk-factor for hermeneutic harm than the confounding of epistemic ones.”

The risk of hermeneutic harm persists even if all transparency problems are solved.

“In these cases, the risk of hermeneutic harm persists even if all transparency problems are solved (i.e. even if it is possible to provide reasonably accurate, comprehensible explanations of the AI systems' decisions).”

Not knowing or not understanding why some event occurred can be a fundamental obstacle to one's efforts to come to terms with it.

“not knowing or not understanding why some event occurred can be a fundamental obstacle to one's efforts to come to terms with it.”

XAI could plausibly reduce the risk of hermeneutic harm in many cases of epistemic transparency.

“XAI could plausibly reduce the risk of hermeneutic harm in many such cases”

Clarifying the structures of responsibility and answerability around AI decision-making would provide some mitigation of the risks of hermeneutic harm.

“Clarifying the structures of responsibility and answerability around AI decision-making would doubtless provide some mitigation of the risks of hermeneutic harm.”