The DOI matches its Crossref record; 87 of the 92 references it cites were found in a registry or an index, and 5 are not checked yet; none of the 27 sources its claims cite is checked for support yet.
'It's Reducing a Human Being to a Percentage': Perceptions of Justice in Algorithmic Decisions
Open accessPublished version of a preprint
Abstract
Data-driven decision-making consequential to individuals raises important questions of accountability and justice. Indeed, European law provides individuals limited rights to 'meaningful information about the logic' behind significant, autonomous decisions such as loan approvals, insurance quotes, and CV filtering. We undertake three experimental studies examining people's perceptions of justice in algorithmic decision-making under different scenarios and explanation styles. Dimensions of justice previously observed in response to human decision-making appear similarly engaged in response to algorithmic decisions. Qualitative analysis identified several concerns and heuristics involved in justice perceptions including arbitrariness, generalisation, and (in)dignity. Quantitative analysis indicates that explanation styles primarily matter to justice perceptions only when subjects are exposed to multiple different styles---under repeated exposure of one style, scenario effects obscure any explanation effects. Our results suggests there may be no 'best' approach to explaining algorithmic decisions, and that reflection on their automated nature both implicates and mitigates justice dimensions.
Checked 2026-10-06 · 5 references not checked yet
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
Three experimental studies found that people apply justice dimensions to algorithmic decisions much as they do to human decisions, but explanation-style effects were mostly visible only when participants compared multiple styles for the same case. Case-based explanations were consistently associated with lower justice perceptions than sensitivity-based explanations, while repeated exposure to a single explanation style largely obscured explanation effects.
Key findings
- Justice constructs correlated similarly to the human decision-making literature: Spearman’s rho was 0.69 for fair process–deserved outcome, 0.60 for fair process–appropriate factors, 0.51 for appropriate factors–deserved outcome, 0.40 for fair process–understanding, 0.40 for appropriate factors–understanding, and 0.36 for understanding–deserved outcome.
- In the between-subjects study (325 participants, 5 conditions of n = 65), explanation styles generally did not significantly affect justice perceptions, except for fair process in loans, F(3, 258) = 2.71, p = 0.046, and appropriate factors, F(3, 258) = 5.35, p = 0.001.
- In the within-subjects study (65 participants), significant explanation effects appeared across all questions and contexts: fair process in loans, F(3, 260) = 7.52, p < .001, and insurance, F(3, 260) = 4.5, p = .004; appropriate factors in loans, F(3, 260) = 3.312, p = .02, and insurance, F(3, 260) = 6.44, p = .0003; deserved outcome in loans, F(3, 260) = 7.31, p = .0001. Case-based explanations produced lower appropriateness, fair process, and loan deservedness than sensitivity-based styles.
- Qualitative themes included lack of human touch/indignity, interpreting the system’s reasoning, acceptability of statistical inference and generalisation, actionability, unaccounted aspects, and the meaning/relevance of moral concepts; some participants saw the algorithmic nature as mitigating justice judgments.
Methods
- Three studies: Phase 1 was an in-person lab study with 19 UK participants, using semi-structured think-aloud interviews over 15 fictional negative-outcome cases across 5 contexts (loan, promotion, car insurance, airline re-routing, bank account freeze), each with one of 4 explanation styles (input influence, sensitivity, case-based, demographic) or a no-explanation control; data were thematically analysed.
- Phase 2 comprised two online Prolific Academic studies with UK participants over 18: a between-subjects study with 325 participants randomly assigned to 5 conditions (n = 65 each) and 12 cases with one explanation style or control, and a within-subjects study with 65 participants using only loan and insurance cases, presenting four different explanation styles for a single case in random order.
- Measures were agreement with five 5-point Likert justice statements (agreement, understanding, appropriateness of factors, fair process, deserved outcome); analysis used Spearman’s rho correlations and ANOVA with Tukey’s post-hoc paired tests.
Limitations
- Stated: samples were not representative—the lab study drew from an affluent, academically dominated UK city, and the online studies were not gender balanced, with almost double the number of females to males; scenarios were hypothetical and did not directly affect participants, lacking first-person consequences and options such as human review or appeal; explanations were not drawn from real machine learning model outputs; differences in explanation text length and format may have confounded results; further confirmatory work is needed on why explanation styles differ.
- Stated/evident: the airline context was excluded from the online studies because lab participants almost universally judged it very unfair and undeserved, preventing useful conclusions; the lab sample was small (19 participants), and online participants were limited to the UK.
- Evident: the between-subjects design repeated a single explanation style, which may have led to habituation, while the within-subjects design made style differences more salient through direct comparison; neither design may fully capture real-world exposure patterns.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
51 Own result · 21 Not checked yet · 17 Own interpretation · 7 Own method · 6 No basis given
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Qualitative analysis identified several concerns and heuristics involved in justice perceptions including arbitrariness, generalisation, and (in)dignity.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Dimensions of justice previously observed in response to human decision-making appear similarly engaged in response to algorithmic decisions.”
hedgedcorrelationaln = 3- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Through iterative testing we found that the use of dialog boxes focused attention on explanations (see Figure 2).”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“In contrast to this apparent lack of humanity (or perhaps to counteract it), some participants attempted to humanise or anthropomorphise the system [48], one participant for instance describing it as 'rude' ( OR ).”
What the sources say
- Not checked yetcites [48] 48. Clifford Nass, Jonathan Steuer, Ellen Tauber, and Heidi Reeder. 1993. Anthropomorphism, agency, and ethopoeia: computers as social actors. In INTERACT'93 and CHI'93 Conference Companion on Human Factors in Computing Systems . ACM, 111-112. source
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The fictionalised scenarios used in the lab and online studies are realistic and representative enough to elicit valid responses about algorithmic decision-making.
- Rests on an assumption: The thematic analysis themes identified by the coders accurately represent participants' interpretations and reasoning.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In judging the appropriateness of certain features, or the fairness of a decision, issues of personal responsibility and the possibility of acting differently were frequently raised.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The fictionalised scenarios used in the lab and online studies are realistic and representative enough to elicit valid responses about algorithmic decision-making.
- Rests on an assumption: The thematic analysis themes identified by the coders accurately represent participants' interpretations and reasoning.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Some participants were happy to endorse negative decisions if they were able to identify a reasonable alternative course of action available to the participant which would resulted in a different outcome, as in sensitivity -based explanations:”
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The fictionalised scenarios used in the lab and online studies are realistic and representative enough to elicit valid responses about algorithmic decision-making.
- Rests on an assumption: The thematic analysis themes identified by the coders accurately represent participants' interpretations and reasoning.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We found that people generally did not pay attention to explanations independent of scenario descriptions.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“An initial reaction from many participants concerned the indignity or 'weirdness' of algorithmic decision-making compared to human decision-making.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The fictionalised scenarios used in the lab and online studies are realistic and representative enough to elicit valid responses about algorithmic decision-making.
- Rests on an assumption: The thematic analysis themes identified by the coders accurately represent participants' interpretations and reasoning.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We also found that people tended to interpret these justice construct statements as a proxy for the simpler question of whether or not they agreed with the decision;”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, many other participants expressed strong objections to the use of statistical inferences to make judgements about people in itself:”
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The fictionalised scenarios used in the lab and online studies are realistic and representative enough to elicit valid responses about algorithmic decision-making.
- Rests on an assumption: The thematic analysis themes identified by the coders accurately represent participants' interpretations and reasoning.
- It may state two separate claims as one. 25
- Probably not a substantial claim. 47
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Quantitative analysis indicates that explanation styles primarily matter to justice perceptions only when subjects are exposed to multiple different styles-under repeated exposure of one style, scenario effects obscure any explanation effects.”
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However in this case, some participants were more comfortable with inductive inferences; AS agreed with the computer's decision to deny a loan applicant with existing debts because they had already 'proven not to be able to pay it back'.”
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The fictionalised scenarios used in the lab and online studies are realistic and representative enough to elicit valid responses about algorithmic decision-making.
- Rests on an assumption: The thematic analysis themes identified by the coders accurately represent participants' interpretations and reasoning.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The airline context was excluded at this stage, because experiment 1 had revealed that participants almost universally judged the cases to be very unfair and undeserved, thus preventing any useful conclusions to be drawn from this context.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Where an individual was deemed at fault for risky or otherwise undesirable past behaviour, participant AS believed the decision was particularly justified:”
plaincorrelationalSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Rests on an assumption: The fictionalised scenarios used in the lab and online studies are realistic and representative enough to elicit valid responses about algorithmic decision-making.
- Rests on an assumption: The thematic analysis themes identified by the coders accurately represent participants' interpretations and reasoning.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“One participant argued that a truly random system could also be seen as more 'fair', at least for certain definitions of fairness:”
plainSingle case- Asserted: fails — The authors may not put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The authors may not put it forward as true. The paper’s conclusions may not depend on it.
- Rests on an assumption: The fictionalised scenarios used in the lab and online studies are realistic and representative enough to elicit valid responses about algorithmic decision-making.
- Rests on an assumption: The thematic analysis themes identified by the coders accurately represent participants' interpretations and reasoning.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In the between-subjects study, explanation styles generally did not significantly affect justice perceptions, with the exception of fair process in loans only (loan F(3, 258) = 2.71, p = 0.046) and appropriate factors (F(3, 258) = 5.35, p = 0.001) showing significant between group differences.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The justice constructs developed for human decision-making (informational, procedural, distributive) are applicable to and valid for evaluating algorithmic decision-making contexts.
- Rests on an assumption: Participants' Likert-scale responses to the justice statements accurately measure their perceived justice.
- Rests on an assumption: The four explanation styles are sufficiently distinct and were correctly perceived by participants as intended.
- Rests on an assumption: The sample of Prolific Academic participants is similar enough to the lab study participants to allow comparison.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“In some cases, what was missing was not just information, but also complexity:”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Rests on an assumption: The fictionalised scenarios used in the lab and online studies are realistic and representative enough to elicit valid responses about algorithmic decision-making.
- Rests on an assumption: The thematic analysis themes identified by the coders accurately represent participants' interpretations and reasoning.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For some participants, the very idea of an algorithmic system making an important decision on the basis of past data seemed unfair;”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Rests on an assumption: The fictionalised scenarios used in the lab and online studies are realistic and representative enough to elicit valid responses about algorithmic decision-making.
- Rests on an assumption: The thematic analysis themes identified by the coders accurately represent participants' interpretations and reasoning.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In others, the missing element was a sense of proportionality.”
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Rests on an assumption: The fictionalised scenarios used in the lab and online studies are realistic and representative enough to elicit valid responses about algorithmic decision-making.
- Rests on an assumption: The thematic analysis themes identified by the coders accurately represent participants' interpretations and reasoning.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Whereas for others, delegation of such responsibility to a computer, and the authority of statistical models, actually removed the relevance of questions of fairness altogether; the system 'just does what it is supposed to', and as long as its inferences are accurate, it could be seen as 'statistically fair' ( PC ).”
hedgeddirectSingle case- Asserted: fails — The authors may not put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The authors may not put it forward as true. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Rests on an assumption: The fictionalised scenarios used in the lab and online studies are realistic and representative enough to elicit valid responses about algorithmic decision-making.
- Rests on an assumption: The thematic analysis themes identified by the coders accurately represent participants' interpretations and reasoning.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The meaning and relevance of moral concepts like 'fairness' and 'desert' in the context of a computer system was occasionally questioned by certain participants.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Rests on an assumption: The fictionalised scenarios used in the lab and online studies are realistic and representative enough to elicit valid responses about algorithmic decision-making.
- Rests on an assumption: The thematic analysis themes identified by the coders accurately represent participants' interpretations and reasoning.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In some cases, while the individual premises seemed understandable, their relevance to the reasoning behind the overall decision was not:”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Rests on an assumption: The fictionalised scenarios used in the lab and online studies are realistic and representative enough to elicit valid responses about algorithmic decision-making.
- Rests on an assumption: The thematic analysis themes identified by the coders accurately represent participants' interpretations and reasoning.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Issues raised in this regard were the importance of scientific rigour, including the use of sufficient sample sizes:”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Rests on an assumption: The fictionalised scenarios used in the lab and online studies are realistic and representative enough to elicit valid responses about algorithmic decision-making.
- Rests on an assumption: The thematic analysis themes identified by the coders accurately represent participants' interpretations and reasoning.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Many criticisms of the decisions came down to things that were missing or unaccounted for.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Rests on an assumption: The fictionalised scenarios used in the lab and online studies are realistic and representative enough to elicit valid responses about algorithmic decision-making.
- Rests on an assumption: The thematic analysis themes identified by the coders accurately represent participants' interpretations and reasoning.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The results (see Table 1) partially confirmed our hypotheses, as the stronger correlations were connected to the construct connections we believed would exist.”
plaincorrelationaln = 325- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The justice constructs developed for human decision-making (informational, procedural, distributive) are applicable to and valid for evaluating algorithmic decision-making contexts.
- Rests on an assumption: Participants' Likert-scale responses to the justice statements accurately measure their perceived justice.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In addition to comparisons between different individuals, participants also commented on the acceptability of judging a single individual's future behaviour on the basis of their past behaviour.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Rests on an assumption: The fictionalised scenarios used in the lab and online studies are realistic and representative enough to elicit valid responses about algorithmic decision-making.
- Rests on an assumption: The thematic analysis themes identified by the coders accurately represent participants' interpretations and reasoning.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Although participant CS was concerned that behaviour change in response to an algorithm could prove dangerous”
hedgedconditionalSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Rests on an assumption: The fictionalised scenarios used in the lab and online studies are realistic and representative enough to elicit valid responses about algorithmic decision-making.
- Rests on an assumption: The thematic analysis themes identified by the coders accurately represent participants' interpretations and reasoning.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Tukey's post-hoc paired tests (see Figure 3) showed that casebased explanations result in lower perceptions of appropriateness, fair process perception, and (in the loans case) deservedness, consistently compared to sensitivity based styles and occasionally compared to other styles.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The justice constructs developed for human decision-making (informational, procedural, distributive) are applicable to and valid for evaluating algorithmic decision-making contexts.
- Rests on an assumption: Participants' Likert-scale responses to the justice statements accurately measure their perceived justice.
- Rests on an assumption: The four explanation styles are sufficiently distinct and were correctly perceived by participants as intended.
- Rests on an assumption: The sample of Prolific Academic participants is similar enough to the lab study participants to allow comparison.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Several responses suggested that explanations ought to help guide future action, something which was particularly lacking in the demographic explanations:”
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Rests on an assumption: The fictionalised scenarios used in the lab and online studies are realistic and representative enough to elicit valid responses about algorithmic decision-making.
- Rests on an assumption: The thematic analysis themes identified by the coders accurately represent participants' interpretations and reasoning.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Others tried to articulate the system's reasoning as a series of rules and consequences:”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Rests on an assumption: The fictionalised scenarios used in the lab and online studies are realistic and representative enough to elicit valid responses about algorithmic decision-making.
- Rests on an assumption: The thematic analysis themes identified by the coders accurately represent participants' interpretations and reasoning.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“There was a significant effect of explanation styles on perceptions of fair process (loan scenario F(3, 260) = 7.52, p < .001; insurance scenario F(3, 260) = 4.5, p = .004), on the perception of appropriate use of factors (loan F(3, 260) = 3.312, p = .02; insurance F(3, 260) = 6.44, p = .0003), and on the perception of deserved outcome (only in the loans case, F(3, 260) = 7.31, p = .0001).”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The justice constructs developed for human decision-making (informational, procedural, distributive) are applicable to and valid for evaluating algorithmic decision-making contexts.
- Rests on an assumption: Participants' Likert-scale responses to the justice statements accurately measure their perceived justice.
- Rests on an assumption: The four explanation styles are sufficiently distinct and were correctly perceived by participants as intended.
- Rests on an assumption: The sample of Prolific Academic participants is similar enough to the lab study participants to allow comparison.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For the same participant, the specificity of the figures used in some explanations (particularly, input influence and sensitivity ) lead her to conclude that the computer must be using arbitrary thresholds, which in itself seemed 'mean':”
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Rests on an assumption: The fictionalised scenarios used in the lab and online studies are realistic and representative enough to elicit valid responses about algorithmic decision-making.
- Rests on an assumption: The thematic analysis themes identified by the coders accurately represent participants' interpretations and reasoning.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Many of the reactions concerned attempts to decipher the logic of the algorithm.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Rests on an assumption: The fictionalised scenarios used in the lab and online studies are realistic and representative enough to elicit valid responses about algorithmic decision-making.
- Rests on an assumption: The thematic analysis themes identified by the coders accurately represent participants' interpretations and reasoning.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Faced with seemingly inexplicable reasoning processes, some participants concluded that the system must be simply 'making it up' ( OR ) and therefore on the face of it unfair.”
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Rests on an assumption: The fictionalised scenarios used in the lab and online studies are realistic and representative enough to elicit valid responses about algorithmic decision-making.
- Rests on an assumption: The thematic analysis themes identified by the coders accurately represent participants' interpretations and reasoning.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Others considered it problematic that with an automated system, 'there's no sense of negotiation' ( DR ), and no opportunity for 'human interaction' ( CS ).”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Rests on an assumption: The fictionalised scenarios used in the lab and online studies are realistic and representative enough to elicit valid responses about algorithmic decision-making.
- Rests on an assumption: The thematic analysis themes identified by the coders accurately represent participants' interpretations and reasoning.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For some, it was the absence of important information:”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Rests on an assumption: The fictionalised scenarios used in the lab and online studies are realistic and representative enough to elicit valid responses about algorithmic decision-making.
- Rests on an assumption: The thematic analysis themes identified by the coders accurately represent participants' interpretations and reasoning.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Participant PC struggled to evaluate the moral dimensions of computer-driven decisions:”
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Rests on an assumption: The fictionalised scenarios used in the lab and online studies are realistic and representative enough to elicit valid responses about algorithmic decision-making.
- Rests on an assumption: The thematic analysis themes identified by the coders accurately represent participants' interpretations and reasoning.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“A series of comments reflected upon the ways in which the very use of an algorithmic decision-making system could be impersonal and dehumanising for the subject of the decision.”
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Rests on an assumption: The fictionalised scenarios used in the lab and online studies are realistic and representative enough to elicit valid responses about algorithmic decision-making.
- Rests on an assumption: The thematic analysis themes identified by the coders accurately represent participants' interpretations and reasoning.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Some emphasised that an individual who seems similar to past individuals may in fact have extenuating circumstances, particularly in response to case-based explanations:”
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Rests on an assumption: The fictionalised scenarios used in the lab and online studies are realistic and representative enough to elicit valid responses about algorithmic decision-making.
- Rests on an assumption: The thematic analysis themes identified by the coders accurately represent participants' interpretations and reasoning.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The think-aloud responses from the lab study resulted in five major themes: the lack of human touch; interpretations of the system's reasoning; the use of statistical inference; the degree of actionability in an explanation; important aspects which were seen to be unaccounted for by the system; and the meaning and relevance of moral concepts.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The fictionalised scenarios used in the lab and online studies are realistic and representative enough to elicit valid responses about algorithmic decision-making.
- Rests on an assumption: The thematic analysis themes identified by the coders accurately represent participants' interpretations and reasoning.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For DR , here was a sense that the very notion of 'fairness' may not be attributable to a system that is primarily designed to serve a different organisational goal like efficiency:”
hedgedSingle case- Asserted: fails — The authors may not put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The authors may not put it forward as true. The paper’s conclusions may not depend on it.
- Rests on an assumption: The fictionalised scenarios used in the lab and online studies are realistic and representative enough to elicit valid responses about algorithmic decision-making.
- Rests on an assumption: The thematic analysis themes identified by the coders accurately represent participants' interpretations and reasoning.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Conversely, where a judgement was made on the basis of circumstances which were outside the individual's control, or the course of action suggested was deemed unrealistic, the outcome was seen as undeserved:”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
- Rests on an assumption: The fictionalised scenarios used in the lab and online studies are realistic and representative enough to elicit valid responses about algorithmic decision-making.
- Rests on an assumption: The thematic analysis themes identified by the coders accurately represent participants' interpretations and reasoning.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“One participant described how she might have it both ways, simultaneously entertaining dual standards of justice, one juridical, the other computational;”
hedgedSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Rests on an assumption: The fictionalised scenarios used in the lab and online studies are realistic and representative enough to elicit valid responses about algorithmic decision-making.
- Rests on an assumption: The thematic analysis themes identified by the coders accurately represent participants' interpretations and reasoning.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Such findings provide a rationale for giving individuals rights to information about significant decisions; requiring decision-makers to explain their decisions should promote informational justice, which should help people assess procedural justice and distributive justice in turn.”
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Work on perceptions of justice reveals much about the role of explanations in ethical assessments of human decisions, but the applicability of these findings to algorithmic decisions and the range of proposed explanation systems proposed for them remains largely unexplored.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Such remarks suggest that if algorithmic decisions are held to norms of social behaviour, interactional justice may be a relevant dimension for evaluation.”
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. It may state two separate claims as one.
- Rests on an assumption: The fictionalised scenarios used in the lab and online studies are realistic and representative enough to elicit valid responses about algorithmic decision-making.
- Rests on an assumption: The thematic analysis themes identified by the coders accurately represent participants' interpretations and reasoning.
In the paper
“According to Colquitt and others, justice perceptions can be broken down into several aspects or antecedents [10].”
What the sources say
- Not checked yetcites [10] 10. Jason A Colquitt, Donald E Conlon, Michael J Wesson, Christopher OLH Porter, and K Yee Ng. 2001. Justice at the Millennium: A Meta-Analytic Review of 25 Years of Organizational Justice Research. Journal of Applied Psychology 86, 3 (2001). DOI: http://dx.doi.org/10.1037//0021-9010.86.3.425 source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
In the paper
“According to the literature on justice perceptions above, we should expect to find correlations between fair process and deserved outcome , understanding and fair process , appropriateness of factors and deserved outcome , and appropriateness of factors and fair process [10].”
What the sources say
- Not checked yetcites [10] 10. Jason A Colquitt, Donald E Conlon, Michael J Wesson, Christopher OLH Porter, and K Yee Ng. 2001. Justice at the Millennium: A Meta-Analytic Review of 25 Years of Organizational Justice Research. Journal of Applied Psychology 86, 3 (2001). DOI: http://dx.doi.org/10.1037//0021-9010.86.3.425 source
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
- Rests on an assumption: The justice constructs developed for human decision-making (informational, procedural, distributive) are applicable to and valid for evaluating algorithmic decision-making contexts.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This is an effect primarily observed, like most effects in the quantitative part of our study, in the within subject study design, indicating that the act of comparison in a particular scenario is important for these differences to become apparent.”
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The justice constructs developed for human decision-making (informational, procedural, distributive) are applicable to and valid for evaluating algorithmic decision-making contexts.
- Rests on an assumption: Participants' Likert-scale responses to the justice statements accurately measure their perceived justice.
- Rests on an assumption: The four explanation styles are sufficiently distinct and were correctly perceived by participants as intended.
- Rests on an assumption: The sample of Prolific Academic participants is similar enough to the lab study participants to allow comparison.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Thematic codes were independently developed by three coders, after which a single set was jointly agreed upon. The data were then re-coded using the new set of codes.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Rests on an assumption: The thematic analysis themes identified by the coders accurately represent participants' interpretations and reasoning.
None of the sources cited for this claim have been checked yet.
In the paper
“In this sense, while algorithmic decision-making implicates dimensions of justice, it may also mitigate them, in so far as people resist imputing morality to a computer system [25, 5].”
What the sources say
- Not checked yetcites [25] 25. Batya Friedman and Peter H Kahn. 1992. Human agency and responsible computing: Implications for computer system design. Journal of Systems and Software 17, 1 (1992), 7-14. source
- Not checked yetcites [5] 5. Joanna J Bryson. 2010. Robots should be slaves. Close Engagements with Artificial Companions: Key social, psychological, ethical and design issues (2010), 63-74.
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The fictionalised scenarios used in the lab and online studies are realistic and representative enough to elicit valid responses about algorithmic decision-making.
- Rests on an assumption: The thematic analysis themes identified by the coders accurately represent participants' interpretations and reasoning.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“to avoid this, an additional statement about agreement (1, agreement ) was also included prior to the others.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
None of the sources cited for this claim have been checked yet.
In the paper
“Much of this research is based on field surveys of individuals with personal experience of being affected by decisions (e.g. [79]), although controlled experiments involving simulated decisionmaking and scenario-based methods have also been used [78].”
What the sources say
- Not checked yetcites [79] 79. Kees Van den Bos, Lynn Van der Velden, and Allan Lind. 2014. On the role of perceived procedural justice in citizens' reactions to government decisions and the handling of conflicts. Utrecht Law Review 10, 4 (2014). DOI: http://dx.doi.org/10.18352/ulr.287
- Not checked yetcites [78] 78. Kees Van den Bos, E Allan Lind, Riël V ermunt, and Henk AM Wilke. 1997. How do I judge my outcome when I do not know the outcome of others? The psychology of the fair process effect. Journal of Personality and Social Psychology 72, 5 (1997), 1034. source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Requiring organisations to explain the logic behind their algorithmic decision-making systems (informational justice) enables affected individuals to assess whether the logic of the system is just (procedural justice), which in turn might moderate their assessments of fairness of the decision outcomes (distributive justice).”
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This was partly due to the hypothetical nature of the scenarios, and partly due to being too human-specific to apply to a computer-based interaction.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Similarly, the United States' Equal Credit Opportunity Act (ECOA) requires lenders to provide 'statements of reasons' when applications are denied loans [32].”
What the sources say
- Not checked yetcites [32] 32. David C Hsia. 1978. Credit scoring and the equal credit opportunity act. Hastings LJ 30 (1978), 371.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Many of the previously mentioned machine learning model explanation systems have not been subjected to user evaluations”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“We also do not yet know much the extent to which findings on correlations between perceptions of justice from human decision-making contexts hold in the context of algorithmic decisions.”
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“However the nuances of the different explanation approaches recently developed in interpretable ML have not yet been explored in relation to perceptions of justice.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
In the paper
“These aspects of justice are distinct but correlated [10].”
What the sources say
- Not checked yetcites [10] 10. Jason A Colquitt, Donald E Conlon, Michael J Wesson, Christopher OLH Porter, and K Yee Ng. 2001. Justice at the Millennium: A Meta-Analytic Review of 25 Years of Organizational Justice Research. Journal of Applied Psychology 86, 3 (2001). DOI: http://dx.doi.org/10.1037//0021-9010.86.3.425 source
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“Statement 2 ( understanding ) aimed to capture informational justice [27]; statements 3 and 4 ( appropriateness of factors, fair process ) pertained to procedural justice; and statement 5 ( deserved outcome ) addressed distributive dimensions [11].”
What the sources say
- Not checked yetcites [27] 27. Jerald Greenberg and R Cropanzano. 1993. The social side of fairness: Interpersonal and informational classes of organizational justice. Justice in the workplace: Approaching fairness in human resource management, Lawrence Erlbaum Associates, Hillsdale, NJ (1993).
- Not checked yetcites [11] 11. Jason A Colquitt and Jessica B Rodell. 2015. Measuring justice and fairness. In Oxford Handbook of Justice in the Workplace , Russell Cropanzano and Maureen L Ambrose (Eds.). Oxford University Press, Oxford, 187-202.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
- Rests on an assumption: The justice constructs developed for human decision-making (informational, procedural, distributive) are applicable to and valid for evaluating algorithmic decision-making contexts.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“One possible explanation is that when people repeatedly see the same explanation style, they become habituated and pay more attention to the features of the cases rather than any specific features of the explanation provided.”
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“the scenarios considered were hypothetical, not affecting the participants directly, and therefore lacked the first-person consequences and significance of a real world decision”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The introduction of algorithmic systems seems likely to shake up previously stable categories and concepts, requiring adjustment and incorporation into one's moral world-view.”
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
- Rests on an assumption: The fictionalised scenarios used in the lab and online studies are realistic and representative enough to elicit valid responses about algorithmic decision-making.
- Rests on an assumption: The thematic analysis themes identified by the coders accurately represent participants' interpretations and reasoning.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The algorithmic nature of these systems results in an array of novel considerations which are not captured by traditional research on perceptions of justice.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“people may realistically face multiple explanations for similar algorithmic decisions when applying for a loan, mortgage or credit”
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“creators of ML explanation systems have not typically designed them with the information needs of those individuals facing significant personal consequences of model outputs in mind”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“When exposed to multiple explanation styles regarding the same case, the differences may become more salient.”
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“differences in the length and format of the text required to impart an explanation between explanation conditions may have been a confounding factor”
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The authors state this as an assumption.
The GDPR automated-decision hurdle ('based solely on automated processing') can be read less restrictively than legal analysis has indicated, so the selected scenarios may fall under GDPR.
“The hurdle that a decision must be 'based solely on automated processing' is here read less restrictively than legal analysis has indicated [81, 20].”
The paper relies on this without support.
Explanations are most likely to be requested in response to a decision with a negative outcome for the individual.
“Because explanations are most likely to be requested in response to a decision which has a negative outcome for the individual, and considerations of procedural justice have been found to be more strongly felt in such cases [4], each scenario ended in a negative outcome for the individual.”
The paper relies on this without support.
The justice constructs developed for human decision-making (informational, procedural, distributive) are applicable to and valid for evaluating algorithmic decision-making contexts.
“The justice constructs used in all three studies were based on those developed in the psychology of justice research, summarised by Colquitt et al [11].”
“The remaining questions pertained to informational, procedural and distributive justice.”
The paper relies on this without support.
Participants' Likert-scale responses to the justice statements accurately measure their perceived justice.
“Participants were asked to rate their agreement with five statements on a 5-point Likert scale”
“Participants were asked to indicate their agreement with the 5 measures described in phase 1 using the same Likert scale.”
The paper relies on this without support.
The fictionalised scenarios used in the lab and online studies are realistic and representative enough to elicit valid responses about algorithmic decision-making.
“The first study consisted of in-person semi-structured interviews focused around a series of fictionalised cases as described above.”
“Cases were presented with a brief description of the individual, the decision outcome, the explanation (except in the control condition) and an illustrative image depicting the context.”
The paper relies on this without support.
The four explanation styles are sufficiently distinct and were correctly perceived by participants as intended.
“Each case was presented with one of the 4 explanation styles described above, or a control condition in which no explanation was provided.”
“Each decision was explained using a different style (in a random order), allowing participants to directly compare multiple styles applied to an individual subject.”
The paper relies on this without support.
The thematic analysis themes identified by the coders accurately represent participants' interpretations and reasoning.
“Thematic analysis was undertaken to identify common themes in participants' interpretation, evaluation and reasoning in response to cases. Thematic codes were independently developed by three coders, after which a single set was jointly agreed upon. The data were then re-coded using the new set of codes.”
The paper relies on this without support.
The sample of Prolific Academic participants is similar enough to the lab study participants to allow comparison.
“Participants were recruited via the study platform Prolific Academic, 4 and filtered to include only individuals over 18 based in the UK, in order to maintain similarity with the participants recruited in phase 1.”
The paper relies on this without support.
There is still much to learn about what people want and need to know about algorithms in order to hold them accountable to justice.
“there is still much to learn about what people want and need to know about algorithms in order to hold them accountable to justice”
The paper relies on this without support.
The explanation styles proposed in the interpretable machine learning literature vary in the information they contain.
“the explanation styles proposed in the interpretable machine learning literature vary in the information they contain”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“and that humans would treat decision subjects with dignity and compassion (e.g., Q.31; also in line with previous research [17,68,94]).”
From Disentangling Fairness Perceptions in Algorithmic Decision-Making: the Effects of Explanations, Human Oversight, and Contestability (2023) · cites this paper as [17]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Furthermore, humans”
From Artificial Intelligence and Management: The Automation–Augmentation Paradox (2021) · cites this paper as Binns, Van Kleek, Veale, Lyngs, Zhao, & Shadbolt, 2018
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This philosophically grounded notion is matched by human perceptions that decisions made by algorithms are less trustworthy and fair than those made by humans and that being evaluated by algorithms is dehumanizing ( Binns et al., 2018 ;Lee, 2018 ).”
From The flaws of policies requiring human oversight of government algorithms (2022) · cites this paper as Binns et al., 2018
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“There is evidence that suggests that decision subjects care about justice-related aspects in algorithmic decisionmaking, as they care in human decision-making [17].”
From Disentangling Fairness Perceptions in Algorithmic Decision-Making: the Effects of Explanations, Human Oversight, and Contestability (2023) · cites this paper as [17]
Statistics recomputed
Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.
- ConsistentF(3, 258) = 2.71reported p = .046 · recomputed p = .046
“F(3, 258) = 2.71, p = .046”
- ConsistentF(3, 258) = 5.35reported p = .001 · recomputed p = .001
“F(3, 258) = 5.35, p = .001”
- ConsistentF(3, 260) = 4.5reported p = .004 · recomputed p = .004
“F(3, 260) = 4.5, p = .004”
- ConsistentF(3, 260) = 3.312reported p = .02 · recomputed p = .021
“F(3, 260) = 3.312, p = .02”
- ConsistentF(3, 260) = 6.44reported p = .0003 · recomputed p < .001
“F(3, 260) = 6.44, p = .0003”
- ConsistentF(3, 260) = 7.31reported p = .0001 · recomputed p < .001
“F(3, 260) = 7.31, p = .0001”
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Do different explanation styles result in differences in perceived levels of justice? · p. 9
“In the within-subjects study-where multiple explanation styles were presented for the same decision-significant effects were observed across all questions and decision-making contexts.”
§ Do different explanation styles result in differences in perceived levels of justice? · p. 9
“on the perception of deserved outcome (only in the loans case, F(3, 260) = 7.31, p = .0001).”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Quantitative results · p. 8
“we calculated Spearman's rho correlation coefficients (with 0.05 confidence level bootstrapped intervals).”
§ Do different explanation styles result in differences in perceived levels of justice? · p. 9
“Table 1. Spearman's rho correlations for justice constructs with bootstrapped (1000 rep) confidence interval at 0.02 level.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Qualitative Results: Themes and reflections · p. 6
“The think-aloud responses from the lab study resulted in five major themes”
§ Qualitative Results: Themes and reflections · p. 6
“the lack of human touch; interpretations of the system's reasoning; the use of statistical inference; the degree of actionability in an explanation; important aspects which were seen to be unaccounted for by the system; and the meaning and relevance of moral concepts.”
Citations
Cites
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555 · cited 4×
“are applicable to any kind of learning algorithm, such as LIME [62], rather than methods which only work for particular ML models (e.”
BackgroundMethodInfluential“Why Should I Trust You?”: Explaining the Predictions of Any Classifier
Ribeiro, M., Singh, S., & Guestrin, C. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations · doi:10.18653/v1/n16-3020 · Cited by 6,682
The Social Psychology of Procedural Justice
Lind, E. A., & Tyler, T. R. (1988) · Critical Issues in Social Justice · doi:10.1007/978-1-4899-2115-4 · Cited by 6,585 · cited 2×
“A significant body of research in legal and organisational psychology suggests that people do not only care about whether the outcome of a decision benefits them, but also whether it meets standards of justice [71, 40, 11].”
BackgroundJustice at the millennium: A meta-analytic review of 25 years of organizational justice research.
Colquitt, J. A., Conlon, D. E., Wesson, M. J., Porter, C. O. L. H., & Ng, K. Y. (2001) · Journal of Applied Psychology · doi:10.1037/0021-9010.86.3.425 · Cited by 5,942 · cited 5×
“According to the literature on justice perceptions above, we should expect to find correlations between fair process and deserved outcome , understanding and fair process , appropriateness of factors and deserved outcome , and appropriateness of factors and fair process [10].”
BackgroundResultInfluentialLipton, Z. C. (2018) · Queue · doi:10.1145/3236386.3241340 · Cited by 5,427 · cited 3×
“2Although we acknowledge this is an important challenge if explanations are to be more than just comforting stories, as cautioned by Lipton [41].”
BackgroundLikert scales, levels of measurement, and the “laws” of statistics
Norman, G. (2010) · Advances in Health Sciences Education · doi:10.1007/s10459-010-9222-y · Cited by 4,850 · cited 1×
“6Some readers might have reservations over parametric tests being applied to ordinal data: we point to recent literature that has emphasised [53] that Likert data are robust to parametric assumptions, and that ‘parametric methods can be utilized without concern for “getting the wrong answer”’.”
BackgroundComparing Individual Means in the Analysis of Variance
Tukey, J. W. (1949) · Biometrics · doi:10.2307/3001913 · Cited by 4,302 · cited 1×
“We used ANOVA tests, followed by Tukey’s post-hoc paired tests [75], to analyse the relative effects of different explanation styles within different study designs.”
MethodBias in random forest variable importance measures: Illustrations, sources and a solution
Strobl, C., Boulesteix, A., Zeileis, A., & Hothorn, T. (2007) · BMC Bioinformatics · doi:10.1186/1471-2105-8-25 · Cited by 3,970
How the machine “thinks”: Understanding opacity in machine learning algorithms
Burrell, J. (2016) · Big Data & Society · doi:10.1177/2053951715622512 · Cited by 3,196
Towards A Rigorous Science of Interpretable Machine Learning
Doshi-Velez, F., & Kim, B. (2017) · arXiv · doi:10.48550/arxiv.1702.08608 · Cited by 3,121 · cited 2×
“Furthermore, as research into practically achievable explanations for machine learning systems accelerates, researchers are turning to consider which explanation styles are more desirable, rather than simply possible to implement [41, 15].”
Background
Research works printed but not linked: 8
- Joanna J Bryson. 2010. Robots should be slaves. Close Engagements with Artificial Companions: Key social, psychological, ethical and design issues (2010), 63--74.
- 10.5555/270318.270324 · DOI
- 10.5555/1386322 · DOI
- Zachary C Lipton. 2016. The Mythos of Model Interpretability. In 2016 ICML Workshop on Human Interpretability in Machine Learning (WHI 2016). New York. https://arxiv.org/abs/1606.03490 · arXiv
- Gregory Mitchell. 2004. Libertarian Paternalism is an Cxymoron. Nw. UL Rev. 99 (2004), 1245.
- 10.5555/3175762 · DOI
- Cathy O'Neill. 2016. How algorithms rule our working lives. The Guardian (2016).
- Frank Pasquale. 2015. Digital Star Chamber. Aeon (2015). https://aeon.co/essays/ judge-jury-and-executioner-the-unaccountable-algorithm
Cited by
Sorted by citations
Items 101–200 of 696
Toward involving end-users in interactive human-in-the-loop AI fairness
Nakao, Y., Stumpf, S., Ahmed, S., Naseer, A., & Strappelli, L. (2022) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3514258 · Cited by 69
“The need to directly involve end-users in assessing the fairness of AI systems and making ML models fairer has been emphasized [10, 21, 40, 69, 78, 80], and we believe our tool is a step in that direction.”
Krupiy, T. (2020) · Computer law & security review · doi:10.1016/j.clsr.2020.105429 · Cited by 81
Schoeffer, J., Kuehl, N., & Machowski, Y. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533218 · Cited by 67
“[10] only measure the understandability aspect of informational fairness for individual explanation styles; Schlicker et al.”
BackgroundInfluentialPan, C. A., Yakhmi, S., Iyer, T., Strasnick, E., Zhang, A. X., & Bernstein, M. S. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3512929 · Cited by 77
Tal, A. S., Kuflik, T., & Kliger, D. (2022) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2022.2095705 · Cited by 73
“We created the explanation styles used in the experiment based on the explanations presented in Binns et al. (2018) and Dodge et al. (2019), and according to the explanation creation guidelines in Gedikli et al. (2014).”
BackgroundMethodInfluentialSoliciting Stakeholders’ Fairness Notions in Child Maltreatment Predictive Systems
Cheng, H.-F., Stapleton, L., Wang, R., Bullock, P., Chouldechova, A., Wu, Z. S. S., & Zhu, H. (2021) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445308 · Cited by 71
“This body of work suggests that fairness principles need to be context-specifc, and the algorithmic systems should embody the fairness notions derived from the community of stakeholders [10, 11, 23, 48, 49, 62].”
BackgroundYu, J., Bekerian, D. A., & Osback, C. (2024) · Encyclopedia · doi:10.3390/encyclopedia4040109 · Cited by 114
Dissonance Between Human and Machine Understanding
Zhang, Z., Singh, J., Gadiraju, U., & Anand, A. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359158 · Cited by 62
I agree with the decision, but they didn't deserve this
Kasinidou, M., Kleanthous, S., Barlas, P., & Otterbacher, J. (2021) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3442188.3445931 · Cited by 66
“Binns et al. [5] ran an experiment using different explanation styles (input influence, sensitivity, case-based, demographics).”
BackgroundMethodInfluentialChiang, C.-W., Lu, Z., Li, Z., & Yin, M. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581015 · Cited by 72
“the error of the AI [99, 120], having people describe the cause of an AI model’s prediction [14, 80, 100], or having people analyze the importance and contributions of diferent features to an AI model’s”
BackgroundThe Material Conditions of Platforms: Monopolization Through Decentralization
Blanke, T., & Pybus, J. (2020) · Social Media + Society · doi:10.1177/2056305120971632 · Cited by 58
“For the user, this means that even when then they are logged out of the Facebooks and Googles, they are probably still logged into their service eco-system via at least one of its developer tools, which have been integrated in over 45% of mobile applications (Binns et al., 2018).”
Bertrand, A., Viard, T., Belloum, R., Eagan, J. R., & Maxwell, W. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581314 · Cited by 71
“Fairness questionnaires from [15] or Lee et al. [68].”
BackgroundMethodHow fair can we go in machine learning ? Assessing the boundaries of accuracy and fairness
Valdivia, A., Sánchez‐Monedero, J., & Casillas, J. (2021) · International Journal of Intelligent Systems · doi:10.1002/int.22354 · Cited by 65
“Fairness in ML has emerged as a community initially motivated to develop technological solutions to the disparate impact and treatment by biased algorithms [8, 9, 10, 11, 5] that also moves to a broader and multi-disciplinary understanding of the issues of socio-technological interventions [12, 13, 14, 15].”
BackgroundAlgorithmic and human decision making: for a double standard of transparency
Günther, M., & Kasirzadeh, A. (2021) · AI & Society · doi:10.1007/s00146-021-01200-5 · Cited by 62
“677) support their claim that algorithms can be given intentional stance explanations? They borrow four types of algorithmic explanations from Binns et al. (2018). The types are meant to be analogues to human intentional stance explanations, in particular the ones they call “input influence-based explanations” and “se…”
BackgroundAn Assessment of the Barriers Impacting Responsible Artificial Intelligence
Merhi, M. I. (2022) · Information Systems Frontiers · doi:10.1007/s10796-022-10276-3 · Cited by 58
“Binns et al (2018) and Bjerring and Busch (2020) argue that the obligation of explanations of the models and the outputs of AI may increase the level of trust toward AI and thus will increase its usage.”
BackgroundInterpretable Data-Based Explanations for Fairness Debugging
Pradhan, R., Zhu, J., Glavic, B., & Salimi, B. (2022) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/3514221.3517886 · Cited by 65
Selective Explanations: Leveraging Human Input to Align Explainable AI
Lai, V. W.-M., Zhang, Y., Chen, C., Liao, Q. V., & Tan, C. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3610206 · Cited by 75
“[49, 50, 102]), and conducting empirical studies to investigate the effects of explanations on people’s task performance [6, 57, 108], efficiency [1, 17, 21, 34, 36, 39, 54, 55, 59, 64, 88, 99, 103], cognitive load [1, 37], understanding [4, 8, 11, 14, 21, 68, 88, 98, 103], subjective perceptions of AI [9, 15, 25, 37,…”
BackgroundSilva: Interactively Assessing Machine Learning Fairness Using Causality
Yan, J. N., Gu, Z., Lin, H., & Rzeszotarski, J. M. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376447 · Cited by 54
Shen, H., Jin, H., Cabrera, Á. A., Perer, A., Zhu, H., & Hong, J. (2020) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3415224 · Cited by 60
Mitigating Bias in Algorithmic Systems—A Fish-eye View
Orphanou, K., Otterbacher, J., Kleanthous, S., Batsuren, K., Giunchiglia, F., Bogina, V., Tal, A. S., Hartman, A., & Kuflik, T. (2022) · ACM Computing Surveys · doi:10.1145/3527152 · Cited by 58
“ML Model Model Explainability [20, 36, 52, 72, 112, 174, 190, 227] [27, 42, 96, 125, 191, 228] Output Outcome Explainability [47, 157, 164, 164, 165, 187, 197] [16, 65, 84, 177, 182, 200, 215, 227, 229] HCI User Model Explainability [88] User/Output Outcome Explainability [15, 56, 67, 159] RecSys User/Output Outcome E…”
BackgroundLanger, M., Hunsicker, T., Feldkamp, T., König, C. J., & Grgić-Hlača, N. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517527 · Cited by 55
Tal, A. S., Kuflik, T., & Kliger, D. (2022) · Ethics and Information Technology · doi:10.1007/s10676-022-09623-4 · Cited by 59
“Furthermore, Binns et al. (2018) suggested that users’ fairness perception may be enhanced only when multiple explanation styles (combining global and local explanations) are presented to the users and that the scenario effects (e.g., system’s domain, results, usage) outweigh the explanation effect…”
BackgroundMethodResultInfluentialSocial impacts of algorithmic decision-making: A research agenda for the social sciences
Gerdon, F., Bach, R. L., Kern, C. A., & Kreuter, F. (2022) · Big Data & Society · doi:10.1177/20539517221089305 · Cited by 71
“…documents how individuals evaluate algorithmic decisions compared to human decisions, fi nding both similarities and differences (Araujo et al., 2020; Binns et al., 2018; Plane et al., 2017; see section “ Data preparation and analysis — from fairness in algorithmic output to fairness in social…”
BackgroundInfluentialA Critical Survey on Fairness Benefits of Explainable AI
Deck, L., Schoeffer, J., De‐Arteaga, M., & Kühl, N. (2024) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3630106.3658990 · Cited by 107
Nakao, Y., Strappelli, L., Stumpf, S., Naseer, A., Regoli, D., & Gamba, G. D. (2022) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2022.2067936 · Cited by 66
“For example, it has been found that while fairness outputs were perceived by participants as statistically fair, some participants were not comfortable with an AI system making high-stakes decisions without a human in the loop (Binns et al., 2018).”
BackgroundResultAoki, N., Tatsumi, T., Naruse, G., & Maeda, K. (2024) · Government Information Quarterly · doi:10.1016/j.giq.2024.101965 · Cited by 75
Saxena, N. A., Huang, K., DeFilippis, E., Radanović, G., Parkes, D. C., & Liu, Y. (2020) · Artificial Intelligence · doi:10.1016/j.artint.2020.103238 · Cited by 57
“As an example, Yaari and Bar-Hillel [35], Pierson [30], Plane et al. [31], Grgi´c-Hlaˇca et al. [20], Grgic-Hlaca et al. [19], Binns et al. [8] used survey methodologies to examine people’s attitudes about fairness and discrimination.”
BackgroundArtificial fairness? Trust in algorithmic police decision-making
Hobson, Z., Yesberg, J. A., Bradford, B., & Jackson, J. (2021) · Journal of Experimental Criminology · doi:10.1007/s11292-021-09484-9 · Cited by 62
“These technologies, and the opaque manner in which they are deployed, raise concerns that they may have unintended consequences and operate outside the scope of traditional oversight and public accountability mechanisms (Binns et al., 2018; Brown et al., 2019).”
BackgroundExplaining Data-Driven Decisions made by AI Systems: The Counterfactual Approach
Fernández-Loría, C., Provost, F. J., & Han, X. (2022) · MIS Quarterly · doi:10.25300/misq/2022/16749 · Cited by 134
Certification Labels for Trustworthy AI: Insights From an Empirical Mixed-Method Study
Scharowski, N., Benk, M., Kühne, S. J., Wettstein, L., & Brühlmann, F. (2023) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3593013.3593994 · Cited by 61
“For both the interviews and the survey, we used a scenario-based approach to investigate people’s attitudes and the effects of a certification label, inspired by past research [5, 32, 35].”
MethodHuman-Centered Approaches to Fair and Responsible AI
Lee, M. K., Grgić-Hlača, N., Tschantz, M. C., Binns, R., Weller, A., Carney, M., & Inkpen, K. M. (2020) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3334480.3375158 · Cited by 57
Self-citationNaveed, S., Stevens, G., & Robin-Kern, D. (2024) · Applied Sciences · doi:10.3390/app142311288 · Cited by 67
How Platform-User Power Relations Shape Algorithmic Accountability
Ramesh, D., Kameswaran, V., Wang, D., & Sambasivan, N. (2022) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3531146.3533237 · Cited by 61
“, of source-code, datasets, models, and processes surrounding the development of ADS) [21, 30, 55, 64, 83, 105] and explainable decisions (i.”
BackgroundPlayer-AI Interaction: What Neural Network Games Reveal About AI as Play
Zhu, J., Villareale, J., Javvaji, N., Risi, S., Löwe, M., Weigelt, R., & Harteveld, C. (2021) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445307 · Cited by 52
“Furthermore, new interdisciplinary research areas have emerged around topics such as explainable AI [9, 66, 83, 101], ethics & fairness [12, 23, 38], and machine learning (ML) as a design material for UX [93, 94].”
BackgroundThe public perceptions of algorithmic decision-making systems: Results from a large-scale survey
Aysolmaz, B., Müller, R., & Meacham, D. (2023) · Telematics and Informatics · doi:10.1016/j.tele.2023.101954 · Cited by 56
“Although perceived transparency is an important aspect of an ADM system on its own, the ways to improve transparency are vast and include a trade-off between increasing complexity and confusion for users (Binns, 2018).”
BackgroundProcedural fairness in algorithmic decision-making: the role of public engagement
Decker, M. C., Wegner, L., & Leicht-Scholten, C. (2024) · Ethics and Information Technology · doi:10.1007/s10676-024-09811-4 · Cited by 69
“…that, even if an ADM incorporates explanations (for example, using XAI), these explanations require specific background knowledge to be understood (Binns et al., 2018 ) – and the necessary background knowledge (as well as the specific design of the explanations) changes as soon as this background…”
This (AI)n’t fair? Employee reactions to artificial intelligence (AI) in career development systems
Köchling, A., Wehner, M., & Ruhle, S. A. (2024) · Review of Managerial Science · doi:10.1007/s11846-024-00789-3 · Cited by 63
“…Hunken-schroer and Luetge 2022; Ochmann et al. 2024), there is less evidence of employees’ interactional justice perceptions depending on whether a human or an AI system makes decisions during career development and promotions (e.g., Binns et al. 2018; Bankins et al. 2022; Newman et al. 2020).”
Ma, S., Chen, Q., Wang, X., Zheng, C., Peng, Z., Yin, M., & Ma, X. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713423 · Cited by 48
Attitudes and Folk Theories of Data Subjects on Transparency and Accuracy in Emotion Recognition
Grill, G., & Andalibi, N. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3512925 · Cited by 49
“Beyond studying algorithms as impactful black-boxes, scholars have also investigated people’s expectations of and attitudes towards algorithms and how they perceive and interact with them [6,17,47,59,118,147].”
BackgroundEhsan, U., Singh, R., Metcalf, J., & Riedl, M. O. (2022) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3531146.3533186 · Cited by 59
Directive Explanations for Actionable Explainability in Machine Learning Applications
Singh, R., Miller, T., Lyons, H., Sonenberg, L., Velloso, E., Vetere, F., Howe, P. D. L., & Dourish, P. (2023) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3579363 · Cited by 57
Towards a multi-stakeholder value-based assessment framework for algorithmic systems
Yurrita, M., Murray-Rust, D., Balayn, A., & Bozzon, A. (2022) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3531146.3533118 · Cited by 50
A Missing Piece in the Puzzle: Considering the Role of Task Complexity in Human-AI Decision Making
Salimzadeh, S., He, G., & Gadiraju, U. (2023) · Proceedings of the 31st ACM Conference on User Modeling, Adaptation and Personalization · doi:10.1145/3565472.3592959 · Cited by 42
“For example, recidivism prediction algorithms have been used to help judges determine whether defendants are likely to re-offend [26, 31, 65, 69], medical diseases are being diagnosed with AI systems [19, 57, 62], and loan risk prediction algorithms are employed to approve or reject loan applications [11, 25, 38, 95].”
BackgroundMethodEvaluating the Promise of Human-Algorithm Collaborations in Everyday Work Practices
Wolf, C., & Blomberg, J. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359245 · Cited by 48
van Berkel, N., Sarsenbayeva, Z., & Gonçalves, J. (2022) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2022.102954 · Cited by 48
He, G., Buijsman, S., & Gadiraju, U. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3610067 · Cited by 56
“The loan prediction task has been widely used to study human-AI decision making where there is a clear risk associated with the decision and a potential benefit in adopting AI advice [5, 9, 27, 60].”
BackgroundFrom human-centered to symbiotic artificial intelligence: a focus on medical applications
Desolda, G., Esposito, A., Lanzilotti, R., Piccinno, A., & Costabile, M. F. (2024) · Multimedia Tools and Applications · doi:10.1007/s11042-024-20414-5 · Cited by 62
Cognitive bias in generative AI influences religious education
Zhang, J., Song, W., & Liu, Y. (2025) · Scientific Reports · doi:10.1038/s41598-025-99121-6 · Cited by 73
“This can exacerbate distrust and conflict between religious groups 31 .”
Street-Level Realities of Data Practices in Homeless Services Provision
Karusala, N., Wilson, J. H., Vayanos, P., & Rice, E. R. W. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359286 · Cited by 41
Explainable software systems: from requirements analysis to system evaluation
Chazette, L., Brunotte, W., & Speith, T. (2022) · Requirements Engineering · doi:10.1007/s00766-022-00393-5 · Cited by 52
“In the literature, many authors refer to this as liability [108] or legal accountability [109].”
BackgroundMethodInherent Limitations of AI Fairness
Buyl, M., & Bie, T. D. (2024) · Communications of the ACM · doi:10.1145/3624700 · Cited by 50
Niforatos, E., Palma, A., Gluszny, R., Vourvopoulos, A., & Liarokapis, F. (2020) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376788 · Cited by 47
Chen, C., & Sundar, S. S. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580805 · Cited by 41
“Second, the positive mediation effect of labeling quality on AI trust shows the value of early communication in shaping user perceptions and trust in AI, which broadens current practices of XAI research that is dominated by post-hoc explanations (see examples [6, 16, 27, 31, 48, 53, 59]).”
BackgroundData-Driven Game Development: Ethical Considerations
El‐Nasr, M. S., & Kleinman, E. (2020) · International Conference on the Foundations of Digital Games (FDG) · doi:10.1145/3402942.3402964 · Cited by 38
“[8] conducted a study in which they found that many participants felt it was dehumanizing or impersonal to allow a machine to make decisions for humans.”
BackgroundTransparency for Whom? Assessing Discriminatory Artificial Intelligence
van Nuenen, T., Ferrer, X., Such, J. M., & Coté, M. (2020) · Computer · doi:10.1109/mc.2020.3002181 · Cited by 38
Dimensions of Diversity in Human Perceptions of Algorithmic Fairness
Grgić-Hlača, N., Lima, G., Weller, A., & Redmiles, E. M. (2022) · ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization (EAAMO) · doi:10.1145/3551624.3555306 · Cited by 55
“Shin [94] found a positive correlation between explainability and perceived fairness, and Binns et al. [11] found that this effect depends on the type of explanation provided.”
BackgroundBenjamin, J. J., Lindley, J., Edwards, E., Rubegni, E., Korjakow, T., Grist, D., & Sharkey, R. (2024) · Designing Interactive Systems Conference · doi:10.1145/3643834.3660677 · Cited by 32
“Further below, parts of this finding also inform limitations we see for future work (see section 6.3).”
BackgroundInfluentialA top-level model of case-based argumentation for explanation: Formalisation and experiments
Prakken, H., & Ratsma, R. (2021) · Argument & Computation · doi:10.3233/aac-210009 · Cited by 32
“Finally, a recent empirical study suggests that a case-based style of explanation may lead to a lower degree of perceived justice than other styles [45].”
BackgroundLewicki, K., Lee, M. S. A., Cobbe, J., & Singh, J. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581463 · Cited by 30
The impact of AI perceived transparency on trust in AI recommendations in healthcare applications
Shabankareh, M., Sahne, S. S. K., Nazarian, A., & Foroudi, P. (2025) · Asia-Pacific Journal of Business Administration · doi:10.1108/apjba-12-2024-0690 · Cited by 37
Designing Civic Technology with Trust
Corbett, E., & Dantec, C. L. (2021) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445341 · Cited by 31
“Public institutions are increasingly partnering with, and drawing from, private industry in designing civic technology platforms [41], smart city infrastructures [1], and automated decision-making systems [9] that mediate interactions between the public and their governments.”
BackgroundA Framework and Benchmarking Study for Counterfactual Generating Methods on Tabular Data
Oliveira, R., & Martens, D. (2021) · Applied Sciences · doi:10.3390/app11167274 · Cited by 36
“Several studies indicate the advantages of counterfactual explanations [7, 14], as they can make algorithmic decisions more understandable for humans [9].”
BackgroundLet the user have a say - voice in automated decision-making
Hellwig, P., Buchholz, V., Kopp, S., & Maier, G. W. (2022) · Computers in Human Behavior · doi:10.1016/j.chb.2022.107446 · Cited by 32
How to Support Users in Understanding Intelligent Systems? Structuring the Discussion
Eiband, M., Buschek, D., & Hußmann, H. (2021) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3397481.3450694 · Cited by 34
“Accountability How fair and controllable is the system? [81] How fair are system decisions? [13] How will algorithmic-decision making impact my (social) work practices? [17] How can I as a designer/developer of decision systems support human values? [45, 96] Users critique the system, often in a wider context beyond u…”
BackgroundTechnology in the Workplace: Opportunities and Challenges
Griep, Y., Vranješ, I., van Hooff, M. L. M., Beckers, D. G. J., & Geurts, S. A. E. (2021) · Springer eBooks · doi:10.1007/978-3-030-74128-0_6 · Cited by 42
“More specifically, the literature suggests that workers seek a “human touch” and resent “being reduced to a percentage” when algorithms are responsible for decisions impacting them (Binns et al. 2018).”
BackgroundCategorical and Continuous Features in Counterfactual Explanations of AI Systems
Warren, G., Byrne, R. M. J., & Keane, M. T. (2023) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3581641.3584090 · Cited by 29
“The burgeoning prevalence of automated decision-making in the public and private sectors has led to increased concerns about the fairness, transparency, and trustworthiness of these Artificial Intelligence (AI) systems [3, 10].”
BackgroundResultPerception of Fairness in Algorithmic Decisions: Future Developers' Perspective
Kleanthous, S., Kasinidou, M., Barlas, P., & Otterbacher, J. (2021) · Patterns · doi:10.1016/j.patter.2021.100380 · Cited by 37
“Binns et al.(22) conducted a study using four different explanation styles (input influence, sensitivity, case-based, demographics).”
BackgroundMethodInfluentialLi, R., Kingsley, S., Fan, C., Sinha, P., Wai, N., Lee, J., Shen, H., Eslami, M., & Hong, J. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3582074 · Cited by 38
“Past work has investigated how these folk theories and existing perceptions can shape and influence people’s attitudes and behaviors towards algorithmic systems [6, 9, 22, 25, 36, 68, 72].”
BackgroundUsing artificial intelligence to enhance patient autonomy in healthcare decision-making
Quiñones, J. L. G. (2024) · AI & Society · doi:10.1007/s00146-024-01956-6 · Cited by 39
“Third, Binns et al. (2018) present four kinds of explanations that could be offered by AI 9 : (a) input influence based, each input variable goes together with a quantitative measure of its influence on the final output; (b) demographic based, the decision subject (i.e., patient) is compared with…”
Co-Writing with AI, on Human Terms: Aligning Research with User Demands Across the Writing Process
Reza, M., Thomas-Mitchell, J., Dushniku, P., Laundry, N., Williams, J. J., & Kuzminykh, A. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3757566 · Cited by 37
“…Nathan Laundry, University of Toronto, Toronto, Ontario, Canada, nathan.laundry@mail.utoronto.ca; Joseph Jay Williams, University of Toronto, Toronto, Ontario, Canada, williams@cs.toronto.edu; Anastasia Kuzminykh, University of Toronto, Toronto, Ontario, Canada, anastasia.kuzminykh@ utoronto.ca.”
“Interfaces for explanations in human-AI interaction: proposing a design evaluation approach,”
Mucha, H., Robert, S., Breitschwerdt, R., & Fellmann, M. (2021) · Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411763.3451759 · Cited by 30
Explainability Auditing for Intelligent Systems: A Rationale for Multi-Disciplinary Perspectives
Langer, M., Baum, K. L., Hartmann, K., Hessel, S., Speith, T., & Wahl, J. (2021) · IEEE International Requirements Engineering Conference Workshops (REW) · doi:10.1109/rew53955.2021.00030 · Cited by 36
“…and outputs can help to improve system usability for users [9], can aid developers to increase system robustness [10], can help regulating bodies to clarify legal and ethical accountability in case of unfavorable outcomes [11], and can lead to more warranted trust in intelligent systems [7], [12].”
Explainable Recommendations in Intelligent Systems: Delivery Methods, Modalities and Risks
Naiseh, M., Jiang, N., Ma, J., & Ali, R. (2020) · Lecture Notes in Business Information Processing · doi:10.1007/978-3-030-50316-1_13 · Cited by 27
Post hoc explanations improve consumer responses to algorithmic decisions
Mourali, M., Novakowski, D., Pogacar, R., & Brigden, N. (2024) · Journal of Business Research · doi:10.1016/j.jbusres.2024.114981 · Cited by 27
Haskard, A., & Herath, D. C. (2025) · ACM Computing Surveys · doi:10.1145/3723050 · Cited by 37
What’s the Appeal? Perceptions of Review Processes for Algorithmic Decisions
Lyons, H., Wijenayake, S., Miller, T., & Velloso, E. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517606 · Cited by 27
Liu, W., & Wang, Y. (2024) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2024.2313921 · Cited by 27
“As found by Binns et al. (2018), participants believed that AI could potentially be more precise and fairer than humans because its decisions are consistently based on the same set of rules.”
Torkamaan, H., Tahaei, M., Buijsman, S., Xiao, Z., Wilkinson, D., & Knijnenburg, B. P. (2024) · Human-computer interaction series · doi:10.1007/978-3-031-55109-3_2 · Cited by 28
Politik in der digitalen Gesellschaft
Hofmann, J., Kersting, N., Ritzi, C., & Schünemann, W. J. (2019) · transcript Verlag eBooks · doi:10.14361/9783839448649 · Cited by 29
Locating the work of artificial intelligence ethics
Slota, S. C., Fleischmann, K. R., Greenberg, S. R., Verma, N., Cummings, B., Li, L., & Shenefiel, C. (2022) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.24638 · Cited by 33
“…can problematize attempts to regulate, and the “ black-boxed ” nature of algorithmic operation has led to calls for more accountable design (Binns et al., 2018; Fink, 2018; Fleischmann & Wallace, 2009; Shah, 2018) and proposals for how to audit and reverse-engineer algorithms already in…”
BackgroundTal, A. S., Kuflik, T., Kliger, D., & Mancini, A. (2024) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2024.2348843 · Cited by 29
“…et al., 2022; Wesche et al., 2022) and for investigating XAI variations and their effect on users’ fairness perceptions (Araujo et al., 2020; Binns et al., 2018; Conati et al., 2021; Dodge et al., 2019; Ebermann et al., 2023; Efendi � c et al., 2024; Kouki et al., 2020; Lee et al., 2019;…”
InfluentialRevillod, G. (2025) · AI & Society · doi:10.1007/s00146-025-02216-x · Cited by 35
“…et al. 2018; Johnson et al. 2022; Shami et al. 2015), skills development (Gross 2022; Höddinghaus et al. 2021; Xu et al. 2017), career development (Binns et al. 2018; Höddinghaus et al. 2021), or “ trans-versal processes ” (Emery and Gonin 2009: 46), where AI that can predict psychosocial risks…”
Pass/Fail Prediction in Programming Courses
Petegem, C. V., Deconinck, L., Mourisse, D., Maertens, R., Strijbol, N., Dhoedt, B., Wever, B. D., Dawyndt, P., & Mesuere, B. (2022) · Journal of Educational Computing Research · doi:10.1177/07356331221085595 · Cited by 30
“Such background information definitely does not explain everything and lowers the perceived fairness of predictions (Binns et al., 2018; Grgić-Hlača et al., 2018).”
BackgroundExplainable AI: Foundations, Applications, Opportunities for Data Management Research
Pradhan, R., Lahiri, A., Galhotra, S., & Salimi, B. (2022) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/3514221.3522564 · Cited by 25
“The development of XAI methods is motivated by technical, social and ethical objectives [9, 14, 36, 38, 44]: (1) increasing societal acceptance of machine learning (ML)-based decision-making algorithms by establishing trust in decision results, (2) providing users with actionable insights to change…”
MethodSchoeffer, J., & Kuehl, N. (2021) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/3462204.3481742 · Cited by 25
“Particularly the organizational justice literature (e.g., [9]) appears to offer a rich set of suitable constructs, which have been increasingly adopted by HCI and CSCW scholars recently [3, 26, 36].”
InfluentialAre we justified attributing a mistake in diagnosis to an AI diagnostic system?
Babushkina, D. (2022) · AI and Ethics · doi:10.1007/s43681-022-00189-x · Cited by 24
The Invisible Arms Race: Digital Trends in Illicit Goods Trafficking and AI-Enabled Responses
Mademlis, I., Mancuso, M., Paternoster, C., Evangelatos, S., Finlay, E., Hughes, J. G., Grammatikis, P. I. R., Sarigiannidis, P. G., Stavropoulos, G., Votis, K., et al. (2024) · IEEE Transactions on Technology and Society · doi:10.1109/tts.2024.3514683 · Cited by 24
“Simultaneously, reducing bias and discrimination is directly linked to the perception of justice [158].”
Recent Studies of XAI - Review
Hu, Z. F., Kuflik, T., Mocanu, I. G., Najafian, S., & Tal, A. S. (2021) · ACM Conference on User Modeling, Adaptation and Personalization (UMAP) · doi:10.1145/3450614.3463354 · Cited by 25
Hauer, M. P., Kevekordes, J., & Haeri, M. A. (2021) · Computer law & security review · doi:10.1016/j.clsr.2021.105583 · Cited by 28
“The mathematical operationalization of fairness leads to a result driven anti-discrimination law in which seemingly individual justice is preprogrammed and predetermined without many possibilities for the individual to contest this predetermination [59].”
BackgroundCounterfactual explanations as interventions in latent space
Crupi, R., Castelnovo, A., Regoli, D., & González, B. S. M. (2022) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-022-00889-2 · Cited by 23
“Binns et al. (2018) and Fernandez et al. (2020) evaluate counterfactual explanations in comparison with other XAI approaches, such as feature importance.”
Designing Visual Markers for Continuous Artificial Intelligence Support
van Berkel, N., Ahmad, O. F., Stoyanov, D. V., Lovat, L. B., & BLANDFORD, A. E. (2020) · ACM Transactions on Computing for Healthcare · doi:10.1145/3422156 · Cited by 22
“The growing use of AI across a range of application domains has led to an increased focus on usable [9, 39], fair [49, 53], and transparent [4] systems within the HCI community as well as the wider public debate [33, 50].”
BackgroundInteractive Text Generation Using GPT-2 with Gradio: An Approach to Accessible NLP Applications
Niharika, V., Nandan, D., Kandadi, R., & Singh, M. K. (2025) · IEEE International Conference on Advanced Computing Technologies (ICACT) · doi:10.1109/icact67549.2025.11351387 · Cited by 22
“This coincides with the changing emphasis on the creation of performant and explainable, usable (and co-creatively designed) AI [7][8].”
Exploring Understandable Algorithms to Suggest Fitness Tracker Goals that Foster Commitment
Woźniak, P. W., Kucharski, P., de Graaf, M. M. A., & Niess, J. (2020) · Nordic Conference on Human-Computer Interaction · doi:10.1145/3419249.3420131 · Cited by 22
“Binns et al. [6] conducted three consecutive experimental studies exploring how users assess the fairness of algorithm decisions and how explanations a � ect that perception.”
BackgroundDignity and use of algorithm in performance evaluation
Zhang, L., & Amos, C. (2023) · Behaviour and Information Technology · doi:10.1080/0144929x.2022.2164214 · Cited by 27
“They are reduced to a number and only represent a means to an end (Binns et al. 2018).”
BackgroundSwain, V. D., Gao, L., Mondal, A., Abowd, G. D., & Choudhury, M. D. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642716 · Cited by 24
James, U. U., Idika, C. N., Enyejo, L. A., Abiodun, K. E., & Enyejo, J. O. (2024) · International Journal of Scientific Research and Modern Technology. · doi:10.38124/ijsrmt.v3i12.644 · Cited by 32
SCREEN-DR: Collaborative platform for diabetic retinopathy
Pedrosa, M., Silva, J. M., Silva, J., Matos, S., & Costa, C. M. A. (2018) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2018.10.005 · Cited by 21
Short Text Classification Using Contextual Analysis
Sulaimani, S. A., & Starkey, A. J. (2021) · IEEE Access · doi:10.1109/access.2021.3125768 · Cited by 25
“It is well recognized in literature [20], [21], that despite these successes, that focus needs to be given to the availability of transparent methods to solve classification problems in general, and is especially important for detecting critical events via social media.”
BackgroundChoi, D., Lee, U., & Hong, H. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517574 · Cited by 23
“Researchers have made multiple eforts to understand how people perceive fairness in algorithmic decision making [10, 40, 47, 56, 61, 73, 83, 90, 94].”
BackgroundLyons, H., Miller, T., & Velloso, E. (2023) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3593013.3594041 · Cited by 26
“Scenarios are frequently used in social psychology, ethics, and HCI research to explore people’s views on various matters [7, 24].”
Background
Metadata
- Authors
- Binns, R., Van Kleek, M., Veale, M., Lyngs, U., Zhao, J., & Shadbolt, N.
- Affiliations
- University of Oxford (GB)
- University of Oxford, Oxford, Oxfordshire , United Kingdom
- University College London (GB)
- Venue
- Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2018-04-21
- Pages
- 1-14
- OpenAlex
- W4296978576
- Semantic Scholar
- aba609e6c5c9b58f9dac272b75983f75a7f190e5
- Cited by
- 691
- References
- 92
- Citations by year
- 2026: 139 · 2025: 204 · 2024: 87 · 2023: 60 · 2022: 76 · 2021: 53 · 2020: 35 · 2019: 31 · 2018: 6
- FWCI
- 56.17
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- Funders
- UK Engineering and Physical Sciences Research Council (EP/J017728/2)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Social Sciences › Law › Artificial Intelligence in Law
- Keywords
- algorithmic decision-making, perceived justice, algorithmic transparency, explanation styles, accountability, arbitrariness, indignity, generalization, automated decision-making, EU data protection law, lending, insurance quotes
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-06, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-06 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
