The DOI matches its Crossref record; 14 of the 14 references it cites were found in a registry or an index; 6 of the 22 sources its claims cite were checked for support.
Bias in computer systems
Open accessAbstract
From an analysis of actual cases, three categories of bias in computer systems have been developed: preexisting, technical, and emergent. Preexisting bias has its roots in social institutions, practices, and attitudes. Technical bias arises from technical constraints of considerations. Emergent bias arises in a context of use. Although others have pointed to bias inparticular computer systems and have noted the general problem, we know of no comparable work that examines this phenomenon comprehensively and which offers a framework for understanding and remedying it. We conclude by suggesting that freedom from bias should by counted amoung the select set of criteria—including reliability, accuracy, and efficiency—according to which the quality of systems in use in society should be judged.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
33 Own interpretation · 6 No full text of the sources · 4 Sources not found · 3 Supported · 2 Partly supported · 2 Own method · 2 No basis given · 1 Supported so far · 1 Not checked yet · accuracy not yet measured
Every source the paper cites for this claim was checked and supports it.
In the paper
“As specified in Section 50-(9) of the British Nationality Act itself and imple-mented in BNAP,”
§ Bias in Computer Systems > 3.3 The British Nationality Act Program (BNAP)
What the sources say
- Supportedcites Sergot et al. 1986 source
The “Formalization by Trial and Error” passage states the same distinction: a man is the father only of his legitimate children, while a woman is the mother of all her children. It also says incorporating that provision requires an adjustment to the existing rules, supporting the claim that it was implemented in the logic program. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“At this point, it was discovered that the original Admissions Algorithm placed married couples at a disadvantage in achieving an optimal placement as compared with their single peers [Roth 1984;1990].”
What the sources say
- Partly supportedcites Roth 1984 sourceExperimental — accuracy not yet measured
Section IV, “Contemporary Issues” (page not supplied), says the couples algorithm used until 1983 could not accept preferences over pairs of positions and could produce outcomes unstable with respect to couples. That supports a disadvantage for couples, but the paper does not establish that couples achieved less optimal placements than single applicants. (LLM, conf=0.84)
- Not checked — no full text of the sourcecites Roth 1990 source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“In this manner the MLSA gives processing attention to all processes as quickly as possible that are beginning a new command.”
§ Bias in Computer Systems > 3.2 A Multilevel Scheduling Algorithm (MLSA)
What the sources say
- Supportedcites Corbato et al. 1962 source
The “A Multi-Level Scheduling Algorithm” section says programs entering the system are placed in priority queues and that scheduling begins with the lowest occupied level, switching to a newly occupied lower level. The IBM 7090 system section also says the supervisor gives top priority to initiating waiting commands. Together, these passages support the paraphrase about prompt attention to processes beginning a command. (LLM, conf=0.88)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“At the time of the BNAP's initial implementation (1983) no mechanism was built into the program to incorporate relevant case law as it came into being [Sergot et al. 1986].”
§ Bias in Computer Systems > 3.3 The British Nationality Act Program (BNAP)
What the sources say
- Supportedcites Sergot et al. 1986 source
The introduction says the Act was free of case law when the original implementation began in summer 1983. It also says the authors deliberately avoided incorporating rules derived from previous cases, leaving treatment of case law for further investigation. These passages support the claim that the initial program had no built-in means of incorporating emerging case law. (LLM, conf=0.88)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The Admissions Algorithm does not take into account geographic distribution when determining a match, considering only the ranked preferences of hospitals and students.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“Although others have pointed to bias in particular computer systems and have noted the general problem, we know of no comparable work that examines this phenomenon comprehensively and which offers a framework for understanding and remedying it.”
- Also stated in: § INTRODUCTION · hedged
“Although others have pointed to bias in particular computer systems and have noted the general problem [Johnson and Mulvey 1993;Moor 1985], we know of no comparable work that focuses exclusively on this phenomenon and examines it comprehensively.”
§ Bias in Computer Systems > Abstract
What the sources say
- Not matchedcites Johnson and Mulvey 1993
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not checked — no full text of the sourcecites Moor 1985 source
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
- Also stated in: § INTRODUCTION · hedged
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“From an analysis of actual cases, three categories of bias in computer systems have been developed: preexisting, technical, and emergent.”
- Also stated in: § INTRODUCTION · plain
“From an analysis of actual computer systems, we have developed three categories: preexisting bias, technical bias, and emergent bias.”
- Also stated in: § 2. FRAMEWORK FOR ANALYZING BIAS IN COMPUTER SYSTEMS · plain
“The framework that emerged from this methodology is comprised of three overarching categories-preexisting bias, technical bias, and emergent bias.”
§ Bias in Computer Systems > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The 17 computer systems examined are representative enough to generalize the framework to computer systems generally.
- Also stated in: § INTRODUCTION · plain
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Emergent bias arises in a context of use.”
- Also stated in: § 2.3 Emergent Bias · plain
“emergent bias arises only in a context of use.”
§ Bias in Computer Systems > Abstract
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 2.3 Emergent Bias · plain
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Thus, in terms of overall service from the processor, these individuals with long-running programs are systematically disadvantaged.”
§ Bias in Computer Systems > 3.2 A Multilevel Scheduling Algorithm (MLSA)
plaindirectn = 2- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Thus the British Nationality Act is biased against the illegitimate descendants of British men.”
§ Bias in Computer Systems > 3.3 The British Nationality Act Program (BNAP)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Probably not a substantial claim. 18
- It may state two separate claims as one. 15
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“For in the same text the authors also state (as noted above) that their goal is to translate the British Nationality Act so that”
§ Bias in Computer Systems > 5. CONCLUSION
What the sources say
- Partly supportedcites Sergot et al. 1986 sourceExperimental — accuracy not yet measured
Split into 2 atoms — 1 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Partly supported
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“One charges that the Admissions Algorithm systematically favors hospital programs over medical students in cases of conflict [Graettinger and Peranson 1981b;Roth 1984;Sudarshan and Zisook 1981;Williams et al. 1981].”
§ Bias in Computer Systems > 3.1 The National Resident Match Program (NRMP)
What the sources say
- Not matchedcites Graettinger and Peranson 1981b
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Supportedcites Roth 1984 source
Section III (Theoretical Analysis; page unavailable) states that the NRMP algorithm produces the best stable outcome for hospitals and the worst for students when preferences are submitted as strict rank-order lists. Section IV (Contemporary Issues; page unavailable) also identifies whether the algorithm unfairly favors hospitals as an issue that had been raised, while noting a qualification when participants may be indifferent between matches. (LLM, conf=0.92)
- Not checked — no full text of the sourcecites Williams et al. 1981 source
hedgeddirect- Asserted: fails — The authors may not put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The authors may not put it forward as true. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Technical bias arises from technical constraints or considerations.”
- Also stated in: § 2.1 Preexisting Bias · plain
“Technical bias arises from technical constraints or technical considerations.”
§ Bias in Computer Systems > Abstract
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
- Also stated in: § 2.1 Preexisting Bias · plain
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Preexisting bias has its roots in social institutions, practices, and attitudes.”
- Also stated in: § INTRODUCTION · plain
“Preexisting bias has its roots in social institutions, practices, and attitudes.”
§ Bias in Computer Systems > Abstract
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
- Also stated in: § INTRODUCTION · plain
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In this case, we ourselves would shy away from viewing this as a bias in the system because we think this discrimination can be defended as having a reasonable basis.Namely, the discrimination reflects the preferences of match participants, and it is reasonable in our view for employment decisions to be determined largely by the choices of employers and employees.”
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The discrimination against rural hospitals reflects the preferences of match participants.
- Rests on an assumption: The best teaching hospitals tend to be in urban areas, and urban areas tend to fill their positions far more effectively, and with better students, than rural hospitals.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Emergent bias poses a potential problem for any legal expert system, especially in a society where legal systems depend on evolving case law.”
§ Bias in Computer Systems > 3.3 The British Nationality Act Program (BNAP)
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We thus infer that in this case the biased designs were unintended and, instead, reflected gender-biases deeply rooted in the culture at large.”
§ Bias in Computer Systems > 4. CONSIDERATIONS FOR MINIMIZING BIAS IN COMPUTER SYSTEM DESIGN
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As Sabre and Apollo illustrate, biases in computer systems can be difficult to identify let alone remedy because of the way the technology engages and extenuates them.Computer systems, for instance, are comparatively inexpensive to disseminate, and thus, once developed, a biased system has the potential for widespread impact.If the system becomes a standard in the field, the bias becomes pervasive”
§ Bias in Computer Systems > INTRODUCTION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Huff and Cooper report that along a number of dimensions the designs proposed by subjects in the gender-unspecified group closely resembled the designs proposed by subjects who designed for boys and were significantly different from the designs proposed by subjects who designed for girls.”
§ Bias in Computer Systems > 4. CONSIDERATIONS FOR MINIMIZING BIAS IN COMPUTER SYSTEM DESIGN
What the sources say
- Not checked — no full text of the sourcecites Huff and Cooper 1987 source
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“One source of this alleged bias lies in Sabre's and Apollo's algorithms for controlling search and display functions.In the algorithms, preference is given to”
§ Bias in Computer Systems > INTRODUCTION
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Thus, a shift in context of use may well create difficulties for a new set of users.”
§ Bias in Computer Systems > 2.2 Technical Bias
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“Thus, the computer systems systematically downgrade and, hence, are biased against international carriers who fly few, if any, internal U.S. flights, and against internal carriers who do not fly international flights [Fotos 1988;Ott 1988].”
§ Bias in Computer Systems > INTRODUCTION
What the sources say
- Not matchedcites Fotos 1988
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites Ott 1988
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This bias typically emerges some time after a design is completed, as a result of changing societal knowledge, population, or cultural values.”
§ Bias in Computer Systems > 2.2 Technical Bias
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
In the paper
“One is that the interface design compounds the bias in the reservation systems.Lists of ranked flight options are displayed screen by screen.Each screen displays only two to five options.The advantage to a carrier of having its flights shown on the first screen is enormous since 90% of the tickets booked by travel agents are booked by the first screen display [Taib 1990].”
§ Bias in Computer Systems > INTRODUCTION
What the sources say
- Not matchedcites Taib 1990
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Or consider again the case of technical bias with the MLSA (which favors users with short jobs over long jobs).While technical bias is embedded in the program, the bias is a phenomenon of the system in use, in a context wherein users with short and long jobs outpace the system's capacity.”
§ Bias in Computer Systems > 4. CONSIDERATIONS FOR MINIMIZING BIAS IN COMPUTER SYSTEM DESIGN
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“In their study, subjects were asked to propose designs for software to teach seventh graders the correct use of commas.One group of subjects was asked to design the software for seventh-grade boys, the second group to design for seventh-grade girls, and the third group to design for seventh-graders, gender unspecified.”
§ Bias in Computer Systems > 4. CONSIDERATIONS FOR MINIMIZING BIAS IN COMPUTER SYSTEM DESIGN
What the sources say
- Not checked — no full text of the sourcecites Huff and Cooper 1987 source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This type of bias can enter a system either through the explicit and conscious efforts of individuals or institutions, or implicitly and unconsciously, even in spite of the best of intentions.”
§ Bias in Computer Systems > 2.1 Preexisting Bias
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Sources of technical bias can be found in several aspects of the design process, including limitations of computer tools such as hardware, software, and peripherals; the process of ascribing social meaning to algorithms developed out of context; imperfections in pseudorandom number generation; and the attempt to make human constructs amenable to computers, when we quantify the qualitative, discret”
§ Bias in Computer Systems > 2.2 Technical Bias
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“The constraint inherent in the NRMP algorithm, in which preference is given to hospital choices when conflicts in rankings occur, duplicates what happens in an actual admissions process without a computerized matching program”
§ Bias in Computer Systems > 3.1 The National Resident Match Program (NRMP)
What the sources say
- Not matchedcites Graettinger and Peranson 1981b
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Whatever ranking algorithm is used, if it systematically places certain airlines' flights on initial screens and other airlines' flights on later screens, the system will exhibit technical bias.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“They can also reflect the personal biases of individuals who have significant input into the design of the system, such as the client or system designer.”
§ Bias in Computer Systems > 2.1 Preexisting Bias
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“those responsible for maintaining the system responded conscientiously to this societal change and modified the system's algorithm to place couples more fairly [Roth 1990].”
§ Bias in Computer Systems > 4. CONSIDERATIONS FOR MINIMIZING BIAS IN COMPUTER SYSTEM DESIGN
What the sources say
- Not checked — no full text of the sourcecites Roth 1990 source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Preexisting biases may originate in society at large, in subcultures, and in formal or informal, private or public organizations and institutions.”
§ Bias in Computer Systems > 2.1 Preexisting Bias
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
In the paper
“Should the accumulation of case law lead to changes in the way the Act is interpreted-say, by granting new subgroups British citizenship-BNAP would systematically misinform members of this subgroup regarding their status as British citizens [Leith 1986].”
§ Bias in Computer Systems > 3.3 The British Nationality Act Program (BNAP)
What the sources say
- Not checked — no full text of the sourcecites Leith 1986 source
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“In total, we examined 17 computer systems from diverse fields including banking, commerce, computer science, education, medicine, and law.”
§ Bias in Computer Systems > 2. FRAMEWORK FOR ANALYZING BIAS IN COMPUTER SYSTEMS
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We derived our framework by examining actual computer systems for bias.Instances of bias were identified and characterized according to their source, and then the characterizations were generalized to more abstract categories.These categories were further refined by their application to other instances of bias in the same or additional computer systems.”
§ Bias in Computer Systems > 2. FRAMEWORK FOR ANALYZING BIAS IN COMPUTER SYSTEMS
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“although we have talked about bias in computer systems, the presence of bias is not so much a feature inherent in the system independent of the context of use, but an aspect of a system in use.”
§ Bias in Computer Systems > 4. CONSIDERATIONS FOR MINIMIZING BIAS IN COMPUTER SYSTEM DESIGN
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
The paper relies on this without support.
In cases of conflict, there is no good rationale for favoring hospital programs at the expense of medical students.
“We are inclined to answer yes because in cases of conflict there does not appear to be a good rationale for favoring hospital programs at the expense of students.”
The paper relies on this without support.
The discrimination against rural hospitals reflects the preferences of match participants.
“Namely, the discrimination reflects the preferences of match participants, and it is reasonable in our view for employment decisions to be determined largely by the choices of employers and employees.”
The paper relies on this without support.
The 17 computer systems examined are representative enough to generalize the framework to computer systems generally.
“In total, we examined 17 computer systems from diverse fields including banking, commerce, computer science, education, medicine, and law.”
The paper relies on this without support.
The illustrative examples in Table I portray plausible cases of bias.
“The illustrative examples portray plausible cases of bias.”
The paper relies on this without support.
The best teaching hospitals tend to be in urban areas, and urban areas tend to fill their positions far more effectively, and with better students, than rural hospitals.
“Because the best teaching hospitals tend to be in urban areas, the urban areas tend to fill their positions far more effectively, and with better students, than rural hospitals.”
The authors state this as an assumption.
The MLSA system is not saturated with too many users.
“assuming the system is not saturated with too many users”
The paper relies on this without support.
BNAP was designed in a research environment among people with sophisticated knowledge of immigration law.
“First, the system was designed in a research environment, among people with sophisticated knowledge of immigration law.”
The paper relies on this without support.
BNAP's users are likely to be at best paralegal or immigration counselors in Britain, if not lay persons in foreign countries with only limited access to British legal expertise.
“Its users, however, are likely to be at best paralegal or immigration counselors in Britain, if not lay persons in foreign countries with only limited access to British legal expertise.”
The paper relies on this without support.
Some of BNAP's queries require expert knowledge to answer.
“Some of the program's queries, for example, require expert knowledge to answer.”
The paper relies on this without support.
Nonexperts advising British citizen hopefuls are not alerted to alternative legal frameworks that complement the British Nationality Act.
“More generally, nonexperts advising British citizen hopefuls are not alerted to alternative legal frameworks that complement the British Nationality Act.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The multi-metric approach (Figure S1) makes explicit potential trade-offs and helps ensure the nonaccuracy desiderata are not treated as second-class citizens to accuracy (see Ref. 13).”
From Holistic Evaluation of Language Models (2023) · cites this paper as [13]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“research has repeatedly shown that these systems can in fact encode biases into decisions 124 .”
From Administrative law and the machines of government: judicial review of automated public-sector decision-making (2019) · cites this paper as B Friedman and H Nissenbaum 'Bias in Computer Systems' (1996)
The cited source says what the claim attributes to it.
In the citing paper
“The risk of”
From Positioning Paradata: A Conceptual Frame for AI Processual Documentation in Archives and Recordkeeping Contexts (2023) · cites this paper as [25]
The cited source says what the claim attributes to it.
In the citing paper
“Bias or opacity are far from newly observed characteristics of computer systems generally [26, 11, 24], but these issues have been exacerbated by the rapidly expanding numbers of data-driven information systems entering decision-making processes.”
From Fairness and Accountability Design Needs for Algorithmic Support in High-Stakes Public Sector Decision-Making (2018) · cites this paper as [26]
In the citing paper
“These decisions may be based on factors of ethical importance (e.g., ethnic, gender, or religious grounds) and irrelevant to the decision-making at hand, or they may be relevant but legally protected as a nondiscriminatory characteristic (Friedman and Nissenbaum 1996).”
From How to Design AI for Social Good: Seven Essential Factors (2020) · cites this paper as Friedman and Nissenbaum 1996
In the citing paper
“On this account, the apparent problem to which data science is a solution determines whether it happens to serve morally defensible ends.”
From Problem Formulation and Fairness (2019) · cites this paper as [17]
In the citing paper
“Corresponding with [14] this outcome systematically disadvantages individuals of one gender by preferring the other at the topk positions.”
From FA*IR: A Fair Top-k Ranking Algorithm (2017) · cites this paper as [14]
In the citing paper
“While technical factors, such as biased data and design choices, play an important role, biases can also arise from the context of use, for example, how human users interpret system outputs or from a mismatch between the capabilities and values assumed in the design of the system and those of its actual users (Danks & London, 2017; Friedman & Nissenbaum, 1996).”
From Digital ageism: challenges and opportunities in artificial intelligence for older adults (2022) · cites this paper as Friedman & Nissenbaum, 1996
In the citing paper
“and to data protection, privacy, and other human rights issues 11”
From Administrative Law and the Machines of Government: Judicial Review of Automated Public-Sector Decision-Making (2018) · cites this paper as [11]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Bias in Computer Systems > 3.1 The National Resident Match Program (NRMP)
“Table II.The Simplest Case of Rank-Order Lists in which the Desires of Students and Programs are in Conflict (Reprinted from Williams et al. [1991], by permission of the New England Journal of Medicine)”
§ Bias in Computer Systems > References
“18. WILLIAMS, K. J., WERTH, V. P., AND WOLFF, J. A. 1981. An analysis of the resident match.”
Citations
Cites
The evolution of the labor market for medical interns and residents: A case study in game theory
Roth, A. E. (1984) · Journal of Political Economy · doi:10.1086/261272 · Cited by 1,219
MOOR, J. H. (1985) · Metaphilosophy · doi:10.1111/j.1467-9973.1985.tb00173.x · Cited by 863
The British Nationality Act as a logic program
Sergot, M. J., Sadri, F., Kowalski, R. A., Kriwaczek, F., Hammond, P., & Cory, H. T. (1986) · Communications of the ACM · doi:10.1145/5689.5920 · Cited by 564
An experimental time-sharing system
Corbató, F. J., Merwin-Daggett, M., & Daley, R. C. (1962) · Proceedings of the May 1-3, 1962, spring joint computer conference on - AIEE-IRE '62 (Spring) · doi:10.1145/1460833.1460871 · Cited by 234
New physicians: A natural experiment in market organization
Roth, A. E. (1990) · Science · doi:10.1126/science.2274783 · Cited by 141
Sex Bias in Educational Software: The Effect of Designers' Stereotypes on the Software They Design1
Huff, C., & Cooper, J. (1987) · Journal of Applied Social Psychology · doi:10.1111/j.1559-1816.1987.tb00328.x · Cited by 85
Fundamental Errors in Legal Logic Programming
Leith, P. (1986) · The Computer Journal · doi:10.1093/comjnl/29.6.545 · Cited by 51
Moor, J. H. (2017) · Routledge eBooks · doi:10.4324/9781315259697-7 · Cited by 40
Moor, J. H. (2020) · Routledge eBooks · doi:10.4324/9781003075011-1 · Cited by 21
National resident matching program
(1981) · New England Journal of Medicine · doi:10.1056/nejm198108273050923 · Cited by 10
Research works printed but not linked: 4
- FISHLOCK , T. 1981. Delhi press detect racism in Nationality Bill. The London Times ( Jan. 20 ) . FISHLOCK, T. 1981. Delhi press detect racism in Nationality Bill. The London Times (Jan. 20).
- GAO . 1992. Patriot Missile defense: Software problem led to system failure at Dhahran , Saudi Arabia. GAO/IMTEC-92-26, U.S. General Accounting Office, Washington, D.C. GAO. 1992. Patriot Missile defense: Software problem led to system failure at Dhahran, Saudi Arabia. GAO/IMTEC-92-26, U.S. General Accounting Office, Washington, D.C.
- JOHNSON , D.G. AND MULVEY , J. M. 1993. Computer decisions: Ethical issues of responsibil-ity and bias. Statistics and Operations Res. Series SOR-93-11 , Dept. of Civil Engineering and Operations Research , Princeton Univ ., Princeton, N.J. JOHNSON,D.G.AND MULVEY, J. M. 1993. Computer decisions: Ethical issues of responsibil-ity and bias. Statistics and Operations Res. Series SOR-93-11, Dept. of Civil Engineering and Operations Research, Princeton Univ., Princeton, N.J.
- OTT , J. 1988. American Airlines settles CRS dispute with British Airways. Aviat. Week Space Tech. ( July 18 ) . OTT, J. 1988. American Airlines settles CRS dispute with British Airways. Aviat. Week Space Tech. (July 18).
Cited by
Sorted by citations
A Survey on Bias and Fairness in Machine Learning
Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., & Galstyan, A. (2022) · ACM Computing Surveys · doi:10.1145/3457607 · Cited by 5,812
The ethics of algorithms: Mapping the debate
Mittelstadt, B. D., Allo, P., Taddeo, M., Wachter, S., & Floridi, L. (2016) · Big Data & Society · doi:10.1177/2053951716679679 · Cited by 4,567
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
Value Sensitive Design and Information Systems
Friedman, B., Kahn, P. H., & Borning, A. H. (2008) · Wiley eBooks · doi:10.1002/9780470281819.ch4 · Cited by 1,149
Self-citationFriedman, B. (1996) · Interactions · doi:10.1145/242485.242493 · Cited by 966
Self-citationBender, E. M., & Friedman, B. (2018) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00041 · Cited by 854
“Friedman and Nissenbaum (1996) show that in some cases, system bias reflects biases in society; these are pre-existing biases with roots in social institutions, practices and attitudes.”
BackgroundMethodSelf-citationBaker, R. S., & Hawn, A. (2022) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-021-00285-9 · Cited by 993
“Landmark early work by Friedman and Nissenbaum (1996) set a precedent for this sequential framing of bias, categorizing biases based on the context where they arise, whether from preexisting social biases, technical limitations, or use of the models by real users.”
BackgroundImproving Fairness in Machine Learning Systems: What Do Industry Practitioners Need?
Holstein, K., Wortman Vaughan, J., Daumé, H., Dudik, M., & Wallach, H. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300830 · Cited by 777
“The design, prototyping, and maintenance of machine learning systems raises many unique challenges [27, 63, 78, 87] not commonly faced with other kinds of intelligent systems or computing systems more broadly [34, 66, 67].”
BackgroundShneiderman, B. (2020) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3419764 · Cited by 768
“An early review by Friedman and Nissenbaum [39] described three kind of bias, which remain as a helpful guide: preexisting biases based on social practices and attitudes, technical bias based on design constraints in hardware and software, and emergent bias that arises from changing the use context.”
BackgroundSocial Data: Biases, Methodological Pitfalls, and Ethical Boundaries
Olteanu, A., Castillo, C., Diaz, F., & Kıcıman, E. (2019) · Frontiers in Big Data · doi:10.3389/fdata.2019.00013 · Cited by 763
“Broadly discussing bias in computer systems, Friedman and Nissenbaum (1996) characterize it according to its source such as societal, technical, or usage related.”
BackgroundEthical Implications and Accountability of Algorithms
Martin, K. (2019) · Journal of Business Ethics · doi:10.1007/s10551-018-3921-3 · Cited by 757
“7 These technologies survive to have biases that are valueladen (Friedman and Nissenbaum 1996; Johnson 2004) or have politics (Winner 1980).”
BackgroundAlgorithmic Accountability: Journalistic investigation of computational power structures
Diakopoulos, N. (2015) · Digital Journalism · doi:10.1080/21670811.2014.976411 · Cited by 733
“Researchers have been aware of and have critiqued the biases embedded in computational systems for many years (Friedman and Nissenbaum 1996).”
BackgroundValue Sensitive Design and Information Systems
Friedman, B., Kahn, P. H., Borning, A., & Huldtgren, A. (2013) · Philosophy of Engineering and Technology · doi:10.1007/978-94-007-7844-3_4 · Cited by 678
“bias (Friedman and Nissenbaum, 1996), universal usability (Shneiderman, 1999, 2000;Thomas, 1997),autonomy (Suchman,1994;Winograd, 1994), informedconsent (Millett et al.”
BackgroundSelf-citationBias in algorithmic filtering and personalization
Bozdag, E. (2013) · Ethics and Information Technology · doi:10.1007/s10676-013-9321-6 · Cited by 649
“Just as any computer system, they can unfairly discriminate against certain individuals or groups of individuals in favor of others (Friedman and Nissenbaum 1996).”
BackgroundInfluentialBias and Debias in Recommender System: A Survey and Future Directions
Chen, J., Dong, H., Wang, X., Feng, F., Wang, M., & He, X. (2023) · ACM Transactions on Information Systems · doi:10.1145/3564284 · Cited by 741
“uired traits” [50], and the unfairness can be defined as follow: Unfairness. The system systematically and unfairly discriminates against certain individuals or groups of individuals in favor others [51]. Unfairness issue has been an obstacle to making recommender systems more entrenched within our society. In particul…”
BackgroundKuziemski, M., & Misuraca, G. (2020) · Telecommunications Policy · doi:10.1016/j.telpol.2020.101976 · Cited by 610
Where fairness fails: data, algorithms, and the limits of antidiscrimination discourse
Hoffmann, A. L. (2019) · Information, Communication & Society · doi:10.1080/1369118x.2019.1573912 · Cited by 535
“In these cases, biases are said to ‘ sneak in ’ (University of Bath, 2017), ‘ whether on purpose or by accident ’ (Barocas & Selbst, 2016, p. 674), or in ways that only emerge over time (Friedman & Nissenbaum, 1996).”
Background“Participant” Perceptions of Twitter Research Ethics
Fiesler, C. L., & Proferes, N. (2018) · Social Media + Society · doi:10.1177/2056305118763366 · Cited by 545
“However, both of these beliefs may be misguided, as re-identification is possible in large data sets (Zimmer, 2010a), and computer algorithms can be biased in both design and application (Friedman & Nissenbaum, 1996).”
BackgroundA Framework of Potential Sources of Harm Throughout the Machine Learning Life Cycle
Suresh, H., & Guttag, J. V. (2021) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 576
Algorithmic Transparency in the News Media
Diakopoulos, N., & Koliska, M. (2017) · Digital Journalism · doi:10.1080/21670811.2016.1208053 · Cited by 515
“One of the main concerns regarding computational systems is the existence of embedded biases, which researchers have long criticized (Friedman and Nissenbaum 1996).”
BackgroundEthical and Bias Considerations in Artificial Intelligence/Machine Learning
Hanna, M. G., Pantanowitz, L., Jackson, B., Palmer, O., Visweswaran, S., Pantanowitz, J., Deebajah, M., & Rashidi, H. H. (2025) · Modern Pathology · doi:10.1016/j.modpat.2024.100686 · Cited by 535
Unequal Representation and Gender Stereotypes in Image Search Results for Occupations
Kay, M., Matuszek, C., & Munson, S. A. (2015) · Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems · doi:10.1145/2702123.2702520 · Cited by 509
“Both algorithms and interfaces, however, can be biased in how they represent the world [9,34].”
BackgroundKöchling, A., & Wehner, M. C. (2020) · Business Research · doi:10.1007/s40685-020-00134-w · Cited by 511
FA*IR: A Fair Top-k Ranking Algorithm
Zehlike, M., Bonchi, F., Castillo, C., Hajian, S., Megahed, M., & Baeza-Yates, R. (2017) · Proceedings of the 2017 ACM on Conference on Information and Knowledge Management · doi:10.1145/3132847.3132938 · Cited by 496
“Corresponding with [14] this outcome systematically disadvantages individuals of one gender by preferring the other at the top-k positions.”
BackgroundHolistic Evaluation of Language Models
Bommasani, R., Liang, P., & Lee, T. (2023) · Annals of the New York Academy of Sciences · doi:10.1111/nyas.15007 · Cited by 480
Veale, M., Van Kleek, M., & Binns, R. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174014 · Cited by 490
Algorithmic Accountability and Public Reason
Binns, R. (2018) · Philosophy & Technology · doi:10.1007/s13347-017-0263-5 · Cited by 457
“In the case of knowledge-based systems, the knowledge that is fed into the system, and assumptions that are involved in modelling it, may reflect biases of the system designers and data collection process (Wiener, 1960; Weizenbaum, 1972; Dutton & Kraemer, 1980; Friedman & Nissenbaum, 1996).”
BackgroundMadaio, M. A., Stark, L., Wortman Vaughan, J., & Wallach, H. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376445 · Cited by 436
“Although HCI researchers have developed methods for designing systems with human values in mind (e.g., [38,39,91,93]), they have not yet become best practices or even propagated into professional training for UX researchers.”
BackgroundMethodHow to Design AI for Social Good: Seven Essential Factors
Floridi, L., Cowls, J., King, T. C., & Taddeo, M. (2020) · Science and Engineering Ethics · doi:10.1007/s11948-020-00213-5 · Cited by 430
“These decisions may be based on factors of ethical importance (e.g., ethnic, gender, or religious grounds) and irrelevant to the decision-making at hand, or they may be relevant but legally protected as a nondiscriminatory characteristic (Friedman and Nissenbaum 1996).”
BackgroundTransparency in Algorithmic and Human Decision-Making: Is There a Double Standard?
Zerilli, J., Knott, A., Maclaurin, J., & Gavaghan, C. (2019) · Philosophy & Technology · doi:10.1007/s13347-018-0330-6 · Cited by 420
“For example, what Friedman and Nissenbaum (1996) call B technical ^ bias arises from the inherent constraints imposed by the technology itself.”
BackgroundGreene, D., Hoffmann, A. L., & Stark, L. (2019) · Proceedings of the Annual Hawaii International Conference on System Sciences · doi:10.24251/hicss.2019.258 · Cited by 386
“longstanding work in computing ethics [11], [36], [37].”
BackgroundA Survey on the Fairness of Recommender Systems
Wang, Y., Ma, W., Zhang, M., Liu, Y., & Ma, S. (2022) · ACM Transactions on Information Systems · doi:10.1145/3547333 · Cited by 377
“Although fairness has been studied in computer science for decades [27], and there is a lot of related work in machine learning [100, 101], fairness in recommendation has its unique problems.”
BackgroundFairness-Aware Ranking in Search & Recommendation Systems with Application to LinkedIn Talent Search
Geyik, S. C., Ambler, S., & Kenthapadi, K. (2019) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3292500.3330691 · Cited by 428
“, see [19, 22, 43] and the references therein).”
Value Sensitive Design: Theory and Methods
Friedman, B., Kahn, P. H., & Borning, A. H. (2002) · Cited by 375
Self-citationEmbodying Values in Technology: Theory and Practice
Flanagan, M., Howe, D. C., & Nissenbaum, H. F. (2008) · Cambridge University Press eBooks · doi:10.1017/cbo9780511498725.017 · Cited by 346
Self-citationRakova, B., Yang, J., Cramer, H., & Chowdhury, R. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3449081 · Cited by 329
“While the academic discussion of algorithmic bias has an over 20-year long history [12], we have now reached a transitional phase in which this debate has taken a practical turn.”
BackgroundOut of the laboratory and into the classroom: the future of artificial intelligence in education
Schiff, D. (2021) · AI & SOCIETY · doi:10.1007/s00146-020-01033-8 · Cited by 334
“In contrast, an emerging fear in artificial intelligence is that of algorithmic bias, which may favor or disfavor certain subgroups by gender, sex, race, ethnicity, socioeconomic grouping, and other more subtle factors (Friedman and Nissenbaum 1996; Howard and Borenstein 2017; Bolukbasi et al. 2016; Buolamwini and Geb…”
BackgroundValues as lived experience: evolving value sensitive design in support of value discovery
Dantec, C. A. L., Poole, E., & Wyche, S. (2009) · International Conference on Human Factors in Computing Systems · doi:10.1145/1518701.1518875 · Cited by 386
“During the past decade, the HCI community has become increasingly interested in accounting for human values in the design of computer systems [3, 16, 18, 19, 23, 40].”
BackgroundOn the apparent conflict between individual and group fairness
Binns, R. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372864 · Cited by 312
“Research on bias, fairness and discrimination in socio-technical systems has a long history which significantly predates specific work on fairness in machine learning (see e.g. [23, 24]).”
BackgroundKopalle, P. K., Gangwar, M., Kaplan, A., Ramachandran, D., Reinartz, W., & Rindfleisch, A. (2021) · International Journal of Research in Marketing · doi:10.1016/j.ijresmar.2021.11.002 · Cited by 361
Digital ageism: challenges and opportunities in artificial intelligence for older adults
Chu, C. H., Nyrup, R., Leslie, K., Shi, J., Bianchi, A., Lyn, A., McNicholl, M., Khan, S., Rahimi, S., & Grenier, A. (2022) · The Gerontologist · doi:10.1093/geront/gnab167 · Cited by 290
“A system discriminates unfairly if it denies an opportunity or a good or if it assigns an undesirable outcome to an individual or group of individuals on grounds that are unreasonable or inappropriate" (Friedman & Nissenbaum, 1996, p. 332).”
DeVito, M. A. (2016) · Digital Journalism · doi:10.1080/21670811.2016.1178592 · Cited by 295
Human Values, Ethics, and Design
Batya Friedman and Peter H. Kahn, Jr. (2002) · CRC Press eBooks · doi:10.1201/9781410606723-48 · Cited by 349
A Qualitative Exploration of Perceptions of Algorithmic Fairness
Woodruff, A., Fox, S. E., Rousso-Schindler, S., & Warshaw, J. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174230 · Cited by 256
Should we welcome robot teachers?
SHARKEY, A. J. C. (2016) · Ethics and Information Technology · doi:10.1007/s10676-016-9387-z · Cited by 277
“Forms of technological bias were already being discussed nearly two decades ago (Friedman and Nissenbaum 1996).”
BackgroundSociotechnical Harms of Algorithmic Systems: Scoping a Taxonomy for Harm Reduction
Shelby, R., Rismani, S., Henne, K., Moon, A., Rostamzadeh, N., Nicholas, P., Yilla-Akbari, N., Gallegos, J., Smart, A., Garcia, E., et al. (2023) · Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3600211.3604673 · Cited by 279
“Computing research has traced how marginalized communities — referring to communities that face structural forms of social exclusion [130] — disproportionately experience sociotechnical harms from algorithmic systems [90, 96].”
BackgroundPerceived risk and the consumer buying process: internet airline reservations
Cunningham, L. F., Gerlach, J. H., Harper, M. D., & Young, C. E. (2005) · International Journal of Service Industry Management · doi:10.1108/09564230510614004 · Cited by 246
“Initial research focused on bias in reservation systems (Friedman and Nussenbaum, 1996) that favored certain carriers and the effectiveness of the electronic marketplaces to increase price competition and reduce seller monopoly power (Bakos, 1997).”
BackgroundMapping value sensitive design onto AI for social good principles
Umbrello, S., & van de Poel, I. (2021) · AI and Ethics · doi:10.1007/s43681-021-00038-3 · Cited by 251
Values Levers: Building Ethics into Design
Shilton, K. (2013) · Science, Technology, & Human Values · doi:10.1177/0162243912436985 · Cited by 240
Algorithmic bias in machine learning-based marketing models
Akter, S., Dwivedi, Y. K., Sajib, S., Biswas, K., Bandara, R. J., & Michael, K. (2022) · Journal of Business Research · doi:10.1016/j.jbusres.2022.01.083 · Cited by 242
Problem Formulation and Fairness
Passi, S., & Barocas, S. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287567 · Cited by 283
“into the system during its development and implementation [17].”
Backgroundvan Giffen, B., Herhausen, D., & Fahse, T. (2022) · Journal of Business Research · doi:10.1016/j.jbusres.2022.01.076 · Cited by 230
“First, a monitoring plan accounts for changes in the algorithm when the context evolves (Corbett-Davies & Goel, 2018; Friedman & Nissenbaum, 1996).”
BackgroundMethodFairness in recommender systems: research landscape and future directions
Deldjoo, Y., Jannach, D., Bellogín, A., Difonzo, A., & Zanzonelli, D. (2023) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-023-09364-z · Cited by 290
“In Friedman and Nissenbaum (1996), for example, bias is used to “ refer to computer systems that systematically and unfairly discriminate against certain individuals or groups of individuals in favor of others ”.”
Passi, S., & Jackson, S. J. (2018) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3274405 · Cited by 225
“professional values, but also from the emergent sites of algorithmic re-use as opposed to their contexts of development [29,43,51].”
BackgroundHouston, L., Jackson, S. J., Rosner, D. K., Ahmed, S. I., Young, M., & Kang, L. (2016) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858470 · Cited by 235
“In the early 1990s, Friedman, Nissenbaum and others opened calls for value-based approaches that were oriented to producing technological systems that were “good” to use in a moral as well as a functional sense [18].”
BackgroundRoles for computing in social change
Abebe, R., Barocas, S., Kleinberg, J. M., Levy, K. E. C., Raghavan, M., & Robinson, D. G. (2020) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3351095.3372871 · Cited by 341
“Some values may be harder to incorporate into formal models than others, perhaps because they are difficult to quantify [43].”
BackgroundA clarification of the nuances in the fairness metrics landscape
Castelnovo, A., Crupi, R., Greco, G., Regoli, D., Penco, I. G., & Cosentini, A. (2022) · Scientific Reports · doi:10.1038/s41598-022-07939-1 · Cited by 292
“It is out of the scope of this manuscript to list and discuss all the possible shades of biases that can hide in the data (Friedman and Nissenbaum, 1996; Crawford, 2013; Mehrabi et al., 2021; BaezaYates, 2018; Hardt, 2014), we want nevertheless to say something about the main cases that can be…”
BackgroundMachine Learning for Chemical Reactivity: The Importance of Failed Experiments
Strieth‐Kalthoff, F., Sandfort, F., Kühnemund, M., Schäfer, F. R., Kuchen, H., & Glorius, F. (2022) · Angewandte Chemie International Edition · doi:10.1002/anie.202204647 · Cited by 236
PI3K and MEK inhibitor combinations: examining the evidence in selected tumor types
Britten, C. D. (2013) · Cancer Chemotherapy and Pharmacology · doi:10.1007/s00280-013-2121-1 · Cited by 215
Roselli, D., Matthews, J., & Talagala, N. (2019) · Companion Proceedings of The 2019 World Wide Web Conference · doi:10.1145/3308560.3317590 · Cited by 237
“Other researchers have proposed taxonomies of bias that serve as an excellent starting point towards managing that bias [38, 39, 40] Cramer et.”
BackgroundDemographic Bias in Biometrics: A Survey on an Emerging Challenge
Drozdowski, P., Rathgeb, C., Dantcheva, A., Damer, N., & Busch, C. (2020) · IEEE Transactions on Technology and Society · doi:10.1109/tts.2020.2992344 · Cited by 263
“A good discussion of the topic of bias was provided by Danks and London [19], as well as Friedman and Nissenbaum [20], both of which explored various sources and types of bias in the context of computer systems.”
Lee, M. K., Kusbit, D., Kahng, A., Kim, J. T., Yuan, X., Chan, A., See, D., Noothigattu, R., Lee, S., Psomas, A., et al. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359283 · Cited by 260
“Fairness is an important value in governing algorithms as algorithms can perpetuate unfair treatment of different populations or stakeholders [27, 35, 85].”
BackgroundWillson, M. (2016) · Information Communication & Society · doi:10.1080/1369118x.2016.1200645 · Cited by 205
Motivations of contributors to Wikipedia
Kuznetsov, S. (2006) · ACM SIGCAS Computers and Society · doi:10.1145/1215942.1215943 · Cited by 205
Zajko, M. (2022) · Sociology Compass · doi:10.1111/soc4.12962 · Cited by 268
Bias of AI-generated content: an examination of news produced by large language models
Fang, X., Che, S., Mao, M., Zhang, H., Zhao, M., & Zhao, X. (2024) · Scientific Reports · doi:10.1038/s41598-024-55686-2 · Cited by 194
“At the sentence level, the bias of AIGC is evaluated as the difference between sentences related to each investigated population group in AIGC and their counterparts in the news articles produced by the selected news agencies, in terms of their expressed sentiments and toxicities.”
BackgroundExclusion by design: intersections of social, digital and data exclusion
Park, S., & Humphry, J. (2019) · Information Communication & Society · doi:10.1080/1369118x.2019.1606266 · Cited by 227
“Our approach adopts the view established in Science and Technology Studies (STS) that technologies are embedded within larger social systems and processes, inscribed with the rules, values and interests of typically dominant groups (Bijker & Law, 1992; Friedman & Nissenbaum, 1996).”
MethodDeep reinforcement learning in recommender systems: A survey and new perspectives
Chen, X., Yao, L., McAuley, J., Zhou, G., & Wang, X. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.110335 · Cited by 288
Democratizing Algorithmic Fairness
Wong, P.-H. (2020) · Philosophy & Technology · doi:10.1007/s13347-019-00355-w · Cited by 182
“” (Friedman & Nissenbaum 1996, 332), and the recognition of bias in computer systems has inspired numerous approaches to detect, scrutinize and prevent biases in computer systems.”
BackgroundMiller, J. K., Friedman, B., & Jancke, G. (2007) · doi:10.1145/1316624.1316668 · Cited by 181
“, informed consent, fairness, security, trust, privacy) with respect to technology [5, 6, 13, 16].”
BackgroundSelf-citationTechnologies of memory: Key issues and critical perspectives
House, N. V., & Churchill, E. F. (2008) · Memory Studies · doi:10.1177/1750698008093795 · Cited by 180
“Like other work (Friedman and Nissenbaum, 1996) showing how power relations and values are instantiated in technologies, we see that explicit and tacit models of social and personal memory are ‘baked into ‘ the design of these technologies.”
BackgroundThe combine will tell the truth: On precision agriculture and algorithmic rationality
Miles, C. (2019) · Big Data & Society · doi:10.1177/2053951719849444 · Cited by 189
Gender Bias in Machine Translation
Savoldi, B., Gaido, M., Bentivogli, L., Negri, M., & Turchi, M. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00401 · Cited by 229
“Hereby, drawing on the human-centered approach of Value Sensitive Design (Friedman and Hendry, 2019), we consider as biased an MT model that systematically and unfairly discriminates against certain individuals or groups in favour of others (Friedman and Nissenbaum, 1996).”
MethodGender bias perpetuation and mitigation in AI technologies: challenges and opportunities
O’Connor, S., & Liu, H. K. (2023) · AI & Society · doi:10.1007/s00146-023-01675-4 · Cited by 216
Designing Fair Ranking Schemes
Asudeh, A., Jagadish, H. V., Stoyanovich, J., & Das, G. (2019) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/3299869.3300079 · Cited by 159
“Interpreting the definition of Friedman and Nissenbaum [21] for rankings, a biased outcome occurs when a ranking decision is based fully or partially on a protected feature.”
BackgroundHow to Design AI for Social Good: Seven Essential Factors
Floridi, L., Cowls, J., King, T. C., & Taddeo, M. (2021) · Philosophical studies series · doi:10.1007/978-3-030-81907-1_9 · Cited by 181
Reducing disparate exposure in ranking: A learning to rank approach
Zehlike, M., & Castillo, C. (2020) · Proceedings of The Web Conference 2020 · doi:10.1145/3366424.3380048 · Cited by 159
“Researchers have become increasingly concerned with various systematic biases [10] against sociallysalient groups, caused by historic and current discriminatory patternsmaking their way into data-drivenmodels.”
BackgroundDecentering technology in discourse on discrimination
Gangadharan, S. P., & Niklas, J. (2019) · Information Communication & Society · doi:10.1080/1369118x.2019.1593484 · Cited by 180
Critiquing the Reasons for Making Artificial Moral Agents
van Wynsberghe, A., & Robbins, S. (2019) · Science and Engineering Ethics · doi:10.1007/s11948-018-0030-8 · Cited by 161
Knobel, C. P., & Bowker, G. C. (2011) · Communications of the ACM · doi:10.1145/1965724.1965735 · Cited by 158
Shen, H., DeVos, A., Eslami, M., & Holstein, K. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3479577 · Cited by 171
Fairness-Aware Tensor-Based Recommendation
Zhu, Z., Hu, X., & Caverlee, J. (2018) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3269206.3271795 · Cited by 156
“Friedman [7] defined that a computer system is biased “if it systematically and unfairly discriminates against certain individuals or groups of individuals in favor of others.”
BackgroundDiscrimination, artificial intelligence, and algorithmic decision-making
Borgesius, F. Z. (2025) · arXiv · doi:10.48550/arxiv.2510.13465 · Cited by 152
“See also Bryson 2017; Friedman and Nissenbaum 1996; Hacker 2018, p. 1149; Kim 2017, p. 884; Vetzo, Gerards, and Nehmelman 2018, p. 145.”
Machine morality: bottom-up and top-down approaches for modelling human moral faculties
Wallach, W., Allen, C., & Smit, I. (2008) · AI & SOCIETY · doi:10.1007/s00146-007-0099-0 · Cited by 152
“Generally these are the explicit values of the institutions who use the systems and the engineers, who design the systems, although many systems also substantiate often unrecognized implicit values (Friedman and Nissenbaum 1996).”
BackgroundFlanagan, M., Howe, D. C., & Nissenbaum, H. F. (2005) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1054972.1055076 · Cited by 149
“Research has focused, for example, on the value of privacy [1,2], community [13], freedom from bias [23], universal usability [46], autonomy [48], informed consent [19], and trust [16,22,38].”
BackgroundSelf-citationManning, L., Brewer, S., Craigon, P. J., Frey, J. G., Gutiérrez, A., Jacobs, N. J., Kanza, S., Munday, S., Sacks, J., & Pearson, S. (2022) · Trends in Food Science & Technology · doi:10.1016/j.tifs.2022.04.025 · Cited by 161
Ethical and legal challenges of AI in marketing: an exploration of solutions
Kumar, D., & Suthar, N. (2024) · Journal of Information, Communication and Ethics in Society · doi:10.1108/jices-05-2023-0068 · Cited by 180
Ethical Framework for Harnessing the Power of AI in Healthcare and Beyond
Nasir, S., Khan, R. A., & Bai, S. (2024) · IEEE Access · doi:10.1109/access.2024.3369912 · Cited by 152
“For example, as societal values evolve, the preferences or priorities of patients and healthcare providers may change, leading to biases in algorithmic decision-making or recommendations [79].”
BackgroundIntegrating ethics in design through the value-sensitive design approach
Cummings, M. L. (2006) · Science and Engineering Ethics · doi:10.1007/s11948-006-0065-0 · Cited by 144
AI-driven healthcare: A review on ensuring fairness and mitigating bias
Chinta, S. V., Wang, Z., Palikhe, A., Zhang, X., Kashif, A., Smith, M. A., Liu, J., & Zhang, W. (2025) · PLOS Digital Health · doi:10.1371/journal.pdig.0000864 · Cited by 268
“Implicit bias , on the other hand, often stems from unintentional or unconscious decisions that lead to skewed datasets, frequently influenced by systemic inequities [68].”
InfluentialA Comprehensive Study on Face Recognition Biases Beyond Demographics
Terhörst, P., Kolf, J. N., Huber, M., Kirchbuchner, F., Damer, N., Morales, A., Fiérrez, J., & Kuijper, A. (2021) · IEEE Transactions on Technology and Society · doi:10.1109/tts.2021.3111823 · Cited by 167
Informed consent in the Mozilla browser: implementing value-sensitive design
Friedman, B., Howe, D. C., & Felten, E. W. (2003) · Hawaii International Conference on System Sciences (HICSS) · doi:10.1109/hicss.2002.994366 · Cited by 149
Self-citationDigital Library Use: Social Practice in Design and Evaluation
Bishop, A. P., VanHouse, N. A., & Buttenfield, B. P. (2003) · CU Scholar (University of Colorado Boulder) · Cited by 139
Addressing age-related bias in sentiment analysis,
Diaz, M., Johnson, I., Lazar, A., Piper, A. M., & Gergle, D. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173986 · Cited by 138
“Much work in critical algorithm studies examines algorithmic bias, which can be defined as “systems that systematically and unfairly discriminate against certain individuals or groups of individuals in favor of others” [27].”
BackgroundGoverning Algorithms: A Provocation PiecePreprint
Barocas, S., Hood, S., & Ziewitz, M. (2013) · SSRN Electronic Journal · doi:10.2139/ssrn.2245322 · Cited by 133
Introna, L. D. (2007) · Ethics and Information Technology · doi:10.1007/s10676-006-9133-z · Cited by 133
“This awareness often only emerges over longer historical epochs, as suggested by Heidegger (1977). The practical and the coconstitutive play themselves out in different horizons of intelligibility.”
BackgroundForlano, L., & Mathew, A. (2014) · Journal of Urban Technology · doi:10.1080/10630732.2014.971525 · Cited by 144
“Value-sensitive design has been applied in a range of academic communities including human-computer interaction, computer-supported collaborative work, and information science (Friedman, 1996; Friedman and Nissenbaum, 1996; Nissenbaum, 2001; Le Dantec et al., 2009; Friedman et al., 2013).”
BackgroundTechnology as empowerment: a capability approach to computer ethics
Johnstone, J. (2007) · Ethics and Information Technology · doi:10.1007/s10676-006-9127-x · Cited by 142
“More recent developments that Bynum identifies include ‘valuesensitive design (e.g. Friedman and Nissenbaum 1996; Introna and Nissenbaum 2000) and Brey s ‘disclosive methodology (Brey 2001).”
BackgroundBuilding and Auditing Fair Algorithms: A Case Study in Candidate Screening
Wilson, C., Ghosh, A., Jiang, S., Mislove, A., Baker, L., Szary, J., Trindel, K., & Polli, F. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445928 · Cited by 133
“Since at least the mid-1990s, critical scholars have been raising alarms about the potential for computer systems to embed, entrench, and compound social biases [24].”
BackgroundHankerson, D., Marshall, A., Booker, J., Elmimouni, H., Walker, I., & Rode, J. A. (2016) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/2851581.2892578 · Cited by 137
Metadata
- Authors
- Friedman, B., & Nissenbaum, H.
- Affiliations
- Colby College (US)
- Princeton University (US)
- Venue
- ACM Transactions on Information Systems
- Published
- 1996-07
- Volume
- 14
- Issue
- 3
- Pages
- 330-347
- ISSN
- 1046-8188 (ISSN-L)
- 1558-2868
- OpenAlex
- W2006447892
- Cited by
- 1,203
- References
- 13
- Citations by year
- 2026: 102 · 2025: 149 · 2024: 151 · 2023: 121 · 2022: 126 · 2021: 133 · 2020: 79 · 2019: 70 · 2018: 50 · 2017: 21 · 2016: 29 · 2015: 18 · 2014: 18 · 2013: 16 · 2012: 8
- FWCI
- 17.25
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- ACM Transactions on Information Systems (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Social Sciences › Business, Management and Accounting › Management Information Systems › Information Systems Theories and Implementation
- Social Sciences › Law › Artificial Intelligence in Law
- Keywords
- technical bias, social institutions, technological constraints, societal impact of technology
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
