The DOI matches its Crossref record; 14 of the 14 references it cites were found in a registry or an index; 6 of the 22 sources its claims cite were checked for support.
Bias in computer systems
Open accessAbstract
From an analysis of actual cases, three categories of bias in computer systems have been developed: preexisting, technical, and emergent. Preexisting bias has its roots in social institutions, practices, and attitudes. Technical bias arises from technical constraints of considerations. Emergent bias arises in a context of use. Although others have pointed to bias inparticular computer systems and have noted the general problem, we know of no comparable work that examines this phenomenon comprehensively and which offers a framework for understanding and remedying it. We conclude by suggesting that freedom from bias should by counted amoung the select set of criteria—including reliability, accuracy, and efficiency—according to which the quality of systems in use in society should be judged.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
6 of 22 cited sources checked · 4 supported · 0 flagged · accuracy not yet measured
Every source the paper cites for this claim was checked and supports it.
In the paper
“As specified in Section 50-(9) of the British Nationality Act itself and imple-mented in BNAP,”
§ Bias in Computer Systems > 3.3 The British Nationality Act Program (BNAP)
What the sources say
- Supportedcites Sergot et al. 1986 source
The “Formalization by Trial and Error” passage states the same distinction: a man is the father only of his legitimate children, while a woman is the mother of all her children. It also says incorporating that provision requires an adjustment to the existing rules, supporting the claim that it was implemented in the logic program. (LLM, conf=0.94)
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“At this point, it was discovered that the original Admissions Algorithm placed married couples at a disadvantage in achieving an optimal placement as compared with their single peers [Roth 1984;1990].”
What the sources say
- Partly supportedcites Roth 1984 sourceExperimental — accuracy not yet measured
Section IV, “Contemporary Issues” (page not supplied), says the couples algorithm used until 1983 could not accept preferences over pairs of positions and could produce outcomes unstable with respect to couples. That supports a disadvantage for couples, but the paper does not establish that couples achieved less optimal placements than single applicants. (LLM, conf=0.84)
- Not checked — no full text of the sourcecites Roth 1990 source
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“In this manner the MLSA gives processing attention to all processes as quickly as possible that are beginning a new command.”
§ Bias in Computer Systems > 3.2 A Multilevel Scheduling Algorithm (MLSA)
What the sources say
- Supportedcites Corbato et al. 1962 source
The “A Multi-Level Scheduling Algorithm” section says programs entering the system are placed in priority queues and that scheduling begins with the lowest occupied level, switching to a newly occupied lower level. The IBM 7090 system section also says the supervisor gives top priority to initiating waiting commands. Together, these passages support the paraphrase about prompt attention to processes beginning a command. (LLM, conf=0.88)
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“At the time of the BNAP's initial implementation (1983) no mechanism was built into the program to incorporate relevant case law as it came into being [Sergot et al. 1986].”
§ Bias in Computer Systems > 3.3 The British Nationality Act Program (BNAP)
What the sources say
- Supportedcites Sergot et al. 1986 source
The introduction says the Act was free of case law when the original implementation began in summer 1983. It also says the authors deliberately avoided incorporating rules derived from previous cases, leaving treatment of case law for further investigation. These passages support the claim that the initial program had no built-in means of incorporating emerging case law. (LLM, conf=0.88)
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes- Supported
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The Admissions Algorithm does not take into account geographic distribution when determining a match, considering only the ranked preferences of hospitals and students.”
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesNone of the sources cited for this claim have been checked yet.
In the paper
“Although others have pointed to bias in particular computer systems and have noted the general problem, we know of no comparable work that examines this phenomenon comprehensively and which offers a framework for understanding and remedying it.”
- Also stated in: § INTRODUCTION · hedged
“Although others have pointed to bias in particular computer systems and have noted the general problem [Johnson and Mulvey 1993;Moor 1985], we know of no comparable work that focuses exclusively on this phenomenon and examines it comprehensively.”
§ Bias in Computer Systems > Abstract
What the sources say
- Not matchedcites Johnson and Mulvey 1993
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not checked — no full text of the sourcecites Moor 1985 source
Load-bearingStrength:hedged· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes? Based: unclear- Also stated in: § INTRODUCTION · hedged
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“From an analysis of actual cases, three categories of bias in computer systems have been developed: preexisting, technical, and emergent.”
- Also stated in: § INTRODUCTION · plain
“From an analysis of actual computer systems, we have developed three categories: preexisting bias, technical bias, and emergent bias.”
- Also stated in: § 2. FRAMEWORK FOR ANALYZING BIAS IN COMPUTER SYSTEMS · plain
“The framework that emerged from this methodology is comprised of three overarching categories-preexisting bias, technical bias, and emergent bias.”
§ Bias in Computer Systems > Abstract
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes- Rests on an assumption: The 17 computer systems examined are representative enough to generalize the framework to computer systems generally.
- Also stated in: § INTRODUCTION · plain
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Emergent bias arises in a context of use.”
- Also stated in: § 2.3 Emergent Bias · plain
“emergent bias arises only in a context of use.”
§ Bias in Computer Systems > Abstract
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes- Also stated in: § 2.3 Emergent Bias · plain
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Thus, in terms of overall service from the processor, these individuals with long-running programs are systematically disadvantaged.”
§ Bias in Computer Systems > 3.2 A Multilevel Scheduling Algorithm (MLSA)
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesA conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Thus the British Nationality Act is biased against the illegitimate descendants of British men.”
§ Bias in Computer Systems > 3.3 The British Nationality Act Program (BNAP)
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
- Probably not a substantial claim. 18
- It may state two separate claims as one. 15
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“For in the same text the authors also state (as noted above) that their goal is to translate the British Nationality Act so that”
§ Bias in Computer Systems > 5. CONCLUSION
What the sources say
- Partly supportedcites Sergot et al. 1986 sourceExperimental — accuracy not yet measured
Split into 2 atoms — 1 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Partly supported
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“One charges that the Admissions Algorithm systematically favors hospital programs over medical students in cases of conflict [Graettinger and Peranson 1981b;Roth 1984;Sudarshan and Zisook 1981;Williams et al. 1981].”
§ Bias in Computer Systems > 3.1 The National Resident Match Program (NRMP)
What the sources say
- Not matchedcites Graettinger and Peranson 1981b
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Supportedcites Roth 1984 source
Section III (Theoretical Analysis; page unavailable) states that the NRMP algorithm produces the best stable outcome for hospitals and the worst for students when preferences are submitted as strict rank-order lists. Section IV (Contemporary Issues; page unavailable) also identifies whether the algorithm unfairly favors hospitals as an issue that had been raised, while noting a qualification when participants may be indifferent between matches. (LLM, conf=0.92)
- Not checked — no full text of the sourcecites Williams et al. 1981 source
Strength:hedged· not yet calibratedCausal:direct· not yet calibratedTests✗ Asserted: fails✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The authors may not put it forward as true. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Technical bias arises from technical constraints or considerations.”
- Also stated in: § 2.1 Preexisting Bias · plain
“Technical bias arises from technical constraints or technical considerations.”
§ Bias in Computer Systems > Abstract
Tests✓ Asserted: passes✗ Checkable: fails✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
- Also stated in: § 2.1 Preexisting Bias · plain
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Preexisting bias has its roots in social institutions, practices, and attitudes.”
- Also stated in: § INTRODUCTION · plain
“Preexisting bias has its roots in social institutions, practices, and attitudes.”
§ Bias in Computer Systems > Abstract
Tests✓ Asserted: passes✗ Checkable: fails✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
- Also stated in: § INTRODUCTION · plain
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In this case, we ourselves would shy away from viewing this as a bias in the system because we think this discrimination can be defended as having a reasonable basis.Namely, the discrimination reflects the preferences of match participants, and it is reasonable in our view for employment decisions to be determined largely by the choices of employers and employees.”
Load-bearingStrength:hedged· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
- Rests on an assumption: The discrimination against rural hospitals reflects the preferences of match participants.
- Rests on an assumption: The best teaching hospitals tend to be in urban areas, and urban areas tend to fill their positions far more effectively, and with better students, than rural hospitals.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Emergent bias poses a potential problem for any legal expert system, especially in a society where legal systems depend on evolving case law.”
§ Bias in Computer Systems > 3.3 The British Nationality Act Program (BNAP)
Tests✓ Asserted: passes✗ Checkable: fails✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We thus infer that in this case the biased designs were unintended and, instead, reflected gender-biases deeply rooted in the culture at large.”
§ Bias in Computer Systems > 4. CONSIDERATIONS FOR MINIMIZING BIAS IN COMPUTER SYSTEM DESIGN
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As Sabre and Apollo illustrate, biases in computer systems can be difficult to identify let alone remedy because of the way the technology engages and extenuates them.Computer systems, for instance, are comparatively inexpensive to disseminate, and thus, once developed, a biased system has the potential for widespread impact.If the system becomes a standard in the field, the bias becomes pervasive”
§ Bias in Computer Systems > INTRODUCTION
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
In the paper
“Huff and Cooper report that along a number of dimensions the designs proposed by subjects in the gender-unspecified group closely resembled the designs proposed by subjects who designed for boys and were significantly different from the designs proposed by subjects who designed for girls.”
§ Bias in Computer Systems > 4. CONSIDERATIONS FOR MINIMIZING BIAS IN COMPUTER SYSTEM DESIGN
What the sources say
- Not checked — no full text of the sourcecites Huff and Cooper 1987 source
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“One source of this alleged bias lies in Sabre's and Apollo's algorithms for controlling search and display functions.In the algorithms, preference is given to”
§ Bias in Computer Systems > INTRODUCTION
Strength:hedged· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Thus, a shift in context of use may well create difficulties for a new set of users.”
§ Bias in Computer Systems > 2.2 Technical Bias
Strength:hedged· not yet calibratedCausal:conditional· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“Thus, the computer systems systematically downgrade and, hence, are biased against international carriers who fly few, if any, internal U.S. flights, and against internal carriers who do not fly international flights [Fotos 1988;Ott 1988].”
§ Bias in Computer Systems > INTRODUCTION
What the sources say
- Not matchedcites Fotos 1988
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites Ott 1988
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This bias typically emerges some time after a design is completed, as a result of changing societal knowledge, population, or cultural values.”
§ Bias in Computer Systems > 2.2 Technical Bias
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✗ Based: failsProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
In the paper
“One is that the interface design compounds the bias in the reservation systems.Lists of ranked flight options are displayed screen by screen.Each screen displays only two to five options.The advantage to a carrier of having its flights shown on the first screen is enormous since 90% of the tickets booked by travel agents are booked by the first screen display [Taib 1990].”
§ Bias in Computer Systems > INTRODUCTION
What the sources say
- Not matchedcites Taib 1990
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Or consider again the case of technical bias with the MLSA (which favors users with short jobs over long jobs).While technical bias is embedded in the program, the bias is a phenomenon of the system in use, in a context wherein users with short and long jobs outpace the system's capacity.”
§ Bias in Computer Systems > 4. CONSIDERATIONS FOR MINIMIZING BIAS IN COMPUTER SYSTEM DESIGN
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
In the paper
“In their study, subjects were asked to propose designs for software to teach seventh graders the correct use of commas.One group of subjects was asked to design the software for seventh-grade boys, the second group to design for seventh-grade girls, and the third group to design for seventh-graders, gender unspecified.”
§ Bias in Computer Systems > 4. CONSIDERATIONS FOR MINIMIZING BIAS IN COMPUTER SYSTEM DESIGN
What the sources say
- Not checked — no full text of the sourcecites Huff and Cooper 1987 source
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This type of bias can enter a system either through the explicit and conscious efforts of individuals or institutions, or implicitly and unconsciously, even in spite of the best of intentions.”
§ Bias in Computer Systems > 2.1 Preexisting Bias
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✗ Based: failsProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Sources of technical bias can be found in several aspects of the design process, including limitations of computer tools such as hardware, software, and peripherals; the process of ascribing social meaning to algorithms developed out of context; imperfections in pseudorandom number generation; and the attempt to make human constructs amenable to computers, when we quantify the qualitative, discret”
§ Bias in Computer Systems > 2.2 Technical Bias
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“The constraint inherent in the NRMP algorithm, in which preference is given to hospital choices when conflicts in rankings occur, duplicates what happens in an actual admissions process without a computerized matching program”
§ Bias in Computer Systems > 3.1 The National Resident Match Program (NRMP)
What the sources say
- Not matchedcites Graettinger and Peranson 1981b
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Whatever ranking algorithm is used, if it systematically places certain airlines' flights on initial screens and other airlines' flights on later screens, the system will exhibit technical bias.”
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“They can also reflect the personal biases of individuals who have significant input into the design of the system, such as the client or system designer.”
§ Bias in Computer Systems > 2.1 Preexisting Bias
Strength:plain· not yet calibratedCausal:correlational· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“those responsible for maintaining the system responded conscientiously to this societal change and modified the system's algorithm to place couples more fairly [Roth 1990].”
§ Bias in Computer Systems > 4. CONSIDERATIONS FOR MINIMIZING BIAS IN COMPUTER SYSTEM DESIGN
What the sources say
- Not checked — no full text of the sourcecites Roth 1990 source
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Preexisting biases may originate in society at large, in subcultures, and in formal or informal, private or public organizations and institutions.”
§ Bias in Computer Systems > 2.1 Preexisting Bias
Strength:hedged· not yet calibratedCausal:conditional· not yet calibratedTests✓ Asserted: passes✗ Checkable: fails✗ Load-bearing: fails✗ Based: failsProbably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
In the paper
“Should the accumulation of case law lead to changes in the way the Act is interpreted-say, by granting new subgroups British citizenship-BNAP would systematically misinform members of this subgroup regarding their status as British citizens [Leith 1986].”
§ Bias in Computer Systems > 3.3 The British Nationality Act Program (BNAP)
What the sources say
- Not checked — no full text of the sourcecites Leith 1986 source
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“In total, we examined 17 computer systems from diverse fields including banking, commerce, computer science, education, medicine, and law.”
§ Bias in Computer Systems > 2. FRAMEWORK FOR ANALYZING BIAS IN COMPUTER SYSTEMS
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We derived our framework by examining actual computer systems for bias.Instances of bias were identified and characterized according to their source, and then the characterizations were generalized to more abstract categories.These categories were further refined by their application to other instances of bias in the same or additional computer systems.”
§ Bias in Computer Systems > 2. FRAMEWORK FOR ANALYZING BIAS IN COMPUTER SYSTEMS
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“although we have talked about bias in computer systems, the presence of bias is not so much a feature inherent in the system independent of the context of use, but an aspect of a system in use.”
§ Bias in Computer Systems > 4. CONSIDERATIONS FOR MINIMIZING BIAS IN COMPUTER SYSTEM DESIGN
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✗ Checkable: fails✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim. It may be too vague to check against evidence.
The paper relies on this without support.
In cases of conflict, there is no good rationale for favoring hospital programs at the expense of medical students.
“We are inclined to answer yes because in cases of conflict there does not appear to be a good rationale for favoring hospital programs at the expense of students.”
The paper relies on this without support.
The discrimination against rural hospitals reflects the preferences of match participants.
“Namely, the discrimination reflects the preferences of match participants, and it is reasonable in our view for employment decisions to be determined largely by the choices of employers and employees.”
The paper relies on this without support.
The 17 computer systems examined are representative enough to generalize the framework to computer systems generally.
“In total, we examined 17 computer systems from diverse fields including banking, commerce, computer science, education, medicine, and law.”
The paper relies on this without support.
The illustrative examples in Table I portray plausible cases of bias.
“The illustrative examples portray plausible cases of bias.”
The paper relies on this without support.
The best teaching hospitals tend to be in urban areas, and urban areas tend to fill their positions far more effectively, and with better students, than rural hospitals.
“Because the best teaching hospitals tend to be in urban areas, the urban areas tend to fill their positions far more effectively, and with better students, than rural hospitals.”
The authors state this as an assumption.
The MLSA system is not saturated with too many users.
“assuming the system is not saturated with too many users”
The paper relies on this without support.
BNAP was designed in a research environment among people with sophisticated knowledge of immigration law.
“First, the system was designed in a research environment, among people with sophisticated knowledge of immigration law.”
The paper relies on this without support.
BNAP's users are likely to be at best paralegal or immigration counselors in Britain, if not lay persons in foreign countries with only limited access to British legal expertise.
“Its users, however, are likely to be at best paralegal or immigration counselors in Britain, if not lay persons in foreign countries with only limited access to British legal expertise.”
The paper relies on this without support.
Some of BNAP's queries require expert knowledge to answer.
“Some of the program's queries, for example, require expert knowledge to answer.”
The paper relies on this without support.
Nonexperts advising British citizen hopefuls are not alerted to alternative legal frameworks that complement the British Nationality Act.
“More generally, nonexperts advising British citizen hopefuls are not alerted to alternative legal frameworks that complement the British Nationality Act.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The multi-metric approach (Figure S1) makes explicit potential trade-offs and helps ensure the nonaccuracy desiderata are not treated as second-class citizens to accuracy (see Ref. 13).”
From Holistic Evaluation of Language Models (2023) · cites this paper as [13]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“research has repeatedly shown that these systems can in fact encode biases into decisions 124 .”
From Administrative law and the machines of government: judicial review of automated public-sector decision-making (2019) · cites this paper as B Friedman and H Nissenbaum 'Bias in Computer Systems' (1996)
The cited source says what the claim attributes to it.
In the citing paper
“The risk of”
From Positioning Paradata: A Conceptual Frame for AI Processual Documentation in Archives and Recordkeeping Contexts (2023) · cites this paper as [25]
The cited source says what the claim attributes to it.
In the citing paper
“Bias or opacity are far from newly observed characteristics of computer systems generally [26, 11, 24], but these issues have been exacerbated by the rapidly expanding numbers of data-driven information systems entering decision-making processes.”
From Fairness and Accountability Design Needs for Algorithmic Support in High-Stakes Public Sector Decision-Making (2018) · cites this paper as [26]
In the citing paper
“These decisions may be based on factors of ethical importance (e.g., ethnic, gender, or religious grounds) and irrelevant to the decision-making at hand, or they may be relevant but legally protected as a nondiscriminatory characteristic (Friedman and Nissenbaum 1996).”
From How to Design AI for Social Good: Seven Essential Factors (2020) · cites this paper as Friedman and Nissenbaum 1996
In the citing paper
“On this account, the apparent problem to which data science is a solution determines whether it happens to serve morally defensible ends.”
From Problem Formulation and Fairness (2019) · cites this paper as [17]
In the citing paper
“Corresponding with [14] this outcome systematically disadvantages individuals of one gender by preferring the other at the topk positions.”
From FA*IR: A Fair Top-k Ranking Algorithm (2017) · cites this paper as [14]
In the citing paper
“While technical factors, such as biased data and design choices, play an important role, biases can also arise from the context of use, for example, how human users interpret system outputs or from a mismatch between the capabilities and values assumed in the design of the system and those of its actual users (Danks & London, 2017; Friedman & Nissenbaum, 1996).”
From Digital ageism: challenges and opportunities in artificial intelligence for older adults (2022) · cites this paper as Friedman & Nissenbaum, 1996
In the citing paper
“and to data protection, privacy, and other human rights issues 11”
From Administrative Law and the Machines of Government: Judicial Review of Automated Public-Sector Decision-Making (2018) · cites this paper as [11]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Table II gives conflicting years for its source
MinorThe table caption dates the Williams et al. source to 1991, while the paper's reference entry for that work dates it to 1981.
“Table II.The Simplest Case of Rank-Order Lists in which the Desires of Students and Programs are in Conflict (Reprinted from Williams et al. [1991], by permission of the New England Journal of Medicine)”
“18. WILLIAMS, K. J., WERTH, V. P., AND WOLFF, J. A. 1981. An analysis of the resident match.”
Citations
Cites
The evolution of the labor market for medical interns and residents: A case study in game theory
Roth, A. E. (1984) · Journal of Political Economy · doi:10.1086/261272 · Cited by 1,219
MOOR, J. H. (1985) · Metaphilosophy · doi:10.1111/j.1467-9973.1985.tb00173.x · Cited by 863
The British Nationality Act as a logic program
Sergot, M. J., Sadri, F., Kowalski, R. A., Kriwaczek, F., Hammond, P., & Cory, H. T. (1986) · Communications of the ACM · doi:10.1145/5689.5920 · Cited by 564
An experimental time-sharing system
Corbató, F. J., Merwin-Daggett, M., & Daley, R. C. (1962) · Proceedings of the May 1-3, 1962, spring joint computer conference on - AIEE-IRE '62 (Spring) · doi:10.1145/1460833.1460871 · Cited by 234
New physicians: A natural experiment in market organization
Roth, A. E. (1990) · Science · doi:10.1126/science.2274783 · Cited by 141
Sex Bias in Educational Software: The Effect of Designers' Stereotypes on the Software They Design1
Huff, C., & Cooper, J. (1987) · Journal of Applied Social Psychology · doi:10.1111/j.1559-1816.1987.tb00328.x · Cited by 85
Fundamental Errors in Legal Logic Programming
Leith, P. (1986) · The Computer Journal · doi:10.1093/comjnl/29.6.545 · Cited by 51
Moor, J. H. (2017) · Routledge eBooks · doi:10.4324/9781315259697-7 · Cited by 40
Moor, J. H. (2020) · Routledge eBooks · doi:10.4324/9781003075011-1 · Cited by 21
National resident matching program
(1981) · New England Journal of Medicine · doi:10.1056/nejm198108273050923 · Cited by 10
Flagged first, then by citations
Research works printed but not linked: 4
- FISHLOCK , T. 1981. Delhi press detect racism in Nationality Bill. The London Times ( Jan. 20 ) . FISHLOCK, T. 1981. Delhi press detect racism in Nationality Bill. The London Times (Jan. 20).
- GAO . 1992. Patriot Missile defense: Software problem led to system failure at Dhahran , Saudi Arabia. GAO/IMTEC-92-26, U.S. General Accounting Office, Washington, D.C. GAO. 1992. Patriot Missile defense: Software problem led to system failure at Dhahran, Saudi Arabia. GAO/IMTEC-92-26, U.S. General Accounting Office, Washington, D.C.
- JOHNSON , D.G. AND MULVEY , J. M. 1993. Computer decisions: Ethical issues of responsibil-ity and bias. Statistics and Operations Res. Series SOR-93-11 , Dept. of Civil Engineering and Operations Research , Princeton Univ ., Princeton, N.J. JOHNSON,D.G.AND MULVEY, J. M. 1993. Computer decisions: Ethical issues of responsibil-ity and bias. Statistics and Operations Res. Series SOR-93-11, Dept. of Civil Engineering and Operations Research, Princeton Univ., Princeton, N.J.
- OTT , J. 1988. American Airlines settles CRS dispute with British Airways. Aviat. Week Space Tech. ( July 18 ) . OTT, J. 1988. American Airlines settles CRS dispute with British Airways. Aviat. Week Space Tech. (July 18).
Cited by
Sorted by citations
Items 401–500 of 1,155
If it is bad, why don’t I quit? Algorithmic recommendation use strategy from folk theories
Chen, K. (2023) · Global Media and China · doi:10.1177/20594364231209354 · Cited by 17
“These con fi gurations often favor certain values and interests over others, thereby revealing the presence of both human and technical biases among today ’ s emerging gatekeepers (Bozdag, 2013; Friedman & Nissenbaum, 1996; Kraemer et al., 2011).”
Empathy: an ethical consideration of AI & others in the workplace
Kleinrichert, D. (2024) · Ai & Society · doi:10.1007/s00146-023-01831-w · Cited by 12
“In citing the earlier work of Friedman and Nissenbaum (1996), “technology has consequences, tests norms, changes what we do or are able to do, acts for us, and makes biased deci-sions” (Martin et al. 2019, p. 307).”
Riebe, T., Schmid, S., & Reuter, C. (2020) · IEEE Technology and Society Magazine · doi:10.1109/mts.2020.3031846 · Cited by 18
“yields theoretical and methodological implications by assuming more or less abstract values to be reflected in interfaces or software and thus indicate the need of interpretative work [70].”
BackgroundBeyond Trust Building — Calibrating Trust in Visual Analytics
Han, W., & Schulz, H. (2020) · IEEE Workshop on TRust and EXpertise in Visual Analytics (TREX) · doi:10.1109/trex51495.2020.00006 · Cited by 21
“In addition to user involvement and understanding, Friedman and Nissenbaum pointed out that technical and social constraints can transfer into issues in computer systems [16].”
Background“It’s Like the Value System in the Loop”: Domain Experts’ Values Expectations for NLP Automation
Showkat, D., & Baumer, E. P. S. (2022) · Designing Interactive Systems Conference · doi:10.1145/3532106.3533483 · Cited by 21
“As noted above by P3, increasing support for autonomy and subjectivity in a system may simultaneously also increase the possibility of introducing bias and other potential misuse [59, 61].”
BackgroundMethodInfluentialMannhardt, F. (2022) · Lecture notes in business information processing · doi:10.1007/978-3-031-08848-3_12 · Cited by 18
Angehrn, Z., Sostar, J., Nordon, C., Turner, A. J., Gove, D., Karcher, H., Keenan, A., Mittelstadt, B., & de Vulpillieres, F. D. R. (2020) · Journal of Alzheimer s Disease · doi:10.3233/jad-191159 · Cited by 21
“…be purely technical (e.g., low granularity of data for prediction affecting the precision of prediction) but can also be rooted in social attitudes and practices, either pre-existing at the time when a predictive model is being developed or emerging during and through the use of this model [67].”
Rhea, A. K., Markey, K., D’Arinzo, L., Schellmann, H., Sloane, M., Squires, P., & Stoyanovich, J. (2022) · AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3514094.3534189 · Cited by 19
“Reports of algorithmic hiring systems acting in ways that are discriminatory or unreliable abound [3, 6, 12, 13, 18, 60].”
BackgroundEthics of Seamless infrastructures: Resources and Future Directions
Ratto, M. (2007) · The International Review of Information Ethics · doi:10.29173/irie93 · Cited by 18
Ethical implications of values embedded in computational models: An exploratory study
Fleischmann, K. R., & Wallace, W. A. (2006) · Proceedings of the American Society for Information Science and Technology · doi:10.1002/meet.14504301254 · Cited by 18
The Indicator Browser: A Web-Based Interface for Visualizing UrbanSim Simulation Results
Schwartzman, Y., & Borning, A. H. (2007) · Hawaii International Conference on System Sciences (HICSS) · doi:10.1109/hicss.2007.540 · Cited by 17
“Next, we committed to several key moral values to support explicitly: fairness and more specifically freedom from bias [7], representativeness, accountability, and support for a democratic society.”
Background“Don’t let me be misunderstood”
Strauß, S. (2021) · TATuP - Zeitschrift für Technikfolgenabschätzung in Theorie und Praxis · doi:10.14512/tatup.30.3.44 · Cited by 16
Challenging the inequitable impacts of edtech
Macgilchrist, F., Potter, J. M., & Williamson, B. (2024) · Learning Media and Technology · doi:10.1080/17439884.2024.2350117 · Cited by 16
Larsen, B. C. (2021) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3461702.3462591 · Cited by 20
“humans are subject to bias [45], this means that the values of the designer can be “frozen into the code, effectively institutionalizing those values’’ [37: 158].”
BackgroundLePere‐Schloop, M., Appe, S., Adjei‐Bamfo, P., Zook, S., & Bawole, J. N. (2021) · Nonprofit and Voluntary Sector Quarterly · doi:10.1177/08997640211057401 · Cited by 16
“Friedman and Nissenbaum (1996) problematize computational methods by considering the ways that social structures and institutions, as well as technological constraints, shape technical choices, whereas changes in context of use influence algorithm application and outcomes.”
BackgroundInfluentialEagle, T., Mehrotra, A., Sharma, A., Zuniga, A., & Whittaker, S. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3555155 · Cited by 16
“, apps assuming normative standards concerning weight, or gender roles) [42,48].”
BackgroundImplications of AI Bias in HRI
Hitron, T., Yaar, N. M., & Erel, H. (2023) · ACM/IEEE International Conference on Human-Robot Interaction (HRI) · doi:10.1145/3568162.3576977 · Cited by 16
“It refers to AI and machine learning algorithms, that due to their training process, involve stereotypes leading to favoring or disfavoring of particular groups or individuals [3, 17, 26].”
BackgroundDahl, Y., Farshchian, B. A., Vilarinho, T., Helbostad, J. L., Nawaz, A., Nygård, A. J., & Wik, P. B. (2015) · Interacting with Computers · doi:10.1093/iwc/iwv036 · Cited by 19
“In this context, biases typically refer to value orientations, approximations, assumptions and prejudices that reflect those held by the designer, the society or both (Friedman and Nissenbaum, 1996).”
BackgroundAxes for Sociotechnical Inquiry in AI Research
Dean, S., Gilbert, T. K., Lambert, N., & Zick, T. (2021) · IEEE Transactions on Technology and Society · doi:10.1109/tts.2021.3074097 · Cited by 20
“mining community [26] and echoing older concerns of bias in computer systems [27].”
BackgroundArtificial Intelligence and People with Disabilities: a Reflection on Human–AI Partnerships
White, J. J. G. (2021) · Springer eBooks · doi:10.1007/978-3-030-72188-6_14 · Cited by 18
Recipes for cookies: how institutions shape communication technologies
Shah, R., & Kesan, J. P. (2009) · New Media & Society · doi:10.1177/1461444808101614 · Cited by 17
“The value-laden nature of technology is evident in the literature on communication technologies (Flanagan et al., 2000; Foner, 2002; Friedman and Nissenbaum, 1996), such as internet search engines (Introna and Nissenbaum, 2000).”
BackgroundGender Bias in Artificial Intelligence
Ruiz, E. L., & Sedeño, E. P. (2023) · Intelligent systems reference library · doi:10.1007/978-3-031-21606-0_4 · Cited by 15
AI4people − an ethical framework for a good AI society: the Ghana (Ga) perspective
Ammah, L., Lütge, C., Kriebitz, A., & Ramkissoon, L. (2024) · Journal of Information Communication and Ethics in Society · doi:10.1108/jices-06-2024-0072 · Cited by 15
Responsible AI, ethics, and the AI lifecycle: how to consider the human influence?
Elia, M., Ziethmann, P., Krumme, J., Schlögl-Flierl, K., & Bauer, B. (2025) · AI and Ethics · doi:10.1007/s43681-025-00666-z · Cited by 16
“Bias-identification and negative bias-effect mitigation are crucial for a trustworthy adoption of AI within society [ 18 ].”
InfluentialFan, X., Xiao, Q., Zhou, X., Pei, J., Sap, M., Lu, Z., & Shen, H. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713477 · Cited by 52
“…this expert-driven approach often fails to uncover significant issues that everyday users of algorithmic systems can quickly detect in actual use [48], as these issues may only arise or be perceived as harmful in specific contexts or usage patterns that auditors may not anticipate [23, 32, 37].”
Bias and comparison framework for abusive language datasets
Wich, M., Eder, T., Kuwatly, H. A., & Groh, G. (2021) · AI and Ethics · doi:10.1007/s43681-021-00081-0 · Cited by 16
“We define bias as a phenomenon in which a system “systematically and unfairly discriminate[s] against certain individuals or groups of individuals in favor of others” [17, p. 332].”
BackgroundRecent Advances in Algorithmic Biases and Fairness in Financial Services: A Survey
Bajracharya, A., Khakurel, U. B., Harvey, B., & Rawat, D. B. (2022) · Lecture notes in networks and systems · doi:10.1007/978-3-031-18461-1_53 · Cited by 16
Helm, P., & Matzner, T. (2023) · New Media & Society · doi:10.1177/14614448231165916 · Cited by 22
“More generally, factors like the demographic of a programming team or their location can import certain assumptions and biases into the program—often in a manner that is difficult to notice (Friedman and Nissenbaum, 1996).”
Kopec, M., Magnani, M., Ricks, V., Torosyan, R., Basl, J., Miklaucic, N., Muzny, F., Sandler, R. L., Wilson, C., Wisniewski-Jensen, A., et al. (2023) · Big Data & Society · doi:10.1177/20539517231176230 · Cited by 36
“A particular emphasis is placed on how design occurs within a value-laden, socio-technical context (Feng and Feenberg, 2008; van de Poel and Royakkers, 2011), how design choices can have serious socio-political effects (Friedman and Nissenbaum, 1996), and how artifacts can, themselves, “have politics” (Winner, 1980).”
BackgroundAlgorithms for All: Can AI in the Mortgage Market Expand Access to Homeownership?
Perry, V. G., Martin, K. E., & Schnare, A. B. (2023) · AI · doi:10.3390/ai4040045 · Cited by 14
“Model developers should take note that the data and models reflect value judgments that may include certain biases [13,56].”
BackgroundJamieson, J., Yamashita, N., & Foong, E. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3623340 · Cited by 19
“That value-related discussions vary by project and over time is consistent with arguments that changes in the social context around software [21] can prompt developers to revisit ethical and social challenges that were previously regarded as settled.”
ResultTackling Language Modelling Bias in Support of Linguistic Diversity
Bella, G., Helm, P., Koch, G., & Giunchiglia, F. (2024) · The 2024 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3630106.3658925 · Cited by 15
Wongvorachan, T., Bulut, O., Liu, J. X., & Mazzullo, E. (2024) · Information · doi:10.3390/info15060326 · Cited by 19
Lockbox: mobility, privacy and values in cloud storage
Stark, L., & Tierney, M. (2013) · Ethics and Information Technology · doi:10.1007/s10676-013-9328-z · Cited by 18
Bias in Geographic Information Systems: The Case of Google Maps
Wagner, B., Human, S., & Winkler, T. (2021) · Hawaii International Conference on System Sciences (HICSS) · doi:10.24251/hicss.2021.103 · Cited by 17
“based on ground truth, which refers to field exploration in cartography, and the predictions made by Google Maps as system bias [9].”
MethodAlgorithmic Accountability in Context. Socio-Technical Perspectives on Structural Causal Models
Poechhacker, N., & Kacianka, S. (2021) · Frontiers in Big Data · doi:10.3389/fdata.2020.519957 · Cited by 19
“Because of the intensifying application of these systems in various social domains, issues of fairness, (in)justice and power relations have become the focus of attention, especially in the form of bias (Friedman and Helen, 1996; Bozdag, 2013; Crawford, 2013).”
BackgroundSystemic Mapping and Design Research: Towards Participatory Democratic Engagement
de la Rosa, J., Ruecker, S., & Nohra, C. G. (2021) · She ji · doi:10.1016/j.sheji.2021.04.001 · Cited by 15
Instructional Methods and Curricula for “Values Conscious Design”
Belman, J., Flanagan, M., & Nissenbaum, H. F. (2009) · Loading · Cited by 15
Self-citationDo You Own a Volkswagen? Values as Non-Functional Requirements
Barn, B. S. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-44902-9_10 · Cited by 15
“Thus design choices that explicitly consider values can change the affordances of resulting technologies [15,28].”
Banerjee, I., Warnier, M., Brazier, F. M. T., & Helbing, D. (2021) · Scientific Reports · doi:10.1038/s41598-021-86635-y · Cited by 20
Huldtgren, A. (2014) · Springer eBooks · doi:10.1007/978-94-007-6970-0_35 · Cited by 17
Bylund, M. (2005) · KTH Publication Database DiVA (KTH Royal Institute of Technology) · Cited by 15
AI and Discrimination: Sources of Algorithmic Biases
Moussawi, S., Deng, X. N., & Joshi, K. D. (2024) · ACM SIGMIS Database the DATABASE for Advances in Information Systems · doi:10.1145/3701613.3701615 · Cited by 28
“According to Friedman and Nissenbaum (1996), bias simply means slant.”
Unmasking Nationality Bias: A Study of Human Perception of Nationalities in AI-Generated Articles
Venkit, P. N., Gautam, S., Panchanadikar, R., Huang, T.-H., & Wilson, S. (2023) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3600211.3604667 · Cited by 21
“For this work, we associate with the definition of bias proposed by Friedman and Nissenbaum [21].”
MethodUnderstanding and evaluating harms of AI-generated image captions in political images
Sarhan, H., & Hegelich, S. (2023) · Frontiers in Political Science · doi:10.3389/fpos.2023.1245684 · Cited by 16
Lay User Involvement in Developing Human-centric Responsible AI Systems: When and How?
Vincenzi, B., Stumpf, S., Taylor, A. S., & Nakao, Y. (2024) · ACM Journal on Responsible Computing · doi:10.1145/3652592 · Cited by 16
“AI development processes present numerous opportunities for bias and unfairness to arise [ 8 , 20 , 31 , 40 , 48 , 55 , 88 ].”
BackgroundAnimation and Artificial Intelligence
Stark, L. (2024) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3630106.3658995 · Cited by 19
“…interactive AI systems as animated entities places these technologies within the ambit of a variety of existing legal, regulatory, policy, and design tools, including extant literature in human-computer interaction [18,38,44,57,60] and scholarship on values in digital design [3,23,26,27,28].”
BackgroundCalvi, A., Malgieri, G., & Kotzinos, D. (2024) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3630106.3659024 · Cited by 20
“(Technical) bias occurs when computer systems “systematically and unfairly discriminate against certain individuals or groups of individuals in favour of others”, by for instance denying opportunities or assigning undesirable outcomes [36].”
BackgroundPhilosophical Investigations into AI Alignment: A Wittgensteinian Framework
Pérez‐Escobar, J. A., & Sarikaya, D. (2024) · Philosophy & Technology · doi:10.1007/s13347-024-00761-9 · Cited by 20
Fairness-Aware Instrumentation of Preprocessing~Pipelines for Machine Learning
Yang, K., Huang, B., Stoyanovich, J., & Schelter, S. (2020) · UvA-DARE (University of Amsterdam) · doi:10.1145/3398730.3399194 · Cited by 18
“Further, these systems are subject to preexisting bias “that exists independently, and usually prior to the creation of the system” [7].”
BackgroundAI Development for the Public Interest: From Abstraction Traps to Sociotechnical Risks
Andrus, M., Dean, S., Gilbert, T. K., Lambert, N., & Zick, T. (2020) · IEEE International Symposium on Technology and Society (ISTAS) · doi:10.1109/istas50296.2020.9462193 · Cited by 15
“A growing awareness of the possibility for bias in data-driven systems developed over the past fifteen years, starting in the data mining community [26] and echoing older concerns of bias in computer systems [27].”
BackgroundExplainable Abusive Language Classification Leveraging User and Network Data
Wich, M., Mosca, E., Gorniak, A., Hingerl, J., & Groh, G. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-86517-7_30 · Cited by 16
On the Role of Value Sensitive Concerns in Software Engineering Practice
Barn, B. S., Barn, R., & Raimondi, F. (2015) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2015.182 · Cited by 14
“Value-sensitive design (VSD) emerged to integrate moral values (and more broadly ethics) with the design of systems.”
BackgroundInfluentialTaming Technical Bias in Machine Learning Pipelines
Schelter, S., & Stoyanovich, J. (2020) · UvA-DARE (University of Amsterdam) · Cited by 14
Ethics and Professional Responsibility in Computing
Loui, M. C., & Miller, K. (2008) · Wiley eBooks · doi:10.1002/9780470050118.ecse909 · Cited by 14
“Because choices involve tradeoffs between competing values, computing professionals should identify potential biases in their design choices (6).”
BackgroundAge and gender bias in pedestrian detection algorithms
Brandao, M. (2019) · arXiv · doi:10.48550/arxiv.1906.10490 · Cited by 32
“Recent research has brought to light some problematic biases in computer systems [11], artificial intelligence (AI) [3, 6] and robotics [13].”
BackgroundEscaping the Impossibility of Fairness: From Formal to Substantive Algorithmic FairnessPreprint
Green, B. (2021) · SSRN Electronic Journal · doi:10.2139/ssrn.3883649 · Cited by 14
Predictive Privacy: Towards an Applied Ethics of Data AnalyticsPreprint
Mühlhoff, R. (2021) · SSRN Electronic Journal · doi:10.2139/ssrn.3724185 · Cited by 14
Moon, D., & Ahn, S. (2025) · IEEE Access · doi:10.1109/access.2025.3556082 · Cited by 13
“Xu et al. demonstrated that AI-generated images can introduce invisible relevance bias in text-image retrieval systems, affecting search accuracy and fairness in content ranking [17].”
Signaling transparency in the era of artificial intelligence
Wang, F., López, C., & Okazaki, S. (2025) · Internet Research · doi:10.1108/intr-11-2023-1041 · Cited by 17
Framework for Assessing Ethical Aspects of Algorithms and Their Encompassing Socio-Technical System
van Bruxvoort, X., & van Keulen, M. (2021) · Applied Sciences · doi:10.3390/app112311187 · Cited by 16
“A well-studied AI issue is algorithmic bias: Outcomes may be skewed by a range of factors and thus might be considered unfair to certain groups or individuals [33].”
BackgroundMuralikumar, M. D., Yang, Y. S., & McDonald, D. W. (2023) · ACM Transactions on Social Computing · doi:10.1145/3582568 · Cited by 14
“A systemic bias in this context is a systematic error or assumption that creates unfair and discriminatory predictions [39].”
BackgroundThe effect of electronic program applications amidst the politics of administrative burden
Johnson, D., & Kroll, A. (2023) · Government Information Quarterly · doi:10.1016/j.giq.2023.101808 · Cited by 13
Outlier bias: AI classification of curb ramps, outliers, and context
Deitz, S. L. (2023) · Big Data & Society · doi:10.1177/20539517231203669 · Cited by 13
“These tools also make systematic errors which are likely to disproportionately affect categories of people, and there is a rich body of work on addressing bias (see, for example, Crawford 2017; Friedman and Nissenbaum 1996; Eubanks 2018).”
BackgroundCai, J., Patel, A., Naderi, A., & Wohn, D. Y. (2024) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3613905.3650882 · Cited by 21
“The findings can potentially benefit social media platforms by furthering their understanding of how their users perceive moderation decisions and enabling them to better current moderation practices with policymaking. to unfairness and injustice in use [13, 39].”
The Ecology of Harmful Design: Risk and Safety of Game Making on a Metaverse Platform
Kou, Y., Zhou, Y., Zhang, Z., & Gui, X. (2024) · Designing Interactive Systems Conference · doi:10.1145/3643834.3660678 · Cited by 14
“Batya Friedman and colleagues proposed VSD in the 1990s based on the observation that limited emphasis had been placed on values in the design of technology [33], but researchers started to raise concerns about human values such as privacy, autonomy, and informed consent [33, 34, 36].”
BackgroundMinimizing bias in computer systems
Friedman, B., Brok, E., Roth, S. K., & Thomas, J. C. (1996) · ACM SIGCHI Bulletin · doi:10.1145/249170.249184 · Cited by 14
Self-citationMühlhoff, R. (2020) · Deutsche Zeitschrift für Philosophie · doi:10.1515/dzph-2020-0059 · Cited by 8
Setting Diversity at the Core of HCI
Dankwa, N. K., & Draude, C. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-78092-0_3 · Cited by 17
Information Resilience: the nexus of responsible and agile approaches to information use
Sadiq, S. W., Aryani, A., Demartini, G., Hua, W., Indulska, M., Burton‐Jones, A., Khosravi, H., Benavides‐Prado, D., Sellis, T., Someh, I., et al. (2022) · The VLDB Journal · doi:10.1007/s00778-021-00720-2 · Cited by 18
“An initial step towards a successful adoption of analytical tools and solutions requires a level of data and algorithmic literacy that can help users understand issues of fairness and bias [62,142].”
BackgroundGender Bias in Machine Translation Systems
Ullmann, S. (2022) · Artificial Intelligence and Its Discontents · doi:10.1007/978-3-030-88615-8_7 · Cited by 7
Softwtre Engineering Ethical Principles based on Islamic Values
Hameed, S. A. (2009) · Journal of Software · doi:10.4304/jsw.4.6.563-570 · Cited by 14
“This way of understanding computer ethics is reflected in recent developments such as Brey's "disclosive computer ethics" methodology [6] and the emerging research field of "value-sensitive computer design" [7, 8 , 9].”
MethodThis Thing Called Fairness: Disciplinary Confusion Realizing a Value in TechnologyPreprint
Mulligan, D. K., Kroll, J. A., Kohli, N., & Wong, R. Y. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.11869 · Cited by 14
“terrogatingthevaluesoftechnicalandsociotechnicalsystems havedeeprootsfrommultipledisciplines,includingscience&technologystudies, law,philosophy, HCI, and computer science & engineering (e.g., [22, 27, 50, 89, 90, 96, 126, 151]). In addition to the critical analysis of understanding values in technical artifact…”
BackgroundAlgorithmic Discrimination in ServicePreprint
Ukanwa, K., & Rust, R. T. (2020) · SSRN Electronic Journal · doi:10.2139/ssrn.3654943 · Cited by 12
Happiness maximizing sets under group fairness constraints
Zheng, J., Ma, Y., Ma, W., Wang, Y., & Wang, X. (2022) · Proceedings of the VLDB Endowment · doi:10.14778/3565816.3565830 · Cited by 12
Barea, M. A., Boeren, D., & Gonçalves, J. (2023) · AI & Society · doi:10.1007/s00146-023-01804-z · Cited by 15
“Independently from the preferred terminology, biased algorithms “systematically and unfairly discriminate against individuals or groups of individuals in favor of others” (Friedman and Nissenbaum 1996, p. 332).”
Silver-Tongued and Sundry: Exploring Intersectional Pronouns with ChatGPT
Fujii, T., Seaborn, K., & Steeds, M. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642303 · Cited by 18
“De-velopers unconsciously embed mental models of the human world, including stereotypes, in their designs [9, 29, 49, 94].”
BackgroundArtificial intelligence for response prediction and personalisation in radiation oncology
Zwanenburg, A., Price, G. J., & Löck, S. (2024) · Strahlentherapie und Onkologie · doi:10.1007/s00066-024-02281-z · Cited by 14
“Biases result in AI systems that perform better in certain populations than in others and can lead to reinforcing existing structures of inequality [57, 58].”
Auditing Flood Vulnerability Geo-Intelligence Workflow for Biases
Masinde, B. K., Gevaert, C. M., Nagenborg, M., den Homberg, M. J. C. V., Margutti, J., Gortzak, I., & Zevenbergen, J. A. (2024) · ISPRS International Journal of Geo-Information · doi:10.3390/ijgi13120419 · Cited by 14
“In addition to using the bias framework by Friedman and Nissenbaum [17] to systematically identify biases that occur in the workflow, we use model cards [19] and datasheets [20], drawing inspiration from the Scoping, Mapping, Artifact Collection, Testing (SMACTR) framework by Raji et al. [21].”
InfluentialYi, F., Singh, A., & Tao, Z. (2024) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000101 · Cited by 13
Zehlike, M., Loosley, A., Jonsson, H., Wiedemann, E., & Hacker, P. (2025) · Artificial Intelligence · doi:10.1016/j.artint.2024.104280 · Cited by 18
Algorithmic Curation and Users’ Civic Attitudes: A Study on Facebook News Feed Results
Papa, V., & Photiadis, T. (2021) · Information · doi:10.3390/info12120522 · Cited by 12
Bias and Fairness in Computer Vision Applications of the Criminal Justice System
Noiret, S., Lumetzberger, J., & Kampel, M. (2021) · IEEE Symposium Series on Computational Intelligence (SSCI) · doi:10.1109/ssci50451.2021.9660177 · Cited by 21
“A rare and thorough analysis of what bias means in the context of algorithms and automated decision making was provided by Friedman and Nissenbaum [6], who consider algorithms as biased, in that they have the potential to ”systematically and unfairly discriminate against certain individuals or…”
BackgroundMethodDesign considerations for a digital service to support prison leavers
Grierson, M., Varghese, D., Bolton, M., & Olivier, P. L. (2022) · Designing Interactive Systems Conference · doi:10.1145/3532106.3533567 · Cited by 14
“While still relatively nascent, sensitive HCI builds on the principles of value-sensitive design, design justice, and participatory design, which propose that an ethical, user-centric, design process is critical to achieving a meaningful design outcome [19, 25, 57].”
BackgroundRespect as a Lens for the Design of AI Systems
Seymour, W., Kleek, M. V., Binns, R., & Murray-Rust, D. (2022) · AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3514094.3534186 · Cited by 18
“While issues of societal bias, discrimination, and fairness in sociotechnical systems have long been studied in a range of disciplines [39, 40, 51], the last decade has seen a significant growth of interest in these topics amongst computer scientists, often borrowing from or in collaboration with other disciplines suc…”
BackgroundThe role of biased data in computerized gender discrimination
Ahmed, M. A., Chatterjee, M., Dadure, P., & Pakray, P. (2022) · doi:10.1145/3524501.3527599 · Cited by 12
“There are 3 types of biases as mentioned by Friedman & Nissenbaum [4] which lead to bias in machine translation (MT): pre-existing bias, technical bias, and emergent bias.”
BackgroundImproving Science That Uses Code
Thimbleby, H. (2023) · The Computer Journal · doi:10.1093/comjnl/bxad067 · Cited by 13
FCJ-189 Reimagining Work: Entanglements and Frictions around Future of Work Narratives
Forlano, L., & Halpern, M. K. (2015) · The Fibreculture Journal · doi:10.15307/fcj.26.189.2015 · Cited by 12
“…socially constructed and mutually constituted (Bijker and Law, 1994; Pinch and Bijker, 1984), embedded in socio-economic and socio-technical systems (Star, 1999), reconfigurations of invisible labour (Latour, 1992) and imbued with society’s values (Friedman and Nissenbaum, 1996; Nissenbaum, 2001).”
BackgroundInglesant, P., Hartswood, M., & Jirotka, M. D. (2016) · Oxford University Research Archive (ORA) (University of Oxford) · Cited by 12
Fairness and Discrimination in Information Access SystemsPreprint
Ekstrand, M. D., Das, A., Burke, R., & Díaz, F. (2021) · arXiv (Cornell University) · Cited by 13
Algorithmic Social Injustices: Antecedents and Mitigations
Tanriverdi̇, H., & Akinyemi, J.-P. O. (2025) · MIS Quarterly · doi:10.25300/misq/2025/18314 · Cited by 15
Leveraging Digital Intelligence for Community Well-Being
Narayan, R. (2020) · International Journal of Community Well-Being · doi:10.1007/s42413-020-00085-4 · Cited by 14
“For instance, Friedman and Nissenbaum (1996) have used actual cases to illustrate how biases in terms of preexisting, technical, and emergent, take shape in computer systems.”
BackgroundStark, L. (2023) · American Literature · doi:10.1215/00029831-10575148 · Cited by 20
The Future of OpenAI Tools: Opportunities and Challenges for Human-AI Collaboration
Kunal, Rana, M., & Bansal, J. (2023) · 2023 2nd International Conference on Futuristic Technologies (INCOFT) · doi:10.1109/incoft60753.2023.10424990 · Cited by 5
Nolin-Lapalme, A., Corbin, D., Tastet, O., Avram, R., & Hussin, J. (2024) · Canadian Journal of Cardiology · doi:10.1016/j.cjca.2024.04.026 · Cited by 12
The role of ETSI in the EU’s regulation and governance of artificial intelligence
Gamito, M. C. (2024) · Innovation The European Journal of Social Science Research · doi:10.1080/13511610.2024.2349627 · Cited by 12
Fairness and Bias in Robot Learning
Londoño, L., Hurtado, J. V., Hertz, N., Kellmeyer, P., Vöeneky, S., & Valada, A. (2024) · Proceedings of the IEEE · doi:10.1109/jproc.2024.3403898 · Cited by 43
“Similarly, in ML and AI, bias refers to when a model systematically and unfairly treats specific individuals or groups of individuals in favor of others [21].”
BackgroundGenerative AI Search Engines as Arbiters of Public Knowledge: An Audit of Bias and Authority
Li, A., & Sinnamon, L. S. (2024) · Proceedings of the Association for Information Science and Technology · doi:10.1002/pra2.1021 · Cited by 37
“Bias in SEs is any type of systematic skew that causes harm (Friedman & Nissenbaum, 1996).”
Helm, P., Michael, L., & Schelenz, L. (2022) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3514094.3534149 · Cited by 11
Stoyanovich, J., Howe, B., Jagadish, H. V., & Miklau, G. (2018) · Proceedings of the VLDB Endowment · doi:10.14778/3229863.3240494 · Cited by 9
“This portion of the panel will be informed by work on bias in computer systems and on algorithmic fairness, including [5, 6, 7, 11, 12, 14, 16, 17, 22, 25, 27].”
Background
Metadata
- Authors
- Friedman, B., & Nissenbaum, H.
- Affiliations
- Colby College (US)
- Princeton University (US)
- Venue
- ACM Transactions on Information Systems
- Published
- 1996-07
- Volume
- 14
- Issue
- 3
- Pages
- 330-347
- ISSN
- 1046-8188 (ISSN-L)
- 1558-2868
- OpenAlex
- W2006447892
- Cited by
- 1,203
- References
- 13
- Citations by year
- 2026: 102 · 2025: 149 · 2024: 151 · 2023: 121 · 2022: 126 · 2021: 133 · 2020: 79 · 2019: 70 · 2018: 50 · 2017: 21 · 2016: 29 · 2015: 18 · 2014: 18 · 2013: 16 · 2012: 8
- FWCI
- 17.25
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- ACM Transactions on Information Systems (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Social Sciences › Business, Management and Accounting › Management Information Systems › Information Systems Theories and Implementation
- Social Sciences › Law › Artificial Intelligence in Law
- Keywords
- technical bias, social institutions, technological constraints, societal impact of technology
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
