Cited by 1,203Ethics and Social Impacts of AIInformation Systems Theories and ImplementationArtificial Intelligence in Law

Bias in computer systems

Open access
Friedman, Batya, Nissenbaum, Helen·1996·ACM Transactions on Information Systems

Abstract

From an analysis of actual cases, three categories of bias in computer systems have been developed: preexisting, technical, and emergent. Preexisting bias has its roots in social institutions, practices, and attitudes. Technical bias arises from technical constraints of considerations. Emergent bias arises in a context of use. Although others have pointed to bias inparticular computer systems and have noted the general problem, we know of no comparable work that examines this phenomenon comprehensively and which offers a framework for understanding and remedying it. We conclude by suggesting that freedom from bias should by counted amoung the select set of criteria—including reliability, accuracy, and efficiency—according to which the quality of systems in use in society should be judged.

Integrity signals

Registry

Crossref · 10.1145/230538.230561

Cites

14 of 14 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

6 of 22 cited sources checked · 4 supported · 0 flagged · accuracy not yet measured

  • Premisecitation

    Section 50-(9) of the British Nationality Act, implemented in BNAP, specifies that a man is the father of only his legitimate children, whereas a woman is the mother of all her children, legitimate or not.

    “As specified in Section 50-(9) of the British Nationality Act itself and imple-mented in BNAP,”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Sergot et al. 1986 source
  • Premisecitation
    § Bias in Computer Systems

    The original NRMP Admissions Algorithm placed married couples at a disadvantage in achieving an optimal placement compared with their single peers.

    “At this point, it was discovered that the original Admissions Algorithm placed married couples at a disadvantage in achieving an optimal placement as compared with their single peers [Roth 1984;1990].”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites Roth 1984 source

      Section IV, “Contemporary Issues” (page not supplied), says the couples algorithm used until 1983 could not accept preferences over pairs of positions and could produce outcomes unstable with respect to couples. That supports a disadvantage for couples, but the paper does not establish that couples achieved less optimal placements than single applicants. (LLM, conf=0.84)

      Experimental — accuracy not yet measured
    • cites Roth 1990 source
  • Premisereasoning

    The MLSA gives processing attention to all processes beginning a new command as quickly as possible.

    “In this manner the MLSA gives processing attention to all processes as quickly as possible that are beginning a new command.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Corbato et al. 1962 source
  • Premisecitation

    At the time of BNAP's initial implementation (1983), no mechanism was built into the program to incorporate relevant case law as it came into being.

    “At the time of the BNAP's initial implementation (1983) no mechanism was built into the program to incorporate relevant case law as it came into being [Sergot et al. 1986].”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Sergot et al. 1986 source
  • Findingnone given
    § Bias in Computer Systems

    The NRMP Admissions Algorithm does not take into account geographic distribution when determining a match, considering only the ranked preferences of hospitals and students.

    “The Admissions Algorithm does not take into account geographic distribution when determining a match, considering only the ranked preferences of hospitals and students.”
    Quote in the paperSubstantial claim
  • Classified as probably not a substantial claim. 18
  • May combine two claims. 15
  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    The BNAP authors state their goal is to translate the British Nationality Act so that the consequences of the act can be determined mechanistically, and they place their work in the context of other legal expert systems designed for use with real users.

    “For in the same text the authors also state (as noted above) that their goal is to translate the British Nationality Act so that”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Sergot et al. 1986 source

      Split into 2 atoms — 1 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

      Experimental — accuracy not yet measured
  • Probably not a substantial claim: may not pass S1 (asserted), S3 (load-bearing).

    Premisecitation

    The NRMP Admissions Algorithm systematically favors hospital programs over medical students in cases of conflict.

    “One charges that the Admissions Algorithm systematically favors hospital programs over medical students in cases of conflict [Graettinger and Peranson 1981b;Roth 1984;Sudarshan and Zisook 1981;Williams et al. 1981].”
    Quote in the paperNot a substantial claim1 of 3 sources checked
    • cites Graettinger and Peranson 1981b

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Roth 1984 source
    • cites Williams et al. 1981 source
  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationnone given

    Technical bias arises from technical constraints or considerations.

    “Technical bias arises from technical constraints or considerations.”
    • Also stated in: § 2.1 Preexisting Bias · plain
      “Technical bias arises from technical constraints or technical considerations.”
    Quote in the paperNot a substantial claim
  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationnone given

    Preexisting bias has its roots in social institutions, practices, and attitudes.

    “Preexisting bias has its roots in social institutions, practices, and attitudes.”
    • Also stated in: § INTRODUCTION · plain
      “Preexisting bias has its roots in social institutions, practices, and attitudes.”
    Quote in the paperNot a substantial claim
  • May combine two claims.

    Interpretationreasoning
    § Bias in Computer Systems

    The NRMP is not biased against rural hospitals in favor of urban ones because the discrimination reflects the preferences of match participants and has a reasonable basis.

    “In this case, we ourselves would shy away from viewing this as a bias in the system because we think this discrimination can be defended as having a reasonable basis.Namely, the discrimination reflects the preferences of match participants, and it is reasonable in our view for employment decisions to be determined largely by the choices of employers and employees.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The discrimination against rural hospitals reflects the preferences of match participants.
    • Rests on an assumption: The best teaching hospitals tend to be in urban areas, and urban areas tend to fill their positions far more effectively, and with better students, than rural hospitals.
  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    Emergent bias poses a potential problem for any legal expert system, especially in a society where legal systems depend on evolving case law.

    “Emergent bias poses a potential problem for any legal expert system, especially in a society where legal systems depend on evolving case law.”
    Quote in the paperNot a substantial claim
  • May combine two claims.

    Interpretationreasoning

    In this case the biased designs were unintended and reflected gender-biases deeply rooted in the culture at large.

    “We thus infer that in this case the biased designs were unintended and, instead, reflected gender-biases deeply rooted in the culture at large.”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Interpretationreasoning

    Biases in computer systems can be difficult to identify and remedy because computer systems are inexpensive to disseminate, can become standards, can hide biases in complex code, and offer no equivalent means for appeal.

    “As Sabre and Apollo illustrate, biases in computer systems can be difficult to identify let alone remedy because of the way the technology engages and extenuates them.Computer systems, for instance, are comparatively inexpensive to disseminate, and thus, once developed, a biased system has the potential for widespread impact.If the system becomes a standard in the field, the bias becomes pervasive”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Premisecitation

    Huff and Cooper report that along a number of dimensions the designs proposed by subjects in the gender-unspecified group closely resembled the designs proposed by subjects who designed for boys and were significantly different from the designs proposed by subjects who designed for girls.

    “Huff and Cooper report that along a number of dimensions the designs proposed by subjects in the gender-unspecified group closely resembled the designs proposed by subjects who designed for boys and were significantly different from the designs proposed by subjects who designed for girls.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Huff and Cooper 1987 source
  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisereasoning

    One source of bias in Sabre and Apollo lies in their algorithms for controlling search and display functions, which give preference to on-line flights.

    “One source of this alleged bias lies in Sabre's and Apollo's algorithms for controlling search and display functions.In the algorithms, preference is given to”
    Quote in the paperNot a substantial claim
  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    A shift in context of use may well create difficulties for a new set of users.

    “Thus, a shift in context of use may well create difficulties for a new set of users.”
    Quote in the paperNot a substantial claim
  • May combine two claims.

    Premisecitation

    Sabre and Apollo systematically downgrade and are biased against international carriers who fly few, if any, internal U.S. flights, and against internal carriers who do not fly international flights.

    “Thus, the computer systems systematically downgrade and, hence, are biased against international carriers who fly few, if any, internal U.S. flights, and against internal carriers who do not fly international flights [Fotos 1988;Ott 1988].”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites Fotos 1988

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Ott 1988

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    Emergent bias typically emerges some time after a design is completed, as a result of changing societal knowledge, population, or cultural values.

    “This bias typically emerges some time after a design is completed, as a result of changing societal knowledge, population, or cultural values.”
    Quote in the paperNot a substantial claim
  • May combine two claims.

    Premisecitation

    The interface design compounds the bias in reservation systems because lists of ranked flight options are displayed screen by screen, and 90% of tickets booked by travel agents are booked by the first screen display.

    “One is that the interface design compounds the bias in the reservation systems.Lists of ranked flight options are displayed screen by screen.Each screen displays only two to five options.The advantage to a carrier of having its flights shown on the first screen is enormous since 90% of the tickets booked by travel agents are booked by the first screen display [Taib 1990].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Taib 1990

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Interpretationreasoning

    Technical bias with the MLSA is embedded in the program, but the bias is a phenomenon of the system in use, in a context wherein users with short and long jobs outpace the system's capacity.

    “Or consider again the case of technical bias with the MLSA (which favors users with short jobs over long jobs).While technical bias is embedded in the program, the bias is a phenomenon of the system in use, in a context wherein users with short and long jobs outpace the system's capacity.”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Premisecitation

    In the Huff and Cooper study, subjects were asked to propose designs for software to teach seventh graders the correct use of commas; one group designed for seventh-grade boys, a second for seventh-grade girls, and a third for seventh-graders with gender unspecified.

    “In their study, subjects were asked to propose designs for software to teach seventh graders the correct use of commas.One group of subjects was asked to design the software for seventh-grade boys, the second group to design for seventh-grade girls, and the third group to design for seventh-graders, gender unspecified.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Huff and Cooper 1987 source
  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Interpretationnone given

    Preexisting bias can enter a system either through explicit and conscious efforts of individuals or institutions, or implicitly and unconsciously, even in spite of the best of intentions.

    “This type of bias can enter a system either through the explicit and conscious efforts of individuals or institutions, or implicitly and unconsciously, even in spite of the best of intentions.”
    Quote in the paperNot a substantial claim
  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationnone given

    Sources of technical bias include limitations of computer tools such as hardware, software, and peripherals; the process of ascribing social meaning to algorithms developed out of context; imperfections in pseudorandom number generation; and the attempt to make human constructs amenable to computers.

    “Sources of technical bias can be found in several aspects of the design process, including limitations of computer tools such as hardware, software, and peripherals; the process of ascribing social meaning to algorithms developed out of context; imperfections in pseudorandom number generation; and the attempt to make human constructs amenable to computers, when we quantify the qualitative, discret”
    Quote in the paperNot a substantial claim
  • Classified as probably not a substantial claim.

    Premisecitation

    The NRMP algorithm's preference for hospital programs in cases of conflict duplicates what happens in an actual admissions process without a computerized matching program.

    “The constraint inherent in the NRMP algorithm, in which preference is given to hospital choices when conflicts in rankings occur, duplicates what happens in an actual admissions process without a computerized matching program”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Graettinger and Peranson 1981b

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationnone given
    § Bias in Computer Systems

    Whatever ranking algorithm is used, if it systematically places certain airlines' flights on initial screens and other airlines' flights on later screens, the system will exhibit technical bias.

    “Whatever ranking algorithm is used, if it systematically places certain airlines' flights on initial screens and other airlines' flights on later screens, the system will exhibit technical bias.”
    Quote in the paperNot a substantial claim
  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given

    Preexisting biases can reflect the personal biases of individuals who have significant input into the design of the system, such as the client or system designer.

    “They can also reflect the personal biases of individuals who have significant input into the design of the system, such as the client or system designer.”
    Quote in the paperNot a substantial claim
  • May combine two claims.

    Premisecitation

    Those responsible for maintaining the NRMP responded conscientiously to this societal change and modified the system's algorithm to place couples more fairly.

    “those responsible for maintaining the system responded conscientiously to this societal change and modified the system's algorithm to place couples more fairly [Roth 1990].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Roth 1990 source
  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Interpretationnone given

    Preexisting biases may originate in society at large, in subcultures, and in formal or informal, private or public organizations and institutions.

    “Preexisting biases may originate in society at large, in subcultures, and in formal or informal, private or public organizations and institutions.”
    Quote in the paperNot a substantial claim
  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationcitation

    Should the accumulation of case law lead to changes in the way the Act is interpreted, such as by granting new subgroups British citizenship, BNAP would systematically misinform members of this subgroup regarding their status as British citizens.

    “Should the accumulation of case law lead to changes in the way the Act is interpreted-say, by granting new subgroups British citizenship-BNAP would systematically misinform members of this subgroup regarding their status as British citizens [Leith 1986].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Leith 1986 source
  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    In total, 17 computer systems from diverse fields including banking, commerce, computer science, education, medicine, and law were examined.

    “In total, we examined 17 computer systems from diverse fields including banking, commerce, computer science, education, medicine, and law.”
    Quote in the paperNot a substantial claim
  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductown data

    The framework was derived by examining actual computer systems for bias; instances were identified and characterized according to source, generalized to abstract categories, and refined by application to other instances.

    “We derived our framework by examining actual computer systems for bias.Instances of bias were identified and characterized according to their source, and then the characterizations were generalized to more abstract categories.These categories were further refined by their application to other instances of bias in the same or additional computer systems.”
    Quote in the paperSubstantial claim
  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    The presence of bias is not so much a feature inherent in the system independent of the context of use, but an aspect of a system in use.

    “although we have talked about bias in computer systems, the presence of bias is not so much a feature inherent in the system independent of the context of use, but an aspect of a system in use.”
    Quote in the paperNot a substantial claim
27 of 27
  • The paper relies on this without support.

    In cases of conflict, there is no good rationale for favoring hospital programs at the expense of medical students.

    “We are inclined to answer yes because in cases of conflict there does not appear to be a good rationale for favoring hospital programs at the expense of students.”
  • The paper relies on this without support.

    The discrimination against rural hospitals reflects the preferences of match participants.

    “Namely, the discrimination reflects the preferences of match participants, and it is reasonable in our view for employment decisions to be determined largely by the choices of employers and employees.”
  • The paper relies on this without support.

    The 17 computer systems examined are representative enough to generalize the framework to computer systems generally.

    “In total, we examined 17 computer systems from diverse fields including banking, commerce, computer science, education, medicine, and law.”
  • The paper relies on this without support.

    The illustrative examples in Table I portray plausible cases of bias.

    “The illustrative examples portray plausible cases of bias.”
  • The paper relies on this without support.

    The best teaching hospitals tend to be in urban areas, and urban areas tend to fill their positions far more effectively, and with better students, than rural hospitals.

    “Because the best teaching hospitals tend to be in urban areas, the urban areas tend to fill their positions far more effectively, and with better students, than rural hospitals.”
  • The authors state this as an assumption.

    The MLSA system is not saturated with too many users.

    “assuming the system is not saturated with too many users”
  • The paper relies on this without support.

    BNAP was designed in a research environment among people with sophisticated knowledge of immigration law.

    “First, the system was designed in a research environment, among people with sophisticated knowledge of immigration law.”
  • The paper relies on this without support.

    BNAP's users are likely to be at best paralegal or immigration counselors in Britain, if not lay persons in foreign countries with only limited access to British legal expertise.

    “Its users, however, are likely to be at best paralegal or immigration counselors in Britain, if not lay persons in foreign countries with only limited access to British legal expertise.”
  • The paper relies on this without support.

    Some of BNAP's queries require expert knowledge to answer.

    “Some of the program's queries, for example, require expert knowledge to answer.”
  • The paper relies on this without support.

    Nonexperts advising British citizen hopefuls are not alerted to alternative legal frameworks that complement the British Nationality Act.

    “More generally, nonexperts advising British citizen hopefuls are not alerted to alternative legal frameworks that complement the British Nationality Act.”
10 of 10

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • HELM's multi-metric approach makes potential trade-offs explicit and helps prevent nonaccuracy desiderata from being treated as second-class.

    “The multi-metric approach (Figure S1) makes explicit potential trade-offs and helps ensure the nonaccuracy desiderata are not treated as second-class citizens to accuracy (see Ref. 13).”

    From Holistic Evaluation of Language Models (2023) · cites this paper as [13]

    The Conclusion argues that freedom from bias should be a criterion for judging computer systems alongside accuracy, and Section 4 describes weighing ease of access against equity of access. These passages support taking nonaccuracy concerns and trade-offs seriously, but the paper does not assess HELM’s multi-metric approach or show that it makes those trade-offs explicit. (LLM, conf=0.82)

    Experimental — accuracy not yet measured
  • Research has repeatedly shown that ADM systems can encode biases into decisions.

    “research has repeatedly shown that these systems can in fact encode biases into decisions 124 .”

    From Administrative law and the machines of government: judicial review of automated public-sector decision-making (2019) · cites this paper as B Friedman and H Nissenbaum 'Bias in Computer Systems' (1996)

  • The risk of emergent bias, by which a tool produces biased outputs through its own functioning, is documented in the literature.

    “The risk of”
  • Bias and opacity in computer systems have been exacerbated by the rapidly expanding numbers of data-driven information systems entering decision-making processes.

    “Bias or opacity are far from newly observed characteristics of computer systems generally [26, 11, 24], but these issues have been exacerbated by the rapidly expanding numbers of data-driven information systems entering decision-making processes.”
  • AI-driven decisions may be based on factors of ethical importance (e.g., ethnic, gender, or religious grounds) and irrelevant to the decision-making at hand, or they may be relevant but legally protected as a nondiscriminatory characteristic.

    “These decisions may be based on factors of ethical importance (e.g., ethnic, gender, or religious grounds) and irrelevant to the decision-making at hand, or they may be relevant but legally protected as a nondiscriminatory characteristic (Friedman and Nissenbaum 1996).”

    From How to Design AI for Social Good: Seven Essential Factors (2020) · cites this paper as Friedman and Nissenbaum 1996

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Table II gives conflicting years for its source

    Minor

    The table caption dates the Williams et al. source to 1991, while the paper's reference entry for that work dates it to 1981.

1 of 1

Cites

Flagged first, then by citations

Research works printed but not linked: 4

  1. FISHLOCK , T. 1981. Delhi press detect racism in Nationality Bill. The London Times ( Jan. 20 ) . FISHLOCK, T. 1981. Delhi press detect racism in Nationality Bill. The London Times (Jan. 20).
  2. GAO . 1992. Patriot Missile defense: Software problem led to system failure at Dhahran , Saudi Arabia. GAO/IMTEC-92-26, U.S. General Accounting Office, Washington, D.C. GAO. 1992. Patriot Missile defense: Software problem led to system failure at Dhahran, Saudi Arabia. GAO/IMTEC-92-26, U.S. General Accounting Office, Washington, D.C.
  3. JOHNSON , D.G. AND MULVEY , J. M. 1993. Computer decisions: Ethical issues of responsibil-ity and bias. Statistics and Operations Res. Series SOR-93-11 , Dept. of Civil Engineering and Operations Research , Princeton Univ ., Princeton, N.J. JOHNSON,D.G.AND MULVEY, J. M. 1993. Computer decisions: Ethical issues of responsibil-ity and bias. Statistics and Operations Res. Series SOR-93-11, Dept. of Civil Engineering and Operations Research, Princeton Univ., Princeton, N.J.
  4. OTT , J. 1988. American Airlines settles CRS dispute with British Airways. Aviat. Week Space Tech. ( July 18 ) . OTT, J. 1988. American Airlines settles CRS dispute with British Airways. Aviat. Week Space Tech. (July 18).

Cited by

Sorted by citations

Authors
Friedman, B., & Nissenbaum, H.
Affiliations
  • Colby College (US)
  • Princeton University (US)
Venue
ACM Transactions on Information Systems
Published
1996-07
Volume
14
Issue
3
Pages
330-347
ISSN
  • 1046-8188 (ISSN-L)
  • 1558-2868
OpenAlex
W2006447892
Cited by
1,203
References
13
Citations by year
2026: 102 · 2025: 149 · 2024: 151 · 2023: 121 · 2022: 126 · 2021: 133 · 2020: 79 · 2019: 70 · 2018: 50 · 2017: 21 · 2016: 29 · 2015: 18 · 2014: 18 · 2013: 16 · 2012: 8
FWCI
17.25
Type
journal article
Language
en
Open access
Locations
ACM Transactions on Information Systems (publishedVersion)
Topics
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Social Sciences › Sociology and Political Science › Information Systems Theories and Implementation
  • Social Sciences › Political Science and International Relations › Artificial Intelligence in Law
Keywords
technical bias, social institutions, technological constraints, societal impact of technology

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-10.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-10.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review