CoChat
Sign Up
Cited by 0Ethics and Social Impacts of AIFeminist Epistemology and Gender StudiesOpen Source Software Innovations

Data Set Failures and Intersectional Data

Open access
Stevens, Nikki·2019·Journal of Cultural Analytics
DOI 10.22148/16.041

Published version of a preprint

Open this paper in CoChat — free accountOpen PDF

Abstract

In 2016, a software developer named David and I met to discuss creating a quan- titative demographic survey of the open source software community to which we were both long-time contributors. David and I did not know each other well, but shared a belief that our open source community (OSC, hereafter) was an un- safe place for anyone who did not identify as white, cisgendered, heterosexual and male. That lack of safety was further complicated by any one individual’s distance from privileged modes of contribution.

AI snapshot

Written from the paper’s full text.

TL;DR

This reflexive case study narrates the design, collection, and assurance of a 2016 demographic survey of an open source software community, arguing that the team's intersectional aims were repeatedly undermined by platform choices, quantification's demand for stable identities, and corporate funding. The author concludes that "intersectional data is messy data" and that the team's foundational failure was undertaking human quantification without confronting its ethical and epistemological implications.

Key findings

  • The author identifies six failures across the data lifecycle: (1) using a code-centric platform (GitHub) for survey design, which barred non-developers; (2) forcing respondents to stabilize identity categories, where "primary gender identity" imposed an artificial hierarchy and collapsing trans/intersex questions excluded trans-nonbinary and other trans* identities; (3) including workplace, education, and socioeconomic questions that carried normative, corporate assumptions (one respondent: "assuming that everyone works for a company or business leaves out those of us working in higher (or other) education, government, or non-profit organizations"); (4) letting survey-vendor plugin limits dictate question design, preventing "check all that apply" plus self-identify; (5) accepting corporate advertising money despite rejecting corporate data control; and (6) engaging in an ethically complicated process without considering its implications.
  • During six months of design work there were 108 commits to the GitHub repository by 8 individuals, plus 25 additional people in discussions; the team did not log the race, class, or gender of its own members.
  • After a few months the survey had fewer than 200 respondents; the author's employer provided $20,000 for advertising and Stack Overflow matched it with $20,000 of in-kind advertising, with both companies given access to the data upon release.
  • In the final open question, 179 respondents entered free text (e.g., "Quaker, polyamorous, Russian-speaking Asian, autistic, former fundamentalist, feminist, abuse survivor, liberal, immigrant, depressed"), which the author reads as evidence that the provided identity markers were insufficient.
  • The team deliberately declined to "clean" the data, making no changes to responses before release, as a counterpoint to shaped datasets like those of GitLab and Stack Overflow.

Methods

  • A participant-observer case study of a single survey project: the author co-initiated the project in May 2016, worked with community members on design, and opened the survey for data collection in October 2016.
  • The project is analyzed through the eight-phase data lifecycle (Plan, Collect, Assure, Describe, Preserve, Discover, Integrate, Analyze), with the paper focusing on five major failures in Planning and Collection plus the state of the data at the end of Assuring.
  • Evidence comes from the author's recollections of meetings and review of digital artifacts such as GitHub issue threads; the paper is explicitly situated and subjective rather than objective.

Limitations

  • Stated: the author is "deeply embedded in this story," was a participant observer at every stage, and treats removal from the narrative as an impossibility; observations derive from recollections of conversations and review of digital artifacts.
  • Stated: the paper is "dealing only with a small case study," and examining the content of every survey question is out of scope.
  • Stated: the team did not log the race, class, or gender of survey team members, so the composition of the designing group cannot be assessed.
  • Evident from the text: the account rests on a single author's retrospective memory of a project conducted with others whom she does not speak for, and no response-level data or response rate beyond "fewer than 200 respondents" is reported.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.22148/16.041

Cites

0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 18 cited sources checked · 0 supported · 0 flagged

  • Findingcitation

    We omitted technology bootcamps-a large source of training for non-traditional and later-in-life technologists.

    “In the education section, we asked about an individual's level of educational achievement, and we omitted technology bootcamps—a large source of training for non-traditional and later-in-life technologists.21”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [21]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Findingreasoning

    Maggie's complaint was taken seriously.

    “Maggie's complaint was taken seriously.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: Intersectionality is about power structures and social position.
  • Findingreasoning

    In evaluating the validity of the claims against Tom, Maggie's history of quantifiable contribution was actively considered.

    “In evaluating the validity of the claims against Tom, Maggie's history of quantifiable contribution was actively considered.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: Intersectionality is about power structures and social position.
  • Findingnone given

    I met many folks who wanted to participate but could not (or did not want to) navigate the Github onboarding process.

    “I met many folks who wanted to participate but could not (or did not want to) navigate the Github onboarding process.”
    Quote in the paperSubstantial claim
  • Findingreasoning

    We excluded trans-nonbinary and other trans* identities.

    “As a result, we excluded trans-nonbinary and other trans* identities.”
    Quote in the paperSubstantial claim
  • Classified as probably not a substantial claim. 39
  • May combine two claims. 14
  • The quote only partly states this claim. 1

  • Probably not a substantial claim: may not pass S4 (based).

    Findingnone given

    By choosing a code-centric platform, despite the goal of validating and encouraging non-code participation, we erected barriers to participation in the planning stage.

    “Failure #1: By choosing a code-centric platform, despite the goal of validating and encouraging non-code participation, we erected barriers to participation in the planning stage.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Findingnone given

    For sexual orientation, we included checkboxes, but still demanded that the respondent explore their definition of the word queer in order to answer the question.

    “For sexual orientation, we included checkboxes, but still demanded that the respondent explore their definition of the word queer in order to answer the question.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning

    Individuals in the community gave Maggie's complaint more weight not only because of her race and gender, but because she was seen as a valuable code contributor.

    “Individuals in the community gave Maggie's complaint more weight not only because of her race and gender, but because she was seen as a valuable code contributor.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: Intersectionality is about power structures and social position.

  • May combine two claims.

    Findingown data

    The community largely ignored Alex's complaint and continued to encourage Tom's participation.

    “The community largely ignored Alex's complaint (despite others coming forward to support and offer similar experiences with Tom) and continued to encourage Tom's participation.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: Intersectionality is about power structures and social position.

  • Probably not a substantial claim: may not pass S4 (based).

    Findingnone given

    By asking users to stabilize aspects of their identity, our questions acted as barriers to intersectional representations of an individual.

    “Failure #2: By asking users to stabilize aspects of their identity, our questions acted as barriers to intersectional representations of an individual.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Findingreasoning

    The authors did not draw a discrete line between the documenters and the documented and as a result did not understand the complexities of counting humans or that the classification of individuals is at the heart of social control.

    “We did not draw a discrete line between the documenters and the documented and as a result, did not understand the complexities of counting humans or that”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [29]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim. May combine two claims.

    Findingnone given

    The responsibility of designing and asking demographic questions, the power dynamics inherent in positioning the survey as by and for open source, and the epistemological challenges of representing individuals in numbers were all lost on the authors at the time.

    “The responsibility of designing and asking demographic questions, the power dynamics inherent in positioning the survey as by and for open source, the epistemological challenges of representing individuals in numbers-all lost on us at the time.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Findingnone given

    The authors were in a uniquely privileged position to be able to start the survey project.

    “While we may have realized that we were in a uniquely privileged position to be able to start the survey project”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Findingnone given

    None of the authors had considered the implications of undertaking such a survey.

    “however none of us had considered the implications of undertaking such a survey.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    Unlike Stack Overflow's closed and restricted approach, the authors did all of their work in the open.

    “Unlike Stack Overflow's closed and restricted approach, we would be doing all of our work in the open.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Findingnone given

    The authors did not have a formal conversation about the nuances of collecting data about people and the responsibilities they might have for the data at every phase of the lifecycle.

    “we did not have a formal conversation about the nuances of collecting data about people and the responsibilities we might have for the data at every phase of the lifecycle.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Findingnone given

    The authors' data imaginary contained amorphous representations of individuals and their many identities.

    “Our data imaginary contained amorphous representations of individuals and their many identities.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingreasoning

    The authors engaged in an ethically complicated process without considering the implications.

    “Failure #6: We engaged in an ethically complicated process without considering the implications.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingreasoning

    In our focus to ensure that we would be able to easily share the data, we made user interface compromises.

    “In our focus to ensure that we would be able to easily share the data, we made user interface compromises.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingreasoning

    From its inception, the authors positioned the survey as counter to the Stack Overflow surveys.

    “From its inception, we positioned the survey as counter to the Stack Overflow surveys.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingreasoning

    We allowed the limitations of media to determine the designs of our questions.

    “Failure #4: We allowed the limitations of media to determine the designs of our questions.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

    Interpretationreasoning

    The expectation that contribution and individual value are quantifiable results in the erasure of work that is not easily measurable and devalues the individuals producing that work.

    “This expectation results in the erasure of work that is not easily measurable and devalues the individuals producing that work.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

    Interpretationreasoning

    Ranking contributors by their lines of code erases the other important ways that people contribute to a software product and elides the relationality inherent in creating a project with others.

    “Second, ranking contributors by their lines of code erases the other important ways that people contribute to a software product and elides the relationality inherent in creating a project with others.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based). May combine two claims.

    Interpretationreasoning

    All of this work directly contributes to the success of the project, but again is not so easily quantifiable.

    “All of this work directly contributes to the success of the project, but again is not so easily quantifiable.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Study design and conductnone given

    Observations about the survey shared below come from my recollections of conversations in meetings and reviewing digital artifacts like Github issue threads.

    “Observations about the survey shared below come from my recollections of conversations in meetings and reviewing digital artifacts like Github issue threads.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    We included questions about a respondent's workplace, size of workplace, and main business function, invoking capitalist expectations that an individual's work should serve a profit-making enterprise.

    “We included questions about a respondent's workplace, size of workplace, and main business function, invoking capitalist expectations that an individual's work should serve a profit-making enterprise.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    Our backgrounds as developers and our long experience with collaborative tools like Github blinded us to the fact that for many people working in technology, Github is not a necessary tool.

    “Our backgrounds as developers and our long experience with collaborative tools like Github blinded us to the fact that for many people working in technology, Github is not a necessary tool.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based). May combine two claims.

    Premisereasoning

    In order to be successful, software projects need documentation, support forums, outreach individuals, community managers.

    “In order to be successful, software projects need documentation, support forums, outreach individuals, community managers.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    We did not log the race, class or gender of members of the survey team.

    “We did not log the race, class or gender of members of the survey team.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    Without an intersectional framework, we were unable to account for all of the ways the community had done a disservice to Alex and their experience.

    “Without an intersectional framework, we were unable to account for all of the ways the community had done a disservice to Alex and their experience.”
    Quote in the paperNot a substantial claim
    • The authors state this as an assumption: Intersectionality is about power structures and social position.

  • Probably not a substantial claim: may not pass S2 (checkable).

    Premisecitation

    We need intersectionality to examine the racialized and gendered ways that knowledge is produced, not just on the internet, but also within open source communities.

    “I further Safiya Noble's assertion that we need intersectionality to examine the racialized and gendered ways that knowledge is produced, not just on the internet, but also within open source communities.15”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [15]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    This elision does violence to all of the contributors whose labor is not so easily quantifiable.

    “This elision does violence to all of the contributors whose labor is not so easily quantifiable.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Premisenone given

    Intersectionality is an analytical framework that ultimately is about exposing power structures and systemic inequalities.

    “Rooted in a history of black feminist scholarship, intersectionality is an analytical framework that ultimately is about exposing power structures and systemic inequalities.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

    Premisecitation

    Intersectional research often deals with complexity, but it also centers those from marginalized groups and addresses social inequality, relationality and social context.

    “Intersectional research often deals with complexity, but it also centers those from marginalized groups and addresses social inequality, relationality and social context.13”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [13]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationcitation

    Given the reproduction of external oppressions in open source communities, and the violent force of meritocratic ideology, open source communities are sites ripe for intersectional examination.

    “Given the reproduction of external oppressions in open source communities, and the violent force of meritocratic ideology, 14 open source communities are sites ripe for intersectional examination.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [14]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • The authors state this as an assumption: OSC was an unsafe place for anyone who did not identify as white, cisgendered, heterosexual and male.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Numerous works assert the importance of having a system for cleaning data and connect clean data with quality data.

    “Numerous works 24 assert the importance of having a system for cleaning data, 23 DataOne.and make a connection between”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [24]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [23]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [25]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S2 (checkable).

    Premisecitation

    Before data can be planned, before it exists as a data point, it exists as an imaginary.

    “Before data can be planned, before it exists as a data point, it exists as an imaginary.30”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [30]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationcitation

    Data as an abstract concept cannot be clean or dirty.

    “However, if we consider that data is abstract and conceptual, 26 then as a concept it cannot be”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [26]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    The team centered expectations that technology most often acts as a tool and not a barrier.

    “This is another example of the team centering expectations that technology most often acts as a tool and not a barrier.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    The authors' choice not to clean the data they obtained was informed by the argument that a focus on data cleanliness controls valid knowledge and is counter to intersectional aims.

    “This informed our choice not to”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    The authors' dataset stands as a counterpoint to the shaped, cleaned, interpreted data produced by surveys like GitLab and Stack Overflow.

    “In this way, I believe that our dataset stands as a counterpoint to the shaped,”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [28]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S2 (checkable). May combine two claims. The quote only partly states this claim.

    Interpretationreasoning

    A focus on data's cleanliness is a way of controlling which knowledge is valid and is directly counter to intersectional aims.

    “I am arguing that a focus on data's cleanliness is a way of controlling which knowledge is”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    The answers to the final survey question reinforced the importance of self-identify boxes and showed that the identity markers provided were insufficient.

    “These answers reinforced for us the importance of self-identify boxes and the ways that the identity markers we provided were insufficient.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    Scholars have expanded the criteria by which we can determine if a research method is intersectional but make no specific mention of whether the artifacts resulting from research can be intersectional themselves.

    “Scholars have expanded the criteria by which we can determine if a research method (qualitative or quantitative) is intersectional 31 but make no specific mention of whether or not the artifacts resulting from research can be intersectional themselves.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [31]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S2 (checkable).

    Premisecitation

    Intersectional data is messy data.

    “Intersectional data is messy data.27”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [27]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Interpretationreasoning

    Ultimately, we were complicit in the gathering of data for corporate interests, despite rationalizing that our involvement with corporate funding was in service of our larger project to benefit the community.

    “Ultimately, we were complicit in the gathering of data for corporate interests, despite rationalizing that our involvement with corporate funding was in service of our larger project to benefit the community.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationreasoning

    Data as a material object or collection of objects cannot be inherently clean or dirty, but can only violate an external standard.

    “If we consider data as a material object, or collection of objects, it similarly cannot be inherently clean or dirty, but simply in violation of an external standard.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    In the assuring phase of the data lifecycle, data quality is assured through checks and inspections, including quality checks and transformations to ensure data is recorded in a particular way and meets certain standards.

    “In the assuring phase,”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [23]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

43 of 43

  • The authors state this as an assumption.

    OSC was an unsafe place for anyone who did not identify as white, cisgendered, heterosexual and male.

    “David and I did not know each other well, but shared a belief that our open source community (OSC, hereafter) was an unsafe place for anyone who did not identify as white, cisgendered, heterosexual and male.”

  • The authors state this as an assumption.

    Knowledge produced by the community should belong to the community.

    “We viewed our open source community not only as a software production system, but a knowledge production network and believed that knowledge produced by the community should belong to the community.”

  • The authors state this as an assumption.

    Intersectionality is about power structures and social position.

    “As previously mentioned, the survey team understood intersectionality to be about power structures and social position.”

  • The paper relies on this without support.

    In open source communities, there is no reason to wait for an invitation to do things, and leadership and opportunity are rarely given but taken by those with the time and inclination to do the work.

    “In OSC, it was emphasized that there was no reason to ever wait for an invitation to do things and that leadership and opportunity were rarely given, but taken by those with the time and inclination to do the work.”
4 of 4

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Survey opening year differs by 90 years

    Minor

    The methodology says the survey opened for data collection in October 2016, but the collecting section gives October 2106.

1 of 1

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 1

  1. Lisa Gitelman, Raw Data Is an Oxymoron (MIT Press, 2013).

Cited by

Still being checked.

Related

Authors
Stevens, N.
Corresponding author
Stevens, Nikki
ORCID
Stevens, Nikki
Venue
Journal of Cultural Analytics
Publisher
CA: Journal of Cultural Analytics
Published
2019-06-12
Volume
4
Issue
1
ISSN
2371-4549 (ISSN-L)
DOI
10.22148/16.041
OpenAlex
W2951029771
Cited by
0
References
0
FWCI
0.00
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • Journal of Cultural Analytics (publishedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
Topics
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Social Sciences › Sociology and Political Science › Feminist Epistemology and Gender Studies
  • Physical Sciences › Computer Science › Computer Science Applications › Open Source Software Innovations
Keywords
intersectionality, open source communities, demographic survey, white privilege, cisnormativity, open source software

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-11-04.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-11-04.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review