CoChat
Sign Up
PreprintCited by 36Explainable Artificial Intelligence (XAI)Ethics and Social Impacts of AIScientific Computing and Data Management

ABOUT ML: Annotation and Benchmarking on Understanding and Transparency of Machine Learning Lifecycles

Open access
Raji, Inioluwa Deborah, Yang, Jingying·2019·arXiv
DOI 10.48550/arxiv.1912.06166arXiv 1912.06166
Open this paper in CoChat — free accountOpen PDF

Abstract

We present the "Annotation and Benchmarking on Understanding and Transparency of Machine Learning Lifecycles" (ABOUT ML) project as an initiative to operationalize ML transparency and work towards a standard ML documentation practice. We make the case for the project's relevance and effectiveness in consolidating disparate efforts across a variety of stakeholders, as well as bringing in the perspectives of currently missing voices that will be valuable in shaping future conversations. We describe the details of the initiative and the gaps we hope this project will help address.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1912.06166

Cites

8 of 8 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Third-party auditors provide independent certification of robustness and security, which serves as an external driving force to the fulfillment of trustworthiness requirements [99, 64].

    “Besides direct participants in the system, third-party auditors provide independent certification of robustness and security, which serves as an external driving force to the fulfillment of trustworthiness requirements [99, 64].”

    From Trustworthy AI: From Principles to Practices (2023) · cites this paper as [64]

    The paper says outside auditing can help hold AI developers accountable (Demand for Transparency in ML Systems, p. 2), and that external documentation can help auditing organizations maintain trust (Transparency Through Documentation, p. 2). It does not say third-party auditors provide independent certification of robustness or security. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • In the context of adversarial attacks, the minimal adversarial perturbation is a core metric of robustness, and its empirical upper bound on a test dataset has been widely used.

    “In the context of adversarial attacks, the minimal adversarial perturbation is a core metric of robustness, and its empirical upper bound, a.k.a. empirical robustness, on a test dataset has been widely used [312, 64].”

    From Trustworthy AI: From Principles to Practices (2023) · cites this paper as [64]

    The Introduction (p. 1) describes the paper as an approach to ML transparency through documentation. Searches for minimal adversarial perturbation, empirical robustness, and adversarial attacks found no passage in our copy addressing the claimed robustness metric or its use on test datasets. (LLM, conf=0.75)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 11

  1. [3][3] Misha Benjamin, Paul Gagnon, Negar Rostamzadeh, Chris Pal, Yoshua Bengio, and Alex Shee. Towards standardization of data licenses: The montreal data license. arXiv preprint arXiv:1903.12262 , 2019. - · arXiv
  2. [7][7] Sarah Holland, Ahmed Hosny, Sarah Newman, Joshua Joseph, and Kasia Chmielinski. The dataset nutrition label: A framework to drive higher data quality standards. arXiv preprint arXiv:1805.03677 , 2018. - · arXiv
  3. [8][8] Kenneth Holstein, Jennifer Wortman Vaughan, Hal Daumé III, Miro Dudik, and Hanna Wallach. Improving fairness in machine learning systems: What do industry practitioners need? In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems , page 600. ACM, 2019. -
  4. [9][9] Anna Jobin, Marcello Ienca, and Effy Vayena. Artificial intelligence: the global landscape of ethics guidelines. arXiv preprint arXiv:1906.11668 , 2019. - · arXiv
  5. [10][10] Patrick Gage Kelley, Joanna Bresee, Lorrie Faith Cranor, and Robert W Reeder. A nutrition label for privacy. In Proceedings of the 5th Symposium on Usable Privacy and Security , page 4. ACM, 2009. -
  6. [11][11] Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. Model cards for model reporting. In Proceedings of the Conference on Fairness, Accountability, and Transparency , pages 220-229. ACM, 2019. -
  7. [14][14] Andrew D Selbst, Danah Boyd, Sorelle A Friedler, Suresh Venkatasubramanian, and Janet Vertesi. Fairness and abstraction in sociotechnical systems. In Proceedings of the Conference on Fairness, Accountability, and Transparency , pages 59-68. ACM, 2019. -
  8. [15][15] Andrea L Thomaz and Cynthia Breazeal. Transparency and socially guided machine learning. In 5th Intl. Conf. on Development and Learning (ICDL) , 2006. -
  9. [16][16] Michael Veale. Logics and practices of transparency and opacity in real-world applications of public sector machine learning. arXiv preprint arXiv:1706.09249 , 2017. - · arXiv
  10. [18][18] Meg Young, Lassana Magassa, and Batya Friedman. Toward inclusive tech policy design: a method for underrepresented voices to strengthen tech policy documents. Ethics and Information Technology , 21(2):89-103, 2019. -
  11. [19][19] Yi Zeng, Enmeng Lu, and Cunqing Huangfu. Linking artificial intelligence principles. arXiv preprint arXiv:1812.04814 , 2018. · arXiv

Cited by

Sorted by citations

Authors
Raji, I. D., & Yang, J.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2019
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1912.06166
arXiv
1912.06166
OpenAlex
W2994816012
Cited by
36
References
8
Citations by year
2025: 2 · 2024: 6 · 2023: 5 · 2022: 6 · 2021: 8 · 2020: 8 · 2019: 1
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
Keywords
machine learning transparency, benchmarking, annotation, standardization, stakeholder engagement, explainability

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review