CoChat
Sign Up
PreprintCited by 5Explainable Artificial Intelligence (XAI)Software Engineering ResearchSoftware Engineering Techniques and Practices

Towards evaluating and eliciting high-quality documentation for intelligent systems

Open access
Piorkowski, David, González, Daniel, Richards, John, Houde, Stephanie·2020·arXiv
DOI 10.48550/arxiv.2011.08774arXiv 2011.08774
Open this paper in CoChat — free accountOpen PDF

Abstract

A vital component of trust and transparency in intelligent systems built on machine learning and artificial intelligence is the development of clear, understandable documentation. However, such systems are notorious for their complexity and opaqueness making quality documentation a non-trivial task. Furthermore, little is known about what makes such documentation "good." In this paper, we propose and evaluate a set of quality dimensions to identify in what ways this type of documentation falls short. Then, using those dimensions, we evaluate three different approaches for eliciting intelligent system documentation. We show how the dimensions identify shortcomings in such documentation and posit how such dimensions can be use to further enable users to provide documentation that is suitable to a given persona or use case.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2011.08774

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • AI documentation is advocated to increase the transparency and facilitate understanding of AI, and is embodied in recently proposed features of AI Factsheets and Model Cards.

    “AI documentation is advocated to increase the transparency and facilitate understanding of AI [35,36,40,75,76,82], and is embodied in recently proposed features of AI Factsheets [8,81] and Model Cards [64].”

    From Investigating Explainability of Generative AI for Code through Scenario-based Design (2022) · cites this paper as [75]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

No citation lists for this paper

No citation lists for this paper yet — checked again daily.

References 23 · Cited by 5

Research works printed but not linked: 23

  1. [1][1] Ritu Agarwal and Mohan R Tanniru. 1990. Knowledge acquisition using structured interviewing: an empirical investigation. Journal of Management Information Systems 7, 1 (1990), 123-140. -
  2. [2][2] Eugene W Anderson. 1994. Cross-category variation in customer satisfaction and retention. Marketing letters 5, 1 (1994), 19-30. -
  3. [3][3] Matthew Arnold, Rachel KE Bellamy, Michael Hind, Stephanie Houde, Sameep Mehta, Aleksandra Mojsilović, Ravi Nair, K Natesan Ramamurthy, Alexandra Olteanu, David Piorkowski, et al. 2019. FactSheets: Increasing trust in AI services through supplier's declarations of conformity. IBM Journal of Research and Development 63, 4/5 (2019), 6-1. -
  4. [4][4] John H Boose. 1990. Knowledge acquisition tools, methods and mediating representations. Knowledge Acquisition for Knowledge-Based Systems (Motoda, H. et al., Eds), IOS Press, Ohinsha Ltd., Tokyo (1990), 123-168. -
  5. [6][6] Vikas S Chomal and Jatinderkumar R Saini. 2015. Software Project Documentation-An Essence of Software Development. International Journal of Advanced Networking and Applications 6, 6 (2015), 2563. -
  6. [9][9] Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, and Kate Crawford. 2018. Datasheets for datasets. arXiv preprint arXiv:1803.09010 (2018). - · arXiv
  7. [10][10] Marek Havrda and Bogdana Rakova. 2020. Enhanced well-being assessment as basis for the practical implementation of ethical and rights-based normative principles for AI. arXiv preprint arXiv:2007.14826 (2020). - · arXiv
  8. [11][11] Michael Hind, Stephanie Houde, Jacquelyn Martino, Aleksandra Mojsilovic, David Piorkowski, John Richards, and Kush R Varshney. 2020. Experiences with Improving the Transparency of AI Models and Services. In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems . 1-8. -
  9. [12][12] Kenneth Holstein, Jennifer Wortman Vaughan, Hal Daumé III, Miro Dudik, and Hanna Wallach. 2019. Improving fairness in machine learning systems: What do industry practitioners need?. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems . 1-16. -
  10. [13][13] Michael I Jordan and Tom M Mitchell. 2015. Machine learning: Trends, perspectives, and prospects. Science 349, 6245 (2015), 255-260. -
  11. [14][14] Noela Jemutai Kipyegen and William PK Korir. 2013. Importance of software documentation. International Journal of Computer Science Issues (IJCSI) 10, 5 (2013), 223. -
  12. [15][15] Geoffrey N Leech. 2016. Principles of pragmatics . Routledge. -
  13. [16][16] Stephen W Littlejohn and Karen A Foss. 2010. Theories of human communication . Waveland press. -
  14. [18][18] Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. 2019. Model cards for model reporting. In Proceedings of the conference on fairness, accountability, and transparency . 220-229. -
  15. [19][19] Arun Parasuraman, Valarie A Zeithaml, and Leonard L Berry. 1994. Alternative scales for measuring service quality: a comparative assessment based on psychometric and diagnostic criteria. Journal of retailing 70, 3 (1994), 201-230. -
  16. [20][20] David Lorge Parnas. 2011. Precise documentation: The key to better software. In The Future of Software Engineering . Springer, 125-148. -
  17. [21][21] Reinhold Plösch, Andreas Dautovic, and Matthias Saft. 2014. The value of software documentation quality. In 2014 14th International Conference on Quality Software . IEEE, 333-342. -
  18. [22][22] Inioluwa Deborah Raji and Jingying Yang. 2019. ABOUT ML: Annotation and Benchmarking on Understanding and Transparency of Machine Learning Lifecycles. arXiv preprint arXiv:1912.06166 (2019). - · arXiv
  19. [23][23] John Richards, David Piorkowski, Michael Hind, Stephanie Houde, and Aleksandra Mojsilović. 2020. A Methodology for Creating AI FactSheets. arXiv preprint arXiv:2006.13796 (2020). - · arXiv
  20. [24][24] Tommy Sandbank, Michal Shmueli-Scheuer, Jonathan Herzig, David Konopnicki, John Richards, and David Piorkowski. 2018. Detecting Egregious Conversations between Customers and Virtual Agents. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) . 1802-1811. -
  21. [25][25] Karl L Smart. 2002. Assessing quality documents. ACM Journal of Computer Documentation (JCD) 26, 3 (2002), 130-140. -
  22. [26][26] Ian Sommerville. 2001. Software documentation. Software engineering 2 (2001), 143-154. -
  23. [27][27] Junji Zhi, Vahid Garousi-Yusifoğlu, Bo Sun, Golara Garousi, Shawn Shahnewaz, and Guenther Ruhe. 2015. Cost, benefits and quality of software development documentation: A systematic mapping. Journal of Systems and Software 99 (2015), 175-198.

Authors
Piorkowski, D., González, D., Richards, J., & Houde, S.
Affiliations
  • IBM (United States) (US)
  • Rensselaer Polytechnic Institute (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2020
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2011.08774
arXiv
2011.08774
OpenAlex
W3106500229
Cited by
5
References
23
Citations by year
2025: 1 · 2023: 2 · 2022: 1 · 2021: 1
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
Keywords
documentation quality, quality dimensions, explainable AI, trust and transparency, personas, use cases, machine learning systems

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review