The DOI matches its Crossref record; 57 of the 57 references it cites were found in a registry or an index.
Opening up ChatGPT: Tracking openness, transparency, and accountability in instruction-tuned text generators
Open accessAbstract
Large language models that exhibit instruction-following behaviour represent one of the biggest recent upheavals in conversational interfaces, a trend in large part fuelled by the release of OpenAI’s ChatGPT, a proprietary large language model for text generation fine-tuned through reinforcement learning from human feedback (LLM+RLHF). We review the risks of relying on proprietary software and survey the first crop of open-source projects of comparable architecture and functionality. The main contribution of this paper is to show that openness is differentiated, and to offer scientific documentation of degrees of openness in this fast-moving field. We evaluate projects in terms of openness of code, training data, model weights, RLHF data, licensing, scientific documentation, and access methods. We find that while there is a fast-growing list of projects billing themselves as ‘open source’, many inherit undocumented data of dubious legality, few share the all-important instruction-tuning (a key site where human annotation labour is involved), and careful scientific documentation is exceedingly rare. Degrees of openness are relevant to fairness and accountability at all points, from data collection and curation to model architecture, and from training and fine-tuning to release and deployment.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“For many scientists and public interest practitioners, this lack of transparency is at best concerning, and often a reason to avoid such models in their work altogether [38, 46].”
From Laboratory-Scale AI: Open-Weight Models are Competitive with ChatGPT Even in Low-Resource Settings (2024) · cites this paper as [38]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“For one, we are already seeing a strong trend towards selective and self-serving forms of openness, both in release strategies and in empirical surveys of model openness ([32, 53, 60], and see below).”
From Rethinking open source generative AI: open washing and the EU AI Act (2024) · cites this paper as [32]
The cited source says what the claim attributes to it.
In the citing paper
“Open generative AI systems are on the rise, with small players and academic initiatives leading the way in open innovation and scientific documentation [20, 32, 61] and several larger corporations joining the fray by releasing models billed as 'open'.”
From Rethinking open source generative AI: open washing and the EU AI Act (2024) · cites this paper as [32]
The cited source says what the claim attributes to it.
In the citing paper
“The multi-faceted nature of openness is also seen in initiatives that aim to codify transparency from upstream resources to downstream uses [8] and in a proposal like the Linux Foundation's model openness framework [59], which shows many likenesses to the dimensions we use here and introduced in [32].”
From Rethinking open source generative AI: open washing and the EU AI Act (2024) · cites this paper as [32]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
The protein data bank: A computer-based archival file for macromolecular structures
Bernstein, F. C., Koetzle, T. F., Williams, G. J. B., Meyer, E. F., Brice, M. D., Rodgers, J. R., Kennard, O., Shimanouchi, T., & Tasumi, M. (1977) · Journal of Molecular Biology · doi:10.1016/s0022-2836(77)80200-3 · Cited by 8,574
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988 · cited 2×
“It is one of the achievements of publicly funded science that it can afford to not jump on the bandwagon and instead make room for reflection [2, 5].”
BackgroundLLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208
Highly accurate protein structure prediction for the human proteome
Tunyasuvunakool, K., Adler, J., Wu, Z., Green, T., Zielinski, M., Žídek, A., Bridgland, A., Cowie, A., Meyer, C., Laydon, A., et al. (2021) · Nature · doi:10.1038/s41586-021-03828-1 · Cited by 3,025 · cited 1×
“system predicting the structure of nearly all known proteins [53], where decades of prior work had reached a comparatively meagre 17%) has only been possible thanks to openly deposited structural data in the Protein Data Bank that goes back half a century [3].”
BackgroundModel Cards for Model Reporting
Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., & Gebru, T. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287596 · Cited by 2,681
Self-Instruct: Aligning Language Models with Self-Generated Instructions
Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A., Khashabi, D., & Hajishirzi, H. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.754 · Cited by 1,243
Frankfurt, H. G. (2005) · doi:10.1515/9781400826537 · Cited by 1,175
How Open Science Helps Researchers Succeed
McKiernan, E. C., Bourne, P. E., Brown, C. T., Buck, S., Kenall, A., Lin, J., McDougall, D., Nosek, B. A., Ram, K., Soderberg, C. K., et al. (2016) · eLife · doi:10.7554/elife.16800 · Cited by 798
“Everyone wants to do the model work, not the data work”: Data Cascades in High-Stakes AI
Sambasivan, N., Kapania, S., Highfill, H., Akrong, D., Paritosh, P., & Aroyo, L. M. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445518 · Cited by 781
Extracting Training Data from Large Language Models
Carlini, N., Tramer, F., Wallace, E., Jagielski, M., Herbert-Voss, A., Lee, K., Roberts, A., Brown, T., Song, D., Erlingsson, U., et al. (2020) · arXiv · doi:10.48550/arxiv.2012.07805 · Cited by 727
Algorithmic injustice: A relational ethics approach
Birhane, A. (2021) · Patterns · doi:10.1016/j.patter.2021.100205 · Cited by 707 · cited 1×
“But work that documents data provenance and traces harmful impacts [4, 49] deserves major scholarly and societal credit.”
BackgroundAdvances, challenges and opportunities in creating data for trustworthy AI
Liang, W., Tadesse, G. A., Ho, D. E., Fei-Fei, L., Zaharia, M., Zhang, C., & Zou, J. (2022) · Nature Machine Intelligence · doi:10.1038/s42256-022-00516-1 · Cited by 617
Trustworthy AI: From Principles to Practices
Li, B., Qi, P., Liu, B., Di, S., Liu, J., Pei, J., Yi, J., & Zhou, B. (2023) · ACM Computing Surveys · doi:10.1145/3555803 · Cited by 612 · cited 2×
“Despite strong evidence of the scientific and engineering benefits of open research practices, openness is not a given in machine learning and AI research [18, 20, 30].”
BackgroundData and its (dis)contents: A survey of dataset development and use in machine learning research
Paullada, A., Raji, I. D., Bender, E. M., Denton, E., & Hanna, A. (2021) · Patterns · doi:10.1016/j.patter.2021.100336 · Cited by 548 · cited 1×
“Here, AI and NLP might benefit from work in software engineering and infrastructure, where strong frameworks already exist to foster accountability for datasets [22, 31, 45].”
BackgroundTransparency and reproducibility in artificial intelligence
Haibe-Kains, B., Adam, G. A., Hosny, A., Khodakarami, F., Massive Analysis Quality Control (MAQC) Society Board of Directors, Shraddha, T., Kusko, R., Sansone, S.-A., Tong, W., Wolfinger, R. D., et al. (2020) · Nature · doi:10.1038/s41586-020-2766-y · Cited by 540 · cited 1×
“Despite strong evidence of the scientific and engineering benefits of open research practices, openness is not a given in machine learning and AI research [18, 20, 30].”
BackgroundA systematic evaluation of large language models of code
Xu, F. F., Alon, U., Neubig, G., & Hellendoorn, V. J. (2022) · Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming · doi:10.1145/3520312.3534862 · Cited by 540
The Pile: An 800GB Dataset of Diverse Text for Language Modeling
Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., et al. (2021) · arXiv · doi:10.48550/arxiv.2101.00027 · Cited by 492
Fine-Tuning Language Models from Human Preferences
Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., Christiano, P., & Irving, G. (2019) · arXiv · doi:10.48550/arxiv.1909.08593 · Cited by 413
Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Model
Luccioni, A. S., Viguier, S., & Ligozat, A.-L. (2022) · arXiv · doi:10.48550/arxiv.2211.02001 · Cited by 378
Crosslingual Generalization through Multitask Finetuning
Muennighoff, N., Wang, T., Sutawika, L., Roberts, A., Biderman, S., Le Scao, T., Bari, M. S., Shen, S., Yong, Z. X., Schoelkopf, H., et al. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.891 · Cited by 361
Baize: An Open-Source Chat Model with Parameter-Efficient Tuning on Self-Chat Data
Xu, C., Guo, D., Duan, N., & McAuley, J. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.385 · Cited by 348
State of the Art: Reproducibility in Artificial Intelligence
Gundersen, O. E., & Kjensmo, S. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11503 · Cited by 293
Hutchinson, B., Smart, A., Hanna, A., Denton, R., Greer, C., Kjartansson, O., Barnes, P., & Mitchell, M. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445918 · Cited by 279
Large image datasets: A pyrrhic win for computer vision?
Birhane, A., & Prabhu, V. U. (2021) · 2021 IEEE Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv48630.2021.00158 · Cited by 257
Burgelman, J.-C., Pascu, C., Szkuta, K., Von Schomberg, R., Karalopoulos, A., Repanas, K., & Schouppe, M. (2019) · Frontiers in Big Data · doi:10.3389/fdata.2019.00043 · Cited by 215
The growing influence of industry in AI research
Ahmed, N., Wahed, M., & Thompson, N. C. (2023) · Science · doi:10.1126/science.ade2420 · Cited by 208
Collaboration challenges in building ML-enabled systems
Nahar, N., Zhou, S., Lewis, G. A., & Kästner, C. (2022) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3510003.3510209 · Cited by 197
Multimodal datasets: misogyny, pornography, and malignant stereotypes
Birhane, A., Prabhu, V. U., & Kahembwe, E. (2021) · arXiv · doi:10.48550/arxiv.2110.01963 · Cited by 160
Augmented Language Models: a Survey
Mialon, G., Dessì, R., Lomeli, M., Nalmpantis, C., Pasunuru, R., Raileanu, R., Rozière, B., Schick, T., Dwivedi-Yu, J., Celikyilmaz, A., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.07842 · Cited by 160
Self-Instruct: Aligning Language Models with Self-Generated Instructions
Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A., Khashabi, D., & Hajishirzi, H. (2022) · arXiv · doi:10.48550/arxiv.2212.10560 · Cited by 118
TAMER: Training an Agent Manually via Evaluative Reinforcement
Bradley Knox, W., & Stone, P. (2008) · 2008 7th IEEE International Conference on Development and Learning · doi:10.1109/devlrn.2008.4640845 · Cited by 117
Towards Decolonising Computational Sciences
Birhane, A., & Guest, O. (2021) · Kvinder Køn & Forskning · doi:10.7146/kkf.v29i2.124899 · Cited by 117
None But Ourselves Can Free Our Minds: Critical Computational Literacy as a Pedagogy of Resistance
Lee, C. H., & Soep, E. (2016) · Equity & Excellence in Education · doi:10.1080/10665684.2016.1227157 · Cited by 109
Why open-source generative AI models are an ethical way forward for science
Spirling, A. (2023) · Nature · doi:10.1038/d41586-023-01295-4 · Cited by 104
Interactive Model Cards: A Human-Centered Approach to Model Documentation
Crisan, A., Drouhard, M., Vig, J., & Rajani, N. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533108 · Cited by 101 · cited 1×
“Interactive model cards [13] offer a promising step towards a humancentered approach to documentation.”
BackgroundGundersen, O. E., Gil, Y., & Aha, D. W. (2018) · AI Magazine · doi:10.1609/aimag.v39i3.2816 · Cited by 99
The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset
Laurençon, H., Saulnier, L., Wang, T., Akiki, C., del Moral, A. V., Scao, T. L., Von Werra, L., Mou, C., Ponferrada, E. G., Nguyen, H., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.03915 · Cited by 93
The Gradient of Generative AI Release: Methods and Considerations
Solaiman, I. (2023) · 2023 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3593013.3593981 · Cited by 90
Pile of Law: Learning Responsible Data Filtering from the Law and a 256GB Open-Source Legal Dataset
Henderson, P., Krass, M., Zheng, L., Guha, N., Manning, C. D., Jurafsky, D., & Ho, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2119 · Cited by 62
Deep TAMER: Interactive Agent Shaping in High-Dimensional State Spaces
Warnell, G., Waytowich, N. R., Lawhern, V. J., & Stone, P. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11485 · Cited by 60
Forgetting Practices in the Data Sciences
Muller, M. J., & Strohmayer, A. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517644 · Cited by 55
Changing the World by Changing the Data
Rogers, A. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.170 · Cited by 51
Pile of Law: Learning Responsible Data Filtering from the Law and a 256GB Open-Source Legal Dataset
Henderson, P., Krass, M. S., Zheng, L., Guha, N., Manning, C. D., Jurafsky, D., & Ho, D. E. (2022) · arXiv · doi:10.48550/arxiv.2207.00220 · Cited by 49
McMillan-Major, A., Osei, S., Rodriguez, J. D., Ammanamanchi, P. S., Gehrmann, S., & Jernite, Y. (2021) · Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics (GEM 2021) · doi:10.18653/v1/2021.gem-1.11 · Cited by 37
Crosslingual Generalization through Multitask Finetuning
Muennighoff, N., Wang, T., Sutawika, L., Roberts, A., Biderman, S., Scao, T. L., Bari, M. S., Shen, S., Yong, Z.-X., Schoelkopf, H., et al. (2022) · arXiv · doi:10.48550/arxiv.2211.01786 · Cited by 29
Church, K. W., & Kordoni, V. (2022) · Natural Language Engineering · doi:10.1017/s1351324922000043 · Cited by 28
Ten years after ImageNet: a 360° perspective on artificial intelligence
Chawla, S., Nakov, P., Ali, A., Hall, W., Khalil, I. M., Ma, X., Sencar, H. T., Weber, I. G., Wooldridge, M., & Yu, T. (2023) · Royal Society Open Science · doi:10.1098/rsos.221414 · Cited by 24
Data Statements: From Technical Concept to Community Practice
McMillan-Major, A., Bender, E. M., & Friedman, B. (2024) · ACM Journal on Responsible Computing · doi:10.1145/3594737 · Cited by 21
Not Directly Stated, Not Explicitly Stored:
Larson, M., Oostdijk, N. H. J., & Borgesius, F. Z. (2021) · ACM Conference on User Modeling, Adaptation and Personalization (UMAP) · doi:10.1145/3450614.3463601 · Cited by 8
Dynamic Planning in Open-Ended Dialogue using Reinforcement LearningPreprint
Cohen, D., Ryu, M., Chow, Y., Keller, O., Greenberg, I., Hassidim, A., Fink, M., Matias, Y., Szpektor, I., Boutilier, C. E., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2208.02294 · Cited by 8
Baize: An Open-Source Chat Model with Parameter-Efficient Tuning on Self-Chat DataPreprint
Xu, C., Guo, D., Duan, N., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2304.01196 · Cited by 7
EleutherAI: Going Beyond "Open Science" to "Science in the Open"
Phang, J., Bradley, H., Gao, L., Castricato, L., & Biderman, S. (2022) · arXiv · Cited by 3
OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.08774
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model
Workshop, B., :, Scao, T. L., Fan, A., Akiki, C., Pavlick, E., Ilić, S., Hesslow, D., Castagné, R., Luccioni, A. S., et al. (2022) · arXiv · doi:10.48550/arxiv.2211.05100
Flagged first, then by citations
Research works printed but not linked: 3
- Ivan Illich. 1973. Tools for conviviality. Harper & Row, New York.
- Nathan Lambert, Louis Castricato, Leandro von Werra, and Alex Havrilla. 2022. Illustrating Reinforcement Learning from Human Feedback (RLHF). Hugging Face Blog (2022).
- Dan McQuillan. 2022. Resisting AI: an anti-fascist approach to artificial intelligence. Bristol University Press, Bristol, UK. OCLC: on1328026349.
Cited by
Sorted by citations
Artificial intelligence and illusions of understanding in scientific research
Messeri, L., & Crockett, M. J. (2024) · Nature · doi:10.1038/s41586-024-07146-0 · Cited by 707
A Survey on Large Language Models for Code Generation
Jiang, J., Wang, F., Shen, J., Kim, S., & Kim, S. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3747588 · Cited by 251
Fields, J., Chovanec, K., & Madiraju, P. (2024) · IEEE Access · doi:10.1109/access.2024.3349952 · Cited by 188
ChatGPT is a Remarkable Tool—For Experts
Azaria, A., Azoulay, R., & Reches, S. (2023) · Data Intelligence · doi:10.1162/dint_a_00235 · Cited by 184
Törnberg, P. (2024) · Social Science Computer Review · doi:10.1177/08944393241286471 · Cited by 173
Metadata
- Authors
- Liesenfeld, A., Lopez, A., & Dingemanse, M.
- Affiliations
- Radboud University Nijmegen (NL)
- Venue
- Proceedings of the 5th International Conference on Conversational User Interfaces
- Publisher
- ACM
- Published
- 2023-07-19
- Pages
- 1-6
- OpenAlex
- W4384261711
- Cited by
- 111
- References
- 53
- Citations by year
- 2026: 17 · 2025: 28 · 2024: 52 · 2023: 12
- FWCI
- 27.28
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- ACM Conference on Conversational User Interfaces (CUI) (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- Radboud Repository (Radboud University) (submittedVersion, cc-by)
- arXiv (Cornell University)
- Funders
- Nederlandse Organisatie voor Wetenschappelijk Onderzoek (016.Vidi.185.205)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Keywords
- ChatGPT, instruction-tuned language models, open-source large language models, openness, transparency, accountability, reinforcement learning from human feedback, model weights, licensing, scientific documentation, proprietary software
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
