Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat
Sign Up
Cited by 4

The DOI matches its Crossref record; the references it cites are not checked yet.

The software heritage license dataset (2022 edition)

Gonzalez-Barahona, Jesus M., Montes-Leon, Sergio, Robles, Gregorio, Zacchiroli, Stefano·2023·Empirical Software Engineering
DOI 10.1007/s10664-023-10377-w
Open this paper in CoChat — free account

Abstract

Context: When software is released publicly, it is common to include with it either the full text of the license or licenses under which it is published, or a detailed reference to them. Therefore public licenses, including FOSS (free, open source software) licenses, are usually publicly available in source code repositories Objective: To compile a dataset containing as many documents as possible that contain the text of software licenses, or references to the license terms. Once compiled, characterize the dataset so that it can be used for further research, or practical purposes related to license analysis Method: Retrieve from Software Heritage—the largest publicly available archive of FOSS source code—all versions of all files whose names are commonly used to convey licensing terms. All retrieved documents will be characterized in various ways, using automated and manual analyses Results: The dataset consists of 6.9 million unique license files. Additional metadata about shipped license files is also provided, making the dataset ready to use in various contexts, including: file length measures, MIME type, SPDX license (detected using ScanCode), and oldest appearance. The results of a manual analysis of 8102 documents is also included, providing a ground truth for further analysis. The dataset is released as open data as an archive file containing all deduplicated license files, plus several portable CSV files with metadata, referencing files via cryptographic checksums Conclusions: Thanks to the extensive coverage of Software Heritage, the dataset presented in this paper covers a very large fraction of all software licenses for public code. We have assembled a large body of software licenses, characterized it quantitatively and qualitatively, and validated that it is mostly composed of licensing information and includes almost all known license texts. The dataset can be used to conduct empirical studies on open source licensing, training of automated license classifiers, natural language processing (NLP) analyses of legal texts, as well as historical and phylogenetic studies on FOSS licensing. It can also be used in practice to improve tools detecting licenses in source code

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1007/s10664-023-10377-w

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

The citation lists haven’t been fetched yet

Cited by 4

Research works printed but not linked: 39

  1. Abramatic JF, Di Cosmo R, Zacchiroli S (2018) Building the universal archive of source code. Communications of the ACM 61(10):29–31 · DOI
  2. Allançon T, A Pietri, S Zacchiroli (2021) The software heritage filesystem (swhfs): Integrating source code archival with development. In 43rd IEEE/ACM International Conference on Software Engineering: Companion Proceedings, ICSE Companion 2021, Madrid, Spain, May 25-28, 2021, pages 45–48. IEEE · DOI
  3. Bird S (2006) NLTK: the natural language toolkit. In Nicoletta Calzolari, Claire Cardie, and Pierre Isabelle, editors, ACL 2006, 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference, Sydney, Australia, 17-21 July 2006. The Association for Computer Linguistics
  4. Boldi P, Pietri A, Vigna S, Zacchiroli S (2020) Ultra-large-scale repository analysis via graph compression. In SANER 2020: The 27th IEEE International Conference on Software Analysis, Evolution and Reengineering. IEEE, 2020 · DOI
  5. Caneill M, Germán DM, Zacchiroli S (2017) The debsources dataset: Two decades of free and open source software. Empirical Software Engineering 22:1405–1437 · DOI
  6. Collet Y (2022) RFC 8878 - Zstandard compression and the “application/zstd” media type, 2021. Accessed 2022-01-24 · DOI
  7. Di Cosmo R, Gruenpeter M, Zacchiroli S (2018) Identifiers for digital objects: the case of software source code preservation. In Proceedings of the 15th International Conference on Digital Preservation, iPRES 2018, Boston, USA
  8. Di Cosmo R, Zacchiroli S (2017) Software Heritage: Why and how to preserve software source code. In Proceedings of the 14th International Conference on Digital Preservation, iPRES 2017
  9. Di Penta M, German DM, Gaël Guéhéneuc Y, Antoniol G (2010) An exploratory study of the evolution of software licensing. In Proceedings of the 32nd ACM/IEEE International Conference on Software Engineering - Volume 1, ICSE ’10, page 145-154, New York, NY, USA, 2010. Association for Computing Machinery · DOI
  10. Dyer R, Nguyen HA, Rajan H, Nguyen TN (2015) Boa: Ultra-large-scale software repository and source-code mining. ACM Trans. Softw Eng Methodol 25(1):7:1–7:34 · DOI
  11. Flint SW, Chauhan J, Dyer R (2021) Escaping the time pit: Pitfalls and guidelines for using time-based git data. In 18th IEEE/ACM International Conference on Mining Software Repositories, MSR 2021, Madrid, Spain, May 17-19, 2021 85–96. IEEE, 2021 · DOI · Paper page
  12. Gandhi RA, Germonprez M, GJP Link (2018) Open data standards for open source software risk management routines: An examination of SPDX. In Forte A, Prilla M, Vivacqua AS, Müller C, and Lionel P. Robert Jr., editors, Proceedings of the 2018 ACM Conference on Supporting Groupwork, GROUP 2018, Sanibel Island, FL, USA, January 07 - 10, pages 219–229. ACM, 2018 · DOI
  13. German DM, Di Penta M, Davies J (2010) Understanding and auditing the licensing of open source software distributions. In 2010 IEEE 18th International Conference on Program Comprehension 84–93 · doi:10.1109/icpc.2010.48
  14. German DM, González-Barahona JM (2009) An empirical study of the reuse of software licensed under the GNU General Public License. In Boldyreff C, Crowston K, Lundell B, and Wasserman AI, editors, Open Source Ecosystems: Diverse Communities Interacting, pages 185–198, Berlin, Heidelberg, 2009. Springer Berlin Heidelberg · doi:10.1007/978-3-642-02032-2_17
  15. German DM, Hassan AE (2009) License integration patterns: Addressing license mismatches in component-based development. In 2009 IEEE 31st International Conference on Software Engineering 188–198 · doi:10.1109/icse.2009.5070520
  16. Germán DM, Manabe Y, Inoue K (2010) A sentence-matching method for automatic license identification of source code files. In Pecheur C, Andrews J, and Di Nitto E, editors, ASE 2010, 25th IEEE/ACM International Conference on Automated Software Engineering, Antwerp, Belgium, September 20-24, pages 437–446. ACM, 2010 · doi:10.1145/1858996.1859088
  17. Germán DM, Di Penta M (2012) A method for open source license compliance of java applications. IEEE Softw 29(3):58–63 · doi:10.1109/ms.2012.50
  18. Gobeille R (2008) The fossology project. In Hassan AE, Lanza M, and Godfrey MW, editors, Proceedings of the 2008 International Working Conference on Mining Software Repositories, MSR 2008 (Co-located with ICSE), Leipzig, Germany, May 10-11, 2008, Proceedings 47–50. ACM · doi:10.1145/1370750.1370763
  19. Gousios G, Spinellis D (2012) Ghtorrent: Github’s data from a firehose. In Lanza M, Di Penta M, and Xie T, editors, 9th IEEE Working Conference of Mining Software Repositories, MSR, pages 12–21. IEEE Computer Society, 2012 · DOI · Paper page
  20. Harutyunyan N (2020) Managing your open source supply chain-why and how? Computer 53(6):77–81 · DOI
  21. Ma Y, Dey T, Bogart C, Amreen S, Valiev M, Tutko A, Kennard D, Zaretzki R, Mockus A (2021) World of code: enabling a research workflow for mining and analyzing the universe of open source VCS data. Empir Softw Eng 26(2):22 · DOI · Paper page
  22. Manabe Y, German DM, Inoue K (2014) Analyzing the relationship between the license of packages and their files in free and open source software. In Corral L, Sillitti A, Succi G, Vlasenko J, and Wasserman AI, editors, Open Source Software: Mobile Open Source Technologies 51–60, Berlin, Heidelberg, 2014. Springer Berlin Heidelberg · DOI
  23. Manabe Y, Hayase Y, Inoue K (2010) Evolutional analysis of licenses in FOSS. In Andrea Capiluppi, Anthony Cleve, and Naouel Moha, editors, Proceedings of the Joint ERCIM Workshop on Software Evolution (EVOL) and International Workshop on Principles of Software Evolution (IWPSE), Antwerp, Belgium, September 20-21, 2010, pages 83–87. ACM, 2010 · doi:10.1145/1862372.1862391
  24. Maryka T, Germán DM, Poo-Caamaño G (2015) On the variability of the BSD and MIT licenses. In Ernesto Damiani, Fulvio Frati, Dirk Riehle, and Anthony I. Wasserman, editors, Open Source Systems: Adoption and Impact - 11th IFIP WG 2.13 International Conference, OSS 2015, Florence, Italy, May 16-17, 2015, Proceedings, volume 451 of IFIP Advances in Information and Communication Technology 146–156. Springer, 2015 · doi:10.1007/978-3-319-17837-0_14
  25. Maryka T, German DM, Poo-Caamaño G (2015) On the variability of the bsd and mit licenses. In: Damiani Ernesto, Frati Fulvio, Riehle Dirk, Wasserman Anthony I (eds) Open Source Systems: Adoption and Impact (OSS 2015). pp. Springer International Publishing, Cham, pp 146–156 · doi:10.1007/978-3-319-17837-0_14
  26. McKinney W et al (2011) Pandas: a foundational python library for data analysis and statistics. Python for high performance and scientific computing 14(9):1–9
  27. Philippe Ombredanne (2020) Free and open source software license compliance: Tools for software composition analysis. Computer 53(10):105–109 · doi:10.1109/mc.2020.3011082
  28. Phipps S, Zacchiroli S (2020) Continuous open source license compliance. Computer 53(12):115–119 · doi:10.1109/mc.2020.3024403
  29. Pietri A, Spinellis D, Zacchiroli S (2019) The Software Heritage graph dataset: public software development under one roof. In Storey MAD, Adams B, and Haiduc S, editors, Proceedings of the 16th International Conference on Mining Software Repositories, MSR 2019, 26-27 May 2019, Montreal, Canada., pages 138–142. IEEE / ACM · DOI
  30. Rosen L (2005) Open source licensing, volume 692. Prentice Hall
  31. Rousseau G, Di Cosmo R, Zacchiroli S (2020) Software provenance tracking at the scale of public source code. Empirical Software Engineering 25(4):2930–2959 · DOI · Paper page
  32. Shafranovich Y (2005) RFC 4180 - common format and MIME type for comma-separated values (CSV) files, 2005. Accessed 2022-01-24 · DOI
  33. Stewart K, P Odence, Rockett E (2010) Software package data exchange (SPDX) specification. IFOSS L Rev 2:191 · DOI
  34. Vendome C, Bavota G, Di Penta M, Vásquez ML, Germán DM, Poshyvanyk D (2017) License usage and changes: a large-scale study on GitHub. Empir Softw Eng 22(3):1537–1577 · DOI
  35. Vendome C, Linares-Vásquez M, Bavota G, Di Penta M, German DM, Poshyvanyk D (2015) When and why developers adopt and change software licenses. In 2015 IEEE International Conference on Software Maintenance and Evolution (ICSME) pages 31–40 · DOI
  36. Vendome C, Vásquez ML, Bavota G, Di Penta M, Germán DM, Poshyvanyk D (2017) Machine learning-based detection of open source license exceptions. In Sebastián Uchitel, Alessandro Orso, and Martin P. Robillard, editors, Proceedings of the 39th International Conference on Software Engineering, ICSE 2017, Buenos Aires, Argentina, May 20-28, 2017, pages 118–129. IEEE / ACM, 2017 · DOI
  37. Xu S, Gao Y, Fan L, Liu Z, Liu Y, and Ji H (2023) Lidetector: License incompatibility detection for open source software. ACM Trans. Softw Eng Methodol 32(1) · doi:10.1145/3518994
  38. Zacchiroli S (2022) A large-scale dataset of (open source) license text variants. In The 2022 Mining Software Repositories Conference (MSR 2022), pages 757–761. ACM, 2022 · DOI · Paper page
  39. Zhang D, Luo P, Tang W, and Zhou M (2021) Osldetector: Identifying open-source libraries through binary analysis. In Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering, ASE ’20, page 1312-1315, New York, NY, USA, 2021. Association for Computing Machinery · doi:10.1145/3324884.3415303

Metadata

Authors
Gonzalez-Barahona, J. M., Montes-Leon, S., Robles, G., & Zacchiroli, S.
Venue
Empirical Software Engineering
Publisher
Springer Science and Business Media LLC
Published
2023-11
Volume
28
Issue
6
ISSN
  • 1382-3256
  • 1573-7616
DOI
10.1007/s10664-023-10377-w
Cited by
4
Language
en
Licence

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: The software heritage license dataset (2022 edition). https://app.cochat.ai/papers/10.1007/s10664-023-10377-w
The paper
Gonzalez-Barahona, J. M., Montes-Leon, S., Robles, G., & Zacchiroli, S. (2023). The software heritage license dataset (2022 edition). Empirical Software Engineering. https://doi.org/10.1007/s10664-023-10377-w

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.