CoChat
Sign Up
PreprintCited by 62Natural Language Processing TechniquesTopic ModelingSpeech Recognition and Synthesis

The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset

Open access
Laurençon, Hugo, Saulnier, Lucile, Wang, Thomas, Akiki, Christopher, del Moral, Albert Villanova, Scao, Teven Le, Von Werra, Leandro, Mou, Chenghao +46 more·2023·arXiv
DOI 10.48550/arxiv.2303.03915arXiv 2303.03915
Open this paper in CoChat — free account

Abstract

As language models grow ever larger, the need for large-scale high-quality text datasets has never been more pressing, especially in multilingual settings. The BigScience workshop, a 1-year international and multidisciplinary initiative, was formed with the goal of researching and training large language models as a values-driven undertaking, putting issues of ethics, harm, and governance in the foreground. This paper documents the data creation and curation efforts undertaken by BigScience to assemble the Responsible Open-science Open-collaboration Text Sources (ROOTS) corpus, a 1.6TB dataset spanning 59 languages that was used to train the 176-billion-parameter BigScience Large Open-science Open-access Multilingual (BLOOM) language model. We further release a large initial subset of the corpus and analyses thereof, and hope to empower large-scale monolingual and multilingual modeling projects with both the data and the processing tools, as well as stimulate research around this large multilingual corpus.

AI snapshot

Written from the paper’s full text.

TL;DR

The paper documents BigScience ROOTS, a 1.6TB multilingual corpus spanning 59 languages (46 natural, 13 programming) used to train the 176B-parameter BLOOM model. It describes the community-sourced and OSCAR-based curation pipeline, releases a gated subset and tooling, and presents initial corpus analyses.

Key findings

  • ROOTS is 1.6TB, covers 59 languages (46 natural, 13 programming), and is composed of 62% community-selected documented sources and 38% filtered OSCAR 21.09; it was used to train the 176-billion-parameter BLOOM model.
  • Natural-language distribution is led by English (30.03%), Simplified Chinese (16.16%), French (12.9%), Spanish (10.85%), Portuguese (4.91%), and Arabic (4.6%); median document size is 1,129 bytes.
  • OSCAR filtering removed 5.2%–48.8% of documents depending on language (e.g., Bengali 48.8%, Basque 5.2%, English 17.2%); deduplication found 0.7% average SimHash near-duplicates and 21.67% average byte duplication via substring dedup.
  • The code subset spans 13 programming languages, with Java, PHP, and C++ accounting for more than half of all documents; 5.23% are configuration files and 7.88% test files, and almost 32% of code data consists of near-duplicates (10.9M duplicate vs. 4.1M unique files).

Methods

  • Design: a collaborative, value-driven BigScience effort organized around four working groups, combining community-selected/documented language resources (62%) with filtered OSCAR 21.09 (38%).
  • Data: 252 catalogue sources (at least 21 per language category), Masader Arabic resources, pseudo-crawling from 18 Common Crawl 2020–2021 snapshots over 614 unique domains (539 domain text datasets; 147 pseudo-crawl datasets after filtering), GitHub BigQuery code, and the February 2021 OSCAR snapshot.
  • Processing and analysis: cleaning, filtering, deduplication, and regex-based PII redaction; native-speaker thresholds for OSCAR filters; statistical breakdowns by language, document size, code file types, and tokenizer tokens-per-byte.

Limitations

  • The paper states that using Common Crawl creates tension with consent and privacy; PII removal is not exhaustive; it is next to impossible to identify individual contributors and ask for consent; similar concerns apply to other datasets including WuDao; and reliance on medium-to-large digitized sources over-represents privileged voices and language varieties.
  • The paper states that the code dataset was filtered for GPL licenses only due to a preprocessing bug, and OSCAR filtering was limited to languages for which native speakers could set hyperparameters and cutoffs (Arabic, Basque, Bengali, Catalan, Chinese, English, French, Hindi, Indonesian, Portuguese, Spanish, Urdu, Vietnamese).
  • Evident from the text: the corpus is heavily skewed toward English and a few high-resource languages; Figure 1’s caption notes the Africa macroarea is 0.4GB and Indonesian is the sole Papunesia representative at 18GB.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2303.03915

Cites

119 of 119 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

16 of 26 cited sources checked · 9 supported · 1 flagged · accuracy not yet measured

  • Premisecitation
    § 3 Processing OSCAR · p. 7

    The quantity of data is a strong factor in a trained model's performance on evaluation tasks.

    “recent work has put a strong emphasis on the quantity of data being a strong factor in a trained model's performance on evaluation tasks (Kaplan et al., 2020; Hoffmann et al., 2022)”
    Quote in the paperSubstantial claim2 of 2 sources checked
    • cites Kaplan et al., 2020
    • cites Hoffmann et al., 2022 source
  • Findingown data
    § 2.1 Obtaining Data from the Identified Resources · p. 6

    Non-trivial dataset overlap included s2orc, Arxiv, and the PubMed Central subset of the Pile.

    “Non-trivial datasets overlap included s2orc (Lo et al., 2020), Arxiv (Clement et al., 2019) and the PubMed Central subset of the Pile (Gao et al., 2020).”
    Quote in the paperNot a substantial claim1 of 3 sources checked
    • cites Lo et al., 2020

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Clement et al., 2019 source
    • cites Gao et al., 2020

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Findingcitation
    § 1.1 Outline of the Paper · p. 4

    38% of ROOTS consists of text extracted from OSCAR, filtered with the help of native speakers.

    “38% consists of text extracted from a pre-processed web crawl, OSCAR (Ortiz Suárez et al. (2020)), filtered with the help of native speakers”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Ortiz Suárez et al. (2020) source
  • Premisecitation
    § 4.2 Programming Languages · p. 10

    Code datasets contain a large fraction of near-duplicates that can inflate performance metrics.

    “Allamanis (2019) and Lopes et al. (2017) highlight the large fraction of near-duplicates present in code datasets and how they can inflate performance metrics.”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites Allamanis (2019) source
    • cites Lopes et al. (2017) source
  • Premisecitation
    § 3 Processing OSCAR · p. 7

    Crawled content over-represents pornographic text across languages, especially in the form of spam ads.

    “Crawled content also over-represents pornographic text across languages (Kreutzer et al., 2022), especially in the form of spam ads.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Kreutzer et al., 2022 source

      The section “Which languages have the lowest quality?” (p. 8) reports notable amounts of pornographic content—over 10% of sampled sentences—in CCAligned for 11 languages. That supports its presence across multiple languages, but does not establish over-representation in crawled content generally or identify spam ads as its predominant form. (LLM, conf=0.86)

      Experimental — accuracy not yet measured
  • Classified as probably not a substantial claim. 27
  • May combine two claims. 23

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 1.2 Related Work · p. 4

    Existing work on dataset processing varies in the level of detail provided, from no details to detailed filtering instructions with or without code release.

    “Existing work tends to fall on a spectrum from no details at all (Brown et al., 2020) to detailed filtering instructions, with (Raffel et al., 2020) or without the dataset release (Rae et al., 2021) to detailed filtering instructions with the accompanying code (Gao et al., 2020; Conneau et al., 2020; Ortiz Suárez et al., 2019).”
    Quote in the paperNot a substantial claim1 of 3 sources checked
    • cites Brown et al., 2020 source

      The citation presents Brown et al. (2020) as providing no dataset-processing details. Section 2.2 (p. 8) describes filtering and deduplication, and Appendix A (p. 43) gives further details of both methods, so the paper says the contrary. (LLM, conf=0.94)

      Checked by a reasoning model — may be wrong.

      Experimental — accuracy not yet measured
    • cites Raffel et al., 2020

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Gao et al., 2020

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim.

    Premisecitation
    § 1.2 Related Work · p. 4

    In the pre-trained model paradigm, performance is directly correlated with both model size and dataset size and quality.

    “performance is directly correlated on both the model size and the dataset size and quality (Kaplan et al., 2020)”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Kaplan et al., 2020

      The Summary (p. 2) says language-model performance improves with model size and dataset size, provided other factors do not become bottlenecks. Section 3.3 (p. 9) examines dataset size using subsets of the same dataset. These passages do not establish that performance is correlated with dataset quality. (LLM, conf=0.88)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § 1.2 Related Work · p. 4

    The Pile authors analyzed and documented syntactic and semantic properties of their dataset and released a datasheet.

    “The most notable example of this is the Pile, for which the authors themselves analyze and document a variety of syntactic and semantic properties of the dataset including structural statistics (n-gram counts, language, document sizes), topical distributions across its components, social bias and sentiment co-occurrence, pejorative content, and information about licensing and authorial consent, in”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Biderman et al., 2022 source

      The Abstract (p. 1) identifies this work as a datasheet for the Pile. The dataset-composition section (p. 8) gives document counts and sizes by component, and the legal and ethical sections (pp. 14–17) discuss licensing, potentially offensive content, and consent. The passages do not substantiate the more specific attribution of n-gram counts, topical distributions, sentiment co-occurrence, or pejorative-content analyses to this datasheet. (LLM, conf=0.88)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Premisecitation
    § 3.2 Deduplication · p. 8

    Deduplication improves performance on downstream tasks and decreases memorization of training data.

    “Data deduplication has become a key tool for language model projects following research showing that it both improves performance on downstream tasks (Lee et al., 2022; Zhang et al., 2021) and decreases memorization of training data (Kandpal et al., 2022).”
    Quote in the paperSubstantial claim3 of 3 sources checked
    • cites Lee et al., 2022 source

      The Introduction and Conclusion report that deduplication sometimes improves language-model perplexity and helps models reach accuracy targets faster. But the paper’s “Contamination of downstream tasks” section explicitly says it does not study effects on downstream benchmark tasks, so Lee et al. do not establish that part of the citing sentence. The memorization claim is attached to Kandpal et al., not Lee et al. (LLM, conf=0.88)

      Experimental — accuracy not yet measured
    • cites Zhang et al., 2021 source

      Zhang et al. (2021) is cited for improved downstream-task performance; the separate Kandpal et al. citation is attached to decreased memorization. Our copy's Appendix C (p. 15) compares memorization in C4 and a deduplicated dataset, but does not report downstream-task performance. Searches for downstream-task performance after deduplication found no supporting passage. (LLM, conf=0.75)

      Experimental — accuracy not yet measured
    • cites Kandpal et al., 2022 source

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

    Premisecitation
    § Putting Our Values into Practice · p. 20

    Participatory approaches play a vital role in bridging the gaps between model development and deployment and in promoting fairness in ML applications.

    “Participatory approaches play a vital role in bridging the gaps between model development and deployment and in promoting fairness in ML applications (Rajkomar et al., 2018).”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Rajkomar et al., 2018 source

  • May combine two claims.

    Premisecitation
    § Putting Our Values into Practice · p. 20

    Multilingual text data curation done without involving language-specific expertise leads to resources that are very different from the intentions of their creators, and these limitations carry on to the models trained on these datasets.

    “First, Kreutzer et al. (2022) have shown in recent work that multilingual text data curation done without involving language-specific expertise leads to resources that are very different from the intentions of their creators, and these limitations carry on to the models trained on these datasets.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Kreutzer et al. (2022) source

      The Human Audit Results (p. 6) say low quality in underrepresented languages may go unnoticed when no proficient speakers are involved in curation. Risks of Low-Quality Data (p. 10) links data quality to translation quality and warns that flawed data may affect downstream technology. The paper does not establish the claim’s definite causal chain from lack of language-specific expertise to defective resources and then to models trained on them. (LLM, conf=0.84)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § BigScience Values · p. 20

    Data efforts in computer vision tend to prioritize efficiency over care, universality over contextuality, and impartiality over positionality.

    “Scheuerman et al. (2021) outline how data efforts in computer vision tend to prioritize ' efficiency [over] care; universality [over] contextuality; impartiality [over] positionality . . . '.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Scheuerman et al. (2021) source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given
    § 2.1 Obtaining Data from the Identified Resources · p. 5

    Due to a bug in the pre-processing pipeline, the GitHub code dataset was filtered for GPL licenses only.

    “Due to a bug in the pre-processing pipeline the dataset was also filtered for GPL licenses only.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given
    § 4.1 Natural Languages · p. 9

    The previous post-processing steps only reduced the size of the documents.

    “At the same time, the previous post-processing steps only reduced the size of the documents.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § 2 (Crowd) Sourcing a Language Resource Catalogue · p. 4

    The first part of the ROOTS corpus accounts for 62% of the final dataset size in bytes.

    “The first part of our corpus, accounting for 62% of the final dataset size (in bytes), was made up of a collection of monolingual and multilingual language resources that were selected and documented collaboratively through various efforts of the BigScience Data Sourcing working group.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § 2.1 Obtaining Data from the Identified Resources · p. 5

    27 domain names were removed because no HTML pages were retrieved for them.

    “27 domain names were additionally removed from the list at this stage as we had not retrieved any HTML pages for them.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 4.2 Programming Languages · p. 10

    Only 1% of the Python data and 2% of the PHP files do not pass the syntax check.

    “We find that only 1% of the Python data and 2% of the PHP files do not pass the syntax check.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 4.2 Programming Languages · p. 9

    The code subset of the corpus spans 13 programming languages, with Java, PHP, and C++ accounting for more than half of all documents.

    “the code subset of the corpus spans 13 programming languages, with Java, PHP, and C++ accounting for more than half of all documents.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § 2.1 Obtaining Data from the Identified Resources · p. 5

    The pseudo-crawling seed domain names came from 192 homepage fields in the metadata of the 252 catalogue entries and 456 websites proposed by participants.

    “These domain names came from two main sources: the homepage field in the metadata of the 252 above-mentioned catalogue entries when available (192 in total), and the 456 websites proposed by participants asynchronously to improve the geographical diversity of our language sources; which yielded a total of 614 unique domain names after deduplication.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingnone given
    § Abstract · p. 1

    The paper releases a large initial subset of ROOTS and analyses thereof.

    “We further release a large initial subset of the corpus and analyses thereof”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given
    § 4.1 Natural Languages · p. 9

    The flagged word filter distribution for OSCAR is skewed right compared to the catalogue even after filtering.

    “A notable example is the flagged word filter, for which the distribution for OSCAR is skewed right compared to the catalogue even after filtering.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingreasoning
    § 2.2 Processing Pipeline for Quality Improvement on Crowdsourced Datasets · p. 6

    The processing resulted in 485 non-empty datasets.

    “This resulted in 485 non-empty datasets.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § 2.1 Obtaining Data from the Identified Resources · p. 6

    After removing domains smaller than 2MB, there were 147 pseudo-crawl-based datasets.

    “We then removed entire domains whose size was less than 2MB after this step, yielding 147 pseudo-crawl-based datasets, and a total of 517 datasets including all three pipelines.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § 2.1 Obtaining Data from the Identified Resources · p. 5

    48 of the 614 initial seed domain names had no matches in the Common Crawl index and were left out.

    “48 of the 614 initial seed domain names had no matches in the index and were therefore left out.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 4.1 Natural Languages · p. 8

    The corpus comprises 46 natural languages spanning 3 macroareas and 9 language families.

    “It comprises of 46 natural languages spanning 3 macroareas and 9 language families: Afro-Asiatic, AustroAsiatic, Austronesian, Basque, Dravidian, Indo-European, Mande, Niger-Congo, Sino-Tibetan.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given
    § 2.2 Processing Pipeline for Quality Improvement on Crowdsourced Datasets · p. 6

    In the HTML-extracted text, not all text is relevant; menus, advertisements, and repeated text on each page are examples of irrelevant content.

    “In the specific case of text extraction from HTML, we observe that not all text are relevant (menus, advertisements, repeated text on each page etc ...).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 4.1 Natural Languages · p. 8

    English constitutes the largest part of the corpus at 30.03%, followed by Simplified Chinese (16.16%), French (12.9%), Spanish (10.85%), Portuguese (4.91%) and Arabic (4.6%).

    “At 30.03%, English constitutes the largest part of the corpus, followed by Simplified Chinese (16.16%), French (12.9%), Spanish (10.85%), Portuguese (4.91%) and Arabic (4.6%).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingreasoning
    § 2.1 Obtaining Data from the Identified Resources · p. 5

    After deduplication, the pseudo-crawling seed domain names totaled 614 unique domain names.

    “These domain names came from two main sources: the homepage field in the metadata of the 252 above-mentioned catalogue entries when available (192 in total), and the 456 websites proposed by participants asynchronously to improve the geographical diversity of our language sources; which yielded a total of 614 unique domain names after deduplication.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 3.1 Data cleaning and filtering · p. 8

    The percentage of documents removed by filtering per language is: AR 20.3, EU 5.2, BN 48.8, CA 21.1, ZH 23.1, EN 17.2, FR 17.0, HI 25.7, ID 10.4, PT 12.6, UR 15.8, VI 21.3, ES 16.9.

    “| AR | EU | BN | CA | ZH | EN | FR | HI | ID | PT | UR | VI | ES |”
    • Also stated in: § 3.1 Data cleaning and filtering · p. 8 · plain
      “| 20.3 | 5.2 | 48.8 | 21.1 | 23.1 | 17.2 | 17.0 | 25.7 | 10.4 | 12.6 | 15.8 | 21.3 | 16.9 |”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 4.2 Programming Languages · p. 10

    5.23% of the data consists of configuration files and 7.88% of test files.

    “We find that 5.23% of the data consists of configuration files and 7.88% of test files.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Configuration and test files are abundant in most GitHub repositories but not as interesting for code modeling.

  • May combine two claims.

    Findingreasoning
    § C Exhaustive list of human curated filters used on OSCAR · p. 27

    The word repetition ratio filter is more robust in finding documents with long exact duplicated sentences in them, while the character repetition ratio filter is used to find short to medium sized repetitions.

    “This filter is more robust in finding documents with long exact duplicated sentences in them, while the previous one is used to find short to medium sized repetitions.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation
    § Limitations of the Approach. · p. 21

    The WuDao web-based corpus makes up a significant part of the Chinese language data.

    “Similar concerns apply to other existing NLP datasets we identified in the catalogue, including notably the WuDao web-based corpus (Yuan et al., 2021) which makes up a significant part of the Chinese language data.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Yuan et al., 2021 source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § C Exhaustive list of human curated filters used on OSCAR · p. 27

    About 1% of the documents for each language are removed by the flagged word ratio filter.

    “About 1% of the documents for each language are removed by this filter.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingnone given
    § Limitations of the Approach. · p. 21

    Our preprocessing removes some categories of PII but is still far from exhaustive.

    “Our preprocessing removes some categories of PII but is still far from exhaustive”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisereasoning
    § 1.2 Related Work · p. 4

    Existing data processing code is typically tailored to its original project, so large projects usually build their own pipeline rather than reuse existing ones on new data.

    “Even when the code is released, it tends to be built and tailored for the project's purpose. Consequently, large projects that do not re-use an existing dataset outright usually build their own pipeline rather than re-use an existing one on new data.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationnone given
    § 3.2 Deduplication · p. 8

    SimHash is essentially a bag-of-words algorithm, so long documents are more likely to end up being similar to each other.

    “because SimHash is essentially a bag-of-words algorithm, long documents are more likely to end up being similar to each other.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning
    § 2.2 Processing Pipeline for Quality Improvement on Crowdsourced Datasets · p. 6

    The datasets came from heterogeneous sources with different properties, requiring each to have its own set of processing functions.

    “As datasets came from heterogeneous sources with different properties, each needs its own set of processing functions to correspond to our definition of natural language documents.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning
    § 2.2 Processing Pipeline for Quality Improvement on Crowdsourced Datasets · p. 6

    The processing pipelines were finetuned iteratively with input from native speakers.

    “This rapid feedback loop enabled us to update the pipeline consequently in an iterative process to finetune each processing pipelines across datasets and languages with the input of native speakers.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data
    § 4.3 Tokenizer analysis of the component datasets · p. 11

    global_voices and no_code_stackexchange are similar to the authors' corpus but not to the Pile or C4.

    “Both are similar to our corpus but not to the Pile or C4.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning
    § 1.1 Outline of the Paper · p. 3

    The language choice for ROOTS was chiefly driven by the communities who participated in the effort, given the importance placed on language expertise.

    “the language choice was chiefly driven by the communities who participated in the effort given the importance we placed on language expertise”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Premisecitation
    § BigScience Values · p. 20

    ML research programs and systems support the development of new technologies that carry these same values when deploying these technologies in production.

    “These ML research programs and systems in turn support the development of new technologies that carry these same values when deploying these technologies in production (Winner, 2017).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Winner, 2017

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

    Interpretationnone given
    § Putting Our Values into Practice · p. 20

    Intentional curation and proper documentation of web-scale corpora takes a significant amount of human work and expertise, which can be distributed between a large number of participants in community efforts.

    “Third, intentional curation and proper documentation of web-scale corpora takes a significant amount of human work and expertise, which can be distributed between a large number of participants in community efforts.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductreasoning
    § D PII filtering initiative · p. 28

    The more complex PII detection approach using regexes with surrounding text context could not be operationalized on ROOTS because of the scale of the data, the inability to fully assess its impact on language modeling, and time constraints due to compute allocation.

    “However, the scale of the data, the fact that the impact on the resulting text could not be fully assessed in terms of language modeling and the time constraint due to compute allocation, meant this approach could not be operationalized on ROOTS.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationnone given
    § C Exhaustive list of human curated filters used on OSCAR · p. 26

    Documents with too few words often contain incorrect sentences, or contain no context for a model to learn correctly.

    “We discarded documents with too few words, as they often contain incorrect sentences, or contain no context for a model to learn correctly.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Interpretationnone given
    § Putting Our Values into Practice · p. 20

    Community involvement can help foster trust and collective ownership of the artifacts created.

    “Finally, community involvement can help foster trust and collective ownership of the artifacts we create.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

    Interpretationcitation
    § BigScience Values · p. 20

    The prioritization of narrow performance and efficiency in NLP and ML research limits the potential positive societal benefits of rapid advances in NLP research while increasing risks considerably.

    “This limits the potential positive societal benefits of the rapid advances of NLP research while increasing risks considerably.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

    Interpretationnone given
    § Putting Our Values into Practice · p. 20

    Resources developed in collaboration with other communities are more likely to be more directly relevant to them, and thus to avoid reduce replication of model development by making the artifacts and tools useful to more people and for more languages.

    “Second, resources that are developed in collaboration with other communities are more likely to be more directly relevant to them, and thus to avoid reduce replication of model development by making the artifacts and tools we develop useful to more people and for more languages.”
    Quote in the paperNot a substantial claim
42 of 42

  • The paper relies on this without support.

    Most large language model datasets are not released.

    “Most such datasets, however, are not released, hindering further research.”

  • The paper relies on this without support.

    Crawled data contains personal information that may constitute a privacy risk.

    “Finally, it contains personal information that may constitute a privacy risk.”

  • The paper relies on this without support.

    In order for a trained model to have an opportunity to learn long dependencies, the training corpus needs to contain long sequences of coherent text.

    “In order for the trained model to have an opportunity to learn long dependencies, the training corpus needs to contain long sequences of coherent text.”

  • The paper relies on this without support.

    Configuration and test files are abundant in most GitHub repositories but not as interesting for code modeling.

    “Configuration and test files are abundant in most GitHub repositories but not as interesting for code modeling.”

  • The paper relies on this without support.

    Exact match deduplication alone can miss a fair amount of near-duplicates.

    “Exact match deduplication alone can miss a fair amount of near-duplicates.”

  • The paper relies on this without support.

    The legal status of webscraped datasets is extremely unclear in many jurisdictions, putting a substantial burden on both data creators and data users who wish to be involved with this process.

    “The legal status of webscraped datasets is extremely unclear in many jurisdictions, putting a substantial burden on both data creators and data users who wish to be involved with this process.”

  • The paper relies on this without support.

    The principle of fair use generally protects academic researchers, but it is not recognized in all jurisdictions and may not cover research carried out in an industry context.

    “While the principle of fair use generally protects academic researchers, it is not recognized in all jurisdictions and may not cover research carried out in an industry context.”

  • The paper relies on this without support.

    The nature of crawled datasets makes it next to impossible to identify individual contributors and ask for their consent.

    “the nature of crawled datasets makes it next to impossible to identify individual contributors and ask for their consent.”

  • The paper relies on this without support.

    Our reliance on medium to large sources of digitized content still over-represents privileged voices and language varieties.

    “our reliance on medium to large sources of digitized content still over-represents privileged voices and language varieties.”

  • The paper relies on this without support.

    Language identification does not perform as well on low-resource languages.

    “the language identification does not perform as well on low-resource languages.”

  • The paper relies on this without support.

    Pornographic documents are in practice much more likely to have shocking and sexist content, and to contain only buzzwords for SEO.

    “pornographic documents, which are in practice much more likely to have shocking and sexist content, and to contain only buzzwords for SEO”

  • The paper relies on this without support.

    The use of data from the Common Crawl represents a point of tension between the drive to present a research artifact comparable to previous work and values of consent and privacy.

    “Among those, we primarily identify the use of data from the Common Crawl which represents a point of tension between our drive to present a research artifact that is comparable to previous work and values of consent and privacy (see Section 3).”
12 of 12

Cites

Flagged first, then by citations

Research works printed but not linked: 6

  1. BigScience Workshop (2022). Bloom (revision 4ab0472).
  2. Bird, S., E. Klein, and E. Loper (2009). Natural Language Processing with Python: Analyzing Text with the Natural Language Toolkit . Beijing: O'Reilly.
  3. Leahy, C. and S. Biderman (2021). The hard problem of aligning AI to human values. In The State of AI Ethics Report , Volume 4, pp. 180-183. The Montreal AI Ethics Institute.
  4. Moeljadi, D. (2012). Usage of indonesian possessive verbal predicates: A statistical analysis based on questionnaire and storytelling surveys. In 5th Conference on Austronesian and Papuan Languages and Linguistics (APLL5) , SOAS, University of London.
  5. Nomoto, H., K. Okano, D. Moeljadi, and H. Sawada (2018). Tufs asian language parallel corpus (talpco). In Proceedings of the Twenty-Fourth Annual Meeting of the Association for Natural Language Processing , pp. 436-439. Association for Natural Language Processing.
  6. Winner, L. (2017). Do artifacts have politics? In Computer Ethics , pp. 177-192. Routledge.

Cited by

Sorted by citations

Related

Authors
Laurençon, H., Saulnier, L., Wang, T., Akiki, C., del Moral, A. V., Scao, T. L., Von Werra, L., Mou, C., Ponferrada, E. G., Nguyen, H., Frohberg, J., Šaško, M., Lhoest, Q., McMillan-Major, A., Dupont, G., Biderman, S., Rogers, A., allal, L. B., De Toni, F., Pistilli, G., Nguyen, O., Nikpoor, S., Masoud, M., Colombo, P., de la Rosa, J., Villegas, P., Thrush, T., Longpre, S., Nagel, S., Weber, L., Muñoz, M., Zhu, J., Van Strien, D., Alyafeai, Z., Almubarak, K., Vu, M. C., Gonzalez-Dios, I., Soroa, A., Lo, K., Dey, M., Suarez, P. O., Gokaslan, A., Bose, S., Adelani, D., Phan, L., Tran, H., Yu, I., Pai, S., Chim, J., Lepercq, V., Ilic, S., Mitchell, M., Luccioni, S. A., & Jernite, Y.
Affiliations
  • Hugging Face (US)
  • University of Michigan (US)
  • Leipzig University (DE)
  • Ferrum College (US)
  • The National Library of Norway (NO)
  • University of Washington (US)
  • Booz Allen Hamilton (United States) (US)
  • University of Copenhagen (DK)
  • ServiceNow (United States) (US)
  • Cicatelli Associates (US)
  • CentraleSupélec (FR)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
DOI
10.48550/arxiv.2303.03915
arXiv
2303.03915
OpenAlex
W4307225507
Cited by
61
References
100
Citations by year
2025: 12 · 2024: 16 · 2023: 32 · 2022: 1
FWCI
12.07
Type
conference paper
Language
en
Open access
Licence
Locations
  • UWA Profiles and Research Repository (University of Western Australia) (publishedVersion, other-oa)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • HAL (Le Centre pour la Communication Scientifique Directe) (submittedVersion, cc-by)
  • IT University Of Copenhagen (IT University of Copenhagen) (submittedVersion, other-oa)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
Keywords
multilingual corpora, large language model training data, data curation, open science, open collaboration, responsible AI

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-06.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-06.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review