The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data Only
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
With adequate filtering, data extracted from the web can outperform datasets constructed from human-curated sources.
“Recent findings show that with adequate filtering, data extracted from the web can outperform datasets constructed from human-curated sources (Penedo et al., 2023).”
From Will we run out of data? Limits of LLM scaling based on human-generated data (2022) · cites this paper as Penedo et al., 2023
The largest datasets of human-generated public text data, such as RefinedWeb, C4, and RedPajama, contain tens of trillions of words collected from billions of web pages.
“The largest datasets of human-generated public text data , such as RefinedWeb, C4, and RedPajama, contain tens of trillions of words collected from billions of web pages (Penedo et al., 2023; Together.ai, 2023).”
From Will we run out of data? Limits of LLM scaling based on human-generated data (2022) · cites this paper as Penedo et al., 2023
The filtering part of Penedo et al.'s process reduced the size of the web dataset by around 30%.
“The filtering part of this process reduced the size of the web dataset by around 30%.”
From Will we run out of data? Limits of LLM scaling based on human-generated data (2022) · cites this paper as Penedo et al., 2023
Penedo et al. created a 5T-token dataset that outperforms curated corpora by carefully filtering and deduplicating raw data from Common Crawl.
“Penedo et al. (2023) create a 5T-token dataset that outperforms curated corpora by carefully filtering and deduplicating raw data from Common Crawl.”
From Will we run out of data? Limits of LLM scaling based on human-generated data (2022) · cites this paper as Penedo et al., 2023
Cited by
Sorted by citations
Detecting hallucinations in large language models using semantic entropy
Farquhar, S., Kossen, J., Kuhn, L., & Gal, Y. (2024) · Nature · doi:10.1038/s41586-024-07421-0 · Cited by 934 · see citing works
Will we run out of data? Limits of LLM scaling based on human-generated data
Villalobos, P., Ho, A., Sevilla, J., Besiroglu, T., Heim, L., & Hobbhahn, M. (2022) · arXiv · doi:10.48550/arxiv.2211.04325 · Cited by 80 · see citing works
RAID: A Shared Benchmark for Robust Evaluation of Machine-Generated Text Detectors
Dugan, L., Hwang, A., Trhlík, F., Zhu, A., Ludan, J. M., Xu, H., Ippolito, D., & Callison-Burch, C. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.674 · Cited by 48 · see citing works
ENEC: A Lossless AI Model Compression Method Enabling Fast Inference on Ascend NPUs
Yang, J., Wu, J., Liu, Z., Ma, X., Zhao, H., Gu, Y., Huang, Y., Liu, X., Huang, W., Wei, Z., et al. (2026) · 2026 ACM/IEEE 53rd Annual International Symposium on Computer Architecture (ISCA) · doi:10.1109/isca66397.2026.00100
- Authors
- Penedo, G., Malartic, Q., Hesslow, D., Cojocaru, R., Alobeidli, H., Cappelli, A., Pannier, B., Almazrouei, E., & Launay, J.
- Venue
- Advances in Neural Information Processing Systems 36
- Published
- 2023
- Pages
- 79155-79172
- ISSN
- 1049-5258 (ISSN-L)
- OpenAlex
- W7133205862
- Cited by
- 25
- References
- 0
- Citations by year
- 2026: 20 · 2025: 3 · 2024: 2
- FWCI
- 3.97
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- neural information processing systems (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Keywords
- large language models, web data, training data
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
