CoChat
Sign Up
Cited by 30Natural Language Processing TechniquesWeb Data Mining and AnalysisTopic Modeling

C4Corpus: Multilingual Web-size Corpus with Free License

Open access
Habernal, Ivan, Zayed, Omnia, Gurevych, Iryna·2016·Proceedings of the Language Resources and Evaluation Conference
DOI 10.63317/3iti4i8msr33
Open this paper in CoChat — free accountOpen PDF

Abstract

Large Web corpora containing full documents with permissive licenses are crucial for many NLP tasks.In this article we present the construction of 12 million-pages Web corpus (over 10 billion tokens) licensed under CreativeCommons license family in 50+ languages that has been extracted from CommonCrawl, the largest publicly available general Web crawl to date with about 2 billion crawled URLs.Our highly-scalable Hadoop-based framework is able to process the full CommonCrawl corpus on 2000+ CPU cluster on the Amazon Elastic Map/Reduce infrastructure.The processing pipeline includes license identification, state-of-the-art boilerplate removal, exact duplicate and near-duplicate document removal, and language detection.The construction of the corpus is highly configurable and fully reproducible, and we provide both the framework (DKPro C4CorpusTools) and the resulting data (C4Corpus) to the research community.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.63317/3iti4i8msr33

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Prior data sets use a more limited set of filtering heuristics, are not publicly available, and/or are different in scope.

    “However, we opted to create a new data set because prior data sets use a more limited set of filtering heuristics, are not publicly available, and/or are different in scope (e.g. are limited to News data (Zellers et al., 2019; Liu et al., 2019c), comprise only Creative Commons content (Habernal et al., 2016), or are focused on parallel training data for machine translation (Smith et al., 2013)).”

    From Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019) · cites this paper as Habernal et al., 2016

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

References 29 · Cited by 30

Research works printed but not linked: 30

  1. Asheghi, N. R., Sharoff, S., and Markert, K. (2016). Crowdsourcing for web genre annotation. Language Re- sources and Evaluation, in press, jan.
  2. Barbaresi, A. and Würzner, K.-M. (2014). For a fistful of blogs: Discovery and comparative benchmarking of re- publishable German content. In Gertrud Faaßet al., ed- itors, Workshop Proceedings of the 12th Edition of the KONVENS Conference, pages 2-10, Hildesheim, Ger- many. Universität Hildesheim.
  3. Baroni, M., Chantree, F., Kilgarriff, A., and Sharoff, S. (2008). Cleaneval: a Competition for Cleaning Web Pages. In Nicoletta Calzolari, et al., editors, Proceed- ings of the Sixth International Conference on Language Resources and Evaluation (LREC'08), Marrakech, Mo- rocco, may. European Language Resources Association (ELRA).
  4. Baroni, M., Bernardini, S., Ferraresi, A., and Zanchetta, E. (2009). The WaCky wide web: a collection of very large linguistically processed web-crawled corpora. Language Resources and Evaluation, 43(3):209-226.
  5. Biemann, C., Bildhauer, F., Evert, S., Goldhahn, D., Quasthoff, U., Schäfer, R., Simon, J., Swiezinski, L., and Zesch, T. (2013). Scalable construction of high- quality web corpora. Journal for Language Technology and Computational Linguistics, 28(2):23-59.
  6. Bing, L., Li, P., Liao, Y., Lam, W., Guo, W., and Pas- sonneau, R. (2015). Abstractive multi-document sum- marization via phrase selection and merging. In Pro- ceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Vol- ume 1: Long Papers), pages 1587-1597, Beijing, China, July. Association for Computational Linguistics.
  7. Boldi, P., Marino, A., Santini, M., and Vigna, S. (2016). BUbiNG: Massive Crawling for the Masses. arXiv preprint. http://arxiv.org/abs/1601.06919. · arXiv
  8. Buck, C., Heafield, K., and van Ooyen, B. (2014). N- gram Counts and Language Models from the Common Crawl. In Nicoletta Calzolari, et al., editors, Proceedings of the Ninth International Conference on Language Re- sources and Evaluation (LREC'14), pages 3579-3584, Reykjavik, Iceland. European Language Resources As- sociation (ELRA).
  9. Charikar, M. S. (2002). Similarity estimation techniques from rounding algorithms. In Proceedings of the Thiry- fourth Annual ACM Symposium on Theory of Comput- ing, STOC '02, pages 380-388, New York, NY, USA. ACM.
  10. Evert, S. (2008). A lightweight and efficient tool for clean- ing web pages. In Nicoletta Calzolari, et al., editors, Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC'08), Mar- rakech, Morocco. European Language Resources Asso- ciation (ELRA).
  11. Faaß, G. and Eckart, K. (2013). Sdewac -a corpus of parsable sentences from the web. In Iryna Gurevych, et al., editors, Proceedings of 25th GSCL International Conference, pages 61-68, Darmstadt, GE. Springer Berlin / Heidelberg.
  12. Gasieniec, L., Jansson, J., and Lingas, A. (2004). Approx- imation algorithms for hamming clustering problems. Journal of Discrete Algorithms, 2(2):289 -301. Com- binatiorial Pattern Matching.
  13. Goldhahn, D., Eckart, T., and Quasthoff, U. (2012). Building large monolingual dictionaries at the Leipzig corpora collection: From 100 to 200 languages. In Nicoletta Calzolari, et al., editors, Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC-2012), pages 759-765, Istanbul, Turkey, May. European Language Resources Associa- tion (ELRA).
  14. Habernal, I. and Gurevych, I. (2015). Exploiting Debate Portals for Semi-Supervised Argumentation Mining in User-Generated Web Discourse. In Proceedings of the 2015 Conference on Empirical Methods in Natural Lan- guage Processing, pages 2127-2137, Lisbon, Portugal. Association for Computational Linguistics.
  15. Hládek, D., Staš, J., and Juhár, J. (2014). Slovak Web Discussion Corpus. In Adam Przepiórkowski et al., edi- tors, 9th International Conference on NLP, PolTAL 2014, pages 463-469, Warsaw, PL. Springer International Pub- lishing.
  16. Lijffijt, J., Nevalainen, T., Säily, T., Papapetrou, P., Puo- lamäki, K., and Mannila, H. (2014). Significance testing of word frequencies in corpora. Digital Scholarship in the Humanities, pages 1-24.
  17. Ljubešić, N. and Klubička, F. (2014). {bs,hr,sr}WaC - Web corpora of Bosnian, Croatian and Serbian. In Fe- lix Bildhauer et al., editors, Proceedings of the 9th Web as Corpus Workshop (WAC9)@EACL 2014, pages 29- 35, Gothenburg, Sweden. Association for Computational Linguistics.
  18. Lyding, V., Stemle, E., Borghetti, C., Brunello, M., Castag- noli, S., Dell'Orletta, F., Dittmann, H., Lenci, A., and Pirrelli, V. (2014). The PAIS À Corpus of Italian Web Texts. In Felix Bildhauer et al., editors, Proceedings of the 9th Web as Corpus Workshop (WAC9) @ EACL 2014, pages 36-43, Gothenburg, Sweden. Association for Computational Linguistics.
  19. Manku, G. S., Jain, A., and Das Sarma, A. (2007). Detect- ing near-duplicates for web crawling. In Proceedings of the 16th International Conference on World Wide Web, WWW '07, pages 141-150, New York, NY, USA. ACM.
  20. Napoles, C., Gormley, M., and Van Durme, B. (2012). An- notated Gigaword. In Proceedings of the Joint Work- shop on Automatic Knowledge Base Construction and Web-scale Knowledge Extraction (AKBC-WEKEX '12), pages 95-100, Montreal, Canada. Association for Com- putational Linguistics.
  21. Petrov, S., Das, D., and McDonald, R. (2012). A universal part-of-speech tagset. In Nicoletta Calzolari, et al., edi- tors, Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC-2012), pages 2089-2096, Istanbul, Turkey, May. European Lan- guage Resources Association (ELRA).
  22. Pomikálek, J. (2011). Removing boilerplate and duplicate content from web corpora. Ph.D. thesis, Masaryk uni- versity, Faculty of informatics, Brno, Czech Republic.
  23. Schäfer, R. and Bildhauer, F. (2012). Building large cor- pora from the web using a new efficient tool chain. In Nicoletta Calzolari, et al., editors, Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC-2012), pages 486-493, Istanbul, Turkey, May. European Language Resources Associa- tion (ELRA).
  24. Schäfer, R. (2015). Processing and querying large web corpora with the COW14 architecture. In Piotr Bański, et al., editors, Proceedings of the 3 rd Workshop on Chal- lenges in the Management of Large Corpora (CMLC-3), pages 28-34, Lancaster, UK, July.
  25. Sharoff, S. (2013). Measuring the Distance Between Com- parable Corpora Between Languages. In Serge Sharoff, et al., editors, Building and Using Comparable Corpora, pages 113-130. Springer Berlin Heidelberg, Berlin, Hei- delberg.
  26. Smith, J. R., Saint-Amand, H., Plamada, M., Koehn, P., Callison-Burch, C., and Lopez, A. (2013). Dirt Cheap Web-Scale Parallel Text from the Common Crawl. In Proceedings of the 51st Annual Meeting of the Associa- tion for Computational Linguistics (Volume 1: Long Pa- pers), pages 1374-1383, Sofia, Bulgaria. Association for Computational Linguistics.
  27. Spoustová, J. and Spousta, M. (2012). A High-Quality Web Corpus of Czech. In Nicoletta Calzolari, et al., edi- tors, Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC-2012), pages 311-315, Istanbul, Turkey. European Language Resources Association (ELRA).
  28. Suchomel, V. and Pomikálek, J. (2012). Efficient web crawling for large text corpora. In Serge Sharoff et al., editors, Proceedings of the seventh Web as Corpus Work- shop (WAC7), pages 39-43, Lyon.
  29. Versley, Y. and Panchenko, Y. (2012). Not Just Bigger: Towards Better-Quality Web Corpora. In Proceedings of the seventh Web as Corpus Workshop (WAC7) @WWW 2012, pages 44-52, Lyon. Association for Computa- tional Linguistics.
  30. White, T. (2015). Hadoop: The Definitive Guide. O'Reilly Media, Inc., Sebastopol, CA, 4th edition.

Authors
Habernal, I., Zayed, O., & Gurevych, I.
Affiliations
Technische Universität Darmstadt (DE)
ORCID
Venue
Proceedings of the Language Resources and Evaluation Conference
Publisher
European Language Resources Association (ELRA)
Published
2016-05
Pages
914-922
ISSN
2522-2686
DOI
10.63317/3iti4i8msr33
OpenAlex
W2578673564
Cited by
30
References
29
Citations by year
2025: 1 · 2024: 1 · 2023: 2 · 2022: 1 · 2021: 5 · 2020: 4 · 2019: 4 · 2018: 3 · 2017: 1 · 2016: 8
FWCI
2.87
Type
conference paper
Language
en
Open access
Licence
cc-by-nc
Locations
  • Language Resources and Evaluation Conference (LREC) (publishedVersion, cc-by-nc)
  • TUbilio (Technical University of Darmstadt) (acceptedVersion)
  • Language Resources and Evaluation
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Information Systems › Web Data Mining and Analysis
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
Creative Commons licenses, Common Crawl, boilerplate removal, language identification, Hadoop

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review