The DOI matches its Crossref record; 16 of the 16 references it cites were found in a registry or an index.
Wrangling messy CSV files by detecting row and type patterns
Open accessAbstract
Data scientists spend the majority of their time on preparing data for analysis. One of the first steps in this preparation phase is to load the data from the raw storage format. Comma-separated value (CSV) files are a popular format for tabular data due to their simplicity and ostensible ease of use. However, formatting standards for CSV files are not followed consistently, so each file requires manual inspection and potentially repair before the data can be loaded, an enormous waste of human effort for a task that should be one of the simplest parts of data science. The first and most essential step in retrieving data from CSV files is deciding on the dialect of the file, such as the cell delimiter and quote character. Existing dialect detection approaches are few and non-robust. In this paper, we propose a dialect detection method based on a novel measure of data consistency of parsed data files. Our method achieves 97% overall accuracy on a large corpus of real-world CSV files and improves the accuracy on messy CSV files by almost 22% compared to existing approaches, including those in the Python standard library. Our measure of data consistency is not specific to the data parsing problem, and has potential for more general applicability.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A relational model of data for large shared data banks
Codd, E. F. (1970) · Communications of the ACM · doi:10.1145/362384.362685 · Cited by 5,547 · cited 2×
“(1) where the sets Vj are the domains of values in the tuples (Codd, 1970).”
BackgroundThe application/json Media Type for JavaScript Object Notation (JSON)
Crockford, D. (2006) · doi:10.17487/rfc4627 · Cited by 963
Wickham, H. (2014) · Journal of Statistical Software · doi:10.18637/jss.v059.i10 · Cited by 901
Exploratory Data Mining and Data Cleaning
Dasu, T., & Johnson, T. (2003) · Wiley Series in Probability and Statistics · doi:10.1002/0471448354 · Cited by 618
Wrangler: interactive visual specification of data transformation scripts
Kandel, S., Paepcke, A., Hellerstein, J., & Heer, J. (2011) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1978942.1979444 · Cited by 581
Table extraction using conditional random fields
Pinto, D., McCallum, A., Wei, X., & Croft, W. B. (2003) · Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval · doi:10.1145/860435.860479 · Cited by 355
Common Format and MIME Type for Comma-Separated Values (CSV) Files
Shafranovich, Y. (2005) · doi:10.17487/rfc4180 · Cited by 295
Guo, P. J., Kandel, S., Hellerstein, J. M., et al. (2011) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/2047196.2047205 · Cited by 110
8th USENIX Workshop on the Theory and Practice of Provenance (TaPP '16)
Abreu, R., Archer, D., Chapman, E., Cheney, J., Eldardiry, H., & Gascón, A. (2016) · Cited by 72
FlashExtract: a framework for data extraction by examples
Le, V., & Gulwani, S. (2014) · ACM SIGPLAN Notices · doi:10.1145/2666356.2594333 · Cited by 59
Research works printed but not linked: 4
- Arnold VI (2003) Catastrophe theory. Springer, Berlin
- Kleene SC (1956) Representation of events in nerve nets and finite automata. In: Shannon CE, McCarthy J (eds) Automata studies. Princeton University Press, Princeton
- Ng HT, Lim CY, Koo JLT (1999) Learning to recognize tables in free text. In: Proceedings of the 37th annual meeting of the association for computational linguistics, ACL, pp. 443–450
- Raymond ES (2003) The art of Unix programming. Addison-Wesley Professional, Boston
Metadata
- Authors
- van den Burg, G. J. J., Nazábal, A., & Sutton, C.
- Affiliations
- The Alan Turing Institute (GB)
- Google (United States) (US)
- Corresponding author
- van den Burg, G. J. J.
- Venue
- Data Mining and Knowledge Discovery
- Published
- 2019-07-26
- Volume
- 33
- Issue
- 6
- Pages
- 1799-1820
- ISSN
- 1384-5810 (ISSN-L)
- 1573-756X
- OpenAlex
- W2902696364
- Cited by
- 42
- References
- 16
- Citations by year
- 2026: 7 · 2025: 12 · 2024: 6 · 2023: 9 · 2022: 3 · 2021: 1 · 2020: 4
- FWCI
- 1.36
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- www.springer.com/tdm (tdm)
- www.springer.com/tdm (vor)
- Locations
- Data Mining and Knowledge Discovery (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- UK Government’s Defense & Security Programme
- Topics
- Physical Sciences › Computer Science › Information Systems › Data Quality and Management
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Physical Sciences › Computer Science › Information Systems › Web Data Mining and Analysis
- Keywords
- CSV files, data parsing, data preparation, data loading
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Wrangling messy CSV files by detecting row and type patterns. https://app.cochat.ai/papers/10.1007/s10618-019-00646-y
- The paper
- van den Burg, G. J. J., Nazábal, A., & Sutton, C. (2019). Wrangling messy CSV files by detecting row and type patterns. Data Mining and Knowledge Discovery. https://doi.org/10.1007/s10618-019-00646-y
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
