The DOI matches its Crossref record; 27 of the 27 references it cites were found in a registry or an index.
NADEEF: a commodity data cleaning system
Abstract
Despite the increasing importance of data quality and the rich theoretical and practical contributions in all aspects of data cleaning, there is no single end-to-end off-the-shelf solution to (semi-)automate the detection and the repairing of violations w.r.t. a set of heterogeneous and ad-hoc quality constraints. In short, there is no commodity platform similar to general purpose DBMSs that can be easily customized and deployed to solve application-specific data quality problems. In this paper, we present NADEEF, an extensible, generalized and easy-to-deploy data cleaning platform. NADEEF distinguishes between a programming interface and a core to achieve generality and extensibility. The programming interface allows the users to specify multiple types of data quality rules, which uniformly define what is wrong with the data and (possibly) how to repair it through writing code that implements predefined classes. We show that the programming interface can be used to express many types of data quality rules beyond the well known CFDs (FDs), MDs and ETL rules. Treating user implemented interfaces as black-boxes, the core provides algorithms to detect errors and to clean data. The core is designed in a way to allow cleaning algorithms to cope with multiple rules holistically, i.e. detecting and repairing data errors without differentiating between various types of rules. We showcase two implementations for core repairing algorithms. These two implementations demonstrate the extensibility of our core, which can also be replaced by other user-provided algorithms. Using real-life data, we experimentally verify the generality, extensibility, and effectiveness of our system.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Data mining and knowledge discovery: making sense out of data
Feyyad, U. M. (1996) · IEEE Expert · doi:10.1109/64.539013 · Cited by 4,612
Consistent query answers in inconsistent databases
Arenas, M., Bertossi, L., & Chomicki, J. (1999) · Proceedings of the eighteenth ACM SIGMOD-SIGACT-SIGART symposium on Principles of database systems · doi:10.1145/303976.303983 · Cited by 868
Real-world Data is Dirty: Data Cleansing and The Merge/Purge Problem
Hernández, M. A., & Stolfo, S. J. (1998) · Data Mining and Knowledge Discovery · doi:10.1023/a:1009761603038 · Cited by 812
Frontiers in Artificial Intelligence and Applications
To, W., Tryfonas, T., & FARTHING, D. W. (2009) · IOS Press eBooks · Cited by 720
A Survey of Indexing Techniques for Scalable Record Linkage and Deduplication
Christen, P. (2012) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2011.127 · Cited by 679
(2006) · Data-Centric Systems and Applications · doi:10.1007/3-540-33173-5 · Cited by 659
Potter's Wheel: An Interactive Data Cleaning System
Raman, V., & Hellerstein, J. L. (2001) · Cited by 492
Swoosh: a generic approach to entity resolution
Benjelloun, O., García-Molina, H., Menestrina, D., Su, Q., Whang, S. E., & Widom, J. (2008) · The VLDB Journal · doi:10.1007/s00778-008-0098-x · Cited by 489
Conditional functional dependencies for capturing data inconsistencies
Fan, W., Geerts, F., Jia, X., & Kementsietsidis, A. (2008) · ACM Transactions on Database Systems · doi:10.1145/1366102.1366103 · Cited by 460
A cost-based model and effective heuristic for repairing constraints by value modification
Bohannon, P., Fan, W., Flaster, M., & Rastogi, R. (2005) · Proceedings of the 2005 ACM SIGMOD international conference on Management of data · doi:10.1145/1066157.1066175 · Cited by 424
Data Quality: Concepts, Methodologies and Techniques (Data-Centric Systems and Applications)
Batini, C., & Scannapieco, M. (2006) · Springer eBooks · Cited by 321
Declarative Data Cleaning: Language, Model, and Algorithms
Galhardas, H., Florescu, D., Shasha, D. E., Simon, E., & Saita, C.-A. (2001) · Cited by 320
Improving data quality: consistency and accuracy
Cong, G., Fan, W., Geerts, F., et al. (2007) · Edinburgh Research Explorer (University of Edinburgh) · Cited by 313
TAILOR: a record linkage toolbox
Elfeky, M. G., Verykios, V. S., & Elmagarmid, A. K. (2003) · Proceedings 18th International Conference on Data Engineering · doi:10.1109/icde.2002.994694 · Cited by 289
Self-citationHandbook of Satisfiability: Volume 185 Frontiers in Artificial Intelligence and Applications
Biere, A., Heule, M. J. H., van Maaren, H., & Walsh, T. (2009) · Cited by 251
Dependencies revisited for improving data quality
Fan, W. (2008) · Proceedings of the twenty-seventh ACM SIGMOD-SIGACT-SIGART symposium on Principles of database systems · doi:10.1145/1376916.1376940 · Cited by 240
Reasoning about record matching rules
Fan, W., Jia, X., Li, J., & Ma, S. (2009) · Proceedings of the VLDB Endowment · doi:10.14778/1687627.1687674 · Cited by 226
Yakout, M., Elmagarmid, A. K., Neville, J., Ouzzani, M., & Ilyas, I. F. (2011) · Proceedings of the VLDB Endowment · doi:10.14778/1952376.1952378 · Cited by 216
Self-citationDatabase repairing using updates
Wijsen, J. (2005) · ACM Transactions on Database Systems · doi:10.1145/1093382.1093385 · Cited by 208
On approximating optimum repairs for functional dependency violations
Kolahi, S., & Lakshmanan, L. V. S. (2009) · Proceedings of the 12th International Conference on Database Theory · doi:10.1145/1514894.1514901 · Cited by 206
Towards certain fixes with editing rules and master data
Fan, W., Li, J., Ma, S., Tang, N., & Yu, W. (2010) · Proceedings of the VLDB Endowment · doi:10.14778/1920841.1920867 · Cited by 202
Self-citationERACER: a database approach for statistical inference and data cleaning
Mayfield, C., Neville, J., & Prabhakar, S. (2010) · Proceedings of the 2010 ACM SIGMOD International Conference on Management of data · doi:10.1145/1807167.1807178 · Cited by 173
Zchaff2004: An Efficient SAT Solver
Mahajan, Y. S., Fu, Z., & Malik, S. (2005) · Lecture notes in computer science · doi:10.1007/11527695_27 · Cited by 147
Towards certain fixes with editing rules and master data
Fan, W., Li, J., Ma, S., Tang, N., & Yu, W. (2011) · The VLDB Journal · doi:10.1007/s00778-011-0253-7 · Cited by 128
Self-citationInteraction between record matching and data repairing
Fan, W., Li, J., Ma, S., Tang, N., & Yu, W. (2011) · Proceedings - ACM-SIGMOD International Conference on Management of Data · doi:10.1145/1989323.1989373 · Cited by 116
Self-citationData Fusion in Three Steps: Resolving Schema, Tuple, and Value Inconsistencies.
Naumann, F., Bilke, A., Bleiholder, J., & Weis, M. (2006) · IEEE Data(base) Engineering Bulletin · Cited by 77
Generic entity resolution with negative rules
Whang, S. E., Benjelloun, O., & García-Molina, H. (2009) · The VLDB Journal · doi:10.1007/s00778-009-0136-3 · Cited by 54
Cited by
Data Cleaning: Overview and Emerging Challenges
Chu, X., Ilyas, I. F., Krishnan, S., & Wang, J. (2016) · Proceedings of the 2016 International Conference on Management of Data · doi:10.1145/2882903.2912574 · Cited by 488
Self-citationHoloClean: holistic data repairs with probabilistic inference
Rekatsinas, T., Chu, X., Ilyas, I. F., & Ré, C. (2017) · Proceedings of the VLDB Endowment · doi:10.14778/3137628.3137631 · Cited by 480
Self-citationProfiling relational data: a survey
Abedjan, Z., Golab, L., & Naumann, F. (2015) · The VLDB Journal · doi:10.1007/s00778-015-0389-y · Cited by 283
KATARA: A Data Cleaning System Powered by Knowledge Bases and Crowdsourcing
Chu, X., Morcos, J., Ilyas, I. F., Ouzzani, M., Papotti, P., Tang, N., & Ye, Y. (2015) · Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data · doi:10.1145/2723372.2749431 · Cited by 279
Self-citationData quality: The other face of Big Data
Saha, B., & Srivastava, D. (2014) · 2014 IEEE 30th International Conference on Data Engineering · doi:10.1109/icde.2014.6816764 · Cited by 262
Metadata
- Authors
- Dallachiesa, M., Ebaid, A., Eldawy, A., Elmagarmid, A., Ilyas, I. F., Ouzzani, M., & Tang, N.
- Venue
- Proceedings of the 2013 ACM SIGMOD International Conference on Management of Data
- Publisher
- ACM
- Published
- 2013-06-22
- Pages
- 541-552
- ISSN
- 0730-8078 (ISSN-L)
- OpenAlex
- W2046298800
- Cited by
- 313
- References
- 27
- FWCI
- 48.95
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.