The DOI matches its Crossref record.
Dark Data: Are we solving the right problems?
Abstract
With the increasing urge of the enterprises to ingest as much data as they can in what's commonly referred to as “Data Lakes”, the new environment presents serious challenges to traditional ETL models and to building analytic layers on top of well-understood global schema. With the recent development of multiple technologies to support this “load-first” paradigm, even traditional enterprises have fairly large HDFS-based data lakes now. They have even had them long enough that their first generation IT projects delivered on some, but not all, of the promise of integrating their enterprise's data assets. In short, we moved from no data to Dark data. Dark data is what enterprises might have in their possession, without the ability to access it or with limited awareness of what this data represents. In particular, business-critical information might still remain out of reach. This panel is about Dark Data and whether we have been focusing on the right data management challenges in dealing with it.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Data quality metrics are extensively studied in the literature but are not widely used by industrial IoT systems having dark data.
“Data quality metrics are extensively studied in the literature but are not widely used by industrial IoT systems having dark data [67].”
From A Systematic Review of Data Quality in CPS and IoT for Industry 4.0 (2023) · cites this paper as [67]
Citations
Cited by
Sorted by citations
A Systematic Review of Data Quality in CPS and IoT for Industry 4.0
Goknil, A., Nguyen, P., Sen, S., Politaki, D., Niavis, H., Pedersen, K. J., Suyuthi, A., Anand, A., & Ziegenbein, A. (2023) · ACM Computing Surveys · doi:10.1145/3593043 · Cited by 84
“Data quality metrics are extensively studied in the literature but are not widely used by industrial IoT systems having dark data [67].”
BackgroundLike a rainbow in the dark: metadata annotation for HPC applications in the age of dark data
Schembera, B. (2021) · The Journal of Supercomputing · doi:10.1007/s11227-020-03602-6 · Cited by 20
“capacity, some authors even speak of a paradigm shift from “no data” to “dark data” [8].”
BackgroundA New Approach to Information Extraction in User-Centric E-Recruitment Systems
Awan, M. N. A., Khan, S., Latif, K., et al. (2019) · Applied Sciences · doi:10.3390/app9142852 · Cited by 11
“Deepdive [7,31] is also an effort for populating knowledge bases from dark data [32].”
BackgroundA Framework for Image Dark Data Assessment
Liu, Y., Wang, Y., Zhou, K., et al. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-26072-9_1 · Cited by 8
Khan, R., & Gubanov, M. N. (2020) · Proceedings of the ... ACM International Conference on Information & Knowledge Management · doi:10.1145/3340531.3417443 · Cited by 7
A Data Market with Decentralized Repositories.
Ivanschitz, B.-P., Lampoltshammer, T. J., Mireles, V., et al. (2018) · Cited by 6
Esmaeili, L., & Golpayegani, A. H. (2019) · Journal of theoretical and applied electronic commerce research · doi:10.4067/s0718-18762020000200107 · Cited by 4
“Dark data is generally in an ideal state, and it is a subset of big data [17].”
BackgroundDeep Hash-based Relevance-aware Data Quality Assessment for Image Dark Data
Liu, Y., Wang, Y., Gao, L., et al. (2021) · ACM/IMS Transactions on Data Science · doi:10.1145/3420038 · Cited by 4
A framework for image dark data assessment
Zhou, K., Wang, Y., Liu, Y., et al. (2020) · World Wide Web · doi:10.1007/s11280-020-00779-x · Cited by 4
Enhancing Real-Time Analytics: Streaming Data Quality Metrics for Continuous Monitoring
Silva, E. C. E., Oliveira, Ó., & Oliveira, B. (2024) · International Conference on Mathematics and Statistics (ICoMS) · doi:10.1145/3686592.3686609 · Cited by 3
Entropy-Based Feature Selection for Network Anomaly Detection
Alabi, R., & Yurtkan, K. (2018) · International Symposium on Multidisciplinary Studies and Innovative Technologies (ISMSIT) · doi:10.1109/ismsit.2018.8566694 · Cited by 2
“As a result of so many devices communicating, a huge volume of data is produced that is not used completely, in recent years, the concept of dark data has been delved into by various researchers ([2]-[4]).”
BackgroundFAIRifying STEM Data Ecosystem to Enhance Data Reuse
Namgay, P., Wangdi, P., Thinley, S., et al. (2023) · 2022 IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie58773.2023.10343320 · Cited by 1
“The STEM community needs to be pragmatic in dealing with one’s digital assets, such as learning resources, teaching materials, and research data sets, for the well-timed harnessing of its value, thereby preventing a ‘dark data’ situation [12, 14].”
BackgroundInfluentialStrong filters: Toward an interpretable architecture for large-scale health data
Zadorozhny, V., Liu, Z., & Elmer, J. (2026) · Journal of Intelligent Information Systems · doi:10.1007/s10844-026-01067-2 · Cited by 1
“Lossless methods (Baskin et al., 2023; Cafarella et al., 2016; Costa et al., 2018, 2023, 2019) are well-suited for long-term archival because they preserve data accuracy in full.”
Dados escuros à luz do controle público
de Albuquerque, A. M., & Dorneles, C. F. (2022) · doi:10.5753/sbbd.2022.224331 · Cited by 1
Gill, A. Q., & Nikiforova, A. (2025) · Lecture notes in business information processing · doi:10.1007/978-3-031-94590-8_12
Natural Language Processing Methodology for Data Generation and Sharing
Kavitha, J., Vethamani, S. E., Harilakshmi, V. S., et al. (2025) · Lecture notes in networks and systems · doi:10.1007/978-981-96-5214-3_18
Metadata
- Authors
- Cafarella, M., Ilyas, I. F., Kornacker, M., Kraska, T., & Re, C.
- Affiliations
- University of Michigan (US)
- University of Waterloo (CA)
- Cloudera (United States) (US)
- Brown University (US)
- Venue
- 2016 IEEE 32nd International Conference on Data Engineering (ICDE)
- Publisher
- IEEE
- Published
- 2016-05
- Pages
- 1444-1445
- ISSN
- 1084-4627 (ISSN-L)
- OpenAlex
- W2438459259
- Cited by
- 16
- References
- 0
- Citations by year
- 2026: 1 · 2025: 2 · 2024: 1 · 2023: 2 · 2022: 1 · 2021: 2 · 2020: 2 · 2019: 3 · 2018: 2
- FWCI
- 2.87
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE International Conference on Data Engineering (ICDE) (publishedVersion)
- Topics
- Social Sciences › Business, Management and Accounting › Management Information Systems › Big Data and Business Intelligence
- Physical Sciences › Computer Science › Information Systems › Data Quality and Management
- Social Sciences › Decision Sciences › Information Systems and Management › Big Data Technologies and Applications
- Keywords
- dark data, data lakes, data management, ETL, HDFS, data integration, data governance
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
