The DOI matches its Crossref record; 94 of the 94 references it cites were found in a registry or an index.
Big Data Integration
Abstract
The Big Data era is upon us: data is being generated, collected and analyzed at an unprecedented scale, and data-driven decision making is sweeping through society. Since the value of data explodes when it can be linked and fused with other data, addressing the big data integration (BDI) challenge is critical to realizing the promise of Big Data. BDI differs from traditional data integration in many dimensions: (i) the number of data sources, even for a single domain, has grown to be in the tens of thousands, (ii) many of the data sources are very dynamic, as a huge amount of newly collected data are continuously made available, (iii) the data sources are extremely heterogeneous in their structure, with considerable variety even for substantially similar entities, and (iv) the data sources are of widely differing qualities, with significant differences in the coverage, accuracy and timeliness of data provided. This tutorial explores the progress that has been made by the data integration community on the topics of schema mapping, record linkage and data fusion in addressing these novel challenges faced by big data integration, and identifies a range of open problems for the community.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538
Artificial intelligence: a modern approach
(1995) · Choice Reviews Online · doi:10.5860/choice.33-1577 · Cited by 21,974
The anatomy of a large-scale hypertextual Web search engine
Brin, S., & Page, L. (1998) · Computer Networks and ISDN Systems · doi:10.1016/s0169-7552(98)00110-x · Cited by 16,004
Davies, D. L., & Bouldin, D. W. (1979) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.1979.4766909 · Cited by 9,145
Authoritative sources in a hyperlinked environment
Kleinberg, J. M. (1999) · Journal of the ACM · doi:10.1145/324133.324140 · Cited by 8,959
Freebase: a collaboratively created graph database for structuring human knowledge
Bollacker, K., Evans, C., Paritosh, P., Sturge, T., & Taylor, J. (2008) · Proceedings of the 2008 ACM SIGMOD international conference on Management of data · doi:10.1145/1376616.1376746 · Cited by 4,852
A survey of approaches to automatic schema matching
Rahm, E., & Bernstein, P. A. (2001) · The VLDB Journal · doi:10.1007/s007780100057 · Cited by 3,352
MapReduce: Simplified Data Processing on Large Cluster
Senior Software Developer, A. U., & Dayalan, M. (2018) · International Journal of Research and Engineering · doi:10.21276/ijre.2018.5.5.4 · Cited by 2,948
Yago: a core of semantic knowledge
Suchanek, F. M., Kasneci, G., & Weikum, G. (2007) · Proceedings of the 16th international conference on World Wide Web · doi:10.1145/1242572.1242667 · Cited by 2,512
Fellegi, I. P., & Sunter, A. B. (1969) · Journal of the American Statistical Association · doi:10.1080/01621459.1969.10501049 · Cited by 2,475
Bleiholder, J., & Naumann, F. (2009) · ACM Computing Surveys · doi:10.1145/1456650.1456651 · Cited by 1,838
Answering queries using views: A survey
Halevy, A. Y. (2001) · The VLDB Journal · doi:10.1007/s007780100054 · Cited by 1,510
Crowdsourcing systems on the World-Wide Web
Doan, A., Ramakrishnan, R., & Halevy, A. Y. (2011) · Communications of the ACM · doi:10.1145/1924421.1924442 · Cited by 1,338
Mining the Web for Synonyms: PMI-IR versus LSA on TOEFL
Turney, P. D. (2001) · Lecture Notes in Computer Science · doi:10.1007/3-540-44795-4_42 · Cited by 1,285
Duplicate Record Detection: A Survey
Elmagarmid, A. K., Ipeirotis, P. G., & Verykios, V. S. (2007) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2007.250581 · Cited by 1,267
Settles, B. (2012) · Cited by 1,183
Efficient clustering of high-dimensional data sets with application to reference matching
McCallum, A., Nigam, K., & Ungar, L. H. (2000) · Proceedings of the sixth ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/347090.347123 · Cited by 1,096
Bansal, N., Blum, A., & Chawla, S. (2004) · Machine Learning · doi:10.1023/b:mach.0000033116.57574.95 · Cited by 1,075
Real-world Data is Dirty: Data Cleansing and The Merge/Purge Problem
Hernández, M. A., & Stolfo, S. J. (1998) · Data Mining and Knowledge Discovery · doi:10.1023/a:1009761603038 · Cited by 812
Min-Wise Independent Permutations
Broder, A. Z., Charikar, M., Frieze, A. M., & Mitzenmacher, M. (2000) · Journal of Computer and System Sciences · doi:10.1006/jcss.1999.1690 · Cited by 802
Probase: a probabilistic taxonomy for text understanding
Wu, W., Li, H., Wang, H., & Zhu, K. Q. (2012) · Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data · doi:10.1145/2213836.2213891 · Cited by 763
From databases to dataspaces: a new abstraction for information management
Franklin, M., Halevy, A., & Maier, D. (2005) · ACM SIGMOD Record · doi:10.1145/1107499.1107502 · Cited by 757
Interactive deduplication using active learning
Sarawagi, S., & Bhamidipaty, A. (2002) · Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/775047.775087 · Cited by 692
(2006) · Data-Centric Systems and Applications · doi:10.1007/3-540-33173-5 · Cited by 659
Truth Discovery with Multiple Conflicting Information Providers on the Web
Yin, X., Han, J., & Yu, P. S. (2008) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2007.190745 · Cited by 648
WebTables: exploring the power of tables on the web
Cafarella, M. J., Halevy, A., Wang, D. Z., Wu, E., & Zhang, Y. (2008) · Proceedings of the VLDB Endowment · doi:10.14778/1453856.1453916 · Cited by 629
Principles of Data Integration
(2012) · Elsevier eBooks · doi:10.1016/c2011-0-06130-6 · Cited by 518
CrowdER: crowdsourcing entity resolution
Wang, J., Kraska, T., Franklin, M. J., & Feng, J. (2012) · Proceedings of the VLDB Endowment · doi:10.14778/2350229.2350263 · Cited by 507
Settles, B. (2012) · Synthesis Lectures on Artificial Intelligence and Machine Learning · doi:10.1007/978-3-031-01560-1 · Cited by 502
Complexity of answering queries using materialized views
Abiteboul, S., & Duschka, O. M. (1998) · Proceedings of the seventeenth ACM SIGACT-SIGMOD-SIGART symposium on Principles of database systems · doi:10.1145/275487.275516 · Cited by 483
Maximizing Non-monotone Submodular Functions
Feige, U., Mirrokni, V. S., & Vondrák, J. (2011) · SIAM Journal on Computing · doi:10.1137/090779346 · Cited by 452
Evaluation of entity resolution approaches on real-world match problems
Köpcke, H., Thor, A., & Rahm, E. (2010) · Proceedings of the VLDB Endowment · doi:10.14778/1920841.1920904 · Cited by 443
The Hungarian Method for the Assignment Problem
Kuhn, H. W. (2010) · 50 Years of Integer Programming 1958-2008 · doi:10.1007/978-3-540-68279-0_2 · Cited by 419
Annotating and searching web tables using entities, types and relationships
Limaye, G., Sarawagi, S., & Chakrabarti, S. (2010) · Proceedings of the VLDB Endowment · doi:10.14778/1920841.1921005 · Cited by 391
PARIS: probabilistic alignment of relations, instances, and schema
Suchanek, F. M., Abiteboul, S., & Senellart, P. (2011) · Proceedings of the VLDB Endowment · doi:10.14778/2078331.2078332 · Cited by 358
Bellahsène, Z., Bonifati, A., & Rahm, E. (2011) · Springer eBooks · doi:10.1007/978-3-642-16518-4 · Cited by 358
Entity resolution: theory, practice & open challenges
Getoor, L., & Machanavajjhala, A. (2012) · Proceedings of the VLDB Endowment · doi:10.14778/2367502.2367564 · Cited by 348
Principles of dataspace systems
Halevy, A., Franklin, M., & Maier, D. (2006) · Proceedings of the twenty-fifth ACM SIGMOD-SIGACT-SIGART symposium on Principles of database systems · doi:10.1145/1142351.1142352 · Cited by 346
Recovering semantics of tables on the web
Venetis, P., Halevy, A., Madhavan, J., Paşca, M., Shen, W., Wu, F., Miao, G., & Wu, C. (2011) · Proceedings of the VLDB Endowment · doi:10.14778/2002938.2002939 · Cited by 340
Knowing What to Believe (when you already know something)
Pasternack, J., & Roth, D. (2010) · International Conference on Computational Linguistics · Cited by 332
He, B., Patel, M., Zhang, Z., & Chang, K. C.-C. (2007) · Communications of the ACM · doi:10.1145/1230819.1241670 · Cited by 327
Data integration with uncertainty
Dong, X. L., Halevy, A. Y., & Yu, C. (2008) · The VLDB Journal · doi:10.1007/s00778-008-0119-9 · Cited by 321
Self-citationCorroborating information from disagreeing views
Galland, A., Abiteboul, S., Marian, A., & Senellart, P. (2010) · Proceedings of the third ACM international conference on Web search and data mining · doi:10.1145/1718487.1718504 · Cited by 318
Integrating conflicting data: the role of source dependence
Dong, X. L., Berti-Equille, L., & Srivastava, D. (2009) · Proceedings of the VLDB Endowment · doi:10.14778/1687627.1687690 · Cited by 312
Self-citationHuman mobility characterization from cellular network data
Becker, R., Cáceres, R., Hanson, K., Isaacman, S., Loh, J. M., Martonosi, M., Rowland, J., Urbanek, S., Varshavsky, A., & Volinsky, C. (2013) · Communications of the ACM · doi:10.1145/2398356.2398375 · Cited by 311
Web-scale Data Integration: You can only afford to Pay As You Go
Madhavan, J., Jeffery, S. R., Cohen, S., Xin, D., Ko, D., Yu, C., & Halevy, A. Y. (2007) · Conference on Innovative Data Systems Research · Cited by 301
Record linkage: similarity measures and algorithms
Koudas, N., Sarawagi, S., & Srivastava, D. (2006) · Proceedings of the 2006 ACM SIGMOD international conference on Management of data · doi:10.1145/1142473.1142599 · Cited by 289
Self-citationA Tale of One City: Using Cellular Network Data for Urban Planning
Becker, R. A., Caceres, R., Hanson, K., Loh, J. M., Urbanek, S., Varshavsky, A., & Volinsky, C. (2011) · IEEE Pervasive Computing · doi:10.1109/mprv.2011.44 · Cited by 275
A Bayesian approach to discovering truth from conflicting sources for data integration
Zhao, B., Rubinstein, B. I. P., Gemmell, J., & Han, J. (2012) · Proceedings of the VLDB Endowment · doi:10.14778/2168651.2168656 · Cited by 264
Bootstrapping pay-as-you-go data integration systems
Das Sarma, A., Dong, X., & Halevy, A. (2008) · Proceedings of the 2008 ACM SIGMOD international conference on Management of data · doi:10.1145/1376616.1376702 · Cited by 254
Self-citationTruth finding on the deep web: is the problem solved?
Li, X., Dong, X. L., Lyons, K., Meng, W., & Srivastava, D. (2012) · Proceedings of the VLDB Endowment · doi:10.14778/2535568.2448943 · Cited by 247
Self-citationFramework for evaluating clustering algorithms in duplicate detection
Hassanzadeh, O., Chiang, F., Lee, H., & Miller, R. J. (2009) · Proceedings of the VLDB Endowment · doi:10.14778/1687627.1687771 · Cited by 238
From data fusion to knowledge fusion
Dong, X. L., Gabrilovich, E., Heitz, G., Horn, W., Murphy, K., Sun, S., & Zhang, W. (2014) · Proceedings of the VLDB Endowment · doi:10.14778/2732951.2732962 · Cited by 232
Self-citationReasoning about record matching rules
Fan, W., Jia, X., Li, J., & Ma, S. (2009) · Proceedings of the VLDB Endowment · doi:10.14778/1687627.1687674 · Cited by 226
Corleone: hands-off crowdsourcing for entity matching
Gokhale, C., Das, S., Doan, A., Naughton, J. F., Rampalli, N., Shavlik, J., & Zhu, X. (2014) · Proceedings of the 2014 ACM SIGMOD International Conference on Management of Data · doi:10.1145/2588555.2588576 · Cited by 219
Pay-as-you-go user feedback for dataspace systems
Jeffery, S. R., Franklin, M. J., & Halevy, A. Y. (2008) · Proceedings of the 2008 ACM SIGMOD international conference on Management of data · doi:10.1145/1376616.1376701 · Cited by 215
Clio: Schema Mapping Creation and Data Exchange
Fagin, R., Haas, L. M., Hernández, M., Miller, R. J., Popa, L., & Velegrakis, Y. (2009) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-02463-4_12 · Cited by 208
Leveraging transitive relations for crowdsourced joins
Wang, J., Li, G., Kraska, T., Franklin, M. J., & Feng, J. (2013) · Proceedings of the 2013 ACM SIGMOD International Conference on Management of Data · doi:10.1145/2463676.2465280 · Cited by 205
Data fusion: resolving data conflicts for integration
Dong, X. L., & Naumann, F. (2009) · Proceedings of the VLDB Endowment · doi:10.14778/1687553.1687620 · Cited by 202
Self-citationMining database structure; or, how to build a data quality browser
Dasu, T., Johnson, T. J., Muthukrishnan, S., & Shkapenyuk, V. (2002) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/564691.564719 · Cited by 191
Truth discovery with multiple conflicting information providers on the web
Yin, X., Han, J., & Yu, P. S. (2007) · Proceedings of the 13th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1281192.1281309 · Cited by 185
Duplicate record elimination in large data files
Bitton, D., & DeWitt, D. J. (1983) · ACM Transactions on Database Systems · doi:10.1145/319983.319987 · Cited by 179
Semi-supervised truth discovery
Yin, X., & Tan, W. (2011) · Proceedings of the 20th international conference on World wide web · doi:10.1145/1963405.1963439 · Cited by 172
Pasternack, J., & Roth, D. (2013) · Proceedings of the 22nd international conference on World Wide Web · doi:10.1145/2488388.2488476 · Cited by 162
Crowdsourcing algorithms for entity resolution
Vesdapunt, N., Bellare, K., & Dalvi, N. (2014) · Proceedings of the VLDB Endowment · doi:10.14778/2732977.2732982 · Cited by 162
Naumann, F. (2014) · ACM SIGMOD Record · doi:10.1145/2590989.2590995 · Cited by 161
Load Balancing for MapReduce-based Entity Resolution
Kolb, L., Thor, A., & Rahm, E. (2012) · 2012 IEEE 28th International Conference on Data Engineering · doi:10.1109/icde.2012.22 · Cited by 158
Xin, D., Saha, B., & Srivastava, D. (2012) · Proceedings of the VLDB Endowment · doi:10.14778/2535568.2448938 · Cited by 155
Self-citationTruth discovery and copying detection in a dynamic world
Dong, X. L., Berti-Equille, L., & Srivastava, D. (2009) · Proceedings of the VLDB Endowment · doi:10.14778/1687627.1687691 · Cited by 146
Self-citationQuestion selection for crowd entity resolution
Whang, S. E., Lofgren, P., & García-Molina, H. (2013) · Proceedings of the VLDB Endowment · doi:10.14778/2536336.2536337 · Cited by 146
Kolb, L., Thor, A., & Rahm, E. (2012) · Proceedings of the VLDB Endowment · doi:10.14778/2367502.2367527 · Cited by 142
From information to knowledge: harvesting entities and relationships from web sources
Weikum, G., & Theobald, M. (2010) · ACM SIGACT-SIGMOD-SIGART Symposium on Principles of Database Systems (PODS) · doi:10.1145/1807085.1807097 · Cited by 142
Pochampally, R., Sarma, A. D., Dong, X. L., Meliou, A., & Srivastava, D. (2014) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/2588555.2593674 · Cited by 131
Self-citationGlobal detection of complex copying relationships between sources
Xin, D., Berti‐Équille, L., Hu, Y., & Srivastava, D. (2010) · Proceedings of the VLDB Endowment · doi:10.14778/1920841.1921008 · Cited by 131
Self-citationAnswering table queries on the web using column keywords
Pimplikar, R., & Sarawagi, S. (2012) · Proceedings of the VLDB Endowment · doi:10.14778/2336664.2336665 · Cited by 129
Harvesting relational tables from lists on the web
Elmeleegy, H., Madhavan, J., & Halevy, A. Y. (2009) · Proceedings of the VLDB Endowment · doi:10.14778/1687627.1687749 · Cited by 128
Answering table augmentation queries from unstructured lists on the web
Gupta, R., & Sarawagi, S. (2009) · Proceedings of the VLDB Endowment · doi:10.14778/1687627.1687661 · Cited by 108
Cafarella, M., Halevy, A. Y., & Madhavan, J. (2011) · Communications of the ACM · doi:10.1145/1897816.1897839 · Cited by 105
Mining collective intelligence in diverse groups
Qi, G.-J., Aggarwal, C. C., Han, J., & Huang, T. S. (2013) · International World Wide Web Conference (WWW) · doi:10.1145/2488388.2488479 · Cited by 101
Li, P., Dong, X. L., Maurino, A., & Srivastava, D. (2011) · Proceedings of the VLDB Endowment · doi:10.14778/3402707.3402733 · Cited by 92
Self-citationLarge-scale linked data integration using probabilistic reasoning and crowdsourcing
Demartini, G., Difallah, D. E., & Cudré-Mauroux, P. (2013) · The VLDB Journal · doi:10.1007/s00778-013-0324-z · Cited by 85
Matching unstructured product offers to structured product specifications
Kannan, A., Givoni, I. E., Agrawal, R., & Fuxman, A. D. (2011) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/2020408.2020474 · Cited by 73
Liu, X., Dong, X. L., Ooi, B. C., & Srivastava, D. (2011) · Proceedings of the VLDB Endowment · doi:10.14778/3402707.3402731 · Cited by 62
Self-citationCharacterizing and selecting fresh data sources
Rekatsinas, T., Dong, X. L., & Srivastava, D. (2014) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/2588555.2610504 · Cited by 60
Self-citationProbabilistic Models to Reconcile Complex Data from Inaccurate Data Sources
Blanco, L., Crescenzi, V., Merialdo, P., & Papotti, P. (2010) · Notes on numerical fluid mechanics and multidisciplinary design · doi:10.1007/978-3-642-13094-6_8 · Cited by 55
Automatically incorporating new sources in keyword search-based data integration
Talukdar, P. P., Ives, Z. G., & Pereira, F. M. B. (2010) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/1807167.1807211 · Cited by 55
Data Modeling in Dataspace Support Platforms
Sarma, A. D., Dong, X. L., & Halevy, A. Y. (2009) · Lecture notes in computer science · doi:10.1007/978-3-642-02463-4_8 · Cited by 36
Self-citationA first tutorial on dataspaces
Franklin, M. J. M., Halevy, A. Y., & Maier, D. (2008) · Proceedings of the VLDB Endowment · doi:10.14778/1454159.1454217 · Cited by 33
Web table taxonomy and formalization
Lautert, L. R., Scheidt, M. M., & Dorneles, C. F. (2013) · ACM SIGMOD Record · doi:10.1145/2536669.2536674 · Cited by 32
Quarrying dataspaces: Schemaless profiling of unfamiliar information sources
Howe, B., Maier, D., Rayner, N., & Rucker, J. (2008) · IEEE International Conference on Data Engineering Workshops (ICDEW) · doi:10.1109/icdew.2008.4498331 · Cited by 23
Li, X., Dong, X. L., Lyons, K. B., et al. (2015) · Proceedings - International Conference on Data Engineering · doi:10.1109/icde.2015.7113275 · Cited by 21
Self-citationHadjieleftheriou, M., & Srivastava, D. (2011) · Foundations and Trends in Databases · doi:10.1561/1900000010 · Cited by 18
Self-citationDong, X. L., & Srivastava, D. (2011) · Proceedings - ACM-SIGMOD International Conference on Management of Data · doi:10.1145/1989323.1989454 · Cited by 8
Self-citation
Cited by
Arrieta, A. B., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., García, S., Gil-López, S., Molina, D., Benjamins, R., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.10045 · Cited by 11,321
Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., et al. (2020) · Information Fusion · doi:10.1016/j.inffus.2019.12.012 · Cited by 10,423
Big data analytics capabilities: A systematic literature review and research agenda
Mikalef, P., Pappas, I. O., Krogstie, J., & Giannakos, M. (2018) · Information Systems and e-Business Management · doi:10.1007/s10257-017-0362-y · Cited by 888
Big Data and cloud computing: innovation opportunities and challenges
Yang, C., Huang, Q., Li, Z., Liu, K., & Hu, F. (2017) · International Journal of Digital Earth · doi:10.1080/17538947.2016.1239771 · Cited by 838
Snorkel: rapid training data creation with weak supervision
Ratner, A., Bach, S. H., Ehrenberg, H., Fries, J., Wu, S., & Ré, C. (2017) · Proceedings of the VLDB Endowment · doi:10.14778/3157794.3157797 · Cited by 777
Metadata
- Authors
- Dong, X. L., & Srivastava, D.
- Venue
- Synthesis Lectures on Data Management
- Publisher
- Springer International Publishing
- Published
- 2015
- Volume
- 6
- Issue
- 1
- Pages
- 1-1189
- ISSN
- 2153-5418 (ISSN-L)
- 2153-5426
- OpenAlex
- W2101654853
- Cited by
- 539
- References
- 94
- FWCI
- 10.21
- Type
- book
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.