The DOI matches its Crossref record; 81 of the 81 references it cites were found in a registry or an index.
Web data extraction, applications and techniques: A survey
Open accessAbstract
Web Data Extraction is an important problem that has been studied by means of different scientific tools and in a broad range of applications. Many approaches to extracting data from the Web have been designed to solve specific problems and operate in ad-hoc domains. Other approaches, instead, heavily reuse techniques and algorithms developed in the field of Information Extraction.This survey aims at providing a structured and comprehensive overview of the literature in the field of Web Data Extraction. We provided a simple classification framework in which existing Web Data Extraction applications are grouped into two main classes, namely applications at the Enterprise level and at the Social Web level. At the Enterprise level, Web Data Extraction techniques emerge as a key tool to perform data analysis in Business and Competitive Intelligence systems as well as for business process re-engineering. At the Social Web level, Web Data Extraction techniques allow to gather a large amount of structured data continuously generated and disseminated by Web 2.0, Social Media and Online Social Network users and this offers unprecedented opportunities to analyze human behavior at a very large scale. We discuss also the potential of cross-fertilization, i.e., on the possibility of re-using Web Data Extraction techniques originally designed to work in a given domain, in other domains.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Data mining: concepts and techniques
(2012) · Choice Reviews Online · doi:10.5860/choice.49-3305 · Cited by 28,531
Pan, S. J., & Yang, Q. (2010) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2009.191 · Cited by 23,387 · cited 1×
“Various strategies have been proposed to perform such a transfer of information and some of these strategies take advantages from models and algorithms developed in the field of transfer of learning [96].”
MethodThe Structure and Function of Complex Networks
Newman, M. E. J. (2003) · SIAM Review · doi:10.1137/s003614450342480 · Cited by 18,719
Foundations of statistical natural language processing
Manning, C. D., & Schütze, H. (1999) · Cited by 9,816
Machine learning in automated text categorization
Sebastiani, F. (2002) · ACM Computing Surveys · doi:10.1145/505282.505283 · Cited by 7,926
What is Twitter, a social network or a news media?
Kwak, H., Lee, C., Park, H., & Moon, S. (2010) · Proceedings of the 19th international conference on World wide web · doi:10.1145/1772690.1772751 · Cited by 6,652
The small-world problem: (400002009-005)
Milgram, S. (1967) · PsycEXTRA Dataset · doi:10.1037/e400002009-005 · Cited by 4,790
Data Mining: Concepts and Techniques
Thamaraiselvi, G., & Kaliammal, A. (2014) · Journal of Information and Knowledge · Cited by 4,088
A survey of approaches to automatic schema matching
Rahm, E., & Bernstein, P. A. (2001) · The VLDB Journal · doi:10.1007/s007780100057 · Cited by 3,352
A maximum entropy approach to natural language processing
Berger, A. L., Pietra, V. J. D., & Pietra, S. A. D. (1996) · doi:10.5555/234285.234289 · Cited by 3,065
Measurement and analysis of online social networks
Mislove, A., Marcon, M., Gummadi, K. P., Druschel, P., & Bhattacharjee, B. (2007) · Proceedings of the 7th ACM SIGCOMM conference on Internet measurement · doi:10.1145/1298306.1298311 · Cited by 3,057
Understanding natural language
Winograd, T. (1972) · Cognitive Psychology · doi:10.1016/0010-0285(72)90002-3 · Cited by 2,799
Mining the peanut gallery: opinion extraction and semantic classification of product reviews
Dave, K., Lawrence, S., & Pennock, D. M. (2003) · Proceedings of the twelfth international conference on World Wide Web - WWW '03 · doi:10.1145/775152.775226 · Cited by 1,916
The small-world phenomenon: an algorithmic perspective
Kleinberg, J. (2000) · Proceedings of the thirty-second annual ACM symposium on Theory of computing · doi:10.1145/335305.335325 · Cited by 1,882
(2000) · Cited by 1,467
RoadRunner: Towards Automatic Data Extraction from Large Web Sites
Crescenzi, V., Mecca, G., & Merialdo, P. (2001) · Iris (Roma Tre University) · Cited by 1,043
Ajax: A New Approach to Web Applications
Garrett, J. J. (2007) · Cited by 1,029
Learning Information Extraction Rules for Semi-Structured and Free Text
Soderland, S. (1999) · Machine Learning · doi:10.1023/a:1007562322031 · Cited by 934
User interactions in social networks and their implications
Wilson, C., Boe, B., Sala, A., Puttaswamy, K. P. N., & Zhao, B. Y. (2009) · Proceedings of the 4th ACM European conference on Computer systems · doi:10.1145/1519065.1519089 · Cited by 879
The State of Record Linkage and Current Research Problems
Winkler, W. E. (1999) · Cited by 810
A Survey of Web Information Extraction Systems
Chang, C.-H., Kayed, M., Girgis, M. R., & Shaalan, K. F. (2006) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2006.152 · Cited by 793
Crandall, D. J., Backstrom, L., Huttenlocher, D., & Kleinberg, J. (2009) · Proceedings of the 18th international conference on World wide web · doi:10.1145/1526709.1526812 · Cited by 769
Sarawagi, S. (2008) · Foundations and Trends in Databases · doi:10.1561/1900000003 · Cited by 707
Walking in Facebook: A Case Study of Unbiased Sampling of OSNs
Gjoka, M., Kurant, M., Butts, C. T., & Markopoulou, A. (2010) · 2010 Proceedings IEEE INFOCOM · doi:10.1109/infcom.2010.5462078 · Cited by 671
A new statistical parser based on bigram lexical dependencies
Collins, M. J. (1996) · Proceedings of the 34th annual meeting on Association for Computational Linguistics - · doi:10.3115/981863.981888 · Cited by 614
Backstrom, L., Boldi, P., Rosa, M., Ugander, J., & Vigna, S. (2012) · Proceedings of the 4th Annual ACM Web Science Conference · doi:10.1145/2380718.2380723 · Cited by 567
Wrapper induction: Efficiency and expressiveness
Kushmerick, N. (2000) · Artificial Intelligence · doi:10.1016/s0004-3702(99)00100-9 · Cited by 553
Relational Learning of Pattern-Match Rules for Information Extraction.
Califf, M. E., & Mooney, R. J. (1997) · Cited by 507
Web data extraction based on partial tree alignment
Zhai, Y., & Liu, B. (2005) · Proceedings of the 14th international conference on World Wide Web - WWW '05 · doi:10.1145/1060745.1060761 · Cited by 496
Boilerplate detection using shallow text features
Kohlschütter, C., Fankhauser, P., & Nejdl, W. (2010) · Proceedings of the third ACM international conference on Web search and data mining · doi:10.1145/1718487.1718542 · Cited by 476
The tree-to-tree editing problem
Selkow, S. M. (1977) · Information Processing Letters · doi:10.1016/0020-0190(77)90064-3 · Cited by 474
Tweets from Justin Bieber's heart: the dynamics of the location field in user profiles
Hecht, B., Hong, L., Suh, B., & Chi, E. H. (2011) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1978942.1978976 · Cited by 465
The power of a good idea: Quantitative modeling of the spread of ideas from epidemiological models
Bettencourt, L. M. A., Cintrón-Arias, A., Kaiser, D. I., & Castillo-Chávez, C. (2006) · Physica A: Statistical Mechanics and its Applications · doi:10.1016/j.physa.2005.08.083 · Cited by 463
XWRAP: an XML-enabled wrapper construction system for Web information sources
Liu, L., Pu, C., & Han, W. (2002) · Proceedings of 16th International Conference on Data Engineering (Cat. No.00CB37073) · doi:10.1109/icde.2000.839475 · Cited by 422
Generating finite-state transducers for semi-structured data extraction from the Web
Hsu, C.-N., & Dung, M.-T. (1998) · Information Systems · doi:10.1016/s0306-4379(98)00027-1 · Cited by 411
A hierarchical approach to wrapper induction
Muslea, I., Minton, S., & Knoblock, C. (1999) · Proceedings of the third annual conference on Autonomous Agents · doi:10.1145/301136.301191 · Cited by 371
Mining the peanut gallery: opinion extraction and semantic classification of product reviews
Dave, K., Lawrence, S., & Pennock, D. M. (2003) · Proceedings of the twelfth international conference on World Wide Web - WWW '03 · doi:10.1145/775224.775226 · Cited by 369
Can movies and books collaborate?: cross-domain collaborative filtering for sparsity reduction
Li, B., Yang, Q., & Xue, X. (2009) · Cited by 361
On the editing distance between unordered labeled trees
Zhang, K., Statman, R., & Shasha, D. E. (1992) · Information Processing Letters · doi:10.1016/0020-0190(92)90136-j · Cited by 359
Machine Learning for Information Extraction in Informal Domains
Freitag, D. (2000) · Machine Learning · doi:10.1023/a:1007601113994 · Cited by 350
The convergence of social and technological networks
Kleinberg, J. (2008) · Communications of the ACM · doi:10.1145/1400214.1400232 · Cited by 346
Identifying syntactic differences between two programs
Yang, W. (1991) · Software: Practice and Experience · doi:10.1002/spe.4380210706 · Cited by 346
Automatic web news extraction using tree edit distance
Reis, D. C., Golgher, P. B., Silva, A. S., & Laender, A. F. (2004) · Proceedings of the 13th international conference on World Wide Web · doi:10.1145/988672.988740 · Cited by 342
Transfer learning for collaborative filtering via a rating-matrix generative model
Li, B., Yang, Q., & Xue, X. (2009) · Proceedings of the 26th Annual International Conference on Machine Learning · doi:10.1145/1553374.1553454 · Cited by 340
Users’ interaction with World Wide Web resources: an exploratory study using a holistic approach
Wang, P., Hawk, W. B., & Tenopir, C. (2000) · Information Processing & Management · doi:10.1016/s0306-4573(99)00059-x · Cited by 307
Exploiting innocuous activity for correlating users across sites
Goga, O., Lei, H., Parthasarathi, S. H. K., Friedland, G., Sommer, R., & Teixeira, R. (2013) · Proceedings of the 22nd international conference on World Wide Web · doi:10.1145/2488388.2488428 · Cited by 243
"I'm eating a sandwich in Glasgow": modeling locations with tweets
Kinsella, S., Murdock, V., & O'Hare, N. (2011) · Proceedings of the 3rd international workshop on Search and mining user-generated contents · doi:10.1145/2065023.2065039 · Cited by 233
Connecting Corresponding Identities across Communities
Zafarani, R., & Liu, H. (2009) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v3i1.13993 · Cited by 224
Crawling Facebook for social network analysis purposes
Catanese, S. A., De Meo, P., Ferrara, E., Fiumara, G., & Provetti, A. (2011) · Proceedings of the International Conference on Web Intelligence, Mining and Semantics · doi:10.1145/1988688.1988749 · Cited by 217
Self-citationUser identification across multiple social networks
Vosecky, J., Hong, D., & Shen, V. Y. (2009) · 2009 First International Conference on Networked Digital Technologies · doi:10.1109/ndt.2009.5272173 · Cited by 217
Cross-system user modeling and personalization on the Social Web
Abel, F., Herder, E., Houben, G.-J., Henze, N., & Krause, D. (2013) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-012-9131-2 · Cited by 204
Identifying Users Across Social Tagging Systems
Iofciu, T., Fankhauser, P., Abel, F., & Bischoff, K. (2021) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v5i1.14153 · Cited by 201
How Unique and Traceable Are Usernames?
Perito, D., Castelluccia, C., Kaafar, M. A., & Manils, P. (2011) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-22263-4_1 · Cited by 194
Adaptive information extraction
Turmo, J., Ageno, A., & Català, N. (2006) · ACM Computing Surveys · doi:10.1145/1132956.1132957 · Cited by 191
The Geospatial Characteristics of a Social Movement Communication Network
Conover, M., Davis, C. A., Ferrara, E., McKelvey, K., Menczer, F., & Flammini, A. (2013) · PLoS ONE · doi:10.1371/journal.pone.0055957 · Cited by 173
Self-citationCI Spider: a tool for competitive intelligence on the Web
Chen, H., Chau, M., & Zeng, D. (2002) · Decision Support Systems · doi:10.1016/s0167-9236(02)00002-7 · Cited by 170
Romero, D. M., & Kleinberg, J. M. (2010) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v4i1.14015 · Cited by 163
On the bias of BFS (Breadth First Search)
Kurant, M., Markopoulou, A. P., & Thiran, P. (2010) · International Teletraffic Congress · doi:10.1109/itc.2010.5608727 · Cited by 151
Abusing Social Networks for Automated User Profiling
Balduzzi, M. `., Platzer, C., Holz, T., Kirda, E., Balzarotti, D., & Kruegel, C. (2010) · Lecture notes in computer science · doi:10.1007/978-3-642-15512-3_22 · Cited by 141
Folks in Folksonomies: Social Link Prediction from Shared MetadataPreprint
Schifanella, R., Barrat, A., Cattuto, C., Markines, B., & Menczer, F. (2010) · arXiv (Cornell University) · doi:10.48550/arxiv.1003.2281 · Cited by 138
A li B aba : PubMed as a graph
Plake, C., Schiemann, T., Pankalla, M., Hakenberg, J., & Leser, U. (2006) · Bioinformatics · doi:10.1093/bioinformatics/btl408 · Cited by 137
You Are What You Like! Information Leakage Through Users' Interests
Chaabane, A., Ács, G., & Kâafar, M. A. (2012) · Cited by 131
Ye, S., Lang, J., & Wu, F. (2010) · Asia-Pacific Web Conference · doi:10.1109/apweb.2010.10 · Cited by 124
Jedi: extracting and synthesizing information from the Web
Huck, G., Frankhauser, P., Aberer, K., & Neuhold, E. J. (1998) · Cooperative Information Systems · doi:10.1109/coopis.1998.706182 · Cited by 124
A large-scale community structure analysis in Facebook
Ferrara, E. (2012) · EPJ Data Science · doi:10.1140/epjds9 · Cited by 117
Self-citationMulti-Domain Collaborative FilteringPreprint
Zhang, Y., Cao, B., & Yeung, D. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1203.3535 · Cited by 105
Matching Algorithms within a Duplicate Detection System.
Monge, A. E. (2000) · IEEE Data(base) Engineering Bulletin · Cited by 101
New Algorithm for Ordered Tree-to-Tree Correction Problem
Chen, W. (2001) · Journal of Algorithms · doi:10.1006/jagm.2001.1170 · Cited by 93
Correlating user profiles from multiple folksonomies
Szomszor, M., Cantador, I., & Alani, H. (2008) · doi:10.1145/1379092.1379103 · Cited by 74
Web Archiving: Issues and Methods
Masanés, J. (2006) · Springer eBooks · doi:10.1007/978-3-540-46332-0_1 · Cited by 63
Clustering memes in social media
Ferrara, E., JafariAsbagh, M., Varol, O., Qazvinian, V., & Flammini, R. (2013) · Cited by 60
Self-citationTraveling Trends: Social Butterflies or Frequent Fliers?Preprint
Ferrara, E., Varol, O., Menczer, F., & Flammini, A. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1310.2671 · Cited by 53
Self-citationSubject-Wise Policy Based Access Control Mechanism for Protection of Personal Information
Mun, H.-J., Um, N.-K., Sun, N., Li, Y., & Lee, S.-H. (2007) · International Conference on Computer Sciences and Convergence Information Technology · doi:10.1109/iccit.2007.398 · Cited by 45
Efficient Information Extraction over Evolving Text Data
Chen, F., Doan, A., Yang, J., & Ramakrishnan, R. (2008) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde.2008.4497503 · Cited by 43
SXPath - Extending XPath towards Spatial Querying on Web Documents
Oro, E., Ruffolo, M., & Staab, S. (2010) · Proceedings of the VLDB Endowment · doi:10.14778/1921071.1921079 · Cited by 43
Analyzing the Facebook Friendship GraphPreprint
Catanese, S., Meo, P. D., Ferrara, E., & Fiumara, G. (2010) · arXiv (Cornell University) · doi:10.48550/arxiv.1011.5168 · Cited by 34
Self-citationPrivacy-Enhancing Technologies
Wang, Y. (2011) · IGI Global eBooks · doi:10.4018/9781605661322.ch013 · Cited by 28
A Unified Framework for Wrapping, Mediating and Restructuring Information from the Web
May, W., Himmeröder, R., Lausen, G., & Ludäscher, B. (1999) · Lecture notes in computer science · doi:10.1007/3-540-48054-4_25 · Cited by 24
Automated data extraction from the web with conditional models
Phan, X.-H., Horiguchi, S., & Ho, T. B. (2005) · International Journal of Business Intelligence and Data Mining · doi:10.1504/ijbidm.2005.008362 · Cited by 11
Semantic Online Tourism Market Monitoring
Walchhofer, N., Hronský, M., Pöttler, M., Baumgartner, R., & Fröschl, K. A. (2010) · Springer eBooks · doi:10.1007/978-3-211-99407-8_52 · Cited by 8
Self-citationSelf-supervised Automated Wrapper Generation for Weblog Data Extraction
Gkotsis, G., Stepanyan, K., Cristea, A. I., et al. (2013) · Lecture notes in computer science · doi:10.1007/978-3-642-39467-6_26 · Cited by 7
Cited by
Hogan, A., Blomqvist, E., Cochez, M., D’amato, C., Melo, G. D., Gutierrez, C., Kirrane, S., Gayo, J. E. L., Navigli, R., Neumaier, S., et al. (2022) · ACM Computing Surveys · doi:10.1145/3447772 · Cited by 1,904
A Survey on Data Collection for Machine Learning: A Big Data - AI Integration Perspective
Roh, Y., Heo, G., & Whang, S. E. (2021) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2019.2946162 · Cited by 968
Artificial intelligence in E-Commerce: a bibliometric study and literature review
Bawack, R. E., Wamba, S. F., Carillo, K. D. A., & Akter, S. (2022) · Electronic Markets · doi:10.1007/s12525-022-00537-z · Cited by 372
Knowledge management and digital transformation for Industry 4.0: a structured literature review
de Bem Machado, A., Secinaro, S., Calandra, D., & Lanzalonga, F. (2022) · Knowledge Management Research & Practice · doi:10.1080/14778238.2021.2015261 · Cited by 347
Metadata
- Authors
- Ferrara, E., De Meo, P., Fiumara, G., & Baumgartner, R.
- Venue
- Knowledge-Based Systems
- Publisher
- Elsevier BV
- Published
- 2014-11
- Volume
- 70
- Pages
- 301-323
- ISSN
- 0950-7051 (ISSN-L)
- OpenAlex
- W2148317291
- Cited by
- 396
- References
- 81
- FWCI
- 116.49
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.