The DOI matches its Crossref record; 72 of the 72 references it cites were found in a registry or an index.
Web Crawling
Abstract
This is a survey of the science and practice of web crawling. While at first glance web crawling may appear to be merely an application of breadth-first-search, the truth is that there are many challenges ranging from systems concerns such as managing very large data structures to theoretical questions such as how often to revisit evolving content sources. This survey outlines the fundamental challenges and describes the state-of-the-art models and solutions. It also highlights avenues for future work.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The anatomy of a large-scale hypertextual Web search engine
Brin, S., & Page, L. (1998) · Computer Networks and ISDN Systems · doi:10.1016/s0169-7552(98)00110-x · Cited by 16,004
(1999) · Cited by 12,854
Authoritative sources in a hyperlinked environment
Kleinberg, J. M. (1999) · Journal of the ACM · doi:10.1145/324133.324140 · Cited by 8,959
Space/time trade-offs in hash coding with allowable errors
Bloom, B. H. (1970) · Communications of the ACM · doi:10.1145/362686.362692 · Cited by 7,492
Cumulated gain-based evaluation of IR techniques
Järvelin, K., & Kekäläinen, J. (2002) · ACM Transactions on Information Systems · doi:10.1145/582415.582418 · Cited by 4,841
Broder, A., Kumar, R., Maghoul, F., Raghavan, P., Rajagopalan, S., Stata, R., Tomkins, A., & Wiener, J. (2000) · Computer Networks · doi:10.1016/s1389-1286(00)00083-9 · Cited by 2,810
Focused crawling: a new approach to topic-specific Web resource discovery
Chakrabarti, S., van den Berg, M., & Dom, B. (1999) · Computer Networks · doi:10.1016/s1389-1286(99)00052-3 · Cited by 1,486
Syntactic clustering of the Web
Broder, A. Z., Glassman, S. C., Manasse, M. S., & Zweig, G. (1997) · Computer Networks and ISDN Systems · doi:10.1016/s0169-7552(97)00031-7 · Cited by 1,343
Efficient crawling through URL ordering
Cho, J., Garcia-Molina, H., & Page, L. (1998) · Computer Networks and ISDN Systems · doi:10.1016/s0169-7552(98)00108-1 · Cited by 837
Automatic resource compilation by analyzing hyperlink structure and associated text
Chakrabarti, S., Dom, B., Raghavan, P., Rajagopalan, S., Gibson, D., & Kleinberg, J. (1998) · Computer Networks and ISDN Systems · doi:10.1016/s0169-7552(98)00087-7 · Cited by 697
Detecting spam web pages through content analysis
Ntoulas, A., Najork, M., Manasse, M., & Fetterly, D. (2006) · Proceedings of the 15th international conference on World Wide Web · doi:10.1145/1135777.1135794 · Cited by 608
Self-citationSearching Distributed Collections With Inference Networks
Callan, J. P., Lu, Z., & Croft, W. B. (2017) · ACM SIGIR Forum · doi:10.1145/3130348.3130363 · Cited by 608
Mercator: A scalable, extensible Web crawler
Heydon, A., & Najork, M. (1999) · World Wide Web · doi:10.1023/a:1019213109274 · Cited by 582
Self-citationUbiCrawler: a scalable fully distributed Web crawler
Boldi, P., Codenotti, B., Santini, M., & Vigna, S. (2004) · Software: Practice and Experience · doi:10.1002/spe.587 · Cited by 560
Focused Crawling Using Context Graphs
Diligenti, M., Coetzee, F., Lawrence, S., Giles, C. L., & Gori, M. (2000) · Cited by 533
The Evolution of the Web and Implications for an Incremental Crawler
Cho, J. J., & García-Molina, H. (2000) · Cited by 499
Finding related pages in the World Wide Web
Dean, J., & Henzinger, M. R. (1999) · Computer Networks · doi:10.1016/s1389-1286(99)00022-5 · Cited by 489
What's new on the web?: the evolution of the web from a search engine perspective
Ntoulas, A., Cho, J., & Olston, C. (2004) · Proceedings of the 13th international conference on World Wide Web · doi:10.1145/988672.988674 · Cited by 452
Self-citationDistributed Information Retrieval
Callan, J. (2005) · Kluwer Academic Publishers eBooks · doi:10.1007/0-306-47019-5_5 · Cited by 389
Query-based sampling of text databases
Callan, J., & Connell, M. (2001) · ACM Transactions on Information Systems · doi:10.1145/382979.383040 · Cited by 368
Design and implementation of a high-performance distributed Web crawler
Shkapenyuk, V., & Suel, T. (2003) · Proceedings 18th International Conference on Data Engineering · doi:10.1109/icde.2002.994750 · Cited by 328
The shark-search algorithm. An application: tailored Web site mapping
Hersovici, M., Jacovi, M., Maarek, Y., Pelleg, D., Shtalhaim, M., & Ur, S. (1998) · Computer Networks and ISDN Systems · doi:10.1016/s0169-7552(98)00038-5 · Cited by 319
Cho, J., & Garcia-Molina, H. (2002) · Proceedings of the 11th international conference on World Wide Web · doi:10.1145/511446.511464 · Cited by 312
Challenges in web search engines
Henzinger, M. R., Motwani, R., & Silverstein, C. (2002) · ACM SIGIR Forum · doi:10.1145/792550.792553 · Cited by 307
Spam, damn spam, and statistics: using statistical analysis to locate spam web pages
Fetterly, D., Manasse, M., & Najork, M. (2004) · Proceedings of the 7th International Workshop on the Web and Databases: colocated with ACM SIGMOD/PODS 2004 · doi:10.1145/1017074.1017077 · Cited by 291
Self-citationHenzinger, M. R., Heydon, A., Mitzenmacher, M., & Najork, M. (2000) · Computer Networks · doi:10.1016/s1389-1286(00)00055-4 · Cited by 269
Self-citationTopical web crawlers: Evaluating adaptive algorithms
Menczer, F., Pant, G., & Srinivasan, P. (2004) · ACM Transactions on Internet Technology · doi:10.1145/1031114.1031117 · Cited by 268
Finding What People Want : Experiences with the WebCrawler
Pinkerton, B. (1994) · Medical Entomology and Zoology · Cited by 266
Breadth-first crawling yields high-quality pages
Najork, M., & Wiener, J. L. (2001) · Proceedings of the 10th international conference on World Wide Web · doi:10.1145/371920.371965 · Cited by 265
Self-citationEffective page refresh policies for Web crawlers
Cho, J., & Garcia-Molina, H. (2003) · ACM Transactions on Database Systems · doi:10.1145/958942.958945 · Cited by 258
GENVL and WWWW: Tools for taming the Web
McBryan, O. A. (1994) · Computer Networks and ISDN Systems · doi:10.1016/s0169-7552(94)90149-x · Cited by 242
Identifying link farm spam pages
Wu, B., & Davison, B. D. (2005) · Special interest tracks and posters of the 14th international conference on World Wide Web - WWW '05 · doi:10.1145/1062745.1062762 · Cited by 241
An adaptive model for optimizing performance of an incremental web crawler
Edwards, J., McCurley, K., & Tomlin, J. (2001) · Proceedings of the 10th international conference on World Wide Web · doi:10.1145/371920.371960 · Cited by 220
Adaptive on-line page importance computation
Abiteboul, S., Preda, M., & Cobena, G. (2003) · Proceedings of the twelfth international conference on World Wide Web - WWW '03 · doi:10.1145/775152.775192 · Cited by 213
Downloading textual hidden web content through keyword queries
Ntoulas, A., Zerfos, P., & Cho, J. (2005) · Proceedings of the 5th ACM/IEEE-CS joint conference on Digital libraries · doi:10.1145/1065385.1065407 · Cited by 206
Estimating frequency of change
Cho, J., & Garcia-Molina, H. (2003) · ACM Transactions on Internet Technology · doi:10.1145/857166.857170 · Cited by 196
A large-scale study of the evolution of web pages
Fetterly, D., Manasse, M., Najork, M., & Wiener, J. (2003) · Proceedings of the twelfth international conference on World Wide Web - WWW '03 · doi:10.1145/775152.775246 · Cited by 189
Self-citationThe effectiveness of GIOSS for the text database discovery problem
Gravano, L., García-Molina, H., & Tomasic, A. (1994) · Proceedings of the 1994 ACM SIGMOD international conference on Management of data · doi:10.1145/191839.191869 · Cited by 175
SpamRank -- Fully Automatic Link Spam Detection.
Benczúr, A. A., Csalogány, K., Sarlós, T., & Uher, M. (2005) · Cited by 170
The web changes everything: understanding the dynamics of web content
Adar, E., Teevan, J. B., Dumais, S., & Elsas, J. L. (2009) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/1498759.1498837 · Cited by 160
Adaptive Retrieval Agents: Internalizing Local Context and Scaling up to the Web
Menczer, F., & Belew, R. K. (2000) · Machine Learning · doi:10.1023/a:1007653114902 · Cited by 156
Recognizing Nepotistic Links on the Web
Davison, B. D. (2000) · Cited by 152
Learning to crawl: Comparing classification schemes
Pant, G., & Srinivasan, P. (2005) · ACM Transactions on Information Systems · doi:10.1145/1095872.1095875 · Cited by 138
Measuring index quality using random walks on the Web
Henzinger, M. R., Heydon, A., Mitzenmacher, M., & Najork, M. (1999) · Computer Networks · doi:10.1016/s1389-1286(99)00016-x · Cited by 134
Self-citationQuery Selection Techniques for Efficient Crawling of Structured Web Sources
Wu, P. K. T., Wen, J.-R., Liu, H., & Ma, W. (2006) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde.2006.124 · Cited by 128
Link-Based Characterization and Detection of Web Spam
Becchetti, L., Castillo, C., Donato, D., Leonardi, S., & Baeza-Yates, R. A. (2006) · Cited by 127
Baeza-Yates, R. A., Castillo, C., Marı́n, M., & Rodriguez, A. (2005) · International World Wide Web Conference (WWW) · doi:10.1145/1062745.1062768 · Cited by 124
Random sampling from a search engine's index
Bar-Yossef, Z., & Gurevich, M. (2006) · International World Wide Web Conference (WWW) · doi:10.1145/1135777.1135833 · Cited by 119
Optimal robot scheduling for Web search engines
Coffman, E. G., Liu, Z., & Weber, R. (1998) · Journal of Scheduling · Cited by 117
Recrawl scheduling based on information longevity
Olston, C., & Pandey, S. (2008) · International World Wide Web Conference (WWW) · doi:10.1145/1367497.1367557 · Cited by 116
Self-citationMirror, mirror on the Web: a study of host pairs with replicated content
Bharat, K. A., & Broder, A. (1999) · Computer Networks · doi:10.1016/s1389-1286(99)00021-3 · Cited by 114
Pandey, S., & Olston, C. (2005) · International World Wide Web Conference (WWW) · doi:10.1145/1060745.1060805 · Cited by 101
Self-citationNajork, M., & Heydon, A. (2002) · Massive computing · doi:10.1007/978-1-4615-0005-6_2 · Cited by 100
Self-citationLycos: design choices in an Internet search service
Mauldin, M. I. (1997) · IEEE Expert · doi:10.1109/64.577466 · Cited by 99
How to build a WebFountain: An architecture for very large-scale text analytics
Gruhl, D., Chavet, L., Gibson, D., Meyer, J., Pattanayak, P. K., Tomkins, A., & Zien, J. Y. (2004) · IBM Systems Journal · doi:10.1147/sj.431.0064 · Cited by 94
A comparison of techniques to find mirrored hosts on the WWW
Bharat, K. A., Broder, A., Dean, J., & Henzinger, M. R. (2000) · Journal of the American Society for Information Science · Cited by 88
Link Evolution: Analysis and Algorithms
Chien, S., Dwork, C., Kumar, R., Simon, D., & Sivakumar, D. (2004) · Internet Mathematics · doi:10.1080/15427951.2004.10129090 · Cited by 79
The discoverability of the web
Dasgupta, A., Ghosh, A., Kumar, R., Olston, C., Pandey, S., & Tomkins, A. (2007) · International World Wide Web Conference (WWW) · doi:10.1145/1242572.1242630 · Cited by 74
Self-citationEfficient URL caching for world wide web crawling
Broder, A., Najork, M., & Wiener, J. L. (2003) · International World Wide Web Conference (WWW) · doi:10.1145/775152.775247 · Cited by 72
Self-citationA taxonomy of JavaScript redirection spam
Chellapilla, K., & Maykov, A. (2007) · Adversarial Information Retrieval on the Web · doi:10.1145/1244408.1244423 · Cited by 62
Characterizing a national community web
Gomes, D., & Silva, M. J. (2005) · ACM Transactions on Internet Technology · doi:10.1145/1084772.1084775 · Cited by 53
Tracking Web Spam with Hidden Style Similarity
Urvoy, T., Lavergne, T., & Filoche, P. (2006) · Adversarial Information Retrieval on the Web · Cited by 44
RankMass crawler: a crawler with high personalized pagerank coverage guarantee
Cho, J. J., & Schonfeld, U. (2007) · Very Large Data Bases · Cited by 42
Geographically focused collaborative crawling
Gao, W., Lee, H. C., & Miao, Y. (2006) · International World Wide Web Conference (WWW) · doi:10.1145/1135777.1135822 · Cited by 36
Crawl ordering by search impact
Pandey, S., & Olston, C. (2008) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/1341531.1341535 · Cited by 31
Self-citationSchonfeld, U., & Shivakumar, N. (2009) · International World Wide Web Conference (WWW) · doi:10.1145/1526709.1526842 · Cited by 31
Baeza-Yates, R. A., & Castillo, C. (2007) · Cited by 23
Incorporating site-level knowledge for incremental crawling of web forums
Yang, J.-M., Cai, R., Wang, C., Huang, H., Zhang, L., & Ma, W. (2009) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/1557019.1557166 · Cited by 20
Paradoxical Effects in PageRank Incremental Computations
Boldi, P., Santini, M., & Vigna, S. (2005) · Internet Mathematics · doi:10.1080/15427951.2005.10129106 · Cited by 17
A Genre-Aware Approach to Focused Crawling
de Assis, G. T., Laender, A. H. F., Gonçalves, M. A., & da Silva, A. S. (2009) · World Wide Web · doi:10.1007/s11280-009-0063-7 · Cited by 16
DSphere: A Source-Centric Approach to Crawling, Indexing and Searching the World Wide Web
Bamba, B., Liu, L., Caverlee, J., Padliya, V. J., Srivatsa, M., Bansal, T., Palekar, M., Patrao, J., Li, S., & Singh, A. (2007) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde.2007.369060 · Cited by 9
(2005)
Cited by
Community Detection and Mining in Social Media
Tang, L., & Liu, H. (2010) · Synthesis lectures on data mining and knowledge discovery · doi:10.2200/s00298ed1v01y201009dmk003 · Cited by 352
Unpacking the Social Media Bot: A Typology to Guide Research and Policy
Gorwa, R., & Guilbeault, D. (2020) · Policy & Internet · doi:10.1002/poi3.184 · Cited by 270
Mining query logs: turning search usage data into knowledge
Silvestri, F. (2009) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000013 · Cited by 194
Trafilatura: A Web Scraping Library and Command-Line Tool for Text Discovery and Extraction
Barbaresi, A. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations · doi:10.18653/v1/2021.acl-demo.15 · Cited by 90
Metadata
- Authors
- Olston, C., & Najork, M.
- Venue
- Foundations and Trends® in Information Retrieval
- Publisher
- Emerald
- Published
- 2010
- Volume
- 4
- Issue
- 3
- Pages
- 175-246
- ISSN
- 1554-0669 (ISSN-L)
- 1554-0677
- OpenAlex
- W2911388033
- Cited by
- 370
- References
- 72
- FWCI
- 63.25
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.