The DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.
Design and implementation of a high-performance distributed Web crawler
Abstract
Broad Web search engines as well as many more specialized search tools rely on Web crawlers to acquire large collections of pages for indexing and analysis. Such a Web crawler may interact with millions of hosts over a period of weeks or months, and thus issues of robustness, flexibility, and manageability are of major importance. In addition, I/O performance, network resources, and OS limits must be taken into account in order to achieve high performance at a reasonable cost. In this paper, we describe the design and implementation of a distributed Web crawler that runs on a network of workstations. The crawler scales to (at least) several hundred pages per second, is resilient against system crashes and other events, and can be adapted to various crawling applications. We present the software architecture of the system, discuss the, performance bottlenecks, and describe efficient techniques for achieving high performance. We also report preliminary experimental results based on a crawl of 120 million pages on 5 million hosts.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The anatomy of a large-scale hypertextual Web search engine
Brin, S., & Page, L. (1998) · Computer Networks and ISDN Systems · doi:10.1016/s0169-7552(98)00110-x · Cited by 16,004
Baeza-Yates, R. A., & Ribeiro-Neto, B. A. (1999) · Cited by 11,505
Focused crawling: a new approach to topic-specific Web resource discovery
Chakrabarti, S., van den Berg, M., & Dom, B. (1999) · Computer Networks · doi:10.1016/s1389-1286(99)00052-3 · Cited by 1,486
Efficient crawling through URL ordering
Cho, J., Garcia-Molina, H., & Page, L. (1998) · Computer Networks and ISDN Systems · doi:10.1016/s0169-7552(98)00108-1 · Cited by 837
Mercator: A scalable, extensible Web crawler
Heydon, A., & Najork, M. (1999) · World Wide Web · doi:10.1023/a:1019213109274 · Cited by 582
Focused Crawling Using Context Graphs
Diligenti, M., Coetzee, F., Lawrence, S., Giles, C. L., & Gori, M. (2000) · Cited by 533
Arasu, A., Cho, J., Garcia-Molina, H., Paepcke, A., & Raghavan, S. (2001) · ACM Transactions on Internet Technology · doi:10.1145/383034.383035 · Cited by 505
Mining the Web's link structure
Chakrabarti, S., Dom, B. E., Kumar, S. R., Raghavan, P., Rajagopalan, S., Tomkins, A., Gibson, D., & Kleinberg, J. (1999) · Computer · doi:10.1109/2.781636 · Cited by 501
The Evolution of the Web and Implications for an Incremental Crawler
Cho, J. J., & García-Molina, H. (2000) · Cited by 499
Synchronizing a database to improve freshness
Cho, J., & Garcia-Molina, H. (2000) · ACM SIGMOD Record · doi:10.1145/335191.335391 · Cited by 346
Henzinger, M. R., Heydon, A., Mitzenmacher, M., & Najork, M. (2000) · Computer Networks · doi:10.1016/s1389-1286(00)00055-4 · Cited by 269
Breadth-first crawling yields high-quality pages
Najork, M., & Wiener, J. L. (2001) · Proceedings of the 10th international conference on World Wide Web · doi:10.1145/371920.371965 · Cited by 265
Using Reinforcement Learning to Spider the Web Efficiently
Rennie, J. D. M., & McCallum, A. (1999) · Cited by 212
The Connectivity Server: fast access to linkage information on the Web
Bharat, K., Broder, A., Henzinger, M., Kumar, P., & Venkatasubramanian, S. (1998) · Computer Networks and ISDN Systems · doi:10.1016/s0169-7552(98)80047-0 · Cited by 188
WebBase: a repository of Web pages
Hirai, J., Raghavan, S., Garcia-Molina, H., & Paepcke, A. (2000) · Computer Networks · doi:10.1016/s1389-1286(00)00063-3 · Cited by 183
Breadth-First Search Crawling Yields High-Quality Pages
Najork, M., & Wiener, J. L. (2001) · Cited by 182
Synchronizing a database to improve freshness
Cho, J. J., & García-Molina, H. (2000) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/342009.335391 · Cited by 170
Measuring index quality using random walks on the Web
Henzinger, M. R., Heydon, A., Mitzenmacher, M., & Najork, M. (1999) · Computer Networks · doi:10.1016/s1389-1286(99)00016-x · Cited by 134
Approximating Aggregate Queries about Web Pages via Random Walks
Bar-Yossef, Z., Berg, A., Chien, S., Fakcharoenphol, J., & Weitz, D. (2000) · Cited by 125
Compressing the graph structure of the Web
Suel, T., & Yuan, J. (2002) · DCC · doi:10.1109/dcc.2001.917152 · Cited by 125
Self-citationDistributed Hypertext Resource Discovery Through Examples
Chakrabarti, S., van den Berg, M., & Dom, B. (1999) · Cited by 61
Cited by
UbiCrawler: a scalable fully distributed Web crawler
Boldi, P., Codenotti, B., Santini, M., & Vigna, S. (2004) · Software: Practice and Experience · doi:10.1002/spe.587 · Cited by 560
Olston, C., & Najork, M. (2010) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000017 · Cited by 370
The Perron-Frobenius theorem: some of its applications
Pillai, S. U., Suel, T., & Cha, S. (2005) · IEEE Signal Processing Magazine · doi:10.1109/msp.2005.1406483 · Cited by 338
Self-citationWeb application security assessment by fault injection and behavior monitoring
Huang, Y.-W., Huang, S.-K., Lin, T.-P., & Tsai, C.-H. (2003) · Proceedings of the twelfth international conference on World Wide Web - WWW '03 · doi:10.1145/775152.775174 · Cited by 319
Towards Scaling Fully Personalized PageRank: Algorithms, Lower Bounds, and Experiments
Fogaras, D., Rácz, B., Csalogány, K., & Sarlós, T. (2005) · Internet Mathematics · doi:10.1080/15427951.2005.10129104 · Cited by 282
Metadata
- Authors
- Shkapenyuk, V., & Suel, T.
- Venue
- Proceedings 18th International Conference on Data Engineering
- Publisher
- IEEE Comput. Soc
- Published
- 2003-06-25
- Pages
- 357-368
- ISSN
- 1084-4627 (ISSN-L)
- OpenAlex
- W2164542999
- Cited by
- 328
- References
- 21
- FWCI
- 18.63
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.