The DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.
Design and implementation of a high-performance distributed Web crawler
Abstract
Broad Web search engines as well as many more specialized search tools rely on Web crawlers to acquire large collections of pages for indexing and analysis. Such a Web crawler may interact with millions of hosts over a period of weeks or months, and thus issues of robustness, flexibility, and manageability are of major importance. In addition, I/O performance, network resources, and OS limits must be taken into account in order to achieve high performance at a reasonable cost. In this paper, we describe the design and implementation of a distributed Web crawler that runs on a network of workstations. The crawler scales to (at least) several hundred pages per second, is resilient against system crashes and other events, and can be adapted to various crawling applications. We present the software architecture of the system, discuss the, performance bottlenecks, and describe efficient techniques for achieving high performance. We also report preliminary experimental results based on a crawl of 120 million pages on 5 million hosts.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The anatomy of a large-scale hypertextual Web search engine
Brin, S., & Page, L. (1998) · Computer Networks and ISDN Systems · doi:10.1016/s0169-7552(98)00110-x · Cited by 16,004
Baeza-Yates, R. A., & Ribeiro-Neto, B. A. (1999) · Cited by 11,505
Focused crawling: a new approach to topic-specific Web resource discovery
Chakrabarti, S., van den Berg, M., & Dom, B. (1999) · Computer Networks · doi:10.1016/s1389-1286(99)00052-3 · Cited by 1,486
Efficient crawling through URL ordering
Cho, J., Garcia-Molina, H., & Page, L. (1998) · Computer Networks and ISDN Systems · doi:10.1016/s0169-7552(98)00108-1 · Cited by 837
Mercator: A scalable, extensible Web crawler
Heydon, A., & Najork, M. (1999) · World Wide Web · doi:10.1023/a:1019213109274 · Cited by 582
Focused Crawling Using Context Graphs
Diligenti, M., Coetzee, F., Lawrence, S., Giles, C. L., & Gori, M. (2000) · Cited by 533
Arasu, A., Cho, J., Garcia-Molina, H., Paepcke, A., & Raghavan, S. (2001) · ACM Transactions on Internet Technology · doi:10.1145/383034.383035 · Cited by 505
Mining the Web's link structure
Chakrabarti, S., Dom, B. E., Kumar, S. R., Raghavan, P., Rajagopalan, S., Tomkins, A., Gibson, D., & Kleinberg, J. (1999) · Computer · doi:10.1109/2.781636 · Cited by 501
The Evolution of the Web and Implications for an Incremental Crawler
Cho, J. J., & García-Molina, H. (2000) · Cited by 499
Synchronizing a database to improve freshness
Cho, J., & Garcia-Molina, H. (2000) · ACM SIGMOD Record · doi:10.1145/335191.335391 · Cited by 346
Cited by
UbiCrawler: a scalable fully distributed Web crawler
Boldi, P., Codenotti, B., Santini, M., & Vigna, S. (2004) · Software: Practice and Experience · doi:10.1002/spe.587 · Cited by 560
Olston, C., & Najork, M. (2010) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000017 · Cited by 370
The Perron-Frobenius theorem: some of its applications
Pillai, S. U., Suel, T., & Cha, S. (2005) · IEEE Signal Processing Magazine · doi:10.1109/msp.2005.1406483 · Cited by 338
Self-citationWeb application security assessment by fault injection and behavior monitoring
Huang, Y.-W., Huang, S.-K., Lin, T.-P., & Tsai, C.-H. (2003) · Proceedings of the twelfth international conference on World Wide Web - WWW '03 · doi:10.1145/775152.775174 · Cited by 319
Towards Scaling Fully Personalized PageRank: Algorithms, Lower Bounds, and Experiments
Fogaras, D., Rácz, B., Csalogány, K., & Sarlós, T. (2005) · Internet Mathematics · doi:10.1080/15427951.2005.10129104 · Cited by 282
Effective page refresh policies for Web crawlers
Cho, J., & Garcia-Molina, H. (2003) · ACM Transactions on Database Systems · doi:10.1145/958942.958945 · Cited by 258
I/O-efficient techniques for computing pagerank
Chen, Y.-Y., Gan, Q., & Suel, T. (2002) · Proceedings of the eleventh international conference on Information and knowledge management · doi:10.1145/584792.584882 · Cited by 59
Self-citationKoloveas, P., Chantzios, T., Tryfonopoulos, C., & Skiadopoulos, S. (2019) · 2019 IEEE World Congress on Services (SERVICES) · doi:10.1109/services.2019.00016 · Cited by 48
Batch query processing for web search engines
Ding, S., Attenberg, J., Baeza-Yates, R., & Suel, T. (2011) · Proceedings of the fourth ACM international conference on Web search and data mining · doi:10.1145/1935826.1935858 · Cited by 22
Self-citation
Metadata
- Authors
- Shkapenyuk, V., & Suel, T.
- Venue
- Proceedings 18th International Conference on Data Engineering
- Publisher
- IEEE Comput. Soc
- Published
- 2003-06-25
- Pages
- 357-368
- ISSN
- 1084-4627 (ISSN-L)
- OpenAlex
- W2164542999
- Cited by
- 328
- References
- 21
- FWCI
- 18.63
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.