The DOI matches its Crossref record; 11 of the 11 references it cites were found in a registry or an index.
Web Scraping or Web Crawling: State of Art, Techniques, Approaches and Application
Open accessAbstract
Web scraping or web crawling refers to the procedure of automatic extraction of data from websites using software. It is a process that is particularly important in fields such as Business Intelligence in the modern age. Web scrapping is a technology that allow us to extract structured data from text such as HTML. Web scrapping is extremely useful in situations where data isn’t provided in machine readable format such as JSON or XML. The use of web scrapping to gather data allows us to gather prices in near real time from retail store sites and provide further details, web scrapping can also be used to gather intelligence of illicit businesses such as drug marketplaces in the darknet to provide law enforcement and researchers valuable data such as drug prices and varieties that would be unavailable with conventional methods. It has been found that using a web scraping program would yield data that is far more thorough, accurate, and consistent than manual entry. Based on the result it has been concluded that Web scraping is a highly useful tool in the information age, and an essential one in the modern fields. Multiple technologies are required to implement web scrapping properly such as spidering and pattern matching which are discussed. This paper is looking into what web scraping is, how it works, web scraping stages, technologies, how it relates to Business Intelligence, artificial intelligence, data science, big data, cyber securityو how it can be done with the Python language, some of the main benefits of web scraping, and what the future of web scraping may look like, and a special degree of emphasis is placed on highlighting the ethical and legal issues. Keywords: Web Scraping, Web Crawling, Python Language, Business Intelligence, Data Science, Artificial Intelligence, Big Data, Cloud Computing, Cybersecurity, legal, ethical.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Landers, R. N., Brusso, R. C., Cavanaugh, K. J., & Collmus, A. B. (2016) · Psychological Methods · doi:10.1037/met0000081 · Cited by 192
Fiesler, C. L., Beard, N., & Keegan, B. (2020) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v14i1.7290 · Cited by 101
Web scraping for food price research
Irek, J. (2019) · British Food Journal · doi:10.1108/bfj-02-2019-0081 · Cited by 77
Web Scraping in the Statistics and Data Science Curriculum: Challenges and Opportunities
Dogucu, M., & Çetinkaya-Rundel, M. (2020) · Journal of Statistics and Data Science Education · doi:10.1080/10691898.2020.1787116 · Cited by 73
Cloud Based Web Scraping for Big Data Applications
Chaulagain, R. S., Pandey, S., Basnet, S. R., & Shakya, S. (2017) · IEEE International Conference on Smart Cloud (SmartCloud) · doi:10.1109/smartcloud.2017.28 · Cited by 67
A Web Scraping Methodology for Bypassing Twitter API RestrictionsPreprint
Hernandez-Suarez, A., Sánchez-Pérez, G., Toscano-Medina, K., Martinez-Hernandez, V., Sánchez, V., & Perez-Meana, H. M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.09875 · Cited by 43
ScrAPIr: Making Web Data APIs Accessible to End Users
Alrashed, T., Almahmoud, J., Zhang, A. X., & Karger, D. R. (2020) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376691 · Cited by 23
Suganya, E., & Vijayarani, S. (2019) · Advances in intelligent systems and computing · doi:10.1007/978-3-030-16660-1_66 · Cited by 21
Using Ontologies in Cybersecurity Field
Georgescu, T. M., & Smeureanu, I. (2017) · Informatica Economica · doi:10.12948/issn14531305/21.3.2017.01 · Cited by 15
Twenty Years of Web Scraping and the Computer Fraud and Abuse Act
Sellars, A. (2018) · bepress Legal Repository · Cited by 13
Cited by
Boedijanto, F. J. O., & Delina, L. L. (2024) · Energy Research & Social Science · doi:10.1016/j.erss.2024.103638 · Cited by 33
Artificial Intelligence into Multimedia Deepfakes Creation and Detection
Khder, M. A., Shorman, S., Aldoseri, D. T., & Saeed, M. M. (2023) · 2023 International Conference on IT Innovation and Knowledge Discovery (ITIKD) · doi:10.1109/itikd56332.2023.10099744 · Cited by 27
Self-citation
Metadata
- Authors
- Khder, M.
- Venue
- International Journal of Advances in Soft Computing and its Applications
- Publisher
- Alzaytoonah University of Jordan
- Published
- 2021-11-28
- Volume
- 13
- Issue
- 3
- Pages
- 145-168
- ISSN
- 2074-8523 (ISSN-L)
- 2710-1274
- OpenAlex
- W4200146754
- Cited by
- 284
- References
- 11
- FWCI
- 34.45
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Web Scraping or Web Crawling: State of Art, Techniques, Approaches and Application. https://app.cochat.ai/papers/10.15849/ijasca.211128.11
- The paper
- Khder, M. (2021). Web Scraping or Web Crawling: State of Art, Techniques, Approaches and Application. International Journal of Advances in Soft Computing and its Applications. https://doi.org/10.15849/ijasca.211128.11
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.