Learning to detect malicious URLs
Open accessThe DOI matches its Crossref record; 39 of the 39 references it cites were found in a registry or an index.
Abstract
Malicious Web sites are a cornerstone of Internet criminal activities. The dangers of these sites have created a demand for safeguards that protect end-users from visiting them. This article explores how to detect malicious Web sites from the lexical and host-based features of their URLs. We show that this problem lends itself naturally to modern algorithms for online learning. Online algorithms not only process large numbers of URLs more efficiently than batch algorithms, they also adapt more quickly to new features in the continuously evolving distribution of malicious URLs. We develop a real-time system for gathering URL features and pair it with a real-time feed of labeled URLs from a large Web mail provider. From these features and labels, we are able to train an online classifier that detects malicious Web sites with 99% accuracy over a balanced dataset.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Advances in Neural Information Processing Systems 19
(2007) · The MIT Press eBooks · doi:10.52202/085713 · Cited by 15,539
The perceptron: A probabilistic model for information storage and organization in the brain.
Rosenblatt, F. (1958) · Psychological Review · doi:10.1037/h0042519 · Cited by 12,280
Pattern Recognition and Machine Learning
(2006) · doi:10.1007/978-0-387-45528-0 · Cited by 9,824
Adaptive Algorithms and Stochastic Approximations
Benveniste, A., Métivier, M., & Priouret, P. (1990) · Springer eBooks · doi:10.1007/978-3-642-75894-2 · Cited by 2,006
Domain names - implementation and specification
Mockapetris, P. V. (1987) · doi:10.17487/rfc1035 · Cited by 1,698 · cited 1×
“The Domain Name System (DNS) is a hierarchical network of servers responsible for translating domain names into IP addresses and other kinds of information [Mockapretis 1987a; 1987b].”
Online Passive-Aggressive Algorithms
Crammer, K., Dekel, O., Keshet, J., Shalev‐Shwartz, S., & Singer, Y. (2003) · Cited by 1,661
Internet Protocol, Version 6 (IPv6) Specification
Deering, S., & Hinden, R. (1998) · doi:10.17487/rfc2460 · Cited by 1,524
Internet Protocol, Version 6 (IPv6) Specification
Deering, S., & Hinden, R. (2017) · doi:10.17487/rfc8200 · Cited by 1,520
Domain names - concepts and facilities
Mockapetris, P. V. (1987) · doi:10.17487/rfc1034 · Cited by 1,467
Dynamic Host Configuration Protocol
Droms, R. (1993) · doi:10.17487/rfc1531 · Cited by 1,025
Online Passive-Aggressive Algorithms
CrammerKoby, DekelOfer, KeshetJoseph, Shalev-ShwartzShai, & SingerYoram (2006) · Journal of Machine Learning Research · doi:10.5555/1248547.1248566 · Cited by 894
Beyond blacklists: learning to detect malicious web sites from suspicious URLs
Ma, J., Saul, L. K., Savage, S., & Voelker, G. M. (2009) · Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1557019.1557153 · Cited by 803
Self-citationDynamic Host Configuration Protocol
Droms, R. (1997) · doi:10.17487/rfc2131 · Cited by 761
Cantina: a content-based approach to detecting phishing web sites
Zhang, Y., Hong, J. I., & Cranor, L. F. (2007) · Proceedings of the 16th international conference on World Wide Web · doi:10.1145/1242572.1242659 · Cited by 757 · cited 2×
“CANTINA classi.es phishing URLs by thresholding a weighted sum of 8 features (4 content-related, 3 lexical, and 1 WHOIS-related) [Zhang et al. 2007].”
MethodProceedings 2002 IEEE International Conference on Data Mining. ICDM 2002
(2002) · 2002 IEEE International Conference on Data Mining, 2002. Proceedings. · doi:10.1109/icdm.2002.1183878 · Cited by 727
Cost-sensitive learning by cost-proportionate example weighting
Zadrozny, B., Langford, J., & Abe, N. (2004) · Third IEEE International Conference on Data Mining · doi:10.1109/icdm.2003.1250950 · Cited by 680
Learning to detect phishing emails
Fette, I., Sadeh, N., & Tomasic, A. (2007) · Proceedings of the 16th international conference on World Wide Web · doi:10.1145/1242572.1242660 · Cited by 604
Sequential updating of conditional probabilities on directed graphical structures
Spiegelhalter, D. J., & Lauritzen, S. L. (1990) · Networks · doi:10.1002/net.3230200507 · Cited by 521
A comparison of machine learning techniques for phishing detection
Abu-Nimeh, S., Nappa, D., Wang, X., & Nair, S. (2007) · Proceedings of the anti-phishing working groups 2nd annual eCrime researchers summit · doi:10.1145/1299015.1299021 · Cited by 502
Identifying suspicious URLs: an application of large-scale online learning
Ma, J., Saul, L. K., Savage, S., & Voelker, G. M. (2009) · Proceedings of the 26th Annual International Conference on Machine Learning · doi:10.1145/1553374.1553462 · Cited by 498
Self-citationA framework for detection and measurement of phishing attacks
Garera, S., Provos, N., Chew, M., & Rubin, A. D. (2007) · Proceedings of the 2007 ACM workshop on Recurring malcode · doi:10.1145/1314389.1314391 · Cited by 489
Provos, N., Mavrommatis, P., Rajab, M. A., Monrose, F., Google Inc, Mavrommatis, P., Abu, M., & Monrose, R. F. (2008) · Cited by 483
(2004) · Cited by 432
Confidence-weighted linear classification
Dredze, M., Crammer, K., & Pereira, F. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390190 · Cited by 375
Bottou, L., & Cun, Y. L. (2003) · Cited by 359
Wang, Y., Beck, D., Jiang, X., Roussev, R., Verbowski, C., Chen, S., & King, S. T. (2006) · Cited by 331
LIBLINEAR: A Library for Large Linear Classification
FanRong-En, ChangKai-Wei, HsiehCho-Jui, WangXiang-Rui, & LinChih-Jen (2008) · Journal of Machine Learning Research · doi:10.5555/1390681.1442794 · Cited by 328
Behind phishing: an examination of phisher modi operandi
McGrath, D. K., & Gupta, M. (2008) · Cited by 282
A Crawler-based Study of Spyware on the Web
Moshchuk, A., Bragin, T., Gribble, S. D., & Levy, H. M. (2005) · Network and Distributed System Security Symposium · Cited by 225
Fast webpage classification using URL features
Kan, M.-Y., & Thi, H. O. N. (2005) · Proceedings of the 14th ACM international conference on Information and knowledge management · doi:10.1145/1099554.1099649 · Cited by 224
Daigle, L. L. (2004) · doi:10.17487/rfc3912 · Cited by 199
SVMs for the Blogosphere: Blog Identification and Splog Detection
Kolari, P., Finin, T., & Joshi, A. (2006) · National Conference on Artificial Intelligence · Cited by 194
Improved Phishing Detection using Model-Based Features.
Bergholz, A., Chang, J. H., Paaß, G., Reichartz, F., & Strobel, S. (2008) · Conference on Email and Anti-Spam · Cited by 181
The Forgetron: A Kernel-Based Perceptron on a Budget
Dekel, O., Shalev-Shwartz, S., & Singer, Y. (2008) · SIAM Journal on Computing · doi:10.1137/060666998 · Cited by 176
SpyProxy: execution-based detection of malicious web content
Moshchuk, A., Bragin, T., Deville, D., Gribble, S. D., & Levy, H. M. (2007) · USENIX Security Symposium · Cited by 129
Exact Convex Confidence-Weighted Learning
Crammer, K., Dredze, M. H., & Pereira, F. M. B. (2008) · Cited by 125
Shades of grey: On the effectiveness of reputation-based “blacklists”
Sinha, S., Bailey, M., & Jahanian, F. (2008) · International Conference on Malicious and Unwanted Software · doi:10.1109/malware.2008.4690858 · Cited by 110 · cited 1×
“While such systems have minimal query overhead (just searching for a URL within the list) they can only offer incomplete protection because no blacklist is perfectly comprehensive and up-to-date [Sinha et al. 2008].”
BackgroundOrabona, F., Keshet, J., & Barbara Caputo, B. (2008) · International Conference on Machine Learning · doi:10.1145/1390156.1390247 · Cited by 102
A Quantitative Study of Forum Spamming Using Context-based Analysis.
Niu, Y., Chen, H., Hsu, F., Wang, Y., & Ma, M. (2007) · Cited by 78
Cited by
Malicious URL Detection using Machine Learning: A Survey
Sahoo, D., Liu, C., & Hoi, S. C. H. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1701.07179 · Cited by 274
Example-dependent cost-sensitive decision trees
Correa Bahnsen, A., Aouada, D., & Ottersten, B. (2015) · Expert Systems with Applications · doi:10.1016/j.eswa.2015.04.042 · Cited by 223
“Lastly, in the case of intrusion detection, classifying a benign connection as malicious have a different cost than when a malicious connection is accepted (Ma et al., 2011).”
Background
Metadata
- Authors
- Ma, J., Saul, L. K., Savage, S., & Voelker, G. M.
- Venue
- ACM Transactions on Intelligent Systems and Technology
- Published
- 2011-04
- Volume
- 2
- Issue
- 3
- Pages
- 1-24
- ISSN
- 2157-6904 (ISSN-L)
- 2157-6912
- OpenAlex
- W1975909792
- Cited by
- 225
- References
- 39
- FWCI
- 30.52
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.