CoChat

Security Data Collection and Data Analytics in the Internet: A Survey

Open access
Jing, Xuyang, Yan, Zheng, Pedrycz, Witold · 2019 · IEEE Communications Surveys & Tutorials
Cited by 130

The DOI matches its Crossref record; 3 of the 3 references it cites were found in a registry or an index.

Abstract

Attacks over the Internet are becoming more and more complex and sophisticated. How to detect security threats and measure the security of the Internet arises a significant research topic. For detecting the Internet attacks and measuring its security, collecting different categories of data and employing methods of data analytics are essential. However, the literature still lacks a thorough review on security-related data collection and analytics on the Internet. Therefore, it becomes a necessity to review the current state of the art in order to gain a deep insight on what categories of data should be collected and which methods should be used to detect the Internet attacks and to measure its security. In this paper, we survey existing studies about security-related data collection and analytics for the purpose of measuring the Internet security. We first divide the data related to network security measurement into four categories: 1) packet-level data; 2) flow-level data; 3) connection-level data; and 4) host-level data. For each category of data, we provide a specific classification and discuss its advantages and disadvantages with regard to the Internet security threat detection. We also propose several additional requirements for security-related data analytics in order to make the analytics flexible and scalable. Based on the usage of data categories and the types of data analytic methods, we review current detection methods for distributed denial of service flooding and worm attacks by applying the proposed requirements to evaluate their performance. Finally, based on the completed review, a list of open issues is outlined and future research directions are identified.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/comst.2018.2863942

Cites

3 of 3 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 3 references

Research works printed but not linked: 160

  1. 10.1016/j.comnet.2007.02.001 · DOI · Paper page
  2. 10.3390/e17042367 · DOI · Paper page
  3. 2018, LoadRunner
  4. 2018, Collectl
  5. 10.1109/SURV.2010.032210.00054 · DOI · Paper page
  6. brutlag, 2000, Aberrant behavior detection in time series for network service monitoring, Proc USENIX 14th Syst Admin Conf, 139
  7. ye, 2003, Computer intrusion detection through EWMA for autocorrelated and uncorrelated data, IEEE Trans Rel, 51, 75
  8. yu, 2012, A nonparametric adaptive CUSUM method and its application in network anomaly detection, Int J Adv Comput Technol, 4, 280
  9. lee, 2001, Information-theoretic measures for anomaly detection, Proc IEEE Symp Security Privacy, 130
  10. 10.1109/INM.2007.374825 · doi:10.1109/inm.2007.374825
  11. 2018, Netflow
  12. li, 2013, A survey of network flow applications, J Netw Comput Appl, 36, 567, 10.1016/j.jnca.2012.12.020 · DOI · Paper page
  13. weber, 2006, The fundamentals of passive monitoring access
  14. 10.1145/948109.948116 · DOI · Paper page
  15. kavisankar, 2014, A pioneer scheme in the detection and defense of DRDoS attack involving spoofed flooding packets, KSII Trans Internet Inf Syst, 8, 1726, 10.3837/tiis.2014.05.013 · DOI
  16. 10.1109/ISSA.2015.7335069 · doi:10.1109/issa.2015.7335069
  17. 10.1016/j.comnet.2012.07.003 · DOI
  18. 10.1109/COMST.2014.2321898 · DOI · Paper page
  19. 10.1007/978-3-642-23235-0_25 · doi:10.1007/978-3-642-23235-0_25
  20. abuhmed, 2008, A survey on deep packet inspection for intrusion detection systems, Mag Korea Telecommun Soc, 24, 25
  21. chen, 2012, CBF: A packet filtering method for DDoS attack defense in cloud environment, Proc IEEE 9th Int Conf Depend Auton Secure Comput, 427
  22. 10.1049/cp.2015.0754 · doi:10.1049/cp.2015.0754
  23. tong, 2016, A hybrid approach of mobile malware detection in Android, J Parallel Distrib Comput, 103, 22, 10.1016/j.jpdc.2016.10.012 · DOI
  24. 10.1109/NOMS.2010.5488473 · doi:10.1109/noms.2010.5488473
  25. 10.1016/j.future.2014.06.010 · DOI
  26. 10.1145/2808206 · DOI
  27. 10.1109/ICAL.2008.4636301 · doi:10.1109/ical.2008.4636301
  28. anbar, 2014, Connection failure message-based approach for detecting sequential and random TCP scanning, Indian J of Sci and Technol, 7, 628, 10.17485/ijst/2014/v7i5.8 · doi:10.17485/ijst/2014/v7i5.8
  29. rasheed, 2009, Intelligent failure connection algorithm for detecting Internet worms, Int J Comput Sci Netw Security, 9, 280
  30. 10.1109/TNET.2009.2020431 · DOI
  31. 10.1145/1162666.1162676 · doi:10.1145/1162666.1162676
  32. rasheed, 2009, A new generation for intelligent anti-Internet worm early system detection, Proc Int Conf Inform Comput, 61
  33. 10.1109/EBISS.2009.5137885 · doi:10.1109/ebiss.2009.5137885
  34. 10.1016/j.comcom.2015.06.012 · DOI
  35. 10.1109/NSS.2010.70 · DOI
  36. 10.1016/j.eswa.2007.01.040 · DOI
  37. 10.2298/CSIS161230032K · DOI
  38. 10.1016/j.eswa.2010.08.137 · DOI · Paper page
  39. vijayasarathy, 2011, A system approach to network modeling for DDoS detection using a naive Bayesian classifier, Proc Int Conf Commun Syst Netw, 1
  40. 10.1016/j.neucom.2015.04.101 · DOI · Paper page
  41. david, 2015, DDoS attack detection using fast entropy approach on flow—Based network traffic, Procedia Comput Sci, 50, 30, 10.1016/j.procs.2015.04.007 · doi:10.1016/j.procs.2015.04.007
  42. 10.1016/j.ins.2015.06.025 · doi:10.1016/j.ins.2015.06.025
  43. 10.1016/j.ins.2016.04.009 · DOI
  44. 10.1016/j.ins.2016.07.008 · doi:10.1016/j.ins.2016.07.008
  45. 10.1109/TSMCC.2012.2236648 · DOI · Paper page
  46. 10.1016/j.future.2016.06.036 · DOI
  47. 10.1109/COMST.2015.2392629 · DOI · Paper page
  48. 10.1016/j.ins.2015.01.025 · DOI · Paper page
  49. 10.1109/SURV.2013.031413.00127 · DOI · Paper page
  50. 10.1145/1216370.1216373 · DOI · Paper page
  51. yan, 2016, Unwanted traffic detection and control based on trust management, Information Fusion for Cyber-Security Analytics Trends and Patterns, 77
  52. 10.1109/COMST.2008.4483668 · DOI
  53. 10.1109/JSAC.2006.877136 · doi:10.1109/jsac.2006.877136
  54. 10.1109/COMST.2015.2487361 · DOI · Paper page
  55. 10.1016/j.cose.2014.03.010 · DOI
  56. 10.1080/13614568.2013.832807 · DOI
  57. 10.1109/SURV.2014.022714.00160 · DOI
  58. 10.1109/TPDS.2007.70786 · doi:10.1109/tpds.2007.70786
  59. 10.1109/GLOCOM.2009.5425454 · doi:10.1109/glocom.2009.5425454
  60. kim, 2006, PacketScore: A statistics-based packet filtering scheme against distributed denial-of-service attacks, IEEE Trans Depend Secure Comput, 3, 141, 10.1109/TDSC.2006.25 · DOI
  61. 10.1007/s11219-017-9368-4 · DOI
  62. 10.1016/j.dcan.2017.10.002 · DOI · Paper page
  63. 10.1109/SURV.2013.091213.00134 · DOI · Paper page
  64. 10.1016/j.jnca.2016.06.012 · DOI
  65. 10.1109/WETICE.2005.35 · DOI · Paper page
  66. collins, 2007, Hit-list worm detection and bot identification in large networks using protocol graphs, 2nd Int Workshop Recent Adv Intrusion Detect, 276, 10.1007/978-3-540-74320-0_15 · DOI
  67. 10.1016/j.comcom.2008.12.010 · doi:10.1016/j.comcom.2008.12.010
  68. 10.1109/TDSC.2010.13 · doi:10.1109/tdsc.2010.13
  69. 10.1109/TDSC.2014.2302298 · doi:10.1109/tdsc.2014.2302298
  70. 10.1016/j.future.2013.08.002 · DOI
  71. 10.1109/ISCC.2006.142 · DOI
  72. 10.1109/WorldCIS.2013.6751036 · doi:10.1109/worldcis.2013.6751036
  73. 10.1007/978-3-540-79232-1_22 · doi:10.1007/978-3-540-79232-1_22
  74. alonso, 2016, Mining IP to domain name interactions to detect DNS flood attacks on recursive DNS servers, SENSORS, 16, 10.3390/s16081311 · DOI
  75. 10.1109/TC.2013.191 · DOI
  76. 10.1016/j.comnet.2007.12.004 · DOI
  77. bhatia, 2017, Ensemble-based model for DDoS attack detection and flash event separation, Proc Future Technol Conf, 958
  78. 10.1109/TIFS.2017.2758754 · doi:10.1109/tifs.2017.2758754
  79. 10.1109/INFCOMW.2011.5928950 · doi:10.1109/infcomw.2011.5928950
  80. 10.1016/j.eswa.2008.06.037 · doi:10.1016/j.eswa.2008.06.037
  81. 10.1109/INFCOM.2013.6567003 · DOI
  82. muraleedharan, 2010, ADRISYA: A flow based anomaly detection system for slow and fast scan, Int J Netw Security Appl, 2, 234
  83. 10.1109/TPDS.2011.262 · DOI · Paper page
  84. farag, 2009, Intelligent system for worm detection, Int Arab J E-Technol, 1, 58
  85. 10.3906/elk-1308-188 · doi:10.3906/elk-1308-188
  86. 10.1109/AINA.2009.132 · doi:10.1109/aina.2009.132
  87. chatzis, 2008, Flow level data mining of DNS query streams for email worm detection, Proc Int Workshop Comput Intell Security Inf Syst (CISIS), 186
  88. 10.1007/s00521-009-0238-2 · DOI
  89. abdulla, 2014, Employing machine learning algorithms to detect unknown scanning and email worms, Int Arab J Inf Technol, 11, 140
  90. wagner, 2011, Machine learning approach for IP-flow record anomaly detection, Proc Int IFIP-TC6 Netw Conf, 28
  91. cui, 2005, BINDER: An extrusion-based break-in detector for personal computers, Proc USENIX Annu Tech Conf, 363
  92. 10.1016/j.comcom.2011.01.012 · DOI · Paper page
  93. 10.1109/CIS.2015.105 · doi:10.1109/cis.2015.105
  94. 10.1049/iet-ifs.2010.0158 · doi:10.1049/iet-ifs.2010.0158
  95. yadav, 2016, DDA: An approach to handle DDoS (ping flood) attack, Proc Int Conf ICT Sustain Develop, 11, 10.1007/978-981-10-0129-1_2 · doi:10.1007/978-981-10-0129-1_2
  96. 10.1587/transinf.E96.D.1716 · doi:10.1587/transinf.e96.d.1716
  97. 10.1109/ICoICT.2013.6574541 · DOI
  98. 10.1109/CyberSecPODS.2016.7502345 · doi:10.1109/cybersecpods.2016.7502345
  99. 10.1109/LANMAN.2015.7114740 · doi:10.1109/lanman.2015.7114740
  100. 10.1109/ICCCNT.2014.6963119 · doi:10.1109/icccnt.2014.6963119
  101. 10.1016/j.comcom.2012.04.002 · doi:10.1016/j.comcom.2012.04.002
  102. 10.1145/2555615 · doi:10.1145/2555615
  103. 10.1109/TNET.2008.926503 · DOI · Paper page
  104. 10.1109/FITME.2008.150 · doi:10.1109/fitme.2008.150
  105. 10.1109/COMST.2015.2474118 · DOI · Paper page
  106. 10.1109/ICWS.2017.58 · DOI
  107. 10.1109/TPDS.2007.1050 · doi:10.1109/tpds.2007.1050
  108. 10.1109/JPROC.2016.2558521 · DOI · Paper page
  109. 10.1109/IMIS.2016.88 · doi:10.1109/imis.2016.88
  110. 10.1109/TIFS.2011.2107320 · DOI · Paper page
  111. 10.1145/2980258.2980431 · DOI
  112. 10.1002/sec.1460 · doi:10.1002/sec.1460
  113. 10.1109/AsiaJCIS.2016.24 · doi:10.1109/asiajcis.2016.24
  114. 10.1109/ICNP.2004.1348109 · DOI
  115. xiao, 2016, A protocol-free detection against cloud oriented reflection DoS attacks, Soft Comput, 21, 3713, 10.1007/s00500-015-2025-6 · DOI
  116. maciá-fernández, 2007, LoRDAS: A low-rate DoS attack against application servers, Proc Int Workshop Critical Inf Infrastruct Secur, 197
  117. 10.1109/LCOMM.2012.121912.122257 · doi:10.1109/lcomm.2012.121912.122257
  118. 10.1109/ICC.2014.6883403 · doi:10.1109/icc.2014.6883403
  119. ryba, 2015, Amplification and DRDoS attack defense—A survey and new perspectives, arXiv 1505 07892
  120. 10.1109/TNET.2006.880180 · DOI
  121. kuhrer, 2014, Exit from hell? Reducing the impact of amplification DDoS attacks, Proc 23rd USENIX Secur Symp (USENIX Secur ), 111
  122. 10.14722/ndss.2014.23233 · doi:10.14722/ndss.2014.23233
  123. 10.1016/j.comcom.2008.05.033 · doi:10.1016/j.comcom.2008.05.033
  124. kambourakis, 2007, Detecting DNS amplification attacks, Proc Int Workshop Critical Inf Infrastruct Secur, 186
  125. 10.1007/978-3-319-17172-2_14 · doi:10.1007/978-3-319-17172-2_14
  126. wu, 2012, MSABMS-based approach of detecting LDoS attack, Comput Security, 31, 402, 10.1016/j.cose.2012.03.002 · doi:10.1016/j.cose.2012.03.002
  127. 10.1155/2017/3691629 · doi:10.1155/2017/3691629
  128. 10.1109/TDSC.2015.2443807 · doi:10.1109/tdsc.2015.2443807
  129. 10.1002/dac.3449 · doi:10.1002/dac.3449
  130. 10.1109/COMST.2015.2457491 · DOI · Paper page
  131. 10.1016/j.cose.2017.05.009 · DOI · Paper page
  132. 10.1587/transinf.E92.D.799 · doi:10.1587/transinf.e92.d.799
  133. 10.1109/SURV.2013.052213.00046 · DOI · Paper page
  134. 10.1016/j.comcom.2017.03.010 · DOI · Paper page
  135. jacobson, 1989, The tcpdump manual page
  136. 10.1145/2535372.2535398 · DOI
  137. combs, 2018, Wireshark
  138. 10.1007/s12652-013-0196-5 · doi:10.1007/s12652-013-0196-5
  139. 10.1109/INFOCOM.2016.7524507 · doi:10.1109/infocom.2016.7524507
  140. 2018, SNORT
  141. 10.1007/978-3-319-46301-8_27 · doi:10.1007/978-3-319-46301-8_27
  142. lyon, 2009, Nmap Network Scanning The Official Nmap Project Guide to Network Discovery and Security Scanning
  143. 10.1109/ICRTIT.2011.5972281 · DOI
  144. 10.1016/j.future.2016.01.020 · DOI
  145. 10.1145/2185376.2185382 · DOI
  146. adi, 2017, Stealthy denial of service (DoS) attack modelling and detection for HTTP/2 services, J Netw Comput Appl, 91, 1, 10.1016/j.jnca.2017.04.015 · DOI
  147. kang, 2013, The crossfire attack, Proc IEEE Symp Security Privacy, 127
  148. huistra, 2013, Detecting reflection attacks in DNS flows, Proc 19th Twente Student Conference on IT
  149. xue, 2014, Towards detecting target link flooding attack, Proc USENIX Large Install Syst Admin Conf, 81
  150. 10.1016/j.compeleceng.2012.07.004 · doi:10.1016/j.compeleceng.2012.07.004
  151. 10.1109/WIFS.2015.7368594 · doi:10.1109/wifs.2015.7368594
  152. 10.1007/s11432-008-0132-z · doi:10.1007/s11432-008-0132-z
  153. 10.1007/978-3-642-31968-6_5 · doi:10.1007/978-3-642-31968-6_5
  154. 10.2991/aiie-15.2015.63 · doi:10.2991/aiie-15.2015.63
  155. 10.1109/TNET.2005.857113 · DOI · Paper page
  156. 10.1145/948134.948136 · doi:10.1145/948134.948136
  157. she, 2016, Application-layer DDoS detection by K-means algorithm, Proc Int Conf Elect Electron Comp Eng, 75
  158. 10.3390/e18100350 · doi:10.3390/e18100350
  159. 10.1016/j.cose.2009.04.007 · doi:10.1016/j.cose.2009.04.007
  160. 10.1002/sec.1236 · DOI

Cited by

Metadata

Authors
Jing, X., Yan, Z., & Pedrycz, W.
Venue
IEEE Communications Surveys & Tutorials
Published
2019
Volume
21
Issue
1
Pages
586-618
ISSN
  • 1553-877X (ISSN-L)
  • 2373-745X
OpenAlex
W2886128149
Cited by
130
References
3
FWCI
9.18
Type
journal article
Language
en
Open access
green
Funders
  • National Key Research and Development Program of China (2016YFB0800704)
  • National Natural Science Foundation of China (61672410, U1536202)
  • Natural Science Foundation of Shaanxi Province (2016ZDJC-06)
  • 111 Project (B08038, B16037)
  • Academy of Finland (308087)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.