Online flow size prediction for improved network routing
Abstract
We describe an emerging application of data mining in the context of computer networks. This application concerns the problem of predicting the size of a flow and detecting elephant flows (very large flows). Flow size is a very important statistic that can be used to improve routing, load balancing and scheduling in computer networks. Flow size prediction is particularly challenging since flow patterns continuously change and predictions must be done in real time (milliseconds) to avoid delays. We describe how to formulate the problem as an online machine learning task to continuously adjust to changes in flow traffic. We evaluate the predictive nature of a set of features and the accuracy of three online predictors based on neural networks, Gaussian process regression and online Bayesian Moment Matching on three datasets of real traffic. We also demonstrate how to use such online predictors to improve routing (i.e., reduced flow completion time) in a network simulation.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
A 14% improvement is observed when the size of the three first packets is used along with the header information.
“A 14% improvement is observed when the size of the three first packets is used along with the header information.”
From A comprehensive survey on machine learning for networking: Evolution, applications and research opportunities (2018) · cites this paper as [365]
Poupart et al. explore the use of different ML techniques for flow size prediction and elephant flow detection, including GPR, oBMM and a (106, 60, 40, 1) MLP-NN.
“Poupart et al. [365] explore the use of different ML techniques for flow size prediction and elephant flow detection.These techniques include gaussian processes regression (GPR), online bayesian moment matching (oBMM) and a (106, 60, 40, 1) MLP-NN.”
From A comprehensive survey on machine learning for networking: Evolution, applications and research opportunities (2018) · cites this paper as [365]
The experiments show noticeable discrepancies in the performance of the approaches with varying datasets.
“The experiments show noticeable discrepancies in the performance of the approaches with varying datasets.”
From A comprehensive survey on machine learning for networking: Evolution, applications and research opportunities (2018) · cites this paper as [365]
Although NN outperforms GPR in terms of robustness to class imbalance by looking at the consistency of its TPR with varying flow size threshold, it has the lowest average TNR of below 80% in all datasets.
“Although NN outperforms GPR in terms of robustness to class imbalance by looking at the consistency of its TPR with varying flow size threshold, it has the lowest average TNR of below 80% in all datasets.”
From A comprehensive survey on machine learning for networking: Evolution, applications and research opportunities (2018) · cites this paper as [365]
oBMM outperforms all other approaches in terms of average TNR in all 3 datasets.
“However, it is worth noting that oBMM outperforms all other approaches in terms of average TNR in all 3 datasets.”
From A comprehensive survey on machine learning for networking: Evolution, applications and research opportunities (2018) · cites this paper as [365]
According to the authors, GPR improves the completion time by 6.6% in average for 99% of elephant flows when only the first packet header information is considered.
“According to the authors, GPR improves the completion time by 6.6% in average for 99% of elephant flows when only the first packet header information is considered.”
From A comprehensive survey on machine learning for networking: Evolution, applications and research opportunities (2018) · cites this paper as [365]
NN and GPR have an average TPR consistently above 80%.
“On the other hand, NN and GPR, have an average TPR consistently above 80%.”
From A comprehensive survey on machine learning for networking: Evolution, applications and research opportunities (2018) · cites this paper as [365]
The motive for flow size prediction in [365] is to discriminate elephant flows from mice flows in routing to speed up elephant flow completion time.
“The motive for flow size prediction in [365], is to discriminate elephant flows from mice flows in routing to speed up elephant flow completion time.”
From A comprehensive survey on machine learning for networking: Evolution, applications and research opportunities (2018) · cites this paper as [365]
Although oBMM outperforms all other approaches in one dataset with an average TPR and TNR very close to 100%, it fails miserably in the other datasets with an average TPR below 50% for one dataset.
“Although oBMM outperforms all other approaches in one dataset with an average TPR and TNR very close to 100%, it fails miserably in the other datasets with an average TPR below 50% for one dataset.”
From A comprehensive survey on machine learning for networking: Evolution, applications and research opportunities (2018) · cites this paper as [365]
Considering the size of the first three packets alone leads to over 13.5% improvement, regardless of whether GPR or oBMM is used, with a very slight impact on the completion time of mice flows.
“It is also noticed that considering the size of the first three packets alone leads to over 13.5% improvement, regardless of whether GPR or oBMM is used, with a very slight impact on the completion time of mice flows.”
From A comprehensive survey on machine learning for networking: Evolution, applications and research opportunities (2018) · cites this paper as [365]
Non-TSF approaches were investigated in [94,274,365] to infer traffic volumes from flow count and packet header fields.
“Non-TSF approaches were investigated in [94,274,365] to infer traffic volumes from flow count and packet header fields.”
From A comprehensive survey on machine learning for networking: Evolution, applications and research opportunities (2018) · cites this paper as [365]
According to [365], a more complex MLP-NN (in terms of number of layers and neurons) might be required to achieve better accuracy in a non-TSF setting, and the performance of supervised learning techniques varies when tested on different datasets.
“According to [365] a more complex MLP-NN (in terms of number of layers and neurons) might be required to achieve better accuracy in a non-TSF setting, and the performance of supervised learning techniques varies when tested on different datasets.”
From A comprehensive survey on machine learning for networking: Evolution, applications and research opportunities (2018) · cites this paper as [365]
Cites
Gaussian Processes for Machine Learning
Rasmussen, C. E., & Williams, C. K. I. (2005) · doi:10.7551/mitpress/3206.001.0001 · Cited by 11,705 · see citing works
McKeown (2008) · doi:10.1145/1355734.1355746 · Cited by 8,513 · see citing works
Pattern recognition and machine learning
(2007) · Choice Reviews Online · doi:10.5860/choice.44-5091 · Cited by 2,782 · see citing works
Network traffic characteristics of data centers in the wild
Benson, T., Akella, A., & Maltz, D. A. (2010) · ACM SIGCOMM Internet Measurement Conference (IMC) · doi:10.1145/1879141.1879175 · Cited by 2,231 · see citing works
Silberschatz (1991) · Cited by 1,605 · see citing works
Silberschatz, A., Galvin, P., & Gagne, G. (1983) · Cited by 1,584 · see citing works
Hedera: dynamic flow scheduling for data center networks
Al-Fares, M., Radhakrishnan, S., Raghavan, B., Huang, N., & Vahdat, A. M. (2010) · doi:10.5555/1855711.1855730 · Cited by 1,529 · see citing works
Analysis of an Equal-Cost Multi-Path Algorithm
Hopps, C. (2000) · doi:10.17487/rfc2992 · Cited by 973 · see citing works
Farrington, N., Porter, G., Radhakrishnan, S., Bazzaz, H. H., Subramanya, V., Fainman, Y., Papen, G. C., & Vahdat, A. M. (2010) · ACM SIGCOMM Conference (SIGCOMM) · doi:10.1145/1851182.1851223 · Cited by 918 · see citing works
Wang, G., Andersen, D. G., Kaminsky, M., Papagiannaki, K., Ng, T. S. E., Kozuch, M. A., & Ryan, M. P. (2010) · ACM SIGCOMM Conference (SIGCOMM) · doi:10.1145/1851182.1851222 · Cited by 653 · see citing works
Mahout: Low-overhead datacenter traffic management using end-host-based elephant detection
Curtis, A. R., Kim, W.-H., & Yalagandula, P. (2011) · IEEE Conference on Computer Communications (INFOCOM) · doi:10.1109/infcom.2011.5934956 · Cited by 404 · see citing works
InMon Corporation's sFlow: A Method for Monitoring Traffic in Switched and Routed Networks
Phaal, P., Panchen, S., & McKee, N. (2001) · doi:10.17487/rfc3176 · Cited by 370 · see citing works
Why flow-completion time is the right metric for congestion control
Dukkipati, N., & McKeown, N. (2006) · ACM SIGCOMM Computer Communication Review · doi:10.1145/1111322.1111336 · Cited by 319 · see citing works
ElephantTrap: A low cost device for identifying large flows
Lu, Y., Prabhakar, B., & Bonomi, F. (2007) · doi:10.1109/hoti.2007.13 · Cited by 63 · see citing works
HadoopWatch: A first step towards comprehensive traffic forecasting in cloud computing
Yang, P., Chen, K., Wang, G., et al. (2014) · doi:10.1109/infocom.2014.6847920 · Cited by 53 · see citing works
Self-citationBuilding a RCP (Rate Control Protocol) Test Network
Dukkipati, N., Gibb, G., McKeown, N., et al. (2007) · doi:10.1109/hoti.2007.15 · Cited by 41 · see citing works
Online Algorithms for Sum-Product Networks with Continuous Variables
Jaini, P., Rashwan, A. M., Zhao, H., et al. (2016) · Probabilistic Graphical Models · Cited by 15 · see citing works
Self-citationOnline Bayesian Learning in Probabilistic Graphical Models using Moment Matching with Applications
Omar, F. (2018) · UWSpace (University of Waterloo) · doi:10.13140/rg.2.2.22951.04003 · Cited by 10 · see citing works
Emilie: Enhance the power of traffic identification
Shao, Y., Yang, B., Jiang, J., et al. (2014) · 2012 International Conference on Computing, Networking and Communications (ICNC) · doi:10.1109/iccnc.2014.6785300 · Cited by 6 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Boutaba, R., Salahuddin, M. A., Limam, N., Ayoubi, S., Shahriar, N., Estrada-Solano, F., & Caicedo, O. M. (2018) · Journal of Internet Services and Applications · doi:10.1186/s13174-018-0087-2 · Cited by 1,036 · see citing works
- Authors
- Poupart, P., Chen, Z., Jaini, P., Fung, F., Susanto, H., Geng, Y., Chen, L., Chen, K., & Jin, H.
- Affiliations
- University of Waterloo (CA)
- Huawei Technologies (China) (CN)
- Hong Kong University of Science and Technology (HK)
- Venue
- 2016 IEEE 24th International Conference on Network Protocols (ICNP)
- Publisher
- IEEE
- Published
- 2016-11
- Pages
- 1-6
- ISSN
- 1092-1648 (ISSN-L)
- OpenAlex
- W2567615407
- Cited by
- 96
- References
- 19
- Citations by year
- 2026: 2 · 2025: 4 · 2024: 15 · 2023: 10 · 2022: 14 · 2021: 10 · 2020: 20 · 2019: 16 · 2018: 3 · 2017: 2
- FWCI
- 3.77
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE International Conference on Network Protocols (ICNP) (publishedVersion)
- Rare & Special e-Zone (The Hong Kong University of Science and Technology) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Internet Traffic Analysis and Secure E-voting
- Physical Sciences › Computer Science › Computer Networks and Communications › Network Traffic and Congestion Control
- Physical Sciences › Computer Science › Computer Networks and Communications › Software-Defined Networks and 5G
- Keywords
- elephant flows, online machine learning, network routing, load balancing, real-time traffic prediction, artificial neural networks, Gaussian process regression, network traffic features, flow completion time
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review