Inter-data-center network traffic prediction with elephant flows
Abstract
With the ever increasing number of large scale Internet applications, inter data center (inter-DC) data transfers are becoming more and more common. Traditional inter-DC transfers suffers from both low-utilization and congestion, and traffic prediction is an important method to optimize these transfers. Inter-DC traffic is harder to predict than many other types of network traffic, because it is dominated by a few large applications. We propose a model that significantly reduces the prediction errors. In our model, we combine wavelet transform with artificial neural network (ANN) to improve prediction accuracy. Specifically, we explicitly add information of elephant flows, the least predictable yet dominating traffic in inter-DC network, into our prediction model. To reduce the amount of monitoring overhead for the elephant flow information, we added interpolation to fill in the unknown values in the elephant flows. We demonstrate that we can reduce prediction errors over existing methods by 5%~10%. Our prediction is already in production at Baidu, one of the largest Internet companies in China, helping reducing the peak network bandwidth.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The linear ARIMA model has prediction error of approximately 8.5 and 6.9% for incoming and outgoing traffic, respectively.
“In contrast, the linear ARIMA model has prediction error of approximately 8.5 and 6.9% for incoming and outgoing traffic, respectively.”
From A comprehensive survey on machine learning for networking: Evolution, applications and research opportunities (2018) · cites this paper as [274]
Wavelet transformation reduces the average prediction errors for different prediction horizons by 5.4 and 2.9% for incoming and outgoing traffic, respectively.
“Evidently, wavelet transformation reduces the average prediction errors for different prediction horizons by 5.4 and 2.9% for incoming and outgoing traffic, respectively.”
From A comprehensive survey on machine learning for networking: Evolution, applications and research opportunities (2018) · cites this paper as [274]
k-step ahead predictions feeding k × 120 features into the NN show a relative RMSE (RRMSE) ranging from 4 to 10% for the NN-Wavelet transformation model as the prediction horizon varies from 30 s to 20 min.
“Thus, k-step ahead predictions, feed k × 120 features into the NN and show a relative RMSE (RRMSE) ranging from 4 to 10% for the NN-Wavelet transformation model as the prediction horizon varies from 30 s to 20 min.”
From A comprehensive survey on machine learning for networking: Evolution, applications and research opportunities (2018) · cites this paper as [274]
The combined NN and wavelet transform model reduces the peak inter-DC link utilization, i.e. the ISP's billed utilization, by about 9%.
“The combined NN and wavelet transform model reduces the peak inter-DC link utilization, i.e. the ISP's billed utilization, by about 9%.”
From A comprehensive survey on machine learning for networking: Evolution, applications and research opportunities (2018) · cites this paper as [274]
Li et al. propose a frequency domain based method for network traffic flows, instead of just traffic volume, focusing on predicting incoming and outgoing traffic volume on an inter-data center link dominated by elephant flows.
“For instance, Li et al. [274] propose a frequency domain based method for network traffic flows, instead of just traffic volume.The focus is on predicting incoming and outgoing traffic volume on an inter-data center link dominated by elephant flows.”
From A comprehensive survey on machine learning for networking: Evolution, applications and research opportunities (2018) · cites this paper as [274]
Non-TSF approaches were investigated in [94,274,365] to infer traffic volumes from flow count and packet header fields.
“Non-TSF approaches were investigated in [94,274,365] to infer traffic volumes from flow count and packet header fields.”
From A comprehensive survey on machine learning for networking: Evolution, applications and research opportunities (2018) · cites this paper as [274]
Cites
A fast learning algorithm for deep belief nets
Hinton, G. E., Osindero, S., & Teh, Y.-W. (2006) · Neural Computation · doi:10.1162/neco.2006.18.7.1527 · Cited by 16,336 · see citing works
de Boor, C. (1978) · Applied mathematical sciences · doi:10.1007/978-1-4612-6333-3 · Cited by 12,089 · see citing works
ENSEMBLE EMPIRICAL MODE DECOMPOSITION: A NOISE-ASSISTED DATA ANALYSIS METHOD
Wu, Z., & Huang, N. E. (2008) · Advances in Adaptive Data Analysis · doi:10.1142/s1793536909000047 · Cited by 8,847 · see citing works
Large-Scale Machine Learning with Stochastic Gradient Descent
Bottou, L. (2010) · Proceedings of COMPSTAT'2010 · doi:10.1007/978-3-7908-2604-3_16 · Cited by 5,795 · see citing works
Time series forecasting using a hybrid ARIMA and neural network model
Zhang, G. P. (2003) · Neurocomputing · doi:10.1016/s0925-2312(01)00702-0 · Cited by 4,576 · see citing works
Forecasting with artificial neural networks:
Zhang, G., Eddy Patuwo, B., & Y. Hu, M. (1998) · International Journal of Forecasting · doi:10.1016/s0169-2070(97)00044-7 · Cited by 4,229 · see citing works
AN INTRODUCTION TO LONG‐MEMORY TIME SERIES MODELS AND FRACTIONAL DIFFERENCING
Granger, C. W. J., & Joyeux, R. (1980) · Journal of Time Series Analysis · doi:10.1111/j.1467-9892.1980.tb00297.x · Cited by 3,435 · see citing works
Fitting autoregressive models for prediction
Akaike, H. (1969) · Annals of the Institute of Statistical Mathematics · doi:10.1007/bf02532251 · Cited by 2,499 · see citing works
STL : A Seasonal-Trend Decomposition Procedure Based on Loess
Cleveland, R. B. (1990) · Medical Entomology and Zoology · Cited by 2,272 · see citing works
Fitting Autoregreesive Models for Prediction
Akaike, H. (1998) · Springer series in statistics · doi:10.1007/978-1-4612-1694-0_10 · Cited by 2,040 · see citing works
Introduction to Wavelets and Wavelet Transforms: A Primer
Burrus, C. S., Gopinath, R. A., Guo, H., Odegard, J. E., & Selesnick, I. (1997) · Cited by 1,794 · see citing works
Jain, S., Kumar, A., Mandal, S., Ong, J. S., Poutievski, L., Singh, A., Venkata, S., Wanderer, J., Zhou, J., Zhu, M., et al. (2013) · ACM SIGCOMM Conference (SIGCOMM) · doi:10.1145/2486001.2486019 · Cited by 1,770 · see citing works
ARIMA models to predict next-day electricity prices
Contreras, J., Espínola, R., Nogales, F. J., & Conejo, A. J. (2003) · IEEE Transactions on Power Systems · doi:10.1109/tpwrs.2002.804943 · Cited by 1,557 · see citing works
Analysis of Microarray Data Using Z Score Transformation
Cheadle, C., Vawter, M. P., Freed, W. J., & Becker, K. G. (2003) · Journal of Molecular Diagnostics · doi:10.1016/s1525-1578(10)60455-2 · Cited by 1,150 · see citing works
Achieving high utilization with software-driven WAN
Hong, C.-Y., Kandula, S., Mahajan, R., Zhang, M., Gill, V., Nanduri, M., & Wattenhofer, R. P. (2013) · ACM SIGCOMM Conference (SIGCOMM) · doi:10.1145/2486001.2486012 · Cited by 1,074 · see citing works
ARIMA Models to Predict Next-Day Electricity Prices
Contreras, J., Espínola, R., Nogales, F. J., et al. (2002) · IEEE Power Engineering Review · doi:10.1109/mper.2002.4312577 · Cited by 934 · see citing works
Adeli, H., & Jiang, X. (2008) · CRC Press eBooks · doi:10.1201/9781482281767-9 · Cited by 665 · see citing works
Forecast of hourly average wind speed with ARMA models in Navarre (Spain)
Torres, J. L., García, A., Blas, M. D., et al. (2004) · Solar Energy · doi:10.1016/j.solener.2004.09.013 · Cited by 660 · see citing works
Benson, T., Anand, A., Akella, A., & Zhang, M. (2011) · ACM International Conference on emerging Networking EXperiments and Technologies (CoNEXT) · doi:10.1145/2079296.2079304 · Cited by 604 · see citing works
A multifractal wavelet model with application to network traffic
Riedi, R. H., Crouse, M. S., Ribeiro, V. J., & Baraniuk, R. G. (1999) · IEEE Transactions on Information Theory · doi:10.1109/18.761337 · Cited by 577 · see citing works
Achieving high utilization with software-driven WAN
Hong, C.-Y., Kandula, S., Mahajan, R., Zhang, M., Gill, V., Nanduri, M., & Wattenhofer, R. P. (2013) · ACM SIGCOMM Computer Communication Review · doi:10.1145/2534169.2486012 · Cited by 343 · see citing works
A comparative study of neural network and Box-Jenkins ARIMA modeling in time series prediction
Ho, S. L., Xie, M., & Goh, T. N. (2002) · Computers & Industrial Engineering · doi:10.1016/s0360-8352(02)00036-0 · Cited by 331 · see citing works
A Self-Organizing Learning Array System for Power Quality Classification Based on Wavelet Transform
He, H., & Starzyk, J. A. (2005) · IEEE Transactions on Power Delivery · doi:10.1109/tpwrd.2005.852392 · Cited by 261 · see citing works
Wang, H. H., Xie, H., Qiu, L., et al. (2006) · ACM SIGCOMM Computer Communication Review · doi:10.1145/1151659.1159926 · Cited by 213 · see citing works
A comparison of species diversity estimators
Mouillot, D., & Leprêtre, A. (1999) · Population Ecology · doi:10.1007/s101440050024 · Cited by 165 · see citing works
Study on network traffic prediction techniques
Feng, H., & Shu, Y. (2005) · International Conference on Wireless Communications, Networking and Mobile Computing · doi:10.1109/wcnm.2005.1544219 · Cited by 132 · see citing works
Multiresolution FIR neural-network-based learning algorithm applied to network traffic prediction
Alarcon‐Aquino, V., & Barria, J. A. (2006) · IEEE Transactions on Systems Man and Cybernetics Part C (Applications and Reviews) · doi:10.1109/tsmcc.2004.843217 · Cited by 123 · see citing works
A pragmatic definition of elephants in internet backbone traffic
Papagiannaki, K., Taft, N., Bhattacharyya, S., et al. (2002) · doi:10.1145/637201.637227 · Cited by 90 · see citing works
SVM-Based Models for Predicting WLAN Traffic
Feng, H., Shu, Y., Wang, S., et al. (2006) · International Conference on Communications · doi:10.1109/icc.2006.254860 · Cited by 62 · see citing works
Fuzzy-Neural Network Traffic Prediction Framework with Wavelet Decomposition
Xiao, H. L., Sun, H., Ran, B., et al. (2003) · Transportation Research Record Journal of the Transportation Research Board · doi:10.3141/1836-03 · Cited by 55 · see citing works
Izenman, A. J. (2009) · Springer texts in statistics · doi:10.1007/978-0-387-78189-1_10 · Cited by 42 · see citing works
On the characteristics of Internet traffic variability: spikes and elephants
Mori, T., Kawahara, R., Naito, S., et al. (2004) · doi:10.1109/saint.2004.1266104 · Cited by 35 · see citing works
On the characteristics of Internet traffic variability: Spikes and Elephants
森, 達., 35108, 60708551, et al. (2004) · Waseda University Repository (Waseda University) · Cited by 32 · see citing works
A pragmatic definition of elephants in internet backbone traffic
Papagiannaki, K., Taft, N., Bhattacharyya, S., et al. (2002) · doi:10.1145/637224.637227 · Cited by 20 · see citing works
Hu, K., Sim, A., Antoniades, D., et al. (2013) · Lecture notes in computer science · doi:10.1007/978-3-642-39712-7_46 · Cited by 19 · see citing works
Nonlinear network traffic prediction based on BP neural network
Huang, Y. (2007) · Journal of Computer Applications · Cited by 19 · see citing works
Network bandwidth utilization forecast model on high bandwidth networks
Yoo, W., & Sim, A. (2015) · 2012 International Conference on Computing, Networking and Communications (ICNC) · doi:10.1109/iccnc.2015.7069393 · Cited by 18 · see citing works
Qian, Y., Xia, J., Fu, K., et al. (2012) · International Conference on Consumer Electronics, Communications and Networks (CECNet) · doi:10.1109/cecnet.2012.6201816 · Cited by 17 · see citing works
A network traffic prediction model based on recurrent wavelet neural network
Zhang, K., Chai, Y., & Fu, X.-A. (2012) · International Conference on Computer Science and Network Technology (ICCSNT) · doi:10.1109/iccsnt.2012.6526232 · Cited by 13 · see citing works
Wavelet Transform-based Network Traffic Prediction: A Fast On-line Approach
Zhao, H. L., & Ansari, N. (2012) · Journal of Computing and Information Technology · doi:10.2498/cit.1001989 · Cited by 11 · see citing works
S-ARMA MODEL FOR NETWORK TRAFFIC PREDICTION IN WIRELESS SENSOR NETWORKS
Periyanayagi, S., & Sumathy, V. (2014) · Cited by 10 · see citing works
Network traffic prediction algorithm research based on PSO-BP neural network
Cheng, W., & Feng, P. (2015) · Advances in intelligent systems research/Advances in Intelligent Systems Research · doi:10.2991/isrme-15.2015.252 · Cited by 6 · see citing works
Network Traffic Prediction Algorithm based on Wavelet Transform
Tan, X., Fang, W., & -, Y. Q. (2013) · International Journal of Advancements in Computing Technology · doi:10.4156/ijact.vol5.issue5.22 · Cited by 5 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Boutaba, R., Salahuddin, M. A., Limam, N., Ayoubi, S., Shahriar, N., Estrada-Solano, F., & Caicedo, O. M. (2018) · Journal of Internet Services and Applications · doi:10.1186/s13174-018-0087-2 · Cited by 1,036 · see citing works
- Authors
- Li, Y., Liu, H., Yang, W., Hu, D., & Xu, W.
- Affiliations
- Tsinghua University (CN)
- Baidu (China) (CN)
- Venue
- NOMS 2016 - 2016 IEEE/IFIP Network Operations and Management Symposium
- Publisher
- IEEE
- Published
- 2016-04
- Pages
- 206-213
- OpenAlex
- W2471456127
- Cited by
- 34
- References
- 43
- Citations by year
- 2024: 1 · 2023: 3 · 2022: 4 · 2021: 7 · 2020: 9 · 2019: 7 · 2018: 3
- FWCI
- 2.85
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE/IFIP Network Operations and Management Symposium (NOMS) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Networks and Communications › Network Traffic and Congestion Control
- Physical Sciences › Computer Science › Artificial Intelligence › Internet Traffic Analysis and Secure E-voting
- Physical Sciences › Computer Science › Computer Networks and Communications › Software-Defined Networks and 5G
- Keywords
- traffic prediction, elephant flows, wavelet transform, artificial neural networks, network congestion
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
