Challenges of Big Data analysis
Open accessAbstract
Big Data bring new opportunities to modern society and challenges to data scientists. On the one hand, Big Data hold great promises for discovering subtle population patterns and heterogeneities that are not possible with small-scale data. On the other hand, the massive sample size and high dimensionality of Big Data introduce unique computational and statistical challenges, including scalability and storage bottleneck, noise accumulation, spurious correlation, incidental endogeneity and measurement errors. These challenges are distinguished and require new computational and statistical paradigm. This paper gives overviews on the salient features of Big Data and how these features impact on paradigm change on statistical and computational methods as well as computing architectures. We also provide various new perspectives on the Big Data analysis and computation. In particular, we emphasize on the viability of the sparsest solution in high-confidence set and point out that exogenous assumptions in most statistical methods for Big Data cannot be validated due to incidental endogeneity. They can lead to wrong statistical inferences and consequently wrong scientific conclusions.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Overfitting is a key risk in machine learning, meaning a machine may learn a model that explains errors and fails to generalize beyond training data.
“Overfitting is a key risk in this context, which means that a machine may learn a complete model that also explains the errors, consequently failing to generalize appropriately beyond its training data (Fan, Han, & Liu, 2014).”
From Artificial Intelligence and Management: The Automation–Augmentation Paradox (2021) · cites this paper as Fan, Han, & Liu, 2014
Section 3.2 (page unavailable) explains that using too many features accumulates noise and worsens classification of new observations, supporting the risk of poor generalization. It does not explicitly describe a machine learning a model that explains training errors, so that part of the attributed definition is not supported by our copy. (LLM, conf=0.82)
Experimental — accuracy not yet measuredThe combination of datasets in the big data age needs to cope with a number of challenges such as efficient data representation, redundancy reduction and data compression, spurious correlations, and noise accumulation.
“In addition, the combination of datasets in the big data age needs to cope with a number of challenges listed in [20,61], for example efficient data representation, redundancy reduction and data compression, spurious correlations, and noise accumulation.”
From Opening up Smart Cities: Citizen-Centric Challenges and Opportunities from GIScience (2016) · cites this paper as [61]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing
Benjamini, Y., & Hochberg, Y. (1995) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1995.tb02031.x · Cited by 109,482 · see citing works
Boyd, S., & Vandenberghe, L. (2004) · doi:10.1017/cbo9780511804441 · Cited by 56,601 · see citing works
A new look at the statistical model identification
Akaike, H. (1974) · IEEE Transactions on Automatic Control · doi:10.1109/tac.1974.1100705 · Cited by 51,879 · see citing works
Regression shrinkage and selection via the lasso
Tibshirani, R. (1996) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1996.tb02080.x · Cited by 51,482 · see citing works
Golub, G. (2012) · doi:10.56021/9781421407944 · Cited by 34,505 · see citing works
The Elements of Statistical Learning
Hastie, T., Tibshirani, R., & Friedman, J. (2009) · Springer Series in Statistics · doi:10.1007/978-0-387-84858-7 · Cited by 25,970 · see citing works
The Elements of Statistical Learning
Hastie, T., Friedman, J., & Tibshirani, R. (2001) · Springer Series in Statistics · doi:10.1007/978-0-387-21606-5 · Cited by 24,162 · see citing works
Donoho, D. L. (2006) · IEEE Transactions on Information Theory · doi:10.1109/tit.2006.871582 · Cited by 23,994 · see citing works
Longitudinal data analysis using generalized linear models
LIANG, K.-Y., & ZEGER, S. L. (1986) · Biometrika · doi:10.1093/biomet/73.1.13 · Cited by 18,550 · see citing works
Boyd, S., Parikh, N., Chu, E., Peleato, B., & Eckstein, J. (2011) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000016 · Cited by 17,075 · see citing works
Donoho, D. L. (2004) · Cited by 17,049 · see citing works
Gene Expression Omnibus: NCBI gene expression and hybridization array data repository
Edgar, R. (2002) · Nucleic Acids Research · doi:10.1093/nar/30.1.207 · Cited by 13,288 · see citing works
Boyd, S. (2010) · now publishers, Inc. eBooks · doi:10.1561/9781601984616 · Cited by 12,863 · see citing works
Indexing by latent semantic analysis
Deerwester, S. C., Dumais, S., Furnas, G. W., Landauer, T. K., & Harshman, R. A. (1990) · Journal of the American Society for Information Science · doi:10.1002/(sici)1097-4571(199009)41:6 · Cited by 12,758 · see citing works
A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems
Beck, A., & Teboulle, M. (2009) · SIAM Journal on Imaging Sciences · doi:10.1137/080716542 · Cited by 11,952 · see citing works
An Introduction To Compressive Sampling
Candes, E. J., & Wakin, M. B. (2008) · IEEE Signal Processing Magazine · doi:10.1109/msp.2007.914731 · Cited by 10,017 · see citing works
Efron, B., Hastie, T., Johnstone, I., & Tibshirani, R. (2004) · The Annals of Statistics · doi:10.1214/009053604000000067 · Cited by 9,357 · see citing works
Variable Selection via Nonconcave Penalized Likelihood and its Oracle Properties
B, J. F. A., & Li, R. (2001) · Journal of the American Statistical Association · doi:10.1198/016214501753382273 · Cited by 8,919 · see citing works
Ideal Spatial Adaptation by Wavelet Shrinkage
Donoho, D. L., & Johnstone, I. M. (1994) · Biometrika · doi:10.1093/biomet/81.3.425 · Cited by 8,730 · see citing works
Sparse MRI: The application of compressed sensing for rapid MR imaging
Lustig, M., Donoho, D., & Pauly, J. M. (2007) · Magnetic Resonance in Medicine · doi:10.1002/mrm.21391 · Cited by 6,952 · see citing works
Robust principal component analysis?
Candès, E. J., Li, X., Ma, Y., & Wright, J. N. (2011) · Journal of the ACM · doi:10.1145/1970392.1970395 · Cited by 6,914 · see citing works
Atomic Decomposition by Basis Pursuit
Chen, S. S., Donoho, D. L., & Saunders, M. A. (1998) · SIAM Journal on Scientific Computing · doi:10.1137/s1064827596304010 · Cited by 6,734 · see citing works
Twitter mood predicts the stock market
Bollen, J., Mao, H., & Zeng, X. (2011) · Journal of Computational Science · doi:10.1016/j.jocs.2010.12.007 · Cited by 5,206 · see citing works
Atomic Decomposition by Basis Pursuit
Chen, S. S., Donoho, D. L., & Saunders, M. A. (2001) · SIAM Review · doi:10.1137/s003614450037906x · Cited by 5,030 · see citing works
An iterative thresholding algorithm for linear inverse problems with a sparsity constraint.
Daubechies, I., Defrise, M., & De Mol, C. (2004) · Communications on Pure and Applied Mathematics · doi:10.1002/cpa.20042 · Cited by 4,958 · see citing works
Enhancing Sparsity by Reweighted ℓ 1 Minimization
Candès, E. J., Wakin, M. B., & Boyd, S. (2008) · Journal of Fourier Analysis and Applications · doi:10.1007/s00041-008-9045-x · Cited by 4,798 · see citing works
ANew Look at the Statistical Model Identification
AKAlKE, H. (2011) · Cited by 4,177 · see citing works
Determining the Number of Factors in Approximate Factor Models
Bai, J.-S., & Ng, S. (2000) · Econometrica · doi:10.1111/1468-0262.00273 · Cited by 4,141 · see citing works
Nearly unbiased variable selection under minimax concave penalty
Zhang, C. (2010) · The Annals of Statistics · doi:10.1214/09-aos729 · Cited by 3,909 · see citing works
Figueiredo, M. A. T., Nowak, R. D., & Wright, S. J. (2007) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2007.910281 · Cited by 3,531 · see citing works
The Dantzig selector: Statistical estimation when p is much larger than n
Candès, E. J., & Tao, T. (2007) · The Annals of Statistics · doi:10.1214/009053606000001523 · Cited by 3,152 · see citing works
Forecasting Using Principal Components From a Large Number of Predictors
Stock, J. H., & Watson, M. W. (2002) · Journal of the American Statistical Association · doi:10.1198/016214502388618960 · Cited by 3,104 · see citing works
Optimally sparse representation in general (nonorthogonal) dictionaries via ℓ 1 minimization
Donoho, D. L., & Elad, M. (2003) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0437847100 · Cited by 2,970 · see citing works
Di Martino, A., Yan, C.-G., Li, Q., Denio, E., Castellanos, F. X., Alaerts, K., Anderson, J. S., Assaf, M., Bookheimer, S. Y., Dapretto, M., et al. (2014) · Molecular Psychiatry · doi:10.1038/mp.2013.78 · Cited by 2,780 · see citing works
Sure Independence Screening for Ultrahigh Dimensional Feature Space
Fan, J., & Lv, J. (2008) · Journal of the Royal Statistical Society Series B (Statistical Methodology) · doi:10.1111/j.1467-9868.2008.00674.x · Cited by 2,643 · see citing works
Extensions of Lipschitz mappings into a Hilbert space
Johnson, W. B., & Lindenstrauss, J. (1984) · Contemporary mathematics - American Mathematical Society · doi:10.1090/conm/026/737400 · Cited by 2,583 · see citing works
Simultaneous analysis of Lasso and Dantzig selector
Bickel, P. J., Ritov, Y., & Tsybakov, A. B. (2009) · The Annals of Statistics · doi:10.1214/08-aos620 · Cited by 2,412 · see citing works
The positive false discovery rate: a Bayesian interpretation and the q-value
Storey, J. D. (2003) · The Annals of Statistics · doi:10.1214/aos/1074290335 · Cited by 2,204 · see citing works
Predicting the future with social media
Asur, S., & Huberman, B. A. (2010) · 2010 IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology · doi:10.1109/wi-iat.2010.63 · Cited by 2,081 · see citing works
Capturing Heterogeneity in Gene Expression Studies by Surrogate Variable Analysis
Leek, J. T., & Storey, J. D. (2007) · PLoS Genetics · doi:10.1371/journal.pgen.0030161 · Cited by 1,930 · see citing works
Pathwise coordinate optimization
Friedman, J. H., Hastie, T. J., Höfling, H., & Tibshirani, R. J. (2007) · The Annals of Applied Statistics · doi:10.1214/07-aoas131 · Cited by 1,906 · see citing works
Discrete Cosine Transform: Algorithms, Advantages, Applications
Rao, K. R., Yip, P. C. Y., & Britaňák, V. (1990) · Cited by 1,873 · see citing works
Statistics for High-Dimensional Data
Bühlmann, P., & van de Geer, S. (2011) · Springer series in statistics · doi:10.1007/978-3-642-20192-9 · Cited by 1,697 · see citing works
Inferential Theory for Factor Models of Large Dimensions
Bai, J. (2003) · Econometrica · doi:10.1111/1468-0262.00392 · Cited by 1,690 · see citing works
Low, Y., Bickson, D., Gonzalez, J. E., Guestrin, C., Kyrola, A., & Hellerstein, J. M. (2012) · Proceedings of the VLDB Endowment · doi:10.14778/2212351.2212354 · Cited by 1,656 · see citing works
Statistics for High-Dimensional Data: Methods, Theory and Applications
Bhlmann, P., & van de Geer, S. (2011) · Cited by 1,527 · see citing works
Database-friendly random projections: Johnson-Lindenstrauss with binary coins
Achlioptas, D. (2003) · Journal of Computer and System Sciences · doi:10.1016/s0022-0000(03)00025-4 · Cited by 1,374 · see citing works
Engle, R. F., Hendry, D. F., & Richard, J.-F. (1983) · Econometrica · doi:10.2307/1911990 · Cited by 1,346 · see citing works
Nonlinear Programming: Analysis and Methods
Avriel, M. (1976) · CERN Document Server (European Organization for Nuclear Research) · Cited by 1,307 · see citing works
Personal omics profiling reveals dynamic molecular and medical phenotypes
Chen, R., Mias, G. I., Li-Pook-Than, J., Jiang, L., Lam, H. Y. K., Chen, R., Miriami, E., Karczewski, K. J., Hariharan, M., Dewey, F. E., et al. (2012) · Cell · doi:10.1016/j.cell.2012.02.009 · Cited by 1,291 · see citing works
Nonlinear Programming Analysis and Methods
Rodin, E. Y. (1977) · Computers & Mathematics with Applications · doi:10.1016/0898-1221(77)90071-2 · Cited by 1,285 · see citing works
Efficiency of Coordinate Descent Methods on Huge-Scale Optimization Problems
Nesterov, Y. (2012) · SIAM Journal on Optimization · doi:10.1137/100802001 · Cited by 1,187 · see citing works
Covariance regularization by thresholding
Bickel, P. J., & Levina, E. (2007) · Cited by 1,118 · see citing works
A Survey of Dimension Reduction Techniques
Fodor, I. K. (2002) · doi:10.2172/15002155 · Cited by 1,109 · see citing works
Multiple Rare Alleles Contribute to Low Plasma Levels of HDL Cholesterol
Cohen, J. C., Kiss, R. S., Pertsemlidis, A., Marcel, Y. L., McPherson, R., & Hobbs, H. H. (2004) · Science · doi:10.1126/science.1099870 · Cited by 1,108 · see citing works
Penalized Regressions: The Bridge versus the Lasso
Fu, W. J. (1998) · Journal of Computational and Graphical Statistics · doi:10.1080/10618600.1998.10474784 · Cited by 1,096 · see citing works
ArrayExpress--a public repository for microarray gene expression data at the EBI
Parkinson, H. (2004) · Nucleic Acids Research · doi:10.1093/nar/gki056 · Cited by 1,075 · see citing works
A Constrained ℓ 1 Minimization Approach to Sparse Precision Matrix Estimation
Cai, T., Liu, W., & Luo, X. (2011) · Journal of the American Statistical Association · doi:10.1198/jasa.2011.tm10155 · Cited by 1,045 · see citing works
Extensions of compressed sensing
Tsaig, Y., & Donoho, D. L. (2005) · Signal Processing · doi:10.1016/j.sigpro.2005.05.029 · Cited by 958 · see citing works
One-step sparse estimates in nonconcave penalized likelihood modelsPreprint
Zou, H., & Li, R. (2008) · Cited by 915 · see citing works
Large Covariance Estimation by Thresholding Principal Orthogonal Complements
B, J. F. A., Liao, Y., & Mincheva, M. (2013) · Journal of the Royal Statistical Society Series B (Statistical Methodology) · doi:10.1111/rssb.12016 · Cited by 913 · see citing works
Covariance regularization by thresholding
Bickel, P. J., & Levina, E. (2008) · The Annals of Statistics · doi:10.1214/08-aos600 · Cited by 895 · see citing works
Breheny, P. J., & Huang, J. (2011) · The Annals of Applied Statistics · doi:10.1214/10-aoas388 · Cited by 838 · see citing works
The Generalized Dynamic Factor Model
Forni, M., Hallin, M., Lippi, M., & Reichlin, L. (2005) · Journal of the American Statistical Association · doi:10.1198/016214504000002050 · Cited by 825 · see citing works
Optimization Transfer Using Surrogate Objective Functions
Lange, K. L., Hunter, D. R., & Yang, I. (2000) · Journal of Computational and Graphical Statistics · doi:10.1080/10618600.2000.10474858 · Cited by 809 · see citing works
CUR matrix decompositions for improved data analysis
Mahoney, M. W., & Drineas, P. (2009) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0803205106 · Cited by 799 · see citing works
ArrayExpress--a public repository for microarray gene expression data at the EBI
Brāzma, A. (2003) · Nucleic Acids Research · doi:10.1093/nar/gkg091 · Cited by 776 · see citing works
Feature Screening via Distance Correlation Learning
Li, R., Zhong, W., & Zhu, L. (2012) · Journal of the American Statistical Association · doi:10.1080/01621459.2012.695654 · Cited by 752 · see citing works
Coordinate descent algorithms for lasso penalized regressionPreprint
Wu, T. T., & Lange, K. L. (2008) · Cited by 740 · see citing works
Database-friendly random projections
Achlioptas, D. (2001) · ACM SIGACT-SIGMOD-SIGART Symposium on Principles of Database Systems (PODS) · doi:10.1145/375551.375608 · Cited by 712 · see citing works
Worthey, E., Mayer, A. N., Syverson, G., Helbling, D., Bonacci, B. B., Decker, B., Serpe, J. M., Dasu, T., Tschannen, M., Veith, R., et al. (2011) · Genetics in Medicine · doi:10.1097/gim.0b013e3182088158 · Cited by 704 · see citing works
High dimensional covariance matrix estimation using a factor model
B, J. F. A., Fan, Y., & Lv, J. (2008) · Journal of Econometrics · doi:10.1016/j.jeconom.2008.09.017 · Cited by 688 · see citing works
One-step sparse estimates in nonconcave penalized likelihood models
Zou, H., & Li, R. (2008) · The Annals of Statistics · doi:10.1214/009053607000000802 · Cited by 684 · see citing works
Risk bounds for model selection via penalization
Barron, A. R., Birgé, L., & Massart, P. (1999) · Probability Theory and Related Fields · doi:10.1007/s004400050210 · Cited by 674 · see citing works
Noisy matrix decomposition via convex relaxation: Optimal rates in high dimensions
Agarwal, A., Negahban, S., & Wainwright, M. J. (2012) · The Annals of Statistics · doi:10.1214/12-aos1000 · Cited by 669 · see citing works
Extensions of Lipschitz mappings into a Hilbert space
Johnson, W. B. (1984) · Medical Entomology and Zoology · Cited by 661 · see citing works
Sure independence screening in generalized linear models with NP-dimensionality
Fan, J., & Song, R. (2010) · The Annals of Statistics · doi:10.1214/10-aos798 · Cited by 644 · see citing works
Adaptive Thresholding for Sparse Covariance Matrix Estimation
Cai, T., & Liu, W. (2011) · Journal of the American Statistical Association · doi:10.1198/jasa.2011.tm10560 · Cited by 601 · see citing works
Coordinate descent algorithms for lasso penalized regression
Wu, T. T., & Lange, K. (2008) · The Annals of Applied Statistics · doi:10.1214/07-aoas147 · Cited by 563 · see citing works
Nonparametric Independence Screening in Sparse Ultra-High-Dimensional Additive Models
Fan, J., Feng, Y., & Song, R. (2011) · Journal of the American Statistical Association · doi:10.1198/jasa.2011.tm09779 · Cited by 547 · see citing works
Factor modeling for high-dimensional time series: Inference for the number of factors
Lam, C., & Yao, Q. (2012) · The Annals of Statistics · doi:10.1214/12-aos970 · Cited by 546 · see citing works
Computational Intelligence: Imitating Life
Marks, R. J., Żurada, J. M., & Robinson, C. J. (1994) · IEEE Press eBooks · Cited by 536 · see citing works
High-dimensional regression with noisy and missing data: Provable guarantees with nonconvexity
Loh, P.-L., & Wainwright, M. J. (2012) · The Annals of Statistics · doi:10.1214/12-aos1018 · Cited by 529 · see citing works
The case for cloud computing in genome informatics
Stein, L. (2010) · Genome biology · doi:10.1186/gb-2010-11-5-207 · Cited by 522 · see citing works
Twitter Catches The Flu: Detecting Influenza Epidemics using Twitter
Aramaki, E., Maskawa, S., & Morita, M. (2011) · Cited by 515 · see citing works
High-dimensional semiparametric Gaussian copula graphical models
Liu, H., Han, F., Yuan, M., Lafferty, J., & Wasserman, L. (2012) · The Annals of Statistics · doi:10.1214/12-aos1037 · Cited by 509 · see citing works
Self-citationRegularization of Wavelet Approximations
Antoniadis, A., & B, J. F. A. (2001) · Journal of the American Statistical Association · doi:10.1198/016214501753208942 · Cited by 498 · see citing works
High-dimensional classification using features annealed independence rules
Fan, J., & Fan, Y. (2008) · The Annals of Statistics · doi:10.1214/07-aos504 · Cited by 489 · see citing works
Self-citationUltrahigh dimensional feature selection: beyond the linear model.
Fan, J., Samworth, R. J., & Wu, Y. (2009) · PubMed · Cited by 422 · see citing works
Discoidin domain receptor tyrosine kinases: new players in cancer progression
Valiathan, R. R., Marco, M., Leitinger, B., Kleer, C. G., & Fridman, R. A. (2012) · Cancer and Metastasis Reviews · doi:10.1007/s10555-012-9346-z · Cited by 384 · see citing works
Invited articles section foreword
Vianu, V. (2010) · Journal of the ACM · doi:10.1145/1870103.1870104 · Cited by 382 · see citing works
Variable selection using MM algorithmsPreprint
Hunter, D. R., & Li, R. (2005) · Cited by 375 · see citing works
On the Class of Elliptical Distributions and their Applications to the Theory of Portfolio Choice
Owen, J. S., & Rabinovitch, R. (1983) · The Journal of Finance · doi:10.1111/j.1540-6261.1983.tb02499.x · Cited by 374 · see citing works
Risk Management: Value at Risk and Beyond
Dempster, M. A. H. (2002) · Cited by 344 · see citing works
Variable selection using MM algorithms
Hunter, D. R., & Li, R. (2005) · The Annals of Statistics · doi:10.1214/009053605000000200 · Cited by 339 · see citing works
Robust rank correlation based screening
Li, G., Peng, H., Zhang, J., & Zhu, L. (2012) · The Annals of Statistics · doi:10.1214/12-aos1024 · Cited by 318 · see citing works
A Data-Adaptive Sum Test for Disease Association with Multiple Common or Rare Variants
Han, F., & Pan, W. (2010) · Human Heredity · doi:10.1159/000288704 · Cited by 306 · see citing works
A Direct Estimation Approach to Sparse Linear Discriminant Analysis
Cai, T., & Liu, W. (2011) · Journal of the American Statistical Association · doi:10.1198/jasa.2011.tm11199 · Cited by 268 · see citing works
Variance Estimation Using Refitted Cross-Validation in Ultrahigh Dimensional Regression
Fan, J., Guo, S., & Hao, N. (2011) · Journal of the Royal Statistical Society Series B (Statistical Methodology) · doi:10.1111/j.1467-9868.2011.01005.x · Cited by 264 · see citing works
Self-citationRegularized rank-based estimation of high-dimensional nonparanormal graphical models
Xue, L., & Zou, H. (2012) · The Annals of Statistics · doi:10.1214/12-aos1041 · Cited by 256 · see citing works
Variable Selection in Finite Mixture of Regression Models
Khalili, A., & Chen, J. (2007) · Journal of the American Statistical Association · doi:10.1198/016214507000000590 · Cited by 241 · see citing works
Principled sure independence screening for Cox models with ultra-high-dimensional covariates
Zhao, S. D., & Li, Y. (2011) · Journal of Multivariate Analysis · doi:10.1016/j.jmva.2011.08.002 · Cited by 239 · see citing works
Wavelets in statistics: A review
Antoniadis, A. (1997) · Statistical Methods & Applications · doi:10.1007/bf03178905 · Cited by 227 · see citing works
Estimating False Discovery Proportion Under Arbitrary Covariance Dependence
Fan, J., Han, X., & Gu, W. (2012) · Journal of the American Statistical Association · doi:10.1080/01621459.2012.720478 · Cited by 227 · see citing works
Ultrahigh Dimensional Feature Selection: Beyond The Linear Model
FanJianqing, SamworthRichard, & WuYichao (2009) · Journal of Machine Learning Research · doi:10.5555/1577069.1755853 · Cited by 216 · see citing works
Parallel Coordinate Descent for L1-Regularized Loss MinimizationPreprint
Bradley, J. K., Kyrola, A., Bickson, D., & Guestrin, C. (2011) · arXiv (Cornell University) · doi:10.48550/arxiv.1105.5379 · Cited by 204 · see citing works
Fast global convergence of gradient methods for high-dimensional statistical recovery
Agarwal, A., Negahban, S. N., & Wainwright, M. J. (2012) · The Annals of Statistics · doi:10.1214/12-aos1032 · Cited by 204 · see citing works
ℓ1-penalization for mixture regression models
Städler, N., Bühlmann, P., & van de Geer, S. (2010) · Test · doi:10.1007/s11749-010-0197-z · Cited by 201 · see citing works
ℓ1-Penalization for Mixture Regression ModelsPreprint
van de Geer, S. (2016) · Cited by 186 · see citing works
Using Generalized Correlation to Effect Variable Selection in Very High Dimensional Problems
Hall, P., & Miller, H. (2009) · Journal of Computational and Graphical Statistics · doi:10.1198/jcgs.2009.08041 · Cited by 181 · see citing works
Regularized M-estimators with nonconvexity: Statistical and algorithmic theory for local optimaPreprint
Loh, P., & Wainwright, M. J. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1305.2436 · Cited by 169 · see citing works
High‐Dimensional Covariance Estimation
Pourahmadi, M. (2013) · Wiley series in probability and statistics · doi:10.1002/9781118573617 · Cited by 163 · see citing works
Large Vector Auto RegressionsPreprint
Song, S., & Bickel, P. J. (2011) · RePEc: Research Papers in Economics · doi:10.18452/4336 · Cited by 144 · see citing works
Correlatedz-Values and the Accuracy of Large-Scale Statistical Estimates
Efron, B. (2010) · Journal of the American Statistical Association · doi:10.1198/jasa.2010.tm09129 · Cited by 133 · see citing works
High-dimensional instrumental variables regression and confidence setsPreprint
Gautier, É., & Rose, C. D. (2011) · arXiv (Cornell University) · doi:10.48550/arxiv.1105.2454 · Cited by 126 · see citing works
Oracle inequalities for the lasso in the Cox model
Huang, J., Sun, T., Ying, Z., Yu, Y., & Zhang, C.-H. (2013) · The Annals of Statistics · doi:10.1214/13-aos1098 · Cited by 107 · see citing works
A comparison of the lasso and marginal regression
Genovese, C. R., Jin, J., Wasserman, L., & Yao, Z. (2012) · doi:10.5555/2188385.2343712 · Cited by 105 · see citing works
Parallel Coordinate Descent for L1-Regularized Loss Minimization
Kyrola, A., Bickson, D., Guestrin, C., & Bradley, J. K. (2011) · International Conference on Machine Learning · Cited by 99 · see citing works
Optimal computational and statistical rates of convergence for sparse nonconvex learning problems
Wang, Z., Liu, H., & Zhang, T. (2014) · The Annals of Statistics · doi:10.1214/14-aos1238 · Cited by 88 · see citing works
Self-citationGradient Directed Regularization for Linear Regression and Classi…cation
FriedmanBogdan, J. H., & Popescu, E. (2004) · Cited by 88 · see citing works
Phase transition in limiting distributions of coherence of high-dimensional random matrices
Cai, T., & Jiang, T. (2011) · Journal of Multivariate Analysis · doi:10.1016/j.jmva.2011.11.008 · Cited by 76 · see citing works
Transelliptical Graphical Models
Liu, H., Han, F., & Zhang, C. (2012) · Cited by 69 · see citing works
Self-citationHall, P., Pittelkow, Y., & Ghosh, M. (2008) · Journal of the Royal Statistical Society Series B (Statistical Methodology) · doi:10.1111/j.1467-9868.2007.00631.x · Cited by 62 · see citing works
Empirical null and false discovery rate analysis in neuroimaging
Schwartzman, A., Dougherty, R. F., Lee, J., Ghahremani, D. G., & Taylor, J. (2008) · NeuroImage · doi:10.1016/j.neuroimage.2008.04.182 · Cited by 60 · see citing works
In Search of Non-Gaussian Components of a High-Dimensional DistributionPreprint
Blanchard, G., Kawanabe, M., Sugiyama, M., Spokoiny, V. G., & Müller, K. (2006) · RePEc: Research Papers in Economics · doi:10.18452/3970 · Cited by 60 · see citing works
Scale-Invariant Sparse PCA on High-Dimensional Meta-Elliptical Data
Han, F., & Liu, H. (2013) · Journal of the American Statistical Association · doi:10.1080/01621459.2013.844699 · Cited by 58 · see citing works
Self-citation(2002) · Cambridge University Press eBooks · doi:10.1017/cbo9780511615337 · Cited by 58 · see citing works
Large Vector Auto RegressionsPreprint
Song, S., & Bickel, P. J. (2011) · arXiv (Cornell University) · doi:10.48550/arxiv.1106.3915 · Cited by 55 · see citing works
Detecting outliers in high-dimensional neuroimaging datasets with robust covariance estimators
Fritsch, V., Varoquaux, G. P., Thyreau, B., Poline, J., & Thirion, B. (2012) · Medical Image Analysis · doi:10.1016/j.media.2012.05.002 · Cited by 53 · see citing works
Transition Matrix Estimation in High Dimensional Time Series
Han, F., & Liu, H. (2013) · Cited by 47 · see citing works
Self-citationCovariate assisted screening and estimation
Ke, Z. T., Jin, J., & Fan, J. (2014) · The Annals of Statistics · doi:10.1214/14-aos1243 · Cited by 43 · see citing works
Self-citationAn overview of recent developments in genomics and associated statistical methods
Bickel, P. J., Brown, J. B., Huang, H., & Li, Q. (2009) · Philosophical Transactions of the Royal Society A Mathematical Physical and Engineering Sciences · doi:10.1098/rsta.2009.0164 · Cited by 38 · see citing works
Posterior consistency of nonparametric conditional moment restricted models
Liao, Y., & Jiang, W. (2011) · The Annals of Statistics · doi:10.1214/11-aos930 · Cited by 30 · see citing works
Features of Big Data and sparsest solution in high confidence set
Fan, J. (2014) · Chapman & Hall/CRC eBooks · doi:10.1201/b16720-50 · Cited by 20 · see citing works
Self-citationWhat has Functional Neuroimaging told us about the Mind? So Many Examples, So Little Space
Jonides, J. J., Nee, D. E., & Berman, M. G. (2006) · Cortex · doi:10.1016/s0010-9452(08)70372-1 · Cited by 18 · see citing works
Principal Component Analysis on non-Gaussian Dependent Data
Han, F., & Liu, H. (2013) · Cited by 14 · see citing works
Self-citationControl of the False Discovery Rate Under Arbitrary Covariance DependencePreprint
Han, X., Gu, W., & Fan, J. (2010) · arXiv (Cornell University) · doi:10.48550/arxiv.1012.4397 · Cited by 14 · see citing works
Self-citationEndogeneity in Ultrahigh DimensionPreprint
Fan, J., & Liao, Y. (2012) · SSRN Electronic Journal · doi:10.2139/ssrn.2045864 · Cited by 13 · see citing works
Self-citationTranselliptical Component Analysis
Han, F., & Liu, H. (2012) · neural information processing systems · Cited by 12 · see citing works
Self-citationWould the field of cognitive neuroscience be advanced by sharing functional MRI data?
Visscher, K., & Weissman, D. H. (2011) · BMC Medicine · doi:10.1186/1741-7015-9-34 · Cited by 12 · see citing works
Factor Modeling for High Dimensional Time Series
Lam, C., Yao, Q., & Bathia, N. (2011) · Contributions to statistics · doi:10.1007/978-3-7908-2736-1_31 · Cited by 11 · see citing works
An overview of recent developments in genomics and associated statistical methods
Eter, B. Y. P., Ickel, J. B., Ames, J., Rown, B. B., & Uang, H. A. H. (2015) · Cited by 4 · see citing works
Hallin, M. (2013) · Data Archiving and Networked Services (DANS) · Cited by 3 · see citing works
Flagged first, then by citations
Research works printed but not linked: 3
- [2]Donoho David High-dimensional data analysis: The curses and blessings of dimensionality. the American Mathematical Society Conference Los Angeles, CA, United States August 7–12, 2000
- [11]Fan Jianqing Lv Jinchi Sure independence screening for ultrahigh dimensional feature space (with disussion). J. Roy. Statist. Soc. Ser. B 2008 70 5 849 911 PMC2709408 19603084
- [48]Cochrane John Asset Pricing 2001 Princeton University Press
Cited by
Sorted by citations
Beyond the hype: Big data concepts, methods, and analytics
Gandomi, A., & Haider, M. (2015) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2014.10.007 · Cited by 4,226 · see citing works
Artificial Intelligence and Management: The Automation–Augmentation Paradox
Raisch, S., & Krakowski, S. (2021) · Academy of Management Review · doi:10.5465/amr.2018.0072 · Cited by 2,755 · see citing works
Majorization-Minimization Algorithms in Signal Processing, Communications, and Machine Learning
Sun, Y., Babu, P., & Palomar, D. P. (2017) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2016.2601299 · Cited by 1,820 · see citing works
Wang, G., Gunasekaran, A., Ngai, E. W. T., & Papadopoulos, T. (2016) · International Journal of Production Economics · doi:10.1016/j.ijpe.2016.03.014 · Cited by 1,545 · see citing works
Global ecosystem thresholds driven by aridity
Berdugo, M., Delgado‐Baquerizo, M., Soliveres, S., Hernández‐Clemente, R., Zhao, Y., Gaitán, J., Gross, N., Sáiz, H., Maire, V., Lehmann, A., et al. (2020) · Science · doi:10.1126/science.aay5958 · Cited by 1,264 · see citing works
On big data, artificial intelligence and smart cities
Allam, Z., & Dhunny, Z. A. (2019) · Cities · doi:10.1016/j.cities.2019.01.032 · Cited by 1,178 · see citing works
A survey on missing data in machine learning
Emmanuel, T., Maupong, T., Mpoeleng, D., Semong, T., Mphago, B., & Tabona, O. (2021) · Journal of Big Data · doi:10.1186/s40537-021-00516-9 · Cited by 1,067 · see citing works
Machine learning with big data: challenges and approaches.
L'Heureux, A., Grolinger, K., Elyamany, H. F., & Capretz, M. A. M. (2017) · IEEE Access · doi:10.1109/access.2017.2696365 · Cited by 1,023 · see citing works
Big Data for Remote Sensing: Challenges and Opportunities
Chi, M., Plaza, A., Benediktsson, J. A., Sun, Z., Shen, J., & Zhu, Y. (2016) · Proceedings of the IEEE · doi:10.1109/jproc.2016.2598228 · Cited by 610 · see citing works
Big data analytics in supply chain management between 2010 and 2016: Insights to industries
Tiwari, S., Wee, H. M., & Daryanto, Y. (2018) · Computers & Industrial Engineering · doi:10.1016/j.cie.2017.11.017 · Cited by 582 · see citing works
Artificial intelligence and big data in entrepreneurship: a new era has begun
Obschonka, M., & Audretsch, D. B. (2019) · Small Business Economics · doi:10.1007/s11187-019-00202-4 · Cited by 586 · see citing works
Real-time big data processing for anomaly detection: A survey
Ariyaluran Habeeb, R. A., Nasaruddin, F., Gani, A., Targio Hashem, I. A., Ahmed, E., & Imran, M. (2019) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2018.08.006 · Cited by 487 · see citing works
Business intelligence and big data in hospitality and tourism: a systematic literature review
Mariani, M. M., Baggio, R., Fuchs, M., & Höpken, W. (2018) · International Journal of Contemporary Hospitality Management · doi:10.1108/ijchm-07-2017-0461 · Cited by 441 · see citing works
Biomarkers for depression: recent insights, current challenges and future prospects
Strawbridge, R., Young, A. H., & Cleare, A. J. (2017) · Neuropsychiatric Disease and Treatment · doi:10.2147/ndt.s114542 · Cited by 415 · see citing works
From hype to reality: data science enabling personalized medicine
Fröhlich, H., Balling, R., Beerenwinkel, N., Kohlbacher, O., Kumar, S., Lengauer, T., Maathuis, M. H., Moreau, Y., Murphy, S. A., Przytycka, T. M., et al. (2018) · BMC Medicine · doi:10.1186/s12916-018-1122-7 · Cited by 413 · see citing works
Mavragani, A., Ochoa, G., & Tsagarakis, K. P. (2018) · Journal of Medical Internet Research · doi:10.2196/jmir.9366 · Cited by 390 · see citing works
Big Data: Deep Learning for financial sentiment analysis
Sohangir, S., Wang, D., Pomeranets, A., & Khoshgoftaar, T. M. (2018) · Journal of Big Data · doi:10.1186/s40537-017-0111-6 · Cited by 390 · see citing works
Data sharing practices and data availability upon request differ across scientific disciplines
Tedersoo, L., Küngas, R., Oras, E., Köster, K., Eenmaa, H., Leijen, Ä., Pedaste, M., Raju, M., Сергеевна, А. А., Lukner, H., et al. (2021) · Scientific Data · doi:10.1038/s41597-021-00981-0 · Cited by 397 · see citing works
An overview of the estimation of large covariance and precision matrices
Fan, J., Liao, Y., & Liu, H. (2016) · Econometrics Journal · doi:10.1111/ectj.12061 · Cited by 390 · see citing works
Self-citationGenuer, R., Poggi, J., Tuleau-Malot, C., & Vialaneix, N. (2017) · Big Data Research · doi:10.1016/j.bdr.2017.07.003 · Cited by 385 · see citing works
Artificial intelligence in Internet of things
Ghosh, A. K., Chakraborty, D., & Law, A. (2018) · CAAI Transactions on Intelligence Technology · doi:10.1049/trit.2018.1008 · Cited by 370 · see citing works
Next-Generation Big Data Analytics: State of the Art, Challenges, and Future Research Topics
Lv, Z., Song, H. H., Val, P. B., Steed, A. J., & Jo, M. (2017) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2017.2650204 · Cited by 366 · see citing works
The impact of big data on world-class sustainable manufacturing
Dubey, R., Gunasekaran, A., Childe, S. J., Wamba, S. F., & Παπαδόπουλος, Θ. (2015) · The International Journal of Advanced Manufacturing Technology · doi:10.1007/s00170-015-7674-1 · Cited by 366 · see citing works
The Australian Geoscience Data Cube — Foundations and lessons learned
Lewis, A., Oliver, S., Lymburner, L., Evans, B., Wyborn, L., Mueller, N., Raevksi, G., Hooke, J., Woodcock, R., Sixsmith, J., et al. (2017) · Remote Sensing of Environment · doi:10.1016/j.rse.2017.03.015 · Cited by 353 · see citing works
Emerging Technologies to Conserve Biodiversity
Pimm, S. L., Alibhai, S. K., Bergl, R. A., Dehgan, A. O., Giri, C. P., Jewell, Z. C., Joppa, L., Kays, R., & Loarie, S. R. (2015) · Trends in Ecology & Evolution · doi:10.1016/j.tree.2015.08.008 · Cited by 355 · see citing works
Big Data sources and methods for social and economic analyses
Blazquez-Soriano, A., & Domènech, J. (2017) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2017.07.027 · Cited by 344 · see citing works
Big Data in Agriculture: A Challenge for the Future
Coble, K. H., Mishra, A. K., Ferrell, S., & Griffin, T. (2017) · Applied Economic Perspectives and Policy · doi:10.1093/aepp/ppx056 · Cited by 326 · see citing works
Innovations in Genomics and Big Data Analytics for Personalized Medicine and Health Care: A Review
Hassan, M., Awan, F. M., Naz, A., deAndrés-Galiana, E. J., Alvarez, O., Cernea, A., Fernández-Brillet, L., Fernández-Martínez, J. L., & Kloczkowski, A. (2022) · International Journal of Molecular Sciences · doi:10.3390/ijms23094645 · Cited by 317 · see citing works
“Although big data analysis promises great advantages and a potential solution to a diverse range of problems, there remain many unique technical, computational, and statistical challenges that must be addressed to fully explore its potential [103].”
BackgroundInfluentialScaling Up Scientific Discovery in Sleep Medicine: The National Sleep Research Resource
Dean, D. A., Goldberger, A. L., Mueller, R., Kim, M. J., Rueschman, M., Mobley, D. R., Sahoo, S. S., Jayapandian, C. P., Cui, L., Morrical, M. G., et al. (2016) · SLEEP · doi:10.5665/sleep.5774 · Cited by 311 · see citing works
A comprehensive survey on regularization strategies in machine learning
Tian, Y., & Zhang, Y. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.11.005 · Cited by 313 · see citing works
SICE: an improved missing data imputation technique
Khan, S. I., & Hoque, A. S. M. L. (2020) · Journal Of Big Data · doi:10.1186/s40537-020-00313-w · Cited by 282 · see citing works
Insect pest monitoring with camera-equipped traps: strengths and limitations
Preti, M., Verheggen, F. J., & Angeli, S. (2020) · Journal of Pest Science · doi:10.1007/s10340-020-01309-4 · Cited by 282 · see citing works
Impact of big data & predictive analytics capability on supply chain sustainability
Jeble, S., Dubey, R., Childe, S., Papadopoulos, T., Roubaud, D., & Prakash, A. K. (2018) · The International Journal of Logistics Management · doi:10.1108/ijlm-05-2017-0134 · Cited by 217 · see citing works
Trends and Future Perspective Challenges in Big Data
Naeem, M., Jamal, T., Díaz-Martínez, J., Butt, S. A., Montesano, N., Tariq, M. I., De-La-Hoz-Franco, E., & Valdiris, E. M. D. L. H. (2021) · Smart innovation, systems and technologies · doi:10.1007/978-981-16-5036-9_30 · Cited by 272 · see citing works
Big Data in Earth system science and progress towards a digital twin
Li, X., Feng, M., Ran, Y., Su, Y., Liu, F., Huang, C., Shen, H., Xiao, Q., Su, J., Yuan, S., et al. (2023) · Nature Reviews Earth & Environment · doi:10.1038/s43017-023-00409-w · Cited by 269 · see citing works
Data-Driven Decision Making in Precision Agriculture: The Rise of Big Data in Agricultural Systems
Tantalaki, N., Souravlas, S. I., & Roumeliotis, Μ. (2019) · Journal of Agricultural & Food Information · doi:10.1080/10496505.2019.1638264 · Cited by 258 · see citing works
The Role of the Gut Microbiome in Cattle Production and Health: Driver or Passenger?
O’Hara, E., Neves, A. L. A., Song, Y., & Guan, L. L. (2020) · Annual Review of Animal Biosciences · doi:10.1146/annurev-animal-021419-083952 · Cited by 251 · see citing works
A Systematic Review of Synthetic Data Generation Techniques Using Generative AI
Goyal, M., & Mahmoud, Q. H. (2024) · Electronics · doi:10.3390/electronics13173509 · Cited by 244 · see citing works
Recent Advances of Volatile Memristors: Devices, Mechanisms, and Applications
Wang, R., Yang, J., Mao, J., Wang, Z., Wu, S., Zhou, M., Chen, T., Zhou, Y., & Han, S. (2020) · Advanced Intelligent Systems · doi:10.1002/aisy.202000055 · Cited by 241 · see citing works
Deep learning for time series forecasting: a survey
Kong, X., Chen, Z., Liu, W., Ning, K., Zhang, L., Marier, S. M., Liu, Y., Chen, Y., & Xia, F. (2025) · International Journal of Machine Learning and Cybernetics · doi:10.1007/s13042-025-02560-w · Cited by 279 · see citing works
Distributed testing and estimation under sparse high dimensional models
Battey, H., B, J. F. A., Liu, H., Lu, J., & Zhu, Z. (2018) · The Annals of Statistics · doi:10.1214/17-aos1587 · Cited by 223 · see citing works
Self-citationBig data and data science: a critical review of issues for educational research.
Daniel, B. K. (2019) · British Journal of Educational Technology · doi:10.1111/bjet.12595 · Cited by 220 · see citing works
Big Data Analytics: A Review on Theoretical Contributions and Tools Used in Literature
Grover, P., & Kar, A. K. (2017) · Global Journal of Flexible Systems Management · doi:10.1007/s40171-017-0159-3 · Cited by 219 · see citing works
Triguero, I., García-Gil, D., Maillo, J., Luengo, J., García, S., & Herrera, F. (2018) · WIREs Data Mining Knowl. Discov. · doi:10.1002/widm.1289 · Cited by 190 · see citing works
Copy-move forgery detection: Survey, challenges and future directions
Warif, N. B. A., Wahab, A. W. A., Idris, M. Y. I., Ramli, R., Salleh, R., Band, S. S., & Choo, K. R. (2016) · Journal of Network and Computer Applications · doi:10.1016/j.jnca.2016.09.008 · Cited by 211 · see citing works
Big data, machine learning, and digital twin assisted additive manufacturing: A review
Jin, L., Zhai, X., Wang, K., Zhang, K., Wu, D., Nazir, A., Jiang, J., & Liao, W. (2024) · Materials & Design · doi:10.1016/j.matdes.2024.113086 · Cited by 220 · see citing works
Big Data Reduction Methods: A Survey
Rehman, M. H. U., Liew, C. S., Abbas, A., Jayaraman, P. P., Wah, T. Y., & Khan, S. U. (2016) · Data Science and Engineering · doi:10.1007/s41019-016-0022-0 · Cited by 208 · see citing works
Vital Directions for Health and Health Care
Dzau, V. J., McClellan, M., McGinnis, J. M., Burke, S. P., Coye, M. J., Díaz, Á., Daschle, T. A., Frist, W. H., Gaines, M. E., Hamburg, M. A., et al. (2017) · JAMA · doi:10.1001/jama.2017.1964 · Cited by 205 · see citing works
Academic social networks: Modeling, analysis, mining and applications
Kong, X., Shi, Y., Yu, S., Liu, J., & Xia, F. (2019) · Journal of Network and Computer Applications · doi:10.1016/j.jnca.2019.01.029 · Cited by 207 · see citing works
Retailing and retailing research in the age of big data analytics
Dekimpe, M. G. (2020) · International Journal of Research in Marketing · doi:10.1016/j.ijresmar.2019.09.001 · Cited by 202 · see citing works
Energy consumption forecasting based on Elman neural networks with evolutive optimization
Ruiz, L. G. B., Rueda, R., Cuéllar, M. P., & Pegalajar, M. C. (2017) · Expert Systems with Applications · doi:10.1016/j.eswa.2017.09.059 · Cited by 198 · see citing works
Pouyanfar, S., Yang, Y., Chen, S., Shyu, M., & Iyengar, S. S. (2018) · ACM Computing Surveys · doi:10.1145/3150226 · Cited by 195 · see citing works
Big Data Analytics of Identifying Geochemical Anomalies Supported by Machine Learning Methods
Zuo, R., & Xiong, Y. (2017) · Natural Resources Research · doi:10.1007/s11053-017-9357-0 · Cited by 188 · see citing works
Wen, F., Chu, L., Liu, P., & Qiu, R. C. (2018) · IEEE Access · doi:10.1109/access.2018.2880454 · Cited by 188 · see citing works
Information leverage in interconnected ecosystems: Overcoming the curse of dimensionality
Ye, H., & Sugihara, G. (2016) · Science · doi:10.1126/science.aag0863 · Cited by 180 · see citing works
Perceived strategic value-based adoption of Big Data Analytics in emerging economy
Verma, S., & Bhattacharyya, S. S. (2017) · Journal of Enterprise Information Management · doi:10.1108/jeim-10-2015-0099 · Cited by 187 · see citing works
A Big Data-as-a-Service Framework: State-of-the-Art and Perspectives
Wang, X., Yang, L. T., Liu, H., & Deen, M. J. (2017) · IEEE Transactions on Big Data · doi:10.1109/tbdata.2017.2757942 · Cited by 178 · see citing works
Self-citationMachine-learning-based patient-specific prediction models for knee osteoarthritis
Jamshidi, A., Pelletier, J.-P., & Martel-Pelletier, J. (2019) · Nature Reviews Rheumatology · doi:10.1038/s41584-018-0130-5 · Cited by 190 · see citing works
Japec, L., Kreuter, F., Berg, M., Biemer, P. P., Decker, P. T., Lampe, C. A. C., Lane, J. I., O’Neil, C., & Usher, A. (2015) · Public Opinion Quarterly · doi:10.1093/poq/nfv039 · Cited by 175 · see citing works
Enabling Smart Data: Noise filtering in Big Data classification
García‐Gil, D., Luengo, J., García, S., & Herrera, F. (2018) · Information Sciences · doi:10.1016/j.ins.2018.12.002 · Cited by 176 · see citing works
Munawar, H. S., Qayyum, S., Ullah, F., & Sepasgozar, S. M. E. (2020) · Big Data and Cognitive Computing · doi:10.3390/bdcc4020004 · Cited by 171 · see citing works
Data Integration Challenges for Machine Learning in Precision Medicine
Martínez-García, M., & Hernández-Lemus, E. (2022) · Frontiers in Medicine · doi:10.3389/fmed.2021.784455 · Cited by 158 · see citing works
How Digital Information Transforms Project Delivery Models
Whyte, J. K. (2019) · Project Management Journal · doi:10.1177/8756972818823304 · Cited by 168 · see citing works
Opening up Smart Cities: Citizen-Centric Challenges and Opportunities from GIScience
Degbelo, A., Granell, C., Trilles, S., Bhattacharya, D., Casteleyn, S., & Kray, C. (2016) · ISPRS International Journal of Geo-Information · doi:10.3390/ijgi5020016 · Cited by 166 · see citing works
Singh, M., Kumar, A., Khanna, N. N., Laird, J. R., Nicolaides, A. N., Faa, G., Johri, A. M., Mantella, L. E., Fernandes, J. F. E., Teji, J. S., et al. (2024) · EClinicalMedicine · doi:10.1016/j.eclinm.2024.102660 · Cited by 186 · see citing works
An open source approach to the design and implementation of Digital Twins for Smart Manufacturing
Damjanović, V., & Behrendt, W. (2019) · International Journal of Computer Integrated Manufacturing · doi:10.1080/0951192x.2019.1599436 · Cited by 165 · see citing works
Optimal subsampling for quantile regression in big data
Wang, H., & Yanyuan Ma, B. (2020) · Biometrika · doi:10.1093/biomet/asaa043 · Cited by 162 · see citing works
Challenges for MapReduce in Big Data
Grolinger, K., Hayes, M., Higashino, W. A., L’Heureux, A., Allison, D. S., & Capretz, M. A. M. (2014) · IEEE World Congress on Services (SERVICES) · doi:10.1109/services.2014.41 · Cited by 160 · see citing works
A Highly Efficient Semismooth Newton Augmented Lagrangian Method for Solving Lasso Problems
Li, X., Sun, D., & Toh, K.-C. (2018) · SIAM Journal on Optimization · doi:10.1137/16m1097572 · Cited by 157 · see citing works
Silva, B. N., Khan, M., Jung, C., Seo, J., Diyan, M., Han, J., Yoon, Y., & Han, K. (2018) · Sensors · doi:10.3390/s18092994 · Cited by 155 · see citing works
Dinov, I. D. (2016) · GigaScience · doi:10.1186/s13742-016-0117-6 · Cited by 153 · see citing works
Going Digital: A Survey on Digitalization and Large-Scale Data Analytics in Healthcare
Tresp, V., Overhage, J. M., Bundschus, M., Rabizadeh, S., Fasching, P. A., & Yu, S. (2016) · Proceedings of the IEEE · doi:10.1109/jproc.2016.2615052 · Cited by 154 · see citing works
Contextual anomaly detection framework for big sensor data
Hayes, M. A., & Capretz, M. A. (2015) · Journal of Big Data · doi:10.1186/s40537-014-0011-y · Cited by 149 · see citing works
Liu, C., Feng, Y., Lin, D., Wu, L., & Guo, M. (2020) · International Journal of Production Research · doi:10.1080/00207543.2019.1677961 · Cited by 146 · see citing works
Design principles for creating digital transparency in government
Matheus, R., Janssen, M., & Janowski, T. (2020) · Government Information Quarterly · doi:10.1016/j.giq.2020.101550 · Cited by 146 · see citing works
Stoll, A., & Benner, P. (2021) · GAMM-Mitteilungen · doi:10.1002/gamm.202100003 · Cited by 147 · see citing works
Navigating the Local Modes of Big Data: The Case of Topic Models
Roberts, M. E., Stewart, B., & Tingley, D. H. (2016) · Computational Social Science · doi:10.1017/cbo9781316257340.004 · Cited by 140 · see citing works
Machine Learning Approaches in Cardiovascular Imaging
Henglin, M., Stein, G., Hushcha, P. V., Snoek, J., Wiltschko, A. B., & Cheng, S. C. (2017) · Circulation Cardiovascular Imaging · doi:10.1161/circimaging.117.005614 · Cited by 142 · see citing works
Statistical methods and computing for big data
Wang, C., Chen, M., Schifano, E., Wu, J., & Yan, J. (2016) · Statistics and Its Interface · doi:10.4310/sii.2016.v9.n4.a1 · Cited by 138 · see citing works
CASEarth Poles: Big Data for the Three Poles
Li, X., Che, T., Li, X., Wang, L., Duan, A., Shangguan, D., Pan, X., Fang, M., & Bao, Q. (2020) · Bulletin of the American Meteorological Society · doi:10.1175/bams-d-19-0280.1 · Cited by 132 · see citing works
Nguyen, B. H., Xue, B., Andreae, P. M., Ishibuchi, H., & Zhang, M. (2019) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2019.2913831 · Cited by 131 · see citing works
Predicting Tourist Demand Using Big Data
Song, H., & Liu, H. (2016) · Tourism on the verge · doi:10.1007/978-3-319-44263-1_2 · Cited by 129 · see citing works
Self-citationBig Data-enabled Customer Relationship Management: A holistic approach
Zerbino, P., Aloini, D., Dulmin, R., & Mininno, V. (2018) · Information Processing & Management · doi:10.1016/j.ipm.2017.10.005 · Cited by 112 · see citing works
Jena, R. K. (2020) · Industrial Marketing Management · doi:10.1016/j.indmarman.2019.12.012 · Cited by 126 · see citing works
From inverse optimal control to inverse reinforcement learning: A historical review
Azar, N. A., Shahmansoorian, A., & Davoudi, M. (2020) · Annual Reviews in Control · doi:10.1016/j.arcontrol.2020.06.001 · Cited by 127 · see citing works
Masoud, M. Z., Jaradat, Y. M., Manasrah, A. A., & Jannoud, I. (2019) · Journal of Sensors · doi:10.1155/2019/6514520 · Cited by 125 · see citing works
Challenges and Benefits of Deploying Big Data Analytics in the Cloud for Business Intelligence
Balachandran, B. M., & Prasad, S. T. (2017) · Procedia Computer Science · doi:10.1016/j.procs.2017.08.138 · Cited by 123 · see citing works
Blending Big Data Analytics: Review on Challenges and a Recent Study
Amalina, F., Hashem, M., Azizul, Z. H., Ang, T. F., Firdaus, A., Imran, M. A., & Anuar, N. B. (2019) · IEEE Access · doi:10.1109/access.2019.2923270 · Cited by 127 · see citing works
Large data sets and machine learning: Applications to statistical arbitrage
Huck, N. (2019) · European Journal of Operational Research · doi:10.1016/j.ejor.2019.04.013 · Cited by 118 · see citing works
Individual-Based Tracking Systems in Ornithology: Welcome to the Era of Big Data
López‐López, P. (2016) · Ardeola · doi:10.13157/arla.63.1.2016.rp5 · Cited by 117 · see citing works
Randomly distributed embedding making short-term high-dimensional data predictable
Ma, H., Leng, S., Aihara, K., Lin, W. J., & Chen, L. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1802987115 · Cited by 103 · see citing works
Zhang, X. (2020) · Springer eBooks · doi:10.1007/978-981-15-2770-8_6 · Cited by 107 · see citing works
Bioprospecting of Novel Extremozymes From Prokaryotes—The Advent of Culture-Independent Methods
Sysoev, M., Grötzinger, S. W., Renn, D., Eppinger, J., Rueping, M., & Karan, R. (2021) · Frontiers in Microbiology · doi:10.3389/fmicb.2021.630013 · Cited by 103 · see citing works
A selective overview of feature screening for ultrahigh-dimensional data
Liu, J., Zhong, W., & Li, R. (2015) · Science China Mathematics · doi:10.1007/s11425-015-5062-9 · Cited by 102 · see citing works
Ngo, J., Hwang, B., & Zhang, C. (2019) · Automation in Construction · doi:10.1016/j.autcon.2019.103042 · Cited by 104 · see citing works
Marcucci, E., Gatta, V., Pira, M. L., Hansson, L., & Bråthen, S. (2020) · Sustainability · doi:10.3390/su122410623 · Cited by 81 · see citing works
The Double-Edged Sword of Big Data in Organizational and Management Research
Wenzel, R., & Quaquebeke, N. V. (2017) · Organizational Research Methods · doi:10.1177/1094428117718627 · Cited by 104 · see citing works
Open Innovation Engineering—Preliminary Study on New Entrance of Technology to Market
Yun, J. J., Kim, D., & Yan, M.-R. (2020) · Electronics · doi:10.3390/electronics9050791 · Cited by 98 · see citing works
Impact of AI-Based Tools and Urban Big Data Analytics on the Design and Planning of Cities
Kamrowska-Załuska, D. (2021) · Land · doi:10.3390/land10111209 · Cited by 95 · see citing works
A semiparametric method for clustering mixed data
Foss, A., Markatou, M., Ray, B. K., & Heching, A. R. (2016) · Machine Learning · doi:10.1007/s10994-016-5575-7 · Cited by 96 · see citing works
- Authors
- Fan, J., Han, F., & Liu, H.
- Affiliations
- Department of Operations Research and Financial Engineering, Princeton University, Princeton, NJ 08544, USA;
- Johns Hopkins University (US)
- Princeton University (US)
- Venue
- National Science Review
- Publisher
- Oxford University Press (OUP)
- Published
- 2014-02-05
- Volume
- 1
- Issue
- 2
- Pages
- 293-314
- ISSN
- 2053-714X (ISSN-L)
- 2095-5138
- PMC
- PMC4236847
- OpenAlex
- W2114060717
- Cited by
- 1,482
- References
- 144
- Citations by year
- 2026: 86 · 2025: 132 · 2024: 142 · 2023: 172 · 2022: 118 · 2021: 122 · 2020: 125 · 2019: 157 · 2018: 162 · 2017: 120 · 2016: 90 · 2015: 40 · 2014: 15 · 2013: 1
- FWCI
- 165.47
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- National Science Review (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Europe PMC (PubMed Central) (acceptedVersion)
- PubMed Central (acceptedVersion)
- arXiv (Cornell University) (submittedVersion)
- HighWire Press Open Archive (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (R01-GM072611, 1206464, 1332109, 1116730)
- National Institutes of Health
- National Institute of Mental Health (R01MH102339)
- National Institute of General Medical Sciences (R01GM072611, R01GM100474)
- Topics
- Social Sciences › Decision Sciences › Information Systems and Management › Big Data Technologies and Applications
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Inference
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Applications
- Keywords
- big data analytics, high-dimensional data, scalability, noise accumulation, spurious correlations, measurement error, statistical inference, computer architecture
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
