The DOI matches its Crossref record; 144 of the 144 references it cites were found in a registry or an index.
Challenges of Big Data analysis
Open accessAbstract
Big Data bring new opportunities to modern society and challenges to data scientists. On the one hand, Big Data hold great promises for discovering subtle population patterns and heterogeneities that are not possible with small-scale data. On the other hand, the massive sample size and high dimensionality of Big Data introduce unique computational and statistical challenges, including scalability and storage bottleneck, noise accumulation, spurious correlation, incidental endogeneity and measurement errors. These challenges are distinguished and require new computational and statistical paradigm. This paper gives overviews on the salient features of Big Data and how these features impact on paradigm change on statistical and computational methods as well as computing architectures. We also provide various new perspectives on the Big Data analysis and computation. In particular, we emphasize on the viability of the sparsest solution in high-confidence set and point out that exogenous assumptions in most statistical methods for Big Data cannot be validated due to incidental endogeneity. They can lead to wrong statistical inferences and consequently wrong scientific conclusions.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Overfitting is a key risk in this context, which means that a machine may learn a complete model that also explains the errors, consequently failing to generalize appropriately beyond its training data (Fan, Han, & Liu, 2014).”
From Artificial Intelligence and Management: The Automation–Augmentation Paradox (2021) · cites this paper as Fan, Han, & Liu, 2014
Experimental — accuracy not yet measuredIn the citing paper
“In addition, the combination of datasets in the big data age needs to cope with a number of challenges listed in [20,61], for example efficient data representation, redundancy reduction and data compression, spurious correlations, and noise accumulation.”
From Opening up Smart Cities: Citizen-Centric Challenges and Opportunities from GIScience (2016) · cites this paper as [61]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing
Benjamini, Y., & Hochberg, Y. (1995) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1995.tb02031.x · Cited by 109,482
Boyd, S., & Vandenberghe, L. (2004) · doi:10.1017/cbo9780511804441 · Cited by 56,601
A new look at the statistical model identification
Akaike, H. (1974) · IEEE Transactions on Automatic Control · doi:10.1109/tac.1974.1100705 · Cited by 51,879
Regression shrinkage and selection via the lasso
Tibshirani, R. (1996) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1996.tb02080.x · Cited by 51,482
Golub, G. (2012) · doi:10.56021/9781421407944 · Cited by 34,505
The Elements of Statistical Learning
Hastie, T., Tibshirani, R., & Friedman, J. (2009) · Springer Series in Statistics · doi:10.1007/978-0-387-84858-7 · Cited by 25,970
The Elements of Statistical Learning
Hastie, T., Friedman, J., & Tibshirani, R. (2001) · Springer Series in Statistics · doi:10.1007/978-0-387-21606-5 · Cited by 24,162
Donoho, D. L. (2006) · IEEE Transactions on Information Theory · doi:10.1109/tit.2006.871582 · Cited by 23,994
Longitudinal data analysis using generalized linear models
LIANG, K.-Y., & ZEGER, S. L. (1986) · Biometrika · doi:10.1093/biomet/73.1.13 · Cited by 18,550
Boyd, S., Parikh, N., Chu, E., Peleato, B., & Eckstein, J. (2011) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000016 · Cited by 17,075
Flagged first, then by citations
Research works printed but not linked: 6
- Donoho, High-dimensional data analysis: the curses and blessings of dimensionality
- Bickel, 2008, Discussion on the paper ‘Sure independence screening for ultrahigh dimensional feature space’ by Fan and Lv, J Roy Stat Soc B, 70, 883
- Friedman, 2003, Gradient directed regularization for linear regression and classification, Technical report
- Milham, 2011, The International Neuroimaging Data-sharing Initiative (INDI) and the Functional Connectomes Project, 17th Annual Meeting of the Organization for Human Brain Mapping
- 2012, The ADHD-200 Consortium. The ADHD-200 consortium: a model to advance the translational potential of neuroimaging in clinical neuroscience, Front Syst Neurosci, 6, 62
- Cochrane, 2001, Asset Pricing
Cited by
Sorted by citations
Beyond the hype: Big data concepts, methods, and analytics
Gandomi, A., & Haider, M. (2015) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2014.10.007 · Cited by 4,226
Artificial Intelligence and Management: The Automation–Augmentation Paradox
Raisch, S., & Krakowski, S. (2021) · Academy of Management Review · doi:10.5465/amr.2018.0072 · Cited by 2,755
Majorization-Minimization Algorithms in Signal Processing, Communications, and Machine Learning
Sun, Y., Babu, P., & Palomar, D. P. (2017) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2016.2601299 · Cited by 1,820
Wang, G., Gunasekaran, A., Ngai, E. W. T., & Papadopoulos, T. (2016) · International Journal of Production Economics · doi:10.1016/j.ijpe.2016.03.014 · Cited by 1,545
Global ecosystem thresholds driven by aridity
Berdugo, M., Delgado‐Baquerizo, M., Soliveres, S., Hernández‐Clemente, R., Zhao, Y., Gaitán, J., Gross, N., Sáiz, H., Maire, V., Lehmann, A., et al. (2020) · Science · doi:10.1126/science.aay5958 · Cited by 1,265
Metadata
- Authors
- Fan, J., Han, F., & Liu, H.
- Affiliations
- Department of Operations Research and Financial Engineering, Princeton University, Princeton, NJ 08544, USA;
- Johns Hopkins University (US)
- Princeton University (US)
- Venue
- National Science Review
- Publisher
- Oxford University Press (OUP)
- Published
- 2014-02-05
- Volume
- 1
- Issue
- 2
- Pages
- 293-314
- ISSN
- 2053-714X (ISSN-L)
- 2095-5138
- PMC
- PMC4236847
- OpenAlex
- W2114060717
- Cited by
- 1,482
- References
- 144
- Citations by year
- 2026: 86 · 2025: 132 · 2024: 142 · 2023: 172 · 2022: 118 · 2021: 122 · 2020: 125 · 2019: 157 · 2018: 162 · 2017: 120 · 2016: 90 · 2015: 40 · 2014: 15
- FWCI
- 165.47
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- National Science Review (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Europe PMC (PubMed Central) (acceptedVersion)
- PubMed Central (acceptedVersion)
- arXiv (Cornell University) (submittedVersion)
- HighWire Press Open Archive (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (R01-GM072611, 1206464, 1332109, 1116730)
- National Institutes of Health
- National Institute of Mental Health (R01MH102339)
- National Institute of General Medical Sciences (R01GM072611, R01GM100474)
- Topics
- Social Sciences › Decision Sciences › Information Systems and Management › Big Data Technologies and Applications
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Inference
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Applications
- Keywords
- big data analytics, high-dimensional data, scalability, noise accumulation, spurious correlations, measurement error, statistical inference, computer architecture
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
