The DOI matches its Crossref record; 47 of the 47 references it cites were found in a registry or an index.
Big Data: New Tricks for Econometrics
Open accessAbstract
Computers are now involved in many economic transactions and can capture data associated with these transactions, which can then be manipulated and analyzed. Conventional statistical and econometric techniques such as regression often work well, but there are issues unique to big datasets that may require different tools. First, the sheer size of the data involved may require more powerful data manipulation tools. Second, we may have more potential predictors than appropriate for estimation, so we need to do some kind of variable selection. Third, large datasets may allow for more flexible relationships than simple linear models. Machine learning techniques such as decision trees, support vector machines, neural nets, deep learning, and so on may allow for more effective ways to model complex relationships. In this essay, I will describe a few of these tools for manipulating and analyzing big data. I believe that these methods have a lot to offer and should be more widely known and used by economists.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Elements of Statistical Learning
Hastie, T., Tibshirani, R., & Friedman, J. (2009) · Springer Series in Statistics · doi:10.1007/978-0-387-84858-7 · Cited by 25,970
Modern Applied Statistics with S
Venables, W. N., & Ripley, B. D. (2002) · Statistics and Computing · doi:10.1007/978-0-387-21706-2 · Cited by 24,305
Classification and Regression Trees.
Gordon, A. D., Breiman, L., Friedman, J. H., Olshen, R. A., & Stone, C. J. (1984) · Biometrics · doi:10.2307/2530946 · Cited by 23,522
An Introduction to Statistical Learning
James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013) · Springer Texts in Statistics · doi:10.1007/978-1-4614-7138-7 · Cited by 12,106
Causality: Models, Reasoning, and Inference
Pearl, J. (2009) · doi:10.1017/cbo9780511803161 · Cited by 11,653
(2019) · AEA Papers and Proceedings · doi:10.1257/pandp.109.612 · Cited by 10,902
Mostly Harmless Econometrics: An Empiricist's Companion
Angrist, J. D., & Pischke, J.-S. (2009) · doi:10.1515/9781400829828 · Cited by 9,467
Estimating causal effects of treatments in randomized and nonrandomized studies
Rubin, D. B. (1974) · Journal of Educational Psychology · doi:10.1037/h0037350 · Cited by 9,436
(2012) · Cited by 9,062
Friedman, J. H. (2002) · Computational Statistics & Data Analysis · doi:10.1016/s0167-9473(01)00065-2 · Cited by 7,067
Mostly Harmless Econometrics: An Empiricist's Companion
Angrist, J. D., & Pischke, J.-S. (2008) · doi:10.2307/j.ctvcm4j72 · Cited by 5,051
Unbiased Recursive Partitioning: A Conditional Inference Framework
Hothorn, T., Hornik, K., & Zeileis, A. (2006) · Journal of Computational and Graphical Statistics · doi:10.1198/106186006x133933 · Cited by 4,021
Why Are There Still So Many Jobs? The History and Future of Workplace Automation
Autor, D. H. (2015) · Journal of Economic Perspectives · doi:10.1257/jep.29.3.3 · Cited by 3,917 · cited 7×
“Source: The table is based on that in Ley and Steel (2009); the data analyzed is from Sala-i-Martín (1997). Notes: We illustrate different methods of variable selection.”
BackgroundMethodInfluentialA few useful things to know about machine learning
Domingos, P. (2012) · Communications of the ACM · doi:10.1145/2347736.2347755 · Cited by 3,326
An Introduction to Statistical Learning: with Applications in R
James, G. M., Witten, D., Hastie, T. J., & Tibshirani, R. J. (2018) · doi:10.25334/q4ht55 · Cited by 2,912
An empirical comparison of supervised learning algorithms
Caruana, R., & Niculescu-Mizil, A. (2006) · Proceedings of the 23rd international conference on Machine learning - ICML '06 · doi:10.1145/1143844.1143865 · Cited by 2,780
Sivia, D. S. (1996) · Cited by 2,669
Angrist, J. D., & Krueger, A. B. (2001) · Journal of Economic Perspectives · doi:10.1257/jep.15.4.69 · Cited by 2,355
Predicting the Present with Google Trends
CHOI, H., & VARIAN, H. (2012) · Economic Record · doi:10.1111/j.1475-4932.2012.00809.x · Cited by 2,354
Self-citationCausal inference in statistics: An overview
Pearl, J. (2009) · Statistics Surveys · doi:10.1214/09-ss057 · Cited by 2,336
I Just Ran Two Million Regressions
Sala-i-Martín, X. (1997) · American Economic Review · Cited by 2,321
Predictive learning via rule ensembles
Friedman, J. H., & Popescu, B. E. (2008) · The Annals of Applied Statistics · doi:10.1214/07-aoas148 · Cited by 1,315
Problems in the Analysis of Survey Data, and a Proposal
Morgan, J. N., & Sonquist, J. A. (1963) · Journal of the American Statistical Association · doi:10.1080/01621459.1963.10500855 · Cited by 1,305
Harvey, A. C. (1981) · Medical Entomology and Zoology · Cited by 1,144
Mortgage Lending in Boston: Interpreting HMDA Data
Munnell, A. H., Tootell, G. M. B., Browne, L. E., & McEneaney, J. P. (1992) · American Economic Review · Cited by 996
Inferring causal impact using Bayesian structural time-series models
Brodersen, K. H., Gallusser, F., Koehler, J., Remy, N., & Scott, S. L. (2015) · The Annals of Applied Statistics · doi:10.1214/14-aoas788 · Cited by 961
The Top Ten Algorithms in Data Mining
(2009) · Chapman & Hall/CRC eBooks · doi:10.1201/9781420089653 · Cited by 670
Predicting consumer behavior with web search
Goel, S., Hofman, J. M., Lahaie, S., Pennock, D. M., & Watts, D. J. (2010) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1005962107 · Cited by 636
Ley, E., & Steel, M. F. J. (2009) · Journal of Applied Econometrics · doi:10.1002/jae.1057 · Cited by 477
Predicting the Present with Google Trends
Varian, H. R., & Choi, H. (2009) · SSRN Electronic Journal · doi:10.2139/ssrn.1659302 · Cited by 426
Self-citationPredicting the present with Bayesian structural time series
Scott, S. L., & Varian, H. R. (2014) · International Journal of Mathematical Modelling and Numerical Optimisation · doi:10.1504/ijmmno.2014.059942 · Cited by 393
Self-citationEvidence on Discrimination in Mortgage Lending
Ladd, H. F. (1998) · Journal of Economic Perspectives · doi:10.1257/jep.12.2.41 · Cited by 380
Nowcasting with Google Trends in an Emerging Market
Carrière‐Swallow, Y., & Labbé, F. (2013) · Journal of Forecasting · doi:10.1002/for.1252 · Cited by 284
Bäck, T. (1996) · Evolutionary Algorithms in Theory and Practice · doi:10.1093/oso/9780195099713.003.0009 · Cited by 233
The Data Revolution and Economic Analysis
Einav, L., & Levin, J. (2014) · Innovation Policy and the Economy · doi:10.1086/674019 · Cited by 227
Machine Learning Methods for Demand Estimation
Bajari, P., Nekipelov, D., Ryan, S. P., & Yang, M. (2015) · American Economic Review · doi:10.1257/aer.p20151021 · Cited by 220
On bagging and nonlinear estimation
Friedman, J. H., & Hall, P. (2007) · Journal of Statistical Planning and Inference · doi:10.1016/j.jspi.2006.06.002 · Cited by 183
Hendry, D. F., & Krolzig, H. (2004) · Oxford Bulletin of Economics and Statistics · doi:10.1111/j.1468-0084.2004.102_1.x · Cited by 168
Bayesian Variable Selection for Nowcasting Economic Time Series
Scott, S. L., & Varian, H. R. (2013) · National Bureau of Economic Research · doi:10.3386/w19567 · Cited by 140
Self-citationUsing internet search data as economic indicators
McLaren, N., & Shanbhogue, R. (2011) · RePEc: Research Papers in Economics · Cited by 126
International Comparisons of the Sources of Economic Growth
Jorgenson, D. W., & Dougherty, C. (1996) · American Economic Review · Cited by 105
Statistical significance of the Netflix challengePreprint
Feuerverger, A., He, Y. H., & Khatri, S. (2016) · Cited by 77
Statistical Significance of the Netflix Challenge
Feuerverger, A., He, Y., & Khatri, S. (2012) · Statistical Science · doi:10.1214/11-sts368 · Cited by 68
Larose, C. D., & Larose, D. T. (2019) · Wiley eBooks · doi:10.1002/9781119526865.ch1 · Cited by 46
Tracking the Future on the Web: Construction of Leading Indicators Using Internet SearchesPreprint
Artola, C., & Lucha, E. G. (2012) · SSRN Electronic Journal · doi:10.2139/ssrn.2027861 · Cited by 44
Bayesian model averaging and forecasting
Steel, M. F. J. (2011) · doi:10.82444/warw.33615439 · Cited by 26
How To Pick The Best Regression Equation: A Review And Comparison Of Model Selection AlgorithmsPreprint
Castle, J. L., Qin, X., & Reed, W. R. (2009) · RePEc: Research Papers in Economics · Cited by 18
Cited by
Biau, G., & Scornet, E. (2016) · Test · doi:10.1007/s11749-016-0481-7 · Cited by 4,149
Why Are There Still So Many Jobs? The History and Future of Workplace Automation
Autor, D. H. (2015) · Journal of Economic Perspectives · doi:10.1257/jep.29.3.3 · Cited by 3,917
Angrist, J. D., & Krueger, A. B. (2001) · Journal of Economic Perspectives · doi:10.1257/jep.15.4.69 · Cited by 2,355
Machine Learning: An Applied Econometric Approach
Mullainathan, S., & Spiess, J. (2017) · Journal of Economic Perspectives · doi:10.1257/jep.31.2.87 · Cited by 1,989
Athey, S., Tibshirani, J., & Wager, S. (2019) · The Annals of Statistics · doi:10.1214/18-aos1709 · Cited by 1,870
Metadata
- Authors
- Varian, H. R.
- Venue
- Journal of Economic Perspectives
- Publisher
- American Economic Association
- Published
- 2014-05-01
- Volume
- 28
- Issue
- 2
- Pages
- 3-28
- ISSN
- 0895-3309 (ISSN-L)
- OpenAlex
- W2155419203
- Cited by
- 1,557
- References
- 47
- FWCI
- 184.85
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.