The Parable of Google Flu: Traps in Big Data Analysis
Open accessThe DOI matches its Crossref record; 31 of the 31 references it cites were found in a registry or an index; 5 of the 8 sources its claims cite were checked for support.
Abstract
Large errors in flu prediction were largely avoidable, which offers lessons for the use of big data.
Overview
CoChat Summary
TL;DR
The paper uses the failure of Google Flu Trends (GFT) to illustrate two traps in big data analysis: “big data hubris” and algorithm dynamics. It argues that GFT’s large flu-prediction errors were largely avoidable and that big data should supplement, not replace, traditional data and methods.
Key findings
- In February 2013, GFT predicted more than double the proportion of doctor visits for influenza-like illness (ILI) than the CDC; from 21 August 2011 to 1 September 2013, GFT reported overly high flu prevalence in 100 out of 108 weeks, and it overshot the 2011–2012 season by more than 50%.
- GFT errors were not random: they showed temporal autocorrelation and seasonality, meaning last week’s errors predicted this week’s errors and error direction/magnitude varied with time of year.
- Simple alternatives outperformed GFT: out-of-sample mean absolute error was 0.486 for GFT, 0.311 for lagged CDC, and 0.232 for combined GFT and CDC; all differences were statistically significant at P < 0.05. Even 3-week-old CDC data did better than GFT at projecting current flu prevalence.
- The authors attribute GFT’s problems to overfitting (initial GFT matched 50 million search terms to 1152 data points and missed the nonseasonal 2009 H1N1 pandemic) and to “algorithm dynamics,” including Google search algorithm changes and user behavior changes.
Methods
- Design: policy forum/case study comparing GFT estimates with CDC ILI surveillance data and alternative forecasting models.
- Data/sample: GFT and CDC flu prevalence estimates, including the period from 21 August 2011 to 1 September 2013; the initial GFT methodology reportedly fit 50 million search terms to 1152 data points.
- Main analysis: comparison of GFT alone, lagged CDC data with 52-week seasonality variables, and a combined GFT + CDC model using lagged CDC estimates, lagged GFT error, and seasonality variables; reported mean absolute error and statistical significance; Google Correlate used to examine correlated search terms. Supplementary materials contain data and methods.
Limitations
- The paper states that GFT never documented the 45 search terms used, released examples appear misleading, and Google Correlate is limited to national-level data whereas GFT was developed using regional correlations; therefore the original GFT analyses cannot be replicated from the information provided.
- The paper states that Google cannot make its full data available to outsiders because of privacy issues, though it argues there is no such constraint on derivative, aggregated data.
- Evident from the text: this is a commentary/policy forum rather than a full empirical article, with key data and methods placed in supplementary materials; the quantitative comparison depends on GFT and CDC data and model specifications not fully detailed in the main text.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
2 Valid · 2 Not valid · 2 Not checked · 8 Own results and interpretations
Some cited sources support this claim and some don’t — read each source’s verdict below.
In the paper
“This ad hoc method of throwing out peculiar search terms failed when GFT completely missed the nonseasonal 2009 infl uenza A–H1N1 pandemic ( 2, 14).”
What the sources say
- Partly supportedcites [2] sourceExperimental — accuracy not yet measured
The Discussion (page unavailable) says the original GFT model missed the spring 2009 pandemic wave, supporting the claim about its failure to detect the nonseasonal outbreak. It does not say GFT completely missed the entire 2009 pandemic. The Introduction describes automated selection of the original model’s search terms, not an ad hoc method of throwing out peculiar terms. (LLM, conf=0.86)
- Disputedcites [14] S. Cook et al., PLOS ONE 6, e23610 (2011). sourceExperimental — accuracy not yet measured
The Abstract (Methodology/Principal Findings) says the original GFT model underestimated the magnitude of illness during the 2009 H1N1 pandemic, not that it completely missed it. The Abstract (Conclusions) says both models performed well during the pandemic, although the updated model performed better. The Discussion notes that pandemic-specific queries were excluded, but does not describe this as an ad hoc method of throwing out peculiar terms. (LLM, conf=0.88)
Checked by a reasoning model — may be wrong.
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“Oddly, the few search terms offered in the papers ( 14) do not seem to be strongly related with either GFT or the CDC data (SM)—we surmise that the authors felt an unarticulated need to cloak the actual search terms identifi ed.”
What the sources say
- Disputedcites [14] S. Cook et al., PLOS ONE 6, e23610 (2011). sourceExperimental — accuracy not yet measured
The Results section (‘Search Behavior During pH1N1’; page unavailable) says that all three search queries shown in Figure 3 closely tracked CDC ILINet data before the pandemic, and ‘symptoms of flu’ continued to do so during it. The paper does report that the bronchitis and pneumonia queries underestimated ILINet data during the pandemic, but that does not support the broad claim that the offered terms were not strongly related to the data. (LLM, conf=0.87)
Checked by a reasoning model — may be wrong.
hedgedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“The Internet has opened the way for improving standard surveys, experiments, and health reporting ( 35).”
What the sources say
- Partly supportedcites [35] A. J. Berinsky et al., Polit. Anal. 20, 351–368 (2012). sourceExperimental — accuracy not yet measured
Section 2 describes using an Internet platform to run surveys with embedded experiments, and the Conclusion says MTurk can make experimental research easier and less expensive. The paper does not address improvements to health reporting; the disease scenario in section 4.3.2 is a framing experiment, not health reporting. Page numbers are unavailable in our copy. (LLM, conf=0.88)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Although not widely reported until 2013, the new GFT has been persistently overestimating flu prevalence for a much longer time.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“From 21 August 2011 to 1 September 2013, GFT reported overly high fl u prevalence 100 out of 108 weeks.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
3 more claims
Create a free account to see all 14 claims and every source verdict.
- It may state two separate claims as one. 5
- Probably not a substantial claim. 2
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Google Trends predicted more than twice the number of influenza-like illnesses than the CDC in February 2013 46 .”
From Unrepresentative big surveys significantly overestimated US vaccine uptake (2021) · cites this paper as [46]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This was one of the main reasons behind the failure of Google Flu Trends, since the data used for training the model in the first few years was not representative of the trends in subsequent years [15].”
From Theory-Guided Data Science: A New Paradigm for Scientific Discovery from Data (2017) · cites this paper as [15]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Social data research often relies on counting entities. These entities can be users, links, or messages, and the description is a summary of these counts (Lazer et al., 2014; Salganik, 2017). Yet, simple counts can mislead if it is unclear what is counted and how.”
From Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019) · cites this paper as Lazer et al., 2014
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This unintelligibility is also subject to functional biases (section 4.1), as it depends on the mechanisms available on each social platform (e.g., having a like button, but not a dislike one), as well as to variations in platform algorithms and mechanisms in response to users actions (Lazer et al., 2014).”
From Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019) · cites this paper as Lazer et al., 2014
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“It also includes temporal validity, which captures to what extent constructs change over time (Howison et al., 2011) and may invalidate previous conclusions about societal and/or platform phenomena; e.g., see the case of Google Flu Trends (Lazer et al., 2014).”
From Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019) · cites this paper as Lazer et al., 2014
6 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 14 MARCH 2014 VOL 343 SCIENCE www.sciencemag.org 1204
“The 2009 version of GFT has weathered other media panics related to the fl u, including the 2005–2006 influenza A/H5N1 ("bird flu") outbreak”
“GFT engineers updated the algorithm in 2009, and this model has run ever since”
Citations
Cites
boyd, D., & Crawford, K. (2012) · Information, Communication & Society · doi:10.1080/1369118x.2012.678878 · Cited by 5,479
Twitter mood predicts the stock market
Bollen, J., Mao, H., & Zeng, X. (2011) · Journal of Computational Science · doi:10.1016/j.jocs.2010.12.007 · Cited by 5,206
Detecting influenza epidemics using search engine query data
Ginsberg, J., Mohebbi, M. H., Patel, R. S., Brammer, L., Smolinski, M. S., & Brilliant, L. (2009) · Nature · doi:10.1038/nature07634 · Cited by 4,295
Cohen, D. (1998) · The Women's Review of Books · doi:10.2307/4022859 · Cited by 4,261
Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk
Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012) · Political Analysis · doi:10.1093/pan/mpr057 · Cited by 4,008
Big Data : The Management Revolution
マカフィー, ア., ブリニョルフソン, エ., & 有賀, 裕. (2013) · Harvard business review · Cited by 3,549
Lazer, D., Pentland, A., Adamic, L., Aral, S., Barabási, A.-L., Brewer, D., Christakis, N., Contractor, N., Fowler, J., Gutmann, M., et al. (2009) · Science · doi:10.1126/science.1167742 · Cited by 3,365
Self-citationPredicting Elections with Twitter: What 140 Characters Reveal about Political Sentiment
Tumasjan, A., Sprenger, T., Sandner, P., & Welpe, I. (2010) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v4i1.14009 · Cited by 2,649
Multiscale mobility networks and the spatial spreading of infectious diseases
Balcan, D., Colizza, V., Gonçalves, B., Hu, H., Ramasco, J. J., & Vespignani, A. (2009) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0906910106 · Cited by 1,380
Self-citationPredicting consumer behavior with web search
Goel, S., Hofman, J. M., Lahaie, S., Pennock, D. M., & Watts, D. J. (2010) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1005962107 · Cited by 636
Research works printed but not linked: 4
- McAfee A., Brynjolfsson E., Harv. Bus. Rev. 90, 60 (2012).
- Copeland P., et al.., Int. Soc. Negl. Trop. Dis. 2013, 3 (2013).
- Mustafaraj E., Metaxas P., in Proceedings of the Web-Sci10, Raleigh, NC, 26 and 27 April 2010 (Web Science Trust, 2010); http://journal.webscience.org/317/.
- Ratkiewicz J., et al.., in Proceedings of 5th International AAAI Conference on Weblogs and Social Media, San Francisco, CA, 7 to 11 August 2011 (AAAI, 2011), p. 297–304.
Cited by
The ethics of algorithms: Mapping the debate
Mittelstadt, B. D., Allo, P., Taddeo, M., Wachter, S., & Floridi, L. (2016) · Big Data & Society · doi:10.1177/2053951716679679 · Cited by 4,567
“The search for causal links is difficult, as correlations established in large, proprietary datasets are frequently not reproducible or falsifiable (cf. Ioannidis, 2005; Lazer et al., 2014).”
Ayers, J. W., Poliak, A., Dredze, M., Leas, E. C., Zhu, Z., Kelley, J. B., Faix, D. J., Goodman, A. M., Longhurst, C. A., Hogarth, M., et al. (2023) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2023.1838 · Cited by 2,613
Human-Centered Artificial Intelligence: Reliable, Safe & Trustworthy
Shneiderman, B. (2020) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2020.1741118 · Cited by 1,743
Theory-Guided Data Science: A New Paradigm for Scientific Discovery from Data
Karpatne, A., Atluri, G., Faghmous, J. H., Steinbach, M., Banerjee, A., Ganguly, A., Shekhar, S., Samatova, N., & Kumar, V. (2017) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2017.2720168 · Cited by 1,752
“two, as measured by the number of influenza-related doctor visits in subsequent years, according to CDC data [15].”
MethodMachine learning in medicine: a practical introduction
Sidey-Gibbons, J. A. M., & Sidey-Gibbons, C. J. (2019) · BMC Medical Research Methodology · doi:10.1186/s12874-019-0681-4 · Cited by 1,271
“The oft-told parable of the failure of the Google Flu Trends model offers an accessible example of the risks and consequences posed by a lack of understanding of ML models deployed ostensibly to improve health [34].”
BackgroundGentzkow, M., Kelly, B., & Taddy, M. (2019) · Journal of Economic Literature · doi:10.1257/jel.20181020 · Cited by 1,236
“Lazer et al. (2014), for example, show that the accuracy of the Google Flu Trends model—which has not been recalibrated or updated based on more recent data—has deteriorated dramatically, and that in recent years it is outperformed by simple extrapolation from prior CDC estimates.”
MethodDo no harm: a roadmap for responsible machine learning for health care
Wiens, J., Saria, S., Sendak, M., Ghassemi, M., Liu, V. X., Doshi-Velez, F., Jung, K., Heller, K., Kale, D., Saeed, M., et al. (2019) · Nature Medicine · doi:10.1038/s41591-019-0548-6 · Cited by 1,266
Acquisti, A., Taylor, C., & Wagman, L. (2016) · Journal of Economic Literature · doi:10.1257/jel.54.2.442 · Cited by 1,283
Machine Learning and Prediction in Medicine — Beyond the Peak of Inflated Expectations
Chen, J. H., & Asch, S. M. (2017) · New England Journal of Medicine · doi:10.1056/nejmp1702071 · Cited by 1,171
If...Then: Algorithmic Power and Politics
Bucher, T. (2018) · doi:10.1093/oso/9780190493028.001.0001 · Cited by 1,223
The use of google trends in health care research: a systematic review
Nuti, S. V., Wayda, B., Ranasinghe, I., Wang, S., Dreyer, R. P., Chen, S. I., & Murugiah, K. (2014) · PLoS ONE · doi:10.1371/journal.pone.0109583 · Cited by 1,086
Luo, W., Phung, D., Tran, T., Gupta, S., Rana, S., Karmakar, C., Shilton, A., Yearwood, J., Dimitrova, N., Ho, T. B., et al. (2016) · Journal of Medical Internet Research · doi:10.2196/jmir.5870 · Cited by 988
Big Data computing and clouds: trends and future directions
Assunção, M. D., Calheiros, R. N., Bianchi, S., Netto, M. A. S., & Buyya, R. (2015) · Journal of Parallel and Distributed Computing · doi:10.1016/j.jpdc.2014.08.003 · Cited by 901
“In business models where high-level analytics services may be delivered by the Cloud, human expertise cannot be easily replaced bymachine learning and Big Data analysis [99]; in certain scenarios, theremay be a need for human analysts to remain in the loop [91].”
BackgroundStatistical physics of vaccination
Wang, Z., Bauch, C. T., Bhattacharyya, S., d'Onofrio, A., Manfredi, P., Perc, M., Perra, N., Salathé, M., & Zhao, D. (2016) · Physics Reports · doi:10.1016/j.physrep.2016.10.006 · Cited by 872
“However, the tool was not able, despite being updated, to predict the unfolding of the pandemic in 2009 [751, 752].”
MethodDeep Learning: A Critical Appraisal
Marcus, G. (2018) · arXiv · doi:10.48550/arxiv.1801.00631 · Cited by 853
Surveillance, Snowden, and Big Data: Capacities, consequences, critique
Lyon, D. (2014) · Big Data & Society · doi:10.1177/2053951714541861 · Cited by 839
“(Ginsberg et al., 2009; Lazer et al., 2014) – along with the capacity to analyze vast quantities of data at steadily shrinking unit costs.”
BackgroundPopulation flow drives spatio-temporal distribution of COVID-19 in China
Jia, J. S., Lu, X., Yuan, Y., Xu, G., Jia, J., & Christakis, N. A. (2020) · Nature · doi:10.1038/s41586-020-2284-y · Cited by 835
Torous, J., Kiang, M. V., Lorme, J., & Onnela, J.-P. (2016) · JMIR Mental Health · doi:10.2196/mental.5165 · Cited by 758
Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries
Olteanu, A., Castillo, C., Diaz, F., & Kiciman, E. (2016) · doi:10.2139/ssrn.2886526 · Cited by 763
Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries
Olteanu, A., Castillo, C., Diaz, F., & Kıcıman, E. (2019) · Frontiers in Big Data · doi:10.3389/fdata.2019.00013 · Cited by 763
“It also includes temporal validity , which captures to what extent constructs change over time (Howison et al., 2011) and may invalidate previous conclusions about societal and/or platform phenomena; e.g., see the case of Google Flu Trends (Lazer et al., 2014).”
BackgroundInfluentialAlgorithmic Accountability: Journalistic investigation of computational power structures
Diakopoulos, N. (2015) · Digital Journalism · doi:10.1080/21670811.2014.976411 · Cited by 733
Predicting poverty and wealth from mobile phone metadata
Blumenstock, J., Cadamuro, G., & On, R. (2015) · Science · doi:10.1126/science.aac4420 · Cited by 698
Big data for development: a review of promises and challenges
Hilbert, M. (2016) · Development Policy Review · doi:10.1111/dpr.12142 · Cited by 679
“Returning to the example of Google’s flu-trend (Ginsberg et al. 2009; Lazer et al. 2014), Google processed an impressive 450 million different mathematical models in order to eventually identify 45 search terms that could predict flu outbreaks better than traditional models.”
Andreu-Perez, J., Poon, C. C. Y., Merrifield, R. D., Wong, S. T. C., & Yang, G.-Z. (2015) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2015.2450362 · Cited by 670
“its for influenza-like illness than that of the Centers for Disease Control and Prevention [124].”
BackgroundChallenges and Future Directions of Big Data and Artificial Intelligence in Education
Luan, H., Geczy, P., Lai, H., Gobert, J., Yang, S. J. H., Ogata, H., Baltes, J., Guerra, R., Li, P., & Tsai, C.-C. (2020) · Frontiers in Psychology · doi:10.3389/fpsyg.2020.580820 · Cited by 667
“Hence, novel and innovative computational technologies are required for the acquisition, storage, distribution, analysis, and management of big data (Lazer et al., 2014; Geczy, 2015).”
BackgroundSocial media for large studies of behavior
Ruths, D., & Pfeffer, J. (2014) · Science · doi:10.1126/science.346.6213.1063 · Cited by 648
Jusup, M., Holme, P., Kanazawa, K., Takayasu, M., Romić, I., Wang, Z., Geček, S., Lipić, T., Podobnik, B., Wang, L., et al. (2022) · Physics Reports · doi:10.1016/j.physrep.2021.10.005 · Cited by 643
Rapid assessment of disaster damage using social media activity
Kryvasheyeu, Y., Chen, H., Obradovich, N., Moro, E., Van Hentenryck, P., Fowler, J., & Cebrian, M. (2016) · Science Advances · doi:10.1126/sciadv.1500779 · Cited by 637
“Moreover, an indirect and potentially nonstationary relationship between social media signals and real-world phenomena, compounded by changing social norms in the use of particular online platforms, calls for caution in developing predictive tools based on Big Data analysis (68).”
MethodGovernance by algorithms: reality construction by algorithmic selection on the Internet
Just, N., & Latzer, M. (2017) · Media, Culture & Society · doi:10.1177/0163443716643157 · Cited by 612
Editorial—Big Data, Data Science, and Analytics: The Opportunity and Challenge for IS Research
Agarwal, R., & Dhar, V. (2014) · Information Systems Research · doi:10.1287/isre.2014.0546 · Cited by 610
“One, the scathing review of Google flu trends (Lazer et al. 2014) (that uses search terms to predict the incidence of flu) with respect to its accuracy as compared with estimates produced by the Centers for Disease Control and Prevention, and two, the ethical debate ignited by experiments conducted…”
MethodLum, K., & Isaac, W. (2016) · Significance · doi:10.1111/j.1740-9713.2016.00960.x · Cited by 611
Can deep learning beat numerical weather prediction?
Schultz, M. G., Betancourt, C., Gong, B., Kleinert, F., Langguth, M., Leufen, L. H., Mozaffari, A., & Stadtler, S. (2021) · Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences · doi:10.1098/rsta.2020.0097 · Cited by 562
Formalizing Trust in Artificial Intelligence: Prerequisites, Causes and Goals of Human Trust in AI
Jacovi, A., Marasović, A., Miller, T., & Goldberg, Y. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445923 · Cited by 545
“the original classification task [42, 46].”
Measuring Emotional Contagion in Social Media
Ferrara, E., & Yang, Z. (2015) · PLOS ONE · doi:10.1371/journal.pone.0142390 · Cited by 532
Jun, S.-P., Yoo, H. S., & Choi, S. (2018) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2017.11.009 · Cited by 530
“Even when analyzing information provided by Google Trends, to obtain a clearer understanding of the intention of utilization, we will also need to be analyzed other social big data as mentioned above (Lazer et al., 2014).”
InfluentialInstagram photos reveal predictive markers of depression
Reece, A. G., & Danforth, C. M. (2017) · EPJ Data Science · doi:10.1140/epjds/s13688-017-0110-z · Cited by 519
The Role of Big Data and Predictive Analytics in Retailing
Bradlow, E. T., Gangwar, M., Kopalle, P., & Voleti, S. (2017) · Journal of Retailing · doi:10.1016/j.jretai.2016.12.004 · Cited by 516
“Later a deeper analysis showed that these 'theory-free' projections were overpredicting actual cases by 100%, among other problems that arose (see Lazer et al. 2014 for a review).”
BackgroundPoint-of-Care Diagnostics: Recent Developments in a Connected Age
Nayak, S., Blumenfeld, N. R., Laksanasopin, T., & Sia, S. K. (2017) · Analytical Chemistry · doi:10.1021/acs.analchem.6b04630 · Cited by 508
Economics in the age of big data
Einav, L., & Levin, J. (2014) · Science · doi:10.1126/science.1243089 · Cited by 498
Management challenges in creating value from business analytics
Vidgen, R., Shaw, S., & Grant, D. B. (2017) · European Journal of Operational Research · doi:10.1016/j.ejor.2017.02.023 · Cited by 493
“…for a bottom-up and atheoretical approach to finding patterns in data - an approach that has led to concerns about ‘big data hubris’, which Lazer et al. (2014) say is the often implicit assumption that big data is a substitute for, rather than a supplement to, traditional data collection…”
BackgroundArtificial Intelligence and Personalized Medicine
Schork, N. J. (2019) · Cancer Treatment and Research · doi:10.1007/978-3-030-16391-4_11 · Cited by 488
Physics-Guided Neural Networks (PGNN): An Application in Lake Temperature Modeling
Daw, A., Karpatne, A., Watkins, W. D., Read, J. S., & Kumar, V. (2022) · Knowledge-Guided Machine Learning · doi:10.1201/9781003143376-15 · Cited by 472
Lee, H., Yune, S., Mansouri, M., Kim, M., Tajmir, S. H., Guerrier, C. E., Ebert, S. A., Pomerantz, S. R., Romero, J. M., Kamalian, S., et al. (2018) · Nature Biomedical Engineering · doi:10.1038/s41551-018-0324-9 · Cited by 441
Process‐Guided Deep Learning Predictions of Lake Water Temperature
Read, J. S., Jia, X., Willard, J., Appling, A. P., Zwart, J. A., Oliver, S. K., Karpatne, A., Hansen, G. J. A., Hanson, P. C., Watkins, W., et al. (2019) · Water Resources Research · doi:10.1029/2019wr024922 · Cited by 447
“…underlying the data, known laws or theory are ignored (such as conservation of energy) and this omission can lead to spurious and inaccurate predictions (e.g., Lazer et al., 2014; Nayak et al., 2013), particularly when predictions are made outside the range of data used to train the DL models.”
BackgroundArtificial intelligence in medicine: current trends and future possibilities
Buch, V. H., Ahmed, I., & Maruthappu, M. (2018) · British Journal of General Practice · doi:10.3399/bjgp18x695213 · Cited by 443
Tenkanen, H., Di Minin, E., Heikinheimo, V., Hausmann, A., Herbst, M., Kajala, L., & Toivonen, T. (2017) · Scientific Reports · doi:10.1038/s41598-017-18007-4 · Cited by 439
Business intelligence and big data in hospitality and tourism: a systematic literature review
Mariani, M., Baggio, R., Fuchs, M., & Höepken, W. (2018) · International Journal of Contemporary Hospitality Management · doi:10.1108/ijchm-07-2017-0461 · Cited by 434
Heading toward Artificial Intelligence 2.0
Pan, Y. (2016) · Engineering · doi:10.1016/j.eng.2016.04.018 · Cited by 433
“[20] suggested that the fusion and intelligent analysis of multiple sources of information were the characteristics associated with knowledge and system evolution and were the necessary means of solving “big data hubris.”
BackgroundDiscriminating Data: Correlation, Neighborhoods, and the New Politics of Recognition
Chun, W. H. K., & Barnett, A. (2021) · The MIT Press eBooks · doi:10.7551/mitpress/14050.001.0001 · Cited by 429
Forecasting the onset and course of mental illness with Twitter data
Reece, A. G., Reagan, A. J., Lix, K. L. M., Dodds, P. S., Danforth, C. M., & Langer, E. J. (2017) · Scientific Reports · doi:10.1038/s41598-017-12961-9 · Cited by 401
Combining Search, Social Media, and Traditional Data Sources to Improve Influenza Surveillance
Santillana, M., Nguyen, A. T., Dredze, M., Paul, M. J., Nsoesie, E. O., & Brownstein, J. S. (2015) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1004513 · Cited by 420
Williams, M. L., Burnap, P., & Sloan, L. (2017) · Sociology · doi:10.1177/0038038517708140 · Cited by 423
“High-profile failures of big data, such as the inability to predict the US housing bubble in 2008 and the spread of influenza across the United States using Google search terms, have resulted in the questioning of the power and longevity of algorithms (Lazer et al., 2014).”
BackgroundData Science: A Comprehensive Overview
Cao, L. (2018) · ACM Computing Surveys · doi:10.1145/3076253 · Cited by 407
“It can enable the discovery of indicators like Google Flu [Lazer et al. 2014] which may not be readily predicted by domain-driven hypothesis and professionals.”
BackgroundWhy general artificial intelligence will not be realized
Fjelland, R. (2020) · Humanities and Social Sciences Communications · doi:10.1057/s41599-020-0494-4 · Cited by 399
Accurate estimation of influenza epidemics using Google search data via ARGO
Yang, S., Santillana, M., & Kou, S. C. (2015) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1515373112 · Cited by 373
“…we calculated these accuracy metrics for (a) GFT estimates (accessed on April 25, 2015), (b) estimates produced using the methodologies of Santillana et al. 2014 [6, 17], (c) estimates produced with an AR(3) autoregressive model ([4, 16]), and (d) a naive method that simply uses the…”
Methodde Albuquerque, J. P., Herfort, B., Brenning, A., & Zipf, A. (2015) · International Journal of Geographical Information Science · doi:10.1080/13658816.2014.996567 · Cited by 385
“As pointed out by Lazer et al. (2014), one should not see ‘ big data ’ as a substitute for all existing data, but rather take the challenge of doing innovative analytics by using data from all traditional and new sources.”
BackgroundMatheus, R., Janssen, M., & Maheshwari, D. (2020) · Government Information Quarterly · doi:10.1016/j.giq.2018.01.006 · Cited by 377
Integrating explanation and prediction in computational social science
Hofman, J. M., Watts, D. J., Athey, S., Garip, F., Griffiths, T. L., Kleinberg, J., Margetts, H., Mullainathan, S., Salganik, M. J., Vazire, S., et al. (2021) · Nature · doi:10.1038/s41586-021-03659-0 · Cited by 371
Self-citationArtificial intelligence vs COVID-19: limitations, constraints and pitfalls
Naudé, W. (2020) · AI & SOCIETY · doi:10.1007/s00146-020-00978-0 · Cited by 368
Insights From Advanced Analytics At The Veterans Health Administration
Fihn, S. D., Francis, J., Clancy, C., Nielson, C., Nelson, K., Rumsfeld, J., Cullen, T., Bates, J., & Graham, G. L. (2014) · Health Affairs · doi:10.1377/hlthaff.2014.0054 · Cited by 367
Xu, W., Dainoff, M. J., Ge, L., & Gao, Z. (2023) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2022.2041900 · Cited by 363
“Many AI professionals are primarily dedicated to studying algorithms, rather than providing useful AI systems to meet user needs, resulting in the failure of many AI systems (Hoffman et al., 2016; Lazer et al., 2014; Lieberman, 2009; Yampolskiy, 2019).”
BackgroundNext-Generation Big Data Analytics: State of the Art, Challenges, and Future Research Topics
Lv, Z., Song, H., Basanta-Val, P., Steed, A., & Jo, M. (2017) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2017.2650204 · Cited by 365
“The current research outcomes have indicated that data are no longer just data [57], [67], [68].”
BackgroundSocial Media– and Internet-Based Disease Surveillance for Public Health
Aiello, A. E., Renson, A., & Zivich, P. N. (2020) · Annual Review of Public Health · doi:10.1146/annurev-publhealth-040119-094402 · Cited by 349
“The inaccurate predictions by Google Flu Trends in multiple seasons raised awareness of potential biases inherent in digital surveillance (60), including changes to search algorithms (stability), nonindependence of data sources (posting and searching can be influenced by others), confounding (of search terms), represe…”
BackgroundAdvances and opportunities in machine learning for process data analytics
Qin, S. J., & Chiang, L. H. (2019) · Computers & Chemical Engineering · doi:10.1016/j.compchemeng.2019.04.003 · Cited by 351
Deep Transformer Models for Time Series Forecasting: The Influenza Prevalence Case
Wu, N., Green, B., Ben, X., & O’Banion, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2001.08317 · Cited by 335
“While initially regarded as a great success, GFT suffered from over-estimation of peak ILI magnitude in subsequent years (Olson et al., 2013; Lazer et al., 2014).”
BackgroundMethodBig Data in Public Health: Terminology, Machine Learning, and Privacy
Mooney, S. J., & Pejaver, V. (2018) · Annual Review of Public Health · doi:10.1146/annurev-publhealth-040617-014208 · Cited by 325
“…104) d) participant measurements taken automatically at extremely frequent intervals as by a GPS device or FitBit,(39) or e) measures compiled from the ‘data effluent’ created by life in an electronic world, such as search term records,(67) social media postings,(7) or cell phone records.”
BackgroundTackling climate change with machine learning
Rolnick, D., Donti, P. L., Kaack, L. H., Kochanski, K., Lacoste, A., Sankaran, K., Ross, A. S., Milojevic-Dupont, N., Jaques, N., Waldman-Brown, A., et al. (2023) · ACM Computing Surveys · doi:10.1145/3485128 · Cited by 336
“While non-survey proxies are observational and selfreported, current research attempts to address these issues [567, 568].”
BackgroundBig data in the policy cycle: Policy decision making in the digital era
Höchtl, J., Parycek, P., & Schöllhammer, R. (2016) · Journal of Organizational Computing and Electronic Commerce · doi:10.1080/10919392.2015.1125187 · Cited by 331
“…extreme caution, in the light of the knowledge that social media activity can influence policy decisions and change the behavior of citizens — thereby possibly distorting the actual salience of an issue for the general public and overemphasizing the concerns of a vocal minority (Lazer et al. 2014).”
BackgroundBento, A. I., Nguyen, T., Wing, C., Lozano-Rojas, F., Ahn, Y.-Y., & Simon, K. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2005335117 · Cited by 311
Readers’ perception of computer-generated news: Credibility, expertise, and readability
Graefe, A., Haim, M., Haarmann, B., & Brosius, H.-B. (2018) · Journalism · doi:10.1177/1464884916641269 · Cited by 315
“For more information on the SoSci Panel, see Leiner (2014). A total of 1107 subjects participated in the study in December 2014.”
BackgroundData ex Machina: Introduction to Big Data
Lazer, D., & Radford, J. (2017) · Annual Review of Sociology · doi:10.1146/annurev-soc-060116-053457 · Cited by 303
“It is worth noting that this granularity is most useful when fused with traditional methods rather than used as a replacement for them (Lazer et al. 2014).”
BackgroundMethodInfluentialSelf-citationCrisis informatics—New data for extraordinary times
Palen, L., & Anderson, K. M. (2016) · Science · doi:10.1126/science.aag2579 · Cited by 309
Coevolution spreading in complex networks
Wang, W., Liu, Q.-H., Liang, J., Hu, Y., & Zhou, T. (2019) · Physics Reports · doi:10.1016/j.physrep.2019.07.001 · Cited by 286
“[247] concluded that two issues, big data hubris and the instability of the algorithm, caused GFT’s errors.”
BackgroundUtilizing big data analytics for information systems research: challenges, promises and guidelines
Müller, O., Junglas, I., Brocke, J. V., & Debortoli, S. (2016) · European Journal of Information Systems · doi:10.1057/ejis.2016.2 · Cited by 281
Agency plus automation: Designing artificial intelligence into interactive systems
Heer, J. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1807184115 · Cited by 279
EPM 20/20: A Review, Framework, and Research Agenda for Electronic Performance Monitoring
Ravid, D. M., Tomczak, D. L., White, J. C., & Behrend, T. S. (2020) · Journal of Management · doi:10.1177/0149206319869435 · Cited by 277
“Theory-free algorithms can produce overfitted models that describe relationships among data points but cannot explain causal mechanisms or endure inevitable changes in the individuals or contexts they describe (Lazer et al., 2014).”
BackgroundValue co-creation between firms and customers: The role of big data-based cooperative assets
Xie, K., Wu, Y., Xiao, J., & Hu, Q. (2016) · Information & Management · doi:10.1016/j.im.2016.06.003 · Cited by 273
Forecasting sales in the supply chain: Consumer analytics in the big data era
Boone, T., Ganeshan, R., Jain, A., & Sanders, N. R. (2019) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2018.09.003 · Cited by 271
Towards felicitous decision making: An overview on challenges and trends of Big Data
Wang, H., Xu, Z., Fujita, H., & Liu, S. (2016) · Information Sciences · doi:10.1016/j.ins.2016.07.007 · Cited by 268
“[82] stated that human analysts are necessary to remain in the loop in certain scenarios.”
BackgroundEthical challenges of big data in public health
Vayena, E., Salathé, M., Madoff, L. C., & Brownstein, J. S. (2015) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1003904 · Cited by 268
Unrepresentative big surveys significantly overestimated US vaccine uptake
Bradley, V. C., Kuriwaki, S., Isakov, M., Sejdinovic, D., Meng, X.-L., & Flaxman, S. (2021) · Nature · doi:10.1038/s41586-021-04198-4 · Cited by 248
Transparent to whom? No algorithmic accountability without a critical audience
Kemper, J., & Kolkman, D. (2019) · Information, Communication & Society · doi:10.1080/1369118x.2018.1477967 · Cited by 260
“The seminal example of the former is the Google Flu Trends algorithmic model that inad-vertently predicted more than double the proportion of doctor visits for in fl uenza-like illness (Lazer, Kennedy, King, & Vespignani, 2014).”
MethodBig Data for Infectious Disease Surveillance and Modeling
Bansal, S., Chowell, G., Simonsen, L., Vespignani, A., & Viboud, C. (2016) · Journal of Infectious Diseases · doi:10.1093/infdis/jiw400 · Cited by 255
“big data must stay away from the pitfalls of “big data hubris” [21] and complement novel approaches with rigorous statistical and hypothesis-driven analyses.”
BackgroundSelf-citationA Review of Challenges and Opportunities in Machine Learning for Health
Ghassemi, M., Naumann, T., Schulam, P., Beam, A. L., Chen, I. Y., & Ranganath, R. (2018) · PubMed · doi:10.48550/arxiv.1806.00388 · Cited by 248
“In a notable example of concept drift, Google Flu Trends persistently overestimated flu due to shifts in search behaviors 80 .”
BackgroundAn Investigation of Brand-Related User-Generated Content on Twitter
Liu, X., Burns, A. C., & Hou, Y. (2017) · Journal of Advertising · doi:10.1080/00913367.2017.1297273 · Cited by 243
“As a caution, no matter how “big” it is, big data may give very misleading information (Lazer et al. 2014; Lin, Lucas, and Shmueli 2013).”
BackgroundCitizen science provides a reliable and scalable tool to track disease-carrying mosquitoes
Palmer, J. R. B., Oltra, A., Collantes, F., Delgado, J. A., Lucientes, J., Delacour, S., Bengoa, M., Eritja, R., & Bartumeus, F. (2017) · Nature Communications · doi:10.1038/s41467-017-00914-9 · Cited by 243
Towards Intelligent Regulation of Artificial Intelligence
BUITEN, M. C. (2019) · European Journal of Risk Regulation · doi:10.1017/err.2019.8 · Cited by 237
Temporal correlation detection using computational phase-change memory
Sebastian, A., Tuma, T., Papandreou, N., Le Gallo, M., Kull, L., Parnell, T., & Eleftheriou, E. (2017) · Nature Communications · doi:10.1038/s41467-017-01481-9 · Cited by 241
Twitter Improves Influenza Forecasting
Paul, M. J., Dredze, M., & Broniatowski, D. (2014) · PLoS Currents · doi:10.1371/currents.outbreaks.90b9ed0f59bae4ccaa683a39865d9117 · Cited by 238
Mergel, I., Rethemeyer, R. K., & Isett, K. (2016) · Public Administration Review · doi:10.1111/puar.12625 · Cited by 241
“(Lazer et al. 2014 , 1203) More professionally infuriating can be the instances in which computational approaches get the right answer—but one that social scientists have known about for 5, or 10, or 20 years.”
BackgroundOnline big data-driven oil consumption forecasting with Google trends
Yu, L., Zhao, Y., Tang, L., & Yang, Z. (2019) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2017.11.005 · Cited by 236
Ecological Rationality: Fast-and-Frugal Heuristics for Managerial Decision Making under Uncertainty
Luan, S., Reb, J., & Gigerenzer, G. (2019) · Academy of Management Journal · doi:10.5465/amj.2018.0172 · Cited by 235
Konapala, G., Kao, S.-C., Painter, S. L., & Lu, D. (2020) · Environmental Research Letters · doi:10.1088/1748-9326/aba927 · Cited by 236
“Also, since these ML models are purely data-driven and have no process representation, they can lead to spurious and inaccurate predictions (e.g. Nayak et al 2013, Lazer et al 2014, Read et al 2019), especially when predictions are outside of the range of data used for ML model training.”
BackgroundDiscovering dynamic patterns from infectious disease data using dynamic mode decomposition
Proctor, J. L., & Eckhoff, P. A. (2015) · International Health · doi:10.1093/inthealth/ihv009 · Cited by 230
Rethinking big data: A review on the data quality and usage issues
Liu, J., Li, J., Li, W., & Wu, J. (2016) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2015.11.006 · Cited by 236
Planning and Conducting Experimental Advertising Research and Questionnaire Design
Geuens, M., & De Pelsmacker, P. (2017) · Journal of Advertising · doi:10.1080/00913367.2016.1225233 · Cited by 234
“…such as bias in collection and interpretation (Bottles, Begoli, and Worley 2014); validity, transparency, replicability, and interpretation of big data (Lazer et al. 2014; Boyd and Crawford 2012; Bollier and Firestone 2010); and lack of theoretical considerations (Bottles, Begoli, and Worley 2014).”
BackgroundEnhancing disease surveillance with novel data streams: challenges and opportunities
Althouse, B. M., Scarpino, S. V., Meyers, L. A., Ayers, J. W., Bargsten, M., Baumbach, J., Brownstein, J. S., Castro, L., Clapham, H., Cummings, D. A., et al. (2015) · EPJ Data Science · doi:10.1140/epjds/s13688-015-0054-0 · Cited by 231
“For public health authorities with access to high-resolution data on reported cases of influenza, simple autoregressive models can be used to nowcast with high accuracy [19].”
BackgroundMethodInfluentialSelf-citationForecasting Destination Weekly Hotel Occupancy with Big Data
Pan, B., & Yang, Y. (2017) · Journal of Travel Research · doi:10.1177/0047287516669050 · Cited by 229
“However, many scholars have also cautioned against becoming overly optimistic about the potential of forecasting using big data because quite often the accuracy is not as good as expected (Lazer et al. 2014).”
BackgroundResultA Big Data Guide to Understanding Climate Change: The Case for Theory-Guided Data Science
Faghmous, J. H., & Kumar, V. (2014) · Big Data · doi:10.1089/big.2014.0026 · Cited by 223
Macroeconomic Nowcasting and Forecasting with Big Data
Bok, B., Caratelli, D., Giannone, D., Sbordone, A. M., & Tambalotti, A. (2018) · Annual Review of Economics · doi:10.1146/annurev-economics-080217-053214 · Cited by 224
Metadata
- Authors
- Lazer, D., Kennedy, R., King, G., & Vespignani, A.
- Affiliations
- Northeastern University (US)
- Harvard University (US)
- Corresponding author
- Lazer, David
- Venue
- Science
- Published
- 2014-03-14
- Volume
- 343
- Issue
- 6176
- Pages
- 1203-1205
- ISSN
- 0036-8075 (ISSN-L)
- 1095-9203
- OpenAlex
- W2068181924
- Semantic Scholar
- 00d681cfff7f0224ace607e515f4ed510e792df9
- Cited by
- 2,523
- References
- 30
- Citations by year
- 2026: 113 · 2025: 101 · 2024: 121 · 2023: 150 · 2022: 168 · 2021: 229 · 2020: 273 · 2019: 261 · 2018: 272 · 2017: 249 · 2016: 271 · 2015: 219 · 2014: 91
- FWCI
- 208.16
- Type
- journal article
- Language
- en
- Open access
- green
- dash.harvard.edu
- Licence
- cc-by
- Locations
- Science (publishedVersion)
- PubMed (publishedVersion)
- Digital Access to Scholarship at Harvard (DASH) (Harvard University) (submittedVersion, cc-by)
- Topics
- Health Sciences › Medicine › Epidemiology › Data-Driven Disease Surveillance
- Social Sciences › Decision Sciences › Information Systems and Management › Big Data Technologies and Applications
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- Google Flu Trends, influenza surveillance, big data, prediction error, data quality
- MeSH terms
- Centers for Disease Control and Prevention, U.S., Data Collection, Data Interpretation, Statistical, Disease Outbreaks, Humans, Influenza, Human, Office Visits, Prognosis, United States, Social Media
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.