The Parable of Google Flu: Traps in Big Data Analysis
Open accessThe DOI matches its Crossref record; 30 of the 31 references it cites were found in a registry or an index; 5 of the 8 sources its claims cite were checked for support.
Abstract
Large errors in flu prediction were largely avoidable, which offers lessons for the use of big data.
Overview
CoChat Summary
TL;DR
The paper uses the failure of Google Flu Trends (GFT) to illustrate two traps in big data analysis: “big data hubris” and algorithm dynamics. It argues that GFT’s large flu-prediction errors were largely avoidable and that big data should supplement, not replace, traditional data and methods.
Key findings
- In February 2013, GFT predicted more than double the proportion of doctor visits for influenza-like illness (ILI) than the CDC; from 21 August 2011 to 1 September 2013, GFT reported overly high flu prevalence in 100 out of 108 weeks, and it overshot the 2011–2012 season by more than 50%.
- GFT errors were not random: they showed temporal autocorrelation and seasonality, meaning last week’s errors predicted this week’s errors and error direction/magnitude varied with time of year.
- Simple alternatives outperformed GFT: out-of-sample mean absolute error was 0.486 for GFT, 0.311 for lagged CDC, and 0.232 for combined GFT and CDC; all differences were statistically significant at P < 0.05. Even 3-week-old CDC data did better than GFT at projecting current flu prevalence.
- The authors attribute GFT’s problems to overfitting (initial GFT matched 50 million search terms to 1152 data points and missed the nonseasonal 2009 H1N1 pandemic) and to “algorithm dynamics,” including Google search algorithm changes and user behavior changes.
Methods
- Design: policy forum/case study comparing GFT estimates with CDC ILI surveillance data and alternative forecasting models.
- Data/sample: GFT and CDC flu prevalence estimates, including the period from 21 August 2011 to 1 September 2013; the initial GFT methodology reportedly fit 50 million search terms to 1152 data points.
- Main analysis: comparison of GFT alone, lagged CDC data with 52-week seasonality variables, and a combined GFT + CDC model using lagged CDC estimates, lagged GFT error, and seasonality variables; reported mean absolute error and statistical significance; Google Correlate used to examine correlated search terms. Supplementary materials contain data and methods.
Limitations
- The paper states that GFT never documented the 45 search terms used, released examples appear misleading, and Google Correlate is limited to national-level data whereas GFT was developed using regional correlations; therefore the original GFT analyses cannot be replicated from the information provided.
- The paper states that Google cannot make its full data available to outsiders because of privacy issues, though it argues there is no such constraint on derivative, aggregated data.
- Evident from the text: this is a commentary/policy forum rather than a full empirical article, with key data and methods placed in supplementary materials; the quantitative comparison depends on GFT and CDC data and model specifications not fully detailed in the main text.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
2 Valid · 2 Not valid · 2 Not checked · 8 Own results and interpretations
Some cited sources support this claim and some don’t — read each source’s verdict below.
In the paper
“This ad hoc method of throwing out peculiar search terms failed when GFT completely missed the nonseasonal 2009 infl uenza A–H1N1 pandemic ( 2, 14).”
What the sources say
- Partly supportedcites [2] matched to “Reassessing Google Flu Trends Data for Detection of Seasonal and Pandemic Influenza: A Comparative Epidemiological Study at Three Geographic Scales” (2013) sourceExperimental — accuracy not yet measured
The Discussion (page unavailable) says the original GFT model missed the spring 2009 pandemic wave, supporting the claim about its failure to detect the nonseasonal outbreak. It does not say GFT completely missed the entire 2009 pandemic. The Introduction describes automated selection of the original model’s search terms, not an ad hoc method of throwing out peculiar terms. (LLM, conf=0.86)
- Disputedcites [14] S. Cook et al., PLOS ONE 6, e23610 (2011). sourceExperimental — accuracy not yet measured
The Abstract (Methodology/Principal Findings) says the original GFT model underestimated the magnitude of illness during the 2009 H1N1 pandemic, not that it completely missed it. The Abstract (Conclusions) says both models performed well during the pandemic, although the updated model performed better. The Discussion notes that pandemic-specific queries were excluded, but does not describe this as an ad hoc method of throwing out peculiar terms. (LLM, conf=0.88)
Checked by a reasoning model — may be wrong.
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“Oddly, the few search terms offered in the papers ( 14) do not seem to be strongly related with either GFT or the CDC data (SM)—we surmise that the authors felt an unarticulated need to cloak the actual search terms identifi ed.”
What the sources say
- Disputedcites [14] S. Cook et al., PLOS ONE 6, e23610 (2011). sourceExperimental — accuracy not yet measured
The Results section (‘Search Behavior During pH1N1’; page unavailable) says that all three search queries shown in Figure 3 closely tracked CDC ILINet data before the pandemic, and ‘symptoms of flu’ continued to do so during it. The paper does report that the bronchitis and pneumonia queries underestimated ILINet data during the pandemic, but that does not support the broad claim that the offered terms were not strongly related to the data. (LLM, conf=0.87)
Checked by a reasoning model — may be wrong.
hedgedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“The Internet has opened the way for improving standard surveys, experiments, and health reporting ( 35).”
What the sources say
- Partly supportedcites [35] A. J. Berinsky et al., Polit. Anal. 20, 351–368 (2012). sourceExperimental — accuracy not yet measured
Section 2 describes using an Internet platform to run surveys with embedded experiments, and the Conclusion says MTurk can make experimental research easier and less expensive. The paper does not address improvements to health reporting; the disease scenario in section 4.3.2 is a framing experiment, not health reporting. Page numbers are unavailable in our copy. (LLM, conf=0.88)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Although not widely reported until 2013, the new GFT has been persistently overestimating flu prevalence for a much longer time.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“From 21 August 2011 to 1 September 2013, GFT reported overly high fl u prevalence 100 out of 108 weeks.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
3 more claims
Create a free account to see all 14 claims and every source verdict.
- It may state two separate claims as one. 5
- Probably not a substantial claim. 2
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Google Trends predicted more than twice the number of influenza-like illnesses than the CDC in February 2013 46 .”
From Unrepresentative big surveys significantly overestimated US vaccine uptake (2021) · cites this paper as [46]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This was one of the main reasons behind the failure of Google Flu Trends, since the data used for training the model in the first few years was not representative of the trends in subsequent years [15].”
From Theory-Guided Data Science: A New Paradigm for Scientific Discovery from Data (2017) · cites this paper as [15]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Social data research often relies on counting entities. These entities can be users, links, or messages, and the description is a summary of these counts (Lazer et al., 2014; Salganik, 2017). Yet, simple counts can mislead if it is unclear what is counted and how.”
From Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019) · cites this paper as Lazer et al., 2014
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This unintelligibility is also subject to functional biases (section 4.1), as it depends on the mechanisms available on each social platform (e.g., having a like button, but not a dislike one), as well as to variations in platform algorithms and mechanisms in response to users actions (Lazer et al., 2014).”
From Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019) · cites this paper as Lazer et al., 2014
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“It also includes temporal validity, which captures to what extent constructs change over time (Howison et al., 2011) and may invalidate previous conclusions about societal and/or platform phenomena; e.g., see the case of Google Flu Trends (Lazer et al., 2014).”
From Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019) · cites this paper as Lazer et al., 2014
6 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 14 MARCH 2014 VOL 343 SCIENCE www.sciencemag.org 1204
“The 2009 version of GFT has weathered other media panics related to the fl u, including the 2005–2006 influenza A/H5N1 ("bird flu") outbreak”
“GFT engineers updated the algorithm in 2009, and this model has run ever since”
Citations
Cites
boyd, D., & Crawford, K. (2012) · Information, Communication & Society · doi:10.1080/1369118x.2012.678878 · Cited by 5,479
Twitter mood predicts the stock market
Bollen, J., Mao, H., & Zeng, X. (2011) · Journal of Computational Science · doi:10.1016/j.jocs.2010.12.007 · Cited by 5,206
Detecting influenza epidemics using search engine query data
Ginsberg, J., Mohebbi, M. H., Patel, R. S., Brammer, L., Smolinski, M. S., & Brilliant, L. (2009) · Nature · doi:10.1038/nature07634 · Cited by 4,295
Cohen, D. (1998) · The Women's Review of Books · doi:10.2307/4022859 · Cited by 4,261
Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk
Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012) · Political Analysis · doi:10.1093/pan/mpr057 · Cited by 4,008
Big Data : The Management Revolution
マカフィー, ア., ブリニョルフソン, エ., & 有賀, 裕. (2013) · Harvard business review · Cited by 3,549
Lazer, D., Pentland, A., Adamic, L., Aral, S., Barabási, A.-L., Brewer, D., Christakis, N., Contractor, N., Fowler, J., Gutmann, M., et al. (2009) · Science · doi:10.1126/science.1167742 · Cited by 3,365
Self-citationPredicting Elections with Twitter: What 140 Characters Reveal about Political Sentiment
Tumasjan, A., Sprenger, T., Sandner, P., & Welpe, I. (2010) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v4i1.14009 · Cited by 2,649
Multiscale mobility networks and the spatial spreading of infectious diseases
Balcan, D., Colizza, V., Gonçalves, B., Hu, H., Ramasco, J. J., & Vespignani, A. (2009) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0906910106 · Cited by 1,380
Self-citationPredicting consumer behavior with web search
Goel, S., Hofman, J. M., Lahaie, S., Pennock, D. M., & Watts, D. J. (2010) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1005962107 · Cited by 636
Research works printed but not linked: 7
- A. McAfee, E. Brynjolfsson, Harv. Bus. Rev. 90, 60 (2012).
- AAAI Conference on Weblogs and Social Media, Atlanta, Georgia, 11 to 15 July 2010 (Association for Advancement of Artifi cial Intelligence, 2010), pp. 178.
- P. Copeland et al., Int. Soc. Negl. Trop. Dis. 2013, 3
- I. M. Hall et al., Epidemiol. Infect. 135, 372 (2007).
- E. Mustafaraj, P. Metaxas, in Proceedings of the Web- Sci10, Raleigh, NC, 26 and 27 April 2010 (Web Science Trust, 2010); http://journal.webscience.org/317/.
- J. Ratkiewicz et al., in Proceedings of 5th International AAAI Conference on Weblogs and Social Media, San Francisco, CA, 7 to 11 August 2011 (AAAI, 2011), p. 297–304.
- 2013; http://chronicle.com/article/Researchers-Struggle- to-Secure/141591/.
Cited by
Items 301–400 of 2,513
Big data are coming to psychiatry: a general introduction
Monteith, S., Glenn, T., Geddes, J., & Bauer, M. (2015) · International Journal of Bipolar Disorders · doi:10.1186/s40345-015-0038-9 · Cited by 86
“These algorithms are not validated publicly and may be modified at any time, such as to improve customer service, which can impact their use in longitudinal studies (Lazer et al. 2014).”
BackgroundWu, Y. (2019) · Journalism Practice · doi:10.1080/17512786.2019.1682940 · Cited by 116
Calvard, T. S. (2016) · Management Learning · doi:10.1177/1350507615592113 · Cited by 86
“…for example, the “big data hubris” and over-reliance on certain algorithmic dynamics that ultimately led to erroneous predictions by the Google Flu Trends algorithm (Lazer et al., 2014), as well as other general possible abuses of metrics and analytical decision making (Davenport et al., 2010).”
BackgroundLi, Z., Liu, T., Zhu, G., Lin, H., Zhang, Y., He, J., Deng, A., Peng, Z., Xiao, J., Rutherford, S., et al. (2017) · PLoS neglected tropical diseases · doi:10.1371/journal.pntd.0005354 · Cited by 90
Big Data Analytics, Infectious Diseases and Associated Ethical Impacts
Garattini, C., Raffle, J., Aisyah, D. N., Sartain, F., & Kozlakidis, Z. (2019) · Philosophy & Technology · doi:10.1007/s13347-017-0278-y · Cited by 84
“Despite these positive outcomes, the routine use of BDA is increasingly challenged with examples of wrongful conclusions (Lazer et al. 2014), potential misuse of personal information, well-publicised privacy breeches and ongoing profiling of individuals for commercial purposes (Smith 2012; Richards…”
BackgroundRajaraman, V. (2016) · Resonance · doi:10.1007/s12045-016-0376-7 · Cited by 96
van Ooijen, C., Ubaldi, B., & Welby, B. (2019) · OECD working papers on public governance · doi:10.1787/09ab162c-en · Cited by 266
Pollett, S., Boscardin, W. J., Azziz‐Baumgartner, E., Tinoco, Y. O., Soto, G. M., Romero, C., Kok, J., Biggerstaff, M., Viboud, C. G., & Rutherford, G. W. (2016) · Clinical Infectious Diseases · doi:10.1093/cid/ciw657 · Cited by 86
Network analysis of big data research in tourism
Li, X., & Law, R. (2019) · Tourism Management Perspectives · doi:10.1016/j.tmp.2019.100608 · Cited by 105
Reinventing Radiology: Big Data and the Future of Medical Imaging
Morris, M. A., Saboury, B., Burkett, B. J., Gao, J. Y., & Siegel, E. L. (2017) · Journal of Thoracic Imaging · doi:10.1097/rti.0000000000000311 · Cited by 92
Big Data and the Innovation Cycle
Lee, H. L. (2018) · Production and Operations Management · doi:10.1111/poms.12845 · Cited by 81
“Lazer et al. (2014) described the interesting challenge of predicting the arrival of the flu season, in which the Center of Disease Control usually used their scientific and statistically based method.”
Bayesian Nonparametric Covariance RegressionPreprint
Fox, E. B., & Dunson, D. B. (2011) · arXiv (Cornell University) · doi:10.5555/2789272.2912079 · Cited by 96
“There has, however, been significant recent debate about the accuracy of the Google Flu Trend estimates (Butler, 2013; Lazer et al., 2014; Harris, 2014).”
ResultAnalyzing Big Data in Psychology: A Split/Analyze/Meta-Analyze Approach
Cheung, M. W. ‐., & Jak, S. (2016) · Frontiers in Psychology · doi:10.3389/fpsyg.2016.00738 · Cited by 99
“Now the question is whether psychologists know how to analyze these datasets to address important questions in their research domains.”
Big Data and medicine: a big deal?
Mayer‐Schönberger, V., & Ingelsson, E. (2017) · Journal of Internal Medicine · doi:10.1111/joim.12721 · Cited by 90
Phillips, C. A., Barz Leahy, A., Li, Y., Schapira, M. M., Bailey, L. C., & Merchant, R. M. (2018) · Journal of Medical Internet Research · doi:10.2196/jmir.8870 · Cited by 81
Social Media for Nowcasting Flu Activity: Spatio-Temporal Big Data Analysis
Zadeh, A. H., Zolbanin, H. M., Sharda, R., & Delen, D. (2019) · Information Systems Frontiers · doi:10.1007/s10796-018-9893-0 · Cited by 98
“Therefore, new developments should be mostly focused on exploring the ways to integrate web-based data sources into existing forecasting/surveillance systems, rather than retrospective analyses of performance (Aslam et al. 2014; Lazer et al. 2014; Milinovich et al. 2014).”
BackgroundRomero-Brufau, S., Wyatt, K. D., Boyum, P., Mickelson, M., Moore, M., & Cognetta-Rieke, C. (2020) · Applied Clinical Informatics · doi:10.1055/s-0040-1715827 · Cited by 81
Woo, H., Cho, Y., Shim, E., Lee, J.-K., Lee, C.-G., & Kim, S. H. (2016) · Journal of Medical Internet Research · doi:10.2196/jmir.4955 · Cited by 81
Salathé, M. (2016) · Journal of Infectious Diseases · doi:10.1093/infdis/jiw281 · Cited by 80
Inflated granularity: Spatial “Big Data” and geodemographics
Dalton, C. M., & Thatcher, J. (2015) · Big Data & Society · doi:10.1177/2053951715601144 · Cited by 79
“While scientists have raised concerns over Big Data’s methodological issues (Lazer et al., 2014), spatial Big Data practitioners lean on market justifications for their products akin to their geodemographic brethren.”
BackgroundUse of Twitter data to improve Zika virus surveillance in the United States during the 2016 epidemic
Masri, S., Jia, J., Li, C., Zhou, G., Lee, M., Yan, G., & Wu, J. (2019) · BMC Public Health · doi:10.1186/s12889-019-7103-8 · Cited by 87
The Big Data Agenda: Data Ethics and Critical Data Studies
Richterich, A. (2018) · University of Westminster Press eBooks · doi:10.16997/book14 · Cited by 162
Data Science: Challenges and Directions
Cao, L. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.16966 · Cited by 79
“After a comprehensive review [5, 6, 9, 10, 12, 15, 18] of hundreds of literature that directly incorporates data science in their scopes, we make the following observations of the big data buzz and data science debate:”
BackgroundBig Data and the danger of being precisely inaccurate
McFarland, D. A., & McFarland, H. R. (2015) · Big Data & Society · doi:10.1177/2053951715602495 · Cited by 81
“However, the model began over -reporting flu levels as soon as 2011 – as high as double the number of actual cases (Butler, 2013; Lazer et al., 2014).”
BackgroundSocial media in Ebola outbreak
HOSSAIN, L., KAM, D., KONG, F., WIGAND, R. T., & BOSSOMAIER, T. (2016) · Epidemiology and Infection · doi:10.1017/s095026881600039x · Cited by 78
“Initially, Google Flu trends was considered a potential source for early detection due to its capability to detect the trends but it started to overpredict flu cases [15].”
BackgroundBig data for secure healthcare system: a conceptual design
Sarkar, B. K. (2017) · Complex & Intelligent Systems · doi:10.1007/s40747-017-0040-1 · Cited by 79
Artificial intelligence and the public arena
Jungherr, A., & Schroeder, R. (2023) · Communication Theory · doi:10.1093/ct/qtad006 · Cited by 78
No! Formal Theory, Causal Inference, and Big Data Are Not Contradictory Trends in Political Science
Monroe, B. L., Pan, J., Roberts, M. E., Sen, M., & Sinclair, B. (2014) · PS Political Science & Politics · doi:10.1017/s1049096514001760 · Cited by 91
“A landmark example of this strength is provided by Lazer et al. ( 2014 ), who demonstrated that “Google Flu Trends” is not the valid leading indicator of flu incidence that it initially appeared to be.”
BackgroundMethodUsing Baidu Search Index to Predict Dengue Outbreak in China
Liu, K., Wang, T., Yang, Z., Huang, X., Milinovich, G. J., Lu, Y., Jing, Q., Xia, Y., Zhao, Z., Yang, Y., et al. (2016) · Scientific Reports · doi:10.1038/srep38040 · Cited by 78
Google dengue trends: An indicator of epidemic behavior. The Venezuelan Case
Strauss, R. A., Castro, J. S., Reintjes, R., & Torres, J. R. (2017) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2017.05.003 · Cited by 79
Utility and potential of rapid epidemic intelligence from internet-based sources
Yan, S.-L. J., Chughtai, A. A., & MacIntyre, C. R. (2017) · International Journal of Infectious Diseases · doi:10.1016/j.ijid.2017.07.020 · Cited by 86
The need for ‘skeptical’ accountants in the era of Big Data
McKinney, E. H., Yoos, C. J., & Snead, K. C. (2017) · Journal of Accounting Education · doi:10.1016/j.jaccedu.2016.12.007 · Cited by 112
The Economics of PrivacyPreprint
Acquisti, A., Taylor, C. R., & Wagman, L. (2016) · SSRN Electronic Journal · doi:10.2139/ssrn.2580411 · Cited by 1,369
“Researchers have mentioned “Big Data Hubris” (Lazer et al. 2014) in reference to the “implicit assumption that big data are a substitute for, rather than a supplement to, traditional data collection and analysis.”
BackgroundLearning from Large-Scale Wearable Device Data for Predicting the Epidemic Trend of COVID-19
Zhu, G., Li, J., Meng, Z., Yu, Y., Li, Y., Tang, X., Dong, Y., Sun, G., Zhou, R., Wang, H., et al. (2020) · Discrete Dynamics in Nature and Society · doi:10.1155/2020/6152041 · Cited by 77
“However, search queries can be greatly influenced by social hotspots, which weakens the correlation between the search queries and the occurrence of influenza-like diseases [17].”
BackgroundIs there a role for statistics in artificial intelligence?
Friedrich, S., Antes, G., Behr, S., Binder, H., Brannath, W., Dumpert, F., Ickstadt, K., Kestler, H. A., Lederer, J., Leitgöb, H., et al. (2021) · Advances in Data Analysis and Classification · doi:10.1007/s11634-021-00455-6 · Cited by 115
Song, S., Zhang, Y., & Yu, B. (2020) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2020.104321 · Cited by 94
Mathematical Models for Predicting Covid-19 Pandemic: A Review
Vytla, V., Ramakuri, S. K., Peddi, A., Srinivas, K., & Ragav, N. N. (2021) · Journal of Physics Conference Series · doi:10.1088/1742-6596/1797/1/012009 · Cited by 100
Bauman, A. E., Kamada, M., Reis, R. S., Troiano, R. P., Ding, D., Milton, K. E., Murphy, N. M., & Hallal, P. C. (2021) · The Lancet · doi:10.1016/s0140-6736(21)01165-x · Cited by 100
Innovations in genomic antimicrobial resistance surveillance
Wheeler, N. E., Price, V., Cunningham-Oakes, E., Tsang, K. K., Nunn, J. G., Midega, J. T., Anjum, M. F., Wade, M. J., Feasey, N., Peacock, S. J., et al. (2023) · The Lancet Microbe · doi:10.1016/s2666-5247(23)00285-9 · Cited by 97
Milinovich, G. J., Avril, S. M. R., Clements, A., Brownstein, J. S., Tong, S., & Hu, W. (2014) · BMC Infectious Diseases · doi:10.1186/s12879-014-0690-1 · Cited by 86
Sharpe, J. D., Hopkins, R. S., Cook, R. L., & Striley, C. W. (2016) · JMIR Public Health and Surveillance · doi:10.2196/publichealth.5901 · Cited by 77
Network representation learning: a systematic literature review
Li, B., & Pi, D. (2020) · Neural Computing and Applications · doi:10.1007/s00521-020-04908-5 · Cited by 85
Data-Driven Computational Social Science: A Survey
Zhang, J., Wang, W. W., Xia, F., Lin, Y., & Tong, H. (2020) · Big Data Research · doi:10.1016/j.bdr.2020.100145 · Cited by 94
“In addition, one important issue we need to consider is the analysis validation [191].”
BackgroundUAV Model-based Flight Control with Artificial Neural Networks: A Survey
Gu, W., Valavanis, K. P., Rutherford, M. J., & Rizzo, A. (2020) · Journal of Intelligent & Robotic Systems · doi:10.1007/s10846-020-01227-8 · Cited by 90
Spatial and temporal epidemiological analysis in the Big Data era
Pfeiffer, D. U., & Stevens, K. B. (2015) · Preventive Veterinary Medicine · doi:10.1016/j.prevetmed.2015.05.012 · Cited by 76
“However, the results generated by this algorithm have been the subject of controversy as predictions were incorrect at specific time points when they particularly mattered (Butler, 2013; Lazer et al., 2014).”
Principles of Bayesian Inference Using General Divergence Criteria
Jewson, J., Smith, J. Q., & Holmes, C. C. (2018) · Entropy · doi:10.3390/e20060442 · Cited by 148
“The model provides the only opportunity to input not only structural, but quantitative expert judgements about the domain, something that is often critical to a successful analysis—see [12] for example.”
BackgroundPappa, D., & Stergioulas, L. K. (2019) · International Journal of Data Science and Analytics · doi:10.1007/s41060-019-00175-3 · Cited by 86
Rieder, B., Abdulla, R. A., Poell, T., Woltering, R. A. F. L., & Zack, L. (2015) · Big Data & Society · doi:10.1177/2053951715614980 · Cited by 111
“In this, we follow the call by Lazer et al. (2014) to ‘‘study the algorithm’’, which we consider the API to be a part of: Twitter, Facebook, Google, and the Internet more generally are constantly changing because of the actions of millions of engineers and consumers.”
BackgroundMind the Scales: Harnessing Spatial Big Data for Infectious Disease Surveillance and Inference
Lee, E. C., Asher, J. M., Goldlust, S., Kraemer, J. D., Lawson, A. B., & Bansal, S. (2016) · Journal of Infectious Diseases · doi:10.1093/infdis/jiw344 · Cited by 75
Youth Perspectives on Critical Data Literacies
Hautea, S., Dasgupta, S., & Hill, B. M. (2017) · Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025823 · Cited by 74
“In particular, these critiques have emphasized the challenges associated with drawing valid conclusions from online behavioral data and the need for researchers and educators to reflect on the ethics of how data is acquired, interpreted, and utilized [5, 33, 34, 41].”
BackgroundBig Data, data integrity, and the fracturing of the control zone
Lagoze, C. (2014) · Big Data & Society · doi:10.1177/2053951714558281 · Cited by 75
“…the data), researchers have at times unwittingly or sloppily applied methodological tools or epistemo-logical understanding to those data that failed to account for the fundamental differences between them and traditional highly-curated and reliable data As pointed out by Lazer et al. (2014), ‘‘ .”
BackgroundHora, M. T., Bouwma-Gearhart, J. L., & Park, H. J. (2017) · Review of higher education/The review of higher education · doi:10.1353/rhe.2017.0013 · Cited by 125
Big Data for Policymaking: Great Expectations, but with Limited Progress?
Poel, M., Meyer, E. T., & Schroeder, R. (2018) · Policy & Internet · doi:10.1002/poi3.176 · Cited by 86
Lee, E. W. J., & Viswanath, K. (2020) · Journal of Medical Internet Research · doi:10.2196/16377 · Cited by 74
Ensuring trustworthy use of artificial intelligence and big data analytics in health insurance
Ho, C. W.-L., Ali, J., & Caals, K. (2020) · Bulletin of the World Health Organization · doi:10.2471/blt.19.234732 · Cited by 99
The New ABCs of Research: Achieving Breakthrough Collaborations
Shneiderman, B. (2016) · doi:10.1093/acprof:oso/9780198758839.001.0001 · Cited by 74
Garrett, K. A., Bebber, D. P., Etherton, B. A., Gold, K. M., Sulá, A. I. P., & Selvaraj, M. G. (2022) · Annual Review of Phytopathology · doi:10.1146/annurev-phyto-021021-042636 · Cited by 89
Learning assessment in the age of big data: Learning analytics in higher education
Caspari-Sadeghi, S. (2023) · Cogent Education · doi:10.1080/2331186x.2022.2162697 · Cited by 74
“Therefore, acquisition, storage, distribution, analysis, and management of big data are conducted through innovative computational technologies called Big Data Analytics (Lazer et al., 2014).”
MethodThe dangers of faulty, biased, or malicious algorithms requires independent oversight
Shneiderman, B. (2016) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1618211113 · Cited by 73
Dong, J., Wu, H., Zhou, D., Li, K., Zhang, Y., Ji, H., Tong, Z., Lou, S., & Liu, Z. (2021) · Journal of Medical Systems · doi:10.1007/s10916-021-01757-0 · Cited by 96
Demand forecasting with user-generated online information
Schaer, O., Kourentzes, N., & Fildes, R. A. (2018) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2018.03.005 · Cited by 92
“Its service was discontinued in 2015, partly because of data reliability concerns (Lazer et al., 2014; Butler, 2013).”
BackgroundMethodForecasting Seasonal Influenza Fusing Digital Indicators and a Mechanistic Disease Model
Zhang, Q., Perra, N., Perrotta, D., Tizzoni, M., Paolotti, D., & Vespignani, A. (2017) · International World Wide Web Conference (WWW) · doi:10.1145/3038912.3052678 · Cited by 78
“Furthermore, they might be affected by non trivial biases in the data used that could significantly lower their accuracy [9, 10, 27].”
BackgroundMethodSelf-citationUsing Clinicians' Search Query Data to Monitor Influenza Epidemics
Santillana, M., Nsoesie, E. O., Mekaru, S. R., Scales, D. C., & Brownstein, J. S. (2014) · Clinical Infectious Diseases · doi:10.1093/cid/ciu647 · Cited by 86
“The potential for the public ’ s search activity to be in fl uenced by anxiety, fears, and rumors raises concerns regarding reliability [10 – 13].”
BackgroundBarros, J. M., Melia, R., Francis, K., Bogue, J., O’Sullivan, M., Young, K. L., Bernert, R. A., Rebholz‐Schuhmann, D., & Duggan, J. (2019) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph16173201 · Cited by 75
“In detail, for search queries, one must also be wary of several factors such as changes to the search algorithm [34] and media events which lead to an unexpected behavior [35].”
BackgroundThe Automation of Society is Next: How to Survive the Digital RevolutionPreprint
Helbing, D. (2015) · SSRN Electronic Journal · doi:10.2139/ssrn.2694312 · Cited by 82
Nestor, B. A., McDermott, M. B. A., Boag, W., Berner, G., Naumann, T., Hughes, M. C., Goldenberg, A., & Ghassemi, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.00690 · Cited by 71
“, 2017) all use time-agnostic evaluation protocols, which do not account for a significant source of error that would affect models during true deployment: namely, the evolution of care practices over time and the resultant concept drift (Žliobaitė, 2010), which are known to induce significant differences in clinical…”
BackgroundBehavior Revealed in Mobile Phone Usage Predicts Loan RepaymentPreprint
Björkegren, D., & Grissen, D. (2015) · SSRN Electronic Journal · doi:10.2139/ssrn.2611775 · Cited by 115
Scrutinizing an algorithmic technique: the Bayes classifier as interested reading of reality
Rieder, B. (2016) · Information Communication & Society · doi:10.1080/1369118x.2016.1181195 · Cited by 88
Medicare Fraud Detection Using Random Forest with Class Imbalanced Big Data
Bauder, R., & Khoshgoftaar, T. (2018) · IEEE International Conference on Information Reuse and Integration · doi:10.1109/iri.2018.00019 · Cited by 65
“Because big data is available and machine learning techniques can readily handle these copious amounts of data, building models directly using the entire dataset, with minimal prior data analysis or preparation, appears to be increasingly common [29].”
BackgroundZhang, B., Cao, D., Hu, W., Ghias, A. M. Y. M., & Chen, Z. (2023) · International Journal of Electrical Power & Energy Systems · doi:10.1016/j.ijepes.2023.109641 · Cited by 92
Using Networks to Combine “Big Data” and Traditional Surveillance to Improve Influenza Predictions
Davidson, M. W., Haim, D. A., & Radin, J. M. (2015) · Scientific Reports · doi:10.1038/srep08154 · Cited by 68
“Using network data from the previous year to inform our predictions, we hypothesized that incorporating this weighted influenza load measure would allow for (1) better assessment of real-time infection levels and (2) more accurate predictions regarding the future spread of influenza.”
MethodCan we trust Big Data? Applying philosophy of science to software
Symons, J., & Alvarado, R. (2016) · Big Data & Society · doi:10.1177/2053951716664747 · Cited by 69
“Because of this, the project is often taken to exemplify ‘‘Big Data hubris’’ (Lazer et al., 2014), the often underlying assumption that large amounts of data and the patterns that are discovered through its analysis can yield results independently from or without the aid of principled theoretical…”
BackgroundMethodInfluentialTranslating Big Data into Smart Data for Veterinary Epidemiology
VanderWaal, K. L., Morrison, R. B., Neuhauser, C. M., Vilalta, C., & Perez, A. M. (2017) · Frontiers in Veterinary Science · doi:10.3389/fvets.2017.00110 · Cited by 75
“Due to the large number of potential variables, care must be taken to identify spurious correlations (5, 66) , and the use of large datasets does not necessarily increase a study's validity.”
BackgroundCrowd-Squared: Amplifying the Predictive Power of Search Trend Data1
Brynjolfsson, E., Geva, T., & Reichman, S. (2016) · MIS Quarterly · doi:10.25300/misq/2016/40.4.07 · Cited by 82
Bigger sociological imaginations: framing big social data theory and methods
Halavais, A. (2015) · Information Communication & Society · doi:10.1080/1369118x.2015.1008543 · Cited by 93
On Some Principles of Statistical Inference
Reid, N., & Cox, D. R. (2015) · International Statistical Review · doi:10.1111/insr.12067 · Cited by 66
“Lazer et al. (2014) describe a setting where seemingly big data relied for its analysis on what was effectively a small data set, which led to overfitting.”
BackgroundCole-Lewis, H. J., Varghese, A., Sanders, A. E., Schwarz, M., Pugatch, J., & Augustson, E. M. (2015) · Journal of Medical Internet Research · doi:10.2196/jmir.4392 · Cited by 68
Google Trends can improve surveillance of Type 2 diabetes
Tkachenko, N., Chotvijit, S., Gupta, N., Bradley, E., Gilks, C., Guo, W., Crosby, H., Shore, E., Thiarai, M., Procter, R., et al. (2017) · Scientific Reports · doi:10.1038/s41598-017-05091-9 · Cited by 76
Do Customers Speak Their Minds? Using Forums and Search for Predicting Sales
Geva, T., Oestreicher-Singer, G., Efron, N., & Shimshoni, Y. (2013) · MIS Quarterly · doi:10.25300/misq/2017/41.1.04 · Cited by 80
Chen, Y., & Han, D. (2016) · Journal of Hydroinformatics · doi:10.2166/hydro.2016.180 · Cited by 64
Lünich, M. (2022) · Springer eBooks · doi:10.1007/978-3-658-36368-0 · Cited by 71
Big Data and Industrial Ecology
Xu, M., Cai, H., & Liang, S. (2015) · Journal of Industrial Ecology · doi:10.1111/jiec.12241 · Cited by 75
“Big data enabled by ICT naturally grows rapidly and becomes massive.”
BackgroundBig Data Analytics = Machine Learning + Cloud Computing
Wu, C., Buyya, R., & Ramamohanarao, K. (2016) · Elsevier eBooks · doi:10.1016/b978-0-12-805394-2.00001-5 · Cited by 113
“Another Big Data critique comes from David Lazer et al. [19].”
Cloud-based Electronic Health Records for Real-time, Region-specific Influenza Surveillance
Santillana, M., Nguyen, A. T., Louie, T., Zink, A., Gray, J., Sung, I., & Brownstein, J. S. (2016) · Scientific Reports · doi:10.1038/srep25732 · Cited by 78
Pfeffer, K., Verrest, H., & Poorthuis, A. (2015) · European Journal of Development Research · doi:10.1057/ejdr.2015.48 · Cited by 68
“The clustering of social media data also enriches our understanding of human interactions among people (social networks) and non-linear relations with the physical environment (for example, Lazer et al , 2014).”
BackgroundJendryke, M., Balz, T., McClure, S. C., & Liao, M. (2016) · Computers Environment and Urban Systems · doi:10.1016/j.compenvurbsys.2016.10.004 · Cited by 65
Identifying Adverse Effects of HIV Drug Treatment and Associated Sentiments Using Twitter
Adrover, C., Bodnar, T. J., Huang, Z., Telenti, A., & Salathé, M. (2015) · JMIR Public Health and Surveillance · doi:10.2196/publichealth.4488 · Cited by 64
Recent progress and trends in predictive visual analytics
Lu, J., Chen, W., Ma, Y., Ke, J., Li, Z., Zhang, F., & Maciejewski, R. (2016) · Frontiers of Computer Science · doi:10.1007/s11704-016-6028-y · Cited by 64
“[6] who recommend several items including increased data and algorithmic transparency to improve analytic intelligence and data linkage between smaller, cleaner datasets, and model construction.”
BackgroundCrowdbreaks: Tracking Health Trends Using Public Social Media Data and Crowdsourcing
Müller, M., & Salathé, M. (2019) · Frontiers in Public Health · doi:10.3389/fpubh.2019.00081 · Cited by 75
“Shortly after, it was discontinued [8, 9].”
Big data from dynamic pricing: A smart approach to tourism demand forecasting
Guizzardi, A., Pons, F., Angelini, G., & Ranieri, E. (2020) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2020.11.006 · Cited by 84
Kim, J., Seo, D., Jang, M., & Kim, J. (2021) · Journal of Hydrology · doi:10.1016/j.jhydrol.2021.126817 · Cited by 71
Research Roadmap of Service Ecosystems: A Crowd Intelligence Perspective
Xue, X., Li, G., Zhou, D., Zhang, Y., Zhang, L., Zhao, Y., Feng, Z., Cui, L., Zhou, Z., Sun, X., et al. (2022) · International Journal of Crowd Science · doi:10.26599/ijcs.2022.9100026 · Cited by 78
When decision support systems fail: Insights for strategic information systems from Formula 1
Aversa, P., Cabantous, L., & Haefliger, S. (2018) · The Journal of Strategic Information Systems · doi:10.1016/j.jsis.2018.03.002 · Cited by 65
Neural-inspired sensors enable sparse, efficient classification of spatiotemporal data
Mohren, T. L., Daniel, T. L., Brunton, S. L., & Brunton, B. W. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1808909115 · Cited by 71
“In both living systems and modern technology, there is a tension between gathering vast and increasing quantities of heterogeneous data (e.g., the internet-of-things), versus acquiring targeted data gathered by specialized sensors [6, 29].”
BackgroundIntroduction to Scientific Computing
Holmes, M. H. (2016) · Texts in computational science and engineering · doi:10.1007/978-3-319-30256-0_1 · Cited by 65
Don’t Mention It? Analyzing User-Generated Content Signals for Early Adverse Event Warnings
Abbasi, A., Li, J., Adjeroh, D., Abate, M. A., & Zheng, W. (2019) · Information Systems Research · doi:10.1287/isre.2019.0847 · Cited by 77
“…channels such as search and Twitter may have a degree of separation from event detection tasks in terms of their primary use cases, sole reliance on such channels would not be prudent, as noted by recent issues with the Google Flu monitoring system (Agarwal and Dhar 2014, Lazer et al. 2014).”
BackgroundDrivers and challenges of internet of things diffusion in smart stores: A field exploration
Roe, M., Spanaki, K., Ioannou, A., Zamani, E. D., & Giannakis, M. (2022) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2022.121593 · Cited by 86
“This relates to the adage ‘bigger data are not necessarily better data’ (Lazer et al., 2014), which supports the process of pre-filtering ‘useful’ data.”
BackgroundThe data revolution in social science needs qualitative research
Grigoropoulou, N., & Small, M. L. (2022) · Nature Human Behaviour · doi:10.1038/s41562-022-01333-7 · Cited by 71
Secondary data for global health digitalisation
Näher, A., Vorisek, C. N., Klopfenstein, S. A. I., Lehne, M., Thun, S., Alsalamah, S., Pujari, S., Heider, D., Ahrens, W., Pigeot, I., et al. (2023) · The Lancet Digital Health · doi:10.1016/s2589-7500(22)00195-9 · Cited by 80
“As well as the advantages of sourcing health-related data from the web, wearables, apps, or electronic health records there are also considerable limitations (table): 32,42 estimates solely resting on data gathered from these sources are prone to sampling bias.”
BackgroundXia, H., Barnes, S., & Golden, B. (2017) · International Transactions in Operational Research · doi:10.1111/itor.12400 · Cited by 84
“…improve the amount and quality of usable data for more accurate estimation of patient wait times, LOS, and service times inside the ED, and provide valuable resources for system analytics, optimization, and validation (Peck et al., 2012; Hsieh et al., 2013; Lazer et al., 2014; Ang et al., 2015).”
Background
Metadata
- Authors
- Lazer, D., Kennedy, R., King, G., & Vespignani, A.
- Affiliations
- Northeastern University (US)
- Harvard University (US)
- Corresponding author
- Lazer, David
- Venue
- Science
- Published
- 2014-03-14
- Volume
- 343
- Issue
- 6176
- Pages
- 1203-1205
- ISSN
- 0036-8075 (ISSN-L)
- 1095-9203
- OpenAlex
- W2068181924
- Semantic Scholar
- 00d681cfff7f0224ace607e515f4ed510e792df9
- Cited by
- 2,523
- References
- 30
- Citations by year
- 2026: 113 · 2025: 101 · 2024: 121 · 2023: 150 · 2022: 168 · 2021: 229 · 2020: 273 · 2019: 261 · 2018: 272 · 2017: 249 · 2016: 271 · 2015: 219 · 2014: 91
- FWCI
- 208.16
- Type
- journal article
- Language
- en
- Open access
- green
- dash.harvard.edu
- Licence
- cc-by
- Locations
- Science (publishedVersion)
- PubMed (publishedVersion)
- Digital Access to Scholarship at Harvard (DASH) (Harvard University) (submittedVersion, cc-by)
- Topics
- Health Sciences › Medicine › Epidemiology › Data-Driven Disease Surveillance
- Social Sciences › Decision Sciences › Information Systems and Management › Big Data Technologies and Applications
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- Google Flu Trends, influenza surveillance, big data, prediction error, data quality
- MeSH terms
- Centers for Disease Control and Prevention, U.S., Data Collection, Data Interpretation, Statistical, Disease Outbreaks, Humans, Influenza, Human, Office Visits, Prognosis, United States, Social Media
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.