The Parable of Google Flu: Traps in Big Data Analysis
Open accessThe DOI matches its Crossref record; 30 of the 31 references it cites were found in a registry or an index; 5 of the 8 sources its claims cite were checked for support.
Abstract
Large errors in flu prediction were largely avoidable, which offers lessons for the use of big data.
Overview
CoChat Summary
TL;DR
The paper uses the failure of Google Flu Trends (GFT) to illustrate two traps in big data analysis: “big data hubris” and algorithm dynamics. It argues that GFT’s large flu-prediction errors were largely avoidable and that big data should supplement, not replace, traditional data and methods.
Key findings
- In February 2013, GFT predicted more than double the proportion of doctor visits for influenza-like illness (ILI) than the CDC; from 21 August 2011 to 1 September 2013, GFT reported overly high flu prevalence in 100 out of 108 weeks, and it overshot the 2011–2012 season by more than 50%.
- GFT errors were not random: they showed temporal autocorrelation and seasonality, meaning last week’s errors predicted this week’s errors and error direction/magnitude varied with time of year.
- Simple alternatives outperformed GFT: out-of-sample mean absolute error was 0.486 for GFT, 0.311 for lagged CDC, and 0.232 for combined GFT and CDC; all differences were statistically significant at P < 0.05. Even 3-week-old CDC data did better than GFT at projecting current flu prevalence.
- The authors attribute GFT’s problems to overfitting (initial GFT matched 50 million search terms to 1152 data points and missed the nonseasonal 2009 H1N1 pandemic) and to “algorithm dynamics,” including Google search algorithm changes and user behavior changes.
Methods
- Design: policy forum/case study comparing GFT estimates with CDC ILI surveillance data and alternative forecasting models.
- Data/sample: GFT and CDC flu prevalence estimates, including the period from 21 August 2011 to 1 September 2013; the initial GFT methodology reportedly fit 50 million search terms to 1152 data points.
- Main analysis: comparison of GFT alone, lagged CDC data with 52-week seasonality variables, and a combined GFT + CDC model using lagged CDC estimates, lagged GFT error, and seasonality variables; reported mean absolute error and statistical significance; Google Correlate used to examine correlated search terms. Supplementary materials contain data and methods.
Limitations
- The paper states that GFT never documented the 45 search terms used, released examples appear misleading, and Google Correlate is limited to national-level data whereas GFT was developed using regional correlations; therefore the original GFT analyses cannot be replicated from the information provided.
- The paper states that Google cannot make its full data available to outsiders because of privacy issues, though it argues there is no such constraint on derivative, aggregated data.
- Evident from the text: this is a commentary/policy forum rather than a full empirical article, with key data and methods placed in supplementary materials; the quantitative comparison depends on GFT and CDC data and model specifications not fully detailed in the main text.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
2 Valid · 2 Not valid · 2 Not checked · 8 Own results and interpretations
Some cited sources support this claim and some don’t — read each source’s verdict below.
In the paper
“This ad hoc method of throwing out peculiar search terms failed when GFT completely missed the nonseasonal 2009 infl uenza A–H1N1 pandemic ( 2, 14).”
What the sources say
- Partly supportedcites [2] matched to “Reassessing Google Flu Trends Data for Detection of Seasonal and Pandemic Influenza: A Comparative Epidemiological Study at Three Geographic Scales” (2013) sourceExperimental — accuracy not yet measured
The Discussion (page unavailable) says the original GFT model missed the spring 2009 pandemic wave, supporting the claim about its failure to detect the nonseasonal outbreak. It does not say GFT completely missed the entire 2009 pandemic. The Introduction describes automated selection of the original model’s search terms, not an ad hoc method of throwing out peculiar terms. (LLM, conf=0.86)
- Disputedcites [14] S. Cook et al., PLOS ONE 6, e23610 (2011). sourceExperimental — accuracy not yet measured
The Abstract (Methodology/Principal Findings) says the original GFT model underestimated the magnitude of illness during the 2009 H1N1 pandemic, not that it completely missed it. The Abstract (Conclusions) says both models performed well during the pandemic, although the updated model performed better. The Discussion notes that pandemic-specific queries were excluded, but does not describe this as an ad hoc method of throwing out peculiar terms. (LLM, conf=0.88)
Checked by a reasoning model — may be wrong.
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“Oddly, the few search terms offered in the papers ( 14) do not seem to be strongly related with either GFT or the CDC data (SM)—we surmise that the authors felt an unarticulated need to cloak the actual search terms identifi ed.”
What the sources say
- Disputedcites [14] S. Cook et al., PLOS ONE 6, e23610 (2011). sourceExperimental — accuracy not yet measured
The Results section (‘Search Behavior During pH1N1’; page unavailable) says that all three search queries shown in Figure 3 closely tracked CDC ILINet data before the pandemic, and ‘symptoms of flu’ continued to do so during it. The paper does report that the bronchitis and pneumonia queries underestimated ILINet data during the pandemic, but that does not support the broad claim that the offered terms were not strongly related to the data. (LLM, conf=0.87)
Checked by a reasoning model — may be wrong.
hedgedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“The Internet has opened the way for improving standard surveys, experiments, and health reporting ( 35).”
What the sources say
- Partly supportedcites [35] A. J. Berinsky et al., Polit. Anal. 20, 351–368 (2012). sourceExperimental — accuracy not yet measured
Section 2 describes using an Internet platform to run surveys with embedded experiments, and the Conclusion says MTurk can make experimental research easier and less expensive. The paper does not address improvements to health reporting; the disease scenario in section 4.3.2 is a framing experiment, not health reporting. Page numbers are unavailable in our copy. (LLM, conf=0.88)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Although not widely reported until 2013, the new GFT has been persistently overestimating flu prevalence for a much longer time.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“From 21 August 2011 to 1 September 2013, GFT reported overly high fl u prevalence 100 out of 108 weeks.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
3 more claims
Create a free account to see all 14 claims and every source verdict.
- It may state two separate claims as one. 5
- Probably not a substantial claim. 2
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Google Trends predicted more than twice the number of influenza-like illnesses than the CDC in February 2013 46 .”
From Unrepresentative big surveys significantly overestimated US vaccine uptake (2021) · cites this paper as [46]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This was one of the main reasons behind the failure of Google Flu Trends, since the data used for training the model in the first few years was not representative of the trends in subsequent years [15].”
From Theory-Guided Data Science: A New Paradigm for Scientific Discovery from Data (2017) · cites this paper as [15]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Social data research often relies on counting entities. These entities can be users, links, or messages, and the description is a summary of these counts (Lazer et al., 2014; Salganik, 2017). Yet, simple counts can mislead if it is unclear what is counted and how.”
From Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019) · cites this paper as Lazer et al., 2014
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This unintelligibility is also subject to functional biases (section 4.1), as it depends on the mechanisms available on each social platform (e.g., having a like button, but not a dislike one), as well as to variations in platform algorithms and mechanisms in response to users actions (Lazer et al., 2014).”
From Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019) · cites this paper as Lazer et al., 2014
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“It also includes temporal validity, which captures to what extent constructs change over time (Howison et al., 2011) and may invalidate previous conclusions about societal and/or platform phenomena; e.g., see the case of Google Flu Trends (Lazer et al., 2014).”
From Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019) · cites this paper as Lazer et al., 2014
6 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 14 MARCH 2014 VOL 343 SCIENCE www.sciencemag.org 1204
“The 2009 version of GFT has weathered other media panics related to the fl u, including the 2005–2006 influenza A/H5N1 ("bird flu") outbreak”
“GFT engineers updated the algorithm in 2009, and this model has run ever since”
Citations
Cites
boyd, D., & Crawford, K. (2012) · Information, Communication & Society · doi:10.1080/1369118x.2012.678878 · Cited by 5,479
Twitter mood predicts the stock market
Bollen, J., Mao, H., & Zeng, X. (2011) · Journal of Computational Science · doi:10.1016/j.jocs.2010.12.007 · Cited by 5,206
Detecting influenza epidemics using search engine query data
Ginsberg, J., Mohebbi, M. H., Patel, R. S., Brammer, L., Smolinski, M. S., & Brilliant, L. (2009) · Nature · doi:10.1038/nature07634 · Cited by 4,295
Cohen, D. (1998) · The Women's Review of Books · doi:10.2307/4022859 · Cited by 4,261
Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk
Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012) · Political Analysis · doi:10.1093/pan/mpr057 · Cited by 4,008
Big Data : The Management Revolution
マカフィー, ア., ブリニョルフソン, エ., & 有賀, 裕. (2013) · Harvard business review · Cited by 3,549
Lazer, D., Pentland, A., Adamic, L., Aral, S., Barabási, A.-L., Brewer, D., Christakis, N., Contractor, N., Fowler, J., Gutmann, M., et al. (2009) · Science · doi:10.1126/science.1167742 · Cited by 3,365
Self-citationPredicting Elections with Twitter: What 140 Characters Reveal about Political Sentiment
Tumasjan, A., Sprenger, T., Sandner, P., & Welpe, I. (2010) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v4i1.14009 · Cited by 2,649
Multiscale mobility networks and the spatial spreading of infectious diseases
Balcan, D., Colizza, V., Gonçalves, B., Hu, H., Ramasco, J. J., & Vespignani, A. (2009) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0906910106 · Cited by 1,380
Self-citationPredicting consumer behavior with web search
Goel, S., Hofman, J. M., Lahaie, S., Pennock, D. M., & Watts, D. J. (2010) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1005962107 · Cited by 636
Research works printed but not linked: 7
- A. McAfee, E. Brynjolfsson, Harv. Bus. Rev. 90, 60 (2012).
- AAAI Conference on Weblogs and Social Media, Atlanta, Georgia, 11 to 15 July 2010 (Association for Advancement of Artifi cial Intelligence, 2010), pp. 178.
- P. Copeland et al., Int. Soc. Negl. Trop. Dis. 2013, 3
- I. M. Hall et al., Epidemiol. Infect. 135, 372 (2007).
- E. Mustafaraj, P. Metaxas, in Proceedings of the Web- Sci10, Raleigh, NC, 26 and 27 April 2010 (Web Science Trust, 2010); http://journal.webscience.org/317/.
- J. Ratkiewicz et al., in Proceedings of 5th International AAAI Conference on Weblogs and Social Media, San Francisco, CA, 7 to 11 August 2011 (AAAI, 2011), p. 297–304.
- 2013; http://chronicle.com/article/Researchers-Struggle- to-Secure/141591/.
Cited by
Items 201–300 of 2,513
Benelli, G., & Duggan, M. F. (2018) · Acta Tropica · doi:10.1016/j.actatropica.2018.02.015 · Cited by 136
“The experiment has not been as successful in detecting outbreaks as hoped, but it pointed out the idea that hotspots, key nodes in a social network, can be exploited in the near future to combat infectious diseases early (Lazer et al., 2014).”
BackgroundFlexible Modeling of Epidemics with an Empirical Bayes Framework
Brooks, L., Farrow, D., Hyun, S., Tibshirani, R. J., & Rosenfeld, R. (2015) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1004382 · Cited by 129
Chinese Social Media and Big Data: Big Data, Big Brother, Big Profit?
Jiang, M., & Fu, K. (2018) · Policy & Internet · doi:10.1002/poi3.187 · Cited by 118
Demographic research with non-representative internet data
Zagheni, E., & Weber, I. (2015) · International Journal of Manpower · doi:10.1108/ijm-12-2014-0261 · Cited by 119
“However, at times, it also produced largely erroneous estimates, typically when the nature of the relationship between searches, news and behaviors changed (Lazer et al., 2014).”
BackgroundA review of the literature on big data analytics in healthcare
Galetsi, P., & Katsaliaki, K. (2019) · Journal of the Operational Research Society · doi:10.1080/01605682.2019.1630328 · Cited by 139
Leitgöb, H., & Wolbring, T. (2021) · Schriftenreihe der ASI · doi:10.1007/978-3-658-34396-5_1 · Cited by 123
Moran, K. R., Fairchild, G. C., Generous, N., Hickmann, K. S., Osthus, D., Priedhorsky, R., Hyman, J. M., & Valle, S. Y. D. (2016) · The Journal of Infectious Diseases · doi:10.1093/infdis/jiw375 · Cited by 112
“demic is an example of this phenomenon [25].”
BackgroundJia, Q., Guo, Y., Wang, G., & Barnes, S. J. (2020) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph17176161 · Cited by 117
“Although the H1N1 outbreak was successfully predicted in 2009 [36], it was also questioned because of the high predicted incidence of influenza [37].”
BackgroundA dynamic linear model to forecast hotel registrations in Puerto Rico using Google Trends data
Rivera, R. (2016) · Tourism Management · doi:10.1016/j.tourman.2016.04.008 · Cited by 133
“Another possible explanation for the poor performance of Google Flu Trends is the fact that Google constantly changes the functionality of searches (Lazer et al. 2014).”
BackgroundJungherr, A., Schoen, H., Posegga, O., & Jürgens, P. (2017) · Social Science Computer Review · doi:10.1177/0894439316631043 · Cited by 114
Big data has big potential for applications to climate change adaptation
Ford, J. D., Tilleard, S. E., Berrang-Ford, L., Araos, M., Biesbroek, R., Lesnikowski, A. C., MacDonald, G. K., Hsu, A., Chen, C., & Bizikova, L. (2016) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1614023113 · Cited by 114
Crowd intelligence in AI 2.0 era
Li, W., Wu, W., Wang, H., Cheng, X., Chen, H., Zhou, Z., & Ding, R. (2017) · Frontiers of Information Technology & Electronic Engineering · doi:10.1631/fitee.1601859 · Cited by 133
Machine-learned epidemiology: real-time detection of foodborne illness at scale
Sadilek, A., Caty, S. N., DiPrete, L., Mansour, R., Schenk, T. L., Bergtholdt, M., Jha, A., Ramaswami, P., & Gabrilovich, E. (2018) · npj Digital Medicine · doi:10.1038/s41746-018-0045-1 · Cited by 161
The dynamics of information-driven coordination phenomena: A transfer entropy analysis
Borge-Holthoefer, J., Perra, N., Gonçalves, B., González-Bailón, S., Arenas, A., Moreno, Y., & Vespignani, A. (2016) · Science Advances · doi:10.1126/sciadv.1501158 · Cited by 109
Self-citationLeveraging Trends in Online Searches for Product Features in Market Response Modeling
Du, R. Y., Hu, Y., & Damangir, S. (2015) · Journal of Marketing · doi:10.1509/jm.12.0459 · Cited by 113
A Frame of Mind: Using Statistical Models for Detection of Framing and Agenda Setting Campaigns
Tsur, O., Calacci, D., & Lazer, D. M. J. (2015) · doi:10.3115/v1/p15-1157 · Cited by 142
Self-citationBig data for forecasting the impacts of global change on plant communities
Franklin, J., Serra‐Diaz, J. M., Syphard, A. D., & Regan, H. M. (2017) · Global Ecology and Biogeography · doi:10.1111/geb.12501 · Cited by 113
“In addition to data that are big in volume, big data also refers to a type of data analysis, called predictive analytics, often applied to data derived from modern digital technology, such as internet searches and social media (Lazer et al. , 2014).”
BackgroundMethodGao, J. G., Zhang, Y., & Zhou, T. (2019) · Physics Reports · doi:10.1016/j.physrep.2019.05.002 · Cited by 138
“Such smart jokes will also reduce the data authenticity and result in incorrect conclusions [709].”
BackgroundBarros, J. M., Duggan, J., & Rebholz-Schuhmann, D. (2020) · Journal of Medical Internet Research · doi:10.2196/13680 · Cited by 111
Morgenstern, J. D., Rosella, L. C., Daley, M. J., Goel, V., Schünemann, H. J., & Piggott, T. (2021) · BMC Public Health · doi:10.1186/s12889-020-10030-x · Cited by 111
“Finally, measurement error can reduce the accuracy of AI models, as was seen with Google Flu Trends when changes in Google’s search algorithms and spurious media events changed what their algorithms were measuring [60].”
MethodForecasting influenza in Hong Kong with Google search queries and statistical model fusion
Xu, Q., Gel, Y. R., Ramírez-Ramírez, L. L., Nezafati, K., Zhang, Q., & Tsui, K. (2017) · PLoS ONE · doi:10.1371/journal.pone.0176690 · Cited by 146
“, sensitivity to fickle media interest) and lack of calibration of online data [27].”
BackgroundReappraising the utility of Google Flu Trends
Kandula, S., & Shaman, J. L. (2019) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1007258 · Cited by 123
Studying health-related internet and mobile device use using web logs and smartphone records
Bach, R. L., & Wenz, A. (2020) · PLoS ONE · doi:10.1371/journal.pone.0234663 · Cited by 136
Social sensing of floods in the UK
Arthur, R., Boulton, C. A., Shotton, H., & Williams, H. T. P. (2018) · PLoS ONE · doi:10.1371/journal.pone.0189327 · Cited by 125
“Web search statistics have also been used to track infectious diseases, but are affected by changes in underlying (proprietary) search algorithms which may change outputs and confound statistical analyses [20].”
MethodThe Challenge of Big Data and Data Science
Brady, H. E. (2019) · Annual Review of Political Science · doi:10.1146/annurev-polisci-090216-023229 · Cited by 139
Chen, Z., Dong, B., Pei, Q., & Zhang, Z. (2021) · Habitat International · doi:10.1016/j.habitatint.2021.102490 · Cited by 122
Bokelmann, B., & Lessmann, S. (2019) · Tourism Management · doi:10.1016/j.tourman.2019.04.015 · Cited by 125
The promise and challenges of new datasets for accounting research
Teoh, S. H. (2018) · Accounting Organizations and Society · doi:10.1016/j.aos.2018.03.008 · Cited by 116
On the nature and types of anomalies: a review of deviations in data
Foorthuis, R. (2021) · International Journal of Data Science and Analytics · doi:10.1007/s41060-021-00265-1 · Cited by 205
Can Big Data Protect a Firm from Competition?Preprint
Lambrecht, A., & Tucker, C. E. (2015) · SSRN Electronic Journal · doi:10.2139/ssrn.2705530 · Cited by 122
A novel surveillance approach for disaster mental health
Gruebner, O., Lowe, S. R., Sykora, M., Shankardass, K., Subramanian, S. V., & Galea, S. (2017) · PLOS ONE · doi:10.1371/journal.pone.0181233 · Cited by 105
Searching for a better life: Predicting international migration with online search keywords
Böhme, M. H., Gröger, A., & Stöhr, T. (2020) · Journal of Development Economics · doi:10.1016/j.jdeveco.2019.04.002 · Cited by 106
“Despite a number of initial criticisms (Lazer et al. 2014), the literature has since grown quickly, including applications to the prediction of aggregate demand (Carriére-Swallow and Labbé 2013) and private consumption (Schmidt and Vosen 2009), the number of food stamp recipients (Fantazzini…”
BackgroundReinventing Innovation Management: The Impact of Self-Innovating Artificial Intelligence
Hutchinson, P. (2020) · IEEE Transactions on Engineering Management · doi:10.1109/tem.2020.2977222 · Cited by 135
Ernala, S. K., Birnbaum, M. L., Candan, K. A., Rizvi, A. F., Sterling, W. A., Kane, J. M., & De Choudhury, M. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300364 · Cited by 106
“A lack of contextualization in psychiatric practice [57] or theory [7] additionally reduces confidence in their construct validity—an issue recognized in prior critiques of big data approaches [13, 55].”
BackgroundLeek, J. T., & Peng, R. D. (2015) · Science · doi:10.1126/science.aaa6146 · Cited by 107
Glenn, T., & Monteith, S. (2014) · Current Psychiatry Reports · doi:10.1007/s11920-014-0523-3 · Cited by 129
Low validity of Google Trends for behavioral forecasting of national suicide rates
Tran, U. S., Andel, R., Niederkrotenthaler, T., Till, B., Ajdacic‐Gross, V., & Voracek, M. (2017) · PLoS ONE · doi:10.1371/journal.pone.0183149 · Cited by 116
“In response to the critique of the scientific community, Google Flu Trends was revised repeatedly in 2009 and in 2013 [36,38].”
BackgroundInfluentialUsing Social Media to Measure Temporal Ambient Population: Does it Help Explain Local Crime Rates?
Hipp, J. R., Bates, C., Lichman, M., & Smyth, P. (2018) · Justice Quarterly · doi:10.1080/07418825.2018.1445276 · Cited by 115
“Although there are cautions about blindly using “big data” (Lazer, Kennedy, King, and Vespignani 2014), we showed that the general pattern of tweets at a small geographic unit of a block over particular hours of the day and the days of the week may provide a relatively robust predictor of various…”
MethodDeep Learning and Big Data in Healthcare: A Double Review for Critical Beginners
Bote-Curiel, L., Muñoz-Romero, S., Guerrero-Curieses, A., & Rojo‐Álvarez, J. L. (2019) · Applied Sciences · doi:10.3390/app9112331 · Cited by 128
“Another exemplary use is Google Flu Trends [114], which was able to predict more than double the proportion of doctor visits for influenza-like illness than the Centers for Disease Control and Prevention by using surveillance reports from laboratories across the United States.”
BackgroundData science: challenges and directions
Cao, L. (2017) · Communications of the ACM · doi:10.1145/3015456 · Cited by 154
“After a comprehensive review [5, 6, 9, 10, 12, 15, 18] of hundreds of literature that directly incorporates data science in their scopes, we make the following observations of the big data buzz and data science debate:”
BackgroundInvisible Search and Online Search Engines: The Ubiquity of Search in Everyday Life
Haider, J., & Sundin, O. (2019) · Routledge eBooks · doi:10.4324/9780429448546 · Cited by 106
Integrating modelling and smart sensors for environmental and human health
Reis, S., Seto, E., Northcross, A., Quinn, N. W. T., Convertino, M., Jones, R. L., Maier, H. R., Schlink, U., Steinle, S., Vieno, M., et al. (2015) · Environmental Modelling & Software · doi:10.1016/j.envsoft.2015.06.003 · Cited by 115
Sociology as a Population Science
Goldthorpe, J. H. (2015) · Cambridge University Press eBooks · doi:10.1017/cbo9781316412565 · Cited by 174
Performance characteristics of the low-cost Plantower PMS optical sensor
He, M., Kuerbanjiang, N., & Dhaniyala, S. (2019) · Aerosol Science and Technology · doi:10.1080/02786826.2019.1696015 · Cited by 104
“…shown to be very effective in reproducing measurements accurately, the accuracy of these models is compromised when it is applied over long time period or at locations away from the calibration region (e.g., Bulot et al. 2019), previously well recognized as “big data hubris” (Lazer et al. 2014).”
BackgroundCorrecting for Selection Bias in Learning-to-rank Systems
Ovaisi, Z., Ahsan, R., Zhang, Y., Vasilaky, K., & Zheleva, E. (2020) · Proceedings of The Web Conference 2020 · doi:10.1145/3366423.3380255 · Cited by 101
“Each figure in the experiments depicts how theARRRornDCG@10 (y axis) changes when the user only observes the first k ∈ [1, 30]”
BackgroundWang, S., Ding, S., & Xiong, L. (2020) · JMIR mHealth and uHealth · doi:10.2196/19457 · Cited by 102
Sharing economy, technological innovation and carbon emissions: Evidence from Chinese cities
Gu, J. (2022) · Journal of Innovation & Knowledge · doi:10.1016/j.jik.2022.100228 · Cited by 112
Anderegg, W. R. L., & Goldsmith, G. R. (2014) · Environmental Research Letters · doi:10.1088/1748-9326/9/5/054005 · Cited by 121
“However, recent criticisms of Google search methods to examine fl u patterns highlight that algorithm transparency and careful validation with traditional data need to be better presented and utilized when using search volume data (Lazer et al 2014).”
Advances in using Internet searches to track dengue
Yang, S., Kou, S. C., Lu, F., Brownstein, J. S., Brooke, N. J., & Santillana, M. (2017) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1005607 · Cited by 115
Cross-media analysis and reasoning: advances and directions
Peng, Y., Zhu, W., Zhao, Y., Xu, C., Huang, Q., Lu, H., Zheng, Q., Huang, T., & Gao, W. (2017) · Frontiers of Information Technology & Electronic Engineering · doi:10.1631/fitee.1601787 · Cited by 108
“Furthermore, lifelong learning (Lazer et al., 2014) is the key capability of advanced intelligence systems.”
BackgroundNext-generation visitation models using social media to estimate recreation on public lands
Wood, S. A., Winder, S. G., Lia, E. H., et al. (2020) · Scientific Reports · doi:10.1038/s41598-020-70829-x · Cited by 103
Progress and Challenges in Infectious Disease Cartography
Kraemer, M. U. G., Hay, S. I., Pigott, D. M., Smith, D. L., Wint, W., & Golding, N. (2015) · Trends in Parasitology · doi:10.1016/j.pt.2015.09.006 · Cited by 118
“Understanding population interconnectedness helps identify key individuals, risk groups, or regions in a network that would benefit most from targeted interventions, such as through vaccination [54,74].”
BackgroundThe State‐of‐the‐Art in Predictive Visual Analytics
Lu, Y., Garcia, R., Hansen, B., Gleicher, M., & Maciejewski, R. (2017) · Computer Graphics Forum · doi:10.1111/cgf.13210 · Cited by 97
“One well-documented example of the complexities in predictive analytics is Google Flu Trends [LKKV14].”
BackgroundSmartphone apps in the COVID-19 pandemic
Pandit, J. A., Radin, J. M., Quer, G., & Topol, E. J. (2022) · Nature Biotechnology · doi:10.1038/s41587-022-01350-x · Cited by 101
“Many of these apps and web dashboards (Table 1) harnessed the power of big data, but also demonstrated potential sources of bias 41 .”
BackgroundImprecision farming? Examining the (in)accuracy and risks of digital agriculture
Виссер, О., Sippel, S. R., & Thiemann, L. (2021) · Journal of Rural Studies · doi:10.1016/j.jrurstud.2021.07.024 · Cited by 140
“laboratories across the US showed, however, that Google Flu Trends later on predicted more than twice the number of flu-related doctor’s visits than occurred, among others due to the continuous change in the configuration of the search algorithm (Lazer et al., 2014).”
BackgroundCavallaro, G., Riedel, M., Richerzhagen, M., Benediktsson, J. A., & Plaza, A. (2015) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2015.2458855 · Cited by 101
“, C = [1, 10, 100, 1000, 10 000] and γ = [2, 4, 8, 16, 32].”
Tree‐Based Models for Political Science Data
Montgomery, J., & Olivella, S. (2018) · American Journal of Political Science · doi:10.1111/ajps.12361 · Cited by 122
“However, it remains a task for the researcher to determine whether inputs or model outputs are sensible or nonsense (see Lazer et al., 2014) Likewise, tree-based methods are no replacement for research design. While the models we discussed allow researchers to more easily discover important subtleties in large dataset…”
BackgroundBig Data in Health Care: Applications and Challenges
Hong, L., Luo, M., Wang, R., Lu, P., Lu, W., & Lu, L. J. (2018) · Data and Information Management · doi:10.2478/dim-2018-0014 · Cited by 158
Adaptive nowcasting of influenza outbreaks using Google searches
Preis, T., & Moat, H. S. (2014) · Royal Society Open Science · doi:10.1098/rsos.140095 · Cited by 110
“In addition, concerns about structural changes affecting how Google presents search results to users have been raised, leading to questions about the continued usefulness of this approach [25,26].”
BackgroundAraz, Ö. M., Bentley, D., & Muelleman, R. L. (2014) · The American Journal of Emergency Medicine · doi:10.1016/j.ajem.2014.05.052 · Cited by 112
Mining Twitter to Assess the Public Perception of the “Internet of Things”
Bian, J., Yoshigoe, K., Hicks, A., Yuan, J., He, Z., Xie, M., Guo, Y. M., Prosperi, M., Salloum, R. G., & Modave, F. P. (2016) · PLoS ONE · doi:10.1371/journal.pone.0158450 · Cited by 110
Ethics and Epistemology in Big Data Research
Lipworth, W. L., Mason, P. H., Kerridge, I., & Ioannidis, J. P. A. (2017) · Journal of Bioethical Inquiry · doi:10.1007/s11673-017-9771-3 · Cited by 117
“2015; Zulman, Shah, and Verghese 2016); & difficulty analysing and interpreting enormous amounts of data that are unstructured and beset with errors, Bnoise,^ and biases (Hemkens, ContopoulosIoannidis, and Ioannidis 2016c; Khoury and Ioannidis 2014; Ioannidis 2013); & highly variable levels of reproducibility and repl…”
BackgroundForecast of Dengue Cases in 20 Chinese Cities Based on the Deep Learning Method
Xu, J., Xu, K., Li, Z., Meng, F., Tu, T., Xu, L., & Liu, Q. (2020) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph17020453 · Cited by 125
Hossain, F. A., Lover, A. A., Corey, G. A., Reich, N. G., & Rahman, T. (2020) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3381014 · Cited by 147
“For example, Google Flu Trends (GFT) faced significant criticisms for dramatically overestimated prediction and lack of reproducibility and reliability during the 2012-13 flu season [25].”
BackgroundArtificial intelligence in higher education: a bibliometric analysis and topic modeling approach
Maphosa, V., & Maphosa, M. (2023) · Applied Artificial Intelligence · doi:10.1080/08839514.2023.2261730 · Cited by 100
“Traditional data management techniques cannot process large heterogeneous data sets; hence, big data analytics is applied in acquiring, analyzing, and evaluating complex and large data sets (Lazer et al. 2014).”
BackgroundRogge, N., Agasisti, T., & De Witte, K. (2017) · Public Policy and Administration · doi:10.1177/0952076716687355 · Cited by 98
“An illustration of a limitation to big data use in policy decision making was discussed given by Lazer et al. (2014).”
BackgroundUsing electronic health records and Internet search information for accurate influenza forecasting
Yang, S., Santillana, M., Brownstein, J. S., Gray, J., Richardson, S., & Kou, S. C. (2017) · BMC Infectious Diseases · doi:10.1186/s12879-017-2424-7 · Cited by 107
“For instance, Google Flu Trends was criticized [9, 10, 42 – 45] not because people questioned the value of online search data [27, 46], but because Google Flu Trends produced mis-leading forecasts in both 2009 and 2012 when it was needed most, due to its sub-optimal method to process the valuable…”
MethodWenzel, R., & Van Quaquebeke, N. (2018) · Organizational Research Methods · doi:10.1177/1094428117718627 · Cited by 98
Passos, I. C., Ballester, P. L., Barros, R. C., Librenza‐Garcia, D., Mwangi, B., Birmaher, B. B., Brietzke, E. M., Hájek, T., López‐Jaramillo, C., Mansur, R. B., et al. (2019) · Bipolar Disorders · doi:10.1111/bdi.12828 · Cited by 111
Behavior Revealed in Mobile Phone Usage Predicts Credit Repayment
Björkegren, D., & Grissen, D. (2019) · The World Bank Economic Review · doi:10.1093/wber/lhz006 · Cited by 191
“The performance of machine learning methods can deteriorate if the underlying environment shifts over time (Butler, 2013; Lazer et al., 2014).”
Fountain, J. E. (2021) · Government Information Quarterly · doi:10.1016/j.giq.2021.101645 · Cited by 151
Shifting Concepts of Value: Designing Algorithmic Decision-Support Systems for Public Services
Holten Møller, N., Shklovski, I., & Hildebrandt, T. T. (2020) · Proceedings of the 11th Nordic Conference on Human-Computer Interaction: Shaping Experiences, Shaping Society · doi:10.1145/3419249.3420149 · Cited by 97
“Algorithmic decision-support systems are considered in safety-critical domains such as diagnostics and critical care [15], crisis management in due time [39], and containment of disease [27].”
BackgroundTracking employment shocks using mobile phone data
Toole, J. L., Lin, Y.-R., Muehlegger, E., Shoag, D., González, M. C., & Lazer, D. (2015) · Journal of The Royal Society Interface · doi:10.1098/rsif.2015.0185 · Cited by 96
Self-citationThe Interactive Visualization Gap in Initial Exploratory Data Analysis
Batch, A., & Elmqvist, N. (2017) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2017.2743990 · Cited by 115
Feng, Y., & Sester, M. (2018) · ISPRS International Journal of Geo-Information · doi:10.3390/ijgi7020039 · Cited by 107
“However, overestimation of such events may sometimes happen [10].”
BackgroundOpen Innovation Engineering—Preliminary Study on New Entrance of Technology to Market
Yun, J. J., Kim, D., & Yan, M.-R. (2020) · Electronics · doi:10.3390/electronics9050791 · Cited by 102
“With the Google Flue Trends (GFT) announcement in February 2013, many people have asserted that there are enormous scientific possibilities in big data, and one can ignore foundational issues of measurement and construct validity and reliability and dependencies among data [50].”
BackgroundThe internet as a data source for advancement in social sciences
Askitas, N., & Zimmermann, K. F. (2015) · International Journal of Manpower · doi:10.1108/ijm-02-2015-0029 · Cited by 129
“While this study has received large attention, it was also found that the structure identified between the searches, the defined keywords and the individual behavior was under change (Lazer et al., 2014).”
BackgroundClarifying questions about “risk factors”: predictors versus explanation
Schooling, C. M., & Jones, H. E. (2018) · Emerging Themes in Epidemiology · doi:10.1186/s12982-018-0080-z · Cited by 111
Lessons for artificial intelligence from the study of natural stupidity
Rich, A. S., & Gureckis, T. M. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0038-z · Cited by 94
A reality check(list) for digital methods
Venturini, T., Bounegru, L., Gray, J., & Rogers, R. (2018) · New Media & Society · doi:10.1177/1461444818769236 · Cited by 158
“Others have already discussed various 'perils' (Bollier & Firestone, 2010), 'provocations' (Boyd & Crawford, 2011), 'challenges' (Rieder & Röhle, 2012), 'problems' (Marcus & Davis, 2014), 'traps' (Lazer et al. 2014) and 'misunderstandings' (Venturini et al. 2014) of digital research.”
Dataset level explanation of heat demand forecasting ANN with SHAP
Białek, J., Bujalski, W., Wojdan, K., Guzek, M., & Kurek, T. (2022) · Energy · doi:10.1016/j.energy.2022.125075 · Cited by 112
Big data business models: Challenges and opportunities
Schroeder, R. (2016) · Cogent Social Sciences · doi:10.1080/23311886.2016.1166924 · Cited by 128
“Recent failures in Google’s flu prediction (Lazer, Kennedy, King, & Vespignani, 2014) are a case in point, and also highlight the general lack of transparency about how the data were arrived at, which presents a barrier to replicability.”
BackgroundComputational Exposure Science: An Emerging Discipline to Support 21st-Century Risk Assessment
Egeghy, P. P., Sheldon, L. S., Isaacs, K., Özkaynak, H., Goldsmith, M., Wambaugh, J. F., Judson, R., & Buckley, T. J. (2015) · Environmental Health Perspectives · doi:10.1289/ehp.1509748 · Cited by 96
Harnessing machine learning for development of microbiome therapeutics
McCoubrey, L. E., Elbadawi, M., Orlu, M., Gaisford, S., & Basit, A. W. (2021) · Gut Microbes · doi:10.1080/19490976.2021.1872323 · Cited by 93
“Next, data in repositories are commonly ‘messy’, containing missing values, unequal scaling, and background noise from poorly performed experiments or incorrect data labeling.(45,46) Clearly, without rectification messy data will adversely impact the predictive power of ML algorithms.”
BackgroundOnline social integration is associated with reduced mortality risk
Hobbs, W., Burke, M., Christakis, N. A., & Fowler, J. H. (2016) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1605554113 · Cited by 106
Lü, G., Chen, M., Yuan, L., Zhou, L., Wen, Y., Wu, M., Hu, B., Yu, Z., Yue, S., & Sheng, Y. (2017) · International Journal of Digital Earth · doi:10.1080/17538947.2017.1374477 · Cited by 90
“…on the interactive channels of VGEs (e.g. Batty 2008; Chen et al. 2008, 2013b; Che, Lin, and Hu 2011; Lai et al. 2011; Priestnall et al. 2012) and collaborative work modes based on VGEs (e.g. Zhu et al. 2007; Hui et al. 2010; Xu et al. 2011; Chen et al. 2012; Li et al. 2015; Zhu et al. 2016).”
BackgroundBauder, R. A., & Khoshgoftaar, T. M. (2018) · Health Information Science and Systems · doi:10.1007/s13755-018-0051-3 · Cited by 119
“The quantity of the information in the healthcare industry today continues to grow with the adoption of EHRs and electronic insurance claims records [7], making big data fairly ubiquitous for data modeling and analytics in the healthcare industry [6].”
BackgroundBaquero, O. S., Santana, L. M. R., & Neto, F. C. (2018) · PLoS ONE · doi:10.1371/journal.pone.0195065 · Cited by 94
Big data-informed energy efficiency assessment of China industry sectors based on K-means clustering
Liu, G., Yang, J. G., Hao, Y., & Zhang, Y. (2018) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2018.02.129 · Cited by 104
Random-Forest-Bagging Broad Learning System With Applications for COVID-19 Pandemic
Zhan, C., Zheng, Y., Zhang, H., & Wen, Q. (2021) · IEEE Internet of Things Journal · doi:10.1109/jiot.2021.3066575 · Cited by 96
“However, a large amount of media data inevitably contains a lot of false information or noise, which has to be filtered to create a training set [40].”
BackgroundTheory-driven or process-driven prediction? Epistemological challenges of big data analytics
Elragal, A. A., & Klischewski, R. (2017) · Journal Of Big Data · doi:10.1186/s40537-017-0079-2 · Cited by 109
“For example, a Google Flu prediction error in February 2013 resulted in doubling the proportion of doctor visits for influenza-like illness in the USA [22].”
BackgroundUsing Google Trends and ambient temperature to predict seasonal influenza outbreaks
Zhang, Y., Bambrick, H., Mengersen, K. L., et al. (2018) · Environment International · doi:10.1016/j.envint.2018.05.016 · Cited by 89
“For instance, Google Flu Trends (GFT) predicted 38 more than double the peak of influenza-like illness (ILI) cases than the Centers for Disease 39 Control and Prevention (CDC) in 2013 (11).”
BackgroundMethodTowards big data driven construction industry
Li, F., Laili, Y., Chen, X., Lou, Y., Wang, C., Yang, H., Gao, X., & Han, H. (2023) · Journal of Industrial Information Integration · doi:10.1016/j.jii.2023.100483 · Cited by 128
A Multi-Level Geographical Study of Italian Political Elections from Twitter Data
Caldarelli, G., Chessa, A., Pammolli, F., Pompa, G., Puliga, M., Riccaboni, M., & Riotta, G. (2014) · PLoS ONE · doi:10.1371/journal.pone.0095809 · Cited by 96
Gentzkow, M. A., Kelly, B. T., & Taddy, M. A. (2017) · National Bureau of Economic Research · doi:10.3386/w23276 · Cited by 130
Can internet searches forecast tourism inflows?
Artola, C., Hernández, F. P., & de Pedraza, P. (2015) · International Journal of Manpower · doi:10.1108/ijm-12-2014-0259 · Cited by 103
Forecasting sales using online review and search engine data: A method based on PCA–DSFOA–BPNN
Zhang, C., Tian, Y., & Fan, Z. (2021) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2021.07.010 · Cited by 98
The application of Big Data in medicine: current implications and future directions
Austin, C. O., & Kusumoto, F. M. (2016) · Journal of Interventional Cardiac Electrophysiology · doi:10.1007/s10840-016-0104-y · Cited by 103
AI-enhanced collective intelligence
Cui, H., & Yasseri, T. (2024) · Patterns · doi:10.1016/j.patter.2024.101074 · Cited by 142
“Smart home AI assistants such as Alexa and Siri can help manage smart home devices, play music, provide weather updates, answer questions, and assist with daily routines.”
Big data in healthcare: Challenges and opportunities
Asri, H., Mousannif, H., Moatassime, H. A., & Noël, T. (2015) · International Conference on Cloud Computing Technologies and Applications (CloudTech) · doi:10.1109/cloudtech.2015.7337020 · Cited by 119
“Google, for instance, has applied algorithms of data mining and machine learning to detect influenza epidemics through search queries [22][23].”
Method
Metadata
- Authors
- Lazer, D., Kennedy, R., King, G., & Vespignani, A.
- Affiliations
- Northeastern University (US)
- Harvard University (US)
- Corresponding author
- Lazer, David
- Venue
- Science
- Published
- 2014-03-14
- Volume
- 343
- Issue
- 6176
- Pages
- 1203-1205
- ISSN
- 0036-8075 (ISSN-L)
- 1095-9203
- OpenAlex
- W2068181924
- Semantic Scholar
- 00d681cfff7f0224ace607e515f4ed510e792df9
- Cited by
- 2,523
- References
- 30
- Citations by year
- 2026: 113 · 2025: 101 · 2024: 121 · 2023: 150 · 2022: 168 · 2021: 229 · 2020: 273 · 2019: 261 · 2018: 272 · 2017: 249 · 2016: 271 · 2015: 219 · 2014: 91
- FWCI
- 208.16
- Type
- journal article
- Language
- en
- Open access
- green
- dash.harvard.edu
- Licence
- cc-by
- Locations
- Science (publishedVersion)
- PubMed (publishedVersion)
- Digital Access to Scholarship at Harvard (DASH) (Harvard University) (submittedVersion, cc-by)
- Topics
- Health Sciences › Medicine › Epidemiology › Data-Driven Disease Surveillance
- Social Sciences › Decision Sciences › Information Systems and Management › Big Data Technologies and Applications
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- Google Flu Trends, influenza surveillance, big data, prediction error, data quality
- MeSH terms
- Centers for Disease Control and Prevention, U.S., Data Collection, Data Interpretation, Statistical, Disease Outbreaks, Humans, Influenza, Human, Office Visits, Prognosis, United States, Social Media
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.