The Parable of Google Flu: Traps in Big Data Analysis
Open accessThe DOI matches its Crossref record; 30 of the 31 references it cites were found in a registry or an index; 5 of the 8 sources its claims cite were checked for support.
Abstract
Large errors in flu prediction were largely avoidable, which offers lessons for the use of big data.
Overview
CoChat Summary
TL;DR
The paper uses the failure of Google Flu Trends (GFT) to illustrate two traps in big data analysis: “big data hubris” and algorithm dynamics. It argues that GFT’s large flu-prediction errors were largely avoidable and that big data should supplement, not replace, traditional data and methods.
Key findings
- In February 2013, GFT predicted more than double the proportion of doctor visits for influenza-like illness (ILI) than the CDC; from 21 August 2011 to 1 September 2013, GFT reported overly high flu prevalence in 100 out of 108 weeks, and it overshot the 2011–2012 season by more than 50%.
- GFT errors were not random: they showed temporal autocorrelation and seasonality, meaning last week’s errors predicted this week’s errors and error direction/magnitude varied with time of year.
- Simple alternatives outperformed GFT: out-of-sample mean absolute error was 0.486 for GFT, 0.311 for lagged CDC, and 0.232 for combined GFT and CDC; all differences were statistically significant at P < 0.05. Even 3-week-old CDC data did better than GFT at projecting current flu prevalence.
- The authors attribute GFT’s problems to overfitting (initial GFT matched 50 million search terms to 1152 data points and missed the nonseasonal 2009 H1N1 pandemic) and to “algorithm dynamics,” including Google search algorithm changes and user behavior changes.
Methods
- Design: policy forum/case study comparing GFT estimates with CDC ILI surveillance data and alternative forecasting models.
- Data/sample: GFT and CDC flu prevalence estimates, including the period from 21 August 2011 to 1 September 2013; the initial GFT methodology reportedly fit 50 million search terms to 1152 data points.
- Main analysis: comparison of GFT alone, lagged CDC data with 52-week seasonality variables, and a combined GFT + CDC model using lagged CDC estimates, lagged GFT error, and seasonality variables; reported mean absolute error and statistical significance; Google Correlate used to examine correlated search terms. Supplementary materials contain data and methods.
Limitations
- The paper states that GFT never documented the 45 search terms used, released examples appear misleading, and Google Correlate is limited to national-level data whereas GFT was developed using regional correlations; therefore the original GFT analyses cannot be replicated from the information provided.
- The paper states that Google cannot make its full data available to outsiders because of privacy issues, though it argues there is no such constraint on derivative, aggregated data.
- Evident from the text: this is a commentary/policy forum rather than a full empirical article, with key data and methods placed in supplementary materials; the quantitative comparison depends on GFT and CDC data and model specifications not fully detailed in the main text.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
2 Valid · 2 Not valid · 2 Not checked · 8 Own results and interpretations
Some cited sources support this claim and some don’t — read each source’s verdict below.
In the paper
“This ad hoc method of throwing out peculiar search terms failed when GFT completely missed the nonseasonal 2009 infl uenza A–H1N1 pandemic ( 2, 14).”
What the sources say
- Partly supportedcites [2] matched to “Reassessing Google Flu Trends Data for Detection of Seasonal and Pandemic Influenza: A Comparative Epidemiological Study at Three Geographic Scales” (2013) sourceExperimental — accuracy not yet measured
The Discussion (page unavailable) says the original GFT model missed the spring 2009 pandemic wave, supporting the claim about its failure to detect the nonseasonal outbreak. It does not say GFT completely missed the entire 2009 pandemic. The Introduction describes automated selection of the original model’s search terms, not an ad hoc method of throwing out peculiar terms. (LLM, conf=0.86)
- Disputedcites [14] S. Cook et al., PLOS ONE 6, e23610 (2011). sourceExperimental — accuracy not yet measured
The Abstract (Methodology/Principal Findings) says the original GFT model underestimated the magnitude of illness during the 2009 H1N1 pandemic, not that it completely missed it. The Abstract (Conclusions) says both models performed well during the pandemic, although the updated model performed better. The Discussion notes that pandemic-specific queries were excluded, but does not describe this as an ad hoc method of throwing out peculiar terms. (LLM, conf=0.88)
Checked by a reasoning model — may be wrong.
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“Oddly, the few search terms offered in the papers ( 14) do not seem to be strongly related with either GFT or the CDC data (SM)—we surmise that the authors felt an unarticulated need to cloak the actual search terms identifi ed.”
What the sources say
- Disputedcites [14] S. Cook et al., PLOS ONE 6, e23610 (2011). sourceExperimental — accuracy not yet measured
The Results section (‘Search Behavior During pH1N1’; page unavailable) says that all three search queries shown in Figure 3 closely tracked CDC ILINet data before the pandemic, and ‘symptoms of flu’ continued to do so during it. The paper does report that the bronchitis and pneumonia queries underestimated ILINet data during the pandemic, but that does not support the broad claim that the offered terms were not strongly related to the data. (LLM, conf=0.87)
Checked by a reasoning model — may be wrong.
hedgedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“The Internet has opened the way for improving standard surveys, experiments, and health reporting ( 35).”
What the sources say
- Partly supportedcites [35] A. J. Berinsky et al., Polit. Anal. 20, 351–368 (2012). sourceExperimental — accuracy not yet measured
Section 2 describes using an Internet platform to run surveys with embedded experiments, and the Conclusion says MTurk can make experimental research easier and less expensive. The paper does not address improvements to health reporting; the disease scenario in section 4.3.2 is a framing experiment, not health reporting. Page numbers are unavailable in our copy. (LLM, conf=0.88)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Although not widely reported until 2013, the new GFT has been persistently overestimating flu prevalence for a much longer time.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“From 21 August 2011 to 1 September 2013, GFT reported overly high fl u prevalence 100 out of 108 weeks.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
3 more claims
Create a free account to see all 14 claims and every source verdict.
- It may state two separate claims as one. 5
- Probably not a substantial claim. 2
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Google Trends predicted more than twice the number of influenza-like illnesses than the CDC in February 2013 46 .”
From Unrepresentative big surveys significantly overestimated US vaccine uptake (2021) · cites this paper as [46]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This was one of the main reasons behind the failure of Google Flu Trends, since the data used for training the model in the first few years was not representative of the trends in subsequent years [15].”
From Theory-Guided Data Science: A New Paradigm for Scientific Discovery from Data (2017) · cites this paper as [15]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Social data research often relies on counting entities. These entities can be users, links, or messages, and the description is a summary of these counts (Lazer et al., 2014; Salganik, 2017). Yet, simple counts can mislead if it is unclear what is counted and how.”
From Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019) · cites this paper as Lazer et al., 2014
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This unintelligibility is also subject to functional biases (section 4.1), as it depends on the mechanisms available on each social platform (e.g., having a like button, but not a dislike one), as well as to variations in platform algorithms and mechanisms in response to users actions (Lazer et al., 2014).”
From Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019) · cites this paper as Lazer et al., 2014
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“It also includes temporal validity, which captures to what extent constructs change over time (Howison et al., 2011) and may invalidate previous conclusions about societal and/or platform phenomena; e.g., see the case of Google Flu Trends (Lazer et al., 2014).”
From Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019) · cites this paper as Lazer et al., 2014
6 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 14 MARCH 2014 VOL 343 SCIENCE www.sciencemag.org 1204
“The 2009 version of GFT has weathered other media panics related to the fl u, including the 2005–2006 influenza A/H5N1 ("bird flu") outbreak”
“GFT engineers updated the algorithm in 2009, and this model has run ever since”
Citations
Cites
boyd, D., & Crawford, K. (2012) · Information, Communication & Society · doi:10.1080/1369118x.2012.678878 · Cited by 5,479
Twitter mood predicts the stock market
Bollen, J., Mao, H., & Zeng, X. (2011) · Journal of Computational Science · doi:10.1016/j.jocs.2010.12.007 · Cited by 5,206
Detecting influenza epidemics using search engine query data
Ginsberg, J., Mohebbi, M. H., Patel, R. S., Brammer, L., Smolinski, M. S., & Brilliant, L. (2009) · Nature · doi:10.1038/nature07634 · Cited by 4,295
Cohen, D. (1998) · The Women's Review of Books · doi:10.2307/4022859 · Cited by 4,261
Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk
Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012) · Political Analysis · doi:10.1093/pan/mpr057 · Cited by 4,008
Big Data : The Management Revolution
マカフィー, ア., ブリニョルフソン, エ., & 有賀, 裕. (2013) · Harvard business review · Cited by 3,549
Lazer, D., Pentland, A., Adamic, L., Aral, S., Barabási, A.-L., Brewer, D., Christakis, N., Contractor, N., Fowler, J., Gutmann, M., et al. (2009) · Science · doi:10.1126/science.1167742 · Cited by 3,365
Self-citationPredicting Elections with Twitter: What 140 Characters Reveal about Political Sentiment
Tumasjan, A., Sprenger, T., Sandner, P., & Welpe, I. (2010) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v4i1.14009 · Cited by 2,649
Multiscale mobility networks and the spatial spreading of infectious diseases
Balcan, D., Colizza, V., Gonçalves, B., Hu, H., Ramasco, J. J., & Vespignani, A. (2009) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0906910106 · Cited by 1,380
Self-citationPredicting consumer behavior with web search
Goel, S., Hofman, J. M., Lahaie, S., Pennock, D. M., & Watts, D. J. (2010) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1005962107 · Cited by 636
Research works printed but not linked: 7
- A. McAfee, E. Brynjolfsson, Harv. Bus. Rev. 90, 60 (2012).
- AAAI Conference on Weblogs and Social Media, Atlanta, Georgia, 11 to 15 July 2010 (Association for Advancement of Artifi cial Intelligence, 2010), pp. 178.
- P. Copeland et al., Int. Soc. Negl. Trop. Dis. 2013, 3
- I. M. Hall et al., Epidemiol. Infect. 135, 372 (2007).
- E. Mustafaraj, P. Metaxas, in Proceedings of the Web- Sci10, Raleigh, NC, 26 and 27 April 2010 (Web Science Trust, 2010); http://journal.webscience.org/317/.
- J. Ratkiewicz et al., in Proceedings of 5th International AAAI Conference on Weblogs and Social Media, San Francisco, CA, 7 to 11 August 2011 (AAAI, 2011), p. 297–304.
- 2013; http://chronicle.com/article/Researchers-Struggle- to-Secure/141591/.
Cited by
Items 401–500 of 2,513
Osthus, D., Gattiker, J., Priedhorsky, R., & Valle, S. Y. D. (2018) · Bayesian Analysis · doi:10.1214/18-ba1117 · Cited by 64
“Such approaches can provide improved prediction but may ignore known features of disease epidemiology and can go astray for periods of time, as was the case with Google Flu Trends (Lazar et al. (2014)).”
BackgroundJin, D.-H., & Kim, H. (2018) · Sustainability · doi:10.3390/su10103778 · Cited by 80
“emphasized in The Parable of Google Flu: Traps in Big Data Analysis, the essential element is turning data into valuable information, not the quantity of data or new data itself [43].”
BackgroundA Scalable Framework to Detect Personal Health Mentions on Twitter
Yin, Z., Fabbri, D., Rosenbloom, S. T., & Malin, B. (2015) · Journal of Medical Internet Research · doi:10.2196/jmir.4305 · Cited by 78
Behavioral insights on big data: using social media for predicting biomedical outcomes
Young, S. D. (2014) · Trends in Microbiology · doi:10.1016/j.tim.2014.08.004 · Cited by 63
A “big data” approach to HIV epidemiology and prevention
Young, S. D. (2014) · Preventive Medicine · doi:10.1016/j.ypmed.2014.11.002 · Cited by 63
“There has already been criticism about some of the current methods of using technology data for monitoring health outcomes, including the reliability and validity of the data and methods (Lazer et al., 2014), making it important to highlight issues that need to be addressed prior to using big data from technologies fo…”
Measuring economic activity in China with mobile big data
Dong, L., Chen, S., Cheng, Y., Wu, Z., Li, C., & Wu, H. (2017) · EPJ Data Science · doi:10.1140/epjds/s13688-017-0125-5 · Cited by 72
Social Monitoring for Public Health
Paul, M. J., & Dredze, M. H. (2017) · Synthesis lectures on information concepts, retrieval, and services · doi:10.1007/978-3-031-02311-8 · Cited by 76
“The entire field came under scrutiny after it was noticed in a widely publicized study that Google Flu Trends, a popular digital flu monitoring system that we discuss throughout this book, had started performing inaccurately and severely misfired in a recent year [Lazer et al., 2014b].”
BackgroundFormalizing Trust in Artificial Intelligence: Prerequisites, Causes and Goals of Human Trust in AIPreprint
Jacovi, A., Marasović, A., Miller, T., & Goldberg, Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.07487 · Cited by 81
“the original classification task [42, 46].”
Evaluating Social Media Networks in Medicines Safety Surveillance: Two Case Studies
Coloma, P. M., Becker, B. F. H., Sturkenboom, M., van Mulligen, E. M., & Kors, J. A. (2015) · Drug Safety · doi:10.1007/s40264-015-0333-5 · Cited by 61
Flu Gone Viral: Syndromic Surveillance of Flu on Twitter Using Temporal Topic Models
Chen, L., Hossain, K. S. M. T., Butler, P., Ramakrishnan, N., & Prakash, B. A. (2014) · Proceedings · doi:10.1109/icdm.2014.137 · Cited by 70
“[17] provide many reasons for Google Flu Trend’s lackluster performance.”
BackgroundSyndromic surveillance of Flu on Twitter using weakly supervised temporal topic models
Chen, L., Hossain, K. S. M. T., Butler, P., Ramakrishnan, N., & Prakash, B. A. (2015) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-015-0434-x · Cited by 70
Using Facebook Ads Audiences for Global Lifestyle Disease Surveillance
Araújo, M., Mejova, Y., Weber, I. G., & Benevenuto, F. (2017) · ACM Web Science Conference (WebSci) · doi:10.1145/3091478.3091513 · Cited by 62
“…as abortion [16] and vaccines [19], others use the data to now-cast diseases [8], while others use search logs, most notably using Google Trends, to predict seasonal flu [6]. large errors in flu prediction from Google Flu Trends offered interesting lessons for the use of this kind of data [12].”
BackgroundHuman-Computer Interaction and Collective Intelligence
Bigham, J. P., Bernstein, M. S., & Adar, E. (2018) · Figshare · doi:10.1184/r1/6470123 · Cited by 61
O'Brien, D. T. (2018) · Journal of Research in Crime and Delinquency · doi:10.1177/0022427818806040 · Cited by 68
Ilan, Y. (2020) · Frontiers in Digital Health · doi:10.3389/fdgth.2020.569178 · Cited by 73
“AI may give undue importance to spurious correlations within past data, as exemplified by a failure to predict seasonal prevalence of influenza (108).”
BackgroundSocial Media as a Catalyst for Policy Action and Social Change for Health and Well-Being: Viewpoint
Yeung, D. (2018) · Journal of Medical Internet Research · doi:10.2196/jmir.8508 · Cited by 73
“Suggesting caution in the use of such data, one analysis suggested that the predictive power of Google Flu Trends’s model was significantly overstated [35] but could be useful in conjunction with other flu-tracking data.”
MethodWu, H., Cai, Y., Wu, Y., Ren, Z., Li, Q., Jing, Z., Lin, D., & Li, Y. (2017) · BioScience Trends · doi:10.5582/bst.2017.01035 · Cited by 65
“Although it achieved an impressive accuracy of 97% in its early stage, Google Flu Trends team no longer published current estimates because of its drop in accuracy in the interval of 2011-2013 (9,10).”
BackgroundFold-stratified cross-validation for unbiased and privacy-preserving federated learning
Bey, R., Goussault, R., Grolleau, F., Benchoufi, M., & Porcher, R. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaa096 · Cited by 70
Modelling microbial infection to address global health challenges
Fitzpatrick, M. C., Bauch, C. T., Townsend, J. P., & Galvani, A. P. (2019) · Nature Microbiology · doi:10.1038/s41564-019-0565-8 · Cited by 63
Forecasting in Light of Big Data
Hosni, H., & Vulpiani, A. (2017) · Philosophy & Technology · doi:10.1007/s13347-017-0265-3 · Cited by 73
“The extent to which those promises are likely to be fulfilled is currently a matter for debate across a number of disciplines [1, 15, 22, 17, 3, 23], while some early success stories rather quickly turned into macroscopic failures [16].”
BackgroundNews media versus FRED‐MD for macroeconomic forecasting
Ellingsen, J., Larsen, V. H., & Thorsrud, L. A. (2021) · Journal of Applied Econometrics · doi:10.1002/jae.2859 · Cited by 79
“by Medeiros et al. (2019). Whereas they use the FRED-MD dataset to compare ML models for inflation forecasting, we focus on the (textual) news versus hard economic data dimension when forecasting National Account Statistics.”
BackgroundMethodInfluentialEstablishing and evaluating trustworthy AI: overview and research challenges
Kowald, D., Scher, S., Pammer-Schindler, V., Müllner, P., Waxnegger, K., Demelius, L., Fessl, A., Toller, M., Mendoza Estrada, I. G., Šimić, I., et al. (2024) · Frontiers in Big Data · doi:10.3389/fdata.2024.1467222 · Cited by 59
Theory In, Theory Out: The Uses of Social Theory in Machine Learning for Social Science
Radford, J., & Joseph, K. (2020) · Frontiers in Big Data · doi:10.3389/fdata.2020.00018 · Cited by 58
“, 2012) or aggregated search data from Google Trends to predict the prevalence of the flu (Lazer et al., 2014).”
BackgroundBuilding truths in AI: Making predictive algorithms doable in healthcare
Henriksen, A., & Bechmann, A. (2020) · Information, Communication & Society · doi:10.1080/1369118x.2020.1751866 · Cited by 58
“…us of the wide use of AI predictive modeling not only to detect disease and forecast viruses as with IBM Watson and Google Flu Trend (see e.g., Lazer et al., 2014) but also to predict the risk of (re)admissions and the actions that should be taken given the condition a patient is in (see…”
MethodSmith, G. (2018) · doi:10.1093/oso/9780198824305.001.0001 · Cited by 81
Nationwide prediction of type 2 diabetes comorbidities
Dworzynski, P., Aasbrenn, M., Rostgaard, K., Melbye, M., Gerds, T. A., Hjalgrim, H., & Pers, T. H. (2020) · Scientific Reports · doi:10.1038/s41598-020-58601-7 · Cited by 58
Early identification of adverse drug reactions from search log data
White, R. W., Wang, S., Pant, A., Harpaz, R., Shukla, P., Sun, W., DuMouchel, W., & Horvitz, E. (2015) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2015.11.005 · Cited by 64
Text and data mining of social media to map wildlife recreation activity
Monkman, G. G., Kaiser, M. J., & Hyder, K. (2018) · Biological Conservation · doi:10.1016/j.biocon.2018.10.010 · Cited by 66
Toward Value-Based Healthcare through Interactive Process Mining in Emergency Rooms: The Stroke Case
Ibáñez-Sánchez, G., Fernández-Llatas, C., Martínez-Millana, A., Celda, A., Mandingorra, J., Aparici-Tortajada, L., Valero-Ramón, Z., Muñoz-Gama, J., Sepúlveda, M., Rojas, E., et al. (2019) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph16101783 · Cited by 62
“However, despite the great advantages in the application of Big Data to healthcare, health professionals are somewhat suspicious of how current expert clinical knowledge will be integrated into automatically learned clinical models [17].”
BackgroundEbert, T., Götz, F. M., Gladstone, J. J., Müller, S. R., & Matz, S. (2021) · Journal of Personality and Social Psychology · doi:10.1037/pspp0000344 · Cited by 66
Jun, S., Yoo, H. S., & Lee, J. (2021) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2021.120592 · Cited by 73
“However, researchers have also steadily pointed out problems in the accuracy of GFT and some of the fundamental limitations of predicting infectious diseases using RSV (Butler, 2013; Lazer et al., 2014; Olson et al., 2013; Pollett et al., 2016).”
BackgroundInternet use that reproduces educational inequalities: Evidence from big data
Zhang, M. (2015) · Computers & Education · doi:10.1016/j.compedu.2015.08.007 · Cited by 72
Google's MIDAS Touch: Predicting UK Unemployment with Internet Search Data
Smith, P. (2015) · Journal of Forecasting · doi:10.1002/for.2391 · Cited by 77
Tsay, D., & Patterson, C. (2018) · Circulation · doi:10.1161/circulationaha.118.031734 · Cited by 64
A geologically-constrained deep learning algorithm for recognizing geochemical anomalies
Zhang, C., Zuo, R., Xiong, Y., Zhao, X.-F., & Zhao, K. (2022) · Computers & Geosciences · doi:10.1016/j.cageo.2022.105100 · Cited by 58
Addressing bias in artificial intelligence for public health surveillance
Flores, L., Kim, S., & Young, S. D. (2023) · Journal of Medical Ethics · doi:10.1136/jme-2022-108875 · Cited by 75
Deep learning based multimodal urban air quality prediction and traffic analytics
Hameed, S., Islam, A., Ahmad, K., Belhouari, S. B., Qadir, J., & Al‐Fuqaha, A. (2023) · Scientific Reports · doi:10.1038/s41598-023-49296-7 · Cited by 80
AI for the people? Embedding AI ethics in HR and people analytics projects
Bar-Gil, O., Ron, T., & Czerniak, O. (2024) · Technology in Society · doi:10.1016/j.techsoc.2024.102527 · Cited by 100
Integrating predictive analytics and social media
Lu, Y., Kruger, R., Thom, D., Wang, F., Koch, S., Ertl, T., & Maciejewski, R. (2014) · IEEE Conference on Visual Analytics Science and Technology (VAST) · doi:10.1109/vast.2014.7042495 · Cited by 66
“[25], there is a need for tools that can improve insight into large data analytics and an increased transparency can potentially lead to improved model efficacy.”
BackgroundCatching the flu: Syndromic surveillance, algorithmic governmentality and global health security
Roberts, S. L., & Elbe, S. (2016) · Security Dialogue · doi:10.1177/0967010616666443 · Cited by 63
“More recently, there has been prominent coverage of the ‘failure’ of the digital surveillance system Google Flu Trends to accurately predict oncoming seasonal influenza arcs (Butler, 2013; Lazer et al., 2014).”
BackgroundSafety in Numbers? Group Privacy and Big Data Analytics in the Developing World
Taylor, L. (2016) · Springer eBooks · doi:10.1007/978-3-319-46608-8_2 · Cited by 85
Monitoring of the Venezuelan exodus through Facebook’s advertising platform
Palotti, J., Adler, N., Morales, A. J., Villaveces, J., Sekara, V., García–Herranz, M., Al-Asad, M., & Weber, I. G. (2020) · PLoS ONE · doi:10.1371/journal.pone.0229175 · Cited by 62
Data Action: Using Data for Public Good
Williams, S. (2020) · The MIT Press eBooks · doi:10.7551/mitpress/12261.001.0001 · Cited by 102
Mobile Network Data for Public Health: Opportunities and Challenges
Oliver, N., Matic, A., & Frías-Martínez, E. (2015) · Frontiers in Public Health · doi:10.3389/fpubh.2015.00189 · Cited by 63
“Note that the vast majority of large-scale human behavioral data used today is used in an opportunistic manner as the data was originally collected for other purposes (86).”
BackgroundBig Data in Medical Science–a Biostatistical View
Binder, H., & Blettner, M. (2015) · Deutsches Ärzteblatt international · doi:10.3238/arztebl.2015.0137 · Cited by 71
“However, later investigations (4) showed serious systematic deviations from predicted values in the period covered by (3).”
BackgroundAlgorithmic discovery of dynamic models from infectious disease data
Horrocks, J., & Bauch, C. T. (2020) · Scientific Reports · doi:10.1038/s41598-020-63877-w · Cited by 66
“And, the risk of overfitting to noise is always present and will only increase as more data becomes available [51].”
BackgroundWang, Z., Ma, D., Pang, R., Xie, F., Zhang, J., & Sun, D. (2020) · ISPRS International Journal of Geo-Information · doi:10.3390/ijgi9110632 · Cited by 64
“[33] used Google Flu Trends (GFT) as an argument to raise questions about big data as an alternative to traditional statistical methods and theories, and proposed two contradictions of Google Flu Trends: Hubris and Dynamics Algorithm.”
MethodBig Data in Organizations and the Role of Human Resource Management
Scholz, T. M. (2017) · Peter Lang D eBooks · doi:10.3726/b10907 · Cited by 80
Rahman, M. M., Khatun, F., Uzzaman, A., Sami, S. I., Bhuiyan, M. A.-A., & Kiong, T. S. (2021) · International Journal of Health Services · doi:10.1177/00207314211017469 · Cited by 62
“Several studies have been performed to gather training data from the present pandemic and to develop a precise prediction model for the pandemic.(61) Google Flu Trends(62) presented an approach that employs online searching for estimating and predicting real-time flu activity using AI and big social data.”
BackgroundLinking Past to Present: Discovering Style in Two Centuries of Architecture
Lee, S., Maisonneuve, N., Crandall, D., Efros, A. A., & Šivic, J. (2015) · IEEE International Conference on Computational Photography (ICCP) · doi:10.1109/iccphot.2015.7168368 · Cited by 63
“For example, collaborations between sociologists and computer scientists are using online social network data to measure human behavior at unprecedented scales [16], while work in health informatics is using online data to monitor outbreaks of diseases [11] and to predict their spread [22] (albeit with some controvers…”
BackgroundLeveraging “big data” to enhance the effectiveness of “one health” in an era of health informatics
Asokan, G. V., & Asokan, V. (2015) · Journal of Epidemiology and Global Health · doi:10.1016/j.jegh.2015.02.001 · Cited by 52
Reassessing the Facebook experiment: critical thinking about the validity of Big Data research
Panger, G. (2015) · Information Communication & Society · doi:10.1080/1369118x.2015.1093525 · Cited by 68
“Researchers may also neglect shifts in language or user behaviour that can degrade the validity of a measure over time (Lazer et al., 2014).”
BackgroundBig data and ophthalmic research
Clark, A., Ng, J. Q., Morlet, N., & Semmens, J. B. (2016) · Survey of Ophthalmology · doi:10.1016/j.survophthal.2016.01.003 · Cited by 55
Developing a Research Agenda for Human-Centered Data Science
Aragon, C., Hutto, C., Echenique, A., Fiore-Gartland, B., Huang, Y., Kim, J., Neff, G., Xing, W., & Bayer, J. (2016) · Proceedings of the 19th ACM Conference on Computer Supported Cooperative Work and Social Computing Companion · doi:10.1145/2818052.2855518 · Cited by 53
“There are clear advantages of each perspective—one can choose methods and techniques which facilitate deep, but narrow analysis, or one can be broad, but shallow [5, 6, 7].”
BackgroundEnhancing Feature Selection Using Word Embeddings: The Case of Flu Surveillance
Lampos, V., Zou, B., & Cox, I. J. (2017) · International World Wide Web Conference (WWW) · doi:10.1145/3038912.3052622 · Cited by 63
“Whereas previous attempts to model ILI rates from search query logs [50, 20] have reportedly produced misleading outputs [38, 45], follow-up research has corroborated that this was due to inadequacy of the applied statistical framework [35, 59].”
BackgroundWhere is the Human?: Bridging the Gap Between AI and HCI
Inkpen, K., Chancellor, S., De Choudhury, M., Veale, M., & Baumer, E. P. S. (2019) · Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290607.3299002 · Cited by 54
“Moreover, human activities and behaviors are deeply contextual, nuanced, and laden with subjectivity – aspects that many current AI/ML systems often fail to account for adequately [29].”
BackgroundMining Social Media Data for Biomedical Signals and Health-Related Behavior
Correia, R. B., Wood, I. B., Bollen, J., & Rocha, L. M. (2020) · Annual Review of Biomedical Data Science · doi:10.1146/annurev-biodatasci-030320-040844 · Cited by 82
“[27] David Lazer, Ryan Kennedy, Gary King, and Alessandro Vespignani.”
BackgroundDetecting and modelling real percolation and phase transitions of information on social media
Xie, J., Meng, F., Sun, J., Ma, X., Yan, G. F., & Hu, Y. (2021) · Nature Human Behaviour · doi:10.1038/s41562-021-01090-z · Cited by 69
“Online social media create large-scale networks that enable information spreading to occur rapidly and widely [1-11] and play an important role in a variety of applications [1,10,12-31].”
BackgroundData deprivations, data gaps and digital divides: Lessons from the COVID-19 pandemic
Naudé, W. A., & Vinuesa, R. (2021) · Big Data & Society · doi:10.1177/20539517211025545 · Cited by 68
“However, during a crisis such as that of COVID-19, social media can become too noisy to extract reliable inferences (Lazer et al., 2014).”
BackgroundBig Data and Health Economics: Strengths, Weaknesses, Opportunities and Threats
Collins, B. (2015) · PharmacoEconomics · doi:10.1007/s40273-015-0306-7 · Cited by 53
“However, despite some early success, it was found in subsequent analyses to be not very accurate in predicting rates of flu cases [12].”
BackgroundBernstam, E. V., Shireman, P. K., Meric‐Bernstam, F., Zozus, M. N., Jiang, X., Brimhall, B. B., Windham, A. K., Schmidt, S., Visweswaran, S., Ye, Y., et al. (2021) · Clinical and Translational Science · doi:10.1111/cts.13175 · Cited by 67
“For example, Google Flu Trends was wrong for 100 out of 108 weeks between 2011 and 2013, 75 and it missed the 2009 swine flu (H1N1) pandemic.”
BackgroundSurveillance of emerging infectious diseases for biosecurity
Hao, R., Liu, Y., Shen, W., Zhao, R., Jiang, B., Song, H., Yan, M., & Ma, H. (2022) · Science China Life Sciences · doi:10.1007/s11427-021-2071-x · Cited by 72
“However, with the predicting results much larger than the actual influenza-like illness provided by the US CDC, there was doubt that its predictions were accurate enough (Ginsberg et al., 2009; Lazer et al., 2014).”
BackgroundSasikumar, A., Ravi, L., Kotecha, K. V., Saini, J. R., Varadarajan, V., & Subramaniyaswamy, V. (2022) · Computational Intelligence and Neuroscience · doi:10.1155/2022/1419360 · Cited by 69
Xu, W., & Gao, Z. (2024) · IEEE Transactions on Technology and Society · doi:10.1109/tts.2024.3486254 · Cited by 112
“…systems trained with distorted data may cause the system to produce bias, easily amplify prejudice and unfairness, and so on, and the “world view” they follow may put certain user groups at a disadvantage and affect social fairness, causing some AI projects to fail [8], [11], [12], [13].”
Bello, N. M., & Renter, D. G. (2018) · Journal of Dairy Science · doi:10.3168/jds.2017-13978 · Cited by 68
Historical evolution of concrete and abstract language revisited
Snefjella, B., Généreux, M., & Kuperman, V. (2018) · Behavior Research Methods · doi:10.3758/s13428-018-1071-2 · Cited by 49
Park, H., Jung, H., On, J., Park, S. K., & Kang, H. (2018) · Healthcare Informatics Research · doi:10.4258/hir.2018.24.4.253 · Cited by 55
The empiricist’s challenge: Asking meaningful questions in political science in the age of big data
Jungherr, A. M., & Theocharis, Y. (2017) · Journal of Information Technology & Politics · doi:10.1080/19331681.2017.1312187 · Cited by 58
Is there a duty to participate in digital epidemiology?
Mittelstadt, B., Benzler, J., Engelmann, L., Prainsack, B., & Vayena, E. (2018) · Life Sciences Society and Policy · doi:10.1186/s40504-018-0074-1 · Cited by 58
Panuganti, B. A., Jafari, A., MacDonald, B. V., & DeConde, A. S. (2020) · Otolaryngology · doi:10.1177/0194599820932128 · Cited by 58
“The method of deriving the search terms and the media’s influence on user behavior were cited reasons for failure and represent an important reason to exercise caution when using this type of data.(5) As such, although significant correlations between Google searches pertaining to anosmia and COVID-19 incidence have a…”
MethodToward the use of neural networks for influenza prediction at multiple spatial resolutions
Aiken, E., Nguyen, A. T., Viboud, C. G., & Santillana, M. (2021) · Science Advances · doi:10.1126/sciadv.abb1237 · Cited by 57
Incorporating FAT and privacy aware AI modeling approaches into business decision making frameworks
Zhdanov, D., Bhattacharjee, S., & Bragin, M. A. (2021) · Decision Support Systems · doi:10.1016/j.dss.2021.113715 · Cited by 61
Zhou, Y. (2022) · Journal of Environmental and Public Health · doi:10.1155/2022/1064558 · Cited by 69
The Synergy Between Remote Sensing and Social Sensing in Urban Studies: Review and perspectives
Xing, X., Yu, B., Kang, C., Huang, B., Gong, J., & Liu, Y. (2024) · IEEE Geoscience and Remote Sensing Magazine · doi:10.1109/mgrs.2023.3343968 · Cited by 64
Trinkley, K. E., An, R., Maw, A., Glasgow, R. E., & Brownson, R. C. (2024) · Implementation Science · doi:10.1186/s13012-024-01346-y · Cited by 63
Schermann, M., Hemsen, H., Buchmüller, C., Bitter, T., Krcmar, H., Markl, V., & Hoeren, T. (2014) · Business & Information Systems Engineering · doi:10.1007/s12599-014-0345-1 · Cited by 52
Epidemiology in wonderland: Big Data and precision medicine
Saracci, R. (2018) · European Journal of Epidemiology · doi:10.1007/s10654-018-0385-9 · Cited by 59
Using big data to track marine oil transportation along the 21st-century Maritime Silk Road
Cheng, L., Yan, Z., Xiao, Y., Chen, Y., Zhang, F., & Li, M. (2018) · Science China Technological Sciences · doi:10.1007/s11431-018-9335-1 · Cited by 51
Are MOOC Learning Analytics Results Trustworthy? With Fake Learners, They Might Not Be!
Alexandron, G., Yoo, L. Y., Ruipérez‐Valiente, J. A., Lee, S., & Pritchard, D. E. (2019) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-019-00183-1 · Cited by 62
A Hierarchy of Limitations in Machine Learning
Malik, M. M. (2020) · RePEc: Research Papers in Economics · doi:10.48550/arxiv.2002.05193 · Cited by 51
“Lazer et al. (2014) phrase this as a problem of overfitting, and indeed we can think of models that fit to anything other than an underlying causal process as “overfitting,” but it is more strongly a lesson about causality (Harford, 2014).”
BackgroundBroniatowski, D. A., Dredze, M. H., Paul, M. J., & Dugas, A. F. (2015) · JMIR Public Health and Surveillance · doi:10.2196/publichealth.4472 · Cited by 55
“In addition, this prior work found that there was insufficient control for temporal autocorrelation and a lack of analysis of Google Flu Trends data at local, rather than national, levels [24].”
BackgroundKatsikopoulos, K. V., Şimşek, Ö., Buckmann, M., & Gigerenzer, G. (2021) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2020.12.006 · Cited by 61
Gonzalez, K. E., James, R., Bjorklund, E. T., & Hill, T. D. (2021) · Social Science Quarterly · doi:10.1111/ssqu.13025 · Cited by 58
“Finally, aggregate Internet search results from Google Search Trends may be impacted by changes in the search algorithm (Lazer et al., 2014).”
BackgroundCan Google Trends data improve forecasting of Lyme disease incidence?
Kapitány‐Fövény, M., Ferenci, T., Sulyok, Z., Kegele, J., Richter, H., Valyi-Nagy, I. T., & Sulyok, M. (2018) · Zoonoses and Public Health · doi:10.1111/zph.12539 · Cited by 63
Marshall, D. A., Burgos-Liz, L., Pasupathy, K. S., Padula, W. V., IJzerman, M. J., Wong, P. K., Higashi, M. K., Engbers, J. D. T., Wiebe, S., Crown, W. H., et al. (2015) · PharmacoEconomics · doi:10.1007/s40273-015-0330-7 · Cited by 51
“partially due to the change in search behaviour of users, an example of agency and subsequent emergent behaviour, and changes in the Google search algorithm itself [23].”
BackgroundHarnessing Big Data for Health Care and Research: Are Urologists Ready?
Ghani, K. R., Zheng, K., Wei, J. T., & Friedman, C. P. (2014) · European Urology · doi:10.1016/j.eururo.2014.07.032 · Cited by 52
“[18] Lazer D, Kennedy R, King G, Vespignani A.”
BackgroundBehavioral Informatics and Computational Modeling in Support of Proactive Health Management and Care
Pavel, M., Jimison, H. B., Korhonen, I., Gordon, C. M., & Saranummi, N. (2015) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2015.2484286 · Cited by 62
“ranging from physical activities [29] to searching the Internet [30].”
BackgroundWang, H.-W., Chen, D.-R., Yu, H.-W., & Chen, Y.-M. (2015) · Journal of Medical Internet Research · doi:10.2196/jmir.4516 · Cited by 50
Monteith, S., Glenn, T., Geddes, J., Whybrow, P. C., & Bauer, M. (2016) · International Journal of Bipolar Disorders · doi:10.1186/s40345-016-0051-7 · Cited by 50
“Predictive models can be wrong as shown repeatedly with Google Flu (Lazer et al. 2014a, b).”
MethodBig data in social and psychological science: theoretical and methodological issues
Qiu, L., Chan, S. H. M., & Chan, D. K. (2017) · Journal of Computational Social Science · doi:10.1007/s42001-017-0013-6 · Cited by 55
“However, researchers later found that the model completely missed nonseasonal influenza, suggesting that it predicted seasonality rather than the actual flu trend [19].”
BackgroundLi, K., Liu, M., Feng, Y., Ning, C., Ou, W., Sun, J., Wei, W., Liang, H., & Shao, Y. (2019) · Scientific Reports · doi:10.1038/s41598-018-35685-w · Cited by 50
Impact of CEO media appearance on corporate performance in social media
Bai, L., Yan, X., & Yu, G. (2019) · The North American Journal of Economics and Finance · doi:10.1016/j.najef.2019.100996 · Cited by 54
Has demography witnessed a data revolution? Promises and pitfalls of a changing data ecosystem
Kashyap, R. (2021) · Population Studies · doi:10.1080/00324728.2021.1969031 · Cited by 62
Google search histories of patients presenting to an emergency department: an observational study
Asch, J. M., Asch, D. A., Klinger, E. V., Marks, J., Sadek, N. H., & Merchant, R. M. (2019) · BMJ Open · doi:10.1136/bmjopen-2018-024791 · Cited by 50
Scientometric trends for coronaviruses and other emerging viral infections
Kagan, D., Moran‐Gilad, J., & Fire, M. (2020) · GigaScience · doi:10.1093/gigascience/giaa085 · Cited by 56
“Lazer et al. [26] noted that these types of methods are problematic because companies like Google, Facebook, and Twitter are constantly changing their products.”
Bibliometric analysis on tendency and topics of artificial intelligence over last decade
Gao, F., Jia, X., Zhao, Z., Chen, C., Xu, F., Geng, Z., & Song, X. (2019) · Microsystem Technologies · doi:10.1007/s00542-019-04426-y · Cited by 63
Turn to the Internet First? Using Online Medical Behavioral Data to Forecast COVID-19 Epidemic Trend
Huang, W., Cao, B., Yang, G., Luo, N., & Chao, N. (2020) · Information Processing & Management · doi:10.1016/j.ipm.2020.102486 · Cited by 49
“…direct “ novel coronavirus ” or “ novel coronavirus pneumonia ” , to avoid the errors of overfitting caused by self-correction in keyword selection (Cook et al., 2011; Lazer et al., 2014), because the strong link between search queries and NCC might result from the fear and panic during pandemic.”
BackgroundA participatory data-centric approach to AI Ethics by Design
Gerdes, A. (2021) · Applied Artificial Intelligence · doi:10.1080/08839514.2021.2009222 · Cited by 77
“Yet , a later version of the GFT algorithm made false predictions as the improvement of the model now caused users to click more on flu-related content, thereby tunneling their search results into flu-searches, which lead to the overfitting of the GFT algorithm (Lazer et al. 2014).”
MethodStolerman, L. M., Clemente, L., Poirier, C., Parag, K. V., Majumder, A., Masyn, S., Resch, B., & Santillana, M. (2023) · Science Advances · doi:10.1126/sciadv.abq0199 · Cited by 51
“…explored valuable information from search engines ( 29 , 32 – 35 ), Twitter microblogs ( 36 – 38 ), and electronic health records ( 39 – 41 ) for real-time estimates of disease incidence and characterized the limitations of those non-traditional data sources in the context of influenza ( 42 , 43 ).”
Scherr, S., Haim, M., & Arendt, F. (2018) · New Media & Society · doi:10.1177/1461444818801010 · Cited by 55
Metadata
- Authors
- Lazer, D., Kennedy, R., King, G., & Vespignani, A.
- Affiliations
- Northeastern University (US)
- Harvard University (US)
- Corresponding author
- Lazer, David
- Venue
- Science
- Published
- 2014-03-14
- Volume
- 343
- Issue
- 6176
- Pages
- 1203-1205
- ISSN
- 0036-8075 (ISSN-L)
- 1095-9203
- OpenAlex
- W2068181924
- Semantic Scholar
- 00d681cfff7f0224ace607e515f4ed510e792df9
- Cited by
- 2,523
- References
- 30
- Citations by year
- 2026: 113 · 2025: 101 · 2024: 121 · 2023: 150 · 2022: 168 · 2021: 229 · 2020: 273 · 2019: 261 · 2018: 272 · 2017: 249 · 2016: 271 · 2015: 219 · 2014: 91
- FWCI
- 208.16
- Type
- journal article
- Language
- en
- Open access
- green
- dash.harvard.edu
- Licence
- cc-by
- Locations
- Science (publishedVersion)
- PubMed (publishedVersion)
- Digital Access to Scholarship at Harvard (DASH) (Harvard University) (submittedVersion, cc-by)
- Topics
- Health Sciences › Medicine › Epidemiology › Data-Driven Disease Surveillance
- Social Sciences › Decision Sciences › Information Systems and Management › Big Data Technologies and Applications
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- Google Flu Trends, influenza surveillance, big data, prediction error, data quality
- MeSH terms
- Centers for Disease Control and Prevention, U.S., Data Collection, Data Interpretation, Statistical, Disease Outbreaks, Humans, Influenza, Human, Office Visits, Prognosis, United States, Social Media
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.