The Parable of Google Flu: Traps in Big Data Analysis
Open accessThe DOI matches its Crossref record; 30 of the 31 references it cites were found in a registry or an index; 5 of the 8 sources its claims cite were checked for support.
Abstract
Large errors in flu prediction were largely avoidable, which offers lessons for the use of big data.
Overview
CoChat Summary
TL;DR
The paper uses the failure of Google Flu Trends (GFT) to illustrate two traps in big data analysis: “big data hubris” and algorithm dynamics. It argues that GFT’s large flu-prediction errors were largely avoidable and that big data should supplement, not replace, traditional data and methods.
Key findings
- In February 2013, GFT predicted more than double the proportion of doctor visits for influenza-like illness (ILI) than the CDC; from 21 August 2011 to 1 September 2013, GFT reported overly high flu prevalence in 100 out of 108 weeks, and it overshot the 2011–2012 season by more than 50%.
- GFT errors were not random: they showed temporal autocorrelation and seasonality, meaning last week’s errors predicted this week’s errors and error direction/magnitude varied with time of year.
- Simple alternatives outperformed GFT: out-of-sample mean absolute error was 0.486 for GFT, 0.311 for lagged CDC, and 0.232 for combined GFT and CDC; all differences were statistically significant at P < 0.05. Even 3-week-old CDC data did better than GFT at projecting current flu prevalence.
- The authors attribute GFT’s problems to overfitting (initial GFT matched 50 million search terms to 1152 data points and missed the nonseasonal 2009 H1N1 pandemic) and to “algorithm dynamics,” including Google search algorithm changes and user behavior changes.
Methods
- Design: policy forum/case study comparing GFT estimates with CDC ILI surveillance data and alternative forecasting models.
- Data/sample: GFT and CDC flu prevalence estimates, including the period from 21 August 2011 to 1 September 2013; the initial GFT methodology reportedly fit 50 million search terms to 1152 data points.
- Main analysis: comparison of GFT alone, lagged CDC data with 52-week seasonality variables, and a combined GFT + CDC model using lagged CDC estimates, lagged GFT error, and seasonality variables; reported mean absolute error and statistical significance; Google Correlate used to examine correlated search terms. Supplementary materials contain data and methods.
Limitations
- The paper states that GFT never documented the 45 search terms used, released examples appear misleading, and Google Correlate is limited to national-level data whereas GFT was developed using regional correlations; therefore the original GFT analyses cannot be replicated from the information provided.
- The paper states that Google cannot make its full data available to outsiders because of privacy issues, though it argues there is no such constraint on derivative, aggregated data.
- Evident from the text: this is a commentary/policy forum rather than a full empirical article, with key data and methods placed in supplementary materials; the quantitative comparison depends on GFT and CDC data and model specifications not fully detailed in the main text.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
2 Valid · 2 Not valid · 2 Not checked · 8 Own results and interpretations
Some cited sources support this claim and some don’t — read each source’s verdict below.
In the paper
“This ad hoc method of throwing out peculiar search terms failed when GFT completely missed the nonseasonal 2009 infl uenza A–H1N1 pandemic ( 2, 14).”
What the sources say
- Partly supportedcites [2] matched to “Reassessing Google Flu Trends Data for Detection of Seasonal and Pandemic Influenza: A Comparative Epidemiological Study at Three Geographic Scales” (2013) sourceExperimental — accuracy not yet measured
The Discussion (page unavailable) says the original GFT model missed the spring 2009 pandemic wave, supporting the claim about its failure to detect the nonseasonal outbreak. It does not say GFT completely missed the entire 2009 pandemic. The Introduction describes automated selection of the original model’s search terms, not an ad hoc method of throwing out peculiar terms. (LLM, conf=0.86)
- Disputedcites [14] S. Cook et al., PLOS ONE 6, e23610 (2011). sourceExperimental — accuracy not yet measured
The Abstract (Methodology/Principal Findings) says the original GFT model underestimated the magnitude of illness during the 2009 H1N1 pandemic, not that it completely missed it. The Abstract (Conclusions) says both models performed well during the pandemic, although the updated model performed better. The Discussion notes that pandemic-specific queries were excluded, but does not describe this as an ad hoc method of throwing out peculiar terms. (LLM, conf=0.88)
Checked by a reasoning model — may be wrong.
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“Oddly, the few search terms offered in the papers ( 14) do not seem to be strongly related with either GFT or the CDC data (SM)—we surmise that the authors felt an unarticulated need to cloak the actual search terms identifi ed.”
What the sources say
- Disputedcites [14] S. Cook et al., PLOS ONE 6, e23610 (2011). sourceExperimental — accuracy not yet measured
The Results section (‘Search Behavior During pH1N1’; page unavailable) says that all three search queries shown in Figure 3 closely tracked CDC ILINet data before the pandemic, and ‘symptoms of flu’ continued to do so during it. The paper does report that the bronchitis and pneumonia queries underestimated ILINet data during the pandemic, but that does not support the broad claim that the offered terms were not strongly related to the data. (LLM, conf=0.87)
Checked by a reasoning model — may be wrong.
hedgedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“The Internet has opened the way for improving standard surveys, experiments, and health reporting ( 35).”
What the sources say
- Partly supportedcites [35] A. J. Berinsky et al., Polit. Anal. 20, 351–368 (2012). sourceExperimental — accuracy not yet measured
Section 2 describes using an Internet platform to run surveys with embedded experiments, and the Conclusion says MTurk can make experimental research easier and less expensive. The paper does not address improvements to health reporting; the disease scenario in section 4.3.2 is a framing experiment, not health reporting. Page numbers are unavailable in our copy. (LLM, conf=0.88)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Although not widely reported until 2013, the new GFT has been persistently overestimating flu prevalence for a much longer time.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“From 21 August 2011 to 1 September 2013, GFT reported overly high fl u prevalence 100 out of 108 weeks.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
3 more claims
Create a free account to see all 14 claims and every source verdict.
- It may state two separate claims as one. 5
- Probably not a substantial claim. 2
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Google Trends predicted more than twice the number of influenza-like illnesses than the CDC in February 2013 46 .”
From Unrepresentative big surveys significantly overestimated US vaccine uptake (2021) · cites this paper as [46]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This was one of the main reasons behind the failure of Google Flu Trends, since the data used for training the model in the first few years was not representative of the trends in subsequent years [15].”
From Theory-Guided Data Science: A New Paradigm for Scientific Discovery from Data (2017) · cites this paper as [15]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Social data research often relies on counting entities. These entities can be users, links, or messages, and the description is a summary of these counts (Lazer et al., 2014; Salganik, 2017). Yet, simple counts can mislead if it is unclear what is counted and how.”
From Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019) · cites this paper as Lazer et al., 2014
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This unintelligibility is also subject to functional biases (section 4.1), as it depends on the mechanisms available on each social platform (e.g., having a like button, but not a dislike one), as well as to variations in platform algorithms and mechanisms in response to users actions (Lazer et al., 2014).”
From Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019) · cites this paper as Lazer et al., 2014
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“It also includes temporal validity, which captures to what extent constructs change over time (Howison et al., 2011) and may invalidate previous conclusions about societal and/or platform phenomena; e.g., see the case of Google Flu Trends (Lazer et al., 2014).”
From Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019) · cites this paper as Lazer et al., 2014
6 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 14 MARCH 2014 VOL 343 SCIENCE www.sciencemag.org 1204
“The 2009 version of GFT has weathered other media panics related to the fl u, including the 2005–2006 influenza A/H5N1 ("bird flu") outbreak”
“GFT engineers updated the algorithm in 2009, and this model has run ever since”
Citations
Cites
boyd, D., & Crawford, K. (2012) · Information, Communication & Society · doi:10.1080/1369118x.2012.678878 · Cited by 5,479
Twitter mood predicts the stock market
Bollen, J., Mao, H., & Zeng, X. (2011) · Journal of Computational Science · doi:10.1016/j.jocs.2010.12.007 · Cited by 5,206
Detecting influenza epidemics using search engine query data
Ginsberg, J., Mohebbi, M. H., Patel, R. S., Brammer, L., Smolinski, M. S., & Brilliant, L. (2009) · Nature · doi:10.1038/nature07634 · Cited by 4,295
Cohen, D. (1998) · The Women's Review of Books · doi:10.2307/4022859 · Cited by 4,261
Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk
Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012) · Political Analysis · doi:10.1093/pan/mpr057 · Cited by 4,008
Big Data : The Management Revolution
マカフィー, ア., ブリニョルフソン, エ., & 有賀, 裕. (2013) · Harvard business review · Cited by 3,549
Lazer, D., Pentland, A., Adamic, L., Aral, S., Barabási, A.-L., Brewer, D., Christakis, N., Contractor, N., Fowler, J., Gutmann, M., et al. (2009) · Science · doi:10.1126/science.1167742 · Cited by 3,365
Self-citationPredicting Elections with Twitter: What 140 Characters Reveal about Political Sentiment
Tumasjan, A., Sprenger, T., Sandner, P., & Welpe, I. (2010) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v4i1.14009 · Cited by 2,649
Multiscale mobility networks and the spatial spreading of infectious diseases
Balcan, D., Colizza, V., Gonçalves, B., Hu, H., Ramasco, J. J., & Vespignani, A. (2009) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0906910106 · Cited by 1,380
Self-citationPredicting consumer behavior with web search
Goel, S., Hofman, J. M., Lahaie, S., Pennock, D. M., & Watts, D. J. (2010) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1005962107 · Cited by 636
Research works printed but not linked: 7
- A. McAfee, E. Brynjolfsson, Harv. Bus. Rev. 90, 60 (2012).
- AAAI Conference on Weblogs and Social Media, Atlanta, Georgia, 11 to 15 July 2010 (Association for Advancement of Artifi cial Intelligence, 2010), pp. 178.
- P. Copeland et al., Int. Soc. Negl. Trop. Dis. 2013, 3
- I. M. Hall et al., Epidemiol. Infect. 135, 372 (2007).
- E. Mustafaraj, P. Metaxas, in Proceedings of the Web- Sci10, Raleigh, NC, 26 and 27 April 2010 (Web Science Trust, 2010); http://journal.webscience.org/317/.
- J. Ratkiewicz et al., in Proceedings of 5th International AAAI Conference on Weblogs and Social Media, San Francisco, CA, 7 to 11 August 2011 (AAAI, 2011), p. 297–304.
- 2013; http://chronicle.com/article/Researchers-Struggle- to-Secure/141591/.
Cited by
Items 1,301–1,400 of 2,513
Medizin und öffentliche Gesundheit
Medizin und öffentliche Gesundheit - Konzepte, A., Schmiedebach, H.-P., & Ankele, M. (2018) · De Gruyter eBooks · doi:10.1515/9783110563269 · Cited by 8
Big Data in Official Statistics
Zeelenberg, K., & Braaksma, B. (2017) · Advances in data mining and database management book series · doi:10.4018/978-1-5225-2512-7.ch011 · Cited by 8
On Predictability of Rare Events Leveraging Social Media
Le, L., Ferrara, E., & Flammini, A. (2015) · Conference on Online Social Networks · doi:10.1145/2817946.2817949 · Cited by 8
“edu/people/pabo/ movie-review-data/ predict exogenous events [11, 12]: in such cases, it is important to keep in mind that the usage of machine learning algorithms or statistical models that function as black boxes can yield to results which are not interpretable and misleading [15].”
BackgroundICT4ICTD: Computational Social Science for Digital Development
Hilbert, M. (2015) · Hawaii International Conference on System Sciences (HICSS) · doi:10.1109/hicss.2015.258 · Cited by 8
On Big Data and Hydroinformatics
Chen, Y., & Han, D. (2016) · Procedia Engineering · doi:10.1016/j.proeng.2016.07.443 · Cited by 8
Protecting Consumers in Privacy and Data Security: A Perspective of Information EconomicsPreprint
Jin, G. Z., & Stivers, A. E. (2017) · SSRN Electronic Journal · doi:10.2139/ssrn.3006172 · Cited by 13
Information Seeking Responses to News of Local COVID-19 Cases: Evidence from Internet Search DataPreprint
Bento, A. I., Nguyen, T. D., Wing, C., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.04591 · Cited by 8
“y apply to Google Trends API for access. Limitations. Although the eventual failure of Google Flu Trends suggested that building and maintaining a complex model for a long period of time is difficult (9), internet search volume has been shown to be a good proxy for many socio-economic behavioral indicators, such as au…”
BackgroundOn the Nature and Types of Anomalies: A Review.Preprint
Foorthuis, R. (2020) · arXiv (Cornell University) · Cited by 10
Fantazzini, D. (2020) · SSRN Electronic Journal · doi:10.2139/ssrn.3671005 · Cited by 8
“This is what happened during the 2012–2013 influenza season, when Google Flu Trends (GFT) overestimated the prevalence of influenza by more than fifty percent, see Lazer et al. (2014) for more details.”
InfluentialDataset Level Explanation of Heat Demand Forecasting Ann with ShapPreprint
Białek, J., Bujalski, W., Wojdan, K., et al. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.4007107 · Cited by 8
Patel, D., Sumner, S. A., Bowen, D. A., Zwald, M. L., Yard, E. E., Wang, J., Law, R., Holland, K. M., Nguyen, T., Mower, G. D., et al. (2024) · npj Mental Health Research · doi:10.1038/s44184-023-00045-8 · Cited by 13
Complex data in tourism analysis: A stochastic approach to price competition
Angelini, G., Costa, M., & Guizzardi, A. (2025) · Big Data Research · doi:10.1016/j.bdr.2025.100520 · Cited by 7
Feng, Y., Chen, Z., Zhang, Y., Huang, W., Zhang, X., & He, S. (2025) · Frontiers in Public Health · doi:10.3389/fpubh.2025.1608241 · Cited by 12
Filtering and prediction of noisy and unstable signals: The case of Google Trends data
Fenga, L. (2019) · Journal of Forecasting · doi:10.1002/for.2626 · Cited by 7
Surveillance of a vomiting outbreak in dogs in the UK using owner‐derived and internet search data
Woolley, C. S. C., Handel, I. G., Bronsvoort, M., Schoenebeck, J. J., & Clements, D. N. (2021) · Veterinary Record · doi:10.1002/vetr.308 · Cited by 6
Beauchamp, N. (2022) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v16i1.19388 · Cited by 9
“The flaws of early efforts to predict public opinion (O’Connor et al. 2010) or influenza (Gins-berg et al. 2009; Dugas et al. 2013) have by now been well-documented (Gayo-Avello, Metaxas, and Mustafaraj 2011; Lazer et al. 2014; Beauchamp 2017).”
BackgroundОрганізація надання електронних послуг в Україні в умовах війни
Chukut, S. A., & Karpenko, Y. (2023) · Public Administration and Regional Development · doi:10.34132/pard2023.20.14 · Cited by 21
Liu, Y.-H., Wolff, K. T., & Lo, T.-Y. (2023) · Methodological Innovations · doi:10.1177/20597991231183962 · Cited by 8
“Lazer et al. (2014) noted that GT flu data were inaccurate at the time of the highest flu peak because the estimates from GT doubled the actual number of flu cases from CDC data.”
BackgroundSolving business problems: the business-driven data-supported process
Rodgers, M. D., Mukherjee, S., Melamed, B., Baveja, A., & Kapoor, A. (2024) · Annals of Operations Research · doi:10.1007/s10479-023-05770-z · Cited by 13
“Actually, there is solid scientific understanding of the flu spreading process, which was not leveraged by GFT, and a simple forecast algorithm, using two-week lagged CDC data sets, proved more accurate than the sophisticated GFT algorithm (Lazer et al., 2014).”
Lee, E. W. J., Tan, W. W., Pham, B. T. P., Kawaja, A., & Theng, Y. L. (2024) · JMIR Serious Games · doi:10.2196/47600 · Cited by 10
New Data Sources for Demographic Research
Breen, C. F., & Feehan, D. M. (2024) · Population and Development Review · doi:10.1111/padr.12671 · Cited by 13
Christophersen, M., Mørck, P., Langhoff, T. O., et al. (2015) · International Conference on Human-Computer Interaction · Cited by 7
Big Data for Monitoring Political Instability
Donnay, K. (2017) · International development policy/Revue internationale de politique de développement · doi:10.4000/poldev.2468 · Cited by 7
Demographic and spatial trends in diabetes-related virtual nursing examinations
Schultze, S. R., Mujica, F. C., & Kleinheksel, A. J. (2019) · Social Science & Medicine · doi:10.1016/j.socscimed.2019.01.002 · Cited by 9
Kirabo, L., Namara, M., & McNeese, N. J. (2021) · African Human Computer Interaction Conference (AfriCHI) · doi:10.1145/3448696.3448746 · Cited by 7
Multichannel Data-Driven Attribution Models: A Review and Research Agenda
Beck, B. B., Petersen, J. A., & Venkatesan, R. (2021) · Emerald Group Publishing Limited eBooks · doi:10.1108/s1548-643520210000018007 · Cited by 18
Artificial Intelligence in Healthcare
Aggarwal, N., Matheny, M. E., Shachar, C., Wang, S. X. Y., & Israni, S. T. (2022) · Oxford University Press eBooks · doi:10.1093/oxfordhb/9780197579329.013.50 · Cited by 7
Wikipedia pageviews as investors’ attention indicator for Nasdaq
Martínez, R. G., Orden‐Cruz, C., & Martínez‐Navalón, J. G. (2022) · Intelligent systems in accounting, finance and management/Intelligent systems in accounting, finance & management · doi:10.1002/isaf.1508 · Cited by 11
Lorenz, R., Beck, J., Horneber, S., Keusch, F., & Antoun, C. (2022) · IMISCOE research series · doi:10.1007/978-3-031-01319-5_10 · Cited by 7
Singh, L., Gresenz, C. R., Wang, Y., & Hu, S. (2022) · Journal of Medical Internet Research · doi:10.2196/38319 · Cited by 7
Folkmedical Technologies and the Sociotechnical Systems of Healthcare
Schoenherr, J. R. (2022) · IEEE Technology and Society Magazine · doi:10.1109/mts.2022.3197273 · Cited by 7
“Despite early successes, a failure to retrain the algorithm with new patterns of HISB led to significant overestimations in the prevalence of the flu [49].”
BackgroundCollective Information Seeking During a Health Crisis : Predictors of Google Trends During COVID-19
Xu, Y., & Margolin, D. (2023) · Health Communication · doi:10.1080/10410236.2023.2167578 · Cited by 7
“We begin by acknowledging the limitation due to the opacity of Google’s algorithms (Lazer et al., 2014), making it difficult to pinpoint what is measured in the “Google Trends” reporting.”
MethodThe Empirical Study of Human Mobility: Potentials and Pitfalls of Using Traditional and Digital Data
Recchi, E., & Tittel, K. (2023) · Springer eBooks · doi:10.1007/978-3-031-16624-2_23 · Cited by 9
Higher education worries and response in the era of artificial intelligence
Li, Y., & Yang, P. (2023) · SCIREA Journal of Education · doi:10.54647/education880398 · Cited by 12
DeepLens: Interactive Out-of-distribution Data Detection in NLP Models
Song, D., Wang, Z., Huang, Y., Ma, L., & Zhang, T. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580741 · Cited by 12
“One signifcant factor of the failure is that GFT did not take into account how users’ search behavior had changed since 2012 [30].”
BackgroundMethodGenerational effects of culture and digital media in former Soviet Republics
Horne, B. D., Rice, N. M., Luther, C. A., Ruck, D. J., Borycz, J., Allard, S. L., Fitzgerald, M., Манаев, О. Т., Prins, B., Taylor, M., et al. (2023) · Humanities and Social Sciences Communications · doi:10.1057/s41599-023-01670-6 · Cited by 10
Raubenheimer, J. (2023) · Acta Sociologica · doi:10.1177/00016993231187489 · Cited by 7
“I have detailed the effects of replicating the GT results of four different studies (including Lazer et al., 2014) using 130 samples of data each (Raubenheimer et al., 2023).”
BackgroundShi, Y., Haller, A., Reeson, A. F., & Li, X. (2024) · International Journal of Information Management Data Insights · doi:10.1016/j.jjimei.2024.100220 · Cited by 10
Intelli-food: Cyberinfrastructure for Real-Time Outbreak Source Detection and Rapid Response
Convertino, M., & Hedberg, C. W. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-08416-9_19 · Cited by 6
The Big To Do About “Big Data”
Schilling, P. L., & Bozic, K. J. (2014) · Clinical Orthopaedics and Related Research · doi:10.1007/s11999-014-3887-0 · Cited by 7
Collective behaviors and networks
Ciampaglia, G. L., Ferrara, E., & Flammini, A. (2014) · EPJ Data Science · doi:10.1140/epjds/s13688-014-0037-6 · Cited by 7
Big data and the historical sciences: A critique
Ebach, M. C., Michael, M. S., Shaw, W. S., GOFF, J. R., Murphy, D. J., & Matthews, S. T. (2016) · Geoforum · doi:10.1016/j.geoforum.2016.02.020 · Cited by 9
Vestergård, G. L. (2016) · Journalism · doi:10.1177/1464884915623171 · Cited by 7
“This methodology, though, has a low level of reliability, as Google algorithms are altered continuously in an enclosed cooperate setting (in 2014 Lazer et al. showed how scientific results based on Google Flu Trends were skewed because of algorithm changes (Lazer et al., 2014)).”
BackgroundMethodFdez-Arróyabe, P., & Royé, D. (2017) · Studies in big data · doi:10.1007/978-3-319-49736-5_9 · Cited by 10
Big Data Analyses in Health and Opportunities for Research in Radiology
Aphinyanaphongs, Y. (2017) · Seminars in Musculoskeletal Radiology · doi:10.1055/s-0036-1597255 · Cited by 7
Thinking and Modeling for Big Data from the Perspective of the I Ching
Lin, C., Li, G., Shan, Z., & Shi, Y. (2017) · International Journal of Information Technology & Decision Making · doi:10.1142/s0219622017500286 · Cited by 7
Applicability of internet search index for asthma admission forecast using machine learning
Luo, L., Liao, C., Zhang, F., Zhang, W., Li, C., Qiu, Z., & Huang, D. (2018) · The International Journal of Health Planning and Management · doi:10.1002/hpm.2525 · Cited by 7
Google searches do not correlate with melanoma incidence in majority English speaking countries
McDonald, L., Simpson, A., Graham, S., et al. (2018) · npj Digital Medicine · doi:10.1038/s41746-018-0050-4 · Cited by 7
City-Wide Influenza Forecasting based on Multi-Source Data
Su, K., Xiong, Y., Li, Q., et al. (2018) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata.2018.8622413 · Cited by 7
Pigeonholes and Mustard Seeds: Growing Capacity to use Data for Society
Ashby, D. (2019) · Journal of the Royal Statistical Society Series A (Statistics in Society) · doi:10.1111/rssa.12483 · Cited by 9
“…on social media (such as searches for symptoms of influenza) have been used for the tracking of epidemics although this turned out to be problematic (Lazer et al. , 2014) and mobile phone data, whichcontaingeographiclocators,canbeused,forexample,toestimatethesizeofcrowds(Botta et al. , 2015) and…”
Duclos, V. (2019) · Medicine Anthropology Theory · doi:10.17157/mat.6.3.660 · Cited by 10
Ziakas, P. D., & Mylonakis, E. E. (2020) · Medicine · doi:10.1097/md.0000000000020005 · Cited by 7
“Notably, analyzing of search trends is not free of caveats,([7]) and the reliability of digital epidemiology using Google Trends may vary depending on the healthcare setting.”
BackgroundCreating a Learning Televillage and Automated Digital Child Health Ecosystem
Chong, N. K., Elaine, C. C. S., & de Korne, D. F. (2020) · Pediatric Clinics of North America · doi:10.1016/j.pcl.2020.04.016 · Cited by 9
Dynamical Origins of Distribution Functions
Zang, C., Cui, P., Zhu, W., & Wang, F. (2019) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3292500.3330842 · Cited by 11
“) in the real world, based on which we can make better prediction [11] and control [12].”
BackgroundThe Evolution of Artificial Intelligence in Medical Informatics: A Bibliometric Analysis
Penteado, B. E., Fornazin, M., & Castro, L. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-86230-5_10 · Cited by 8
Examining the Correlation of Google Influenza Trend with Hospital Data: Retrospective Study
Jabour, A. M., Varghese, J., Damad, A. H., Ghailan, K. Y., & Mehmood, A. (2021) · Journal of Multidisciplinary Healthcare · doi:10.2147/jmdh.s322185 · Cited by 10
From Big Data to Smart Data: Application to performance management
Souifi, A., Cherfi-Boulanger, Z., Zolghadri, M., Barkallah, M., & Haddar, M. (2021) · IFAC-PapersOnLine · doi:10.1016/j.ifacol.2021.08.100 · Cited by 9
Schoenherr, J. R. (2021) · IEEE International Symposium on Technology and Society (ISTAS) · doi:10.1109/istas52410.2021.9629118 · Cited by 9
Vidal-García, J., Vidal, M., & Barros, R. H. (2018) · IGI Global eBooks · doi:10.4018/978-1-5225-7501-6.ch055 · Cited by 7
Can Big Data Forecast North Korean Military Aggression?
Kim, Y. H., Kang, H., & Lee, J. K. (2017) · Defence and Peace Economics · doi:10.1080/10242694.2016.1270736 · Cited by 8
“…of a conflict’s outbreak, such as terrorist attacks (Blomberg, Hess, and Weerapana 2004; Chen et al. 2013; Ghatak, Gold, and Prins 2017; Hendrix 2015; Housel, El Sawy, and Donovan 1986; Lazer et al. 2014; Li and Rao 2010; Pan, Pan, and Devadoss 2005; Piazza 2015; Shmueli and Koppius 2011).”
InfluentialMao, H. (2015) · Cambridge University Press eBooks · doi:10.1017/cbo9781316182635.005 · Cited by 7
Desai, B. C. (2014) · International Database Engineering and Applications Symposium (IDEAS) · doi:10.1145/2790755.2790769 · Cited by 7
Rainwater‐Lovett, K., Rodríguez-Barraquer, I., & Moss, W. J. (2016) · Elsevier eBooks · doi:10.1016/b978-0-12-800964-2.00018-5 · Cited by 7
“While initial efforts using data from the Internet for viral disease surveillance offer promising results, concerns have been raised regarding the utility and robustness of these approaches (Lazer, 2014) .”
BackgroundHafferty, J. D., Smıth, D. J., & McIntosh, A. M. (2016) · Social Psychiatry and Psychiatric Epidemiology · doi:10.1007/s00127-016-1294-4 · Cited by 6
“Google Flu Trends is a recent example of a machine-learning ‘big data’ approach which has fallen short of the initial promise [1, 5].”
BackgroundMapping Knowledge Domain Research in Big Data: From 2006 to 2016
Zeng, L., Li, Z., Wu, T., & Yang, L. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-61845-6_24 · Cited by 7
Representation and Analysis of Twitter Activity: A Dynamic Network Perspective
Falzon, L., McCurrie, C. H., & Dunn, J. D. (2017) · IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM) · doi:10.1145/3110025.3122118 · Cited by 8
“The data mining approach is seen to be a substitute for, rather than a complement to, traditional methods [9].”
MethodNunes, M. (2017) · Journal of Data and Information Science · doi:10.1515/jdis-2017-0017 · Cited by 6
“Lazer et al. (2014) proposed in a very highly cited Science article in 2014 that there is a “big data hubris.””
BackgroundTSAR: A Fully-Distributed Trustless Data ShARing Platform
Wu, H., Cao, J., Jiang, S., Yang, R., Yang, Y., & Hey, J. (2018) · IEEE International Conference on Smart Computing · doi:10.1109/smartcomp.2018.00028 · Cited by 9
“However, data sharing is performed in a primitive way in most of the big data analytics applications [7].”
BackgroundDifferentially Private Event Sequences over Infinite Streams with Relaxed Privacy Guarantee
Ren, X., Wang, S., Yao, X., et al. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-23597-0_22 · Cited by 7
Foursquare to the Rescue: Predicting Ambulance Calls Across Geographies
Noulas, A., Moffatt, C., Hristova, D., & Gonçalves, B. (2018) · International Conference on Digital Health · doi:10.1145/3194658.3194665 · Cited by 8
“Despite their biases [14], web and online social media sources, provide ample opportunity to break away many of the barriers that characterize traditional experimental methodology in medical studies.”
BackgroundGeiger, R. S., & Halfaker, A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.07273 · Cited by 7
“For example, in a Science editorial on the shortcomings of the Google flu trends study, Lazer, Kennedy, King, and Vespignani — influential researchers and early champions of the promise of “big data” — critique this approach and “the often implicit assumption that big data are a substitute for, rather than a supplemen…”
BackgroundIntroduction to Big Data and Its Applications in Insurance
Billot, R., Bothorel, C., & Lenca, P. (2018) · Wiley eBooks · doi:10.1002/9781119489368.ch1 · Cited by 7
Lists of Ethical, Legal, Societal and Economic Issues of Big Data TechnologiesPreprint
Custers, B. H. M., Fors, K. L., Jóźwiak, M., Keymolen, E., Bachlechner, D., Friedewald, M., & Aguzzi, S. (2018) · SSRN Electronic Journal · doi:10.2139/ssrn.3091018 · Cited by 10
Reproducibility in the Era of Big Data
McCoach, D. B., Dineen, J. N., Chafouleas, S. M., & Briesch, A. M. (2020) · Wiley eBooks · doi:10.1002/9781118976357.ch21 · Cited by 7
Inflated Granularity: Spatial Big Dataa and GeodemographicsPreprint
Dalton, C. M., & Thatcher, J. (2015) · SSRN Electronic Journal · doi:10.2139/ssrn.2544638 · Cited by 7
Applications of Big Data Science and Analytic Techniques for Health Disparities Research
Dankwa‐Mullan, I., Zhang, X., Le, P., & Riley, W. T. (2021) · Wiley eBooks · doi:10.1002/9781119374855.ch14 · Cited by 11
Whatever it takes to understand a central banker - Embedding their words using neural networksPreprint
Baumgaertner, M., & Zahner, J. (2021) · RePEc: Research Papers in Economics · Cited by 10
The epistemological foundations of data science: a critical analysisPreprint
Desai, J., Watson, D., Wang, V., Taddeo, M., & Floridi, L. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.4008316 · Cited by 7
Ethnography and Machine Learning: Synergies and New Directions
Li, Z., & Abramson, C. M. (2023) · Oxford University Press eBooks · doi:10.1093/oxfordhb/9780197653609.013.36 · Cited by 12
Snaphaan, T. (2025) · Methodological Innovations · doi:10.1177/20597991251336070 · Cited by 6
Wu, J., Tanim, S. H., Woo, M., Ahammed, T., Bleichrodt, A. M., & Rennert, L. (2025) · Epidemics · doi:10.1016/j.epidem.2025.100865 · Cited by 10
“This has the potential to predict outbreaks earlier, pinpoint high-risk areas, and inform targeted interventions (Lazer et al., 2014), ultimately improving public health preparedness and response.”
Using routinely collected data for research purposes: challenges and mitigation strategies
Hoffmann, S., Morris, T. P., Herrmann, M., Heinze, G., Wynants, L., CALSTER, B. V., Bischl, B., Schmid, M., Shaw, P. A., Mathes, T., et al. (2026) · BMJ · doi:10.1136/bmj-2025-087812 · Cited by 12
Service Ecosystem: A Lens of Smart Digital Society
Xue, X., Feng, Z., Chen, S., Zhou, Z., Qin, C., Li, B., Wang, Z., Wang, S., Wu, H., Zhang, L., et al. (2021) · IEEE International Conference on Services Computing · doi:10.1109/scc53864.2021.00039 · Cited by 8
Real-Time Detection of Flu Season Onset: A Novel Approach to Flu Surveillance
Liu, J., & Suzuki, S. (2022) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph19063681 · Cited by 8
“However, the GFT’s original algorithm for flu activity estimations was found to lack reliability and accuracy compared to those measured by CDC in subsequent years [11,21].”
ResultLi, J., Huang, W. W., Sia, C. L., Chen, Z., Wu, T., & Wang, Q. (2022) · JMIR Public Health and Surveillance · doi:10.2196/35266 · Cited by 7
Giusti, M., & Samuelsson, K. (2022) · Heliyon · doi:10.1016/j.heliyon.2023.e15751 · Cited by 8
“These results also rely on undisclosed and proprietary algorithms that are at odds with scientific transparency and replicability [10].”
BackgroundKanashina, O., Huertas-García, R., & Jiménez‐Zarco, A. I. (2023) · Intangible Capital · doi:10.3926/ic.2312 · Cited by 11
“…population participating in different social networks may lead to a bias in the interpretation of opinions and ratings (Moon et al., 2021); e.g., Lazer, Kennedy, King and Vespignani (2014) warn that many of the forecasts and analyses generated by data from social media misrepresent the real…”
BackgroundThe rise of online dating and racial homogamy in marriage
Kornrich, S., & Robbins, B. G. (2024) · Social Science Research · doi:10.1016/j.ssresearch.2023.102976 · Cited by 8
Laurer, M., van Atteveldt, W., Casas, A., & Welbers, K. (2024) · Communication Methods and Measures · doi:10.1080/19312458.2024.2378690 · Cited by 13
“…in text but actually measure incumbency (Hirst et al., 2014), we may believe to measure populism in text but actually only identify party names (Jankowski & Huber, 2023), or we may believe to measure a flu outbreak but actually measure unrelated search terms like “basketball” (Lazer et al., 2014).”
Untapped Potential: Designed Digital Trace Data in Online Survey Experiments
Macke, E., Daviss, C., & Williams-Baron, E. (2024) · Sociological Methods & Research · doi:10.1177/00491241241268770 · Cited by 12
“Academic researchers have used search data (e.g., Ginsberg et al. 2009; Goel et al. 2010; Lazer et al. 2014), social media posts (e.g., Bail 2017; Bail et al. 2020; King, Pan, and Roberts 2013; McCormick et al. 2017; Schober et al. 2016), email data (e.g., Bhatt, Goldberg, and Srivastava 2022;…”
When Infodemic Meets Epidemic: Systematic Literature Review
Asaad, C., Khaouja, I., Ghogho, M., & Bäına, K. (2025) · JMIR Public Health and Surveillance · doi:10.2196/55642 · Cited by 13
Peligros del uso de los big data en la investigación en salud pública y en epidemiología
Pérez, G. (2015) · Gaceta Sanitaria · doi:10.1016/j.gaceta.2015.09.007 · Cited by 8
Big Data and Business Decision Making
Vidal, M., Vidal-García, J., & Barros, R. H. (2016) · Advances in data mining and database management book series · doi:10.4018/978-1-5225-0182-4.ch010 · Cited by 6
Marketing-Automation-Projekte erfolgreich umsetzen
Findeisen, D. (2017) · Springer eBooks · doi:10.1007/978-3-658-15260-4_8 · Cited by 6
Survey on Big Data and Cloud Computing
Prabha, M. S., & Sarojini, B. (2017) · World Congress on Computing and Communication Technologies (WCCCT) · doi:10.1109/wccct.2016.36 · Cited by 6
Using Computational Intelligence for Tracking COVID-19 Outbreak in Online Social Networks
Qazi, S., Ahmad, S., & Raza, K. (2020) · Studies in computational intelligence · doi:10.1007/978-981-15-8534-0_3 · Cited by 6
Long-Term Influenza Outbreak Forecast Using Time-Precedence Correlation of Web Data
Jang, B., Kim, I., & Kim, J. W. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2021.3106637 · Cited by 8
“GFT showed remarkable performance in ILI prediction using a simple linear regression model but later erroneously predicted influenza outbreaks that were two times higher than CDC’s ILI [27].”
MethodTrevino, J., Malik, S., & Schmidt, M. (2022) · JMIR Infodemiology · doi:10.2196/32386 · Cited by 9
Scali, S. T., & Stone, D. H. (2023) · Seminars in Vascular Surgery · doi:10.1053/j.semvascsurg.2023.03.003 · Cited by 8
Challenges and Opportunities in One Health: Google Trends Search Data
Wisnieski, L. C., Gruszynski, K., Faulkner, V., et al. (2023) · Pathogens · doi:10.3390/pathogens12111332 · Cited by 6
“GFT trends overestimated flu prevalence by over 50% in 2011–2012, which some researchers blamed on the increased media coverage and google searches for “swine flu” and “bird flu” [1].”
BackgroundMethodInfluential
Metadata
- Authors
- Lazer, D., Kennedy, R., King, G., & Vespignani, A.
- Affiliations
- Northeastern University (US)
- Harvard University (US)
- Corresponding author
- Lazer, David
- Venue
- Science
- Published
- 2014-03-14
- Volume
- 343
- Issue
- 6176
- Pages
- 1203-1205
- ISSN
- 0036-8075 (ISSN-L)
- 1095-9203
- OpenAlex
- W2068181924
- Semantic Scholar
- 00d681cfff7f0224ace607e515f4ed510e792df9
- Cited by
- 2,523
- References
- 30
- Citations by year
- 2026: 113 · 2025: 101 · 2024: 121 · 2023: 150 · 2022: 168 · 2021: 229 · 2020: 273 · 2019: 261 · 2018: 272 · 2017: 249 · 2016: 271 · 2015: 219 · 2014: 91
- FWCI
- 208.16
- Type
- journal article
- Language
- en
- Open access
- green
- dash.harvard.edu
- Licence
- cc-by
- Locations
- Science (publishedVersion)
- PubMed (publishedVersion)
- Digital Access to Scholarship at Harvard (DASH) (Harvard University) (submittedVersion, cc-by)
- Topics
- Health Sciences › Medicine › Epidemiology › Data-Driven Disease Surveillance
- Social Sciences › Decision Sciences › Information Systems and Management › Big Data Technologies and Applications
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- Google Flu Trends, influenza surveillance, big data, prediction error, data quality
- MeSH terms
- Centers for Disease Control and Prevention, U.S., Data Collection, Data Interpretation, Statistical, Disease Outbreaks, Humans, Influenza, Human, Office Visits, Prognosis, United States, Social Media
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.