The Parable of Google Flu: Traps in Big Data Analysis
Open accessThe DOI matches its Crossref record; 30 of the 31 references it cites were found in a registry or an index; 5 of the 8 sources its claims cite were checked for support.
Abstract
Large errors in flu prediction were largely avoidable, which offers lessons for the use of big data.
Overview
CoChat Summary
TL;DR
The paper uses the failure of Google Flu Trends (GFT) to illustrate two traps in big data analysis: “big data hubris” and algorithm dynamics. It argues that GFT’s large flu-prediction errors were largely avoidable and that big data should supplement, not replace, traditional data and methods.
Key findings
- In February 2013, GFT predicted more than double the proportion of doctor visits for influenza-like illness (ILI) than the CDC; from 21 August 2011 to 1 September 2013, GFT reported overly high flu prevalence in 100 out of 108 weeks, and it overshot the 2011–2012 season by more than 50%.
- GFT errors were not random: they showed temporal autocorrelation and seasonality, meaning last week’s errors predicted this week’s errors and error direction/magnitude varied with time of year.
- Simple alternatives outperformed GFT: out-of-sample mean absolute error was 0.486 for GFT, 0.311 for lagged CDC, and 0.232 for combined GFT and CDC; all differences were statistically significant at P < 0.05. Even 3-week-old CDC data did better than GFT at projecting current flu prevalence.
- The authors attribute GFT’s problems to overfitting (initial GFT matched 50 million search terms to 1152 data points and missed the nonseasonal 2009 H1N1 pandemic) and to “algorithm dynamics,” including Google search algorithm changes and user behavior changes.
Methods
- Design: policy forum/case study comparing GFT estimates with CDC ILI surveillance data and alternative forecasting models.
- Data/sample: GFT and CDC flu prevalence estimates, including the period from 21 August 2011 to 1 September 2013; the initial GFT methodology reportedly fit 50 million search terms to 1152 data points.
- Main analysis: comparison of GFT alone, lagged CDC data with 52-week seasonality variables, and a combined GFT + CDC model using lagged CDC estimates, lagged GFT error, and seasonality variables; reported mean absolute error and statistical significance; Google Correlate used to examine correlated search terms. Supplementary materials contain data and methods.
Limitations
- The paper states that GFT never documented the 45 search terms used, released examples appear misleading, and Google Correlate is limited to national-level data whereas GFT was developed using regional correlations; therefore the original GFT analyses cannot be replicated from the information provided.
- The paper states that Google cannot make its full data available to outsiders because of privacy issues, though it argues there is no such constraint on derivative, aggregated data.
- Evident from the text: this is a commentary/policy forum rather than a full empirical article, with key data and methods placed in supplementary materials; the quantitative comparison depends on GFT and CDC data and model specifications not fully detailed in the main text.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
2 Valid · 2 Not valid · 2 Not checked · 8 Own results and interpretations
Some cited sources support this claim and some don’t — read each source’s verdict below.
In the paper
“This ad hoc method of throwing out peculiar search terms failed when GFT completely missed the nonseasonal 2009 infl uenza A–H1N1 pandemic ( 2, 14).”
What the sources say
- Partly supportedcites [2] matched to “Reassessing Google Flu Trends Data for Detection of Seasonal and Pandemic Influenza: A Comparative Epidemiological Study at Three Geographic Scales” (2013) sourceExperimental — accuracy not yet measured
The Discussion (page unavailable) says the original GFT model missed the spring 2009 pandemic wave, supporting the claim about its failure to detect the nonseasonal outbreak. It does not say GFT completely missed the entire 2009 pandemic. The Introduction describes automated selection of the original model’s search terms, not an ad hoc method of throwing out peculiar terms. (LLM, conf=0.86)
- Disputedcites [14] S. Cook et al., PLOS ONE 6, e23610 (2011). sourceExperimental — accuracy not yet measured
The Abstract (Methodology/Principal Findings) says the original GFT model underestimated the magnitude of illness during the 2009 H1N1 pandemic, not that it completely missed it. The Abstract (Conclusions) says both models performed well during the pandemic, although the updated model performed better. The Discussion notes that pandemic-specific queries were excluded, but does not describe this as an ad hoc method of throwing out peculiar terms. (LLM, conf=0.88)
Checked by a reasoning model — may be wrong.
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“Oddly, the few search terms offered in the papers ( 14) do not seem to be strongly related with either GFT or the CDC data (SM)—we surmise that the authors felt an unarticulated need to cloak the actual search terms identifi ed.”
What the sources say
- Disputedcites [14] S. Cook et al., PLOS ONE 6, e23610 (2011). sourceExperimental — accuracy not yet measured
The Results section (‘Search Behavior During pH1N1’; page unavailable) says that all three search queries shown in Figure 3 closely tracked CDC ILINet data before the pandemic, and ‘symptoms of flu’ continued to do so during it. The paper does report that the bronchitis and pneumonia queries underestimated ILINet data during the pandemic, but that does not support the broad claim that the offered terms were not strongly related to the data. (LLM, conf=0.87)
Checked by a reasoning model — may be wrong.
hedgedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“The Internet has opened the way for improving standard surveys, experiments, and health reporting ( 35).”
What the sources say
- Partly supportedcites [35] A. J. Berinsky et al., Polit. Anal. 20, 351–368 (2012). sourceExperimental — accuracy not yet measured
Section 2 describes using an Internet platform to run surveys with embedded experiments, and the Conclusion says MTurk can make experimental research easier and less expensive. The paper does not address improvements to health reporting; the disease scenario in section 4.3.2 is a framing experiment, not health reporting. Page numbers are unavailable in our copy. (LLM, conf=0.88)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Although not widely reported until 2013, the new GFT has been persistently overestimating flu prevalence for a much longer time.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“From 21 August 2011 to 1 September 2013, GFT reported overly high fl u prevalence 100 out of 108 weeks.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
3 more claims
Create a free account to see all 14 claims and every source verdict.
- It may state two separate claims as one. 5
- Probably not a substantial claim. 2
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Google Trends predicted more than twice the number of influenza-like illnesses than the CDC in February 2013 46 .”
From Unrepresentative big surveys significantly overestimated US vaccine uptake (2021) · cites this paper as [46]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This was one of the main reasons behind the failure of Google Flu Trends, since the data used for training the model in the first few years was not representative of the trends in subsequent years [15].”
From Theory-Guided Data Science: A New Paradigm for Scientific Discovery from Data (2017) · cites this paper as [15]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Social data research often relies on counting entities. These entities can be users, links, or messages, and the description is a summary of these counts (Lazer et al., 2014; Salganik, 2017). Yet, simple counts can mislead if it is unclear what is counted and how.”
From Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019) · cites this paper as Lazer et al., 2014
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This unintelligibility is also subject to functional biases (section 4.1), as it depends on the mechanisms available on each social platform (e.g., having a like button, but not a dislike one), as well as to variations in platform algorithms and mechanisms in response to users actions (Lazer et al., 2014).”
From Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019) · cites this paper as Lazer et al., 2014
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“It also includes temporal validity, which captures to what extent constructs change over time (Howison et al., 2011) and may invalidate previous conclusions about societal and/or platform phenomena; e.g., see the case of Google Flu Trends (Lazer et al., 2014).”
From Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019) · cites this paper as Lazer et al., 2014
6 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 14 MARCH 2014 VOL 343 SCIENCE www.sciencemag.org 1204
“The 2009 version of GFT has weathered other media panics related to the fl u, including the 2005–2006 influenza A/H5N1 ("bird flu") outbreak”
“GFT engineers updated the algorithm in 2009, and this model has run ever since”
Citations
Cites
boyd, D., & Crawford, K. (2012) · Information, Communication & Society · doi:10.1080/1369118x.2012.678878 · Cited by 5,479
Twitter mood predicts the stock market
Bollen, J., Mao, H., & Zeng, X. (2011) · Journal of Computational Science · doi:10.1016/j.jocs.2010.12.007 · Cited by 5,206
Detecting influenza epidemics using search engine query data
Ginsberg, J., Mohebbi, M. H., Patel, R. S., Brammer, L., Smolinski, M. S., & Brilliant, L. (2009) · Nature · doi:10.1038/nature07634 · Cited by 4,295
Cohen, D. (1998) · The Women's Review of Books · doi:10.2307/4022859 · Cited by 4,261
Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk
Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012) · Political Analysis · doi:10.1093/pan/mpr057 · Cited by 4,008
Big Data : The Management Revolution
マカフィー, ア., ブリニョルフソン, エ., & 有賀, 裕. (2013) · Harvard business review · Cited by 3,549
Lazer, D., Pentland, A., Adamic, L., Aral, S., Barabási, A.-L., Brewer, D., Christakis, N., Contractor, N., Fowler, J., Gutmann, M., et al. (2009) · Science · doi:10.1126/science.1167742 · Cited by 3,365
Self-citationPredicting Elections with Twitter: What 140 Characters Reveal about Political Sentiment
Tumasjan, A., Sprenger, T., Sandner, P., & Welpe, I. (2010) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v4i1.14009 · Cited by 2,649
Multiscale mobility networks and the spatial spreading of infectious diseases
Balcan, D., Colizza, V., Gonçalves, B., Hu, H., Ramasco, J. J., & Vespignani, A. (2009) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0906910106 · Cited by 1,380
Self-citationPredicting consumer behavior with web search
Goel, S., Hofman, J. M., Lahaie, S., Pennock, D. M., & Watts, D. J. (2010) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1005962107 · Cited by 636
Research works printed but not linked: 7
- A. McAfee, E. Brynjolfsson, Harv. Bus. Rev. 90, 60 (2012).
- AAAI Conference on Weblogs and Social Media, Atlanta, Georgia, 11 to 15 July 2010 (Association for Advancement of Artifi cial Intelligence, 2010), pp. 178.
- P. Copeland et al., Int. Soc. Negl. Trop. Dis. 2013, 3
- I. M. Hall et al., Epidemiol. Infect. 135, 372 (2007).
- E. Mustafaraj, P. Metaxas, in Proceedings of the Web- Sci10, Raleigh, NC, 26 and 27 April 2010 (Web Science Trust, 2010); http://journal.webscience.org/317/.
- J. Ratkiewicz et al., in Proceedings of 5th International AAAI Conference on Weblogs and Social Media, San Francisco, CA, 7 to 11 August 2011 (AAAI, 2011), p. 297–304.
- 2013; http://chronicle.com/article/Researchers-Struggle- to-Secure/141591/.
Cited by
Items 601–700 of 2,513
Big Data Analytics for Medical Applications
Das, N., Das, L., Rautaray, S. S., & Pandey, M. (2018) · International Journal of Modern Education and Computer Science · doi:10.5815/ijmecs.2018.02.04 · Cited by 42
“Using this big data population health management influence of data to direct our most challenging healthcare problems [30].”
BackgroundUnderstanding Medication Nonadherence from Social Media: A Sentiment-Enriched Deep Learning Approach
Xie, J., Liu, X., Zeng, D. D., & Fang, X. (2017) · MIS Quarterly · doi:10.25300/misq/2022/15336 · Cited by 59
The Big Data Paradox in Clinical Practice
Msaouel, P. (2022) · Cancer Investigation · doi:10.1080/07357907.2022.2084621 · Cited by 42
Introduction to supervised machine learning in clinical epidemiology
Ono, S., & Goto, T. (2022) · Annals of Clinical Epidemiology · doi:10.37737/ace.22009 · Cited by 57
Collective Attention and Stock Prices: Evidence from Google Trends Data on Standard and Poor's 100
Heiberger, R. H. (2015) · PLoS ONE · doi:10.1371/journal.pone.0135311 · Cited by 46
Regional Level Influenza Study with Geo-Tagged Twitter Data
Wang, F., Wang, H., Xu, K., Raymond, R., Chon, J., Fuller, S., & Debruyn, A. (2016) · Journal of Medical Systems · doi:10.1007/s10916-016-0545-y · Cited by 38
“However, GFT modeling has recently been criticized due to the lack of transparency of Google flu data and the overestimation of influenza prevalence by conflating signals of influenza awareness (such as media attention) with signals of actual infection [2].”
BackgroundGamma, A., Schleifer, R., Weinmann, W., Buadze, A., & Liebrenz, M. (2016) · PLoS ONE · doi:10.1371/journal.pone.0166566 · Cited by 38
Regulatory learning: How to supervise machine learning models? An application to credit scoring
Guégan, D., & Hassani, B. K. (2018) · The Journal of Finance and Data Science · doi:10.1016/j.jfds.2018.04.001 · Cited by 46
Ho, H. T., Carvajal, T. M., Bautista, J. R., Capistrano, J. D. R., Viacrusis, K. M., Hernandez, L. F. T., & Watanabe, K. (2018) · Tropical Medicine and Infectious Disease · doi:10.3390/tropicalmed3040118 · Cited by 41
Chen, S., Liu, X., Wu, Y., et al. (2018) · The Science of The Total Environment · doi:10.1016/j.scitotenv.2018.10.304 · Cited by 37
Rethinking Resilience Analytics
Eisenberg, D. A., Seager, T. P., & Alderson, D. (2019) · Risk Analysis · doi:10.1111/risa.13328 · Cited by 41
“The service, which once was heralded as a “poster child” for the power of data analytics, is now regarded as a cautionary tale of “big data hubris” (Lazer et al., 2014) for modelers and users alike.”
BackgroundHidden big data analytics issues in the healthcare industry
Strang, K. D., & Sun, Z. (2019) · Health Informatics Journal · doi:10.1177/1460458219854603 · Cited by 48
“54 These types of problems are programming errors made by Google.”
BackgroundSmith, G. (2020) · Journal of Information Technology · doi:10.1177/0268396220915600 · Cited by 76
“(Helft, 2008) However, after issuing its report, Google Flu Trends over-estimated the number of flu cases for 100 of the next 108 weeks, by an average of nearly 100% (Lazer et al., 2014).”
BackgroundFlow Pattern Transition in Pipes Using Data-Driven and Physics-Informed Machine Learning
Quintino, A. M., da Rocha, D. L. L. N., da Fonseca, R., & Rodríguez, O. M. H. (2020) · Journal of Fluids Engineering · doi:10.1115/1.4048876 · Cited by 37
Google Trends Data and COVID‐19 in Europe: Correlations and model enhancement are European wide
Sulyok, M., Ferenci, T., & Walker, M. (2021) · Transboundary and Emerging Diseases · doi:10.1111/tbed.13887 · Cited by 35
Choi, D., Sumner, S. A., Holland, K. M., Draper, J., Murphy, S. F., Bowen, D. A., Zwald, M. L., Wang, J., Law, R., Taylor, J., et al. (2020) · JAMA Network Open · doi:10.1001/jamanetworkopen.2020.30932 · Cited by 45
Lu, T., & Reis, B. Y. (2021) · npj Digital Medicine · doi:10.1038/s41746-021-00396-6 · Cited by 35
Precision Psychiatry: The Future Is Now
Passos, I. C., Ballester, P. L., Rabelo‐da‐Ponte, F. D., & Kapczinski, F. P. (2021) · The Canadian Journal of Psychiatry · doi:10.1177/0706743721998044 · Cited by 45
“Later on, a study reported that the algorithm consistently overestimated flu prevalence in subsequent years in part because the terms collectively searched in 2008 are different from the terms commonly searched in 2011.(13) Within psychiatry, nonstationarity is a challenge for generating digital phenotypes, as behavio…”
BackgroundInfluenza surveillance with Baidu index and attention-based long short-term memory model
Dai, S., & Han, L. (2023) · PLoS ONE · doi:10.1371/journal.pone.0280834 · Cited by 40
“Hence, considering the feasibility of using search engine data for influenza monitoring [22] and the advantages of RNN-kind models in time series data processing, based on Baidu Index data and real influenza statistics, our study explores supervised machine learning experiments by using less data to get excellent fitt…”
MethodBig Data for cardiology: novel discovery?
Mayer‐Schönberger, V. (2015) · European Heart Journal · doi:10.1093/eurheartj/ehv648 · Cited by 37
“A more iterative, process-oriented approach, however, also necessitates an infrastructure of continuous data collection and analyses that is very different from what is in place today.22 This is challenging, even for engineers at Google.23 This emerging emphasis on complexity and iteration also refocuses the attention…”
BackgroundInfluentialQuantitative Agent Based Model of Opinion Dynamics: Polish Elections of 2015
Sobkowicz, P. (2016) · PLoS ONE · doi:10.1371/journal.pone.0155098 · Cited by 43
“Because the political events are largely driven by the communications, such approach could have a better chance than the Google Flu prediction [99 – 103].”
Using Social Media to Predict the Future: A Systematic Literature Review
Phillips, L., Dowling, C., Shaffer, K., Hodas, N. O., & Volkova, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.06134 · Cited by 100
“EMBERS makes use of a wide variety of publicly available data including Tweets, RSS news and blog feeds, meteorological data, and Google Flu Trends among many others.”
BackgroundInfluentialEpistemic Entitlements and the Practice of Computer Simulation
Symons, J., & Alvarado, R. (2019) · Minds and Machines · doi:10.1007/s11023-018-9487-0 · Cited by 46
Caveat Emptor: The Risks of Using Big Data for Human Development
Latif, S., Qayyum, A., Usama, M., Qadir, J., Zwitter, A., & Shahzad, M. (2019) · IEEE Technology and Society Magazine · doi:10.1109/mts.2019.2930273 · Cited by 36
“[15] for predicting flu cases in the US using 50 million Google search terms suffered from activity bias where the model has started over-predicting the flu in the US [7], [24].”
Extracting medical entities from social media
Šćepanović, S., Martín-López, E., Quercia, D., & Baykaner, K. (2020) · ACM Conference on Health, Inference, and Learning (CHIL) · doi:10.1145/3368555.3384467 · Cited by 53
“This was the main reason Google terminated its ‘flu trends’ project, in which flu outbreaks were tracked from people’s searches – by considering any word useful for prediction, the system ended up failing to be robust to exogenous events [32].”
BackgroundResearch status of deep learning methods for rumor detection
Tan, L., Wang, G., Jia, F., & Lian, X. (2022) · Multimedia Tools and Applications · doi:10.1007/s11042-022-12800-8 · Cited by 46
Fusing algorithms and analysts: open-source intelligence in the age of ‘Big Data’
Eldridge, C., Hobbs, C., & Moran, M. (2017) · Intelligence & National Security · doi:10.1080/02684527.2017.1406677 · Cited by 56
The epistemological foundations of data science: a critical analysis
Desai, J., Watson, D., Wang, V., Taddeo, M., & Floridi, L. (2022) · Synthese · doi:10.1007/s11229-022-03933-2 · Cited by 68
“Lazer et al. (2014) stress the emergence of “computational social science”, and Miller (2010) observes the proliferation of data in the context of regional and urban science.”
BackgroundThe next generation of machine learning for tracking adaptation texts
Sietsma, A. J., Ford, J. D., & Minx, J. C. (2023) · Nature Climate Change · doi:10.1038/s41558-023-01890-3 · Cited by 55
“Similarly, ML methods can certainly be used to generate novel insights, but it is ‘data hubris’ 78 to think that such insights will simply reveal themselves with sufficient data and algorithms.”
Generalization in neural networks: A broad survey
Rohlfs, C. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.128701 · Cited by 68
AI-driven epidemic intelligence: the future of outbreak detection and response
Kaur, J., & Butt, Z. A. (2025) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2025.1645467 · Cited by 61
The Promise of Urban Informatics: Some Speculations
Thrift, N. J. (2014) · Environment and Planning A Economy and Space · doi:10.1068/a472c · Cited by 35
Geng, P., Liu, Y., Wang, J., et al. (2016) · Journal of Systems Science and Systems Engineering · doi:10.1007/s11518-016-5311-7 · Cited by 35
Sagarra, O., Gutiérrez-Roig, M., Bonhoure, I., & Perelló, J. (2016) · Frontiers in Physics · doi:10.3389/fphy.2015.00093 · Cited by 59
“…on the data gathering process (by the public in general and by scientists in particular), inherent This is a provisional file, not the final typeset article 2 population and sampling biases (Lazer et al., 2014) and lack of reproducibility among other systemic problems (Boyd and Crawford, 2012).”
BackgroundPilny, A. N., McAninch, K. G., Slone, A., & Moore, K. (2019) · Communication Methods and Measures · doi:10.1080/19312458.2019.1650166 · Cited by 45
The Advantages and Challenges of Using Real‐World Data for Patient Care
Ho, Y., Hu, F., & Lee, P. (2019) · Clinical and Translational Science · doi:10.1111/cts.12683 · Cited by 38
“9 The tide of science and technology is flowing inexorably in favor of data sciences, such as artificial intelligence, big data, and machine learning.”
BackgroundAssessing Canadians Health Activity and Nutritional Habits Through Social Media
Shah, N. J., Srivastava, G., Savage, D. W., & Mago, V. (2020) · Frontiers in Public Health · doi:10.3389/fpubh.2019.00400 · Cited by 39
“If individuals search for a solution for the flu or any medical information related to the flu, the algorithm uses that information and considers their location as a potential flu affected area (6).”
BackgroundMethodSofiev, M. A., Palamarchuk, Y., Bédard, A., Basagaña, X., Antó, J. M., Kouznetsov, R. D., Urzua, R. D., Bergmann, K., Fonseca, J., de Vries, G., et al. (2020) · Chinese Medical Journal · doi:10.1097/cm9.0000000000000916 · Cited by 37
A Google–Wikipedia–Twitter Model as a Leading Indicator of the Numbers of Coronavirus Deaths
O’Leary, D. E., & Storey, V. C. (2020) · Intelligent systems in accounting, finance and management/Intelligent systems in accounting, finance & management · doi:10.1002/isaf.1482 · Cited by 37
“Although the Google flu trends team terminated their efforts in 2015 (Google, 2015), there is continued research using Google searches, as a way of investigating disease outbreaks (e.g. Butler, 2013; Lazer et al., 2014; Kugler, 2016).”
BackgroundBig data and Wikipedia research: social science knowledge across disciplinary divides
Schroeder, R., & Taylor, L. (2015) · Information, Communication & Society · doi:10.1080/1369118x.2015.1008538 · Cited by 35
Optimizing Online Suicide Prevention: A Search Engine-Based Tailored Approach
Arendt, F., & Scherr, S. (2016) · Health Communication · doi:10.1080/10410236.2016.1224451 · Cited by 39
“In fact, algorithm-based decision-making by search engines is constantly tested and improved upon (Lazer, Kennedy, King, & Vespignani, 2014).”
MethodSocial prediction: a new research paradigm based on machine learning
Chen, Y., Wu, X., Hu, A., He, G., & Ju, G. (2021) · The Journal of Chinese Sociology · doi:10.1186/s40711-021-00152-z · Cited by 54
“The black box mechanism (such as the well-known Google Flu prediction; see Butler 2013) and prediction errors of machine learning are often criticized (Lazer et al. 2014).”
Using Internet Search Data to Produce State-level Measures: The Case of Tea Party Mobilization
DiGrazia, J. (2015) · Sociological Methods & Research · doi:10.1177/0049124115610348 · Cited by 38
“For a recent criticism of Google Flu Trends (GFT), see Lazer et al. (2014).”
BackgroundBig Data in Public Affairs Education
Mergel, I. A. (2016) · Journal of Public Affairs Education · doi:10.1080/15236803.2016.12002243 · Cited by 49
“Journal of Public Affairs Education 233 Lazer et al. (2009) focus on real-time availability and see these big data sets as opportunities to create a “second-by-second picture of interactions over extended periods of time, providing information about both the structure and content of relationships” (p.”
BackgroundBig data in forensic science and medicine
Lefèvre, T. (2017) · Journal of Forensic and Legal Medicine · doi:10.1016/j.jflm.2017.08.001 · Cited by 40
“When Google claimed its search engine performed better than the Centers for Disease Control (CDC) (15), it was then opposed that it was not doing it accurately and only provided correct trends, not precise estimates (16).”
MethodFuller, D., Buote, R., & Stanley, K. G. (2017) · Journal of Epidemiology & Community Health · doi:10.1136/jech-2017-209608 · Cited by 37
Conceptual framework for investigating causal effects from observational data in livestock1
Bello, N. M., Ferreira, V. C. A., Gianola, D., & Rosa, G. J. M. (2018) · Journal of Animal Science · doi:10.1093/jas/sky277 · Cited by 41
Rees, E. E., Ng, V., Gachon, P., Mawudeku, A., McKenney, D. W., Pedlar, J., Yemshanov, D., Parmely, J., & Knox, J. D. (2019) · Canada Communicable Disease Report · doi:10.14745/ccdr.v45i05a02 · Cited by 42
Aging biomarkers and the measurement of health and risk
Green, S., & Hillersdal, L. (2021) · History & Philosophy of the Life Sciences · doi:10.1007/s40656-021-00367-w · Cited by 40
Forecasting influenza-like illness trends in Cameroon using Google Search Data
Nsoesie, E. O., Oladeji, O., Abah, A. S. A., & Ndeffo-Mbah, M. L. (2021) · Scientific Reports · doi:10.1038/s41598-021-85987-9 · Cited by 44
On a Certain Research Gap in Big Data Mining for Customer Insights
Mach‐Król, M., & Hadasik, B. (2021) · Applied Sciences · doi:10.3390/app11156993 · Cited by 62
“With the amount of data, discovering dependencies and valuable patterns becomes extremely difficult [87,118].”
BackgroundAddressing Google Trends inconsistencies
Cebrián, E., & Domènech, J. (2024) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2024.123318 · Cited by 55
Avian Influenza Risk Surveillance in North America with Online Media
Robertson, C., & Yee, L. (2016) · PLoS ONE · doi:10.1371/journal.pone.0165688 · Cited by 33
Big Data: A New Empiricism and its Epistemic and Socio-Political Consequences
Rieder, G., & Simon, J. (2017) · Springer eBooks · doi:10.1007/978-3-658-12153-2_4 · Cited by 53
Bernard, R., Bowsher, G. M., Milner, C., Boyle, P., Patel, P., & Sullivan, R. J. (2018) · Journal of Public Health · doi:10.1007/s10389-018-0899-3 · Cited by 33
The added value of online user-generated content in traditional methods for influenza surveillance
Wagner, M. G., Lampos, V., Cox, I. J., & Pebody, R. G. (2018) · Scientific Reports · doi:10.1038/s41598-018-32029-6 · Cited by 37
Drivers of walleye recruitment in Minnesota’s large lakes
Honsey, A. E., Feiner, Z. S., & Hansen, G. J. A. (2020) · Canadian Journal of Fisheries and Aquatic Sciences · doi:10.1139/cjfas-2019-0453 · Cited by 38
“…to overfitting the data, and they can infer relationships that run counter to 58 well-established principles (e.g., conservation of energy; Read et al. 2019), especially when predicting 59 outside the range of the training data (see also Lazer et al. 2014; Karpatne et al. 2017; Chen et al. 2018).”
BackgroundData science on the ground: Hype, criticism, and everyday work
Carter, D. L., & Sholler, D. (2015) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.23563 · Cited by 46
Computational Social Science Methodology, Anyone?
Hox, J. J. (2017) · Methodology · doi:10.1027/1614-2241/a000127 · Cited by 33
Smart Cities, Big Data, and Sustainability Union
Kudva, S., & Ye, X. (2017) · Big Data and Cognitive Computing · doi:10.3390/bdcc1010004 · Cited by 37
Future Shock: Generative AI and the International AI Policy and Governance Crisis
Leslie, D., & Perini, A. M. (2024) · Harvard Data Science Review · doi:10.1162/99608f92.88b4cc98 · Cited by 33
Surveillance of communicable diseases using social media: A systematic review
Pilipiec, P., Samsten, I., & Bóta, A. (2023) · PLoS ONE · doi:10.1371/journal.pone.0282101 · Cited by 54
Predicting food crises using news streams
Balashankar, A., Subramanian, L., & Fraiberger, S. P. (2023) · Science Advances · doi:10.1126/sciadv.abm3449 · Cited by 54
“…for selecting news indicators that are predictive, valid, and interpretable could be extended to other domains, from disease surveillance to the impact of climate change, where big data and machine learning are being used to predict policy outcomes in data-scarce environments ( 13 , 25 , 26 ).”
He, F., Hu, Z., Zhang, W., et al. (2017) · Scientific Reports · doi:10.1038/s41598-017-07475-3 · Cited by 32
Using big data to advance personality theory
Bleidorn, W., Hopwood, C. J., & Wright, A. G. C. (2017) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2017.08.004 · Cited by 41
Research on Web Search Behavior: How Online Query Data Inform Social Psychology
Lai, K., Lee, Y. X., Chen, H., & Yu, R. (2017) · Cyberpsychology Behavior and Social Networking · doi:10.1089/cyber.2017.0261 · Cited by 31
The use of ambient humidity conditions to improve influenza forecast
Shaman, J. L., Kandula, S., Yang, W., & Karspeck, A. R. (2017) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1005844 · Cited by 38
Mavragani, A., Sampri, A., Sypsa, K., & Tsagarakis, K. P. (2018) · JMIR Public Health and Surveillance · doi:10.2196/publichealth.8726 · Cited by 31
Machine Decisions and Human Consequences
Scantamburlo, T., Charlesworth, A., & Cristianini, N. (2019) · Oxford University Press eBooks · doi:10.1093/oso/9780198838494.003.0003 · Cited by 56
“15 Overfitting was one of the reasons that led Google Flu trends to predict “more than double the proportion of doctor visits for influenza like-illness (ILI) than the Centers for Disease Control and Prevention (CDC)” (Lazer et al. 2014: 1203) 16 E.g. according to Kantar UK Insights: https://uk.kantar.com/ge2017/2017/…”
BackgroundWell-being through the lens of the internet
Algan, Y., Murtin, F., Beasley, E., Higa, K., & Sénik, C. (2019) · PLoS ONE · doi:10.1371/journal.pone.0209562 · Cited by 37
Risk Analysis in the Age of Big Data: The Promises and Pitfalls
Nateghi, R., & Aven, T. (2021) · Risk Analysis · doi:10.1111/risa.13682 · Cited by 34
“…explanatory (retrospective) power versus (prospective) predictive power could be dangerous, as high-dimensional data lends itself to overfitting, resulting in inferences based in spurious correlations (for an example of a failed big data project attributed to overfitting refer to Lazer, 2014).”
BackgroundPietsch, W. (2021) · Cambridge University Press eBooks · doi:10.1017/9781108588676 · Cited by 50
Sumner, S. A., Bowen, D. A., Holland, K. M., Zwald, M. L., Vivolo-Kantor, A. M., Guy, G. P., Heuett, W. J., Pressley, D., & Jones, C. M. (2022) · JAMA Network Open · doi:10.1001/jamanetworkopen.2022.23033 · Cited by 40
Heat stroke internet searches can be a new heatwave health warning surveillance indicator
Li, T., Ding, F., Sun, Q., Zhang, Y., & Kinney, P. L. (2016) · Scientific Reports · doi:10.1038/srep37294 · Cited by 36
Taxonomy for Humans or Computers? Cognitive Pragmatics for Big Data
Sterner, B., & Franz, N. M. (2017) · Biological Theory · doi:10.1007/s13752-017-0259-5 · Cited by 33
“In contrast, critics have argued that the value of an aggregated dataset cannot be reliably separated from how its component parts were produced (boyd and Crawford 2012; Lazer et al. 2014; Leonelli 2014; Meng 2014).”
BackgroundPence, J., Sakurahara, T., Zhu, X., Mohaghegh, Z., Ertem, M., Ostroff, C. L., & Kee, E. J. (2018) · Reliability Engineering & System Safety · doi:10.1016/j.ress.2018.12.020 · Cited by 41
van den Heuvel, L., Dorsey, R., Prainsack, B., Post, B., Stiggelbout, A. M., Meinders, M. J., & Bloem, B. R. (2019) · Journal of Parkinson s Disease · doi:10.3233/jpd-191712 · Cited by 37
Putting big data in its place: aRegional Studies and Regional Scienceperspective
Rae, A., & Singleton, A. D. (2015) · Regional Studies Regional Science · doi:10.1080/21681376.2014.990678 · Cited by 33
“This would involve a combination of ‘small data’ and big data approaches of the kind recently advocated by Lazer et al. (2014) (2) In relation to method, infrastructure and knowledge, we think there is a real need to increase interdisciplinary thinking and working.”
BackgroundHölzl, J., Keusch, F., & Sajons, C. (2024) · Social Science Research · doi:10.1016/j.ssresearch.2024.103099 · Cited by 63
“These suggestions are partly based on what other users have already googled, leading to possible search externalities and making individual queries dependent on others ’ searches (Lazer et al., 2014).”
Introducing textual analysis tools for policy informatics: a case study of e-petitions
Hagen, L., Harrison, T. M., Uzuner, Ö., Fake, T., LaManna, D., & Kotfila, C. (2015) · Annual International Conference on Digital Government Research · doi:10.1145/2757401.2757421 · Cited by 38
“We do not know whether mentioning location names is structurally correlated (or unrelated) with signature counts [33].”
BackgroundHeiberger, R. H., & Riebling, J. R. (2016) · Methodological Innovations · doi:10.1177/2059799115622763 · Cited by 34
“This is done in other articles in greater detail (Lazer et al., 2009, 2014; Ruths and Pfeffer, 2014).”
MethodNational Surveys of Population Health: Big Data Analytics for Mobile Health Monitors
Schatz, B. R. (2015) · Big Data · doi:10.1089/big.2015.0021 · Cited by 31
“are not adequate for health monitors since they reflect global interest, not local activity, as shown by confounding effects of people in regions with little flu reading articles about people in regions with much flu.(41) Carried phones can accurately monitor personal status,”
BackgroundMoss, R. G., Zarebski, A. E., DAWSON, P., & McCaw, J. M. (2016) · Epidemiology and Infection · doi:10.1017/s0950268816002053 · Cited by 33
“Techniques for estimating true incidence exist [29] and have been applied to, e.g. the 2009 H1N1 pandemic in the UK [30, 31] and the United States [32], but are better suited to retrospective analyses than to providing a benchmark for forecast evaluation during an epidemic.”
BackgroundRohart, F., Milinovich, G. J., Avril, S. M. R., Cao, K. L., Tong, S., & Hu, W. (2016) · Scientific Reports · doi:10.1038/srep38522 · Cited by 33
Measuring Global Disease with Wikipedia: Success, Failure, and a Research Agenda
Priedhorsky, R., Osthus, D., Daughton, A. R., Moran, K. R., Generous, N., Fairchild, G. C., Deshpande, A., & Valle, S. Y. D. (2017) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/2998181.2998183 · Cited by 37
“For example, after significant accuracy problems [24] and scientific criticism [95], Flu and Dengue Trends folded in the summer of 2015 [154].”
BackgroundWhy Expertise Matters: A Response to the Challenges
Klein, G. A., Shneiderman, B., Hoffman, R. R., & Ford, K. M. (2017) · IEEE Intelligent Systems · doi:10.1109/mis.2017.4531230 · Cited by 39
Hadoop Oriented Smart Cities Architecture
Dıaconıța, V., Bologa, A.-R., & Bologa, R. (2018) · Sensors · doi:10.3390/s18041181 · Cited by 33
“…there is the danger of falling into the “Big Data Hubris”, the impression that having large quantities of data and apparently robust algorithms mean that the more traditional ways of ensuring data quality, determining valid and trustworthy dependencies among data, can be overlooked [6,7].”
BackgroundPreparing for the future: integrating spatial ecology into ecosystem-based management
Lowerre‐Barbieri, S. K., Catalán, I. A., Opdal, A. F., & Jørgensen, C. (2019) · ICES Journal of Marine Science · doi:10.1093/icesjms/fsy209 · Cited by 35
“However, AI computer models and “learning feedback loops” have great potential (Hamet and Tremblay, 2017) and have been successfully applied to continuous flows of real-time data from predicting flu epidemics (Lazer et al., 2014) to swaying elections (Gonzalez, 2017).”
Clemente, L., Lu, F., & Santillana, M. (2019) · JMIR Public Health and Surveillance · doi:10.2196/12214 · Cited by 39
Predicting tick-borne encephalitis using Google Trends
Sulyok, M., Richter, H., Sulyok, Z., Kapitány‐Fövény, M., & Walker, M. D. (2019) · Ticks and Tick-borne Diseases · doi:10.1016/j.ttbdis.2019.101306 · Cited by 31
Big geodata mining: Objective, connotations and research issues
Pei, T., Song, C., Guo, S., Shu, H., Liu, Y., Du, Y., Ma, T., & Zhou, C. (2020) · Journal of Geographical Sciences · doi:10.1007/s11442-020-1726-7 · Cited by 34
The social life of COVID-19: Early insights from social media monitoring data collected in Poland
Burzyńska, J., Bartosiewicz, A., & Rękas, M. (2020) · Health Informatics Journal · doi:10.1177/1460458220962652 · Cited by 37
Universal patterns in passenger flight departure delays
Wang, Y., Cao, Y., Zhu, C.-P., Wu, F., Hu, M., Dương, V., Watkins, M., Barzel, B., & Stanley, E. (2020) · Scientific Reports · doi:10.1038/s41598-020-62871-6 · Cited by 31
Applications and challenges of AI-based algorithms in the COVID-19 pandemic
Khemasuwan, D., & Colt, H. G. (2021) · BMJ Innovations · doi:10.1136/bmjinnov-2020-000648 · Cited by 37
Rabiolo, A., Alladio, E., Morales, E., McNaught, A. I., Bandello, F. M., Afifi, A. A., & Marchese, A. (2021) · Journal of Medical Internet Research · doi:10.2196/28876 · Cited by 34
Engines of Order: A Mechanology of Algorithmic Techniques
Rieder, B. (2020) · Amsterdam University Press eBooks · doi:10.1515/9789048537419 · Cited by 103
Internet of connected ServGoods: Considerations, consequences and concerns
Tien, J. M. (2015) · Journal of Systems Science and Systems Engineering · doi:10.1007/s11518-015-5273-1 · Cited by 31
Nowcasting Intraseasonal Recreational Fishing Harvest with Internet Search Volume
Carter, D. W., Crosson, S., & Liese, C. (2015) · PLoS ONE · doi:10.1371/journal.pone.0137752 · Cited by 35
Koplenig, A., & Müller-Spitzer, C. (2016) · PLoS ONE · doi:10.1371/journal.pone.0150771 · Cited by 30
Assessing the impact of a health intervention via user-generated Internet content
Lampos, V., Yom‐Tov, E., Pebody, R. G., & Cox, I. J. (2015) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-015-0427-9 · Cited by 33
Metadata
- Authors
- Lazer, D., Kennedy, R., King, G., & Vespignani, A.
- Affiliations
- Northeastern University (US)
- Harvard University (US)
- Corresponding author
- Lazer, David
- Venue
- Science
- Published
- 2014-03-14
- Volume
- 343
- Issue
- 6176
- Pages
- 1203-1205
- ISSN
- 0036-8075 (ISSN-L)
- 1095-9203
- OpenAlex
- W2068181924
- Semantic Scholar
- 00d681cfff7f0224ace607e515f4ed510e792df9
- Cited by
- 2,523
- References
- 30
- Citations by year
- 2026: 113 · 2025: 101 · 2024: 121 · 2023: 150 · 2022: 168 · 2021: 229 · 2020: 273 · 2019: 261 · 2018: 272 · 2017: 249 · 2016: 271 · 2015: 219 · 2014: 91
- FWCI
- 208.16
- Type
- journal article
- Language
- en
- Open access
- green
- dash.harvard.edu
- Licence
- cc-by
- Locations
- Science (publishedVersion)
- PubMed (publishedVersion)
- Digital Access to Scholarship at Harvard (DASH) (Harvard University) (submittedVersion, cc-by)
- Topics
- Health Sciences › Medicine › Epidemiology › Data-Driven Disease Surveillance
- Social Sciences › Decision Sciences › Information Systems and Management › Big Data Technologies and Applications
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- Google Flu Trends, influenza surveillance, big data, prediction error, data quality
- MeSH terms
- Centers for Disease Control and Prevention, U.S., Data Collection, Data Interpretation, Statistical, Disease Outbreaks, Humans, Influenza, Human, Office Visits, Prognosis, United States, Social Media
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.