The Parable of Google Flu: Traps in Big Data Analysis
Open accessThe DOI matches its Crossref record; 30 of the 31 references it cites were found in a registry or an index; 5 of the 8 sources its claims cite were checked for support.
Abstract
Large errors in flu prediction were largely avoidable, which offers lessons for the use of big data.
Overview
CoChat Summary
TL;DR
The paper uses the failure of Google Flu Trends (GFT) to illustrate two traps in big data analysis: “big data hubris” and algorithm dynamics. It argues that GFT’s large flu-prediction errors were largely avoidable and that big data should supplement, not replace, traditional data and methods.
Key findings
- In February 2013, GFT predicted more than double the proportion of doctor visits for influenza-like illness (ILI) than the CDC; from 21 August 2011 to 1 September 2013, GFT reported overly high flu prevalence in 100 out of 108 weeks, and it overshot the 2011–2012 season by more than 50%.
- GFT errors were not random: they showed temporal autocorrelation and seasonality, meaning last week’s errors predicted this week’s errors and error direction/magnitude varied with time of year.
- Simple alternatives outperformed GFT: out-of-sample mean absolute error was 0.486 for GFT, 0.311 for lagged CDC, and 0.232 for combined GFT and CDC; all differences were statistically significant at P < 0.05. Even 3-week-old CDC data did better than GFT at projecting current flu prevalence.
- The authors attribute GFT’s problems to overfitting (initial GFT matched 50 million search terms to 1152 data points and missed the nonseasonal 2009 H1N1 pandemic) and to “algorithm dynamics,” including Google search algorithm changes and user behavior changes.
Methods
- Design: policy forum/case study comparing GFT estimates with CDC ILI surveillance data and alternative forecasting models.
- Data/sample: GFT and CDC flu prevalence estimates, including the period from 21 August 2011 to 1 September 2013; the initial GFT methodology reportedly fit 50 million search terms to 1152 data points.
- Main analysis: comparison of GFT alone, lagged CDC data with 52-week seasonality variables, and a combined GFT + CDC model using lagged CDC estimates, lagged GFT error, and seasonality variables; reported mean absolute error and statistical significance; Google Correlate used to examine correlated search terms. Supplementary materials contain data and methods.
Limitations
- The paper states that GFT never documented the 45 search terms used, released examples appear misleading, and Google Correlate is limited to national-level data whereas GFT was developed using regional correlations; therefore the original GFT analyses cannot be replicated from the information provided.
- The paper states that Google cannot make its full data available to outsiders because of privacy issues, though it argues there is no such constraint on derivative, aggregated data.
- Evident from the text: this is a commentary/policy forum rather than a full empirical article, with key data and methods placed in supplementary materials; the quantitative comparison depends on GFT and CDC data and model specifications not fully detailed in the main text.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
2 Valid · 2 Not valid · 2 Not checked · 8 Own results and interpretations
Some cited sources support this claim and some don’t — read each source’s verdict below.
In the paper
“This ad hoc method of throwing out peculiar search terms failed when GFT completely missed the nonseasonal 2009 infl uenza A–H1N1 pandemic ( 2, 14).”
What the sources say
- Partly supportedcites [2] matched to “Reassessing Google Flu Trends Data for Detection of Seasonal and Pandemic Influenza: A Comparative Epidemiological Study at Three Geographic Scales” (2013) sourceExperimental — accuracy not yet measured
The Discussion (page unavailable) says the original GFT model missed the spring 2009 pandemic wave, supporting the claim about its failure to detect the nonseasonal outbreak. It does not say GFT completely missed the entire 2009 pandemic. The Introduction describes automated selection of the original model’s search terms, not an ad hoc method of throwing out peculiar terms. (LLM, conf=0.86)
- Disputedcites [14] S. Cook et al., PLOS ONE 6, e23610 (2011). sourceExperimental — accuracy not yet measured
The Abstract (Methodology/Principal Findings) says the original GFT model underestimated the magnitude of illness during the 2009 H1N1 pandemic, not that it completely missed it. The Abstract (Conclusions) says both models performed well during the pandemic, although the updated model performed better. The Discussion notes that pandemic-specific queries were excluded, but does not describe this as an ad hoc method of throwing out peculiar terms. (LLM, conf=0.88)
Checked by a reasoning model — may be wrong.
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“Oddly, the few search terms offered in the papers ( 14) do not seem to be strongly related with either GFT or the CDC data (SM)—we surmise that the authors felt an unarticulated need to cloak the actual search terms identifi ed.”
What the sources say
- Disputedcites [14] S. Cook et al., PLOS ONE 6, e23610 (2011). sourceExperimental — accuracy not yet measured
The Results section (‘Search Behavior During pH1N1’; page unavailable) says that all three search queries shown in Figure 3 closely tracked CDC ILINet data before the pandemic, and ‘symptoms of flu’ continued to do so during it. The paper does report that the bronchitis and pneumonia queries underestimated ILINet data during the pandemic, but that does not support the broad claim that the offered terms were not strongly related to the data. (LLM, conf=0.87)
Checked by a reasoning model — may be wrong.
hedgedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“The Internet has opened the way for improving standard surveys, experiments, and health reporting ( 35).”
What the sources say
- Partly supportedcites [35] A. J. Berinsky et al., Polit. Anal. 20, 351–368 (2012). sourceExperimental — accuracy not yet measured
Section 2 describes using an Internet platform to run surveys with embedded experiments, and the Conclusion says MTurk can make experimental research easier and less expensive. The paper does not address improvements to health reporting; the disease scenario in section 4.3.2 is a framing experiment, not health reporting. Page numbers are unavailable in our copy. (LLM, conf=0.88)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Although not widely reported until 2013, the new GFT has been persistently overestimating flu prevalence for a much longer time.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“From 21 August 2011 to 1 September 2013, GFT reported overly high fl u prevalence 100 out of 108 weeks.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
3 more claims
Create a free account to see all 14 claims and every source verdict.
- It may state two separate claims as one. 5
- Probably not a substantial claim. 2
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Google Trends predicted more than twice the number of influenza-like illnesses than the CDC in February 2013 46 .”
From Unrepresentative big surveys significantly overestimated US vaccine uptake (2021) · cites this paper as [46]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This was one of the main reasons behind the failure of Google Flu Trends, since the data used for training the model in the first few years was not representative of the trends in subsequent years [15].”
From Theory-Guided Data Science: A New Paradigm for Scientific Discovery from Data (2017) · cites this paper as [15]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Social data research often relies on counting entities. These entities can be users, links, or messages, and the description is a summary of these counts (Lazer et al., 2014; Salganik, 2017). Yet, simple counts can mislead if it is unclear what is counted and how.”
From Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019) · cites this paper as Lazer et al., 2014
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This unintelligibility is also subject to functional biases (section 4.1), as it depends on the mechanisms available on each social platform (e.g., having a like button, but not a dislike one), as well as to variations in platform algorithms and mechanisms in response to users actions (Lazer et al., 2014).”
From Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019) · cites this paper as Lazer et al., 2014
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“It also includes temporal validity, which captures to what extent constructs change over time (Howison et al., 2011) and may invalidate previous conclusions about societal and/or platform phenomena; e.g., see the case of Google Flu Trends (Lazer et al., 2014).”
From Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019) · cites this paper as Lazer et al., 2014
6 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 14 MARCH 2014 VOL 343 SCIENCE www.sciencemag.org 1204
“The 2009 version of GFT has weathered other media panics related to the fl u, including the 2005–2006 influenza A/H5N1 ("bird flu") outbreak”
“GFT engineers updated the algorithm in 2009, and this model has run ever since”
Citations
Cites
boyd, D., & Crawford, K. (2012) · Information, Communication & Society · doi:10.1080/1369118x.2012.678878 · Cited by 5,479
Twitter mood predicts the stock market
Bollen, J., Mao, H., & Zeng, X. (2011) · Journal of Computational Science · doi:10.1016/j.jocs.2010.12.007 · Cited by 5,206
Detecting influenza epidemics using search engine query data
Ginsberg, J., Mohebbi, M. H., Patel, R. S., Brammer, L., Smolinski, M. S., & Brilliant, L. (2009) · Nature · doi:10.1038/nature07634 · Cited by 4,295
Cohen, D. (1998) · The Women's Review of Books · doi:10.2307/4022859 · Cited by 4,261
Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk
Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012) · Political Analysis · doi:10.1093/pan/mpr057 · Cited by 4,008
Big Data : The Management Revolution
マカフィー, ア., ブリニョルフソン, エ., & 有賀, 裕. (2013) · Harvard business review · Cited by 3,549
Lazer, D., Pentland, A., Adamic, L., Aral, S., Barabási, A.-L., Brewer, D., Christakis, N., Contractor, N., Fowler, J., Gutmann, M., et al. (2009) · Science · doi:10.1126/science.1167742 · Cited by 3,365
Self-citationPredicting Elections with Twitter: What 140 Characters Reveal about Political Sentiment
Tumasjan, A., Sprenger, T., Sandner, P., & Welpe, I. (2010) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v4i1.14009 · Cited by 2,649
Multiscale mobility networks and the spatial spreading of infectious diseases
Balcan, D., Colizza, V., Gonçalves, B., Hu, H., Ramasco, J. J., & Vespignani, A. (2009) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0906910106 · Cited by 1,380
Self-citationPredicting consumer behavior with web search
Goel, S., Hofman, J. M., Lahaie, S., Pennock, D. M., & Watts, D. J. (2010) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1005962107 · Cited by 636
Research works printed but not linked: 7
- A. McAfee, E. Brynjolfsson, Harv. Bus. Rev. 90, 60 (2012).
- AAAI Conference on Weblogs and Social Media, Atlanta, Georgia, 11 to 15 July 2010 (Association for Advancement of Artifi cial Intelligence, 2010), pp. 178.
- P. Copeland et al., Int. Soc. Negl. Trop. Dis. 2013, 3
- I. M. Hall et al., Epidemiol. Infect. 135, 372 (2007).
- E. Mustafaraj, P. Metaxas, in Proceedings of the Web- Sci10, Raleigh, NC, 26 and 27 April 2010 (Web Science Trust, 2010); http://journal.webscience.org/317/.
- J. Ratkiewicz et al., in Proceedings of 5th International AAAI Conference on Weblogs and Social Media, San Francisco, CA, 7 to 11 August 2011 (AAAI, 2011), p. 297–304.
- 2013; http://chronicle.com/article/Researchers-Struggle- to-Secure/141591/.
Cited by
Items 1,001–1,100 of 2,513
Mortality, morbidity and health in developed societies: a review of data sources
Wunsch, G. J., & Gourbin, C. (2018) · Genus · doi:10.1186/s41118-018-0027-9 · Cited by 27
“A good (i.e. bad!) example of erroneous causal inference is the wellknown Google Flu Trends study, matching search terms and flu propensity; the structurally unrelated matches were much too numerous (Lazer et al.”
BackgroundKeller‐McNulty, S., Korkmaz, G., Robbins, C., et al. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1800467115 · Cited by 16
Hope, hype and harms of Big Data
Scott, I. A. (2019) · Internal Medicine Journal · doi:10.1111/imj.14172 · Cited by 15
Air Quality Forecast through Integrated Data Assimilation and Machine Learning
Lin, H. X., Jin, J., & van den Herik, H. J. (2019) · International Conference on Agents and Artificial Intelligence (ICAART) · doi:10.5220/0007555207870793 · Cited by 16
“However, we have to admit that the notion of a black-box application within data science has so far met only limited success, e.g., (Caldwell et al., 2014; Lazer et al., 2014).”
BackgroundBig data: critical questions for sport and society
Spaaij, R., & Thiel, A. (2017) · European Journal for Sport and Society · doi:10.1080/16138171.2017.1288374 · Cited by 16
“This includes the (often implicit) assumption that big data are a substitute for, rather than a supplement to, traditional data collection and analysis (Lazer et al., 2014).”
BackgroundEditorial: Big data and machine learning in sociology
Leitgöb, H., Prandner, D., & Wolbring, T. (2023) · Frontiers in Sociology · doi:10.3389/fsoc.2023.1173155 · Cited by 28
“, 2014) illustrated the demand for a flexible methodological framework with theory, traditional data sources and methods, as well as DBD and algorithmic approaches as complementary elements to be integrated to maximize knowledge gain (Lazer et al., 2014; Schnell, 2019).”
BackgroundConsiderations on explainable AI and users’ mental models
Rutjes, H., Willemsen, M. C., & IJsselsteijn, W. A. (2019) · TU/e Research Portal · Cited by 62
Häckel, B. S., Karnebogen, P., & Ritter, C. (2021) · Decision Support Systems · doi:10.1016/j.dss.2021.113653 · Cited by 20
Wang, S., Wu, Y. J., & Li, R. (2022) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph19159752 · Cited by 15
Jia, X. (2023) · Ecology and Society · doi:10.5751/es-14125-280216 · Cited by 18
Nowcasting with panels and alternative data: The OECD weekly tracker
Woloszko, N. (2023) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2023.11.005 · Cited by 23
Woelmer, W. M., Thomas, R. Q., Olsson, F., Steele, B. G., Weathers, K. C., & Carey, C. C. (2024) · Ecological Informatics · doi:10.1016/j.ecoinf.2024.102825 · Cited by 18
Harmonizing Multisource Data to Inform Vector-Borne Disease Risk Management Strategies
Lowe, R., & Codeço, C. T. (2024) · Annual Review of Entomology · doi:10.1146/annurev-ento-040124-015101 · Cited by 26
Pharmacovigilance for a Revolving World: Prospects of Patient-Generated Data on the Internet
Norén, G. N. (2014) · Drug Safety · doi:10.1007/s40264-014-0205-4 · Cited by 16
“This has been placed in new light following recent analyses indicating that the predictive value of internet search patterns alone was previously over-estimated; however, their combination with traditional surveillance methods has been found to outperform each method when used separately [17].”
BackgroundMethodDatafication and the Seductive Power of Uncertainty—A Critical Exploration of Big Data Enthusiasm
Strauß, S. (2015) · Information · doi:10.3390/info6040836 · Cited by 24
Shi, Y., Shan, Z., Li, J., et al. (2017) · Annals of Data Science · doi:10.1007/s40745-017-0129-9 · Cited by 15
Damangir, S., Du, R. Y., & Hu, Y. (2018) · Journal of Interactive Marketing · doi:10.1016/j.intmar.2017.11.002 · Cited by 17
“Such noises are inevitable when online activities are used as proxies of intents and no other contextual information is available about the individual behind those activities (Lazer et al. 2014).”
BackgroundInfluentialWang, H.-W., & Chen, D. (2018) · JMIR Public Health and Surveillance · doi:10.2196/publichealth.7314 · Cited by 17
Emergence of polarized opinions from free association networks
File, B., Keczer, Z., Vancsó, A., Bőthe, B., Tóth‐Király, I., Hunyadi, M., Ujhelyi, A., Ulbert, I., Góth, J., & Orosz, G. (2018) · Behavior Research Methods · doi:10.3758/s13428-018-1090-z · Cited by 16
Using Google Health Trends to investigate COVID-19 incidence in Africa
Fulk, A., Romero-Álvarez, D., Abu-Saymeh, Q., Onge, J. M. S., Peterson, A. T., & Agusto, F. B. (2022) · PLoS ONE · doi:10.1371/journal.pone.0269573 · Cited by 20
Xu, W. W., Tshimula, J. M., Dubé, È., Graham, J., Greyson, D. L., MacDonald, N. E., & Meyer, S. B. (2022) · JMIR Infodemiology · doi:10.2196/41198 · Cited by 26
Societal Attention to Heat Waves Can Indicate Public Health Impacts
Bogdanovich, E., Guenther, L., Reichstein, M., Frank, D., Ruhrmann, G., Brenning, A., Denissen, J., & Orth, R. (2023) · Weather Climate and Society · doi:10.1175/wcas-d-22-0147.1 · Cited by 14
“First, Google algorithms are not transparent and change through time (Nuti et al. 2014; Lazer et al. 2014).”
MethodWiki-worthy: collective judgment of candidate notability
Margolin, D., Goodman, S., Keegan, B., Lin, Y., & Lazer, D. M. J. (2015) · Information Communication & Society · doi:10.1080/1369118x.2015.1069871 · Cited by 15
“Changes or ‘drifts’ in Wikipedia norms or the extent of its use might shift these relationships (Lazer et al., 2014).”
BackgroundSelf-citationORBiT: Oak Ridge biosurveillance toolkit for public health dynamics
Ramanathan, A., Pullum, L., Hobson, T. C., Steed, C. A., Quinn, S. P., Chennubhotla, C. S., & Valkova, S. (2015) · BMC Bioinformatics · doi:10.1186/1471-2105-16-s17-s4 · Cited by 15
“While initial studies have shown that tools that make use of “proxy” datasets can serve as useful monitors for emerging diseases [14-16,8,17], recent studies have demonstrated that the estimates from internet search patterns can over-estimate the severity of the outbreak [18,19].”
ResultPence, J., Mohaghegh, Z., Ostroff, C. L., Kee, E. J., Yılmaz, F., Grantom, R., & Johnson, D. (2014) · Adelaide Research & Scholarship (AR&S) (University of Adelaide) · doi:10.13140/rg.2.1.1523.3760 · Cited by 16
New data strategies: nonprobability sampling, mobile, big data
Link, M. (2018) · Quality Assurance in Education · doi:10.1108/qae-06-2017-0029 · Cited by 15
Marjanovic, O., Cecez-Kecmanovic, D., & Vidgen, R. (2018) · IFIP Advances in Information and Communication Technology · doi:10.1007/978-3-030-04091-8_4 · Cited by 15
“While machine learning is an un-doubtedly powerful technique, it is not without dangers, as evidenced by the failure of Google flu, which failed as a result of model over-fitting, relying on unreliable data, and failing to update the model to take account of underlying changes [20].”
Altenburger, K. M., & Ho, D. E. (2019) · International World Wide Web Conference (WWW) · doi:10.1145/3308558.3313683 · Cited by 23
“As put in Science, big data “does not mean that one can ignore measurement and construct validity and reliability” [36].”
BackgroundFace Recognition Technologies: Designing Systems that Protect Privacy and Prevent Bias
Yeung, D., Balebako, R., Gaviria, C. G., & Chaykowsky, M. (2020) · RAND Corporation eBooks · doi:10.7249/rr4226 · Cited by 22
Forecasting tuberculosis using diabetes-related google trends data
Frauenfeld, L., Nann, D., Sulyok, Z., Feng, Y., & Sulyok, M. (2020) · Pathogens and Global Health · doi:10.1080/20477724.2020.1767854 · Cited by 16
“Despite the failure of GFT, the number of studies utilizing Google searches has increased [41, 42, 43, 27, 28, 29, 44, 45, 30].”
BackgroundResultInfluentialFractional Diffusion Emulates a Human Mobility Network during a Simulated Disease Outbreak
Gustafson, K., Bayati, B. S., & Eckhoff, P. A. (2017) · Frontiers in Ecology and Evolution · doi:10.3389/fevo.2017.00035 · Cited by 13
“While the predictive power of GFT data has recently been challenged (Lazer et al., 2014), the qualitative pattern of the GFT data is descriptive of influenza”
BackgroundBig Data for Advancing Dementia Research
Deetjen, U., Meyer, E. T., & Schroeder, R. (2015) · OECD digital economy papers · doi:10.1787/5js4sbddf7jk-en · Cited by 15
“, 2013), to using Google search data to predict everything from flu (Lazer et al., 2014) to house prices (Wu and Brynjolfsson, 2013).”
BackgroundMethodDynamic Bayesian Influenza Forecasting in the United States with Hierarchical DiscrepancyPreprint
Osthus, D., Gattiker, J., Priedhorsky, R., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1708.09481 · Cited by 15
“Recently, the value of digital surveillance data with respect to flu forecasting has been curbed [e.g., Lazer et al., 2014, Priedhorsky et al., 2017].”
BackgroundMunk, A. K., Jacomy, M., Ficozzi, M., & Jensen, T. E. (2024) · Big Data & Society · doi:10.1177/20539517241255107 · Cited by 25
“…disturbances ” (Whatmore, 2009), of events that render knowledge claims available for scrutiny, such as the failure of Google Flu Trends (Lazer et al., 2014), the crash of the self-driving Tesla Model S (Stilgoe, 2018) or the botched introduction of AI-enhanced photography in Huawei…”
Epidemiological methods in transition: Minimizing biases in classical and digital approaches
Mesquita, S., Perfeito, L., Paolotti, D., & Gonçalves-Sá, J. (2025) · PLOS Digital Health · doi:10.1371/journal.pdig.0000670 · Cited by 26
Machine Learning Modeling: A New Way to do Quantitative Research in Social Sciences in the Era of AI
Zhang, J., & Feng, S. (2021) · Journal of Web Engineering · doi:10.13052/jwe1540-9589.2023 · Cited by 19
Infodemiology of Influenza-like Illness: Utilizing Google Trends’ Big Data for Epidemic Surveillance
Shih, D., Wu, Y.-H., Wu, T.-W., Chang, S.-C., & Shih, M. (2024) · Journal of Clinical Medicine · doi:10.3390/jcm13071946 · Cited by 17
“Previous studies have pointed out that there are potential errors in models constructed using only internet search data or climate factors [5,30–33].”
BackgroundTime evolution of predictability of epidemics on networks
Holme, P., & Takaguchi, T. (2015) · Physical Review E · doi:10.1103/physreve.91.042811 · Cited by 14
Bayesian data fusion approaches to predicting spatial tracks: Application to marine mammals
Liu, Y., Zidek, J. V., Trites, A. W., & Battaile, B. C. (2016) · The Annals of Applied Statistics · doi:10.1214/16-aoas945 · Cited by 14
“Recent technological advances have made it feasible to track many things, such as the foraging trips of endangered animals [Wilson et al. (2007)], the movements of basketball players Miller et al. (2014) and the spread of infectious diseases [Ginsberg et al. (2009), Lazer et al. (2014a)].”
BackgroundMethodPublic Health Emergency Management and Multi-Source Data Technology in China
Wang, X., Liu, Y., Zhang, H., Ma, Q., & Cao, Z. (2017) · Intelligent Automation & Soft Computing · doi:10.1080/10798587.2016.1267246 · Cited by 14
An Information Theory-Based Scheme for Efficient Classification of Remote Sensing Data
Marinoni, A., Iannelli, G. C., & Gamba, P. E. (2017) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2017.2716187 · Cited by 14
“However, when large-scale and/or heterogeneous data sets are considered, these assumptions might not hold anymore, and very complex relationships among records could show up, such as non-Gaussian noise distribution, nonEuclidean pattern clustering, and nonlinear projection between the samples and their corresponding l…”
BackgroundArendt, F., & Fawzi, N. (2018) · Information Communication & Society · doi:10.1080/1369118x.2018.1473459 · Cited by 14
Mitchell, S. S. D. (2018) · Online Information Review · doi:10.1108/oir-03-2018-0075 · Cited by 15
Nonannual seasonality of influenza‐like illness in a tropical urban setting
Lam, H. M., Wesolowski, A., Hùng, N. T., Nguyen, T. D., Nhat, N. T. D., Todd, S., Vinh, D. N., Vy, N. H. T., Thao, T. T. N., Thanh, N. T. L., et al. (2018) · Influenza and Other Respiratory Viruses · doi:10.1111/irv.12595 · Cited by 13
A monetary policy prediction model based on deep learning
Lu, M. (2019) · Neural Computing and Applications · doi:10.1007/s00521-019-04319-1 · Cited by 16
Lifestyle Disease Surveillance Using Population Search Behavior: Feasibility Study
Memon, S. A., Razak, S., & Weber, I. G. (2019) · Journal of Medical Internet Research · doi:10.2196/13347 · Cited by 14
Predictability limit of partially observed systems
Abeliuk, A., Huang, Z., Ferrara, E., & Lerman, K. (2020) · Scientific Reports · doi:10.1038/s41598-020-77091-1 · Cited by 18
Moving beyond operations: Leveraging big data for urban planning decisions
French, S., Barchers, C., & Zhang, W. (2015) · Annual Conference on Computers · Cited by 14
Exploring the effect of streamed social media data variations on social network analysis
Weber, D., Nasim, M., Mitchell, L., & Falzon, L. (2021) · Social Network Analysis and Mining · doi:10.1007/s13278-021-00770-y · Cited by 21
“Taking a purely “big data” approach can also lead to inappropriate generalisations and conclusions (Lazer et al. 2014; Tufekci 2014).”
BackgroundEthical Issues in the Use of Big Data for Social Research
Weinhardt, M. (2020) · Historical social research · doi:10.12759/hsr.45.2020.3.342-368 · Cited by 28
Algorithms and Discrimination: The Case of Credit Scoring in Brazil
Mendes, L. S., & Mattiuzzo, M. (2022) · Ius gentium · doi:10.1007/978-3-030-90331-2_17 · Cited by 14
Christophersen, M., Mørck, P., Langhoff, T. O., & Bjørn, P. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-20684-4_28 · Cited by 20
Digital supply chain surveillance
Brintrup, A. M., Kosasih, E. E., MacCarthy, B. L., & Demirel, G. (2022) · Elsevier eBooks · doi:10.1016/b978-0-323-91614-1.00022-8 · Cited by 19
Who caught a cold ? - Identifying the subject of a symptom
Kanouchi, S., Komachi, M., Okazaki, N., Aramaki, E., & Ishikawa, H. (2015) · Annual Meeting of the Association for Computational Linguistics · doi:10.3115/v1/p15-1160 · Cited by 14
“” Lazer et al. (2014) reported large errors in Google Flu Trends (Carneiro and Mylonakis, 2009) on the basis of a comparison with the proportion of doctor visits for influenza-like illnesses.”
BackgroundICARES: a real-time automated detection tool for clusters of infectious diseases in the Netherlands
Groeneveld, G. H., Dalhuijsen, A., Kara‐Zaitri, C., Hamilton, B., de Waal, M. W. M., van Dissel, J. T., & van Steenbergen, J. (2017) · BMC Infectious Diseases · doi:10.1186/s12879-017-2300-5 · Cited by 16
The Salience and Urgency of Enterprise Data Management In the Public Sector
Harrison, T. M., Pardo, T. A., Gascó‐Hernández, M., et al. (2018) · Proceedings of the ... Annual Hawaii International Conference on System Sciences/Proceedings of the Annual Hawaii International Conference on System Sciences · doi:10.24251/hicss.2018.282 · Cited by 14
Big Data in soziologischer Perspektive
Weyer, J., Delisle, M., Kappler, K. E., Kiehl, M., Merz, C., & Schrape, J.-F. (2018) · Technikzukünfte, Wissenschaft und Gesellschaft · doi:10.1007/978-3-658-21665-8_2 · Cited by 24
Global Research on Syndromic Surveillance from 1993 to 2017: Bibliometric Analysis and Visualization
Musa, I., Park, H. W., Munkhdalai, L., & Ryu, K. H. (2018) · Sustainability · doi:10.3390/su10103414 · Cited by 16
Toward Area‐Smart Data Science: Critical Questions for Working With Big Data From China
Stockmann, D. (2016) · Policy & Internet · doi:10.1002/poi3.192 · Cited by 16
Zhang, S., Zhao, B., Tian, Y., & Chen, S. (2020) · Annals of the American Association of Geographers · doi:10.1080/24694452.2020.1782166 · Cited by 18
“We agree with many that the so-called big data revolution is indeed a revolution (Honig 2012; Silver 2012; Barnes 2013; Lazer et al. 2014) but argue that it is not simply due to the abundance and easy availability of data.”
BackgroundPredicting seasonal influenza using supermarket retail records
Miliou, I., Xiong, X., Rinzivillo, S., Zhang, Q., Rossetti, G., Giannotti, F., Pedreschi, D., & Vespignani, A. (2021) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1009087 · Cited by 20
Self-citationTriggering Interventions for Influenza: The ALERT Algorithm
Reich, N. G., Cummings, D. A. T., Lauer, S. A., Zorn, M., Robinson, C. C., Nyquist, A., Price, C. S., Simberkoff, M. S., Radonovich, L. J., & Perl, T. M. (2014) · Clinical Infectious Diseases · doi:10.1093/cid/ciu749 · Cited by 13
“However, Google Flu Trends has shown poor performance in estimating the burden of influenza [17], correlates better with influenza-like illness than with laboratory-confirmed influenza [18], cannot accommodate data from healthcare settings, and is considered a supplementary rather than authoritative source for influen…”
BackgroundText As DataPreprint
Gentzkow, M. A., & Kelly, B. T. (2017) · SSRN Electronic Journal · doi:10.2139/ssrn.2934001 · Cited by 18
Rai, A. (2016) · MIS Quarterly · doi:10.25300/misq/2016/40.2.e0 · Cited by 22
Sources and Types of Big Data for Macroeconomic Forecasting
Garboden, P. M. E. (2019) · Advanced studies in theoretical and applied econometrics · doi:10.1007/978-3-030-31150-6_1 · Cited by 15
“Although attempts have not been wholly successful (Lazer et al., 2014), many researchers have attempted to use search terms for disease symptoms to track epidemics in real time (Ginsberg et al.”
BackgroundInfluentialData, global development, and COVID-19: Lessons and consequences
Naudé, W. A., & Vinuesa, R. (2020) · WIDER Working Paper Series · doi:10.35188/unu-wider/2020/866-5 · Cited by 15
Culturas algorítmicas: conceptos y métodos para su estudio social
Sued, G. (2022) · Revista Mexicana de Ciencias Políticas y Sociales · doi:10.22201/fcpys.2448492xe.2022.246.78422 · Cited by 36
Patton, T., Abramovitz, D. A., Johnson, D. C., Leas, E. C., Nobles, A. L., Caputi, T. L., Ayers, J. W., Strathdee, S. A., & Bórquez, A. (2022) · Journal of Medical Internet Research · doi:10.2196/41527 · Cited by 17
“Search patterns change over time due to the thousands of decisions being made by Google’s programmers as the company strives to test and improve its search algorithm [40].”
BackgroundInference with non-probability samples and survey data integration: a science mapping study
Salvatore, C. (2023) · METRON · doi:10.1007/s40300-023-00243-6 · Cited by 22
“There are, however, selectivity and measurement issues that cannot be ignored, as demonstrated by the Google Flu experiment, which initially appeared promising but then failed to predict outbreaks [55].”
BackgroundGeneral theory of data, artificial intelligence and governance
de Pedraza, P., & Vollbracht, I. (2023) · Humanities and Social Sciences Communications · doi:10.1057/s41599-023-02096-w · Cited by 23
Allen, B., Schell, R. C., Jent, V. A., Krieger, M. S., Pratty, C., Hallowell, B. D., Goedel, W. C., Basta, M., Yedinak, J. L., Li, Y., et al. (2024) · Epidemiology · doi:10.1097/ede.0000000000001695 · Cited by 14
Geographical big data and data mining: A new opportunity for “water-energy-food” nexus analysis
Yang, J., Cao, X., Yao, J. P., Kang, Z., Chang, J., & Wang, Y. (2024) · Journal of Geographical Sciences · doi:10.1007/s11442-024-2202-6 · Cited by 16
“The existence of errors may lead to cognitive biases and even fallacies, as demonstrated by Google’s success and failure in flu prediction (Lazer et al ., 2014).”
Aliferis, C., & Simon, G. (2024) · Health informatics · doi:10.1007/978-3-031-39355-6_12 · Cited by 19
Wyatt, K. D., Alexander, N., Hills, G. D., Liang, W. H., Kadauke, S., Volchenboum, S. L., Mian, A., & Phillips, C. (2024) · Pediatric Blood & Cancer · doi:10.1002/pbc.31143 · Cited by 16
Luo, J., Wang, X., Fan, X., He, Y., Du, X., Chen, Y.-Q., & Zhao, Y. (2025) · BMC Public Health · doi:10.1186/s12889-025-21618-6 · Cited by 22
“However, the reliability of GFT’s predictions was compromised by changes in Google’s search algorithms and user behavior [10].”
Callahan, A., Pernek, I., Štiglic, G., Leskovec, J., Strasberg, H. R., & Shah, N. H. (2015) · Journal of Medical Internet Research · doi:10.2196/jmir.4427 · Cited by 13
Empirical Study of User Preferences Based on Rating Data of Movies
Zhao, Y., & Shen, B. (2016) · PLoS ONE · doi:10.1371/journal.pone.0146541 · Cited by 14
“Furthermore, researches about big data also imply that the prediction error would be further reduced by combining the historical data based prediction with other nearreal-time data, such as feedback of users [54, 55].”
BackgroundBig Data Infrastructures for Processing Sentinel Data
Week, P., & Fritsch, D. (2015) · Cited by 13
Limits of use of social media for monitoring biosecurity events
Welvaert, M., Al-Ghattas, O., Cameron, M. A., et al. (2017) · PLoS ONE · doi:10.1371/journal.pone.0172457 · Cited by 13
The Opportunities and Challenges with Social Media and Big Data for Research in Human Dynamics
Nara, A., Tsou, M., Yang, J., & Huang, C.-C. (2018) · Human dynamics in smart cities · doi:10.1007/978-3-319-73247-3_12 · Cited by 15
Era of Intelligent Systems in Healthcare
Belciug, S., & Gorunescu, F. (2019) · Intelligent systems reference library · doi:10.1007/978-3-030-14354-1_1 · Cited by 18
“The problems in implementation must be identified and ameliorated [153, 154] , in order to build up confidence.”
BackgroundDodds, P. S., Minot, J. R., Arnold, M., Alshaabi, T., Adams, J. L., Dewhurst, D. R., Reagan, A. J., & Danforth, C. M. (2022) · Journal of Quantitative Description Digital Media · doi:10.51685/jqd.2022.004 · Cited by 18
A Large-Scale Characterization of How Readers Browse Wikipedia
Piccardi, T., Gerlach, M., Arora, A., & West, R. (2023) · ACM Transactions on the Web · doi:10.1145/3580318 · Cited by 21
“Previous work indicates that big-data analyses are not immune to biases introduced by algorithmic dynamics [38, 74], data collection problems, preprocessing errors, and measurement errors [37, 74].”
BackgroundLiu, B. (2014) · Frontiers of Medicine · doi:10.1007/s11684-014-0364-9 · Cited by 13
“In September 2008, a special issue was published by Nature devoted to big data storage, management, and analysis, and it signified the coming of the era of big data [1,2].”
BackgroundOrellano, P. W., Reynoso, J. I., Antman, J., & Argibay, O. (2015) · Cadernos de Saúde Pública · doi:10.1590/0102-311x00072814 · Cited by 15
Role of data warehousing in healthcare epidemiology
Wyllie, D. H., & Davies, J. (2015) · Journal of Hospital Infection · doi:10.1016/j.jhin.2015.01.005 · Cited by 15
Big Data: What Is It and What Does It Mean for Cardiovascular Research and Prevention Policy
Pah, A. R., Rasmussen‐Torvik, L. J., Goel, S., Greenland, P., & Kho, A. (2014) · Current Cardiovascular Risk Reports · doi:10.1007/s12170-014-0424-3 · Cited by 16
Quantum Mechanisms: Expanding the Boundaries of Power, Space, and Time in Global Security Studies
Montgomery, A. H. (2016) · Journal of Global Security Studies · doi:10.1093/jogss/ogv011 · Cited by 16
Managerial work in the realm of the digital universe: The role of the data triad
Khatri, V. (2016) · Business Horizons · doi:10.1016/j.bushor.2016.06.001 · Cited by 16
Shah, M. P., Lopman, B. A., Tate, J. E., Harris, J. P., Esparza‐Aguilar, M., Sánchez-Uribe, E., Richardson, V., Steiner, C., & Parashar, U. D. (2017) · Journal of the Pediatric Infectious Diseases Society · doi:10.1093/jpids/pix004 · Cited by 14
Guo, D., Zhu, Y., & Yin, W. (2017) · GeoInformatica · doi:10.1007/s10707-017-0308-z · Cited by 13
Big Data: Challenge and Opportunity for Translational and Industrial Research in Healthcare
Rossi, R. L., & Grifantini, R. (2018) · Frontiers in Digital Humanities · doi:10.3389/fdigh.2018.00013 · Cited by 18
“…assumption that big data are a substitute for traditional data collection and analysis: the textbook example is the Google Flu Trends project (Lazer et al., 2014), which claimed to be able to anticipate seasonal flu outbreaks just by tracking internet user searches for flu-related terms:…”
BackgroundEvaluation of Nowcasting for Detecting and Predicting Local Influenza Epidemics, Sweden, 2009–2014
Sprečo, A., Eriksson, O., Dahlström, Ö., Cowling, B. J., & Timpka, T. (2018) · Emerging infectious diseases · doi:10.3201/eid2410.171940 · Cited by 13
Weber, I. G., & State, B. (2017) · International World Wide Web Conference (WWW) · doi:10.1145/3041021.3051104 · Cited by 20
“…have combined call records and field surveys [5], and have shown that combining a traditional epidemiological dataset provided by the U.S. Center for Disease Control with Google Flu Trends predictions may have helped avoid the now-infamous misfiring of that previously well-regarded algo-rithm [21].”
BackgroundSurvey vs Scraped Data: Comparing Time Series Properties of Web and Survey Vacancy Data
de Pedraza, P., Visintin, S., Tijdens, K. G., & Kismihók, G. (2019) · IZA Journal of Labor Economics · doi:10.2478/izajole-2019-0004 · Cited by 18
“…as Twitter (Barbera and Rivero, 2015; Blank, 2017; Rafali, 2018), Wikipedia (Martin, 2018), Google searches (Choi and Variant, 2012; Butler, 2013; Lazer et al., 2014), voluntary web surveys (Pedraza et al., 2010), Open Data Government portals (Sáez Martín et al., 2016), the combination of…”
BackgroundClemens, T., & Viechtbauer-Gruber, M. (2020) · SPE Reservoir Evaluation & Engineering · doi:10.2118/200540-pa · Cited by 17
Towards Computational and Behavioral Social Science
Conte, R., & Giardini, F. (2016) · European Psychologist · doi:10.1027/1016-9040/a000257 · Cited by 17
Liu, S., Triantis, K. P., Zhao, L., & Wang, Y. (2018) · PLoS ONE · doi:10.1371/journal.pone.0194687 · Cited by 18
Santillana, M. (2016) · Clinical Infectious Diseases · doi:10.1093/cid/ciw660 · Cited by 12
Jia, X., Willard, J., Karpatne, A., Read, J. S., Zwart, J. A., Steinbach, M. S., & Kumar, V. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.13075 · Cited by 21
“However, capturing this opportunity is contingent on a paradigm shift in data-intensive scientific discovery since the black box use of ML often leads to serious false discoveries in scientific applications [9, 12].”
Background
Metadata
- Authors
- Lazer, D., Kennedy, R., King, G., & Vespignani, A.
- Affiliations
- Northeastern University (US)
- Harvard University (US)
- Corresponding author
- Lazer, David
- Venue
- Science
- Published
- 2014-03-14
- Volume
- 343
- Issue
- 6176
- Pages
- 1203-1205
- ISSN
- 0036-8075 (ISSN-L)
- 1095-9203
- OpenAlex
- W2068181924
- Semantic Scholar
- 00d681cfff7f0224ace607e515f4ed510e792df9
- Cited by
- 2,523
- References
- 30
- Citations by year
- 2026: 113 · 2025: 101 · 2024: 121 · 2023: 150 · 2022: 168 · 2021: 229 · 2020: 273 · 2019: 261 · 2018: 272 · 2017: 249 · 2016: 271 · 2015: 219 · 2014: 91
- FWCI
- 208.16
- Type
- journal article
- Language
- en
- Open access
- green
- dash.harvard.edu
- Licence
- cc-by
- Locations
- Science (publishedVersion)
- PubMed (publishedVersion)
- Digital Access to Scholarship at Harvard (DASH) (Harvard University) (submittedVersion, cc-by)
- Topics
- Health Sciences › Medicine › Epidemiology › Data-Driven Disease Surveillance
- Social Sciences › Decision Sciences › Information Systems and Management › Big Data Technologies and Applications
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- Google Flu Trends, influenza surveillance, big data, prediction error, data quality
- MeSH terms
- Centers for Disease Control and Prevention, U.S., Data Collection, Data Interpretation, Statistical, Disease Outbreaks, Humans, Influenza, Human, Office Visits, Prognosis, United States, Social Media
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.