The Parable of Google Flu: Traps in Big Data Analysis
Open accessThe DOI matches its Crossref record; 30 of the 31 references it cites were found in a registry or an index; 5 of the 8 sources its claims cite were checked for support.
Abstract
Large errors in flu prediction were largely avoidable, which offers lessons for the use of big data.
Overview
CoChat Summary
TL;DR
The paper uses the failure of Google Flu Trends (GFT) to illustrate two traps in big data analysis: “big data hubris” and algorithm dynamics. It argues that GFT’s large flu-prediction errors were largely avoidable and that big data should supplement, not replace, traditional data and methods.
Key findings
- In February 2013, GFT predicted more than double the proportion of doctor visits for influenza-like illness (ILI) than the CDC; from 21 August 2011 to 1 September 2013, GFT reported overly high flu prevalence in 100 out of 108 weeks, and it overshot the 2011–2012 season by more than 50%.
- GFT errors were not random: they showed temporal autocorrelation and seasonality, meaning last week’s errors predicted this week’s errors and error direction/magnitude varied with time of year.
- Simple alternatives outperformed GFT: out-of-sample mean absolute error was 0.486 for GFT, 0.311 for lagged CDC, and 0.232 for combined GFT and CDC; all differences were statistically significant at P < 0.05. Even 3-week-old CDC data did better than GFT at projecting current flu prevalence.
- The authors attribute GFT’s problems to overfitting (initial GFT matched 50 million search terms to 1152 data points and missed the nonseasonal 2009 H1N1 pandemic) and to “algorithm dynamics,” including Google search algorithm changes and user behavior changes.
Methods
- Design: policy forum/case study comparing GFT estimates with CDC ILI surveillance data and alternative forecasting models.
- Data/sample: GFT and CDC flu prevalence estimates, including the period from 21 August 2011 to 1 September 2013; the initial GFT methodology reportedly fit 50 million search terms to 1152 data points.
- Main analysis: comparison of GFT alone, lagged CDC data with 52-week seasonality variables, and a combined GFT + CDC model using lagged CDC estimates, lagged GFT error, and seasonality variables; reported mean absolute error and statistical significance; Google Correlate used to examine correlated search terms. Supplementary materials contain data and methods.
Limitations
- The paper states that GFT never documented the 45 search terms used, released examples appear misleading, and Google Correlate is limited to national-level data whereas GFT was developed using regional correlations; therefore the original GFT analyses cannot be replicated from the information provided.
- The paper states that Google cannot make its full data available to outsiders because of privacy issues, though it argues there is no such constraint on derivative, aggregated data.
- Evident from the text: this is a commentary/policy forum rather than a full empirical article, with key data and methods placed in supplementary materials; the quantitative comparison depends on GFT and CDC data and model specifications not fully detailed in the main text.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
2 Valid · 2 Not valid · 2 Not checked · 8 Own results and interpretations
Some cited sources support this claim and some don’t — read each source’s verdict below.
In the paper
“This ad hoc method of throwing out peculiar search terms failed when GFT completely missed the nonseasonal 2009 infl uenza A–H1N1 pandemic ( 2, 14).”
What the sources say
- Partly supportedcites [2] matched to “Reassessing Google Flu Trends Data for Detection of Seasonal and Pandemic Influenza: A Comparative Epidemiological Study at Three Geographic Scales” (2013) sourceExperimental — accuracy not yet measured
The Discussion (page unavailable) says the original GFT model missed the spring 2009 pandemic wave, supporting the claim about its failure to detect the nonseasonal outbreak. It does not say GFT completely missed the entire 2009 pandemic. The Introduction describes automated selection of the original model’s search terms, not an ad hoc method of throwing out peculiar terms. (LLM, conf=0.86)
- Disputedcites [14] S. Cook et al., PLOS ONE 6, e23610 (2011). sourceExperimental — accuracy not yet measured
The Abstract (Methodology/Principal Findings) says the original GFT model underestimated the magnitude of illness during the 2009 H1N1 pandemic, not that it completely missed it. The Abstract (Conclusions) says both models performed well during the pandemic, although the updated model performed better. The Discussion notes that pandemic-specific queries were excluded, but does not describe this as an ad hoc method of throwing out peculiar terms. (LLM, conf=0.88)
Checked by a reasoning model — may be wrong.
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“Oddly, the few search terms offered in the papers ( 14) do not seem to be strongly related with either GFT or the CDC data (SM)—we surmise that the authors felt an unarticulated need to cloak the actual search terms identifi ed.”
What the sources say
- Disputedcites [14] S. Cook et al., PLOS ONE 6, e23610 (2011). sourceExperimental — accuracy not yet measured
The Results section (‘Search Behavior During pH1N1’; page unavailable) says that all three search queries shown in Figure 3 closely tracked CDC ILINet data before the pandemic, and ‘symptoms of flu’ continued to do so during it. The paper does report that the bronchitis and pneumonia queries underestimated ILINet data during the pandemic, but that does not support the broad claim that the offered terms were not strongly related to the data. (LLM, conf=0.87)
Checked by a reasoning model — may be wrong.
hedgedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“The Internet has opened the way for improving standard surveys, experiments, and health reporting ( 35).”
What the sources say
- Partly supportedcites [35] A. J. Berinsky et al., Polit. Anal. 20, 351–368 (2012). sourceExperimental — accuracy not yet measured
Section 2 describes using an Internet platform to run surveys with embedded experiments, and the Conclusion says MTurk can make experimental research easier and less expensive. The paper does not address improvements to health reporting; the disease scenario in section 4.3.2 is a framing experiment, not health reporting. Page numbers are unavailable in our copy. (LLM, conf=0.88)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Although not widely reported until 2013, the new GFT has been persistently overestimating flu prevalence for a much longer time.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“From 21 August 2011 to 1 September 2013, GFT reported overly high fl u prevalence 100 out of 108 weeks.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
3 more claims
Create a free account to see all 14 claims and every source verdict.
- It may state two separate claims as one. 5
- Probably not a substantial claim. 2
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Google Trends predicted more than twice the number of influenza-like illnesses than the CDC in February 2013 46 .”
From Unrepresentative big surveys significantly overestimated US vaccine uptake (2021) · cites this paper as [46]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This was one of the main reasons behind the failure of Google Flu Trends, since the data used for training the model in the first few years was not representative of the trends in subsequent years [15].”
From Theory-Guided Data Science: A New Paradigm for Scientific Discovery from Data (2017) · cites this paper as [15]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Social data research often relies on counting entities. These entities can be users, links, or messages, and the description is a summary of these counts (Lazer et al., 2014; Salganik, 2017). Yet, simple counts can mislead if it is unclear what is counted and how.”
From Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019) · cites this paper as Lazer et al., 2014
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This unintelligibility is also subject to functional biases (section 4.1), as it depends on the mechanisms available on each social platform (e.g., having a like button, but not a dislike one), as well as to variations in platform algorithms and mechanisms in response to users actions (Lazer et al., 2014).”
From Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019) · cites this paper as Lazer et al., 2014
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“It also includes temporal validity, which captures to what extent constructs change over time (Howison et al., 2011) and may invalidate previous conclusions about societal and/or platform phenomena; e.g., see the case of Google Flu Trends (Lazer et al., 2014).”
From Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019) · cites this paper as Lazer et al., 2014
6 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 14 MARCH 2014 VOL 343 SCIENCE www.sciencemag.org 1204
“The 2009 version of GFT has weathered other media panics related to the fl u, including the 2005–2006 influenza A/H5N1 ("bird flu") outbreak”
“GFT engineers updated the algorithm in 2009, and this model has run ever since”
Citations
Cites
boyd, D., & Crawford, K. (2012) · Information, Communication & Society · doi:10.1080/1369118x.2012.678878 · Cited by 5,479
Twitter mood predicts the stock market
Bollen, J., Mao, H., & Zeng, X. (2011) · Journal of Computational Science · doi:10.1016/j.jocs.2010.12.007 · Cited by 5,206
Detecting influenza epidemics using search engine query data
Ginsberg, J., Mohebbi, M. H., Patel, R. S., Brammer, L., Smolinski, M. S., & Brilliant, L. (2009) · Nature · doi:10.1038/nature07634 · Cited by 4,295
Cohen, D. (1998) · The Women's Review of Books · doi:10.2307/4022859 · Cited by 4,261
Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk
Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012) · Political Analysis · doi:10.1093/pan/mpr057 · Cited by 4,008
Big Data : The Management Revolution
マカフィー, ア., ブリニョルフソン, エ., & 有賀, 裕. (2013) · Harvard business review · Cited by 3,549
Lazer, D., Pentland, A., Adamic, L., Aral, S., Barabási, A.-L., Brewer, D., Christakis, N., Contractor, N., Fowler, J., Gutmann, M., et al. (2009) · Science · doi:10.1126/science.1167742 · Cited by 3,365
Self-citationPredicting Elections with Twitter: What 140 Characters Reveal about Political Sentiment
Tumasjan, A., Sprenger, T., Sandner, P., & Welpe, I. (2010) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v4i1.14009 · Cited by 2,649
Multiscale mobility networks and the spatial spreading of infectious diseases
Balcan, D., Colizza, V., Gonçalves, B., Hu, H., Ramasco, J. J., & Vespignani, A. (2009) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0906910106 · Cited by 1,380
Self-citationPredicting consumer behavior with web search
Goel, S., Hofman, J. M., Lahaie, S., Pennock, D. M., & Watts, D. J. (2010) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1005962107 · Cited by 636
Research works printed but not linked: 7
- A. McAfee, E. Brynjolfsson, Harv. Bus. Rev. 90, 60 (2012).
- AAAI Conference on Weblogs and Social Media, Atlanta, Georgia, 11 to 15 July 2010 (Association for Advancement of Artifi cial Intelligence, 2010), pp. 178.
- P. Copeland et al., Int. Soc. Negl. Trop. Dis. 2013, 3
- I. M. Hall et al., Epidemiol. Infect. 135, 372 (2007).
- E. Mustafaraj, P. Metaxas, in Proceedings of the Web- Sci10, Raleigh, NC, 26 and 27 April 2010 (Web Science Trust, 2010); http://journal.webscience.org/317/.
- J. Ratkiewicz et al., in Proceedings of 5th International AAAI Conference on Weblogs and Social Media, San Francisco, CA, 7 to 11 August 2011 (AAAI, 2011), p. 297–304.
- 2013; http://chronicle.com/article/Researchers-Struggle- to-Secure/141591/.
Cited by
Items 801–900 of 2,513
Khemnani, M., Tripathi, B., Thakkar, P., et al. (2023) · ACS Applied Electronic Materials · doi:10.1021/acsaelm.3c01038 · Cited by 25
Modeling and Predicting Human Infectious Diseases
Perra, N., & Gonçalves, B. (2015) · Springer eBooks · doi:10.1007/978-3-319-14011-7_4 · Cited by 24
“An intense research activity, across different disciplines, is clearly showing the potential, as well as the challenges and risks, of such digital traces for epidemiology [61] .”
BackgroundFrom Smallpox to Big Data: The Next 100 Years of Epidemiologic Methods
Gange, S. J., & Golub, E. T. (2015) · American Journal of Epidemiology · doi:10.1093/aje/kwv150 · Cited by 27
Identification of Keywords From Twitter and Web Blog Posts to Detect Influenza Epidemics in Korea
Woo, H., Cho, H. S., Shim, E., Lee, J., Lee, K., Song, G., & Cho, Y. (2017) · Disaster Medicine and Public Health Preparedness · doi:10.1017/dmp.2017.84 · Cited by 25
Leveraging hospital big data to monitor flu epidemics
Bouzillé, G., Poirier, C., Campillo‐Gimenez, B., Aubert, M.-L., Chabot, M., Chazard, E., Lavenu, A., & Marc Cuggia, A. (2017) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2017.11.012 · Cited by 35
Scarborough, W. J. (2018) · Socius Sociological Research for a Dynamic World · doi:10.1177/2378023118780760 · Cited by 34
“Its size and convenience are seductive for social scientists, but as Lazer et al. (2014) warned, we must be careful of “big data hubris,” which has prompted some to wholly embrace big data, reject other forms of inquiry, and criticize those who continue to use surveys and interviews.”
BackgroundInfluentialMachine Learning in Aging Research
Odden, M. C., & Melzer, D. R. (2019) · The Journals of Gerontology Series A · doi:10.1093/gerona/glz074 · Cited by 26
“In other words, the algorithm was confounded by seasonality (13).”
BackgroundDevelopmental Idealism in Internet Search Data
Dorius, S., & Swindle, J. (2019) · Sociology of Development · doi:10.1525/sod.2019.5.3.286 · Cited by 23
Improving Google Flu Trends Estimates for the United States through Transformation
Martin, L. J., Xu, B., & Yasui, Y. (2014) · PLoS ONE · doi:10.1371/journal.pone.0109209 · Cited by 27
BIG DATA: NEW SCIENCE, NEW CHALLENGES, NEW DIALOGICAL OPPORTUNITIES
Fuller, M. (2015) · Zygon® · doi:10.1111/zygo.12187 · Cited by 24
“The core challenge is that most big data that have received popular attention are not the output of instruments designed to produce valid and reliable data amenable for scientific analysis” (Lazer et al. 2014: 1203).”
InfluentialPandey, M. K., & Subbiah, K. (2016) · IEEE International Conference on Computer and Information Technology (CIT) · doi:10.1109/cit.2016.86 · Cited by 27
The Changing Science of HIV Epidemiology in the United States
DʼSouza, G., Golub, E. T., & Gange, S. J. (2019) · American Journal of Epidemiology · doi:10.1093/aje/kwz211 · Cited by 27
“Future epidemiologic research will increasingly use populations constructed from those individuals who contribute data to repositories of electronic health records, social media, and search engines (68).”
BackgroundReshaping the Arc of Quantitative Educational Research: It’s Time to Broaden Our Paradigm
Singer, J. D. (2019) · Journal of Research on Educational Effectiveness · doi:10.1080/19345747.2019.1658835 · Cited by 26
Zangani, C., Schifano, F., Napoletano, F., Arillotta, D., Gilgar, L., Guirguis, A., Corkery, J., Gambini, O., & Vento, A. E. (2020) · Current Neuropharmacology · doi:10.2174/1570159x18666200302125146 · Cited by 27
Decentralized but Globally Coordinated Biodiversity Data
Sterner, B., Gilbert, E., & Franz, N. M. (2020) · Frontiers in Big Data · doi:10.3389/fdata.2020.519133 · Cited by 30
“, by aggregating observations across many sources—unfortunately does not necessarily lead to improved predictions or discoveries, despite claims that suggest theoretically informed statistical analyses and experimental designs are no longer relevant (Philippe et al., 2011; Lazer et al., 2014; Leonelli, 2016; Sterner a…”
BackgroundTracking COVID-19 using taste and smell loss Google searches is not a reliable strategy
Asseo, K., Fierro, F., Slavutsky, Y., Frasnelli, J., & Niv, M. Y. (2020) · Scientific Reports · doi:10.1038/s41598-020-77316-3 · Cited by 24
Digital epidemiology, deep phenotyping and the enduring fantasy of pathological omniscience
Engelmann, L. (2022) · Big Data & Society · doi:10.1177/20539517211066451 · Cited by 31
“Nonetheless, Google Flu closed down as it lacked in accuracy what it offered in ef fi ciency (Lazer et al., 2014).”
BackgroundSchäfer, M. S., & Hase, V. (2022) · Wiley Interdisciplinary Reviews Climate Change · doi:10.1002/wcc.806 · Cited by 30
Weller, K., & Kinder-Kurlanda, K. E. (2021) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v9i4.14687 · Cited by 40
“Other authors discuss the pitfalls of divorcing theory from Big Data analysis (e.g. Lazer 2014; Frické 2014).”
BackgroundGlobal mapping of artificial intelligence in Google and Google Scholar
Omar, M., Mehmood, A., Choi, G. S., & Park, H. W. (2017) · Scientometrics · doi:10.1007/s11192-017-2534-4 · Cited by 26
“Google query data have been used to identify flu trends (Ginsberg et al. 2009; Lazer et al. 2014), dengue fever (Chan et al. 2011; Gluskin et al. 2014), markets (Arditi et al. 2015; Ball 2013; Drake et al. 2012; Preis and Moat 2015), and even suicidal behavior (Gunn and Lester 2013).”
BackgroundMethodInfluentialGeospatial Data Science Techniques and Applications
Karimi, B. (2017) · CRC Press eBooks · doi:10.1201/9781315228396 · Cited by 41
Algorithm Analysis for Big Data in Education Based on Depth Learning
Zhang, W., & Jiang, L. (2018) · Wireless Personal Communications · doi:10.1007/s11277-018-5331-3 · Cited by 24
Broniatowski, D. A., & Tucker, C. S. (2017) · Systems Engineering · doi:10.1002/sys.21414 · Cited by 32
Association Between HIV-Related Tweets and HIV Incidence in the United States: Infodemiology Study
Stevens, R. C., Bonett, S., Bannon, J. A., Chittamuru, D., Slaff, B., Browne, S., Huang, S. X. L., & Bauermeister, J. A. (2020) · Journal of Medical Internet Research · doi:10.2196/17196 · Cited by 23
Using digital technologies to improve the design and enforcement of public policies
OECD (2019) · OECD digital economy papers · doi:10.1787/99b9ba70-en · Cited by 45
Data Quality in the Age of AI: A Review of Governance, Ethics, and the FAIR Principles
Guillen-Aguinaga, M., Aguinaga-Ontoso, E., Guillen-Aguinaga, L., Guillen-Grima, F., & Aguinaga-Ontoso, I. (2025) · Data · doi:10.3390/data10120201 · Cited by 37
Enhancing data-driven soil moisture modeling with physically-guided LSTM networks
Geng, Q., Yan, S., Li, Q., & Zhang, C. (2024) · Frontiers in Forests and Global Change · doi:10.3389/ffgc.2024.1353011 · Cited by 29
wei, S., Lin, S., Wen-jing, Z., et al. (2024) · BMC Public Health · doi:10.1186/s12889-024-17978-0 · Cited by 23
“However, the forecast model was not stable because the influenza forecast trend exceeded the peak of the epidemic by more than 140% in 2013 in the U.S. and sparked a hot discussion about the limitation of search data in infectious disease research [5].”
BackgroundBrigo, F., Otte, W. M., Igwe, S. C., Ausserer, H., Nardone, R., Tezzon, F., & Trinka, E. (2015) · Epileptic Disorders · doi:10.1684/epd.2015.0772 · Cited by 24
Pruyt, E. (2016) · International Journal of System Dynamics Applications · doi:10.4018/ijsda.2016010101 · Cited by 25
Google Search Queries About Neurosurgical Topics: Are They a Suitable Guide for Neurosurgeons?
McLean, A. C. L., McLean, A. L., Kalff, R., & Walter, J. (2016) · World Neurosurgery · doi:10.1016/j.wneu.2016.02.045 · Cited by 23
Tracking Natural Events through Social Media and Computer Vision
Wang, J., Korayem, M., Blanco, S. A., et al. (2016) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/2964284.2984067 · Cited by 23
“Keywords Computer vision; social media; event detection; ecology”
BackgroundZhao, N., Cao, G., Vanos, J., & Vecellio, D. J. (2017) · International Journal of Biometeorology · doi:10.1007/s00484-017-1306-4 · Cited by 28
“However, documented problems of the original GFT include missing the 2009 spring influenza pandemic (Cook et al. 2011; Olson et al. 2013) and overestimating the intensity of 2012/2013 flu seasons (Butler 2013; Lazer et al. 2014; Olson et al. 2013).”
BackgroundWang, S., & Zhang, T. (2018) · Cluster Computing · doi:10.1007/s10586-018-2184-1 · Cited by 29
RetractedSensing global tourism numbers with millions of publicly shared online photographs
Preis, T., Botta, F., & Moat, H. S. (2019) · Environment and Planning A Economy and Space · doi:10.1177/0308518x19872772 · Cited by 25
“…for on Google, pages they access on Wikipedia and content they share on Twitter have fuelled the rapidly developing interdisciplinary field of computational social science (Botta et al., 2015; Ginsberg et al., 2009; King, 2011; Lazer et al., 2009, 2014; Preis and Moat, 2014; Vespignani, 2009).”
BackgroundComputational Intelligence in Vaccine Design Against COVID-19
Bharadwaj, K. K., Srivastava, A., Panda, M. K., Singh, Y. D., Maharana, R., Mandal, K., Singh, B. S. M., Singh, D., Das, M., Murmu, D., et al. (2020) · Studies in computational intelligence · doi:10.1007/978-981-15-8534-0_16 · Cited by 31
Ontrup, G., Schempp, P. S., & Kluge, A. (2021) · Journal of Organizational Effectiveness People and Performance · doi:10.1108/joepp-03-2021-0064 · Cited by 29
Does ‘bigger’ mean ‘better’? Pitfalls and shortcuts associated with big data for social research
Giardullo, P. (2015) · Quality & Quantity · doi:10.1007/s11135-015-0162-8 · Cited by 23
Campaign Klout: Measuring Online Influence During the 2012 Election
Bode, L., & Epstein, B. (2014) · Journal of Information Technology & Politics · doi:10.1080/19331681.2014.994157 · Cited by 25
Estimating Voter Registration Deadline Effects with Web Search Data
Street, A., Murray, T. A., Blitzer, J. C., & Patel, R. S. (2015) · Political Analysis · doi:10.1093/pan/mpv002 · Cited by 28
“Web search data can “predict the present” by providing evidence on time-varying outcomes more quickly than other methods (Ginsberg et al. 2008; Choi and Varian 2012; Varian 2014; but see also Lazer et al. 2014).”
BackgroundMethodResultInfluentialTowards Identifying and Reducing the Bias of Disease Information Extracted from Search Engine Data
Huang, D., Wang, J., Huang, J., Sui, D. Z., Zhang, H., Hu, M.-G., & Xu, C. (2016) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1004876 · Cited by 27
“However, their limitations have attracted increasing attention from the broader scientific community [3, 4].”
InfluentialBig Data and Survey Research: Supplement or Substitute?
Johnson, T. P., & Smith, T. (2016) · Springer geography · doi:10.1007/978-3-319-40902-3_7 · Cited by 32
Demography in the Big Data Revolution: Changing the Culture to Forge New Frontiers
Bohon, S. A. (2018) · Population Research and Policy Review · doi:10.1007/s11113-018-9464-6 · Cited by 25
Big Data-Driven Materials Science and Its FAIR Data Infrastructure
Draxl, C., & Scheffler, M. (2019) · Springer eBooks · doi:10.1007/978-3-319-42913-7_104-1 · Cited by 25
“Reasons may be, among others, anxiety-stimulated searches caused by reports in public and social media and changes in google search technology, e.g. recommending searches based on what others have searched (Lazer et al. 2014).”
BackgroundPeng, Y., Li, C., Rong, Y., Chen, X., & Chen, H. (2020) · Journal of Global Health · doi:10.7189/jogh.10.020511 · Cited by 24
Ferres, J. L., Meirick, T. M., Lomazow, W., et al. (2021) · JAMA Ophthalmology · doi:10.1001/jamaophthalmol.2021.4852 · Cited by 23
Conditional Hyper-Network for Blind Super-Resolution With Multiple Degradations
Yin, G., Wang, W., Yuan, Z., Ji, W., Yu, D., Sun, S., Chua, T., & Wang, C. (2022) · IEEE Transactions on Image Processing · doi:10.1109/tip.2022.3176526 · Cited by 29
“When the data distributions at test time mismatch the training distributions (referred to as distribution shift [9], [10]), these learning-based models will suffer severe performance drop [11].”
Feldman, M. P., Kenney, M. F., & Lissoni, F. (2015) · Research Policy · doi:10.1016/j.respol.2015.02.007 · Cited by 23
“1 For example, see the saga of the Google Flu prediction algorithm (Lazer et al., 2014).”
BackgroundBig Data in Environment and Human Health
Fleming, L. E., Tempini, N., Gordon-Brown, H., Nichols, G. L., Sarran, C. E., Vineis, P., Leonardi, G. S., Golding, B. W., Haines, A., Kessel, A. S., et al. (2016) · Oxford University Press eBooks · doi:10.1093/acrefore/9780199389414.013.541 · Cited by 29
Online and ubiquitous HCI research
Lazar, J. K., Feng, J. H., & Hochheiser, H. (2017) · Elsevier eBooks · doi:10.1016/b978-0-12-805390-4.00014-5 · Cited by 22
Sequence to Sequence with Attention for Influenza Prevalence Prediction using Google Trends
Kondo, K., Ishikawa, A., & Kimura, M. (2019) · International Conference on Computational Biology and Bioinformatics (ICCBB) · doi:10.1145/3365966.3365967 · Cited by 22
“[3] also reported the prediction of ILI using Google Flu Trends along with the data disclosed by the Centers for Disease Control and Prevention (CDC)(1).”
MethodQuantifying the sustainability of Bitcoin and Blockchain
Fry, J. M., & Serbera, J. (2020) · Journal of Enterprise Information Management · doi:10.1108/jeim-06-2018-0134 · Cited by 27
Tracking and predicting U.S. influenza activity with a real-time surveillance network
Leuba, S. I., Yaesoubi, R., Antillón, M., Cohen, T., & Zimmer, C. (2020) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1008180 · Cited by 22
Biases in using social media data for public health surveillance: A scoping review
Zhao, Y., He, X., Feng, Z., Bost, S. J., Prosperi, M., Wu, Y., Guo, Y. M., & Bian, J. (2022) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2022.104804 · Cited by 32
What Is (Not) Big Data Based on Its 7Vs Challenges: A Survey
García, C. G., & Álvarez-Fernández, E. (2022) · Big Data and Cognitive Computing · doi:10.3390/bdcc6040158 · Cited by 33
“However, it should be kept in mind that not always because of having larger datasets will one get better predictions; this can be classified as arrogance, because Big Data is not the substitute for data collection and Data Mining [156].”
Bardak, B., & Tan, M. (2015) · IEEE International Conference on Bioinformatics and Bioengineering (BIBE) · doi:10.1109/bibe.2015.7367640 · Cited by 23
“For instance, in 2012/2013 season, GFT significantly overpredicted flu levels compared to CDC’s report [13].”
ResultRobertson, C., & Feick, R. (2015) · Cartography and Geographic Information Science · doi:10.1080/15230406.2015.1088801 · Cited by 26
“As discussed in Lazer et al. (2014), modifications to the Google search algorithm and suggested search terms, likely intended to increase ad revenues, may have inadvertently fooled the flu detection algo-rithm.”
Jendryke, M., Balz, T., & Liao, M. (2017) · Transactions in GIS · doi:10.1111/tgis.12266 · Cited by 24
Google as a cancer control tool in Queensland
Huang, X., Baade, P. D., Youlden, D. R., Youl, P., Hu, W., & Kimlin, M. G. (2017) · BMC Cancer · doi:10.1186/s12885-017-3828-x · Cited by 22
Reimaging Research Methodology as Data Science
Daniel, B. K. (2018) · Big Data and Cognitive Computing · doi:10.3390/bdcc2010004 · Cited by 27
Crowdsourcing architectural beauty: Online photo frequency predicts building aesthetic ratings
Sáiz, A., Miranda, A. S., & Bernard, J. (2018) · PLoS ONE · doi:10.1371/journal.pone.0194369 · Cited by 29
Google Trends Analysis of COVID-19 Pandemic
Pan, Z., Nguyen, H. L., Abu-gellban, H., & Zhang, Y. (2020) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata50022.2020.9377852 · Cited by 22
Marks, C., Carrasco‐Escobar, G., Carrasco-Hernández, R., Johnson, D. C., Ciccarone, D., Strathdee, S. A., Smith, D. M., & Bórquez, A. (2021) · Translational research · doi:10.1016/j.trsl.2021.03.018 · Cited by 27
Using Facebook Ads Audiences for Global Lifestyle Disease Surveillance: Promises and LimitationsPreprint
Araújo, M., Mejova, Y., Weber, I. G., & Benevenuto, F. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.04045 · Cited by 27
“…as abortion [16] and vaccines [19], others use the data to now-cast diseases [8], while others use search logs, most notably using Google Trends, to predict seasonal flu [6]. large errors in flu prediction from Google Flu Trends offered interesting lessons for the use of this kind of data [12].”
BackgroundWho Is Doing Computational Social Science? Trends in Big Data Research A SAGE White Paper
Metzler, K., Kim, D. A., Allum, N., & Denman, A. (2016) · doi:10.4135/wp160926 · Cited by 31
Coston, A., Guha, N., Ouyang, D., Lu, L., Chouldechova, A., & Ho, D. E. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445881 · Cited by 21
“Ground truth is often lacking, which is precisely why policymakers and academics have flocked toward big data, on the implicit assumption that scale can overcome more conventional questions of data reliability, sampling bias, and the like [2, 35].”
BackgroundMu, X., Zhang, X., Yeh, A. G. O., & Wang, J. (2024) · Applied Geography · doi:10.1016/j.apgeog.2024.103260 · Cited by 22
Liu, Q., Li, Y., Jie, Y., et al. (2024) · International Journal of Geographical Information Systems · doi:10.1080/13658816.2024.2342321 · Cited by 21
“Therefore, these models may yield results that are inconsistent with physical laws (Lazer et al. 2014).”
BackgroundImmediate and long-term effects of 2016 Zika Outbreak: A Twitter-based study
Khatua, A., & Khatua, A. (2016) · IEEE International Conference on e-Health Networking, Applications and Services (Healthcom) · doi:10.1109/healthcom.2016.7749496 · Cited by 23
“In recent times social media data have gained immense popularity for syndromic surveillance during global health crises such as the 2009 H1N1 pandemic [1], [3], [4], [5], [7], [11], [13] or the 2014 Ebola outbreak [2], [9], [10], [14], [15], [21].”
BackgroundMethodHarnessing Social Media in the Modelling of Pandemics—Challenges and Opportunities
Sooknanan, J., & Mays, N. (2021) · Bulletin of Mathematical Biology · doi:10.1007/s11538-021-00895-3 · Cited by 27
“However, they overestimated the prevalence of influenza in the 2012–2013 and 2011–2012 seasons by more than 50% when compared to surveillance reports from the Centers for Disease Control and Prevention (CDC) (Lazer et al. 2014).”
BackgroundAn open‐source image classifier for characterizing recreational activities across landscapes
Winder, S. G., Lee, H., Seo, B., Lia, E. H., & Wood, S. A. (2022) · People and Nature · doi:10.1002/pan3.10382 · Cited by 23
Crime, inequality and public health: a survey of emerging trends in urban data science
Luca, M., Campedelli, G. M., Centellegher, S., Tizzoni, M., & Lepri, B. (2023) · Frontiers in Big Data · doi:10.3389/fdata.2023.1124526 · Cited by 33
Cohort bias in predictive risk assessments of future criminal justice system involvement
Montana, E., Nagin, D. S., Neil, R., & Sampson, R. J. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2301990120 · Cited by 24
“In a well-known case, changes to Google’s search algorithm led Google Flu Trends to severely overpredict flu patterns over time (36).”
Sherlock: Experimental Evaluation of a Conversational Agent for Mobile Information Tasks
Preece, A. D., Webberley, W., Braines, D., Zaroukian, E. G., & Bakdash, J. Z. (2017) · IEEE Transactions on Human-Machine Systems · doi:10.1109/thms.2017.2700625 · Cited by 21
“Consequently, inferences made using social sensing can be quite wrong [16].”
BackgroundSchiavoni, C., Palm, F. C., Smeekes, S., & van den Brakel, J. (2020) · Journal of the Royal Statistical Society Series A (Statistics in Society) · doi:10.1111/rssa.12626 · Cited by 27
Analyzing Medical Emergency Processes with Process Mining: The Stroke Case
Fernández-Llatas, C., Ibáñez-Sánchez, G., Celda, A., Mandingorra, J., Aparici-Tortajada, L., Martínez-Millana, A., Muñoz-Gama, J., Sepúlveda, M., Rojas, E., Galvez-Yanjari, V., et al. (2019) · Lecture notes in business information processing · doi:10.1007/978-3-030-11641-5_17 · Cited by 23
Linking Big Data and Prediction Strategies: Tools, Pitfalls, and Lessons Learned
Yang, S., Stansbury, L. G., Rock, P., Scalea, T. M., & Hu, P. (2019) · Critical Care Medicine · doi:10.1097/ccm.0000000000003739 · Cited by 24
Policy and Science for Global Health Security: Shaping the Course of International Health
Berger, K. M., Wood, J. L. N., Jenkins, B. D., Olsen, J. M., Morse, S. S., Gresham, L. S., Root, J. J., Rush, M., Pigott, D. M., Winkleman, T., et al. (2019) · Tropical Medicine and Infectious Disease · doi:10.3390/tropicalmed4020060 · Cited by 27
“These platforms vary by the purpose, their intended stakeholders, the data they integrate, their analytic capabilities and methodologies, their accuracy, and other factors, all of which have different utility to public health decision-makers [23] [24] [25] .”
BackgroundThe Fever Coach Mobile App for Participatory Influenza Surveillance in Children: Usability Study
Kim, M., Yune, S., Chang, S., Jung, Y., Sa, S., & Han, H. W. (2019) · JMIR mhealth and uhealth · doi:10.2196/14276 · Cited by 20
Soziologie des Digitalen - Digitale Soziologie? Soziale Welt - Sonderband 23
(2020) · Nomos Verlagsgesellschaft mbH & Co. KG eBooks · doi:10.5771/9783845295008 · Cited by 20
Big Data: transforming drug development and health policy decision making
Alemayehu, D., & Berger, M. L. (2016) · Health Services and Outcomes Research Methodology · doi:10.1007/s10742-016-0144-x · Cited by 21
“Furthermore, uncritical use of modern algorithmic tools is likely to lead to unacceptable results with unpredictable consequences (Lazer et al. 2014).”
BackgroundChowell, G., Cleaton, J. M., & Viboud, C. G. (2016) · The Journal of Infectious Diseases · doi:10.1093/infdis/jiw356 · Cited by 21
A conceptual distributed framework for improved and secured healthcare system
Sarkar, B. K., & Sana, S. S. (2018) · International Journal of Healthcare Management · doi:10.1080/20479700.2017.1422338 · Cited by 21
Shin, S.-Y., Kim, T., Seo, D., Sohn, C. H., Kim, S.-H., Ryoo, S. M., Lee, Y., Lee, J., Kim, W. Y., & Lim, K. S. (2016) · PLoS ONE · doi:10.1371/journal.pone.0158539 · Cited by 21
“However, there are some criticisms about digital surveillance, such as Google Flu Trends [20, 21, 23].”
BackgroundInfluentialPolicy and population behavior in the age of Big Data
Ruggeri, K., Yoon, H., Kácha, O., van der Linden, S., & Muennig, P. A. (2017) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2017.05.010 · Cited by 23
Use Internet search data to accurately track state level influenza epidemics
Yang, S., Ning, S., & Kou, S. C. (2021) · Scientific Reports · doi:10.1038/s41598-021-83084-5 · Cited by 21
Box-office forecasting in Korea using search trend data: a modified generalized Bass diffusion model
Kang, D. (2021) · Electronic Commerce Research · doi:10.1007/s10660-020-09456-7 · Cited by 23
The Dangers of Data Colonialism in Precision Public Health
Ferryman, K. (2021) · Global Policy · doi:10.1111/1758-5899.12953 · Cited by 20
Comparing minds and machines: implications for financial stability
Buckmann, M., Haldane, A., & Hüser, A. (2021) · Oxford Review of Economic Policy · doi:10.1093/oxrep/grab017 · Cited by 29
Vieira, F. A. D. S., Santos, D. T. V., Bragagnolo, C., Campos‐Silva, J. V., Correia, R. A., Jepson, P. R., Malhado, A. C. M., & Ladle, R. J. (2021) · Ocean & Coastal Management · doi:10.1016/j.ocecoaman.2021.105918 · Cited by 30
Fundamental limitations on efficiently forecasting certain epidemic measures in network models
Rosenkrantz, D. J., Vullikanti, A. K. S., Ravi, S. S., Stearns, R. E., Levin, S. A., Poor, H. V., & MARATHE, M. V. (2022) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2109228119 · Cited by 22
“• Lazer, Kennedy, King and Vespignani [53] use Google Flu Trends as an example to caution against the use of social media and search information as a substitute for traditional data collection and analysis methods to predict epidemic measures.”
BackgroundThe illusion of data validity: Why numbers about people are likely wrong
Jansen, B. J. J., Salminen, J. O., Jung, S., & Almerekhi, H. (2022) · Data and Information Management · doi:10.1016/j.dim.2022.100020 · Cited by 27
Meng, J., Abed, A. M., Elsehrawy, M. G., et al. (2024) · Heliyon · doi:10.1016/j.heliyon.2024.e31244 · Cited by 20
RetractedBig Data and IT-Enabled Services: Ecosystem and Coevolution
Chae, B. K. (2015) · IT Professional · doi:10.1109/mitp.2015.17 · Cited by 20
‘Big data’ and policy learning
Dunleavy, P. (2017) · Bristol University Press eBooks · doi:10.2307/j.ctt1t89d4k.15 · Cited by 22
“But the apparent association did not last – the advent of a different strain of flu made the Google-based model perform very poorly, forecasting almost twice the level of flu in 2012 than the CDC system found (Lazer et al, 2014).”
BackgroundVaccine Design for H5N1 Based on B- and T-cell Epitope Predictions
Tambunan, U. S. F., Sipahutar, F. R. P., Parikesit, A. A., & Kerami, D. (2016) · Bioinformatics and Biology Insights · doi:10.4137/bbi.s38378 · Cited by 20
Vollmar, H. C., Krämer, U., Müller, H., Griemmert, M., Noelle, G., & Schrappe, M. (2017) · Das Gesundheitswesen · doi:10.1055/s-0043-122233 · Cited by 23
Measuring Employment Demand Using Internet Search Data
Chancellor, S., & Counts, S. (2018) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173696 · Cited by 26
“caution against “the hazards of predicting the future” [1], and we are aware of the methodological limitations of such analyses [12, 53, 54].”
BackgroundBroos, T., Verbert, K., Langie, G., Soom, C. V., & Laet, T. D. (2017) · Journal of Research in Innovative Teaching & Learning · doi:10.1108/jrit-05-2017-0010 · Cited by 30
“It may come to no surprise that many scientists feel uncomfortable about this notion (Kitchin, 2013; Lazer et al., 2014) and that it may contribute to a distrust of LA from educational scientists.”
BackgroundCross-Sectoral Big Data: The Application of an Ethics Framework for Big Data in Health and Research
Laurie, G. T. (2019) · Asian Bioethics Review · doi:10.1007/s41649-019-00093-3 · Cited by 22
“However, the system has by no means been infallible, with under-reporting of H1N1 (swine flu) pandemic in 2009, and a discrepancy of almost double CDC estimates for flu in 2011 – 2012 (Butler 2013; Lazer et al. 2014).”
Metadata
- Authors
- Lazer, D., Kennedy, R., King, G., & Vespignani, A.
- Affiliations
- Northeastern University (US)
- Harvard University (US)
- Corresponding author
- Lazer, David
- Venue
- Science
- Published
- 2014-03-14
- Volume
- 343
- Issue
- 6176
- Pages
- 1203-1205
- ISSN
- 0036-8075 (ISSN-L)
- 1095-9203
- OpenAlex
- W2068181924
- Semantic Scholar
- 00d681cfff7f0224ace607e515f4ed510e792df9
- Cited by
- 2,523
- References
- 30
- Citations by year
- 2026: 113 · 2025: 101 · 2024: 121 · 2023: 150 · 2022: 168 · 2021: 229 · 2020: 273 · 2019: 261 · 2018: 272 · 2017: 249 · 2016: 271 · 2015: 219 · 2014: 91
- FWCI
- 208.16
- Type
- journal article
- Language
- en
- Open access
- green
- dash.harvard.edu
- Licence
- cc-by
- Locations
- Science (publishedVersion)
- PubMed (publishedVersion)
- Digital Access to Scholarship at Harvard (DASH) (Harvard University) (submittedVersion, cc-by)
- Topics
- Health Sciences › Medicine › Epidemiology › Data-Driven Disease Surveillance
- Social Sciences › Decision Sciences › Information Systems and Management › Big Data Technologies and Applications
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- Google Flu Trends, influenza surveillance, big data, prediction error, data quality
- MeSH terms
- Centers for Disease Control and Prevention, U.S., Data Collection, Data Interpretation, Statistical, Disease Outbreaks, Humans, Influenza, Human, Office Visits, Prognosis, United States, Social Media
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.