The Parable of Google Flu: Traps in Big Data Analysis
Open accessThe DOI matches its Crossref record; 30 of the 31 references it cites were found in a registry or an index; 5 of the 8 sources its claims cite were checked for support.
Abstract
Large errors in flu prediction were largely avoidable, which offers lessons for the use of big data.
Overview
CoChat Summary
TL;DR
The paper uses the failure of Google Flu Trends (GFT) to illustrate two traps in big data analysis: “big data hubris” and algorithm dynamics. It argues that GFT’s large flu-prediction errors were largely avoidable and that big data should supplement, not replace, traditional data and methods.
Key findings
- In February 2013, GFT predicted more than double the proportion of doctor visits for influenza-like illness (ILI) than the CDC; from 21 August 2011 to 1 September 2013, GFT reported overly high flu prevalence in 100 out of 108 weeks, and it overshot the 2011–2012 season by more than 50%.
- GFT errors were not random: they showed temporal autocorrelation and seasonality, meaning last week’s errors predicted this week’s errors and error direction/magnitude varied with time of year.
- Simple alternatives outperformed GFT: out-of-sample mean absolute error was 0.486 for GFT, 0.311 for lagged CDC, and 0.232 for combined GFT and CDC; all differences were statistically significant at P < 0.05. Even 3-week-old CDC data did better than GFT at projecting current flu prevalence.
- The authors attribute GFT’s problems to overfitting (initial GFT matched 50 million search terms to 1152 data points and missed the nonseasonal 2009 H1N1 pandemic) and to “algorithm dynamics,” including Google search algorithm changes and user behavior changes.
Methods
- Design: policy forum/case study comparing GFT estimates with CDC ILI surveillance data and alternative forecasting models.
- Data/sample: GFT and CDC flu prevalence estimates, including the period from 21 August 2011 to 1 September 2013; the initial GFT methodology reportedly fit 50 million search terms to 1152 data points.
- Main analysis: comparison of GFT alone, lagged CDC data with 52-week seasonality variables, and a combined GFT + CDC model using lagged CDC estimates, lagged GFT error, and seasonality variables; reported mean absolute error and statistical significance; Google Correlate used to examine correlated search terms. Supplementary materials contain data and methods.
Limitations
- The paper states that GFT never documented the 45 search terms used, released examples appear misleading, and Google Correlate is limited to national-level data whereas GFT was developed using regional correlations; therefore the original GFT analyses cannot be replicated from the information provided.
- The paper states that Google cannot make its full data available to outsiders because of privacy issues, though it argues there is no such constraint on derivative, aggregated data.
- Evident from the text: this is a commentary/policy forum rather than a full empirical article, with key data and methods placed in supplementary materials; the quantitative comparison depends on GFT and CDC data and model specifications not fully detailed in the main text.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
2 Valid · 2 Not valid · 2 Not checked · 8 Own results and interpretations
Some cited sources support this claim and some don’t — read each source’s verdict below.
In the paper
“This ad hoc method of throwing out peculiar search terms failed when GFT completely missed the nonseasonal 2009 infl uenza A–H1N1 pandemic ( 2, 14).”
What the sources say
- Partly supportedcites [2] matched to “Reassessing Google Flu Trends Data for Detection of Seasonal and Pandemic Influenza: A Comparative Epidemiological Study at Three Geographic Scales” (2013) sourceExperimental — accuracy not yet measured
The Discussion (page unavailable) says the original GFT model missed the spring 2009 pandemic wave, supporting the claim about its failure to detect the nonseasonal outbreak. It does not say GFT completely missed the entire 2009 pandemic. The Introduction describes automated selection of the original model’s search terms, not an ad hoc method of throwing out peculiar terms. (LLM, conf=0.86)
- Disputedcites [14] S. Cook et al., PLOS ONE 6, e23610 (2011). sourceExperimental — accuracy not yet measured
The Abstract (Methodology/Principal Findings) says the original GFT model underestimated the magnitude of illness during the 2009 H1N1 pandemic, not that it completely missed it. The Abstract (Conclusions) says both models performed well during the pandemic, although the updated model performed better. The Discussion notes that pandemic-specific queries were excluded, but does not describe this as an ad hoc method of throwing out peculiar terms. (LLM, conf=0.88)
Checked by a reasoning model — may be wrong.
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“Oddly, the few search terms offered in the papers ( 14) do not seem to be strongly related with either GFT or the CDC data (SM)—we surmise that the authors felt an unarticulated need to cloak the actual search terms identifi ed.”
What the sources say
- Disputedcites [14] S. Cook et al., PLOS ONE 6, e23610 (2011). sourceExperimental — accuracy not yet measured
The Results section (‘Search Behavior During pH1N1’; page unavailable) says that all three search queries shown in Figure 3 closely tracked CDC ILINet data before the pandemic, and ‘symptoms of flu’ continued to do so during it. The paper does report that the bronchitis and pneumonia queries underestimated ILINet data during the pandemic, but that does not support the broad claim that the offered terms were not strongly related to the data. (LLM, conf=0.87)
Checked by a reasoning model — may be wrong.
hedgedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“The Internet has opened the way for improving standard surveys, experiments, and health reporting ( 35).”
What the sources say
- Partly supportedcites [35] A. J. Berinsky et al., Polit. Anal. 20, 351–368 (2012). sourceExperimental — accuracy not yet measured
Section 2 describes using an Internet platform to run surveys with embedded experiments, and the Conclusion says MTurk can make experimental research easier and less expensive. The paper does not address improvements to health reporting; the disease scenario in section 4.3.2 is a framing experiment, not health reporting. Page numbers are unavailable in our copy. (LLM, conf=0.88)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Although not widely reported until 2013, the new GFT has been persistently overestimating flu prevalence for a much longer time.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“From 21 August 2011 to 1 September 2013, GFT reported overly high fl u prevalence 100 out of 108 weeks.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
3 more claims
Create a free account to see all 14 claims and every source verdict.
- It may state two separate claims as one. 5
- Probably not a substantial claim. 2
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Google Trends predicted more than twice the number of influenza-like illnesses than the CDC in February 2013 46 .”
From Unrepresentative big surveys significantly overestimated US vaccine uptake (2021) · cites this paper as [46]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This was one of the main reasons behind the failure of Google Flu Trends, since the data used for training the model in the first few years was not representative of the trends in subsequent years [15].”
From Theory-Guided Data Science: A New Paradigm for Scientific Discovery from Data (2017) · cites this paper as [15]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Social data research often relies on counting entities. These entities can be users, links, or messages, and the description is a summary of these counts (Lazer et al., 2014; Salganik, 2017). Yet, simple counts can mislead if it is unclear what is counted and how.”
From Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019) · cites this paper as Lazer et al., 2014
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This unintelligibility is also subject to functional biases (section 4.1), as it depends on the mechanisms available on each social platform (e.g., having a like button, but not a dislike one), as well as to variations in platform algorithms and mechanisms in response to users actions (Lazer et al., 2014).”
From Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019) · cites this paper as Lazer et al., 2014
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“It also includes temporal validity, which captures to what extent constructs change over time (Howison et al., 2011) and may invalidate previous conclusions about societal and/or platform phenomena; e.g., see the case of Google Flu Trends (Lazer et al., 2014).”
From Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019) · cites this paper as Lazer et al., 2014
6 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 14 MARCH 2014 VOL 343 SCIENCE www.sciencemag.org 1204
“The 2009 version of GFT has weathered other media panics related to the fl u, including the 2005–2006 influenza A/H5N1 ("bird flu") outbreak”
“GFT engineers updated the algorithm in 2009, and this model has run ever since”
Citations
Cites
boyd, D., & Crawford, K. (2012) · Information, Communication & Society · doi:10.1080/1369118x.2012.678878 · Cited by 5,479
Twitter mood predicts the stock market
Bollen, J., Mao, H., & Zeng, X. (2011) · Journal of Computational Science · doi:10.1016/j.jocs.2010.12.007 · Cited by 5,206
Detecting influenza epidemics using search engine query data
Ginsberg, J., Mohebbi, M. H., Patel, R. S., Brammer, L., Smolinski, M. S., & Brilliant, L. (2009) · Nature · doi:10.1038/nature07634 · Cited by 4,295
Cohen, D. (1998) · The Women's Review of Books · doi:10.2307/4022859 · Cited by 4,261
Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk
Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012) · Political Analysis · doi:10.1093/pan/mpr057 · Cited by 4,008
Big Data : The Management Revolution
マカフィー, ア., ブリニョルフソン, エ., & 有賀, 裕. (2013) · Harvard business review · Cited by 3,549
Lazer, D., Pentland, A., Adamic, L., Aral, S., Barabási, A.-L., Brewer, D., Christakis, N., Contractor, N., Fowler, J., Gutmann, M., et al. (2009) · Science · doi:10.1126/science.1167742 · Cited by 3,365
Self-citationPredicting Elections with Twitter: What 140 Characters Reveal about Political Sentiment
Tumasjan, A., Sprenger, T., Sandner, P., & Welpe, I. (2010) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v4i1.14009 · Cited by 2,649
Multiscale mobility networks and the spatial spreading of infectious diseases
Balcan, D., Colizza, V., Gonçalves, B., Hu, H., Ramasco, J. J., & Vespignani, A. (2009) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0906910106 · Cited by 1,380
Self-citationPredicting consumer behavior with web search
Goel, S., Hofman, J. M., Lahaie, S., Pennock, D. M., & Watts, D. J. (2010) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1005962107 · Cited by 636
Research works printed but not linked: 7
- A. McAfee, E. Brynjolfsson, Harv. Bus. Rev. 90, 60 (2012).
- AAAI Conference on Weblogs and Social Media, Atlanta, Georgia, 11 to 15 July 2010 (Association for Advancement of Artifi cial Intelligence, 2010), pp. 178.
- P. Copeland et al., Int. Soc. Negl. Trop. Dis. 2013, 3
- I. M. Hall et al., Epidemiol. Infect. 135, 372 (2007).
- E. Mustafaraj, P. Metaxas, in Proceedings of the Web- Sci10, Raleigh, NC, 26 and 27 April 2010 (Web Science Trust, 2010); http://journal.webscience.org/317/.
- J. Ratkiewicz et al., in Proceedings of 5th International AAAI Conference on Weblogs and Social Media, San Francisco, CA, 7 to 11 August 2011 (AAAI, 2011), p. 297–304.
- 2013; http://chronicle.com/article/Researchers-Struggle- to-Secure/141591/.
Cited by
Items 701–800 of 2,513
An, J., & Weber, I. G. (2015) · EPJ Data Science · doi:10.1140/epjds/s13688-015-0058-9 · Cited by 32
“When it comes to now-casting, the poster child usually mentioned is Google Flu Trends [16], though limitations have been pointed out [21].”
BackgroundA data-driven model for influenza transmission incorporating media effects
Mitchell, L., & Ross, J. V. (2016) · Royal Society Open Science · doi:10.1098/rsos.160481 · Cited by 36
“On the other hand, whilst pure statistical models of media and prevalence are potentially of use for detecting and tracking disease incidence, they are subject to typical criticisms of “big data” analyses [23] as containing biases and tending towards overfitting.”
BackgroundUnsupervised Data Driven Feature Extraction by Means of Mutual Information Maximization
Marinoni, A., & Gamba, P. E. (2017) · IEEE Transactions on Computational Imaging · doi:10.1109/tci.2017.2669731 · Cited by 30
“cently become hot topics in EO data ingestion, analysis and visualization [5], [7], [14], [24]–[32].”
BackgroundArendt, F. (2020) · Health Communication · doi:10.1080/10410236.2020.1748820 · Cited by 29
Real-time estimation of disease activity in emerging outbreaks using internet search information
Aiken, E., McGough, S. F., Majumder, M. S., Wachtel, G., Nguyen, A. T., Viboud, C. G., & Santillana, M. (2020) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1008117 · Cited by 32
Using social media audience data to analyse the drivers of low-carbon diets
Eker, S., García, D., Valin, H., & van Ruijven, B. J. (2021) · Environmental Research Letters · doi:10.1088/1748-9326/abf770 · Cited by 36
Spikes and Variance: Using Google Trends to Detect and Forecast Protests
Timoneda, J. C., & Wibbels, E. M. (2021) · Political Analysis · doi:10.1017/pan.2021.7 · Cited by 40
“First, Google search trends can be responsive to media trends rather than the underlying phenomenon of interest (Lazer et al., 2014).”
BackgroundGrill, G. (2021) · Computer Supported Cooperative Work (CSCW) · doi:10.1007/s10606-021-09409-0 · Cited by 37
“Also, failures of promising predictive big data systems such as Google Flu Trends (Lazer et al. 2014) have further shown the volatility of such data.”
BackgroundJia, T., Yang, K., Peng, Z., Tang, L., Duan, H., & Luo, Y. (2022) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2022.3188788 · Cited by 37
A Practical Approach to Big Data in Tourism: A Low Cost Raspberry Pi Cluster
d’Amore, M., Baggio, R., & Valdani, E. (2014) · Springer eBooks · doi:10.1007/978-3-319-14343-9_13 · Cited by 30
“One example is the recent revision to the Google Flu Trends indicator that has been found to be contaminated by a number of additional factors (Lazer et al., 2014).”
BackgroundWikipedia and neurological disorders
Brigo, F., Igwe, S. C., Nardone, R., Lochner, P., Tezzon, F., & Otte, W. M. (2015) · Journal of Clinical Neuroscience · doi:10.1016/j.jocn.2015.02.006 · Cited by 30
Bauckhage, C., Hadiji, F., & Kersting, K. (2021) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v9i1.14598 · Cited by 30
“This way, we follow arguments brought forth, for instance, by Bauckhage et al. (2013) or Lazer et al. (2014) who criticized the lack of interpretability and the ‘big data hubris’ of purely data driven approaches for their potential of over-fitting and misleading results.”
BackgroundInfluentialDalgic, O. O., Özaltın, O. Y., Ciccotelli, W. A., & Erenay, F. S. (2017) · PLoS ONE · doi:10.1371/journal.pone.0172261 · Cited by 30
Governing the Complexity of Smart Data Cities: Setting a Research Agenda
Edelenbos, J., Hirzalla, F., van Zoonen, L., van Dalen, J., Bouma, G., Slob, A. F. L., & Woestenburg, A. (2017) · Public administration and information technology · doi:10.1007/978-3-319-58577-2_3 · Cited by 35
Influenza Activity Surveillance Based on Multiple Regression Model and Artificial Neural Network
Xue, H., Bai, Y., Hu, H., & Liang, H. (2017) · IEEE Access · doi:10.1109/access.2017.2771798 · Cited by 29
“The lead author David Lazer asserts that the disparity was caused by ‘‘big data arrogance’’ and algorithm variations [18].”
BackgroundResultForeign arrivals nowcasting in Italy with Google Trends data
Antolini, F., & Grassini, L. (2018) · Quality & Quantity · doi:10.1007/s11135-018-0748-z · Cited by 36
“In reference to the searching algorithm and the release of GT data (point 2), special attention must be taken when using Web search data (Butler 2013; Lazer et al. 2014; Has-sani and Silva 2015).”
BackgroundMethodValidating the Use of Google Trends to Enhance Pertussis Surveillance in California
Pollett, S., Wood, N., Boscardin, W. J., Bengtsson, H. L., Schwarcz, S. K., Harriman, K. H., Winter, K., & Rutherford, G. W. (2015) · PLoS Currents · doi:10.1371/currents.outbreaks.7119696b3e7523faa4543faac87c56c2 · Cited by 32
Big Data Technologies: New Opportunities for Diabetes Management
Bellazzi, R., Dagliati, A., Sacchi, L., & Segagni, D. (2015) · Journal of Diabetes Science and Technology · doi:10.1177/1932296815583505 · Cited by 29
“The potential biases and confounding factors hidden in the data will have to be carefully taken into account.(23) As it is often happening in the present times, when also media play an important role in emphasizing the importance of new concepts and ideas and thus in creating public expectations (that often reflect in…”
BackgroundBig Data and International Relations
Zwitter, A. (2015) · Ethics & International Affairs · doi:10.1017/s0892679415000362 · Cited by 37
The Lexicocalorimeter: Gauging public health through caloric input and output on social media
Alajajian, S. E., Williams, J. R., Reagan, A. J., Alajajian, S., Frank, M. R., Mitchell, L., Lahne, J., Danforth, C. M., & Dodds, P. S. (2017) · PLoS ONE · doi:10.1371/journal.pone.0168893 · Cited by 35
“Initially based very simply on search terms, the instrument proved unsurprisingly to be imperfect and in need of a more sophisticated approach [14].”
BackgroundThe Dark Web and cannabis use in the United States: Evidence from a big data research design
Jardine, E., & Lindner, A. M. (2019) · International Journal of Drug Policy · doi:10.1016/j.drugpo.2019.102627 · Cited by 29
Deep Learning for Adverse Event Detection from Web Search
Ahmad, F., Abbasi, A., Kitchens, B., Adjeroh, D., & Zeng, D. D. (2020) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2020.3017786 · Cited by 36
“Recent studies have shed light on a major challenge – the context problem [9] [26].”
BackgroundMemory Devices for Flexible and Neuromorphic Device Applications
Kim, D., Kim, I., & Lee, J. (2021) · Advanced Intelligent Systems · doi:10.1002/aisy.202000206 · Cited by 31
Satpathy, P., Kumar, S., & Prasad, P. (2021) · Disaster Medicine and Public Health Preparedness · doi:10.1017/dmp.2021.249 · Cited by 34
Zhang, X., Hou, L., Liu, J., Yang, K., Chai, C., Li, Y., & He, S. (2022) · Energy · doi:10.1016/j.energy.2022.124382 · Cited by 32
Early and Real-Time Detection of Seasonal Influenza Onset
Won, M., Marques-Pita, M., Louro, C., & Gonçalves-Sá, J. (2017) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1005330 · Cited by 27
Potentials and Challenges of the Health Data Cooperative Model
van Roessel, I., Reumann, M., & Brand, A. M. (2017) · Public Health Genomics · doi:10.1159/000489994 · Cited by 40
Tran, V. N., Ivanov, V. Y., Xu, D., & Kim, J. (2023) · Geophysical Research Letters · doi:10.1029/2023gl104464 · Cited by 30
Using cognitive psychology to understand GPT-like models needs to extend beyond human biases
Stella, M., Hills, T. T., & Kenett, Y. N. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2312911120 · Cited by 47
Converting Big Data into public health
Fung, I. C., Tse, Z. T. H., & Fu, K. (2015) · Science · doi:10.1126/science.347.6222.620-b · Cited by 28
Jie, Q., & Peng, T. (2016) · Internet Research · doi:10.1108/intr-04-2014-0104 · Cited by 29
Cato, K., Bockting, W. O., & Larson, E. (2016) · Journal of Empirical Research on Human Research Ethics · doi:10.1177/1556264616661611 · Cited by 30
“As the highly publicized errors in GFT have demonstrated, transparency to big data algorithms can allow exter-nal data scientists to point out possible errors (Lazer, Kennedy, King, & Vespignani, 2014).”
BackgroundPelham, B. W., Shimizu, M., Arndt, J., Carvallo, M., Solomon, S., & Greenberg, J. (2017) · Personality and Social Psychology Bulletin · doi:10.1177/0146167217736047 · Cited by 27
Socioeconomic bias in influenza surveillance
Scarpino, S. V., Scott, J. G., Eggo, R. M., Clements, B., Dimitrov, N. B., & Ancel, L. W. (2020) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1007941 · Cited by 33
Measuring Tourism: Methods, Indicators, and Needs
Baggio, R. (2018) · Springer eBooks · doi:10.1007/978-3-319-89941-1_13 · Cited by 42
A Deep Learning-Based Weather Forecast System for Data Volume and Recency Analysis
Booz, J., Yu, W., Xu, G., GRIFFITH, D. W. T., & Golmie, N. (2019) · 2012 International Conference on Computing, Networking and Communications (ICNC) · doi:10.1109/iccnc.2019.8685584 · Cited by 35
“more suitable for the training process? [22].”
Regional Infoveillance of COVID-19 Case Rates: Analysis of Search-Engine Query Patterns
Cousins, H. C., Cousins, C. C., Harris, A., & Pasquale, L. R. (2020) · Journal of Medical Internet Research · doi:10.2196/19483 · Cited by 32
Statistical Analysis of Glycosylation Reactions
Chang, C., Lin, M., & Wang, C. (2020) · Chemistry - A European Journal · doi:10.1002/chem.202003105 · Cited by 29
Big Data and the Little Big Bang: An Epistemological (R)evolution
Balazka, D., & Rodighiero, D. (2020) · Frontiers in Big Data · doi:10.3389/fdata.2020.00031 · Cited by 44
“This undermines our ability to identify instances of misuse of data and threatens two of the basic assumptions of science: comparability and replicability of findings (Gelman, 2013; Lazer et al., 2014; Biblić, 2016; Leonelli, 2018).”
BackgroundChong, M., & Park, H. W. (2021) · Scientometrics · doi:10.1007/s11192-021-04054-2 · Cited by 31
Introduction to Machine Learning in Obstetrics and Gynecology
Shazly, S. A., Trabuco, E. C., Ngufor, C. G., & Famuyide, A. O. (2022) · Obstetrics and Gynecology · doi:10.1097/aog.0000000000004706 · Cited by 33
“44 In addition, large databases are typically retrospective; thus, machine learning may be prone to the inherent drawbacks of retrospective studies.”
BackgroundSafety of proprotein convertase subtilisin/kexin 9 inhibitors: a systematic review and meta-analysis
Li, J., Du, H., Wang, Y., Aertgeerts, B., Guyatt, G. H., Hao, Q., Shen, Y., Li, L., Su, N., Delvaux, N., et al. (2022) · Heart · doi:10.1136/heartjnl-2021-320556 · Cited by 31
Age-Related Differences in the Accuracy of Web Query-Based Predictions of Influenza-Like Illness
Domnich, A., Panatto, D., Signori, A., Lai, P. L., Gasparini, R., & Amicizia, D. (2015) · PLoS ONE · doi:10.1371/journal.pone.0127754 · Cited by 32
Convex Risk Minimization to Infer Networks from probabilistic diffusion data at multiple scales
Sefer, E., & Kingsford, C. (2015) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde.2015.7113323 · Cited by 28
Ping, Q., Yang, C. C., Marshall, S. A., Avis, N. E., & Ip, E. H. (2016) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2016.2615850 · Cited by 30
“Fan et al. [22] utilized principal component analysis to identify three symptom clusters in clinical surveys of patients with metastatic cancer.”
BackgroundVolunteer Science: An Online Laboratory for Experiments in Social Psychology
Radford, J. P., Pilny, A., Reichelmann, A. V., Keegan, B., Welles, B. F., Hoye, J., Ognyanova, K., Meleis, W. M., & Lazer, D. M. J. (2016) · Social Psychology Quarterly · doi:10.1177/0190272516675866 · Cited by 39
Self-citationEditor's comments: synergies between big data and theory
Rai, A. (2016) · MIS Quarterly · doi:10.5555/3177617.3177618 · Cited by 32
Where did I get dengue? Detecting spatial clusters of infection risk with social network data
Souza, R. C. S. N. P., Assunção, R., Oliveira, D. M., Neill, D. B., & Meira, W. (2018) · Spatial and Spatio-temporal Epidemiology · doi:10.1016/j.sste.2018.11.005 · Cited by 31
Exploring crime patterns in Mexico City
Piña-García, C. A., & Ramírez-Ramírez, L. (2019) · Journal Of Big Data · doi:10.1186/s40537-019-0228-x · Cited by 31
Arendt, F., & Scherr, S. (2019) · Journal of Health Communication · doi:10.1080/10810730.2019.1652709 · Cited by 28
“Although Google Trends data have limitations (Lazer, Kennedy, King, & Vespignani, 2014), there are already studies that used search volume data as a proxy for public attention (e.g., Arendt & Scherr, 2017; Fond et al., 2015).”
MethodResultA gridded establishment dataset as a proxy for economic activity in China
Dong, L., Yuan, X., Li, M., Ratti, C., & Liu, Y. (2021) · Scientific Data · doi:10.1038/s41597-020-00792-9 · Cited by 35
Kershaw, D., Rowe, M., & Stacey, P. (2014) · ACM Web Science Conference (WebSci) · doi:10.1145/2615569.2615678 · Cited by 29
“[25] which critiqued the use of ’big data’ commenting that systems such as this and Google Flu should be in support of exsisting systems, and not a supplement.”
BackgroundPoirier, C., Lavenu, A., Bertaud, V., Campillo‐Gimenez, B., Chazard, E., Cuggia, M., & Bouzillé, G. (2018) · JMIR Public Health and Surveillance · doi:10.2196/11361 · Cited by 36
Chu, K., Colditz, J. B., Malik, M. M., et al. (2019) · Journal of Medical Internet Research · doi:10.2196/12443 · Cited by 27
Disambiguating the benefits and risks from public health data in the digital economy
Cheung, S. (2020) · Big Data & Society · doi:10.1177/2053951720933924 · Cited by 34
Is There a Place for Responsible Artificial Intelligence in Pandemics? A Tale of Two Countries
El-Haddadeh, R., Fadlalla, A. M., & Hindi, N. M. (2021) · Information Systems Frontiers · doi:10.1007/s10796-021-10140-w · Cited by 30
“Hence, offering opportunities to help contribute to the advancement of knowledge (Lazer et al., 2014).”
BackgroundUnderstanding Well-being Data: Improving Social and Cultural Policy, Practice and Research
Oman, S. (2021) · New directions in cultural policy research · doi:10.1007/978-3-030-72937-0 · Cited by 49
Measuring sustainable tourism with online platform data
Hoffmann, F. J., Braesemann, F., & Teubner, T. (2022) · EPJ Data Science · doi:10.1140/epjds/s13688-022-00354-6 · Cited by 37
The Detection of Emerging Trends Using Wikipedia Traffic Data and Context Networks
Kämpf, M., Tessenow, E., Kenett, D. Y., & Kantelhardt, J. W. (2015) · PLoS ONE · doi:10.1371/journal.pone.0141892 · Cited by 35
“Currently, Google Trends seems to be increasingly used in several areas of science, including, in particular, health sciences [6 – 9, 20, 21], and economic research [10 – 14, 17 – 19 ], to identify changes in attention towards various Secondly, we think that the ambiguity of search keywords can…”
BackgroundMethodInfluentialKhan, G. F., Lee, S., Park, J. Y., & Park, H. W. (2016) · Scientometrics · doi:10.1007/s11192-015-1822-0 · Cited by 28
“Google started its Search API service in 2002, and the reliability of Google's API service has been verified in previous research (Park 2014; Lazer et al. 2014; Delgado and Repiso 2013; Dugas et al. 2012; Olson et al. 2013; Lee et al. 2010a, b).”
MethodBail, C. A., Merhout, F., & Ding, P. (2018) · Science Advances · doi:10.1126/sciadv.aao5948 · Cited by 35
Optimal multi-source forecasting of seasonal influenza
Ertem, Z., Raymond, D., & Ancel, L. W. (2018) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1006236 · Cited by 35
Big Data Opportunities: System Health Monitoring and Management
Tsui, K., Zhao, Y., & Wang, D. (2019) · IEEE Access · doi:10.1109/access.2019.2917891 · Cited by 30
“6 depicts the ILI trend predicted by GFT and the actual CDC data over those dates [118].”
BackgroundMethodResultImproved forecasts of influenza-associated hospitalization rates with Google Search Trends
Kandula, S., Pei, S., & Shaman, J. L. (2019) · Journal of The Royal Society Interface · doi:10.1098/rsif.2019.0080 · Cited by 29
Carrignon, S., Bentley, R. A., & Ruck, D. J. (2019) · Humanities and Social Sciences Communications · doi:10.1057/s41599-019-0295-9 · Cited by 32
Editorial: Uncertainty in learning from big data
Wang, X., & Huang, J. Z. (2014) · Fuzzy Sets and Systems · doi:10.1016/j.fss.2014.10.010 · Cited by 27
Behavior Revealed in Mobile Phone Usage Predicts Credit Repayment
Björkegren, D., & Grissen, D. (2019) · World Bank, Washington, DC eBooks · doi:10.1596/1813-9450-9074 · Cited by 36
Nestor, B. A., McDermott, M. B. A., Chauhan, G., Naumann, T., Hughes, M. C., Goldenberg, A., & Ghassemi, M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.12583 · Cited by 26
“Care practices evolve over time via concept drift [2] and are serious in their own right [3].”
BackgroundSocial Data Mining and Seasonal Influenza Forecasts: The FluOutlook Platform
Zhang, Q., Gioannini, C., Paolotti, D., et al. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-23461-8_21 · Cited by 26
Self-citationPorter, N. D., Verdery, A. M., & Gaddis, S. M. (2020) · PLoS ONE · doi:10.1371/journal.pone.0233154 · Cited by 30
“For instance, compare the critique of sentiment analysis in the aforementioned Facebook experiment [16, 19] or concerns about search term inclusion in Google Flu [11, 55].”
BackgroundResultGoogle Trends and pollen concentrations in allergy and airway diseases in France
Bousquet, J. J., Onorato, G. L., Oliver, G., Basagaña, X., Annesi‐Maesano, I., Arnavielhe, S., Besancenot, J., Bossé, I., Bousquet, P., Charpin, D., et al. (2019) · Allergy · doi:10.1111/all.13804 · Cited by 26
The imaginary of precision public health
Kenney, M., & Mamo, L. (2019) · Medical Humanities · doi:10.1136/medhum-2018-011597 · Cited by 42
“David Lazer et al. (2014), “The Parable of Google Flu: Traps in Big Data Analysis,””
BackgroundPedersen, T., & Johansen, C. (2019) · AI & Society · doi:10.1007/s00146-019-00928-5 · Cited by 32
“GFT was initially fed with the officially recorded prevalence of Influenza (the condition is identified by physicians and then registered in national health databases) and simultaneously fueled with data from the Google Search Engine (where people typically search for symptoms related to Influenza).”
BackgroundInfluentialLymelight: forecasting Lyme disease risk using web search data
Sadilek, A., Hswen, Y., Bavadekar, S., Shekel, T., Brownstein, J. S., & Gabrilovich, E. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-0222-x · Cited by 29
Lee, E. W. J., & Yee, A. Z. H. (2020) · Frontiers in Communication · doi:10.3389/fcomm.2020.00011 · Cited by 31
“It is a consequence of an implicit adherence to radical empiricism, the belief that inductive pattern recognition from big data can substitute and perhaps overtake the traditional hypothetico-deductive model of science (Kitchin, 2014; Lazer et al., 2014).”
BackgroundDegeling, C., Carter, S. M., van Oijen, A. M., McAnulty, J., Sintchenko, V., Braunack‐Mayer, A., Yarwood, T., Johnson, J., & Gilbert, G. L. (2020) · BMC Medical Ethics · doi:10.1186/s12910-020-00474-6 · Cited by 26
Global Crises and the Role of BISE
Thomas, O., Hagen, S., Frank, U., Recker, J., Wessel, L., Kammler, F., Zarvić, N., & Timm, I. J. (2020) · Business & Information Systems Engineering · doi:10.1007/s12599-020-00657-w · Cited by 31
Dunnington, D., Trueman, B. F., Raseman, W. J., Anderson, L. E., & Gagnon, G. A. (2020) · ACS ES&T Engineering · doi:10.1021/acsestengg.0c00053 · Cited by 28
Location-Based Mobile Gaming and Local Depression Trends: A Study of Pokémon Go
Cheng, Z., Greenwood, B. N., & Pavlou, P. A. (2022) · Journal of Management Information Systems · doi:10.1080/07421222.2021.2023407 · Cited by 31
“The epidemiology literature acknowledges that construct validity is a common issue in internet data studies [36, 62].”
BackgroundAn Analysis of Body Language of Patients Using Artificial Intelligence
Abdulghafor, R. A. A., Abdelmohsen, A., Turaev, S., Ali, M. A. H., & Wani, S. (2022) · Healthcare · doi:10.3390/healthcare10122504 · Cited by 36
““Big data hubris and algorithm dynamics”, as [103] put it.”
BackgroundBig data, big problems: Why scientists should refrain from using Google Trends
Franzén, A. (2023) · Acta Sociologica · doi:10.1177/00016993221151118 · Cited by 37
“…Trends was considerably worse than traditional surveillance data from medical laboratories at making correct predictions about the prevalence of patients looking for medical appointments with doctors regarding in fl uenza-like sickness in the United States (Butler, 2013; Lazer et al., 2014: 1203).”
BackgroundFox, P., & Hendler, J. (2014) · Big Data · doi:10.1089/big.2014.0011 · Cited by 25
Lin, J. (2015) · The Annals of the American Academy of Political and Social Science · doi:10.1177/0002716215569174 · Cited by 32
Surveillance of pertussis: methods and implementation
Guiso, N., & von König, C. H. W. (2016) · Expert Review of Anti-infective Therapy · doi:10.1080/14787210.2016.1190272 · Cited by 26
Zhao, H. (2016) · International Journal of Strategic Communication · doi:10.1080/1553118x.2016.1244061 · Cited by 33
Internet search query data improve forecasts of daily emergency department volume
Tideman, S., Santillana, M., Bickel, J. P., & Reis, B. Y. (2019) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocz154 · Cited by 31
“However, GFT’s early predictive performance was disappointing and led to doubts about the validity of its original approach.(35) A growing body of research has attempted to expand upon GFT’s results and has demonstrated noticeable improvements.”
ResultGoogle Searches and Suicide Rates in Spain, 2004-2013: Correlation Study
Jiménez, A., Germán, M. Á. S., & Ramos, V. (2019) · JMIR Public Health and Surveillance · doi:10.2196/10919 · Cited by 33
Big Data in Medizin und Gesundheitswesen
Stefan Rüping, A. (2015) · Bundesgesundheitsblatt - Gesundheitsforschung - Gesundheitsschutz · doi:10.1007/s00103-015-2181-y · Cited by 36
Big Data and Population Processes: A Revolution?Preprint
Billari, F. C., & Zagheni, E. (2017) · SocArXiv (OSF Preprints) · doi:10.31235/osf.io/f9vzp · Cited by 31
Optimising the use of electronic medical records for large scale research in psychiatry
Newby, D., Taylor, N., Joyce, D. W., & Winchester, L. M. (2024) · Translational Psychiatry · doi:10.1038/s41398-024-02911-1 · Cited by 39
Atias, D., Ashri, S., Goldbourt, U., Benyamini, Y., Gilad-Bachrach, R., Hasin, T., Gerber, Y., & Obolski, U. (2025) · Annals of Epidemiology · doi:10.1016/j.annepidem.2025.07.024 · Cited by 37
Stevens, K., & Pfeiffer, D. U. (2015) · Spatial and Spatio-temporal Epidemiology · doi:10.1016/j.sste.2015.04.003 · Cited by 30
“…its prevalence estimates to be highly correlated with actual disease risk (Cook et al., 2011; Wilson et al., 2009; Ortiz et al., 2011; Dugas et al., 2012; Thompson et al., 2014) while others suggest GFT is not as reliable as CDC estimates (Lazer et al., 2014; Olson et al., 2013; Butler, 2013).”
Informatics: Make sense of health data
Elliott, J. H., Grimshaw, J., Altman, R., Bero, L., Goodman, S. N., Henry, D., Macleod, M., Tovey, D., Tugwell, P., White, H., et al. (2015) · Nature · doi:10.1038/527031a · Cited by 24
Madsen, A. K. (2015) · Big Data & Society · doi:10.1177/2053951714568727 · Cited by 28
“…dynamics of so-called blue team dynamics that must be dealt Such deliberate attempts at gaining information are known as red team dynamics (Lazer et al., 2014) and there is often no way to know whether such dynamics are at play in web-based data. means that the lack of con-trol of data…”
BackgroundThe perilous path from publication to practice
Chekroud, A. M., & Koutsouleris, N. (2017) · Molecular Psychiatry · doi:10.1038/mp.2017.227 · Cited by 30
Watad, ., Watad, S., Mahroum, N., Sharif, Κ., Amital, H., Bragazzi, N. L., & Adawi, M. (2019) · JMIR Public Health and Surveillance · doi:10.2196/publichealth.9176 · Cited by 27
Schneider, P. P., van Gool, C. J., Spreeuwenberg, P. M. M., Hooiveld, M., Donker, G., Barnett, D. J. M., & Paget, J. (2020) · Eurosurveillance · doi:10.2807/1560-7917.es.2020.25.21.1900221 · Cited by 29
Computer Simulations as Scientific Instruments
Alvarado, R. (2021) · Foundations of Science · doi:10.1007/s10699-021-09812-2 · Cited by 28
Deml, M. J., Jungo, K. T., Maessen, M., Martani, A., & Ulytė, A. (2022) · Public health reviews · doi:10.3389/phrs.2022.1604434 · Cited by 37
Toward a smart health: big data analytics and IoT for real-time miscarriage prediction
Asri, H., & Jarir, Z. (2023) · Journal Of Big Data · doi:10.1186/s40537-023-00704-9 · Cited by 32
“In fact, Google compares between people who search for the flu and people who have flu symptoms [2].”
Background
Metadata
- Authors
- Lazer, D., Kennedy, R., King, G., & Vespignani, A.
- Affiliations
- Northeastern University (US)
- Harvard University (US)
- Corresponding author
- Lazer, David
- Venue
- Science
- Published
- 2014-03-14
- Volume
- 343
- Issue
- 6176
- Pages
- 1203-1205
- ISSN
- 0036-8075 (ISSN-L)
- 1095-9203
- OpenAlex
- W2068181924
- Semantic Scholar
- 00d681cfff7f0224ace607e515f4ed510e792df9
- Cited by
- 2,523
- References
- 30
- Citations by year
- 2026: 113 · 2025: 101 · 2024: 121 · 2023: 150 · 2022: 168 · 2021: 229 · 2020: 273 · 2019: 261 · 2018: 272 · 2017: 249 · 2016: 271 · 2015: 219 · 2014: 91
- FWCI
- 208.16
- Type
- journal article
- Language
- en
- Open access
- green
- dash.harvard.edu
- Licence
- cc-by
- Locations
- Science (publishedVersion)
- PubMed (publishedVersion)
- Digital Access to Scholarship at Harvard (DASH) (Harvard University) (submittedVersion, cc-by)
- Topics
- Health Sciences › Medicine › Epidemiology › Data-Driven Disease Surveillance
- Social Sciences › Decision Sciences › Information Systems and Management › Big Data Technologies and Applications
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- Google Flu Trends, influenza surveillance, big data, prediction error, data quality
- MeSH terms
- Centers for Disease Control and Prevention, U.S., Data Collection, Data Interpretation, Statistical, Disease Outbreaks, Humans, Influenza, Human, Office Visits, Prognosis, United States, Social Media
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.