The Parable of Google Flu: Traps in Big Data Analysis
Open accessThe DOI matches its Crossref record; 30 of the 31 references it cites were found in a registry or an index; 5 of the 8 sources its claims cite were checked for support.
Abstract
Large errors in flu prediction were largely avoidable, which offers lessons for the use of big data.
Overview
CoChat Summary
TL;DR
The paper uses the failure of Google Flu Trends (GFT) to illustrate two traps in big data analysis: “big data hubris” and algorithm dynamics. It argues that GFT’s large flu-prediction errors were largely avoidable and that big data should supplement, not replace, traditional data and methods.
Key findings
- In February 2013, GFT predicted more than double the proportion of doctor visits for influenza-like illness (ILI) than the CDC; from 21 August 2011 to 1 September 2013, GFT reported overly high flu prevalence in 100 out of 108 weeks, and it overshot the 2011–2012 season by more than 50%.
- GFT errors were not random: they showed temporal autocorrelation and seasonality, meaning last week’s errors predicted this week’s errors and error direction/magnitude varied with time of year.
- Simple alternatives outperformed GFT: out-of-sample mean absolute error was 0.486 for GFT, 0.311 for lagged CDC, and 0.232 for combined GFT and CDC; all differences were statistically significant at P < 0.05. Even 3-week-old CDC data did better than GFT at projecting current flu prevalence.
- The authors attribute GFT’s problems to overfitting (initial GFT matched 50 million search terms to 1152 data points and missed the nonseasonal 2009 H1N1 pandemic) and to “algorithm dynamics,” including Google search algorithm changes and user behavior changes.
Methods
- Design: policy forum/case study comparing GFT estimates with CDC ILI surveillance data and alternative forecasting models.
- Data/sample: GFT and CDC flu prevalence estimates, including the period from 21 August 2011 to 1 September 2013; the initial GFT methodology reportedly fit 50 million search terms to 1152 data points.
- Main analysis: comparison of GFT alone, lagged CDC data with 52-week seasonality variables, and a combined GFT + CDC model using lagged CDC estimates, lagged GFT error, and seasonality variables; reported mean absolute error and statistical significance; Google Correlate used to examine correlated search terms. Supplementary materials contain data and methods.
Limitations
- The paper states that GFT never documented the 45 search terms used, released examples appear misleading, and Google Correlate is limited to national-level data whereas GFT was developed using regional correlations; therefore the original GFT analyses cannot be replicated from the information provided.
- The paper states that Google cannot make its full data available to outsiders because of privacy issues, though it argues there is no such constraint on derivative, aggregated data.
- Evident from the text: this is a commentary/policy forum rather than a full empirical article, with key data and methods placed in supplementary materials; the quantitative comparison depends on GFT and CDC data and model specifications not fully detailed in the main text.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
2 Valid · 2 Not valid · 2 Not checked · 8 Own results and interpretations
Some cited sources support this claim and some don’t — read each source’s verdict below.
In the paper
“This ad hoc method of throwing out peculiar search terms failed when GFT completely missed the nonseasonal 2009 infl uenza A–H1N1 pandemic ( 2, 14).”
What the sources say
- Partly supportedcites [2] matched to “Reassessing Google Flu Trends Data for Detection of Seasonal and Pandemic Influenza: A Comparative Epidemiological Study at Three Geographic Scales” (2013) sourceExperimental — accuracy not yet measured
The Discussion (page unavailable) says the original GFT model missed the spring 2009 pandemic wave, supporting the claim about its failure to detect the nonseasonal outbreak. It does not say GFT completely missed the entire 2009 pandemic. The Introduction describes automated selection of the original model’s search terms, not an ad hoc method of throwing out peculiar terms. (LLM, conf=0.86)
- Disputedcites [14] S. Cook et al., PLOS ONE 6, e23610 (2011). sourceExperimental — accuracy not yet measured
The Abstract (Methodology/Principal Findings) says the original GFT model underestimated the magnitude of illness during the 2009 H1N1 pandemic, not that it completely missed it. The Abstract (Conclusions) says both models performed well during the pandemic, although the updated model performed better. The Discussion notes that pandemic-specific queries were excluded, but does not describe this as an ad hoc method of throwing out peculiar terms. (LLM, conf=0.88)
Checked by a reasoning model — may be wrong.
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“Oddly, the few search terms offered in the papers ( 14) do not seem to be strongly related with either GFT or the CDC data (SM)—we surmise that the authors felt an unarticulated need to cloak the actual search terms identifi ed.”
What the sources say
- Disputedcites [14] S. Cook et al., PLOS ONE 6, e23610 (2011). sourceExperimental — accuracy not yet measured
The Results section (‘Search Behavior During pH1N1’; page unavailable) says that all three search queries shown in Figure 3 closely tracked CDC ILINet data before the pandemic, and ‘symptoms of flu’ continued to do so during it. The paper does report that the bronchitis and pneumonia queries underestimated ILINet data during the pandemic, but that does not support the broad claim that the offered terms were not strongly related to the data. (LLM, conf=0.87)
Checked by a reasoning model — may be wrong.
hedgedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“The Internet has opened the way for improving standard surveys, experiments, and health reporting ( 35).”
What the sources say
- Partly supportedcites [35] A. J. Berinsky et al., Polit. Anal. 20, 351–368 (2012). sourceExperimental — accuracy not yet measured
Section 2 describes using an Internet platform to run surveys with embedded experiments, and the Conclusion says MTurk can make experimental research easier and less expensive. The paper does not address improvements to health reporting; the disease scenario in section 4.3.2 is a framing experiment, not health reporting. Page numbers are unavailable in our copy. (LLM, conf=0.88)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Although not widely reported until 2013, the new GFT has been persistently overestimating flu prevalence for a much longer time.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“From 21 August 2011 to 1 September 2013, GFT reported overly high fl u prevalence 100 out of 108 weeks.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
3 more claims
Create a free account to see all 14 claims and every source verdict.
- It may state two separate claims as one. 5
- Probably not a substantial claim. 2
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Google Trends predicted more than twice the number of influenza-like illnesses than the CDC in February 2013 46 .”
From Unrepresentative big surveys significantly overestimated US vaccine uptake (2021) · cites this paper as [46]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This was one of the main reasons behind the failure of Google Flu Trends, since the data used for training the model in the first few years was not representative of the trends in subsequent years [15].”
From Theory-Guided Data Science: A New Paradigm for Scientific Discovery from Data (2017) · cites this paper as [15]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Social data research often relies on counting entities. These entities can be users, links, or messages, and the description is a summary of these counts (Lazer et al., 2014; Salganik, 2017). Yet, simple counts can mislead if it is unclear what is counted and how.”
From Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019) · cites this paper as Lazer et al., 2014
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This unintelligibility is also subject to functional biases (section 4.1), as it depends on the mechanisms available on each social platform (e.g., having a like button, but not a dislike one), as well as to variations in platform algorithms and mechanisms in response to users actions (Lazer et al., 2014).”
From Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019) · cites this paper as Lazer et al., 2014
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“It also includes temporal validity, which captures to what extent constructs change over time (Howison et al., 2011) and may invalidate previous conclusions about societal and/or platform phenomena; e.g., see the case of Google Flu Trends (Lazer et al., 2014).”
From Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019) · cites this paper as Lazer et al., 2014
6 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 14 MARCH 2014 VOL 343 SCIENCE www.sciencemag.org 1204
“The 2009 version of GFT has weathered other media panics related to the fl u, including the 2005–2006 influenza A/H5N1 ("bird flu") outbreak”
“GFT engineers updated the algorithm in 2009, and this model has run ever since”
Citations
Cites
boyd, D., & Crawford, K. (2012) · Information, Communication & Society · doi:10.1080/1369118x.2012.678878 · Cited by 5,479
Twitter mood predicts the stock market
Bollen, J., Mao, H., & Zeng, X. (2011) · Journal of Computational Science · doi:10.1016/j.jocs.2010.12.007 · Cited by 5,206
Detecting influenza epidemics using search engine query data
Ginsberg, J., Mohebbi, M. H., Patel, R. S., Brammer, L., Smolinski, M. S., & Brilliant, L. (2009) · Nature · doi:10.1038/nature07634 · Cited by 4,295
Cohen, D. (1998) · The Women's Review of Books · doi:10.2307/4022859 · Cited by 4,261
Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk
Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012) · Political Analysis · doi:10.1093/pan/mpr057 · Cited by 4,008
Big Data : The Management Revolution
マカフィー, ア., ブリニョルフソン, エ., & 有賀, 裕. (2013) · Harvard business review · Cited by 3,549
Lazer, D., Pentland, A., Adamic, L., Aral, S., Barabási, A.-L., Brewer, D., Christakis, N., Contractor, N., Fowler, J., Gutmann, M., et al. (2009) · Science · doi:10.1126/science.1167742 · Cited by 3,365
Self-citationPredicting Elections with Twitter: What 140 Characters Reveal about Political Sentiment
Tumasjan, A., Sprenger, T., Sandner, P., & Welpe, I. (2010) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v4i1.14009 · Cited by 2,649
Multiscale mobility networks and the spatial spreading of infectious diseases
Balcan, D., Colizza, V., Gonçalves, B., Hu, H., Ramasco, J. J., & Vespignani, A. (2009) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0906910106 · Cited by 1,380
Self-citationPredicting consumer behavior with web search
Goel, S., Hofman, J. M., Lahaie, S., Pennock, D. M., & Watts, D. J. (2010) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1005962107 · Cited by 636
Research works printed but not linked: 7
- A. McAfee, E. Brynjolfsson, Harv. Bus. Rev. 90, 60 (2012).
- AAAI Conference on Weblogs and Social Media, Atlanta, Georgia, 11 to 15 July 2010 (Association for Advancement of Artifi cial Intelligence, 2010), pp. 178.
- P. Copeland et al., Int. Soc. Negl. Trop. Dis. 2013, 3
- I. M. Hall et al., Epidemiol. Infect. 135, 372 (2007).
- E. Mustafaraj, P. Metaxas, in Proceedings of the Web- Sci10, Raleigh, NC, 26 and 27 April 2010 (Web Science Trust, 2010); http://journal.webscience.org/317/.
- J. Ratkiewicz et al., in Proceedings of 5th International AAAI Conference on Weblogs and Social Media, San Francisco, CA, 7 to 11 August 2011 (AAAI, 2011), p. 297–304.
- 2013; http://chronicle.com/article/Researchers-Struggle- to-Secure/141591/.
Cited by
Items 501–600 of 2,513
Big data analytics in health: an overview and bibliometric study of research activity
Galetsi, P., & Katsaliaki, K. (2019) · Health Information & Libraries Journal · doi:10.1111/hir.12286 · Cited by 58
Retooling Politics: How Digital Media Are Shaping Democracy
Jungherr, A. M., Rivero, G., & Gayo-Avello, D. (2020) · Cited by 48
Statistically-Robust Clustering Techniques for Mapping Spatial Hotspots: A Survey
Xie, Y., Shekhar, S., & Li, Y. (2022) · ACM Computing Surveys · doi:10.1145/3487893 · Cited by 68
“, algorithm dynamics which are widely believed to have partially caused the failure of Google Flu Trends [93]) and representativeness problems (e.”
BackgroundThick Big Data: Doing Digital Social Sciences
Jemielniak, D. (2020) · doi:10.1093/oso/9780198839705.001.0001 · Cited by 70
Increasing pressures and support for public healthcare in Europe
Jensen, C., & Naumann, E. (2016) · Health Policy · doi:10.1016/j.healthpol.2016.04.015 · Cited by 49
Social Monitoring for Public Health
Paul, M. J., & Dredze, M. H. (2017) · Synthesis lectures on information concepts, retrieval, and services · doi:10.2200/s00791ed1v01y201707icr060 · Cited by 88
Privacy behaviors after Snowden
Preibusch, S. (2015) · Communications of the ACM · doi:10.1145/2663341 · Cited by 71
Jung, K., & Park, H. W. (2016) · Government Information Quarterly · doi:10.1016/j.giq.2015.09.010 · Cited by 50
Pettit, M. (2016) · History of Psychology · doi:10.1037/hop0000023 · Cited by 41
Zhang, Y., Yakob, L., Bonsall, M. B., & Hu, W. (2019) · Scientific Reports · doi:10.1038/s41598-019-39871-2 · Cited by 50
Reality mining and predictive analytics for building smart applications
Asri, H., Mousannif, H., & Moatassime, H. A. (2019) · Journal Of Big Data · doi:10.1186/s40537-019-0227-y · Cited by 48
Rampazzo, F., Bijak, J., Vitali, A., Weber, I. G., & Zagheni, E. (2021) · Demography · doi:10.1215/00703370-9578562 · Cited by 54
Lu, S.-C., Xu, C., Nguyen, C., Geng, Y., Pfob, A., & Sidey-Gibbons, C. J. (2022) · JMIR Medical Informatics · doi:10.2196/33182 · Cited by 56
News and Narratives in Financial Systems: Exploiting Big Data for Systemic Risk AssessmentPreprint
Nyman, R., Kapadia, S., Tuckett, D., Gregory, D., Ormerod, P., & Smith, R. E. (2018) · SSRN Electronic Journal · doi:10.2139/ssrn.3098729 · Cited by 61
Big Data – Eine Revolution, die unser Leben verändern wird
Mayer‐Schönberger, V. (2015) · Bundesgesundheitsblatt - Gesundheitsforschung - Gesundheitsschutz · doi:10.1007/s00103-015-2180-z · Cited by 45
Spatialities of data: mapping social media ‘beyond the geotag’
Shelton, T. (2016) · GeoJournal · doi:10.1007/s10708-016-9713-3 · Cited by 52
“…tweets in particular, remain an incredibly limited data source in many respects, and because of these biases, is extremely problematic for purposes of predicting social behavior or inferring collective sentiment about a given issue (Lazer et al. 2014; Ruths and Pfeffer 2014; Hargittai 2015).”
BackgroundChapter 2: Users’ Views of Ethics in Social Media Research: Informed Consent, Anonymity, and Harm
Williams, M. L., Burnap, P., Sloan, L., Jessop, C., & Lepps, H. (2017) · Advances in research ethics and integrity · doi:10.1108/s2398-601820180000002002 · Cited by 81
Why policymakers should care about “big data” in healthcare
Bates, D. W., Heitmueller, A., Kakad, M., & Saria, S. (2018) · Health Policy and Technology · doi:10.1016/j.hlpt.2018.04.006 · Cited by 46
“A well-known example is the Google Flu Trends (GFT) algorithm that relied on people’s web-search patterns to infer flu rates.(10) However, for periods of time, GFT results were found to be divergent from rates measured by more formal approaches”
ResultA Bibliometrics analysis on big data research (2009–2018)
Xu, Z., & Yu, D. (2019) · Journal of Data Information and Management · doi:10.1007/s42488-019-00001-2 · Cited by 56
Attitudes and Folk Theories of Data Subjects on Transparency and Accuracy in Emotion Recognition
Grill, G., & Andalibi, N. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3512925 · Cited by 57
“However, various scholars have found that big social media data is insufficient to ensure high accuracy [97,112,135].”
BackgroundPsychological AI: Designing Algorithms Informed by Human Psychology
Gigerenzer, G. (2023) · Perspectives on Psychological Science · doi:10.1177/17456916231180597 · Cited by 69
“For instance, GFT has been criticized for its lack of transparency: The variables used were not revealed publicly, nor were the details of the three model updates (Lazer et al., 2014).”
BackgroundCharacterizing Ebola Transmission Patterns Based on Internet News Reports
Cleaton, J. M., Viboud, C. G., Simonsen, L., Hurtado, A. M., & Chowell, G. (2015) · Clinical Infectious Diseases · doi:10.1093/cid/civ748 · Cited by 47
Think Before You Collect: Setting Up a Data Collection Approach for Social Media Studies
Mayr, P., & Weller, K. (2016) · SAGE Publications Ltd eBooks · doi:10.4135/9781473983847.n8 · Cited by 82
Borah, P. (2015) · New Media & Society · doi:10.1177/1461444815621512 · Cited by 47
“However, scholars have also indicated that using big data does not mean researchers can ignore appropriate “measurement, construct validity and reliability, and dependencies among data” (Lazer et al., 2014: 8).”
BackgroundGlobal trends in the awareness of sepsis: insights from search engine data between 2012 and 2017
Jabaley, C. S., Blum, J. M., Groff, R. F., & O’Reilly-Shah, V. N. (2018) · Critical Care · doi:10.1186/s13054-017-1914-8 · Cited by 46
“Specifically, fits of the trailing data for forecasting purposes led to systematic overestimates of flu rates [46].”
BackgroundHjelmsø, M. H., Mollerup, S., Jensen, R. H., Pietroni, C., Lukjančenko, O., Schultz, A. C., Aarestrup, F. M., & Hansen, A. J. (2019) · PLoS ONE · doi:10.1371/journal.pone.0210368 · Cited by 45
Trust in Public Policy Algorithms
Kennedy, R. P., Waggoner, P. D., & Ward, M. M. (2022) · The Journal of Politics · doi:10.1086/716283 · Cited by 46
Self-citationUsing Twitter to Measure Public Discussion of Diseases: A Case Study
Weeg, C., Schwartz, H. A., Hill, S., Merchant, R. M., Arango, C. S., & Ungar, L. (2015) · JMIR Public Health and Surveillance · doi:10.2196/publichealth.3953 · Cited by 46
Conceptions of Good Science in Our Data-Rich World
Elliott, K. C., Cheruvelil, K. S., Montgomery, G. M., & Soranno, P. A. (2016) · BioScience · doi:10.1093/biosci/biw115 · Cited by 57
Huang, D., & Wang, J. (2017) · The Science of The Total Environment · doi:10.1016/j.scitotenv.2017.09.017 · Cited by 46
“Therefore, search engine query data and other novel data sources should serve as a supplement, but not as a substitute, for traditional data sources (Lazer et al., 2014).”
BackgroundOsthus, D., Daughton, A. R., & Priedhorsky, R. (2019) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1006599 · Cited by 43
Xie, G., Chen, T., Li, Y., Chen, T., Li, X., & Liu, Z. (2019) · Kidney Diseases · doi:10.1159/000504600 · Cited by 51
Themes in data mining, big data, and crime analytics
Oatley, G. (2021) · Wiley Interdisciplinary Reviews Data Mining and Knowledge Discovery · doi:10.1002/widm.1432 · Cited by 70
“In relation to data there are several vulnerabilities (human error, fragmented, and biased data), discussed at length by Lazer et al. (2014), Lartey (2016), and Robinson and Koepke (2016).”
BackgroundLearning As It Happens: A Decade of Analyzing and Shaping a Large-Scale Online Learning System
Brinkhuis, M. J. S., Savi, A. O., Hofman, A. D., Coomans, F., van der Maas, H. L. J., & Maris, G. (2018) · Journal of Learning Analytics · doi:10.18608/jla.2018.52.3 · Cited by 49
Reimsbach-Kounatze, C. (2015) · OECD digital economy papers · doi:10.1787/5js7t9wqzvg8-en · Cited by 70
“As Lazer et al. (2014) further explains, one major cause of the failures (in the case of Google Flu Trends) may have been that the Internet constantly changes and as a result the Google search engine itself constantly changes.”
BackgroundNelson, M. I., Lloyd‐Smith, J. O., Simonsen, L., Rambaut, A., Holmes, E. C., Chowell, G., Miller, M. A., Spiro, D. J., Grenfell, B. T., & Viboud, C. G. (2018) · Epidemics · doi:10.1016/j.epidem.2018.10.004 · Cited by 45
Data-Driven Innovation for Growth and Well-being
Hill, L. A., Levy, F. S., Kundra, V., Laki, B., & Smith, J. F. (2015) · Repository of the Academy's Library (Library of the Hungarian Academy of Sciences) · Cited by 47
Big Data and Social Indicators: Actual Trends and New Perspectives
di Bella, E., Leporatti, L., & Maggino, F. (2016) · Social Indicators Research · doi:10.1007/s11205-016-1495-y · Cited by 52
“Although other articles (e.g. Lazer et al. 2014) demonstrated that the model implemented in Ginsberg et al. (2009) could be much less accurate than what was initially stated, the widespread of the paper and the new opportunities that it opened for the analysis of complex phenomena gave rise to a…”
ResultBig Data approaches in social and behavioral science: four key trade-offs and a call for integration
Mahmoodi, J., Leckelt, M., van Zalk, M. H. W., Geukes, K., & Back, M. D. (2017) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2017.07.001 · Cited by 56
“[14] even warn that ‘big data hubris’ leads to the assumption that data quantity is a substitute for knowledge-driven methodologies and theories (also see [15]).”
BackgroundMethods Using Social Media and Search Queries to Predict Infectious Disease Outbreaks
Seo, D., & Shin, S.-Y. (2017) · Healthcare Informatics Research · doi:10.4258/hir.2017.23.4.343 · Cited by 43
“Second, as in the case of Google Flu, this system can fail to predict disease outbreaks correctly [20].”
BackgroundThe Digital Phenotype: a Philosophical and Ethical Exploration
Loi, M. (2018) · Philosophy & Technology · doi:10.1007/s13347-018-0319-1 · Cited by 47
“The B parable of Google Flu ^ (Lazer et al. 2014) is a remainder of the fact that these platforms are not optimized to produce knowledge that serves the purposes of public health.”
BackgroundMethodZhao, Z., Shaw, S., Yin, L., Fang, Z., Yang, X., Zhang, F., & Wu, S. (2019) · International Journal of Geographical Information Systems · doi:10.1080/13658816.2019.1584805 · Cited by 47
The Personality Panorama: Conceptualizing Personality through Big Behavioural Data
Boyd, R. L., Pasca, P., & Lanning, K. (2020) · European Journal of Personality · doi:10.1002/per.2254 · Cited by 55
“, “icy roads”, “holidays”, “high school basketball”) that were not true signals of the actual influenza virus (Lazer et al., 2014; Lazer & Kennedy, 2015; O’Neil & Schutt, 2013).”
BackgroundShelton, T. (2020) · Big Data & Society · doi:10.1177/2053951720965612 · Cited by 68
“Nevermind, of course, the countless biases in the data that make such systems less-than-perfect reflections of the material realities they purport to represent (cf. Crawford, 2013; Lazer et al., 2014).”
BackgroundClassification in the Wild: The Science and Art of Transparent Decision Making
Katsikopoulos, K. V., Şimşek, Ö., Buckmann, M., & Gigerenzer, G. (2021) · doi:10.7551/mitpress/11790.001.0001 · Cited by 42
Exploiting big data for critical care research
Docherty, A. B., & Lone, N. I. (2015) · Current Opinion in Critical Care · doi:10.1097/mcc.0000000000000228 · Cited by 42
Shen, H., & Zhang, L. (2023) · Science China Earth Sciences · doi:10.1007/s11430-022-9999-9 · Cited by 54
“However, some scholars insist that the “big data hubris” problem (Lazer et al., 2014) is serious, and the effectiveness of machine learning is overestimated.”
BackgroundVicente, P. (2023) · Quality & Quantity · doi:10.1007/s11135-023-01615-w · Cited by 57
“…by several challenging factors, related above all to the representation of human populations and human behavior, as well as by methodological issues, such as coverage (e.g., Gayo-Avello 2011), measurement (e.g., Cohen and Ruths 2013; Lazer et al. 2014) and generalizability (e.g., Tufekci 2014).”
BackgroundBagshaw, S. M., Goldstein, S. L., Ronco, C., Kellum, J. A., & for the ADQI 15 Consensus Group (2016) · Canadian Journal of Kidney Health and Disease · doi:10.1186/s40697-016-0103-z · Cited by 51
Twitter Population Sample Bias and its impact on predictive outcomes: a case study on elections
Filho, R. M., Almeida, J. M., & Pappa, G. L. (2015) · IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM) · doi:10.1145/2808797.2809328 · Cited by 45
“In this way, all the hopes of these methods are put into large amounts of data, but previous cases, such as the Google Flu [2], have already showed that is not always the case.”
Temporal Topic Modeling to Assess Associations between News Trends and Infectious Disease Outbreaks
Ghosh, S., Chakraborty, P., Nsoesie, E. O., Cohn, E. L., Mekaru, S. R., Brownstein, J. S., & Ramakrishnan, N. (2017) · Scientific Reports · doi:10.1038/srep40841 · Cited by 41
Automation, Big Data and Politics: A Research Review
Shorey, S., & Howard, P. N. (2016) · International journal of communication · Cited by 42
An unexpected journey: A few lessons from sciences Po médialab's experience
Venturini, T., Jacomy, M., Meunier, A., & Latour, B. (2017) · Big Data & Society · doi:10.1177/2053951717720949 · Cited by 51
“Yet, such abundance of traces does not always convert into valuable data (Lazer et al., 2014) and mainstream social media are not always the most interesting sources.”
BackgroundMonitoring Pertussis Infections Using Internet Search Queries
Zhang, Y., Milinovich, G. J., Xu, Z., Bambrick, H., Mengersen, K. L., Tong, S., & Hu, W. (2017) · Scientific Reports · doi:10.1038/s41598-017-11195-z · Cited by 51
Venturini, T., Cardon, D., & Cointet, J. (2015) · Réseaux · doi:10.3917/res.188.0009 · Cited by 58
Gianfredi, V., Bragazzi, N. L., Nucci, D., Martini, M., Rosselli, R., Minelli, L., & Moretti, M. (2018) · Frontiers in Public Health · doi:10.3389/fpubh.2018.00090 · Cited by 49
Emerging Challenges and Opportunities in Infectious Disease Epidemiology
Lewnard, J. A., & Reingold, A. L. (2018) · American Journal of Epidemiology · doi:10.1093/aje/kwy264 · Cited by 50
The evolution of software development orchestration: current state and an agenda for future research
Maruping, L. M., & Matook, S. (2020) · European Journal of Information Systems · doi:10.1080/0960085x.2020.1831834 · Cited by 41
Influenza Virus: Tracking, Predicting, and Forecasting
Ali, S. T., & Cowling, B. J. (2020) · Annual Review of Public Health · doi:10.1146/annurev-publhealth-010720-021049 · Cited by 49
The dark side of data ecosystems: A longitudinal study of the DAMD project
Aaen, J., Nielsen, J. A., & Carugati, A. (2021) · European Journal of Information Systems · doi:10.1080/0960085x.2021.1947753 · Cited by 54
“Another example is the use of Google search terms to predict flu trends; although the original collection of those data was clearly purposeful, their later use was not envisioned at collection time (Lazer et al., 2014).”
BackgroundEthical Issues in Social Science Research Employing Big Data
Hosseini, M., Wieczorek, M., & Gordijn, B. (2022) · Science and Engineering Ethics · doi:10.1007/s11948-022-00380-7 · Cited by 62
“Especially in cases where data sets are not linked with a published manuscript or lack supplementary information about the used methodology, researchers are unaware (and unable to become aware) of biases and limitations (Mittelstadt & Floridi, 2016; Lazer et al., 2014).”
BackgroundAI in Computational Pathology of Cancer: Improving Diagnostic Workflows and Clinical Outcomes?
Çifçi, D., Veldhuizen, G. P., Foersch, S., & Kather, J. N. (2023) · Annual Review of Cancer Biology · doi:10.1146/annurev-cancerbio-061521-092038 · Cited by 54
Forecasting influenza outbreak dynamics in Melbourne from Internet search query surveillance data
Moss, R. G., Zarebski, A. E., Dawson, P., & McCaw, J. M. (2016) · Influenza and Other Respiratory Viruses · doi:10.1111/irv.12376 · Cited by 42
“24,25 As the algorithm is based on Internet searches, 9 differences in Internet penetration and/or utilisation between populations presumably influence the quality of the data, 26 which highlights the importance of evaluating these forecasting techniques in an Australian context.”
BackgroundBragazzi, N. L., Dini, G., Toletone, A., Brigo, F., & Durando, P. (2016) · PLoS ONE · doi:10.1371/journal.pone.0166051 · Cited by 46
Forecasting state-level premature deaths from alcohol, drugs, and suicides using Google Trends data
Parker, J., Cuthbertson, C. A., Loveridge, S., Skidmore, M. L., & Dyar, W. (2016) · Journal of Affective Disorders · doi:10.1016/j.jad.2016.10.038 · Cited by 46
Scheider, S., Ostermann, F., & Adams, B. (2017) · Future Generation Computer Systems · doi:10.1016/j.future.2017.02.046 · Cited by 49
Communicable Disease Surveillance Ethics in the Age of Big Data and New Technology
Gilbert, G. L., Degeling, C., & Johnson, J. (2019) · Asian Bioethics Review · doi:10.1007/s41649-019-00087-1 · Cited by 41
Boulos, M. N. K., & Koh, K. P. (2021) · International Journal of Health Geographics · doi:10.1186/s12942-021-00266-0 · Cited by 41
Zhang, D., Wang, D., Peng, Q., Lin, J., Jin, T., Yang, T., Sorooshian, S., & Liu, Y. (2022) · Journal of Hydrology · doi:10.1016/j.jhydrol.2022.127427 · Cited by 43
International Gender Differences and Gaps in Online Social Networks
Magno, G., & Weber, I. G. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-13734-6_9 · Cited by 47
“Recently, its use as a figurehead has however been questioned [24].”
BackgroundCan Big Data Solve the Fundamental Problem of Causal Inference?
Titiunik, R. (2014) · PS Political Science & Politics · doi:10.1017/s1049096514001772 · Cited by 49
Jia, X., Willard, J., Karpatne, A., Read, J. S., Zwart, J. A., Steinbach, M. S., & Kumar, V. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2001.11086 · Cited by 40
“However, capturing this opportunity is contingent on a paradigm shift in data-intensive scientific discovery since the ”black box” use of ML often leads to serious false discoveries in scientific applications (Karpatne et al. 2017a; Lazer et al. 2014).”
BackgroundNews and Narratives in Financial Systems: Exploiting Big Data for Systemic Risk AssessmentPreprint
Nyman, R., Kapadia, S., Tuckett, D., Gregory, D., Ormerod, P., & Smith, R. E. (2018) · SSRN Electronic Journal · doi:10.2139/ssrn.3135262 · Cited by 61
Stacked kinship CNN vs. GBLUP for genomic predictions of additive and complex continuous phenotypes
Nazzicari, N., & Biscarini, F. (2022) · Scientific Reports · doi:10.1038/s41598-022-24405-0 · Cited by 44
“This is illustrated by the GoogleFlu predictions of influenza-like cases in 2013 in the United States which, in spite of highly correlated trends, were predicted to be the double of the actual cases 54 .”
MethodComputational modeling of heterogeneity and function of CD4+ T cells
Carbo, A., Hontecillas, R., Andrew, T., Eden, K. B., Mei, Y., Hoops, S., & Bassaganya-Riera, J. (2014) · Frontiers in Cell and Developmental Biology · doi:10.3389/fcell.2014.00031 · Cited by 57
“The so called “Big Data Hubris” (the often implicit assumption that big data are a substitute, rather than a supplement to, traditional data collection and analysis) already triggered an overestimation of Google’s assessment on flu prevalence in 2013 (Lazer et al., 2014).”
BackgroundSupplementing Public Health Inspection via Social Media
Schomberg, J., Haimson, O. L., Hayes, G. R., & Anton‐Culver, H. (2016) · PLoS ONE · doi:10.1371/journal.pone.0152117 · Cited by 40
“This model running on simulated data would then yield sensitivity, specificity, positive predictive value, and AUC using the R “ROCR”[39] module for each iteration of the dataset [28].”
MethodPTML Model of Enzyme Subclasses for Mining the Proteome of Biofuel Producing Microorganisms
Concu, R., Cordeiro, M. N. D. S., Munteanu, C. R., & González‐Díaz, H. (2019) · Journal of Proteome Research · doi:10.1021/acs.jproteome.8b00949 · Cited by 40
Fine-grained tourism prediction: Impact of social and environmental features
Khatibi, A., Belém, F. M., da Silva, A. P. C., Almeida, J. M., & Gonçalves, M. A. (2019) · Information Processing & Management · doi:10.1016/j.ipm.2019.102057 · Cited by 44
Godfree, R. C., Knerr, N. J., Godfree, D., Busby, J., Robertson, B. C., & Encinas‐Viso, F. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1902046116 · Cited by 38
Mapping Out Human-Centered Data Science: Methods, Approaches, and Best Practices
Kogan, M., Halfaker, A., Guha, S., Aragon, C., Muller, M., & Geiger, S. (2020) · Companion of the 2020 ACM International Conference on Supporting Group Work · doi:10.1145/3323994.3369898 · Cited by 40
“…data science approaches have been increasingly documented to fall short of producing a nuanced understanding of human behavior, as exemplified by failure of the Google flu trends [23] and the surprising outcome of the 2016 US presidential election, which was not predicted by most models [24].”
BackgroundBig Data-Driven Materials Science and Its FAIR Data Infrastructure
Draxl, C., & Scheffler, M. (2020) · Springer eBooks · doi:10.1007/978-3-319-44677-6_104 · Cited by 55
What’s So Different about Big Data?. A Primer for Clinicians Trained to Think Epidemiologically
Iwashyna, T. J., & Liu, V. X. (2014) · Annals of the American Thoracic Society · doi:10.1513/annalsats.201405-185as · Cited by 43
Big data and prediction: Four case studies
Northcott, R. (2019) · Studies in History and Philosophy of Science Part A · doi:10.1016/j.shpsa.2019.09.002 · Cited by 42
“Moreover, there is a new source of non-stationarity, namely that Google’s search algorithms themselves change frequently (Lazer et al. 2014).”
BackgroundData Mining Methods and Applications
Tsui, K., Chen, V., Jiang, W., Yang, F., & Kan, C. (2023) · Springer handbooks · doi:10.1007/978-1-4471-7503-2_38 · Cited by 48
Big Data: An Introduction for Librarians
Hoy, M. B. (2014) · Medical Reference Services Quarterly · doi:10.1080/02763869.2014.925709 · Cited by 48
Zahroh, H., Ma’rup, A., Tambunan, U. S. F., & Parikesit, A. A. (2016) · Drug Target Insights · doi:10.4137/dti.s38458 · Cited by 46
“11,12 The epitope-based vaccine has several advantages over conventional vaccines, and some are moving forward to the clinical trial pipeline.”
BackgroundUsing Participatory Web-based Surveillance Data to Improve Seasonal Influenza Forecasting in Italy
Perrotta, D., Tizzoni, M., & Paolotti, D. (2017) · International World Wide Web Conference (WWW) · doi:10.1145/3038912.3052670 · Cited by 36
“Following the same methodology adopted in previous works [27, 34, 38], we considered as our reference a baseline model that considers only ILI data from the traditional GP surveillance Influnet.”
MethodSaha, K., Weber, I., Birnbaum, M. L., & De Choudhury, M. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.6815 · Cited by 39
Malware in the future? Forecasting of analyst detection of cyber events
Bakdash, J. Z., Hutchinson, S. E., Zaroukian, E. G., Marusich, L. R., Thirumuruganathan, S., Sample, C., Hoffman, B., & Das, G. (2018) · Journal of Cybersecurity · doi:10.1093/cybsec/tyy007 · Cited by 49
“This issue is illustrated by the initial accuracy and then inaccuracy of Google Flu Trends (GFT) [47].”
BackgroundInfluentialPublic Values in the Age of Big Data: A Public Information Perspective
Ingrams, A. R. (2018) · Policy & Internet · doi:10.1002/poi3.193 · Cited by 55
Forecasting innovations in science, technology, and education
Börner, K., Rouse, W. B., Trunfio, P. A., & Stanley, E. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1818750115 · Cited by 42
Translational Health Disparities Research in a Data-Rich World
Breen, N., Jackson, J. S., Wood, F. B., Wong, D. W. S., & Zhang, X. (2019) · American Journal of Public Health · doi:10.2105/ajph.2019.304951 · Cited by 38
Participatory Disease Surveillance Systems: Ethical Framework
Geneviève, L. D., Martani, A., Wangmo, T., Paolotti, D., Koppeschaar, C. E., Kjelsø, C., Guerrisi, C., Hirsch, M., Woolley-Meza, O., Lukowicz, P., et al. (2019) · Journal of Medical Internet Research · doi:10.2196/12273 · Cited by 49
“Nonetheless, big data surveillance offers an unprecedented opportunity to monitor in a timely manner the spatial and temporal evolutions of epidemics with increased granularity compared with more traditional surveillance systems, provided that the potential flaws of big data analytics are taken into consideration [12,…”
Kalimeri, K., Delfino, M., Cattuto, C., Perrotta, D., Colizza, V., Guerrisi, C., Turbelin, C., Duggan, J., Edmunds, J., Obi, C., et al. (2019) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1006173 · Cited by 46
Huang, Y., Li, J., Wu, G., & Fei, T. (2020) · Cities · doi:10.1016/j.cities.2020.102719 · Cited by 39
Hogan, K. J., Macedo, B., Macha, V., Barman, A., & Jiang, X. (2021) · JMIR Medical Informatics · doi:10.2196/27449 · Cited by 47
Engines of Order: A Mechanology of Algorithmic Techniques
Rieder, B. (2020) · Amsterdam University Press eBooks · doi:10.2307/j.ctv12sdvf1 · Cited by 78
Zhao, I. Y., Ma, Y. X., Yu, M. W. C., Liu, J., Dong, W., Pang, Q., Lu, X. Q., Molassiotis, A., Holroyd, E. A., & Wong, W. C. W. (2021) · Journal of Medical Internet Research · doi:10.2196/32328 · Cited by 50
DEFENDER: Detecting and Forecasting Epidemics Using Novel Data-Analytics for Enhanced Response
Thapen, N., Simmie, D., Hankin, C., & Gillard, J. J. (2016) · PLoS ONE · doi:10.1371/journal.pone.0155417 · Cited by 44
A survey of social media data analysis for physical activity surveillance
Liu, S., & Young, S. D. (2016) · Journal of Forensic and Legal Medicine · doi:10.1016/j.jflm.2016.10.019 · Cited by 40
Metadata
- Authors
- Lazer, D., Kennedy, R., King, G., & Vespignani, A.
- Affiliations
- Northeastern University (US)
- Harvard University (US)
- Corresponding author
- Lazer, David
- Venue
- Science
- Published
- 2014-03-14
- Volume
- 343
- Issue
- 6176
- Pages
- 1203-1205
- ISSN
- 0036-8075 (ISSN-L)
- 1095-9203
- OpenAlex
- W2068181924
- Semantic Scholar
- 00d681cfff7f0224ace607e515f4ed510e792df9
- Cited by
- 2,523
- References
- 30
- Citations by year
- 2026: 113 · 2025: 101 · 2024: 121 · 2023: 150 · 2022: 168 · 2021: 229 · 2020: 273 · 2019: 261 · 2018: 272 · 2017: 249 · 2016: 271 · 2015: 219 · 2014: 91
- FWCI
- 208.16
- Type
- journal article
- Language
- en
- Open access
- green
- dash.harvard.edu
- Licence
- cc-by
- Locations
- Science (publishedVersion)
- PubMed (publishedVersion)
- Digital Access to Scholarship at Harvard (DASH) (Harvard University) (submittedVersion, cc-by)
- Topics
- Health Sciences › Medicine › Epidemiology › Data-Driven Disease Surveillance
- Social Sciences › Decision Sciences › Information Systems and Management › Big Data Technologies and Applications
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- Google Flu Trends, influenza surveillance, big data, prediction error, data quality
- MeSH terms
- Centers for Disease Control and Prevention, U.S., Data Collection, Data Interpretation, Statistical, Disease Outbreaks, Humans, Influenza, Human, Office Visits, Prognosis, United States, Social Media
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.