The Parable of Google Flu: Traps in Big Data Analysis
Open accessThe DOI matches its Crossref record; 31 of the 31 references it cites were found in a registry or an index; 5 of the 8 sources its claims cite were checked for support.
Abstract
Large errors in flu prediction were largely avoidable, which offers lessons for the use of big data.
Overview
CoChat Summary
TL;DR
The paper uses the failure of Google Flu Trends (GFT) to illustrate two traps in big data analysis: “big data hubris” and algorithm dynamics. It argues that GFT’s large flu-prediction errors were largely avoidable and that big data should supplement, not replace, traditional data and methods.
Key findings
- In February 2013, GFT predicted more than double the proportion of doctor visits for influenza-like illness (ILI) than the CDC; from 21 August 2011 to 1 September 2013, GFT reported overly high flu prevalence in 100 out of 108 weeks, and it overshot the 2011–2012 season by more than 50%.
- GFT errors were not random: they showed temporal autocorrelation and seasonality, meaning last week’s errors predicted this week’s errors and error direction/magnitude varied with time of year.
- Simple alternatives outperformed GFT: out-of-sample mean absolute error was 0.486 for GFT, 0.311 for lagged CDC, and 0.232 for combined GFT and CDC; all differences were statistically significant at P < 0.05. Even 3-week-old CDC data did better than GFT at projecting current flu prevalence.
- The authors attribute GFT’s problems to overfitting (initial GFT matched 50 million search terms to 1152 data points and missed the nonseasonal 2009 H1N1 pandemic) and to “algorithm dynamics,” including Google search algorithm changes and user behavior changes.
Methods
- Design: policy forum/case study comparing GFT estimates with CDC ILI surveillance data and alternative forecasting models.
- Data/sample: GFT and CDC flu prevalence estimates, including the period from 21 August 2011 to 1 September 2013; the initial GFT methodology reportedly fit 50 million search terms to 1152 data points.
- Main analysis: comparison of GFT alone, lagged CDC data with 52-week seasonality variables, and a combined GFT + CDC model using lagged CDC estimates, lagged GFT error, and seasonality variables; reported mean absolute error and statistical significance; Google Correlate used to examine correlated search terms. Supplementary materials contain data and methods.
Limitations
- The paper states that GFT never documented the 45 search terms used, released examples appear misleading, and Google Correlate is limited to national-level data whereas GFT was developed using regional correlations; therefore the original GFT analyses cannot be replicated from the information provided.
- The paper states that Google cannot make its full data available to outsiders because of privacy issues, though it argues there is no such constraint on derivative, aggregated data.
- Evident from the text: this is a commentary/policy forum rather than a full empirical article, with key data and methods placed in supplementary materials; the quantitative comparison depends on GFT and CDC data and model specifications not fully detailed in the main text.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
2 Valid · 2 Not valid · 2 Not checked · 8 Own results and interpretations
Some cited sources support this claim and some don’t — read each source’s verdict below.
In the paper
“This ad hoc method of throwing out peculiar search terms failed when GFT completely missed the nonseasonal 2009 infl uenza A–H1N1 pandemic ( 2, 14).”
What the sources say
- Partly supportedcites [2] matched to “Reassessing Google Flu Trends Data for Detection of Seasonal and Pandemic Influenza: A Comparative Epidemiological Study at Three Geographic Scales” (2013) sourceExperimental — accuracy not yet measured
The Discussion (page unavailable) says the original GFT model missed the spring 2009 pandemic wave, supporting the claim about its failure to detect the nonseasonal outbreak. It does not say GFT completely missed the entire 2009 pandemic. The Introduction describes automated selection of the original model’s search terms, not an ad hoc method of throwing out peculiar terms. (LLM, conf=0.86)
- Disputedcites [14] S. Cook et al., PLOS ONE 6, e23610 (2011). sourceExperimental — accuracy not yet measured
The Abstract (Methodology/Principal Findings) says the original GFT model underestimated the magnitude of illness during the 2009 H1N1 pandemic, not that it completely missed it. The Abstract (Conclusions) says both models performed well during the pandemic, although the updated model performed better. The Discussion notes that pandemic-specific queries were excluded, but does not describe this as an ad hoc method of throwing out peculiar terms. (LLM, conf=0.88)
Checked by a reasoning model — may be wrong.
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“Oddly, the few search terms offered in the papers ( 14) do not seem to be strongly related with either GFT or the CDC data (SM)—we surmise that the authors felt an unarticulated need to cloak the actual search terms identifi ed.”
What the sources say
- Disputedcites [14] S. Cook et al., PLOS ONE 6, e23610 (2011). sourceExperimental — accuracy not yet measured
The Results section (‘Search Behavior During pH1N1’; page unavailable) says that all three search queries shown in Figure 3 closely tracked CDC ILINet data before the pandemic, and ‘symptoms of flu’ continued to do so during it. The paper does report that the bronchitis and pneumonia queries underestimated ILINet data during the pandemic, but that does not support the broad claim that the offered terms were not strongly related to the data. (LLM, conf=0.87)
Checked by a reasoning model — may be wrong.
hedgedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“The Internet has opened the way for improving standard surveys, experiments, and health reporting ( 35).”
What the sources say
- Partly supportedcites [35] A. J. Berinsky et al., Polit. Anal. 20, 351–368 (2012). sourceExperimental — accuracy not yet measured
Section 2 describes using an Internet platform to run surveys with embedded experiments, and the Conclusion says MTurk can make experimental research easier and less expensive. The paper does not address improvements to health reporting; the disease scenario in section 4.3.2 is a framing experiment, not health reporting. Page numbers are unavailable in our copy. (LLM, conf=0.88)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Although not widely reported until 2013, the new GFT has been persistently overestimating flu prevalence for a much longer time.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“From 21 August 2011 to 1 September 2013, GFT reported overly high fl u prevalence 100 out of 108 weeks.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
3 more claims
Create a free account to see all 14 claims and every source verdict.
- It may state two separate claims as one. 5
- Probably not a substantial claim. 2
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Google Trends predicted more than twice the number of influenza-like illnesses than the CDC in February 2013 46 .”
From Unrepresentative big surveys significantly overestimated US vaccine uptake (2021) · cites this paper as [46]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This was one of the main reasons behind the failure of Google Flu Trends, since the data used for training the model in the first few years was not representative of the trends in subsequent years [15].”
From Theory-Guided Data Science: A New Paradigm for Scientific Discovery from Data (2017) · cites this paper as [15]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Social data research often relies on counting entities. These entities can be users, links, or messages, and the description is a summary of these counts (Lazer et al., 2014; Salganik, 2017). Yet, simple counts can mislead if it is unclear what is counted and how.”
From Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019) · cites this paper as Lazer et al., 2014
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This unintelligibility is also subject to functional biases (section 4.1), as it depends on the mechanisms available on each social platform (e.g., having a like button, but not a dislike one), as well as to variations in platform algorithms and mechanisms in response to users actions (Lazer et al., 2014).”
From Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019) · cites this paper as Lazer et al., 2014
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“It also includes temporal validity, which captures to what extent constructs change over time (Howison et al., 2011) and may invalidate previous conclusions about societal and/or platform phenomena; e.g., see the case of Google Flu Trends (Lazer et al., 2014).”
From Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019) · cites this paper as Lazer et al., 2014
6 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 14 MARCH 2014 VOL 343 SCIENCE www.sciencemag.org 1204
“The 2009 version of GFT has weathered other media panics related to the fl u, including the 2005–2006 influenza A/H5N1 ("bird flu") outbreak”
“GFT engineers updated the algorithm in 2009, and this model has run ever since”
Citations
Cites
boyd, D., & Crawford, K. (2012) · Information, Communication & Society · doi:10.1080/1369118x.2012.678878 · Cited by 5,479
Twitter mood predicts the stock market
Bollen, J., Mao, H., & Zeng, X. (2011) · Journal of Computational Science · doi:10.1016/j.jocs.2010.12.007 · Cited by 5,206
Detecting influenza epidemics using search engine query data
Ginsberg, J., Mohebbi, M. H., Patel, R. S., Brammer, L., Smolinski, M. S., & Brilliant, L. (2009) · Nature · doi:10.1038/nature07634 · Cited by 4,295
Cohen, D. (1998) · The Women's Review of Books · doi:10.2307/4022859 · Cited by 4,261
Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk
Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012) · Political Analysis · doi:10.1093/pan/mpr057 · Cited by 4,008
Big Data : The Management Revolution
マカフィー, ア., ブリニョルフソン, エ., & 有賀, 裕. (2013) · Harvard business review · Cited by 3,549
Lazer, D., Pentland, A., Adamic, L., Aral, S., Barabási, A.-L., Brewer, D., Christakis, N., Contractor, N., Fowler, J., Gutmann, M., et al. (2009) · Science · doi:10.1126/science.1167742 · Cited by 3,365
Self-citationPredicting Elections with Twitter: What 140 Characters Reveal about Political Sentiment
Tumasjan, A., Sprenger, T., Sandner, P., & Welpe, I. (2010) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v4i1.14009 · Cited by 2,649
Multiscale mobility networks and the spatial spreading of infectious diseases
Balcan, D., Colizza, V., Gonçalves, B., Hu, H., Ramasco, J. J., & Vespignani, A. (2009) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0906910106 · Cited by 1,380
Self-citationPredicting consumer behavior with web search
Goel, S., Hofman, J. M., Lahaie, S., Pennock, D. M., & Watts, D. J. (2010) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1005962107 · Cited by 636
Research works printed but not linked: 4
- McAfee A., Brynjolfsson E., Harv. Bus. Rev. 90, 60 (2012).
- Copeland P., et al.., Int. Soc. Negl. Trop. Dis. 2013, 3 (2013).
- Mustafaraj E., Metaxas P., in Proceedings of the Web-Sci10, Raleigh, NC, 26 and 27 April 2010 (Web Science Trust, 2010); http://journal.webscience.org/317/.
- Ratkiewicz J., et al.., in Proceedings of 5th International AAAI Conference on Weblogs and Social Media, San Francisco, CA, 7 to 11 August 2011 (AAAI, 2011), p. 297–304.
Cited by
Items 901–1,000 of 2,513
Ackley, S. F., Pilewski, S., Petrovic, V. S., Worden, L., Murray, E. L., & Porco, T. C. (2020) · Health Informatics Journal · doi:10.1177/1460458219897152 · Cited by 21
“23,24 Nonetheless, crowdsourced data and data from cloud-based devices may provide unique benefits, which may not be available for other data collection methods, 25,26 such as real-time availability and geopreci-sion.”
BackgroundBig Data Statistical Analysis of Facial Fractures in Korea
Park, C.-H., Chung, K. J., Kim, T. G., Lee, J. H., Kim, I., & Kim, Y. (2020) · Journal of Korean Medical Science · doi:10.3346/jkms.2020.35.e57 · Cited by 19
Arnaboldi, M., & AZZONE, G. F. (2020) · Heliyon · doi:10.1016/j.heliyon.2020.e04300 · Cited by 30
“It can, however, also be used to support decision-making in the public sector [7, 8], helping us to gain a deeper and more transparent understanding of our world [9], while improving the way we identify and assemble the choices made by people when faced with a number of”
BackgroundTension in big data using machine learning: Analysis and applications
Wang, H., Yao, Y., & Salhi, S. (2020) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2020.120175 · Cited by 25
Fantazzini, D. (2020) · Applied Econometrics · doi:10.22394/1993-7601-2020-59-33-54 · Cited by 29
Velsko, S. P., & Bates, T. (2016) · Health Security · doi:10.1089/hs.2015.0063 · Cited by 20
Infoveillance of infectious diseases in USA: STDs, tuberculosis, and hepatitis
Mavragani, A., & Ochoa, G. (2018) · Journal Of Big Data · doi:10.1186/s40537-018-0140-9 · Cited by 23
Yang, C.-Y., Chen, R., Chou, W.-L., Lee, Y., & Lo, Y.-S. (2019) · Journal of Medical Internet Research · doi:10.2196/12341 · Cited by 20
A Tutorial for Using Twitter Data in the Social Sciences: Data Collection, Preparation, and AnalysisPreprint
JJrgens, P., & Jungherr, A. M. (2016) · SSRN Electronic Journal · doi:10.2139/ssrn.2710146 · Cited by 25
Deregulating Collection: Must Privacy Give Way to Use Regulation?Preprint
Nissenbaum, H. F. (2017) · SSRN Electronic Journal · doi:10.2139/ssrn.3092282 · Cited by 24
Bail, C. A., Argyle, L. P., Brown, T., Bumpus, J. P., Chen, H., Hunzaker, M. B. F., Lee, J., Mann, M., Merhout, F., & Volfovsky, A. (2018) · SocArXiv (OSF Preprints) · doi:10.31235/osf.io/4ygux · Cited by 27
Bollenbach, J., Neubig, S., Hein, A., Keller, R., & Krcmar, H. (2024) · Information Technology & Tourism · doi:10.1007/s40558-024-00291-2 · Cited by 37
“Thus, the integration of search query data may enhance the predicting performance but should be evaluated in detail for each use case (Lazer et al. 2014; Li et al. 2021; Önder et al. 2019).”
ALGAN: Time Series Anomaly Detection with Adjusted-LSTM GAN
Bashar, M. A., & Nayak, R. (2025) · International Journal of Data Science and Analytics · doi:10.1007/s41060-025-00810-2 · Cited by 40
Do Google Trends forecast bitcoins? Stylized facts and statistical evidence
Arratia, A., & López-Barrantes, A. X. (2021) · Journal of Banking and Financial Technology · doi:10.1007/s42786-021-00027-4 · Cited by 19
Disruption and Legitimacy: Big Data in Society
Ferreira, C., Merendino, A., & Meadows, M. (2021) · Information Systems Frontiers · doi:10.1007/s10796-021-10155-3 · Cited by 28
A Review of Nonprobability Sampling Using Mobile Apps for Fishing Effort and Catch Surveys
Brick, J. M., Andrews, W. R., & Foster, J. R. (2021) · Transactions of the American Fisheries Society · doi:10.1002/tafs.10342 · Cited by 19
Now-casting Romanian migration into the United Kingdom by using Google Search engine data
Avramescu, A., & Wiśniowski, A. (2021) · Demographic Research · doi:10.4054/demres.2021.45.40 · Cited by 21
“We also follow the recommendations from literature that involve integrating new forms of data and traditional sources (see, e.g., the criticism of using Google Flu Trends by Lazer et al. 2014).”
MethodGao, C., Zhang, R., Chen, X., Yao, T., Song, Q., Ye, W., Li, P., Wang, Z., Yi, D., & Wu, Y. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00704-8 · Cited by 20
Integrating Artificial Intelligence and Machine Learning Into Cancer Clinical Trials
Kang, J., Chowdhry, A. K., Pugh, S. L., & Park, J. H. (2023) · Seminars in Radiation Oncology · doi:10.1016/j.semradonc.2023.06.004 · Cited by 23
Can data science achieve the ideal of evidence-based decision-making in environmental regulation?
Kim, E. S. (2024) · Technology in Society · doi:10.1016/j.techsoc.2024.102615 · Cited by 24
FPGA based nonlinear Support Vector Machine training using an ensemble learning
Rabieah, M. B., & Bouganis, C.-S. (2015) · International Conference on Field Programmable Logic and Applications (FPL) · doi:10.1109/fpl.2015.7293972 · Cited by 19
“Evidence of this can be seen in the google flu trend (GFT) where the need for model retraining was emphasized [4].”
BackgroundSchofield, P. (2017) · BJPsych Bulletin · doi:10.1192/pb.bp.116.055053 · Cited by 18
When everything becomes intelligence: machine learning and the connected world
Brantly, A. F. (2018) · Intelligence & National Security · doi:10.1080/02684527.2018.1452555 · Cited by 19
A Belief Rule Based Flood Risk Assessment Expert System using Real Time Sensor Data Streaming
Monrat, A. A., Islam, R. U., Hossain, M. S., & Andersson, K. (2018) · IEEE Conference on Local Computer Networks Workshops (LCN Workshops) · doi:10.1109/lcnw.2018.8628607 · Cited by 22
“Data sets are called big data when data becomes so large and complex that traditional data processing and management methods are not capable of handling it [43].”
What to know before forecasting the flu
Chakraborty, P., Lewis, B. L., Eubank, S., Brownstein, J. S., MARATHE, M. V., & Ramakrishnan, N. (2018) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1005964 · Cited by 24
Correlated impulses: Using Facebook interests to improve predictions of crime rates in urban areas
Fatehkia, M., O'Brien, D. T., & Weber, I. G. (2019) · PLoS ONE · doi:10.1371/journal.pone.0211350 · Cited by 25
Knebel, S., & Seele, P. (2019) · International Journal of Strategic Communication · doi:10.1080/1553118x.2019.1637878 · Cited by 20
“As a result, the relative lack of computational power will increase, while the relative amount of processable data decreases (Helbing, 2015; Lazer, Kennedy, King, & Vespignani, 2014) Strategic communication ’ s task to keep a well-balanced overview of all communication that is substantial for the…”
BackgroundHochman, A., Alpert, P., Negev, M., Abdeen, Z., Abdeen, A. M., Pinto, J. G., & Levine, H. (2020) · The Science of The Total Environment · doi:10.1016/j.scitotenv.2020.141686 · Cited by 22
Research challenges and directions in HVAC fault prevalence
Kim, J.-H., Trenbath, K., Granderson, J., Chen, Y., Crowe, E., Reeve, H. M., Newman, S., & Ehrlich, P. (2021) · Science and Technology for the Built Environment · doi:10.1080/23744731.2021.1898243 · Cited by 17
Improving Google Flu Trends for COVID-19 estimates using Weibo posts
Guo, S., Fang, F., Zhou, T., Zhang, W., Guo, Q., Zeng, R., Chen, X., Liu, J.-G., & Lu, X. B. (2021) · Data Science and Management · doi:10.1016/j.dsm.2021.07.001 · Cited by 26
“However, its predicted result was still slightly higher than that of CDC in the United States, and in the 31 weeks after the implementation of GFT 3.0, the prediction result was higher in 23 weeks (Lazer et al., 2014a).”
InfluentialAre Australia’s suburbs swamped by Asians and Muslims? Countering political claims with data
Forrest, J., Johnston, R. J., Siciliano, F., et al. (2017) · Australian Geographer · doi:10.1080/00049182.2017.1329383 · Cited by 19
“This post-truth situation has become an increasing problem with the growing volume of commercially collected big data now available, much of it with questionable statistical validity (Lazer et al. 2014) and whose interpretation by (often self-defined) ‘experts’ is”
BackgroundAragona, B., & Rosa, R. D. (2018) · Mathematical Population Studies · doi:10.1080/08898480.2017.1418113 · Cited by 26
Consumer Information for Data-Driven Decision Making: Teaching Socially Responsible Use of Data
Walker, K., & Moran, N. (2018) · Journal of Marketing Education · doi:10.1177/0273475318813176 · Cited by 29
Responsible and Ethical Use of AI in Education: Are We Forcing a Square Peg into a Round Hole?
Amigud, A., & Pell, D. J. (2025) · World · doi:10.3390/world6020081 · Cited by 24
“Technology that relies on big data suffers from what Lazer et al. [44] described as “big data hubris”.”
Eschmann, R., Groshek, J., Chanderdatt, R., Chang, K., & Whyte, M. (2020) · Social Media + Society · doi:10.1177/2056305120975716 · Cited by 27
“The combination of our mixed-methods approach answers a call (Lazer et al., 2014) but also provides a methodological roadmap for future studies seeking to make similar contributions that are topically agnostic.”
BackgroundA Scaling Perspective on AI Startups
Schulte‐Althoff, M., Fürstenau, D., & Lee, G. M. (2021) · Hawaii International Conference on System Sciences (HICSS) · doi:10.24251/hicss.2021.784 · Cited by 30
“While it is not entirely clear whether the business value of large data has unlimited positive returns [43, 63], access to enough relevant data is unquestionably a bottleneck factor for many AI startups to scale.”
BackgroundDolan, E. H., Goulding, J., Marshall, H., Smith, G., Long, G., & Tata, L. J. (2023) · Nature Communications · doi:10.1038/s41467-023-42776-4 · Cited by 24
Looking in the Wrong (La)place? The Promise and Perils of Becoming Big Data
Busch, L. (2016) · Science Technology & Human Values · doi:10.1177/0162243916677835 · Cited by 20
“The failure of Google Flu was due in part to what Lazer et al. (2014) call ‘‘big data Hubris.’’”
Geographic information science I: Geographic information observatories and opportunistic GIScience
Miller, H. J. (2017) · Progress in Human Geography · doi:10.1177/0309132517710741 · Cited by 24
“Avoiding getting tripped-up by data hubris requires understanding the processes that generate the data, including the measurement processes and technologies (Lazer et al., 2014).”
BackgroundBig data and data science in health care: What nurses and midwives need to know
O’Connor, S. (2017) · Journal of Clinical Nursing · doi:10.1111/jocn.14164 · Cited by 18
Use of daily Internet search query data improves real-time projections of influenza epidemics
Zimmer, C., Leuba, S. I., Yaesoubi, R., & Cohen, T. (2018) · Journal of The Royal Society Interface · doi:10.1098/rsif.2018.0220 · Cited by 17
“The validity of Internet search queries, which have included Google [6–12], Twitter [13,14] and Wikipedia [15,16], as proxy measures of influenza activity have been the subject of some debate [12].”
BackgroundDevelopments in Survey Research over the Past 60 Years: A Personal Perspective
Kalton, G. (2018) · International Statistical Review · doi:10.1111/insr.12287 · Cited by 33
Big Data and Dahl’s Challenge of Democratic Governance
Ingrams, A. R. (2019) · Review of Policy Research · doi:10.1111/ropr.12331 · Cited by 24
An automated quantitative content analysis process for humanitarian logistics research
Kunz, N. (2019) · Journal of Humanitarian Logistics and Supply Chain Management · doi:10.1108/jhlscm-06-2018-0051 · Cited by 22
Google Health Trends performance reflecting dengue incidence for the Brazilian states
Romero-Álvarez, D., Parikh, N., Osthus, D., Martinez, K. M., Generous, N., Valle, S. Y. D., & Manore, C. A. (2020) · BMC Infectious Diseases · doi:10.1186/s12879-020-04957-0 · Cited by 22
A Review on Predictive Systems and Data Models for COVID-19
Khan, F. N., Khanam, A., Ramlal, A., & Ahmad, S. (2020) · Studies in computational intelligence · doi:10.1007/978-981-15-8534-0_7 · Cited by 25
Big Data: Some Ethical Concerns for the Social Sciences
Weinhardt, M. (2021) · Social Sciences · doi:10.3390/socsci10020036 · Cited by 28
“Despite these potential benefits, it is not yet clear whether the new “data scape” available to social researchers really advances the possibilities of social research to a significant extent (Hamid Ekbia et al. 2015; Crawford 2013; Lazer et al. 2014).”
BackgroundBreiger, R. L. (2015) · Big Data & Society · doi:10.1177/2053951715602497 · Cited by 18
“Recognizing and addressing them should lead to additional progress in advancing the study of what Lazer et al. (2014) term an ‘‘all data revolution,’’ wherein innovative analytics using data from all traditional and new sources are developed and used to further our understanding of our world.”
BackgroundDoes Big Data Change the Privacy Landscape? A Review of the Issues
Keller‐McNulty, S., Shipp, S. S., & Schroeder, A. D. (2016) · Annual Review of Statistics and Its Application · doi:10.1146/annurev-statistics-041715-033453 · Cited by 19
Edelstein, M. L., Wallensten, A., Zetterqvist, I., & Hulth, A. (2014) · PLoS ONE · doi:10.1371/journal.pone.0100309 · Cited by 20
Lwin, M. O., Vijaykumar, S., Foo, S. S.-B., Fernando, O. N. N., Lim, G., Panchapakesan, C., & Wimalaratne, P. (2015) · Health Education Research · doi:10.1093/her/cyv065 · Cited by 22
O'Brien, D. T. (2016) · American Journal of Community Psychology · doi:10.1002/ajcp.12093 · Cited by 19
Johnson, A. K., Mikati, T., & Mehta, S. D. (2016) · Scientific Reports · doi:10.1038/srep36503 · Cited by 17
“Ginsberg et al. developed the Google-based application Google Flu Trends which can predict influenza 7–10 days earlier than traditional surveillance systems 2 .”
BackgroundHealthcare, Big Data and Cloud Computing
Alexandru, A., Alexandru, C., Coardoş, D., & Tudora, E. (2016) · Edinburgh Research Explorer (University of Edinburgh) · Cited by 32
Keeler, L. W., Bernstein, M. J., & Selin, C. L. (2019) · Technikzukünfte, Wissenschaft und Gesellschaft · doi:10.1007/978-3-658-27155-8_12 · Cited by 23
The Complex Economics of Artificial IntelligencePreprint
Mateos-Garcia, J. C. (2018) · SSRN Electronic Journal · doi:10.2139/ssrn.3294552 · Cited by 30
He, L., Shi, L., Song, W., Shen, J., Wang, L., Hu, X., & Zha, Y. (2024) · Water Resources Research · doi:10.1029/2024wr037582 · Cited by 21
“Unfortunately, from the standpoint of advancing scientific knowledge, the excellent simulation performance of existing AI techniques is not enough to dispel the mainstream geoscientific community's skepticism over their black‐box properties (Ebert‐Uphoff et al., 2019; Lazer et al., 2014).”
Urban Vitality Measurement Through Big Data and Internet of Things Technologies
Kim, Y.-L. (2025) · ISPRS International Journal of Geo-Information · doi:10.3390/ijgi14010014 · Cited by 26
“For instance, the biases inherent in data collection processes can skew the understandings of urban vitality, potentially leading to misguided policy interventions [13,14].”
Using google trends to assess for seasonal variation in knee injuries
Dewan, V., & Sur, H. (2018) · Journal of Arthroscopy and Joint Surgery · doi:10.1016/j.jajs.2018.02.002 · Cited by 16
Twitter vs. Zika—The role of social media in epidemic outbreaks surveillance
Abouzahra, M., & Tan, J. (2020) · Health Policy and Technology · doi:10.1016/j.hlpt.2020.10.014 · Cited by 22
A Systematic Review of Machine Learning for Assessment and Feedback of Treatment Fidelity
Ahmadi, A., Noetel, M., Schellekens, M., Parker, P. D., Antczak, D., Beauchamp, M. R., Dicke, T., Diezmann, C. M., Maeder, A., Ntoumanis, N., et al. (2021) · Psychosocial Intervention · doi:10.5093/pi2021a4 · Cited by 20
“For example, Google Flu Trends was able to predict historial flu rates from their search data, but it failed to accurately predict future data because methods became too sensitive to noise in the historical data (Lazer et al., 2014).”
BackgroundXie, W., Kimura, M., Takaki, K., Asada, Y., Iida, T., & Jia, X. (2022) · Water Resources Research · doi:10.1029/2021wr030493 · Cited by 22
“Additionally, because the learning process remains agnostic of theories and limitations, omissions may lead to spurious or inaccurate predictions, further weakening the generalizability and applicability of neural-network models (Lazer et al., 2014; Nayak et al., 2013).”
BackgroundAutographic Design: The Matter of Data in a Self-Inscribing World
Offenhuber, D. (2023) · The MIT Press eBooks · doi:10.7551/mitpress/14666.001.0001 · Cited by 35
Correlating Regional Aeroallergen Effects on Internet Search Activity
Willson, T. J., Lospinoso, J. A., Weitzel, E. K., & McMains, K. C. (2014) · Otolaryngology · doi:10.1177/0194599814560149 · Cited by 17
“When compared with actual CDC data, the Google tool was not found to be as accurate as previous studies and articles had initially reported.(12) In light of this, caution should be exercised when considering its implications.”
ResultSprečo, A., Eriksson, O., Dahlström, Ö., Cowling, B. J., & Timpka, T. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.7101 · Cited by 17
Local spatial obesity analysis and estimation using online social network sensors
Sun, Q., Wang, N., Li, S., & Zhou, H. (2018) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2018.03.010 · Cited by 15
Criminology in the age of data explosion: New directions
Ozkan, T. (2018) · The Social Science Journal · doi:10.1016/j.soscij.2018.10.010 · Cited by 25
Gu, W., Valavanis, K. P., Rutherford, M. J., & Rizzo, A. (2019) · International Conference on Unmanned Aircraft Systems (ICUAS) · doi:10.1109/icuas.2019.8797853 · Cited by 19
“• Requirement for a large number of training data; • Easy to learn spurious relationships which may lead to poor generalization capabilities [20]; • Lack of interpretability due to black-box characteristics; • No systematic approach for ANN architecture design: For instance, decisions on the number of layers and neuro…”
BackgroundDaughton, A. R., Chunara, R., & Paul, M. J. (2020) · JMIR Public Health and Surveillance · doi:10.2196/14986 · Cited by 17
Torrens, P. M. (2022) · Urban Informatics · doi:10.1007/s44212-022-00009-x · Cited by 25
“…somewhat, but thus far these big data are dominated by social media, which can be reasonably viewed as a relatively lackluster indicator of real-world behavior (Lazer et al., 2014), or even wholly unreliable (Jagadish et al., 2014), and perhaps particularly so in urban science (Kitchin, 2014).”
BackgroundA cultural evolutionary theory that explains both gradual and punctuated change
Vidiella, B., Carrignon, S., Bentley, R. A., O’Brien, M. J., & Valverde, S. (2022) · Journal of The Royal Society Interface · doi:10.1098/rsif.2022.0570 · Cited by 22
“For example, clear pay-offs often lose out to the spread of misinformation [119,124,130 – 136].”
BackgroundWilburn, J. K., O’Connor, C., Walsh, A. L., & Morgan, D. (2019) · International Journal of Infectious Diseases · doi:10.1016/j.ijid.2019.10.011 · Cited by 18
“A wellknown example of the limitations of automated ‘big data analysis’ systems to provide accurate, usable information is Google Flu Trends (Lazer et al., 2014).”
Promises and Pitfalls of Using Digital Traces for Demographic ResearchPreprint
Cesare, N., Lee, H., McCormick, T., et al. (2016) · SSRN Electronic Journal · doi:10.2139/ssrn.2839585 · Cited by 17
Zou, Z., Tang, F., Qiao, L., Wang, S., & Zhang, H. (2025) · Frontiers in Microbiology · doi:10.3389/fmicb.2025.1528696 · Cited by 21
Joyce, K. A., & Senier, L. (2017) · Environmental Sociology · doi:10.1080/23251042.2017.1281375 · Cited by 18
“It quickly became evident, however, that Google’s results were flawed (Lazer et al. 2014), although the shaky findings have hardly doused enthusiasm for Big Data among policymakers.”
Backgroundde Vries, I., & Radford, J. P. (2021) · The British Journal of Criminology · doi:10.1093/bjc/azab077 · Cited by 24
Keusch, F., & Kreuter, F. (2021) · Routledge eBooks · doi:10.4324/9781003024583-8 · Cited by 31
Probabilistic forecasts of international bilateral migration flows
Welch, N. G., & Raftery, A. E. (2022) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2203822119 · Cited by 26
Harnessing Google Health Trends Data for Epidemiologic Research
Neumann, K., Mason, S. M., Farkas, K., Santaularia, N. J., Ahern, J., & Riddell, C. A. (2022) · American Journal of Epidemiology · doi:10.1093/aje/kwac171 · Cited by 25
Zhang, X., Hou, L., Zhu, Z., et al. (2023) · Geoenergy Science and Engineering · doi:10.1016/j.geoen.2023.212013 · Cited by 16
Forecasting virus outbreaks with social media data via neural ordinary differential equations
Núñez, M., Barreiro, N. L., Barrio, R. A., & Rackauckas, C. (2023) · Scientific Reports · doi:10.1038/s41598-023-37118-9 · Cited by 24
Machine learning applications in vadose zone hydrology: A review
Li, X., Nieber, J. L., & Kumar, V. (2024) · Vadose Zone Journal · doi:10.1002/vzj2.20361 · Cited by 23
“An example of such failure is the epic failure of the Google flu trend (Lazer et al., 2014).”
Error correction mechanisms in social networks can reduce accuracy and encourage innovation
Brashears, M. E., & Gladstone, E. (2015) · Social Networks · doi:10.1016/j.socnet.2015.07.007 · Cited by 18
Martin, L. J., Lee, B. E., & Yasui, Y. (2015) · Epidemiology and Infection · doi:10.1017/s0950268815001478 · Cited by 16
Bordonaro, S. F., McGillicuddy, D. C., Pompei, F., Burmistrov, D., Harding, C., & Sánchez, L. D. (2016) · BMC Emergency Medicine · doi:10.1186/s12873-016-0080-7 · Cited by 16
Early social science research about Big Data
Youtie, J., Porter, A. L., & Huang, Y. (2016) · Science and Public Policy · doi:10.1093/scipol/scw021 · Cited by 19
Model selection for seasonal influenza forecasting
Zarebski, A. E., Dawson, P., McCaw, J. M., & Moss, R. G. (2017) · Infectious Disease Modelling · doi:10.1016/j.idm.2016.12.004 · Cited by 16
Vidal-García, J., Vidal, M., & Barros, R. H. (2017) · Advances in e-business research series · doi:10.4018/978-1-5225-2104-4.ch013 · Cited by 17
Wang, Y. (2017) · Education Policy Analysis Archives · doi:10.14507/epaa.25.3037 · Cited by 25
Translational medicine in the Age of Big Data
Tatonetti, N. P. (2017) · Briefings in Bioinformatics · doi:10.1093/bib/bbx116 · Cited by 15
Guidelines for the responsible application of data analytics
Clarke, R. (2017) · Computer law & security review · doi:10.1016/j.clsr.2017.11.002 · Cited by 20
Towards an Archaeological‐Ethnographic Approach to Big Data: Rethinking Data Veracity
Zhang, S., Zhao, B., & Ventrella, J. (2018) · Ethnographic Praxis in Industry Conference Proceedings · doi:10.1111/1559-8918.2018.01197 · Cited by 16
Ouyang, Z., Sargeant, J. M., Thomas, A., Wycherley, K., Ma, R., Esmaeilbeigi, R., Versluis, A. M., Stacey, D., STONE, E. A., Poljak, Z., et al. (2019) · Animal Health Research Reviews · doi:10.1017/s1466252319000136 · Cited by 20
“…existing big datasets in veterinary medicine and animal health, like human health, may have been extracted from data sources that were not designed to answer questions currently held by researchers (Lazer et al ., 2014; Chen and Asch, 2017), making it difficult to conduct studies that use big data.”
BackgroundInfectious Disease Forecasting for Public Health
Lauer, S. A., Brown, A., & Reich, N. G. (2020) · Oxford University Press eBooks · doi:10.1093/oso/9780198853244.003.0004 · Cited by 20
“Perhaps the most famous and controversial example of using digital data streams to support infectious disease prediction surround the early promising performance [117,118] and later dismal failure [119] of Google Flu trends to predict the influenza-like-illness in the US.”
BackgroundRaubenheimer, J., Riordan, B. C., Merrill, J. E., Winter, T. J., Ward, R. M., Scarf, D., & Buckley, N. A. (2020) · International Journal of Drug Policy · doi:10.1016/j.drugpo.2020.103083 · Cited by 16
Predictive analytics and machine learning for medical informatics: A survey of tasks and techniques
Lamba, D., Hsu, W. H., & Alsadhan, M. (2021) · Elsevier eBooks · doi:10.1016/b978-0-12-821777-1.00023-9 · Cited by 33
“The authors cite Google Flu Trends as an example of the need to update machine learning models to account for this data temporality, as it persistently overestimated flu [99].”
BackgroundDecoding Human Behavior with Big Data? Critical, Constructive Input from the Decision Sciences
Katsikopoulos, K. V., & Canellas, M. C. (2022) · AI Magazine · doi:10.1002/aaai.12034 · Cited by 25
Dong, X., & Bollen, J. (2015) · PLoS ONE · doi:10.1371/journal.pone.0120039 · Cited by 16
“We caution again that similar methods have been shown to be subject to systematic challenges [19], which we have painstakingly sought to avoid.”
MethodWhat happens when big data blunders?
Kugler, L. (2016) · Communications of the ACM · doi:10.1145/2911975 · Cited by 16
Uhl, A., Kolleck, N., & Schiebel, E. (2016) · European Journal of Futures Research · doi:10.1007/s40309-016-0102-4 · Cited by 20
Metadata
- Authors
- Lazer, D., Kennedy, R., King, G., & Vespignani, A.
- Affiliations
- Northeastern University (US)
- Harvard University (US)
- Corresponding author
- Lazer, David
- Venue
- Science
- Published
- 2014-03-14
- Volume
- 343
- Issue
- 6176
- Pages
- 1203-1205
- ISSN
- 0036-8075 (ISSN-L)
- 1095-9203
- OpenAlex
- W2068181924
- Semantic Scholar
- 00d681cfff7f0224ace607e515f4ed510e792df9
- Cited by
- 2,523
- References
- 30
- Citations by year
- 2026: 113 · 2025: 101 · 2024: 121 · 2023: 150 · 2022: 168 · 2021: 229 · 2020: 273 · 2019: 261 · 2018: 272 · 2017: 249 · 2016: 271 · 2015: 219 · 2014: 91
- FWCI
- 208.16
- Type
- journal article
- Language
- en
- Open access
- green
- dash.harvard.edu
- Licence
- cc-by
- Locations
- Science (publishedVersion)
- PubMed (publishedVersion)
- Digital Access to Scholarship at Harvard (DASH) (Harvard University) (submittedVersion, cc-by)
- Topics
- Health Sciences › Medicine › Epidemiology › Data-Driven Disease Surveillance
- Social Sciences › Decision Sciences › Information Systems and Management › Big Data Technologies and Applications
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- Google Flu Trends, influenza surveillance, big data, prediction error, data quality
- MeSH terms
- Centers for Disease Control and Prevention, U.S., Data Collection, Data Interpretation, Statistical, Disease Outbreaks, Humans, Influenza, Human, Office Visits, Prognosis, United States, Social Media
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.