The Parable of Google Flu: Traps in Big Data Analysis
Open accessThe DOI matches its Crossref record; 31 of the 31 references it cites were found in a registry or an index; 5 of the 8 sources its claims cite were checked for support.
Abstract
Large errors in flu prediction were largely avoidable, which offers lessons for the use of big data.
Overview
CoChat Summary
TL;DR
The paper uses the failure of Google Flu Trends (GFT) to illustrate two traps in big data analysis: “big data hubris” and algorithm dynamics. It argues that GFT’s large flu-prediction errors were largely avoidable and that big data should supplement, not replace, traditional data and methods.
Key findings
- In February 2013, GFT predicted more than double the proportion of doctor visits for influenza-like illness (ILI) than the CDC; from 21 August 2011 to 1 September 2013, GFT reported overly high flu prevalence in 100 out of 108 weeks, and it overshot the 2011–2012 season by more than 50%.
- GFT errors were not random: they showed temporal autocorrelation and seasonality, meaning last week’s errors predicted this week’s errors and error direction/magnitude varied with time of year.
- Simple alternatives outperformed GFT: out-of-sample mean absolute error was 0.486 for GFT, 0.311 for lagged CDC, and 0.232 for combined GFT and CDC; all differences were statistically significant at P < 0.05. Even 3-week-old CDC data did better than GFT at projecting current flu prevalence.
- The authors attribute GFT’s problems to overfitting (initial GFT matched 50 million search terms to 1152 data points and missed the nonseasonal 2009 H1N1 pandemic) and to “algorithm dynamics,” including Google search algorithm changes and user behavior changes.
Methods
- Design: policy forum/case study comparing GFT estimates with CDC ILI surveillance data and alternative forecasting models.
- Data/sample: GFT and CDC flu prevalence estimates, including the period from 21 August 2011 to 1 September 2013; the initial GFT methodology reportedly fit 50 million search terms to 1152 data points.
- Main analysis: comparison of GFT alone, lagged CDC data with 52-week seasonality variables, and a combined GFT + CDC model using lagged CDC estimates, lagged GFT error, and seasonality variables; reported mean absolute error and statistical significance; Google Correlate used to examine correlated search terms. Supplementary materials contain data and methods.
Limitations
- The paper states that GFT never documented the 45 search terms used, released examples appear misleading, and Google Correlate is limited to national-level data whereas GFT was developed using regional correlations; therefore the original GFT analyses cannot be replicated from the information provided.
- The paper states that Google cannot make its full data available to outsiders because of privacy issues, though it argues there is no such constraint on derivative, aggregated data.
- Evident from the text: this is a commentary/policy forum rather than a full empirical article, with key data and methods placed in supplementary materials; the quantitative comparison depends on GFT and CDC data and model specifications not fully detailed in the main text.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
2 Valid · 2 Not valid · 2 Not checked · 8 Own results and interpretations
Some cited sources support this claim and some don’t — read each source’s verdict below.
In the paper
“This ad hoc method of throwing out peculiar search terms failed when GFT completely missed the nonseasonal 2009 infl uenza A–H1N1 pandemic ( 2, 14).”
What the sources say
- Partly supportedcites [2] sourceExperimental — accuracy not yet measured
The Discussion (page unavailable) says the original GFT model missed the spring 2009 pandemic wave, supporting the claim about its failure to detect the nonseasonal outbreak. It does not say GFT completely missed the entire 2009 pandemic. The Introduction describes automated selection of the original model’s search terms, not an ad hoc method of throwing out peculiar terms. (LLM, conf=0.86)
- Disputedcites [14] S. Cook et al., PLOS ONE 6, e23610 (2011). sourceExperimental — accuracy not yet measured
The Abstract (Methodology/Principal Findings) says the original GFT model underestimated the magnitude of illness during the 2009 H1N1 pandemic, not that it completely missed it. The Abstract (Conclusions) says both models performed well during the pandemic, although the updated model performed better. The Discussion notes that pandemic-specific queries were excluded, but does not describe this as an ad hoc method of throwing out peculiar terms. (LLM, conf=0.88)
Checked by a reasoning model — may be wrong.
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“Oddly, the few search terms offered in the papers ( 14) do not seem to be strongly related with either GFT or the CDC data (SM)—we surmise that the authors felt an unarticulated need to cloak the actual search terms identifi ed.”
What the sources say
- Disputedcites [14] S. Cook et al., PLOS ONE 6, e23610 (2011). sourceExperimental — accuracy not yet measured
The Results section (‘Search Behavior During pH1N1’; page unavailable) says that all three search queries shown in Figure 3 closely tracked CDC ILINet data before the pandemic, and ‘symptoms of flu’ continued to do so during it. The paper does report that the bronchitis and pneumonia queries underestimated ILINet data during the pandemic, but that does not support the broad claim that the offered terms were not strongly related to the data. (LLM, conf=0.87)
Checked by a reasoning model — may be wrong.
hedgedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“The Internet has opened the way for improving standard surveys, experiments, and health reporting ( 35).”
What the sources say
- Partly supportedcites [35] A. J. Berinsky et al., Polit. Anal. 20, 351–368 (2012). sourceExperimental — accuracy not yet measured
Section 2 describes using an Internet platform to run surveys with embedded experiments, and the Conclusion says MTurk can make experimental research easier and less expensive. The paper does not address improvements to health reporting; the disease scenario in section 4.3.2 is a framing experiment, not health reporting. Page numbers are unavailable in our copy. (LLM, conf=0.88)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Although not widely reported until 2013, the new GFT has been persistently overestimating flu prevalence for a much longer time.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“From 21 August 2011 to 1 September 2013, GFT reported overly high fl u prevalence 100 out of 108 weeks.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
3 more claims
Create a free account to see all 14 claims and every source verdict.
- It may state two separate claims as one. 5
- Probably not a substantial claim. 2
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Google Trends predicted more than twice the number of influenza-like illnesses than the CDC in February 2013 46 .”
From Unrepresentative big surveys significantly overestimated US vaccine uptake (2021) · cites this paper as [46]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This was one of the main reasons behind the failure of Google Flu Trends, since the data used for training the model in the first few years was not representative of the trends in subsequent years [15].”
From Theory-Guided Data Science: A New Paradigm for Scientific Discovery from Data (2017) · cites this paper as [15]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Social data research often relies on counting entities. These entities can be users, links, or messages, and the description is a summary of these counts (Lazer et al., 2014; Salganik, 2017). Yet, simple counts can mislead if it is unclear what is counted and how.”
From Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019) · cites this paper as Lazer et al., 2014
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This unintelligibility is also subject to functional biases (section 4.1), as it depends on the mechanisms available on each social platform (e.g., having a like button, but not a dislike one), as well as to variations in platform algorithms and mechanisms in response to users actions (Lazer et al., 2014).”
From Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019) · cites this paper as Lazer et al., 2014
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“It also includes temporal validity, which captures to what extent constructs change over time (Howison et al., 2011) and may invalidate previous conclusions about societal and/or platform phenomena; e.g., see the case of Google Flu Trends (Lazer et al., 2014).”
From Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019) · cites this paper as Lazer et al., 2014
6 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 14 MARCH 2014 VOL 343 SCIENCE www.sciencemag.org 1204
“The 2009 version of GFT has weathered other media panics related to the fl u, including the 2005–2006 influenza A/H5N1 ("bird flu") outbreak”
“GFT engineers updated the algorithm in 2009, and this model has run ever since”
Citations
Cites
boyd, D., & Crawford, K. (2012) · Information, Communication & Society · doi:10.1080/1369118x.2012.678878 · Cited by 5,479
Twitter mood predicts the stock market
Bollen, J., Mao, H., & Zeng, X. (2011) · Journal of Computational Science · doi:10.1016/j.jocs.2010.12.007 · Cited by 5,206
Detecting influenza epidemics using search engine query data
Ginsberg, J., Mohebbi, M. H., Patel, R. S., Brammer, L., Smolinski, M. S., & Brilliant, L. (2009) · Nature · doi:10.1038/nature07634 · Cited by 4,295
Cohen, D. (1998) · The Women's Review of Books · doi:10.2307/4022859 · Cited by 4,261
Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk
Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012) · Political Analysis · doi:10.1093/pan/mpr057 · Cited by 4,008
Big Data : The Management Revolution
マカフィー, ア., ブリニョルフソン, エ., & 有賀, 裕. (2013) · Harvard business review · Cited by 3,549
Lazer, D., Pentland, A., Adamic, L., Aral, S., Barabási, A.-L., Brewer, D., Christakis, N., Contractor, N., Fowler, J., Gutmann, M., et al. (2009) · Science · doi:10.1126/science.1167742 · Cited by 3,365
Self-citationPredicting Elections with Twitter: What 140 Characters Reveal about Political Sentiment
Tumasjan, A., Sprenger, T., Sandner, P., & Welpe, I. (2010) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v4i1.14009 · Cited by 2,649
Multiscale mobility networks and the spatial spreading of infectious diseases
Balcan, D., Colizza, V., Gonçalves, B., Hu, H., Ramasco, J. J., & Vespignani, A. (2009) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0906910106 · Cited by 1,380
Self-citationPredicting consumer behavior with web search
Goel, S., Hofman, J. M., Lahaie, S., Pennock, D. M., & Watts, D. J. (2010) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1005962107 · Cited by 636
Research works printed but not linked: 4
- McAfee A., Brynjolfsson E., Harv. Bus. Rev. 90, 60 (2012).
- Copeland P., et al.., Int. Soc. Negl. Trop. Dis. 2013, 3 (2013).
- Mustafaraj E., Metaxas P., in Proceedings of the Web-Sci10, Raleigh, NC, 26 and 27 April 2010 (Web Science Trust, 2010); http://journal.webscience.org/317/.
- Ratkiewicz J., et al.., in Proceedings of 5th International AAAI Conference on Weblogs and Social Media, San Francisco, CA, 7 to 11 August 2011 (AAAI, 2011), p. 297–304.
Cited by
Items 101–200 of 2,513
On the Internet of Things, smart cities and the WHO Healthy Cities
Kamel Boulos, M. N., & Al-Shorbaji, N. M. (2014) · International Journal of Health Geographics · doi:10.1186/1476-072x-13-10 · Cited by 224
“But 'big data' methods can only be truly useful if they are paired with traditional forms of information collection, or what some researchers call 'small data' [56,57].”
BackgroundPassi, S., & Jackson, S. J. (2018) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3274405 · Cited by 225
“Several data correlations established in large datasets are neither reproducible nor falsifiable [45,61].”
BackgroundMachine learning in agricultural and applied economics
Storm, H., Baylis, K., & Heckelei, T. (2020) · European Review of Agricultural Economics · doi:10.1093/erae/jbz033 · Cited by 216
“The Google flu prediction is an example in this respect (Lazer et al., 2014).”
BackgroundMethodBig data and data science: a critical review of issues for educational research.
Daniel, B. K. (2019) · British Journal of Educational Technology · doi:10.1111/bjet.12595 · Cited by 220
Digital epidemiology: what is it, and where is it going?
Salathé, M. (2018) · Life Sciences, Society and Policy · doi:10.1186/s40504-017-0065-7 · Cited by 214
“The specific problems with GFT have been well described and discussed (Cook et al., 2011; Olson et al., 2013; Lazer et al., 2014), but the larger problem with GFT was the private ownership of the underlying data, which meant that the algorithm could not be reproduced and investigated independently.”
BackgroundBIG data – BIG gains? Understanding the link between big data analytics and innovation
Niebel, T., Rasel, F., & Viete, S. (2019) · Economics of Innovation and New Technology · doi:10.1080/10438599.2018.1493075 · Cited by 213
“One problem big data poses to decision-making is that correlations identi fi ed from the raw data are erroneously interpreted as causal relationships or that misleading patterns are found in the data (McAfee and Brynjolfsson 2012; Lazer et al. 2014).”
BackgroundCollective Emotions and Social Resilience in the Digital Traces After a Terrorist Attack
Garcia, D., & Rimé, B. (2019) · Psychological Science · doi:10.1177/0956797619831964 · Cited by 211
Sabottke, C. F., Suciu, O., & Dumitraş, T. (2015) · USENIX Security Symposium · Cited by 425
A Taxonomy of Ethical Tensions in Inferring Mental Health States from Social Media
Chancellor, S., Birnbaum, M. L., Caine, E. D., Silenzio, V. M. B., & De Choudhury, M. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287587 · Cited by 214
Using Twitter for Demographic and Social Science Research: Tools for Data Collection and Processing
McCormick, T. H., Lee, H., Cesare, N., Shojaie, A., & Spiro, E. S. (2017) · Sociological Methods & Research · doi:10.1177/0049124115605339 · Cited by 207
Hemingway, H., Asselbergs, F. W., Danesh, J., Dobson, R., Maniadakis, N., Maggioni, A., van Thiel, G. J. M., Cronin, M., Brobert, G., Vardas, P., et al. (2018) · European Heart Journal · doi:10.1093/eurheartj/ehx487 · Cited by 209
“…content in social media are inherently public and the language used in twitter can be used to predict community heart disease rates ( Table 2 , Eichstaedt et al. 23 ) and it is plausible that Google searches 24 might give clues to environmental pollution triggers of acute cardiovascular events.”
BackgroundData Science: A Comprehensive Overview
Cao, L. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.03606 · Cited by 210
“It can enable the discovery of indicators like Google Flu [Lazer et al. 2014] which may not be readily predicted by domain-driven hypothesis and professionals.”
BackgroundSelective responsiveness: Online public demands and government responsiveness in authoritarian China
Su, Z., & Meng, T. (2016) · Social Science Research · doi:10.1016/j.ssresearch.2016.04.017 · Cited by 202
“Just like other ways, the validity and reliability is not guaranteed in generating data using big data approach (Lazer et al., 2014; Manovich, 2012).”
BackgroundLeveraging Facebook's Advertising Platform to Monitor Stocks of Migrants
Zagheni, E., Weber, I., & Gummadi, K. (2017) · Population and Development Review · doi:10.1111/padr.12102 · Cited by 199
“These approaches also stimulated reflection about the problem of relying solely on non-representative sources, when in fact the combination of traditional and new data, within a unified framework, would yield more accurate estimates and predictions (Lazer et al. 2014).”
BackgroundBig Data Methods: Leveraging Modern Data Analytic Techniques to Build Organizational Science
Tonidandel, S., King, E. B., & Cortina, J. M. (2018) · Organizational Research Methods · doi:10.1177/1094428116677299 · Cited by 199
Big Data’s Role in Precision Public Health
Dolley, S. (2018) · Frontiers in Public Health · doi:10.3389/fpubh.2018.00068 · Cited by 203
“One early example of using big data for public health prediction, Google Flu Trends, was a well-publicized failure (99).”
BackgroundPINNeik: Eikonal solution using physics-informed neural networks
Waheed, U. B., Haghighat, E., Alkhalifah, T., Song, C., & Hao, Q. (2021) · Computers & Geosciences · doi:10.1016/j.cageo.2021.104833 · Cited by 199
“Despite its success in the initial years that were used to train the model, it soon started overestimating by several factors to the point that it was eventually taken down [31].”
BackgroundBig data hurdles in precision medicine and precision public health
Prosperi, M., Min, J. S., Bian, J., & Modave, F. (2018) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-018-0719-2 · Cited by 200
“However, research using non-traditional health-related data from these domains have been conducted with some success as well as with some controversy [46, 47].”
BackgroundKwan, M.-P. (2016) · Annals of the American Association of Geographers · doi:10.1080/00045608.2015.1117937 · Cited by 195
Zhang, X., Pérez-Stable, E. J., Bourne, P. E., Peprah, E., Duru, O. K., Breen, N., Berrigan, D., Wood, F., Jackson, J. S., Wong, D. W. S., et al. (2017) · Ethnicity & Disease · doi:10.18865/ed.27.2.95 · Cited by 195
Artificial Communication: How Algorithms Produce Social Intelligence
Esposito, E. (2022) · The MIT Press eBooks · doi:10.7551/mitpress/14189.001.0001 · Cited by 195
Using data-driven agent-based models for forecasting emerging infectious diseases
Venkatramanan, S., Lewis, B., Chen, J., Higdon, D., Vullikanti, A., & Marathe, M. (2018) · Epidemics · doi:10.1016/j.epidem.2017.02.010 · Cited by 189
“However, as [5] points out, such big data approaches often fail to account for the actual disease and social dynamics on the ground.”
BackgroundA bibliometric analysis of research on Big Data analytics for business and management
Ardito, L., Scuotto, V., Del Giudice, M., & Petruzzelli, A. M. (2019) · Management Decision · doi:10.1108/md-07-2018-0754 · Cited by 192
Sustainable Supply Chains in the Age of AI and Digitization: Research Challenges and Opportunities
Sanders, N. R., Boone, T., Ganeshan, R., & Wood, J. D. (2019) · Journal of Business Logistics · doi:10.1111/jbl.12224 · Cited by 193
Big Data in the Public Sector: Lessons for Practitioners and Scholars
Desouza, K. C., & Jacob, B. (2017) · Administration & Society · doi:10.1177/0095399714555751 · Cited by 188
Social Media Analyses for Social Measurement
Schober, M. F., Pasek, J., Guggenheim, L., Lampe, C., & Conrad, F. G. (2016) · Public Opinion Quarterly · doi:10.1093/poq/nfv048 · Cited by 187
The Dark Side of Information Proliferation
Hills, T. T. (2019) · Perspectives on Psychological Science · doi:10.1177/1745691618803647 · Cited by 183
Prediction of Number of Cases of 2019 Novel Coronavirus (COVID-19) Using Social Media Search Index
Qin, L., Sun, Q., Wang, Y., Wu, K.-F., Chen, M., Shia, B.-C., & Wu, S.-Y. (2020) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph17072365 · Cited by 188
Big Data and Biomedical Informatics: A Challenging Opportunity
Bellazzi, R. (2014) · Yearbook of Medical Informatics · doi:10.15265/iy-2014-0024 · Cited by 184
What Can Digital Disease Detection Learn from (an External Revision to) Google Flu Trends?
Santillana, M., Zhang, D. W., Althouse, B. M., & Ayers, J. W. (2014) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2014.05.020 · Cited by 183
“Moreover, by making the inner workings of GFT and the data behind GFT more public, such improvements may be more quickly realized by other external teams.(19,20)”
BackgroundDemographic Inference and Representative Population Estimates from Multilingual Social Media Data
Wang, Z., Hale, S. A., Adelani, D. I., Grabowicz, P. A., Hartman, T., Flöck, F., & Jurgens, D. (2019) · Publications of the UdS (Saarland University) · doi:10.22028/d291-29212 · Cited by 173
Information leverage in interconnected ecosystems: Overcoming the curse of dimensionality
Ye, H., & Sugihara, G. (2016) · Science · doi:10.1126/science.aag0863 · Cited by 179
Artificial Intelligence and Democracy: A Conceptual Framework
Jungherr, A. (2023) · Social Media + Society · doi:10.1177/20563051231186353 · Cited by 180
“This points to two problems: unobserved temporal shifts between variables (Lazer et al., 2014) and the dangers of relying on purely correlative evidence without the support of causal models (Pearl, 2019; Schölkopf et al., 2021).”
BackgroundThree maps and three misunderstandings: A digital mapping of climate diplomacy
Venturini, T., Baya Laffite, N., Cointet, J.-P., Gray, I., Zabban, V., & De Pryck, K. (2014) · Big Data & Society · doi:10.1177/2053951714543804 · Cited by 178
Kim, Y., Huang, J., & Emery, S. (2016) · Journal of Medical Internet Research · doi:10.2196/jmir.4738 · Cited by 176
Visualization analysis of big data research based on Citespace
Wang, W., & Lu, C. (2020) · Soft Computing · doi:10.1007/s00500-019-04384-7 · Cited by 178
Rahman, M. S., Peeri, N. C., Shrestha, N., Zaki, R., Haque, U., & Hamid, S. H. A. (2020) · Health Policy and Technology · doi:10.1016/j.hlpt.2020.04.005 · Cited by 178
A Big Data-as-a-Service Framework: State-of-the-Art and Perspectives
Wang, X., Yang, L. T., Liu, H., & Deen, M. J. (2018) · IEEE Transactions on Big Data · doi:10.1109/tbdata.2017.2757942 · Cited by 178
“We have moved from collecting small data or samples to collecting huge data [2]; from having clean data to structured and unstructured or messy data [3]; and from causations to associations or correlations [4].”
BackgroundKogan, N. E., Clemente, L., Liautaud, P., Kaashoek, J., Link, N. B., Nguyen, A. T., Lu, F. S., Huybers, P., Resch, B., Havas, C., et al. (2021) · Science Advances · doi:10.1126/sciadv.abd6989 · Cited by 172
Self-citationEvaluating and characterizing urban vibrancy using spatial big data: Shanghai as a case study
Huang, B., Zhou, Y., Li, Z., Song, Y., Cai, J., & Tu, W. (2020) · Environment and Planning B: Urban Analytics and City Science · doi:10.1177/2399808319828730 · Cited by 173
“The use of spatial big data sources offers enormous possibilities for understanding urban dynamics with large sample sizes and rich spatial and temporal details, and for detecting complex and non-linear variables (Lazar, 2014).”
BackgroundBig Data in Survey Research: AAPOR Task Force Report
Japec, L., Kreuter, F., Berg, M., Biemer, P., Decker, P., Lampe, C., Lane, J., O’Neil, C., & Usher, A. (2015) · Public Opinion Quarterly · doi:10.1093/poq/nfv039 · Cited by 174
Chancellor, S., Baumer, E. P. S., & De Choudhury, M. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359249 · Cited by 172
“In sum, we strongly advocate for HCML researchers to consider these approaches, in part to avoid the risks of “big data hubris” [112], as well as to do right by the communities and individuals our work claims to help.”
Backgroundfor the Influenza Forecasting Contest Working Group, Biggerstaff, M., Alper, D., Dredze, M., Fox, S., Fung, I. C.-H., Hickmann, K. S., Lewis, B., Rosenfeld, R., Shaman, J., et al. (2016) · BMC Infectious Diseases · doi:10.1186/s12879-016-1669-x · Cited by 170
Self-citationBig Data and the brave new world of social media research
Schroeder, R. (2014) · Big Data & Society · doi:10.1177/2053951714563194 · Cited by 170
“As Lazer et al. (2014) point out, there are a number methodological problems with these studies, but a major problem is simply that the data cannot be replicated since it is unclear how the data was arrived at (see also Borgman, forthcoming).”
BackgroundCommentary: Epidemiology in the Era of Big Data
Mooney, S. J., Westreich, D. J., & El-Sayed, A. M. (2015) · Epidemiology · doi:10.1097/ede.0000000000000274 · Cited by 170
An empirical study of the rise of big data in business scholarship
Frizzo-Barker, J., Chow-White, P. A., Mozafari, M., & Ha, D. (2016) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2016.01.006 · Cited by 168
Inference of seasonal and pandemic influenza transmission dynamics
Yang, W., Lipsitch, M., & Shaman, J. (2015) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1415012112 · Cited by 159
Implications of the Data Revolution for Statistics Education
Ridgway, J. (2016) · International Statistical Review · doi:10.1111/insr.12110 · Cited by 166
“Goel et al. (2010) showed that the predictions from GFT were little better than predictions from simple-lagged CDC data; Lazer et al. (2014) showed that lagged models of CDC data gave better predictions than GFT and that a combination of the two sets of data gave better predictions still.”
ResultAdvances in Artificial Intelligence for Infectious-Disease Surveillance
Brownstein, J. S., Rader, B., Astley, C. M., & Tian, H. (2023) · New England Journal of Medicine · doi:10.1056/nejmra2119215 · Cited by 167
Tracking COVID-19 using online search
Lampos, V., Majumder, M. S., Yom-Tov, E., Edelstein, M., Moura, S., Hamada, Y., Rangaka, M. X., McKendry, R. A., & Cox, I. J. (2021) · npj Digital Medicine · doi:10.1038/s41746-021-00384-w · Cited by 164
Artificial Intelligence for COVID-19: Rapid Review
(2020) · doi:10.2196/21476 · Cited by 160
Retooling Politics: How Digital Media Are Shaping Democracy
Jungherr, A., Rivero, G., & Gayo-Avello, D. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108297820 · Cited by 161
Gozzi, N., Tizzani, M., Starnini, M., Ciulla, F., Paolotti, D., Panisson, A., & Perra, N. (2020) · Journal of Medical Internet Research · doi:10.2196/21597 · Cited by 160
“On the other hand, despite some important limitations [27], modern media has become a key data source to observe and monitor health.”
BackgroundBig data in IBD: a look into the future
Olivera, P., Danese, S., Jay, N., Natoli, G., & Peyrin-Biroulet, L. (2019) · Nature Reviews Gastroenterology & Hepatology · doi:10.1038/s41575-019-0102-5 · Cited by 151
Speaking Sociologically with Big Data: Symphonic Social Science and the Future for Big Data Research
Halford, S., & Savage, M. (2017) · Sociology · doi:10.1177/0038038517698639 · Cited by 211
Nagar, R., Yuan, Q., Freifeld, C. C., Santillana, M., Nojima, A., Chunara, R., & Brownstein, J. S. (2014) · Journal of Medical Internet Research · doi:10.2196/jmir.3416 · Cited by 154
“Though providing impetus for increased vigilance during one of the worst flu seasons to date, this case demonstrates that Google Flu Trends alone has limitations for influenza prediction, and motivates search for improved models [17,18].”
Information and reformation in KM systems: big data and strategic decision-making
Intezari, A., & Gressel, S. (2017) · Journal of Knowledge Management · doi:10.1108/jkm-07-2015-0293 · Cited by 150
A tale of two epidemics: Contextual Word2Vec for classifying twitter streams during outbreaks
Khatua, A., Khatua, A., & Cambria, E. (2018) · Information Processing & Management · doi:10.1016/j.ipm.2018.10.010 · Cited by 165
Making sense of Big Data – can it transform operations management?
Matthias, O., Fouweather, I., Gregory, I., & Vernon, A. (2017) · International Journal of Operations & Production Management · doi:10.1108/ijopm-02-2015-0084 · Cited by 147
“The Google Flu Trends Project (GFT) reveals just how such claims can prove disappointing (Lazer et al. 2014).”
Kirchner, T. R., & Shiffman, S. (2016) · Social Psychiatry and Psychiatric Epidemiology · doi:10.1007/s00127-016-1277-5 · Cited by 147
“This important work is establishing precise metrics for the study human mobility, although it suffers in some ways from an aggregated, macro perspective some have used to question its societal impact [60, 61].”
BackgroundGoldberg, S. B., Flemotomos, N., Martinez, V. R., Tanana, M. J., Kuo, P. B., Pace, B. T., Villatte, J. L., Georgiou, P. G., Van Epps, J., Imel, Z. E., et al. (2020) · Journal of Counseling Psychology · doi:10.1037/cou0000382 · Cited by 141
What types of COVID-19 conspiracies are populated by Twitter bots?
Ferrara, E. (2020) · First Monday · doi:10.5210/fm.v25i6.10633 · Cited by 223
“ve a digital trail associated with the data center used to dispatch tweets. RQ2: CHARACTERIZING BOT FUELED CAMPAIGNS Data-driven computational sensemaking can fail when using human communication data [66]. Common reasons include population and sampling bias, platform-design bias, distortion of human and nonhuman behav…”
Inductive reasoning: The promise of big data
McAbee, S. T., Landis, R. S., & Burke, M. I. (2016) · Human Resource Management Review · doi:10.1016/j.hrmr.2016.08.005 · Cited by 195
Going Digital: A Survey on Digitalization and Large-Scale Data Analytics in Healthcare
Tresp, V., Marc Overhage, J., Bundschus, M., Rabizadeh, S., Fasching, P. A., & Yu, S. (2016) · Proceedings of the IEEE · doi:10.1109/jproc.2016.2615052 · Cited by 149
“Google flu for example, predicted well initially but the fit was very poor later [73], [74].”
BackgroundInfectious Disease Surveillance in the Big Data Era: Towards Faster and Locally Relevant Systems
Simonsen, L., Gog, J. R., Olson, D., & Viboud, C. (2016) · Journal of Infectious Diseases · doi:10.1093/infdis/jiw376 · Cited by 146
Exploring Limits to Prediction in Complex Social Systems
Martin, T., Hofman, J. M., Sharma, A., Anderson, A., & Watts, D. J. (2016) · Proceedings of the 25th International Conference on World Wide Web · doi:10.1145/2872427.2883001 · Cited by 136
The value of Big Data in government: The case of ‘smart cities’
Löfgren, K., & Webster, C. W. R. (2020) · Big Data & Society · doi:10.1177/2053951720912775 · Cited by 198
“The result of the Google Flu Trends analysis, as described in the seminal piece by Lazer et al. (2014), is a good example of this issue and shows how Google’s attempt to detect and predict flu outbreaks (in the US) eventually failed and primarily predicted the arrival of winter.”
Advances in nowcasting influenza-like illness rates using search query logs
Lampos, V., Miller, A. C., Crossan, S., & Stefansen, C. (2015) · Scientific Reports · doi:10.1038/srep12760 · Cited by 143
“Recent works indicated that this basic model mispredicted ILI in several flu seasons, with significant errors happening during 2012–13 19,20 .”
BackgroundMethodInfluentialWilliams, M. L., Burnap, P., & Sloan, L. (2016) · British Journal of Criminology · doi:10.1093/bjc/azw031 · Cited by 142
“Yet high-profile failures of big data, such as the inability to predict the US housing bubble in 2008 and the spread of influenza across the United States using Google search terms, have resulted in many questioning the power of these new forms of data (Lazer et al. 2014).”
BackgroundConceptualising the right to data protection in an era of Big Data
McDermott, Y. (2017) · Big Data & Society · doi:10.1177/2053951716686994 · Cited by 141
“We can point to a number of examples, including Google’s unsuccessful attempts at health diagnostics (Lazer et al., 2014) and, most recently, the use of analytics to predict the US election results (Carter, 2016), to question the utility”
BackgroundDigital data and management accounting: why we need to rethink research methods
Bhimani, A. (2020) · Journal of Management Control · doi:10.1007/s00187-020-00295-z · Cited by 284
Daw, A., Thomas, R. Q., Carey, C. C., Read, J. S., Appling, A., & Karpatne, A. (2020) · Society for Industrial and Applied Mathematics eBooks · doi:10.1137/1.9781611976236.60 · Cited by 213
“Despite dramatic advances in many commercial fields, current standards of deep learning has seen limited success in scientific applications (e.g., [3, 18, 22]), sometimes even leading to spectacular failures (e.g., [18]).”
BackgroundPromises and Pitfalls of Using Digital Traces for Demographic Research
Cesare, N., Lee, H., McCormick, T., Spiro, E., & Zagheni, E. (2018) · Demography · doi:10.1007/s13524-018-0715-2 · Cited by 137
“Following the burst of initial research advocating the possibilities of using digital trace data to study social phenomena, a recent body of literature has emerged outlining ways in which these data have been misused, cautioning researchers about the unique challenges that they present (Adams and Brueckner 2015; boyd…”
News and narratives in financial systems: Exploiting big data for systemic risk assessment
Nyman, R., Kapadia, S., & Tuckett, D. (2021) · Journal of Economic Dynamics and Control · doi:10.1016/j.jedc.2021.104119 · Cited by 167
Tampering with Twitter’s Sample API
Pfeffer, J., Mayer, K., & Morstatter, F. (2018) · EPJ Data Science · doi:10.1140/epjds/s13688-018-0178-0 · Cited by 135
Ayers, J. W., Leas, E. C., Allem, J.-P., Benton, A., Dredze, M. H., Althouse, B. M., Cruz, T. B., & Unger, J. B. (2017) · PLoS ONE · doi:10.1371/journal.pone.0170702 · Cited by 145
Meaningful measures of human society in the twenty-first century
Lazer, D. M. J., Hargittai, E., Freelon, D., González‐Bailón, S., Munger, K., Ognyanova, K., & Radford, J. P. (2021) · Nature · doi:10.1038/s41586-021-03660-7 · Cited by 180
Self-citationJohansson, M. A., Reich, N. G., Hota, A., Brownstein, J. S., & Santillana, M. (2016) · Scientific Reports · doi:10.1038/srep33707 · Cited by 120
Predicting and Interpolating State‐Level Polls Using Twitter Textual Data
Beauchamp, N. (2017) · American Journal of Political Science · doi:10.1111/ajps.12274 · Cited by 132
“2012), but they are generally plagued by small N, ad hoc parameter settings, and presumably high selection bias (Lazer et al. 2014) [S, B, T, O].”
BackgroundThe INTERSPEECH 2017 Computational Paralinguistics Challenge: Addressee, Cold & Snoring
Schuller, B., Steidl, S., Batliner, A., Bergelson, E., Krajewski, J., Janott, C., Amatuni, A., Casillas, M., Seidl, A., Soderstrom, M., et al. (2017) · Interspeech 2017 · doi:10.21437/interspeech.2017-43 · Cited by 134
“This can enable the monitoring of call-centre and other telephone interactions in order to better predict propagation of a cold [4, 5].”
BackgroundImran, R., Alraja, M. N., & Khashab, B. M. (2021) · IEEE Transactions on Engineering Management · doi:10.1109/tem.2021.3114256 · Cited by 172
“used analytical tools was not enough capable to overcome the causal misconceptions resulting in correlation for causation [156]–[158].”
MethodSuppressing disease spreading by using information diffusion on multiplex networks
Wang, W., Liu, Q.-H., Cai, S., Tang, M., Braunstein, L. A., & Stanley, E. (2016) · Scientific Reports · doi:10.1038/srep29259 · Cited by 162
“Figure 1(a) shows the real-data time series of information n G ( t ) and disease n D ( t ) indicating that macroscopically the two systems exhibit similar trends and confirming that the GFT effectively predicts disease spreading [26, 27]—although some researchers have expressed skepticism [28].”
ResultWang, H., Xu, Z., & Pedrycz, W. (2016) · Knowledge-Based Systems · doi:10.1016/j.knosys.2016.11.008 · Cited by 150
Ethical perspectives on recommending digital technology for patients with mental illness
Bauer, M., Glenn, T., Monteith, S., Bauer, R., Whybrow, P. C., & Geddes, J. (2017) · International Journal of Bipolar Disorders · doi:10.1186/s40345-017-0073-9 · Cited by 130
“As shown with Google Flu Trends (flu tracking algorithm), the published results could not be replicated with publically available information (Lazer et al. 2014).”
ResultFung, I. C., Fu, K., Chan, C., Chan, B. S. B., Cheung, C.-N., Abraham, T., & Tse, Z. T. H. (2016) · Public Health Reports · doi:10.1177/003335491613100312 · Cited by 149
Studying the Digital: Directions and Challenges for Digital Methods
Hampton, K. N. (2017) · Annual Review of Sociology · doi:10.1146/annurev-soc-060116-053505 · Cited by 127
Corbett, C. J. (2018) · Production and Operations Management · doi:10.1111/poms.12837 · Cited by 128
“This danger of the mis-use of big data is well examined in Lazer et al. (2014).”
BackgroundState of the art review: the data revolution in critical care
Ghassemi, M., Celi, L. A., & Stone, D. J. (2015) · Critical Care · doi:10.1186/s13054-015-0801-4 · Cited by 128
Characterizing Dietary Choices, Nutrition, and Language in Food Deserts via Social Media
Choudhury, M. D., Sharma, S. S., & Kıcıman, E. (2016) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/2818048.2819956 · Cited by 130
Jones, J. J., Bond, R. M., Bakshy, E., Eckles, D., & Fowler, J. H. (2017) · PLOS ONE · doi:10.1371/journal.pone.0173851 · Cited by 127
Big Data-enabled Customer Relationship Management: A holistic approach
Zerbino, P., Aloini, D., Dulmin, R., & Mininno, V. (2018) · Information Processing & Management · doi:10.1016/j.ipm.2017.10.005 · Cited by 162
“…acknowledgement of this relationship, one reason of the lack of clarity could be that " most big data that received popular attention are not the output of instruments designed to produce valid and reliable data amenable for scientific analysis " (Lazer, Kennedy, King, & Vespignani, 2014, p. 1204).”
Tavoschi, L., Quattrone, F., D’Andrea, E., Ducange, P., Vabanesi, M., Marcelloni, F., & Lopalco, P. L. (2020) · Human Vaccines & Immunotherapeutics · doi:10.1080/21645515.2020.1714311 · Cited by 148
Predicting lake surface water phosphorus dynamics using process-guided machine learning
Hanson, P. C., Stillman, A. B., Jia, X., Karpatne, A., Dugan, H. A., Carey, C. C., Stachelek, J., Ward, N. K., Zhang, Y., Read, J. S., et al. (2020) · Ecological Modelling · doi:10.1016/j.ecolmodel.2020.109136 · Cited by 136
Shen, C., Chen, A., Luo, C., Zhang, J., Feng, B., & Liao, W. (2020) · Journal of Medical Internet Research · doi:10.2196/19421 · Cited by 127
Zhang, Y. E., Xu, S., Zhang, L., & Yang, M. (2021) · Journal of Business Research · doi:10.1016/j.jbusres.2021.04.019 · Cited by 178
Predictive analytics in mental health: applications, guidelines, challenges and perspectives
Hahn, T., Nierenberg, A. A., & Whitfield, S. (2016) · Molecular Psychiatry · doi:10.1038/mp.2016.201 · Cited by 150
A review of influenza detection and prediction through social networking sites
Alessa, A., & Faezipour, M. (2018) · Theoretical Biology and Medical Modelling · doi:10.1186/s12976-017-0074-5 · Cited by 127
O'Brien, D. T., Sampson, R. J., & Winship, C. (2015) · Sociological Methodology · doi:10.1177/0081175015576601 · Cited by 155
“Big data hubris is indeed a problem (Lazer et al. 2014).”
BackgroundInfluentialBig data stream computing in healthcare real-time analytics
Ta, V.-D., Liu, C.-M., & Nkabinde, G. W. (2016) · International Conference on Cloud Computing and Big Data Analytics (ICCCBDA) · doi:10.1109/icccbda.2016.7529531 · Cited by 125
“In [8] Google Flu Trend based on big data analytics collected from many laboratory tests enabled the prediction of flu spread trend.”
MethodClinical epidemiology in the era of big data: new opportunities, familiar challenges
Ehrenstein, V., Nielsen, H., Pedersen, A. B., Johnsen, S. P., & Pedersen, L. (2017) · Clinical Epidemiology · doi:10.2147/clep.s129779 · Cited by 122
Metadata
- Authors
- Lazer, D., Kennedy, R., King, G., & Vespignani, A.
- Affiliations
- Northeastern University (US)
- Harvard University (US)
- Corresponding author
- Lazer, David
- Venue
- Science
- Published
- 2014-03-14
- Volume
- 343
- Issue
- 6176
- Pages
- 1203-1205
- ISSN
- 0036-8075 (ISSN-L)
- 1095-9203
- OpenAlex
- W2068181924
- Semantic Scholar
- 00d681cfff7f0224ace607e515f4ed510e792df9
- Cited by
- 2,523
- References
- 30
- Citations by year
- 2026: 113 · 2025: 101 · 2024: 121 · 2023: 150 · 2022: 168 · 2021: 229 · 2020: 273 · 2019: 261 · 2018: 272 · 2017: 249 · 2016: 271 · 2015: 219 · 2014: 91
- FWCI
- 208.16
- Type
- journal article
- Language
- en
- Open access
- green
- dash.harvard.edu
- Licence
- cc-by
- Locations
- Science (publishedVersion)
- PubMed (publishedVersion)
- Digital Access to Scholarship at Harvard (DASH) (Harvard University) (submittedVersion, cc-by)
- Topics
- Health Sciences › Medicine › Epidemiology › Data-Driven Disease Surveillance
- Social Sciences › Decision Sciences › Information Systems and Management › Big Data Technologies and Applications
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- Google Flu Trends, influenza surveillance, big data, prediction error, data quality
- MeSH terms
- Centers for Disease Control and Prevention, U.S., Data Collection, Data Interpretation, Statistical, Disease Outbreaks, Humans, Influenza, Human, Office Visits, Prognosis, United States, Social Media
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.