The Parable of Google Flu: Traps in Big Data Analysis
Open accessThe DOI matches its Crossref record; 30 of the 31 references it cites were found in a registry or an index; 5 of the 8 sources its claims cite were checked for support.
Abstract
Large errors in flu prediction were largely avoidable, which offers lessons for the use of big data.
Overview
CoChat Summary
TL;DR
The paper uses the failure of Google Flu Trends (GFT) to illustrate two traps in big data analysis: “big data hubris” and algorithm dynamics. It argues that GFT’s large flu-prediction errors were largely avoidable and that big data should supplement, not replace, traditional data and methods.
Key findings
- In February 2013, GFT predicted more than double the proportion of doctor visits for influenza-like illness (ILI) than the CDC; from 21 August 2011 to 1 September 2013, GFT reported overly high flu prevalence in 100 out of 108 weeks, and it overshot the 2011–2012 season by more than 50%.
- GFT errors were not random: they showed temporal autocorrelation and seasonality, meaning last week’s errors predicted this week’s errors and error direction/magnitude varied with time of year.
- Simple alternatives outperformed GFT: out-of-sample mean absolute error was 0.486 for GFT, 0.311 for lagged CDC, and 0.232 for combined GFT and CDC; all differences were statistically significant at P < 0.05. Even 3-week-old CDC data did better than GFT at projecting current flu prevalence.
- The authors attribute GFT’s problems to overfitting (initial GFT matched 50 million search terms to 1152 data points and missed the nonseasonal 2009 H1N1 pandemic) and to “algorithm dynamics,” including Google search algorithm changes and user behavior changes.
Methods
- Design: policy forum/case study comparing GFT estimates with CDC ILI surveillance data and alternative forecasting models.
- Data/sample: GFT and CDC flu prevalence estimates, including the period from 21 August 2011 to 1 September 2013; the initial GFT methodology reportedly fit 50 million search terms to 1152 data points.
- Main analysis: comparison of GFT alone, lagged CDC data with 52-week seasonality variables, and a combined GFT + CDC model using lagged CDC estimates, lagged GFT error, and seasonality variables; reported mean absolute error and statistical significance; Google Correlate used to examine correlated search terms. Supplementary materials contain data and methods.
Limitations
- The paper states that GFT never documented the 45 search terms used, released examples appear misleading, and Google Correlate is limited to national-level data whereas GFT was developed using regional correlations; therefore the original GFT analyses cannot be replicated from the information provided.
- The paper states that Google cannot make its full data available to outsiders because of privacy issues, though it argues there is no such constraint on derivative, aggregated data.
- Evident from the text: this is a commentary/policy forum rather than a full empirical article, with key data and methods placed in supplementary materials; the quantitative comparison depends on GFT and CDC data and model specifications not fully detailed in the main text.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
2 Valid · 2 Not valid · 2 Not checked · 8 Own results and interpretations
Some cited sources support this claim and some don’t — read each source’s verdict below.
In the paper
“This ad hoc method of throwing out peculiar search terms failed when GFT completely missed the nonseasonal 2009 infl uenza A–H1N1 pandemic ( 2, 14).”
What the sources say
- Partly supportedcites [2] matched to “Reassessing Google Flu Trends Data for Detection of Seasonal and Pandemic Influenza: A Comparative Epidemiological Study at Three Geographic Scales” (2013) sourceExperimental — accuracy not yet measured
The Discussion (page unavailable) says the original GFT model missed the spring 2009 pandemic wave, supporting the claim about its failure to detect the nonseasonal outbreak. It does not say GFT completely missed the entire 2009 pandemic. The Introduction describes automated selection of the original model’s search terms, not an ad hoc method of throwing out peculiar terms. (LLM, conf=0.86)
- Disputedcites [14] S. Cook et al., PLOS ONE 6, e23610 (2011). sourceExperimental — accuracy not yet measured
The Abstract (Methodology/Principal Findings) says the original GFT model underestimated the magnitude of illness during the 2009 H1N1 pandemic, not that it completely missed it. The Abstract (Conclusions) says both models performed well during the pandemic, although the updated model performed better. The Discussion notes that pandemic-specific queries were excluded, but does not describe this as an ad hoc method of throwing out peculiar terms. (LLM, conf=0.88)
Checked by a reasoning model — may be wrong.
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“Oddly, the few search terms offered in the papers ( 14) do not seem to be strongly related with either GFT or the CDC data (SM)—we surmise that the authors felt an unarticulated need to cloak the actual search terms identifi ed.”
What the sources say
- Disputedcites [14] S. Cook et al., PLOS ONE 6, e23610 (2011). sourceExperimental — accuracy not yet measured
The Results section (‘Search Behavior During pH1N1’; page unavailable) says that all three search queries shown in Figure 3 closely tracked CDC ILINet data before the pandemic, and ‘symptoms of flu’ continued to do so during it. The paper does report that the bronchitis and pneumonia queries underestimated ILINet data during the pandemic, but that does not support the broad claim that the offered terms were not strongly related to the data. (LLM, conf=0.87)
Checked by a reasoning model — may be wrong.
hedgedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“The Internet has opened the way for improving standard surveys, experiments, and health reporting ( 35).”
What the sources say
- Partly supportedcites [35] A. J. Berinsky et al., Polit. Anal. 20, 351–368 (2012). sourceExperimental — accuracy not yet measured
Section 2 describes using an Internet platform to run surveys with embedded experiments, and the Conclusion says MTurk can make experimental research easier and less expensive. The paper does not address improvements to health reporting; the disease scenario in section 4.3.2 is a framing experiment, not health reporting. Page numbers are unavailable in our copy. (LLM, conf=0.88)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Although not widely reported until 2013, the new GFT has been persistently overestimating flu prevalence for a much longer time.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“From 21 August 2011 to 1 September 2013, GFT reported overly high fl u prevalence 100 out of 108 weeks.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
3 more claims
Create a free account to see all 14 claims and every source verdict.
- It may state two separate claims as one. 5
- Probably not a substantial claim. 2
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Google Trends predicted more than twice the number of influenza-like illnesses than the CDC in February 2013 46 .”
From Unrepresentative big surveys significantly overestimated US vaccine uptake (2021) · cites this paper as [46]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This was one of the main reasons behind the failure of Google Flu Trends, since the data used for training the model in the first few years was not representative of the trends in subsequent years [15].”
From Theory-Guided Data Science: A New Paradigm for Scientific Discovery from Data (2017) · cites this paper as [15]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Social data research often relies on counting entities. These entities can be users, links, or messages, and the description is a summary of these counts (Lazer et al., 2014; Salganik, 2017). Yet, simple counts can mislead if it is unclear what is counted and how.”
From Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019) · cites this paper as Lazer et al., 2014
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This unintelligibility is also subject to functional biases (section 4.1), as it depends on the mechanisms available on each social platform (e.g., having a like button, but not a dislike one), as well as to variations in platform algorithms and mechanisms in response to users actions (Lazer et al., 2014).”
From Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019) · cites this paper as Lazer et al., 2014
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“It also includes temporal validity, which captures to what extent constructs change over time (Howison et al., 2011) and may invalidate previous conclusions about societal and/or platform phenomena; e.g., see the case of Google Flu Trends (Lazer et al., 2014).”
From Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019) · cites this paper as Lazer et al., 2014
6 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 14 MARCH 2014 VOL 343 SCIENCE www.sciencemag.org 1204
“The 2009 version of GFT has weathered other media panics related to the fl u, including the 2005–2006 influenza A/H5N1 ("bird flu") outbreak”
“GFT engineers updated the algorithm in 2009, and this model has run ever since”
Citations
Cites
boyd, D., & Crawford, K. (2012) · Information, Communication & Society · doi:10.1080/1369118x.2012.678878 · Cited by 5,479
Twitter mood predicts the stock market
Bollen, J., Mao, H., & Zeng, X. (2011) · Journal of Computational Science · doi:10.1016/j.jocs.2010.12.007 · Cited by 5,206
Detecting influenza epidemics using search engine query data
Ginsberg, J., Mohebbi, M. H., Patel, R. S., Brammer, L., Smolinski, M. S., & Brilliant, L. (2009) · Nature · doi:10.1038/nature07634 · Cited by 4,295
Cohen, D. (1998) · The Women's Review of Books · doi:10.2307/4022859 · Cited by 4,261
Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk
Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012) · Political Analysis · doi:10.1093/pan/mpr057 · Cited by 4,008
Big Data : The Management Revolution
マカフィー, ア., ブリニョルフソン, エ., & 有賀, 裕. (2013) · Harvard business review · Cited by 3,549
Lazer, D., Pentland, A., Adamic, L., Aral, S., Barabási, A.-L., Brewer, D., Christakis, N., Contractor, N., Fowler, J., Gutmann, M., et al. (2009) · Science · doi:10.1126/science.1167742 · Cited by 3,365
Self-citationPredicting Elections with Twitter: What 140 Characters Reveal about Political Sentiment
Tumasjan, A., Sprenger, T., Sandner, P., & Welpe, I. (2010) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v4i1.14009 · Cited by 2,649
Multiscale mobility networks and the spatial spreading of infectious diseases
Balcan, D., Colizza, V., Gonçalves, B., Hu, H., Ramasco, J. J., & Vespignani, A. (2009) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0906910106 · Cited by 1,380
Self-citationPredicting consumer behavior with web search
Goel, S., Hofman, J. M., Lahaie, S., Pennock, D. M., & Watts, D. J. (2010) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1005962107 · Cited by 636
Research works printed but not linked: 7
- A. McAfee, E. Brynjolfsson, Harv. Bus. Rev. 90, 60 (2012).
- AAAI Conference on Weblogs and Social Media, Atlanta, Georgia, 11 to 15 July 2010 (Association for Advancement of Artifi cial Intelligence, 2010), pp. 178.
- P. Copeland et al., Int. Soc. Negl. Trop. Dis. 2013, 3
- I. M. Hall et al., Epidemiol. Infect. 135, 372 (2007).
- E. Mustafaraj, P. Metaxas, in Proceedings of the Web- Sci10, Raleigh, NC, 26 and 27 April 2010 (Web Science Trust, 2010); http://journal.webscience.org/317/.
- J. Ratkiewicz et al., in Proceedings of 5th International AAAI Conference on Weblogs and Social Media, San Francisco, CA, 7 to 11 August 2011 (AAAI, 2011), p. 297–304.
- 2013; http://chronicle.com/article/Researchers-Struggle- to-Secure/141591/.
Cited by
Items 1,401–1,500 of 2,513
Ethical Data Science: Prediction in the Public Interest
Washington, A. L. (2023) · doi:10.1093/oso/9780197693025.001.0001 · Cited by 12
Historical Roots of Dissemination and Implementation Science
Dearing, J. W., Kee, K. F., & Peng, T. (2023) · Oxford University Press eBooks · doi:10.1093/oso/9780197660690.003.0003 · Cited by 9
Measuring vaccination coverage and concerns of vaccine holdouts from web search logs
Chang, S. Y., Fourney, A., & Horvitz, E. (2024) · Nature Communications · doi:10.1038/s41467-024-50614-4 · Cited by 12
On the viability of pro-active automated PII breach detection: A South African case study
Swart, I., Irwin, B. V. W., & Grobler, M. M. (2014) · South African Institute of Computer Scientists and Information Technologists · doi:10.1145/2664591.2664600 · Cited by 6
“At least one researcher has shown that the reliance on third party data can lead to incorrect conclusions such as the case with the Google Flu Tracker [28].”
BackgroundSaito, S., Hirata, Y., Sasahara, K., & Suzuki, H. (2015) · PLoS ONE · doi:10.1371/journal.pone.0139085 · Cited by 8
Big data e news online: possibilità e limiti per la ricerca sociale
Giuffrida, G., Rinaldi, F. M., & Zarba, C. G. (2016) · SOCIOLOGIA E RICERCA SOCIALE · doi:10.3280/sr2016-109013 · Cited by 6
A declarative query processing system for nowcasting
Antenucci, D., Anderson, M. R., & Cafarella, M. (2016) · Proceedings of the VLDB Endowment · doi:10.14778/3021924.3021931 · Cited by 6
“While nowcasting has faced some criticism due to model failures [24] and the cherry-picking of results [25], we view nowcasting as a tool that can be used either effectively or maliciously—like statistics in general—and with proper domain knowledge and training, these issues can be avoided.”
BackgroundSarkar, B. K. (2017) · International Journal of Knowledge-Based Organizations · doi:10.4018/ijkbo.2017100104 · Cited by 6
Personicle: Personal Chronicle of Life Events
Jalali, L., Huo, D., Oh, H., Tang, M., Pongpaichet, S., & Jain, R. (2014) · Cited by 8
Managerial Controversies in Artificial Intelligence and Big Data Analytics
Strang, K. D., & Sun, Z. (2019) · Advances in data mining and database management book series · doi:10.4018/978-1-5225-7277-0.ch004 · Cited by 6
Big Data and Biodefense: Prospects and Pitfalls
Vogel, K. M. (2019) · Springer eBooks · doi:10.1007/978-3-030-03053-7_15 · Cited by 6
Introductory Chapter: Geographic Information Systems and Science
Viana, C. M., Abrantes, P. C. D. R. M., & Rocha, J. (2019) · IntechOpen eBooks · doi:10.5772/intechopen.86121 · Cited by 9
“To avoid being trapped in a kind of data dependency, we need to understand the processes (including measurement methods and technologies) that generate it [72].”
BackgroundStudying Psychology using Big Data
Kim, C. (2019) · 한국심리학회지: 일반 · doi:10.22257/kjp.2019.12.38.4.519 · Cited by 6
Big Data and Reality Mining in Healthcare: Promise and Potential
Asri, H., Mousannif, H., Moatassime, H. A., & Zahir, J. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-51935-3_13 · Cited by 8
A general method for estimating the prevalence of influenza-like-symptoms with Wikipedia data
Toni, G. D., Consonni, C., & Montresor, A. (2021) · PLoS ONE · doi:10.1371/journal.pone.0256858 · Cited by 8
“This issue was also the reason behind the failure of several other similar solutions, like Google Flu Trends [6, 9].”
ResultNew developments in central bank statistics around the world
Rosolia, A., Stapel-Weber, S., & Tissot, B. (2021) · Statistical Journal of the IAOS · doi:10.3233/sji-210887 · Cited by 6
“The failure of Google Flu Trends provides a good example of these perils, as it was initially intended to provide estimates of in-fluenza activity based on Google Search queries but was discontinued in the mid-2010s [20].”
The Influence of Big Data Analytics on E‐Commerce: Case Study of the U.S. and China
Zhuang, W. (2021) · Wireless Communications and Mobile Computing · doi:10.1155/2021/2888673 · Cited by 9
“However, big data hubris and algorithm dynamics issues may contribute to analysis mistakes [7] because of human subjective prejudice, technological objective limitations, and the need to enhance artificial intelligence by processing data more efficiently for e-commerce transactions.”
BackgroundA Novel Intelligent Hybrid Optimized Analytics and Streaming Engine for Medical Big Data
Thilagaraj, M., Dwarakanath, B., Pandimurugan, V., Naveen, P., S, H. M., Hariharasitaraman, S., Arunkumar, N., & Govindan, P. (2022) · Computational and Mathematical Methods in Medicine · doi:10.1155/2022/7120983 · Cited by 8
“The privileged massive data in healthcare utilized by big data streaming plays a vital role in analytics using many predictions and recommendation systems [7, 8].”
BackgroundRetractedEarly Experience of Land Readjustment in Hong Kong: Case Study of the Kowloon Walled City
Chua, M. H., & Yau, Y. (2022) · Buildings · doi:10.3390/buildings12060757 · Cited by 8
Machine learning in agricultural economics
Baylis, K., Heckelei, T., & Storm, H. (2021) · Handbook of agricultural economics · doi:10.1016/bs.hesagr.2021.10.007 · Cited by 12
Clinical Artificial Intelligence
McDermott, M. B. A., Nestor, B., & Szolovits, P. (2022) · Clinics in Laboratory Medicine · doi:10.1016/j.cll.2022.09.004 · Cited by 7
A Two‐Step Procedure to Measure Representativeness of Internet Data Sources
Beręsewicz, M. (2017) · International Statistical Review · doi:10.1111/insr.12217 · Cited by 7
“Internet data sources are present in the literature devoted to information systems and technology, e-commerce, sociology or political studies (cf. Ginsberg et al., 2008; Miller, 2011; Lazer et al., 2014; Couper, 2013).”
BackgroundVisual Analytics Solutions as ‘off-the-Shelf’ Libraries
Ventocilla, E., & Riveiro, M. (2017) · International Conference on Information Visualisation (IV) · doi:10.1109/iv.2017.77 · Cited by 6
“Analytic intelligence for actionable insight has also been associated with algorithmic transparency [28].”
BackgroundFinancial attention and the demand for information
Qadan, M., & Zoua’bi, M. (2019) · Journal of Behavioral and Experimental Economics · doi:10.1016/j.socec.2019.101450 · Cited by 6
How Cognitive Biases Influence the Data Verification of Safety Indicators: A Case Study in Rail
Burggraaf, J., Groeneweg, J., Sillem, S., & van Gelder, P. H. A. J. M. (2019) · Safety · doi:10.3390/safety5040069 · Cited by 8
“Additionally, the constant improvements in Google’s search algorithm has likely had an effect on the quality of the predictions [33].”
BackgroundModelling epidemics: Technical and critical issues in the context of COVID-19
Brunsdon, C. (2020) · Dialogues in Human Geography · doi:10.1177/2043820620934328 · Cited by 6
“This standpoint has many opponents – and for the current situation, a cautionary tale of the dangers of analysis without considering process and issues of data collection lies in the recent ‘Google Flu’ episode (Lazer et al., 2014).”
BackgroundForschungsdatenmanagement sozialwissenschaftlicher Umfragedaten
Jensen, U., Netscher, S., Weller, K., & Budrich, V. B. (2019) · Verlag Barbara Budrich eBooks · doi:10.3224/84742233 · Cited by 15
Nowcasting with Google Trends, the more is not always the better
Combes, S., & Bortoli, C. (2016) · International Conference on Advanced Research Methods and Analytics (CARMA) · doi:10.4995/carma2016.2016.4226 · Cited by 14
Bonenfant, M., & Meurs, M. (2019) · Springer eBooks · doi:10.1007/978-94-024-1555-1_39 · Cited by 12
Exploratory Models in a time of Big Data
Mazur, M., & Manley, E. (2016) · Interdisciplinary Science Reviews · doi:10.1080/03080188.2016.1257196 · Cited by 8
“Thus, theory should ensure that adequate parameters calibrate the exploratory model (Lazer et al .”
BackgroundInfluentialDemography, Demographers and the ‘Data Revolution’ in Africa
Moultrie, T. A. (2017) · Afrique contemporaine · doi:10.3917/afco.258.0025 · Cited by 6
Beaude, B., & Nova, N. (2016) · Réseaux · doi:10.3917/res.195.0053 · Cited by 17
On predictability of rare events leveraging social media: a machine learning perspectivePreprint
Le, L., Ferrara, E., & Flammini, A. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1502.05886 · Cited by 6
“edu/people/pabo/ movie-review-data/ predict exogenous events [11, 12]: in such cases, it is important to keep in mind that the usage of machine learning algorithms or statistical models that function as black boxes can yield to results which are not interpretable and misleading [15].”
BackgroundAdaptive Cities: A Cybernetic Perspective on Urban Systems.Preprint
Gershenson, C., Santi, P., & Ratti, C. (2016) · arXiv (Cornell University) · Cited by 9
Network Effects in Contagion Processes: Identification and ControlPreprint
Drakopoulos, K., & Zheng, F. (2017) · SSRN Electronic Journal · doi:10.2139/ssrn.3091313 · Cited by 5
“Although an early version of this tool produced data with low accuracy on rare occasions [48], multiple studies have found that the overall accuracy level of the data to be high [49, 50, 51].”
MethodThe Meanings of “Meaning” in Social Scientific Text Analysis
Monroe, B. L. (2019) · Sociological Methodology · doi:10.1177/0081175019865231 · Cited by 6
Mutual radicalisation of opposing extremist groups via the InternetPreprint
Gallacher, J., & Heerdink, M. W. (2021) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/dtfc5 · Cited by 8
Jony, R. I. (2023) · Queensland University of Technology · doi:10.5204/thesis.eprints.238058 · Cited by 9
How do Generative Language Models Answer Opinion Polls?Preprint
Boelaert, J., Coavoux, S., Ollion, É., Petev, I. D., & Präg, P. (2024) · SocArXiv (OSF Preprints) · doi:10.31235/osf.io/r2pnb · Cited by 15
Causal Inferences from Digital Behavioral Data: Methodological Implications
Leitgöb, H., & Keusch, F. (2026) · KZfSS Kölner Zeitschrift für Soziologie und Sozialpsychologie · doi:10.1007/s11577-026-01050-3 · Cited by 8
“A well-known illustration of this problem is Google Flu Trends (Lazer et al. 2014).”
Wang, X., Song, Y., & Su, Y. (2022) · Social Science Computer Review · doi:10.1177/08944393211058112 · Cited by 9
Automated image analysis for studying online behaviour
Schwemmer, C., Unger, S., & Heiberger, R. H. (2023) · Edward Elgar Publishing eBooks · doi:10.4337/9781789906769.00023 · Cited by 14
AI and BDA impact on Stakeholders' Responses to Education Technology Adoption
Elshamly, A., & Gameel, Z. A. A. (2023) · MIGRATION LETTERS · doi:10.59670/ml.v20i8.5738 · Cited by 9
“…2020; Cheung & Jak, 2018; Daniel, 2014, 2017; Harlow & Oswald, 2016; Huang et al., 2019; Kalgotra & Sharda, 2021; Klašnja-Milićević et al., 2017; Lazer et al., 2014; Luan et al., 2020; Lutfi et al., 2023; Lutfi, Alsyouf, et al., 2022; Renz & Hilbig, 2020; Sivarajah et al., 2017; Tsai et al.,…”
Google trends as an early indicator of African swine fever outbreaks in Southeast Asia
Hsu, C.-H., Yang, C.-H., & Perez, A. M. (2024) · Frontiers in Veterinary Science · doi:10.3389/fvets.2024.1425394 · Cited by 7
Internet-based surveillance to track trends in seasonal allergies across the United States
Stallard-Olivera, E., & Fierer, N. (2024) · PNAS Nexus · doi:10.1093/pnasnexus/pgae430 · Cited by 4
“While internet-based disease surveillance shows promise from a technical perspective, there have been problems with its accuracy and applicability, for example the widely publicized failure of Google Flu Trends to predict flu hospitalizations despite years of development (21).”
PoN: Open source solution for real-time data analysis
Venkatesan, N. J., Kim, E., & Shin, D. R. (2016) · doi:10.1109/dipdmwc.2016.7529409 · Cited by 5
“The authors et al. [5] explained how Google can predict the spread of any flu virus, specifically narrowing down even to particular regions and states.”
BackgroundCaine, K. E. (2016) · IEEE Pervasive Computing · doi:10.1109/mprv.2016.61 · Cited by 5
Difficulties of predicting the timing, size and severity of influenza seasons
Barr, I., & CHENG, A. C. (2018) · Respirology · doi:10.1111/resp.13310 · Cited by 5
Reconstructing the MERS disease outbreak from news
Balashankar, A., Dugar, A., Subramanian, L., & Fraiberger, S. P. (2019) · ACM SIGCAS Conference on Computing and Sustainable Societies (COMPASS) · doi:10.1145/3314344.3332498 · Cited by 6
“Other auxiliary data like internet search history [12] and the web [5] have been used for disease surveillance, but the limitations of a fully unsupervised system without validation can cause spurious correlations as noted in [16].”
BackgroundThe perils of working with big data, and a SMALL checklist you can use to recognize them
Brave, S. A., Butters, R. A., & Fogarty, M. S. (2021) · Business Horizons · doi:10.1016/j.bushor.2021.06.004 · Cited by 5
Fotopoulou, A., Barratt, H., & Marandet, E. (2021) · Health Promotion International · doi:10.1093/heapro/daab166 · Cited by 6
In.To. COVID-19 socio-epidemiological co-causality
Galbraith, E., Li, J., Vilas, V. D. R., & Convertino, M. (2022) · Scientific Reports · doi:10.1038/s41598-022-09656-1 · Cited by 6
“Keyword-based strategies, as we employed, are problematic 44 so it would be more accurate to describe our results as the value of the topic “misinformation” rather the value of specific misinforming messages.”
MethodBiased Bytes: On the Validity of Estimating Food Consumption from Digital Traces
Gligorić, K., Ðordevic, I., & West, R. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3555660 · Cited by 9
“For example, if one is measuring physical activity based on mobile phone location traces, how consequential is the omission of stationary activities such as treadmill or yoga [57]? If one is tracking influenza with Web search logs of symptoms, how consequential are searches from persons not experiencing any symptoms […”
BackgroundShao, N., & Hu, H. (2022) · Journal of Environmental and Public Health · doi:10.1155/2022/2288321 · Cited by 5
“The coming of age of big data has given education a great opportunity [1, 2].”
BackgroundRetractedDistributional Shift Adaptation using Domain-Specific Features
Tahir, A., Lu, C. Z., Guo, R., & Liu, H. (2022) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata55660.2022.10020444 · Cited by 5
“Thus, in scenarios involving distribution shift, ERM performance degrades significantly [14,15].”
BackgroundAssessing Spurious Correlations in Big Search Data
Richman, J. T., & Roberts, R. J. (2023) · Forecasting · doi:10.3390/forecast5010015 · Cited by 5
“Consistent ith previous theoretical argu ents [19], e find that the agnitude of the problem is very large.”
Ziakas, P. D., & Mylonakis, E. E. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000271 · Cited by 7
“In events of profound public interest, such as the Covid-19 pandemic, the study of web search behaviors could add to our understanding of the disease trajectory and serve as a prediction tool within the limits and reliability of digital epidemiology [10,11].”
BackgroundGoverning digital health for infectious disease outbreaks
Roberts, S. L., & Kelman, I. (2023) · Global Public Health · doi:10.1080/17441692.2023.2241894 · Cited by 8
“Despite this initial promise, the project faltered in its forecasting activities and following a series of huge overestima-tions in in fl uenza patterns in the coming years, eventually closed in 2015, itself a victim of what has been termed ‘ big data hubris ’ (Duclos, 2019; Lazer et al., 2014).”
Data Flourishing: Developing Human‐Centered Data Science through Communities of Ethical Practice
Poole, A. H. (2023) · Proceedings of the Association for Information Science and Technology · doi:10.1002/pra2.793 · Cited by 6
“swamped current social, political, and legal institutions; Lazer et al. (2014) warned of the conceit that Big Data supplants conventional data practices (e.g., validity and reliability).”
BackgroundA combination of annual and nonannual forces drive respiratory disease in the tropics
Yang, F., Servadio, J. L., Thanh, N. T. L., Lam, H. M., Choisy, M., Thái, P. Q., Thao, T. T. N., Vy, N. H. T., Phuong, H. T., Nguyen, T. D., et al. (2023) · BMJ Global Health · doi:10.1136/bmjgh-2023-013054 · Cited by 4
Artificial Intelligence and Public Health Context: What We Should Know?
Madli, F., Janin, Y., Gulabdin, S. B., Lada, S., Yun, W. S., Adis, A.-A. A., & Jafar, A. (2024) · Journal of Advanced Research in Applied Sciences and Engineering Technology · doi:10.37934/araset.39.2.96109 · Cited by 8
Big Data in der amtlichen Statistik
Zwick, M. (2015) · Bundesgesundheitsblatt - Gesundheitsforschung - Gesundheitsschutz · doi:10.1007/s00103-015-2188-4 · Cited by 5
Li, N., Zhao, C., Choe, E. K., et al. (2015) · doi:10.1145/2702613.2732804 · Cited by 5
“Object D is a button that allows users who declined GPS location service to report their location manually.”
BackgroundStrategic Treaty Management: Practice and Implications
McInerney, T. F. (2015) · Cited by 5
How to Engrain a Big Data Mindset into Our Managers' DNA
Carillo, K. (2015) · Advances in public policy and administration (APPA) book series · doi:10.4018/978-1-4666-9649-5.ch005 · Cited by 5
Lugmayr, A. R., Stockleben, B., & Scheib, C. (2016) · eSpace (Curtin University) · Cited by 5
Extracting signals from news streams for disease outbreak prediction
Chakraborty, S., & Subramanian, L. (2016) · IEEE Global Conference on Signal and Information Processing · doi:10.1109/globalsip.2016.7906051 · Cited by 6
“While that seemed to work initially, over the years such an approach failed to produce the same results as increased awareness of diseases led users to search without the occurrence of the disease, leading to overestimation of flu cases [13, 20].”
MethodResultBig Data, Small Sample: Edgeworth Expansions Provide a Cautionary Tale
Gerlovina, I., van der Laan, M. J., & Hubbard, A. (2017) · The International Journal of Biostatistics · doi:10.1515/ijb-2017-0012 · Cited by 7
Task assignment optimization in geographically distributed data centers
Zhang, B., & Hwang, J. (2017) · IFIP/IEEE International Symposium on Integrated Network Management (IM) · doi:10.23919/inm.2017.7987318 · Cited by 5
“The recent years have witnessed the explosive growth of data generated by various applications such as web search, health monitoring, and social networks [1, 2] .”
BackgroundTrends detection of flu based on ensemble models with emotional factors from social networks
Sun, X. N., Ren, F., & Ye, J. (2016) · IEEJ Transactions on Electrical and Electronic Engineering · doi:10.1002/tee.22389 · Cited by 5
Suicide rates and information seeking via search engines: A cross-national correlational approach
Arendt, F. (2017) · Death Studies · doi:10.1080/07481187.2017.1388305 · Cited by 5
“…query share volume data are not available, transparency issues related to Google’s full arsenal of data, lack of stability over time, lack of knowledge regarding the most adequate search terms, uncertainty of who searches for specific search terms, and why (Lazer et al., 2014; Tran et al., 2017).”
BackgroundWhen Algorithms go Wrong, Who is Liable?
Olhede, S. C., & Wolfe, P. J. (2017) · Significance · doi:10.1111/j.1740-9713.2017.01085.x · Cited by 7
e-Health, social media, and rheumatology: Can they get along?
Berenbaum, F. (2017) · Joint Bone Spine · doi:10.1016/j.jbspin.2017.12.007 · Cited by 6
Predicting Antimicrobial Drug Consumption using Web Search Data
Hansen, N. D., Mølbak, K., Cox, I. J., & Lioma, C. (2018) · International Conference on Digital Health · doi:10.1145/3194658.3194667 · Cited by 9
“ictionoffset.Thesubscriptdenotesthe offsetfromthecurrentpredictionpoint.Englishtranslationsinbrackets. but not seasonal, as has been previously observed for the prediction of influenza like illnesses [5, 18]. Such spurious correlations cannot be expected to reliably model rapid changes in antimicrobial drug consumptio…”
BackgroundPatterns of Consumption and Connectedness in GIS Web Sources
Ballatore, A., Scheider, S., & Lemmens, R. L. G. (2018) · Lecture notes in geoinformation and cartography · doi:10.1007/978-3-319-78208-9_7 · Cited by 5
“Foundational critique about the big data hype was raised in recent years (Boyd and Crawford 2011), addressing the representational bias in human language texts on the Web (Caliskan et al. 2017), which can lead to severe estimation errors in a data-driven science (Lazer et al. 2014).”
INNOVATION STRATEGIES AND ORGANIZATIONAL PERFORMANCE: A CASE STUDY OF TELKOM KENYA LIMITED
Wambui, S. K., KAHUTHIA, D. J., & GAKENIA, D. J. (2018) · Strategic Journal of Business & Change Management · doi:10.61426/sjbcm.v5i3.805 · Cited by 5
Shah, N., Malensek, M., Harshil, S., Pallickara, S., & Pallickara, S. L. (2018) · Concurrency and Computation Practice and Experience · doi:10.1002/cpe.4998 · Cited by 6
“The system demonstrated faster results compared to traditional disease surveillance methods, but also suffered high-profile failures due to potential over-fitting, changes in the underlying Google search algorithm, and the opaque nature of the system (42).”
ResultMeasuring Professional Skill Development in U.S. Cities Using Internet Search Queries
Jhaver, S., Cranshaw, J., & Counts, S. (2019) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v13i01.3228 · Cited by 12
“However, we warn that internet search data may not contain information offered by community knowledge and macroeconomic measures provided by government websites (Lazer et al. 2014), and therefore, skill search data should be seen as a supplemental, rather than a substitute, data source.”
BackgroundThe Application of Big Data in Cyberspace: A Survey
He, J., Wang, H., Xian, M., et al. (2019) · International Conference on Communications, Information System and Computer Engineering (CISCE) · doi:10.1109/cisce.2019.00132 · Cited by 5
“(4) Parable of Google Flu: Traps in Big Data Analysis (2014)[10]”
BackgroundChoi, E., & Horvát, E.-Á. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-34971-4_1 · Cited by 7
Pitfalls Analyzer: Quality Control for Model-Driven Data Science Pipelines
Rajbahadur, G. K., Oliva, G. A., Hassan, A. E., & Juergen Dingel, A. (2019) · ACM/IEEE International Conference on Model Driven Engineering Languages and Systems (MODELS) · doi:10.1109/models.2019.00-19 · Cited by 7
“flu trends correctly, because they did not account for common data science pitfalls [20].”
BackgroundThe Healthy States of America: Creating a Health Taxonomy with Social Media
Šćepanović, S., Aiello, L. M., Zhou, K., Joglekar, S. P., & Quercia, D. (2021) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v15i1.18089 · Cited by 8
“That is to do with whether “the instrumentation is actually capturing the theoretical construct of interest” (Lazer et al. 2014).”
BackgroundInfluentialYu, K. O. N. G., Zhen, F., Zhao-zhong, L. I., & Xing-xing, F. U. (2019) · 自然资源学报 · doi:10.31497/zrzyxb.20191014 · Cited by 5
Li, J., Sia, C., Chen, Z., & Huang, W. (2021) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph18126591 · Cited by 7
“Finally, we did not compare our model with methods based on search query data, such as Google Flu Trends [7], in our research.”
BackgroundMethodInfluentialCharles, V., Emrouznejad, A., & Gherman, T. (2022) · Studies in big data · doi:10.1007/978-3-030-87304-2_2 · Cited by 6
Okunoye, B., Ning, S., & Jemielniak, D. (2022) · JMIR Formative Research · doi:10.2196/29819 · Cited by 5
Environmental change and human mobility: Opportunities and challenges of big data
Martin, S. F., & Singh, L. (2022) · International Migration · doi:10.1111/imig.13002 · Cited by 6
Sun, H., Zhang, Y., Gao, G., & Wu, D. (2022) · Frontiers in Public Health · doi:10.3389/fpubh.2022.958835 · Cited by 7
Nelson, L. K. (2022) · New Literary History · doi:10.1353/nlh.2022.a898331 · Cited by 4
Bishal, M. M., Chowdory, M. R. H., Das, A., & Kabir, M. A. (2024) · Heliyon · doi:10.1016/j.heliyon.2024.e34103 · Cited by 5
“User-generated content from social media platforms has gained substantial recognition for syndromic surveillance during global health emergencies, such as the 2009 H1N1 pandemic [10–16], the 2014 Ebola outbreak [17–22], the 2003 SARS epidemic [23], and recently COVID-19 pandemic [24,25].”
Big Data Analytics: Views from Statistical and Computational Perspectives
Pyne, S., Рао, Б. Л. С. П., & Rao, S. (2016) · Big Data Analytics · doi:10.1007/978-81-322-3628-3_1 · Cited by 5
Big Data in Structure-Property Studies—From Definitions to Models
Polański, J. (2017) · Challenges and advances in computational chemistry and physics · doi:10.1007/978-3-319-56850-8_15 · Cited by 8
Symptom-based data preprocessing for the detection of disease outbreak
Duangchaemkarn, K., Chaovatut, V., Wiwatanadate, P., & Boonchieng, E. (2017) · Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/embc.2017.8037393 · Cited by 5
Barriers to the Implementation of Big Data
Wang, S. J., & Moriarty, P. (2018) · Springer eBooks · doi:10.1007/978-3-319-73610-5_4 · Cited by 5
Crowd Replication: Sensing-Assisted Quantification of Human Behavior in Public Spaces
Hemminki, S., Kuribayashi, K., Konomi, S., Nurmi, P., & Tarkoma, S. (2019) · ACM Transactions on Spatial Algorithms and Systems · doi:10.1145/3317666 · Cited by 5
“Such biases are often difficult to avoid and understand with pervasive sensing and opportunistic data collection methods [33].”
BackgroundLiang, B., Wang, Y., & Tsou, M.-H. (2019) · Journal of health communication · doi:10.1080/10810730.2019.1657526 · Cited by 3
Mobile Phone Data for Children on the Move: Challenges and Opportunities
Sekara, V., Omodei, E., Healy, L., Beise, J., Hansen, C., You, D., Blume, S., & García–Herranz, M. (2019) · Springer eBooks · doi:10.1007/978-3-030-12554-7_3 · Cited by 11
Mango: Exploratory Data Analysis for Large-Scale Sequencing Datasets
Morrow, A., He, G. Z., Nothaft, F. A., Tu, E. T., Paschall, J. E., Yosef, N., & Joseph, A. D. (2019) · Cell Systems · doi:10.1016/j.cels.2019.11.002 · Cited by 9
Recognizing Indonesian Acronym and Expansion Pairs with Supervised Learning and MapReduce
Abidin, T. F., Mahazir, A., Subianto, M., Munadi, K., & Ferdhiana, R. (2020) · Information · doi:10.3390/info11040210 · Cited by 6
“Many queries are posed by millions of users from across the globe each day on Google’s search engine, which has attracted considerable attention from researchers who have analyzed the query logs using machine learning techniques to track and predict phenomena, including the spread patterns of flu symptoms in the United…”
BackgroundArendt, F., & Scherr, S. (2020) · Social Science & Medicine · doi:10.1016/j.socscimed.2020.113495 · Cited by 5
Metadata
- Authors
- Lazer, D., Kennedy, R., King, G., & Vespignani, A.
- Affiliations
- Northeastern University (US)
- Harvard University (US)
- Corresponding author
- Lazer, David
- Venue
- Science
- Published
- 2014-03-14
- Volume
- 343
- Issue
- 6176
- Pages
- 1203-1205
- ISSN
- 0036-8075 (ISSN-L)
- 1095-9203
- OpenAlex
- W2068181924
- Semantic Scholar
- 00d681cfff7f0224ace607e515f4ed510e792df9
- Cited by
- 2,523
- References
- 30
- Citations by year
- 2026: 113 · 2025: 101 · 2024: 121 · 2023: 150 · 2022: 168 · 2021: 229 · 2020: 273 · 2019: 261 · 2018: 272 · 2017: 249 · 2016: 271 · 2015: 219 · 2014: 91
- FWCI
- 208.16
- Type
- journal article
- Language
- en
- Open access
- green
- dash.harvard.edu
- Licence
- cc-by
- Locations
- Science (publishedVersion)
- PubMed (publishedVersion)
- Digital Access to Scholarship at Harvard (DASH) (Harvard University) (submittedVersion, cc-by)
- Topics
- Health Sciences › Medicine › Epidemiology › Data-Driven Disease Surveillance
- Social Sciences › Decision Sciences › Information Systems and Management › Big Data Technologies and Applications
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- Google Flu Trends, influenza surveillance, big data, prediction error, data quality
- MeSH terms
- Centers for Disease Control and Prevention, U.S., Data Collection, Data Interpretation, Statistical, Disease Outbreaks, Humans, Influenza, Human, Office Visits, Prognosis, United States, Social Media
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.