The Parable of Google Flu: Traps in Big Data Analysis
Open accessThe DOI matches its Crossref record; 30 of the 31 references it cites were found in a registry or an index; 5 of the 8 sources its claims cite were checked for support.
Abstract
Large errors in flu prediction were largely avoidable, which offers lessons for the use of big data.
Overview
CoChat Summary
TL;DR
The paper uses the failure of Google Flu Trends (GFT) to illustrate two traps in big data analysis: “big data hubris” and algorithm dynamics. It argues that GFT’s large flu-prediction errors were largely avoidable and that big data should supplement, not replace, traditional data and methods.
Key findings
- In February 2013, GFT predicted more than double the proportion of doctor visits for influenza-like illness (ILI) than the CDC; from 21 August 2011 to 1 September 2013, GFT reported overly high flu prevalence in 100 out of 108 weeks, and it overshot the 2011–2012 season by more than 50%.
- GFT errors were not random: they showed temporal autocorrelation and seasonality, meaning last week’s errors predicted this week’s errors and error direction/magnitude varied with time of year.
- Simple alternatives outperformed GFT: out-of-sample mean absolute error was 0.486 for GFT, 0.311 for lagged CDC, and 0.232 for combined GFT and CDC; all differences were statistically significant at P < 0.05. Even 3-week-old CDC data did better than GFT at projecting current flu prevalence.
- The authors attribute GFT’s problems to overfitting (initial GFT matched 50 million search terms to 1152 data points and missed the nonseasonal 2009 H1N1 pandemic) and to “algorithm dynamics,” including Google search algorithm changes and user behavior changes.
Methods
- Design: policy forum/case study comparing GFT estimates with CDC ILI surveillance data and alternative forecasting models.
- Data/sample: GFT and CDC flu prevalence estimates, including the period from 21 August 2011 to 1 September 2013; the initial GFT methodology reportedly fit 50 million search terms to 1152 data points.
- Main analysis: comparison of GFT alone, lagged CDC data with 52-week seasonality variables, and a combined GFT + CDC model using lagged CDC estimates, lagged GFT error, and seasonality variables; reported mean absolute error and statistical significance; Google Correlate used to examine correlated search terms. Supplementary materials contain data and methods.
Limitations
- The paper states that GFT never documented the 45 search terms used, released examples appear misleading, and Google Correlate is limited to national-level data whereas GFT was developed using regional correlations; therefore the original GFT analyses cannot be replicated from the information provided.
- The paper states that Google cannot make its full data available to outsiders because of privacy issues, though it argues there is no such constraint on derivative, aggregated data.
- Evident from the text: this is a commentary/policy forum rather than a full empirical article, with key data and methods placed in supplementary materials; the quantitative comparison depends on GFT and CDC data and model specifications not fully detailed in the main text.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
2 Valid · 2 Not valid · 2 Not checked · 8 Own results and interpretations
Some cited sources support this claim and some don’t — read each source’s verdict below.
In the paper
“This ad hoc method of throwing out peculiar search terms failed when GFT completely missed the nonseasonal 2009 infl uenza A–H1N1 pandemic ( 2, 14).”
What the sources say
- Partly supportedcites [2] matched to “Reassessing Google Flu Trends Data for Detection of Seasonal and Pandemic Influenza: A Comparative Epidemiological Study at Three Geographic Scales” (2013) sourceExperimental — accuracy not yet measured
The Discussion (page unavailable) says the original GFT model missed the spring 2009 pandemic wave, supporting the claim about its failure to detect the nonseasonal outbreak. It does not say GFT completely missed the entire 2009 pandemic. The Introduction describes automated selection of the original model’s search terms, not an ad hoc method of throwing out peculiar terms. (LLM, conf=0.86)
- Disputedcites [14] S. Cook et al., PLOS ONE 6, e23610 (2011). sourceExperimental — accuracy not yet measured
The Abstract (Methodology/Principal Findings) says the original GFT model underestimated the magnitude of illness during the 2009 H1N1 pandemic, not that it completely missed it. The Abstract (Conclusions) says both models performed well during the pandemic, although the updated model performed better. The Discussion notes that pandemic-specific queries were excluded, but does not describe this as an ad hoc method of throwing out peculiar terms. (LLM, conf=0.88)
Checked by a reasoning model — may be wrong.
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“Oddly, the few search terms offered in the papers ( 14) do not seem to be strongly related with either GFT or the CDC data (SM)—we surmise that the authors felt an unarticulated need to cloak the actual search terms identifi ed.”
What the sources say
- Disputedcites [14] S. Cook et al., PLOS ONE 6, e23610 (2011). sourceExperimental — accuracy not yet measured
The Results section (‘Search Behavior During pH1N1’; page unavailable) says that all three search queries shown in Figure 3 closely tracked CDC ILINet data before the pandemic, and ‘symptoms of flu’ continued to do so during it. The paper does report that the bronchitis and pneumonia queries underestimated ILINet data during the pandemic, but that does not support the broad claim that the offered terms were not strongly related to the data. (LLM, conf=0.87)
Checked by a reasoning model — may be wrong.
hedgedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“The Internet has opened the way for improving standard surveys, experiments, and health reporting ( 35).”
What the sources say
- Partly supportedcites [35] A. J. Berinsky et al., Polit. Anal. 20, 351–368 (2012). sourceExperimental — accuracy not yet measured
Section 2 describes using an Internet platform to run surveys with embedded experiments, and the Conclusion says MTurk can make experimental research easier and less expensive. The paper does not address improvements to health reporting; the disease scenario in section 4.3.2 is a framing experiment, not health reporting. Page numbers are unavailable in our copy. (LLM, conf=0.88)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Although not widely reported until 2013, the new GFT has been persistently overestimating flu prevalence for a much longer time.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“From 21 August 2011 to 1 September 2013, GFT reported overly high fl u prevalence 100 out of 108 weeks.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
3 more claims
Create a free account to see all 14 claims and every source verdict.
- It may state two separate claims as one. 5
- Probably not a substantial claim. 2
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Google Trends predicted more than twice the number of influenza-like illnesses than the CDC in February 2013 46 .”
From Unrepresentative big surveys significantly overestimated US vaccine uptake (2021) · cites this paper as [46]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This was one of the main reasons behind the failure of Google Flu Trends, since the data used for training the model in the first few years was not representative of the trends in subsequent years [15].”
From Theory-Guided Data Science: A New Paradigm for Scientific Discovery from Data (2017) · cites this paper as [15]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Social data research often relies on counting entities. These entities can be users, links, or messages, and the description is a summary of these counts (Lazer et al., 2014; Salganik, 2017). Yet, simple counts can mislead if it is unclear what is counted and how.”
From Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019) · cites this paper as Lazer et al., 2014
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This unintelligibility is also subject to functional biases (section 4.1), as it depends on the mechanisms available on each social platform (e.g., having a like button, but not a dislike one), as well as to variations in platform algorithms and mechanisms in response to users actions (Lazer et al., 2014).”
From Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019) · cites this paper as Lazer et al., 2014
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“It also includes temporal validity, which captures to what extent constructs change over time (Howison et al., 2011) and may invalidate previous conclusions about societal and/or platform phenomena; e.g., see the case of Google Flu Trends (Lazer et al., 2014).”
From Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019) · cites this paper as Lazer et al., 2014
6 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 14 MARCH 2014 VOL 343 SCIENCE www.sciencemag.org 1204
“The 2009 version of GFT has weathered other media panics related to the fl u, including the 2005–2006 influenza A/H5N1 ("bird flu") outbreak”
“GFT engineers updated the algorithm in 2009, and this model has run ever since”
Citations
Cites
boyd, D., & Crawford, K. (2012) · Information, Communication & Society · doi:10.1080/1369118x.2012.678878 · Cited by 5,479
Twitter mood predicts the stock market
Bollen, J., Mao, H., & Zeng, X. (2011) · Journal of Computational Science · doi:10.1016/j.jocs.2010.12.007 · Cited by 5,206
Detecting influenza epidemics using search engine query data
Ginsberg, J., Mohebbi, M. H., Patel, R. S., Brammer, L., Smolinski, M. S., & Brilliant, L. (2009) · Nature · doi:10.1038/nature07634 · Cited by 4,295
Cohen, D. (1998) · The Women's Review of Books · doi:10.2307/4022859 · Cited by 4,261
Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk
Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012) · Political Analysis · doi:10.1093/pan/mpr057 · Cited by 4,008
Big Data : The Management Revolution
マカフィー, ア., ブリニョルフソン, エ., & 有賀, 裕. (2013) · Harvard business review · Cited by 3,549
Lazer, D., Pentland, A., Adamic, L., Aral, S., Barabási, A.-L., Brewer, D., Christakis, N., Contractor, N., Fowler, J., Gutmann, M., et al. (2009) · Science · doi:10.1126/science.1167742 · Cited by 3,365
Self-citationPredicting Elections with Twitter: What 140 Characters Reveal about Political Sentiment
Tumasjan, A., Sprenger, T., Sandner, P., & Welpe, I. (2010) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v4i1.14009 · Cited by 2,649
Multiscale mobility networks and the spatial spreading of infectious diseases
Balcan, D., Colizza, V., Gonçalves, B., Hu, H., Ramasco, J. J., & Vespignani, A. (2009) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0906910106 · Cited by 1,380
Self-citationPredicting consumer behavior with web search
Goel, S., Hofman, J. M., Lahaie, S., Pennock, D. M., & Watts, D. J. (2010) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1005962107 · Cited by 636
Research works printed but not linked: 7
- A. McAfee, E. Brynjolfsson, Harv. Bus. Rev. 90, 60 (2012).
- AAAI Conference on Weblogs and Social Media, Atlanta, Georgia, 11 to 15 July 2010 (Association for Advancement of Artifi cial Intelligence, 2010), pp. 178.
- P. Copeland et al., Int. Soc. Negl. Trop. Dis. 2013, 3
- I. M. Hall et al., Epidemiol. Infect. 135, 372 (2007).
- E. Mustafaraj, P. Metaxas, in Proceedings of the Web- Sci10, Raleigh, NC, 26 and 27 April 2010 (Web Science Trust, 2010); http://journal.webscience.org/317/.
- J. Ratkiewicz et al., in Proceedings of 5th International AAAI Conference on Weblogs and Social Media, San Francisco, CA, 7 to 11 August 2011 (AAAI, 2011), p. 297–304.
- 2013; http://chronicle.com/article/Researchers-Struggle- to-Secure/141591/.
Cited by
Items 1,101–1,200 of 2,513
Collective response to the media coverage of COVID-19 Pandemic on Reddit and WikipediaPreprint
Gozzi, N., Tizzani, M., Starnini, M., Ciulla, F., Paolotti, D., Panisson, A., & Perra, N. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.06446 · Cited by 12
“[27] David Lazer, Ryan Kennedy, Gary King, and Alessandro Vespignani, “The parable of google flu: traps in big data analysis,” Science 343, 1203–1205 (2014).”
BackgroundSurvey data and human computation for improved flu tracking
Wojcik, S., Bijral, A. S., Johnston, R., et al. (n.d.) · RePEc: Research Papers in Economics · Cited by 13
Self-citationA Reality Check(-List) for Digital MethodsPreprint
Venturini, T., Bounegru, L., Gray, J., et al. (2018) · SSRN Electronic Journal · doi:10.2139/ssrn.3168664 · Cited by 13
Do Customers Speak Their Minds? Using Forums and Search for Predicting SalesPreprint
Geva, T., Oestreicher-Singer, G., Efron, N., & Shimshoni, Y. (2013) · SSRN Electronic Journal · doi:10.2139/ssrn.2294609 · Cited by 13
Time to audit your AI algorithms
Sandu, I., Wiersma, M., & Manichand, D. (2022) · Maandblad voor Accountancy en Bedrijfseconomie · doi:10.5117/mab.96.90108 · Cited by 12
“Finally, the evolving nature of many AI algorithms, that is, the updates to the algorithm as more data comes in, to further train the algorithm, means that the performance of the algorithm can change over time (Lazer et al. 2014).”
BackgroundGong, X., & Zha, Y. (2025) · Water Resources Research · doi:10.1029/2025wr040040 · Cited by 13
“Yet DL often requires extensive, high–quality data sets, raising concerns about interpretability and generalizability (Alzubaidi et al., 2023; Lazer et al., 2014; Loyola‐Gonzalez, 2019).”
Data Science in Public Health: Building Next Generation Capacity
Mirin, N., Mattie, H., Jackson, L., Samad, Z., & Chunara, R. (2022) · Harvard Data Science Review · doi:10.1162/99608f92.18da72db · Cited by 13
“…recent work has surfaced flaws in the use of machine learning approaches in a siloed way, such as algorithms built without an understanding of whom or what the data represent (Buolamwini & Gebru, 2018), limiting internal and external validity of conclusions drawn from analyses (Lazer et al., 2014).”
Schwitter, N., Pretari, A., Marwa, W., Lombardini, S., & Liebe, U. (2022) · Frontiers in Sociology · doi:10.3389/fsoc.2022.909458 · Cited by 14
“While big data come with their traps and biases (Lazer et al., 2014), particularly around representativeness due to access to the internet and/or devices, which are not neutral to race, class, gender, and geography, they can provide a uniquely unobtrusive way to access information from people who are in positions of m…”
BackgroundBig Data and Infectious Disease Epidemiology: Bibliometric Analysis and Research Agenda
Amusa, L. B., Twinomurinzi, H., Phalane, E., & Phaswana-Mafuya, N. R. (2023) · Interactive Journal of Medical Research · doi:10.2196/42292 · Cited by 20
Probability vs. Nonprobability Sampling: From the Birth of Survey Sampling to the Present Day
Kalton, G. (2023) · Statistics in Transition New Series · doi:10.59170/stattrans-2023-029 · Cited by 30
Insight into Facile Ion Diffusion in Resistive Switching Medium toward Low Operating Voltage Memory
Do, D. P., Bui, V. Q., Nguyen, M. C., Seo, S., Do, V. D., Kim, J., Choi, J., Ko, H., Yu, W. J., Kawazoe, Y., et al. (2024) · Nano Letters · doi:10.1021/acs.nanolett.4c01629 · Cited by 12
Timely, Granular, and Actionable: Designing a Social Listening Platform for Public Health 3.0
Kitchens, B., Claggett, J. L., & Abbasi, A. (2024) · MIS Quarterly · doi:10.25300/misq/2023/17381 · Cited by 18
“Despite initial promise, the project failed (Lazer et al. 2014).”
Social Media as a New Vital Sign: Commentary
Young, S. D. (2018) · Journal of Medical Internet Research · doi:10.2196/jmir.8563 · Cited by 16
Potential uses of an infodemiology approach for health-care services for rheumatology
Martinez-Arroyo, G., Ramos-Gomez, S., Rojero-Gil, E. K., Rojas-Gongora, J. A., Barajas‐Ochoa, A., Bustamante-Montes, L. P., Yáñez, J. L., & Ramos-Remus, C. (2018) · Clinical Rheumatology · doi:10.1007/s10067-018-4364-z · Cited by 12
Revisiting the use of web search data for stock market movements
Zhong, X., & Raghib, M. (2019) · Scientific Reports · doi:10.1038/s41598-019-50131-1 · Cited by 15
Applying Big Data visualization to detect trends in 30 years of performance reports
Raveh, E., Ofek, Y., Bekkerman, R., & Cohen, H. (2020) · Evaluation · doi:10.1177/1356389020905322 · Cited by 14
“Therefore, queries will constantly need to be refined, leading to a constant increase in the number of keywords (see Lazer et al., 2014).”
BackgroundBroe, S. D., Struijs, P., Daas, P., van Delden, A., Burger, J., van den Brakel, J., ten Bosch, O., Zeelenberg, K., & Ypma, W. F. H. (2020) · Statistical Journal of the IAOS · doi:10.3233/sji-200711 · Cited by 13
“The example of the google flu case [17,18] shows that it is challenging to validate findings.”
Jaidka, K., Eichstaedt, J. C., Giorgi, S., Schwartz, H. A., & Ungar, L. (2020) · Telematics and Informatics · doi:10.1016/j.tele.2020.101540 · Cited by 18
Zhu, G., Li, L., Zheng, Y., Zhang, X., & Zou, H. (2021) · Journal of Advanced Computational Intelligence and Intelligent Informatics · doi:10.20965/jaciii.2021.p0138 · Cited by 12
Zhang, Y., Bambrick, H., Mengersen, K. L., Tong, S., & Hu, W. (2021) · International Journal of Biometeorology · doi:10.1007/s00484-021-02155-4 · Cited by 16
Castro, V. M., Hart, K. L., Sacks, C. A., Murphy, S. N., Perlis, R. H., & McCoy, T. H. (2021) · General Hospital Psychiatry · doi:10.1016/j.genhosppsych.2021.10.005 · Cited by 14
“These underlying changes in patient characteristics are an instance of concept drift and demonstrate the associated compromise of internal validity consistent with a rapidly evolving pandemic [ 108 – 111] .”
A survey of data element perspective: Application of artificial intelligence in health big data
Xiong, H., Chen, H., Xu, L. D., Liu, H., Fan, L., Tang, Q., & Cho, H. (2022) · Frontiers in Neuroscience · doi:10.3389/fnins.2022.1031732 · Cited by 13
“However, recent research showed that the above-mentioned model in the prediction of flu outbreak existed big defects due to the instability of social network data (Lazer et al., 2014).”
ResultTaube, J. C., Susswein, Z., & Bansal, S. (2023) · JMIR Public Health and Surveillance · doi:10.2196/42128 · Cited by 14
“In our work, we sought to steer away from “big data hubris” [50] by applying rigorous statistical methods to manage concerns about representativeness and bias and by conducting an internal validation of our model-based estimates [51].”
Recent progress in neuromorphic and memory devices based on graphdiyne
Zhang, Z., Chen, X., & Lu, T. (2023) · Science and Technology of Advanced Materials · doi:10.1080/14686996.2023.2196240 · Cited by 13
Whole Person Modeling: a transdisciplinary approach to mental health research
Felsky, D., Cannitelli, A., & Pipitone, J. (2023) · Discover Mental Health · doi:10.1007/s44192-023-00041-6 · Cited by 16
CAFA: Class-Aware Feature Alignment for Test-Time Adaptation
Jung, S., Lee, J., Kim, N., Shaban, A., Boots, B., & Choo, J. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01747 · Cited by 34
Vanathi, R., & Khadir, A. S. A. (2017) · World Congress on Computing and Communication Technologies (WCCCT) · doi:10.1109/wccct.2016.32 · Cited by 13
“In [10] Google Flu Trend based on big data analytics collected from many laboratory tests and it created a scenario to understand the flu spread trend.”
MethodAn automated approach towards sparse single-equation cointegration modelling
Smeekes, S., & Wijler, E. (2020) · Journal of Econometrics · doi:10.1016/j.jeconom.2020.07.021 · Cited by 24
“However, there are additional peculiarities specific to the use of Google trends such as normalization, data hubris and search algorithm dynamics, all of which might result in unstable performance (cf. Lazer et al., 2014).”
BackgroundA Generic Survey on Medical Big Data Analysis Using Internet of Things
Kuila, S., Dhanda, N., Joardar, S., Neogy, S., & Kuila, J. (2018) · Advances in intelligent systems and computing · doi:10.1007/978-981-13-1580-0_26 · Cited by 12
Exploring the Specificities and Challenges of Testing Big Data Systems
Staegemann, D. G., Volk, M., Nahhas, A., Abdallah, M. M. A., & Turowski, K. (2019) · International Conference on Signal-Image Technology & Internet-Based Systems (SITIS) · doi:10.1109/sitis.2019.00055 · Cited by 13
Clemens, T., & Viechtbauer-Gruber, M. (2020) · EAGE Conference and Exhibition incorporating SPE EUROPEC · doi:10.2118/200540-ms · Cited by 12
Graeff, P., & Baur, N. (2020) · Historical social research · doi:10.12759/hsr.45.2020.3.244-269 · Cited by 22
Beaude, B. (2015) · Presses des Mines eBooks · doi:10.4000/books.pressesmines.2000 · Cited by 27
Riddell, C. A., Neumann, K., Santaularia, N. J., Farkas, K., Ahern, J., & Mason, S. M. (2022) · Journal of Medical Internet Research · doi:10.2196/36445 · Cited by 15
State of the Art Review: The Data Revolution in Critical Care
Ghassemi, M., Celi, L. A., & Stone, D. J. (2015) · Annual update in intensive care and emergency medicine · doi:10.1007/978-3-319-13761-2_42 · Cited by 12
Kakarmath, S. S., Golas, S. B., Felsted, J. A., Kvedar, J. C., Jethwani, K., & Agboola, S. O. (2018) · JMIR Research Protocols · doi:10.2196/resprot.9466 · Cited by 13
Coordinating judgmental forecasting: Coping with intentional biases
Pennings, C. L. P., van Dalen, J., & Rook, L. (2018) · Omega · doi:10.1016/j.omega.2018.08.007 · Cited by 12
Dong, J., Dai, W., Liu, Y., Yu, L., & Wang, J. (2019) · International Journal of Information Technology & Decision Making · doi:10.1142/s0219622019500287 · Cited by 12
Ormerod, P., Nyman, R., & Bentley, R. A. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1408.0699 · Cited by 12
Public-Private Partnerships for Statistics: Lessons Learned, Future Steps
Robin, N., Klein, T., & Jütting, J. (2016) · OECD development co-operation working papers · doi:10.1787/5jm3nqp1g8wf-en · Cited by 21
Perils and Challenges of Social Media and Election Manipulation Analysis: The 2018 US MidtermsPreprint
Deb, A., Luceri, L., Badawy, A., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.00043 · Cited by 12
“, the outcomes of political elections [22–26, 38], or public health issues [2, 36, 57].”
BackgroundNIST Big Data Interoperability Framework: volume 4, security and privacy version 3
NIST Big Data Public Working Group, D. A. T. S. (2019) · doi:10.6028/nist.sp.1500-4r2 · Cited by 13
Keegan, B., & Tan, C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.08899 · Cited by 19
“While some studies emphasize the feasibility of using Wikipedia data to forecast influenza [10, 55, 68, 96, 137, 153], dengue [67], zika [140], and other diseases [39, 135], other research concludes behavioral heterogeneity on social platforms like Wikipedia cannot reliably beat canonical epidemiological forecasting m…”
BackgroundNIST Big Data Interoperability Framework: Volume 4, Security and Privacy
NIST Big Data Public Working Group Security and Privacy Subgroup (2015) · doi:10.6028/nist.sp.1500-4 · Cited by 13
Rai, A. (2020) · MIS Quarterly · doi:10.25300/misq/2020/441e0 · Cited by 13
Digitalization, autonomy and the future of energy policy
Heymann, F., Küfeoğlu, S., & Galus, M. D. (2025) · Energy Research & Social Science · doi:10.1016/j.erss.2025.104167 · Cited by 11
What Big Data May Mean for Surveys
Couper, M. P. (2014) · Cited by 11
Big Data : Do Google Searches Predict Unemployment?
Tuhkuri, J. (2015) · Työväentutkimus Vuosikirja · Cited by 12
Bogdanovich, E., Brenning, A., Reichstein, M., Polt, K. D., Guenther, L., Frank, D., & Orth, R. (2023) · International Journal of Disaster Risk Reduction · doi:10.1016/j.ijdrr.2023.104206 · Cited by 13
Li, J., Yan, X., Chu, X., et al. (2025) · Journal of Medical Internet Research · doi:10.2196/71786 · Cited by 11
“However, the model was not stable because the influenza trend exceeded the peak of the epidemic by more than 140% in 2013 in the United States and sparked a hot discussion about the limitation of search data in infectious disease research [18].”
Emrouznejad, A., & Marra, M. (2016) · Studies in big data · doi:10.1007/978-3-319-30265-2_1 · Cited by 13
Huang, L., Ni, X., Ditto, W. L., Spano, M. L., Carney, P. R., & Lai, Y. (2017) · Royal Society Open Science · doi:10.1098/rsos.160741 · Cited by 11
Infectious Disease Research in the Era of Big Data
Kasson, P. M. (2020) · Annual Review of Biomedical Data Science · doi:10.1146/annurev-biodatasci-121219-025722 · Cited by 12
“These predictions later showed some systematic overestimation versus CDC estimates and have been criticized by some as an example of big tech hubris (89).”
ResultIn search of art: rapid estimates of gallery and museum visits using Google Trends
Botta, F., Preis, T., & Moat, H. S. (2020) · EPJ Data Science · doi:10.1140/epjds/s13688-020-00232-z · Cited by 11
COVID-19 early-alert signals using human behavior alternative data
Bari, A., Khubchandani, A., Wang, J., Heymann, M., & Coffee, M. P. (2021) · Social Network Analysis and Mining · doi:10.1007/s13278-021-00723-5 · Cited by 13
Pötzschke, S., & Rinken, S. (2022) · IMISCOE research series · doi:10.1007/978-3-031-01319-5 · Cited by 18
Menchen-Trevino, E., Struett, T., Weeks, B. E., & Wojcieszak, M. (2022) · The Information Society · doi:10.1080/01972243.2022.2152915 · Cited by 10
The Ethics of Computational Social Science
Leslie, D. (2023) · Springer eBooks · doi:10.1007/978-3-031-16624-2_4 · Cited by 20
Nobel Citation Effects on Scientific Publications: A Case Study in Physics
Dong, X., Lin, K., Gao, Y., & Hu, B. (2023) · Information Processing & Management · doi:10.1016/j.ipm.2023.103410 · Cited by 14
The evolution of the COVID-19 pandemic through the lens of google searches
Marty, R., Ramos-Maqueda, M., Khan, N., & Reichert, A. R. (2023) · Scientific Reports · doi:10.1038/s41598-023-41675-4 · Cited by 14
de Melo, P. O. S. V. (2015) · PLoS ONE · doi:10.1371/journal.pone.0140217 · Cited by 13
Urban Informatics: Critical Data and Technology Considerations
Krishnamurthy, R., Smith, K. L., & Desouza, K. C. (2016) · Springer geography · doi:10.1007/978-3-319-40902-3_10 · Cited by 11
Crowd-sourcing for smart cities
Chowdhury, S. N., Dhawan, S., & Agnihotri, A. (2016) · IEEE International Conference on Recent Trends in Electronics, Information & Communication Technology · doi:10.1109/rteict.2016.7807842 · Cited by 11
“It would make the feedback mechanism stronger and allow the citizens to enjoy the services and the administrators to put those services in place for them.”
BackgroundCharalampopoulos, I., Nastos, P. T., & Didaskalou, E. (2017) · Urban Science · doi:10.3390/urbansci1010008 · Cited by 11
L’involution géographique : des données géosociales aux algorithmes
Quesnot, T. (2016) · Netcom · doi:10.4000/netcom.2545 · Cited by 22
Uncovering the relationships between military community health and affects expressed in social media
Volkova, S., Charles, L., Harrison, J., & Corley, C. D. (2017) · EPJ Data Science · doi:10.1140/epjds/s13688-017-0102-z · Cited by 13
Data Quality Challenges in Twitter Content Analysis for Informing Policy Making in Health Care
Soto, A. J., Ryan, C., Silva, F. P., Das, T., Wołkowicz, J., Milios, E., & Brooks, S. (2018) · Hawaii International Conference on System Sciences (HICSS) · doi:10.24251/hicss.2018.096 · Cited by 14
“Social media is an exemplary representative of the inherent challenges of Big Data [4, 5], where vast amounts of data are published at an ever-increasing rate.”
BackgroundChallenges to representing the population from new forms of consumer data
Lansley, G., & Cheshire, J. A. (2018) · Geography Compass · doi:10.1111/gec3.12374 · Cited by 16
“A large share of research on data pertaining to individuals is undertaken within the commercial sector where the focus is on efficiency rather than scientific robustness (Lazer et al., 2014).”
BackgroundProblems with research methods in medical device big data analytics
Strang, K. D. (2019) · International Journal of Data Science and Analytics · doi:10.1007/s41060-019-00176-2 · Cited by 12
“Apparently the healthcare big data-generating engine at Google was modified in such a way that the formerly highly predictive search terms eventually failed to work, for example, when a user searched on fever or cough, Google’s other programs started recommending searches for flu symptoms and treatments, which had a d…”
BackgroundMethodResultInfluentialArehart, C. H., David, M. Z., & Dukić, V. (2019) · Scientific Reports · doi:10.1038/s41598-019-56385-z · Cited by 10
Modernizing the Methods and Analytics Curricula for Health Science Doctoral Programs
Dinov, I. D. (2020) · Frontiers in Public Health · doi:10.3389/fpubh.2020.00022 · Cited by 12
“The 2011 Google Flu Trends (GFT) report (38) was an example where GFT prediction problems were identified in 2013 (39) and partially attributed to overfitting.”
Monitoring sick leave data for early detection of influenza outbreaks
Duchemin, T., Bastard, J., Ante-Testard, P. A., Assab, R., Daouda, O. S., Duval, A., Garsi, J., Lounissi, R., Nekkab, N., Neynaud, H., et al. (2021) · BMC Infectious Diseases · doi:10.1186/s12879-020-05754-5 · Cited by 12
COVID-19 Outbreak and Fashion Communication Strategies on Instagram: A Content Analysis
Noris, A., & Cantoni, L. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-78227-6_25 · Cited by 11
Reporting and Transparency in Big Data: The Nexus of Ethics and Methodology
Nicholls, S. G., Langan, S., & Benchimol, E. I. (2016) · Law, governance and technology series · doi:10.1007/978-3-319-33525-4_15 · Cited by 15
Sparse seasonal and periodic vector autoregressive modeling
Baek, C., Davis, R. A., & Pipiras, V. (2016) · Computational Statistics & Data Analysis · doi:10.1016/j.csda.2016.09.005 · Cited by 11
“For a recent critique, see e.g. Lazer et al. (2014).”
Recovering Markov models from closed-loop data
Epperlein, J. P., Zhuk, S. M., & Shorten, R. N. (2019) · Automatica · doi:10.1016/j.automatica.2019.01.022 · Cited by 11
From big data to big analysis: a perspective of geographical conditions monitoring
Zhai, L., Zhang, N., Hou, W., Feng, Z., Qiao, Q., & Luo, M. (2018) · International Journal of Image and Data Fusion · doi:10.1080/19479832.2018.1482965 · Cited by 11
Big Data in Official Statistics
Braaksma, B., Zeelenberg, K., & Broe, S. D. (2020) · Wiley eBooks · doi:10.1002/9781118976357.ch10 · Cited by 14
“A couple of years ago, there was a lot of enthusiasm concerning Google Flu, but more recently the nowcasting performance of Google Flu has decreased significantly (Lazer et al, 2014).”
BackgroundTrust in Public Policy AlgorithmsPreprint
Kennedy, R., Waggoner, P., & Ward, M. (2018) · SSRN Electronic Journal · doi:10.2139/ssrn.3339475 · Cited by 11
Self-citationUnderstanding digital platform evolution using compartmental models
Szalkowski, G. A., & Mikalef, P. (2023) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2023.122600 · Cited by 20
Mezić, I., Drmač, Z., Črnjarić, N., Mačešić, S., Fonoberova, M., Mohr, R., Avila, A. M., Manojlović, I., & Andrejčuk, A. (2024) · Scientific Reports · doi:10.1038/s41598-024-55798-9 · Cited by 22
“However, predicting the number of influenza cases accurately is a notoriously hard problem 20 , exacerbated by the possibility that a vaccine designed in a particular year does not effectively protect against infection.”
Social media for palliative and end-of-life care research: a systematic review
Wang, Y., Koffman, J., Gao, W., Zhou, Y., Chukwusa, E., & Ćurčin, V. (2024) · BMJ Supportive & Palliative Care · doi:10.1136/spcare-2023-004579 · Cited by 12
“79 We identified among our included studies that developing and accessing search filters during the data collection process were sometimes not present or standardised.”
The canary in the city: indicator groups as predictors of local rent increases
Steentoft, A. A., Poorthuis, A., Lee, B., & Schläpfer, M. (2018) · EPJ Data Science · doi:10.1140/epjds/s13688-018-0151-y · Cited by 10
Spatial Positioning Token (SPToken) for Smart Mobility
Overko, R., Ordóñez-Hurtado, R., Zhuk, S. M., Ferraro, P., Cullen, A., & Shorten, R. N. (2020) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2020.3029537 · Cited by 14
Cahill, T. J., Wertz, B., Zhong, Q., Parlato, A., Donegan, J., Forman, R., Manot, S., Wu, T., Xu, Y., Cummings, J. J., et al. (2019) · Journal of Medical Internet Research · doi:10.2196/12980 · Cited by 14
Preece, A. D., Gwilliams, C., Parizas, C., Pizzocaro, D., Bakdash, J. Z., & Braines, D. (2014) · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · doi:10.1117/12.2053283 · Cited by 10
“In the context of current concerns regarding transparency in big data approaches [17], the latter kind of dialogues seem particularly promising.”
BackgroundData Sovereignty: A Perspective From Zimbabwe
Mawere, M., & van Stam, G. (2020) · ACM Web Science Conference (WebSci) · doi:10.1145/3394332.3402823 · Cited by 12
Sharma, A., & Colonna, G. (2021) · Molecular Diagnosis & Therapy · doi:10.1007/s40291-020-00505-3 · Cited by 13
Jang, B., Kim, I., & Kim, J. W. (2021) · JMIR Medical Informatics · doi:10.2196/23305 · Cited by 15
Tozzi, A. E., Gesualdo, F., Urbani, E., Sbenaglia, A., Ascione, R., Procopio, N., Croci, I., & Rizzo, C. (2021) · Journal of Medical Internet Research · doi:10.2196/29556 · Cited by 12
Click, click boom: Using Wikipedia data to predict changes in battle-related deaths
Oswald, C., & Ohrenhofer, D. (2022) · International Interactions · doi:10.1080/03050629.2022.2061969 · Cited by 14
“Third, Google is commercial and subject to both financial interests and changes to the search algorithm, which occur frequently, which affect the data-generating process (Lazer et al. 2014).”
BackgroundRETRACTED: Does the matching degree between data and human capital affect firm innovation?
Hu, X. T., & Fei, Y. (2023) · Finance research letters · doi:10.1016/j.frl.2023.103985 · Cited by 10
RetractedReal-time estimation of the effective reproduction number of COVID-19 from behavioral data
Bokányi, E., Vizi, Z., Koltai, J., Röst, G., & Karsai, M. (2023) · Scientific Reports · doi:10.1038/s41598-023-46418-z · Cited by 13
Effect of environmental factors on Internet searches related to sinusitis
Willson, T. J., Lospinoso, J. A., Weitzel, E. K., et al. (2015) · The Laryngoscope · doi:10.1002/lary.25420 · Cited by 10
Wears, R. L., & Williams, D. J. (2015) · Annals of Emergency Medicine · doi:10.1016/j.annemergmed.2015.09.019 · Cited by 10
‘Big data’ approaches for novel anti-cancer drug discovery
Benstead-Hume, G., Wooller, S. K., & Pearl, F. M. G. (2017) · Expert Opinion on Drug Discovery · doi:10.1080/17460441.2017.1319356 · Cited by 12
Role of Online Data from Search Engine and Social Media in Healthcare Informatics
Nawaz, M. S., Mustafa, R. U., & Lali, M. I. U. (2017) · Advances in bioinformatics and biomedical engineering book series · doi:10.4018/978-1-5225-2607-0.ch011 · Cited by 10
“Furthermore, errors discovered in Google Flu Trends serves as reminder that this big online data paradigm required further critical investigation and the development of more empirical methodologies for exploring the predictive utility of Internet data (Lazer et al., 2014a; Lazer et al., 2014b).”
BackgroundSeasonal Web Search Query Selection for Influenza-Like Illness (ILI) Estimation
Hansen, N. D., Mølbak, K., Cox, I. J., & Lioma, C. (2017) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3077136.3080760 · Cited by 10
“Similarly to [8, 11], we evaluate estimation performance by reporting the root mean squared error (RMSE) and Pearson correlation between the estimations and the observed historical ILI activity.”
BackgroundMethodInfluentialEstimate suggests many infant deaths in sub-Saharan Africa attributable to air pollution
Waller, L. A. (2018) · Nature · doi:10.1038/d41586-018-05394-5 · Cited by 9
Erosheva, E. A. (2018) · Biometrics · doi:10.1111/biom.12900 · Cited by 10
Metadata
- Authors
- Lazer, D., Kennedy, R., King, G., & Vespignani, A.
- Affiliations
- Northeastern University (US)
- Harvard University (US)
- Corresponding author
- Lazer, David
- Venue
- Science
- Published
- 2014-03-14
- Volume
- 343
- Issue
- 6176
- Pages
- 1203-1205
- ISSN
- 0036-8075 (ISSN-L)
- 1095-9203
- OpenAlex
- W2068181924
- Semantic Scholar
- 00d681cfff7f0224ace607e515f4ed510e792df9
- Cited by
- 2,523
- References
- 30
- Citations by year
- 2026: 113 · 2025: 101 · 2024: 121 · 2023: 150 · 2022: 168 · 2021: 229 · 2020: 273 · 2019: 261 · 2018: 272 · 2017: 249 · 2016: 271 · 2015: 219 · 2014: 91
- FWCI
- 208.16
- Type
- journal article
- Language
- en
- Open access
- green
- dash.harvard.edu
- Licence
- cc-by
- Locations
- Science (publishedVersion)
- PubMed (publishedVersion)
- Digital Access to Scholarship at Harvard (DASH) (Harvard University) (submittedVersion, cc-by)
- Topics
- Health Sciences › Medicine › Epidemiology › Data-Driven Disease Surveillance
- Social Sciences › Decision Sciences › Information Systems and Management › Big Data Technologies and Applications
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- Google Flu Trends, influenza surveillance, big data, prediction error, data quality
- MeSH terms
- Centers for Disease Control and Prevention, U.S., Data Collection, Data Interpretation, Statistical, Disease Outbreaks, Humans, Influenza, Human, Office Visits, Prognosis, United States, Social Media
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.