The Parable of Google Flu: Traps in Big Data Analysis
Open accessThe DOI matches its Crossref record; 30 of the 31 references it cites were found in a registry or an index; 5 of the 8 sources its claims cite were checked for support.
Abstract
Large errors in flu prediction were largely avoidable, which offers lessons for the use of big data.
Overview
CoChat Summary
TL;DR
The paper uses the failure of Google Flu Trends (GFT) to illustrate two traps in big data analysis: “big data hubris” and algorithm dynamics. It argues that GFT’s large flu-prediction errors were largely avoidable and that big data should supplement, not replace, traditional data and methods.
Key findings
- In February 2013, GFT predicted more than double the proportion of doctor visits for influenza-like illness (ILI) than the CDC; from 21 August 2011 to 1 September 2013, GFT reported overly high flu prevalence in 100 out of 108 weeks, and it overshot the 2011–2012 season by more than 50%.
- GFT errors were not random: they showed temporal autocorrelation and seasonality, meaning last week’s errors predicted this week’s errors and error direction/magnitude varied with time of year.
- Simple alternatives outperformed GFT: out-of-sample mean absolute error was 0.486 for GFT, 0.311 for lagged CDC, and 0.232 for combined GFT and CDC; all differences were statistically significant at P < 0.05. Even 3-week-old CDC data did better than GFT at projecting current flu prevalence.
- The authors attribute GFT’s problems to overfitting (initial GFT matched 50 million search terms to 1152 data points and missed the nonseasonal 2009 H1N1 pandemic) and to “algorithm dynamics,” including Google search algorithm changes and user behavior changes.
Methods
- Design: policy forum/case study comparing GFT estimates with CDC ILI surveillance data and alternative forecasting models.
- Data/sample: GFT and CDC flu prevalence estimates, including the period from 21 August 2011 to 1 September 2013; the initial GFT methodology reportedly fit 50 million search terms to 1152 data points.
- Main analysis: comparison of GFT alone, lagged CDC data with 52-week seasonality variables, and a combined GFT + CDC model using lagged CDC estimates, lagged GFT error, and seasonality variables; reported mean absolute error and statistical significance; Google Correlate used to examine correlated search terms. Supplementary materials contain data and methods.
Limitations
- The paper states that GFT never documented the 45 search terms used, released examples appear misleading, and Google Correlate is limited to national-level data whereas GFT was developed using regional correlations; therefore the original GFT analyses cannot be replicated from the information provided.
- The paper states that Google cannot make its full data available to outsiders because of privacy issues, though it argues there is no such constraint on derivative, aggregated data.
- Evident from the text: this is a commentary/policy forum rather than a full empirical article, with key data and methods placed in supplementary materials; the quantitative comparison depends on GFT and CDC data and model specifications not fully detailed in the main text.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
2 Valid · 2 Not valid · 2 Not checked · 8 Own results and interpretations
Some cited sources support this claim and some don’t — read each source’s verdict below.
In the paper
“This ad hoc method of throwing out peculiar search terms failed when GFT completely missed the nonseasonal 2009 infl uenza A–H1N1 pandemic ( 2, 14).”
What the sources say
- Partly supportedcites [2] matched to “Reassessing Google Flu Trends Data for Detection of Seasonal and Pandemic Influenza: A Comparative Epidemiological Study at Three Geographic Scales” (2013) sourceExperimental — accuracy not yet measured
The Discussion (page unavailable) says the original GFT model missed the spring 2009 pandemic wave, supporting the claim about its failure to detect the nonseasonal outbreak. It does not say GFT completely missed the entire 2009 pandemic. The Introduction describes automated selection of the original model’s search terms, not an ad hoc method of throwing out peculiar terms. (LLM, conf=0.86)
- Disputedcites [14] S. Cook et al., PLOS ONE 6, e23610 (2011). sourceExperimental — accuracy not yet measured
The Abstract (Methodology/Principal Findings) says the original GFT model underestimated the magnitude of illness during the 2009 H1N1 pandemic, not that it completely missed it. The Abstract (Conclusions) says both models performed well during the pandemic, although the updated model performed better. The Discussion notes that pandemic-specific queries were excluded, but does not describe this as an ad hoc method of throwing out peculiar terms. (LLM, conf=0.88)
Checked by a reasoning model — may be wrong.
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“Oddly, the few search terms offered in the papers ( 14) do not seem to be strongly related with either GFT or the CDC data (SM)—we surmise that the authors felt an unarticulated need to cloak the actual search terms identifi ed.”
What the sources say
- Disputedcites [14] S. Cook et al., PLOS ONE 6, e23610 (2011). sourceExperimental — accuracy not yet measured
The Results section (‘Search Behavior During pH1N1’; page unavailable) says that all three search queries shown in Figure 3 closely tracked CDC ILINet data before the pandemic, and ‘symptoms of flu’ continued to do so during it. The paper does report that the bronchitis and pneumonia queries underestimated ILINet data during the pandemic, but that does not support the broad claim that the offered terms were not strongly related to the data. (LLM, conf=0.87)
Checked by a reasoning model — may be wrong.
hedgedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“The Internet has opened the way for improving standard surveys, experiments, and health reporting ( 35).”
What the sources say
- Partly supportedcites [35] A. J. Berinsky et al., Polit. Anal. 20, 351–368 (2012). sourceExperimental — accuracy not yet measured
Section 2 describes using an Internet platform to run surveys with embedded experiments, and the Conclusion says MTurk can make experimental research easier and less expensive. The paper does not address improvements to health reporting; the disease scenario in section 4.3.2 is a framing experiment, not health reporting. Page numbers are unavailable in our copy. (LLM, conf=0.88)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Although not widely reported until 2013, the new GFT has been persistently overestimating flu prevalence for a much longer time.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“From 21 August 2011 to 1 September 2013, GFT reported overly high fl u prevalence 100 out of 108 weeks.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
3 more claims
Create a free account to see all 14 claims and every source verdict.
- It may state two separate claims as one. 5
- Probably not a substantial claim. 2
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Google Trends predicted more than twice the number of influenza-like illnesses than the CDC in February 2013 46 .”
From Unrepresentative big surveys significantly overestimated US vaccine uptake (2021) · cites this paper as [46]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This was one of the main reasons behind the failure of Google Flu Trends, since the data used for training the model in the first few years was not representative of the trends in subsequent years [15].”
From Theory-Guided Data Science: A New Paradigm for Scientific Discovery from Data (2017) · cites this paper as [15]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Social data research often relies on counting entities. These entities can be users, links, or messages, and the description is a summary of these counts (Lazer et al., 2014; Salganik, 2017). Yet, simple counts can mislead if it is unclear what is counted and how.”
From Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019) · cites this paper as Lazer et al., 2014
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This unintelligibility is also subject to functional biases (section 4.1), as it depends on the mechanisms available on each social platform (e.g., having a like button, but not a dislike one), as well as to variations in platform algorithms and mechanisms in response to users actions (Lazer et al., 2014).”
From Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019) · cites this paper as Lazer et al., 2014
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“It also includes temporal validity, which captures to what extent constructs change over time (Howison et al., 2011) and may invalidate previous conclusions about societal and/or platform phenomena; e.g., see the case of Google Flu Trends (Lazer et al., 2014).”
From Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019) · cites this paper as Lazer et al., 2014
6 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 14 MARCH 2014 VOL 343 SCIENCE www.sciencemag.org 1204
“The 2009 version of GFT has weathered other media panics related to the fl u, including the 2005–2006 influenza A/H5N1 ("bird flu") outbreak”
“GFT engineers updated the algorithm in 2009, and this model has run ever since”
Citations
Cites
boyd, D., & Crawford, K. (2012) · Information, Communication & Society · doi:10.1080/1369118x.2012.678878 · Cited by 5,479
Twitter mood predicts the stock market
Bollen, J., Mao, H., & Zeng, X. (2011) · Journal of Computational Science · doi:10.1016/j.jocs.2010.12.007 · Cited by 5,206
Detecting influenza epidemics using search engine query data
Ginsberg, J., Mohebbi, M. H., Patel, R. S., Brammer, L., Smolinski, M. S., & Brilliant, L. (2009) · Nature · doi:10.1038/nature07634 · Cited by 4,295
Cohen, D. (1998) · The Women's Review of Books · doi:10.2307/4022859 · Cited by 4,261
Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk
Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012) · Political Analysis · doi:10.1093/pan/mpr057 · Cited by 4,008
Big Data : The Management Revolution
マカフィー, ア., ブリニョルフソン, エ., & 有賀, 裕. (2013) · Harvard business review · Cited by 3,549
Lazer, D., Pentland, A., Adamic, L., Aral, S., Barabási, A.-L., Brewer, D., Christakis, N., Contractor, N., Fowler, J., Gutmann, M., et al. (2009) · Science · doi:10.1126/science.1167742 · Cited by 3,365
Self-citationPredicting Elections with Twitter: What 140 Characters Reveal about Political Sentiment
Tumasjan, A., Sprenger, T., Sandner, P., & Welpe, I. (2010) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v4i1.14009 · Cited by 2,649
Multiscale mobility networks and the spatial spreading of infectious diseases
Balcan, D., Colizza, V., Gonçalves, B., Hu, H., Ramasco, J. J., & Vespignani, A. (2009) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0906910106 · Cited by 1,380
Self-citationPredicting consumer behavior with web search
Goel, S., Hofman, J. M., Lahaie, S., Pennock, D. M., & Watts, D. J. (2010) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1005962107 · Cited by 636
Research works printed but not linked: 7
- A. McAfee, E. Brynjolfsson, Harv. Bus. Rev. 90, 60 (2012).
- AAAI Conference on Weblogs and Social Media, Atlanta, Georgia, 11 to 15 July 2010 (Association for Advancement of Artifi cial Intelligence, 2010), pp. 178.
- P. Copeland et al., Int. Soc. Negl. Trop. Dis. 2013, 3
- I. M. Hall et al., Epidemiol. Infect. 135, 372 (2007).
- E. Mustafaraj, P. Metaxas, in Proceedings of the Web- Sci10, Raleigh, NC, 26 and 27 April 2010 (Web Science Trust, 2010); http://journal.webscience.org/317/.
- J. Ratkiewicz et al., in Proceedings of 5th International AAAI Conference on Weblogs and Social Media, San Francisco, CA, 7 to 11 August 2011 (AAAI, 2011), p. 297–304.
- 2013; http://chronicle.com/article/Researchers-Struggle- to-Secure/141591/.
Cited by
Items 1,201–1,300 of 2,513
Concept Drift Adaptive Physical Event Detection for Social Media Streams
Suprem, A., Musaev, A., & Pu, C. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-23381-5_7 · Cited by 14
“However such assumptions, made in Google Flu Trends [9, 10] degrade in the long term.”
BackgroundMethodPoschmann, P., & Goldenstein, J. (2019) · Sociological Methods & Research · doi:10.1177/0049124119882481 · Cited by 13
“Even if big data are not necessarily better data (Lazer et al. 2014), we maintain that already established methodologies, especially for text analysis, can be complemented in a meaningful way and scaled up by using Wikipedia as a data source for demographic information.”
MethodSurvey data and human computation for improved flu tracking
Wojcik, S., Bijral, A. S., Johnston, R., Ferres, J. M. L., King, G., Kennedy, R., Vespignani, A., & Lazer, D. (2021) · Nature Communications · doi:10.1038/s41467-020-20206-z · Cited by 18
Self-citationShan, S., Yan, Q., & Wei, Y. (2020) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph17186853 · Cited by 11
“According to the latest estimate by the United States Centers for Disease Control and Prevention (US-CDC), as many as 650,000 people worldwide die from seasonal flu-related respiratory diseases each year.”
BackgroundMethodInfluentialLee, J., Kwan, Y., Lee, J. Y., Shin, J. I., Lee, K. H., Hong, S. H., Han, Y. J., Kronbichler, A., Smith, L., Koyanagi, A., et al. (2021) · Journal of Medical Internet Research · doi:10.2196/26368 · Cited by 14
Sqalli, M. T., Al‐Thani, D., Qaraqe, M., & Fernández-Luque, L. (2021) · Lecture notes in bioengineering · doi:10.1007/978-3-030-67303-1_4 · Cited by 11
United States Influenza Search Patterns Since the Emergence of COVID-19: Infodemiology Study
Cai, O., & Sousa‐Pinto, B. (2022) · JMIR Public Health and Surveillance · doi:10.2196/32364 · Cited by 11
The updating methods of object-induced three-way concept in dynamic formal contexts
Hu, Q., Qin, K., & Yang, L. (2022) · Applied Intelligence · doi:10.1007/s10489-022-03646-6 · Cited by 10
“…storage technology and the improvement of computer computing ability, the amount of data generated by various industries such as industrial manufacturing, medical and health care, life and entertainment has increased rapidly with the change of time, and the big data era is coming [11, 13, 15].”
BackgroundEarly detection of COVID-19 outbreaks using textual analysis of electronic medical records
Shapiro, M. J., Landau, R., Shay, S., Kaminsky, M., & Verhovsky, G. (2022) · Journal of Clinical Virology · doi:10.1016/j.jcv.2022.105251 · Cited by 9
“Medical text written by trained health care professional is likely to be better structured and less biased, mitigating the set-backs that plagued previously mentioned systems [22].”
McClymont, H., Si, X., & Hu, W. (2023) · Heliyon · doi:10.1016/j.heliyon.2023.e13782 · Cited by 10
“…for error in this context due to media bias, daily media coverage likely influenced trends in information-seeking behaviour throughout the outbreak in Mel-bourne, with high levels of COVID-related searches by unaffected individuals similar to the effects observed during influenza outbreaks [49,50].”
Sun, X. N., Ye, J., & Ren, F. (2016) · Neurocomputing · doi:10.1016/j.neucom.2016.01.107 · Cited by 10
Does Big Data Lead to Smarter Cities? Problems, Pitfalls and Opportunities
Batty, M. (2015) · The Knowledge Bank (The Ohio State University) · Cited by 10
Arendt, F. (2017) · Crisis · doi:10.1027/0227-5910/a000467 · Cited by 12
“Second, Google Trends data have well-known limitations (Lazer et al., 2014) such as there being a difference between the social structure of web users and the real (offline) population.”
BackgroundMa‐Kellams, C., Bishop, B., Zhang, M. F., & Villagrana, B. (2017) · Psychological Reports · doi:10.1177/0033294117736318 · Cited by 14
“Taken together, the existing literature suggests that Google search is a useful but less than perfect predictor (Butler, 2013; Lazer et al., 2014; Yang et al., 2015).”
BackgroundData, tools and resources for mining social media drug chatter
Sarker, A., & Gonzalez‐Hernandez, G. (2016) · International Conference on Computational Linguistics · Cited by 9
The Role of an Artificial Intelligence Ecosystem in Radiology
Allen, B., Gish, R., & Dreyer, K. (2019) · Springer eBooks · doi:10.1007/978-3-319-94878-2_19 · Cited by 11
Big Data – What is it and why it matters
Tattersall, A., & Grant, M. J. (2016) · Health Information & Libraries Journal · doi:10.1111/hir.12147 · Cited by 10
Artificial intelligence versus COVID-19 in developing countries: Priorities and trade-offs
Naudé, W. A. (2020) · doi:10.35188/unu-wider/wbn/2020-4 · Cited by 10
Xiang, Y., Jiang, Q., Zhang, Y., & Zhou, W. (2022) · Buildings · doi:10.3390/buildings13010100 · Cited by 14
The measurement errors of google trends data
Liu, K. (2024) · Discover Data · doi:10.1007/s44248-024-00013-3 · Cited by 9
“To the author’s knowledge, Lazer et al. [6] is the earliest study that has discussed this issue.”
InfluentialWilde, J., Chen, W., Lohmann, S., et al. (2024) · Population and Development Review · doi:10.1111/padr.12647 · Cited by 9
Zhou, H.-Y., Li, Y., Li, J., Meng, J., & Wu, A. (2024) · The Innovation Medicine · doi:10.59717/j.xinn-med.2024.100091 · Cited by 18
Causality-Guided Deep learning for streamflow predicting in a mountainous region
Tang, X., Qin, G., Wu, X., Zhao, Y., & Li, H. (2025) · Journal of Hydrology · doi:10.1016/j.jhydrol.2025.132719 · Cited by 13
Zulli, A., Zhang, Z., Ruedaflores, M., Sahly, J., Angel, D. M., Rohatgi, K., Malik, W., Hao, R. M., Shepherd, J., & Peccia, J. (2025) · Environmental Science & Technology · doi:10.1021/acs.est.4c05723 · Cited by 9
Adaptive Process-Guided Learning: An Application in Predicting Lake DO Concentrations
Yu, R., Qiu, C., Ladwig, R., Hanson, P. C., Xie, Y., Li, Y., & Jia, X. (2024) · Proceedings · doi:10.1109/icdm59182.2024.00065 · Cited by 11
Nurturing Capabilities: Unpacking the Gap in Human-Centered Evaluations of AI-Based Systems
Khullar, A., Nalin, N., Prasad, A., Mampilli, A. J., & Kumar, N. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713278 · Cited by 16
“Verification workers—who verify the veracity of the output of an AI model [107]—are critical in a high-stakes domain like public health to mitigate AI’s innate fallibility [7, 42, 59] from mitigating harms like non-evidentiary health-related information [7].”
The Hitchhiker's Guide to the Galaxy of Dynamic Ridesharing
Bonazzi, R., & Pigneur, Y. (2015) · Proceedings of the ... Annual Hawaii International Conference on System Sciences/Proceedings of the Annual Hawaii International Conference on System Sciences · doi:10.1109/hicss.2015.146 · Cited by 9
Zhang, Y., Arab, A., Cowling, B. J., & Stoto, M. A. (2014) · BMC Public Health · doi:10.1186/1471-2458-14-850 · Cited by 14
Kmetty, Z., Koltai, J., Bokányi, E., & Bozsonyi, K. (2017) · Intersections · doi:10.17356/ieejsp.v3i1.302 · Cited by 9
“Because of the reasons above, Twitter data is not really suitable for the exact measurement of social phenomena or making predictions (Gayo-Avello, 2012; Lazer et al., 2014), but it can be considered more of a measurement of latent mood.”
BackgroundThe Pearls and Perils of Google Trends: A Housing Market Application
Steegmans, J. (2021) · Big Data · doi:10.1089/big.2020.0183 · Cited by 9
“1204), and they demonstrate these risks for the discontinued Google Flu Trends.(20) Google Flu Trends was a separate service offered by Google, aiming at predicting influenza outbreaks, and it should”
InfluentialReal-time clinician text feeds from electronic health records
Teo, J., Dinu, V., Bernal, W. E., Davidson, P., Oliynyk, V., Breen, C. P., Barker, R. D., & Dobson, R. (2021) · npj Digital Medicine · doi:10.1038/s41746-021-00406-7 · Cited by 9
Testing the Relationship Between Information and Knowledge in Computer-Aided Decision-Making
Li, Y., & Kettinger, W. J. (2021) · Information Systems Frontiers · doi:10.1007/s10796-021-10205-w · Cited by 8
Methodological issues related to the use of online labour market data
Fabo, B., & Kureková, L. M. (2022) · International Labour Office eBooks · doi:10.54394/zzbc8484 · Cited by 16
“Google Flu Trends is a good example: for a considerable time, it predicted the actual doctor visits fairly well, but strongly overestimated the growth of infections, in reaction to a flurry of media reports about a flu pandemic causing people to search for flu symptoms even when not feeling sick (Lazer et al. 2014).”
BackgroundHou, M., Wei, J., Chu, H., Shi, Y., Ayantobo, O. O., Xu, J., Zhu, X., & Ren, Y. Q. (2022) · Journal of Hydrology Regional Studies · doi:10.1016/j.ejrh.2022.101271 · Cited by 10
Favaretto, M., Clercq, E. D., Caplan, A. L., & Elger, B. S. (2023) · PLoS ONE · doi:10.1371/journal.pone.0280542 · Cited by 15
“However, scholars argued that the lack of transparency of Google regarding their supporting materials and methods presented a barrier to replicability for researchers outside of the company that prevented the initial vision of GFT from being developed and perfected into a more accurate or even working model for flu pr…”
BackgroundAn introductory guide for conducting psychological research with big data.
Vezzoli, M., & Zogmaister, C. (2023) · Psychological Methods · doi:10.1037/met0000513 · Cited by 12
Ma, M. Z., Chen, S. X., & Wang, X. (2023) · Applied Psychology Health and Well-Being · doi:10.1111/aphw.12519 · Cited by 9
Değer, Z. T., Taşkın, G., & Wallace, J. W. (2024) · Bulletin of Earthquake Engineering · doi:10.1007/s10518-024-01968-z · Cited by 12
“This issue has been highlighted in various scientific and engineering applications, where black-box models have led to misleading conclusions (Lazer et al. 2014; Douglas and Aochi 2008; Karpatne et al. 2022).”
Drumright, L. N., Frost, S. D. W., Elliot, A. J., Catchpole, M., Pebody, R. G., Atkins, M., Harrison, J., Parker, P., & Holmes, A. H. (2015) · BMC Infectious Diseases · doi:10.1186/s12879-015-0789-z · Cited by 9
“[25].”
Shifting Perspectives: A Process Model for Sense Making Under Uncertainty
Hill, G. E., Datta, P. M., & Acar, W. (2015) · International Journal of Strategic Decision Sciences · doi:10.4018/ijsds.2015010103 · Cited by 11
Predicting well-being with geo-referenced data collected from social media platforms
Loff, J. F., Reis, M., & Martins, B. (2015) · ACM Symposium on Applied Computing · doi:10.1145/2695664.2695939 · Cited by 9
“monplace, many recent studies have also highlighted the fact that the quality of stand-alone monitors, built from social media data, can be somewhat questionable [15, 12].”
BackgroundSpatio-temporal filtering techniques for the detection of disaster-related communication
Fitzhugh, S. M., Gibson, C. B., Spiro, E. S., & Butts, C. T. (2016) · Social Science Research · doi:10.1016/j.ssresearch.2016.04.023 · Cited by 10
“Failure to understand the social processes underlying activity observed at large scale is dangerous andmay lead tomisleading or spurious results (Back et al., 2011; Boyd and Crawford, 2012; Johnson, 2014; Lazer et al., 2014; Leinweber, 2007), such as misclassifying failure-to-connect error messages as “anger” messages…”
BackgroundTsunami databases: The problems of acceptance and absence
GOFF, J. R., & Cain, G. (2016) · Geoforum · doi:10.1016/j.geoforum.2016.09.005 · Cited by 10
Vokorokos, L., Uchnár, M., & Baláž, A. (2017) · IEEE International Conference on Intelligent Engineering Systems (INES) · doi:10.1109/ines.2017.8118530 · Cited by 9
“MongoDB database system does not have firm scheme [6], [7], so it is not easy to map the data structure in database if there are many documents.”
BackgroundPence, J., Sun, Y., Zhu, X., et al. (2017) · Adelaide Research & Scholarship (AR&S) (University of Adelaide) · Cited by 9
Not in my back yard: Egocentrism and climate change skepticism across the globe
Pelham, B. W. (2018) · Environmental Science & Policy · doi:10.1016/j.envsci.2018.09.004 · Cited by 10
Ethical and Statistical Considerations in Models of Moral Judgments
Sivill, T. (2019) · Frontiers in Robotics and AI · doi:10.3389/frobt.2019.00039 · Cited by 12
“However, recent studies have warned of the dangers of using unqualified big data to hype experimental results (Lazer et al., 2014).”
BackgroundBig Data and Trust in Public Policy Automation
Waggoner, P., Kennedy, R., Le, H., & Shiran, M. (2019) · Statistics Politics and Policy · doi:10.1515/spp-2019-0005 · Cited by 12
“The increase in the scale and resolution of data available is important for a range of reasons (Lazer and Radford 2017), but, at the same time, research suggests that “big data hubris” has led some scholars to view big data as allowing them to ignore issues like the validity and reliability of their data (Lazer et al.…”
BackgroundSelf-citationPrecision global health: a roadmap for augmented action
Sheath, D., de Castañeda, R. R., Bempong, N.-E., Raviglione, M. C., Machalaba, C. C., Pepper, M. S., Vayena, E., Ray, N., Wernli, D., Escher, G., et al. (2020) · Journal of Public Health and Emergency · doi:10.21037/jphe.2020.01.01 · Cited by 10
“For example, the huge overestimation of flu cases in the 2012– 2013 season in New York City by Google Flu Trends was partly attributed to a failure in the programme’s algorithms to integrate the search behaviour of its users (14).”
BackgroundCan Google Trends and Wikipedia help traditional surveillance? A pilot study on measles.
Santangelo, O. E., Provenzano, S., Grigis, D., et al. (2020) · Acta bio-medica dell'Ateneo parmense/Acta biomedica · doi:10.23750/abm.v91i4.8888 · Cited by 9
Возможности и ограничения цифровых следов и методов машинного обучения в социологии
Богданов, М. Б., & Smirnov, I. (2021) · Monitoring obŝestvennogo mneniâ: èkonomičeskie i socialʹnye peremeny · doi:10.14515/monitoring.2021.1.1760 · Cited by 15
The effect of drone strikes on civilian communication: evidence from Yemen
Christia, F., Zoumpoulis, S. I., Freedman, M., Yao, L., & Jadbabaie, A. (2021) · Political Science Research and Methods · doi:10.1017/psrm.2021.22 · Cited by 11
“But leveraging CDRs to study conflict is both novel and, we believe, a promising way to shed light on the impact of opaque phenomena, such as drone strikes, on civilians in data poor contexts (Lazer et al ., 2014; Bertolotti et al ., 2020).”
BackgroundWill I die of coronavirus? Google Trends data reveal that politics determine virus fears
Timoneda, J. C., & Vera, S. V. (2021) · PLoS ONE · doi:10.1371/journal.pone.0258189 · Cited by 9
Predictive Analytics in Clinical Practice: Advantages and Disadvantages
Mijderwijk, H.-J., & Steiger, H. (2021) · Acta neurochirurgica. Supplementum · doi:10.1007/978-3-030-85292-4_30 · Cited by 10
Convexity arbitrage – the idea which does not work
Stádník, B. (2022) · Cogent Economics & Finance · doi:10.1080/23322039.2021.2019361 · Cited by 9
“…statistical arbitrage has attracted renewed interest in emerging areas such as bitcoin (Payne and Tresl (2015), Lintilhac and Tourin (2017)), big data (McAfee et al. (2012), Lazer et al. (2014), Chaboud et al. (2014), and Nardo et al. (2016)) and factor investing (Maeso and Martellini (2017)).”
BackgroundThe Use of Big Data Analytics in Medical Applications
Pasupuleti, M. B. (2016) · Malaysian Journal of Medical and Biological Research · doi:10.18034/mjmbr.v3i2.615 · Cited by 11
“Our most difficult healthcare challenges may be directed by using the effect of data from big data population health management (Lazer, 2014) and other sources.”
BackgroundReview content type and hotel review helpfulness: direct and moderating effects
Liu, R., Zhang, Y., Luo, C., Tan, S., & Gong, Y. (2023) · Information Technology and Management · doi:10.1007/s10799-023-00392-0 · Cited by 11
“Second, we used an LDA approach to extract dimensions from helpful reviews, but big data may give misleading information [49].”
Sifting the sand on the river bank: Social media as a source for research data
Metaxas, P., & Mustafaraj, E. (2014) · it - Information Technology · doi:10.1515/itit-2014-1047 · Cited by 11
Schermann, M., Hemsen, H., Buchmüller, C., Bitter, T., Krcmar, H., Markl, V., & Hoeren, T. (2014) · Wirtschaftsinf. · doi:10.1007/s11576-014-0434-2 · Cited by 3,583
“So hat Google Flu Trends zum Teil das Ausmaß von Grippewellen um 50 Prozent überschätzt und nichtsaisonale Grippeausbrüche gar nicht erkannt (Lazer et al. 2014).”
BackgroundInfluentialThe GDPR beyond Privacy: Data-Driven Challenges for Social Scientists, Legislators and Policy-Makers
Vestoso, M. (2018) · Future Internet · doi:10.3390/fi10070062 · Cited by 14
“On the other hand, even when data are available in large amounts and can be processed by means of innovative techniques, predictions led by data correlations risk missing the chance of explaining the causal dynamics of a phenomenon, increasing the possibility of making rough errors [69].”
BackgroundMacro Forecasting Using Alternative Data
Jain, A. (2018) · Elsevier eBooks · doi:10.1016/b978-0-12-813524-2.00011-1 · Cited by 8
Berke, O., Trotz‐Williams, L. A., & de Montigny, S. (2020) · Canada Communicable Disease Report · doi:10.14745/ccdr.v46i06a07 · Cited by 10
Chlaily, S., Mura, M. D., Chanussot, J., et al. (2020) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2020.3014138 · Cited by 9
“Nonetheless, it has been proved, either in remote sensing [4], [5] or in other data science research fields [3], [10], that processing more data and records does not always result in detailed information extraction because of nonidealities, mismatches, and estimation imperfections [5], [11], [12].”
BackgroundGuo, X., Xiong, N. N., Wang, H., & Ren, J. (2021) · IEEE Transactions on Systems Man and Cybernetics Systems · doi:10.1109/tsmc.2020.3048946 · Cited by 10
“high ILI (in the interval of 2011-2013) have led to doubts about the utility of these data [11], [12].”
BackgroundDoes Online Search Predict Sales? Evidence from Big Data for Car Markets in Germany and the UKPreprint
von Graevenitz, G., Helmers, C., Millot, V., & Turnbull, O. D. N. (2016) · SSRN Electronic Journal · doi:10.2139/ssrn.2832004 · Cited by 12
Lessons from Nowcasting GDP across the World
Cascaldi-Garcia, D., Luciani, M., & Modugno, M. (2023) · International Finance Discussion Paper · doi:10.17016/ifdp.2023.1385 · Cited by 8
Behnert, J., Lajic, D., & Bauer, P. C. (2024) · Journal Of Big Data · doi:10.1186/s40537-023-00868-4 · Cited by 8
Diagnostics for Public Health — Infectious Disease Surveillance and Control
Lipsitch, M., & Grad, Y. H. (2024) · NEJM Evidence · doi:10.1056/evidra2300271 · Cited by 6
Aronis, J. M., Ye, Y., Espino, J. U., Hochheiser, H., Michaels, M. G., & Cooper, G. F. (2024) · JMIR Public Health and Surveillance · doi:10.2196/57349 · Cited by 8
Big Data Sharing: A Comprehensive Survey
Jiang, S. (2025) · Data · doi:10.3390/data10110182 · Cited by 13
“A paradigmatic illustration of big data’s economic and epidemiological value is Google LLC’s now-classic 2009 influenza study [35].”
Processing unstructured documents and social media using Big Data techniques
Dıaconıța, V. (2015) · Economic Research-Ekonomska Istraživanja · doi:10.1080/1331677x.2015.1095110 · Cited by 8
ExpoQual: Evaluating measured and modeled human exposure data
LaKind, J. S., O’Mahony, C., Armstrong, T. W., et al. (2019) · Environmental Research · doi:10.1016/j.envres.2019.01.039 · Cited by 8
Fortunati, L., Edwards, A. P., Manganelli, A. M., Edwards, C. C., & de Luca, F. (2022) · Human-Machine Communication · doi:10.30658/hmc.5.3 · Cited by 16
An Insight into the State of Big Data Research: A Bibliometric Study of Scientific Publications
Islam, M. N., & Hu, G. (2023) · Science & Technology Libraries · doi:10.1080/0194262x.2023.2185919 · Cited by 11
Porcu, G., Chen, Y. X., Bonaugurio, A. S., Villa, S., Riva, L., Messina, V., Bagarella, G., Maistrello, M., Leoni, O., Cereda, D., et al. (2023) · Frontiers in Public Health · doi:10.3389/fpubh.2023.1141688 · Cited by 10
Unveiling Venice’s hotels competition networks from dynamic pricing digital market
Armillotta, M., Fokianos, K., & Guizzardi, A. (2023) · Journal of the Royal Statistical Society Series A (Statistics in Society) · doi:10.1093/jrsssa/qnad085 · Cited by 11
Predicting COVID-19 new cases in California with Google Trends data and a machine learning approach
Habibdoust, A., Seifaddini, M., Tatar, M., Araz, Ö. M., & Wilson, F. A. (2024) · Informatics for Health and Social Care · doi:10.1080/17538157.2024.2315246 · Cited by 11
“3 17 Although these predictions sometimes were inaccurate, 14,17 they can provide timely and accurate estimates of influenza activity, especially during peak activity, in the United States.”
BackgroundUse of Internet Search Logs to Evaluate Potential Drug Adverse Events
Sarntivijai, S., & Abernethy, D. R. (2014) · Clinical Pharmacology & Therapeutics · doi:10.1038/clpt.2014.115 · Cited by 8
“7 Lazer and colleagues note that use of Internet search logs has substantial limitations, in both the analytical search algorithms used, which are modified for various reasons over time, and the implicit assumption that the incorporation of more data into an analysis will somehow improve its…”
BackgroundRichterich, A. (2016) · Law, governance and technology series · doi:10.1007/978-3-319-33525-4_3 · Cited by 7
Investigating an Architectural Framework for Small Data Platforms
Thinyane, M. (2017) · UNU Collections (United Nations University) · Cited by 8
Chan, M. S., Lohmann, S., Morales, A., Zhai, C., Ungar, L., Holtgräve, D. R., & Albarracín, D. (2018) · AIDS and Behavior · doi:10.1007/s10461-018-2046-0 · Cited by 8
“Furthermore, people are likely to communicate on social media using informal, constantly changing writing conventions that suit their needs, culture, and idiosyncrasy [40, 41] and that can emerge in ways that researchers and practitioners cannot foresee.”
BackgroundDoes Twitter language reliably predict heart disease? A commentary on Eichstaedt et al. (2015a)
Brown, N. J. L., & Coyne, J. (2018) · PeerJ · doi:10.7717/peerj.5656 · Cited by 5
Cristea, A. I., Alamri, A., Kayama, M., Stewart, C. D., Alshehri, M., & Shi, L. (2018) · Durham Research Online (Durham University) · doi:10.25405/ncl.34464930 · Cited by 7
Sun, Y., Jin, Q., Cheng, Q., & Guo, K. (2019) · Industrial Management & Data Systems · doi:10.1108/imds-03-2019-0125 · Cited by 12
What does digitalization hold for the creation of real-world evidence?
Pisaniello, H. L., & Dixon, W. G. (2019) · Rheumatology · doi:10.1093/rheumatology/kez068 · Cited by 12
Johnson, W. G. (2020) · The Journal of Law Medicine & Ethics · doi:10.1177/1073110520979374 · Cited by 14
Global Dieting Trends and Seasonality: Social Big-Data Analysis May Be a Useful Tool
Park, M.-B., Wang, J., & Bulwer, B. E. (2021) · Nutrients · doi:10.3390/nu13041069 · Cited by 11
“Big-data analyses have limitations, but upgrades and improvements are constantly provided with the aim of improving accuracy and precision [16,17].”
BackgroundA research and policy agenda for the post-pandemic world
Yiu, C., Macon-Cooney, B., & Fingerhut, H. (2021) · Future Healthcare Journal · doi:10.7861/fhj.2021-0082 · Cited by 9
Big Data: Challenges and opportunities for clinical pharmacology
Flockhart, D., Bies, R. R., Gastonguay, M., & Schwartz, S. (2016) · British Journal of Clinical Pharmacology · doi:10.1111/bcp.12896 · Cited by 9
Prediction Using Propagation: From Flu Trends to Cybersecurity
Prakash, B. A. (2016) · IEEE Intelligent Systems · doi:10.1109/mis.2016.1 · Cited by 8
Going Beyond GDP to Nowcast Well-Being Using Retail Market Data
Guidotti, R., Coscia, M., Pedreschi, D., et al. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-28361-6_3 · Cited by 8
“Such studies are not exempt from criticisms: [18] proved that nowcasting with Google queries alone is not enough and the data must be integrated with other models.”
BackgroundSoziale Medien in der empirischen Forschung
Zeller, F. (2016) · Springer eBooks · doi:10.1007/978-3-658-03765-9_21 · Cited by 10
Hybrid Model Based Influenza Detection with Sentiment Analysis from Social Networks
Sun, X. N., Ye, J., & Ren, F. (2015) · Communications in computer and information science · doi:10.1007/978-981-10-0080-5_5 · Cited by 8
Buying Drugs Online in the Age of Social Media
Bachhuber, M. A., & Merchant, R. M. (2017) · American Journal of Public Health · doi:10.2105/ajph.2017.304137 · Cited by 8
Digital Data-Centric Geography: Implications for Geography's Frontier
Bowlick, F. J., & Wright, D. J. (2018) · The Professional Geographer · doi:10.1080/00330124.2018.1443478 · Cited by 9
“Big data and analytical products from such data are not panaceas and clear solutions on their own, however; instead, they are avenues toward decision making, not clear decisions themselves (Lazer et al. 2014).”
BackgroundBig data: what it can and cannot achieve
Schofield, P., & Das‐Munshi, J. (2018) · BJPsych Advances · doi:10.1192/bja.2018.15 · Cited by 7
Research on Escaping the Big-Data Traps in O2O Service Recommendation Strategy
Xue, X., Huangfu, S., Zhang, L., & Wang, S. (2019) · IEEE Transactions on Big Data · doi:10.1109/tbdata.2019.2915798 · Cited by 8
Extraordinary issue: Coronavirus, crisis and communication
Athique, A. M. (2020) · Media International Australia · doi:10.1177/1329878x20960300 · Cited by 10
“Things went quiet as it became apparent that recording one outbreak in detail through proxy inputs does not provide predictive capacity (Lazer et al, 2014).”
BackgroundA method to evaluate the reliability of social media data for social network analysis
Weber, D., Nasim, M., Mitchell, L., & Falzon, L. (2020) · IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM) · doi:10.1109/asonam49781.2020.9381461 · Cited by 10
“from a sampling perspective [1]–[4], biases [5]–[8] and the danger of making invalid generalisations using “big data” approaches lacking nuanced interpretation of the data [9], [10].”
BackgroundSarkar, S., Ghosh, K., & Petter, S. (2020) · Communications of the Association for Information Systems · doi:10.17705/1cais.04705 · Cited by 15
Metadata
- Authors
- Lazer, D., Kennedy, R., King, G., & Vespignani, A.
- Affiliations
- Northeastern University (US)
- Harvard University (US)
- Corresponding author
- Lazer, David
- Venue
- Science
- Published
- 2014-03-14
- Volume
- 343
- Issue
- 6176
- Pages
- 1203-1205
- ISSN
- 0036-8075 (ISSN-L)
- 1095-9203
- OpenAlex
- W2068181924
- Semantic Scholar
- 00d681cfff7f0224ace607e515f4ed510e792df9
- Cited by
- 2,523
- References
- 30
- Citations by year
- 2026: 113 · 2025: 101 · 2024: 121 · 2023: 150 · 2022: 168 · 2021: 229 · 2020: 273 · 2019: 261 · 2018: 272 · 2017: 249 · 2016: 271 · 2015: 219 · 2014: 91
- FWCI
- 208.16
- Type
- journal article
- Language
- en
- Open access
- green
- dash.harvard.edu
- Licence
- cc-by
- Locations
- Science (publishedVersion)
- PubMed (publishedVersion)
- Digital Access to Scholarship at Harvard (DASH) (Harvard University) (submittedVersion, cc-by)
- Topics
- Health Sciences › Medicine › Epidemiology › Data-Driven Disease Surveillance
- Social Sciences › Decision Sciences › Information Systems and Management › Big Data Technologies and Applications
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- Google Flu Trends, influenza surveillance, big data, prediction error, data quality
- MeSH terms
- Centers for Disease Control and Prevention, U.S., Data Collection, Data Interpretation, Statistical, Disease Outbreaks, Humans, Influenza, Human, Office Visits, Prognosis, United States, Social Media
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.