The Parable of Google Flu: Traps in Big Data Analysis
Open accessThe DOI matches its Crossref record; 30 of the 31 references it cites were found in a registry or an index; 5 of the 8 sources its claims cite were checked for support.
Abstract
Large errors in flu prediction were largely avoidable, which offers lessons for the use of big data.
Overview
CoChat Summary
TL;DR
The paper uses the failure of Google Flu Trends (GFT) to illustrate two traps in big data analysis: “big data hubris” and algorithm dynamics. It argues that GFT’s large flu-prediction errors were largely avoidable and that big data should supplement, not replace, traditional data and methods.
Key findings
- In February 2013, GFT predicted more than double the proportion of doctor visits for influenza-like illness (ILI) than the CDC; from 21 August 2011 to 1 September 2013, GFT reported overly high flu prevalence in 100 out of 108 weeks, and it overshot the 2011–2012 season by more than 50%.
- GFT errors were not random: they showed temporal autocorrelation and seasonality, meaning last week’s errors predicted this week’s errors and error direction/magnitude varied with time of year.
- Simple alternatives outperformed GFT: out-of-sample mean absolute error was 0.486 for GFT, 0.311 for lagged CDC, and 0.232 for combined GFT and CDC; all differences were statistically significant at P < 0.05. Even 3-week-old CDC data did better than GFT at projecting current flu prevalence.
- The authors attribute GFT’s problems to overfitting (initial GFT matched 50 million search terms to 1152 data points and missed the nonseasonal 2009 H1N1 pandemic) and to “algorithm dynamics,” including Google search algorithm changes and user behavior changes.
Methods
- Design: policy forum/case study comparing GFT estimates with CDC ILI surveillance data and alternative forecasting models.
- Data/sample: GFT and CDC flu prevalence estimates, including the period from 21 August 2011 to 1 September 2013; the initial GFT methodology reportedly fit 50 million search terms to 1152 data points.
- Main analysis: comparison of GFT alone, lagged CDC data with 52-week seasonality variables, and a combined GFT + CDC model using lagged CDC estimates, lagged GFT error, and seasonality variables; reported mean absolute error and statistical significance; Google Correlate used to examine correlated search terms. Supplementary materials contain data and methods.
Limitations
- The paper states that GFT never documented the 45 search terms used, released examples appear misleading, and Google Correlate is limited to national-level data whereas GFT was developed using regional correlations; therefore the original GFT analyses cannot be replicated from the information provided.
- The paper states that Google cannot make its full data available to outsiders because of privacy issues, though it argues there is no such constraint on derivative, aggregated data.
- Evident from the text: this is a commentary/policy forum rather than a full empirical article, with key data and methods placed in supplementary materials; the quantitative comparison depends on GFT and CDC data and model specifications not fully detailed in the main text.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
2 Valid · 2 Not valid · 2 Not checked · 8 Own results and interpretations
Some cited sources support this claim and some don’t — read each source’s verdict below.
In the paper
“This ad hoc method of throwing out peculiar search terms failed when GFT completely missed the nonseasonal 2009 infl uenza A–H1N1 pandemic ( 2, 14).”
What the sources say
- Partly supportedcites [2] matched to “Reassessing Google Flu Trends Data for Detection of Seasonal and Pandemic Influenza: A Comparative Epidemiological Study at Three Geographic Scales” (2013) sourceExperimental — accuracy not yet measured
The Discussion (page unavailable) says the original GFT model missed the spring 2009 pandemic wave, supporting the claim about its failure to detect the nonseasonal outbreak. It does not say GFT completely missed the entire 2009 pandemic. The Introduction describes automated selection of the original model’s search terms, not an ad hoc method of throwing out peculiar terms. (LLM, conf=0.86)
- Disputedcites [14] S. Cook et al., PLOS ONE 6, e23610 (2011). sourceExperimental — accuracy not yet measured
The Abstract (Methodology/Principal Findings) says the original GFT model underestimated the magnitude of illness during the 2009 H1N1 pandemic, not that it completely missed it. The Abstract (Conclusions) says both models performed well during the pandemic, although the updated model performed better. The Discussion notes that pandemic-specific queries were excluded, but does not describe this as an ad hoc method of throwing out peculiar terms. (LLM, conf=0.88)
Checked by a reasoning model — may be wrong.
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“Oddly, the few search terms offered in the papers ( 14) do not seem to be strongly related with either GFT or the CDC data (SM)—we surmise that the authors felt an unarticulated need to cloak the actual search terms identifi ed.”
What the sources say
- Disputedcites [14] S. Cook et al., PLOS ONE 6, e23610 (2011). sourceExperimental — accuracy not yet measured
The Results section (‘Search Behavior During pH1N1’; page unavailable) says that all three search queries shown in Figure 3 closely tracked CDC ILINet data before the pandemic, and ‘symptoms of flu’ continued to do so during it. The paper does report that the bronchitis and pneumonia queries underestimated ILINet data during the pandemic, but that does not support the broad claim that the offered terms were not strongly related to the data. (LLM, conf=0.87)
Checked by a reasoning model — may be wrong.
hedgedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“The Internet has opened the way for improving standard surveys, experiments, and health reporting ( 35).”
What the sources say
- Partly supportedcites [35] A. J. Berinsky et al., Polit. Anal. 20, 351–368 (2012). sourceExperimental — accuracy not yet measured
Section 2 describes using an Internet platform to run surveys with embedded experiments, and the Conclusion says MTurk can make experimental research easier and less expensive. The paper does not address improvements to health reporting; the disease scenario in section 4.3.2 is a framing experiment, not health reporting. Page numbers are unavailable in our copy. (LLM, conf=0.88)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Although not widely reported until 2013, the new GFT has been persistently overestimating flu prevalence for a much longer time.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“From 21 August 2011 to 1 September 2013, GFT reported overly high fl u prevalence 100 out of 108 weeks.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
3 more claims
Create a free account to see all 14 claims and every source verdict.
- It may state two separate claims as one. 5
- Probably not a substantial claim. 2
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Google Trends predicted more than twice the number of influenza-like illnesses than the CDC in February 2013 46 .”
From Unrepresentative big surveys significantly overestimated US vaccine uptake (2021) · cites this paper as [46]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This was one of the main reasons behind the failure of Google Flu Trends, since the data used for training the model in the first few years was not representative of the trends in subsequent years [15].”
From Theory-Guided Data Science: A New Paradigm for Scientific Discovery from Data (2017) · cites this paper as [15]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Social data research often relies on counting entities. These entities can be users, links, or messages, and the description is a summary of these counts (Lazer et al., 2014; Salganik, 2017). Yet, simple counts can mislead if it is unclear what is counted and how.”
From Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019) · cites this paper as Lazer et al., 2014
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This unintelligibility is also subject to functional biases (section 4.1), as it depends on the mechanisms available on each social platform (e.g., having a like button, but not a dislike one), as well as to variations in platform algorithms and mechanisms in response to users actions (Lazer et al., 2014).”
From Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019) · cites this paper as Lazer et al., 2014
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“It also includes temporal validity, which captures to what extent constructs change over time (Howison et al., 2011) and may invalidate previous conclusions about societal and/or platform phenomena; e.g., see the case of Google Flu Trends (Lazer et al., 2014).”
From Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019) · cites this paper as Lazer et al., 2014
6 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 14 MARCH 2014 VOL 343 SCIENCE www.sciencemag.org 1204
“The 2009 version of GFT has weathered other media panics related to the fl u, including the 2005–2006 influenza A/H5N1 ("bird flu") outbreak”
“GFT engineers updated the algorithm in 2009, and this model has run ever since”
Citations
Cites
boyd, D., & Crawford, K. (2012) · Information, Communication & Society · doi:10.1080/1369118x.2012.678878 · Cited by 5,479
Twitter mood predicts the stock market
Bollen, J., Mao, H., & Zeng, X. (2011) · Journal of Computational Science · doi:10.1016/j.jocs.2010.12.007 · Cited by 5,206
Detecting influenza epidemics using search engine query data
Ginsberg, J., Mohebbi, M. H., Patel, R. S., Brammer, L., Smolinski, M. S., & Brilliant, L. (2009) · Nature · doi:10.1038/nature07634 · Cited by 4,295
Cohen, D. (1998) · The Women's Review of Books · doi:10.2307/4022859 · Cited by 4,261
Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk
Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012) · Political Analysis · doi:10.1093/pan/mpr057 · Cited by 4,008
Big Data : The Management Revolution
マカフィー, ア., ブリニョルフソン, エ., & 有賀, 裕. (2013) · Harvard business review · Cited by 3,549
Lazer, D., Pentland, A., Adamic, L., Aral, S., Barabási, A.-L., Brewer, D., Christakis, N., Contractor, N., Fowler, J., Gutmann, M., et al. (2009) · Science · doi:10.1126/science.1167742 · Cited by 3,365
Self-citationPredicting Elections with Twitter: What 140 Characters Reveal about Political Sentiment
Tumasjan, A., Sprenger, T., Sandner, P., & Welpe, I. (2010) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v4i1.14009 · Cited by 2,649
Multiscale mobility networks and the spatial spreading of infectious diseases
Balcan, D., Colizza, V., Gonçalves, B., Hu, H., Ramasco, J. J., & Vespignani, A. (2009) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0906910106 · Cited by 1,380
Self-citationPredicting consumer behavior with web search
Goel, S., Hofman, J. M., Lahaie, S., Pennock, D. M., & Watts, D. J. (2010) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1005962107 · Cited by 636
Research works printed but not linked: 7
- A. McAfee, E. Brynjolfsson, Harv. Bus. Rev. 90, 60 (2012).
- AAAI Conference on Weblogs and Social Media, Atlanta, Georgia, 11 to 15 July 2010 (Association for Advancement of Artifi cial Intelligence, 2010), pp. 178.
- P. Copeland et al., Int. Soc. Negl. Trop. Dis. 2013, 3
- I. M. Hall et al., Epidemiol. Infect. 135, 372 (2007).
- E. Mustafaraj, P. Metaxas, in Proceedings of the Web- Sci10, Raleigh, NC, 26 and 27 April 2010 (Web Science Trust, 2010); http://journal.webscience.org/317/.
- J. Ratkiewicz et al., in Proceedings of 5th International AAAI Conference on Weblogs and Social Media, San Francisco, CA, 7 to 11 August 2011 (AAAI, 2011), p. 297–304.
- 2013; http://chronicle.com/article/Researchers-Struggle- to-Secure/141591/.
Cited by
Items 1,801–1,900 of 2,513
Search engines, news wires and digital epidemiology: Presumptions and facts
Kaveh-Yazdy, F., & Zareh-Bidoki, A.-M. (2018) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2018.03.017 · Cited by 2
A Study on Delivery Evaluation under Asymmetric Information in the Mail-order Industry
Takahashi, M., Azuma, H., & Tsuda, K. (2018) · Procedia Computer Science · doi:10.1016/j.procs.2018.08.079 · Cited by 2
The Role of Event-Based Biosurveillance in Biodefense
Hartley, D., Mui, W.-L., & Corley, C. D. (2019) · Springer eBooks · doi:10.1007/978-3-030-03053-7_3 · Cited by 2
Naudé, W. A. (2020) · Perspektiven der Wirtschaftspolitik · doi:10.1515/pwp-2020-0021 · Cited by 5
Lawrence, H., Robertson, C., Feick, R., & NELSON, T. A. (2020) · ISPRS International Journal of Geo-Information · doi:10.3390/ijgi9090497 · Cited by 3
Die Renaissance der „Unobtrusive Methods“ im digitalen Zeitalter
Diekmann, A. (2020) · Springer eBooks · doi:10.1007/978-3-658-15629-9_8 · Cited by 2
Characterizing Online Media on COVID-19 during the Early Months of the Pandemic
Dambanemuya, H. K., Lin, H., & Horvát, Á. (2021) · Journal of Quantitative Description Digital Media · doi:10.51685/jqd.2021.014 · Cited by 2
“…Media 1(2021) Characterizing COVID-19 Media 5 when online sources have failed to provide reliable predictions e.g., in Google Flu Trends’ inability to accurately detect seasonal and pandemic influenza at different temporal and geographic scales (Olson et al. 2013; Butler 2013; Lazer et al. 2014).”
BackgroundThe Spatial Dimension in Social Media Analysis: Theoretical and Methodological Characteristics
Falco, C. C. D., Crescentini, N., & Ferracci, M. (2021) · Advances in knowledge acquisition, transfer, and management book series/Advances in knowledge acquisition, transfer and management book series · doi:10.4018/978-1-7998-8473-6.ch029 · Cited by 5
Exploring Gentrification Through Social Media Data and Text Clustering Techniques
Huang, C.-C., Nara, A., Gibbons, J., et al. (2021) · Human dynamics in smart cities · doi:10.1007/978-3-030-83010-6_13 · Cited by 2
Big Data as a Mediator in Science Teaching: A ProposalPreprint
dos Santos, R. P. (2014) · SSRN Electronic Journal · doi:10.2139/ssrn.2441534 · Cited by 2
“On the other hand, as pointed by Lazer et al. (2014), Google and its derivatives Google Trends and Google Correlate are constantly changing their algorithms for numerous technical, strategic, and commercial reasons.”
BackgroundFlyverbom, M., & Rasche, A. (2015) · SSRN Electronic Journal · doi:10.2139/ssrn.2698026 · Cited by 2
The Impact of Pharmacy-Based Immunization on Vaccination Rates and Prevalence of Influenza Virus
McConeghy, K. W. (2015) · Figshare · Cited by 2
Online Learning for Distribution-Free PredictionPreprint
Zachariah, D., Stoica, P., & Schön, T. B. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.05060 · Cited by 4
Glitch Studies and the Ambiguous Objectivity of AlgorithmsPreprint
Kolkman, D., & Kemper, J. (2017) · SSRN Electronic Journal · doi:10.2139/ssrn.2985424 · Cited by 3
Deceptiveness of internet data for disease surveillancePreprint
Priedhorsky, R., Osthus, D., Daughton, A. R., Moran, K. R., & Culotta, A. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.06241 · Cited by 5
“For example, internet systems actively manipulate activity [23], such as Google’s search recommendations that update live while a query is being typed [16].”
BackgroundInfluentialBIG Data BIG Gains? Empirical Evidence on the Link Between Big Data Analytics and InnovationPreprint
Niebel, T., Rasel, F., & Viete, S. (2017) · SSRN Electronic Journal · doi:10.2139/ssrn.3078590 · Cited by 2
“One problem big data poses to decision-making is that correlations identified from the raw data are erroneously interpreted as causal relationships or that misleading patterns are found in the data (Lazer et al. 2014; McAfee and Brynjolfsson 2012).”
BackgroundPublic participation in the Geoweb era: Geosocial media use in local government
Zhang, S. (2018) · UWSpace (University of Waterloo) · Cited by 4
Schneider, P. P., van Gool, C., Spreeuwenberg, P. M. M., et al. (2018) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/440867 · Cited by 2
Big Data, Big Brother, Big Profit?Preprint
Jiang, M., & Fu, K. (2018) · SSRN Electronic Journal · doi:10.2139/ssrn.3239434 · Cited by 2
Becker, B. F. H. (2019) · EUR Research Repository (Erasmus University Rotterdam) · Cited by 2
Botha, A. (2017) · UpSpace Institutional Repository (University of Pretoria) · Cited by 4
Early Detection of Influenza outbreaks in the United StatesPreprint
Liu, K., Srinivasan, R., & Ancel, L. W. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1903.01048 · Cited by 2
Brashears, M. E. (2019) · Wiley eBooks · doi:10.1002/9781119485001.ch5 · Cited by 2
Google searches accurately forecast RSV hospitalizationsPreprint
Althouse, B. M., Weinberger, D. M., Scarpino, S. V., Pitzer, V. E., Ayers, J. W., Wenger, E. A., Fung, I. C., Dredze, M. H., & Hu, H. (2019) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/607119 · Cited by 2
“For example, does a Google search query for ‘flu symptoms’ or ‘depression symptoms’ indicate the searcher is suffering from influenza or depression themselves? Are they searching for general information, or for a loved one? This confusion around search intent led to errors in Google’s Flu Trends algorithm [3] that wer…”
BackgroundBrown, N. (2019) · University of Groningen research database (University of Groningen / Centre for Information Technology) · doi:10.33612/diss.99196913 · Cited by 2
Digital innovation management : investigating digital trace data in online communities
Mankevich, V. N. (2019) · DiVA at Umeå University (Umeå University) · Cited by 10
Searching for the Peak Google Trends and the COVID-19 Outbreak in ItalyPreprint
Brunori, P., Resce, G., & Serlenga, L. (2021) · Research Square · doi:10.21203/rs.3.rs-156833/v1 · Cited by 2
Use Internet Search Data to Accurately Track State-Level Influenza EpidemicsPreprint
Yang, S., Ning, S., & Kou, S. C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.02927 · Cited by 2
Algorithmic Discrimination - The Challenge of Unveiling Inequality in Brazil
Mattiuzzo, M. (2019) · Digital Library of Theses and Dissertations (Universidade de São Paulo) · doi:10.11606/d.2.2019.tde-16072020-174508 · Cited by 2
Progressing Towards Responsible AIPreprint
Scantamburlo, T., Cortés, À., & Schacht, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.07326 · Cited by 11
Using Google Health Trends to investigate COVID-19 incidence in AfricaPreprint
Fulk, A., Romero-Álvarez, D., Abu-Saymeh, Q., Onge, J. M. S., Peterson, A. T., & Agusto, F. B. (2021) · medRxiv · doi:10.1101/2021.03.26.21254369 · Cited by 2
What do fact checkers fact-check when?Preprint
Ribeiro, M. H., Zannettou, S., Goga, O., Benevenuto, F., & West, R. (2021) · arXiv (Cornell University) · Cited by 2
(2018) · Wiley eBooks · doi:10.1002/9781118280249.refs · Cited by 2
de Melo, P. O. S. V. (2016) · PLoS ONE · doi:10.1371/journal.pone.0147656 · Cited by 2
Protecting Health Privacy through Reasonable Inferences
Mittelstadt, B. (2022) · The American Journal of Bioethics · doi:10.1080/15265161.2022.2075980 · Cited by 4
“…of other health conditions are alleged to be detectable and predictable on Internet platforms including depression on Twitter (Nadeem 2016), flu outbreaks on Google (Olson et al. 2013; Lazer et al. 2014), and neurodegenerative disorders on Microsoft Bing (White, Doraiswamy, and Horvitz 2018).”
BackgroundStacked kinship CNN vs. GBLUP for genomic predictions of additive and complex continuous phenotypesPreprint
Nazzicari, N., & Biscarini, F. (2022) · Research Square · doi:10.21203/rs.3.rs-1932443/v1 · Cited by 2
‘Big data’ and policy learning
Dunleavy, P. (2016) · Bristol University Press eBooks · doi:10.46692/9781447329381.009 · Cited by 2
Accurate Measures of Vaccination and Concerns of Vaccine Holdouts from Web Search LogsPreprint
Chang, S. Y., Fourney, A., & Horvitz, E. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2306.07457 · Cited by 2
Woelmer, W. M., Thomas, R. Q., Olsson, F., Steele, B. G., Weathers, K. C., & Carey, C. C. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4699835 · Cited by 2
Bucher, T. (2018) · Oxford University Press eBooks · doi:10.1093/oso/9780190493028.002.0004 · Cited by 2
Why big-data analysis of police activity is inherently biased
Isaac, W., & Dixon, A. (2017) · The Conversation · doi:10.64628/aai.f9fn5dvpq · Cited by 2
Modeling Influenza-Like Illness Activity in the United States
Bollmann, L., & Scherer, M. A. (2017) · North American Actuarial Journal · doi:10.1080/10920277.2017.1284001 · Cited by 1
“Assuming a one-week lag, the autoregressive model slightly outperformed GFT. Lazar et al. (2014) claims that since 2010 simple autoregressive models have in fact even performed better than GFT.”
MethodBig Data and Doctoral Research: Opportunities, Challenges, and Cautions
Berry, R. C., & Johnston, L. (2018) · Advances in library and information science (ALIS) book series · doi:10.4018/978-1-5225-7065-3.ch004 · Cited by 1
Methodologische Entwicklungen in der Gewaltforschung. Videodatenanalyse, Mixed Methods und Big Data
Nassauer, A., & Legewie, N. M. (2020) · Österreichische Zeitschrift für Soziologie · doi:10.1007/s11614-020-00412-1 · Cited by 3
Shin, J. (2020) · Journal of Digital Convergence · doi:10.14400/jdc.2020.18.9.039 · Cited by 1
“빅데이터는 사용자의 관심에 따라 사회 전체를 관측할 수 있게도 하고 드러나지 않고 있던 현상을 발견할 수 있게도 한다[1].”
Introduction: Human Dynamics Research with Social Media and Geospatial Data Analytics
Nara, A. (2021) · Human dynamics in smart cities · doi:10.1007/978-3-030-83010-6_1 · Cited by 1
Practices and barriers for big data projects: A case study on a large insurance company
Terlizzi, M. A., Oliveira, F., & Francisco, E. D. R. (2024) · Revista de Gestão e Projetos · doi:10.5585/gep.v15i1.24673 · Cited by 8
Dahdah, M. A., & Duclos, V. (2024) · Anthropologie et santé · doi:10.4000/11r33 · Cited by 2
The Controversy About Risks and Threats in Artificial Intelligence
Kreissl, R., & von Laufenberg, R. (2024) · Springer eBooks · doi:10.1007/978-3-658-45708-2_9 · Cited by 2
Towards an estimate of the impact of censorship on biomedical literature
Kronk, C. A., Keyes, O., & Marathe, M. (2025) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaf089 · Cited by 1
Scale and Complexity of Air Pollution
Vallero, D. A. (2025) · Elsevier eBooks · doi:10.1016/b978-0-443-31446-9.00002-x · Cited by 1
Robust Checkpoint Selection by Exponential Moving Averaging for Domain Generalized Segmentation
Bätje, M., Schwonberg, M., Bohlke, H., & Leucker, M. (2025) · IEEE Intelligent Vehicles Symposium · doi:10.1109/iv64158.2025.11097429 · Cited by 1
“Despite these advancements, the deployment of machine learning models trained on synthetic data in real-world environments remains challenging and often results in a significant performance drop [10], [11], [12].”
Briziarelli, M., & Stefanizzi, S. (2025) · Quality & Quantity · doi:10.1007/s11135-025-02332-2 · Cited by 1
Jesus’ Second Coming: Some Empirical Evidence from the Perspective of the U.S. Public
Liu, K. (2025) · Journal of Religion & Society · doi:10.64667/ecld5581 · Cited by 1
Yang, F., Huang, X., Huang, W., & Jiang, T. (2025) · Frontiers in Public Health · doi:10.3389/fpubh.2025.1677306 · Cited by 2
“The “AI Q-learning Public Health Early Warning Framework” proposed in 2024 simulates the dynamic game of epidemic transmission and intervention through state-action reward mapping, achieving accurate identification of high-risk areas (14).”
THE UNINTENDED CONSEQUENCES OF ALGORITHMIC TRANSPARENCY ON TRUST: A MULTI-DISCIPLINARY ANALYSIS
Čvokić, D. D. (2025) · Acta scientifica Balcanica. · doi:10.63356/asb.2025.002 · Cited by 1
“When Google revealed how its flu prediction algorithm worked, “public trust declined rather than increased once limitations were exposed” (Lazer et al. , 2014).”
Frequency-aligned loss and spectral filtering improve long-range influenza forecasting
Feng, T., Luo, C. H., & Huang, Y. (2026) · Frontiers in Public Health · doi:10.3389/fpubh.2025.1721883 · Cited by 1
“First, many methods perform well at short horizons but degrade rapidly for longer-range forecasts (24).”
LLM-Assisted Keyword Curation for Early Disease Outbreak Detection Using Search Trends
Jerik Brahmos, A., & Princess, P. J. B. (2025) · International Conference on Applied Artificial Intelligence and Computing (ICAAIC) · doi:10.1109/icaaic64647.2025.11330303 · Cited by 1
“But the 2013 Google Flu Trends fiasco brought to light serious issues with keyword selection techniques [5].”
Measuring Well-Being: A Multidimensional Framework
Li, H., & Shen, G. (2026) · Springer eBooks · doi:10.1007/978-981-95-4937-5_3 · Cited by 2
Dong, H., Li, X., Liu, Y., & Wang, C. (2026) · Humanities and Social Sciences Communications · doi:10.1057/s41599-026-06747-6 · Cited by 1
Engineering collective attention in the age of artificial intelligence
Zhu, R. (2026) · Discover Artificial Intelligence · doi:10.1007/s44163-026-01084-8 · Cited by 2
Moving From Representativeness Toward Transportability in an Era of Digital and Big Data
Chioléro, A. (2026) · International Journal of Public Health · doi:10.3389/ijph.2026.1609306 · Cited by 1
“Third, reproducibility over time is poor because the composition of users and search-engine designs change unpredictably [15].”
A survey on opinion dynamics: From rule-based models to data-driven and hybrid approaches
Wang, J., Zhang, J., Dong, Z., & Jin, L. (2026) · Neurocomputing · doi:10.1016/j.neucom.2026.133705 · Cited by 1
Agyemang, E. F., Rodrigo, H., & Agbenyeavu, V. (2026) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2026.111736 · Cited by 7
Peng, X., Yi, W., Yang, Y., & Chan, A. P. C. (2026) · Journal of Construction Engineering and Management · doi:10.1061/jcemd4.coeng-17732 · Cited by 1
Artificial intelligence in diagnostics and medical decision making
Levi, M. M. (2026) · European Journal of Internal Medicine · doi:10.1016/j.ejim.2026.107056 · Cited by 2
Othmani, W., Houimli, H., & Medien, O. B. (2026) · Cities · doi:10.1016/j.cities.2026.107312 · Cited by 2
Revolutionizing Consumer Insights With AI and the Insight Equation
Mariyappan, M. S. R., Rupigaa, K., & Syahputra, H. (2025) · Advances in marketing, customer relationship management, and e-services book series · doi:10.4018/979-8-3693-8588-3.ch010 · Cited by 2
Manipulation-Robust Prediction
Björkegren, D., Blumenstock, J. E., & Knight, S. (2026) · American Economic Review · doi:10.1257/aer.20241087 · Cited by 71
An Agent-Based Artificial Society Approach to Analyzing Social Propagation
Tang, M., & Mao, X. (2015) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/smc.2015.145 · Cited by 1
“Meanwhile, especially big data, data science [4] has also been applied to foreseeing the spread of the infectious flu or information propagation, e.”
BackgroundBig Data—What Is It and What Use Is It?
Atkinson, G. (2014) · Journal of Ambulatory Care Management · doi:10.1097/jac.0000000000000039 · Cited by 1
Data Analytics for Pharmaceutical Discoveries
Fakhraei, S., Onukwugha, E., & Getoor, L. C. (2015) · Cited by 4
Using Social Media as a Pedagogical Tool in Graduate Public Health Education and Training
Patel, P., & Sibbald, S. (2016) · Journal of Healthcare Communications · doi:10.4172/2472-1654.100012 · Cited by 2
Forecasting model for the fourth generation of cotton bollworm in corn fields based on big data
Zhao, L., Yang, B., Liu, Y., et al. (2016) · DOAJ (DOAJ: Directory of Open Access Journals) · doi:10.11959/j.issn.2096-0271.2016008 · Cited by 1
Estimation of the Health Status of People in the Vicinity of Pharmacies Using Pharmacy Big Data
Hayashi, Y., Saito, M., & Yajima, T. (2016) · YAKUGAKU ZASSHI · doi:10.1248/yakushi.15-00268-4 · Cited by 1
An Interdisciplinary Opportunity for Information Systems Research
Schermann, M., & Krcmar, H. (2014) · Cited by 1
Efficient detection of anomaly patterns through global search in remotely sensed big data
Marinoni, A., & Gamba, P. E. (2016) · Journal of Applied Remote Sensing · doi:10.1117/1.jrs.10.045012 · Cited by 1
“Thus, the quality of the considered analysis can be dramatically degraded.(4,13) The substantial trade-off between process quality and computational cost creates the urgent need for more efficient methods for data processing.”
BackgroundTuckett, D., Ormerod, P., Nyman, R., & Smith, R. E. (2015) · Cited by 2
Risk analysis for GB rail: today and tomorrow
Dacre, M., Harrison, C., & Hunt, M. (2016) · Safety and Reliability · doi:10.1080/09617353.2016.1252081 · Cited by 1
Digital Literacy in Architecture
Savić, S., & Bühlmann, V. (2017) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3027063.3053237 · Cited by 1
“However, there are indications of shortcomings of big-data analysis to account for interactions on the network, context and meaning of the content [8,11,15].”
BackgroundBig Data in Healthcare: A Wide Look at a Broad Subject
Wilson, M. L., Weaver, C. A., Procter, P. M., & Beene, M. S. (2017) · Health informatics · doi:10.1007/978-3-319-53300-1_2 · Cited by 2
Popularity and Geospatial Spread of Trends on Twitter: A Middle Eastern Case Study
Albishry, N., Crick, T., Fagade, T., & Tryfonas, T. (2018) · Lecture notes in computer science · doi:10.1007/978-3-319-98443-8_16 · Cited by 1
“public health issues, such as the spread of influenza [1, 2].”
BackgroundFrom sensors to sense‐making: Opportunities and challenges for information science
Singh, V. K., Shah, C. S., Gwizdka, J., et al. (2017) · Proceedings of the Association for Information Science and Technology · doi:10.1002/pra2.2017.14505401083 · Cited by 1
“This would include the opportunities for research, potential pitfalls like “big-data hubris” (Lazer et al., 2014) and the identification of multiple information science relevant areas that are likely to be impacted by these trends.”
BackgroundBig Data e Valutazione: una relazione ancora da costruire
Rinaldi, F. M. (2018) · RIV Rassegna Italiana di Valutazione · doi:10.3280/riv2017-068002 · Cited by 1
Investor Perspectives on Future Priorities
Crouch, J. (2018) · World Scientific series in grand public policy challenges of the 21st century · doi:10.1142/9789813278356_0011 · Cited by 1
Big Data-Driven Public Service in the Twenty-First Century: The Politics of Big Data
Everest‐Phillips, M. (2019) · Springer eBooks · doi:10.1007/978-981-13-3215-9_9 · Cited by 2
Systematic Analysis of the Factors Contributing to the Variation and Change of the Microbiome
Aiello, K. D. (2018) · Cited by 1
Rees, E. E., Ng, V., Gachon, P., Mawudeku, A., McKenney, D. W., Pedlar, J., Yemshanov, D., Parmely, J., & Knox, J. (2019) · Relevé des maladies transmissibles au Canada · doi:10.14745/ccdr.v45i05a02f · Cited by 3
L’apport du Big Data (Mégadonnée) et des nouvelles données de la recherche en finance
Renault, T. (2019) · Vie & sciences de l entreprise · doi:10.3917/vse.206.0009 · Cited by 1
Algorithm Integration Behavior for Discovering Group Membership Rules
Silva, J., Rodríguez, C. R., Abuabara, C. O., et al. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-26369-0_3 · Cited by 1
Big and Lean is Beautiful: A Conceptual Framework for Data-based Learning in Marketing Management
Soyer, E., Pauwels, K. H., & Seggie, S. H. (2019) · Emerald Group Publishing Limited eBooks · doi:10.1108/s1548-643520190000016006 · Cited by 2
Smith, G. (2018) · Oxford University Press eBooks · doi:10.1093/oso/9780198824305.003.0004 · Cited by 1
Helbing, D. (2021) · Springer eBooks · doi:10.1007/978-3-030-62330-2_4 · Cited by 1
Jungherr, A. M., Rivero, G., & Gayo-Avello, D. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108297820.009 · Cited by 1
Hersh, W. (2020) · Health informatics · doi:10.1007/978-3-030-47686-1_8 · Cited by 1
Managerial Controversies in Artificial Intelligence and Big Data Analytics
Strang, K. D., & Sun, Z. (2021) · IGI Global eBooks · doi:10.4018/978-1-6684-3662-2.ch085 · Cited by 1
Rodríguez, O. M. H., Quintino, A. M., Cely, M. M. H., & do Nascimento, F. J. (2022) · Lecture notes in mechanical engineering · doi:10.1007/978-3-030-93456-9_7 · Cited by 1
Jemielniak, D. (2020) · Oxford University Press eBooks · doi:10.1093/oso/9780198839705.003.0004 · Cited by 1
Using “big data” to improve health care services and research
Dennis, J. A. (2016) · The Southwest Respiratory and Critical Care Chronicles · doi:10.12746/swrccc2016.0413.177 · Cited by 1
Metadata
- Authors
- Lazer, D., Kennedy, R., King, G., & Vespignani, A.
- Affiliations
- Northeastern University (US)
- Harvard University (US)
- Corresponding author
- Lazer, David
- Venue
- Science
- Published
- 2014-03-14
- Volume
- 343
- Issue
- 6176
- Pages
- 1203-1205
- ISSN
- 0036-8075 (ISSN-L)
- 1095-9203
- OpenAlex
- W2068181924
- Semantic Scholar
- 00d681cfff7f0224ace607e515f4ed510e792df9
- Cited by
- 2,523
- References
- 30
- Citations by year
- 2026: 113 · 2025: 101 · 2024: 121 · 2023: 150 · 2022: 168 · 2021: 229 · 2020: 273 · 2019: 261 · 2018: 272 · 2017: 249 · 2016: 271 · 2015: 219 · 2014: 91
- FWCI
- 208.16
- Type
- journal article
- Language
- en
- Open access
- green
- dash.harvard.edu
- Licence
- cc-by
- Locations
- Science (publishedVersion)
- PubMed (publishedVersion)
- Digital Access to Scholarship at Harvard (DASH) (Harvard University) (submittedVersion, cc-by)
- Topics
- Health Sciences › Medicine › Epidemiology › Data-Driven Disease Surveillance
- Social Sciences › Decision Sciences › Information Systems and Management › Big Data Technologies and Applications
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- Google Flu Trends, influenza surveillance, big data, prediction error, data quality
- MeSH terms
- Centers for Disease Control and Prevention, U.S., Data Collection, Data Interpretation, Statistical, Disease Outbreaks, Humans, Influenza, Human, Office Visits, Prognosis, United States, Social Media
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.