The Parable of Google Flu: Traps in Big Data Analysis
Open accessThe DOI matches its Crossref record; 30 of the 31 references it cites were found in a registry or an index; 5 of the 8 sources its claims cite were checked for support.
Abstract
Large errors in flu prediction were largely avoidable, which offers lessons for the use of big data.
Overview
CoChat Summary
TL;DR
The paper uses the failure of Google Flu Trends (GFT) to illustrate two traps in big data analysis: “big data hubris” and algorithm dynamics. It argues that GFT’s large flu-prediction errors were largely avoidable and that big data should supplement, not replace, traditional data and methods.
Key findings
- In February 2013, GFT predicted more than double the proportion of doctor visits for influenza-like illness (ILI) than the CDC; from 21 August 2011 to 1 September 2013, GFT reported overly high flu prevalence in 100 out of 108 weeks, and it overshot the 2011–2012 season by more than 50%.
- GFT errors were not random: they showed temporal autocorrelation and seasonality, meaning last week’s errors predicted this week’s errors and error direction/magnitude varied with time of year.
- Simple alternatives outperformed GFT: out-of-sample mean absolute error was 0.486 for GFT, 0.311 for lagged CDC, and 0.232 for combined GFT and CDC; all differences were statistically significant at P < 0.05. Even 3-week-old CDC data did better than GFT at projecting current flu prevalence.
- The authors attribute GFT’s problems to overfitting (initial GFT matched 50 million search terms to 1152 data points and missed the nonseasonal 2009 H1N1 pandemic) and to “algorithm dynamics,” including Google search algorithm changes and user behavior changes.
Methods
- Design: policy forum/case study comparing GFT estimates with CDC ILI surveillance data and alternative forecasting models.
- Data/sample: GFT and CDC flu prevalence estimates, including the period from 21 August 2011 to 1 September 2013; the initial GFT methodology reportedly fit 50 million search terms to 1152 data points.
- Main analysis: comparison of GFT alone, lagged CDC data with 52-week seasonality variables, and a combined GFT + CDC model using lagged CDC estimates, lagged GFT error, and seasonality variables; reported mean absolute error and statistical significance; Google Correlate used to examine correlated search terms. Supplementary materials contain data and methods.
Limitations
- The paper states that GFT never documented the 45 search terms used, released examples appear misleading, and Google Correlate is limited to national-level data whereas GFT was developed using regional correlations; therefore the original GFT analyses cannot be replicated from the information provided.
- The paper states that Google cannot make its full data available to outsiders because of privacy issues, though it argues there is no such constraint on derivative, aggregated data.
- Evident from the text: this is a commentary/policy forum rather than a full empirical article, with key data and methods placed in supplementary materials; the quantitative comparison depends on GFT and CDC data and model specifications not fully detailed in the main text.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
2 Valid · 2 Not valid · 2 Not checked · 8 Own results and interpretations
Some cited sources support this claim and some don’t — read each source’s verdict below.
In the paper
“This ad hoc method of throwing out peculiar search terms failed when GFT completely missed the nonseasonal 2009 infl uenza A–H1N1 pandemic ( 2, 14).”
What the sources say
- Partly supportedcites [2] matched to “Reassessing Google Flu Trends Data for Detection of Seasonal and Pandemic Influenza: A Comparative Epidemiological Study at Three Geographic Scales” (2013) sourceExperimental — accuracy not yet measured
The Discussion (page unavailable) says the original GFT model missed the spring 2009 pandemic wave, supporting the claim about its failure to detect the nonseasonal outbreak. It does not say GFT completely missed the entire 2009 pandemic. The Introduction describes automated selection of the original model’s search terms, not an ad hoc method of throwing out peculiar terms. (LLM, conf=0.86)
- Disputedcites [14] S. Cook et al., PLOS ONE 6, e23610 (2011). sourceExperimental — accuracy not yet measured
The Abstract (Methodology/Principal Findings) says the original GFT model underestimated the magnitude of illness during the 2009 H1N1 pandemic, not that it completely missed it. The Abstract (Conclusions) says both models performed well during the pandemic, although the updated model performed better. The Discussion notes that pandemic-specific queries were excluded, but does not describe this as an ad hoc method of throwing out peculiar terms. (LLM, conf=0.88)
Checked by a reasoning model — may be wrong.
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“Oddly, the few search terms offered in the papers ( 14) do not seem to be strongly related with either GFT or the CDC data (SM)—we surmise that the authors felt an unarticulated need to cloak the actual search terms identifi ed.”
What the sources say
- Disputedcites [14] S. Cook et al., PLOS ONE 6, e23610 (2011). sourceExperimental — accuracy not yet measured
The Results section (‘Search Behavior During pH1N1’; page unavailable) says that all three search queries shown in Figure 3 closely tracked CDC ILINet data before the pandemic, and ‘symptoms of flu’ continued to do so during it. The paper does report that the bronchitis and pneumonia queries underestimated ILINet data during the pandemic, but that does not support the broad claim that the offered terms were not strongly related to the data. (LLM, conf=0.87)
Checked by a reasoning model — may be wrong.
hedgedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“The Internet has opened the way for improving standard surveys, experiments, and health reporting ( 35).”
What the sources say
- Partly supportedcites [35] A. J. Berinsky et al., Polit. Anal. 20, 351–368 (2012). sourceExperimental — accuracy not yet measured
Section 2 describes using an Internet platform to run surveys with embedded experiments, and the Conclusion says MTurk can make experimental research easier and less expensive. The paper does not address improvements to health reporting; the disease scenario in section 4.3.2 is a framing experiment, not health reporting. Page numbers are unavailable in our copy. (LLM, conf=0.88)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Although not widely reported until 2013, the new GFT has been persistently overestimating flu prevalence for a much longer time.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“From 21 August 2011 to 1 September 2013, GFT reported overly high fl u prevalence 100 out of 108 weeks.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
3 more claims
Create a free account to see all 14 claims and every source verdict.
- It may state two separate claims as one. 5
- Probably not a substantial claim. 2
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Google Trends predicted more than twice the number of influenza-like illnesses than the CDC in February 2013 46 .”
From Unrepresentative big surveys significantly overestimated US vaccine uptake (2021) · cites this paper as [46]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This was one of the main reasons behind the failure of Google Flu Trends, since the data used for training the model in the first few years was not representative of the trends in subsequent years [15].”
From Theory-Guided Data Science: A New Paradigm for Scientific Discovery from Data (2017) · cites this paper as [15]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Social data research often relies on counting entities. These entities can be users, links, or messages, and the description is a summary of these counts (Lazer et al., 2014; Salganik, 2017). Yet, simple counts can mislead if it is unclear what is counted and how.”
From Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019) · cites this paper as Lazer et al., 2014
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This unintelligibility is also subject to functional biases (section 4.1), as it depends on the mechanisms available on each social platform (e.g., having a like button, but not a dislike one), as well as to variations in platform algorithms and mechanisms in response to users actions (Lazer et al., 2014).”
From Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019) · cites this paper as Lazer et al., 2014
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“It also includes temporal validity, which captures to what extent constructs change over time (Howison et al., 2011) and may invalidate previous conclusions about societal and/or platform phenomena; e.g., see the case of Google Flu Trends (Lazer et al., 2014).”
From Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019) · cites this paper as Lazer et al., 2014
6 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 14 MARCH 2014 VOL 343 SCIENCE www.sciencemag.org 1204
“The 2009 version of GFT has weathered other media panics related to the fl u, including the 2005–2006 influenza A/H5N1 ("bird flu") outbreak”
“GFT engineers updated the algorithm in 2009, and this model has run ever since”
Citations
Cites
boyd, D., & Crawford, K. (2012) · Information, Communication & Society · doi:10.1080/1369118x.2012.678878 · Cited by 5,479
Twitter mood predicts the stock market
Bollen, J., Mao, H., & Zeng, X. (2011) · Journal of Computational Science · doi:10.1016/j.jocs.2010.12.007 · Cited by 5,206
Detecting influenza epidemics using search engine query data
Ginsberg, J., Mohebbi, M. H., Patel, R. S., Brammer, L., Smolinski, M. S., & Brilliant, L. (2009) · Nature · doi:10.1038/nature07634 · Cited by 4,295
Cohen, D. (1998) · The Women's Review of Books · doi:10.2307/4022859 · Cited by 4,261
Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk
Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012) · Political Analysis · doi:10.1093/pan/mpr057 · Cited by 4,008
Big Data : The Management Revolution
マカフィー, ア., ブリニョルフソン, エ., & 有賀, 裕. (2013) · Harvard business review · Cited by 3,549
Lazer, D., Pentland, A., Adamic, L., Aral, S., Barabási, A.-L., Brewer, D., Christakis, N., Contractor, N., Fowler, J., Gutmann, M., et al. (2009) · Science · doi:10.1126/science.1167742 · Cited by 3,365
Self-citationPredicting Elections with Twitter: What 140 Characters Reveal about Political Sentiment
Tumasjan, A., Sprenger, T., Sandner, P., & Welpe, I. (2010) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v4i1.14009 · Cited by 2,649
Multiscale mobility networks and the spatial spreading of infectious diseases
Balcan, D., Colizza, V., Gonçalves, B., Hu, H., Ramasco, J. J., & Vespignani, A. (2009) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0906910106 · Cited by 1,380
Self-citationPredicting consumer behavior with web search
Goel, S., Hofman, J. M., Lahaie, S., Pennock, D. M., & Watts, D. J. (2010) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1005962107 · Cited by 636
Research works printed but not linked: 7
- A. McAfee, E. Brynjolfsson, Harv. Bus. Rev. 90, 60 (2012).
- AAAI Conference on Weblogs and Social Media, Atlanta, Georgia, 11 to 15 July 2010 (Association for Advancement of Artifi cial Intelligence, 2010), pp. 178.
- P. Copeland et al., Int. Soc. Negl. Trop. Dis. 2013, 3
- I. M. Hall et al., Epidemiol. Infect. 135, 372 (2007).
- E. Mustafaraj, P. Metaxas, in Proceedings of the Web- Sci10, Raleigh, NC, 26 and 27 April 2010 (Web Science Trust, 2010); http://journal.webscience.org/317/.
- J. Ratkiewicz et al., in Proceedings of 5th International AAAI Conference on Weblogs and Social Media, San Francisco, CA, 7 to 11 August 2011 (AAAI, 2011), p. 297–304.
- 2013; http://chronicle.com/article/Researchers-Struggle- to-Secure/141591/.
Cited by
Items 2,101–2,200 of 2,513
Web data mining for public health purposesPreprint
Hansen, N. D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.00829 · Cited by 1
“The most prevalent prediction models are linear models [43, 79, 107], but other non-linear models such as random forests [52, 1] or Gaussian Processes [75, 76] have also been used.”
BackgroundMethodInfluentialJacobi, M. (2018) · Portuguese National Funding Agency for Science, Research and Technology (RCAAP Project by FCT) · Cited by 1
On Selecting Stable Predictors in Time Series ModelsPreprint
Bijral, A. S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.07659
Fatores prognósticos e estratégias de gerenciamento de fluxo para o manejo da sepse
Pires, H. H. G. (2018) · Digital Library of Theses and Dissertations (Universidade de São Paulo) · doi:10.11606/t.17.2018.tde-17042018-153309
Towards an Idiographic EducationPreprint
Savi, A. O. (2019) · Thesis Commons (OSF Preprints) · doi:10.31237/osf.io/7gqhz
Quantifying human behaviour with online images
Alanyali, M. (2026) · Warwick Research Archive Portal (University of Warwick) · doi:10.82444/warw.33779596
Survey of Big Data Application Technology on Multimedia Data of Public Security
Li, H., Jiang, Y., Yang, Y., Guo, J., Hu, X., Guo, K., Zhang, B., & Cheng, J. (2019) · Lecture notes in electrical engineering · doi:10.1007/978-981-13-6508-9_14
DEFENDER: Detecting and Forecasting Epidemics using Novel Data-analytics for Enhanced ResponsePreprint
Simmie, D., Thapen, N., & Hankin, C. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1504.04357
A Dynamic Linear Model to Forecast Hotel Registrations in Puerto Rico Using Google Trends DataPreprint
Rivera, R. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1512.08097
“Another possible explanation for the poor performance of Google Flu Trends is the fact that Google constantly changes the functionality of searches (Lazer et al. 2014).”
BackgroundMorelli, V., & Conley, N. B. (2019) · Elsevier eBooks · doi:10.1016/b978-0-323-66130-0.00019-3
2. Données massives et recherche en économie : une (r)évolution ?
Renault, T. (2019) · Regards croisés sur l'économie/Regards croisés sur l'économie · doi:10.3917/rce.023.0032
A Comparative Study of Statistical Analysis Techniques
Zou, Z. (2019)
Harnessing Heterogeneity To Improve Patient Outcomes.
Hibbard, J. C. (2017) · Carolina Digital Repository (University of North Carolina at Chapel Hill) · doi:10.17615/ngp1-s502
An appraisal of health datasets to enhance the surveillance of Lyme disease in the United Kingdom
Tulloch, J. (2019) · University of Liverpool · doi:10.17638/03046553 · Cited by 1
Empirical analysis of the US consumer
Haughwout, A. F., & Mandel, B. R. (2019) · Elsevier eBooks · doi:10.1016/b978-0-12-813524-2.00001-9
Online mapping and forecasting of epidemics using open-source indicators
Sandia National Laboratories (SNL-CA), ;. C. L., Ray, J., Administration, U. A. S. F. H. R. A., Sandia National Laboratories (SNL), A., Lefantzi, S., Bauer, J. R., Khalil, M., Rothfuss, A., Cauthen, K., Finley, P. D., et al. (2016) · OSTI OAI (U.S. Department of Energy Office of Scientific and Technical Information) · doi:10.2172/1562406
The semicircular flow of the data economy
de Pedraza, P., & Vollbracht, I. (2019) · Joint Research Centre (European Commission) · doi:10.2760/668 · Cited by 1
Concept Drift Adaptive Physical Event Detection for Social Media StreamsPreprint
Suprem, A., Musaev, A., & Pu, C. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.05494
“However such assumptions, made in Google Flu Trends [9, 10] degrade in the long term.”
BackgroundMethodEvent Detection in Noisy Streaming Data with Combination of Corroborative and Probabilistic SourcesPreprint
Suprem, A., & Pu, C. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.09281
Dym, B., & Fiesler, C. L. (2019) · International ACM SIGGROUP Conference on Supporting Group Work (GROUP) · doi:10.1145/3323994.3369881 · Cited by 1
Research Ethics in Data: New Technologies, New Challenges
Combe, C. G. (2019) · Springer eBooks · doi:10.1007/978-3-319-76040-7_13-1
Ródenas, M. J. D. L. (2019) · Hispania Sacra · doi:10.3989/hs.2019.037 · Cited by 2
Die Survey-Praxis – Schätzen unter Realbedingungen
Quatember, A. (2019) · Springer eBooks · doi:10.1007/978-3-662-60274-4_3
Learning Analytics og udvikling af studerendes autonomi og autenticitet
Christiansen, F. V., Mathiasen, H., & Johansen, M. (2019) · Dansk Universitetspaedagogisk Tidsskrift · doi:10.7146/dut.v14i27.108002
“…design eller i måden, underviserne bruger den på, kan føre til adfærdsændringer, der gør algoritmens prædiktioner upålidelige (se fx Lazer et al 2014 for et konkret eksempel på fejlslagen brug af Big Data samt Papamitsiou og Economides 2014 for en mere udfoldet SWOT-analyse).”
BackgroundDoes the blue bird get the flu? using Twitter for flu surveillance
Grüninger, S. (2017) · Zurich Open Repository and Archive (University of Zurich) · doi:10.5167/uzh-152442
A DIVISION OF LABOR: THE ROLE OF BIG DATA ANALYSIS IN THE REPERTOIRE OF INTERNET RESEARCH METHODS
Helles, R., Ørmen, J., Jensen, K. B., Lai, S. S., Menchen-Trevino, E., Taneja, H., Wu, A. X., & Bruns, A. (2020) · AoIR Selected Papers of Internet Research · doi:10.5210/spir.v2018i0.10467
MengRan, L. I., Liu, X., & Ao, J. (2020) · The international archives of the photogrammetry, remote sensing and spatial information sciences/International archives of the photogrammetry, remote sensing and spatial information sciences · doi:10.5194/isprs-archives-xlii-3-w10-731-2020
ERS International Congress, Madrid, 2019: highlights from the Epidemiology and Environment Assembly
Jankowski, M., & Amaral, A. F. S. (2020) · ERJ Open Research · doi:10.1183/23120541.00320-2019
Elaborating Role of Big Data Analytics in Sports
Kaur, G., & Laxmi, V. (2019) · Journal of Emerging Technologies and Innovative Research
Theory In, Theory Out: The uses of social theory in machine learning for social sciencePreprint
Radford, J. P., & Joseph, K. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2001.03203
“, 2012) or aggregated search data from Google Trends to predict the prevalence of the flu (Lazer et al., 2014).”
BackgroundMethods of Researching Online Communities
Jemielniak, D. (2020) · Oxford University Press eBooks · doi:10.1093/oso/9780198839705.003.0003
Berke, O., Trotz‐Williams, L. A., & de Montigny, S. (2020) · Relevé des maladies transmissibles au Canada · doi:10.14745/ccdr.v46i06a07f
Scales, D. A. (2020) · Online Journal of Public Health Informatics · doi:10.5210/ojphi.v12i1.10579 · Cited by 3
“For example, Google Flu Trends, a tracking system designed to predict influenza, was initially heralded as a success before it was found to have significant flaws and was suspended [39].”
BackgroundA Dual Layer Regression Model for Cross-border E-commerce Industry Sale and Hot Product Prediction
Luo, W., Su, H., Liu, Y., & Xu, R. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-59585-2_5
Validating Social Media Monitoring: Statistical Pitfalls and Opportunities from Public Opinion
Smith, M. C., Mazzuchi, T. A., & Broniatowski, D. A. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-61255-9_7
THE DYNAMICS OF DIGITAL CAPTURE: HOW INDUSTRIES TIE AUDIENCES TO EMERGING TECHNOLOGIES
Einstein, M., Yeo, S., Turow, J., & Jobin, A. (2020) · AoIR Selected Papers of Internet Research · doi:10.5210/spir.v2020i0.11116
“With hindsight, Lazer et al. (2014) were able to point out that GFT had been particularly accurate in predicting winter (p. 1203) and that its creators had failed to recognize that web search does not happen in a vacuum but is influenced by factors both external (e.g. media reporting) and internal…”
BackgroundReplicable Privacy: Enabling Replication on Sensitive Internet DataPreprint
Baleato, S., Honaker, J., & Crosas, M. (2020) · doi:10.33774/apsa-2020-mpq30
Zeng, S., Bayır, M. A., J.Pfeiffer, J., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.08710
“Other domains with distribution or covariate shifts include text and image classification [9, 33], information extraction [4], as well as prediction and now-casting [19].”
BackgroundSaha, K. (2020) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/3406865.3418367
“However, before diving into that, we should ask how these algorithms would perform if we conduct prospective data collection? Despite the potential, the validity of adopting computational social science on big data has been critiqued regarding its applicability in real-world [3, 9] (see left for more description).”
BackgroundÜber die Unfähigkeit, die Zukunft vorhersagen zu können
Hilbig, H. (2020) · Springer eBooks · doi:10.1007/978-3-658-31246-6_1
If You Torture the Data Long Enough
Smith, G. E. (2018) · Oxford University Press eBooks · doi:10.1093/oso/9780198824305.003.0008
Smith, G. (2018) · Oxford University Press eBooks · doi:10.1093/oso/9780198824305.003.0012
Smith, G. (2018) · Oxford University Press eBooks · doi:10.1093/oso/9780198824305.003.0014
Smith, G. (2018) · Oxford University Press eBooks · doi:10.1093/oso/9780198824305.003.0007
Smith, G. (2018) · Oxford University Press eBooks · doi:10.1093/oso/9780198824305.003.0009
Smith, G. (2018) · Oxford University Press eBooks · doi:10.1093/oso/9780198824305.003.0011
Smith, G. (2018) · Oxford University Press eBooks · doi:10.1093/oso/9780198824305.003.0003
Using big data to enhance pertussis surveillance and response in Shandong Province, China
Zhang, Y. (2020) · Queensland University of Technology · doi:10.5204/thesis.eprints.206172
Smith, G. (2018) · Oxford University Press eBooks · doi:10.1093/oso/9780198824305.003.0005
Arvutiteaduse kaasamine humanitaarharidusse / Integrating Computation into Humanities Education
Tolonen, M., Eetu Mäkelä, A., Marjanen, J., & Tahko, T. (2020) · Methis Studia humaniora Estonica · doi:10.7592/methis.v21i26.16909
KORNIEIEVA, I. (2020) · Naukovi pratsi NDFI · doi:10.33763/npndfi2020.03.128
The Flow of Political Information
Jungherr, A. M., Rivero, G., & Gayo-Avello, D. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108297820.002
The perils of working with Big Data and a SMALL framework you can use to avoid them
Brave, S. A., Butters, R. A., & Fogarty, M. S. (2020) · doi:10.21033/wp-2020-35 · Cited by 1
Flahault, A. (2021) · Springer eBooks · doi:10.1007/978-3-030-05325-3_70-1
Imputing Unreported Hate Crimes using Google Search DataPreprint
Dharmapala, D., & Huq, A. Z. (2018) · SSRN Electronic Journal · doi:10.2139/ssrn.3197913
Reber, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.02018
“…2012) and reviews (Mukherjee et al., 2013), • Text completion (Diakopoulos, 2013b), correction (Keller, 2013) and detection (Sap et al., 2019), • Perception of online environments (Hannak et al., 2014; Krafft et al., 2017; Larson & Shaw, 2012), • Finance (Lazer et al., 2014; Pulliam & Barry, 2012).”
BackgroundLearning from pandemics: using extraordinary events can improve disease now-casting modelsPreprint
Mesquita, S., Vieira, C. H., Perfeito, L., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.06774
Jungherr, A. M., Rivero, G., & Gayo-Avello, D. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108297820.007
Social Forces: Revealing the Causes of Success and Disaster
Helbing, D. (2021) · Springer eBooks · doi:10.1007/978-3-030-62330-2_3 · Cited by 1
Conclusion: Toward Technical Culture
Rieder, B. (2020) · Amsterdam University Press eBooks · doi:10.5117/9789462986190_concl
Disease and Social Media in Post-Natural Disaster Recovery PhilippinesPreprint
Charles, L., & Corley, C. D. (2021) · medRxiv · doi:10.1101/2021.03.22.21254137 · Cited by 1
Modern Survey Estimation with Social Media and Auxiliary Data
Ferg, R. A. (2020) · Deep Blue (University of Michigan)
The Effects of Political Information
Jungherr, A. M., Rivero, G., & Gayo-Avello, D. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108297820.004
Jove, D. (2021) · Revista Española de Derecho Constitucional · doi:10.18042/cepc/redc.121.10 · Cited by 1
Piccelli, R. R. (2017) · Digital Library of Theses and Dissertations (Universidade de São Paulo) · doi:10.11606/d.2.2017.tde-27112020-163616
Quantifying the differences incall detail records
Botta, F. (2021) · Royal Society Open Science · doi:10.1098/rsos.201443 · Cited by 1
“Models based on new forms of data have already shown the importance of training and calibrating [28,29], so a better understanding on the limitations of mobile phone data is crucial for researchers and stakeholders alike.”
BackgroundAnalysis of complex socio-economical systems
Bokányi, E. (2019) · Eötvös Loránd Tudományegyetem · doi:10.15476/elte.2019.054
Single Model for Influenza Forecasting of Multiple Countries by Multi-task LearningPreprint
Murayama, T., Wakamiya, S., & Aramaki, E. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.01760
“Also, the training of a single model using various flu-related data can capture the nature of flu epidemics in each country by escaping from overfitting, which is caused by a lesser degree of historical data in one country for training [27,26].”
BackgroundUncertainty quantification for complex systems : application to the study of cities
Steentoft, A. A. (2021) · DR-NTU (Nanyang Technological University) · doi:10.32657/10356/152120
Jardine, E. (2021) · Journal of Cyber Policy · doi:10.1080/23738871.2021.1953557 · Cited by 1
BIG DATA ERASINDA RƏQƏMSAL DEMOQRAFİYANIN FORMALAŞMASI XÜSUSİYYƏTLƏRİ
İnstitutu, A. İ. T., Hacırəhimova, M., Əliyeva, A., & İnstitutu, A. İ. T. (2021) · Problems of Information Technology · doi:10.25045/jpit.v12.i2.07
Proximity in face-to-face interaction is associated with mobile phone communicationPreprint
Bornakke, T., Aledavood, T., Saramäki, J., & Roberts, S. G. B. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.08872 · Cited by 2
Reverberações simbólicas no Google Trends: uma análise do imaginário na internet
Contrera, M. S., & Torres, L. (2021) · Intexto · doi:10.19132/1807-8583202152.98924
Phase III: CSDS Gap-Prescriptions—Design Science Problem-Solving
Mongeau, S., & Hajdasinski, A. (2021) · Springer eBooks · doi:10.1007/978-3-030-74896-8_4 · Cited by 1
Social physicsPreprint
Jusup, M., Holme, P., Kanazawa, K., Takayasu, M., Romić, I., Wang, Z., Geček, S., Lipić, T., Boris Podobnik, A., Wang, L., et al. (2021) · arXiv (Cornell University)
Umsetzbarkeit der Digitalisierung in Controlling-Prozessen
Müller, F. (2021) · Quick Guide · doi:10.1007/978-3-658-35041-3_2
Experimenting: What Makes a Good Business Experiment?
Thomke, S. H. (2021) · Springer eBooks · doi:10.1007/978-3-030-76897-3_10
Computational Social Science: A Thematic Review
Meckin, R., & Elliot, M. J. (2021) · NCRM EPrints Repository (National Centre for Research Methods) · Cited by 5
Unrepresentative Big Surveys Significantly Overestimate US Vaccine UptakePreprint
Bradley, V. C., Kuriwaki, S., Isakov, M., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.05818
Getting a Sense of Big Data and Well-being
Oman, S. (2021) · New directions in cultural policy research · doi:10.1007/978-3-030-72937-0_5
Investigación económica y datos masivos: mercados, fines sociales y colaboración público-privada
Montalvo, J. G. (2021) · Cuadernos Económicos de ICE · doi:10.32796/cice.2021.102.7311 · Cited by 1
(2021) · Elsevier eBooks · doi:10.1016/b978-0-12-816341-2.00009-5
Mazari, A. A. (2021) · IGI Global eBooks · doi:10.4018/978-1-6684-3662-2.ch072
An infodemiological study on Novel Coronavirus (COVID-19) in Iran using Google Trends (Preprint)Preprint
Zeinali, V., Abbasianchavari, A., & Javadi, V. (2020) · doi:10.2196/preprints.21337
Trauma Spillover in Social NetworksPreprint
Sadish, D. (2021) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/t6b2y
COVID-19 forecasting with deep learning: a distressing surveyPreprint
Gutierrez, L., de Medrano, R., & Aznarte, J. L. (2021) · doi:10.36227/techrxiv.16855189 · Cited by 1
Qi, X., Su-Zhen, W. A. N. G., Feng, J., Zhu, G., Liu, Y., Mao, Q., Wang, Z., & Guan, P. (2021) · doi:10.2196/preprints.29578
The Future of Personalisation: Algorithmic Foretelling and Its Limits
Kohl, U., & Eisler, J. (2021) · Cambridge University Press eBooks · doi:10.1017/9781108891325.018
Satpathy, P., Kumar, S., & Prasad, P. (2020) · doi:10.2196/preprints.25985
COVID-19 forecasting with deep learning: a distressing surveyPreprint
Gutiérrez, L. J. C., de Medrano, R., & Aznarte, J. L. (2021) · doi:10.36227/techrxiv.16855189.v1
“Predictive models must be carefully evaluated not only on their ability to capture historical events but on their exactitude in forecasting future trends, fostering a stronger appreciation of the technology’s capabilities and limitations [3].”
BackgroundResearch Methods for Supporting Engineering Systems Design
Szajnfarber, Z., & Broniatowski, D. A. (2021) · Springer eBooks · doi:10.1007/978-3-030-46054-9_22-1
About the Inability to Predict the Future
Hilbig, H. (2022) · Springer eBooks · doi:10.1007/978-3-658-35349-0_1
Editor in Chief’s Pick – Engines of Order: A Mechanology of Algorithmic Techniques
Rieder, B. (2020) · Stedelijk Studies Journal · doi:10.54533/stedstud.vol010.art12 · Cited by 1
Juola, P. (2022) · Springer eBooks · doi:10.1007/978-3-319-32010-6_307
Developmental Idealism in Internet Search DataPreprint
Dorius, S., & Swindle, J. (2018) · doi:10.31235/osf.io/axc3z
(2020) · Cambridge University Press eBooks · doi:10.1017/9781108297820.010
Lacey, K. (2022) · Springer eBooks · doi:10.1007/978-3-319-32010-6_106
Bakdash, J. Z. (2022) · Springer eBooks · doi:10.1007/978-3-319-32010-6_176
Geospatial Data Science: A Transdisciplinary Approach
Eftelioglu, E., Ali, R. Y., Tang, X., Xie, Y., Yan, L., & Shekhar, S. (2017) · CRC Press eBooks · doi:10.1201/9781315228396-2
Metadata
- Authors
- Lazer, D., Kennedy, R., King, G., & Vespignani, A.
- Affiliations
- Northeastern University (US)
- Harvard University (US)
- Corresponding author
- Lazer, David
- Venue
- Science
- Published
- 2014-03-14
- Volume
- 343
- Issue
- 6176
- Pages
- 1203-1205
- ISSN
- 0036-8075 (ISSN-L)
- 1095-9203
- OpenAlex
- W2068181924
- Semantic Scholar
- 00d681cfff7f0224ace607e515f4ed510e792df9
- Cited by
- 2,523
- References
- 30
- Citations by year
- 2026: 113 · 2025: 101 · 2024: 121 · 2023: 150 · 2022: 168 · 2021: 229 · 2020: 273 · 2019: 261 · 2018: 272 · 2017: 249 · 2016: 271 · 2015: 219 · 2014: 91
- FWCI
- 208.16
- Type
- journal article
- Language
- en
- Open access
- green
- dash.harvard.edu
- Licence
- cc-by
- Locations
- Science (publishedVersion)
- PubMed (publishedVersion)
- Digital Access to Scholarship at Harvard (DASH) (Harvard University) (submittedVersion, cc-by)
- Topics
- Health Sciences › Medicine › Epidemiology › Data-Driven Disease Surveillance
- Social Sciences › Decision Sciences › Information Systems and Management › Big Data Technologies and Applications
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- Google Flu Trends, influenza surveillance, big data, prediction error, data quality
- MeSH terms
- Centers for Disease Control and Prevention, U.S., Data Collection, Data Interpretation, Statistical, Disease Outbreaks, Humans, Influenza, Human, Office Visits, Prognosis, United States, Social Media
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.