The DOI matches its Crossref record; 51 of the 51 references it cites were found in a registry or an index.
Reassessing Google Flu Trends Data for Detection of Seasonal and Pandemic Influenza: A Comparative Epidemiological Study at Three Geographic Scales
Open accessAbstract
The goal of influenza-like illness (ILI) surveillance is to determine the timing, location and magnitude of outbreaks by monitoring the frequency and progression of clinical case incidence. Advances in computational and information technology have allowed for automated collection of higher volumes of electronic data and more timely analyses than previously possible. Novel surveillance systems, including those based on internet search query data like Google Flu Trends (GFT), are being used as surrogates for clinically-based reporting of influenza-like-illness (ILI). We investigated the reliability of GFT during the last decade (2003 to 2013), and compared weekly public health surveillance with search query data to characterize the timing and intensity of seasonal and pandemic influenza at the national (United States), regional (Mid-Atlantic) and local (New York City) levels. We identified substantial flaws in the original and updated GFT models at all three geographic scales, including completely missing the first wave of the 2009 influenza A/H1N1 pandemic, and greatly overestimating the intensity of the A/H3N2 epidemic during the 2012/2013 season. These results were obtained for both the original (2008) and the updated (2009) GFT algorithms. The performance of both models was problematic, perhaps because of changes in internet search behavior and differences in the seasonality, geographical heterogeneity and age-distribution of the epidemics between the periods of GFT model-fitting and prospective use. We conclude that GFT data may not provide reliable surveillance for seasonal or pandemic influenza and should be interpreted with caution until the algorithm can be improved and evaluated. Current internet search query data are no substitute for timely local clinical and laboratory surveillance, or national surveillance based on local data collection. New generation surveillance systems such as GFT should incorporate the use of near-real time electronic health data and computational methods for continued model-fitting and ongoing evaluation and improvement.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This ad hoc method of throwing out peculiar search terms failed when GFT completely missed the nonseasonal 2009 infl uenza A–H1N1 pandemic ( 2, 14).”
From The Parable of Google Flu: Traps in Big Data Analysis (2014) · cites this paper as [2]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Detecting influenza epidemics using search engine query data
Ginsberg, J., Mohebbi, M. H., Patel, R. S., Brammer, L., Smolinski, M. S., & Brilliant, L. (2009) · Nature · doi:10.1038/nature07634 · Cited by 4,295
Pandemic Potential of a Strain of Influenza A (H1N1): Early Findings
Fraser, C., Donnelly, C. A., Cauchemez, S., Hanage, W. P., Kerkhove, M. D. V., Hollingsworth, T. D., Griffin, J. T., Baggaley, R. F., Jenkins, H. E., Lyons, E., et al. (2009) · Science · doi:10.1126/science.1176062 · Cited by 1,915
Synchrony, Waves, and Spatial Hierarchies in the Spread of Influenza
Viboud, C. G., Bjørnstad, O. N., Smith, D. L., Simonsen, L., Miller, M. A., & Grenfell, B. T. (2006) · Science · doi:10.1126/science.1125237 · Cited by 832
Self-citationSevere Respiratory Disease Concurrent with the Circulation of H1N1 Influenza
Chowell, G., Bertozzi, S., Colchero, M. A., López‐Gatell, H., Alpuche-Aranda, C. M., Hernández, M., & Miller, M. A. (2009) · New England Journal of Medicine · doi:10.1056/nejmoa0904023 · Cited by 685
Methods for current statistical analysis of excess pneumonia-influenza deaths
Serfling, R. E. (1963) · Public Health Reports (1896-1970) · doi:10.2307/4591848 · Cited by 597
The impact of influenza epidemics on mortality: introducing a severity index.
Simonsen, L., Clarke, M. J., Williamson, G. D., Stroup, D. F., Arden, N. H., & Schonberger, L. B. (1997) · American Journal of Public Health · doi:10.2105/ajph.87.12.1944 · Cited by 587
Self-citationUsing Internet Searches for Influenza Surveillance
Polgreen, P. M., Chen, Y., Pennock, D. M., & Nelson, F. D. (2008) · Clinical Infectious Diseases · doi:10.1086/593098 · Cited by 567
Butler, D. (2013) · Nature · doi:10.1038/494155a · Cited by 528
Salathé, M., Bengtsson, L., Bodnar, T. J., Brewer, D. D., Brownstein, J. S., Buckee, C., Campbell, E. M., Cattuto, C., Khandelwal, S., Mabry, P. L., et al. (2012) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1002616 · Cited by 500
Forecasting seasonal outbreaks of influenza
Shaman, J. L., & Karspeck, A. R. (2012) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1208772109 · Cited by 419
Infodemiology: tracking flu-related searches on the web for syndromic surveillance.
Eysenbach, G. (2006) · PubMed · Cited by 414
Cook, S., Conrad, C., Fowlkes, A. L., & Mohebbi, M. H. (2011) · PLoS ONE · doi:10.1371/journal.pone.0023610 · Cited by 381
Syndromic Surveillance in Public Health Practice, New York City
Heffernan, R. T., Mostashari, F., Das, D., Karpati, A. M., Kulldorff, M., & Weiss, D. (2004) · Emerging infectious diseases · doi:10.3201/eid1005.030646 · Cited by 355
The Signature Features of Influenza Pandemics — Implications for Policy
Miller, M. A., Viboud, C. G., Balinska, M., & Simonsen, L. (2009) · New England Journal of Medicine · doi:10.1056/nejmp0903906 · Cited by 349
Self-citationInfluenza Forecasting with Google Flu Trends
Dugas, A. F., Jalalpour, M., Gel, Y. R., Levin, S., Torcaso, F., Igusa, T., & Rothman, R. E. (2013) · PLoS ONE · doi:10.1371/journal.pone.0056176 · Cited by 349
What is Syndromic Surveillance?
Henning, K. J. (2004) · PsycEXTRA Dataset · doi:10.1037/e307182005-001 · Cited by 323
More Diseases Tracked by Using Google Trends
Pelat, C., Turbelin, C., Bar‐Hen, A., Flahault, A., & Valleron, A. (2009) · Emerging infectious diseases · doi:10.3201/eid1508.090299 · Cited by 244
Web Queries as a Source for Syndromic Surveillance
Hulth, A., Rydevik, G., & Linde, A. (2009) · PLoS ONE · doi:10.1371/journal.pone.0004378 · Cited by 223
Olson, D. R., Heffernan, R. T., Paladini, M., Konty, K., Weiss, D., & Mostashari, F. (2007) · PLoS Medicine · doi:10.1371/journal.pmed.0040247 · Cited by 197
Self-citationUsing Google Trends for Influenza Surveillance in South China
Kang, M., Zhong, H., He, J., Rutherford, S., & Yang, F. (2013) · PLoS ONE · doi:10.1371/journal.pone.0055205 · Cited by 171
Lazarus, R. M., KLEINMAN, K. P., Dashevsky, I., DeMaria, A., & Platt, R. G. (2001) · BMC Public Health · doi:10.1186/1471-2458-1-9 · Cited by 162
Ortiz, J. R., Zhou, H., Shay, D. K., Neuzil, K. M., Fowlkes, A. L., & Goss, C. H. (2011) · PLoS ONE · doi:10.1371/journal.pone.0018687 · Cited by 157
Virtual surveillance of communicable diseases: a 20-year experience in France
Flahault, A., Blanchon, T., Dorléans, Y., Toubiana, L., Vibert, J. F., & Valleron, A. (2006) · Statistical Methods in Medical Research · doi:10.1177/0962280206071639 · Cited by 151
A computer network for the surveillance of communicable diseases: the French experiment.
Valleron, A., Bouvet, É., Garnerin, P., Ménarés, J., Heard, I., Letrait, S., & Lefaucheux, J. (1986) · American Journal of Public Health · doi:10.2105/ajph.76.11.1289 · Cited by 148
A routine tool for detection and assessment of epidemics of influenza-like syndromes in France.
Costagliola, D. G., Flahault, A., Galinec, D., Garnerin, P., Ménarés, J., & Valleron, A. (1991) · American Journal of Public Health · doi:10.2105/ajph.81.1.97 · Cited by 129
Merler, S., Ajelli, M., Pugliese, A., & Ferguson, N. M. (2011) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1002205 · Cited by 123
Lessons from 40 years' surveillance of influenza in England and Wales
Fleming, D. M., & Elliot, A. J. (2007) · Epidemiology and Infection · doi:10.1017/s0950268807009910 · Cited by 109
Collecting And Sharing Data For Population Health: A New Paradigm
Diamond, C. C., Mostashari, F., & Shirky, C. (2009) · Health Affairs · doi:10.1377/hlthaff.28.2.454 · Cited by 107
Valdivia, A., López‐Alcalde, J., Vicente, M., Pichiule‐Castañeda, M., Ruiz, M., & Ordobás, M. (2010) · Eurosurveillance · doi:10.2807/ese.15.29.19621-en · Cited by 100
Brooks‐Pollock, E., Tilston‐Lunel, N. L., Edmunds, W. J., & Eames, K. T. D. (2011) · BMC Infectious Diseases · doi:10.1186/1471-2334-11-68 · Cited by 96
Ellis, R. H., Hong, T., & Roberts, Ε. H. (1985) · International Board for Plant Genetic Resources eBooks · Cited by 94
Dalton, C. B., Dürrheim, D. N., Fejsa, J., Francis, L., Carlson, S., d’Espaignet, E. T., & Tuyl, F. (2009) · Communicable Diseases Intelligence · doi:10.33321/cdi.2009.33.34 · Cited by 90
Gripenet: an internet-based system to monitor influenza-like illness uniformly across Europe
van Noort, S. P., Muehlen, M., Rebelo de Andrade, H., Koppeschaar, C., Lima Lourenço, J. M., & Gomes, M. G. (2007) · Eurosurveillance · doi:10.2807/esm.12.07.00722-en · Cited by 84
Syndromic Surveillance: a Local Perspective
Mostashari, F. (2003) · Journal of Heredity · doi:10.1093/jurban/jtg042 · Cited by 83
Interpreting “Google Flu Trends” data for pandemic H1N1 influenza: The New Zealand experience
Wilson, N., Mason, K., Tobias, M., Peacey, M., Huang, Q. S., & Baker, M. G. (2009) · Eurosurveillance · doi:10.2807/ese.14.44.19386-en · Cited by 78
The Epidemiological Basis for the Control of Influenza
Langmuir, A. D., Henderson, D. A., & Serfling, R. E. (1964) · American Journal of Public Health and the Nations Health · doi:10.2105/ajph.54.4.563 · Cited by 77
Online detection and quantification of epidemics
Pelat, C., Boëlle, P., Cowling, B. J., Carrat, F., Flahault, A., Ansart, S., & Valleron, A. (2007) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-7-29 · Cited by 66
HANNOUN, C., Dab, W., & Cohen, J. (1989) · European Journal of Epidemiology · doi:10.1007/bf00144828 · Cited by 64
Can syndromic thresholds provide early warning of national influenza outbreaks?
Cooper, D. L., Verlander, N. Q., Elliot, A. J., Joseph, C. A., & Smith, G. (2007) · Journal of Public Health · doi:10.1093/pubmed/fdm068 · Cited by 59
Zheng, W., Aitken, R. J., Muscatello, D. J., & Churches, T. (2007) · BMC Public Health · doi:10.1186/1471-2458-7-250 · Cited by 58
The evolution of influenza surveillance in Europe and prospects for the next 10 years
Fleming, D. (2003) · Vaccine · doi:10.1016/s0264-410x(03)00066-5 · Cited by 52
(2002) · PubMed · Cited by 47
Epidemic Influenza — Responding to the Expected but Unpredictable
Bresee, J., & Hayden, F. G. (2013) · New England Journal of Medicine · doi:10.1056/nejmp1300375 · Cited by 39
van den Wijngaard, C. C., van Asten, L., Meijer, A., van Pelt, W., Nagelkerke, N. J. D., Donker, G., van der Sande, M. A. B., & Koopmans, M. (2010) · American Journal of Public Health · doi:10.2105/ajph.2009.168245 · Cited by 34
Sentiweb: French communicable disease surveillance on the world wide web
Boussard, E., Flahault, A., Vibert, J. F., & Valleron, A. (1996) · BMJ · doi:10.1136/bmj.313.7069.1381 · Cited by 33
Use of Internet Search Data to Monitor Impact of Rotavirus Vaccination in the United States
Desai, R., Lopman, B. A., Shimshoni, Y., Harris, J. P., Patel, M. M., & Parashar, U. D. (2012) · Clinical Infectious Diseases · doi:10.1093/cid/cis121 · Cited by 25
Olson, D. R., Paladini, M., Lober, W. B., Buckeridge, D. L., & Isds Distribute Working Group (2011) · PLoS Currents · doi:10.1371/currents.rrn1251 · Cited by 24
Self-citationEye-Opening Approach to Norovirus Surveillance
Hulth, A., Andersson, Y., Hedlund, K., & Andersson, M. (2010) · Emerging infectious diseases · doi:10.3201/eid1608.100093 · Cited by 18
QFLU: new influenza monitoring in UK primary care to support pandemic influenza planning
Hippisley‐Cox, J., Smith, S., Smith, G., Porter, A., Heaps, M., Holland, R., Fenty, J., Harcourt, S., George, R., Charlett, A., et al. (2006) · Weekly releases (1997–2007) · doi:10.2807/esw.11.25.02980-en · Cited by 10
Olson, D., Baer, A., Coletta, M. A., Deyneka, L., Gentry, R., Ising, A., Murray, E. L., Paladini, M., Pendarvis, J., Soetebier, K. A., et al. (2009) · Nature Precedings · doi:10.1038/npre.2009.3493.1 · Cited by 4
Self-citationWeek 50 Influenza Forecast for the 2012-2013 U.S. SeasonPreprint
Shaman, J., Karspeck, A., & Lipsitch, M. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1212.5750 · Cited by 3
Flagged first, then by citations
Research works printed but not linked: 6
- M Helft, 2008, Google Uses Searches to Track Flu's Spread, The New York Times; Sect A, 1
- DR Olson, 2009, Searching for better flu surveillance? A brief communication arising, Nature Precedings, 3493
- Karon AE, Heffernan RT (2009) Google Flu Trends versus ED Syndromic Surveillance, Wisconsin 2009. Annual Conference of the International Society for Disease Surveillance (ISDS), Miami, FL, December 2009.
- M Santora, 2013, New York Declares Health Emergency, The New York Times;Sect A, 21
- M Secret, 2013, Pharmacies Pressed to Meet High Demand for Flu Vaccine, The New York Times; Sect A, 16
- Simonsen L, Olson DR, Viboud C, Heiman E, Taylor R, <etal>et al</etal>.. (2005) Pandemic Influenza and Mortality: Past Evidence and Projections for the Future. In The Threat of Pandemic Influenza: Are We Ready? Board on Global Health, Institute of Medicine, National Academies Press, DC, 2005, pp 89–106.
Cited by
Sorted by citations
The Parable of Google Flu: Traps in Big Data Analysis
Lazer, D., Kennedy, R., King, G., & Vespignani, A. (2014) · Science · doi:10.1126/science.1248506 · Cited by 2,523
Statistical physics of vaccination
Wang, Z., Bauch, C. T., Bhattacharyya, S., d’Onofrio, A., Manfredi, P., Perc, M., Perra, N., Salathé, M., & Zhao, D. (2016) · Physics Reports · doi:10.1016/j.physrep.2016.10.006 · Cited by 973
Jusup, M., Holme, P., Kanazawa, K., Takayasu, M., Romić, I., Wang, Z., Geček, S., Lipić, T., Boris Podobnik, A., Wang, L., et al. (2022) · Physics Reports · doi:10.1016/j.physrep.2021.10.005 · Cited by 688
Jun, S., Yoo, H. S., & Choi, S. (2017) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2017.11.009 · Cited by 587
Combining Search, Social Media, and Traditional Data Sources to Improve Influenza Surveillance
Santillana, M., Nguyen, A., Dredze, M., Paul, M., Nsoesie, E., & Brownstein, J. (2015) · PLoS Comput. Biol. · doi:10.1371/journal.pcbi.1004513 · Cited by 441
Metadata
- Authors
- Olson, D. R., Konty, K. J., Paladini, M., Viboud, C., & Simonsen, L.
- Affiliations
- New York City Department of Health and Mental Hygiene (US)
- National Institutes of Health (US)
- Corresponding author
- Olson, Donald R.
- Venue
- PLoS Computational Biology
- Publisher
- Public Library of Science (PLoS)
- Published
- 2013-10-17
- Volume
- 9
- Issue
- 10
- Pages
- e1003256
- ISSN
- 1553-734X (ISSN-L)
- 1553-7358
- PMC
- PMC3798275
- OpenAlex
- W2080321187
- Cited by
- 320
- References
- 51
- Citations by year
- 2026: 9 · 2025: 9 · 2024: 5 · 2023: 9 · 2022: 14 · 2021: 32 · 2020: 38 · 2019: 28 · 2018: 32 · 2017: 39 · 2016: 40 · 2015: 31 · 2014: 32 · 2013: 2
- FWCI
- 25.55
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS Computational Biology (publishedVersion, public-domain)
- Europe PMC (PubMed Central) (publishedVersion, public-domain)
- PubMed Central (publishedVersion, public-domain)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Figshare (submittedVersion, cc-by)
- Funders
- U.S. Department of Homeland Security
- Alfred P. Sloan Foundation (2010-12-14)
- New York City Department of Health and Mental Hygiene
- Science and Technology Directorate
- Fogarty International Center
- Topics
- Health Sciences › Medicine › Epidemiology › Data-Driven Disease Surveillance
- Health Sciences › Medicine › Epidemiology › Influenza Virus Research Studies
- Life Sciences › Pharmacology, Toxicology and Pharmaceutics › Toxicology › Pharmacovigilance and Adverse Drug Reactions
- Keywords
- Google Flu Trends, influenza surveillance, influenza-like illness, influenza A (H1N1) pandemic, seasonal influenza, H3N2, influenza seasonality, electronic health data, New York City, Mid-Atlantic region
- MeSH terms
- Algorithms, Computer Simulation, Humans, Influenza, Human, Population Surveillance, Seasons, United States, Sentinel Surveillance, Computational Biology, Search Engine, Pandemics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
