CoChat
Sign Up
Cited by 320Data-Driven Disease SurveillanceInfluenza Virus Research StudiesPharmacovigilance and Adverse Drug Reactions

The DOI matches its Crossref record; 51 of the 51 references it cites were found in a registry or an index.

Reassessing Google Flu Trends Data for Detection of Seasonal and Pandemic Influenza: A Comparative Epidemiological Study at Three Geographic Scales

Open access
Olson, Donald R., Konty, Kevin J., Paladini, Marc, Viboud, Cecile, Simonsen, Lone·2013·PLoS Computational Biology
DOI 10.1371/journal.pcbi.1003256PMC PMC3798275
Open this paper in CoChat — free accountOpen PDF

Abstract

The goal of influenza-like illness (ILI) surveillance is to determine the timing, location and magnitude of outbreaks by monitoring the frequency and progression of clinical case incidence. Advances in computational and information technology have allowed for automated collection of higher volumes of electronic data and more timely analyses than previously possible. Novel surveillance systems, including those based on internet search query data like Google Flu Trends (GFT), are being used as surrogates for clinically-based reporting of influenza-like-illness (ILI). We investigated the reliability of GFT during the last decade (2003 to 2013), and compared weekly public health surveillance with search query data to characterize the timing and intensity of seasonal and pandemic influenza at the national (United States), regional (Mid-Atlantic) and local (New York City) levels. We identified substantial flaws in the original and updated GFT models at all three geographic scales, including completely missing the first wave of the 2009 influenza A/H1N1 pandemic, and greatly overestimating the intensity of the A/H3N2 epidemic during the 2012/2013 season. These results were obtained for both the original (2008) and the updated (2009) GFT algorithms. The performance of both models was problematic, perhaps because of changes in internet search behavior and differences in the seasonality, geographical heterogeneity and age-distribution of the epidemics between the periods of GFT model-fitting and prospective use. We conclude that GFT data may not provide reliable surveillance for seasonal or pandemic influenza and should be interpreted with caution until the algorithm can be improved and evaluated. Current internet search query data are no substitute for timely local clinical and laboratory surveillance, or national surveillance based on local data collection. New generation surveillance systems such as GFT should incorporate the use of near-real time electronic health data and computational methods for continued model-fitting and ongoing evaluation and improvement.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1371/journal.pcbi.1003256

Cites

51 of 51 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 Time-series of weekly influenza-like illness (ILI) surveillance and Google Flu Trends (GFT) search query estimates, June 2003–March 2013. Observed weekly ILI proportions (black lines) are shown with Serfling model baseline (gray lines) and 95% epidemic threshold (dashed lines). The periods of the early wave of the 2009 pandemic and the 2012/2013 epidemic are shaded in grey. Sentinel ILI-Net surveillance is shown for (A) the United States and (B) Mid-Atlantic States (New Jersey, New York, Pennsylvania). Local ILI surveillance from emergency department visits is shown for (C) New York City. Scaled GFT internet search query estimates are shown for model-fitting periods for the original (thin red line) and updated (thin blue line) GFT models, and for the periods of prospective operation of the original (thick red line) and updated (thick blue line) GFT models. For Mid-Atlantic States the updated GFT model data represents ILI proportions only for New Jersey and New York (see Supporting Information).
Figure 1 Time-series of weekly influenza-like illness (ILI) surveillance and Google Flu Trends (GFT) search query estimates, June 2003–March 2013. Observed weekly ILI proportions (black lines) are shown with Serfling model baseline (gray lines) and 95% epidemic threshold (dashed lines). The periods of the early wave of the 2009 pandemic and the 2012/2013 epidemic are shaded in grey. Sentinel ILI-Net surveillance is shown for (A) the United States and (B) Mid-Atlantic States (New Jersey, New York, Pennsylvania). Local ILI surveillance from emergency department visits is shown for (C) New York City. Scaled GFT internet search query estimates are shown for model-fitting periods for the original (thin red line) and updated (thin blue line) GFT models, and for the periods of prospective operation of the original (thick red line) and updated (thick blue line) GFT models. For Mid-Atlantic States the updated GFT model data represents ILI proportions only for New Jersey and New York (see Supporting Information).CC0 · PubMed Central
Figure 2 Scatter plots of weekly excess influenza-like illness (ILI) visit proportions against original Google Flu Trends (GFT) model search query estimates, 2003–2009. Weekly excess percent-ILI is calculated as Serfling estimates subtracted from observed proportions. Plots show original GFT model estimates compared with weighted CDC ILI-Net data for (A) the United States, and (B) Mid-Atlantic Census Region States (New Jersey, New York, Pennsylvania), and local ILI surveillance from emergency department visits for (C) New York City. Plots are shown for pre-pandemic influenza seasons, June 1, 2003 to April 25, 2009 (grey circles) and the early wave of the A/H1N1 pandemic, April 26 to August 1, 2009 (red diamonds). Lines representing equivalent axes for X = Y are shown (grey dashed line). Regression lines are shown for seasonal influenza 2003–2009 (black line) and the early 2009 wave of the pandemic (red line).
Figure 2 Scatter plots of weekly excess influenza-like illness (ILI) visit proportions against original Google Flu Trends (GFT) model search query estimates, 2003–2009. Weekly excess percent-ILI is calculated as Serfling estimates subtracted from observed proportions. Plots show original GFT model estimates compared with weighted CDC ILI-Net data for (A) the United States, and (B) Mid-Atlantic Census Region States (New Jersey, New York, Pennsylvania), and local ILI surveillance from emergency department visits for (C) New York City. Plots are shown for pre-pandemic influenza seasons, June 1, 2003 to April 25, 2009 (grey circles) and the early wave of the A/H1N1 pandemic, April 26 to August 1, 2009 (red diamonds). Lines representing equivalent axes for X = Y are shown (grey dashed line). Regression lines are shown for seasonal influenza 2003–2009 (black line) and the early 2009 wave of the pandemic (red line).CC0 · PubMed Central
Figure 3 Scatter plots of weekly excess influenza-like illness (ILI) visit proportions against updated Google Flu Trends (GFT) model search query estimates, 2003–2013. Weekly excess percent-ILI is calculated as Serfling estimates subtracted from observed proportions. Plots show updated GFT model estimates compared with weighted CDC ILI-Net data for (A) the United States, and (B) Mid-Atlantic HHS-2 Region States (New Jersey, New York), and local ILI surveillance from emergency department ILI visit data for (C) New York City. Plots are shown for weeks June 1, 2003 to April 25, 2009 (grey circles), April 26 to January 2, 2010 (red diamonds), January 3, 2010 to Oct 6, 2012 (grey squares), and October 7, 2012 to March 30, 2013 (blue triangles). Lines representing equivalent axes for X = Y are shown (grey dashed line). Regression lines are shown for the 2003/2004–2008/2009 seasons (black line), 2009 pandemic (red line), 2010/2011–2010/2012 seasons (grey solid line) and the 2012/2013 season (blue line).
Figure 3 Scatter plots of weekly excess influenza-like illness (ILI) visit proportions against updated Google Flu Trends (GFT) model search query estimates, 2003–2013. Weekly excess percent-ILI is calculated as Serfling estimates subtracted from observed proportions. Plots show updated GFT model estimates compared with weighted CDC ILI-Net data for (A) the United States, and (B) Mid-Atlantic HHS-2 Region States (New Jersey, New York), and local ILI surveillance from emergency department ILI visit data for (C) New York City. Plots are shown for weeks June 1, 2003 to April 25, 2009 (grey circles), April 26 to January 2, 2010 (red diamonds), January 3, 2010 to Oct 6, 2012 (grey squares), and October 7, 2012 to March 30, 2013 (blue triangles). Lines representing equivalent axes for X = Y are shown (grey dashed line). Regression lines are shown for the 2003/2004–2008/2009 seasons (black line), 2009 pandemic (red line), 2010/2011–2010/2012 seasons (grey solid line) and the 2012/2013 season (blue line).CC0 · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 51 references

Research works printed but not linked: 6

  1. M Helft, 2008, Google Uses Searches to Track Flu's Spread, The New York Times; Sect A, 1
  2. DR Olson, 2009, Searching for better flu surveillance? A brief communication arising, Nature Precedings, 3493
  3. Karon AE, Heffernan RT (2009) Google Flu Trends versus ED Syndromic Surveillance, Wisconsin 2009. Annual Conference of the International Society for Disease Surveillance (ISDS), Miami, FL, December 2009.
  4. M Santora, 2013, New York Declares Health Emergency, The New York Times;Sect A, 21
  5. M Secret, 2013, Pharmacies Pressed to Meet High Demand for Flu Vaccine, The New York Times; Sect A, 16
  6. Simonsen L, Olson DR, Viboud C, Heiman E, Taylor R, <etal>et al</etal>.. (2005) Pandemic Influenza and Mortality: Past Evidence and Projections for the Future. In The Threat of Pandemic Influenza: Are We Ready? Board on Global Health, Institute of Medicine, National Academies Press, DC, 2005, pp 89–106.

Cited by

Sorted by citations

Related

See all

Metadata

Authors
Olson, D. R., Konty, K. J., Paladini, M., Viboud, C., & Simonsen, L.
Affiliations
  • New York City Department of Health and Mental Hygiene (US)
  • National Institutes of Health (US)
Corresponding author
Olson, Donald R.
ORCID
Venue
PLoS Computational Biology
Publisher
Public Library of Science (PLoS)
Published
2013-10-17
Volume
9
Issue
10
Pages
e1003256
ISSN
  • 1553-734X (ISSN-L)
  • 1553-7358
DOI
10.1371/journal.pcbi.1003256
PMC
PMC3798275
OpenAlex
W2080321187
Cited by
320
References
51
Citations by year
2026: 9 · 2025: 9 · 2024: 5 · 2023: 9 · 2022: 14 · 2021: 32 · 2020: 38 · 2019: 28 · 2018: 32 · 2017: 39 · 2016: 40 · 2015: 31 · 2014: 32 · 2013: 2
FWCI
25.55
Type
journal article
Language
en
Open access
Licence
Locations
  • PLoS Computational Biology (publishedVersion, public-domain)
  • Europe PMC (PubMed Central) (publishedVersion, public-domain)
  • PubMed Central (publishedVersion, public-domain)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • Figshare (submittedVersion, cc-by)
Funders
  • U.S. Department of Homeland Security
  • Alfred P. Sloan Foundation (2010-12-14)
  • New York City Department of Health and Mental Hygiene
  • Science and Technology Directorate
  • Fogarty International Center
Topics
  • Health Sciences › Medicine › Epidemiology › Data-Driven Disease Surveillance
  • Health Sciences › Medicine › Epidemiology › Influenza Virus Research Studies
  • Life Sciences › Pharmacology, Toxicology and Pharmaceutics › Toxicology › Pharmacovigilance and Adverse Drug Reactions
Keywords
Google Flu Trends, influenza surveillance, influenza-like illness, influenza A (H1N1) pandemic, seasonal influenza, H3N2, influenza seasonality, electronic health data, New York City, Mid-Atlantic region
MeSH terms
Algorithms, Computer Simulation, Humans, Influenza, Human, Population Surveillance, Seasons, United States, Sentinel Surveillance, Computational Biology, Search Engine, Pandemics

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review