Assessing Google Flu Trends Performance in the United States during the 2009 Influenza Virus A (H1N1) Pandemic
Open accessAbstract
BACKGROUND: Google Flu Trends (GFT) uses anonymized, aggregated internet search activity to provide near-real time estimates of influenza activity. GFT estimates have shown a strong correlation with official influenza surveillance data. The 2009 influenza virus A (H1N1) pandemic [pH1N1] provided the first opportunity to evaluate GFT during a non-seasonal influenza outbreak. In September 2009, an updated United States GFT model was developed using data from the beginning of pH1N1. METHODOLOGY/PRINCIPAL FINDINGS: We evaluated the accuracy of each U.S. GFT model by comparing weekly estimates of ILI (influenza-like illness) activity with the U.S. Outpatient Influenza-like Illness Surveillance Network (ILINet). For each GFT model we calculated the correlation and RMSE (root mean square error) between model estimates and ILINet for four time periods: pre-H1N1, Summer H1N1, Winter H1N1, and H1N1 overall (Mar 2009-Dec 2009). We also compared the number of queries, query volume, and types of queries (e.g., influenza symptoms, influenza complications) in each model. Both models' estimates were highly correlated with ILINet pre-H1N1 and over the entire surveillance period, although the original model underestimated the magnitude of ILI activity during pH1N1. The updated model was more correlated with ILINet than the original model during Summer H1N1 (r = 0.95 and 0.29, respectively). The updated model included more search query terms than the original model, with more queries directly related to influenza infection, whereas the original model contained more queries related to influenza complications. CONCLUSIONS: Internet search behavior changed during pH1N1, particularly in the categories "influenza complications" and "term for influenza." The complications associated with pH1N1, the fact that pH1N1 began in the summer rather than winter, and changes in health-seeking behavior each may have played a part. Both GFT models performed well prior to and during pH1N1, although the updated model performed better during pH1N1, especially during the summer months.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The ad hoc method of throwing out peculiar search terms failed when GFT completely missed the nonseasonal 2009 influenza A-H1N1 pandemic.
“This ad hoc method of throwing out peculiar search terms failed when GFT completely missed the nonseasonal 2009 infl uenza A–H1N1 pandemic ( 2, 14).”
From The Parable of Google Flu: Traps in Big Data Analysis (2014) · cites this paper as [14]
The Abstract (Methodology/Principal Findings) says the original GFT model underestimated the magnitude of illness during the 2009 H1N1 pandemic, not that it completely missed it. The Abstract (Conclusions) says both models performed well during the pandemic, although the updated model performed better. The Discussion notes that pandemic-specific queries were excluded, but does not describe this as an ad hoc method of throwing out peculiar terms. (LLM, conf=0.88)
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe few search terms offered in the GFT papers do not seem to be strongly related with either GFT or the CDC data.
“Oddly, the few search terms offered in the papers ( 14) do not seem to be strongly related with either GFT or the CDC data (SM)—we surmise that the authors felt an unarticulated need to cloak the actual search terms identifi ed.”
From The Parable of Google Flu: Traps in Big Data Analysis (2014) · cites this paper as [14]
The Results section (‘Search Behavior During pH1N1’; page unavailable) says that all three search queries shown in Figure 3 closely tracked CDC ILINet data before the pandemic, and ‘symptoms of flu’ continued to do so during it. The paper does report that the bronchitis and pneumonia queries underestimated ILINet data during the pandemic, but that does not support the broad claim that the offered terms were not strongly related to the data. (LLM, conf=0.87)
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredGFT has never documented the 45 search terms used, and the examples that have been released appear misleading.
“GFT has never documented the 45 search terms used, and the examples that have been released appear misleading ( 14) (SM).”
From The Parable of Google Flu: Traps in Big Data Analysis (2014) · cites this paper as [14]
The Results (Model Composition) show that only 14% of original-model queries contained “flu” and that complication-related queries accounted for 48% of its query volume, supporting concern that flu-focused examples could misrepresent the model. But the paper does not say that GFT never documented its search terms or identify previously released examples as misleading; it describes the original model as having approximately 40 queries, not 45. (LLM, conf=0.72)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
MapReduce: simplified data processing on large clusters
Dean, J., & Ghemawat, S. (2008) · Communications of the ACM · doi:10.1145/1327452.1327492 · Cited by 18,476 · see citing works
Detecting influenza epidemics using search engine query data
Ginsberg, J., Mohebbi, M. H., Patel, R. S., Brammer, L., Smolinski, M. S., & Brilliant, L. (2009) · Nature · doi:10.1038/nature07634 · Cited by 4,295 · see citing works
Self-citationEmergence of a Novel Swine-Origin Influenza A (H1N1) Virus in Humans
Novel Swine-Origin Influenza A (H1N1) Virus Investigation Team (2009) · New England Journal of Medicine · doi:10.1056/nejmoa0903810 · Cited by 3,047 · see citing works
Infodemiology: tracking flu-related searches on the web for syndromic surveillance.
Eysenbach, G. (2006) · PubMed · Cited by 414 · see citing works
Kelly, H., & Grant, K. A. (2009) · Eurosurveillance · doi:10.2807/ese.14.31.19288-en · Cited by 193 · see citing works
Interpreting “Google Flu Trends” data for pandemic H1N1 influenza: The New Zealand experience
Wilson, N., Mason, K., Tobias, M., Peacey, M., Huang, Q. S., & Baker, M. G. (2009) · Eurosurveillance · doi:10.2807/ese.14.44.19386-en · Cited by 78 · see citing works
Brammer, L., Blanton, L., Epperson, S., Mustaquim, D., Bishop, A., Kniss, K. L., Dhara, R., Nowell, M., Kamimoto, L., & Finelli, L. (2010) · Clinical Infectious Diseases · doi:10.1093/cid/ciq009 · Cited by 67 · see citing works
Influenza activity--United States and worldwide, 2007-08 season.
(2008) · PubMed · Cited by 63 · see citing works
Reed, C., Angulo, F. J., Biggerstaff, M., Swerdlow, D. L., & Finelli, L. (2010) · Clinical Infectious Diseases · doi:10.1093/cid/ciq024 · Cited by 16 · see citing works
Flagged first, then by citations
1 of the 10 references couldn’t be matched to a known paper.
Research works printed but not linked: 1
- [5]Wilson N Mason K Tobias M Peacey M Huang QS 2009 Interpreting “GFT” Data for Pandemic H1N1: The New Zealand Experience. EuroSurveill 14 44 pii = 19386 Available: http://www.eurosurveillance.org/ViewArticle.aspx?ArticleId=19386. Accessed 2011 July 28 19941777
Cited by
Sorted by citations
The Parable of Google Flu: Traps in Big Data Analysis
Lazer, D., Kennedy, R., King, G., & Vespignani, A. (2014) · Science · doi:10.1126/science.1248506 · Cited by 2,523 · see citing works
Towards a genomics-informed, real-time, global pathogen surveillance system
Gardy, J. L., & Loman, N. J. (2018) · Nature Reviews Genetics · doi:10.1038/nrg.2017.88 · Cited by 771 · see citing works
Jusup, M., Holme, P., Kanazawa, K., Takayasu, M., Romić, I., Wang, Z., Geček, S., Lipić, T., Boris Podobnik, A., Wang, L., et al. (2022) · Physics Reports · doi:10.1016/j.physrep.2021.10.005 · Cited by 662 · see citing works
Reviewing the History of Pandemic Influenza: Understanding Patterns of Emergence and Transmission
Saunders‐Hastings, P., & Krewski, D. R. (2016) · Pathogens · doi:10.3390/pathogens5040066 · Cited by 566 · see citing works
Broniatowski, D. A., Paul, M., & Dredze, M. (2013) · PLoS ONE · doi:10.1371/journal.pone.0083672 · Cited by 474 · see citing works
Combining Search, Social Media, and Traditional Data Sources to Improve Influenza Surveillance
Santillana, M., Nguyen, A., Dredze, M., Paul, M., Nsoesie, E., & Brownstein, J. (2015) · PLoS Comput. Biol. · doi:10.1371/journal.pcbi.1004513 · Cited by 441 · see citing works
Accurate estimation of influenza epidemics using Google search data via ARGO
Yang, S., Santillana, M., & Kou, S. (2015) · Proceedings of the National Academy of Sciences of the United States of America · doi:10.1073/pnas.1515373112 · Cited by 372 · see citing works
Social Media– and Internet-Based Disease Surveillance for Public Health
Aiello, A. E., Renson, A., & Zivich, P. N. (2020) · Annual Review of Public Health · doi:10.1146/annurev-publhealth-040119-094402 · Cited by 363 · see citing works
Influenza Forecasting with Google Flu Trends
Dugas, A. F., Jalalpour, M., Gel, Y. R., Levin, S., Torcaso, F., Igusa, T., & Rothman, R. E. (2013) · PLoS ONE · doi:10.1371/journal.pone.0056176 · Cited by 349 · see citing works
Olson, D. R., Konty, K. J., Paladini, M., Viboud, C., & Simonsen, L. (2013) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1003256 · Cited by 320 · see citing works
Internet-based surveillance systems for monitoring emerging infectious diseases
Milinovich, G. J., Williams, G. M., Clements, A., & Hu, W. (2013) · The Lancet Infectious Diseases · doi:10.1016/s1473-3099(13)70244-5 · Cited by 306 · see citing works
Association of the COVID-19 pandemic with internet search volumes: a Google Trends™ analysis
Effenberger, M., Kronbichler, A., Shin, J. I., Mayer, G., Tilg, H., & Perco, P. (2020) · International Journal of Infectious Diseases · doi:10.1016/j.ijid.2020.04.033 · Cited by 270 · see citing works
Ethical challenges of big data in public health
Vayena, E., Salathé, M., Madoff, L. C., & Brownstein, J. S. (2015) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1003904 · Cited by 268 · see citing works
Separating Fact from Fear: Tracking Flu Infections on Twitter
Lamb, A., Paul, M. J., & Dredze, M. H. (2013) · Cited by 247 · see citing works
Mass Media and the Contagion of Fear: The Case of Ebola in America
Towers, S., Afzal, S., Bernal, G., Bliss, N. T., Brown, S., Espinoza, B., Jackson, J., Judson-Garcia, J., Khan, M., Lin, M. R., et al. (2015) · PLoS ONE · doi:10.1371/journal.pone.0129179 · Cited by 224 · see citing works
Digital epidemiology: what is it, and where is it going?
Salathé, M. (2018) · Life Sciences Society and Policy · doi:10.1186/s40504-017-0065-7 · Cited by 215 · see citing works
Yang, W., Karspeck, A. R., & Shaman, J. L. (2014) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1003583 · Cited by 201 · see citing works
McIver, D. J., & Brownstein, J. S. (2014) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1003581 · Cited by 195 · see citing works
Using data-driven agent-based models for forecasting emerging infectious diseases
Venkatramanan, S., Lewis, B. L., Chen, J., Higdon, D., Vullikanti, A. K. S., & MARATHE, M. V. (2017) · Epidemics · doi:10.1016/j.epidem.2017.02.010 · Cited by 190 · see citing works
Social Media Analyses for Social Measurement
Schober, M. F., Josh Pasek, B., Guggenheim, L., Lampe, C. A. C., & Conrad, F. G. (2016) · Public Opinion Quarterly · doi:10.1093/poq/nfv048 · Cited by 188 · see citing works
What Can Digital Disease Detection Learn from (an External Revision to) Google Flu Trends?
Santillana, M., Wendong, D., Althouse, B. M., & Ayers, J. W. (2014) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2014.05.020 · Cited by 183 · see citing works
Predicting Asthma-Related Emergency Department Visits Using Big Data
Ram, S., Zhang, W., Williams, M., & Pengetnze, Y. (2015) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2015.2404829 · Cited by 180 · see citing works
McGough, S. F., Brownstein, J. S., Hawkins, J. B., & Santillana, M. (2017) · PLoS neglected tropical diseases · doi:10.1371/journal.pntd.0005295 · Cited by 176 · see citing works
Using Google Trends for Influenza Surveillance in South China
Kang, M., Zhong, H., He, J., Rutherford, S., & Yang, F. (2013) · PLoS ONE · doi:10.1371/journal.pone.0055205 · Cited by 171 · see citing works
for the Influenza Forecasting Contest Working Group, Biggerstaff, M., Alper, D., Dredze, M. H., Fox, S. J., Fung, I. C., Hickmann, K. S., Lewis, B. L., Rosenfeld, R., Shaman, J. L., et al. (2016) · BMC Infectious Diseases · doi:10.1186/s12879-016-1669-x · Cited by 165 · see citing works
Social Listening: A Content Analysis of E-Cigarette Discussions on Twitter
Cole-Lewis, H. J., Pugatch, J., Sanders, A. E., Varghese, A., Posada, S., Yun, C., Schwarz, M., & Augustson, E. M. (2015) · Journal of Medical Internet Research · doi:10.2196/jmir.4969 · Cited by 161 · see citing works
New Data Sources in Social Science Research: Things to Know Before Working With Reddit Data
Amaya, A., Bach, R. L., Keusch, F., & Kreuter, F. (2019) · Social Science Computer Review · doi:10.1177/0894439319893305 · Cited by 158 · see citing works
The Measles Vaccination Narrative in Twitter: A Quantitative Analysis
Radzikowski, J., Stefanidis, A., Jacobsen, K. H., Croitoru, A., Crooks, A. T., & Delamater, P. L. (2016) · JMIR Public Health and Surveillance · doi:10.2196/publichealth.5059 · Cited by 147 · see citing works
Infectious Disease Surveillance in the Big Data Era: Towards Faster and Locally Relevant Systems
Simonsen, L., Gog, J. R., Olson, D., & Viboud, C. G. (2016) · The Journal of Infectious Diseases · doi:10.1093/infdis/jiw376 · Cited by 146 · see citing works
Is the Sky Falling? New Technology, Changing Media, and the Future of Surveys
Couper, M. P. (2013) · Survey research methods · doi:10.18148/srm/2013.v7i3.5751 · Cited by 144 · see citing works
Advances in nowcasting influenza-like illness rates using search query logs
Lampos, V., Miller, A. C., Crossan, S., & Stefansen, C. (2015) · Scientific Reports · doi:10.1038/srep12760 · Cited by 143 · see citing works
Choi, J., Cho, Y., Shim, E., & Woo, H. (2016) · BMC Public Health · doi:10.1186/s12889-016-3893-0 · Cited by 140 · see citing works
Surveillance for emerging respiratory viruses
Al‐Tawfiq, J. A., Zumla, A. I., Gautret, P., Gray, G. C., Hui, D. S. C., Al-Rabeeah, A. A., & Memish, Z. A. (2014) · The Lancet Infectious Diseases · doi:10.1016/s1473-3099(14)70840-0 · Cited by 139 · see citing works
Machine-learned epidemiology: real-time detection of foodborne illness at scale
Sadilek, A., Caty, S. N., DiPrete, L., Mansour, R., Schenk, T. L., Bergtholdt, M., Jha, A., Ramaswami, P., & Gabrilovich, E. (2018) · npj Digital Medicine · doi:10.1038/s41746-018-0045-1 · Cited by 136 · see citing works
Correlation between National Influenza Surveillance Data and Google Trends in South Korea
Cho, S. J., Sohn, C. H., Jo, M. W., Shin, S.-Y., Lee, J.-H., Ryoo, S. M., Kim, W. Y., & Seo, D. (2013) · PLoS ONE · doi:10.1371/journal.pone.0081422 · Cited by 127 · see citing works
A review of influenza detection and prediction through social networking sites
Alessa, A., & Faezipour, M. (2018) · Theoretical Biology and Medical Modelling · doi:10.1186/s12976-017-0074-5 · Cited by 126 · see citing works
Flexible Modeling of Epidemics with an Empirical Bayes Framework
Brooks, L., Farrow, D., Hyun, S., Tibshirani, R. J., & Rosenfeld, R. (2015) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1004382 · Cited by 119 · see citing works
Skip the Trip: Air Travelers' Behavioral Responses to Pandemic Influenza
Fenichel, E. P., Kuminoff, N. V., & Chowell, G. (2013) · PLoS ONE · doi:10.1371/journal.pone.0058249 · Cited by 114 · see citing works
Gao, J. G., Zhang, Y., & Zhou, T. (2019) · Physics Reports · doi:10.1016/j.physrep.2019.05.002 · Cited by 118 · see citing works
Can Big Data Protect a Firm from Competition?Preprint
Lambrecht, A., & Tucker, C. E. (2015) · SSRN Electronic Journal · doi:10.2139/ssrn.2705530 · Cited by 109 · see citing works
Ng, K. Y., & Gui, M. M. (2020) · Physica D Nonlinear Phenomena · doi:10.1016/j.physd.2020.132599 · Cited by 108 · see citing works
Low validity of Google Trends for behavioral forecasting of national suicide rates
Tran, U. S., Andel, R., Niederkrotenthaler, T., Till, B., Ajdacic‐Gross, V., & Voracek, M. (2017) · PLoS ONE · doi:10.1371/journal.pone.0183149 · Cited by 105 · see citing works
Smartphone apps in the COVID-19 pandemic
Pandit, J. A., Radin, J. M., Quer, G., & Topol, E. J. (2022) · Nature Biotechnology · doi:10.1038/s41587-022-01350-x · Cited by 101 · see citing works
Advances in using Internet searches to track dengue
Yang, S., Kou, S. C., Lu, F., Brownstein, J., Brooke, N., & Santillana, M. (2016) · PLoS Comput. Biol. · doi:10.1371/journal.pcbi.1005607 · Cited by 94 · see citing works
Adaptive nowcasting of influenza outbreaks using Google searches
Preis, T., & Moat, H. S. (2014) · Royal Society Open Science · doi:10.1098/rsos.140095 · Cited by 97 · see citing works
Using electronic health records and Internet search information for accurate influenza forecasting
Yang, S., Santillana, M., Brownstein, J. S., Gray, J., Richardson, S., & Kou, S. C. (2017) · BMC Infectious Diseases · doi:10.1186/s12879-017-2424-7 · Cited by 91 · see citing works
Precision global health in the digital age
Flahault, A., Geissbühler, A., Guessous, I., Guerin, P. J., Bolon, I., Salathé, M., & Escher, G. (2017) · Swiss Medical Weekly · doi:10.4414/smw.2017.14423 · Cited by 91 · see citing works
Bayesian Nonparametric Covariance RegressionPreprint
Fox, E. B., & Dunson, D. B. (2011) · arXiv (Cornell University) · doi:10.5555/2789272.2912079 · Cited by 88 · see citing works
Yesterday, Today and Tomorrow of Big Data
Özköse, H., Arı, E. S., & Gencer, C. (2015) · Procedia - Social and Behavioral Sciences · doi:10.1016/j.sbspro.2015.06.147 · Cited by 87 · see citing works
Kostkova, P., Szomszor, M., & Louis, C. S. (2014) · ACM Transactions on Management Information Systems · doi:10.1145/2597892 · Cited by 89 · see citing works
Phillips, C., Leahy, A. B., Li, Y., Schapira, M. M., Bailey, L. C., & Merchant, R. M. (2018) · Journal of Medical Internet Research · doi:10.2196/jmir.8870 · Cited by 82 · see citing works
Woo, H., Cho, Y., Shim, E., Lee, J., Lee, C.-G., & Kim, S. H. (2016) · Journal of Medical Internet Research · doi:10.2196/jmir.4955 · Cited by 81 · see citing works
Salathé, M. (2016) · The Journal of Infectious Diseases · doi:10.1093/infdis/jiw281 · Cited by 81 · see citing works
Total Error in a Big Data World: Adapting the TSE Framework to Big Data
Amaya, A., Biemer, P. P., & Kinyon, D. L. (2019) · Journal of Survey Statistics and Methodology · doi:10.1093/jssam/smz056 · Cited by 77 · see citing works
Milinovich, G. J., Avril, S. M. R., Clements, A., Brownstein, J. S., Tong, S., & Hu, W. (2014) · BMC Infectious Diseases · doi:10.1186/s12879-014-0690-1 · Cited by 77 · see citing works
Cloud-based Electronic Health Records for Real-time, Region-specific Influenza Surveillance
Santillana, M., Nguyen, A. T., Louie, T., Zink, A., Gray, J., Sung, I., & Brownstein, J. (2015) · Scientific Reports · doi:10.1038/srep25732 · Cited by 84 · see citing works
Sharpe, J. D., Hopkins, R. S., Cook, R. L., & Striley, C. W. (2016) · JMIR Public Health and Surveillance · doi:10.2196/publichealth.5901 · Cited by 74 · see citing works
Using Clinicians’ Search Query Data to Monitor Influenza Epidemics
Santillana, M., Nsoesie, E., Mekaru, S. R., Scales, D., & Brownstein, J. (2014) · Clinical Infectious Diseases · doi:10.1093/cid/ciu647 · Cited by 81 · see citing works
Influenza-Like Illness Surveillance on Twitter through Automated Learning of Naïve Language
Gesualdo, F., Stilo, G., Agricola, E., Gonfiantini, M. V., Pandolfi, E., Velardi, P., & Tozzi, A. E. (2013) · PLoS ONE · doi:10.1371/journal.pone.0082489 · Cited by 68 · see citing works
How to Exploit Twitter for Public Health Monitoring?
Krieck, M., Otrusina, L., Smrž, P., Dolog, P., Nejdl, W., Velasco, E., & Denecke, K. (2013) · Methods of Information in Medicine · doi:10.3414/me12-02-0010 · Cited by 70 · see citing works
Can we trust Big Data? Applying philosophy of science to software
Symons, J., & Alvarado, R. (2016) · Big Data & Society · doi:10.1177/2053951716664747 · Cited by 69 · see citing works
Cole-Lewis, H. J., Varghese, A., Sanders, A. E., Schwarz, M., Pugatch, J., & Augustson, E. M. (2015) · Journal of Medical Internet Research · doi:10.2196/jmir.4392 · Cited by 68 · see citing works
Using Networks to Combine “Big Data” and Traditional Surveillance to Improve Influenza Predictions
Davidson, M. W., Haim, D. A., & Radin, J. M. (2015) · Scientific Reports · doi:10.1038/srep08154 · Cited by 68 · see citing works
Data-driven methods for present and future pandemics: Monitoring, modelling and managing
Álamo, T., Reina, D. G., Millán, P., Preciado, V. M., & Giordano, G. (2021) · Annual Reviews in Control · doi:10.1016/j.arcontrol.2021.05.003 · Cited by 68 · see citing works
Paguio, J. A., Yao, J. S., & Dee, E. C. (2020) · Vaccine · doi:10.1016/j.vaccine.2020.06.069 · Cited by 64 · see citing works
Using search queries for malaria surveillance, Thailand
Ocampo, A., Chunara, R., & Brownstein, J. S. (2013) · Malaria Journal · doi:10.1186/1475-2875-12-390 · Cited by 65 · see citing works
A “big data” approach to HIV epidemiology and prevention
Young, S. D. (2014) · Preventive Medicine · doi:10.1016/j.ypmed.2014.11.002 · Cited by 63 · see citing works
Viboud, C. G., Charu, V., Olson, D. R., Ballesteros, S. P., Gog, J. R., Khan, F. L., Grenfell, B. T., & Simonsen, L. (2014) · PLoS ONE · doi:10.1371/journal.pone.0102429 · Cited by 63 · see citing works
Social Monitoring for Public Health
Paul, M., & Dredze, M. (2017) · Synthesis Lectures on Information Concepts Retrieval and Services · doi:10.1007/978-3-031-02311-8 · Cited by 109 · see citing works
Awaiting the Second Big Data Revolution: From Digital Noise to Value Creation
Huberty, M. (2015) · Journal of Industry Competition and Trade · doi:10.1007/s10842-014-0190-4 · Cited by 59 · see citing works
Early Stage Influenza Detection from TwitterPreprint
Li, J., & Cardie, C. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1309.7340 · Cited by 59 · see citing works
Kundu, S., Ganganaik, P. B., Louis, J., Chalamalasetty, H., & Rao, B. P. (2022) · IEEE Transactions on Very Large Scale Integration (VLSI) Systems · doi:10.1109/tvlsi.2022.3161847 · Cited by 57 · see citing works
Early identification of adverse drug reactions from search log data
White, R. W., Wang, S., Pant, A., Harpaz, R., Shukla, P., Sun, W., DuMouchel, W., & Horvitz, E. (2015) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2015.11.005 · Cited by 57 · see citing works
Disease surveillance data sharing for public health: the next ethical frontiers
Kostkova, P. (2018) · Life Sciences Society and Policy · doi:10.1186/s40504-018-0078-x · Cited by 57 · see citing works
Jun, S., Yoo, H. S., & Lee, J. (2021) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2021.120592 · Cited by 62 · see citing works
Implications of non-stationarity on predictive modeling using EHRs
Jung, K., & Shah, N. H. (2015) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2015.10.006 · Cited by 55 · see citing works
Big Data in Organizations and the Role of Human Resource Management
Scholz, T. M. (2017) · Peter Lang D eBooks · doi:10.3726/b10907 · Cited by 69 · see citing works
Williams, S. (2020) · The MIT Press eBooks · doi:10.7551/mitpress/12261.001.0001 · Cited by 65 · see citing works
Google Flu Trends Still Appears Sick: An Evaluation of the 2013-2014 Flu SeasonPreprint
Lazer, D., Kennedy, R., King, G., & Vespignani, A. (2014) · SSRN Electronic Journal · doi:10.2139/ssrn.2408560 · Cited by 53 · see citing works
Big data in safety management: An overview
Wang, B., & Wang, Y. (2021) · Safety Science · doi:10.1016/j.ssci.2021.105414 · Cited by 51 · see citing works
Predicting New Diagnoses of HIV Infection Using Internet Search Engine Data
Jena, A. B., Karaca‐Mandic, P., Weaver, L., & Seabury, S. A. (2013) · Clinical Infectious Diseases · doi:10.1093/cid/cit022 · Cited by 50 · see citing works
Ethical Issues in Health Research With Novel Online Sources
Vayena, E., Mastroianni, A. C., & Kahn, J. P. (2012) · American Journal of Public Health · doi:10.2105/ajph.2012.300813 · Cited by 50 · see citing works
Wang, H.-W., Chen, D., Yu, H.-W., & Chen, Y. (2015) · Journal of Medical Internet Research · doi:10.2196/jmir.4516 · Cited by 50 · see citing works
Using Social Media to Predict the Future: A Systematic Literature Review
Phillips, L., Dowling, C., Shaffer, K., Hodas, N. O., & Volkova, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.06134 · Cited by 76 · see citing works
Toward the use of neural networks for influenza prediction at multiple spatial resolutions
Aiken, E., Nguyen, A. T., Viboud, C. G., & Santillana, M. (2021) · Science Advances · doi:10.1126/sciadv.abb1237 · Cited by 50 · see citing works
Can Google Trends data improve forecasting of Lyme disease incidence?
Kapitány‐Fövény, M., Ferenci, T., Sulyok, Z., Kegele, J., Richter, H., Valyi-Nagy, I. T., & Sulyok, M. (2018) · Zoonoses and Public Health · doi:10.1111/zph.12539 · Cited by 49 · see citing works
Turn to the Internet First? Using Online Medical Behavioral Data to Forecast COVID-19 Epidemic Trend
Huang, W., Cao, B., Yang, G., Luo, N., & Chao, N. (2020) · Information Processing & Management · doi:10.1016/j.ipm.2020.102486 · Cited by 48 · see citing works
Baltrusaitis, K., Brownstein, J. S., Scarpino, S. V., Bakota, E. V., Crawley, A. W., Conidi, G., Gunn, J. E., Gray, J., Zink, A., & Santillana, M. (2018) · BMC Infectious Diseases · doi:10.1186/s12879-018-3322-3 · Cited by 48 · see citing works
Katsikopoulos, K. V., Şimşek, Ö., Buckmann, M., & Gigerenzer, G. (2021) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2020.12.006 · Cited by 48 · see citing works
Social Monitoring for Public Health
Paul, M. J., & Dredze, M. H. (2017) · Synthesis lectures on information concepts, retrieval, and services · doi:10.2200/s00791ed1v01y201707icr060 · Cited by 50 · see citing works
Psychological AI: Designing Algorithms Informed by Human Psychology
Gigerenzer, G. (2023) · Perspectives on Psychological Science · doi:10.1177/17456916231180597 · Cited by 50 · see citing works
Crisis Standards of Care: A Toolkit for Indicators and Triggers
Hanfling, D., Hick, J. L., Stroud, C., & Triggers (2013) · Cited by 43 · see citing works
Classification in the Wild: The Science and Art of Transparent Decision Making
Katsikopoulos, K. V., Şimşek, Ö., Buckmann, M., & Gigerenzer, G. (2021) · doi:10.7551/mitpress/11790.001.0001 · Cited by 42 · see citing works
Infodemiology and Infoveillance of Multiple Sclerosis in Italy
Bragazzi, N. L. (2013) · Multiple Sclerosis International · doi:10.1155/2013/924029 · Cited by 41 · see citing works
Influenza Virus: Tracking, Predicting, and Forecasting
Ali, S. T., & Cowling, B. J. (2020) · Annual Review of Public Health · doi:10.1146/annurev-publhealth-010720-021049 · Cited by 41 · see citing works
Introduction to supervised machine learning in clinical epidemiology
Ono, S., & Goto, T. (2022) · Annals of Clinical Epidemiology · doi:10.37737/ace.22009 · Cited by 40 · see citing works
Ayers, J. W., Althouse, B. M., Johnson, M. L., Dredze, M. H., & Cohen, J. E. (2014) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2014.02.003 · Cited by 40 · see citing works
Cumulative Query Method for Influenza Surveillance Using Search Engine Data
Seo, D., Jo, M., Sohn, C. H., Shin, S.-Y., Lee, J., Yu, M., Kim, W., Lim, K. S., & Lee, S. (2014) · Journal of Medical Internet Research · doi:10.2196/jmir.3680 · Cited by 39 · see citing works
Predicting Attributes of Nodes Using Network Structure
Ali, S., Shakeel, M. H., Khan, I., Faizullah, S., & Khan, M. A. (2021) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3442390 · Cited by 39 · see citing works
Doan, S., Lucila Ohno-machado, A., & Collier, N. (2012) · Cited by 39 · see citing works
- Authors
- Cook, S., Conrad, C., Fowlkes, A. L., & Mohebbi, M. H.
- Affiliations
- Google (United States) (US)
- Google (United Kingdom) (GB)
- Centers for Disease Control and Prevention (US)
- Corresponding author
- Conrad, Corrie
- ORCID
- Fowlkes, Ashley L.
- Venue
- PLoS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2011-08-19
- Volume
- 6
- Issue
- 8
- Pages
- e23610
- ISSN
- 1932-6203 (ISSN-L)
- PMC
- PMC3158788
- OpenAlex
- W1998956117
- Cited by
- 381
- References
- 10
- Citations by year
- 2026: 5 · 2025: 4 · 2024: 7 · 2023: 8 · 2022: 25 · 2021: 33 · 2020: 50 · 2019: 25 · 2018: 26 · 2017: 34 · 2016: 41 · 2015: 48 · 2014: 38 · 2013: 27 · 2012: 9
- FWCI
- 18.08
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS ONE (publishedVersion, public-domain)
- Europe PMC (PubMed Central) (publishedVersion, public-domain)
- PubMed Central (publishedVersion, public-domain)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Topics
- Health Sciences › Medicine › Epidemiology › Data-Driven Disease Surveillance
- Health Sciences › Medicine › Epidemiology › Influenza Virus Research Studies
- Health Sciences › Medicine › Epidemiology › Respiratory viral infections research
- Keywords
- Google Flu Trends, influenza surveillance, 2009 H1N1 pandemic, influenza-like illness, online search behavior, health-seeking behavior, model accuracy assessment
- MeSH terms
- Humans, Influenza, Human, Models, Biological, Population Surveillance, Time Factors, United States, Influenza A Virus, H1N1 Subtype, Search Engine, Pandemics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
