Discovering foodborne illness in online restaurant reviews
Open accessAbstract
Objective: We developed a system for the discovery of foodborne illness mentioned in online Yelp restaurant reviews using text classification. The system is used by the New York City Department of Health and Mental Hygiene (DOHMH) to monitor Yelp for foodborne illness complaints. Materials and Methods: We built classifiers for 2 tasks: (1) determining if a review indicated a person experiencing foodborne illness and (2) determining if a review indicated multiple people experiencing foodborne illness. We first developed a prototype classifier in 2012 for both tasks using a small labeled dataset. Over years of system deployment, DOHMH epidemiologists labeled 13 526 reviews selected by this classifier. We used these biased data and a sample of complementary reviews in a principled bias-adjusted training scheme to develop significantly improved classifiers. Finally, we performed an error analysis of the best resulting classifiers. Results: We found that logistic regression trained with bias-adjusted augmented data performed best for both classification tasks, with F1-scores of 87% and 66% for tasks 1 and 2, respectively. Discussion: Our error analysis revealed that the inability of our models to account for long phrases caused the most errors. Our bias-adjusted training scheme illustrates how to improve a classification system iteratively by exploiting available biased labeled data. Conclusions: Our system has been instrumental in the identification of 10 outbreaks and 8523 complaints of foodborne illness associated with New York City restaurants since July 2012. Our evaluation has identified strong classifiers for both tasks, whose deployment will allow DOHMH epidemiologists to more effectively monitor Yelp for foodborne illness investigations.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Social media has an opportune disease surveillance area, improving its capability to detect disease outbreaks.
“The platform has an opportune disease surveillance area, improving its capability to detect disease outbreaks.”
From Social Media Role and Its Impact on Public Health: A Narrative Review (2023) · cites this paper as [19]
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538 · see citing works
Cortes, C., & Vapnik, V. (1995) · Machine Learning · doi:10.1007/bf00994018 · Cited by 59,421 · see citing works
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1994) · doi:10.1201/9780429246593 · Cited by 40,500 · see citing works
Cortes, C., & Vapnik, V. N. (1995) · Machine Learning · doi:10.1023/a:1022627411411 · Cited by 34,565 · see citing works
(1992) · Cited by 23,431 · see citing works
An Introduction to the Bootstrap.
Buckland, S. T., Efron, B., & Tibshirani, R. J. (1994) · Biometrics · doi:10.2307/2532810 · Cited by 3,236 · see citing works
The Regression Analysis of Binary Sequences
Cox, D. R. (1958) · Journal of the Royal Statistical Society Series B (Statistical Methodology) · doi:10.1111/j.2517-6161.1958.tb00292.x · Cited by 2,909 · see citing works
Surveillance for Foodborne Disease Outbreaks—United States, 2006
Gould, L. H., Walsh, K. A., Vieira, A. R., Herman, K., Williams, I. T., Hall, A. J., & Cole, D. J. (2009) · Annals of Emergency Medicine · doi:10.1016/j.annemergmed.2009.11.004 · Cited by 2,372 · see citing works
Rajaraman, A., & Ullman, J. D. (2011) · Cambridge University Press eBooks · doi:10.1017/cbo9781139058452 · Cited by 2,307 · see citing works
Leskovec, J., Rajaraman, A., & Ullman, J. D. (2014) · Cambridge University Press eBooks · doi:10.1017/cbo9781139924801 · Cited by 1,083 · see citing works
Foodborne Illness Acquired in the United States—Unspecified Agents
Scallan, E., Griffin, P. M., Angulo, F. J., Tauxe, R. V., & Hoekstra, R. M. (2010) · Emerging infectious diseases · doi:10.3201/eid1701.p21101 · Cited by 1,048 · see citing works
Surveillance for Foodborne Disease Outbreaks—United States, 2009-2010
del Portal, D. A., & Karras, D. J. (2013) · Annals of Emergency Medicine · doi:10.1016/j.annemergmed.2013.04.001 · Cited by 719 · see citing works
Freifeld, C. C., Mandl, K. D., Reis, B. Y., & Brownstein, J. S. (2007) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m2544 · Cited by 505 · see citing works
Combining Search, Social Media, and Traditional Data Sources to Improve Influenza Surveillance
Santillana, M., Nguyen, A., Dredze, M., Paul, M., Nsoesie, E., & Brownstein, J. (2015) · PLoS Comput. Biol. · doi:10.1371/journal.pcbi.1004513 · Cited by 441 · see citing works
Meerhoff, T. J., Simaku, A., Ulqinaku, D., Torosyan, L., Gribkova, N. V., Shimanovich, V., Chakhunashvili, G., Karseladze, I., Yesmagambetova, A., Kuatbayeva, A. M., et al. (2015) · BMC Infectious Diseases · doi:10.1186/s12879-014-0722-x · Cited by 304 · see citing works
Online reports of foodborne illness capture foods implicated in official foodborne outbreak reports
Nsoesie, E. O., Kluberg, S. A., & Brownstein, J. S. (2014) · Preventive Medicine · doi:10.1016/j.ypmed.2014.08.003 · Cited by 89 · see citing works
Karr, J. R., Williams, A. H., Zucker, J., Raue, A., Steiert, B., Timmer, J., Kreutz, C., Wilkinson, S., Allgood, B., Bot, B. M., et al. (2015) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1004096 · Cited by 73 · see citing works
Health department use of social media to identify foodborne illness - Chicago, Illinois, 2013-2014.
Harris, J. K., Mansour, R., Choucair, B., Olson, J., Nissen, C., & Bhatt, J. (2014) · PubMed Central · Cited by 70 · see citing works
Harrison, C., Jorder, M., Stern, H., Stavinsky, F., Reddy, V., Hanson, H., Waechter, H., Lowe, L., Gravano, L., & Balter, S. (2014) · PubMed Central · Cited by 67 · see citing works
Self-citationDeploying nEmesis: Preventing Foodborne Illness by Data Mining Social Media
Sadilek, A., Kautz, H., DiPrete, L., Labus, B., Portman, E., Teitel, J., & Silenzio, V. (2017) · AI Magazine · doi:10.1609/aimag.v38i1.2711 · Cited by 56 · see citing works
Deploying nEmesis: Preventing Foodborne Illness by Data Mining Social Media
Sadilek, A., Kautz, H., DiPrete, L., Labus, B., Portman, E., Teitel, J., & Silenzio, V. (2016) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i2.19072 · Cited by 35 · see citing works
Comparing timeliness, content, and disease severity of formal and informal source outbreak reporting
Bahk, C. Y., Scales, D. A., Mekaru, S. R., Brownstein, J. S., & Freifeld, C. C. (2015) · BMC Infectious Diseases · doi:10.1186/s12879-015-0885-0 · Cited by 28 · see citing works
Flagged first, then by citations
2 of the 24 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Transparency in food supply chains: A review of enabling technology solutions
Astill, J., Dara, R. A., Campbell, M., Farber, J. M., Fraser, E., Sharif, S., & Yada, R. Y. (2019) · Trends in Food Science & Technology · doi:10.1016/j.tifs.2019.07.024 · Cited by 547 · see citing works
Utilization of text mining as a big data analysis tool for food science and nutrition
Tao, D., Yang, P., & Feng, H. (2020) · Comprehensive Reviews in Food Science and Food Safety · doi:10.1111/1541-4337.12540 · Cited by 251 · see citing works
Social Media Role and Its Impact on Public Health: A Narrative Review
Kanchan, S., & Gaidhane, A. (2023) · Cureus · doi:10.7759/cureus.33737 · Cited by 221 · see citing works
“The New York City Department of Health and Mental Hygiene (DOHMH) uses the system to track concerns about foodborne illnesses on Yelp [19].”
BackgroundEmerging Applications of Machine Learning in Food Safety
Deng, X., Cao, S., & Horn, A. L. (2021) · Annual Review of Food Science and Technology · doi:10.1146/annurev-food-071720-024112 · Cited by 187 · see citing works
How Can AI Help Improve Food Safety?
Qian, C., Murphy, S. I., Orsi, R. H., & Wiedmann, M. (2022) · Annual Review of Food Science and Technology · doi:10.1146/annurev-food-060721-013815 · Cited by 154 · see citing works
Blockchain: An emerging novel technology to upgrade the current fresh fruit supply chain
Zhang, Y., Chen, L., Battino, M. A., Farag, M. A., Xiao, J., Simal‐Gándara, J., Gao, H., & Jiang, W. (2022) · Trends in Food Science & Technology · doi:10.1016/j.tifs.2022.03.030 · Cited by 119 · see citing works
Artificial intelligence, big data, and blockchain in food safety
Zhou, Q., Zhang, H., & Wang, S. (2021) · International Journal of Food Engineering · doi:10.1515/ijfe-2021-0299 · Cited by 93 · see citing works
Wang, H., Cui, W., Guo, Y., Du, Y., & Zhou, Y. (2020) · JMIR Medical Informatics · doi:10.2196/24924 · Cited by 59 · see citing works
Using big data to promote precision oral health in the context of a learning healthcare system
Finkelstein, J., Zhang, F., Levitin, S. A., & Cappelli, D. P. (2020) · Journal of Public Health Dentistry · doi:10.1111/jphd.12354 · Cited by 53 · see citing works
Big data analytics in food industry: a state-of-the-art literature review
Siddique, A., Gupta, A., Sawyer, J. T., Huang, T., & Morey, A. (2025) · npj Science of Food · doi:10.1038/s41538-025-00394-y · Cited by 54 · see citing works
A perspective on data sharing in digital food safety systems
Qian, C., Liu, Y., Barnett-Neefs, C., Salgia, S., Serbetci, O., Adalja, A., Acharya, J., Zhao, Q. Q., Ivanek, R., & Wiedmann, M. (2022) · Critical Reviews in Food Science and Nutrition · doi:10.1080/10408398.2022.2103086 · Cited by 38 · see citing works
Tao, D., Zhang, D., Hu, R., Rundensteiner, E. A., & Feng, H. (2021) · Scientific Reports · doi:10.1038/s41598-021-00766-w · Cited by 37 · see citing works
Artificial Intelligence for Surveillance in Public Health
Thiébaut, R., Cossin, S., & Informatics, S. E. F. T. I. Y. S. O. P. H. A. E. (2019) · Yearbook of Medical Informatics · doi:10.1055/s-0039-1677939 · Cited by 28 · see citing works
Zhang, P., Cui, W., Wang, H., Du, Y., & Zhou, Y. (2021) · Foodborne Pathogens and Disease · doi:10.1089/fpd.2020.2913 · Cited by 29 · see citing works
Yasami, M., Phetvaroon, K., & Zhu, H. (2021) · Journal of Foodservice Business Research · doi:10.1080/15378020.2021.1964340 · Cited by 27 · see citing works
Scores on doors: restaurant hygiene ratings and public health policy
Fleetwood, J. (2019) · Journal of Public Health Policy · doi:10.1057/s41271-019-00183-4 · Cited by 26 · see citing works
Digital epidemiology, deep phenotyping and the enduring fantasy of pathological omniscience
Engelmann, L. (2022) · Big Data & Society · doi:10.1177/20539517211066451 · Cited by 26 · see citing works
Tian, L., Wu, M., Guo, W., Li, H., Gai, Z., & Gong, G. (2021) · Molecules · doi:10.3390/molecules26216748 · Cited by 22 · see citing works
Biases in using social media data for public health surveillance: A scoping review
Zhao, Y., He, X., Feng, Z., Bost, S. J., Prosperi, M., Wu, Y., Guo, Y. M., & Bian, J. (2022) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2022.104804 · Cited by 22 · see citing works
Predicting Food Safety Compliance for Informed Food Outlet Inspections: A Machine Learning Approach
Oldroyd, R. A., Morris, M., & Birkin, M. H. (2021) · White Rose Research Online (University of Leeds, The University of Sheffield, University of York) · Cited by 19 · see citing works
A Novel Foodborne Illness Detection and Web Application Tool Based on Social Media
Tao, D., Hu, R., Zhang, D., Laber, J., Lapsley, A., Kwan, T., Rathke, L. M., Rundensteiner, E. A., & Feng, H. (2023) · Foods · doi:10.3390/foods12142769 · Cited by 17 · see citing works
Foodborne Event Detection Based on Social Media Mining: A Systematic Review
Salaris, S., Ocagli, H., Casamento, A., Lanera, C., & Gregori, D. (2025) · Foods · doi:10.3390/foods14020239 · Cited by 19 · see citing works
Du, Y., Wang, H., Cui, W., Zhu, H., Guo, Y., Dharejo, F. A., & Zhou, Y. (2021) · JMIR Medical Informatics · doi:10.2196/29433 · Cited by 19 · see citing works
Analysis of Consumers Perceptions of Food Safety Risk in Social Networks
Amato, A., Balzano, W., Cozzolino, G., & Moscato, F. (2019) · Advances in intelligent systems and computing · doi:10.1007/978-3-030-15032-7_102 · Cited by 14 · see citing works
Detecting Foodborne Disease Outbreaks in Florida through Consumer Complaints
Li, X., Sapp, A. C., Singh, N., Matthias, L., Bailey, C., DeMent, J., & Havelaar, A. H. (2020) · Journal of Food Protection · doi:10.4315/jfp-20-138 · Cited by 13 · see citing works
A Survey of the Applications of Text Mining for the Food Domain
Xiong, S., Tian, W., Si, H., Zhang, G., & Shi, L. (2024) · Algorithms · doi:10.3390/a17050176 · Cited by 16 · see citing works
Weakly Supervised Attention Networks for Fine-Grained Opinion Mining and Public Health
Karamanolakis, G., Hsu, D., & Gravano, L. (2019) · Workshop on Noisy User-generated Text (W-NUT) · doi:10.18653/v1/d19-5501 · Cited by 11 · see citing works
Self-citationA food safety prescreening method with domain-specific information using online reviews
Zuo, E., Aysa, A., Muhammat, M., Zhao, Y., Chen, B., & Ubul, K. (2022) · Journal of Consumer Protection and Food Safety · doi:10.1007/s00003-022-01367-z · Cited by 10 · see citing works
Rajamani, S., Firestone, M. J., & Hedberg, C. W. (2021) · Online Journal of Public Health Informatics · doi:10.5210/ojphi.v13i1.11087 · Cited by 8 · see citing works
Adopting Big Data Analysis in the Agricultural Sector: Financial and Societal Impacts
Kashef, R. F. (2019) · Studies in big data · doi:10.1007/978-981-15-0663-5_7 · Cited by 8 · see citing works
Trust Analysis for Information Concerning Food-Related Risks
Amato, A., & Cozzolino, G. (2019) · Lecture notes on data engineering and communications technologies · doi:10.1007/978-3-030-12839-5_31 · Cited by 6 · see citing works
Hughes, D. F., GREEN, M. L., Warner, J. K., & Davidson, P. (2019) · The Science of The Total Environment · doi:10.1016/j.scitotenv.2019.03.254 · Cited by 6 · see citing works
Hu, R., Zhang, D., Tao, D., Zhang, H., Feng, H., & Rundensteiner, E. A. (2023) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata59044.2023.10386694 · Cited by 6 · see citing works
Internet-based surveillance to track trends in seasonal allergies across the United States
Stallard-Olivera, E., & Fierer, N. (2024) · PNAS Nexus · doi:10.1093/pnasnexus/pgae430 · Cited by 5 · see citing works
NYC HANES 2013–14 and Reflections on Future Population Health Surveillance
Perlman, S. E., Gwynn, R. C., Greene, C. M., Freeman, A. L., Chernov, C., & Thorpe, L. E. (2018) · Journal of Urban Health · doi:10.1007/s11524-018-0284-0 · Cited by 4 · see citing works
Lynch, M., & Mah, C. L. (2021) · Journal of Hunger & Environmental Nutrition · doi:10.1080/19320248.2021.2002747 · Cited by 3 · see citing works
Innovating at the human–technology interface in disasters and disease outbreaks
Bernardo, T. M., Gutiérrez, E. P., Hachborn, G., Forrest, R. O., & Sobkowich, K. E. (2020) · Scientific and Technical Review (World Organisation for Animal Health) · doi:10.20506/rst.39.2.3100 · Cited by 3 · see citing works
LLM-based Hierarchical Label Annotation for Foodborne Illness Detection on Social Media
Zhang, D., Hu, R., Tao, D., Feng, H., & Rundensteiner, E. A. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10825369 · Cited by 2 · see citing works
Applications of Machine Learning in Food Safety
Pujahari, R. M., & Khan, R. (2022) · Advances in computational intelligence and robotics book series · doi:10.4018/978-1-6684-5141-0.ch012 · Cited by 3 · see citing works
Contents IMIA Yearbook of Medical Informatics 2019
(2019) · Yearbook of Medical Informatics · doi:10.1055/s-0039-1694828 · Cited by 2 · see citing works
Beltrán-Ostos, A., Urdaneta, A. M., & González, J. A. (2022) · Medicina · doi:10.56050/01205498.1647 · Cited by 1 · see citing works
Detecting Foodborne Illness Complaints in Multiple Languages Using English Annotations Only
Liu, Z., Karamanolakis, G., Hsu, D., & Gravano, L. (2020) · International Workshop on Health Text Mining and Information Analysis (LOUHI) · doi:10.18653/v1/2020.louhi-1.15 · Cited by 3 · see citing works
Self-citationOpportunities for Qualitative Analysis of Social Media Platforms in Dietetic Research and Practice
Lynch, M., Knežević, I., & Ryan, K. L. (2020) · Canadian Journal of Dietetic Practice and Research · doi:10.3148/cjdpr-2020-035 · Cited by 1 · see citing works
Four Short Teaching Cases on Automated Decision-Making in the Public SectorPreprint
Strauss, S., & Wonnenberg, I. (2021) · SSRN Electronic Journal · doi:10.2139/ssrn.3904984 · Cited by 1 · see citing works
How to Meet the Diverse Needs of Consumers: Big Data Mining based on Online Review
Gu, W., Hu, R. B., & Song, Y. (2020) · Hawaii International Conference on System Sciences (HICSS) · doi:10.24251/hicss.2020.349
Predicting Food Safety Compliance for Informed Food Outlet Inspections: A Machine Learning Approach
Oldroyd, R. A., Morris, M. A., & Birkin, M. H. (2021) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph182312635
Leveraging Customer Feedback for Product Improvement and Production Quality ControlPreprint
Gavirneni, S., Gomez, M. I., Yonezawa, K., & Zhu, X. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4546968
Chinese Food Safety Entity Recognition Based on BERT Model
Xiong, S., & Bai, G. (2023) · IEEE International Conference on Frontiers Technology of Information and Computer (ICFTIC) · doi:10.1109/icftic59930.2023.10456298
From Calls to Action: Leveraging Call Center Data for Product and Production ManagementPreprint
Gavirneni, S., Gomez, M. I., Yonezawa, K., & Zhu, X. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5029776
Factors Influencing Foodborne Illness Self-Identification From User-Generated Data – Minnesota, 2024
Krampf, L., Hedberg, C. W., Hauber, B., Walmsley, A., & Firestone, M. J. (2025) · Journal of Food Protection · doi:10.1016/j.jfp.2025.100612
Natural language processing in food safety research: A systematic reviewPreprint
Özen, N., Papadopoulou, F., Mutlu, O., Öztürk, B., Mu, W., van den Bulk, L. M., van der Velden, B. H. M., & Hürriyetoğlu, A. (2026) · Research Square · doi:10.21203/rs.3.rs-9305529/v1
Enhancing food quality, processing, and preservation through AI and microbiological interventions
Singh, P., & GANGWAR, L. K. (2026) · Elsevier eBooks · doi:10.1016/b978-0-443-40362-0.00004-6
Dixit, Y., Reis, M. G., & dos Reis, M. M. (2026) · Journal of Food Protection · doi:10.1016/j.jfp.2026.100854
Natural language processing in food safety research
Özen, N., Papadopoulou, F., Mutlu, O., Öztürk, B., Mu, W., van den Bulk, L. M., van der Velden, B. H. M., & Hürriyetoğlu, A. (2026) · Future Foods · doi:10.1016/j.fufo.2026.101152
Shuai, Z., & Xu, X. (2026) · Agricultural and Food Economics · doi:10.1186/s40100-026-00516-4
- Authors
- Effland, T., Lawson, A., Balter, S., Devinney, K., Reddy, V., Waechter, H., Gravano, L., & Hsu, D.
- Affiliations
- Columbia University (US)
- New York City Department of Health and Mental Hygiene (US)
- Corresponding author
- Effland, Thomas
- Venue
- Journal of the American Medical Informatics Association
- Publisher
- Oxford University Press (OUP)
- Published
- 2018-01-10
- Volume
- 25
- Issue
- 12
- Pages
- 1586-1592
- ISSN
- 1067-5027 (ISSN-L)
- 1527-974X
- PMC
- PMC7647154
- OpenAlex
- W2783923625
- Cited by
- 55
- References
- 24
- Citations by year
- 2026: 5 · 2025: 4 · 2024: 3 · 2023: 5 · 2022: 8 · 2021: 12 · 2020: 8 · 2019: 9 · 2018: 1
- FWCI
- 3.85
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Licence
- academic.oup.com/journals/pages/open_access/funder_policies/chorus/standard_publication_model (vor)
- Locations
- Journal of the American Medical Informatics Association (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Funders
- National Science Foundation (IIS-15-63785)
- Alfred P. Sloan Foundation (G-2015-14017)
- Centers for Disease Control and Prevention (NU90TP000546)
- Topics
- Health Sciences › Medicine › Epidemiology › Data-Driven Disease Surveillance
- Physical Sciences › Computer Science › Artificial Intelligence › Text and Document Classification Technologies
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Keywords
- food poisoning, online restaurant reviews, Yelp, text classification, outbreak detection, biased training data, error analysis, food safety monitoring, New York City Department of Health and Mental Hygiene
- MeSH terms
- Disease Outbreaks, Foodborne Diseases, Humans, New York City, Population Surveillance, Restaurants, Models, Statistical, Logistic Models, Data Mining
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
