Disease Inference from Health-Related Questions via Sparse Deep Learning
The DOI matches its Crossref record; 49 of the 51 references it cites were found in a registry or an index.
Abstract
Automatic disease inference is of importance to bridge the gap between what online health seekers with unusual symptoms need and what busy human doctors with biased expertise can offer. However, accurately and efficiently inferring diseases is non-trivial, especially for community-based health services due to the vocabulary gap, incomplete information, correlated medical concepts, and limited high quality training samples. In this paper, we first report a user study on the information needs of health seekers in terms of questions and then select those that ask for possible diseases of their manifested symptoms for further analytic. We next propose a novel deep learning scheme to infer the possible diseases given the questions of health seekers. The proposed scheme is comprised of two key components. The first globally mines the discriminant medical signatures from raw features. The second deems the raw features and their signatures as input nodes in one layer and hidden nodes in the subsequent layer, respectively. Meanwhile, it learns the inter-relations between these two layers via pre-training with pseudo-labeled data. Following that, the hidden nodes serve as raw features for the more abstract signature mining. With incremental and alternative repeating of these two components, our scheme builds a sparsely connected deep architecture with three hidden layers. Overall, it well fits specific tasks with fine-tuning. Extensive experiments on a real-world dataset labeled by online doctors show the significant performance gains of our scheme.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Representation Learning: A Review and New Perspectives
Bengio, Y., Courville, A., & Vincent, P. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2013.50 · Cited by 13,001
Learning Deep Architectures for AI
Bengio, Y. (2009) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000006 · Cited by 9,239
Learning Deep Architectures for AI
Bengio, Y. (2009) · now publishers, Inc. eBooks · doi:10.1561/9781601982957 · Cited by 4,941
Why Does Unsupervised Pre-training Help Deep Learning?
Erhan, D., Bengio, Y., Courville, A. C., Manzagol, P.-A., Vincent, P., & Bengio, S. (2010) · Journal of Machine Learning Research · Cited by 2,631
Lee, H., Grosse, R., Ranganath, R., & Ng, A. Y. (2009) · Proceedings of the 26th Annual International Conference on Machine Learning · doi:10.1145/1553374.1553453 · Cited by 2,412
Consumer health information seeking on the internet: the state of the art
Cline, R. J. W. (2001) · Health Education Research · doi:10.1093/her/16.6.671 · Cited by 1,708 · cited 1×
“To make more informed decisions towards better health, health seekers are getting increasingly savvywith their information needs [34].”
BackgroundAn overview of MetaMap: historical perspective and recent advances
Aronson, A. R., & Lang, F.-M. (2010) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.2009.002733 · Cited by 1,519
(1992) · Cited by 1,405
A multi-task learning formulation for predicting disease progression
Zhou, J., Yuan, L., Liu, J., & Ye, J. (2011) · Proceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/2020408.2020549 · Cited by 254
Large scale semi-supervised linear SVMs
Sindhwani, V., & Keerthi, S. S. (2006) · Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/1148170.1148253 · Cited by 230
Inequalities between multi-rater kappas
Warrens, M. J. (2010) · Advances in Data Analysis and Classification · doi:10.1007/s11634-010-0073-4 · Cited by 226
Aligning temporal data by sentinel events: discovering patterns in electronic health records
Wang, T. D., Plaisant, C., Quinn, A. J., Stanchak, R., Murphy, S., & Shneiderman, B. (2008) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1357054.1357129 · Cited by 220
An integrated machine learning approach to stroke prediction
Khosla, A., Cao, Y., Lin, C. C.-Y., Chiu, H.-K., Hu, J., & Lee, H. (2010) · Proceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1835804.1835830 · Cited by 199
Using decision tree for diagnosing heart disease patients
Shouman, M., Turner, T., & Stocker, R. (2011) · Australasian Data Mining Conference · Cited by 185
Modeling disease progression via fused sparse group lasso
Zhou, J., Liu, J., Narayan, V. A., & Ye, J. (2012) · Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/2339530.2339702 · Cited by 179
Harvesting visual concepts for image search with complex queries
Nie, L., Yan, S., Wang, M., Hong, R., & Chua, T.-S. (2012) · Proceedings of the 20th ACM international conference on Multimedia · doi:10.1145/2393347.2393363 · Cited by 148
Self-citationPredicting individual disease risk based on medical history
Davis, D. A., Chawla, N. V., Blumm, N., Christakis, N. A., & Barabási, A.-Ĺ. (2008) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/1458082.1458185 · Cited by 129
Beyond Text QA: Multimedia Answer Generation by Harvesting Web Information
Nie, L., Wang, M., Gao, Y., Zha, Z.-J., & Chua, T. (2012) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2012.2229971 · Cited by 126
Self-citationBridging the Vocabulary Gap between Health Seekers and Healthcare Knowledge
Nie, L., Zhao, Y., Akbari, M., Shen, J., & Chua, T. (2014) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2014.2330813 · Cited by 121
Self-citationOverview of the TREC 2012 Medical Records Track.
Voorhees, E. M., & Hersh, W. (2012) · Text REtrieval Conference · Cited by 116
Multimedia answering: enriching text QA with media information
Nie, L., Wang, M., Zha, Z.-J., Li, G., & Chua, T. (2011) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/2009916.2010010 · Cited by 106
Self-citationA classification-based approach to question routing in community question answering
Zhou, T. C., Lyu, M. R.-T., & King, I. (2012) · International World Wide Web Conference (WWW) · doi:10.1145/2187980.2188201 · Cited by 99
A bayesian learning approach to promoting diversity in ranking for biomedical information retrieval
Huang, X., & Hu, Q. (2009) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/1571941.1571995 · Cited by 96
Intentions and attention in exploratory health search
Cartright, M.-A., White, R. W., & Horvitz, E. (2011) · Proceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval · doi:10.1145/2009916.2009929 · Cited by 95
A Framework for Mining Signatures from Event Sequences and Its Applications in Healthcare Data
Wang, F., Lee, N., Hu, J., Sun, J., Ebadollahi, S., & Laine, A. F. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2012.111 · Cited by 89
Robust Graph Mode Seeking by Graph Shift
Liu, H., & Yan, S. (2010) · Cited by 88
Self-citationHeart Disease Diagnosis using Support Vector Machine
Ghumbre, S. U., Patil, C., & Ghatol, A. A. (2011) · Cited by 86
Learning to Recommend Descriptive Tags for Questions in Social Forums
Nie, L., Zhao, Y., Wang, X., Shen, J., & Chua, T. (2014) · ACM Transactions on Information Systems · doi:10.1145/2559157 · Cited by 80
Self-citationApproaches for dealing with missing data in health care studies
Penny, K. I., & Atkinson, I. (2011) · Journal of Clinical Nursing · doi:10.1111/j.1365-2702.2011.03854.x · Cited by 77
Zhao, Y., Nie, L., Wang, X., & Chua, T. (2014) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/2532439 · Cited by 77
Self-citationTowards heterogeneous temporal clinical event pattern discovery: a convolutional approach
Wang, F., Lee, N., Hu, J., Sun, J., & Ebadollahi, S. (2012) · Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/2339530.2339605 · Cited by 76
Extracting key-substring-group features for text classification
Zhang, D., & Lee, W. S. (2006) · Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1150402.1150455 · Cited by 56
On iterative intelligent medical search
Luo, G., & Tang, C. (2008) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/1390334.1390338 · Cited by 47
SympGraph: a framework for mining clinical notes through symptom relation graphs
Sondhi, P., Sun, J., Tong, H., & Zhai, C. (2012) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/2339530.2339712 · Cited by 46
A Joint Local-Global Approach for Medical Terminology Assignment
Nie, L., Akbari, M., Li, T., & Chua, T. (2014) · Cited by 42
Self-citationDoan, S., & Xu, H. (2010) · PubMed · Cited by 41
WenZher: comprehensive vertical search for healthcare domain
Nie, L., Li, T., Akbari, M., Shen, J., & Chua, T. (2014) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/2600428.2611176 · Cited by 41
Self-citationFast Detection of Dense Subgraphs with Iterative Shrinking and Expansion
Liu, H., Latecki, L. J., & Yan, S. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2013.16 · Cited by 39
Self-citationStudies of the onset and persistence of medical concerns in search logs
White, R. W., & Horvitz, E. (2012) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/2348283.2348322 · Cited by 38
A temporal abstraction framework for classifying clinical temporal data.
Batal, I., Sacchi, L., Bellazzi, R., & Hauskrecht, M. (2009) · PubMed · Cited by 34
A Semi-Supervised Bayesian Network Model for Microblog Topic Classification
Chen, Y., Li, Z., Nie, L., Hu, X., Wang, X., Chua, T., & Zhang, X. (2012) · Cited by 29
Self-citationCengage Learning at TREC 2011 Medical Track
King, B., Wang, L., Provalov, I., & Zhou, J. (2011) · Text REtrieval Conference · Cited by 28
Learning to combine representations for medical records search
Limsopatham, N., Macdonald, C., & Ounis, I. (2013) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/2484028.2484177 · Cited by 27
Automated diagnosis of Alzheimer's disease using image similarity and user feedback
Akgül, C. B., Ünay, D., & Ekin, A. (2009) · ACM International Conference on Image and Video Retrieval (CIVR) · doi:10.1145/1646396.1646438 · Cited by 24
Evaluating medical information retrieval
Koopman, B., Bruza, P., Sitbon, L., & Lawley, M. (2011) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/2009916.2010088 · Cited by 18
Bridging the Language Gap: Topic Adaptation for Documents with Different Technicality
Yang, S.-H., Crain, S. P., & Hongyuan Zha, B. (2011) · International Conference on Artificial Intelligence and Statistics · Cited by 16
An adaptive evidence weighting method for medical record search
Zhu, D., & Carterette, B. (2013) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/2484028.2484175 · Cited by 13
The bag-of-repeats representation of documents
Gallé, M. (2013) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/2484028.2484142 · Cited by 12
Semantic text classification of disease reporting
Zhang, Y., & Liu, B. (2007) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/1277741.1277889 · Cited by 8
Cited by
Deep Learning for Health Informatics
Ravì, D., Wong, C., Deligianni, F., Berthelot, M., Andreu-Perez, J., Lo, B., & Yang, G.-Z. (2017) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2016.2636665 · Cited by 2,000
Ali, S., Abuhmed, T., El-Sappagh, S., Muhammad, K., Alonso-Moral, J. M., Confalonieri, R., Guidotti, R., Del Ser, J., Díaz-Rodríguez, N., & Herrera, F. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101805 · Cited by 1,762
Shickel, B., Tighe, P. J., Bihorac, A., & Rashidi, P. (2018) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2017.2767063 · Cited by 1,688
Explainable Deep Learning: A Field Guide for the Uninitiated
Ras, G., Xie, N., Van Gerven, M., & Doran, D. (2022) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.13200 · Cited by 446
Deep learning for lung Cancer detection and classification
Asuntha, A., & Srinivasan, A. (2020) · Multimedia Tools and Applications · doi:10.1007/s11042-019-08394-3 · Cited by 382
Metadata
- Authors
- Nie, L., Wang, M., Zhang, L., Yan, S., Zhang, B., & Chua, T.-S.
- Affiliations
- National University of Singapore (SG)
- Hefei University of Technology (CN)
- Tsinghua University (CN)
- Venue
- IEEE Transactions on Knowledge and Data Engineering
- Published
- 2015-08-01
- Volume
- 27
- Issue
- 8
- Pages
- 2107-2119
- ISSN
- 1041-4347 (ISSN-L)
- OpenAlex
- W2013787102
- Cited by
- 195
- References
- 51
- Citations by year
- 2026: 2 · 2025: 3 · 2024: 5 · 2023: 17 · 2022: 17 · 2021: 20 · 2020: 15 · 2019: 27 · 2018: 30 · 2017: 22 · 2016: 29 · 2015: 8
- FWCI
- 57.18
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- IEEE Transactions on Knowledge and Data Engineering (publishedVersion)
- Funders
- NUS-Tsinghua Extreme Search Project (R-252-300-001-490)
- Topics
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Keywords
- symptom-based disease prediction, pseudo-labeling, virtual care, vocabulary gap, fine-tuning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.