ProKnow: Process knowledge for safety constrained and explainable question generation for mental health diagnostic assistance
Open accessAbstract
Virtual Mental Health Assistants (VMHAs) are utilized in health care to provide patient services such as counseling and suggestive care. They are not used for patient diagnostic assistance because they cannot adhere to safety constraints and specialized clinical process knowledge (ProKnow) used to obtain clinical diagnoses. In this work, we define ProKnow as an ordered set of information that maps to evidence-based guidelines or categories of conceptual understanding to experts in a domain. We also introduce a new dataset of diagnostic conversations guided by safety constraints and ProKnow that healthcare professionals use (ProKnow-data). We develop a method for natural language question generation (NLG) that collects diagnostic information from the patient interactively (ProKnow-algo). We demonstrate the limitations of using state-of-the-art large-scale language models (LMs) on this dataset. ProKnow-algo incorporates the process knowledge through explicitly modeling safety, knowledge capture, and explainability. As computational metrics for evaluation do not directly translate to clinical settings, we involve expert clinicians in designing evaluation metrics that test four properties: safety, logical coherence, and knowledge capture for explainability while minimizing the standard cross entropy loss to preserve distribution semantics-based similarity to the ground truth. LMs with ProKnow-algo generated 89% safer questions in the depression and anxiety domain (tested property: safety). Further, without ProKnow-algo generations question did not adhere to clinical process knowledge in ProKnow-data (tested property: knowledge capture). In comparison, ProKnow-algo-based generations yield a 96% reduction in our metrics to measure knowledge capture. The explainability of the generated question is assessed by computing similarity with concepts in depression and anxiety knowledge bases. Overall, irrespective of the type of LMs, ProKnow-algo achieved an averaged 82% improvement over simple pre-trained LMs on safety, explainability, and process-guided question generation. For reproducibility, we will make ProKnow-data and the code repository of ProKnow-algo publicly available upon acceptance.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
ProKnow-algo demonstrated a high level of safety and explainability in the context of diagnosing depression and anxiety.
“ProKnow-algo demonstrated a high level of safety and explainability in the context of diagnosing depression and anxiety (as depicted in Fig. 6).”
From Large language models in medical and healthcare fields: applications, advances, and challenges (2024) · cites this paper as Roy et al. (2023)
The average number of unsafe matches offers a way to measure the effectiveness of the harm or severe consequences of the generated questions.
“The average number of unsafe matches (Roy et al. 2023) offers a way to measure the effectiveness of the harm or severe consequences of the generated questions.”
From Large language models in medical and healthcare fields: applications, advances, and challenges (2024) · cites this paper as Roy et al. 2023
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Jakubův, J., Chvalovský, K., Goertzel, Z., et al. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,648 · see citing works
Distilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204 · see citing works
Transformers: State-of-the-Art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2020.emnlp-demos.6 · Cited by 8,741 · see citing works
The PHQ-9: A New Depression Diagnostic and Severity Measure
Kroenke, K., & Spitzer, R. L. (2002) · Psychiatric Annals · doi:10.3928/0048-5713-20020901-06 · Cited by 5,898 · see citing works
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. (2018) · Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/w18-5446 · Cited by 4,222 · see citing works
Universal Language Model Fine-tuning for Text Classification
Howard, J., & Ruder, S. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1031 · Cited by 4,100 · see citing works
Retrofitting Word Vectors to Semantic Lexicons
Faruqui, M., Dodge, J., Jauhar, S. K., Dyer, C., Hovy, E. H., & Smith, N. A. (2015) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.3115/v1/n15-1184 · Cited by 785 · see citing works
LaMDA: Language Models for Dialog Applications
Thoppilan, R., De Freitas, D., Hall, J., Shazeer, N., Kulshreshtha, A., Cheng, H.-T., Jin, A., Bos, T., Baker, L., Du, Y., et al. (2022) · arXiv · doi:10.48550/arxiv.2201.08239 · Cited by 778 · see citing works
Patient Knowledge Distillation for BERT Model Compression
Sun, S., Cheng, Y., Gan, Z., & Liu, J. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1441 · Cited by 758 · see citing works
Ethical and social risks of harm from Language Models
Weidinger, L., Mellor, J., Rauh, M., Griffin, C., Uesato, J., Huang, P.-S., Cheng, M., Glaese, M., Balle, B., Kasirzadeh, A., et al. (2021) · arXiv · doi:10.48550/arxiv.2112.04359 · Cited by 477 · see citing works
Counter-fitting Word Vectors to Linguistic Constraints
Mrkšić, N., Séaghdha, D. Ó., Thomson, B., Gašić, M., Rojas-Barahona, L. M., Su, P.-H., Vandyke, D., Wen, T.-H., & Young, S. J. (2016) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n16-1018 · Cited by 356 · see citing works
Althoff, T., Clark, K., & Leskovec, J. (2016) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00111 · Cited by 322 · see citing works
Ethical research protocols for social media health research
Benton, A., Coppersmith, G., & Dredze, M. (2017) · Proceedings of the First ACL Workshop on Ethics in Natural Language Processing · doi:10.18653/v1/w17-1612 · Cited by 219 · see citing works
Knowledge-aware Assessment of Severity of Suicide Risk for Early Intervention
Gaur, M., Alambo, A., Sain, J. P., Kursuncu, U., Thirunarayan, K., Kavuluru, R., Sheth, A., Welton, R., & Pathak, J. (2019) · International World Wide Web Conference (WWW) · doi:10.1145/3308558.3313698 · Cited by 193 · see citing works
Self-citationSemi-Supervised Approach to Monitoring Clinical Depressive Symptoms in Social Media
Yazdavar, A. H., Al-Olimat, H. S., Ebrahimi, M., Bajaj, G., Banerjee, T., Thirunarayan, K., Pathak, J., & Sheth, A. (2017) · IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM) · doi:10.1145/3110025.3123028 · Cited by 169 · see citing works
Self-citationGaur, M., Faldu, K., & Sheth, A. (2021) · IEEE Internet Computing · doi:10.1109/mic.2020.3031769 · Cited by 140 · see citing works
Self-citationSezgın, E., Sirrianni, J., & Linwood, S. L. (2022) · JMIR Medical Informatics · doi:10.2196/32875 · Cited by 129 · see citing works
Addressing Semantic Drift in Question Generation for Semi-Supervised Question Answering
Zhang, S., & Bansal, M. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1253 · Cited by 115 · see citing works
Learning to Generate Questions by LearningWhat not to Generate
Liu, B., Zhao, M. M., Niu, D., Lai, K., He, Y., Wei, H., & Xu, Y. (2019) · International World Wide Web Conference (WWW) · doi:10.1145/3308558.3313737 · Cited by 104 · see citing works
Gaur, M., Gunaratna, K., Srinivasan, V., & Jin, H. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i10.21312 · Cited by 57 · see citing works
Self-citationKnowledge-Intensive Language Understanding for Explainable AI
Sheth, A., Gaur, M., Roy, K., & Faldu, K. (2021) · IEEE Internet Computing · doi:10.1109/mic.2021.3101919 · Cited by 35 · see citing works
Self-citationGupta, S., Agarwal, A., Gaur, M., Roy, K., Narayanan, V., Kumaraguru, P., & Sheth, A. (2022) · Proceedings of the Eighth Workshop on Computational Linguistics and Clinical Psychology · doi:10.18653/v1/2022.clpsych-1.12 · Cited by 27 · see citing works
Self-citationSaha, T., Gupta, D., Saha, S., & Bhattacharyya, P. (2020) · Expert Systems with Applications · doi:10.1016/j.eswa.2020.113650 · Cited by 21 · see citing works
Towards augmenting crisis counselor training by improving message retrieval
DeMasi, O., Hearst, M. A., & Recht, B. (2019) · Workshop on Computational Linguistics and Clinical Psychology (CLPsych) · doi:10.18653/v1/w19-3001 · Cited by 18 · see citing works
Evaluation of In-Person Counseling Strategies To Develop Physical Activity Chatbot for Women
Liang, K.-H., Lange, P. L., Oh, Y. J., Zhang, J., Fukuoka, Y., & Yu, Z. (2021) · Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/2021.sigdial-1.5 · Cited by 12 · see citing works
Spinning words as disguise: Shady services for ethical research?
Reagle, J. M., & Gaur, M. (2022) · First Monday · doi:10.5210/fm.v27i1.12350 · Cited by 12 · see citing works
Self-citationReddy, R. G., Sultan, M. A., Franz, M., Sil, A., & Ji, H. (2022) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3477495.3531878 · Cited by 4 · see citing works
NLP is Not enough -- Contextualization of User Input in ChatbotsPreprint
Dolbir, N., Dastidar, T. G., & Roy, K. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.06511 · Cited by 2 · see citing works
Self-citationPersonalized Prediction of Suicide Risk for Web-based Intervention
Alambo, A., Gaur, M., Kursuncu, U., Thirunarayan, K., Schumm, J. A., Pathak, J., & Sheth, A. (2018) · Journal of Bioresource Management · Cited by 1 · see citing works
Self-citationOn Refining BERT Contextualized Embeddings using Semantic Lexicons
Zervakis, G., Vincent, E., Couceiro, M., et al. (2021) · HAL (Le Centre pour la Communication Scientifique Directe) · Cited by 1 · see citing works
Flagged first, then by citations
Research works printed but not linked: 2
- Huang R. (2015). Language use in teenage crisis intervention and the immediate outcome: A machine automated analysis of large scale text data. (Ph.D. thesis, Master's thesis). Columbia University.
- Vaswani A. Shazeer N. Parmar N. Uszkoreit J. Jones L. Gomez A. N.. (2017). Attention is all you need, in Advances in Neural Information Processing Systems, 5998– 6008.
Cited by
Sorted by citations
Large language models in medical and healthcare fields: applications, advances, and challenges
Wang, D., & Zhang, S. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10921-0 · Cited by 246 · see citing works
Sarkar, S., Gaur, M., Chen, L. K., Garg, M., & Srivastava, B. (2023) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2023.1229805 · Cited by 38 · see citing works
Self-citationDalal, S., Tilwani, D., Gaur, M., Jain, S., Shalin, V. L., & Sheth, A. P. (2025) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2024.3483577 · Cited by 19 · see citing works
Self-citation
- Authors
- Roy, K., Gaur, M., Soltani, M., Rawte, V., Kalyan, A., & Sheth, A.
- Affiliations
- University of South Carolina (US)
- University of Maryland, Baltimore County (US)
- Allen Institute (US)
- Corresponding author
- Roy, Kaushik
- Venue
- Frontiers in Big Data
- Publisher
- Frontiers Media SA
- Published
- 2023-01-09
- Volume
- 5
- Pages
- 1056728-1056728
- ISSN
- 2624-909X (ISSN-L)
- PMC
- PMC9869802
- OpenAlex
- W4315478863
- Cited by
- 18
- References
- 30
- Citations by year
- 2025: 5 · 2024: 7 · 2023: 6
- FWCI
- 0.72
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Frontiers in Big Data (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- Scholar Commons (University of South Carolina) (submittedVersion, cc-by)
- Scholar Commons (University of South Carolina) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Maryland Shared Open Access Repository (USMAI Consortium) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Digital Repository at the University of Maryland (University of Maryland College Park)
- Funders
- National Science Foundation (1761931, 2133842)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- safety constraints, explainability, psychiatric diagnosis, depression, anxiety, large language models, clinical knowledge base, logical coherence
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
