Putting ChatGPT’s Medical Advice to the (Turing) Test: Survey Study
Open accessPublished version of a preprint
The DOI matches its Crossref record; 30 of the 30 references it cites were found in a registry or an index.
Abstract
Background: Chatbots are being piloted to draft responses to patient questions, but patients’ ability to distinguish between provider and chatbot responses and patients’ trust in chatbots’ functions are not well established. Objective: This study aimed to assess the feasibility of using ChatGPT (Chat Generative Pre-trained Transformer) or a similar artificial intelligence–based chatbot for patient-provider communication. Methods: A survey study was conducted in January 2023. Ten representative, nonadministrative patient-provider interactions were extracted from the electronic health record. Patients’ questions were entered into ChatGPT with a request for the chatbot to respond using approximately the same word count as the human provider’s response. In the survey, each patient question was followed by a provider- or ChatGPT-generated response. Participants were informed that 5 responses were provider generated and 5 were chatbot generated. Participants were asked—and incentivized financially—to correctly identify the response source. Participants were also asked about their trust in chatbots’ functions in patient-provider communication, using a Likert scale from 1-5. Results: A US-representative sample of 430 study participants aged 18 and older were recruited on Prolific, a crowdsourcing platform for academic studies. In all, 426 participants filled out the full survey. After removing participants who spent less than 3 minutes on the survey, 392 respondents remained. Overall, 53.3% (209/392) of respondents analyzed were women, and the average age was 47.1 (range 18-91) years. The correct classification of responses ranged between 49% (192/392) to 85.7% (336/392) for different questions. On average, chatbot responses were identified correctly in 65.5% (1284/1960) of the cases, and human provider responses were identified correctly in 65.1% (1276/1960) of the cases. On average, responses toward patients’ trust in chatbots’ functions were weakly positive (mean Likert score 3.4 out of 5), with lower trust as the health-related complexity of the task in the questions increased. Conclusions: ChatGPT responses to patient questions were weakly distinguishable from provider responses. Laypeople appear to trust the use of chatbots to answer lower-risk health questions. It is important to continue studying patient-chatbot interaction as chatbots move from administrative to more clinical roles in health care.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“GPT-4 can generate fluent, human-like text, even about complex topics like medicine58.”
From Text classification models for assessing the completeness of randomized controlled trial publications based on CONSORT reporting guidelines (2024) · cites this paper as [58]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988 · cited 1×
“Research to mitigate risks of AI chatbot–generated responses, including the potential for patient harm caused by incorrect answers; cybersecurity vulnerabilities [31]; and environmental, social, and financial risks [32] should also be further explored.”
BackgroundSurvey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023) · ACM Computing Surveys · doi:10.1145/3571730 · Cited by 5,025 · cited 1×
“These include biases from the underlying data (eg, social biases such as racism and sexism) [7,8], the ability to “hallucinate” information that is untrue [9], and the lack of mental models that would allow for true reasoning rather than simply probabilistic text generation (leading it to make errors in response to qu…”
BackgroundFisher, R. A. (1936) · BMJ · doi:10.1136/bmj.1.3923.554-a · Cited by 4,980
Large Language Models Encode Clinical Knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2212.13138 · Cited by 4,083
Large language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083 · cited 3×
“Recent studies have had health care professionals judge ChatGPT’s responses to health-related questions [14-16], with findings such as 84% of answers to cardiovascular disease prevention questions being appropriate [15] and ChatGPT overall scoring higher for quality and empathy than health care…”
BackgroundMethodInfluentialAyers, J. W., Poliak, A., Dredze, M., Leas, E. C., Zhu, Z., Kelley, J. B., Faix, D. J., Goodman, A. M., Longhurst, C. A., Hogarth, M., et al. (2023) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2023.1838 · Cited by 2,613
Benefits, Limits, and Risks of GPT-4 as an AI Chatbot for Medicine
Lee, P., Bubeck, S., & Petro, J. (2023) · New England Journal of Medicine · doi:10.1056/nejmsr2214184 · Cited by 1,738
Nadarzynski, T., Miles, O., Cowie, A., & Ridge, D. (2019) · DIGITAL HEALTH · doi:10.1177/2055207619871808 · Cited by 810
Role of Chat GPT in public health
Biswas, S. S. (2023) · Annals of Biomedical Engineering · doi:10.1007/s10439-023-03172-7 · Cited by 673 · cited 1×
“Recently, the launch of ChatGPT (Chat Generative Pre-trained Transformer; OpenAI) [1] has prompted a surge of interest in AI-based chatbots, both from the health care field [2,3] and the general public [4,5].”
BackgroundPhysician stress and burnout: the impact of health information technology
Gardner, R. L., Cooper, E., Haskell, J., Harris, D. A., Poplau, S., Kroth, P. J., & Linzer, M. (2019) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocy145 · Cited by 566
Research works printed but not linked: 1
- BolukbasiTChangKWZouJSaligramaVKalaiAMan is to computer programmer as woman is to homemaker? debiasing word embeddings2016125NIPS'16: 30th International Conference on Neural Information Processing SystemsDecember 5-10, 2016Barcelona, Spain43564364
Cited by
Gender bias and stereotypes in Large Language Models
Kotek, H., Dockum, R., & Sun, D. (2023) · Proceedings of The ACM Collective Intelligence Conference · doi:10.1145/3582269.3615599 · Cited by 381
The impact of generative artificial intelligence on socioeconomic inequalities and policy making
Capraro, V., Lentsch, A., Acemoglu, D., Akgun, S., Akhmedova, A., Bilancini, E., Bonnefon, J.-F., Brañas-Garza, P., Butera, L., Douglas, K. M., et al. (2024) · PNAS Nexus · doi:10.1093/pnasnexus/pgae191 · Cited by 371
Nikolic, S., Daniel, S., Haque, R., Belkina, M., Hassan, G. M., Grundy, S., Lyden, S., Neal, P., & Sandison, C. (2023) · European Journal of Engineering Education · doi:10.1080/03043797.2023.2213169 · Cited by 336
Large language models in patient education: a scoping review of applications in medicine
Aydin, S., Karabacak, M., Vlachos, V., & Margetis, K. (2024) · Frontiers in Medicine · doi:10.3389/fmed.2024.1477898 · Cited by 288
How Can Recommender Systems Benefit from Large Language Models: A Survey
Lin, J., Dai, X., Xi, Y., Liu, W., Chen, B., Zhang, H., Liu, Y., Wu, C., Li, X., Zhu, C., et al. (2025) · ACM Transactions on Information Systems · doi:10.1145/3678004 · Cited by 242
Younis, H. A., Eisa, T. A. E., Nasser, M., Sahib, T. M., Noor, A. A., Alyasiri, O. M., Salisu, S., Hayder, I. M., & Younis, H. A. (2024) · Diagnostics · doi:10.3390/diagnostics14010109 · Cited by 242
Assessing Health Students' Attitudes and Usage of ChatGPT in Jordan: Validation Study
Sallam, M., Salim, N. A., Barakat, M., Al-Mahzoum, K., Al-Tammemi, A. B., Malaeb, D., Hallit, R., & Hallit, S. (2023) · JMIR Medical Education · doi:10.2196/48254 · Cited by 234
“The publicly available and user-friendly ChatGPT from OpenAI exemplifies the widespread attention and scrutiny received in academia and among health professionals [9,17,19-21].”
Ning, Y., Teixayavong, S., Shang, Y., Savulescu, J., Nagaraj, V., Miao, D., Mertens, M., Ting, D. S. W., Ong, J. C. L., Liu, M., et al. (2024) · The Lancet Digital Health · doi:10.1016/s2589-7500(24)00143-2 · Cited by 212
Thirunavukarasu, A. J., Hassan, R., Mahmood, S., Sanghera, R., Barzangi, K., El Mukashfi, M., & Shah, S. (2023) · JMIR Medical Education · doi:10.2196/46599 · Cited by 187
Wang, L., Wan, Z., Ni, C., Song, Q., Li, Y., Clayton, E., Malin, B., & Yin, Z. (2024) · Journal of Medical Internet Research · doi:10.2196/22769 · Cited by 165
Early applications of ChatGPT in medical practice, education and research
Sedaghat, S. (2023) · Clinical Medicine · doi:10.7861/clinmed.2023-0078 · Cited by 163
Assessing the Effectiveness of ChatGPT in Delivering Mental Health Support: A Qualitative Study
Alanezi, F. (2024) · Journal of Multidisciplinary Healthcare · doi:10.2147/jmdh.s447368 · Cited by 139
Generative Artificial Intelligence Through ChatGPT and Other Large Language Models in Ophthalmology
Tan, T. F., Thirunavukarasu, A. J., Campbell, J. P., Keane, P. A., Pasquale, L. R., Abràmoff, M. D., Kalpathy–Cramer, J., Lum, F., Kim, J. E., Baxter, S. L., et al. (2023) · Ophthalmology Science · doi:10.1016/j.xops.2023.100394 · Cited by 144
Meyer, A., Riese, J., & Streichert, T. (2024) · JMIR Medical Education · doi:10.2196/50965 · Cited by 110
“In a previous study by Nov et al [57], considering that 59% of participants trusted chatbots more than traditional search engines, it must be noted that GPT-3.”
Large Language Model–Based Responses to Patients’ In-Basket Messages
Small, W. R., Wiesenfeld, B. M., Brandfield-Harvey, B., Jonassen, Z., Mandal, S., Stevens, E. R., Major, V. J., Lostraglio, E., Szerencsy, A. C., Jones, S., et al. (2024) · JAMA Network Open · doi:10.1001/jamanetworkopen.2024.22399 · Cited by 130
Self-citationEvolution and Prospects of Foundation Models: From Large Language Models to Large Multimodal Models
Chen, Z., Xu, L., Zheng, H., Chen, L., Tolba, A. K. R., Zhao, L., Yu, K., & Feng, H. (2024) · Computers, materials & continua/Computers, materials & continua (Print) · doi:10.32604/cmc.2024.052618 · Cited by 176
Nastasi, A. J., Courtright, K. R., Halpern, S. D., & Weissman, G. E. (2023) · Scientific Reports · doi:10.1038/s41598-023-45223-y · Cited by 139
Knowledge sharing in manufacturing using LLM-powered tools: user study and model benchmarking
Freire, S. K., Wang, C., Foosherian, M., Wellsandt, S., Ruiz-Arenas, S., & Niforatos, E. (2024) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2024.1293084 · Cited by 143
An exploratory survey about using ChatGPT in education, healthcare, and research
Hosseini, M., Gao, C. A., Liebovitz, D. M., Carvalho, A. M., Ahmad, F. S., Luo, Y., MacDonald, N., Holmes, K. L., & Kho, A. (2023) · doi:10.1101/2023.03.31.23287979 · Cited by 238
Integration of large language models and federated learning
Chen, C., Feng, X., Li, Y., Lyu, L., Zhou, J., Zheng, X., & Yin, J. (2024) · Patterns · doi:10.1016/j.patter.2024.101098 · Cited by 174
Hulmán, Á., Dollerup, O. L., Mortensen, J. F., Fenech, M. E., Norman, K., Støvring, H., & Hansen, T. K. (2023) · PLoS ONE · doi:10.1371/journal.pone.0290773 · Cited by 84
Mendel, T., Singh, N., Mann, D. M., Wiesenfeld, B., & Nov, O. (2025) · Journal of Medical Internet Research · doi:10.2196/64290 · Cited by 64
“LLMs have advanced state-of-the-art performance in generating human-like text based on user health questions [19].”
Self-citationWang, L., Li, J., Zhuang, B., Huang, S., Fang, M., Wang, C., Li, W., Zhang, M., & Gong, S. (2025) · Journal of Medical Internet Research · doi:10.2196/64486 · Cited by 51
Alanzi, T. M. (2023) · Journal of Multidisciplinary Healthcare · doi:10.2147/jmdh.s419847 · Cited by 58
Analysis of LLMs for educational question classification and generation
Faraby, S. A., Romadhony, A., & Adiwijaya, A. (2024) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2024.100298 · Cited by 114
Kerr, W. T., & McFarlane, K. N. (2023) · Current Neurology and Neuroscience Reports · doi:10.1007/s11910-023-01318-7 · Cited by 63
Large language models in oncology: a review
Chen, D., Parsa, R., Swanson, K. E., Nuñez, J.-J., Critch, A., Bitterman, D. S., Liu, F., & Raman, S. (2025) · BMJ Oncology · doi:10.1136/bmjonc-2025-000759 · Cited by 76
Moëll, B., & Aronsson, F. S. (2025) · Journal of Medical Internet Research · doi:10.2196/75849 · Cited by 71
Data-driven building load prediction and large language models: Comprehensive overview
Zhang, Y., Wang, D., Wang, G., Xu, P., & Zhu, Y. (2024) · Energy and Buildings · doi:10.1016/j.enbuild.2024.115001 · Cited by 52
AI language model rivals expert ethicist in perceived moral expertise
Dillion, D., Mondal, D., Tandon, N., & Gray, K. J. (2025) · Scientific Reports · doi:10.1038/s41598-025-86510-0 · Cited by 107
Hillmann, H. A. K., Angelini, E., Karfoul, N., Feickert, S., Müller‐Leisse, J., & Duncker, D. (2023) · EP Europace · doi:10.1093/europace/euad369 · Cited by 60
Attributions toward artificial agents in a modified Moral Turing Test
Aharoni, E., Fernandes, S., Brady, D. J., Alexander, C., Criner, M., Queen, K., Rando, J., Nahmias, E., & Crespo, V. (2024) · Scientific Reports · doi:10.1038/s41598-024-58087-7 · Cited by 46
Open AI meets open notes: surveillance capitalism, patient privacy and online record access
Blease, C. (2023) · Journal of Medical Ethics · doi:10.1136/jme-2023-109574 · Cited by 48
Perceptions of ChatGPT in healthcare: usefulness, trust, and risk
Chen, S.-Y., Kuo, H. Y., & Chang, S.-H. (2024) · Frontiers in Public Health · doi:10.3389/fpubh.2024.1457131 · Cited by 41
Tailor, P. D., Dalvin, L. A., Chen, J. J.-W., Iezzi, R., Olsen, T. W., Scruggs, B. A., Barkmeier, A. J., Bakri, S. J., Ryan, E. H., Tang, P. H., et al. (2024) · Ophthalmology Science · doi:10.1016/j.xops.2024.100485 · Cited by 46
Xu, R., & Wang, Z. (2024) · Heliyon · doi:10.1016/j.heliyon.2024.e32364 · Cited by 64
ChatGPT’s advice is perceived as better than that of professional advice columnists
Howe, P. D. L., Fay, N., Saletta, M., & Hovy, E. (2023) · Frontiers in Psychology · doi:10.3389/fpsyg.2023.1281255 · Cited by 42
A Systematic Review of ChatGPT and Other Conversational Large Language Models in HealthcarePreprint
Wang, L., Wan, Z., Ni, C., Song, Q., Li, Y., Clayton, E. W., Malin, B., & Yin, Z. (2024) · medRxiv · doi:10.1101/2024.04.26.24306390 · Cited by 41
Clinical insights: A comprehensive review of language models in medicine
Neveditsin, N., Lingras, P., & Mago, V. (2025) · PLOS Digital Health · doi:10.1371/journal.pdig.0000800 · Cited by 37
“…BART [175], RALL [180], fine-tuned Llama [173,179] Conversation Mental Health Bots [182–184], Medical Chatbots and Health Assis-tants [80,185,186], Triaging [187], Differential Diagnosis [188] MotiVAte [189], Depression_Reddit [190], CLPsych [191], Dreaddit [192], Clinical Vignettes [187]…”
Engin, C. D., Karataş, E., & Öztürk, T. (2024) · Children · doi:10.3390/children11060750 · Cited by 39
Can ChatGPT Guide Parents on Tympanostomy Tube Insertion?
Moise, A., Centomo-Bozzo, A., Orishchak, O., Alnoury, M. K., & Daniel, S. J. (2023) · Children · doi:10.3390/children10101634 · Cited by 40
Detecting the corruption of online questionnaires by artificial intelligence
Lebrun, B., Temtsin, S., Vonasch, A. J., & Bartneck, C. (2024) · Frontiers in Robotics and AI · doi:10.3389/frobt.2023.1277635 · Cited by 54
Scquizzato, T., Semeraro, F. F., Swindell, P., Simpson, R. F. G., Angelini, M., Gazzato, A., Sajjad, U., Bignami, E. G., Landoni, G., Keeble, T., et al. (2023) · Resuscitation · doi:10.1016/j.resuscitation.2023.110077 · Cited by 34
ChatGraph: Interpretable Text Classification by Converting ChatGPT Knowledge to Graphs
Shi, Y., Ma, H., Zhong, W., Tan, Q., Mai, G., Li, X., Liu, T., & Huang, J. (2023) · IEEE ... International Conference on Data Mining workshops · doi:10.1109/icdmw60847.2023.00073 · Cited by 69
The opportunities and challenges of adopting ChatGPT in medical research
Alsadhan, A. A., Al-Anezi, F., AlMohanna, A. M., Alnaim, N. M., Alzahrani, H. A., shinawi, R., AboAlsamh, H. M., Bakhshwain, A. M., Alenazy, M. F., Arif, W. M., et al. (2023) · Frontiers in Medicine · doi:10.3389/fmed.2023.1259640 · Cited by 40
Yapar, D., Avcı, Y. D., Sonuvar, E. T., Eğerci, Ö. F., & Yapar, A. (2023) · Joint Diseases and Related Surgery · doi:10.52312/jdrs.2023.1402 · Cited by 40
Leslie‐Miller, C. J., Simon, S. L., Dean, K. M., Mokhallati, N., & Cushing, C. C. (2024) · Journal of Pediatric Psychology · doi:10.1093/jpepsy/jsae075 · Cited by 38
Meyer, A., Soleman, A., Riese, J., & Streichert, T. (2024) · Clinical Chemistry and Laboratory Medicine (CCLM) · doi:10.1515/cclm-2024-0246 · Cited by 30
Utilization of Generative AI-drafted Responses for Managing Patient-Provider Communication
Mandal, S., Wiesenfeld, B. M., Szerencsy, A. C., Small, W. R., Major, V. J., Richardson, S. I., Schoenthaler, A., Mann, D., & Nov, O. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01972-w · Cited by 40
Self-citationPan, R., García-Díaz, J. A., & Valencia‐García, R. (2024) · Computer Modeling in Engineering & Sciences · doi:10.32604/cmes.2024.049631 · Cited by 48
A Systematic Review of User Attitudes Toward GenAI: Influencing Factors and Industry Perspectives
Chen, J., Wei, X., Xie, Q., Hu, A., Qiao, Y., Wan, R., & Liu, Y. (2025) · Journal of Intelligence · doi:10.3390/jintelligence13070078 · Cited by 49
A systematic review of early evidence on generative AI for drafting responses to patient messages
Hu, D., Guo, Y., Zhou, Y., Flores, L., & Zheng, K. (2025) · npj Health Systems · doi:10.1038/s44401-025-00032-5 · Cited by 40
Innovations in Medicine: Exploring ChatGPT’s Impact on Rare Disorder Management
Zampatti, S., Peconi, C., Megalizzi, D., Calvino, G., Trastulli, G., Cascella, R., Strafella, C., Caltagirone, C., & Giardina, E. (2024) · Genes · doi:10.3390/genes15040421 · Cited by 37
Exploring the use of ChatGPT as a virtual health coach for chronic disease management
Al-Anezi, F. (2024) · Learning Health Systems · doi:10.1002/lrh2.10406 · Cited by 36
Mandal, S., Wiesenfeld, B. M., Mann, D. M., Szerencsy, A. C., Iturrate, E., & Nov, O. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01001-2 · Cited by 34
Self-citationAlanzi, T. M., Alsalem, A. A., Alzahrani, H., Almudaymigh, N., Alessa, A., Mulla, R., Alqahtani, L., Bajonaid, R., Alharthi, A. K. Z., Alnahdi, O. S., et al. (2023) · Cureus · doi:10.7759/cureus.49486 · Cited by 40
Ethical Considerations for Generative Artificial Intelligence in Plastic Surgery
Dhawan, R., Brooks, K. D., Shauly, O., Shay, D., & Losken, A. (2025) · Plastic & Reconstructive Surgery Global Open · doi:10.1097/gox.0000000000006825 · Cited by 29
ChatGPT and Health Communication: A Systematic Literature Review
Li, M., & Guenier, A. W. (2024) · International Journal of E-Health and Medical Communications · doi:10.4018/ijehmc.349980 · Cited by 34
Demir, S. (2024) · Eye & Contact Lens Science & Clinical Practice · doi:10.1097/icl.0000000000001158 · Cited by 23
Tailor, P. D., Dalvin, L. A., Starr, M. R., Tajfirouz, D. A., Chodnicki, K. D., Brodsky, M. C., Mansukhani, S. A., Moss, H. E., Lai, K. E., Ko, M. W., et al. (2024) · Journal of Neuro-Ophthalmology · doi:10.1097/wno.0000000000002145 · Cited by 23
Ferrari‐Light, D., Merritt, R. E., D’Souza, D. M., Ferguson, M. K., Harrison, S. W., Madariaga, M. L. L., Lee, B. E., Moffatt, S. D., & Kneuertz, P. J. (2024) · Journal of Thoracic and Cardiovascular Surgery · doi:10.1016/j.jtcvs.2024.09.030 · Cited by 23
Gessinger, I., Seaborn, K., Steeds, M., & Cowan, B. R. (2024) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2024.103400 · Cited by 33
Tracking ChatGPT Research: Insights From the Literature and the Web
Mubin, O., Alnajjar, F., Trabelsi, Z., Ali, L., Parambil, M. M. A., & Zou, Z. (2024) · IEEE Access · doi:10.1109/access.2024.3356584 · Cited by 26
Connors, C., Gupta, K., Khusid, J. A., Khargi, R., Yaghoubian, A. J., Levy, M., Gallante, B. E., Atallah, W. M., & Gupta, M. (2024) · Journal of Endourology · doi:10.1089/end.2023.0696 · Cited by 22
Large Language Models in Orthopaedics: Definitions, Uses, and Limitations
Yao, J. J., Aggarwal, M., Lopez, R. D., & Namdari, S. (2024) · Journal of Bone and Joint Surgery · doi:10.2106/jbjs.23.01417 · Cited by 21
Suárez, A., Arena, S., Calzada, A. H., Varón, A. I. C., García, V. D., & Freire, Y. (2025) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2025.04.010 · Cited by 23
Soroudi, D., Gozali, A., Knox, J. A., Parmeshwar, N., Sadjadi, R., Wilson, J. C., Lee, S. A., & Piper, M. L. (2024) · Annals of Plastic Surgery · doi:10.1097/sap.0000000000004090 · Cited by 19
ChatGPT Yields a Passing Score on a Pediatric Board Preparatory Exam but Raises Red Flags
Le, M., & Davis, M. K. (2024) · Global Pediatric Health · doi:10.1177/2333794x241240327 · Cited by 26
ChatGPT can help guide and empower patients after prostate cancer diagnosis
Collin, H., Keogh, K., Basto, M. Y., Loeb, S., & Roberts, M. J. (2024) · Prostate Cancer and Prostatic Diseases · doi:10.1038/s41391-024-00864-6 · Cited by 19
Çıracıoğlu, A. M., & Erdoğan, S. D. (2025) · European Journal of Orthopaedic Surgery & Traumatology · doi:10.1007/s00590-025-04198-4 · Cited by 19
Varanasi, R. A., Wiesenfeld, B. M., & Nov, O. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714035 · Cited by 26
Self-citationJiang, L., Lan, M., Menke, J. D., Vorland, C. J., & Kilicoglu, H. (2024) · Scientific Reports · doi:10.1038/s41598-024-72130-7 · Cited by 18
“GPT-4 can generate fluent, human-like text, even about complex topics like medicine 58 .”
BackgroundGams, M., & Kramar, S. (2024) · Journal of Computer and Communications · doi:10.4236/jcc.2024.123014 · Cited by 15
Bentegeac, R., Guellec, B. L., Kuchcinski, G., Amouyel, P., & Hamroun, A. (2025) · Journal of Medical Internet Research · doi:10.2196/64348 · Cited by 39
Mendel, T., Nov, O., & Wiesenfeld, B. M. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3686925 · Cited by 23
Self-citationDifferences in Care Team Response to Patient Portal Messages by Patient Race and Ethnicity
Tang, M., Mishuris, R. G., Payvandi, L., & Stern, A. D. (2024) · JAMA Network Open · doi:10.1001/jamanetworkopen.2024.2618 · Cited by 14
Can ChatGPT Replace an Otolaryngologist in Guiding Parents on Tonsillectomy?
Moise, A., Centomo-Bozzo, A., Orishchak, O., Alnoury, M. K., & Daniel, S. J. (2024) · Ear Nose & Throat Journal · doi:10.1177/01455613241230841 · Cited by 15
ChatGPT and other Chatbots in Psychiatry
Arbanas, G. (2024) · Archives of Psychiatry Research · doi:10.20471/june.2024.60.02.07 · Cited by 15
Investigating the role of large language models on questions about refractive surgery
Demir, S. (2025) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2025.105787 · Cited by 15
Martela, F. (2025) · AI and Ethics · doi:10.1007/s43681-025-00740-6 · Cited by 19
Wong, L. H., Park, H., & Looi, C. (2024) · Journal of Computer Assisted Learning · doi:10.1111/jcal.12962 · Cited by 20
Ermiş, S., Özal, E., Karapapak, M., Kumantaş, E., & Özal, S. A. (2024) · Journal of Pediatric Ophthalmology & Strabismus · doi:10.3928/01913913-20240911-05 · Cited by 12
Imtiaz, A., King, J. M., Holmes, S., Gupta, A., Bafadhel, M., Melcher, M. L., Hurst, J. R., Farewell, D. M., Bolton, C. E., & Duckers, J. M. (2024) · European Respiratory Journal · doi:10.1183/13993003.00163-2024 · Cited by 14
A Survey of Lay People's Willingness to Generate Legal Advice using Large Language Models (LLMs)
Seabrooke, T., Schneiders, E., Dowthwaite, L., Krook, J., Leesakul, N., Clos, J., Maior, H. A., & Fischer, J. E. (2024) · International Symposium on Trustworthy Autonomous Systems (TAS) · doi:10.1145/3686038.3686043 · Cited by 23
Ghanta, S. N., Alaref, S. J., Lala, A., Nadkarni, G. N., Ganatra, S., Dani, S. S., & Mehta, J. L. (2024) · Diagnostics · doi:10.3390/diagnostics14212393 · Cited by 16
A Survey on the Impact of Pre-Trained Language Models in Sentiment Classification Task
Gautam, H., Gaur, A., & Yadav, D. K. (2025) · International Journal of Data Science and Analytics · doi:10.1007/s41060-025-00805-z · Cited by 25
Elyoseph, Z., Gur, T., Haber, Y., Simon, T., Angert, T., Navon, Y., Tal, A., & Asman, O. (2024) · doi:10.2196/preprints.58011 · Cited by 11
Dillion, D., Mondal, D., Tandon, N., & Gray, K. J. (2024) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/w7236 · Cited by 28
Allen, J. W., Wilkinson, D. J. C., & Savulescu, J. (2025) · Bioethics · doi:10.1111/bioe.70032 · Cited by 15
Collin, H., Tong, C., Srinivas, A., Pegler, A. H., Allan, P. D., & Hagley, D. (2025) · ANZ Journal of Surgery · doi:10.1111/ans.70053 · Cited by 10
Unlocking the potential of ChatGPT in academic libraries
Hussain, A. (2024) · IP Indian Journal of Library Science and Information Technology · doi:10.18231/j.ijlsit.2024.015 · Cited by 17
Geracitano, J., Anderson, B., Coffel, M., Rosenzweig, M., Dorn, S. D., Khairat, S., & Conklin, J. (2025) · Advances in Health Information Science and Practice · doi:10.63116/vxul2925 · Cited by 14
Liu, Z., Hu, L., Zhou, T., Tang, Y., & Cai, Z. (2025) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp61157.2025.00092 · Cited by 22
Wang, B., Shibo, B. W., & Kafle, J. (2025) · Journal of Health Communication · doi:10.1080/10810730.2025.2540864 · Cited by 12
Zhang, L., Yang, J., & Fang, G. (2025) · Frontiers in Public Health · doi:10.3389/fpubh.2025.1637270 · Cited by 15
Jordan, R. F., Sandrone, S., & Southerland, A. M. (2024) · Neurology Education · doi:10.1212/ne9.0000000000200116 · Cited by 8
OpenAI ChatGPT: A Potential Medical Application
Chaddad, A., Jiang, Y., & He, C. (2023) · IEEE International Conference on e-Health Networking, Applications and Services (Healthcom) · doi:10.1109/healthcom56612.2023.10472387 · Cited by 13
Ata, A. M., Aras, B., Taşdelen, Ö. Y., Çelik, C., & Çulha, C. (2024) · Physical & Occupational Therapy In Pediatrics · doi:10.1080/01942638.2024.2316178 · Cited by 8
Zeljković, I., Novák, A., Lisičić, A., Jordán, A., Šerman, A., Jurin, I., Pavlović, N., & Manola, Š. (2025) · Canadian Journal of Cardiology · doi:10.1016/j.cjca.2025.01.036 · Cited by 24
Paging Dr. ChatGPT: safety, accuracy and readability of ChatGPT in ENT emergencies
Soon, S., & Perry, B. (2025) · Australian Journal of Otolaryngology · doi:10.21037/ajo-24-56 · Cited by 10
Metadata
- Authors
- Nov, O., Singh, N., & Mann, D.
- Affiliations
- New York University (US)
- NYU Langone Health (US)
- Corresponding author
- Nov, Oded
- Venue
- JMIR Medical Education
- Publisher
- JMIR Publications Inc.
- Published
- 2023-07-10
- Volume
- 9
- Pages
- e46939
- ISSN
- 2369-3762 (ISSN-L)
- PMC
- PMC10366957
- OpenAlex
- W4380575774
- Cited by
- 198
- References
- 28
- Citations by year
- 2026: 35 · 2025: 67 · 2024: 68 · 2023: 28
- FWCI
- 7.91
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- JMIR Medical Education (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- National Science Foundation (1928614, 2129076)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
- Social Sciences › Business, Management and Accounting › Marketing › AI in Service Interactions
- Keywords
- ChatGPT, large language models, chatbots, patient-provider communication, source identification, trust in chatbots, patient questions, health information seeking, artificial intelligence in healthcare, Turing test, health literacy, task complexity
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.