Evaluating the use of large language model in identifying top research questions in gastroenterology
Open accessAbstract
The field of gastroenterology (GI) is constantly evolving. It is essential to pinpoint the most pressing and important research questions. To evaluate the potential of chatGPT for identifying research priorities in GI and provide a starting point for further investigation. We queried chatGPT on four key topics in GI: inflammatory bowel disease, microbiome, Artificial Intelligence in GI, and advanced endoscopy in GI. A panel of experienced gastroenterologists separately reviewed and rated the generated research questions on a scale of 1–5, with 5 being the most important and relevant to current research in GI. chatGPT generated relevant and clear research questions. Yet, the questions were not considered original by the panel of gastroenterologists. On average, the questions were rated 3.6 ± 1.4, with inter-rater reliability ranging from 0.80 to 0.98 (p < 0.001). The mean grades for relevance, clarity, specificity, and originality were 4.9 ± 0.1, 4.6 ± 0.4, 3.1 ± 0.2, 1.5 ± 0.4, respectively. Our study suggests that Large Language Models (LLMs) may be a useful tool for identifying research priorities in the field of GI, but more work is needed to improve the novelty of the generated research questions.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
LLMs demonstrate potential applications in medical assistance, including identifying gastrointestinal diseases, dementia diagnosis, accelerating COVID-19 literature evaluation, and overall healthcare.
“In the field of medical assistance, LLMs demonstrate potential applications, including research on identifying gastrointestinal diseases [99], dementia diagnosis [217], accelerating the evaluation of COVID-19 literature [93], and their overall potential in healthcare [17].”
From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [99]
The Abstract and Discussion describe ChatGPT’s potential to identify research priorities and generate research questions in gastroenterology. They do not report using an LLM to identify gastrointestinal diseases, which is the application attributed to this source by citation [99]. The dementia, COVID-19, and broader healthcare examples have separate citations in the citing sentence. (LLM, conf=0.84)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
A guideline of selecting and reporting intraclass correlation coefficients for reliability research
Koo, T. K., & Li, M. Y. (2016) · Journal of Chiropractic Medicine · doi:10.1016/j.jcm.2016.02.012 · Cited by 29,639 · see citing works
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · see citing works
Advances in neural information processing systems 7
Cios, K. J., & Shields, M. E. (1997) · Neurocomputing · doi:10.1016/s0925-2312(97)90018-7 · Cited by 22,089 · see citing works
Evaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436 · see citing works
The Effectiveness of Artificial Intelligence Conversational Agents in Health Care: Systematic Review
Milne-Ives, M., de Cock, C., Lim, E., Shehadeh, M. H., de Pennington, N., Mole, G., Normando, E., & Meinert, E. (2020) · Journal of Medical Internet Research · doi:10.2196/20346 · Cited by 626 · see citing works
Abstracts written by ChatGPT fool scientists
Else, H. (2023) · Nature · doi:10.1038/d41586-023-00056-7 · Cited by 611 · see citing works
On the State of the Art of Evaluation in Neural Language ModelsPreprint
Melis, G., Dyer, C., & Blunsom, P. (2017) · International Conference on Learning Representations · doi:10.48550/arxiv.1707.05589 · Cited by 578 · see citing works
News Summarization and Evaluation in the Era of GPT-3
Goyal, T., Li, J. J., & Durrett, G. (2022) · arXiv · doi:10.48550/arxiv.2209.12356 · Cited by 175 · see citing works
Are ChatGPT and AlphaCode going to replace programmers?
Castelvecchi, D. (2022) · Nature · doi:10.1038/d41586-022-04383-z · Cited by 138 · see citing works
Evaluating the Text-to-SQL Capabilities of Large Language Models
Rajkumar, N., Li, R., & Bahdanau, D. (2022) · arXiv · doi:10.48550/arxiv.2204.00498 · Cited by 56 · see citing works
Topic Modeling for Interpretable Text Classification From EHRs
Rijcken, E., Kaymak, U., Scheepers, F. E., et al. (2022) · Frontiers in Big Data · doi:10.3389/fdata.2022.846930 · Cited by 37 · see citing works
Evaluating Commonsense in Pre-trained Language ModelsPreprint
Zhou, X., Zhang, Y., Cui, L., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.11931 · Cited by 27 · see citing works
Innovation in Gastroenterology—Can We Do Better?
Klang, E., Soffer, S., Tsur, A., Shachar, E., & Lahat, A. (2022) · Biomimetics · doi:10.3390/biomimetics7010033 · Cited by 8 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 1
- [6]King, M. & Cook, P. Evaluating approaches to personalizing language models. In Proceedings of the 12th Language Resources and Evaluation Conference 2461–2469 (2020).
Cited by
Sorted by citations
A Survey on Evaluation of Large Language Models
Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3641289 · Cited by 2,842 · see citing works
“Medical assistants: (Lahat et al., 2023) / (Cascella et al.”
ChatGPT in healthcare: A taxonomy and systematic review
Li, J., Dada, A., Puladi, B., Kleesiek, J., & Egger, J. (2024) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2024.108013 · Cited by 452 · see citing works
Sohail, S. S., Farhat, F., Himeur, Y., Nadeem, M., Madsen, D. Ø., Singh, Y., Atalla, S., & Mansoor, W. (2023) · Journal of King Saud University - Computer and Information Sciences · doi:10.1016/j.jksuci.2023.101675 · Cited by 192 · see citing works
He, K., Mao, R., Lin, Q., Ruan, Y., Lan, X., Feng, M., & Cambria, E. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.102963 · Cited by 147 · see citing works
Eppler, M., Ganjavi, C., Ramacciotti, L. S., Piazza, P., Rodler, S., Checcucci, E., Gomez Rivas, J., Kowalewski, K. F., Belenchón, I. R., Puliatti, S., et al. (2024) · European Urology · doi:10.1016/j.eururo.2023.10.014 · Cited by 117 · see citing works
Large Language Model‐Based Chatbots in Higher Education
Yigci, D., Eryilmaz, M., Yetisen, A. K., Tasoglu, S., & Ozcan, A. (2025) · Advanced Intelligent Systems · doi:10.1002/aisy.202400429 · Cited by 113 · see citing works
ChatGPT as a Complementary Mental Health Resource: A Boon or a Bane
Farhat, F. (2023) · doi:10.20944/preprints202307.1479.v1 · Cited by 76 · see citing works
He, K., Mao, R., Lin, Q., Ruan, Y., Lan, X., Feng, M., & Cambria, E. (2024) · doi:10.2139/ssrn.4809363 · Cited by 69 · see citing works
Gravina, A. G., Pellegrino, R., Cipullo, M., Palladino, G., Imperio, G., Ventura, A., Auletta, S., Ciamarra, P., & Federico, A. (2024) · World Journal of Gastroenterology · doi:10.3748/wjg.v30.i1.17 · Cited by 56 · see citing works
ChatGPT applications in Academic Research: A Review of Benefits, Concerns, and Recommendations
AlZaabi, A., ALAmri, A., Albalushi, H., Aljabri, R., & AalAbdulsalam, A. (2023) · doi:10.1101/2023.08.17.553688 · Cited by 32 · see citing works
- Authors
- Lahat, A., Shachar, E., Avidan, B., Shatz, Z., Glicksberg, B. S., & Klang, E.
- Affiliations
- Tel Aviv University (IL)
- Icahn School of Medicine at Mount Sinai (US)
- Corresponding author
- Lahat, Adi
- Venue
- Scientific Reports
- Published
- 2023-03-13
- Volume
- 13
- Issue
- 1
- Pages
- 4164-4164
- ISSN
- 2045-2322 (ISSN-L)
- PMC
- PMC10011374
- OpenAlex
- W4324129637
- Cited by
- 111
- References
- 13
- Citations by year
- 2026: 12 · 2025: 29 · 2024: 43 · 2023: 27
- FWCI
- 4.47
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- Scientific Reports (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- large language models, ChatGPT, research question formulation, research priorities, gastroenterology, inflammatory bowel disease, gut microbiota, advanced endoscopy, inter-rater reliability
- MeSH terms
- Gastroenterologists, Artificial Intelligence, Gastroenterology, Humans, Reproducibility of Results, Inflammatory Bowel Diseases
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
