The DOI matches its Crossref record; the references it cites are not checked yet.
Risks and Benefits of AI-generated Text Summarization for Expert Level Content in Graduate Health Informatics
Abstract
AI-generated text summarization (AI-GTS) is now a popular topic in applied computer science education. It has proven helpful in various sectors, but its benefits and risks in education have not been thoroughly investigated. Few researchers have demonstrated the benefits of employing AI-generated text summaries in learning to generate ideas swiftly and to explore insights and hidden knowledge. AI-GTS has made it easier for students to understand electronically-available critical information. On the other hand, the risks linked with its implementation in education are understudied. Some anticipated risks include harming pupils' writing skills, overdependence, reduced critical thinking capacity, and increased plagiarism. This paper presents the application of AI-generated text summarization in a graduate health informatics course and discusses the risks and benefits to students. Furthermore, utilizing the Bidirectional Encoder Representations from Transformers (BERT) model, we demonstrate that the current state-of-the-art AI-generated text summarization has the potential to create expert knowledge content. We conducted a study with 58 health informatics graduate students in the Fall of 2019 to write annotated bibliography for 25 articles each, to which we also added the AI-generated article summaries. We then asked the students to peer grade and distinguish the AI-generated annotations from the student-written summary. Using the Kruskal-Wallis test, we found no significant difference in the peer grades between the two. The robustness of such AI-generated text summarization raises important questions for educators teaching in health informatics.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Several studies highlight the risks of underperforming or biased AI models, which can negatively impact human learning and perpetuate systemic biases.
“Several studies highlight the risks of underperforming or biased AI models, which can negatively impact human learning and perpetuate systemic biases, such as gender, racial, and social class biases [87,88].”
From Promises and challenges of generative artificial intelligence for human learning (2024) · cites this paper as [88]
Text summaries generated using BERT had no significant differences compared with student-generated summaries and cannot be differentiated by graduate students.
“Text summaries generated using BERT had no significant differences compared with studentgenerated summaries and can not be differentiated by graduate students (Merine & Purkayastha, 2022).”
From Practical and ethical challenges of large language models in education: A systematic scoping review (2024) · cites this paper as Merine & Purkayastha, 2022
LLMs-based innovations are subject to data privacy issues but were rarely mentioned or investigated in the literature.
“LLMs-based innovations are subject to data privacy issues but were rarely mentioned or investigated in the literature (Merine & Purkayastha, 2022)”
From Practical and ethical challenges of large language models in education: A systematic scoping review (2024) · cites this paper as Merine & Purkayastha, 2022
Two studies have suggested that potential bias and discrimination issues may occur if adopting a model that is accurate but unfair.
“Apart from issues with adopting inaccurate models, two studies have suggested that potential bias and discrimination issues may occur if adopting a model that is accurate but unfair (Merine & Purkayastha, 2022;Sha et al., 2021).”
From Practical and ethical challenges of large language models in education: A systematic scoping review (2024) · cites this paper as Merine & Purkayastha, 2022
Citations
The citation lists haven’t been fetched yet
References 38 · Cited by 16
Research works printed but not linked: 43
- xiao, 0, Ehrd 371 - applied learning principles (spring 2022) How to prepare an annotated bibliography
- engle, 2018, How to prepare an annotated bibliography The annotated bibliography
- aries, 2019, Automatic text summarization: What has been done and what has to be done, ArXiv Preprint
- 10.1016/j.sbspro.2015.06.013 · DOI
- sri, 0, A survey on automatic text summarization techniques, Journal of Physics Conference Series, 2040, 12044
- 10.3917/rfla.131.0063 · DOI
- 10.3390/app11146287 · DOI
- 10.1016/j.pec.2013.04.019 · DOI
- lehto, 2019, Automatic text summarization of swedish news articles
- wedi, 0, Exploring the imple-mentation of automatic text summarization in online learning setting, Proceedings of the 1 st International Conference on Information Technology and Education (ICITE 2020), 5
- liu, 2019, Fine-tune bert for extractive summarization, ArXiv Preprint
- tarnpradab, 0, Toward extractive summarization of online forum discussions via hierarchical attention networks, The Thirtieth International Flairs Conference
- siu, 2016, Duplicate question detection using online learning
- 10.1155/2013/945623 · DOI
- miller, 2019, Leveraging bert for extractive text summarization on lectures, ArXiv Preprint
- 0, How important is text summarization for the next generation?, suntecai
- 2020, Benefits of automatic text summarization
- pramudianto, 0, Assessing the quality of automatic summarization for peer review in education, EDM (Workshops)
- 10.13189/ujer.2018.061018 · DOI
- 10.1016/j.sbspro.2009.01.440 · DOI
- mani, 1999, Advances in Automatic Text Summarization MIT Press
- pedro, 2019, Artificial Intelligence in Education Challenges and Opportunities for Sustainable Development
- 10.37398/JSR.2020.640148 · DOI
- sharma, 2016, A survey on extractive text summarization, International Journal of Advanced Research in Computer Science and Software Engineering, 6, 461
- ter hoeve, 2020, What makes a good summary? investigating the focus of automatic summarization in an educational context, ArXiv Preprint
- 10.35940/ijrte.C4996.098319 · DOI
- barbella, 2021, A comparison of methods for the evaluation of text summarization techniques
- widyassari, 2020, Review of automatic text summarization techniques & methods, Journal of King Saud University-Computer and Information Sciences
- wanjale, 0, Comprehensive survey on abstractive text summarization
- 10.5220/0009723306480655 · DOI
- dafoe, 2018, Ai governance: a research agenda, Governance of AI Program Future of Humanity Institute University of Oxford Oxford UK, 1442, 1443
- bazerman, 2010, Summarizing: The author's main ideas, Informed writer Using sources in the disciplines, 5, 50
- horev, 2018, Bert explained: State of the art language model for nlp, Towards Data Science, 10
- levy, 2020, Less is more? how to teach summary writing
- xiao, 2021, Primer: Pyramid-based masked sentence pre-training for multi-document summarization, ArXiv Preprint
- belet, 2007, Effectiveness of learning strategies over reading comprehension, writing skills and learners' attitudes towards turkish course, Journal of Theory and Practice in Education, 3, 69
- van aken, 0, How does bert an-swer questions? a layer-wise analysis of transformer representations, Proceedings of ACM International Conference on Information and Knowledge Management, 1823
- 10.1186/s40862-020-00090-2 · DOI
- singh, 2020, Multi-label natural language processing to identify diagnosis and procedure codes from mimic-iii inpatient notes, ArXiv Preprint
- 10.17507/tpls.1006.11 · DOI
- 10.1109/TAAI48200.2019.8959920 · DOI
- moiyadi, 2016, Nlp based text summarization using semantic analysis, International Journal of Advanced Engineering Management and Science, 2, 239678
- vincent, 2021, Google is using ai to help users explore the topics they're searching for
Metadata
- Authors
- Merine, R., & Purkayastha, S.
- Affiliations
- Indiana University Indianapolis (US)
- Venue
- 2022 IEEE 10th International Conference on Healthcare Informatics (ICHI)
- Publisher
- IEEE
- Published
- 2022-06
- Pages
- 567-574
- OpenAlex
- W4297685771
- Cited by
- 16
- References
- 38
- Citations by year
- 2026: 4 · 2025: 5 · 2024: 4 · 2023: 3
- FWCI
- 2.68
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE International Conference on Healthcare Informatics (ICHI) (publishedVersion)
- Topics
- Social Sciences › Education › Artificial Intelligence in Education
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Advanced Text Analysis Techniques
- Keywords
- health informatics education, graduate students, annotated bibliography, BERT, peer grading, student writing ability, plagiarism risk, overreliance on AI, critical thinking
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
