CoChat
Sign Up
Cited by 8Sentiment Analysis and Opinion MiningOnline Learning and AnalyticsComputational and Text Analysis Methods

Automated Analysis of Preceptor Comments: A Pilot Study Using Sentiment Analysis to Identify Potential Student Issues in Experiential Education

Open access
Fuller, Kathryn, Lupton-Smith, Carly, Hubal, Robert, McLaughlin, Jacqueline E.·2023·American Journal of Pharmaceutical Education
DOI 10.1016/j.ajpe.2023.02.005PMC PMC11826384
Open this paper in CoChat — free accountOpen PDF

Abstract

Objective The purpose of this paper is to describe a sentiment analysis program that aids in identifying pharmacy students at risk for progression issues by automatically scoring preceptor comments as positive or negative. Methods An R-based program to analyze advanced pharmacy practice experiences and introductory pharmacy practice experiences midpoint evaluation of preceptor comments was piloted in phase 1 by comparing the sentiment analysis algorithm results to human coding. The algorithm was refined in phase 2. In phase 3, the validation phase, the final sentiment analysis algorithm analyzed all midpoint student evaluations (n = 1560). Sentiment scores were generated for each preceptor comment, and correlations were performed between sentiment scores and the quantitative scoring provided on the assessment. Results In phase 1, agreement between faculty coders and sentiment analysis was 96%, and in phase 2, agreement between the final codes and sentiment analysis was 92.4% once keywords were added to the sentiment dictionary. In phase 3, a total of 3919 comments from 1560 evaluations were analyzed, and overall, the sentiment analysis results aligned with the quantitative data. Conclusion This sentiment analysis algorithm was accurate in capturing positive and negative comments corresponding to pharmacy student performance. Given the accuracy of this preliminary validation for flagging preceptor comments, there are numerous implications when considering the use of sentiment analysis in pharmacy education. Using a sentiment analysis program minimizes the number of qualitative preceptor comments needing review by experiential faculty, as this program can aid in identifying students at risk of progression issues.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1016/j.ajpe.2023.02.005

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • RoBERTa-driven sentiment analysis of crisis statements shifted communication training from post hoc scoring to in-process calibration-mirroring real-time monitoring paradigms in clinical diagnostics.

    “RoBERTa-driven sentiment analysis of crisis statements shifted communication training from post hoc scoring to in-process calibration—mirroring real-time monitoring paradigms in clinical diagnostics [22].”

    From Reforming hospital management education with the P-MASE pedagogy: a randomized controlled trial (2026) · cites this paper as [22]

    The Discussion (page unavailable) says the sentiment-analysis tool was used in real time to flag potential student progression issues, supporting the general idea of monitoring during training. But the Methods and Discussion describe analysis of pharmacy preceptor comments at midpoint evaluations using an R-based program and the sentimentr package—not RoBERTa analysis of crisis statements, a shift in communication training from post hoc scoring to in-process calibration, or a comparison with clinical diagnostics. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 18

  1. [1]Tekian A, Watling CJ, Roberts TE, Steinert Y, Norcini J. Qualitative and quantitative feedback in the context of competency-based education. Med Teach. 2017; 39(12): 1245– 1249. 10.1080/0142159X.2017.1372564 28927332 · DOI
  2. [2]Hwang GJ, Chu HC, Yin C. Objectives, methodologies and research issues of learning analytics. Interact Learn Environ. 2017; 25(2): 143– 146. 10.1080/10494820.2017.1287338 · DOI
  3. [3]Dyckhoff AL, Lukarov V, Muslim A, Chatti MA, Schroeder U. Supporting action research with learning analytics. In: Proceedings of the Third International Conference on Learning Analytics and Knowledge. Association for Computing Machinery; 2013: 220– 229. doi: 10.1145/2460296.2460340. · DOI
  4. [4]Wong BTM. Learning analytics in higher education: an analysis of case studies. AAOUJ. 2017; 12(1): 21– 40. 10.1108/AAOUJ-01-2017-0009 · DOI
  5. [5]Chan TM, Sebok-Syer SS, Cheung WJ, Workplace-based assessment data in emergency medicine: A scoping review of the literature. AEM Educ Train. 2021; 5(3):e10544 10.1002/aet2.10544 PMC8166307 34099992 · DOI
  6. [6]Greaves FD, Ramirez-Cano DP, Millett CP, Darzi AP, Donaldson LP. Machine learning and sentiment analysis of unstructured free-text information about patient experience online. Lancet. 2012; 380: S10– S. 10.1016/S0140-6736(13)60366-9 · DOI
  7. [7]Denecke K. Health web science: social media data for healthcare. 1st ed. Cham Springer International Publishing; 2015.
  8. [8]Ariaeinejad A, Patel R, Chan TM, Samavi R. P031: using machine learning algorithms for predicting future performance of emergency medicine residents. Can J Emerg Med. 2017; 19(suppl 1): S88– S. 10.1017/cem.2017.233 · DOI
  9. [9]Balfour SP. Assessing writing in MOOCs: automated essay scoring and calibrated peer review[TM]. Res Pract Assess. 2013; 8(1): 40.
  10. [10]Gierl MJ, Latifi S, Lai H, Boulais AP, De Champlain A. Automated essay scoring and the future of educational assessment in medical education. Med Educ. 2014; 48(10): 950– 962. 10.1111/medu.12517 25200016 · DOI · Paper page
  11. [12]ten Cate O, Chen HC. The ingredients of a rich entrustment decision. Med Teach. 2020; 42(12): 1413– 1420. 10.1080/0142159X.2020.1817348 33016803 · DOI
  12. [13]El-Haddad C, Damodaran A, McNeil HP, Hu W. The ABCs of entrustable professional activities: an overview of ‘entrustable professional activities’ in medical education. Intern Med J. 2016; 46(9): 1006– 1010. 10.1111/imj.12914 26388198 · DOI
  13. [14]Abeyaratne C, Galbraith K. A review of entrustable professional activities in pharmacy education. Am J Pharm Educ. 2021: 8872. 10.5688/ajpe8872 PMC10159551 34911704 · DOI
  14. [16]Stahl CC, Jung SA, Rosser AA, Natural language processing and entrustable professional activity text feedback in surgery: A machine learning model of resident autonomy. Am J Surg. 2021; 221(2): 369– 375. 10.1016/j.amjsurg.2020.11.044 33256944 PMC7969407 · DOI
  15. [17]van Atteveldt W, van der Velden MACG, Boukes M. The validity of sentiment analysis: comparing manual annotation, crowd-coding, dictionary approaches, and machine learning algorithms. Commun Meth Meas. 2021; 15(2): 121– 140. 10.1080/19312458.2020.1869198 · DOI
  16. [18]Royal KD. Forty-five common rater errors in medical and health professions education. Educ Health Prof. 2018; 1(2): 33– 35. 10.4103/EHP.EHP_27_18 · DOI
  17. [19]Wolf M, Kurvers RHJM, Ward AJW, Krause, Krause J. Accurate decisions in an uncertain world: collective cognition increases true positives while decreasing false positives. Proc Biol Sci. 2013; 280(1756):20122777 10.1098/rspb.2012.2777 PMC3574371 23407830 · DOI
  18. [20]Liu B. Sentiment analysis: a multifaceted problem. IEEE Intel Syst. 2010; 25(3): 76– 80.

Cited by

Sorted by citations

Authors
Fuller, K., Lupton-Smith, C., Hubal, R., & McLaughlin, J. E.
Affiliations
University of North Carolina at Chapel Hill (US)
ORCID
Venue
American Journal of Pharmaceutical Education
Publisher
Elsevier BV
Published
2023-09
Volume
87
Issue
9
Pages
100005
ISSN
0002-9459 (ISSN-L)
DOI
10.1016/j.ajpe.2023.02.005
PMC
PMC11826384
OpenAlex
W4367153427
Cited by
8
References
1
Citations by year
2026: 2 · 2025: 3 · 2024: 3
FWCI
0.75
Type
journal article
Language
en
Open access
Licence
Locations
  • American Journal of Pharmaceutical Education (publishedVersion)
  • PubMed (publishedVersion)
  • PubMed Central (acceptedVersion)
Funders
National Institute on Aging (T32AG000247)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
  • Physical Sciences › Computer Science › Computer Science Applications › Online Learning and Analytics
  • Social Sciences › General Social Sciences › Computational and Text Analysis Methods
Keywords
sentiment analysis, pharmacy students, experiential education, student assessment, at-risk students, computational text analysis, advanced pharmacy practice experience, introductory pharmacy practice experience
MeSH terms
Sentiment Analysis, Education, Pharmacy, Humans, Pilot Projects, Students, Pharmacy, Problem-Based Learning

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review