Artificial intelligence-based conversational agent to support medication prescribing
Open accessThe DOI matches its Crossref record; 25 of the 25 references it cites were found in a registry or an index.
Abstract
Objective: This article describes the system architecture, training, initial use, and performance of Watson Assistant (WA), an artificial intelligence-based conversational agent, accessible within Micromedex®. Materials and methods: The number and frequency of intents (target of a user’s query) triggered in WA during its initial use were examined; intents triggered over 9 months were compared to the frequency of topics accessed via keyword search of Micromedex. Accuracy of WA intents assigned to 400 queries was compared to assignments by 2 independent subject matter experts (SMEs), with inter-rater reliability measured by Cohen’s kappa. Results: In over 126 000 conversations with WA, intents most frequently triggered involved dosing (N = 30 239, 23.9%) and administration (N = 14 520, 11.5%). SMEs with substantial inter-rater agreement (kappa = 0.71) agreed with intent mapping in 247 of 400 queries (62%), including 16 queries related to content that WA and SMEs agreed was unavailable in WA. SMEs found 57 (14%) of 400 queries incorrectly mapped by WA; 112 (28%) queries unanswerable by WA included queries that were either ambiguous, contained unrecognized typographical errors, or addressed topics unavailable to WA. Of the queries answerable by WA (288), SMEs determined 231 (80%) were correctly linked to an intent. Discussion: A conversational agent successfully linked most queries to intents in Micromedex. Ongoing system training seeks to widen the scope of WA and improve matching capabilities. Conclusion: WA enabled Micromedex users to obtain answers to many medication-related questions using natural language, with the conversational agent facilitating mapping to a broader distribution of topics than standard keyword searches.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures



Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Similar manual processes have been reported in chatbot development studies to fine-tune the system.67”
From A chatbot for hypertension self-management support: user-centered design, development, and usability testing (2023) · cites this paper as [67]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1994) · doi:10.1201/9780429246593 · Cited by 40,500
Garg, A. X., Adhikari, N. K. J., McDonald, H., Rosas-Arellano, M. P., Devereaux, P. J., Beyene, J., Sam, J., & Haynes, R. B. (2005) · JAMA · doi:10.1001/jama.293.10.1223 · Cited by 2,936
Inter-Coder Agreement for Computational Linguistics
Artstein, R., & Poesio, M. (2008) · Computational Linguistics · doi:10.1162/coli.07-034-r2 · Cited by 1,665
Conversational agents in healthcare: a systematic review
Laranjo, L., Dunn, A. G., Tong, H. L., Kocaballi, A. B., Chen, J., Bashir, R., Surian, D., Gallego, B., Magrabi, F., Lau, A. Y. S., et al. (2018) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocy072 · Cited by 1,388 · cited 1×
“There are few rigorous studies of conversational agents in healthcare, and none related to pharmacy.(28) Thus, the current study is the first to report the technical performance of a conversational agent aimed at answering medication-related questions using ground-truth intents established by 2 independent SMEs, under…”
BackgroundAgreement, the F-Measure, and Reliability in Information Retrieval
Hripcsak, G. (2005) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1733 · Cited by 949
Mowry, J. B., Spyker, D. A., Cantilena, L. R., McMillan, N., & Ford, M. (2014) · Clinical Toxicology · doi:10.3109/15563650.2014.987397 · Cited by 849
Introduction to “This is Watson”
Ferrucci, D. A. (2012) · IBM Journal of Research and Development · doi:10.1147/jrd.2012.2184356 · Cited by 484
Jiang, M., Chen, Y., Liu, M., Rosenbloom, S. T., Mani, S., Denny, J. C., & Xu, H. (2011) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000163 · Cited by 291
Garg, A. X., Adhikari, N. K. J., McDonald, H., Rosas-Arellano, M. P., Devereaux, P. J., Beyene, J., Sam, J., & Haynes, R. B. (2007) · Cited by 253
Measuring agreement in medical informatics reliability studies
Hripcsak, G., & Heitjan, D. F. (2002) · Journal of Biomedical Informatics · doi:10.1016/s1532-0464(02)00500-2 · Cited by 226
Evaluation of personal digital assistant software for drug interactions
Barrons, R. (2004) · American Journal of Health-System Pharmacy · doi:10.1093/ajhp/61.4.380 · Cited by 122
Types and frequency of errors in the preparation and administration of drugs
Mendes, J. R., Lopes, M. C. B. T., Campanharo, C. R. V., Okuno, M. F. P., & Batista, R. E. A. (2018) · Einstein (São Paulo) · doi:10.1590/s1679-45082018ao4146 · Cited by 122
Clinical decision support tools: analysis of online drug information databases
Clauson, K. A., Marsh, W. A., Polen, H. H., Seamon, M. J., & Ortiz, B. I. (2007) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-7-7 · Cited by 117
Wolfe, D., Yazdi, F., Kanji, S., Burry, L., Beck, A., Butler, C., Esmaeilisaraji, L., Hamel, C., Hersi, M., Skidmore, B., et al. (2018) · PLOS ONE · doi:10.1371/journal.pone.0205426 · Cited by 101
Structured data and inference in DeepQA
Kalyanpur, A. A., Boguraev, B., Patwardhan, S., Murdock, J. W., Lally, A., Welty, C., Prager, J. M., Coppola, B., Fokoue, A., Zhang, L., et al. (2012) · IBM Journal of Research and Development · doi:10.1147/jrd.2012.2188737 · Cited by 77
Yang, X., Bian, J., Gong, Y., Hogan, W. R., & Wu, Y. (2019) · Drug Safety · doi:10.1007/s40264-018-0761-0 · Cited by 66
Co-medication and potential drug interactions among patients with epilepsy
Bosak, M., Słowik, A., Iwańska, A., Lipińska, M., & Turaj, W. (2019) · Seizure · doi:10.1016/j.seizure.2019.01.014 · Cited by 46
Doan, S., & Xu, H. (2010) · PubMed · Cited by 41
Consistency of psychotropic drug–drug interactions listed in drug monographs
Liu, X., Hatton, R. C., Zhu, Y., Hincapie‐Castillo, J. M., Bussing, R., Barnicoat, M., & Winterstein, A. G. (2017) · Journal of the American Pharmacists Association · doi:10.1016/j.japh.2017.07.008 · Cited by 26
Lexicomp Online and Micromedex 2.0
Chatfield, A. J. (2015) · Journal of the Medical Library Association JMLA · doi:10.3163/1536-5050.103.2.016 · Cited by 23
Knollmann, B. C., SMYTH, B., Garnett, C. E., et al. (2005) · Clinical Pharmacology & Therapeutics · doi:10.1016/j.clpt.2005.03.007 · Cited by 17
Ontology-Based Natural Language Query Interfaces for Data Exploration.
Lei, C., Özcan, F., Quamar, A., et al. (2018) · IEEE Data(base) Engineering Bulletin · Cited by 16
Grion, A. M., Gallo, U., Tinjala, D. D., Daragjati, J., Loreggian, M., Cardaci, G., Mangoni, A. A., & Pilotto, A. (2016) · Drugs & Aging · doi:10.1007/s40266-015-0340-3 · Cited by 13
Evaluation of drug information resources for drug-ethanol and drug-tobacco interactions
Beckett, R. D., Stump, C. D., & Dyer, M. A. (2019) · Journal of the Medical Library Association JMLA · doi:10.5195/jmla.2019.549 · Cited by 9
Online information retrieval in pharmacy and related fields
Perry, C. A. (1986) · American Journal of Health-System Pharmacy · doi:10.1093/ajhp/43.6.1509 · Cited by 8
Cited by
Griffin, A. C., Khairat, S., Bailey, S. C., & Chung, A. E. (2023) · JAMIA Open · doi:10.1093/jamiaopen/ooad073 · Cited by 46
Metadata
- Authors
- Preininger, A. M., South, B., Heiland, J., Buchold, A., Baca, M., Wang, S., Nipper, R., Kutub, N., Bohanan, B., & Jackson, G. P.
- Affiliations
- IBM (United States) (US)
- Corresponding author
- Preininger, Anita M
- Venue
- JAMIA Open
- Publisher
- Oxford University Press (OUP)
- Published
- 2020-05-01
- Volume
- 3
- Issue
- 2
- Pages
- 225-232
- ISSN
- 2574-2531 (ISSN-L)
- PMC
- PMC7382615
- OpenAlex
- W3022394003
- Semantic Scholar
- 54955f4f47baf62a859d5ec5b426d10d0cf61036
- Cited by
- 16
- References
- 25
- Citations by year
- 2026: 1 · 2025: 1 · 2024: 4 · 2023: 4 · 2022: 2 · 2021: 2 · 2020: 2
- FWCI
- 0.74
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- JAMIA Open (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Funders
- IBM
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- chatbots, drug prescribing, Micromedex, intent classification, natural language question answering, drug information retrieval, inter-rater reliability, clinical decision support
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.