CoChat

Artificial intelligence-based conversational agent to support medication prescribing

Open access

The DOI matches its Crossref record; 25 of the 25 references it cites were found in a registry or an index.

Abstract

Objective: This article describes the system architecture, training, initial use, and performance of Watson Assistant (WA), an artificial intelligence-based conversational agent, accessible within Micromedex®. Materials and methods: The number and frequency of intents (target of a user’s query) triggered in WA during its initial use were examined; intents triggered over 9 months were compared to the frequency of topics accessed via keyword search of Micromedex. Accuracy of WA intents assigned to 400 queries was compared to assignments by 2 independent subject matter experts (SMEs), with inter-rater reliability measured by Cohen’s kappa. Results: In over 126 000 conversations with WA, intents most frequently triggered involved dosing (N = 30 239, 23.9%) and administration (N = 14 520, 11.5%). SMEs with substantial inter-rater agreement (kappa = 0.71) agreed with intent mapping in 247 of 400 queries (62%), including 16 queries related to content that WA and SMEs agreed was unavailable in WA. SMEs found 57 (14%) of 400 queries incorrectly mapped by WA; 112 (28%) queries unanswerable by WA included queries that were either ambiguous, contained unrecognized typographical errors, or addressed topics unavailable to WA. Of the queries answerable by WA (288), SMEs determined 231 (80%) were correctly linked to an intent. Discussion: A conversational agent successfully linked most queries to intents in Micromedex. Ongoing system training seeks to widen the scope of WA and improve matching capabilities. Conclusion: WA enabled Micromedex users to obtain answers to many medication-related questions using natural language, with the conversational agent facilitating mapping to a broader distribution of topics than standard keyword searches.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1093/jamiaopen/ooaa009

Cites

25 of 25 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1. System architecture. CMS, content management system; DB, database; HIDS, health internet delivery system; UI, user interface. Dotted lines, data flow; red lines, representational state transfer application (REST) services; solid lines, data retrieval.
Figure 1. System architecture. CMS, content management system; DB, database; HIDS, health internet delivery system; UI, user interface. Dotted lines, data flow; red lines, representational state transfer application (REST) services; solid lines, data retrieval.CC BY · PubMed Central
Figure 2. User interface. (A) Micromedex home page. Red arrow to “learn more” under “Ask Watson” explains what information Watson can provide (light green box), including examples of real-world questions, the types of information Micromedex with Watson understands (ie, drug information, drug interactions, and IV compatibility) and what Micromedex with Watson does not understand (ie, in-depth answers, NeoFax/pediatrics, toxicology, diseases, laboratory, alternative medicine, reproductive risks and third party content such as Martindale and Index Nominum). (B) WA chat window key features.
Figure 2. User interface. (A) Micromedex home page. Red arrow to “learn more” under “Ask Watson” explains what information Watson can provide (light green box), including examples of real-world questions, the types of information Micromedex with Watson understands (ie, drug information, drug interactions, and IV compatibility) and what Micromedex with Watson does not understand (ie, in-depth answers, NeoFax/pediatrics, toxicology, diseases, laboratory, alternative medicine, reproductive risks and third party content such as Martindale and Index Nominum). (B) WA chat window key features.CC BY · PubMed Central
Figure 3. Frequency of intents. Utterances that mapped to queries (42 771 total queries) collected between October of 2018 through January 2019, grouped by category or intent to which it was mapped by WA.
Figure 3. Frequency of intents. Utterances that mapped to queries (42 771 total queries) collected between October of 2018 through January 2019, grouped by category or intent to which it was mapped by WA.CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Cited by

See all 16

Metadata

Authors
Preininger, A. M., South, B., Heiland, J., Buchold, A., Baca, M., Wang, S., Nipper, R., Kutub, N., Bohanan, B., & Jackson, G. P.
Affiliations
IBM (United States) (US)
Corresponding author
Preininger, Anita M
Venue
JAMIA Open
Published
2020-05-01
Volume
3
Issue
2
Pages
225-232
ISSN
2574-2531 (ISSN-L)
OpenAlex
W3022394003
Cited by
16
References
25
Citations by year
2026: 1 · 2025: 1 · 2024: 4 · 2023: 4 · 2022: 2 · 2021: 2 · 2020: 2
FWCI
0.74
Type
journal article
Language
en
Open access
Locations
  • JAMIA Open (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
Funders
IBM
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
  • Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
Keywords
chatbots, drug prescribing, Micromedex, intent classification, natural language question answering, drug information retrieval, inter-rater reliability, clinical decision support

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.