CoChat

ExaCT: automatic extraction of clinical trial characteristics from journal publications

Open access
Kiritchenko, Svetlana, de Bruijn, Berry, Carini, Simona, Martin, Joel, Sim, Ida · 2010 · BMC Medical Informatics and Decision Making

The DOI matches its Crossref record; 39 of the 39 references it cites were found in a registry or an index.

Abstract

BACKGROUND: Clinical trials are one of the most important sources of evidence for guiding evidence-based practice and the design of new trials. However, most of this information is available only in free text - e.g., in journal publications - which is labour intensive to process for systematic reviews, meta-analyses, and other evidence synthesis studies. This paper presents an automatic information extraction system, called ExaCT, that assists users with locating and extracting key trial characteristics (e.g., eligibility criteria, sample size, drug dosage, primary outcomes) from full-text journal articles reporting on randomized controlled trials (RCTs). METHODS: ExaCT consists of two parts: an information extraction (IE) engine that searches the article for text fragments that best describe the trial characteristics, and a web browser-based user interface that allows human reviewers to assess and modify the suggested selections. The IE engine uses a statistical text classifier to locate those sentences that have the highest probability of describing a trial characteristic. Then, the IE engine's second stage applies simple rules to these sentences to extract text fragments containing the target answer. The same approach is used for all 21 trial characteristics selected for this study. RESULTS: We evaluated ExaCT using 50 previously unseen articles describing RCTs. The text classifier (first stage) was able to recover 88% of relevant sentences among its top five candidates (top5 recall) with the topmost candidate being relevant in 80% of cases (top1 precision). Precision and recall of the extraction rules (second stage) were 93% and 91%, respectively. Together, the two stages of the extraction engine were able to provide (partially) correct solutions in 992 out of 1050 test tasks (94%), with a majority of these (696) representing fully correct and complete answers. CONCLUSIONS: Our experiments confirmed the applicability and efficacy of ExaCT. Furthermore, they demonstrated that combining a statistical method with 'weak' extraction rules can identify a variety of study characteristics. The system is flexible and can be extended to handle other characteristics and document types (e.g., study protocols).

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1186/1472-6947-10-56

Cites

39 of 39 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 Example of an abstract and the corresponding template filling. The top part of the figure shows the abstract of a journal article regarding an RCT. The bottom part shows the template with the slots filled in with text excerpts from the abstract. Some slots are left empty as the information is not present in the abstract.
Figure 1 Example of an abstract and the corresponding template filling. The top part of the figure shows the abstract of a journal article regarding an RCT. The bottom part shows the template with the slots filled in with text excerpts from the abstract. Some slots are left empty as the information is not present in the abstract.CC BY · PubMed Central
Figure 2 Example of the system's output. The publication details of an article are retrieved directly from PubMed. For other information elements, the system outputs five best candidate sentences in decreasing order of confidence. The text fragments identified by the system as containing the target information are highlighted in the retrieved sentences whose confidence score is above a certain threshold. For eligibility criteria, the whole sentence is considered the target, so no fragments are highlighted in those sentences. Sentences in black were confirmed as correct answers by the field expert.
Figure 2 Example of the system's output. The publication details of an article are retrieved directly from PubMed. For other information elements, the system outputs five best candidate sentences in decreasing order of confidence. The text fragments identified by the system as containing the target information are highlighted in the retrieved sentences whose confidence score is above a certain threshold. For eligibility criteria, the whole sentence is considered the target, so no fragments are highlighted in those sentences. Sentences in black were confirmed as correct answers by the field expert.CC BY · PubMed Central
Figure 3 Hierarchy of information elements. The hierarchy of the 21 information elements used in ExaCT. The elements are grouped semantically in the four-level hierarchical structure to enhance the sentence classification method.
Figure 3 Hierarchy of information elements. The hierarchy of the 21 information elements used in ExaCT. The elements are grouped semantically in the four-level hierarchical structure to enhance the sentence classification method.CC BY · PubMed Central
Figure 4 Interactive user interface for curation. The user interface consists of two panels simultaneously displayed on a screen: the left panel displays the system's suggestions and the right panel displays the original article. A button next to each sentence on the left panel highlights the same sentence within the article on the right panel. The information elements are divided into five tabs: publication information, meta information, enrolment, interventions, and outcomes. For each element, the top-scored solution with a high confidence score or a "not found" message is initially displayed. A user can expand the list of the system's suggestions to the five highest scoring solutions and choose one or more sentences as the most relevant to the element. If a target sentence is not among the system's five choices, a curator can add a sentence directly from the article by copying-and-pasting or dragging-and-dropping it into the corresponding element text box, or by right-clicking and selecting the relevant element from the menu. Automatically extracted fragments are highlighted in the sentences. The highlighting can be modified by a curator using the mouse.
Figure 4 Interactive user interface for curation. The user interface consists of two panels simultaneously displayed on a screen: the left panel displays the system's suggestions and the right panel displays the original article. A button next to each sentence on the left panel highlights the same sentence within the article on the right panel. The information elements are divided into five tabs: publication information, meta information, enrolment, interventions, and outcomes. For each element, the top-scored solution with a high confidence score or a "not found" message is initially displayed. A user can expand the list of the system's suggestions to the five highest scoring solutions and choose one or more sentences as the most relevant to the element. If a target sentence is not among the system's five choices, a curator can add a sentence directly from the article by copying-and-pasting or dragging-and-dropping it into the corresponding element text box, or by right-clicking and selecting the relevant element from the menu. Automatically extracted fragments are highlighted in the sentences. The highlighting can be modified by a curator using the mouse.CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

17 more claims about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 39 references

Research works printed but not linked: 5

  1. Science Commons. --- Either ISSN or Journal title must be supplied.. [ http://sciencecommons.org/ ]
  2. The Human Study Database. --- Either ISSN or Journal title must be supplied.. [ http://hsdbwiki.org ]
  3. Demner-Fushman D, Lin J: Knowledge extraction for clinical question answering: preliminary results. Proceedings of the AAAI Workshop on Question Answering in Restricted Domains: 10 July 2005; Pittsburgh, PA. Edited by: Molla D, Vicedo JL. 2005, AAAI Press, 1-9.
  4. Hansen MJ, Rasmussen NO, Chung G: Extracting number of trial participants from abstracts of randomized controlled trials. Proceedings of Tromsø Telemedicine and eHealth Conference: 9-11 June 2008; Tromsø, Norway. 2008
  5. Van Rijsbergen CJ: Information Retrieval. 1979, London: Buttersworth, 2

Cited by

See all 191

Metadata

Authors
Kiritchenko, S., de Bruijn, B., Carini, S., Martin, J., & Sim, I.
Affiliations
  • National Research Council Canada (CA)
  • University of California, San Francisco (US)
Corresponding author
Kiritchenko, Svetlana
Venue
BMC Medical Informatics and Decision Making
Published
2010-09-28
Volume
10
Issue
1
Pages
56-56
ISSN
1472-6947 (ISSN-L)
OpenAlex
W2020959315
Cited by
188
References
39
Citations by year
2026: 8 · 2025: 17 · 2024: 20 · 2023: 16 · 2022: 9 · 2021: 25 · 2020: 16 · 2019: 13 · 2018: 8 · 2017: 8 · 2016: 11 · 2015: 11 · 2014: 11 · 2013: 8 · 2012: 5
FWCI
1.96
Type
journal article
Language
en
Open access
Licence
  • cc-by
  • creativecommons.org/licenses/by/2.0 (tdm)
Locations
  • BMC Medical Informatics and Decision Making (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • NPARC (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
  • eScholarship (California Digital Library) (submittedVersion)
Funders
U.S. National Library of Medicine (LM 06780, R01LM006780)
Topics
  • Health Sciences › Medicine › Health Informatics › Biomedical Text Mining and Ontologies
  • Physical Sciences › Computer Science › Artificial Intelligence › Advanced Text Analysis Techniques
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
Keywords
clinical trial characteristics, information extraction, randomized controlled trials, full-text journal articles, text classification, rule-based extraction, eligibility criteria, sample size, primary outcomes, evidence synthesis, EXACT
MeSH terms
Humans, Periodicals as Topic, Reproducibility of Results, Randomized Controlled Trials as Topic, Information Storage and Retrieval

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: ExaCT: automatic extraction of clinical trial characteristics from journal publications. https://app.cochat.ai/papers/10.1186/1472-6947-10-56
The paper
Kiritchenko, S., de Bruijn, B., Carini, S., Martin, J., & Sim, I. (2010). ExaCT: automatic extraction of clinical trial characteristics from journal publications. BMC Medical Informatics and Decision Making. https://doi.org/10.1186/1472-6947-10-56

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.