ExaCT: automatic extraction of clinical trial characteristics from journal publications
Open accessThe DOI matches its Crossref record; 39 of the 39 references it cites were found in a registry or an index.
Abstract
BACKGROUND: Clinical trials are one of the most important sources of evidence for guiding evidence-based practice and the design of new trials. However, most of this information is available only in free text - e.g., in journal publications - which is labour intensive to process for systematic reviews, meta-analyses, and other evidence synthesis studies. This paper presents an automatic information extraction system, called ExaCT, that assists users with locating and extracting key trial characteristics (e.g., eligibility criteria, sample size, drug dosage, primary outcomes) from full-text journal articles reporting on randomized controlled trials (RCTs). METHODS: ExaCT consists of two parts: an information extraction (IE) engine that searches the article for text fragments that best describe the trial characteristics, and a web browser-based user interface that allows human reviewers to assess and modify the suggested selections. The IE engine uses a statistical text classifier to locate those sentences that have the highest probability of describing a trial characteristic. Then, the IE engine's second stage applies simple rules to these sentences to extract text fragments containing the target answer. The same approach is used for all 21 trial characteristics selected for this study. RESULTS: We evaluated ExaCT using 50 previously unseen articles describing RCTs. The text classifier (first stage) was able to recover 88% of relevant sentences among its top five candidates (top5 recall) with the topmost candidate being relevant in 80% of cases (top1 precision). Precision and recall of the extraction rules (second stage) were 93% and 91%, respectively. Together, the two stages of the extraction engine were able to provide (partially) correct solutions in 992 out of 1050 test tasks (94%), with a majority of these (696) representing fully correct and complete answers. CONCLUSIONS: Our experiments confirmed the applicability and efficacy of ExaCT. Furthermore, they demonstrated that combining a statistical method with 'weak' extraction rules can identify a variety of study characteristics. The system is flexible and can be extended to handle other characteristics and document types (e.g., study protocols).
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures




Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“In some instances, however, this evaluation was qualitative where the models were applied to large unlabelled, real-life datasets. 46,58,69,48,95,101,102,164”
From Data extraction methods for systematic review (semi)automation: Update of a living systematic review (2025) · cites this paper as [46]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe cited source doesn’t support the claim as written.
In the citing paper
“In some instances, however, this evaluation was qualitative where the models were applied to large unlabelled, real-life datasets. 46 , 58 , 69 , 48 , 95 , 101 , 102 , 164”
From Data extraction methods for systematic review (semi)automation: a living systematic review (2021) · cites this paper as [46]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Commonly, the relevant tools employ classifiers to identify articles possessing specific characteristics (Marshall et al. 2018) or implement named-entity recognition for extracting specific entities or concepts (Kiritchenko et al. 2010)”
From Artificial intelligence for literature reviews: opportunities and challenges (2024) · cites this paper as Kiritchenko et al. 2010
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Length of input text 59 and position of a sentence within a paragraph or abstract, e.g. up to 10% lower classification scores for certain sentence combinations in unstructured abstracts, were shown in several publications. 46,66,102”
From Data extraction methods for systematic review (semi)automation: Update of a living systematic review (2025) · cites this paper as [46]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“There is considerable variation in how an entity is reported, for example between intervention types (drugs, therapies, routes of application) 56 or in outcome measures.”
From Data extraction methods for systematic review (semi)automation: Update of a living systematic review (2025) · cites this paper as [46]
Experimental — accuracy not yet measured
17 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Moher, D., Schulz, K. F., & Altman, D. G. (2001) · The Lancet · doi:10.1016/s0140-6736(00)04337-3 · Cited by 5,373
The Revised CONSORT Statement for Reporting Randomized Trials: Explanation and Elaboration
Altman, D. G., Schulz, K. F., Moher, D., Egger, M., Davidoff, F., Elbourne, D., Gøtzsche, P. C., Lang, T., & for the CONSORT Group (2001) · Annals of Internal Medicine · doi:10.7326/0003-4819-134-8-200104170-00012 · Cited by 3,835
Users' guides to the medical literature
Guyatt, G. H. (1993) · JAMA · doi:10.1001/jama.270.17.2096 · Cited by 2,970
Moher, D., Schulz, K. F., & Altman, D. G. (2001) · BMC Medical Research Methodology · doi:10.1186/1471-2288-1-2 · Cited by 2,755
Savova, G. K., Kipper-Schuler, K. C., Hurdle, J. F., & Meystre, S. M. (2008) · Yearbook of Medical Informatics · doi:10.1055/s-0038-1638592 · Cited by 895
A survey of current work in biomedical text mining
Cohen, A. M. (2005) · Briefings in Bioinformatics · doi:10.1093/bib/6.1.57 · Cited by 765
Guyatt, G. H., Haynes, R. B., Jaeschke, R. Z., Cook, D. J., Green, L., Naylor, C. D., Wilson, M. C., Richardson, W. S., & for the Evidence-Based Medicine Working Group (2000) · JAMA · doi:10.1001/jama.284.10.1290 · Cited by 748
Clinical decision support systems for the practice of evidence-based medicine
Sim, I., Gorman, P., Greenes, R. A., Haynes, R. B., Kaplan, B., Lehmann, H., & Tang, P. C. (2001) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.2001.0080527 · Cited by 732
Self-citationInformation retrieval: the role of equipment and an assessment of the technology
Berul, L. H. (2024) · doi:10.17918/00005341 · Cited by 289
Chen, E. S., Hripcsak, G., Xu, H., Markatou, M., & Friedman, C. (2008) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m2401 · Cited by 210
Research works printed but not linked: 5
- Science Commons. --- Either ISSN or Journal title must be supplied.. [ http://sciencecommons.org/ ]
- The Human Study Database. --- Either ISSN or Journal title must be supplied.. [ http://hsdbwiki.org ]
- Demner-Fushman D, Lin J: Knowledge extraction for clinical question answering: preliminary results. Proceedings of the AAAI Workshop on Question Answering in Restricted Domains: 10 July 2005; Pittsburgh, PA. Edited by: Molla D, Vicedo JL. 2005, AAAI Press, 1-9.
- Hansen MJ, Rasmussen NO, Chung G: Extracting number of trial participants from abstracts of randomized controlled trials. Proceedings of Tromsø Telemedicine and eHealth Conference: 9-11 June 2008; Tromsø, Norway. 2008
- Van Rijsbergen CJ: Information Retrieval. 1979, London: Buttersworth, 2
Cited by
Reducing waste from incomplete or unusable reports of biomedical research
Glasziou, P., Altman, D. G., Bossuyt, P., Boutron, I., Clarke, M., Julious, S., Michie, S., Moher, D., & Wager, E. (2014) · The Lancet · doi:10.1016/s0140-6736(13)62228-x · Cited by 1,319
O’Mara-Eves, A., Thomas, J., McNaught, J., Miwa, M., & Ananiadou, S. (2015) · Systematic Reviews · doi:10.1186/2046-4053-4-5 · Cited by 681
“The use of automation to assist in study selection is possibly the most advanced of all the areas where automation in systematic reviews is being developed; but others range from writing sections of the report, formulating the review question and automated data extraction and quality assessment [90-93].”
BackgroundMarshall, I. J., & Wallace, B. C. (2019) · Systematic Reviews · doi:10.1186/s13643-019-1074-9 · Cited by 638
Systematic review automation technologies
Tsafnat, G., Glasziou, P., Choong, M. K., Dunn, A., Galgani, F., & Coiera, E. (2014) · Systematic Reviews · doi:10.1186/2046-4053-3-74 · Cited by 472
“Often, information highlighting algorithms focus on PICO elements reported in the trial [95-97] but may also focus on the size of the trial [97,98], and its randomization procedure [95,97].”
BackgroundMethodO’Mara-Eves, A., Thomas, J., McNaught, J., Miwa, M., & Ananiadou, S. (2015) · Systematic Reviews · doi:10.1186/s13643-015-0031-5 · Cited by 353
Metadata
- Authors
- Kiritchenko, S., de Bruijn, B., Carini, S., Martin, J., & Sim, I.
- Affiliations
- National Research Council Canada (CA)
- University of California, San Francisco (US)
- Corresponding author
- Kiritchenko, Svetlana
- Venue
- BMC Medical Informatics and Decision Making
- Published
- 2010-09-28
- Volume
- 10
- Issue
- 1
- Pages
- 56-56
- ISSN
- 1472-6947 (ISSN-L)
- PMC
- PMC2954855
- OpenAlex
- W2020959315
- Cited by
- 188
- References
- 39
- Citations by year
- 2026: 8 · 2025: 17 · 2024: 20 · 2023: 16 · 2022: 9 · 2021: 25 · 2020: 16 · 2019: 13 · 2018: 8 · 2017: 8 · 2016: 11 · 2015: 11 · 2014: 11 · 2013: 8 · 2012: 5
- FWCI
- 1.96
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/2.0 (tdm)
- Locations
- BMC Medical Informatics and Decision Making (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- NPARC (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- eScholarship (California Digital Library) (submittedVersion)
- Funders
- U.S. National Library of Medicine (LM 06780, R01LM006780)
- Topics
- Health Sciences › Medicine › Health Informatics › Biomedical Text Mining and Ontologies
- Physical Sciences › Computer Science › Artificial Intelligence › Advanced Text Analysis Techniques
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Keywords
- clinical trial characteristics, information extraction, randomized controlled trials, full-text journal articles, text classification, rule-based extraction, eligibility criteria, sample size, primary outcomes, evidence synthesis, EXACT
- MeSH terms
- Humans, Periodicals as Topic, Reproducibility of Results, Randomized Controlled Trials as Topic, Information Storage and Retrieval
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: ExaCT: automatic extraction of clinical trial characteristics from journal publications. https://app.cochat.ai/papers/10.1186/1472-6947-10-56
- The paper
- Kiritchenko, S., de Bruijn, B., Carini, S., Martin, J., & Sim, I. (2010). ExaCT: automatic extraction of clinical trial characteristics from journal publications. BMC Medical Informatics and Decision Making. https://doi.org/10.1186/1472-6947-10-56
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.