The DOI matches its Crossref record; 14 of the 14 references it cites were found in a registry or an index.
Deploying an interactive machine learning system in an evidence-based practice center: abstrackr
Abstract
Medical researchers looking for evidence pertinent to a specific clinical question must navigate an increasingly voluminous corpus of published literature. This data deluge has motivated the development of machine learning and data mining technologies to facilitate efficient biomedical research. Despite the obvious labor-saving potential of these technologies and the concomitant academic interest therein, however, adoption of machine learning techniques by medical researchers has been relatively sluggish. One explanation for this is that while many machine learning methods have been proposed and retrospectively evaluated, they are rarely (if ever) actually made accessible to the practitioners whom they would benefit. In this work, we describe the ongoing development of an end-to-end interactive machine learning system at the Tufts Evidence-based Practice Center. More specifically, we have developed abstrackr, an online tool for the task of citation screening for systematic reviews. This tool provides an interface to our machine learning methods. The main aim of this work is to provide a case study in deploying cutting-edge machine learning methods that will actually be used by experts in a clinical research setting.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Currently available systems include Abstrackr [30], SWIFT-Review,9 EPPI reviewer [31], and RobotAnalyst [32] (see Table 1).”
From Toward systematic review automation: a practical guide to using machine learning tools in research synthesis (2019) · cites this paper as [30]
In the citing paper
“Several statistical stopping metrics have been proposed in the literature [15, 21, 22, 35, 38, 49, 50, 53, 55]).”
From The SAFE procedure: a practical stopping heuristic for active learning-based screening in systematic reviews and meta-analyses (2024) · cites this paper as [49]
In the citing paper
“Priority screening via active learning has been successfully implemented in various software tools such as Abstrackr [49], ASReview [45], Colandr [12], EPPI-Reviewer [41], FASTREAD [54], Rayyan [28], RobotAnalyst [32], Research Screener [10], DistillerSR [20], and robotreviewer [25].”
From The SAFE procedure: a practical stopping heuristic for active learning-based screening in systematic reviews and meta-analyses (2024) · cites this paper as [49]
In the citing paper
“These include computing an inflection point [16, 21, 35, 38, 49, 50], estimating recall for the sequential screening of a ranked list of references [22], or computing the lengths of consecutive spans of excluded documents that occur between each relevant document during screening [38, 53, 55].”
From The SAFE procedure: a practical stopping heuristic for active learning-based screening in systematic reviews and meta-analyses (2024) · cites this paper as [49]
In the citing paper
“ready-to-use software tools implementing such models (Abstrackr [22], Colandr [23], FASTREAD [11], Rayyan [24], and RobotAnalyst [25]) currently use the same classification technique to predict relevance of publications, namely support vector machines (SVM).”
10 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
(2009) · Minds at UW (University of Wisconsin) · Cited by 4,791
Sentiment analysis of blogs by combining lexical knowledge with text classification
Melville, P., Gryc, W., & Lawrence, R. D. (2009) · Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1557019.1557156 · Cited by 503
Support Vector Machine Active Learning with Application sto Text Classification
Tong, S., & Koller, D. (2000) · International Conference on Machine Learning · Cited by 369
Semi-automated screening of biomedical citations for systematic reviews
Wallace, B. C., Trikalinos, T. A., Lau, J., Brodley, C., & Schmid, C. H. (2010) · BMC Bioinformatics · doi:10.1186/1471-2105-11-55 · Cited by 351
Self-citationFrontiers of biomedical text mining: current progress
Zweigenbaum, P., Demner-Fushman, D., Yu, H., & Cohen, K. B. (2007) · Briefings in Bioinformatics · doi:10.1093/bib/bbm045 · Cited by 282
Proactive learning: cost-sensitive active learning with multiple imperfect oracles
Donmez, P., & Carbonell, J. G. (2008) · Proceedings of the 17th ACM conference on Information and knowledge management · doi:10.1145/1458082.1458165 · Cited by 228
Settles, B. (2011) · Cited by 218
Inactive learning? difficulties employing active learning in practice
Attenberg, J., & Provost, F. J. (2011) · ACM SIGKDD Explorations Newsletter · doi:10.1145/1964897.1964906 · Cited by 107
Who Should Label What? Instance Allocation in Multiple Expert Active Learning
Wallace, B., Small, K., Brodley, C. E., & Trikalinos, T. A. (2011) · SIAM International Conference on Data Mining · doi:10.1137/1.9781611972818.16 · Cited by 75
Self-citation
Cited by
An open source machine learning framework for efficient and transparent systematic reviews
van de Schoot, R., de Bruin, J., Schram, R., Zahedi, P., de Boer, J., Weijdema, F., Kramer, B., Huijts, M., Hoogerwerf, M., Ferdinands, G., et al. (2021) · Nature Machine Intelligence · doi:10.1038/s42256-020-00287-7 · Cited by 1,157
Human-in-the-loop machine learning: a state of the art
Mosqueira-Rey, E., Hernández-Pereira, E., Alonso-Ríos, D., Bobes-Bascarán, J., & Fernández-Leal, Á. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-022-10246-w · Cited by 1,025
What factors contribute to the acceptance of artificial intelligence? A systematic review
Kelly, S., Kaye, S.-A., & Oviedo-Trespalacios, O. (2023) · Telematics and Informatics · doi:10.1016/j.tele.2022.101925 · Cited by 1,008
Mapping the effectiveness of nature‐based solutions for climate change adaptation
Chausson, A., Turner, B., Seddon, D., Chabaneix, N., Girardin, C. A. J., Kapos, V., Key, I., Roe, D., Smith, A., Woroniecki, S., et al. (2020) · Global Change Biology · doi:10.1111/gcb.15310 · Cited by 707
O’Mara-Eves, A., Thomas, J., McNaught, J., Miwa, M., & Ananiadou, S. (2015) · Systematic Reviews · doi:10.1186/2046-4053-4-5 · Cited by 681
Metadata
- Authors
- Wallace, B. C., Small, K., Brodley, C. E., Lau, J., & Trikalinos, T. A.
- Affiliations
- Tufts University (US)
- Tufts Medical Center (US)
- Venue
- Proceedings of the 2nd ACM SIGHIT International Health Informatics Symposium
- Publisher
- ACM
- Published
- 2012-01-28
- Pages
- 819-824
- OpenAlex
- W2100053037
- Cited by
- 532
- References
- 14
- Citations by year
- 2026: 21 · 2025: 51 · 2024: 61 · 2023: 57 · 2022: 51 · 2021: 75 · 2020: 82 · 2019: 40 · 2018: 35 · 2017: 17 · 2016: 15 · 2015: 14 · 2014: 10 · 2013: 1 · 2012: 2
- FWCI
- 25.05
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- International Health Informatics Symposium (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Health Sciences › Medicine › Health Informatics › Biomedical Text Mining and Ontologies
- Keywords
- interactive machine learning, evidence-based practice, systematic reviews, citation screening, clinical research, literature searching
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Deploying an interactive machine learning system in an evidence-based practice center: abstrackr. https://app.cochat.ai/papers/10.1145/2110363.2110464
- The paper
- Wallace, B. C., Small, K., Brodley, C. E., Lau, J., & Trikalinos, T. A. (2012). Deploying an interactive machine learning system in an evidence-based practice center: abstrackr. Proceedings of the 2nd ACM SIGHIT International Health Informatics Symposium. https://doi.org/10.1145/2110363.2110464
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.