Selected articles from the BioCreative/OHNLP challenge 2018
Open accessAbstract
The wide adoption of electronic health records (EHRs) has led to an improvement in healthcare quality by electronically documenting a patient’s medical conditions, thoughts and actions among the care providers [ 1 ]. Those EHR data, with the vast majority being free-texts (e.g., clinical notes, discharge summaries, radiology reports, and pathology reports), have been utilized for primary and secondary purposes, such as documentation need in care process, clinical decision support, outcome improvement, biomedical research and epidemiologic monitoring of the nation’s health. The application of natural language processing (NLP) methods and resources to clinical and biomedical text has received growing attention over the past years, but progress has been limited by difficulties to access shared tools and resources, partially caused by patient privacy and data confidentiality constraints. Efforts to increase sharing and interoperability of the few existing resources are needed to facilitate the progress observed in the general NLP domain. Towards this goal, we organized the BioCreative/OHNLP Challenge 2018 workshop ( https://sites.google.com/view/ohnlp2018/home ) to promote community efforts on methodological advancements and data curation mechanisms in clinical NLP. The challenge consists of two independent clinical NLP tasks: 1) Family History Extraction; and 2) Clinical Semantic Textual Similarity. The top performing teams were invited to present their solutions during the BioCreative/OHNLP Challenge 2018 workshop in conjunction with the 9th ACM Conference on Bioinformatics, Computational Biology, and Health Informatics (ACM BCB) ( http://acm-bcb.org/2018/ ) on August 30th, 2018. This supplement collects the system descriptions of top-performing solutions of the tasks.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The 2018 BioCreative/OHNLP Challenge is the first shared task focusing on family history extraction.
“The 2018 BioCreative/OHNLP Challenge [33,34] is the first shared task focusing on FH extraction.”
From Extracting Family History of Patients From Clinical Narratives: Exploring an End-to-End Solution With Deep Learning Models (2020) · cites this paper as [33]
The Introduction and “Task 1: family history extraction” section identify family history extraction as a shared task in the 2018 BioCreative/OHNLP Challenge. Neither section says it was the first shared task focused on family history extraction, so the priority claim is not supported by our copy. (LLM, conf=0.84)
Experimental — accuracy not yet measuredThe OHNLP consortium has created multiple deidentified data sets for a couple of NLP tasks based on real clinical sentences.
“Leveraging the research in corpus analysis and deidentification, the Open Health Natural Language Processing (OHNLP) consortium has created multiple deidentified data sets for a couple of NLP tasks based on real clinical sentences [14-16].”
From Family History Extraction From Synthetic Clinical Narratives Using Natural Language Processing: Overview and Evaluation of a Challenge Data Set and Solutions for the 2019 National NLP Clinical Challenges (n2c2)/Open Health Natural Language Processing (OHNLP) Competition (2021) · cites this paper as [14]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Implementation of the Federal Health Information Technology Initiative
Blumenthal, D. (2011) · New England Journal of Medicine · doi:10.1056/nejmsr1112158 · Cited by 177 · see citing works
MedSTS: a resource for clinical semantic textual similarity
Wang, Y., Afzal, N., Fu, S., Wang, L., Shen, F., Rastegar-Mojarad, M., & Liu, H. (2020) · Language Resources and Evaluation · doi:10.1007/s10579-018-9431-1 · Cited by 124 · see citing works
Self-citationOverview of the BioCreative/OHNLP 2018 Family History Extraction Task
Liu, S., Rastegar-Mojarad, M., Wang, Y., Wang, L., Shen, F., Fu, S., & Liu, H. D. (2018) · doi:10.13140/rg.2.2.23326.79680 · Cited by 4 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 1
- [2]McCarthy JJ Mendelsohn BA Family history. In: precision medicine: a guide to genomics in clinical practice 2016 New York McGraw-Hill Education
Cited by
Sorted by citations
Clinical concept extraction: A methodology review
Fu, S., Chen, D. J., He, H., Liu, S., Moon, S., Peterson, K. J., Shen, F., Wang, L., Wang, Y., Wen, A., et al. (2020) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2020.103526 · Cited by 182 · see citing works
Self-citationYang, X., Zhang, H., He, X., Bian, J., & Wu, Y. (2020) · JMIR Medical Informatics · doi:10.2196/22982 · Cited by 29 · see citing works
Kades, K., Sellner, J., Koehler, G., Full, P. M., Lai, T. Y. E., Kleesiek, J., & Maier-Hein, K. H. (2020) · JMIR Medical Informatics · doi:10.2196/22795 · Cited by 24 · see citing works
Shen, F., Liu, S., Fu, S., Wang, Y., Henry, S., Uzuner, O., & Liu, H. (2021) · JMIR Medical Informatics · doi:10.2196/24008 · Cited by 21 · see citing works
Self-citationRybinski, M., Dai, X., Singh, S., Karimi, S., & Nguyen, A. N. (2021) · JMIR Medical Informatics · doi:10.2196/24020 · Cited by 18 · see citing works
Dai, H.-J., Lee, Y.-Q., Nekkantti, C., & Jonnagaddala, J. (2020) · JMIR Medical Informatics · doi:10.2196/21750 · Cited by 9 · see citing works
Zlabinger, M., Sabou, M., Hofstätter, S., & Hanbury, A. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.274 · Cited by 7 · see citing works
Clinical Natural Language Processing in Secondary Use of EHR for Research
Fu, S., Wen, A., & Liu, H. (2023) · Health informatics · doi:10.1007/978-3-031-27173-1_21 · Cited by 4 · see citing works
Self-citationFaramarzi, N. S., Dara, A., & Banerjee, R. (2022) · IEEE International Conference on Healthcare Informatics (ICHI) · doi:10.1109/ichi54592.2022.00023 · Cited by 2 · see citing works
Methodology for Design and Analysis of Machine Learning Competitions
Pavão, A. (2023) · theses.fr (ABES) · doi:10.70675/9234d2f0zebc5z4c1czaf3bzb0ad7a481e04
- Authors
- Liu, S., Wang, Y., & Liu, H.
- Affiliations
- Mayo Clinic (US)
- Corresponding author
- Liu, Sijia
- Venue
- BMC Medical Informatics and Decision Making
- Published
- 2019-12
- Volume
- 19
- Issue
- S10
- Pages
- 262-262
- ISSN
- 1472-6947 (ISSN-L)
- PMC
- PMC6933636
- OpenAlex
- W2998580448
- Cited by
- 10
- References
- 3
- Citations by year
- 2023: 2 · 2022: 1 · 2021: 1 · 2020: 6
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- BMC Medical Informatics and Decision Making (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- National Institute of General Medical Sciences
- U.S. National Library of Medicine (R01LM11934)
- National Center for Advancing Translational Sciences (U01TR02062)
- Topics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Biomedical Text Mining and Ontologies
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Keywords
- clinical natural language processing, electronic health records, clinical text mining, clinical notes, data interoperability
- MeSH terms
- Humans, Natural Language Processing, Information Storage and Retrieval, Computational Biology, Electronic Health Records
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
