12 years on – Is the NLM medical text indexer still useful and relevant?
Open accessThe DOI matches its Crossref record; 16 of the 18 references it cites were found in a registry or an index.
Abstract
Background: Facing a growing workload and dwindling resources, the US National Library of Medicine (NLM) created the Indexing Initiative project in 1996. This cross-library team’s mission is to explore indexing methodologies for ensuring quality and currency of NLM document collections. The NLM Medical Text Indexer (MTI) is the main product of this project and has been providing automated indexing recommendations since 2002. After all of this time, the questions arise whether MTI is still useful and relevant. Methods: To answer the question about MTI usefulness, we track a wide variety of statistics related to how frequently MEDLINE indexers refer to MTI recommendations, how well MTI performs against human indexing, and how often MTI is used. To answer the question of MTI relevancy compared to other available tools, we have participated in the 2013 and 2014 BioASQ Challenges. The BioASQ Challenges have provided us with an unbiased comparison between the MTI system and other systems performing the same task. Results: Indexers have continually increased their use of MTI recommendations over the years from 15.75% of the articles they index in 2002 to 62.44% in 2014 showing that the indexers find MTI to be increasingly useful. The MTI performance statistics show significant improvement in Precision ( +0.2992 ) and F 1 ( +0.1997 ) with modest gains in Recall ( +0.0454 ) over the years. MTI consistency is comparable to the available indexer consistency studies. MTI performed well in both of the BioASQ Challenges ranking within the top tier teams. Conclusions: Based on our findings, yes, MTI is still relevant and useful, and needs to be improved and expanded. The BioASQ Challenge results have shown that we need to incorporate more machine learning into MTI while still retaining the indexing rules that have earned MTI the indexers’ trust over the years. We also need to expand MTI through the use of full text, when and where it is available, to provide coverage of indexing terms that are typically only found in the full text. The role of MTI at NLM is also expanding into new areas, further reinforcing the idea that MTI is increasingly useful and relevant.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures






Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Indeed, as of October 2018, PubMed itself has adopted a hybrid approach, where some articles are assigned MeSH terms automatically using their Medical Text Indexer (MTI) system [27], which uses a combination of machine learning and manually crafted rules to assign terms without human intervention [28].”
From Toward systematic review automation: a practical guide to using machine learning tools in research synthesis (2019) · cites this paper as [28]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
An overview of MetaMap: historical perspective and recent advances
Aronson, A. R., & Lang, F.-M. (2010) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.2009.002733 · Cited by 1,519
Self-citationTsatsaronis, G., Balikas, G., Malakasiotis, P., Partalas, I., Zschunke, M., Alvers, M. R., Weissenborn, D., Krithara, A., Petridis, S., Polychronopoulos, D., et al. (2015) · BMC Bioinformatics · doi:10.1186/s12859-015-0564-6 · Cited by 598 · cited 3×
“For an external evaluation, MTI participated in the “Large-scale online biomedical semantic indexing” task of the 2013 and 2014 BioASQ Challenges [23].”
BackgroundMethodInfluentialPubMed related articles: a probabilistic topic-based model for content similarity
Lin, J., & Wilbur, W. J. (2007) · BMC Bioinformatics · doi:10.1186/1471-2105-8-423 · Cited by 243 · cited 1×
“PubMed related citations [18] The related citations of a document are those documents in theMEDLINE/PubMed database that are themost similar to it.”
BackgroundIndexing consistency in MEDLINE.
Funk, M. E., & Reid, C. A. (1983) · Europe PMC (PubMed Central) · Cited by 180
TREC genomics special issue overview
Hersh, W., & Voorhees, E. (2009) · Information Retrieval · doi:10.1007/s10791-008-9076-6 · Cited by 180
The nlm medical text indexer system for indexing biomedical literature
Mork, J. G., Yepes, A. J., & Aronson, A. R. (2013) · Cited by 118
Self-citationRoberts, K. E., Simpson, M. S., Demner‐Fushman, D., Voorhees, E. M., & Hersh, W. (2015) · Information Retrieval · doi:10.1007/s10791-015-9259-x · Cited by 113
Self-citationBeyond synonymy: exploiting the UMLS semantics in mapping vocabularies.
Bodenreider, O., Nelson, S. J., Hole, W. T., & Chang, H. F. (1998) · Europe PMC (PubMed Central) · Cited by 113
Evaluation of the MEDLARS Demand Search Service
Brodman, E. (1969) · Bulletin of the Medical Library Association · Cited by 62
Evaluation of the MEDLARS demand search service
Lancaster, F. W. (1980) · Medical Entomology and Zoology · Cited by 49
Research works printed but not linked: 7
- Pubmed. https://www.ncbi.nlm.nih.gov/pubmed. Accessed 2 Dec 2016.
- Pubmed help. https://www.ncbi.nlm.nih.gov/books/nbk3827/#_pubmedhelp_filters_. Accessed 2 Dec 2016.
- Jimeno-Yepes A, Mork J, Demner-Fushman D, Aronson A. Automatic algorithm selection for mesh heading indexing based on meta-learning. International Symposium on Languages in Biology and Medicine. 2011. https://ii.nlm.nih.gov/Publications/Papers/Antonio_MTI_ISLBM_2011.pdf.
- Mork J, Demner-Fushman D, Schmidt S, Aronson A. Recent enhancements to the nlm medical text indexer. In: CLEF2014 Working Notes: Working Notes for CLEF 2014 Conference. Sheffield, UK, September 15–18, 2014;1180:1328–36. http://ceur-ws.org/Vol-1180/CLEF2014wn-QA-MorkEt2014.pdf.
- Tsoumakas G, Laliotis M, Markantonatos N, Vlahavas I. Large-scale semantic indexing of biomedical publications at bioasq. A Post-Conference Workshop of Conference and Labs of the Evaluation Forum 2013 (CLEF 2013) Valencia, Spain, September 27th, 2013;1094. http://ceur-ws.org/Vol-1094/bioasq2013_submission_6.pdf.
- Mao Y, Wei C, Lu Z. Ncbi at the 2014 bioasq challenge task: Large-scale biomedical semantic indexing and question answering. CLEF2014 Working Notes: Working Notes for CLEF 2014 Conference. Sheffield, UK, September 15–18. 2014;1180:1319–27. http://ceur-ws.org/Vol-1180/CLEF2014wn-QA-MaoEt2014.pdf.
- Liu K, Wu J, Peng S, Zhai C, Zhu S. The fudan-uiuc participation in the bioasq challenge task 2a: The antinomyra system. CLEF2014 Working Notes: Working Notes for CLEF 2014 Conference. Sheffield, UK, September 15–18, 2014;1180:1311–18. http://ceur-ws.org/Vol-1180/CLEF2014wn-QA-LiuEt2014.pdf.
Cited by
Marshall, I. J., & Wallace, B. C. (2019) · Systematic Reviews · doi:10.1186/s13643-019-1074-9 · Cited by 638
PubTator central: automated concept annotation for biomedical full text articles
Wei, C.-H., Allot, A., Leaman, R., & Lu, Z. (2019) · Nucleic Acids Research · doi:10.1093/nar/gkz389 · Cited by 400
Who do we invent for? Patents by women focus more on women’s health, but few women get to invent
Koning, R., Samila, S., & Ferguson, J.-P. (2021) · Science · doi:10.1126/science.aba6990 · Cited by 188
Bernstam, E. V., Shireman, P. K., Meric‐Bernstam, F., Zozus, M. N., Jiang, X., Brimhall, B. B., Windham, A. K., Schmidt, S., Visweswaran, S., Ye, Y., et al. (2021) · Clinical and Translational Science · doi:10.1111/cts.13175 · Cited by 67
You, R., Liu, Y., Mamitsuka, H., & Zhu, S. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btaa837 · Cited by 60
Metadata
- Authors
- Mork, J., Aronson, A., & Demner-Fushman, D.
- Affiliations
- United States National Library of Medicine (US)
- Corresponding author
- Mork, James
- ORCID
- Mork, James
- Venue
- Journal of Biomedical Semantics
- Published
- 2017-02-23
- Volume
- 8
- Issue
- 1
- Pages
- 8-8
- ISSN
- 2041-1480 (ISSN-L)
- PMC
- PMC5324252
- OpenAlex
- W2589937067
- Cited by
- 73
- References
- 18
- Citations by year
- 2026: 5 · 2025: 4 · 2024: 6 · 2023: 7 · 2022: 12 · 2021: 10 · 2020: 13 · 2019: 8 · 2018: 3 · 2017: 5
- FWCI
- 14.75
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- Journal of Biomedical Semantics (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Funders
- National Institutes of Health
- U.S. National Library of Medicine
- Topics
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Health Sciences › Medicine › Health Informatics › Biomedical Text Mining and Ontologies
- Physical Sciences › Computer Science › Information Systems › Library Science and Information Systems
- Keywords
- automatic indexing, MEDLINE, indexing consistency, full-text indexing, indexing rules, bibliographic indexing
- MeSH terms
- Abstracting and Indexing, Machine Learning, Humans, National Library of Medicine (U.S.), United States, Medical Subject Headings
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: 12 years on – Is the NLM medical text indexer still useful and relevant? https://app.cochat.ai/papers/10.1186/s13326-017-0113-5
- The paper
- Mork, J., Aronson, A., & Demner-Fushman, D. (2017). 12 years on – Is the NLM medical text indexer still useful and relevant? Journal of Biomedical Semantics. https://doi.org/10.1186/s13326-017-0113-5
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.