The DOI matches its Crossref record; 39 of the 39 references it cites were found in a registry or an index.
CogStack - experiences of deploying integrated information retrieval and extraction services in a large National Health Service Foundation Trust hospital
Open accessPublished version of a preprint
Abstract
BACKGROUND: Traditional health information systems are generally devised to support clinical data collection at the point of care. However, as the significance of the modern information economy expands in scope and permeates the healthcare domain, there is an increasing urgency for healthcare organisations to offer information systems that address the expectations of clinicians, researchers and the business intelligence community alike. Amongst other emergent requirements, the principal unmet need might be defined as the 3R principle (right data, right place, right time) to address deficiencies in organisational data flow while retaining the strict information governance policies that apply within the UK National Health Service (NHS). Here, we describe our work on creating and deploying a low cost structured and unstructured information retrieval and extraction architecture within King's College Hospital, the management of governance concerns and the associated use cases and cost saving opportunities that such components present. RESULTS: To date, our CogStack architecture has processed over 300 million lines of clinical data, making it available for internal service improvement projects at King's College London. On generated data designed to simulate real world clinical text, our de-identification algorithm achieved up to 94% precision and up to 96% recall. CONCLUSION: We describe a toolkit which we feel is of huge value to the UK (and beyond) healthcare community. It is the only open source, easily deployable solution designed for the UK healthcare environment, in a landscape populated by expensive proprietary systems. Solutions such as these provide a crucial foundation for the genomic revolution in medicine.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“because about 80% of patient data are found in freetext format.1,2”
From Foresight-a generative pretrained transformer for modelling of patient timelines using electronic health records: a retrospective modelling study (2024) · cites this paper as [1]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, structured datasets are not always available and, even when they are, they can provide a narrow view of a patient's journey, because about 80% of patient data are found in freetext format.1,2”
From Foresight-a generative pretrained transformer for modelling of patient timelines using electronic health records: a retrospective modelling study (2024) · cites this paper as [1]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
MIMIC-III, a freely accessible critical care database
Johnson, A. E. W., Pollard, T. J., Shen, L., Lehman, L.-W. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., & Mark, R. G. (2016) · Scientific Data · doi:10.1038/sdata.2016.35 · Cited by 8,141
The unified medical language system (UMLS): integrating biomedical terminology
Bodenreider, O. (2004) · Nucleic Acids Research · doi:10.1093/nar/gkh061 · Cited by 4,356
An Overview of the Tesseract OCR Engine
Smith, R. W. (2007) · Proceedings of the International Conference on Document Analysis and Recognition · doi:10.1109/icdar.2007.4376991 · Cited by 2,457
Studies in Health Technology and Informatics
Aphinyanaphongs, Y., & Aliferis, C. (2007) · Cited by 1,247
A Simple Algorithm for Identifying Negated Findings and Diseases in Discharge Summaries
Chapman, W. W., Bridewell, W. B., Hanbury, P., Cooper, G. F., & Buchanan, B. G. (2001) · Journal of Biomedical Informatics · doi:10.1006/jbin.2001.1029 · Cited by 1,121
The Human Phenotype Ontology project: linking molecular biology and disease through phenotype data
Köhler, S., Doelken, S., Mungall, C. J., Bauer, S., Firth, H., Bailleul-Forestier, I., Black, G., Brown, D. L., Brudno, M., Campbell, J., et al. (2013) · Nucleic Acids Res. · doi:10.1093/nar/gkt1026 · Cited by 840
Perera, G., Broadbent, M. T. M., Callard, F. J., Chang, C., Downs, J. M., Dutta, R., Fernandes, A. C., Hayes, R. D., Henderson, M., Jackson, R., et al. (2016) · BMJ Open · doi:10.1136/bmjopen-2015-008721 · Cited by 492
Self-citationStewart, R., Soremekun, M., Perera, G., Broadbent, M., Callard, F., Denis, M., Hotopf, M., Thornicroft, G., & Lovestone, S. (2009) · BMC Psychiatry · doi:10.1186/1471-244x-9-51 · Cited by 445
Self-citationAutomated de-identification of free-text medical records
Neamatullah, I., Douglass, M. M., Lehman, L.-W. H., Reisner, A., Villarroel, M., Long, W. J., Szolovits, P., Moody, G. B., Mark, R. G., & Clifford, G. D. (2008) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-8-32 · Cited by 413
Secondary Use of EHR: Data Quality Issues and Informatics Opportunities.
(2010) · PubMed Central · Cited by 332
Stubbs, A., Kotfila, C., & Uzuner, Ö. (2015) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2015.06.007 · Cited by 243
Caulfield, M. J., Davies, J., Dennys, M., Elbahy, L., Fowler, T., Hill, S., Hubbard, T., Jostins, L., Maltby, N., Mahon-Pearson, J., et al. (2017) · Figshare · doi:10.6084/m9.figshare.4530893 · Cited by 223
Two biomedical sublanguages: a description based on the theories of Zellig Harris
Friedman, C., Kra, P., & Rzhetsky, A. (2002) · Journal of Biomedical Informatics · doi:10.1016/s1532-0464(03)00012-1 · Cited by 220
Jones, K. H., Ford, D. V., Jones, C., Dsilva, R., Thompson, S., Brooks, C. J., Heaven, M. L., Thayer, D. S., McNerney, C., & Lyons, R. A. (2014) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2014.01.003 · Cited by 218
Annotating longitudinal clinical narratives for de-identification: The 2014 i2b2/UTHealth corpus
Stubbs, A., & Uzuner, Ö. (2015) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2015.07.020 · Cited by 216
Fernandes, A. C., Cloete, D., Broadbent, M. T. M., Hayes, R. D., Chang, C., Jackson, R., Roberts, A., Tsang, J., Soncul, M., Liebscher, J., et al. (2013) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-13-71 · Cited by 194
Self-citationBiomedical data privacy: problems, perspectives, and recent advances
Malin, B., Emam, K. E., & O’Keefe, C. M. (2012) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2012-001509 · Cited by 159
A study of active learning methods for named entity recognition in clinical text
Chen, Y., Lasko, T. A., Mei, Q., Denny, J. C., & Xu, H. (2015) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2015.09.010 · Cited by 157
Towards comprehensive syntactic and semantic annotations of the clinical narrative
Albright, D., Lanfranchi, A., Fredriksen, A., Styler, W. F., Warner, C., Hwang, J. D., Choi, J. D., Dligach, D., Nielsen, R. D., Martin, J., et al. (2013) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2012-001317 · Cited by 145
Wu, S., Miller, T. A., Masanz, J. J., Coarr, M., Halgrim, S. R., Carrell, D., & Clark, C. (2014) · PLoS ONE · doi:10.1371/journal.pone.0112774 · Cited by 119
Evaluating Natural Language Processors in the Clinical Domain
Hripcsak, G. M., & Friedman, C. (1998) · Methods of Information in Medicine · doi:10.1055/s-0038-1634566 · Cited by 104
Automatic concept recognition using the Human Phenotype Ontology reference and test suite corpora
Groza, T., Köhler, S., Doelken, S. C., Collier, N., Oellrich, A., Smedley, D. P., Couto, F. M., Baynam, G. S., Zankl, A., & Robinson, P. N. (2015) · Database · doi:10.1093/database/bav005 · Cited by 71
Self-citationPseudonymization of patient identifiers for translational research
Aamot, H., Kohl, C. D., Richter, D., & Knaup, P. (2013) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-13-75 · Cited by 54
DʼAvolio, L., Nguyen, T. M., Goryachev, S. D., & Fiore, L. D. (2011) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000183 · Cited by 51
Callard, F. J., Broadbent, M. T. M., Denis, M., Hotopf, M. H., Soncul, M., Wykes, T., Lovestone, S., & Stewart, R. J. (2014) · BMJ Open · doi:10.1136/bmjopen-2014-005654 · Cited by 50
Self-citationMoen, H., Ginter, F., Marsi, E., Peltonen, L., Salakoski, T., & Salanterä, S. (2015) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-15-s2-s2 · Cited by 46
Ferraro, J. P., Daumé, H., DuVall, S. L., Chapman, W. W., Harkema, H., & Haug, P. J. (2013) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2012-001453 · Cited by 44
Mattmann, C. A., & Zitting, J. (2011) · Cited by 44
Fifty years' development and future perspectives of psychiatric register research
Munk‐Jørgensen, P., Okkels, N., Golberg, D., Ruggeri, M., & Thornicroft, G. (2014) · Acta Psychiatrica Scandinavica · doi:10.1111/acps.12281 · Cited by 39
Consequences of impaired data quality on information retrieval in electronic patient records
Mikkelsen, G., & Aasly, J. O. (2004) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2004.11.001 · Cited by 29
McEwan, R., Melton, G. B., Knoll, B. C., Wang, Y., Hultman, G. M., Dale, J. L., Meyer, T., & Pakhomov, S. (2016) · PubMed Central · Cited by 27
Standardization of Health Informatics – Results and Challenges
Klein, G. O. (2002) · Methods of Information in Medicine · doi:10.1055/s-0038-1634486 · Cited by 24
Domain adaptation for semantic role labeling of clinical text
Zhang, Y., Tang, B., Jiang, M., Wang, J., & Xu, H. (2015) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocu048 · Cited by 20
Lessons learned from the implementation of clinical messaging systems.
Barnes, M. (2007) · PubMed Central · Cited by 14
Applying natural language processing toolkits to electronic health records - an experience report.
Barrett, N. E., & Weber-Jahnke, J. H. (2009) · PubMed · Cited by 13
An emerging standard for health communications: the HL7 standard.
Dw, S. (1987) · PubMed · Cited by 13
Automatic annotation of medical records.
Antolík, J. (2005) · PubMed · Cited by 8
Simplifying HL7 Version 3 messages
Worden, R., & Scott, P. (2011) · Studies in health technology and informatics · doi:10.3233/978-1-60750-806-9-709 · Cited by 7
cognition: DB binary-to-text converter and pseudonymiser for clinical research
Kartoglu, I. E. (2015) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.594228 · Cited by 1
Self-citation
Research works printed but not linked: 1
- Jackson R, Kartoglu I. A Open Pipeline for Masking Patient Identifiers in Electronic Health Records, The Farr Institute International Conference 2015. 2015.
Cited by
Sorted by citations
Holmes, E. A., O'Connor, R. C., Perry, V. H., Tracey, I., Wessely, S., Arseneault, L., Ballard, C., Christensen, H., Cohen Silver, R., Everall, I., et al. (2020) · The Lancet Psychiatry · doi:10.1016/s2215-0366(20)30168-1 · Cited by 5,715
Natural language processing in medicine: A review
Locke, S., Bashall, A., Al-Adely, S., Moore, J., Wilson, A. J., & Kitchen, G. B. (2021) · Trends in Anaesthesia and Critical Care · doi:10.1016/j.tacc.2021.02.007 · Cited by 270
Bean, D. M., Kraljević, Ž., Searle, T., Bendayan, R., O’Gallagher, K., Pickles, A. R., Folarin, A., Roguski, Ł., Noor, K., Shek, A., et al. (2020) · European Journal of Heart Failure · doi:10.1002/ejhf.1924 · Cited by 231
Self-citationUK phenomics platform for developing and validating electronic health record phenotypes: CALIBER
Denaxas, S., González-Izquierdo, A., Direk, K., Fitzpatrick, N. K., Fatemifar, G., Banerjee, A., Dobson, R., Howe, L. J., Kuan, V., Lumbers, R. T., et al. (2019) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocz105 · Cited by 219
Self-citationKraljević, Ž., Searle, T., Shek, A., Roguski, Ł., Noor, K., Bean, D. M., Mascio, A., Zhu, L., Folarin, A., Roberts, A., et al. (2021) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2021.102083 · Cited by 255
Self-citation
Metadata
- Authors
- Jackson, R., Kartoglu, I., Stringer, C., Gorrell, G., Roberts, A., Song, X., Wu, H., Agrawal, A., Lui, K., Groza, T., Lewsley, D., Northwood, D., Folarin, A., Stewart, R., & Dobson, R.
- Affiliations
- King's College London (GB)
- InterDigital (United Kingdom) (GB)
- Farr Institute (GB)
- King's College Hospital (GB)
- Garvan Institute of Medical Research (AU)
- Corresponding author
- Jackson, Richard
- ORCID
- Venue
- BMC Medical Informatics and Decision Making
- Published
- 2018-06-25
- Volume
- 18
- Issue
- 1
- Pages
- 47-47
- ISSN
- 1472-6947 (ISSN-L)
- PMC
- PMC6020175
- OpenAlex
- W2949308662
- Cited by
- 134
- References
- 39
- Citations by year
- 2026: 11 · 2025: 22 · 2024: 12 · 2023: 18 · 2022: 13 · 2021: 28 · 2020: 15 · 2019: 13 · 2018: 2
- FWCI
- 18.29
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- BMC Medical Informatics and Decision Making (publishedVersion, cc-by)
- bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, other-oa)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- White Rose Research Online (University of Leeds, The University of Sheffield, University of York) (acceptedVersion)
- Edinburgh Research Explorer (University of Edinburgh) (submittedVersion, public-domain)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- UCL Discovery (University College London) (submittedVersion)
- Research Portal (King's College London) (submittedVersion, cc-by)
- Funders
- Wellcome Trust (MR/K006584/1)
- UK Infrastructure for Large-scale Clinical Genomics Research (MC_PC_14089)
- National Institute for Health Research (NIHR) Biomedical Research Centre at South London and Maudsley NHS Foundation Trust
- National Institute for Health Research (NIHR) Biomedical Research Centre at University College London Hospital
- NHS England Enablement
- Horizon 2020 Framework Programme (644753)
- Topics
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Health Sciences › Medicine › Health Informatics › Biomedical Text Mining and Ontologies
- Health Sciences › Health Professions › Health Information Management › Medical Coding and Health Information
- Keywords
- medical information retrieval, clinical text extraction, health information systems, National Health Service, information governance, de-identification, medical data processing, open source software, genomic medicine, clinical text mining
- MeSH terms
- United Kingdom, Hospitals, Humans, National Health Programs, Natural Language Processing, Information Storage and Retrieval, Electronic Health Records
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
