The DOI matches its Crossref record; 31 of the 31 references it cites were found in a registry or an index.
Overcoming the Challenges of Unstructured Data in Multisite, Electronic Medical Record-based Abstraction
Open accessAbstract
Background: Unstructured data encountered during retrospective electronic medical record (EMR) abstraction has routinely been identified as challenging to reliably abstract, as these data are often recorded as free text, without limitations to format or structure. There is increased interest in reliably abstracting this type of data given its prominent role in care coordination and communication, yet limited methodological guidance exists. Objectives: As standard abstraction approaches resulted in substandard data reliability for unstructured data elements collected as part of a multisite, retrospective EMR study of hospital discharge communication quality, our goal was to develop, apply and examine the utility of a phase-based approach to reliably abstract unstructured data. This approach is examined using the specific example of discharge communication for warfarin management. Research Design: We adopted a “fit-for-use” framework to guide the development and evaluation of abstraction methods using a 4-step, phase-based approach including (1) team building; (2) identification of challenges; (3) adaptation of abstraction methods; and (4) systematic data quality monitoring. Measures: Unstructured data elements were the focus of this study, including elements communicating steps in warfarin management (eg, warfarin initiation) and medical follow-up (eg, timeframe for follow-up). Results: After implementation of the phase-based approach, interrater reliability for all unstructured data elements demonstrated κ’s of ≥0.89—an average increase of +0.25 for each unstructured data element. Conclusions: As compared with standard abstraction methodologies, this phase-based approach was more time intensive, but did markedly increase abstraction reliability for unstructured data elements within multisite EMR documentation.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Abstraction followed standardized procedures using manuals and tools, with training and guidance based on prior work.
“Abstraction procedures were performed in a standardized manner using abstraction manuals and tools with training and guidance for medical record review based on prior work.40, 41”
From Discharge Communication of Dementia-Related Neuropsychiatric Symptoms and Care Management Strategies during Hospital to Skilled Nursing Facility Transitions (2021) · cites this paper as [41]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Qualitative Research: Consensus methods for medical and health services research
Jones, J., & Hunter, D. (1995) · BMJ · doi:10.1136/bmj.311.7001.376 · Cited by 3,471
High agreement but low Kappa: I. the problems of two paradoxes
Feinstein, A. R., & Cicchetti, D. V. (1990) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(90)90158-l · Cited by 3,034
Computing inter‐rater reliability and its variance in the presence of high agreement
Gwet, K. L. (2008) · British Journal of Mathematical and Statistical Psychology · doi:10.1348/000711006x126600 · Cited by 2,167
High agreement but low kappa: II. Resolving the paradoxes
Cicchetti, D. V., & Feinstein, A. R. (1990) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(90)90159-m · Cited by 1,838
Coefficient Kappa: Some Uses, Misuses, and Alternatives
Brennan, R. L., & Prediger, D. J. (1981) · Educational and Psychological Measurement · doi:10.1177/001316448104100307 · Cited by 1,676
Estimating Hospital Deaths Due to Medical Errors
Hayward, R. A., & Hofer, T. P. (2001) · JAMA · doi:10.1001/jama.286.4.415 · Cited by 586
Automated Encoding of Clinical Documents Based on Natural Language Processing
Friedman, C., Shagina, L., Lussier, Y. A., & Hripcsak, G. M. (2004) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1552 · Cited by 552
Kahn, M. G., Raebel, M. A., Glanz, J. M., Riedlinger, K., & Steiner, J. F. (2012) · Medical Care · doi:10.1097/mlr.0b013e318257dd67 · Cited by 260
Continuity of care and patient outcomes after hospital discharge
van Walraven, C., Mamdani, M., Fang, J., & Austin, P. C. (2004) · Journal of General Internal Medicine · doi:10.1111/j.1525-1497.2004.30082.x · Cited by 234
King, B. J., Gilmore‐Bykovskyi, A. L., Roiland, R. A., Polnaszek, B. E., Bowers, B. J., & Kind, A. J. H. (2013) · Journal of the American Geriatrics Society · doi:10.1111/jgs.12328 · Cited by 210
Self-citationThe Art and Science of Chart Review
Allison, J. J., Wall, T. C., Spettell, C. M., Calhoun, J., Fargason, C. A., Kobylinski, R. W., Farmer, R. M., & Kiefe, C. I. (2000) · The Joint Commission Journal on Quality Improvement · doi:10.1016/s1070-3241(00)26009-4 · Cited by 183
Natural language processing and its future in medicine
Friedman, C., & Hripcsak, G. M. (1999) · Academic Medicine · doi:10.1097/00001888-199908000-00012 · Cited by 179
Variation in Electronic Health Record Adoption and Readiness for Meaningful Use: 2008–2011
Patel, V., Jamoom, E. W., Hsiao, C., Furukawa, M. F., & Buntin, M. J. B. (2013) · Journal of General Internal Medicine · doi:10.1007/s11606-012-2324-x · Cited by 101
Howard, J., Clark, E. C., Friedman, A., Crosson, J. C., Pellerano, M. B., Crabtree, B. F., Karsh, B., Jaén, C. R., Bell, D. S., & Cohen, D. J. (2012) · Journal of General Internal Medicine · doi:10.1007/s11606-012-2192-4 · Cited by 94
Dictated versus database-generated discharge summaries: a randomized clinical trial.
van Walraven, C., Laupacis, A., Seth, R., & Wells, G. A. (1999) · PubMed Central · Cited by 92
Methods to Achieve High Interrater Reliability in Data Collection From Primary Care Medical Records
Liddy, C. E., Wiens, M., & Hogg, W. E. (2011) · The Annals of Family Medicine · doi:10.1370/afm.1195 · Cited by 85
Voorham, J., Denig, P., & Groningen Initiative to Analyse Type 2 Diabetes Treatment (GIANTT) group (2007) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m2128 · Cited by 81
Training, Quality Assurance, and Assessment of Medical Record Abstraction in a Multisite Study
Reisch, L. M. (2003) · American Journal of Epidemiology · doi:10.1093/aje/kwg016 · Cited by 79
Pan, L., Fergusson, D. A., Schweitzer, I., & Hébert, P. C. (2005) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2005.02.004 · Cited by 70
Meeting Meaningful Use Criteria and Managing Patient Populations
DesRoches, C. M., Audet, A., Painter, M. W., & Donelan, K. (2013) · Annals of Internal Medicine · doi:10.7326/0003-4819-158-11-201306040-00003 · Cited by 69
Stein, H. D., Nadkarni, P., Erdos, J., & Miller, P. L. (2000) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.2000.0070042 · Cited by 65
Standardized or narrative discharge summaries. Which do family physicians prefer?
van Walraven, C., Duke, S. M., Weinberg, A. L., & Wells, P. S. (1998) · PubMed Central · Cited by 63
Designing medical record abstraction forms
Banks, N. J. (1998) · International Journal for Quality in Health Care · doi:10.1093/intqhc/10.2.163 · Cited by 61
Electronic Medical Records And Health Care Transformation
Walker, J. M. (2005) · Health Affairs · doi:10.1377/hlthaff.24.5.1118 · Cited by 53
Omission of Dysphagia Therapies in Hospital Discharge Communications
Kind, A., Anderson, P. A., Hind, J. A., Robbins, J., & Smith, M. A. (2010) · Dysphagia · doi:10.1007/s00455-009-9266-4 · Cited by 46
Self-citationKind, A., Thorpe, C. T., Sattin, J. A., Walz, S. E., & Smith, M. A. (2011) · Journal of General Internal Medicine · doi:10.1007/s11606-011-1860-0 · Cited by 39
Self-citationEngel, L., Henderson, C., Salerno, J., & Colantonio, A. (2009) · Evaluation & the Health Professions · doi:10.1177/0163278709338561 · Cited by 38
Pragmatic strategies that enhance the reliability of data abstracted from medical records
Eder, C., Fullerton, J. T., Benroth, R., et al. (2005) · Applied Nursing Research · doi:10.1016/j.apnr.2004.04.005 · Cited by 36
Mishuris, R. G., & Linder, J. A. (2012) · Journal of General Internal Medicine · doi:10.1007/s11606-012-2304-1 · Cited by 22
Kung, H.-C., Hanzlick, R. L., & Spitler, J. F. (2001) · Journal of Forensic Sciences · doi:10.1520/jfs15110j · Cited by 21
Data abstraction: designing the tools, recruiting and training the data abstractors.
Simmons, B., Bennett, F., Nelson, A. L., et al. (2002) · PubMed · Cited by 5
Flagged first, then by citations
Research works printed but not linked: 1
- Engel, 2008, Eleven steps to improving data collection: guidelines for a retrospective medical record review, Occup Ther Now, 10, 17
Cited by
Sorted by citations
Prevalence of Missing Data in the National Cancer Database and Association with Overall Survival
Yang, D. X., Khera, R., Miccio, J. A., Jairam, V., Chang, E., Yu, J. B., Park, H. S., Krumholz, H. M., & Aneja, S. (2021) · JAMA Network Open · doi:10.1001/jamanetworkopen.2021.1793 · Cited by 135
Cho, H., Nguyen, O. T., Weaver, M., Pruitt, J., Marcelle, C., Salloum, R. G., & Keenan, G. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae239 · Cited by 55
Gilmore-Bykovskyi, A. L., Block, L. M., Walljasper, L., Hill, N., Gleason, C., & Shah, M. N. (2018) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocy070 · Cited by 21
Self-citationGilmore-Bykovskyi, A. L., Hovanes, M., Mirr, J., & Block, L. (2021) · Journal of Geriatric Psychiatry and Neurology · doi:10.1177/0891988720944245 · Cited by 14
“Abstraction procedures were performed in a standardized manner using abstraction manuals and tools with training and guidance for medical record review based on prior work.(40,41) Guidance on identifying NPS within clinical documentation contained within abstraction manuals and tools was informed by the Neuropsychiatr…”
MethodSelf-citationGilmore-Bykovskyi, A. L., Block, L., Hovanes, M., Mirr, J., & Kolanowski, A. (2019) · Research in Gerontological Nursing · doi:10.3928/19404921-20190122-01 · Cited by 11
Self-citation
Metadata
- Authors
- Polnaszek, B., Gilmore-Bykovskyi, A., Hovanes, M., Roiland, R., Ferguson, P., Brown, R., & Kind, A. J. H.
- Affiliations
- United States Department of Veterans Affairs (US)
- Department of Medicine, Geriatrics Division, University of Wisconsin School of Medicine and Public Health
- University of Wisconsin–Madison (US)
- Corresponding author
- Polnaszek, Brock
- Hovanes, Melissa
- Kind, Amy J.H.
- Venue
- Medical Care
- Published
- 2016-10
- Volume
- 54
- Issue
- 10
- Pages
- e65-e72
- ISSN
- 0025-7079 (ISSN-L)
- PMC
- PMC5024721
- OpenAlex
- W2324745248
- Cited by
- 63
- References
- 31
- Citations by year
- 2026: 3 · 2025: 6 · 2024: 9 · 2023: 12 · 2022: 7 · 2021: 10 · 2020: 4 · 2019: 4 · 2018: 4 · 2016: 2
- FWCI
- 2.43
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- Medical Care (publishedVersion)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- Funders
- National Institutes of Health (UL1TR000427)
- National Institute on Aging (K23AG034551)
- National Center for Research Resources (UL1RR025011)
- National Center for Advancing Translational Sciences
- Topics
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Health Sciences › Health Professions › Health Information Management › Medical Coding and Health Information
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Keywords
- unstructured data, electronic health records, inter-rater reliability, hospital discharge communication, data quality monitoring, care coordination
- MeSH terms
- Abstracting and Indexing, Data Accuracy, Aftercare, Ethyl Chloride, Humans, Time Factors, Reproducibility of Results, Multicenter Studies as Topic, Observer Variation, Electronic Health Records
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
