Documentation and coding of ED patient encounters: an evaluation of the accuracy of an electronic medical record
Abstract
Objective The aim of the study was to describe a paper-based, template-driven and an electronic medical record used for capturing emergency care clinical information and to compare the accuracy of these documentation systems for coding patient encounters using the American Medical Association Current Procedural Terminology-2004 (AMA CPT-2004) evaluation and management codes intended for provider reimbursement.Methods A retrospective, cross-sectional study of 4-consecutive-day samples of ED patient encounter records from 2 similar community hospitals was done. For clinical documentation, hospital A uses an electronic medical record, whereas hospital B uses a paper-based template-driven record. Using a simple analytic model, expert coders A and B, respectively, coded the records from hospitals A and B for completeness. First, power analysis determined the acceptability of the patient record sample sizes (1 - beta = .90 at 1% significance level), and the frequency of AMA CPT-2004 primary evaluation and management codes 99281 through 99285 was calculated. Second, the completeness discrepancy rates for hospitals A and B were compared to determine the accuracy of both the paper-based, template-driven record and the electronic medical record in documenting and representing the clinical encounter. Third, interrater reliability between expert coders A and B was calculated to assess the level of agreement between each expert coder in determining the completeness discrepancy rates between hospitals A and B. Finally, the frequency of primary evaluation and management codes was analyzed to determine if there was a statistically significant difference between the paper-based, template-driven record and the electronic medical record representation of the clinical information, and if that difference could be attributable to the differing clinical documentation systems used in hospitals A and B.Results First, descriptive display demonstrated a difference in the frequency of the primary evaluation and management codes 99283 and 99284 within hospital A (expert coder A assessment, 36.1% vs 39.1%; expert coder B assessment, 36.6% vs 38.7%) and hospital B (expert coder A assessment, 47.8% vs 21.9%; expert coder B assessment, 48.6% vs 21.4%) was noted with the median, primary evaluation, and management code for hospital A of 99284 and the median, primary evaluation, and management code for hospital B of 99283. Second, Fisher exact test compared the completeness discrepancy rates between hospitals A and B as assessed by each expert coder and demonstrated no statistically significant difference in the completeness discrepancy rates (accuracy) between the paper-based, template-driven record and the electronic medical record documentation and coding system when assessed by either expert coder A (P = .370) or expert coder B (P = .819). Third, interrater reliability between expert coders A and B was evaluated using Cohen's kappa statistic. When evaluated both individually and jointly with respect to hospitals A and B, expert coders A and B had a good strength of agreement in their assessments of the accuracy of the documentation and coding system for hospital A (kappa = 0.6200) and hospital B (kappa = 0.6906) as well as for both hospitals evaluated together (kappa = 0.6616). Finally, interhospital differences in the frequency of primary evaluation and management codes were evaluated using Pearson chi(2) test with 3 df. The results for expert coder A (chi(2) = 47.4160; P < .001) and expert coder B (chi(2) = 46.5946; P < .001) recognize that there is a statistically significant degree of difference between hospitals A and B in the frequency distribution of primary evaluation and management codes, probably because of the dispersion of codes 99283 and 99284.Conclusions A keystroke-driven, electronic medical record that resides on a knowledge platform that incorporates a clinical structured terminology, administrative coding schemata, AMA CPT-2004 codes and uses object-oriented, open-ended, branching chain clinical algorithms that "force" physician documentation of the clinical elements provides an equally accurate capture and representation of ED clinical encounter data as a paper-based, template-driven documentation system both in terms of the presence or absence of both the medically necessary, discrete data elements and the textual documentation-dependent, medical decision-making elements.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Existing evaluation metrics for healthcare chatbots lack an understanding of medical concepts (e.g., symptoms, diagnostic tests, diagnoses, and treatments), their interplay, and the priority for the well-being of the patient.
“Although these metrics are model-based, they lack an understanding of medical concepts (e.g., symptoms, diagnostic tests, diagnoses, and treatments), their interplay, and the priority for the well-being of the patient, all of which are crucial for medical decision-making10.”
From Foundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI (2024) · cites this paper as [10]
Cites
Accuracy of Data in Computer-based Patient Records
Hogan, W. R., & Wagner, M. M. (1997) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.1997.0040342 · Cited by 389 · see citing works
Porcheret, M., Hughes, R. W., Evans, D., Jordan, K. P., Whitehurst, T., & Ogden, H. (2003) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1362 · Cited by 145 · see citing works
Stausberg, J., Koch, D., Ingenerf, J., & Betzler, M. (2003) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1290 · Cited by 128 · see citing works
Hripcsak, G. M., & Wilcox, A. B. (2002) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.2002.0090001 · Cited by 89 · see citing works
Studying the Human--Computer--Terminology Interface
Cimino, J. J., Patel, V. L., & Kushniruk, A. W. (2001) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.2001.0080163 · Cited by 85 · see citing works
Sujansky, W. (1998) · PubMed Central · Cited by 72 · see citing works
Marill, K. A., Gauharou, E. S., Nelson, B. K., Peterson, M. A., Curtis, R. L., & Gonzalez, M. R. (1999) · Annals of Emergency Medicine · doi:10.1016/s0196-0644(99)70336-7 · Cited by 53 · see citing works
Computerized Reminders Reduce the Use of Medications during Shortages
Bogucki, B. (2004) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1531 · Cited by 33 · see citing works
The Next-Generation Emergency Department
Smith, M. S., & Feied, C. F. (1998) · Annals of Emergency Medicine · doi:10.1016/s0196-0644(98)70101-5 · Cited by 28 · see citing works
Regional Variation in Medical Classification Agreement: Benchmarking the Coding Gap
Lorence, D. P. (2003) · Journal of Medical Systems · doi:10.1023/a:1025607805588 · Cited by 23 · see citing works
Reliability of assigning correct current procedural terminology—4 E/M codes
Bentley, P. N., Wilson, A. G., Derwin, M. E., Scodellaro, R., & Jackson, R. E. (2002) · Annals of Emergency Medicine · doi:10.1067/mem.2002.123692 · Cited by 14 · see citing works
Coding Clinical Information: Analysis of Clinicians Using Computerized Coding
Kadlec, P., Puerner, F., & Hohnloser, J. H. (1996) · Methods of Information in Medicine · doi:10.1055/s-0038-1634643 · Cited by 8 · see citing works
Hohnloser, J. H., Puerner, F., & Soltanian, H. T. (1996) · Computers and Biomedical Research · doi:10.1006/cbmr.1996.0004 · Cited by 8 · see citing works
Winn, D. (2001) · PubMed · Cited by 1 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Foundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI
Abbasian, M., Khatibi, E., Azimi, I., Oniani, D., Shakeri Hossein Abad, Z., Thieme, A., Sriram, R., Yang, Z., Wang, Y., Lin, B., et al. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01074-z · Cited by 213 · see citing works
“10 For this reason, they inadequately capture vital aspects like semantic nuances, contextual relevance, long-range dependencies, changes in critical semantic ordering, and human-centric perspectives, 11 thereby limiting their e ff ectiveness in evaluating healthcare chatbots.”
BackgroundStrategies for improving physician documentation in the emergency department: a systematic review
Lorenzetti, D., Quan, H., Lucyk, K., Cunningham, C., Hennessy, D. A., Jiang, J., & Beck, C. A. (2018) · BMC Emergency Medicine · doi:10.1186/s12873-018-0188-z · Cited by 106 · see citing works
Vuokko, R., Mäkelä-Bengs, P., Hyppönen, H., Lindqvist, M., & Doupi, P. (2016) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2016.10.004 · Cited by 76 · see citing works
Electronic health records improve clinical note quality
BURKE, H. B., Sessums, L. L., Hoang, A., Becher, D. A., Fontelo, P. A., Liu, F., Stephens, M. B., Pangaro, L. N., O’Malley, P. G., Baxi, N. S., et al. (2014) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2014-002726 · Cited by 72 · see citing works
Structured Radiology Reporting: Are We There Yet?
Langlotz, C. P. (2009) · Radiology · doi:10.1148/radiol.2531091088 · Cited by 71 · see citing works
Farzandipour, M., Sheikhtaheri, A., & Sadoughi, F. (2009) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2009.07.002 · Cited by 63 · see citing works
Wu, L.-T. T., Ghitza, U. E., Batch, B. C., Pencina, M. J., Rojas, L. F., Goldstein, B. A., Schibler, T., Dunham, A. A., Rusincovitch, S. A., & Brady, K. T. (2015) · Drug and Alcohol Dependence · doi:10.1016/j.drugalcdep.2015.09.003 · Cited by 60 · see citing works
“The long-term use of an EMR to enhance healthcare and the longitudinally captured EHRs may improve completeness of diagnostic and treatment data (Horvath et al., 2014; Pringle et al., 1995; Silfen, 2006; Silow-Carroll et al., 2012; Weiner et al., 2007).”
BackgroundConcordance Between Electronic Clinical Documentation and Physicians’ Observed Behavior
Berdahl, C. T., Moran, G. J., McBride, O., Santini, A. M., Verzhbinsky, I. A., & Schriger, D. L. (2019) · JAMA Network Open · doi:10.1001/jamanetworkopen.2019.11390 · Cited by 47 · see citing works
“For ROS, physicians documented a median (interquartile range [IQR]) of 14 (8-14) systems, while audio recordings confirmed a median (IQR) of 5 (3-6) systems.”
BackgroundBabu, M. A., Nahed, B. V., DeMoya, M. A., & Curry, W. T. (2011) · Neurosurgery · doi:10.1227/neu.0b013e31821bc667 · Cited by 46 · see citing works
“ISS was categorized into 3 groups: mild (0-9), moderate (10-25), and severe (26-75).”
BackgroundQuality of outpatient clinical notes: a stakeholder definition derived through qualitative research
Hanson, J. L., Stephens, M. B., Pangaro, L. N., & Gimbel, R. W. (2012) · BMC Health Services Research · doi:10.1186/1472-6963-12-407 · Cited by 37 · see citing works
“Background Quality of outpatient clinical encounter notes relates to patient safety [1,2], medical education [3-5], medico-legal issues [6,7], and justification of reimbursement [6-9].”
BackgroundDuszak, R., Nossal, M., Schofield, L., & Picus, D. D. (2012) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2012.01.006 · Cited by 37 · see citing works
“The use of structured templates has een described in the emergency department setting as a ool to facilitate appropriate CPT coding [24].”
MethodInferring conceptual relationships to improve medical records search
Limsopatham, N., Macdonald, C., & Ounis, I. (2013) · doi:10.5555/2491748.2491750 · Cited by 33 · see citing works
Wu, L.-T. T., Gersing, K. R., Swartz, M. S., Burchett, B. M., Li, T., & Blazer, D. G. (2013) · Journal of Psychiatric Research · doi:10.1016/j.jpsychires.2012.12.009 · Cited by 33 · see citing works
“Further, compared with use of insurance claim data for research, EHRs include complete and potentially more accurate treatment encounter data (Silfen, 2006; Weiner et al., 2007).”
Wu, L.-T. T., Blazer, D. G., Gersing, K. R., Burchett, B. M., Swartz, M. S., & Mannelli, P. (2013) · Journal of Psychiatric Research · doi:10.1016/j.jpsychires.2013.08.022 · Cited by 31 · see citing works
“Although EHRs include complete and potentially more accurate treatment-encounter data for research use than insurance claim data and abstractedmedical record data (Silfen, 2006;Weiner et al., 2007), results from EHRs may underestimate the treatment needs of AAs, NHs/PIs, and MRs.”
A Task-Specific Query and Document Representation for Medical Records Search
Limsopatham, N., Macdonald, C., & Ounis, I. (2013) · Lecture notes in computer science · doi:10.1007/978-3-642-36973-5_75 · Cited by 28 · see citing works
“Hence, we propose to use only the medical concepts related to four aspects of the medical decision criteria [7] to represent medical records and queries.”
BackgroundMethodAl-Mujaini, A. (2011) · Oman Medical Journal · doi:10.5001/omj.2011.81 · Cited by 27 · see citing works
Farzandipour, M., & Sheikhtaheri, A. (2009) · PubMed Central · Cited by 23 · see citing works
Limsopatham, N., McCreadie, R., Albakour, M., Macdonald, C., Santos, R. L. T., & Ounis, I. (2012) · Text REtrieval Conference · Cited by 21 · see citing works
Analisis Akurasi Koding Pada Pengembalian Klaim BPJS Rawat Inap Di RSUP Fatmawati Tahun 2016
Indawati, L. (2019) · Jurnal Manajemen Informasi Kesehatan Indonesia (JMIKI) · doi:10.33560/jmiki.v7i2.230 · Cited by 19 · see citing works
Peivandi, S., Ahmadian, L., Farokhzadian, J., & Jahani, Y. (2022) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-022-01835-4 · Cited by 23 · see citing works
“These documents are also important in matters of patient safety [3–7], quality assurance [7–9], legal processes [7, 10, 11], financial issues [5, 10–12], and medical”
BackgroundHosseini, N., Kimiafar, K., Mostafavi, S. M., Kiani, B., Zendehdel, K., Zareiyan, A., & Eslami, S. (2021) · The International Journal of Health Planning and Management · doi:10.1002/hpm.3254 · Cited by 13 · see citing works
Emergency Medicine Quality Improvement and Patient Safety Curriculum
Kelly, J. J., Thallner, E. A., Broida, R. I., Cheung, D., Meisl, H., Hamedani, A. G., Klauer, K. M., Welch, S. J., & Beach, C. (2010) · Academic Emergency Medicine · doi:10.1111/j.1553-2712.2010.00897.x · Cited by 13 · see citing works
Chalmers, D. J., Deakyne, S., Payan, M., Torok, M. R., Kahn, M. G., & Vemulakonda, V. M. (2014) · The Journal of Urology · doi:10.1016/j.juro.2014.04.091 · Cited by 12 · see citing works
Emergency department information system diagnosis: how accurate is it?
Mitchell, J. S., & Westerduin, F. (2008) · Emergency Medicine Journal · doi:10.1136/emj.2007.050104 · Cited by 12 · see citing works
“A more recent study comparing electronic records with hand-written records found them to be of equal accuracy,”
BackgroundSkelly, K., & BERGUS, G. R. (2010) · PubMed · Cited by 10 · see citing works
Kolšek, M. (2009) · Srpski arhiv za celokupno lekarstvo · doi:10.2298/sarh0912664k · Cited by 8 · see citing works
“There are already EMR that are at least as good as old paper-based ones [32].”
BackgroundEvidence-Based Guidelines for Interface Design for Data Entry in Electronic Health Records
Wilbanks, B. A., & Moss, J. A. (2017) · CIN Computers Informatics Nursing · doi:10.1097/cin.0000000000000387 · Cited by 8 · see citing works
“Paper-based and electronic templates have been shown to produce equal levels of completeness in generated documentation.(30) Documentation templates are useful for promoting evidence-based practice, improving documentation quality, making a secondary data analysis of EHR data sets easier, and helping to prove meaningf…”
BackgroundAn Integrative Literature Review on Accuracy in Anesthesia Information Management Systems
Wilbanks, B. A. (2014) · CIN Computers Informatics Nursing · doi:10.1097/nxn.0b013e3182a041f7 · Cited by 6 · see citing works
Gkiala, A. (2022) · Cureus · doi:10.7759/cureus.33000 · Cited by 6 · see citing works
“Quality has been found to be negatively affected by patient volume, the complexity of cases, and the variety and number of healthcare professionals involved in the care of the patient [12,13].”
BackgroundHaglin, J. M., Zeller, J. L., Egol, K. A., & Phillips, D. P. (2017) · The Spine Journal · doi:10.1016/j.spinee.2017.06.009 · Cited by 6 · see citing works
“with comparing documentation pre- and post-EMR implementation rather than assessing what 14 was specifically documented in the medical record [17-19].”
BackgroundEshel, R., Bellolio, F., Boggust, A. J., Shapiro, N. I., Mullan, A. F., Heaton, H. A., Madsen, B. E., Homme, J. L., Iliff, B. W., Sunga, K., et al. (2022) · The American Journal of Emergency Medicine · doi:10.1016/j.ajem.2022.10.009 · Cited by 5 · see citing works
Docheva, N., Heimberger, S., Mueller, A., Bisson, C. M., Arenas, G. A., Perdigao, J. L., Kordik, A., Stewart, K., Goodall, P., Lengyel, E. R., et al. (2023) · Reproductive Sciences · doi:10.1007/s43032-023-01174-3 · Cited by 3 · see citing works
Tillman, D., Jewell, C., Hekman, D. J., Nicholson, A. M., Schnapp, B. H., Lasarev, M. R., Alexandridis, R. A., Hess, J. M., & Westergaard, M. C. (2022) · AEM Education and Training · doi:10.1002/aet2.10741 · Cited by 3 · see citing works
Improving Documentation Using a Real-Time Location System in a Pediatric Emergency Department
Overmann, K. M., Barrick, L., & Porter, S. C. (2021) · Applied Clinical Informatics · doi:10.1055/s-0041-1730028 · Cited by 3 · see citing works
Shaver, K. J., Weiss, S. J., & Braude, D. A. (2010) · The American Journal of Emergency Medicine · doi:10.1016/j.ajem.2008.11.019 · Cited by 2 · see citing works
“Perhaps more pertinent and focused teaching about the unique aspects of documentation and billing/coding could have improved charting [8].”
BackgroundMsami, K., Okoroafor, U., Mbwana, M., Ngowi, J. R., Lwanga, C., & Luziga, H. (2023) · medRxiv · doi:10.1101/2023.09.18.23295750 · Cited by 2 · see citing works
Patnaik, D., Yadav, S. L., & Rajderkar, N. P. (2022) · Interdisciplinary Research in Technology and Management (IRTM) · doi:10.1109/irtm54583.2022.9791597 · Cited by 1 · see citing works
A systematic review of quality components of patients’ medical record in Iranian Hospital
Gommnami, N., Juni, M. H., Said, S. M., Hassan, S. T. S., & Arya, E. (2015) · Universiti Putra Malaysia Institutional Repository (Universiti Putra Malaysia)
Coding and Messaging Systems for Women's Health Informatics
Parry, D. (2009) · IGI Global eBooks · doi:10.4018/978-1-60566-078-3.ch003
Li, J., Wang, Z., Gao, Z., & Zhou, X. (2020) · International Conference on Intelligent Computation Technology and Automation · doi:10.1109/icicta51737.2020.00130
Susanto, A., & Pribadi, F. (2023) · Archives of Business Research · doi:10.14738/abr.111.12755
Is Technology the Cure for Soaring Demand and the High Costs of Healthcare?
Baran, G. R., Kiani, M. F., & Samuel, S. P. (2013) · Springer eBooks · doi:10.1007/978-1-4614-8541-4_1
Nichols, L. (2026) · Australasian Journal of Neuroscience · doi:10.2478/ajon-2026-0008
Child Welfare Involvement and Health Outcomes in Infants With Prenatal Substance Exposure
Chen, W., Connell, C. M., & Font, S. A. (2026) · JAMA Health Forum · doi:10.1001/jamahealthforum.2026.1302
- Authors
- Silfen, E.
- Affiliations
- Columbia University (US)
- Corresponding author
- Silfen, Eric
- ORCID
- Silfen, Eric
- Venue
- The American Journal of Emergency Medicine
- Publisher
- Elsevier BV
- Published
- 2006-10
- Volume
- 24
- Issue
- 6
- Pages
- 664-678
- ISSN
- 0735-6757 (ISSN-L)
- OpenAlex
- W2105949371
- Cited by
- 44
- References
- 14
- Citations by year
- 2026: 2 · 2024: 1 · 2023: 3 · 2022: 5 · 2021: 2 · 2020: 1 · 2019: 2 · 2018: 1 · 2017: 2 · 2016: 1 · 2015: 2 · 2014: 3 · 2013: 5 · 2012: 3
- FWCI
- 4.36
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- The American Journal of Emergency Medicine (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Health Sciences › Health Professions › Health Information Management › Medical Coding and Health Information
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Health Sciences › Nursing › Issues, ethics and legal aspects › Nursing Diagnosis and Documentation
- Keywords
- electronic health records, emergency department documentation, paper medical records, documentation accuracy, documentation completeness, inter-rater reliability, medical coding
- MeSH terms
- Cross-Sectional Studies, Decision Making, Documentation, Emergency Service, Hospital, Forms and Records Control, Humans, Retrospective Studies, Chi-Square Distribution, Medical Records Systems, Computerized, Current Procedural Terminology
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
