The 9-Item Physician Documentation Quality Instrument (PDQI-9) score is not useful in evaluating EMR (scribe) note quality in Emergency Medicine
Open accessAbstract
Summary: Background: Scribes are assisting Emergency Physicians by writing their electronic clinical notes at the bedside during consultations. They increase physician productivity and improve their working conditions. The quality of Emergency scribe notes is unevaluated and important to determine. Objective: The primary objective of the study was to determine if the quality of Emergency Department scribe notes was equivalent to physician only notes, using the Physician Documentation Quality Instrument, Nine-item tool (PDQI-9). Methods: This was a retrospective, observational study comparing 110 scribed to 110 non-scribed Emergency Physician notes written at Cabrini Emergency Department, Australia. Consultations during a randomised controlled trial of scribe/doctor productivity in 2016 were used. Emergency physicians and nurses rated randomly selected, blinded and de-identified notes, 2 raters per note. Comparisons were made between paired scribed and unscribed notes and between raters of each note. Characteristics of individual raters were examined. The ability of the tool to discriminate between good and poor notes was tested. Results: The PDQI-9 tool has significant issues. Individual items had good internal consistency (Cronbach’s alpha=0.93), but there was very poor agreement between raters (Pearson’s r=0.07, p=0.270). There were substantial differences in PDQI-9 scores allocated by each rater, with some giving typically lower scores than others, F(25,206)=1.93, p=0.007. The tool was unable to distinguish good from poor notes, F(3,34)=1.15, p=0.342. There was no difference in PDQI-9 score between scribed and non-scribed notes. Conclusions: The PDQI-9 documentation quality tool did not demonstrate reliability or validity in evaluating Emergency Medicine consultation notes. We found no evidence that scribed notes were of poorer quality than non-scribed notes, however Emergency scribe note quality has not yet been determined. Citation: Walker KJ, Wang A, Dunlop W, Rodda H, Ben-Meir M, Staples M. The 9-Item Physician Documentation Quality Instrument (PDQI-9) score is not useful in evaluating EMR (scribe) note quality in Emergency Medicine. Appl Clin Inform 2017; 8: 981–993 https://doi.org/10.4338/ACI2017052017050080
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Inter-user variability has been reported with the use of the physician documentation quality instrument (PDQI-9 tool).
“Inter-user variability has been reported with the use of the physician documentation quality instrument (PDQI-9 tool) [34]”
From Artificial Intelligence (AI) – Powered Documentation Systems in Healthcare: A Systematic Review (2025) · cites this paper as [34]
Cites
Assessing electronic note quality using the physician documentation quality instrument (PDQI-9)
Bakken, S., Wrenn, J. O., Siegler, E. L., & Stetson, P. D. (2012) · Applied Clinical Informatics · doi:10.4338/aci-2011-11-ra-0070 · Cited by 182 · see citing works
Impact of Scribes on Performance Indicators in the Emergency Department
Arya, R., Salovich, D. M., Ohman‐Strickland, P., & Merlin, M. A. (2010) · Academic Emergency Medicine · doi:10.1111/j.1553-2712.2010.00718.x · Cited by 99 · see citing works
Bank, A. J., Obetz, C. L., Konrardy, A., Khan, A. H., Pillai, K., McKinley, B., Gage, R., Turnbull, M., & Kenney, B. (2013) · ClinicoEconomics and Outcomes Research · doi:10.2147/ceor.s49010 · Cited by 95 · see citing works
Scribes in an Ambulatory Urology Practice: Patient and Physician Satisfaction
Koshy, S., Feustel, P. J., Hong, M., & Kogan, B. A. (2010) · The Journal of Urology · doi:10.1016/j.juro.2010.03.040 · Cited by 83 · see citing works
QNOTE: an instrument for measuring the quality of EHR clinical notes
BURKE, H. B., Hoang, A., Becher, D. A., Fontelo, P. A., Liu, F., Stephens, M. B., Pangaro, L. N., Sessums, L. L., O’Malley, P. G., Baxi, N. S., et al. (2014) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2013-002321 · Cited by 79 · see citing works
Preliminary Development of the Physician Documentation Quality Instrument
Stetson, P. D., Morrison, F. P., Henry, S. B., & Johnson, S. B. (2008) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m2404 · Cited by 73 · see citing works
An ED scribe program is able to improve throughput time and patient satisfaction
Bastani, A., Shaqiri, B., Palomba, K., Bananno, D., & Anderson, W. C. (2014) · The American Journal of Emergency Medicine · doi:10.1016/j.ajem.2013.03.040 · Cited by 73 · see citing works
Hess, J., Wallenstein, J., Ackerman, J. D., Akhter, M., Ander, D. S., Keadey, M. T., & Capes, J. (2015) · Western Journal of Emergency Medicine · doi:10.5811/westjem.2015.6.25432 · Cited by 67 · see citing works
Physician, Scribe, and Patient Perspectives on Clinical Scribes in Primary Care
Chen, Y., Rose, S. L., Rothberg, M. B., Mercer, M. B., Goodman, K., & Misra‐Hebert, A. D. (2016) · Journal of General Internal Medicine · doi:10.1007/s11606-016-3719-x · Cited by 64 · see citing works
Annual impact of scribes on physician productivity and revenue in a cardiology clinic
Bank, A. J., & Gage, R. (2015) · ClinicoEconomics and Outcomes Research · doi:10.2147/ceor.s89329 · Cited by 56 · see citing works
Physician, Scribe, and Patient Perspectives on Clinical Scribes in Primary Care
Misra‐Hebert, A. D., Chen, Y., & Rothberg, M. B. (2016) · Journal of General Internal Medicine · doi:10.1007/s11606-016-3888-7 · Cited by 50 · see citing works
Medical scribes: How do their notes stack up?
Misra‐Hebert, A. D., Amah, L., Rabovsky, A., Morrison, S. L., Cantave, M. D., Hu, B., Sinsky, C. A., & Rothberg, M. B. (2016) · PubMed · Cited by 49 · see citing works
Shuaib, W., Hilmi, J., Caballero, J., Rashid, I., Stanazai, H., Ajanovic, A., Moshtaghi, A., Amari, A., Tawfeek, K., Khurana, A., et al. (2017) · Health Informatics Journal · doi:10.1177/1460458217692930 · Cited by 39 · see citing works
An economic evaluation of the costs of training a medical scribe to work in Emergency Medicine
Walker, K., Dunlop, W. A., Liew, D., Staples, M., Johnson, M., Ben‐Meir, M., Rodda, H., Turner, I., & Phillips, D. G. (2016) · Emergency Medicine Journal · doi:10.1136/emermed-2016-205934 · Cited by 35 · see citing works
Self-citationImpact of scribes on emergency department patient throughput one year after implementation
Heaton, H. A., Nestler, D. M., Lohse, C. M., & Sadosty, A. T. (2016) · The American Journal of Emergency Medicine · doi:10.1016/j.ajem.2016.11.017 · Cited by 35 · see citing works
An electronic documentation system improves the quality of admission notes: a randomized trial
Jamieson, T., Ailon, J., Chien, V., & Mourad, O. (2016) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocw064 · Cited by 34 · see citing works
Scribes in an A ustralian private emergency department: A description of physician productivity
Walker, K., Ben‐Meir, M., O’Mullane, P. A., Phillips, D. G., & Staples, M. (2014) · Emergency Medicine Australasia · doi:10.1111/1742-6723.12314 · Cited by 32 · see citing works
Self-citationWalker, K., Ben‐Meir, M., Phillips, D. G., & Staples, M. (2016) · Emergency Medicine Australasia · doi:10.1111/1742-6723.12562 · Cited by 26 · see citing works
Self-citationMedical scribes have no impact on the patient experience of an emergency department
Dunlop, W. A., Hegarty, L., Staples, M., Levinson, M., Ben‐Meir, M., & Walker, K. (2017) · Emergency Medicine Australasia · doi:10.1111/1742-6723.12818 · Cited by 25 · see citing works
Self-citationImpact of scribes on patient throughput in adult and pediatric academic EDs
Heaton, H. A., Nestler, D. M., Jones, D. D., Lohse, C. M., Goyal, D. G., Kallis, J. S., & Sadosty, A. T. (2016) · The American Journal of Emergency Medicine · doi:10.1016/j.ajem.2016.07.011 · Cited by 24 · see citing works
Impact of Scribes on Billed Relative Value Units in an Academic Emergency Department
Heaton, H. A., Nestler, D. M., Jones, D. D., Varghese, R. S., Lohse, C. M., Williamson, E. S., & Sadosty, A. T. (2016) · Journal of Emergency Medicine · doi:10.1016/j.jemermed.2016.11.017 · Cited by 23 · see citing works
Bierman, J. A., Hufmeyer, K. K., Liss, D. T., Weaver, A. C., & Heiman, H. L. (2017) · Teaching and Learning in Medicine · doi:10.1080/10401334.2017.1303385 · Cited by 23 · see citing works
Kargul, G. J., Wright, S. M., Knight, A. M., McNichol, M., & RIGGIO, J. M. (2012) · American Journal of Medical Quality · doi:10.1177/1062860612445307 · Cited by 13 · see citing works
Carnes, K. M., de Riese, C. S., & de Riese, W. T. W. (2015) · Urology Practice · doi:10.1016/j.urpr.2014.10.006 · Cited by 12 · see citing works
Diagnosenverschlüsselung durch Medizinische Dokumentationsassistentin oder Stationsarzt
Tischendorf, J. J., Crede, S., Herrmann, P., Bach, N., Bömeke, C., Manns, M. P., Schaefer, O., & Trautwein, C. (2004) · DMW - Deutsche Medizinische Wochenschrift · doi:10.1055/s-2004-829024 · Cited by 9 · see citing works
TF11 Emergency Department Scribes: A Two-Step Training Program
Heaton, H. A., Samuel, R., Farrell, K. J., & Colletti, J. E. (2015) · Annals of Emergency Medicine · doi:10.1016/j.annemergmed.2015.07.480 · Cited by 9 · see citing works
Hess, J. J., Akhter, M., Ackerman, J. D., Wallenstein, J., Ander, D. S., Keadey, M. T., & Capes, J. (2013) · Annals of Emergency Medicine · doi:10.1016/j.annemergmed.2013.07.288 · Cited by 6 · see citing works
Feasibility evaluation of a pilot scribe-training program in an Australian emergency department*
Walker, K., Johnson, M. D., Dunlop, W. A., Staples, M., Rodda, H., Turner, I., & Ben‐Meir, M. (2017) · Australian Health Review · doi:10.1071/ah16188 · Cited by 5 · see citing works
Self-citationRecognizing the Characteristics of Quality Documentation.
Combs, T. (2016) · PubMed · Cited by 1 · see citing works
Flagged first, then by citations
1 of the 30 references couldn’t be matched to a known paper.
Research works printed but not linked: 1
- JJ Tischendorf, 2004, Encoding of diagnosis by medical documentation assistant or ward physician, Dtsch med Wochenschr, 129, 1731
Cited by
Sorted by citations
Ebbers, T., Kool, R. B., Smeele, L. E., Dirven, R., den Besten, C. A., Karssemakers, L. H. E., Verhoeven, T., Herruer, J. M., van den Broek, G. B., & Takes, R. P. (2022) · Journal of Medical Systems · doi:10.1007/s10916-022-01837-9 · Cited by 144 · see citing works
ChatGPT's Ability to Assist with Clinical Documentation: A Randomized Controlled Trial
Baker, H. P., Dwyer, E., Kalidoss, S., Hynes, K., Wolf, J., & Strelzow, J. A. (2024) · Journal of the American Academy of Orthopaedic Surgeons · doi:10.5435/jaaos-d-23-00474 · Cited by 115 · see citing works
Artificial Intelligence (AI) – Powered Documentation Systems in Healthcare: A Systematic Review
Bracken, A., Reilly, C., Feeley, A., Sheehan, E., Merghani, K., & Feeley, I. (2025) · Journal of Medical Systems · doi:10.1007/s10916-025-02157-4 · Cited by 113 · see citing works
“Inter-user variability has been reported with the use of the physician documentation quality instrument (PDQI-9 tool) [34] When compared to previous literature AI-driven documentation varied between “bad (29.7)” and “excellent (35.6)” highlighting the variability in documentation produced by AI…”
InfluentialThe emergence of new data work occupations in healthcare: The case of medical scribes
Bossen, C., Chen, Y., & Pine, K. H. (2019) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2019.01.001 · Cited by 80 · see citing works
Walker, K., Ben‐Meir, M., Dunlop, W. A., Rosler, R., West, A., O’Connor, G., Chan, T., Badcock, D., Putland, M., Hansen, K., et al. (2019) · BMJ · doi:10.1136/bmj.l121 · Cited by 56 · see citing works
Self-citationPalm, E., Manikantan, A., Mahal, H., Belwadi, S. S., & Pepin, M. E. (2025) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2025.1691499 · Cited by 67 · see citing works
Evaluating Hospital Course Summarization by an Electronic Health Record–Based Large Language Model
Small, W. R., Austrian, J. S., O’Donnell, L., Burk‐Rafel, J., Hochman, K. A., Goodman, A. J., Zaretsky, J., Martin, J. A., Johnson, S. B., Major, V. J., et al. (2025) · JAMA Network Open · doi:10.1001/jamanetworkopen.2025.26339 · Cited by 40 · see citing works
Prater, L., Sanchez, A., Modan, G. G., Burgess, J. L., Frier, K. A., Richards, N., & Bose‐Brill, S. (2019) · Applied Clinical Informatics · doi:10.1055/s-0039-1683986 · Cited by 21 · see citing works
The Effect of Medical Scribes in Emergency Departments: A Systematic Review
Ullman, K. E., McKenzie, L. G., Bart, B. A., Park, G., MacDonald, R. H., Linskens, E. J., & Wilt, T. J. (2021) · Journal of Emergency Medicine · doi:10.1016/j.jemermed.2021.02.024 · Cited by 16 · see citing works
Medico-legal issues related to emergency physicians’ documentation in Canadian emergency departments
Smith, J. D., Lemay, K., Lee, S., Nuth, J., Ji, J., Montague, K., & Garber, G. E. (2023) · Canadian Journal of Emergency Medicine · doi:10.1007/s43678-023-00576-1 · Cited by 13 · see citing works
Do You Know What Your Scribe Did Last Spring? The Impact of COVID-19 on Medical Scribe Workflow
Gold, J. A., Becton, J., Ash, J. S., Corby, S., & Mohan, V. (2020) · Applied Clinical Informatics · doi:10.1055/s-0040-1721396 · Cited by 13 · see citing works
Cowan, T. L., Dunlop, W. A., Ben‐Meir, M., Staples, M., Treadwell, A., Gardner-Brunton, E., & Walker, K. (2017) · Emergency Medicine Journal · doi:10.1136/emermed-2017-206637 · Cited by 10 · see citing works
Self-citationChart Completion Time of Attending Physicians While Using Medical Scribes.
Florig, S. T., Corby, S., Rosson, N. T., Devara, T., Weiskopf, N. G., Gold, J. A., & Mohan, V. (2021) · PubMed Central · Cited by 10 · see citing works
Perotte, R., Hajicharalambous, C., Sugalski, G., & Underwood, J. P. (2021) · PubMed Central · Cited by 9 · see citing works
Comparing Scribed and Non-scribed Outpatient Progress Notes.
Rule, A., Florig, S. T., Bedrick, S. D., Mohan, V., Gold, J. A., & Hribar, M. R. (2021) · PubMed Central · Cited by 8 · see citing works
Chiang, Y.-L., Yang, K.-F., Su, P.-Y., Tsai, S., & Liang, K.-L. (2025) · Applied Clinical Informatics · doi:10.1055/a-2707-2959 · Cited by 6 · see citing works
Eshel, R., Bellolio, F., Boggust, A. J., Shapiro, N. I., Mullan, A. F., Heaton, H. A., Madsen, B. E., Homme, J. L., Iliff, B. W., Sunga, K., et al. (2022) · The American Journal of Emergency Medicine · doi:10.1016/j.ajem.2022.10.009 · Cited by 5 · see citing works
Schaeffer, C. V., Park, R., Lane, J. C., & DeGeorge, B. R. (2021) · Plastic & Reconstructive Surgery Global Open · doi:10.1097/gox.0000000000003726 · Cited by 5 · see citing works
Effects of medical scribes on patients, physicians, and safety: A scoping review
Shah, L. M., Borycki, E. M., Shah, J. A., Kushniruk, A. W., & |, |. (2021) · Knowledge Management & E-Learning An International Journal · doi:10.34105/j.kmel.2021.13.030 · Cited by 5 · see citing works
Mousavi, A. S., & Baigi, S. F. M. (2022) · Frontiers in Health Informatics · doi:10.30699/fhi.v11i1.348 · Cited by 5 · see citing works
Primary Care Productivity: Findings from the Literature and Perspectives from a Stakeholder Panel
Hempel, S., Curtis, I., Fihn, S. D., Brothers, A., Danz, M., Nelson, K. M., Motala, A., & Rubenstein, L. V. (2021) · RAND Corporation eBooks · doi:10.7249/rra703-1 · Cited by 2 · see citing works
mousavi, A., Yousefi, M., Jamali, J., Meraji, M., & Fazaeli, S. (2022) · Journal of Modern Medical Information Sciences · doi:10.52547/jmis.7.4.52 · Cited by 2 · see citing works
The Evidence Base for Scribes and the Disruptions of COVID-19
Walker, K., & Heaton, H. A. (2020) · Annals of Emergency Medicine · doi:10.1016/j.annemergmed.2020.09.438 · Cited by 2 · see citing works
Self-citationDevelopment of a scoring tool for emergency department note quality: a modified Delphi approach
Foster, D. Z., Douglas, S. L., & Rajaram, A. (2025) · Canadian Journal of Emergency Medicine · doi:10.1007/s43678-025-00914-5 · Cited by 1 · see citing works
Collaborative group reasoning in ward rounds: A critical realist case study
Perversi, P. (2019) · Deakin Research Online (Deakin University) · Cited by 1 · see citing works
Impact of an ambient digital scribe on typing and note quality: the AutoscriberValidate studyPreprint
Bauer, M. P., van Tol, E. M., Constansia, T. K. M., King, L., & van Buchem, M. M. (2026) · medRxiv · doi:10.64898/2026.02.19.26346634 · Cited by 1 · see citing works
Pilot trial of semi-automated medical note writing using lexeme hypotheses
Gugel, D., Lentz, S. R., Perepu, U. S., Sharathkumar, A. A., Staber, J. M. R., Sutamtewagul, G., & Macfarlane, D. E. (2020) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2020.104095
Zyloney, C., Modica, J. S., Manou, D., Mooney, C. J., & Stone, R. T. (2020) · doi:10.22541/au.158291215.59753652
Carral, P., Benítez, S., del Valle, M. P., Sotelo, L. E. D., Ortega, V. A., Lenta, M. D. R., Monopoli, M. J., & Diomedi, S. S. (2025) · Methods of Information in Medicine · doi:10.1055/a-2841-4662
The inter-rater reliability of the TEDNQ for evaluating emergency department note quality
Rajaram, A., Rosas, K. P., Foster, D., Szulewski, A., Wolfrom, B., & Douglas, S. L. (2026) · Canadian Journal of Emergency Medicine · doi:10.1007/s43678-026-01187-2
Tolunay, İ., Erhan, M. D., & Kula, S. (2026) · Journal of Health Systems and Policies · doi:10.52675/jhesp.1914015
Nassani, L. M., McLellan, A., Trotman, C. A., Amer, R. S., Tavazozadeh, E., Harding, S., & Jamilian, A. (2026) · The British Journal of Translational Global Health · doi:10.5005/bjotgh-11016-0046
Bergman, H. I., Liu, V., & Ali, S. (2026) · medRxiv · doi:10.64898/2026.10.16.26365719
Beyond Time Saved: A Framework for Comprehensive Evaluation of AI Clinical Documentation
Ekstrom, V. S. M. (2026) · JMIR Medical Informatics · doi:10.2196/86091
- Authors
- Wang, A., Dunlop, W., Rodda, H., Ben-Meir, M., Staples, M., & Walker, K. J.
- Affiliations
- Monash University (AU)
- Cabrini Hospital (AU)
- Box Hill Hospital (AU)
- Venue
- Applied Clinical Informatics
- Publisher
- Georg Thieme Verlag KG
- Published
- 2017-07
- Volume
- 08
- Issue
- 03
- Pages
- 981-993
- ISSN
- 1869-0327 (ISSN-L)
- PMC
- PMC6220701
- OpenAlex
- W2759593086
- Cited by
- 35
- References
- 30
- Citations by year
- 2026: 7 · 2025: 6 · 2023: 2 · 2022: 4 · 2021: 7 · 2020: 4 · 2019: 4 · 2017: 1
- FWCI
- 3.38
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Applied Clinical Informatics (publishedVersion)
- Europe PMC (PubMed Central) (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Funders
- The Cabrini Medical Staff Association
- Topics
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Health Sciences › Nursing › Research and Theory › Nursing Diagnosis and Documentation
- Health Sciences › Health Professions › Health Information Management › Medical Coding and Health Information
- Keywords
- electronic health records, documentation quality, emergency department, inter-rater reliability, assessment validity, physician productivity
- MeSH terms
- Documentation, Emergency Medicine, Humans, Physicians, Quality Control, Retrospective Studies, Electronic Health Records
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
