Using multiclass classification to automate the identification of patient safety incident reports by type and severity
Open accessThe DOI matches its Crossref record; 8 of the 8 references it cites were found in a registry or an index.
Abstract
Approximately 10% of admissions to acute-care hospitals are associated with an adverse event. Analysis of incident reports helps to understand how and why incidents occur and can inform policy and practice for safer care. Unfortunately our capacity to monitor and respond to incident reports in a timely manner is limited by the sheer volumes of data collected. In this study, we aim to evaluate the feasibility of using multiclass classification to automate the identification of patient safety incidents in hospitals. Text based classifiers were applied to identify 10 incident types and 4 severity levels. Using the one-versus-one (OvsO) and one-versus-all (OvsA) ensemble strategies, we evaluated regularized logistic regression, linear support vector machine (SVM) and SVM with a radial-basis function (RBF) kernel. Classifiers were trained and tested with “balanced” datasets (n_ Type = 2860, n_ SeverityLevel = 1160) from a state-wide incident reporting system. Testing was also undertaken with imbalanced “stratified” datasets (n_ Type = 6000, n_ SeverityLevel =5950) from the state-wide system and an independent hospital reporting system. Classifier performance was evaluated using a confusion matrix, as well as F-score, precision and recall. The most effective combination was a OvsO ensemble of binary SVM RBF classifiers with binary count feature extraction. For incident type, classifiers performed well on balanced and stratified datasets (F-score: 78.3, 73.9%), but were worse on independent datasets (68.5%). Reports about falls, medications, pressure injury, aggression and blood products were identified with high recall and precision. “Documentation” was the hardest type to identify. For severity level, F-score for severity assessment code (SAC) 1 (extreme risk) was 87.3 and 64% for SAC4 (low risk) on balanced data. With stratified data, high recall was achieved for SAC1 (82.8–84%) but precision was poor (6.8–11.2%). High risk incidents (SAC2) were confused with medium risk incidents (SAC3). Binary classifier ensembles appear to be a feasible method for identifying incidents by type and severity level. Automated identification should enable safety problems to be detected and addressed in a more timely manner. Multi-label classifiers may be necessary for reports that relate to more than one incident type.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A systematic analysis of performance measures for classification tasks
Sokolova, M., & Lapalme, G. (2009) · Information Processing & Management · doi:10.1016/j.ipm.2009.03.002 · Cited by 7,515
A Survey of Predictive Modeling on Imbalanced Domains
Branco, P., Torgo, L., & Ribeiro, R. P. (2017) · ACM Computing Surveys · doi:10.1145/2907070 · Cited by 1,129 · cited 2×
“Sensitivity can also be improved by increasing misclassification costs for rare classes so that they gain more importance during classifier training [47].”
BackgroundTowards an International Classification for Patient Safety: key concepts and terms
Runciman, W., Hibbert, P., Thomson, R., Van Der Schaaf, T., Sherman, H., & Lewalle, P. (2009) · International Journal for Quality in Health Care · doi:10.1093/intqhc/mzn057 · Cited by 647 · cited 1×
“An incident type is a descriptive term for a category of incidents with a common nature, grouped because of shared, agreed features [8] .”
BackgroundSelf-citationEfficient Visual Search of Videos Cast as Text Retrieval
Sivic, J., & Zisserman, A. (2009) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2008.111 · Cited by 499
Using FDA reports to inform a classification for health information technology safety problems
Magrabi, F., Ong, M.-S., Runciman, W., & Coiera, E. (2012) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000369 · Cited by 193 · cited 1×
“Current methods, which rely upon the retrospective manual review of reports, can no longer keep up with the growing volume of incidents being reported by healthcare workers [5 – 7].”
MethodSelf-citationRunciman, W. B., Williamson, J. A. H., Deakin, A., Benveniste, K. A., Bannon, K., & Hibbert, P. D. (2006) · Quality in Health Care · doi:10.1136/qshc.2005.017467 · Cited by 156
Self-citationMagrabi, F., Baker, M., Sinha, I., Ong, M.-S., Harrison, S., Kidd, M. R., Runciman, W. B., & Coiera, E. (2015) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2014.12.003 · Cited by 100
Self-citationApplying active learning to assertion classification of concepts in clinical text
Chen, Y., Mani, S., & Xu, H. (2012) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2011.11.003 · Cited by 57
Cited by
Role of artificial intelligence in patient safety outcomes: systematic literature review
Choudhury, A., & Asan, O. (2020) · JMIR Medical Informatics · doi:10.2196/18599 · Cited by 440
Clinical text data in machine learning: systematic review
Spasic, I., & Nenadic, G. (2020) · JMIR Medical Informatics · doi:10.2196/17984 · Cited by 358
“Similarly, applications related to patient safety focused on relevant documents such as adverse event reports [15], patient safety event reports [35], and incident reports [101].”
BackgroundInfluentialClinical text classification research trends: Systematic literature review and open issues
Mujtaba, G., Shuib, L., Idris, N., Hoo, W. L., Raj, R. G., Khowaja, K., Shaikh, K., & Nweke, H. F. (2019) · Expert Systems with Applications · doi:10.1016/j.eswa.2018.09.034 · Cited by 123
El Morr, C. (2019) · International Journal of Extreme Automation and Connectivity in Healthcare · doi:10.4018/ijeach.2019010102 · Cited by 10
Wang, H.-H., Huang, C.-C., Talley, P. C., & Kuo, K.-M. (2022) · Journal of Healthcare Engineering · doi:10.1155/2022/3100618 · Cited by 6
“Second, most studies focus on assessing the types of patient safety incidents [12, 13, 15, 16, 23, 26, 29].”
BackgroundMethodInfluential
Metadata
- Authors
- Wang, Y., Coiera, E., Runciman, W., & Magrabi, F.
- Venue
- BMC Medical Informatics and Decision Making
- Published
- 2017-06-12
- Volume
- 17
- Issue
- 1
- Pages
- 84-84
- ISSN
- 1472-6947 (ISSN-L)
- OpenAlex
- W2621588820
- Cited by
- 63
- References
- 8
- FWCI
- 4.26
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Using multiclass classification to automate the identification of patient safety incident reports by type and severity. https://app.cochat.ai/papers/10.1186/s12911-017-0483-8
- The paper
- Wang, Y., Coiera, E., Runciman, W., & Magrabi, F. (2017). Using multiclass classification to automate the identification of patient safety incident reports by type and severity. BMC Medical Informatics and Decision Making. https://doi.org/10.1186/s12911-017-0483-8
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.