The elephant in the record: On the multiplicity of data recording work
Open accessThe DOI matches its Crossref record; 10 of the 10 references it cites were found in a registry or an index.
Abstract
This article focuses on the production side of clinical data work, or data recording work, and in particular, on its multiplicity in terms of data variability. We report the findings from two case studies aimed at assessing the multiplicity that can be observed when the same medical phenomenon is recorded by multiple competent experts, yet the recorded data enable the knowledgeable management of illness trajectories. Often framed in terms of the latent unreliability of medical data, and then treated as a problem to solve, we argue that practitioners in the health informatics field must gain a greater awareness of the natural variability of data inscribing work, assess it, and design solutions that allow actors on both sides of clinical data work, that is, the production and care, as well as the primary and secondary uses of data to aptly inform each other’s practices.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Answering the Call for a Standard Reliability Measure for Coding Data
Hayes, A. F., & Krippendorff, K. (2007) · Communication Methods and Measures · doi:10.1080/19312450709336664 · Cited by 4,225 · cited 6×
“For the sections of the forms that encompass the mutually exclusive type of multiple choice items (rendered as radio buttons on Web forms), we employed a chance-adjusted measure, Krippendorff’s agreement coefficient alpha .”
MethodInfluentialAllocation of Physician Time in Ambulatory Practice: A Time and Motion Study in 4 Specialties
Sinsky, C., Colligan, L., Li, L., Prgomet, M., Reynolds, S., Goeders, L., Westbrook, J., Tutty, M., & Blike, G. (2016) · Annals of Internal Medicine · doi:10.7326/m16-0961 · Cited by 1,494 · cited 3×
“Even isolating “direct clinical face time” 1 in single encounters would neglect the individual doctors’ need to take written notes of encounters so conditions and interpretations can be recalled during future encounters 9 and, in so doing, formulate an understanding of the evolution of the…”
BackgroundResultArndt, B. G., Beasley, J. W., Watkinson, M. D., Temte, J. L., Tuan, W.-J., Sinsky, C. A., & Gilchrist, V. J. (2017) · The Annals of Family Medicine · doi:10.1370/afm.2121 · Cited by 972 · cited 1×
“This finding confirms other recent studies 1,2,6,7 that found how medical doctors spend slightly more than one-half of their daily work shift “on the record,” and have to deal with forms and charts even visiting their patients, when data work accounts for approximately one-third of the time spent…”
ResultGREENHALGH, T., POTTS, H. W. W., WONG, G., BARK, P., & SWINGLEHURST, D. (2009) · The Milbank Quarterly · doi:10.1111/j.1468-0009.2009.00578.x · Cited by 541
Tai-Seale, M., Olson, C. W., Li, J., Chan, A. S., Morikawa, C., Durbin, M., Wang, W., & Luft, H. S. (2017) · Health Affairs · doi:10.1377/hlthaff.2016.0811 · Cited by 379 · cited 1×
“A recent study 3 analyzed the usage logs of almost one million patients’ electronic records by 471 primary care physicians and found that physicians split time evenly between seeing patients and documenting activities.”
BackgroundTemporality in Medical Work: Time also Matters
Reddy, M. C., Dourish, P., & Pratt, W. (2006) · Computer Supported Cooperative Work (CSCW) · doi:10.1007/s10606-005-9010-z · Cited by 164 · cited 1×
“…acknowledged in the specialist literature and widely discussed in many studies from Garfinkle’s work in 2008 to more recent ones: 14,15 different readers of the same record can depict to themselves different narratives 16 of the patient’s illness trajectory, 17 also according to their knowledge.”
BackgroundThe Use of Medical Scribes in Health Care Settings: A Systematic Review and Future Directions
Shultz, C. G., & Holmstrom, H. L. (2015) · The Journal of the American Board of Family Medicine · doi:10.3122/jabfm.2015.03.140224 · Cited by 139 · cited 1×
“…training sessions to improve the coders’ acquaintance with classification schemas, coding standards, and compilation conventions; and, finally, hiring (or asking for) medical assistants, or scribes, 49 who are specialized in data recording work and committed to high-quality record-keeping.”
BackgroundTime requirements for electronic health record use in an academic ophthalmology center
Read-Brown, S., Hribar, M. R., Reznick, L. G., Lombardi, L. H., Parikh, M., Chamberlain, W. D., Bailey, S. T., Wallace, J. B., Yackel, T. R., & Chiang, M. F. (2017) · JAMA Ophthalmology · doi:10.1001/jamaophthalmol.2017.4187 · Cited by 85 · cited 2×
“This information could be conveyed at the interface level in terms of α , A , or any other suitable measure or even just rendered in terms of color saturation of form backgrounds as depicted in Figures 5 and 6.”
BackgroundResultSwinglehurst, D., Roberts, C., & Greenhalgh, T. (2011) · Communication & Medicine · doi:10.1558/cam.v8i1.3 · Cited by 44 · cited 3×
“…record encounter” (EPRE) to denote medical interactions with a further interlocutor (i.e. the electronic medical record) other than the patient 5 and quantified that residents have approximately 10 EPREs daily, which last little more than half an hour each, for a total of 5 h in data entry…”
BackgroundThe Reliability of Clinical Diagnoses: State of the Art
Kraemer, H. C. (2014) · Annual Review of Clinical Psychology · doi:10.1146/annurev-clinpsy-032813-153739 · Cited by 39 · cited 1×
“32 In this study, to assess the agreement with which the clinicians reported the cases, we chose two common measures from the specialist literature.”
Method
Cited by
Toward a Perspectivist Turn in Ground Truthing for Predictive Computing
Cabitza, F., Campagner, A., & Basile, V. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i6.25840 · Cited by 114
“In fact, real-world settings show that disagreement is unavoidable and essentially irreducible, especially when the objects to classify are so complex that most of the raters can actually get them wrong, and the real experts are a minority (Basile 2021; Cabitza et al. 2019); or when the objects are so ambiguous, as it…”
BackgroundResultInfluentialSelf-citationBridging the “last mile” gap between AI implementation and operation: “data awareness” that matters
Cabitza, F., Campagner, A., & Balsano, C. (2020) · Annals of Translational Medicine · doi:10.21037/atm.2020.03.63 · Cited by 113
“In particular, we proposed highlighting the fields that presented low IRR scores during adoption, in a way that is not too different from that shown in Figure 3 (28).”
BackgroundResultSelf-citationCabitza, F., & Zeitoun, J.-D. (2019) · Annals of Translational Medicine · doi:10.21037/atm.2019.04.07 · Cited by 72
“Or rather, will the AI be unable to provide an accurate diagnosis in any case where some data are missing, incomplete or less than totally accurate or where the observer variability is high (19)?”
BackgroundSelf-citationInvisible to Machines: Designing AI that Supports Vision Work in Radiology
Anichini, G., Natali, C., & Cabitza, F. (2024) · Computer Supported Cooperative Work (CSCW) · doi:10.1007/s10606-024-09491-0 · Cited by 18
“These solutions are informed by the concept of Frictional AI: an AI design principle that leverages programmed inefficiencies (Cabitza et al 2019a) to stimulate human cognitive activation and mitigate overreliance on automated systems (Cab-itza et al. 2024).”
Self-citation
Metadata
- Authors
- Cabitza, F., Locoro, A., Alderighi, C., Rasoini, R., Compagnone, D., & Berjano, P.
- Venue
- Health Informatics Journal
- Publisher
- SAGE Publications
- Published
- 2019-01-22
- Volume
- 25
- Issue
- 3
- Pages
- 475-490
- ISSN
- 1355-4743 (ISSN-L)
- 1460-4582
- 1741-2811
- OpenAlex
- W2913345658
- Cited by
- 31
- References
- 10
- FWCI
- 3.84
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.