The ALFA (Activity Log Files Aggregation) toolkit: a method for precise observation of the consultation
Open accessAbstract
BACKGROUND: There is a lack of tools to evaluate and compare Electronic patient record (EPR) systems to inform a rational choice or development agenda. OBJECTIVE: To develop a tool kit to measure the impact of different EPR system features on the consultation. METHODS: We first developed a specification to overcome the limitations of existing methods. We divided this into work packages: (1) developing a method to display multichannel video of the consultation; (2) code and measure activities, including computer use and verbal interactions; (3) automate the capture of nonverbal interactions; (4) aggregate multiple observations into a single navigable output; and (5) produce an output interpretable by software developers. We piloted this method by filming live consultations (n = 22) by 4 general practitioners (GPs) using different EPR systems. We compared the time taken and variations during coded data entry, prescribing, and blood pressure (BP) recording. We used nonparametric tests to make statistical comparisons. We contrasted methods of BP recording using Unified Modeling Language (UML) sequence diagrams. RESULTS: We found that 4 channels of video were optimal. We identified an existing application for manual coding of video output. We developed in-house tools for capturing use of keyboard and mouse and to time stamp speech. The transcript is then typed within this time stamp. Although we managed to capture body language using pattern recognition software, we were unable to use this data quantitatively. We loaded these observational outputs into our aggregation tool, which allows simultaneous navigation and viewing of multiple files. This also creates a single exportable file in XML format, which we used to develop UML sequence diagrams. In our pilot, the GP using the EMIS LV (Egton Medical Information Systems Limited, Leeds, UK) system took the longest time to code data (mean 11.5 s, 95% CI 8.7-14.2). Nonparametric comparison of EMIS LV with the other systems showed a significant difference, with EMIS PCS (Egton Medical Information Systems Limited, Leeds, UK) (P = .007), iSoft Synergy (iSOFT, Banbury, UK) (P = .014), and INPS Vision (INPS, London, UK) (P = .006) facilitating faster coding. In contrast, prescribing was fastest with EMIS LV (mean 23.7 s, 95% CI 20.5-26.8), but nonparametric comparison showed no statistically significant difference. UML sequence diagrams showed that the simplest BP recording interface was not the easiest to use, as users spent longer navigating or looking up previous blood pressures separately. Complex interfaces with free-text boxes left clinicians unsure of what to add. CONCLUSIONS: The ALFA method allows the precise observation of the clinical consultation. It enables rigorous comparison of core elements of EPR systems. Pilot data suggests its capacity to demonstrate differences between systems. Its outputs could provide the evidence base for making more objective choices between systems.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Established methodology was used for multi-channel video recordings of consultations.
“We used established methodology to undertake multi-channel video (MCV) recordings of consultations.[42][43][44]”
From ‘Too much, too late’: mixed methods multi-channel video recording study of computerized decision support systems and GP prescribing (2013) · cites this paper as [43]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Human-Computer Interaction (3rd Edition)
Dix, A. J., Finlay, J., Abowd, G. D., & Beale, R. (2003) · Prentice-Hall, Inc eBooks · Cited by 497 · see citing works
Evaluation of health information systems—problems and challenges
Ammenwerth, E., Gräber, S., Herrmann, G., Bürkle, T., & König, J. (2003) · International Journal of Medical Informatics · doi:10.1016/s1386-5056(03)00131-x · Cited by 414 · see citing works
Layered representations for learning and inferring office activity from multiple sensory channels
Oliver, N., Garg, A., & Horvitz, E. (2004) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2004.02.004 · Cited by 316 · see citing works
Behaviour and information technology
(1983) · Design Studies · doi:10.1016/0142-694x(83)90017-0 · Cited by 266 · see citing works
Four rules for the reinvention of health care
Coiera, E. W. (2004) · BMJ · doi:10.1136/bmj.328.7449.1197 · Cited by 221 · see citing works
Haux, R. (2006) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2006.05.045 · Cited by 86 · see citing works
Westbrook, J. I., Braithwaite, J., Iedema, R. A. M., & Coiera, E. W. (2004) · Studies in health technology and informatics · doi:10.3233/978-1-60750-949-3-1323 · Cited by 46 · see citing works
Variation in clinical coding lists in UK general practice: a barrier to consistent data entry?
Waize, T., Anandarajah, S., Dhoul, N., & de Lusignan, S. (2007) · Journal of Innovation in Health Informatics · doi:10.14236/jhi.v15i3.652 · Cited by 41 · see citing works
de Lusignan, S., Teasdale, S., Little, D., Zapp, J., Zuckerman, A. E., Bates, D. W., & Steele, A. (2004) · Journal of Innovation in Health Informatics · doi:10.14236/jhi.v12i4.133 · Cited by 33 · see citing works
Theadom, A., de Lusignan, S., Wilson, E., & Chan, T. (2003) · Journal of Innovation in Health Informatics · doi:10.14236/jhi.v11i3.563 · Cited by 29 · see citing works
A gaze and speech multimodal interface
Zhang, Q., Imamiya, A., Mao, X., & Go, K. (2004) · IEEE International Conference on Distributed Computing Systems Workshops (ICDCSW) · doi:10.1109/icdcsw.2004.1284033 · Cited by 24 · see citing works
The computer in the consulting room: a psychological framework
FITTER, M. J., & CRUICKSHANK, P. J. (1982) · Behaviour and Information Technology · doi:10.1080/01449298208914438 · Cited by 22 · see citing works
Leong, A., Koczan, P., de Lusignan, S., & Sheeler, I. (2006) · Medical Informatics and the Internet in Medicine · doi:10.1080/14639230600991668 · Cited by 17 · see citing works
López, D. M., & Blobel, B. G. M. E. (2007) · PubMed · Cited by 11 · see citing works
de Lusignan, S., Wells, S., Russell, C., BEVINGTON, W. P., & Arrowsmith, P. (2002) · Medical Informatics and the Internet in Medicine · doi:10.1080/1463923031000081603 · Cited by 10 · see citing works
Refsum, C., Kumarapeli, P., Gunaratne, A. V. C., Dodds, R., Hasan, A., & de Lusignan, S. (2008) · Journal of Innovation in Health Informatics · doi:10.14236/jhi.v16i2.683 · Cited by 10 · see citing works
Low-cost three-channel video for assessment of the clinical consultation.
Sheeler, I., Koczan, P., Wallage, W., & de Lusignan, S. (2007) · PubMed · Cited by 8 · see citing works
Kumarapeli, P., de Lusignan, S., Koczan, P., Jones, B., & Sheeler, I. (2008) · Journal of Innovation in Health Informatics · doi:10.14236/jhi.v15i4.665 · Cited by 8 · see citing works
de Lusignan, S., Wilson, E., Dyble, A., Grant, T., Theadom, A., & Chan, T. (2003) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-3-12 · Cited by 8 · see citing works
Moulene, M. V., de Lusignan, S., Freeman, G. K., van Vlymen, J. N., Sheeler, I., Singleton, A., & Kumarapeli, P. (2007) · PubMed · Cited by 6 · see citing works
Pflug, B., Kumarapeli, P., van Vlymen, J. N., Chan, T., Ammenwerth, E., & de Lusignan, S. (2008) · Research Repository (Kingston University London) · Cited by 3 · see citing works
Progress and challenges of ubiquitous informatics in health care.
France, F. H. R. (2006) · PubMed · Cited by 2 · see citing works
The use of silence in the computer mediated consultation
Bhanahbai, H., Kumarapeli, P., & de Lusignan, S. (2008) · Research Repository (Kingston University London) · Cited by 1 · see citing works
Flagged first, then by citations
Research works printed but not linked: 4
- [17]Theadom Alice de Lusignan Simon Wilson Eben Chan Tom Using three-channel video to evaluate the impact of the use of the computer on the patient-centredness of the general practice consultation Inform Prim Care 2003 11 3 149 56 14680538 14680538
- [18]Sheeler Ian Koczan Phil Wallage William de Lusignan Simon Low-cost three-channel video for assessment of the clinical consultation Inform Prim Care 2007 15 1 25 31 17612472 17612472
- [22]Kumarapeli P Dodds R S de Lusignan S Observing and Modelling the Core Computer-based Activities of the General Practice Consultation Using Unified Modelling Language (UML) Proceedings of European Federation for Medical Informatics, – Special Topic conference (EFMI STC2007) 2007 EFMI STC2007 30 May - 1 Jun, 2007 Brijuni, Croatia Akademische Verlagsgesellschaft Aka (Aka-Verlag), Berlin and IOS, Amsterdam 43 9
- [24]Chapter 2: Human-Computer Interaction ACM SIGCHI Curricula for Human-Computer Interaction http://sigchi.org/cdg/cdg2.html
Cited by
Sorted by citations
Asan, O., & Montague, E. N. H. (2014) · Journal of Innovation in Health Informatics · doi:10.14236/jhi.v21i4.72 · Cited by 158 · see citing works
Kumarapeli, P., & de Lusignan, S. (2012) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2012-001081 · Cited by 81 · see citing works
Drennan, V. M., Halter, M., Brearley, S. A., Carneiro, W., Gabe, J., Gage, H., Grant, R., Joly, L., & de Lusignan, S. (2014) · Health Services and Delivery Research · doi:10.3310/hsdr02160 · Cited by 71 · see citing works
Hayward, J., Thomson, F., Milne, H., Buckingham, S., Sheikh, A., Fernando, B., Cresswell, K., Williams, R., & Pinnock, H. (2013) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2012-001484 · Cited by 48 · see citing works
An Evaluation Framework for Defining the Contributions of Telestration in Surgical Telementoring
Budrionis, A., Augestad, K. M., PATEL, H. R. H., & Bellika, J. G. (2013) · Interactive Journal of Medical Research · doi:10.2196/ijmr.2611 · Cited by 38 · see citing works
de Lusignan, S., Liaw, S., Dedman, D., Khunti, K. K., Sadek, K., & Jones, S. (2015) · Journal of Innovation in Health Informatics · doi:10.14236/jhi.v22i2.79 · Cited by 23 · see citing works
Vaziri, A., Connor, E. F., Shepherd, I. J., Jones, R., Chan, T., & de Lusignan, S. (2009) · Journal of Innovation in Health Informatics · doi:10.14236/jhi.v17i3.732 · Cited by 20 · see citing works
The pattern of silent time in the clinical consultation: an observational multichannel video study
Gibbings-Isaac, D., Iqbal, M., Tahir, M. I., Kumarapeli, P., & de Lusignan, S. (2012) · Family Practice · doi:10.1093/fampra/cms001 · Cited by 20 · see citing works
Physician Associate and General Practitioner Consultations: A Comparative Observational Video Study
de Lusignan, S., McGovern, A. P., Tahir, M. A., Hassan, S., Jones, S., Halter, M., Joly, L., & Drennan, V. M. (2016) · PLoS ONE · doi:10.1371/journal.pone.0160902 · Cited by 20 · see citing works
Reporting Observational Studies of the Use of Information Technology in the Clinical Consultation
Pearce, C. M., Kumarapeli, P., Stavropoulou, C., Kushniruk, A. W., Sheikh, A., Shachak, A., Mendis, K., & de Lusignan, S. (2011) · Yearbook of Medical Informatics · doi:10.1055/s-0038-1638735 · Cited by 14 · see citing works
de Lusignan, S., McGovern, A. P., Hinton, W., Whyte, M., Munro, N. M., Williams, E. D., Marcu, A., Williams, J., Ferreira, F. M., Mount, J., et al. (2022) · Diabetes Therapy · doi:10.1007/s13300-022-01306-z · Cited by 13 · see citing works
Pflug, B., Kumarapeli, P., van Vlymen, J., Ammenwerth, E., & de Lusignan, S. (2010) · Informatics for Health and Social Care · doi:10.3109/17538150903358693 · Cited by 8 · see citing works
de Lusignan, S., Kumarapeli, P., Debar, S., Kushniruk, A. W., & Pearce, C. (2009) · Studies in health technology and informatics · doi:10.3233/978-1-60750-044-5-1017 · Cited by 8 · see citing works
“How long does it take?” A mixed methods evaluation of computer-related work in GP consultations
Hayward, J., Buckingham, S. C., Thomson, F., Milne, H., Sheikh, A., Fernando, B., Cresswell, K. M., Williams, R., & Pinnock, H. (2015) · Journal of Innovation in Health Informatics · doi:10.14236/jhi.v22i4.95 · Cited by 8 · see citing works
Hall, S., Kulendran, M., Sadek, A.-R., Green, S. A., & de Lusignan, S. (2011) · Family Practice · doi:10.1093/fampra/cmq095 · Cited by 7 · see citing works
Intensification to injectable therapy in type 2 diabetes: mixed methods study (protocol)
de Lusignan, S., Hinton, W., Konstantara, E., Munro, N. M., Whyte, M., Mount, J., & Feher, M. D. (2019) · BMC Health Services Research · doi:10.1186/s12913-019-4112-3 · Cited by 6 · see citing works
The electronic patient record : a linguistic ethnographic study in general practice
Swinglehurst, D. (2012) · Cited by 5 · see citing works
Appleton, A., Sadek, K., Dawson, I. G. J., & de Lusignan, S. (2013) · Journal of Innovation in Health Informatics · doi:10.14236/jhi.v20i4.13 · Cited by 4 · see citing works
de Lusignan, S., Pearce, C. M., Kumarapeli, P., Stavropoulos, C., Kushniruk, A. W., Sheikh, A., Shachak, A., & Mendis, K. (2011) · Yearbook of Medical Informatics · Cited by 2 · see citing works
The use of event logs for collaborative practices reflection
Babič, F., Wagner, J., & Paralič, J. (2012) · International Journal of Intelligent Information and Database Systems · doi:10.1504/ijiids.2012.049313 · Cited by 1 · see citing works
Kinley, E., Pinnock, H., Steed, L., & McClatchey, K. (2024) · Health Expectations · doi:10.1111/hex.70100 · Cited by 1 · see citing works
Glew, S., Ford, E. M., & Smith, H. E. (2018) · International Journal for Population Data Science · doi:10.23889/ijpds.v3i1.430 · Cited by 1 · see citing works
[Signs and diagnosis of occlusal disorders].
Rigolet, D. A. (1976) · PubMed
- Authors
- de Lusignan, S., Kumarapeli, P., Chan, T., Pflug, B., van Vlymen, J., Jones, B., & Freeman, G. K.
- Affiliations
- St George's, University of London (GB)
- Kingston University London (GB)
- Venue
- Journal of Medical Internet Research
- Publisher
- JMIR Publications Inc.
- Published
- 2008-09-08
- Volume
- 10
- Issue
- 4
- Pages
- e27
- ISSN
- 1438-8871 (ISSN-L)
- PMC
- PMC2629369
- OpenAlex
- W1977276812
- Cited by
- 24
- References
- 23
- Citations by year
- 2024: 1 · 2022: 1 · 2019: 1 · 2018: 1 · 2016: 1 · 2015: 2 · 2014: 2 · 2013: 3 · 2012: 4
- FWCI
- 5.13
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- Journal of Medical Internet Research (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Surrey Research Insight Open Access (The University of Surrey) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Research Repository (Kingston University London) (submittedVersion)
- Topics
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Technology and Human Factors in Education and Health
- Health Sciences › Health Professions › Health Information Management › Medical Coding and Health Information
- Keywords
- electronic health records, nonverbal interaction, data entry time, blood pressure measurement, sequence diagrams, EMIS, general practitioners, XML data export
- MeSH terms
- Attitude to Computers, Computers, Family Practice, Humans, Programming Languages, Referral and Consultation, Sensitivity and Specificity, Software, User-Computer Interface, Medical Records Systems, Computerized
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
