Exploring healthcare professionals’ perceptions of artificial intelligence: Validating a questionnaire using the e-Delphi method
Open accessThe DOI matches its Crossref record; 23 of the 23 references it cites were found in a registry or an index.
Abstract
Objective: The aim of this study was to draw upon the collective knowledge of experts in the fields of health and technology to develop a questionnaire that measured healthcare professionals’ perceptions of Artificial Intelligence (AI). Methods: The panel for this study were carefully selected participants who demonstrated an interest and/or involvement in AI from the fields of health or information technology. Recruitment was accomplished via email which invited the panel member to participate and included study and consent information. Data were collected from three rounds in the form of an online survey, an online group meeting and email communication. A 75% median threshold was used to define consensus. Results: Between January and March 2019, five healthcare professionals and three IT experts participated in three rounds of study to reach consensus on the structure and content of the questionnaire. In Round 1 panel members identified issues about general understanding of AI and achieved consensus on nine draft questionnaire items. In Round 2 the panel achieved consensus on demographic questions and comprehensive group discussion resulted in the development of two further questionnaire items for inclusion. In a final e-Delphi round, a draft of the final questionnaire was distributed via email to the panel members for comment. No further amendments were put forward and 100% consensus was achieved. Conclusion: A modified e-Delphi method was used to validate and develop a questionnaire to explore healthcare professionals’ perceptions of AI. The e-Delphi method was successful in achieving consensus from an interdisciplinary panel of experts from health and IT. Further research is recommended to test the reliability of this questionnaire.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Perceived usefulness, perceived ease of use, and user acceptance of information technology
Davis, F. D. (1989) · MIS Quarterly · doi:10.2307/249008 · Cited by 68,483
The potential for artificial intelligence in healthcare
Davenport, T., & Kalakota, R. (2019) · Future Healthcare Journal · doi:10.7861/futurehosp.6-2-94 · Cited by 3,800 · cited 1×
“It is not well understood how much AI technology currently exists in the healthcare setting, although estimates are that it is currently used in diagnosis and treatment recommendations, patient engagement and adherence, and administrative activities.(68) A connection between technology perception and use is thought to…”
BackgroundDiamond, I. R., Grant, R. C., Feldman, B. M., Pencharz, P. B., Ling, S. C., Moore, A. M., & Wales, P. W. (2014) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2013.12.002 · Cited by 3,740
The role of artificial intelligence in achieving the Sustainable Development Goals
Vinuesa, R., Azizpour, H., Leite, I., Balaam, M., Dignum, V., Domisch, S., Felländer, A., Langhans, S. D., Tegmark, M., & Fuso Nerini, F. (2020) · Nature Communications · doi:10.1038/s41467-019-14108-y · Cited by 3,294
Artificial intelligence in medicine
Hamet, P., & Tremblay, J. (2017) · Metabolism · doi:10.1016/j.metabol.2017.01.011 · Cited by 2,334
Boulkedid, R., Abdoul, H., Loustau, M., Sibony, O., & Alberti, C. (2011) · PLoS ONE · doi:10.1371/journal.pone.0020476 · Cited by 2,045 · cited 1×
“Traditionally, the Delphi method uses a paper-based questionnaire to collect data from participants, however digital methods, called e-Delphi methods, are now being used.(50,51) The use of electronic and internet-based questionnaires allows for a faster response time, points of anonymity and a reduction of resource co…”
MethodConsensus measurement in Delphi studies
von der Gracht, H. A. (2012) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2012.04.013 · Cited by 1,831
Yardley, L., Morrison, L., Bradbury, K., & Muller, I. (2015) · Journal of Medical Internet Research · doi:10.2196/jmir.4055 · Cited by 1,442
Chan, K. S., & Zary, N. (2019) · JMIR Medical Education · doi:10.2196/13930 · Cited by 877 · cited 1×
“This could be really interesting” 65–67 These studies acknowledge that understanding of AI is limited amongst medical students and that deciding on the content of education is challenging, but training will be necessary to realise the full capacity of this technology.”
BackgroundNadarzynski, T., Miles, O., Cowie, A., & Ridge, D. (2019) · DIGITAL HEALTH · doi:10.1177/2055207619871808 · Cited by 810
Patient care information systems and health care work: a sociotechnical approach
Berg, M. (1999) · International Journal of Medical Informatics · doi:10.1016/s1386-5056(99)00011-8 · Cited by 786
Medical students' attitude towards artificial intelligence: a multicentre survey
Pinto dos Santos, D., Giese, D., Brodehl, S., Chon, S. H., Staab, W., Kleinert, R., Maintz, D., & Baeßler, B. (2019) · European Radiology · doi:10.1007/s00330-018-5601-1 · Cited by 754 · cited 1×
“This could be really interesting” 65–67 These studies acknowledge that understanding of AI is limited amongst medical students and that deciding on the content of education is challenging, but training will be necessary to realise the full capacity of this technology.”
BackgroundAdapting to Artificial Intelligence: Radiologists and Pathologists as Information Specialists
Jha, S., & Topol, E. J. (2016) · JAMA · doi:10.1001/jama.2016.17438 · Cited by 622
Google DeepMind and healthcare in an age of algorithms
Powles, J., & Hodson, H. (2017) · Health and Technology · doi:10.1007/s12553-017-0179-1 · Cited by 511 · cited 1×
“contribute to the improvement of patient outcomes, safety and care.(8,9) Public perception of AI has changed in the last 10 years, with concern arising around its ethical implications and the lack of expected progress, but also more optimistic attitudes about AI hopes for healthcare and a focus on its inclusion in edu…”
BackgroundSit, C., Srinivasan, R., Amlani, A., Muthuswamy, K., Azam, A., Monzon, L., & Poon, D. S. (2020) · Insights into Imaging · doi:10.1186/s13244-019-0830-7 · Cited by 507 · cited 2×
“35 36-40 A questionnaire that measures health-care professionals’ perception of AI more broadly does not yet exist.”
BackgroundLong-Term Trends in the Public Perception of Artificial Intelligence
Fast, E., & Horvitz, E. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.10635 · Cited by 467 · cited 1×
“Public perception of AI has changed in the last 10 years, with concern arising around its ethical implications and the lack of expected progress, but also more optimistic attitudes about AI hopes for healthcare and a focus on its inclusion in education.(10) Research into the merits of AI technology in healthcare is in…”
BackgroundArtificial Intelligence: American Attitudes and Trends
Zhang, B., & Dafoe, A. (2019) · SSRN Electronic Journal · doi:10.2139/ssrn.3312874 · Cited by 464 · cited 1×
“57 Healthcare workforce perceptions will be a key factor in determining successful implementation and will impact future societal applications of AI in healthcare.”
BackgroundBlease, C., Kaptchuk, T. J., Bernstein, M. H., Mandl, K. D., Halamka, J. D., & DesRoches, C. M. (2019) · Journal of Medical Internet Research · doi:10.2196/12802 · Cited by 346
Fan, W., Liu, J., Zhu, S., & Pardalos, P. M. (2020) · Annals of Operations Research · doi:10.1007/s10479-018-2818-y · Cited by 346 · cited 2×
“28,29 The difficulty lies in finding an appropriate way to understand all stakeholder perceptions when implementing new technology like AI into healthcare.”
BackgroundLiyanage, H., Liaw, S.-T., Jonnagaddala, J., Schreiber, R., Kuziemsky, C., Terry, A. L., & de Lusignan, S. (2019) · Yearbook of Medical Informatics · doi:10.1055/s-0039-1677901 · Cited by 200
Artificial Intelligence-Assisted Online Social Therapy for Youth Mental Health
D'Alfonso, S., Santesteban-Echarri, O., Rice, S., Wadley, G., Lederman, R., Miles, C., Gleeson, J., & Alvarez-Jimenez, M. (2017) · Frontiers in Psychology · doi:10.3389/fpsyg.2017.00796 · Cited by 192 · cited 1×
“Machine learning and deep learning techniques are being tested in a wide range of areas within health, for example medical image analysis,(4) disease epidemic surveillance,(5) pathology classification(6) and treatment support in community healthcare settings.(7) The capabilities of AI are significant and compelling, a…”
BackgroundShinners, L., Aggar, C., Grace, S., & Smith, S. (2020) · Health Informatics Journal · doi:10.1177/1460458219874641 · Cited by 148
Self-citationBlease, C., Locher, C., Leon-Carlyle, M., & Doraiswamy, M. (2020) · DIGITAL HEALTH · doi:10.1177/2055207620968355 · Cited by 121 · cited 1×
“A qualitative study measuring psychiatrists perceptions of AI role replacement, 32 found that their views were divergent about the value and impact of AI, but that it would never replace the relational aspects of psychiatric care.”
Background
Cited by
Ahmed, Z., Bhinder, K. K., Tariq, A., Tahir, M. J., Mehmood, Q., Tabassum, M. S., Malik, M., Aslam, S., Asghar, M. S., & Yousaf, Z. (2022) · Annals of Medicine & Surgery · doi:10.1016/j.amsu.2022.103493 · Cited by 208
“Machine learning and deep learning are two subsets of AI that are being explored globally in the health sector [6].”
BackgroundPotential of large language models in health care: Delphi study
Denecke, K., May, R., LLMHealthGroup, & Rivera Romero, O. (2024) · Journal of Medical Internet Research · doi:10.2196/52399 · Cited by 111
Are physicians and medical students ready for artificial intelligence applications in healthcare?
AlZaabi, A., AlMaskari, S., & AalAbdulsalam, A. (2023) · DIGITAL HEALTH · doi:10.1177/20552076231152167 · Cited by 93
Catalina, Q. M., Fuster-Casanovas, A., Vidal-Alaball, J., Escalé-Besa, A., Marin-Gomez, F. X., Femenia, J., & Solé-Casals, J. (2023) · DIGITAL HEALTH · doi:10.1177/20552076231180511 · Cited by 71
Application of artificial intelligence in medical technologies: a systematic review of main trends.
Bitkina, O. V., Park, J., & Kim, H. K. (2023) · Digital Health · doi:10.1177/20552076231189331 · Cited by 66
Metadata
- Authors
- Shinners, L., Aggar, C., Grace, S., & Smith, S.
- Venue
- DIGITAL HEALTH
- Publisher
- SAGE Publications
- Published
- 2021-01
- Volume
- 7
- Pages
- 20552076211003433-20552076211003433
- ISSN
- 2055-2076 (ISSN-L)
- OpenAlex
- W3136425357
- Cited by
- 62
- References
- 23
- FWCI
- 2.15
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.