Determinants of Laypersons’ Trust in Medical Decision Aids: Randomized Controlled Trial
Open accessPublished version of a preprint
Abstract
Background: Symptom checker apps are patient-facing decision support systems aimed at providing advice to laypersons on whether, where, and how to seek health care (disposition advice). Such advice can improve laypersons’ self-assessment and ultimately improve medical outcomes. Past research has mainly focused on the accuracy of symptom checker apps’ suggestions. To support decision-making, such apps need to provide not only accurate but also trustworthy advice. To date, only few studies have addressed the question of the extent to which laypersons trust symptom checker app advice or the factors that moderate their trust. Studies on general decision support systems have shown that framing automated systems (anthropomorphic or emphasizing expertise), for example, by using icons symbolizing artificial intelligence (AI), affects users’ trust. Objective: This study aims to identify the factors influencing laypersons’ trust in the advice provided by symptom checker apps. Primarily, we investigated whether designs using anthropomorphic framing or framing the app as an AI increases users’ trust compared with no such framing. Methods: Through a web-based survey, we recruited 494 US residents with no professional medical training. The participants had to first appraise the urgency of a fictitious patient description (case vignette). Subsequently, a decision aid (mock symptom checker app) provided disposition advice contradicting the participants’ appraisal, and they had to subsequently reappraise the vignette. Participants were randomized into 3 groups: 2 experimental groups using visual framing (anthropomorphic, 160/494, 32.4%, vs AI, 161/494, 32.6%) and a neutral group without such framing (173/494, 35%). Results: Most participants (384/494, 77.7%) followed the decision aid’s advice, regardless of its urgency level. Neither anthropomorphic framing (odds ratio 1.120, 95% CI 0.664-1.897) nor framing as AI (odds ratio 0.942, 95% CI 0.565-1.570) increased behavioral or subjective trust (P=.99) compared with the no-frame condition. Even participants who were extremely certain in their own decisions (ie, 100% certain) commonly changed it in favor of the symptom checker’s advice (19/34, 56%). Propensity to trust and eHealth literacy were associated with increased subjective trust in the symptom checker (propensity to trust b=0.25; eHealth literacy b=0.2), whereas sociodemographic variables showed no such link with either subjective or behavioral trust. Conclusions: Contrary to our expectation, neither the anthropomorphic framing nor the emphasis on AI increased trust in symptom checker advice compared with that of a neutral control condition. However, independent of the interface, most participants trusted the mock app’s advice, even when they were very certain of their own assessment. Thus, the question arises as to whether laypersons use such symptom checkers as substitutes rather than as aids in their own decision-making. With trust in symptom checkers already high at baseline, the benefit of symptom checkers depends on interface designs that enable users to adequately calibrate their trust levels during usage. Trial Registration: Deutsches Register Klinischer Studien DRKS00028561; https://tinyurl.com/rv4utcfb (retrospectively registered).
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Confidence in the initial decision is a key factor influencing decision-making in high-stakes environments such as medical diagnosis.
“In addition to these psychological factors and medical experience, confidence in the initial decision is a key factor influencing decision-making processes in high-stakes environments, such as medical diagnosis [ 23 - 25 ].”
From Psychological Factors Influencing Appropriate Reliance on AI-enabled Clinical Decision Support Systems: Experimental Web-Based Study Among Dermatologists (2025) · cites this paper as [24]
When humans are confident in their own decisions, they are less likely to change after receiving advice.
“When humans are confident in their own decisions, they are less likely to change it after receiving advice [ 23 , 24 ].”
From Psychological Factors Influencing Appropriate Reliance on AI-enabled Clinical Decision Support Systems: Experimental Web-Based Study Among Dermatologists (2025) · cites this paper as [24]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Wickham, H., Averick, M., Bryan, J., Chang, W., McGowan, L., François, R., Grolemund, G., Hayes, A., Henry, L., Hester, J., et al. (2019) · Journal of Open Source Software · doi:10.21105/joss.01686 · Cited by 22,138 · see citing works
Bundesgesundheitsblatt - Gesundheitsforschung - Gesundheitsschutz
(2013) · Bundesgesundheitsblatt - Gesundheitsforschung - Gesundheitsschutz · doi:10.1007/103.1437-1588 · Cited by 11,769 · see citing works
Beyond the Turk: Alternative platforms for crowdsourcing behavioral research
Peer, E., Brandimarte, L., Samat, S., & Acquisti, A. (2017) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2017.01.006 · Cited by 2,957 · see citing works
Trust in Automation: Integrating Empirical Evidence on Factors That Influence Trust
Hoff, K. A., & Bashir, M. (2015) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1177/0018720814547570 · Cited by 2,817 · see citing works
eHEALS: The eHealth Literacy Scale
Norman, C. D., & Skinner, H. A. (2006) · Journal of Medical Internet Research · doi:10.2196/jmir.8.4.e27 · Cited by 2,680 · see citing works
Foundations for an Empirically Determined Scale of Trust in Automated Systems
Jian, J.-Y., Bisantz, A. M., & Drury, C. G. (2000) · International Journal of Cognitive Ergonomics · doi:10.1207/s15327566ijce0401_04 · Cited by 1,799 · see citing works
Trust, control strategies and allocation of function in human-machine systems
LEE, J., & MORAY, N. (1992) · Ergonomics · doi:10.1080/00140139208967392 · Cited by 1,459 · see citing works
Trust, self-confidence, and operators' adaptation to automation
Lee, J. D., & Moray, N. (1994) · International Journal of Human-Computer Studies · doi:10.1006/ijhc.1994.1007 · Cited by 932 · see citing works
Schaefer, K. E., Chen, J. Y. C., Szalma, J. L., & Hancock, P. A. (2016) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1177/0018720816634228 · Cited by 768 · see citing works
Parasuraman, R., Sheridan, T., & Wickens, C. (2008) · Journal of Cognitive Engineering and Decision Making · doi:10.1518/155534308x284417 · Cited by 729 · see citing works
Almost human: Anthropomorphism increases trust resilience in cognitive agents.
de Visser, E. J., Monfort, S. S., McKendrick, R., Smith, M. A. B., McKnight, P. E., Krueger, F., & Parasuraman, R. (2016) · Journal of Experimental Psychology: Applied · doi:10.1037/xap0000092 · Cited by 614 · see citing works
Evaluation of symptom checkers for self diagnosis and triage: audit study
Semigran, H. L., Linder, J. A., Gidengil, C., & Mehrotra, A. (2015) · BMJ · doi:10.1136/bmj.h3480 · Cited by 489 · see citing works
Artificial intelligence and machine learning for medical imaging: A technology review
Montero, A. M. B., Javaid, U., Valdés, G., Nguyen, D. T. D., Desbordes, P., Macq, B. M. M., Willems, S., Vandewinckele, L., Holmström, M., Löfman, F., et al. (2021) · Physica Medica · doi:10.1016/j.ejmp.2021.04.016 · Cited by 481 · see citing works
Jussupow, E., Spohrer, K., Heinzl, A., & Gawlitza, J. (2021) · Information Systems Research · doi:10.1287/isre.2020.0980 · Cited by 422 · see citing works
Expertise in context: human and machine
Feltovich, P. J., Ford, K. M., & Hoffman, R. R. (1997) · Cited by 286 · see citing works
Yee, N., Bailenson, J. N., & Rickertsen, K. (2007) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1240624.1240626 · Cited by 243 · see citing works
Chambers, D., Cantrell, A. J., Johnson, M., Preston, L., Baxter, S. K., Booth, A., & Turner, J. (2019) · BMJ Open · doi:10.1136/bmjopen-2018-027743 · Cited by 218 · see citing works
The diagnostic and triage accuracy of digital and online symptom checker tools: a systematic review
Wallace, W. J., Chan, C., Chidambaram, S., Hanna, L., Iqbal, F. M., Acharya, A., Normahani, P., Ashrafian, H., Markar, S. R., Sounderajah, V., et al. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00667-w · Cited by 189 · see citing works
Pak, R., Fink, N., Price, M. M., Bass, B., & Sturre, L. (2012) · Ergonomics · doi:10.1080/00140139.2012.691554 · Cited by 183 · see citing works
Meyer, A. N. D., Giardina, T. D., Spitzmueller, C., Shahid, U., Scott, T. M. T., & Singh, H. (2019) · Journal of Medical Internet Research · doi:10.2196/14679 · Cited by 162 · see citing works
Comparison of Physician and Computer Diagnostic Accuracy
Semigran, H. L., Levine, D. M., Nundy, S., & Mehrotra, A. (2016) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2016.6001 · Cited by 145 · see citing works
Jungmann, S. M., Klan, T., Kuhn, S., & Jungmann, F. (2019) · JMIR Formative Research · doi:10.2196/13863 · Cited by 127 · see citing works
Accuracy of online symptom checkers and the potential impact on service utilisation
Ceney, A., Tolond, S., Glowinski, A., Marks, B., Swift, S., & Palser, T. (2021) · PLOS ONE · doi:10.1371/journal.pone.0254088 · Cited by 125 · see citing works
Casey, L. S., Chandler, J., Levine, A. S., Proctor, A., & Strolovitch, D. Z. (2017) · SAGE Open · doi:10.1177/2158244017712774 · Cited by 122 · see citing works
Miller, S., Gilbert, S., Virani, V., & Wicks, P. (2020) · JMIR Human Factors · doi:10.2196/19713 · Cited by 117 · cited 1× · see citing works
“However, both findings stand in contrast to those indicating a low behavioral trust in symptom checkers; for example, Meyer et al [15] reported that only a minority of those advised by a symptom checker to visit the emergency department followed this advice, and Miller et al [61] found that most patients presenting to…”
Dr Google in the ED: Searching for online health information by adult emergency department patients
Cocco, A. M., Zordan, R., Taylor, D. M., Weiland, T. J., Dilley, S. J., Kant, J., Dombagolla, M., Hendarto, A., Lai, F., & Hutton, J. (2018) · Medical Journal of Australia · doi:10.5694/mja17.00889 · Cited by 114 · see citing works
Hill, M. G., Sim, M., & Mills, B. (2020) · Medical Journal of Australia · doi:10.5694/mja2.50600 · Cited by 114 · see citing works
The World is not Enough: Trust in Cognitive Agents
de Visser, E. J., Krueger, F. W., McKnight, P. E., Scheid, S., Smith, M. A. B., Chalk, S., & Parasuraman, R. (2012) · Proceedings of the Human Factors and Ergonomics Society Annual Meeting · doi:10.1177/1071181312561062 · Cited by 107 · see citing works
Triage Accuracy of Symptom Checker Apps: 5-Year Follow-up Evaluation
Schmieding, M. L., Kopka, M., Schmidt, K. F. R., Schulz-Niethammer, S., Balzer, F., & Feufel, M. A. (2022) · Journal of Medical Internet Research · doi:10.2196/31810 · Cited by 99 · see citing works
Self-citationBaumann, E., Czerwinski, F., Rosset, M., Seelig, M., & Suhr, R. (2020) · Bundesgesundheitsblatt - Gesundheitsforschung - Gesundheitsschutz · doi:10.1007/s00103-020-03192-x · Cited by 92 · see citing works
Beyond Dr. Google: the evidence on consumer-facing digital tools for diagnosis
Millenson, M. L., Baldwin, J. L., Zipperer, L., & Singh, H. (2018) · Diagnosis · doi:10.1515/dx-2018-0009 · Cited by 91 · see citing works
The Measurement of the Propensity to Trust Automation
Jessup, S. A., Schneider, T. R., Alarcon, G. M., Ryan, T. J., & Capiola, A. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-21565-1_32 · Cited by 90 · see citing works
Morse, K., Ostberg, N. P., Jones, V. G., & Chan, A. S. (2020) · Journal of Medical Internet Research · doi:10.2196/20549 · Cited by 86 · see citing works
Gottliebsen, K., & Petersson, G. (2020) · BMJ Health & Care Informatics · doi:10.1136/bmjhci-2019-100114 · Cited by 80 · see citing works
Self-triage for acute primary care via a smartphone application: Practical, safe and efficient?
Verzantvoort, N. C. M., Teunis, T., Verheij, T. J., & van der Velden, A. W. (2018) · PLoS ONE · doi:10.1371/journal.pone.0199284 · Cited by 77 · see citing works
Campbell, J. L., Fletcher, E., Britten, N., Green, C., Holt, T., Lattimer, V., Richards, D. A., Richards, S. H., Salisbury, C., Taylor, R. S., et al. (2015) · Health Technology Assessment · doi:10.3310/hta19130 · Cited by 71 · see citing works
Good advice is beyond all price, but what if it comes from a machine?
Hertz, N., & Wiese, E. (2019) · Journal of Experimental Psychology Applied · doi:10.1037/xap0000205 · Cited by 67 · see citing works
Association of Use of Online Symptom Checkers With Patients’ Plans for Seeking Care
Winn, A. N., Somai, M. M., Fergestrom, N. M., & Crotty, B. H. (2019) · JAMA Network Open · doi:10.1001/jamanetworkopen.2019.18561 · Cited by 67 · see citing works
Pak, R., McLaughlin, A., & Bass, B. (2014) · Ergonomics · doi:10.1080/00140139.2014.928750 · Cited by 59 · see citing works
Woodcock, C., Mittelstadt, B., Busbridge, D., & Blank, G. (2021) · Journal of Medical Internet Research · doi:10.2196/29386 · Cited by 57 · see citing works
Gutzwiller, R. S., Chiou, E. K., Craig, S. D., Lewis, C. M., Lematta, G. J., & Hsiung, C.-P. (2019) · Proceedings of the Human Factors and Ergonomics Society Annual Meeting · doi:10.1177/1071181319631201 · Cited by 54 · see citing works
Ćirković, A. (2020) · Journal of Medical Internet Research · doi:10.2196/18097 · Cited by 52 · see citing works
Foundations for an Empirically Determined Scale of Trust in Automated Systems
Jian, J.-Y., Bisantz, A. M., Drury, C. G., et al. (1998) · doi:10.21236/ada388787 · Cited by 49 · see citing works
Benchmarking Triage Capability of Symptom Checkers Against That of Medical Laypersons: Survey Study
Schmieding, M. L., Mörgeli, R., Schmieding, M. A. L., Feufel, M. A., & Balzer, F. (2021) · Journal of Medical Internet Research · doi:10.2196/24475 · Cited by 44 · see citing works
Self-citationTrust Me, I’m a Doctor – User Perceptions of AI-Driven Apps for Mobile Health Diagnosis
Baldauf, M., Fröehlich, P., & Endl, R. (2020) · International Conference on Mobile and Ubiquitous Multimedia (MUM) · doi:10.1145/3428361.3428362 · Cited by 42 · see citing works
Levine, D. M., & Mehrotra, A. (2021) · JAMA Network Open · doi:10.1001/jamanetworkopen.2021.3287 · Cited by 35 · see citing works
The World is not Enough: Trust in Cognitive Agents: (572172013-055)
de Visser, E. J., Krueger, F., McKnight, P., Scheid, S., Smith, M., Chalk, S., & Parasuraman, R. (2012) · PsycEXTRA Dataset · doi:10.1037/e572172013-055 · Cited by 34 · see citing works
Triage accuracy of online symptom checkers for Accident and Emergency Department patients
Yu, S. W. Y., Ma, A., Tsang, V. H. M., Chung, L. S. W., Leung, S. C., & Leung, L. (2019) · Hong Kong Journal of Emergency Medicine · doi:10.1177/1024907919842486 · Cited by 33 · see citing works
Incentivizing high quality crowdwork
Ho, C.-J., Slivkins, A., Suri, S., & Vaughan, J. W. (2016) · ACM SIGecom Exchanges · doi:10.1145/2904104.2904108 · Cited by 29 · see citing works
Donovan, E., Wilcox, C. R., Patel, S. V., Hay, A. D., Little, P. S., & Willcox, M. L. (2020) · British Journal of General Practice · doi:10.3399/bjgp20x708209 · Cited by 29 · see citing works
Relationship Between Trust and Usability in Virtual Environments: An Ongoing Study
Salanitri, D., Hare, C., Borsci, S., Lawson, G., Sharples, S., & Waterfield, B. C. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-20901-2_5 · Cited by 27 · see citing works
The Triage Capability of Laypersons: Retrospective Exploratory Analysis
Kopka, M., Feufel, M. A., Balzer, F., & Schmieding, M. L. (2022) · JMIR Formative Research · doi:10.2196/38977 · Cited by 22 · see citing works
Self-citationThe effects of pitch contour and flanging on trust in speaking cognitive agents
Muralidharan, L., de Visser, E. J., & Parasuraman, R. (2014) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/2559206.2581231 · Cited by 18 · see citing works
Gilbert, S., Fenech, M. E., Upadhyay, S., Wicks, P. J., & Novorol, C. (2021) · Australian Journal of Primary Health · doi:10.1071/py21032 · Cited by 17 · see citing works
Virtual, Augmented and Mixed Reality. Applications and Case Studies
(2019) · Lecture notes in computer science · doi:10.1007/978-3-030-21565-1 · Cited by 15 · see citing works
The uncertainty is killing me: Self-triage decision making and information availability
Cooper, A. A., & Humphreys, K. R. (2008) · E-Journal of Applied Psychology · doi:10.7790/ejap.v4i1.124 · Cited by 14 · see citing works
Under-Reliance on the Decision Aid: A Difference in Calibration and Attribution between Self and Aid
Kees Van Dongen, A., & van Maanen, P.-P. (2006) · Proceedings of the Human Factors and Ergonomics Society Annual Meeting · doi:10.1177/154193120605000304 · Cited by 12 · see citing works
Zou, X., Lv, C., & Zhang, J. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-49183-3_18 · Cited by 7 · see citing works
Feasibility of Web-Based Self-Triage by Parents of Children With Influenza-Like Illness
Price, R. A., Fagbuyi, D. B., Harris, R., Hanfling, D., Place, F., Taylor, T. B., & Kellermann, A. L. (2012) · Archives of Pediatrics and Adolescent Medicine · doi:10.1001/jamapediatrics.2013.1573 · Cited by 2 · see citing works
Under-reliance on the decision aid: A difference in calibration and attribution between self and aid
van Dongen, K., & van Maanen, P.-P. (2006) · PsycEXTRA Dataset · doi:10.1037/e577572012-004 · Cited by 2 · see citing works
Wempen, F. (2010) · doi:10.1002/9781118983959.ch24 · Cited by 1 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Psychological Traits and Appropriate Reliance: Factors Shaping Trust in AI
Küper, A., & Krämer, N. (2024) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2024.2348216 · Cited by 49 · see citing works
Küper, A., Lodde, G. C., Livingstone, E., Schadendorf, D., & Krämer, N. (2025) · Journal of Medical Internet Research · doi:10.2196/58660 · Cited by 31 · see citing works
“Previous studies [23,24,47] have found a similar relationship between confidence and following advice.”
Peven, K., Wickham, A. P., Wilks, O., Kaplan, Y. C., Marhol, A., Ahmed, S., Bamford, R., Cunningham, A. C., Prentice, C., Meczner, A., et al. (2023) · JMIR mHealth and uHealth · doi:10.2196/46718 · Cited by 9 · see citing works
- Authors
- Kopka, M., Schmieding, M. L., Rieger, T., Roesler, E., Balzer, F., & Feufel, M. A.
- Affiliations
- Humboldt-Universität zu Berlin (DE)
- Technische Universität Berlin (DE)
- Corresponding author
- Kopka, Marvin
- Venue
- JMIR Human Factors
- Publisher
- JMIR Publications Inc.
- Published
- 2022-05-03
- Volume
- 9
- Issue
- 2
- Pages
- e35219
- ISSN
- 2292-9495 (ISSN-L)
- PMC
- PMC9115664
- OpenAlex
- W4225396572
- Cited by
- 37
- References
- 62
- Citations by year
- 2026: 9 · 2025: 10 · 2024: 11 · 2023: 6 · 2022: 1
- FWCI
- 2.11
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- JMIR Human Factors (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Refubium (Universitätsbibliothek der Freien Universität Berlin) (submittedVersion, cc-by)
- Universitätsbibliothek der FU Berlin Hochschulschriftenstelle u. Dokumentenserver (cc-by)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › General Health Professions › Mobile Health and mHealth Applications
- Health Sciences › Health Professions › General Health Professions › Patient-Provider Communication in Healthcare
- Keywords
- visual framing, laypeople, digital health literacy, trust propensity, trust in automation, health information seeking, decision support system, case vignettes, mobile health apps
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
