The value of participatory development to support antimicrobial stewardship with a clinical decision support system
Open accessAbstract
Background Current clinical decision support systems (CDSSs) for antimicrobial stewardship programs (ASPs) are guideline- or expert-driven. They are focused on (clinical) content, not on supporting real-time workflow. Thus, CDSSs fail to optimally support prudent antimicrobial prescribing in daily practice. Our aim was to demonstrate why and how participatory development (involving end-users and other stakeholders) can contribute to the success of CDSSs in ASPs.Methods A mixed-methods approach was applied, combining scenario-based prototype evaluations (to support verbalization of work processes and out-of-the-box thinking) among 6 medical resident physicians with an online questionnaire (to cross-reference findings of the prototype evaluations) among 54 Dutch physicians.Results The prototype evaluations resulted in insight into the end-users and their way of working, as well as their needs and expectations. The online questionnaire that was distributed among a larger group of medical specialists, including lung and infection experts, complemented the findings of the prototype evaluations. It revealed a say/do problem concerning the unrecognized need of support for selecting diagnostic tests.Conclusions Low-fidelity prototypes of a technology allow researchers to get to know the end-users, their way of working, and their work context. Involving experts allows technology developers to continuously check the fit between technology and clinical practice. The combination enables the participatory development of technology to successfully support ASPs.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Multiple authors in the included studies indicated that usability tests should be conducted with a broad range of stakeholders, not only end users such as patients, but also caregivers, managers, technology developers, and experts on content and design.
“First, multiple authors indicated that usability tests should be conducted with a broad range of stakeholders: not only end users such as patients, but also caregivers, managers, technology developers, and experts on content and design”
From Methods for Human-Centered eHealth Development: Narrative Scoping Review (2022) · cites this paper as [131]
Multiple studies indicated that values or requirements of the to-be-developed eHealth technology can be used to guide usability tests.
“Third, in multiple studies, authors indicated that values or requirements of the to-be-developed eHealth technology can be used to guide usability tests.”
From Methods for Human-Centered eHealth Development: Narrative Scoping Review (2022) · cites this paper as [131]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Surviving Sepsis Campaign: International Guidelines for Management of Severe Sepsis and Septic Shock
Dellinger, R. P., Levy, M. M., Rhodes, A., Annane, D., Gerlach, H., Opal, S. M., Sevransky, J. E., Sprung, C. L., Douglas, I. S., Jaeschke, R., et al. (2013) · Critical Care Medicine · doi:10.1097/ccm.0b013e31827e83af · Cited by 9,694 · see citing works
Dellinger, R. P., Levy, M. M., Rhodes, A., Annane, D., Gerlach, H., Opal, S. M., Sevransky, J., Sprung, C. L., Douglas, I. S., Jaeschke, R., et al. (2013) · Intensive Care Medicine · doi:10.1007/s00134-012-2769-8 · Cited by 7,282 · see citing works
Lim, W. S. (2003) · Thorax · doi:10.1136/thorax.58.5.377 · Cited by 3,066 · see citing works
The Delphi technique: myths and realities
Powell, C. (2003) · Journal of Advanced Nursing · doi:10.1046/j.1365-2648.2003.02537.x · Cited by 2,611 · see citing works
Interventions to improve antibiotic prescribing practices for hospital inpatients
Davey, P. G., Brown, E. M., Charani, E., Fenelon, L. E., Gould, I. M., Holmes, A. H., Ramsay, C. R., Wiffen, P. J., & Wilcox, M. A. (2013) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd003543.pub3 · Cited by 1,872 · see citing works
Persuasive Systems Design: Key Issues, Process Model, and System Features
Oinas-Kukkonen, H., & Harjumaa, M. (2009) · Communications of the Association for Information Systems · doi:10.17705/1cais.02428 · Cited by 1,663 · see citing works
Antimicrobial Stewardship Programs in Health Care Systems
MacDougall, C., & Polk, R. E. (2005) · Clinical Microbiology Reviews · doi:10.1128/cmr.18.4.638-656.2005 · Cited by 705 · see citing works
Antibiotic prescribing in hospitals: a social and behavioural scientific approach
Hulscher, M., Grol, R. P. T. M., & van der Meer, J. W. M. (2010) · The Lancet Infectious Diseases · doi:10.1016/s1473-3099(10)70027-x · Cited by 365 · see citing works
Participatory IT Design: Designing for Business and Workplace Realities
Bødker, K., Kensing, F., & Simonsen, J. (2004) · Cited by 346 · see citing works
Kesselheim, A. S., Cresswell, K., Phansalkar, S., Bates, D. W., & Sheikh, A. (2011) · Health Affairs · doi:10.1377/hlthaff.2010.1111 · Cited by 277 · see citing works
Procalcitonin Levels Predict Bacteremia in Patients With Community-Acquired Pneumonia
Müller, F., Christ‐Crain, M., Bregenzer, T., Krause, M., Zimmerli, W., Müeller, B., & Schüetz, P. (2010) · CHEST Journal · doi:10.1378/chest.09-2920 · Cited by 256 · see citing works
interventions to improve antibiotic prescribing practices for hospital inpatients (updated protocol)
Davey, P. G., Scott, C., Brown, E. M., Charani, E., Michie, S., Ramsay, C. R., & Marwick, C. (2017) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd011236.pub2 · Cited by 217 · see citing works
Requirements development in scenario-based design
Carroll, J. M., Rosson, M. B., Chin, G., & Koenemann, J. (1998) · IEEE Transactions on Software Engineering · doi:10.1109/32.738344 · Cited by 112 · see citing works
Jarosławski, S., & Saberwal, G. (2014) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-14-1 · Cited by 110 · see citing works
Diagnosis and management of community-acquired pneumonia in adults.
Watkins, R., & Lemonovich, T. L. (2011) · PubMed · Cited by 110 · see citing works
Emerging infectious diseases: the role of social sciences
Janes, C. R., Corbett, K. K., Jones, J. H., & Trostle, J. (2012) · The Lancet · doi:10.1016/s0140-6736(12)61725-5 · Cited by 100 · see citing works
An Integrated Stewardship Model: Antimicrobial, Infection Prevention and Diagnostic (AID)
Dik, J.-W. H., Poelman, R., Friedrich, A. W., Panday, P. V. N., Foe, J. R. L. T., van Assen, S., van Gemert‐Pijnen, J. E. W. C., Niesters, H. G. M., Hendrix, R., & Sinha, B. N. M. (2015) · Future Microbiology · doi:10.2217/fmb.15.99 · Cited by 95 · see citing works
Self-citationParticipatory eHealth development to support nurses in antimicrobial stewardship
Wentzel, J., van Velsen, L., van Limburg, M., de Jong, N., Karreman, J., Hendrix, R., & van Gemert-Pijnen, J. J. E. W. C. (2014) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-14-45 · Cited by 73 · see citing works
Self-citationBarriers to uptake of antimicrobial advice in a UK hospital: a qualitative study
Broom, J. K., Broom, A. F., Plage, S., Adams, K., & Post, J. J. (2016) · Journal of Hospital Infection · doi:10.1016/j.jhin.2016.03.011 · Cited by 72 · see citing works
Nachtigall, I., Tafelski, S., Deja, M., Halle, E., Grebe, M. C., Tamarkin, A., Rothbart, A., Uhrig, A., Meyer, E., Musial‐Bright, L., et al. (2014) · BMJ Open · doi:10.1136/bmjopen-2014-005370 · Cited by 67 · see citing works
Bourgeois, F. C., Linder, J. A., Johnson, S. A., Co, J. P. T., Fiskio, J. M., & Ferris, T. G. G. (2010) · Clinical Pediatrics · doi:10.1177/0009922810373649 · Cited by 42 · see citing works
Chow, A. L., Lye, D. C., & Arah, O. A. (2014) · International Journal of Antimicrobial Agents · doi:10.1016/j.ijantimicag.2014.10.009 · Cited by 36 · see citing works
Cato, K., Sheehan, B., Patel, S. J., Duchon, J. M., DeLaMora, P. A., Ferng, Y., Graham, P., Vawdrey, D. K., Perlman, J. M., Larson, E., et al. (2014) · Applied Clinical Informatics · doi:10.4338/aci-2013-09-ra-0069 · Cited by 35 · see citing works
Antibiotic information application offers nurses quick support
Wentzel, J., Drie‐Pierik, R. J. H. M., Nijdam, L. C., Geesing, J., Sanderman, R., & van Gemert‐Pijnen, J. E. W. C. (2016) · American Journal of Infection Control · doi:10.1016/j.ajic.2015.12.038 · Cited by 32 · see citing works
Self-citationChow, A. L., Lye, D. C., & Arah, O. A. (2015) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocv120 · Cited by 19 · see citing works
Verhoeven, F. (2009) · doi:10.3990/1.9789036528689 · Cited by 17 · see citing works
Nikolaus, S., Bode, C., Taal, E., et al. (2011) · Quality of Life Research · doi:10.1007/s11136-011-9982-8 · Cited by 12 · see citing works
Keeping an eye on the context: participatory development of eHealth to support clinical practice
Wentzel, J. (2015) · doi:10.3990/1.9789036539784 · Cited by 6 · see citing works
Self-citationJong, N. B.-D., Eikelenboom-Boskamp, A., Voss, A., & van Gemert‐Pijnen, J. E. W. C. (2014) · doi:10.13140/2.1.4886.1127 · Cited by 6 · see citing works
van Gemert‐Pijnen, J. E. W. C. (2003) · University of Twente Research Information · Cited by 4 · see citing works
Implementing antibiotic stewardship using eHealth and business modeling
Limburg, V. M. (2016) · doi:10.3990/1.9789036540841 · Cited by 2 · see citing works
Flagged first, then by citations
Research works printed but not linked: 1
- Beerlage-de Jong N. Opening the black box of Clinical Decision Support Systems for Antimicrobial Stewardship-a scoping review. Int J Med Inform 2017;Submitted.
Cited by
Sorted by citations
eHealth in Treatment of Offenders in Forensic Mental Health: A Review of the Current State
Kip, H., Bouman, Y. H. A., Kelders, S. M., & van Gemert-Pijnen, L. J. E. W. C. (2018) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2018.00042 · Cited by 89 · see citing works
Methods for Human-Centered eHealth Development: Narrative Scoping Review
Kip, H., Keizer, J., da Silva, M. C., Beerlage-de Jong, N., Köhle, N., & Kelders, S. M. (2022) · Journal of Medical Internet Research · doi:10.2196/31858 · Cited by 67 · see citing works
Self-citationMethods for Human-Centered eHealth Development: Narrative Scoping Review (Preprint)
Kip, H., Keizer, J., da Silva, M. C., Beerlage-de Jong, N., Köhle, N., & Kelders, S. M. (2021) · doi:10.2196/preprints.31858 · Cited by 67 · see citing works
Bayor, A. A., Li, J., Yang, I. A., & Varnfield, M. (2025) · Journal of Medical Internet Research · doi:10.2196/63733 · Cited by 46 · see citing works
“Users expressed concerns regarding how recommendations are presented and accessed [46] and inconsistencies in workflow integration [49,62].”
InfluentialKip, H., Kelders, S. M., Bouman, Y. H. A., & van Gemert-Pijnen, L. J. E. W. C. (2019) · Journal of Medical Internet Research · doi:10.2196/12972 · Cited by 40 · see citing works
Self-citationKip, H., Kelders, S. M., & Van Gemert-Pijnen, L. J. E. W. C. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300365 · Cited by 11 · see citing works
- Authors
- Beerlage-de Jong, N., Wentzel, J., Hendrix, R., & van Gemert-Pijnen, L.
- Affiliations
- University of Twente (NL)
- University Medical Center Groningen (NL)
- Venue
- American Journal of Infection Control
- Publisher
- Elsevier BV
- Published
- 2017-04
- Volume
- 45
- Issue
- 4
- Pages
- 365-371
- ISSN
- 0196-6553 (ISSN-L)
- OpenAlex
- W2574572273
- Cited by
- 24
- References
- 31
- Citations by year
- 2025: 1 · 2024: 6 · 2023: 2 · 2021: 4 · 2020: 2 · 2019: 5 · 2018: 3 · 2016: 1
- FWCI
- 2.06
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- American Journal of Infection Control (publishedVersion)
- PubMed (publishedVersion)
- University of Twente Research Information (submittedVersion, other-oa)
- University of Groningen research database (University of Groningen / Centre for Information Technology) (submittedVersion)
- Funders
- Interreg
- Topics
- Life Sciences › Immunology and Microbiology › Applied Microbiology and Biotechnology › Antibiotic Use and Resistance
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Keywords
- antimicrobial stewardship, clinical decision support systems, participatory development, end-user involvement, stakeholder engagement, antimicrobial prescribing, low-fidelity prototypes, scenario-based assessment, diagnostic test selection, medical residents, clinical workflow
- MeSH terms
- Adult, Anti-Infective Agents, Drug Utilization, Female, Humans, Male, Netherlands, Physicians, Surveys and Questionnaires, Organizational Policy, Decision Support Systems, Clinical
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
