The DOI matches its Crossref record; 94 of the 94 references it cites were found in a registry or an index.
Evaluating informatics applications – some alternative approaches: theory, social interactionism, and call for methodological pluralism.
Abstract
A review of evaluation literature concerning CDSSs indicates that randomized controlled clinical trials (RCTs) are the 'gold standard' for evaluation. While this approach is excellent for studying system or clinical performance, it is not well suited to answering questions concerning whether systems will be used or how they will be used. Because lack of use of CDSS has been of concern for some years, other evaluation research designs are needed to address those issues. This paper critiques RCT and experimental evaluation approaches and presents alternative approaches to evaluation that address questions outside the scope of the usual RCT and experimental designs. A wide range of literature is summarized to illustrate the value of evaluations that take into account social, organizational, professional, and other contextual considerations. Many of these studies go beyond the usual measures of systems performance or physicians' behavior by focusing on 'fit' of the system with other aspects of professional and organizational life. Because there is little explicit theory that informs many evaluations, the paper then reviews CDSS evaluations informed by social science theories. Lastly, it proposes a theoretical social science base of social interactionism. An example of such an approach is given. It involves a CDSS in psychiatry and is based on Kaplan's 4Cs, which focus on communication, control, care, and context. Although the example is a CDSS, the evaluation approach also is use…
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“An interactive sociotechnical approach incorporating Kaplan's 4Cs of communication, care, control, and context that draw attention to ELSI could investigate the dynamic interactions between these issues [53,[57], [58]].”
From REVISITING HEALTH INFORMATION TECHNOLOGY ETHICAL, LEGAL, and SOCIAL ISSUES and EVALUATION: TELEHEALTH/TELEMEDICINE and COVID-19 (2020) · cites this paper as [53]
In the citing paper
“Incorporating interpretive, critical, sociotechnical approaches into evaluation mitigates against both information technology and evaluation failures [14,[52], [53], [54], [55], [56]] and would allow for more ELSI exploration.”
From REVISITING HEALTH INFORMATION TECHNOLOGY ETHICAL, LEGAL, and SOCIAL ISSUES and EVALUATION: TELEHEALTH/TELEMEDICINE and COVID-19 (2020) · cites this paper as [53]
In the citing paper
“Although a purely theory-based approach—which might for instance add the compatibility variable from Rogers’ Innovation Diffusion Theory [63]—can be useful, empirical work may still be necessary to (a) determine whether those variables are relevant to health care, and (b) contextualize these new variables in order to determine, for example, what “compatibility” means in health care [57, 135].”
From The Technology Acceptance Model: Its past and its future in health care (2010) · cites this paper as [135]
In the citing paper
“As a result, problems have been framed too narrowly, leading to impoverished designs and disappointing ' solutions ' . 85”
From Health information technology: fallacies and sober realities (2010) · cites this paper as [85]
In the citing paper
“But whilst RCTs and similar experimental design methods may be appropriate for studying interventions under controlled clinical conditions, these design methodologies alone are often less well suited to evaluate the impact of eHealth interventions in a complex environment or to study the effect they have on the delivery of care [16],[17].”
From Evaluating eHealth Interventions: The Need for Continuous Systemic Evaluation (2009) · cites this paper as [16]
5 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Rogers, E. M., Singhal, A., & Quinlan, M. M. (2019) · An Integrated Approach to Communication Theory and Research · doi:10.4324/9780203710753-35 · Cited by 37,322
Why Don't Physicians Follow Clinical Practice Guidelines?: A Framework for Improvement
Cabana, M. D., Rand, C. S., Powe, N. R., Wu, A. W., Wilson, M. H., Abboud, P.-A. C., & Rubin, H. R. (1999) · JAMA · doi:10.1001/jama.282.15.1458 · Cited by 6,677
Plans and Situated Actions: The Problem of Human-Machine Communication
Kantowitz, B., & Suchman, L. A. (1990) · The American Journal of Psychology · doi:10.2307/1423221 · Cited by 5,299
Latour, B. (1987) · Cited by 4,106
On understanding computers and cognition: A new foundation for design
Winograd, T., & Flores, F. (1987) · Artificial Intelligence · doi:10.1016/0004-3702(87)90026-9 · Cited by 3,677
Randomized, Controlled Trials, Observational Studies, and the Hierarchy of Research Designs
Concato, J., Shah, N., & Horwitz, R. I. (2000) · New England Journal of Medicine · doi:10.1056/nejm200006223422507 · Cited by 3,563
Thompson, V. A., & March, J. G. (1966) · American Sociological Review · doi:10.2307/2090830 · Cited by 3,351
Users' guides to the medical literature
Guyatt, G. H. (1993) · JAMA · doi:10.1001/jama.270.17.2096 · Cited by 2,970
Effect of clinical guidelines on medical practice: a systematic review of rigorous evaluations
Grimshaw, J. M., & Russell, I. T. (1993) · The Lancet · doi:10.1016/0140-6736(93)92244-n · Cited by 2,770
Clinical guidelines: Potential benefits, limitations, and harms of clinical guidelines
Woolf, S. H., Grol, R., Hutchinson, A., Eccles, M., & Grimshaw, J. (1999) · BMJ · doi:10.1136/bmj.318.7182.527 · Cited by 2,548
Research works printed but not linked: 11
- J.G. Anderson, C.E. Aydin, S.J. Jay (Eds.), Evaluating health care information systems: approaches and applications. Sage, Thousand Oaks, CA, 1994.
- E.M.S. van Gennip, J.L. Talmon (Eds.), Assessment and evaluation of information technologies, Studies in Health Technology and Informatics, vol. 17, Amsterdam IOS Press, 1995
- F. Grémy, M. Bonnin, Evaluation of automatic health information systems. What and how?, In: E.M.S. van Gennip, J.L. Talmon (Eds.), Assessment and evaluation of information technologies, Studies in Health Technology and Informatics, vol 17, Amsterdam IOS Press, 1995, pp. 9–20.
- T. Jørgensen, Measuring effects, In: E.M.S. van Gennip, J.L. Talmon (Eds.), Assessment and evaluation of information technologies, Studies in Health Technology and Informatics, vol. 17, Amsterdam IOS Press, 1995, pp. 99–109.
- C. Aydin, Computerized order entry in a large medical center: evaluating interactions between departments, in: J.G., Anderson, C.E., Aydin, S.J., Jay, (Eds.), Evaluating health care information systems: approaches and applications, Sage, Thousand Oaks, CA, 1994, pp. 260–275.
- Covvey, 2001, When is a system only a system?, Healthcare Inform. Manage. Commun. Canada, 15, 46
- Kaplan, 2001, Computer rorschach test: what do you see when you look at a computer?, Physicians Comput., 18, 12
- L.A. Moore, Reaction of medical personnel to a medical expert system for stroke. In: J.G. Anderson, C.E. Aydin, S.J. Jay (Eds.), Evaluating health care information systems: Approaches and applications, Sage, Thousand Oaks, CA, 1994, pp. 226–244.
- E.M. Rogers, Diffusion of innovations, third ed., The Free Press, New York, 1983.
- R.E. Rice, J.G. Anderson, Social networks and health care information systems: a structural approach to evaluation. In: J.G. Anderson, C.E. Aydin, S.J. Jay (Eds.), Evaluating health care information systems: approaches and applications, Sage, Thousand Oaks, CA, 1994, pp. 135–163.
- R.R. Weaver, Evaluating the Problem Knowledge Coupler: a case study. In: J.G. Anderson, C.E. Aydin, S.J. Jay (Eds.), Evaluating health care information systems: approaches and applications, Sage, Thousand Oaks, CA, 1994, pp. 203–225.
Cited by
Kawamoto, K., Houlihan, C. A., Balas, E. A., & Lobach, D. F. (2005) · BMJ · doi:10.1136/bmj.38398.500764.8f · Cited by 2,703
The Technology Acceptance Model: Its past and its future in health care
Holden, R. J., & Karsh, B.-T. (2010) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2009.07.002 · Cited by 2,669
The Impact of eHealth on the Quality and Safety of Health Care: A Systematic Overview
Black, A. D., Car, J., Pagliari, C., Anandan, C., Cresswell, K., Bokun, T., McKinstry, B., Procter, R., Majeed, A., & Sheikh, A. (2011) · PLoS Medicine · doi:10.1371/journal.pmed.1000387 · Cited by 1,442
A Holistic Framework to Improve the Uptake and Impact of eHealth Technologies
van Gemert-Pijnen, J. E., Nijland, N., van Limburg, M., Ossebaard, H. C., Kelders, S. M., Eysenbach, G., & Seydel, E. R. (2011) · Journal of Medical Internet Research · doi:10.2196/jmir.1672 · Cited by 1,126
Qualitative Research Methods for Evaluating Computer Information Systems
Kaplan, B., & Maxwell, J. A. (2006) · Health Informatics · doi:10.1007/0-387-30329-4_2 · Cited by 821
Self-citation
Metadata
- Authors
- Kaplan, B.
- Affiliations
- Yale University (US)
- Corresponding author
- Kaplan, Bonnie
- ORCID
- Kaplan, Bonnie
- Venue
- International Journal of Medical Informatics
- Publisher
- Elsevier BV
- Published
- 2001-11
- Volume
- 64
- Issue
- 1
- Pages
- 39-56
- ISSN
- 1386-5056 (ISSN-L)
- OpenAlex
- W2090760643
- Cited by
- 380
- References
- 93
- Citations by year
- 2026: 1 · 2025: 4 · 2024: 4 · 2023: 5 · 2022: 11 · 2021: 5 · 2020: 9 · 2019: 12 · 2018: 13 · 2017: 10 · 2016: 15 · 2015: 22 · 2014: 20 · 2013: 16 · 2012: 21
- FWCI
- 39.51
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- www.elsevier.com/tdm/userlicense/1.0/ (tdm)
- Locations
- International Journal of Medical Informatics (publishedVersion)
- PubMed (publishedVersion)
- Funders
- National Institute of Standards and Technology
- Topics
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Social Sciences › Business, Management and Accounting › Management Information Systems › Information Systems Theories and Implementation
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Clinical practice guidelines implementation
- Keywords
- clinical decision support systems, randomized controlled trials, methodological pluralism, social interactionism, social science theory, system adoption, system use, contextual factors, guideline implementation, organizational context
- MeSH terms
- Attitude, Humans, Quality of Health Care, Social Conditions, User-Computer Interface, Organizational Culture, Program Evaluation, Randomized Controlled Trials as Topic, Practice Guidelines as Topic, Decision Support Systems, Clinical
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Evaluating informatics applications – some alternative approaches: theory, social interactionism, and call for methodological pluralism. https://app.cochat.ai/papers/10.1016/s1386-5056(01)00184-8
- The paper
- Kaplan, B. (2001). Evaluating informatics applications – some alternative approaches: theory, social interactionism, and call for methodological pluralism. International Journal of Medical Informatics. https://doi.org/10.1016/s1386-5056(01)00184-8
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.