Beyond effectiveness: the evaluation of information systems using a comprehensive health technology assessment framework
Abstract
A Comprehensive Health Technology Assessment Framework is presented as a conceptual tool for decision-making about health technologies, including information technologies. The aim of the model is to provide an empirical, evidence-based foundation for health technology decisions. The major framework dimensions are (1) population at risk, (2) population impact, (3) economic concerns, (4) social context (including ethical, legal, and political concerns), and (5) technology assessment information. This multi-disciplinary approach provides guidelines on use of appropriate information in aligning 'stakeholder wants' and 'population needs'.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
While growth in mHealth interventions has been rapid, advances in existing evaluation frameworks have not been seen.
“Nevertheless, while growth in mHealth interventions has been rapid, advances in the existing evaluation frameworks have not been seen [25–27].”
Cites
Renkema, T. J. W., & Berghout, E. W. (1997) · Information and Software Technology · doi:10.1016/0950-5849(96)85006-3 · Cited by 223 · see citing works
Decision Research: A Field Guide
Mayne, A. J. (1992) · Journal of the Operational Research Society · doi:10.1057/jors.1992.8 · Cited by 189 · see citing works
Addressing organizational issues into the evaluation of medical systems
Kaplan, B. (1997) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.1997.0040094 · Cited by 175 · see citing works
Evaluating the Impact of Health Care Information Systems
Anderson, J. G., & Aydin, C. E. (1997) · International Journal of Technology Assessment in Health Care · doi:10.1017/s0266462300010436 · Cited by 95 · see citing works
Rationing Health Care in America: Perceptions and Principles of Justice.
(1988) · Annals of Internal Medicine · doi:10.7326/0003-4819-108-2-316_1 · Cited by 93 · see citing works
Finding Audiences, Changing Beliefs: The Structure of Research Use in Canadian Health Policy
Lomas, J. (1990) · Journal of Health Politics Policy and Law · doi:10.1215/03616878-15-3-525 · Cited by 80 · see citing works
Health Decision Support Systems
Tan, J., & Sheps, S. S. (1998) · Cited by 56 · see citing works
Systems to Support Health Policy Analysis: Theory Models and Uses
Gustafson, D. H., Alemi, F., & Cats‐Baril, W. L. (1992) · Medical Entomology and Zoology · Cited by 56 · see citing works
The Political Dimension in Health Care Technology Assessment Programs
Banta, H. D., & Andreasen, P. B. (1990) · International Journal of Technology Assessment in Health Care · doi:10.1017/s0266462300008977 · Cited by 42 · see citing works
Drummond, M. (1990) · International Journal of Technology Assessment in Health Care · doi:10.1017/s0266462300008941 · Cited by 30 · see citing works
Cheung, S.-T., Davies, R. F., SMITH, K., MARSH, R., Sherrard, H., & Keon, W. J. (1998) · Telemedicine Journal · doi:10.1089/tmj.1.1998.4.259 · Cited by 25 · see citing works
Defusing Technology:Technology Diffusion in British Columbia
Kazanjian, A., & Friesen, K. (1993) · International Journal of Technology Assessment in Health Care · doi:10.1017/s0266462300003020 · Cited by 8 · see citing works
Self-citationAn evaluation model for clinical information systems: clinical imaging systems.
Kaplan, B. (1995) · PubMed · Cited by 6 · see citing works
Technology assessment and political decision-making
Bozeman, B. L., & Rossini, F. A. (1979) · Technological Forecasting and Social Change · doi:10.1016/0040-1625(79)90063-5 · Cited by 6 · see citing works
The need for a holistic view of telemedicine, focused on patients and society as prime stakeholders.
Roberts, R., & Rigby, M. (1998) · PubMed · Cited by 5 · see citing works
Flagged first, then by citations
4 of the 19 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Greenhalgh, T., Wherton, J., Papoutsi, C., Lynch, J., Hughes, G., A'Court, C., Hinder, S., Fahy, N., Procter, R., & Shaw, S. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.8775 · Cited by 2,887 · see citing works
“Our search for evidence-based approaches to guide our empirical work on the above case studies identified 28 technology implementation frameworks, informed by several theoretical perspectives, which we sorted into a simple taxonomy [13,43,47-72]; see Multimedia Appendix 1.”
A Holistic Framework to Improve the Uptake and Impact of eHealth Technologies
van Gemert-Pijnen, J. E., Nijland, N., van Limburg, M., Ossebaard, H. C., Kelders, S. M., Eysenbach, G., & Seydel, E. R. (2011) · Journal of Medical Internet Research · doi:10.2196/jmir.1672 · Cited by 1,126 · see citing works
Costs and benefits of health information technology.
Shekelle, P. G., Morton, S. C., & Keeler, E. B. (2006) · doi:10.23970/ahrqepcerta132 · Cited by 501 · see citing works
Demiris, G., Afrin, L. B., Speedie, S. M., Courtney, K. L., Sondhi, M. M., Vimarlund, V., Lovis, C., Goossen, W. T. F., & Lynch, C. (2007) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m2492 · Cited by 310 · see citing works
Information technology and systems justification: A review for research and applications
Gunasekaran, A., Ngai, E. W. T., & McGaughey, R. (2006) · European Journal of Operational Research · doi:10.1016/j.ejor.2005.06.002 · Cited by 278 · see citing works
Investigating evaluation frameworks for health information systems
Yusof, M. M., Papazafeiropoulou, A., Paul, R. J., & Stergioulas, L. K. (2008) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2007.08.004 · Cited by 304 · see citing works
Healthcare system: Moving forward with artificial intelligence
Dıcuonzo, G., Donofrio, F., Fusco, A., & Shini, M. (2022) · Technovation · doi:10.1016/j.technovation.2022.102510 · Cited by 193 · see citing works
Future Directions in Evaluation Research: People, Organizational, and Social Issues
Shaw, N. T., & Kaplan, B. (2004) · Methods of Information in Medicine · doi:10.1055/s-0038-1633862 · Cited by 141 · see citing works
Health information systems evaluation frameworks: A systematic review
Andargoli, A. E., Scheepers, H., Rajendran, D., & Sohal, A. S. (2016) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2016.10.008 · Cited by 130 · see citing works
Guideline for good evaluation practice in health informatics (GEP-HI)
Nykänen, P., Brender, J., Talmon, J., de Keizer, N., Rigby, M., Beuscart-Zephir, M.-C., & Ammenwerth, E. (2011) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2011.08.004 · Cited by 105 · see citing works
Singh, N., Jain, M., Kamal, M. M., Bodhi, R., & Gupta, B. (2023) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2023.122967 · Cited by 104 · see citing works
The IT productivity paradox in health: A stakeholder's perspective
Lapointe, L., Mignerat, M., & Vedel, I. (2010) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2010.11.004 · Cited by 92 · see citing works
MEDICAL DEVICES EARLY ASSESSMENT METHODS: SYSTEMATIC LITERATURE REVIEW
Markiewicz, K., van Til, J. A., & IJzerman, M. J. (2014) · International Journal of Technology Assessment in Health Care · doi:10.1017/s0266462314000026 · Cited by 87 · see citing works
A framework for m-health service development and success evaluation
Sadegh, S. S., Saadat, P. K., Sepehri, M. M., & Assadi, V. (2018) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2018.01.003 · Cited by 54 · see citing works
Khalifa, M., Magrabi, F., & Gallego, B. (2019) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-019-0940-7 · Cited by 54 · see citing works
Riaz, M. S., & Atreja, A. (2016) · Clinical Gastroenterology and Hepatology · doi:10.1016/j.cgh.2016.05.009 · Cited by 51 · see citing works
Nijland, N. (2010) · doi:10.3990/1.9789036531337 · Cited by 45 · see citing works
Economic analyses for ICT in elderly healthcare: questions and challenges
Vimarlund, V., & Olve, N.-G. (2005) · Health Informatics Journal · doi:10.1177/1460458205058758 · Cited by 42 · see citing works
Health Services Research Evaluation Principles
Crawford, P., Lehmann, H. P., & SOCKOLOW, P. S. (2012) · Methods of Information in Medicine · doi:10.3414/me10-01-0066 · Cited by 37 · see citing works
Wentink, M. M., van Bodegom‐Vos, L., Brouns, B., Arwert, H. J., Houdijk, S., Kewalbansing, P., Boyce, L. W., Vlieland, T. P. M. V., de Kloet, A. J., & Meesters, J. J. L. (2019) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-019-0871-3 · Cited by 36 · see citing works
Understanding the role of technology in health information systems.
Lewis, D. P., Hodge, N., Gamage, D., & Whittaker, M. A. (2012) · PubMed · Cited by 29 · see citing works
User-Centric Design Methodology for mHealth Apps: The PainApp Paradigm for Chronic Pain
Koumpouros, Y. (2022) · Technologies · doi:10.3390/technologies10010025 · Cited by 27 · see citing works
A new approach in the evaluation of hospital information systems
Gürsel, G., Zayim, N., Gülkesen, K. H., Arifoğlu, A., & Saka, O. (2014) · TURKISH JOURNAL OF ELECTRICAL ENGINEERING & COMPUTER SCIENCES · doi:10.3906/elk-1110-15 · Cited by 22 · see citing works
Critical Issues in EHR Implementation: Provider and Vendor Perspectives
Palvia, P., Jacks, T., & Brown, W. S. (2015) · Communications of the Association for Information Systems · doi:10.17705/1cais.03636 · Cited by 22 · see citing works
Anastasiadou, D., Lupiañez-Villanueva, F., Faulí, C., Arcal Cunillera, J., & Serrano-Troncoso, E. (2018) · BMC Psychiatry · doi:10.1186/s12888-018-1664-4 · Cited by 16 · see citing works
Developing an Evaluation Framework for Selecting Optimal Medical Devices
Park, J., Kim, E., & Shin, K. (2019) · Journal of Open Innovation Technology Market and Complexity · doi:10.3390/joitmc5030064 · Cited by 17 · see citing works
Fitterer, R., Rohner, P., Mettler, T., & Winter, R. (2010) · Hawaii International Conference on System Sciences (HICSS) · doi:10.1109/hicss.2010.37 · Cited by 14 · see citing works
Taxonomy for multi-perspective assessment of the value of health information systems
Fitterer, R., Mettler, T., Rohner, P., & Winter, R. (2010) · International Journal of Healthcare Technology and Management · doi:10.1504/ijhtm.2011.037220 · Cited by 10 · see citing works
Approaching information system evaluation methodology and techniques: a comprehensive review
Ababneh, H., Shrafat, F. D., & Zeglat, D. (2016) · International Journal of Business Information Systems · doi:10.1504/ijbis.2017.080943 · Cited by 10 · see citing works
Lin, C., Lin, H.-C. K., Huang, Y., & Jalleh, G. (2011) · Global Journal of Business Management · doi:10.5897/ajbm10.883 · Cited by 9 · see citing works
Exploring New Factors and The Question of 'Which' in User Acceptance Studies of Healthcare Software
Azizah, N., & Mohamadali, K. (2013) · Nottingham ePrints (University of Nottingham) · Cited by 8 · see citing works
The rationale of e-health evaluation: the case of NHS Direct
Hamid, A., & Alshawi, S. (2012) · International Journal of Business Information Systems · doi:10.1504/ijbis.2012.046297 · Cited by 7 · see citing works
BUSINESS MODELS FOR SUSTAINED EHEALTH IMPLEMENTATION: LESSONS FROM TWO CONTINENTS
Dyk, L. V., Wentzel, M. J., van Limburg, A. H. M., van Gemert‐Pijnen, J. E. W. C., & Schutte, C. (2012) · SUNScholar (Stellenbosch University) · Cited by 7 · see citing works
Vannieuwenborg, F., Auwermeulen, T. V. D., Ooteghem, J. V., Jacobs, A., Verbrugge, S., & Colle, D. (2016) · JMIR Medical Informatics · doi:10.2196/medinform.5012 · Cited by 7 · see citing works
Towards a B2B E-Commerce Evaluation Management Model to Assess Organizational Drivers in Hospitals
Lin, C., Lin, H.-C. K., Huang, Y., Jalleh, G., Liu, Y.-C., Huang, Y.-H., Su, S.-H., & Chao, C.-J. (2014) · The International Technology Management Review · doi:10.2991/itmr.2014.4.3.1 · Cited by 7 · see citing works
The Challenges of the IS/IT Projects in the Healthcare Sector
Gomes, J., & Rom�ão, M. (2018) · International Journal of Applied Research on Public Health Management · doi:10.4018/ijarphm.2019010105 · Cited by 4 · see citing works
Clinical Trials of Information Interventions
Balas, E. A., & Boren, S. A. (2007) · Springer eBooks · doi:10.1007/978-0-387-38319-4_7 · Cited by 4 · see citing works
Health technology assessment within a public accountability framework
Kazanjian, A., & Green, C. J. (2004) · Clinical Governance An International Journal · doi:10.1108/14777270410517737 · Cited by 4 · see citing works
Self-citationCrowe, B. L., & Sim, L. (2004) · International Congress Series · doi:10.1016/j.ics.2004.03.361 · Cited by 4 · see citing works
Improving the Success of IS/IT Projects in Healthcare: Benefits and Project Management Approaches
Gomes, J., & Rom�ão, M. (2016) · Advances in intelligent systems and computing · doi:10.1007/978-3-319-31232-3_51 · Cited by 4 · see citing works
Marceglia, S. R. F., Simona, F., Bonacina, S., Pinciroli, F., Lasorsa, I., Savino, C., & Pozzi, G. (2016) · Studies in health technology and informatics · doi:10.3233/978-1-61499-635-4-63 · Cited by 3 · see citing works
A holistic nonlinear framework for developing a national eHealth vision based on Moroccan experience
Fakhkhari, H., Bounabat, B., Gaizi, Z., Fatemi, S., Bennani, M., & Bakkali, R. (2021) · International Conference on Networking, Information Systems & Security (NISS) · doi:10.1145/3454127.3457635 · Cited by 3 · see citing works
CROSS DISCIPLINARY EVALUATION FRAMEWORK FOR E-HEALTH SERVICES
Hamid, A. (2010) · Brunel University Research Archive (BURA) (Brunel University London) · Cited by 3 · see citing works
Olve, N.-G., & Vimarlund, V. (2005) · Cited by 2 · see citing works
Beyan, O., & Baykal, N. (2007) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2007.01.006 · Cited by 2 · see citing works
The Routine Health Information System in Palestine: Determinants and Performance
Mimi, Y. (2015) · City Research Online (City University London) · Cited by 2 · see citing works
Yismaw, A., Alemu, Z. A., & Eshete, T. (2021) · doi:10.2196/preprints.32612 · Cited by 2 · see citing works
ADVANCES IN TRANSLATIONAL SCIENCE
Sellin, J. H., Murthy, S. K., Goetz, M., Hoffman, A. W., & Kießlich, R. (2012) · Cited by 1 · see citing works
How information system investment evaluation is being initiated: an actor network theory perspective
Shrafat, F. D., Ababneh, H., & Alrawabdeh, W. A. (2016) · RePEc: Research Papers in Economics · doi:10.1504/ijbis.2016.076872 · Cited by 1 · see citing works
Was ist der Nutzen von eHealth
Fitterer, R., Mettler, T., & Rohner, P. (2009) · Alexandria (UniSG) (University of St.Gallen) · Cited by 1 · see citing works
Health technology assessment of medical devices during development
Markiewicz, K. (2017) · doi:10.3990/1.9789036543712 · Cited by 1 · see citing works
Khalifa, M., Magrabi, F., & Gallego, B. (2019) · arXiv (Cornell University) · doi:10.13140/rg.2.2.16448.89600 · Cited by 1 · see citing works
THE MIDDLE EAST TECHNICAL UNIVERSITY
Gürsel, G. (2012)
Towards a Post-Implementation Evaluation Framework of Outpatient Electronic Drug Prescribing
Oliveira, T. C., Oliveira, M. D., & Peña, M. T. (2013) · IGI Global eBooks · doi:10.4018/978-1-4666-3990-4.ch007
CONTRIBUTION OF FUZZY LOGIC TO EVALUATION
Gürsel, G., & Gülkesen, K. H. (2017) · AJIT-e Academic Journal of Information Technology · doi:10.5824/1309-1581.2017.2.001.x
Eatemadololama, H. (2018) · Universal Journal of Pharmaceutical Research · doi:10.22270/ujpr.v3i1.r13
Gomes, J., & Rom�ão, M. (2023) · Advances in computational intelligence and robotics book series · doi:10.4018/978-1-6684-9151-5.ch014
ICT and Its Roles in Health Development
Gashu, K. D. (2024) · Sustainable development goals series · doi:10.1007/978-3-031-71118-3_1
Design and evaluation of a multidisciplinary cancer conferencing platform
Ghaznavi, F. (2012) · Chalmers Publication Library (Chalmers University of Technology)
- Authors
- Kazanjian, A., & Green, C. J.
- Affiliations
- University of British Columbia (CA)
- Corresponding author
- Green, Carolyn J
- ORCID
- Kazanjian, Arminée
- Venue
- Computers in Biology and Medicine
- Publisher
- Elsevier BV
- Published
- 2002-05
- Volume
- 32
- Issue
- 3
- Pages
- 165-177
- ISSN
- 0010-4825 (ISSN-L)
- OpenAlex
- W2091500337
- Cited by
- 60
- References
- 19
- Citations by year
- 2024: 1 · 2023: 2 · 2022: 2 · 2021: 2 · 2019: 4 · 2018: 4 · 2017: 3 · 2016: 7 · 2015: 2 · 2014: 3 · 2013: 3 · 2012: 7
- FWCI
- 4.85
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Computers in Biology and Medicine (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Health Systems, Economic Evaluations, Quality of Life
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Technology and Human Factors in Education and Health
- Keywords
- health technology assessment, information system evaluation, health information technology, decision-making framework, vulnerable populations, population effects, economic evaluation, social environment, stakeholder needs, evidence-based decision making
- MeSH terms
- Decision Making, Organizational, Evaluation Studies as Topic, Medical Informatics, Technology Assessment, Biomedical
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review