Cited by 623Human Resource Development and Performance EvaluationEvaluation and Performance AssessmentEvaluation of Teaching Practices

The DOI matches its Crossref record; 41 of the 44 references it cites were found in a registry or an index.

A critical analysis of evaluation practice: the Kirkpatrick model and the principle of beneficence

Bates, Reid·2004·Evaluation and Program Planning

DOI 10.1016/j.evalprogplan.2004.04.011

Open this paper in CoChat — free account

Integrity signals

Registry

Crossref · 10.1016/j.evalprogplan.2004.04.011

Cites

41 of 41 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Evaluating Training Programs : The Four Levels
Kirkpatrick, D. L. (2009) · Cited by 4,720
Principles of Biomedical Ethics.
(1984) · Annals of Internal Medicine · doi:10.7326/0003-4819-100-1-168_1 · Cited by 4,559
Utilization-Focused Evaluation
(2018) · Sage eBooks · doi:10.4135/9781506326139.n729 · Cited by 2,315
Application of cognitive, skill-based, and affective theories of learning outcomes to new methods of training evaluation.
Kraiger, K., Ford, J. K., & Salas, E. (1993) · Journal of Applied Psychology · doi:10.1037/0021-9010.78.2.311 · Cited by 1,831 · cited 1×
“Kraiger, Ford and Salas (1993) have forwarded a multidimensional model of learning outcomes from training and have described a process training evaluators could use to developing learning evaluation measures (Kraiger & Jung, 1997).”
Background
The science of training: a decade of progress
Salas, E., & Cannon-Bowers, J. A. (2001) · Annual Review of Psychology · doi:10.1146/annurev.psych.52.1.471 · Cited by 1,703
Techniques for Evaluating Training Programs
Kirkpatrick, D. L. (1975) · Training and development journal · Cited by 1,234
Training and Development in Work Organizations
Tannenbaum, S. I., & YUKL, G. A. (1992) · Annual Review of Psychology · doi:10.1146/annurev.ps.43.020192.002151 · Cited by 1,023
The adult learner
Knowles, M. S., Holton, E. F., & Swanson, R. A. (1973) · Cited by 970
The flawed four‐level evaluation model
Holton, E. F. (1996) · Human Resource Development Quarterly · doi:10.1002/hrdq.3920070103 · Cited by 965
Principles of Biomedical Ethics
Beauchamp, T. L., & Childless, J. F. (1991) · International Clinical Psychopharmacology · doi:10.1097/00004850-199100620-00013 · Cited by 964

Show all 44 references

Research works printed but not linked: 3

Ford, 1995, The application of cognitive constructs and principles to the instructional systems design model of training: Implications for needs assessment, design, and transfer, International Review of Industrial and Organizational Psychology, 10, 1
Grider, 1988, Evaluating valuations, Training and Development Journal, 42, 11
Schwandt, T. A., (1998). How we think about evaluation practice. Paper presented at the American Evaluation Association Conference, Chicago, IL.

Sorted by citations

Program evaluation models and related theories: AMEE Guide No. 67
FRYE, A. W., & Hemmer, P. A. (2012) · Medical Teacher · doi:10.3109/0142159x.2012.668637 · Cited by 641
Machine Learning in Mental Health: A Systematic Review of the HCI Literature to Support the Development of Effective and Implementable ML Systems
Thieme, A., Belgrave, D., & Doherty, G. (2020) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3398069 · Cited by 406
“The principle of beneficence however does not only entail encouraging human flourishing and wellbeing by doing the right thing, but also suggests to ‘do it well’ [13].”
Background
Effective Dementia Education and Training for the Health and Social Care Workforce: A Systematic Review of the Literature
Surr, C. A., Gates, C., Irving, D., Oyebode, J., Smith, S. J., Parveen, S., Drury, M., & Dennison, A. (2017) · Review of Educational Research · doi:10.3102/0034654317723305 · Cited by 327
“Kirkpatrick’s (1979, 1984) “Return on Investment” model developed specifically for evaluation of training consists of four levels and is widely adopted in the evaluation of training and education provision (Bates, 2004).”
A Critique of Kirkpatrick's Evaluation Model
Reio, T. G., Rocco, T. S., Smith, D. H., & Chang, E. (2017) · New Horizons in Adult Education and Human Resource Development · doi:10.1002/nha3.20178 · Cited by 227
“, Bates, 2004; Holton, 1996; Swanson, 1994). Some researchers, however, have developed their own models based on Kirkpatrick’s. Bushnell (1990) and Hayes, Scott, Braczinskas, Scaccia, Stout, and Wandersman (2016) contend that Kirkpatrick’s model focuses only on what happens after the training, and therefore is not suf…”
BackgroundMethodInfluential
HCI and Affective Health
Sanches, P., Janson, A., Karpashevich, P., Nadal, C., Qu, C., Roquet, C. D., Umair, M., Windlin, C., Doherty, G. J., Höök, K., et al. (2019) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300475 · Cited by 248
“Others argued for the value of increasing user’s feeling of control by augmenting tracking technologies with support for reflection [18], or increasing user’s engagement with technology through DIY approaches to design [148].”
BackgroundInfluential
Evaluating training programs: development and correlates of the Q uestionnaire for P rofessional T raining E valuation
Grohmann, A., & Kauffeld, S. (2013) · International Journal of Training and Development · doi:10.1111/ijtd.12005 · Cited by 193
“In organizations often broader evaluation issues, such as exploring determinants of successful training transfer, have to be addressed in order to optimize future training courses (e.g. Bates, 2004).”
Background
Using Kirkpatrick’s model to measure the effect of a new teaching and learning methods workshop for health care staff
Heydari, M. R., Taghva, F., Amini, M., & Delavari, S. (2019) · BMC Research Notes · doi:10.1186/s13104-019-4421-y · Cited by 175
Low- versus high-fidelity simulations in teaching and assessing clinical skills
Munshi, F. M., Lababidi, H. M., & Alyousef, S. (2015) · Journal of Taibah University Medical Sciences · doi:10.1016/j.jtumed.2015.01.008 · Cited by 180
“Fourth, the learning impact of simulation studies can be categorized by Kirkpatrick’s four levels of training evaluation: reaction, learning, behaviour, and results.(30) The first level includes the assessment of participants’ reactions to the use of simulation.”
Kirkpatrick Model: Its Limitations as Used in Higher Education Evaluation
Cahapay, M. B. (2021) · International Journal of Assessment Tools in Education · doi:10.21449/ijate.856143 · Cited by 215
“…higher education fields and aspects (see Quintas et al., 2017 on instructional approach; Baskin, 2001 The reviews of Alliger and Janak (1989), Bates (2004), and Reio et al., (2017) help understand the current state of the Kirkpatrick model by overtly tackling its inherent limitations in the…”
Background
Evaluation in health professions education—Is measuring outcomes enough?
Allen, L., Hay, M., & Palermo, C. E. (2021) · Medical Education · doi:10.1111/medu.14654 · Cited by 136
Achieving Digital Wellbeing Through Digital Self-control Tools: A Systematic Review and Meta-analysis
Roffarello, A. M., & De Russis, L. (2023) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3571810 · Cited by 137
“The principle of beneicence, however, is not only related to łdoing good,ž but also to łdo it wellž [23], i.”
Background
Effectiveness of community health worker training programmes for cardiovascular disease management in low-income and middle-income countries: a systematic review
Abdel-All, M., Putica, B., Praveen, D., Abímbọ́lá, Ṣ., & Joshi, R. (2017) · BMJ Open · doi:10.1136/bmjopen-2016-015529 · Cited by 133
Adaptation of Kirkpatrick’s Four-Level Model of Training Criteria to Evaluate Training Programmes for Head Teachers
Alsalamah, A., & Callinan, C. J. (2021) · Education Sciences · doi:10.3390/educsci11030116 · Cited by 127
“Although Holton [43], and Alliger and Janak [49], have criticised the hierarchical nature of the model, studies and empirical results do not provide sufficient evidence to support this assumption of its hierarchical nature [40].”
BackgroundMethodInfluential
Risking connection trauma training: A pathway toward trauma-informed care in child congregate care settings.
Brown, S. M., Baker, C. N., & Wilcox, P. D. (2011) · Psychological Trauma Theory Research Practice and Policy · doi:10.1037/a0025269 · Cited by 126
Evaluating the effectiveness of virtual reality for safety-relevant training: a systematic review
Stefan, H., Mortimer, M., & Horan, B. (2023) · Virtual Reality · doi:10.1007/s10055-023-00843-7 · Cited by 171
“Despite its criticisms over the last five decades and the proposals of newer evaluation models, Kirkpatrick’s model continues to be the most relevant and widely used model by different organizations as the basis for training evaluation (Bates 2004; Reio et al. 2017).”
Method
Has the new Kirkpatrick generation built a better hammer for our evaluation toolbox?
Moreau, K. A. (2017) · Medical Teacher · doi:10.1080/0142159x.2017.1337874 · Cited by 114
“Criticism 3: Kirkpatrick’s causal chain is unfounded Bates (2004) highlights the limited evidence to support the causal chain embedded within Kirkpatrick’s original model.”
Background
Trainees' reactions to training: an analysis of the factors affecting overall satisfaction with training
Giangreco, A., Sebastiano, A., & Peccei, R. (2009) · The International Journal of Human Resource Management · doi:10.1080/09585190802528417 · Cited by 112
“…Allan and Birdi 1999; Blanchard, Thacker and Way 2000; Tracey, Hinkin, Tannenbaum and Mathieu 2001), including a few criticisms (e.g. Clement 1982; Alliger and Janak 1989; Tannenbaum and Woods 1992; Bates 2004; Brown 2005), and a muscular confrontation between Holton (1996) and Kirkpatrick (1996b).”
Background
Approaching evaluation from a multilevel perspective: A comprehensive analysis of the indicators of training effectiveness
Sitzmann, T., & Weinhardt, J. M. (2017) · Human Resource Management Review · doi:10.1016/j.hrmr.2017.04.001 · Cited by 140
The Kirkpatrick model for training evaluation: bibliometric analysis after 60 years (1959–2020)
Alsalamah, A., & Callinan, C. J. (2021) · Industrial and Commercial Training · doi:10.1108/ict-12-2020-0115 · Cited by 166
Training and employee development for improved performance.
Brown, K. G., & Sitzmann, T. (2010) · American Psychological Association eBooks · doi:10.1037/12170-016 · Cited by 125
What works in delivering dementia education or training to hospital staff? A critical synthesis of the evidence
Surr, C. A., & Gates, C. (2017) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2017.08.002 · Cited by 107
“A model widely adopted in the evaluation of training and education provision (Bates, 2004) is Kirkpatrick’s (1984, 1979) four level ‘Return on Investment’ model.”
Method
Learning in context: Identifying gaps in research on the transfer of medical communication skills to the clinical workplace
van den Eertwegh, V., van Dulmen, S., van Dalen, J., Scherpbier, A. J. J. A., & van der Vleuten, C. P. M. (2012) · Patient Education and Counseling · doi:10.1016/j.pec.2012.06.008 · Cited by 103
“In recent years however the suitability of Kirkpatrick’s levels to evaluate the effectiveness of interventions has been questioned [24–26,27].”
Background
The influence of training and training transfer factors on organisational learning and performance
Dermol, V., & Čater, T. (2013) · Personnel Review · doi:10.1108/00483481311320435 · Cited by 97
Kirkpatrick evaluation model for in-service training on cardiopulmonary resuscitation
Akbari, M., Dorri, S., & Sedeh, M. D. (2016) · Iranian Journal of Nursing and Midwifery Research · doi:10.4103/1735-9066.193396 · Cited by 97
“Hence, this model simplifies the complex process of evaluation of training programs.[10]”
Method
A systematic review of approaches to assessing cybersecurity awareness
Rahim, N. H. A., Hamid, S. B., Kiah, M. L. M., Shamshirband, S., & Furnell, S. (2015) · Kybernetes · doi:10.1108/k-12-2014-0283 · Cited by 94
“This refers to a mixture of quantitative and qualitative methodologies, which can help simplify the complex process of understanding a programme ’ s effectiveness (Bates, 2004).”
Background
Research methodology workshops evaluation using the Kirkpatrick’s model: Translating theory into practice
Abdulghani, H. M., Shaik, S. A., Khamis, N., Al-Drees, A. A., Irshad, M., Khalil, M. S., AlHaqwi, A. I., & Isnani, A. C. (2014) · Medical Teacher · doi:10.3109/0142159x.2014.886012 · Cited by 96
“However Donald Kirkpatrick model (Kirkpatrick & Kirkpatrick 2006) has served as the primary organizing design for the evaluations of training since thirty years (Bates 2004).”
Method
Measuring the effectiveness of virtual training: A systematic review
Strojny, P., & Dużmańska-Misiarczyk, N. (2023) · Computers & Education X Reality · doi:10.1016/j.cexr.2022.100006 · Cited by 107
E-Learning Theories, Components, and Cloud Computing-Based Learning Platforms
Kumar, V., & Sharma, D. (2021) · International Journal of Web-Based Learning and Teaching Technologies · doi:10.4018/ijwltt.20210501.oa1 · Cited by 73
The use of virtual reality and augmented reality to enhance cardio-pulmonary resuscitation: a scoping review
Kuyt, K., Park, S. H., Chang, T. P., Jung, T. H., & MACKINNON, R. J. (2021) · Advances in Simulation · doi:10.1186/s41077-021-00158-0 · Cited by 72
“It consists of four levels of evidence, assessing the reaction to training, skills learnt, behaviours and outcomes [16] (Table 2).”
Background
Training professionals' usage and understanding of K irkpatrick's Level 3 and Level 4 evaluations
Kennedy, P. E., Chyung, S. Y., Winiecki, D. J., & Brinkerhoff, R. O. (2013) · International Journal of Training and Development · doi:10.1111/ijtd.12023 · Cited by 87
Evaluation and impact of cardiotocography training programmes: a systematic review
Pehrson, C., Sorensen, J., & Amer‐Wåhlin, I. (2011) · BJOG An International Journal of Obstetrics & Gynaecology · doi:10.1111/j.1471-0528.2011.03021.x · Cited by 75
Revisiting Kirkpatrick's model ― an evaluation of an academic training course
Rajeev, P., Madan, M., & Jayarajan, K. (2009) · Cited by 72
Are we doing the right thing?
Giangreco, A., Carugati, A., & Sebastiano, A. (2010) · Personnel Review · doi:10.1108/00483481011017390 · Cited by 68
“Of the many scholars who have commented, Bates's (2004)”
Background
Evaluation of a training program of hypertension for accredited social health activists (ASHA) in rural India
Abdel-All, M., Thrift, A. G., Riddell, M. A., Thankappan, K. R., Mini, G., Chow, C. K., Maulik, P. K., Mahal, A. S., Guggilla, R. K., Kalyanram, K., et al. (2018) · BMC Health Services Research · doi:10.1186/s12913-018-3140-8 · Cited by 68
“Use of these models also draws attention to the learning transfer process and the behavioural change of the trainee to achieve the desired outcome of the training [39].”
Background
SeisTutor: A Custom-Tailored Intelligent Tutoring System and Sustainable Education
Singh, N., Gunjan, V. K., Mishra, A. K., Mishra, R. K., & Nawaz, N. (2022) · Sustainability · doi:10.3390/su14074167 · Cited by 76
Non-technical skills assessments in undergraduate medical education: A focused BEME systematic review: BEME Guide No. 54
Gordon, M. A., Farnan, J. M., Grafton‐Clarke, C., Ahmed, R., Gurbutt, D., McLachlan, J. C., & Daniel, M. M. (2019) · Medical Teacher · doi:10.1080/0142159x.2018.1562166 · Cited by 60
“The impacts of the interventions were classified in accordance with Kirkpatrick’s adapted hierarchy (Bates 2004), in line with guidance provided by BEME (Hammick et al.”
Method
A Mixed Methods Examination of the Influence of Dimensions of Support on Training Transfer
Schindler, L. A., & Burkholder, G. (2016) · Journal of Mixed Methods Research · doi:10.1177/1558689814557132 · Cited by 59
“Many organizations rely on the Kirkpatrick (1976) model as a framework to evaluate their training programs (Bates, 2004; Patel, 2010; Salas & Cannon-Bowers, 2001).”
Method
Evaluating protection motivation based cybersecurity awareness training on Kirkpatrick's Model
Khan, N. F., Ikram, N., Murtaza, H., & Javed, M. (2022) · Computers & Security · doi:10.1016/j.cose.2022.103049 · Cited by 57
Enhancing training effectiveness for organizations through blockchain-enabled training effectiveness measurement (BETEM)
Jain, G., Sharma, N., & Shrivastava, A. (2021) · Journal of Organizational Change Management · doi:10.1108/jocm-10-2020-0303 · Cited by 56
The limits of programmed professional development on integration of information and communication technology in education
Peeraer, J., & Petegem, P. V. (2012) · Australasian Journal of Educational Technology · doi:10.14742/ajet.809 · Cited by 56
“The distinction in the model between learning and behaviour has drawn increased attention to the importance of the learning transfer process in making training truly effective (Bates, 2004).”
Background
Evaluating Executive Leadership Programs
Watkins, K. E., Lysø, I. H., & deMarrais, K. (2011) · Advances in Developing Human Resources · doi:10.1177/1523422311415643 · Cited by 68
“Therefore, transfer is not only influenced by the conditions in the organization after a development program but also by events both prior to and during training (Bates, 2004; Holton & Baldwin, 2003; Tannenbaum & Yukl, 1992). These factors led to the inclusion of interviews with supervisors in the U.S. case study to e…”
BackgroundInfluential
Digital literacy knowledge and needs of pharmacy staff: A systematic review
MacLure, K., & Stewart, D. C. (2016) · Journal of Innovation in Health Informatics · doi:10.14236/jhi.v23i3.840 · Cited by 54
Training evaluation in Italian corporate universities: a stakeholder‐based analysis
Guerci, M., Bartezzaghi, E., & Solari, L. (2010) · International Journal of Training and Development · doi:10.1111/j.1468-2419.2010.00359.x · Cited by 53
Strengthening health district management competencies in Ghana, Tanzania and Uganda: lessons from using action research to improve health workforce performance
Martineau, T., Raven, J., Aikins, M., Alonso‐Garbayo, A., Baine, S. O., Huss, R., Maluka, S. O., & Wyss, K. (2018) · BMJ Global Health · doi:10.1136/bmjgh-2017-000619 · Cited by 53
“This is a well-established framework for the evaluation of training, in spite of some criticism.(34) 35 It emphasises multiple (rather than single) measures of effectiveness and highlights the importance of transferring learning to behaviour change, and ultimately the impact on workforce performance and service delive…”
Method
Interprofessional Substance Use Disorder Education in Health Professions Education Programs: A Scoping Review
Muzyk, A. J., Smothers, Z., Andolsek, K. M., Bradner, M. K., Bratberg, J. P., Clark, S. A., Collins, K., Greskovic, G. A., Gruppen, L. D., MacEachern, M., et al. (2019) · Academic Medicine · doi:10.1097/acm.0000000000003053 · Cited by 52
Effectiveness of dementia education for professional care staff and factors influencing staff-related outcomes: An overview of systematic reviews
Rasmussen, B. M., Andersen, P. T., Waldorff, F. B., & Berg‐Beckhoff, G. (2023) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2023.104469 · Cited by 56
International Classification of Functioning, Disability and Health: Catalyst for interprofessional education and collaborative practice
Snyman, S., Pressentin, K. B. V., & Clarke, M. (2015) · Journal of Interprofessional Care · doi:10.3109/13561820.2015.1004041 · Cited by 51
Training Outcome Evaluation Model
Moldovan, L. (2016) · Procedia Technology · doi:10.1016/j.protcy.2016.01.166 · Cited by 49
Serious games: Quality characteristics evaluation framework and case study
Assaf, A. J., McCollum, B., & McMullan, P. (2018) · IEEE Integrated STEM Education Conference (ISEC) · doi:10.1109/isecon.2018.8340460 · Cited by 47
Evaluation of teacher professional learning workshops on the use of technology - a systematic review
Ahadi, A., Bower, M., Lai, J. W. M., Singh, A. K., & Garrett, M. J. (2021) · Professional Development in Education · doi:10.1080/19415257.2021.2011773 · Cited by 57
“Although Kirkpatrick’s method is well renown for its clear taxonomy of programme outcomes, this method potentially provides educators with only limited support to perform a complete evaluation of the professional development programme due to its linear evaluation nature (Bates 2004).”
Method
Trainees' reactions to training: shaping groups and courses for happier trainees
Giangreco, A., Carugati, A., Sebastiano, A., & Bella, D. D. (2010) · The International Journal of Human Resource Management · doi:10.1080/09585192.2010.516598 · Cited by 46
“Several scholars, mostly critics of the Kirkpatrick model (e.g., Tannenbaum and Woods 1992; Holton 1996; Bates 2004; Brown 2005; Blanchard and Thacker 2007), cite a lack of a proven causeand-effect relationship, as well as the inability to demonstrate the progressive importance of information moving from Level 1 (reac…”
Background
Using simulation-based learning to provide interprofessional education in diabetes to nutrition and dietetics and exercise physiology students through telehealth
O'Shea, M., Reeves, N. E., Bialocerkowski, A. E., & Cardell, E. (2019) · Advances in Simulation · doi:10.1186/s41077-019-0116-7 · Cited by 46
Evaluation of Capacity-Building Program of District Health Managers in India: A Contextualized Theoretical Framework
Srinivas, P. N., Marchal, B., Macq, J., Kegels, G., & Criel, B. (2014) · Frontiers in Public Health · doi:10.3389/fpubh.2014.00089 · Cited by 48
“In a critical analysis of evaluation practice, Bates (38) summarizes the common assumption (in literature) of causal linkages between these four levels as follows: the four “.”
BackgroundInfluential
Interventions for undergraduate and postgraduate medical learners with academic difficulties: A BEME systematic review: BEME Guide No. 56
Lacasse, M., Audétat, M., Boileau, É., Fon, N. C., Dufour, M.-H., Laferrière, M.-C., Lafleur, A., Rue, È. L., Lee, S., NENDAZ, M. R., et al. (2019) · Medical Teacher · doi:10.1080/0142159x.2019.1596239 · Cited by 49
Correction to: An Interprofessional Framework for Telebehavioral Health Competencies
Maheu, M. M., Drude, K. P., Hertlein, K. M., Lipschutz, R., Wall, K. M., & Hilty, D. M. (2018) · Journal of Technology in Behavioral Science · doi:10.1007/s41347-018-0046-6 · Cited by 44
“…model is that the greater the demonstration of the competency, the better learned that topic and that there is a causal linkage between the different levels when the research to date supports there is no such linkage and these levels may in fact operate independent of one another (Bates 2004).”
Background
Training evaluation levels and ROI: the case of a small logistics company
Curado, C., & Teixeira, S. M. (2014) · European journal of training and development · doi:10.1108/ejtd-05-2014-0037 · Cited by 43
Planning, construction and use of handmade simulators to enhance the teaching and learning in Obstetrics*
Knobel, R., Menezes, M. D. O., Santos, D. D. S., & Takemoto, M. L. S. (2020) · Revista Latino-Americana de Enfermagem · doi:10.1590/1518-8345.3684.3302 · Cited by 45
The evaluation of a home-based paediatric nursing service: concept and design development using the Kirkpatrick model
Jones, C. L., Fraser, J., & Randall, S. (2018) · Journal of research in nursing · doi:10.1177/1744987118786019 · Cited by 43
“These limitations have been highlighted in a critical analysis of KM (Bates, 2004).”
Background
Benefits of Massive Open Online Course Participation: Deductive Thematic Analysis
Blum, E. R., Stenfors, T., & Palmgren, P. J. (2020) · Journal of Medical Internet Research · doi:10.2196/17318 · Cited by 47
“It has been argued that the four-level model depicts an oversimplified view of learning and training effectiveness that does not take individual or contextual influences into account in the evaluation of the learning that occurs [46].”
Background
The impact of organizational support on the transfer of learning to practice
Futris, T. G., Schramm, D. G., Richardson, E. W., & Lee, T. K. (2015) · Children and Youth Services Review · doi:10.1016/j.childyouth.2015.01.019 · Cited by 42
Systematic review and narrative synthesis of the impact of Appreciative Inquiry in healthcare
Merriel, A., Wilson, A., Decker, E., Hussein, J., Larkin, M., Barnard, K., O’Dair, M., Costello, A. M. D. L., Malata, A. M., & Coomarasamy, A. (2022) · BMJ Open Quality · doi:10.1136/bmjoq-2022-001911 · Cited by 41
“61 However, it was a useful framework to consider heterogenous outcomes.”
Perception of educational value in clinical rotations by medical students
Kandiah, D. A. (2017) · Advances in Medical Education and Practice · doi:10.2147/amep.s129183 · Cited by 40
Simulation-based training and learning: A review on technology-enhanced education for the minerals industry
Bergamo, P. A. D. S., Streng, E. S., de Carvalho, M. A., Rosenkranz, J., & Ghorbani, Y. (2021) · Minerals Engineering · doi:10.1016/j.mineng.2021.107272 · Cited by 40
Evaluation of training programs in Russian manufacturing companies
Kucherov, D., & Manokhina, D. (2017) · European journal of training and development · doi:10.1108/ejtd-10-2015-0084 · Cited by 39
The game-based learning evaluation model (GEM): measuring the effectiveness of serious games using a standardised method
Oprins, E. A. P. B., Visschedijk, G. C., Roozeboom, M. B., Dankbaar, M. E. W., Trooster, W., & Schuit, S. C. E. (2015) · International Journal of Technology Enhanced Learning · doi:10.1504/ijtel.2015.074189 · Cited by 39
Training Evaluation in an Organization using Kirkpatrick Model: A Case Study of PIA
Rafiq, M. (2015) · Journal of Entrepreneurship & Organization Management · doi:10.4172/2169-026x.1000151 · Cited by 37
“Bates [16] presented the critical analysis of Kirkpatrick’s Model, according to him initially the model was developed in 1959 to identify the likeness and not likeness of training program by participants and”
Method
Aligning Objectives and Assessment in Responsible Conduct of Research Instruction
Antes, A. L., & DuBois, J. M. (2014) · Journal of Microbiology and Biology Education · doi:10.1128/jmbe.v15i2.852 · Cited by 36
Factors influencing construction workers’ intention to transfer occupational health and safety training
Pham, T. T., Lingard, H. C., & Zhang, R. P. (2023) · Safety Science · doi:10.1016/j.ssci.2023.106288 · Cited by 36
Evaluating A Continuing Medical Education Program: New World Kirkpatrick Model Approach
Liao, S.-C., & Hsu, S.-Y. (2019) · International Journal of Management Economics and Social Sciences · doi:10.32327/ijmess/8.4.2019.17 · Cited by 35
“First, the links between the levels are not strong and a causal relationship cannot be assumed (Alliger and Janak, 1989; Bates, 2004; Dixon, 1990).”
Background
The effects of interactive training of healthcare providers on the management of life-threatening emergencies in hospital
Merriel, A., Ficquet, J., Barnard, K., Kunutsor, S. K., Soar, J., Lenguerrand, E., Caldwell, D. M., Burden, C., Winter, C., Draycott, T. J., et al. (2019) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd012177.pub2 · Cited by 34
Changes in academic performance in the online, integrated system-based curriculum implemented due to the COVID-19 pandemic in a medical school in Korea
Kim, D.-H., Lee, H. J., Lin, Y., & Kang, Y. J. (2021) · Journal of Educational Evaluation for Health Professions · doi:10.3352/jeehp.2021.18.24 · Cited by 34
“The fact that high satisfaction does not necessarily guarantee high academic achievement is another reason why it is difficult to make a convincing argument for academic achievement despite students’ positive responses to online courses [6].”
Background
Effects of Individual and Work Environment Characteristics on Training Effectiveness: Evidence from Skill Certification System for Automotive Industry in Thailand
Homklin, T., Takahashi, Y., & Techakanont, K. (2013) · International Business Research · doi:10.5539/ibr.v6n12p1 · Cited by 33
“This finding is consistent with previous research on supervisor and peer support, such as that by Bates et al. (2000), Cromwell and Kolb (2004), Facteau et al.”
Result
Evaluating the Effectiveness of a Continuing Education Program for Prevention of Occupational Exposure to Needle Stick Injuries in Nursing Staff Based on Kirkpatrick's Model
Bijani, M., Rostami, K., Momennasab, M., & Yektatalab, S. (2017) · Journal of the National Medical Association · doi:10.1016/j.jnma.2017.11.002 · Cited by 33
An interprofessional framework for telebehavioral health competencies
Maheu, M. M., Drude, K. P., Hertlein, K. M., Lipschutz, R., Wall, K., & Hilty, D. M. (2017) · Journal of Technology in Behavioral Science · doi:10.1007/s41347-017-0038-y · Cited by 33
Impact of Using a 3D Visual Metaphor Serious Game to Teach History-Taking Content to Medical Students: Longitudinal Mixed Methods Pilot Study
Alyami, H., Alawami, M., Lyndon, M., Alyami, M. M., Coomarasamy, C., Henning, M. A., Hill, A. G., & Sundram, F. (2019) · JMIR Serious Games · doi:10.2196/13748 · Cited by 32
“CLT design principles mostly target the extraneous cognitive load imposed by poorly designed material [38,39].”
Background
Beyond Developing Leaders: Toward a Multinarrative Understanding of the Value of Leadership Development Programs
Vongswasdi, P., Leroy, H., Claeys, J., Anisman‐Razin, M., & van Dierendonck, D. (2023) · Academy of Management Learning and Education · doi:10.5465/amle.2021.0231 · Cited by 31
Assessing the Performance of a Tourism MOOC Using the Kirkpatrick Model: A Supplier’s Point of View
Lin, J., & Cantoni, L. (2017) · Springer eBooks · doi:10.1007/978-3-319-51168-9_10 · Cited by 31
The performance evaluation of corporate universities
Cappiello, G., & Pedrini, G. (2017) · Tertiary Education and Management · doi:10.1080/13583883.2017.1329452 · Cited by 31
Revue des modèles en évaluation de formation : approches conceptuelles individuelles et sociales
Gilibert, D., & Gillet, I. (2010) · Pratiques Psychologiques · doi:10.1016/j.prps.2009.03.006 · Cited by 21
“Ce modèle a, par ailleurs, l’avantage de synthétiser le processus complexe de l’évaluation des formations et propose une démarche rationnelle et suffisamment large (mais synthétique) pour correspondre aux besoins des professionnels de la formation (Bates, 2004).”
BackgroundInfluential
A practitioner friendly and scientifically robust training evaluation approach
Griffin, R. P. (2012) · Journal of Workplace Learning · doi:10.1108/13665621211250298 · Cited by 33
Evaluating the impact of healthcare provider training to improve tuberculosis management: a systematic review of methods and outcome indicators used
Wu, S., Roychowdhury, I., & Khan, M. S. (2016) · International Journal of Infectious Diseases · doi:10.1016/j.ijid.2016.11.421 · Cited by 30
SimLife® technology in surgical training – a dynamic simulation model
Danion, J., Brèque, C., Oriot, D., Faure, J., & Richer, J.-P. (2020) · Journal of Visceral Surgery · doi:10.1016/j.jviscsurg.2020.02.013 · Cited by 30
Linking the Use of Ergonomics Methods to Workplace Social Sustainability: The Ovako Working Posture Assessment System and Rapid Entire Body Assessment Method
Gajšek, B., Drăghici, A., Boatcă, M. E., Găureanu, A., & Robescu, D. F. (2022) · Sustainability · doi:10.3390/su14074301 · Cited by 30
An evaluation of approaches used to teach quality improvement to pre-registration healthcare professionals: An integrative review
Armstrong, L., Shepherd, A. J., & Harris, F. (2017) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2017.05.005 · Cited by 30
The effects of graduate competency-based education and mastery learning on patient care and return on investment: a narrative review of basic anesthetic procedures
Bisgaard, C. H., Rubak, S., Rodt, S. A., Petersen, J. A. K., & Musaeus, P. (2018) · BMC Medical Education · doi:10.1186/s12909-018-1262-7 · Cited by 32
“Positive results at a lower level are necessary for causal inference of a superior level effect as the result of an education intervention [54].”
BackgroundMethod
Did you learn something useful today? An analysis of how perceived utility relates to perceived learning and their predictiveness of satisfaction with training
Roszkowski, M. J., & Soven, M. K. (2010) · Performance Improvement Quarterly · doi:10.1002/piq.20082 · Cited by 29
“Whatever the case, most evaluations of training do not advance beyond the first level of reactions (Bassi, Benson, & Cheney, 1996; Lee & Pershing, 1999; Phillips, 1991; Ruona, Leimbach, Holton, & Bates, 2002; Sugrue & Rivera, 2005; Twitchell, Holton, & Trott, 2001). Newby (1992) coined the derisive-sounding term react…”
BackgroundMethod
Analysis of executive coaching effectiveness: a study from the coachee perspective
Albizu, E., Rekalde, I., Landeta, J., & Ferrín, P. F. (2019) · Cuadernos de Gestión · doi:10.5295/cdg.170876ea · Cited by 27
Integration of ultrasound in medical School: Effects on Physical Examination Skills of Undergraduates
Oteri, V., Occhipinti, F., Gribaudo, G., Marastoni, F., & Chisari, E. (2020) · Medical Science Educator · doi:10.1007/s40670-020-00921-4 · Cited by 28
“The impacts of the interventions were classified following Kirkpatrick’s adapted hierarchy [28], in line with guidance provided by BEME [29].”
Method
Teaching virtual apparel technology through industry collaboration: an assessment of pedagogical process and outcomes
Hodges, N. N., Watchravesringkan, K., Min, S., Lee, Y., & Seo, S. (2020) · International Journal of Fashion Design Technology and Education · doi:10.1080/17543266.2020.1742388 · Cited by 28
Training evaluation: a configurational analysis of success and failure of trainers and trainees
Bernardino, G., & Curado, C. (2020) · European journal of training and development · doi:10.1108/ejtd-10-2019-0177 · Cited by 29
Crew Resource Management and Teamwork Training in Health Care: A Review of the Literature and Recommendations for how to Leverage such Interventions to Enhance Patient Safety
Maynard, M. T., Marshall, D., & Dean, M. D. (2012) · Advances in health care management · doi:10.1108/s1474-8231(2012)0000013008 · Cited by 28
Measuring the impact of information skills training: a survey of health libraries in E ngland
Ayre, S. J., Barbrook, J., Engel, C., Lacey, P., Phul, A., Stevenson, P., & Toft, S. (2014) · Health Information & Libraries Journal · doi:10.1111/hir.12079 · Cited by 27
Effects of Using Immersive Media on the Effectiveness of Training to Prevent Ergonomics Risks
Diego-Más, J. A., Alcaide-Marzal, J., & Poveda-Bautista, R. (2020) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph17072592 · Cited by 30
“Although there are more recent models to assess training programs [41–43], Kirkpatrick’s model is the most commonly used [44], acting as the fundamental scheme for educational evaluation [45,46].”
Method
Training evaluation: an analysis of the stakeholders' evaluation needs
Guerci, M., & Vinante, M. (2011) · Journal of European Industrial Training · doi:10.1108/03090591111128342 · Cited by 27
“This assumption has generated “the perception among training evaluators that establishing level four results will provide the most useful information about training program effectiveness” ( Bates, 2004, p. 342”
Background
Collusion with denial: leadership development and its evaluation
King, E., & Nesbit, P. (2015) · Journal of Management Development · doi:10.1108/jmd-02-2013-0023 · Cited by 27
Asthma education for school staff
Kew, K. M., Carr, R., Donovan, T., & Gordon, M. A. (2017) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd012255.pub2 · Cited by 27
Feasibility of Training Physical Therapists to Deliver the Theory-Based Self-Management of Osteoarthritis and Low Back Pain Through Activity and Skills (SOLAS) Intervention Within a Trial
Keogh, A., Matthews, J. J., Segurado, R., & Hurley, D. A. (2017) · Physical Therapy · doi:10.1093/ptj/pzx105 · Cited by 30
Evaluation of Advanced Field Epidemiology Training Programs in the Eastern Mediterranean Region: A Multi-Country Study
Nsour, M. A., Khader, Y. S., Bashier, H., & Alsoukhni, M. A. (2021) · Frontiers in Public Health · doi:10.3389/fpubh.2021.684174 · Cited by 26
“It mainly provides a straightforward systematic way for evaluating training outcomes and provides information that can be used to assess the extent to which training programs have achieved certain objectives (23).”
Background
In‐company training in Catalonia: organizational structure, funding, evaluation and economic impact*
Huerta, M. E., Audet, X. L., & Peregort, O. P. (2006) · International Journal of Training and Development · doi:10.1111/j.1468-2419.2005.00249.x · Cited by 26
Evaluating student learning outcomes in oral health knowledge and skills
Lewis, A., Edwards, S., Whiting, G., & Donnelly, F. (2017) · Journal of Clinical Nursing · doi:10.1111/jocn.14082 · Cited by 26
“Level two evaluation involves measuring whether learning has taken place in terms of participants’ knowledge and/or skills (Barr et al., 1999; Bates, 2004; Beech & Leather, 2006; Curran & Fleet, 2005; Sargent et al., 2011; Smidt et al., 2009).”
Influential

Authors: Bates, R.
Affiliations: Louisiana State University (US)
Corresponding author: Bates, Reid
Venue: Evaluation and Program Planning
Publisher: Elsevier BV
Published: 2004-08
Volume: 27
Issue: 3
Pages: 341-347
ISSN: 0149-7189 (ISSN-L)
DOI: 10.1016/j.evalprogplan.2004.04.011
OpenAlex: W2097592370
Cited by: 623
References: 44
Citations by year: 2026: 53 · 2025: 23 · 2024: 42 · 2023: 53 · 2022: 49 · 2021: 39 · 2020: 43 · 2019: 44 · 2018: 32 · 2017: 45 · 2016: 32 · 2015: 32 · 2014: 30 · 2013: 29 · 2012: 22
FWCI: 4.09
Type: journal article
Language: en
Open access: closed
Licence: www.elsevier.com/tdm/userlicense/1.0/ (tdm)
Locations: Evaluation and Program Planning (publishedVersion)
RePEc: Research Papers in Economics (submittedVersion)
Topics: Social Sciences › Psychology › Applied Psychology › Human Resource Development and Performance Evaluation
Social Sciences › Public Administration › Evaluation and Performance Assessment
Social Sciences › Education › Evaluation of Teaching Practices
Keywords: Kirkpatrick model, training evaluation, beneficence, moral obligation, formative assessment, summative assessment, training effectiveness, human resource development, evaluation limitations

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

A critical analysis of evaluation practice: the Kirkpatrick model and the principle of beneficence

Overview

Integrity signals

Verification

Claims the paper makes

Claims other papers make about this one

Citations

Cites

Cited by

Related

Metadata

Record

About these checks