A critical analysis of evaluation practice: the Kirkpatrick model and the principle of beneficence
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Course evaluation using questionnaires and assessment of participant satisfaction are level 1 criteria of Kirkpatrick's evaluation model of training programs (subjective evaluation).
“Course evaluation using questionnaires and assessment of participant satisfaction are level 1 criteria of Kirkpatrick’s evaluation model of training programs (subjective evaluation).”
From Neurosurgery resident training using blended learning concepts: Course development and participant evaluation (2022) · cites this paper as [18]
It remains to be evaluated whether these skills will be transferred actively into clinical practice where they can have an impact on patient outcome (level 3).
“it remains to be evaluated whether these skills will be transferred actively into clinical practice where they can have an impact on patient outcome (level 3).18”
From Neurosurgery resident training using blended learning concepts: Course development and participant evaluation (2022) · cites this paper as [18]
Analysis of objective, automated measures of the e-learning account for level 2.
“Analysis of objective, automated measures of the e-learning account for level 2.”
From Neurosurgery resident training using blended learning concepts: Course development and participant evaluation (2022) · cites this paper as [18]
Cites
Evaluating Training Programs : The Four Levels
Kirkpatrick, D. L. (2009) · Cited by 4,720 · see citing works
Principles of Biomedical Ethics.
(1984) · Annals of Internal Medicine · doi:10.7326/0003-4819-100-1-168_1 · Cited by 4,559 · see citing works
Utilization-Focused Evaluation
(2018) · The SAGE Encyclopedia of Educational Research, Measurement, and Evaluation · doi:10.4135/9781506326139.n729 · Cited by 2,315 · see citing works
Kraiger, K., Ford, J. K., & Salas, E. (1993) · Journal of Applied Psychology · doi:10.1037/0021-9010.78.2.311 · Cited by 1,831 · cited 1× · see citing works
“Kraiger, Ford and Salas (1993) have forwarded a multidimensional model of learning outcomes from training and have described a process training evaluators could use to developing learning evaluation measures (Kraiger & Jung, 1997).”
BackgroundThe science of training: a decade of progress
Salas, E., & Cannon-Bowers, J. A. (2001) · Annual Review of Psychology · doi:10.1146/annurev.psych.52.1.471 · Cited by 1,703 · see citing works
Techniques for Evaluating Training Programs
Kirkpatrick, D. L. (1975) · Training and development journal · Cited by 1,234 · see citing works
Training and Development in Work Organizations
Tannenbaum, S. I., & YUKL, G. A. (1992) · Annual Review of Psychology · doi:10.1146/annurev.ps.43.020192.002151 · Cited by 1,023 · see citing works
Knowles, M. S., Holton, E. F., & Swanson, R. A. (1973) · Cited by 970 · see citing works
The flawed four‐level evaluation model
Holton, E. F. (1996) · Human Resource Development Quarterly · doi:10.1002/hrdq.3920070103 · Cited by 965 · see citing works
Principles of Biomedical Ethics
Beauchamp, T. L., & Childless, J. F. (1991) · International Clinical Psychopharmacology · doi:10.1097/00004850-199100620-00013 · Cited by 964 · see citing works
A META‐ANALYSIS OF THE RELATIONS AMONG TRAINING CRITERIA
ALLIGER, G. M., TANNENBAUM, S. I., BENNETT, W., TRAVER, H., & SHOTLAND, A. (1997) · Personnel Psychology · doi:10.1111/j.1744-6570.1997.tb00911.x · Cited by 948 · see citing works
Reeves, M. (1993) · Cited by 906 · see citing works
Development of a generalized learning transfer system inventory
Holton, E. F., Bates, R. A., & Ruona, W. E. A. (2000) · Human Resource Development Quarterly · doi:10.1002/1532-1096(200024)11:4 · Cited by 881 · see citing works
Applying trained skills on the job: The importance of the work environment.
Tracey, J. B., Tannenbaum, S. I., & Kavanagh, M. J. (1995) · Journal of Applied Psychology · doi:10.1037/0021-9010.80.2.239 · Cited by 826 · see citing works
Shakhov, A. N., Collart, M. A., Vassalli, P., Nedospasov, S. A., & Jongeneel, C. V. (1990) · The Journal of Experimental Medicine · doi:10.1084/jem.171.1.35 · Cited by 819 · see citing works
The relationship between organizational transfer climate and positive transfer of training
Rouiller, J. Z., & Goldstein, I. L. (1993) · Human Resource Development Quarterly · doi:10.1002/hrdq.3920040408 · Cited by 771 · see citing works
Foundations of Human Resource Development
McGuire, D. (2014) · SAGE Publications Ltd eBooks · doi:10.4135/9781473906891.n1 · Cited by 725 · see citing works
KIRKPATRICK'S LEVELS OF TRAINING CRITERIA: THIRTY YEARS LATER
Alliger, G. M., & Janak, E. A. (1989) · Personnel Psychology · doi:10.1111/j.1744-6570.1989.tb00661.x · Cited by 722 · see citing works
Kitchener, K. S. (1984) · The Counseling Psychologist · doi:10.1177/0011000084123005 · Cited by 560 · see citing works
MODELING JOB PERFORMANCE IN A POPULATION OF JOBS
Campbell, J. P., McHenry, J. J., & Wise, L. L. (1990) · Personnel Psychology · doi:10.1111/j.1744-6570.1990.tb01561.x · Cited by 507 · see citing works
FACTORS AFFECTING THE OPPORTUNITY TO PERFORM TRAINED TASKS ON THE JOB
Ford, J. K., Quiñones, M. A., Sego, D. J., & Sorra, J. (1992) · Personnel Psychology · doi:10.1111/j.1744-6570.1992.tb00858.x · Cited by 498 · see citing works
Training and development handbook: A guide to human resource development
Craig, R. L. (1976) · Medical Entomology and Zoology · Cited by 452 · see citing works
Smith, M. C. (1991) · doi:10.1007/978-1-349-21542-3_4 · Cited by 400 · see citing works
Utilization-focused evaluation
Westbrook, L. (1998) · Library & Information Science Research · doi:10.1016/s0740-8188(98)90032-9 · Cited by 389 · see citing works
SIOP (2002) · PsycEXTRA Dataset · doi:10.1037/e553002012-001 · Cited by 277 · see citing works
Student Evaluations of Teachers
Rodin, M. J., & Rodin, B. (1972) · Science · doi:10.1126/science.177.4055.1164 · Cited by 247 · see citing works
Cannon‐Bowers, J. A., Eduardo Salas, A., Tannenbaum, S. I., & Mathieu, J. E. (1995) · Military Psychology · doi:10.1207/s15327876mp0703_1 · Cited by 206 · see citing works
Bates, R. A., Holton, E. F., Seyler, D. L., & Carvalho, M. A. (2000) · Human Resource Development International · doi:10.1080/136788600361920 · Cited by 142 · see citing works
Foundations of Human Resource Development
McGuire, D. (2011) · SAGE Publications Ltd eBooks · doi:10.4135/9781446251065.n1 · Cited by 126 · see citing works
Measuring Knowledge Organization as a Method for Assessing Learning during Training
Kraiger, K., Eduardo Salas, A., & Cannon‐Bowers, J. A. (1995) · Human Factors The Journal of the Human Factors and Ergonomics Society · doi:10.1518/001872095778995535 · Cited by 103 · see citing works
Ethical Issues in Graduate Education: Faculty and Student Responsibilities
Brown, R. D., & Krager, L. (1985) · The Journal of Higher Education · doi:10.2307/1981303 · Cited by 94 · see citing works
Applied ethics of program evaluation
Sieber, J. E. (1997) · Evaluation and Program Planning · doi:10.1016/s0149-7189(97)85673-x · Cited by 66 · see citing works
Michalski, G. V., & Cousins, J. B. (2000) · Evaluation and Program Planning · doi:10.1016/s0149-7189(00)00005-7 · Cited by 58 · see citing works
Who's Afraid of Level 4 Evaluation? A Practical Approach.
Shelton, S. K., & Alliger, G. M. (1993) · Cited by 50 · see citing works
A quantitative needs assessment technique for cross-cultural work adjustment training
Selmer, J. (2000) · Human Resource Development Quarterly · doi:10.1002/1532-1096(200023)11:3 · Cited by 50 · see citing works
Does Your Training Make a Difference? Prove It!.
Geber, B. (1995) · Cited by 35 · see citing works
Newstrom, J. W. (1995) · Human Resource Development Quarterly · doi:10.1002/hrdq.3920060310 · Cited by 34 · see citing works
Linking training objectives to evaluation criteria.
Kraiger, K., & Jung, K. M. (1997) · American Psychological Association eBooks · doi:10.1037/10260-006 · Cited by 32 · see citing works
Invited reaction: Linking learning with financial performance
Baldwin, T. T., & Danielson, C. (2002) · Human Resource Development Quarterly · doi:10.1002/hrdq.1011 · Cited by 26 · see citing works
Implementing an ROI Measurement Process at Dell Computer
Tesoro, F. (2008) · Performance Improvement Quarterly · doi:10.1111/j.1937-8327.1998.tb00110.x · Cited by 15 · see citing works
Valuing Learning: The Measurement Journey.
Wang, G. (2003) · Educational Technology archive · Cited by 12 · see citing works
Flagged first, then by citations
3 of the 44 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Program evaluation models and related theories: AMEE Guide No. 67
FRYE, A. W., & Hemmer, P. A. (2012) · Medical Teacher · doi:10.3109/0142159x.2012.668637 · Cited by 641 · see citing works
Thieme, A., Belgrave, D., & Doherty, G. (2020) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3398069 · Cited by 406 · see citing works
“The principle of beneficence however does not only entail encouraging human flourishing and wellbeing by doing the right thing, but also suggests to ‘do it well’ [13].”
BackgroundSurr, C. A., Gates, C., Irving, D., Oyebode, J., Smith, S. J., Parveen, S., Drury, M., & Dennison, A. (2017) · Review of Educational Research · doi:10.3102/0034654317723305 · Cited by 327 · see citing works
“Kirkpatrick’s (1979, 1984) “Return on Investment” model developed specifically for evaluation of training consists of four levels and is widely adopted in the evaluation of training and education provision (Bates, 2004).”
A Critique of Kirkpatrick's Evaluation Model
Reio, T. G., Rocco, T. S., Smith, D. H., & Chang, E. (2017) · New Horizons in Adult Education and Human Resource Development · doi:10.1002/nha3.20178 · Cited by 227 · see citing works
“, Bates, 2004; Holton, 1996; Swanson, 1994). Some researchers, however, have developed their own models based on Kirkpatrick’s. Bushnell (1990) and Hayes, Scott, Braczinskas, Scaccia, Stout, and Wandersman (2016) contend that Kirkpatrick’s model focuses only on what happens after the training, and therefore is not suf…”
BackgroundMethodInfluentialSanches, P., Janson, A., Karpashevich, P., Nadal, C., Qu, C., Roquet, C. D., Umair, M., Windlin, C., Doherty, G. J., Höök, K., et al. (2019) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300475 · Cited by 227 · see citing works
“Others argued for the value of increasing user’s feeling of control by augmenting tracking technologies with support for reflection [18], or increasing user’s engagement with technology through DIY approaches to design [148].”
BackgroundInfluentialGrohmann, A., & Kauffeld, S. (2013) · International Journal of Training and Development · doi:10.1111/ijtd.12005 · Cited by 193 · see citing works
“In organizations often broader evaluation issues, such as exploring determinants of successful training transfer, have to be addressed in order to optimize future training courses (e.g. Bates, 2004).”
BackgroundHeydari, M. R., Taghva, F., Amini, M., & Delavari, S. (2019) · BMC Research Notes · doi:10.1186/s13104-019-4421-y · Cited by 175 · see citing works
Low- versus high-fidelity simulations in teaching and assessing clinical skills
Munshi, F. M., Lababidi, H. M., & Alyousef, S. (2015) · Journal of Taibah University Medical Sciences · doi:10.1016/j.jtumed.2015.01.008 · Cited by 163 · see citing works
“Fourth, the learning impact of simulation studies can be categorized by Kirkpatrick’s four levels of training evaluation: reaction, learning, behaviour, and results.(30) The first level includes the assessment of participants’ reactions to the use of simulation.”
Kirkpatrick Model: Its Limitations as Used in Higher Education Evaluation
Cahapay, M. B. (2021) · International Journal of Assessment Tools in Education · doi:10.21449/ijate.856143 · Cited by 162 · see citing works
“…higher education fields and aspects (see Quintas et al., 2017 on instructional approach; Baskin, 2001 The reviews of Alliger and Janak (1989), Bates (2004), and Reio et al., (2017) help understand the current state of the Kirkpatrick model by overtly tackling its inherent limitations in the…”
BackgroundEvaluation in health professions education—Is measuring outcomes enough?
Allen, L., Hay, M., & Palermo, C. E. (2021) · Medical Education · doi:10.1111/medu.14654 · Cited by 136 · see citing works
Roffarello, A. M., & De Russis, L. (2023) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3571810 · Cited by 137 · see citing works
“The principle of beneicence, however, is not only related to łdoing good,ž but also to łdo it wellž [23], i.”
BackgroundAbdel-All, M., Putica, B., Praveen, D., Abímbọ́lá, Ṣ., & Joshi, R. (2017) · BMJ Open · doi:10.1136/bmjopen-2016-015529 · Cited by 133 · see citing works
Alsalamah, A., & Callinan, C. J. (2021) · Education Sciences · doi:10.3390/educsci11030116 · Cited by 127 · see citing works
“Although Holton [43], and Alliger and Janak [49], have criticised the hierarchical nature of the model, studies and empirical results do not provide sufficient evidence to support this assumption of its hierarchical nature [40].”
BackgroundMethodInfluentialBrown, S. M., Baker, C. N., & Wilcox, P. D. (2011) · Psychological Trauma Theory Research Practice and Policy · doi:10.1037/a0025269 · Cited by 126 · see citing works
Evaluating the effectiveness of virtual reality for safety-relevant training: a systematic review
Stefan, H., Mortimer, M., & Horan, B. (2023) · Virtual Reality · doi:10.1007/s10055-023-00843-7 · Cited by 124 · see citing works
“Despite its criticisms over the last five decades and the proposals of newer evaluation models, Kirkpatrick’s model continues to be the most relevant and widely used model by different organizations as the basis for training evaluation (Bates 2004; Reio et al. 2017).”
MethodHas the new Kirkpatrick generation built a better hammer for our evaluation toolbox?
Moreau, K. A. (2017) · Medical Teacher · doi:10.1080/0142159x.2017.1337874 · Cited by 114 · see citing works
“Criticism 3: Kirkpatrick’s causal chain is unfounded Bates (2004) highlights the limited evidence to support the causal chain embedded within Kirkpatrick’s original model.”
BackgroundGiangreco, A., Sebastiano, A., & Peccei, R. (2009) · The International Journal of Human Resource Management · doi:10.1080/09585190802528417 · Cited by 112 · see citing works
“…Allan and Birdi 1999; Blanchard, Thacker and Way 2000; Tracey, Hinkin, Tannenbaum and Mathieu 2001), including a few criticisms (e.g. Clement 1982; Alliger and Janak 1989; Tannenbaum and Woods 1992; Bates 2004; Brown 2005), and a muscular confrontation between Holton (1996) and Kirkpatrick (1996b).”
BackgroundSitzmann, T., & Weinhardt, J. M. (2017) · Human Resource Management Review · doi:10.1016/j.hrmr.2017.04.001 · Cited by 100 · see citing works
The Kirkpatrick model for training evaluation: bibliometric analysis after 60 years (1959–2020)
Alsalamah, A., & Callinan, C. (2021) · Industrial and Commercial Training · doi:10.1108/ict-12-2020-0115 · Cited by 114 · see citing works
Training and employee development for improved performance.
Brown, K. G., & Sitzmann, T. (2010) · American Psychological Association eBooks · doi:10.1037/12170-016 · Cited by 112 · see citing works
Surr, C. A., & Gates, C. (2017) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2017.08.002 · Cited by 107 · see citing works
“A model widely adopted in the evaluation of training and education provision (Bates, 2004) is Kirkpatrick’s (1984, 1979) four level ‘Return on Investment’ model.”
Methodvan den Eertwegh, V., van Dulmen, S., van Dalen, J., Scherpbier, A. J. J. A., & van der Vleuten, C. P. M. (2012) · Patient Education and Counseling · doi:10.1016/j.pec.2012.06.008 · Cited by 103 · see citing works
“In recent years however the suitability of Kirkpatrick’s levels to evaluate the effectiveness of interventions has been questioned [24–26,27].”
BackgroundThe influence of training and training transfer factors on organisational learning and performance
Dermol, V., & Čater, T. (2013) · Personnel Review · doi:10.1108/00483481311320435 · Cited by 97 · see citing works
Kirkpatrick evaluation model for in-service training on cardiopulmonary resuscitation
Akbari, M., Dorri, S., & Sedeh, M. D. (2016) · Iranian Journal of Nursing and Midwifery Research · doi:10.4103/1735-9066.193396 · Cited by 97 · see citing works
“Hence, this model simplifies the complex process of evaluation of training programs.[10]”
MethodA systematic review of approaches to assessing cybersecurity awareness
Rahim, N. H. A., Hamid, S. B., Kiah, M. L. M., Shamshirband, S., & Furnell, S. (2015) · Kybernetes · doi:10.1108/k-12-2014-0283 · Cited by 94 · see citing works
“This refers to a mixture of quantitative and qualitative methodologies, which can help simplify the complex process of understanding a programme ’ s effectiveness (Bates, 2004).”
BackgroundAbdulghani, H. M., Shaik, S. A., Khamis, N., Al-Drees, A. A., Irshad, M., Khalil, M. S., AlHaqwi, A. I., & Isnani, A. C. (2014) · Medical Teacher · doi:10.3109/0142159x.2014.886012 · Cited by 96 · see citing works
“However Donald Kirkpatrick model (Kirkpatrick & Kirkpatrick 2006) has served as the primary organizing design for the evaluations of training since thirty years (Bates 2004).”
MethodMeasuring the effectiveness of virtual training: A systematic review
Strojny, P., & Dużmańska-Misiarczyk, N. (2023) · Computers & Education X Reality · doi:10.1016/j.cexr.2022.100006 · Cited by 107 · see citing works
E-Learning Theories, Components, and Cloud Computing-Based Learning Platforms
Kumar, V., & Sharma, D. (2021) · International Journal of Web-Based Learning and Teaching Technologies · doi:10.4018/ijwltt.20210501.oa1 · Cited by 73 · see citing works
Kuyt, K., Park, S. H., Chang, T. P., Jung, T. H., & MACKINNON, R. J. (2021) · Advances in Simulation · doi:10.1186/s41077-021-00158-0 · Cited by 72 · see citing works
“It consists of four levels of evidence, assessing the reaction to training, skills learnt, behaviours and outcomes [16] (Table 2).”
BackgroundTraining professionals' usage and understanding of K irkpatrick's Level 3 and Level 4 evaluations
Kennedy, P. E., Chyung, S. Y., Winiecki, D. J., & Brinkerhoff, R. O. (2013) · International Journal of Training and Development · doi:10.1111/ijtd.12023 · Cited by 72 · see citing works
Evaluation and impact of cardiotocography training programmes: a systematic review
Pehrson, C., Sorensen, J., & Amer‐Wåhlin, I. (2011) · BJOG An International Journal of Obstetrics & Gynaecology · doi:10.1111/j.1471-0528.2011.03021.x · Cited by 75 · see citing works
Revisiting Kirkpatrick's model ― an evaluation of an academic training course
Rajeev, P., Madan, M., & Jayarajan, K. (2009) · Cited by 72 · see citing works
Giangreco, A., Carugati, A., & Sebastiano, A. (2010) · Personnel Review · doi:10.1108/00483481011017390 · Cited by 68 · see citing works
“Of the many scholars who have commented, Bates's (2004)”
BackgroundAbdel-All, M., Thrift, A. G., Riddell, M. A., Thankappan, K. R., Mini, G., Chow, C. K., Maulik, P. K., Mahal, A. S., Guggilla, R. K., Kalyanram, K., et al. (2018) · BMC Health Services Research · doi:10.1186/s12913-018-3140-8 · Cited by 67 · see citing works
“Use of these models also draws attention to the learning transfer process and the behavioural change of the trainee to achieve the desired outcome of the training [39].”
BackgroundSeisTutor: A Custom-Tailored Intelligent Tutoring System and Sustainable Education
Singh, N., Gunjan, V. K., Mishra, A. K., Mishra, R. K., & Nawaz, N. (2022) · Sustainability · doi:10.3390/su14074167 · Cited by 76 · see citing works
Gordon, M. A., Farnan, J. M., Grafton‐Clarke, C., Ahmed, R., Gurbutt, D., McLachlan, J. C., & Daniel, M. M. (2019) · Medical Teacher · doi:10.1080/0142159x.2018.1562166 · Cited by 60 · see citing works
“The impacts of the interventions were classified in accordance with Kirkpatrick’s adapted hierarchy (Bates 2004), in line with guidance provided by BEME (Hammick et al.”
MethodA Mixed Methods Examination of the Influence of Dimensions of Support on Training Transfer
Schindler, L. A., & Burkholder, G. (2016) · Journal of Mixed Methods Research · doi:10.1177/1558689814557132 · Cited by 59 · see citing works
“Many organizations rely on the Kirkpatrick (1976) model as a framework to evaluate their training programs (Bates, 2004; Patel, 2010; Salas & Cannon-Bowers, 2001).”
MethodEvaluating protection motivation based cybersecurity awareness training on Kirkpatrick's Model
Khan, N. F., Ikram, N., Murtaza, H., & Javed, M. (2022) · Computers & Security · doi:10.1016/j.cose.2022.103049 · Cited by 57 · see citing works
Jain, G., Sharma, N., & Shrivastava, A. (2021) · Journal of Organizational Change Management · doi:10.1108/jocm-10-2020-0303 · Cited by 56 · see citing works
Peeraer, J., & Petegem, P. V. (2012) · Australasian Journal of Educational Technology · doi:10.14742/ajet.809 · Cited by 56 · see citing works
“The distinction in the model between learning and behaviour has drawn increased attention to the importance of the learning transfer process in making training truly effective (Bates, 2004).”
BackgroundEvaluating Executive Leadership Programs
Watkins, K. E., Lysø, I. H., & deMarrais, K. (2011) · Advances in Developing Human Resources · doi:10.1177/1523422311415643 · Cited by 56 · see citing works
“Therefore, transfer is not only influenced by the conditions in the organization after a development program but also by events both prior to and during training (Bates, 2004; Holton & Baldwin, 2003; Tannenbaum & Yukl, 1992). These factors led to the inclusion of interviews with supervisors in the U.S. case study to e…”
BackgroundInfluentialDigital literacy knowledge and needs of pharmacy staff: A systematic review
MacLure, K., & Stewart, D. C. (2016) · Journal of Innovation in Health Informatics · doi:10.14236/jhi.v23i3.840 · Cited by 54 · see citing works
Training evaluation in Italian corporate universities: a stakeholder‐based analysis
Guerci, M., Bartezzaghi, E., & Solari, L. (2010) · International Journal of Training and Development · doi:10.1111/j.1468-2419.2010.00359.x · Cited by 53 · see citing works
Martineau, T., Raven, J., Aikins, M., Alonso‐Garbayo, A., Baine, S. O., Huss, R., Maluka, S. O., & Wyss, K. (2018) · BMJ Global Health · doi:10.1136/bmjgh-2017-000619 · Cited by 53 · see citing works
“This is a well-established framework for the evaluation of training, in spite of some criticism.(34) 35 It emphasises multiple (rather than single) measures of effectiveness and highlights the importance of transferring learning to behaviour change, and ultimately the impact on workforce performance and service delive…”
MethodMuzyk, A. J., Smothers, Z., Andolsek, K. M., Bradner, M. K., Bratberg, J. P., Clark, S. A., Collins, K., Greskovic, G. A., Gruppen, L. D., MacEachern, M., et al. (2019) · Academic Medicine · doi:10.1097/acm.0000000000003053 · Cited by 52 · see citing works
Rasmussen, B. M., Andersen, P. T., Waldorff, F. B., & Berg‐Beckhoff, G. (2023) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2023.104469 · Cited by 56 · see citing works
Snyman, S., Pressentin, K. B. V., & Clarke, M. (2015) · Journal of Interprofessional Care · doi:10.3109/13561820.2015.1004041 · Cited by 51 · see citing works
Training Outcome Evaluation Model
Moldovan, L. (2016) · Procedia Technology · doi:10.1016/j.protcy.2016.01.166 · Cited by 49 · see citing works
Serious games: Quality characteristics evaluation framework and case study
Assaf, A. J., McCollum, B., & McMullan, P. (2018) · IEEE Integrated STEM Education Conference (ISEC) · doi:10.1109/isecon.2018.8340460 · Cited by 47 · see citing works
Evaluation of teacher professional learning workshops on the use of technology - a systematic review
Ahadi, A., Bower, M., Lai, J. W. M., Singh, A. K., & Garrett, M. J. (2021) · Professional Development in Education · doi:10.1080/19415257.2021.2011773 · Cited by 57 · see citing works
“Although Kirkpatrick’s method is well renown for its clear taxonomy of programme outcomes, this method potentially provides educators with only limited support to perform a complete evaluation of the professional development programme due to its linear evaluation nature (Bates 2004).”
MethodTrainees' reactions to training: shaping groups and courses for happier trainees
Giangreco, A., Carugati, A., Sebastiano, A., & Bella, D. D. (2010) · The International Journal of Human Resource Management · doi:10.1080/09585192.2010.516598 · Cited by 46 · see citing works
“Several scholars, mostly critics of the Kirkpatrick model (e.g., Tannenbaum and Woods 1992; Holton 1996; Bates 2004; Brown 2005; Blanchard and Thacker 2007), cite a lack of a proven causeand-effect relationship, as well as the inability to demonstrate the progressive importance of information moving from Level 1 (reac…”
BackgroundO'Shea, M., Reeves, N. E., Bialocerkowski, A. E., & Cardell, E. (2019) · Advances in Simulation · doi:10.1186/s41077-019-0116-7 · Cited by 46 · see citing works
Prashanth, N. S., Marchal, B., Kegels, G., & Criel, B. (2014) · Frontiers in Public Health · doi:10.3389/fpubh.2014.00089 · Cited by 46 · see citing works
“In a critical analysis of evaluation practice, Bates (38) summarizes the common assumption (in literature) of causal linkages between these four levels as follows: the four “.”
BackgroundInfluentialLacasse, M., Audétat, M., Boileau, É., Fon, N. C., Dufour, M.-H., Laferrière, M.-C., Lafleur, A., Rue, È. L., Lee, S., NENDAZ, M. R., et al. (2019) · Medical Teacher · doi:10.1080/0142159x.2019.1596239 · Cited by 45 · see citing works
Correction to: An Interprofessional Framework for Telebehavioral Health Competencies
Maheu, M. M., Drude, K. P., Hertlein, K. M., Lipschutz, R., Wall, K. M., & Hilty, D. M. (2018) · Journal of Technology in Behavioral Science · doi:10.1007/s41347-018-0046-6 · Cited by 44 · see citing works
“…model is that the greater the demonstration of the competency, the better learned that topic and that there is a causal linkage between the different levels when the research to date supports there is no such linkage and these levels may in fact operate independent of one another (Bates 2004).”
BackgroundTraining evaluation levels and ROI: the case of a small logistics company
Curado, C., & Teixeira, S. M. (2014) · European journal of training and development · doi:10.1108/ejtd-05-2014-0037 · Cited by 43 · see citing works
Knobel, R., Menezes, M. D. O., Santos, D. D. S., & Takemoto, M. L. S. (2020) · Revista Latino-Americana de Enfermagem · doi:10.1590/1518-8345.3684.3302 · Cited by 45 · see citing works
Jones, C. L., Fraser, J., & Randall, S. (2018) · Journal of research in nursing · doi:10.1177/1744987118786019 · Cited by 43 · see citing works
“These limitations have been highlighted in a critical analysis of KM (Bates, 2004).”
BackgroundBenefits of Massive Open Online Course Participation: Deductive Thematic Analysis
Blum, E. R., Stenfors, T., & Palmgren, P. J. (2020) · Journal of Medical Internet Research · doi:10.2196/17318 · Cited by 43 · see citing works
“It has been argued that the four-level model depicts an oversimplified view of learning and training effectiveness that does not take individual or contextual influences into account in the evaluation of the learning that occurs [46].”
BackgroundThe impact of organizational support on the transfer of learning to practice
Futris, T. G., Schramm, D. G., Richardson, E. W., & Lee, T. K. (2015) · Children and Youth Services Review · doi:10.1016/j.childyouth.2015.01.019 · Cited by 42 · see citing works
Systematic review and narrative synthesis of the impact of Appreciative Inquiry in healthcare
Merriel, A., Wilson, A., Decker, E., Hussein, J., Larkin, M., Barnard, K., O’Dair, M., Costello, A. M. D. L., Malata, A. M., & Coomarasamy, A. (2022) · BMJ Open Quality · doi:10.1136/bmjoq-2022-001911 · Cited by 41 · see citing works
“61 However, it was a useful framework to consider heterogenous outcomes.”
Perception of educational value in clinical rotations by medical students
Kandiah, D. A. (2017) · Advances in Medical Education and Practice · doi:10.2147/amep.s129183 · Cited by 40 · see citing works
Bergamo, P. A. D. S., Streng, E. S., de Carvalho, M. A., Rosenkranz, J., & Ghorbani, Y. (2021) · Minerals Engineering · doi:10.1016/j.mineng.2021.107272 · Cited by 40 · see citing works
Evaluation of training programs in Russian manufacturing companies
Kucherov, D., & Manokhina, D. (2017) · European journal of training and development · doi:10.1108/ejtd-10-2015-0084 · Cited by 39 · see citing works
Oprins, E. A. P. B., Visschedijk, G. C., Roozeboom, M. B., Dankbaar, M. E. W., Trooster, W., & Schuit, S. C. E. (2015) · International Journal of Technology Enhanced Learning · doi:10.1504/ijtel.2015.074189 · Cited by 39 · see citing works
Training Evaluation in an Organization using Kirkpatrick Model: A Case Study of PIA
Rafiq, M. (2015) · Journal of Entrepreneurship & Organization Management · doi:10.4172/2169-026x.1000151 · Cited by 37 · see citing works
“Bates [16] presented the critical analysis of Kirkpatrick’s Model, according to him initially the model was developed in 1959 to identify the likeness and not likeness of training program by participants and”
MethodAligning Objectives and Assessment in Responsible Conduct of Research Instruction
Antes, A. L., & DuBois, J. M. (2014) · Journal of Microbiology and Biology Education · doi:10.1128/jmbe.v15i2.852 · Cited by 36 · see citing works
Pham, T. T., Lingard, H. C., & Zhang, R. P. (2023) · Safety Science · doi:10.1016/j.ssci.2023.106288 · Cited by 36 · see citing works
Evaluating A Continuing Medical Education Program: New World Kirkpatrick Model Approach
Liao, S.-C., & Hsu, S.-Y. (2019) · International Journal of Management Economics and Social Sciences · doi:10.32327/ijmess/8.4.2019.17 · Cited by 35 · see citing works
“First, the links between the levels are not strong and a causal relationship cannot be assumed (Alliger and Janak, 1989; Bates, 2004; Dixon, 1990).”
BackgroundMerriel, A., Ficquet, J., Barnard, K., Kunutsor, S. K., Soar, J., Lenguerrand, E., Caldwell, D. M., Burden, C., Winter, C., Draycott, T. J., et al. (2019) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd012177.pub2 · Cited by 34 · see citing works
Kim, D.-H., Lee, H. J., Lin, Y., & Kang, Y. J. (2021) · Journal of Educational Evaluation for Health Professions · doi:10.3352/jeehp.2021.18.24 · Cited by 34 · see citing works
“The fact that high satisfaction does not necessarily guarantee high academic achievement is another reason why it is difficult to make a convincing argument for academic achievement despite students’ positive responses to online courses [6].”
BackgroundHomklin, T., Takahashi, Y., & Techakanont, K. (2013) · International Business Research · doi:10.5539/ibr.v6n12p1 · Cited by 33 · see citing works
“This finding is consistent with previous research on supervisor and peer support, such as that by Bates et al. (2000), Cromwell and Kolb (2004), Facteau et al.”
ResultBijani, M., Rostami, K., Momennasab, M., & Yektatalab, S. (2017) · Journal of the National Medical Association · doi:10.1016/j.jnma.2017.11.002 · Cited by 33 · see citing works
An interprofessional framework for telebehavioral health competencies
Maheu, M. M., Drude, K. P., Hertlein, K. M., Lipschutz, R., Wall, K., & Hilty, D. M. (2017) · Journal of Technology in Behavioral Science · doi:10.1007/s41347-017-0038-y · Cited by 33 · see citing works
Alyami, H., Alawami, M., Lyndon, M., Alyami, M. M., Coomarasamy, C., Henning, M. A., Hill, A. G., & Sundram, F. (2019) · JMIR Serious Games · doi:10.2196/13748 · Cited by 32 · see citing works
“CLT design principles mostly target the extraneous cognitive load imposed by poorly designed material [38,39].”
BackgroundVongswasdi, P., Leroy, H., Claeys, J., Anisman‐Razin, M., & van Dierendonck, D. (2023) · Academy of Management Learning and Education · doi:10.5465/amle.2021.0231 · Cited by 31 · see citing works
Assessing the Performance of a Tourism MOOC Using the Kirkpatrick Model: A Supplier’s Point of View
Lin, J., & Cantoni, L. (2017) · Springer eBooks · doi:10.1007/978-3-319-51168-9_10 · Cited by 31 · see citing works
The performance evaluation of corporate universities
Cappiello, G., & Pedrini, G. (2017) · Tertiary Education and Management · doi:10.1080/13583883.2017.1329452 · Cited by 31 · see citing works
Revue des modèles en évaluation de formation : approches conceptuelles individuelles et sociales
Gilibert, D., & Gillet, I. (2010) · Pratiques Psychologiques · doi:10.1016/j.prps.2009.03.006 · Cited by 21 · see citing works
“Ce modèle a, par ailleurs, l’avantage de synthétiser le processus complexe de l’évaluation des formations et propose une démarche rationnelle et suffisamment large (mais synthétique) pour correspondre aux besoins des professionnels de la formation (Bates, 2004).”
BackgroundInfluentialA practitioner friendly and scientifically robust training evaluation approach
Griffin, R. P. (2012) · Journal of Workplace Learning · doi:10.1108/13665621211250298 · Cited by 30 · see citing works
Wu, S., Roychowdhury, I., & Khan, M. S. (2016) · International Journal of Infectious Diseases · doi:10.1016/j.ijid.2016.11.421 · Cited by 30 · see citing works
SimLife® technology in surgical training – a dynamic simulation model
Danion, J., Brèque, C., Oriot, D., Faure, J., & Richer, J.-P. (2020) · Journal of Visceral Surgery · doi:10.1016/j.jviscsurg.2020.02.013 · Cited by 30 · see citing works
Gajšek, B., Drăghici, A., Boatcă, M. E., Găureanu, A., & Robescu, D. F. (2022) · Sustainability · doi:10.3390/su14074301 · Cited by 30 · see citing works
Armstrong, L., Shepherd, A. J., & Harris, F. (2017) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2017.05.005 · Cited by 30 · see citing works
Bisgaard, C. H., Rubak, S., Rodt, S. A., Petersen, J. A. K., & Musaeus, P. (2018) · BMC Medical Education · doi:10.1186/s12909-018-1262-7 · Cited by 29 · see citing works
“Positive results at a lower level are necessary for causal inference of a superior level effect as the result of an education intervention [54].”
BackgroundMethodRoszkowski, M. J., & Soven, M. K. (2010) · Performance Improvement Quarterly · doi:10.1002/piq.20082 · Cited by 29 · see citing works
“Whatever the case, most evaluations of training do not advance beyond the first level of reactions (Bassi, Benson, & Cheney, 1996; Lee & Pershing, 1999; Phillips, 1991; Ruona, Leimbach, Holton, & Bates, 2002; Sugrue & Rivera, 2005; Twitchell, Holton, & Trott, 2001). Newby (1992) coined the derisive-sounding term react…”
BackgroundMethodAnalysis of executive coaching effectiveness: a study from the coachee perspective
Albizu, E., Rekalde, I., Landeta, J., & Ferrín, P. F. (2019) · Cuadernos de Gestión · doi:10.5295/cdg.170876ea · Cited by 27 · see citing works
Oteri, V., Occhipinti, F., Gribaudo, G., Marastoni, F., & Chisari, E. (2020) · Medical Science Educator · doi:10.1007/s40670-020-00921-4 · Cited by 28 · see citing works
“The impacts of the interventions were classified following Kirkpatrick’s adapted hierarchy [28], in line with guidance provided by BEME [29].”
MethodHodges, N. N., Watchravesringkan, K., Min, S., Lee, Y., & Seo, S. (2020) · International Journal of Fashion Design Technology and Education · doi:10.1080/17543266.2020.1742388 · Cited by 28 · see citing works
Training evaluation: a configurational analysis of success and failure of trainers and trainees
Bernardino, G., & Curado, C. (2020) · European journal of training and development · doi:10.1108/ejtd-10-2019-0177 · Cited by 29 · see citing works
Maynard, M. T., Marshall, D., & Dean, M. D. (2012) · Advances in health care management · doi:10.1108/s1474-8231(2012)0000013008 · Cited by 28 · see citing works
Measuring the impact of information skills training: a survey of health libraries in E ngland
Ayre, S. J., Barbrook, J., Engel, C., Lacey, P., Phul, A., Stevenson, P., & Toft, S. (2014) · Health Information & Libraries Journal · doi:10.1111/hir.12079 · Cited by 27 · see citing works
Effects of Using Immersive Media on the Effectiveness of Training to Prevent Ergonomics Risks
Diego-Más, J. A., Alcaide-Marzal, J., & Poveda-Bautista, R. (2020) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph17072592 · Cited by 30 · see citing works
“Although there are more recent models to assess training programs [41–43], Kirkpatrick’s model is the most commonly used [44], acting as the fundamental scheme for educational evaluation [45,46].”
MethodTraining evaluation: an analysis of the stakeholders' evaluation needs
Guerci, M., & Vinante, M. (2011) · Journal of European Industrial Training · doi:10.1108/03090591111128342 · Cited by 27 · see citing works
“This assumption has generated “the perception among training evaluators that establishing level four results will provide the most useful information about training program effectiveness” ( Bates, 2004, p. 342”
BackgroundCollusion with denial: leadership development and its evaluation
King, E., & Nesbit, P. (2015) · Journal of Management Development · doi:10.1108/jmd-02-2013-0023 · Cited by 27 · see citing works
Asthma education for school staff
Kew, K. M., Carr, R., Donovan, T., & Gordon, M. A. (2017) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd012255.pub2 · Cited by 27 · see citing works
Keogh, A., Matthews, J. J., Segurado, R., & Hurley, D. A. (2017) · Physical Therapy · doi:10.1093/ptj/pzx105 · Cited by 26 · see citing works
Nsour, M. A., Khader, Y. S., Bashier, H., & Alsoukhni, M. A. (2021) · Frontiers in Public Health · doi:10.3389/fpubh.2021.684174 · Cited by 26 · see citing works
“It mainly provides a straightforward systematic way for evaluating training outcomes and provides information that can be used to assess the extent to which training programs have achieved certain objectives (23).”
BackgroundIn‐company training in Catalonia: organizational structure, funding, evaluation and economic impact*
Huerta, M. E., Audet, X. L., & Peregort, O. P. (2006) · International Journal of Training and Development · doi:10.1111/j.1468-2419.2005.00249.x · Cited by 26 · see citing works
Evaluating student learning outcomes in oral health knowledge and skills
Lewis, A., Edwards, S., Whiting, G., & Donnelly, F. (2017) · Journal of Clinical Nursing · doi:10.1111/jocn.14082 · Cited by 26 · see citing works
“Level two evaluation involves measuring whether learning has taken place in terms of participants’ knowledge and/or skills (Barr et al., 1999; Bates, 2004; Beech & Leather, 2006; Curran & Fleet, 2005; Sargent et al., 2011; Smidt et al., 2009).”
Influential
- Authors
- Bates, R.
- Affiliations
- Louisiana State University (US)
- Corresponding author
- Bates, Reid
- Venue
- Evaluation and Program Planning
- Publisher
- Elsevier BV
- Published
- 2004-08
- Volume
- 27
- Issue
- 3
- Pages
- 341-347
- ISSN
- 0149-7189 (ISSN-L)
- OpenAlex
- W2097592370
- Cited by
- 623
- References
- 44
- Citations by year
- 2026: 53 · 2025: 23 · 2024: 42 · 2023: 53 · 2022: 49 · 2021: 39 · 2020: 43 · 2019: 44 · 2018: 32 · 2017: 45 · 2016: 32 · 2015: 32 · 2014: 30 · 2013: 29 · 2012: 22
- FWCI
- 4.09
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Evaluation and Program Planning (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Topics
- Social Sciences › Psychology › Applied Psychology › Human Resource Development and Performance Evaluation
- Social Sciences › Decision Sciences › Management Science and Operations Research › Evaluation and Performance Assessment
- Social Sciences › Education › Evaluation of Teaching Practices
- Keywords
- Kirkpatrick model, training evaluation, beneficence, moral obligation, formative assessment, summative assessment, training effectiveness, human resource development, evaluation limitations
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
