Programmatic assessment of competency-based workplace learning: when theory meets practice
Open accessAbstract
BACKGROUND: In competency-based medical education emphasis has shifted towards outcomes, capabilities, and learner-centeredness. Together with a focus on sustained evidence of professional competence this calls for new methods of teaching and assessment. Recently, medical educators advocated the use of a holistic, programmatic approach towards assessment. Besides maximum facilitation of learning it should improve the validity and reliability of measurements and documentation of competence development. We explored how, in a competency-based curriculum, current theories on programmatic assessment interacted with educational practice. METHODS: In a development study including evaluation, we investigated the implementation of a theory-based programme of assessment. Between April 2011 and May 2012 quantitative evaluation data were collected and used to guide group interviews that explored the experiences of students and clinical supervisors with the assessment programme. We coded the transcripts and emerging topics were organised into a list of lessons learned. RESULTS: The programme mainly focuses on the integration of learning and assessment by motivating and supporting students to seek and accumulate feedback. The assessment instruments were aligned to cover predefined competencies to enable aggregation of information in a structured and meaningful way. Assessments that were designed as formative learning experiences were increasingly perceived as summative by students. Peer feedback was experienced as a valuable method for formative feedback. Social interaction and external guidance seemed to be of crucial importance to scaffold self-directed learning. Aggregating data from individual assessments into a holistic portfolio judgement required expertise and extensive training and supervision of judges. CONCLUSIONS: A programme of assessment with low-stakes assessments providing simultaneously formative feedback and input for summative decisions proved not easy to implement. Careful preparation and guidance of the implementation process was crucial. Assessment for learning requires meaningful feedback with each assessment. Special attention should be paid to the quality of feedback at individual assessment moments. Comprehensive attention for faculty development and training for students is essential for the successful implementation of an assessment programme.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Programmatic assessment has met some difficulties in implementation.
“although it has also met some difficulties in implementation [69]”
From Guidelines: The do’s, don’ts and don’t knows of feedback for clinical education. (2015) · cites this paper as [69]
Durable teacher-learner relationships prompt learners to seek feedback more readily.
“As Bok et al. showed, durable teacher-learner relationships also prompt learners to seek feedback more readily [69].”
From Guidelines: The do’s, don’ts and don’t knows of feedback for clinical education. (2015) · cites this paper as [69]
Institutions can create opportunities for longitudinal teacher-learner relationships to flourish, such as extended placements.
“Institutions can create opportunities for longitudinal teacher-learner relationships to flourish, such as extended placements [68–70].”
From Guidelines: The do’s, don’ts and don’t knows of feedback for clinical education. (2015) · cites this paper as [69]
Evidence is emerging from qualitative evaluations of programmatic assessment curricula that raises questions about the mixing of formative and summative assessment.
“Evidence is now emerging from qualitative evaluations of programmatic assessment curricula which raises questions about the mixing of formative and summative assessment.”
From Guidelines: The do’s, don’ts and don’t knows of feedback for clinical education. (2015) · cites this paper as [69]
As a result, the formative assessments did not play the powerful assessment-for-learning role they are meant to have in a curriculum based on programmatic assessment.
“As a result, the formative assessments did not play the powerful assessment-for-learning role they are meant to have in a curriculum based on programmatic assessment [69].”
From Guidelines: The do’s, don’ts and don’t knows of feedback for clinical education. (2015) · cites this paper as [69]
In a qualitative study with clinical undergraduate veterinary students and their supervisors, both struggled with formative assessments that are used as 'data points' for a final summative judgment.
“A qualitative study with clinical undergraduate veterinary students and their supervisors highlighted that both struggled with formative assessments that are used as ‘data points’ for a final summative judgment.”
From Guidelines: The do’s, don’ts and don’t knows of feedback for clinical education. (2015) · cites this paper as [69]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Defining and Assessing Professional Competence
Epstein, R. M. (2002) · JAMA · doi:10.1001/jama.287.2.226 · Cited by 3,114 · see citing works
Competency-based medical education: Theory to practice.
Frank, J. R., Snell, L. S., Cate, O. T., Holmboe, E. S., Carraccio, C., Swing, S. R., Harris, P., Glasgow, N. J., Campbell, C., Dath, D., et al. (2010) · Medical Teacher · doi:10.3109/0142159x.2010.501190 · Cited by 2,883 · see citing works
Design Research: Theoretical and Methodological Issues
Collins, A., Joseph, D., & Bielaczyc, K. (2004) · Journal of the Learning Sciences · doi:10.1207/s15327809jls1301_2 · Cited by 1,946 · see citing works
Assessing professional competence: from methods to programmes
Van Der Vleuten, C. P. M., & Schuwirth, L. W. T. (2005) · Medical Education · doi:10.1111/j.1365-2929.2005.02094.x · Cited by 1,268 · see citing works
The assessment of professional competence: Developments, research and practical implications
Van Der Vleuten, C. P. M. (1996) · Advances in Health Sciences Education · doi:10.1007/bf00596229 · Cited by 1,247 · see citing works
Workplace-based assessment as an educational tool: AMEE Guide No. 31
Norcini, J., & Burch, V. (2007) · Medical Teacher · doi:10.1080/01421590701775453 · Cited by 984 · see citing works
Carraccio, C. L., Wolfsthal, S. D., Englander, R., Ferentz, K. S., & Martin, C. (2002) · Academic Medicine · doi:10.1097/00001888-200205000-00003 · Cited by 979 · see citing works
Programmatic assessment: From assessment of learning to assessment for learning
Schuwirth, L. W. T., & Van der Vleuten, C. P. M. (2011) · Medical Teacher · doi:10.3109/0142159x.2011.565828 · Cited by 902 · see citing works
A model for programmatic assessment fit for purpose.
van der Vleuten, C. P. M., Schuwirth, L. W. T., Driessen, E. W., Dijkstra, J., Tigelaar, D., Baartman, L. K. J., & van Tartwijk, J. (2012) · Medical Teacher · doi:10.3109/0142159x.2012.652239 · Cited by 892 · see citing works
Peer teaching in medical education: twelve reasons to move from theory to practice
ten Cate, O., & Durning, S. J. (2007) · Medical Teacher · doi:10.1080/01421590701606799 · Cited by 699 · see citing works
Tools for Direct Observation and Assessment of Clinical Skills of Medical Trainees
Kogan, J. R., Holmboe, E. S., & Hauer, K. E. (2009) · JAMA · doi:10.1001/jama.2009.1365 · Cited by 602 · see citing works
Qualitative methods in research on healthcare quality
Pope, C. J. (2002) · BMJ Quality & Safety · doi:10.1136/qhc.11.2.148 · Cited by 602 · see citing works
Portfolios in medical education: why do they meet with mixed success? A systematic review
Driessen, E. W., van Tartwijk, J., van der Vleuten, C. P. M., & Wass, V. (2007) · Medical Education · doi:10.1111/j.1365-2923.2007.02944.x · Cited by 337 · see citing works
From principles to practice: An embedded assessment system
Wilson, M., & Sloane, K. (2000) · Applied Measurement in Education · doi:10.1207/s15324818ame1302_4 · Cited by 333 · see citing works
Assessment in Health Professions Education
Downing, S. M., & Yudkowsky, R. (2009) · Routledge eBooks · doi:10.4324/9780203880135 · Cited by 328 · see citing works
Assessment in Health Professions Education
(2019) · Routledge eBooks · doi:10.4324/9781315166902 · Cited by 296 · see citing works
Peer assisted learning: a planning and implementation framework: AMEE Guide no. 30
Ross, M., & Cameron, H. S. (2007) · Medical Teacher · doi:10.1080/01421590701665886 · Cited by 294 · see citing works
Workplace-based assessment as an educational tool
Norcini, J. J., Burch, V. C., & Gibbs, T. J. (2007) · Medical Entomology and Zoology · Cited by 266 · see citing works
Driessen, E. W., van der Vleuten, C. P. M., Schuwirth, L., van Tartwijk, J., & Vermunt, J. D. H. M. (2005) · Medical Education · doi:10.1111/j.1365-2929.2004.02059.x · Cited by 207 · see citing works
Dannefer, E. F., & Henson, L. C. (2007) · Academic Medicine · doi:10.1097/acm.0b013e31803ead30 · Cited by 181 · see citing works
van Merriënboer, J. J. G., & Sluijsmans, D. M. A. (2009) · Educational Psychology Review · doi:10.1007/s10648-008-9092-5 · Cited by 156 · see citing works
Clarifying students’ feedback‐seeking behaviour in clinical clerkships
Bok, H. G. J., Teunissen, P. W., Spruijt, A., Fokkema, J. P. I., van Beukelen, P., Jaarsma, D. A. D. C., & van der Vleuten, C. P. M. (2013) · Medical Education · doi:10.1111/medu.12054 · Cited by 155 · see citing works
Self-citationA new framework for designing programmes of assessment
Dijkstra, J., van der Vleuten, C. P. M., & Schuwirth, L. (2009) · Advances in Health Sciences Education · doi:10.1007/s10459-009-9205-z · Cited by 133 · see citing works
A closer look at cueing effects in multiple‐choice questions
Schuwirth, L., van der Vleuten, C. P. M., & Donkers, J. (1996) · Medical Education · doi:10.1111/j.1365-2923.1996.tb00716.x · Cited by 132 · see citing works
The process of feedback in workplace‐based assessment: organisation, delivery, continuity
Pelgrim, E. A. M., Kramer, A. W. M., Mokkink, H. G. A., & van der Vleuten, C. P. M. (2012) · Medical Education · doi:10.1111/j.1365-2923.2012.04266.x · Cited by 128 · see citing works
Development and Validation of a Competency Framework for Veterinarians
Bok, H. G. J., Jaarsma, D. A. D. C., Teunissen, P. W., van der Vleuten, C. P. M., & van Beukelen, P. (2011) · Journal of Veterinary Medical Education · doi:10.3138/jvme.38.3.262 · Cited by 115 · see citing works
Self-citationLarge‐scale Portfolio Assessments in the US: evidence pertaining to the quality of measurement
Koretz, D. M. (1998) · Assessment in Education Principles Policy and Practice · doi:10.1080/0969595980050302 · Cited by 95 · see citing works
When enough is enough: a conceptual basis for fair and defensible practice performance assessment
Schuwirth, L., Southgate, L. J., Page, G. G., Paget, N. S., Lescop, J., Lew, S. R., Wade, W. B., & Maldonado, M. B. (2002) · Medical Education · doi:10.1046/j.1365-2923.2002.01313.x · Cited by 95 · see citing works
Prescott, L. E., Norcini, J. J., McKinlay, P., & Rennie, J. S. (2002) · Medical Education · doi:10.1046/j.1365-2923.2002.01099.x · Cited by 74 · see citing works
Curriculum development in medical education: From acronyms to dynamism
Prideaux, D. J. (2007) · Teaching and Teacher Education · doi:10.1016/j.tate.2006.12.017 · Cited by 44 · see citing works
Ringsted, C., Henriksen, A. H., Skaarup, A. M., & Van der Vleuten, C. P. M. (2004) · Medical Education · doi:10.1111/j.1365-2929.2004.01841.x · Cited by 41 · see citing works
Martinez, M. E., & Lipson, J. I. (1989) · Educational leadership · Cited by 32 · see citing works
Ringsted, C. V., Østergaard, D., & Scherpbier, A. J. J. A. (2003) · Medical Teacher · doi:10.1080/0142159021000061431 · Cited by 28 · see citing works
Implementing feedback cards in core clerkships
Kogan, J. R., & Shea, J. A. (2008) · Medical Education · doi:10.1111/j.1365-2923.2008.03158.x · Cited by 26 · see citing works
Ringsted, C. V., Skaarup, A. M., Henriksen, A., & Davis, D. (2006) · Medical Teacher · doi:10.1080/01421590500237721 · Cited by 21 · see citing works
Flagged first, then by citations
1 of the 36 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Assessment, feedback and the alchemy of learning
Watling, C., & Ginsburg, S. (2018) · Medical Education · doi:10.1111/medu.13645 · Cited by 434 · see citing works
Guidelines: The do’s, don’ts and don’t knows of feedback for clinical education.
Lefroy, J., Watling, C., Teunissen, P. W., & Brand, P. (2015) · Perspectives on Medical Education · doi:10.1007/s40037-015-0231-7 · Cited by 347 · see citing works
Self-citationTwelve Tips for programmatic assessment
Van Der Vleuten, C. P. M., Schuwirth, L. W. T., Driessen, E. W., Govaerts, M. J. B., & Heeneman, S. (2015) · Medical Teacher · doi:10.3109/0142159x.2014.973388 · Cited by 333 · see citing works
Rekman, J. F., Gofton, W. T., Dudek, N. L., Gofton, T., & Hamstra, S. J. (2015) · Academic Medicine · doi:10.1097/acm.0000000000001045 · Cited by 238 · see citing works
Educational design research: Portraying, conducting, and enhancing productive scholarship
McKenney, S. E., & Reeves, T. C. (2020) · Medical Education · doi:10.1111/medu.14280 · Cited by 328 · see citing works
The impact of programmatic assessment on student learning: theory versus practice
Heeneman, S., Pool, A. O., Schuwirth, L., van der Vleuten, C. P. M., & Driessen, E. W. (2015) · Medical Education · doi:10.1111/medu.12645 · Cited by 213 · see citing works
Schellekens, L. H., Bok, H. G. J., de Jong, L. H., van der Schaaf, M. F., Kremer, W. D. J., & van der Vleuten, C. P. M. (2021) · Studies In Educational Evaluation · doi:10.1016/j.stueduc.2021.101094 · Cited by 229 · see citing works
Self-citationKogan, J. R., Hatala, R. A., Hauer, K. E., & Holmboe, E. S. (2017) · Perspectives on Medical Education · doi:10.1007/s40037-017-0376-7 · Cited by 161 · see citing works
Massie, J., & Ali, J. M. (2016) · Advances in Health Sciences Education · doi:10.1007/s10459-015-9614-0 · Cited by 147 · see citing works
Guidelines: The dos, don’ts and don’t knows of remediation in medical education
Chou, C. L., Kalet, A. L., Costa, M. J., Cleland, J., & Winston, K. (2019) · Perspectives on Medical Education · doi:10.1007/s40037-019-00544-5 · Cited by 144 · see citing works
Schut, S., Driessen, E. W., van Tartwijk, J., van der Vleuten, C. P. M., & Heeneman, S. (2018) · Medical Education · doi:10.1111/medu.13532 · Cited by 141 · see citing works
Ottawa 2020 consensus statement for programmatic assessment – 1. Agreement on the principles
Heeneman, S., de Jong, L. H., Dawson, L. J., WILKINSON, T. J., Ryan, A., Tait, G. R., Rice, N. E., Torre, D. M., Freeman, A., & van der Vleuten, C. P. M. (2021) · Medical Teacher · doi:10.1080/0142159x.2021.1957088 · Cited by 150 · see citing works
Schut, S., Maggio, L. A., Heeneman, S., van Tartwijk, J., van der Vleuten, C. P. M., & Driessen, E. W. (2020) · Perspectives on Medical Education · doi:10.1007/s40037-020-00625-w · Cited by 132 · see citing works
Nousiainen, M. T., Caverzagie, K. J., Ferguson, P. C., Frank, J. R., & Collaborators, O. B. O. T. I. (2017) · Medical Teacher · doi:10.1080/0142159x.2017.1315077 · Cited by 128 · see citing works
Learning Analytics in Medical Education Assessment: The Past, the Present, and the Future
Chan, T. M., Sebok‐Syer, S. S., Thoma, B., Wise, A. F., Sherbino, J., & Pusic, M. V. (2018) · AEM Education and Training · doi:10.1002/aet2.10087 · Cited by 126 · see citing works
Harrison, C., Könings, K. D., Dannefer, E. F., Schuwirth, L., Wass, V., & van der Vleuten, C. P. M. (2016) · Perspectives on Medical Education · doi:10.1007/s40037-016-0297-x · Cited by 124 · see citing works
Beyond “formative”: assessments to enrich student learning
Kulasegaram, K. M., & Rangachari, P. K. (2018) · AJP Advances in Physiology Education · doi:10.1152/advan.00122.2017 · Cited by 143 · see citing works
Branfield Day, L., Miles, A., Ginsburg, S., & Melvin, L. (2020) · Academic Medicine · doi:10.1097/acm.0000000000003315 · Cited by 120 · see citing works
Changing the culture of assessment: the dominance of the summative assessment paradigm
Harrison, C., Könings, K. D., Schuwirth, L., Wass, V., & van der Vleuten, C. P. M. (2017) · BMC Medical Education · doi:10.1186/s12909-017-0912-5 · Cited by 104 · see citing works
Assessment in the context of problem-based learning
van der Vleuten, C. P. M., & Schuwirth, L. (2019) · Advances in Health Sciences Education · doi:10.1007/s10459-019-09909-1 · Cited by 109 · see citing works
Managing tensions in assessment: moving beyond either–or thinking
Govaerts, M., van der Vleuten, C. P. M., & Holmboe, E. S. (2018) · Medical Education · doi:10.1111/medu.13656 · Cited by 88 · see citing works
The impact of entrustment assessments on feedback and learning: Trainee perspectives
Martin, L. J., Sibbald, M., Vegas, D. B., Russell, D., & Govaerts, M. (2019) · Medical Education · doi:10.1111/medu.14047 · Cited by 87 · see citing works
Rekman, J. F., Hamstra, S. J., Dudek, N. L., Wood, T. J., Seabrook, C., & Gofton, W. T. (2016) · Journal of surgical education · doi:10.1016/j.jsurg.2016.02.003 · Cited by 85 · see citing works
Theoretical considerations on programmatic assessment
Torre, D. M., Schuwirth, L., & van der Vleuten, C. P. M. (2019) · Medical Teacher · doi:10.1080/0142159x.2019.1672863 · Cited by 84 · see citing works
Duijn, C. C. M. A., Welink, L. S., Mándoki, M., Kremer, W. D. J., ten Cate, O., & Bok, H. G. J. (2017) · Perspectives on Medical Education · doi:10.1007/s40037-017-0361-1 · Cited by 86 · see citing works
Self-citation‘Playing the game’: How do surgical trainees seek feedback using workplace‐based assessment?
Gaunt, A., Patel, A., Rusius, V., Royle, T. J., Markham, D. H., & Pawlikowska, T. R. B. (2017) · Medical Education · doi:10.1111/medu.13380 · Cited by 73 · see citing works
ten Cate, O., Balmer, D. F., Caretta‐Weyer, H. A., Hatala, R. A., Hennus, M. P., & West, D. C. (2021) · Academic Medicine · doi:10.1097/acm.0000000000004106 · Cited by 71 · see citing works
Anderson, H. L. K., Kurtz, J. B., & West, D. C. (2021) · Academic Medicine · doi:10.1097/acm.0000000000004366 · Cited by 69 · see citing works
Broadening the Scope of Feedback to Promote Its Relevance to Workplace Learning
van der Leeuw, R. M., Teunissen, P. W., & van der Vleuten, C. P. M. (2017) · Academic Medicine · doi:10.1097/acm.0000000000001962 · Cited by 69 · see citing works
Self-citationRich, J. V., Young, S. F., Donnelly, C. A., Hall, A. K., Dagnone, D., Weersink, K., Caudle, J. M., Melle, E. V., & Klinger, D. A. (2019) · Journal of Evaluation in Clinical Practice · doi:10.1111/jep.13328 · Cited by 67 · see citing works
Duijn, C. C. M. A., Welink, L. S., Bok, H. G. J., & ten Cate, O. (2018) · Perspectives on Medical Education · doi:10.1007/s40037-018-0430-0 · Cited by 65 · see citing works
Self-citationGinsburg, S., Watling, C., Schumacher, D. J., Gingerich, A., & Hatala, R. A. (2021) · Academic Medicine · doi:10.1097/acm.0000000000004089 · Cited by 64 · see citing works
Validity evidence for programmatic assessment in competency-based education
Bok, H. G. J., de Jong, L. H., O’Neill, T., Maxey, C., & Hecker, K. G. (2018) · Perspectives on Medical Education · doi:10.1007/s40037-018-0481-2 · Cited by 63 · see citing works
Self-citationBetween trust and control: Teachers' assessment conceptualisations within programmatic assessment
Schut, S., Heeneman, S., Bierer, S. B., Driessen, E. W., van Tartwijk, J., & van der Vleuten, C. P. M. (2020) · Medical Education · doi:10.1111/medu.14075 · Cited by 57 · see citing works
van der Vleuten, C. P. M., Lindemann, I., & Schmidt, L. (2018) · The Medical Journal of Australia · doi:10.5694/mja17.00926 · Cited by 56 · see citing works
Trust, power and learning in workplace‐based assessment: The trainee perspective
Castanelli, D. J., Weller, J. M., Molloy, E., & Bearman, M. (2021) · Medical Education · doi:10.1111/medu.14631 · Cited by 56 · see citing works
Ottawa 2020 consensus statements for programmatic assessment – 2. Implementation and practice
Torre, D. M., Rice, N. E., Ryan, A., Bok, H. G. J., Dawson, L. J., Bierer, S. B., WILKINSON, T. J., Tait, G. R., Laughlin, T., Veerapen, K., et al. (2021) · Medical Teacher · doi:10.1080/0142159x.2021.1956681 · Cited by 55 · see citing works
Self-citationFeedback-giving behaviour in performance evaluations during clinical clerkships
Bok, H. G. J., Jaarsma, D. A. D. C., Spruijt, A., van Beukelen, P., van der Vleuten, C. P. M., & Teunissen, P. W. (2015) · Medical Teacher · doi:10.3109/0142159x.2015.1017448 · Cited by 54 · see citing works
Self-citationDeconstructing programmatic assessment
WILKINSON, T. J., & Tweed, M. (2018) · Advances in Medical Education and Practice · doi:10.2147/amep.s144449 · Cited by 53 · see citing works
Competence Assessment as Learner Support in Education
van der Vleuten, C. P. M., Sluijsmans, D. M. A., & Brinke, D. J.-T. (2016) · Technical and vocational education and training · doi:10.1007/978-3-319-41713-4_28 · Cited by 52 · see citing works
Improving Learner Handovers in Medical Education
Warm, E. J., Englander, R., Pereira, A. G., & Barach, P. R. (2016) · Academic Medicine · doi:10.1097/acm.0000000000001457 · Cited by 52 · see citing works
Some assembly required: tracing the interpretative work of Clinical Competency Committees
Pack, R., Lingard, L. A., Watling, C., Chahine, S., & Cristancho, S. (2019) · Medical Education · doi:10.1111/medu.13884 · Cited by 52 · see citing works
Barrett, A., Galvin, R., Steinert, Y., Scherpbier, A. J. J. A., O’Shaughnessy, A., Horgan, M. M., & Horsley, T. (2016) · Medical Teacher · doi:10.1080/0142159x.2016.1215413 · Cited by 50 · see citing works
Castanelli, D. J., Jowsey, T., Chen, Y., & Weller, J. M. (2016) · Canadian Journal of Anesthesia/Journal canadien d anesthésie · doi:10.1007/s12630-016-0740-9 · Cited by 50 · see citing works
Programmatic Assessment of Level 1 Milestones in Incoming Interns
Hauff, S. R., Hopson, L. R., Losman, E. D., Perry, M. A., Lypson, M. L., Fischer, J. P., & Santen, S. A. (2014) · Academic Emergency Medicine · doi:10.1111/acem.12393 · Cited by 49 · see citing works
de Jonge, L. P. J. W. M., Timmerman, A. A., Govaerts, M., Muris, J. W. M., Muijtjens, A. M. M., Kramer, A. W. M., & van der Vleuten, C. P. M. (2017) · Advances in Health Sciences Education · doi:10.1007/s10459-017-9760-7 · Cited by 49 · see citing works
Workplace-Based Assessment and Assessment for Learning: Threats to Validity
Govaerts, M. (2015) · Journal of Graduate Medical Education · doi:10.4300/jgme-d-15-00101.1 · Cited by 48 · see citing works
Li, S., Sherbino, J., & Chan, T. M. (2016) · AEM Education and Training · doi:10.1002/aet2.10009 · Cited by 48 · see citing works
Ross, S., Hauer, K. E., Wycliffe-Jones, K., Hall, A. K., Molgaard, L. K., Richardson, D. L., Oswald, A., Bhanji, F., & Collaborators, O. B. O. T. I. (2021) · Medical Teacher · doi:10.1080/0142159x.2021.1925099 · Cited by 49 · see citing works
Torre, D. M., Schuwirth, L., van der Vleuten, C. P. M., & Heeneman, S. (2022) · Medical Teacher · doi:10.1080/0142159x.2022.2083487 · Cited by 44 · see citing works
Danilovich, N., Kitto, S. C., Price, D. W., Campbell, C. B., Hodgson, A., & HENDRY, P. J. (2021) · Family Medicine · doi:10.22454/fammed.2021.453158 · Cited by 46 · see citing works
Understanding the influence of teacher–learner relationships on learners’ assessment perception
Schut, S., van Tartwijk, J., Driessen, E. W., van der Vleuten, C. P. M., & Heeneman, S. (2019) · Advances in Health Sciences Education · doi:10.1007/s10459-019-09935-z · Cited by 40 · see citing works
The power of subjectivity in competency-based assessment
Virk, A., Joshi, A., Mahajan, R., & Singh, T. (2020) · Journal of Postgraduate Medicine · doi:10.4103/jpgm.jpgm_591_20 · Cited by 40 · see citing works
Díaz-Pérez, J. A., Raju, S. C., & Echeverri, J. H. (2014) · Journal of Pathology Informatics · doi:10.4103/2153-3539.137729 · Cited by 40 · see citing works
How Trainees Come to Trust Supervisors in Workplace-Based Assessment: A Grounded Theory Study
Castanelli, D. J., Weller, J. M., Molloy, E., & Bearman, M. (2021) · Academic Medicine · doi:10.1097/acm.0000000000004501 · Cited by 39 · see citing works
Mastenbroek, N. J. J. M. (2017) · Journal of Veterinary Medical Education · doi:10.3138/jvme.0216-041r1 · Cited by 41 · see citing works
Lee, V., Brain, K., & Martin, J. A. (2017) · Academic Medicine · doi:10.1097/acm.0000000000001537 · Cited by 39 · see citing works
Pugh, D., Desjardins, I., & Eva, K. W. (2017) · Medical Teacher · doi:10.1080/0142159x.2017.1388502 · Cited by 38 · see citing works
Duijn, C. C. M. A., ten Cate, O., Kremer, W. D. J., & Bok, H. G. J. (2018) · Journal of Veterinary Medical Education · doi:10.3138/jvme.0617-073r · Cited by 37 · see citing works
Self-citationWork-based Assessment and Co-production in Postgraduate Medical Training
Holmboe, E. S. (2017) · Europe PMC (PubMed Central) · doi:10.3205/zma001135 · Cited by 37 · see citing works
Abraham, R. M., & Singaram, V. S. (2019) · BMC Medical Education · doi:10.1186/s12909-019-1547-5 · Cited by 40 · see citing works
Shadow systems in assessment: how supervisors make progress decisions in practice
Castanelli, D. J., Weller, J. M., Molloy, E., & Bearman, M. (2019) · Advances in Health Sciences Education · doi:10.1007/s10459-019-09913-5 · Cited by 36 · see citing works
Students’ perceptions of general medicine following community‐based medical education in rural Japan
Ohta, R., Ryu, Y., Katsube, T., Moriwaki, Y., & Otani, J. (2019) · Journal of General and Family Medicine · doi:10.1002/jgf2.274 · Cited by 35 · see citing works
Students’ motivation toward feedback-seeking in the clinical workplace
de Jong, L. H., Favier, R. P., van der Vleuten, C. P. M., & Bok, H. G. J. (2017) · Medical Teacher · doi:10.1080/0142159x.2017.1324948 · Cited by 35 · see citing works
Self-citationDriving lesson or driving test?: A metaphor to help faculty separate feedback from assessment
Brand, P. L. P., Jaarsma, D. A. D. C., & van der Vleuten, C. P. M. (2020) · Perspectives on Medical Education · doi:10.1007/s40037-020-00617-w · Cited by 36 · see citing works
Self-citationEstruturação de Sistemas para Avaliação Programática do Estudante de Medicina
Troncon, L. E. D. A. (2016) · Revista Brasileira de Educação Médica · doi:10.1590/1981-52712015v40n1e01392015 · Cited by 33 · see citing works
Ross, S., Binczyk, N. M., Hamza, D. M., Schipper, S., Humphries, P., Nichols, D. N., & Donoff, M. G. (2018) · JAMA Network Open · doi:10.1001/jamanetworkopen.2018.4581 · Cited by 33 · see citing works
Pool, A. O., Jaarsma, D. A. D. C., Driessen, E. W., & Govaerts, M. (2020) · Perspectives on Medical Education · doi:10.1007/s40037-020-00571-7 · Cited by 42 · see citing works
Self-citationDesigning programmes of assessment: A participatory approach
Jamieson, J., Jenkins, G., BEATTY, S. E., & Palermo, C. E. (2017) · Medical Teacher · doi:10.1080/0142159x.2017.1355447 · Cited by 32 · see citing works
Janssens, O., Embo, M., Valcke, M., & Haerens, L. L. (2023) · BMC Medical Education · doi:10.1186/s12909-023-04446-3 · Cited by 44 · see citing works
McDonald, J. A., Lai, C. J., Lin, M. Y. C., O’Sullivan, P., & Hauer, K. E. (2020) · Academic Medicine · doi:10.1097/acm.0000000000003730 · Cited by 31 · see citing works
Programmatic Assessment in Emergency Medicine: Implementation of Best Practices
Perry, M. A., Linn, A. M. J., Munzer, B., Hopson, L. R., Amlong, A., Cole, M., & Santen, S. A. (2017) · Journal of Graduate Medical Education · doi:10.4300/jgme-d-17-00094.1 · Cited by 31 · see citing works
Barrett, A., Galvin, R., Scherpbier, A. J. J. A., Teunissen, P. W., O’Shaughnessy, A., & Horgan, M. M. (2016) · Postgraduate Medical Journal · doi:10.1136/postgradmedj-2015-133917 · Cited by 31 · see citing works
Self-citationQuality of WIL assessment design in higher education: a systematic literature review
Lasen, M., Evans, N. S., Tsey, K., Campbell, C. M., & Kinchin, I. (2018) · Higher Education Research & Development · doi:10.1080/07294360.2018.1450359 · Cited by 30 · see citing works
Faculty and Resident Perspectives on Using Entrustment Anchors for Workplace-Based Assessment
Dudek, N. L., Gofton, W. T., Rekman, J. F., & McDougall, A. (2019) · Journal of Graduate Medical Education · doi:10.4300/jgme-d-18-01003.1 · Cited by 30 · see citing works
Young, J. Q., Sugarman, R., Schwartz, J., & O’Sullivan, P. (2020) · Teaching and Learning in Medicine · doi:10.1080/10401334.2020.1767107 · Cited by 30 · see citing works
A philosophical history of programmatic assessment: tracing shifting configurations
Pearce, J., & Tavares, W. (2021) · Advances in Health Sciences Education · doi:10.1007/s10459-021-10050-1 · Cited by 32 · see citing works
Medical education research: a vibrant community of research and education practice
van der Vleuten, C. P. M. (2014) · Medical Education · doi:10.1111/medu.12508 · Cited by 29 · see citing works
The challenge of changing to an assessment for learning culture
Harrison, C., & Wass, V. (2016) · Medical Education · doi:10.1111/medu.13058 · Cited by 30 · see citing works
De ning Uniform Processes for Remediation, Probation and Termination in Residency Training
Smith, J., Lypson, M. L., Silverberg, M., Weizberg, M., Murano, T., Lukela, M. P., & Santen, S. A. (2017) · Western Journal of Emergency Medicine · doi:10.5811/westjem.2016.10.31483 · Cited by 30 · see citing works
Bok, H. G. J. (2015) · Perspectives on Medical Education · doi:10.1007/s40037-015-0172-1 · Cited by 29 · see citing works
Self-citationMcIntyre‐Hite, L. (2016) · The Journal of Competency-Based Education · doi:10.1002/cbe2.1029 · Cited by 29 · see citing works
Entrustment Unpacked: Aligning Purposes, Stakes, and Processes to Enhance Learner Assessment
Kinnear, B., Warm, E. J., Caretta‐Weyer, H. A., Holmboe, E. S., Turner, D. A., van der Vleuten, C. P. M., & Schumacher, D. J. (2021) · Academic Medicine · doi:10.1097/acm.0000000000004108 · Cited by 29 · see citing works
A practical approach to programmatic assessment design
Timmerman, A. A., & Dijkstra, J. (2017) · Advances in Health Sciences Education · doi:10.1007/s10459-017-9756-3 · Cited by 28 · see citing works
The Clinical Learning Environment and Workplace-Based Assessment
Henry, D., & West, D. C. (2019) · Pediatric Clinics of North America · doi:10.1016/j.pcl.2019.03.010 · Cited by 28 · see citing works
How workplace‐based assessments guide learning in postgraduate education: A scoping review
Martin, L. J., Blissett, S., Johnston, B. K., Tsang, M., Gauthier, S. R., Ahmed, Z., & Sibbald, M. (2022) · Medical Education · doi:10.1111/medu.14960 · Cited by 29 · see citing works
A mobile application to facilitate implementation of programmatic assessment in anaesthesia training
Marty, A., Braun, J., Schick, C., Zalunardo, M. P., Spahn, D. R., & Breckwoldt, J. (2022) · British Journal of Anaesthesia · doi:10.1016/j.bja.2022.02.038 · Cited by 27 · see citing works
de Jong, L. H., Bok, H. G. J., Schellekens, L. H., Kremer, W. D. J., Jonker, F. H., & van der Vleuten, C. P. M. (2022) · BMC Medical Education · doi:10.1186/s12909-022-03257-2 · Cited by 24 · see citing works
Self-citationTen caveats of learning analytics in health professions education: A consumer’s perspective
ten Cate, O., Dahdal, S., Lambert, T., Neubauer, F. B., Pless, A., Pohlmann, P.-F., van Rijen, H. V. M., & Gurtner, C. (2020) · Medical Teacher · doi:10.1080/0142159x.2020.1733505 · Cited by 24 · see citing works
How Trainees Come to Trust Supervisors in Workplace-Based Assessment
Castanelli, D. J., Weller, J. M., Molloy, E., & Bearman, M. (2021) · Own your potential (DEAKIN) · Cited by 24 · see citing works
van der Gulden, R., Timmerman, A. A., Muris, J. W. M., Thoonen, B. P. A., Heeneman, S., & Haan, N. S. (2022) · Perspectives on Medical Education · doi:10.1007/s40037-022-00727-7 · Cited by 24 · see citing works
Chakroun, M., Dion, V., Ouellet, K., Graillon, A., Désilets, V. A., Xhignesse, M., & St‐Onge, C. (2022) · Academic Medicine · doi:10.1097/acm.0000000000004755 · Cited by 24 · see citing works
Bullock, J. L., Seligman, L., Lai, C. J., O’Sullivan, P., & Hauer, K. E. (2021) · Teaching and Learning in Medicine · doi:10.1080/10401334.2021.1922285 · Cited by 23 · see citing works
Exploring students’ feedback seeking behavior in the context of programmatic assessment
Baartman, L. K. J., Baukema, H., & Prins, F. J. (2022) · Assessment & Evaluation in Higher Education · doi:10.1080/02602938.2022.2100875 · Cited by 22 · see citing works
Programmatic assessment design choices in nine programs in higher education
Baartman, L. K. J., van Schilt-Mol, T., & van der Vleuten, C. P. M. (2022) · Frontiers in Education · doi:10.3389/feduc.2022.931980 · Cited by 27 · see citing works
The reliability of a portfolio of workplace-based assessments in anesthesia training
Castanelli, D. J., van Loon, J., Jolly, B. C., & Weller, J. M. (2018) · Canadian Journal of Anesthesia/Journal canadien d anesthésie · doi:10.1007/s12630-018-1251-7 · Cited by 22 · see citing works
Ryan, M. S., Richards, A., Perera, R. A., Park, Y. S., Stringer, J. K., Waterhouse, E. J., Dubinsky, B., Khamishon, R., & Santen, S. A. (2021) · Academic Medicine · doi:10.1097/acm.0000000000003921 · Cited by 22 · see citing works
King, E., Turpin, M. J., Green, W., & Schull, D. N. (2019) · Advances in Health Sciences Education · doi:10.1007/s10459-019-09891-8 · Cited by 21 · see citing works
Ryan, M. S., Khan, A. R., Park, Y. S., Chastain, C. A., Phillipi, C. A., Santen, S. A., Barron, B., Obeso, V. T., Yingling, S. L., & Program, F. T. C. E. P. A. F. E. R. P. (2021) · Academic Medicine · doi:10.1097/acm.0000000000004222 · Cited by 21 · see citing works
Programmatic assessment: Can we provide evidence for saturation of information?
de Jong, L. H., Bok, H. G. J., Kremer, W. D. J., & van der Vleuten, C. P. M. (2019) · Medical Teacher · doi:10.1080/0142159x.2018.1555369 · Cited by 21 · see citing works
Self-citation
- Authors
- Bok, H. G., Teunissen, P. W., Favier, R. P., Rietbroek, N. J., Theyse, L. F., Brommer, H., Haarhuis, J. C., van Beukelen, P., van der Vleuten, C. P., & Jaarsma, D. A.
- Affiliations
- Utrecht University (NL)
- Maastricht University (NL)
- University of Amsterdam (NL)
- Corresponding author
- Bok, Harold GJ
- Venue
- BMC Medical Education
- Published
- 2013-09-11
- Volume
- 13
- Issue
- 1
- Pages
- 123-123
- ISSN
- 1472-6920 (ISSN-L)
- PMC
- PMC3851012
- OpenAlex
- W2153422278
- Cited by
- 303
- References
- 36
- Citations by year
- 2026: 20 · 2025: 14 · 2024: 24 · 2023: 31 · 2022: 36 · 2021: 24 · 2020: 29 · 2019: 29 · 2018: 25 · 2017: 26 · 2016: 25 · 2015: 13 · 2014: 5
- FWCI
- 14.62
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- BMC Medical Education (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Pure Amsterdam UMC (publishedVersion)
- Amsterdam UMC (VU Amsterdam) - Institutional Repository (publishedVersion)
- Data Archiving and Networked Services (DANS) (submittedVersion)
- Research Publications (Maastricht University) (submittedVersion, cc-by)
- Radboud Repository (Radboud University) (submittedVersion)
- Radboud Repository (Radboud University) (submittedVersion, other-oa)
- Research Publications (Maastricht University) (submittedVersion, other-oa)
- Funders
- Flinders University
- Topics
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Innovations in Medical Education
- Social Sciences › Education › Student Assessment and Feedback
- Social Sciences › Psychology › Social Psychology › Competency Development and Evaluation
- Keywords
- competency-based medical education, programmatic assessment, workplace-based assessment, formative assessment, summative assessment, feedback, peer feedback, self-directed learning, portfolio assessment, low-stakes assessment, assessment implementation, faculty development, clinical supervisors, medical students
- MeSH terms
- Competency-Based Education, Curriculum, Education, Medical, Education, Veterinary, Educational Measurement, Humans, Netherlands, Surveys and Questionnaires, Schools, Veterinary
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
