Twelve Tips for programmatic assessment

Van Der Vleuten, C.P.M.; Schuwirth, L.W.T.; Driessen, E.W.; Govaerts, M.J.B.; Heeneman, S.

doi:10.3109/0142159x.2014.973388

Cited by 333Innovations in Medical EducationStudent Assessment and FeedbackEducational Assessment and Pedagogy

Twelve Tips for programmatic assessment

Van Der Vleuten, C.P.M., Schuwirth, L.W.T., Driessen, E.W., Govaerts, M.J.B., Heeneman, S.·2015·Medical Teacher

DOI 10.3109/0142159x.2014.973388

Open this paper in CoChat — free account

Abstract

Programmatic assessment is an integral approach to the design of an assessment program with the intent to optimise its learning function, its decision-making function and its curriculum quality-assurance function. Individual methods of assessment, purposefully chosen for their alignment with the curriculum outcomes and their information value for the learner, the teacher and the organisation, are seen as individual data points. The information value of these individual data points is maximised by giving feedback to the learner. There is a decoupling of assessment moment and decision moment. Intermediate and high-stakes decisions are based on multiple data points after a meaningful aggregation of information and supported by rigorous organisational procedures to ensure their dependability. Self-regulation of learning, through analysis of the assessment information and the attainment of the ensuing learning goals, is scaffolded by a mentoring system. Programmatic assessment-for-learning can be applied to any part of the training continuum, provided that the underlying learning conception is constructivist. This paper provides concrete recommendations for implementation of programmatic assessment.

Integrity signals

Registry

Crossref · 10.3109/0142159x.2014.973388

Cites

33 of 33 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Programmatic assessment shifts the focus from isolated assessment events to the strategic collection and integration of multiple low-stakes data points over time.
“Gaining prominence in medical education, programmatic assessment shifts the focus from isolated assessment events to the strategic collection and integration of multiple low-stakes data points over time [16,27].”
From Formative assessment and feedback in medical education: A practical guide: AMEE Guide No. 189 (2026) · cites this paper as [27]
In programmatic assessment, each piece of evidence, whether formative or summative, is deliberately gathered to support learner development and guide high-stakes judgments about competence.
“Each piece of evidence, whether formative or summative, is deliberately gathered to support learner development and guide high-stakes judgments about competence.”
From Formative assessment and feedback in medical education: A practical guide: AMEE Guide No. 189 (2026) · cites this paper as [27]
CBME and programmatic assessment promote the longitudinal compilation of performance data to coach the learner's continuing development and to make evidence-based decisions about the learner's progress.
“Finally, they promote the longitudinal compilation of performance data to coach the learner's continuing development (assessment for learning) and to make evidence-based decisions about the learner's progress (assessment of learning).[11][12][13][14][15]”
From The R2C2 Model in Residency Education: How Does It Foster Coaching and Promote Feedback Use? (2018) · cites this paper as [13]
The principles of programmatic assessment and CBME direct attention to actively engaging learners in their assessment, feedback, and development activities.
“Foundationally, the principles of programmatic assessment and CBME direct our attention to actively engaging learners in their assessment, feedback, and development activities.9,13,15”
From The R2C2 Model in Residency Education: How Does It Foster Coaching and Promote Feedback Use? (2018) · cites this paper as [13]

The Power of Feedback
Hattie, J., & Timperley, H. (2007) · Review of Educational Research · doi:10.3102/003465430298487 · Cited by 12,905 · see citing works
Focus on Formative Feedback
Shute, V. J. (2008) · Review of Educational Research · doi:10.3102/0034654307313795 · Cited by 4,631 · cited 1× · see citing works
“Mostly, conventional feedback from assessments, that is, grades and pass/fail decisions, are poor information carriers (Shute 2008).”
Background
Deliberate Practice and the Acquisition and Maintenance of Expert Performance in Medicine and Related Domains
Ericsson, K. A. (2004) · Academic Medicine · doi:10.1097/00001888-200410001-00022 · Cited by 2,988 · see citing works
Assessing professional competence: from methods to programmes
Van Der Vleuten, C. P. M., & Schuwirth, L. W. T. (2005) · Medical Education · doi:10.1111/j.1365-2929.2005.02094.x · Cited by 1,268 · see citing works
Self-citation
The assessment of professional competence: Developments, research and practical implications
Van Der Vleuten, C. P. M. (1996) · Advances in Health Sciences Education · doi:10.1007/bf00596229 · Cited by 1,247 · see citing works
Problem‐based learning: future challenges for educational practice and research
Dolmans, D. H. J. M., de Grave, W. S., Wolfhagen, I. H. A. P., & van der Vleuten, C. P. M. (2005) · Medical Education · doi:10.1111/j.1365-2929.2005.02205.x · Cited by 1,103 · see citing works
Programmatic assessment: From assessment of learning to assessment for learning
Schuwirth, L. W. T., & Van der Vleuten, C. P. M. (2011) · Medical Teacher · doi:10.3109/0142159x.2011.565828 · Cited by 902 · see citing works
Self-citation
A model for programmatic assessment fit for purpose.
van der Vleuten, C. P. M., Schuwirth, L. W. T., Driessen, E. W., Dijkstra, J., Tigelaar, D., Baartman, L. K. J., & van Tartwijk, J. (2012) · Medical Teacher · doi:10.3109/0142159x.2012.652239 · Cited by 892 · see citing works
Self-citation
Program evaluation models and related theories: AMEE Guide No. 67
FRYE, A. W., & Hemmer, P. A. (2012) · Medical Teacher · doi:10.3109/0142159x.2012.668637 · Cited by 641 · see citing works
A Practical Guide for Medical Teachers
Dent, J. A., & Harden, R. M. (2001) · Cited by 344 · see citing works
The assessment of professional competence: building blocks for theory development
van der Vleuten, C. P. M., Schuwirth, L., Scheele, F., Driessen, E. W., & Hodges, B. D. (2010) · Best Practice & Research Clinical Obstetrics & Gynaecology · doi:10.1016/j.bpobgyn.2010.04.001 · Cited by 341 · see citing works
Self-citation
Programmatic assessment of competency-based workplace learning: when theory meets practice
Bok, H. G., Teunissen, P. W., Favier, R. P., Rietbroek, N. J., Theyse, L. F., Brommer, H., Haarhuis, J. C., van Beukelen, P., van der Vleuten, C. P., & Jaarsma, D. A. (2013) · BMC Medical Education · doi:10.1186/1472-6920-13-123 · Cited by 303 · see citing works
Reflection: a link between receiving and using assessment feedback
Sargeant, J. M., Mann, K. V., van der Vleuten, C. P., & Metsemakers, J. F. (2009) · Advances in Health Sciences Education · doi:10.1007/s10459-008-9124-4 · Cited by 296 · see citing works
Pitfalls in the pursuit of objectivity: issues of reliability
van der Vleuten, C. P. M., Norman, G. R., & de Graaff, E. (1991) · Medical Education · doi:10.1111/j.1365-2923.1991.tb00036.x · Cited by 285 · see citing works
Broadening Perspectives on Clinical Performance Assessment: Rethinking the Nature of In-training Assessment
Govaerts, M., van der Vleuten, C. P. M., Schuwirth, L., & Muijtjens, A. M. M. (2006) · Advances in Health Sciences Education · doi:10.1007/s10459-006-9043-1 · Cited by 284 · see citing works
Self-citation
Toward an empirical research agenda for sustainability in higher education: exploring the transition management framework
Stephens, J. C., & Graham, A. C. (2009) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2009.07.009 · Cited by 270 · see citing works
Learning from clinical work: the roles of learning cues and credibility judgements
Watling, C., Driessen, E., van der Vleuten, C. P. M., & Lingard, L. (2012) · Medical Education · doi:10.1111/j.1365-2923.2011.04126.x · Cited by 261 · see citing works
Self-citation
Validity in work‐based assessment: expanding our horizons
Govaerts, M., & van der Vleuten, C. P. M. (2013) · Medical Education · doi:10.1111/medu.12289 · Cited by 250 · see citing works
Self-citation
Towards more powerful learning environments through combining the perspectives of designers, teachers, and students
Könings, K. D., Brand‐Gruwel, S., & van Merriënboer, J. J. G. (2005) · British Journal of Educational Psychology · doi:10.1348/000709905x43616 · Cited by 226 · see citing works
A Practical Guide for Medical Teachers
Martyn, C. (2001) · Journal of the Royal Society of Medicine · doi:10.1177/014107680109401222 · Cited by 224 · see citing works
Influencing variables and moderators of transfer of learning to the workplace within the area of staff development in higher education: Research review
Rijdt, C. D., Stes, A., van der Vleuten, C. P. M., & Dochy, F. J. R. C. (2012) · Educational Research Review · doi:10.1016/j.edurev.2012.05.007 · Cited by 205 · see citing works
The Portfolio Approach to Competency-Based Assessment at the Cleveland Clinic Lerner College of Medicine
Dannefer, E. F., & Henson, L. C. (2007) · Academic Medicine · doi:10.1097/acm.0b013e31803ead30 · Cited by 181 · see citing works
The Wiley Blackwell Encyclopedia of Health, Illness, Behavior, and Society
Claro, M., & Bretin, H. (2013) · Wiley eBooks · doi:10.1002/9781118410868 · Cited by 170 · see citing works
Programmatic assessment and Kane’s validity perspective
Schuwirth, L., & van der Vleuten, C. P. M. (2011) · Medical Education · doi:10.1111/j.1365-2923.2011.04098.x · Cited by 165 · see citing works
Self-citation
The use of programmatic assessment in the clinical workplace: A Maastricht case report
Driessen, E. W., van Tartwijk, J., Govaerts, M., Teunissen, P., & van der Vleuten, C. P. M. (2012) · Medical Teacher · doi:10.3109/0142159x.2012.652242 · Cited by 116 · see citing works
Self-citation
The effectiveness of immediate feedback during the Objective Structured Clinical Examination
Hodder, R., Rivington, R. N., CALCUTT, L. E., & Hart, I. R. (1989) · Medical Education · doi:10.1111/j.1365-2923.1989.tb00884.x · Cited by 107 · see citing works
Web‐based feedback after summative assessment: how do students engage?
Harrison, C., Könings, K. D., Molyneux, A., Schuwirth, L., Wass, V., & van der Vleuten, C. P. M. (2013) · Medical Education · doi:10.1111/medu.12209 · Cited by 90 · see citing works
Self-citation
Oxford Textbook of Medical Education
Walsh, K. (2013) · Oxford University Press eBooks · doi:10.1093/med/9780199652679.001.0001 · Cited by 89 · see citing works
Expert validation of fit-for-purpose guidelines for designing programmes of assessment
Dijkstra, J., Galbraith, R. M., Hodges, B. D., McAvoy, P. A., McCrorie, P., Southgate, L. J., van der Vleuten, C. P. M., Wass, V., & Schuwirth, L. (2012) · BMC Medical Education · doi:10.1186/1472-6920-12-20 · Cited by 64 · see citing works
Self-citation
Flexible electronic feedback using the virtues of progress testing
Muijtjens, A. M. M., Timmermans, I., Donkers, J. H. H. L. M., Peperkamp, R., Medema, H., Cohen‐Schotanus, J., Thoben, A. J. N. M., Wenink, A. C. G., & van der Vleuten, C. P. M. (2010) · Medical Teacher · doi:10.3109/0142159x.2010.486058 · Cited by 35 · see citing works
The 2011 P rogram E valuation S tandards: a framework for quality in medical education programme evaluations
Ruhe, V., & Boudreau, J. D. (2012) · Journal of Evaluation in Clinical Practice · doi:10.1111/j.1365-2753.2012.01879.x · Cited by 25 · see citing works
Mentoring
Driessen, E. W., & Overeem, K. (2013) · Oxford University Press eBooks · doi:10.1093/med/9780199652679.003.0023 · Cited by 8 · see citing works
Self-citation
Medical Education and Faculty Development
Steinert, Y. (2014) · The Wiley Blackwell Encyclopedia of Health, Illness, Behavior, and Society · doi:10.1002/9781118410868.wbehibs374 · Cited by 2 · see citing works

Flagged first, then by citations

Sorted by citations

Applications and Challenges of Implementing Artificial Intelligence in Medical Education: Integrative Review
Chan, K. S., & Zary, N. (2019) · JMIR Medical Education · doi:10.2196/13930 · Cited by 765 · see citing works
“Programmatic assessment involves the use of an AI system to design an assessment program tailored to optimize learning outcomes and ensure curriculum quality at a systemic level [59].”
2018 Consensus framework for good assessment
Norcini, J. J., Anderson, M. B., Bóllela, V. R., Burch, V. C., Costa, M. J., Duvivier, R. J., Hays, R., Mackay, M. F. P., Roberts, T. E., & Swanson, D. B. (2018) · Medical Teacher · doi:10.1080/0142159x.2018.1500016 · Cited by 330 · see citing works
Implementing an Entrustable Professional Activities Framework in Undergraduate Medical Education: Early Lessons From the AAMC Core Entrustable Professional Activities for Entering Residency Pilot
Lomis, K. D., Amiel, J. M., Ryan, M. S., Esposito, K. F., Green, M., Stagnaro‐Green, A., Bull, J., & Mejicano, G. C. (2017) · Academic Medicine · doi:10.1097/acm.0000000000001543 · Cited by 242 · see citing works
A scoping review on the notions of Assessment as Learning (AaL), Assessment for Learning (AfL), and Assessment of Learning (AoL)
Schellekens, L. H., Bok, H. G. J., de Jong, L. H., van der Schaaf, M. F., Kremer, W. D. J., & van der Vleuten, C. P. M. (2021) · Studies In Educational Evaluation · doi:10.1016/j.stueduc.2021.101094 · Cited by 229 · see citing works
The R2C2 Model in Residency Education: How Does It Foster Coaching and Promote Feedback Use?
Sargeant, J., Lockyer, J. M., Mann, K., Armson, H., Warren, A., Zetkulic, M., Soklaridis, S., Könings, K. D., Ross, K., Silver, I., et al. (2018) · Academic Medicine · doi:10.1097/acm.0000000000002131 · Cited by 148 · see citing works
A history of assessment in medical education
Schuwirth, L., & van der Vleuten, C. P. M. (2020) · Advances in Health Sciences Education · doi:10.1007/s10459-020-10003-0 · Cited by 149 · see citing works
Self-citation
Ottawa 2020 consensus statement for programmatic assessment – 1. Agreement on the principles
Heeneman, S., de Jong, L. H., Dawson, L. J., WILKINSON, T. J., Ryan, A., Tait, G. R., Rice, N. E., Torre, D. M., Freeman, A., & van der Vleuten, C. P. M. (2021) · Medical Teacher · doi:10.1080/0142159x.2021.1957088 · Cited by 150 · see citing works
Self-citation
Learning Analytics in Medical Education Assessment: The Past, the Present, and the Future
Chan, T. M., Sebok‐Syer, S. S., Thoma, B., Wise, A. F., Sherbino, J., & Pusic, M. V. (2018) · AEM Education and Training · doi:10.1002/aet2.10087 · Cited by 126 · see citing works
Feedback and coaching
Atkinson, A. R., Watling, C., & Brand, P. L. P. (2021) · European Journal of Pediatrics · doi:10.1007/s00431-021-04118-8 · Cited by 133 · see citing works
Cognitive load theory: Practical implications and an important challenge
Leppink, J. (2017) · Journal of Taibah University Medical Sciences · doi:10.1016/j.jtumed.2017.05.003 · Cited by 120 · see citing works
Healthcare Simulation Standards of Best PracticeTM Evaluation of Learning and Performance
McMahon, E., Jimenez, F. A., Lawrence, K., & Victor, J. (2021) · Clinical Simulation in Nursing · doi:10.1016/j.ecns.2021.08.016 · Cited by 106 · see citing works
Entrusting Observable Practice Activities and Milestones Over the 36 Months of an Internal Medicine Residency
Warm, E. J., Held, J. D., Hellmann, M., Kelleher, M., Kinnear, B., Lee, C., O’Toole, J. K., Mathis, B. R., Mueller, C. V., Sall, D. R., et al. (2016) · Academic Medicine · doi:10.1097/acm.0000000000001292 · Cited by 89 · see citing works
Crossing the Gap: Using Competency-Based Assessment to Determine Whether Learners Are Ready for the Undergraduate-to-Graduate Transition
Murray, K. E., Lane, J. L., Carraccio, C. L., Glasgow, T. S., Long, M., West, D. C., O’Connor, M., Hobday, P. M., Schwartz, A., & Englander, R. (2018) · Academic Medicine · doi:10.1097/acm.0000000000002535 · Cited by 80 · see citing works
Ten key features of the future medical school—not an impossible dream
Harden, R. M. (2018) · Medical Teacher · doi:10.1080/0142159x.2018.1498613 · Cited by 78 · see citing works
What we measure … and what we should measure in medical education
Boulet, J. R., & Durning, S. J. (2018) · Medical Education · doi:10.1111/medu.13652 · Cited by 77 · see citing works
Twelve tips to maximize the value of a clinical competency committee in postgraduate medical education
Kinnear, B., Warm, E. J., & Hauer, K. E. (2018) · Medical Teacher · doi:10.1080/0142159x.2018.1474191 · Cited by 71 · see citing works
Implementation and Use of Workplace-Based Assessment in Clinical Learning Environments: A Scoping Review
Anderson, H. L. K., Kurtz, J. B., & West, D. C. (2021) · Academic Medicine · doi:10.1097/acm.0000000000004366 · Cited by 69 · see citing works
Competency‐based education calls for programmatic assessment: But what does this look like in practice?
Rich, J. V., Young, S. F., Donnelly, C. A., Hall, A. K., Dagnone, D., Weersink, K., Caudle, J. M., Melle, E. V., & Klinger, D. A. (2019) · Journal of Evaluation in Clinical Practice · doi:10.1111/jep.13328 · Cited by 67 · see citing works
Numbers Encapsulate, Words Elaborate: Toward the Best Use of Comments for Assessment and Feedback on Entrustment Ratings
Ginsburg, S., Watling, C., Schumacher, D. J., Gingerich, A., & Hatala, R. A. (2021) · Academic Medicine · doi:10.1097/acm.0000000000004089 · Cited by 64 · see citing works
The ABCs of entrustable professional activities: an overview of ‘entrustable professional activities’ in medical education
El‐Haddad, C., Damodaran, A. K., McNeil, H. P., & Hu, W. C. Y. (2015) · Internal Medicine Journal · doi:10.1111/imj.12914 · Cited by 63 · see citing works
Multiple‐role mentoring: mentors’ conceptualisations, enactments and role conflicts
Meeuwissen, S. N. E., Stalmeijer, R. E., & Govaerts, M. (2019) · Medical Education · doi:10.1111/medu.13811 · Cited by 62 · see citing works
Self-citation
Artificial intelligence in medical education: Are we ready for it?
Imran, N., & Jawaid, M. (2020) · Pakistan Journal of Medical Sciences · doi:10.12669/pjms.36.5.3042 · Cited by 56 · see citing works
USMLE Step 2 CK: Best Predictor of Multimodal Performance in an Internal Medicine Residency
Sharma, A., Schauer, D. P., Kelleher, M., Kinnear, B., Sall, D. R., & Warm, E. J. (2019) · Journal of Graduate Medical Education · doi:10.4300/jgme-d-19-00099.1 · Cited by 60 · see citing works
Online Project-Based Learning and Formative Assessment
Chanpet, P., Chomsuwan, K., & Murphy, E. (2018) · Technology Knowledge and Learning · doi:10.1007/s10758-018-9363-2 · Cited by 67 · see citing works
Assessment of communication skills
van der Vleuten, C. P. M., van den Eertwegh, V., & Giroldi, E. (2019) · Patient Education and Counseling · doi:10.1016/j.pec.2019.07.007 · Cited by 67 · see citing works
Programmatic assessment: the process, rationale and evidence for modern evaluation approaches in medical education
van der Vleuten, C. P. M., Lindemann, I., & Schmidt, L. (2018) · The Medical Journal of Australia · doi:10.5694/mja17.00926 · Cited by 56 · see citing works
Ottawa 2020 consensus statements for programmatic assessment – 2. Implementation and practice
Torre, D. M., Rice, N. E., Ryan, A., Bok, H. G. J., Dawson, L. J., Bierer, S. B., WILKINSON, T. J., Tait, G. R., Laughlin, T., Veerapen, K., et al. (2021) · Medical Teacher · doi:10.1080/0142159x.2021.1956681 · Cited by 55 · see citing works
Self-citation
Twelve tips for embedding assessment for and as learning practices in a programmatic assessment system
Sein, A. S., Rashid, H., Meka, J., Amiel, J. M., & Pluta, W. J. (2020) · Medical Teacher · doi:10.1080/0142159x.2020.1789081 · Cited by 55 · see citing works
How ‘Testing’ Has Become ‘Programmatic Assessment for Learning’
Schuwirth, L., & van der Vleuten, C. P. M. (2018) · Health Professions Education · doi:10.1016/j.hpe.2018.06.005 · Cited by 53 · see citing works
Self-citation
Stakeholder perspectives on workplace-based performance assessment: towards a better understanding of assessor behaviour
de Jonge, L. P. J. W. M., Timmerman, A. A., Govaerts, M., Muris, J. W. M., Muijtjens, A. M. M., Kramer, A. W. M., & van der Vleuten, C. P. M. (2017) · Advances in Health Sciences Education · doi:10.1007/s10459-017-9760-7 · Cited by 49 · see citing works
Self-citation
Fairness in human judgement in assessment: a hermeneutic literature review and conceptual framework
Valentine, N. A., Durning, S. J., Shanahan, E. M., & Schuwirth, L. (2020) · Advances in Health Sciences Education · doi:10.1007/s10459-020-10002-1 · Cited by 50 · see citing works
Self-citation
McMaster Modular Assessment Program (McMAP) Through the Years: Residents' Experience With an Evolving Feedback Culture Over a 3‐year Period
Li, S., Sherbino, J., & Chan, T. M. (2016) · AEM Education and Training · doi:10.1002/aet2.10009 · Cited by 48 · see citing works
Assessment Pearls for Competency-Based Medical Education
Humphrey-Murto, S. M., Wood, T. J., Ross, S., Tavares, W., Kvern, B. L., Sidhu, R. S., Sargeant, J. M., & Touchie, C. (2017) · Journal of Graduate Medical Education · doi:10.4300/jgme-d-17-00365.1 · Cited by 47 · see citing works
Very short answer questions: a viable alternative to multiple choice questions
Puthiaparampil, T., & Rahman, M. M. (2020) · BMC Medical Education · doi:10.1186/s12909-020-02057-w · Cited by 48 · see citing works
Key considerations in planning and designing programmatic assessment in competency-based medical education
Ross, S., Hauer, K. E., Wycliffe-Jones, K., Hall, A. K., Molgaard, L. K., Richardson, D. L., Oswald, A., Bhanji, F., & Collaborators, O. B. O. T. I. (2021) · Medical Teacher · doi:10.1080/0142159x.2021.1925099 · Cited by 49 · see citing works
Development of a Postgraduate Community Pharmacist Specialization Program Using CanMEDS Competencies, and Entrustable Professional Activities
Westein, M. P. D., de Vries, H., Floor‐Schreudering, A., Koster, A. S., & Buurma, H. (2018) · American Journal of Pharmaceutical Education · doi:10.5688/ajpe6863 · Cited by 46 · see citing works
Next Steps in the Implementation of Learning Analytics in Medical Education: Consensus From an International Cohort of Medical Educators
Thoma, B., Warm, E. J., Hamstra, S. J., Cavalcanti, R. B., Pusic, M. V., Shaw, T. J., Verma, A., Frank, J. R., & Hauer, K. E. (2020) · Journal of Graduate Medical Education · doi:10.4300/jgme-d-19-00493.1 · Cited by 44 · see citing works
An international study on the implementation of programmatic assessment: Understanding challenges and exploring solutions
Torre, D. M., Schuwirth, L., van der Vleuten, C. P. M., & Heeneman, S. (2022) · Medical Teacher · doi:10.1080/0142159x.2022.2083487 · Cited by 44 · see citing works
Self-citation
Video-based feedback using real consultations for a formative assessment in communication skills
Döhms, M., Collares, C. F., & Tibério, I. D. F. L. C. (2020) · BMC Medical Education · doi:10.1186/s12909-020-1955-6 · Cited by 46 · see citing works
Using Natural Language Processing to Evaluate the Quality of Supervisor Narrative Comments in Competency-Based Medical Education
Spadafore, M., Yılmaz, Y., Rally, V., Chan, T. M., Russell, M., Thoma, B., Singh, S., Monteiro, S., Pardhan, A., Martin, L. J., et al. (2024) · Academic Medicine · doi:10.1097/acm.0000000000005634 · Cited by 41 · see citing works
On the validity of summative entrustment decisions
Touchie, C., Kinnear, B., Schumacher, D. J., Caretta‐Weyer, H. A., Hamstra, S. J., Hart, D. E., Gruppen, L. D., Ross, S., Warm, E. J., ten Cate, O., et al. (2021) · Medical Teacher · doi:10.1080/0142159x.2021.1925642 · Cited by 40 · see citing works
The learning environment in remediation: a review
Cleland, J., Cilliers, F. J., & Schalkwyk, S. C. V. (2017) · The Clinical Teacher · doi:10.1111/tct.12739 · Cited by 40 · see citing works
Generalizability of Competency Assessment Scores Across and Within Clerkships: How Students, Assessors, and Clerkships Matter
Zaidi, N. L. B., Kreiter, C. D., Castaneda, P., Schiller, J. H., Yang, J., Grum, C. M., Hammoud, M. M., Gruppen, L. D., & Santen, S. A. (2018) · Academic Medicine · doi:10.1097/acm.0000000000002262 · Cited by 40 · see citing works
The power of subjectivity in competency-based assessment
Virk, A., Joshi, A., Mahajan, R., & Singh, T. (2020) · Journal of Postgraduate Medicine · doi:10.4103/jpgm.jpgm_591_20 · Cited by 40 · see citing works
How Trainees Come to Trust Supervisors in Workplace-Based Assessment: A Grounded Theory Study
Castanelli, D. J., Weller, J. M., Molloy, E., & Bearman, M. (2021) · Academic Medicine · doi:10.1097/acm.0000000000004501 · Cited by 39 · see citing works
The pursuit of fairness in assessment: Looking beyond the objective
Valentine, N. A., Durning, S. J., Shanahan, E. M., van der Vleuten, C. P. M., & Schuwirth, L. (2022) · Medical Teacher · doi:10.1080/0142159x.2022.2031943 · Cited by 39 · see citing works
Self-citation
Nuance and Noise: Lessons Learned From Longitudinal Aggregated Assessment Data
Chan, T. M., Sherbino, J., & Mercuri, M. (2017) · Journal of Graduate Medical Education · doi:10.4300/jgme-d-17-00086.1 · Cited by 39 · see citing works
Assessment and Feedback in Higher Education: A Review of Literature for the Higher Education Academy
Jackel, B., Pearce, J., Radloff, A., & Edwards, D. (2017) · ACEReSearch Repository (Australian Council for Educational Research) · Cited by 37 · see citing works
Programmatic Assessment of Competence in Dietetics: A New Frontier
Palermo, C. E., Gibson, S., Dart, J., Whelan, K., & Hay, M. (2016) · Journal of the Academy of Nutrition and Dietetics · doi:10.1016/j.jand.2016.03.022 · Cited by 36 · see citing works
Shadow systems in assessment: how supervisors make progress decisions in practice
Castanelli, D. J., Weller, J. M., Molloy, E., & Bearman, M. (2019) · Advances in Health Sciences Education · doi:10.1007/s10459-019-09913-5 · Cited by 36 · see citing works
Implementing Systematic Faculty Development to Support an EPA-Based Program of Assessment: Strategies, Outcomes, and Lessons Learned
Bray, M. J., Bradley, E. B., Martindale, J. R., & Gusic, M. E. (2020) · Teaching and Learning in Medicine · doi:10.1080/10401334.2020.1857256 · Cited by 38 · see citing works
Status of portfolios in undergraduate medical education in the LCME accredited US medical school
Chertoff, J., Wright, A., Novak, M. A., Fantone, J. C., Fleming, A., Ahmed, T., Green, M. M., Kalet, A. L., Linsenmeyer, M., Jacobs, J. L., et al. (2015) · Medical Teacher · doi:10.3109/0142159x.2015.1114595 · Cited by 34 · see citing works
A Reliability Analysis of Entrustment-Derived Workplace-Based Assessments
Kelleher, M., Kinnear, B., Sall, D. R., Schumacher, D. J., Schauer, D. P., Warm, E. J., & Kelcey, B. (2019) · Academic Medicine · doi:10.1097/acm.0000000000002997 · Cited by 33 · see citing works
Driving lesson or driving test?: A metaphor to help faculty separate feedback from assessment
Brand, P. L. P., Jaarsma, D. A. D. C., & van der Vleuten, C. P. M. (2020) · Perspectives on Medical Education · doi:10.1007/s40037-020-00617-w · Cited by 36 · see citing works
Formative assessment and feedback in medical education: A practical guide: AMEE Guide No. 189
Lipnevich, A. A., Mattern, K., & Feddock, C. (2026) · Medical Teacher · doi:10.1080/0142159x.2025.2569623 · Cited by 37 · see citing works
Student perspectives on programmatic assessment in a large medical programme: A critical realist analysis
Roberts, C., Khanna, P., Bleasel, J. F., Lane, S., Burgess, A. W., Charles, K. A., Howard, R., O'Mara, D. A., Haq, I., & Rutzou, T. (2022) · Medical Education · doi:10.1111/medu.14807 · Cited by 33 · see citing works
Estruturação de Sistemas para Avaliação Programática do Estudante de Medicina
Troncon, L. E. D. A. (2016) · Revista Brasileira de Educação Médica · doi:10.1590/1981-52712015v40n1e01392015 · Cited by 33 · see citing works
The Impact of Assessment for Learning on Learner Performance in Life Science
Oyinloye, O. M., & Imenda, S. N. (2019) · Eurasia Journal of Mathematics Science and Technology Education · doi:10.29333/ejmste/108689 · Cited by 39 · see citing works
Designing programmes of assessment: A participatory approach
Jamieson, J., Jenkins, G., BEATTY, S. E., & Palermo, C. E. (2017) · Medical Teacher · doi:10.1080/0142159x.2017.1355447 · Cited by 32 · see citing works
On Lampposts, Sneetches, and Stars: A Call to Go Beyond Bibliometrics for Determining Academic Value
Chan, T. M., & Kuehl, D. (2019) · Academic Emergency Medicine · doi:10.1111/acem.13707 · Cited by 32 · see citing works
Programmatic Assessment in Emergency Medicine: Implementation of Best Practices
Perry, M. A., Linn, A. M. J., Munzer, B., Hopson, L. R., Amlong, A., Cole, M., & Santen, S. A. (2017) · Journal of Graduate Medical Education · doi:10.4300/jgme-d-17-00094.1 · Cited by 31 · see citing works
Moving Toward Summative Competency Assessment to Individualize the Postclerkship Phase
Keeley, M., Gusic, M. E., Morgan, H. K., Aagaard, E., & Santen, S. A. (2019) · Academic Medicine · doi:10.1097/acm.0000000000002830 · Cited by 30 · see citing works
Mentors’ Beliefs About Their Roles in Health Care Education: A Qualitative Study of Mentors’ Personal Interpretative Framework
Loosveld, L. M., Gerven, P. W. M. V., Vanassche, E., & Driessen, E. W. (2020) · Academic Medicine · doi:10.1097/acm.0000000000003159 · Cited by 30 · see citing works
Self-citation
Nationwide introduction of a new competency framework for undergraduate medical curricula: a collaborative approach
Sohrmann, M., Berendonk, C., NENDAZ, M. R., Bonvin, R., & Swiss Working Group For Profiles Implementation (2020) · Swiss Medical Weekly · doi:10.57187/smw.2020.20201 · Cited by 30 · see citing works
A philosophical history of programmatic assessment: tracing shifting configurations
Pearce, J., & Tavares, W. (2021) · Advances in Health Sciences Education · doi:10.1007/s10459-021-10050-1 · Cited by 32 · see citing works
Entrustment Unpacked: Aligning Purposes, Stakes, and Processes to Enhance Learner Assessment
Kinnear, B., Warm, E. J., Caretta‐Weyer, H. A., Holmboe, E. S., Turner, D. A., van der Vleuten, C. P. M., & Schumacher, D. J. (2021) · Academic Medicine · doi:10.1097/acm.0000000000004108 · Cited by 29 · see citing works
What Behaviors Define a Good Physician? Assessing and Communicating About Noncognitive Skills
Warm, E. J., Kinnear, B., Lance, S. H., Schauer, D. P., & Brenner, J. M. (2021) · Academic Medicine · doi:10.1097/acm.0000000000004215 · Cited by 29 · see citing works
A practical approach to programmatic assessment design
Timmerman, A. A., & Dijkstra, J. (2017) · Advances in Health Sciences Education · doi:10.1007/s10459-017-9756-3 · Cited by 28 · see citing works
Identifying low test-taking effort during low-stakes tests with the new Test-taking Effort Short Scale (TESS) – development and psychometrics
Schüttpelz-Brauns, K., Kadmon, M., Kiessling, C., Karay, Y., Gestmann, M., & Kämmer, J. E. (2018) · BMC Medical Education · doi:10.1186/s12909-018-1196-0 · Cited by 20 · see citing works
The Clinical Learning Environment and Workplace-Based Assessment
Henry, D., & West, D. C. (2019) · Pediatric Clinics of North America · doi:10.1016/j.pcl.2019.03.010 · Cited by 28 · see citing works
Mobile technologies to support workplace-based assessment for entrustment decisions: Guidelines for programs and educators: AMEE Guide No. 154
Marty, A., Linsenmeyer, M., George, B. C., Young, J. Q., Breckwoldt, J., & ten Cate, O. (2023) · Medical Teacher · doi:10.1080/0142159x.2023.2168527 · Cited by 28 · see citing works
Development of and Preliminary Validity Evidence for the EFeCT Feedback Scoring Tool
Ross, S., Hamza, D. M., Zulla, R. T., Stasiuk, S., & Nichols, D. N. (2022) · Journal of Graduate Medical Education · doi:10.4300/jgme-d-21-00602.1 · Cited by 27 · see citing works
Considerations that will determine if competency-based assessment is a sustainable innovation
Dauphinée, W. D., Boulet, J. R., & Norcini, J. J. (2018) · Advances in Health Sciences Education · doi:10.1007/s10459-018-9833-2 · Cited by 26 · see citing works
Entrustment Decision Making in the Core Entrustable Professional Activities: Results of a Multi-Institutional Study
Brown, D. R., Moeller, J. J., Grbic, D., Biskobing, D. M., Crowe, R., Cutrer, W. B., Green, M. L., Obeso, V. T., Wagner, D. P., Warren, J. B., et al. (2021) · Academic Medicine · doi:10.1097/acm.0000000000004242 · Cited by 26 · see citing works
On the issue of costs in programmatic assessment
van der Vleuten, C. P. M., & Heeneman, S. (2016) · Perspectives on Medical Education · doi:10.1007/s40037-016-0295-z · Cited by 25 · see citing works
Self-citation
Why assessment in medical education needs a solid foundation in modern test theory
Schauber, S. K., Hecht, M., & Nouns, Z. M. (2017) · Advances in Health Sciences Education · doi:10.1007/s10459-017-9771-4 · Cited by 25 · see citing works
Identifying Core Components of EPA Implementation: A Path to Knowing if a Complex Intervention Is Being Implemented as Intended
Carraccio, C. L., Martini, A., Melle, E. V., & Schumacher, D. J. (2021) · Academic Medicine · doi:10.1097/acm.0000000000004075 · Cited by 25 · see citing works
Shaping the right conditions in programmatic assessment: how quality of narrative information affects the quality of high-stakes decision-making
de Jong, L. H., Bok, H. G. J., Schellekens, L. H., Kremer, W. D. J., Jonker, F. H., & van der Vleuten, C. P. M. (2022) · BMC Medical Education · doi:10.1186/s12909-022-03257-2 · Cited by 24 · see citing works
Competency Committees in Undergraduate Medical Education: Approaching Tensions Using a Polarity Management Framework
Monrad, S. U., Mangrulkar, R. S., Woolliscroft, J. O., Daniel, M. M., Hartley, S. E., Gay, T. L., Highet, A., Vijayakumar, N., & Santen, S. A. (2019) · Academic Medicine · doi:10.1097/acm.0000000000002816 · Cited by 24 · see citing works
Ten caveats of learning analytics in health professions education: A consumer’s perspective
ten Cate, O., Dahdal, S., Lambert, T., Neubauer, F. B., Pless, A., Pohlmann, P.-F., van Rijen, H. V. M., & Gurtner, C. (2020) · Medical Teacher · doi:10.1080/0142159x.2020.1733505 · Cited by 24 · see citing works
Blended programmatic assessment for competency based curricula
Mahajan, R., Saiyad, S. N., Virk, A., Joshi, A., & Singh, T. (2021) · Journal of Postgraduate Medicine · doi:10.4103/jpgm.jpgm_1061_20 · Cited by 27 · see citing works
Exploring complexities in the reform of assessment practice: a critical realist perspective
Roberts, C., Khanna, P., Lane, S., Reimann, P., & Schuwirth, L. (2021) · Advances in Health Sciences Education · doi:10.1007/s10459-021-10065-8 · Cited by 24 · see citing works
Self-citation
How Trainees Come to Trust Supervisors in Workplace-Based Assessment
Castanelli, D. J., Weller, J. M., Molloy, E., & Bearman, M. (2021) · Own your potential (DEAKIN) · Cited by 24 · see citing works
Monitoring communication skills progress of medical students: Establishing a baseline has value, predicting the future is difficult
Hanley, K., Gillespie, C. C., Zabar, S. R., Adams, J., & Kalet, A. L. (2018) · Patient Education and Counseling · doi:10.1016/j.pec.2018.09.010 · Cited by 24 · see citing works
Using the Five Core Components of Competency-Based Medical Education to Support Implementation of CBVE
Chaney, K. P., & Hodgson, J. (2021) · Frontiers in Veterinary Science · doi:10.3389/fvets.2021.689356 · Cited by 23 · see citing works
An authentic learner-centered planetary health assignment: A five-year evaluation of student choices to address Sustainable Development Goal 13 (Climate Action)
McLean, M., Phelps, C., Smith, J. K., Maheshwari, N., Veer, V., Bushell, D., Matthews, R. S., Craig, B. M., & Moro, C. (2022) · Frontiers in Public Health · doi:10.3389/fpubh.2022.1049932 · Cited by 23 · see citing works
Feasibility of peer assessment and clinical audit to self-regulate the quality of physiotherapy services: a mixed methods study
Maas, M. J. M., der Sanden, M. W. G. N., Driehuis, F., Heerkens, Y. F., van der Vleuten, C. P. M., & van der Wees, P. J. (2017) · BMJ Open · doi:10.1136/bmjopen-2016-013726 · Cited by 21 · see citing works
Programmatic assessment: Can we provide evidence for saturation of information?
de Jong, L. H., Bok, H. G. J., Kremer, W. D. J., & van der Vleuten, C. P. M. (2019) · Medical Teacher · doi:10.1080/0142159x.2018.1555369 · Cited by 21 · see citing works
Entrustable Professional Activities: Reflecting on Where We Are to Define a Path for the Next Decade
Schumacher, D. J., & Turner, D. A. (2021) · Academic Medicine · doi:10.1097/acm.0000000000004097 · Cited by 21 · see citing works
A Call for an Integrated Program of Assessment
Fielding, D. W., & Regehr, G. (2017) · American Journal of Pharmaceutical Education · doi:10.5688/ajpe81477 · Cited by 21 · see citing works
The education avenue of the clinical learning environment: A pragmatic approach
Chan, M., Snell, L. S., & Philibert, I. (2019) · Medical Teacher · doi:10.1080/0142159x.2019.1566602 · Cited by 20 · see citing works
Juggling roles and generating solutions; practice-based educators’ perceptions of performance-based assessment of physiotherapy students
O’Connor, A., Cantillon, P., Parker, M., & McCurtin, A. (2018) · Physiotherapy · doi:10.1016/j.physio.2018.11.008 · Cited by 19 · see citing works
A Program of Assessment Model for Point-of-Care Ultrasound Training for Pediatric Critical Care Providers: A Comprehensive Approach to Enhance Competency-Based Point-of-Care Ultrasound Training
Maxson, I., Su, E. R., Brown, K. A., Tcharmtchi, M. H., Ginsburg, S., Bhargava, V., Wenger, J. L., Centers, G. I., Alade, K. H., Leung, S. K., et al. (2023) · Pediatric Critical Care Medicine · doi:10.1097/pcc.0000000000003288 · Cited by 18 · see citing works
Implementing Competence Committees on a National Scale: Design and Lessons Learned
Oswald, A., Dubois, D. G., Snell, L. S., Anderson, R., Karpinski, J., Hall, A. K., Frank, J. R., & Cheung, W. J. (2024) · Perspectives on Medical Education · doi:10.5334/pme.961 · Cited by 19 · see citing works
Preschool Communication: Early Identification of Concerns About Preschool Language Development and Social Participation
Doove, B. M., FERON, F. J. M., van Os, J., & Drukker, M. (2021) · Frontiers in Public Health · doi:10.3389/fpubh.2020.546536 · Cited by 23 · see citing works
Challenges and opportunities in cultivating medical students’ competencies: Participatory action research from a hierarchical cultural setting
Findyartini, A., Syah, N. A., Susilo, A. P., Nurokhmanti, H., Qomariyah, N. U., Greviana, N., Ainin, D. Q., Sari, S. M., & Claramita, M. (2023) · Medical Education Online · doi:10.1080/10872981.2023.2185122 · Cited by 17 · see citing works
Transforming Resident Assessment: An Analysis Using Deming’s System of Profound Knowledge
Warm, E. J., Kinnear, B., Kelleher, M., Sall, D. R., & Holmboe, E. S. (2018) · Academic Medicine · doi:10.1097/acm.0000000000002499 · Cited by 17 · see citing works
Development and assessment of an online virtual orthodontic curriculum
Liu, L., Yuan, S., Zhang, W., Wang, Z., Zhao, C., Pan, Y., Yan, B., & Wang, L. (2021) · Journal of Dental Education · doi:10.1002/jdd.12825 · Cited by 17 · see citing works
“We just don’t have the resources”: Supervisor perspectives on introducing workplace-based assessments into medical specialist training in South Africa
Ras, T., Jenkins, L. S., Lazarus, C., van Rensburg, J. J., Cooke, R., Senkubuge, F., Dlova, A. N., Singaram, V. S., Daitz, E., Buch, E., et al. (2023) · BMC Medical Education · doi:10.1186/s12909-023-04840-x · Cited by 21 · see citing works
Ambulatory Long Block: A Model of Precision Education and Assessment for Internal Medicine Residents
Warm, E. J., Kinnear, B., Knopp, M. I., Powers‐Fletcher, M. V., Segev, M., Willauer, A. N., & Schauer, D. P. (2023) · Academic Medicine · doi:10.1097/acm.0000000000005610 · Cited by 16 · see citing works

A new holistic way of assessment: programmatic assessment
van der Vleuten, C. P. M., & Heeneman, S. (2016) · Educación Médica · doi:10.33588/fem.196.860 · Cited by 5 · see citing works
Self-citation
Programmatic Assessment: An Avenue to a Different Assessment Culture
van der Vleuten, C. P. M., Heeneman, S., & Schut, S. (2019) · Routledge eBooks · doi:10.4324/9781315166902-16 · Cited by 8 · see citing works
Self-citation
A model for programmatic assessment fit for purpose.
van der Vleuten, C. P. M., Schuwirth, L. W. T., Driessen, E. W., Dijkstra, J., Tigelaar, D., Baartman, L. K. J., & van Tartwijk, J. (2012) · Medical Teacher · doi:10.3109/0142159x.2012.652239 · Cited by 892 · see citing works
Self-citation
Deconstructing programmatic assessment
WILKINSON, T. J., & Tweed, M. (2018) · Advances in Medical Education and Practice · doi:10.2147/amep.s144449 · Cited by 53 · see citing works
Unraveling programmatic assessment
de Jong, L. H. (2022) · doi:10.33540/1033

Authors: Van Der Vleuten, C. P. M., Schuwirth, L. W. T., Driessen, E. W., Govaerts, M. J. B., & Heeneman, S.
Affiliations: Flinders University (AU)
Maastricht University (NL)
ORCID: Schuwirth, L.W.T.
Driessen, E.W.
Govaerts, M.J.B.
Heeneman, S.
Venue: Medical Teacher
Publisher: Informa UK Limited
Published: 2014-11-20
Volume: 37
Issue: 7
Pages: 641-646
ISSN: 0142-159X (ISSN-L)
1466-187X
DOI: 10.3109/0142159x.2014.973388
OpenAlex: W1738312775
Cited by: 333
References: 33
Citations by year: 2026: 32 · 2025: 21 · 2024: 29 · 2023: 42 · 2022: 26 · 2021: 47 · 2020: 33 · 2019: 32 · 2018: 30 · 2017: 22 · 2016: 10 · 2015: 5 · 2013: 2
FWCI: 12.27
Type: journal article
Language: en
Open access: closed
Locations: Medical Teacher (publishedVersion)
PubMed (publishedVersion)
Research Publications (Maastricht University) (publishedVersion)
Research Publications (Maastricht University) (submittedVersion)
Topics: Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Innovations in Medical Education
Social Sciences › Education › Student Assessment and Feedback
Social Sciences › Education › Educational Assessment and Pedagogy
Keywords: programmatic assessment, assessment for learning, assessment design, feedback to learners, curriculum quality assurance, high-stakes decision making, information aggregation, student self-regulation, mentoring, constructivist learning, medical education

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Twelve Tips for programmatic assessment

Abstract

Integrity signals

Claims the paper makes

Claims other papers make about this one

Cites

Cited by

Related

Record

About these checks