The DOI matches its Crossref record; 69 of the 69 references it cites were found in a registry or an index.
A model for programmatic assessment fit for purpose.
Abstract
We propose a model for programmatic assessment in action, which simultaneously optimises assessment for learning and assessment for decision making about learner progress. This model is based on a set of assessment principles that are interpreted from empirical research. It specifies cycles of training, assessment and learner support activities that are complemented by intermediate and final moments of evaluation on aggregated assessment data points. A key principle is that individual data points are maximised for learning and feedback value, whereas high-stake decisions are based on the aggregation of many data points. Expert judgement plays an important role in the programme. Fundamental is the notion of sampling and bias reduction to deal with the inevitable subjectivity of this type of judgement. Bias reduction is further sought in procedural assessment strategies derived from criteria for qualitative research. We discuss a number of challenges and opportunities around the proposed model. One of its prime virtues is that it enables assessment to move, beyond the dominant psychometric discourse with its focus on individual instruments, towards a systems approach to assessment design underpinned by empirically grounded theory.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Finally, they promote the longitudinal compilation of performance data to coach the learner's continuing development (assessment for learning) and to make evidence-based decisions about the learner's progress (assessment of learning).[11][12][13][14][15]”
From The R2C2 Model in Residency Education: How Does It Foster Coaching and Promote Feedback Use? (2018) · cites this paper as [12]
In the citing paper
“Programmatic assessment (a system of frequent formative assessments also used for end-of-year summative judgements) is designed to optimize learning and reduce exam stress [86].”
From Guidelines: The do’s, don’ts and don’t knows of feedback for clinical education. (2015) · cites this paper as [86]
In the citing paper
“Instead, a novel assessment system has been proposed, by which the utility of assessment can complement the former approach by integrating the results of various types of feedback opportunities programmatically [42].”
From Developing Medical Education Curriculum Reform Strategies to Address the Impact of Generative AI: Qualitative Study (2023) · cites this paper as [42]
Citations
Cites
Hattie, J., & Timperley, H. (2007) · Review of Educational Research · doi:10.3102/003465430298487 · Cited by 12,905
Kluger, A. N., & DeNisi, A. (1996) · Psychological Bulletin · doi:10.1037/0033-2909.119.2.254 · Cited by 5,882
The assessment of clinical skills/competence/performance
Miller, G. E. (1990) · Academic Medicine · doi:10.1097/00001888-199009000-00045 · Cited by 4,883
Shute, V. J. (2008) · Review of Educational Research · doi:10.3102/0034654307313795 · Cited by 4,631
Klein, G. A. (2008) · Human Factors The Journal of the Human Factors and Ergonomics Society · doi:10.1518/001872008x288385 · Cited by 2,434
Harvey, L., & Green, D. (1993) · Assessment & Evaluation in Higher Education · doi:10.1080/0260293930180102 · Cited by 1,281
Assessing professional competence: from methods to programmes
Van Der Vleuten, C. P. M., & Schuwirth, L. W. T. (2005) · Medical Education · doi:10.1111/j.1365-2929.2005.02094.x · Cited by 1,268
Self-citationLinking Theory and Practice: Changing the Pedagogy of Teacher Education
Korthagen, F. A. J., & Kessels, J. P. A. M. (1999) · Educational Researcher · doi:10.3102/0013189x028004004 · Cited by 1,266
The assessment of professional competence: Developments, research and practical implications
Van Der Vleuten, C. P. M. (1996) · Advances in Health Sciences Education · doi:10.1007/bf00596229 · Cited by 1,247
Student Peer Assessment in Higher Education: A Meta-Analysis Comparing Peer and Teacher Marks
Falchikov, N., & Goldfinch, J. (2000) · Review of Educational Research · doi:10.3102/00346543070003287 · Cited by 1,170
A Theory of Unconscious Thought
Dijksterhuis, A., & Nordgren, L. F. (2006) · Perspectives on Psychological Science · doi:10.1111/j.1745-6916.2006.00007.x · Cited by 1,089
The CanMEDS initiative: implementing an outcomes-based framework of physician competencies
Frank, J. R., & Danoff, D. S. (2007) · Medical Teacher · doi:10.1080/01421590701746983 · Cited by 1,005
Self-Assessment in the Health Professions: A Reformulation and Research Agenda
Eva, K. W., & Regehr, G. (2005) · Academic Medicine · doi:10.1097/00001888-200510001-00015 · Cited by 992
Workplace-based assessment as an educational tool: AMEE Guide No. 31
Norcini, J., & Burch, V. (2007) · Medical Teacher · doi:10.1080/01421590701775453 · Cited by 984
What every teacher needs to know about clinical reasoning
Eva, K. W. (2004) · Medical Education · doi:10.1111/j.1365-2929.2004.01972.x · Cited by 939
A review of rubric use in higher education
Reddy, Y. M., & Andrade, H. (2010) · Assessment & Evaluation in Higher Education · doi:10.1080/02602930902862859 · Cited by 917
Programmatic assessment: From assessment of learning to assessment for learning
Schuwirth, L. W. T., & Van der Vleuten, C. P. M. (2011) · Medical Teacher · doi:10.3109/0142159x.2011.565828 · Cited by 902
Self-citationMindful Judgment and Decision Making
Weber, E. U., & Johnson, E. J. (2008) · Annual Review of Psychology · doi:10.1146/annurev.psych.60.110707.163633 · Cited by 782
Educational strategies in curriculum development: the SPICES model
HARDEN, R. M., SOWDEN, S., & DUNN, W. R. (1984) · Medical Education · doi:10.1111/j.1365-2923.1984.tb01024.x · Cited by 772
An admissions OSCE: the multiple mini-interview
Eva, K. W., Rosenfeld, J., Reiter, H. I., & Norman, G. R. (2004) · Medical Education · doi:10.1046/j.1365-2923.2004.01776.x · Cited by 617
Failure to Fail: The Perspectives of Clinical Supervisors
Dudek, N. L., Marks, M. B., & Regehr, G. (2005) · Academic Medicine · doi:10.1097/00001888-200510001-00023 · Cited by 422
SPECIAL ARTICLE: Cognitive, Social and Environmental Sources of Bias in Clinical Performance Ratings
Williams, R. G., Klamen, D. L., & McGaghie, W. C. (2003) · Teaching and Learning in Medicine · doi:10.1207/s15328015tlm1504_11 · Cited by 392
Determinants of linear judgment: A meta-analysis of lens model studies.
Karelaia, N., & Hogarth, R. M. (2008) · Psychological Bulletin · doi:10.1037/0033-2909.134.3.404 · Cited by 380
Assessment of professionalism: Recommendations from the Ottawa 2010 Conference
Hodges, B. D., Ginsburg, S., Cruess, R. L., Cruess, S. R., Delport, R., Hafferty, F. W., Ho, M., Holmboe, E. S., Holtman, M. C., Ohbu, S., et al. (2011) · Medical Teacher · doi:10.3109/0142159x.2011.577300 · Cited by 359
The assessment of professional competence: building blocks for theory development
van der Vleuten, C. P. M., Schuwirth, L., Scheele, F., Driessen, E. W., & Hodges, B. D. (2010) · Best Practice & Research Clinical Obstetrics & Gynaecology · doi:10.1016/j.bpobgyn.2010.04.001 · Cited by 356
Self-citationPortfolios in medical education: why do they meet with mixed success? A systematic review
Driessen, E. W., van Tartwijk, J., van der Vleuten, C. P. M., & Wass, V. (2007) · Medical Education · doi:10.1111/j.1365-2923.2007.02944.x · Cited by 354
Self-citationFrom principles to practice: An embedded assessment system
Wilson, M., & Sloane, K. (2000) · Applied Measurement in Education · doi:10.1207/s15324818ame1302_4 · Cited by 333
The processes and dimensions of informed self-assessment: A conceptual model.
Sargeant, J., Armson, H., Chesluk, B., Dornan, T., Eva, K., Holmboe, E., Lockyer, J., Loney, E., Mann, K., & van der Vleuten, C. (2010) · Academic Medicine · doi:10.1097/acm.0b013e3181d85a4e · Cited by 318
Pitfalls in the pursuit of objectivity: issues of reliability
van der Vleuten, C. P. M., Norman, G. R., & de Graaff, E. (1991) · Medical Education · doi:10.1111/j.1365-2923.1991.tb00036.x · Cited by 285
Govaerts, M., van der Vleuten, C. P. M., Schuwirth, L., & Muijtjens, A. M. M. (2006) · Advances in Health Sciences Education · doi:10.1007/s10459-006-9043-1 · Cited by 284
Self-citationWorkplace-based assessment as an educational tool
Norcini, J. J., Burch, V. C., & Gibbs, T. J. (2007) · Medical Entomology and Zoology · Cited by 266
van Merriënboer, J. J. G. (2007) · Routledge eBooks · doi:10.4324/9781410618054 · Cited by 255
Norcini, J. J. (2003) · BMJ · doi:10.1136/bmj.326.7392.753 · Cited by 253
The portfolio as a tool for stimulating reflection by student teachers
Mansvelder-Longayroux, D. D., Beijaard, D., & Verloop, N. (2006) · Teaching and Teacher Education · doi:10.1016/j.tate.2006.04.033 · Cited by 239
Portfolios for assessment and learning: AMEE Guide no. 45
van Tartwijk, J., & Driessen, E. W. (2009) · Medical Teacher · doi:10.1080/01421590903139201 · Cited by 233
Self-citationMedical Education: Theory and Practice
Dornan, T., Mann, K. V., Scherpbier, A. J. J. A., & Spencer, J. (2010) · Newcastle University ePrints (Newcastle Univesity) · doi:10.25405/ncl.34176165 · Cited by 227
Workplace-based assessment: effects of rater expertise
Govaerts, M., Schuwirth, L., van der Vleuten, C. P. M., & Muijtjens, A. M. M. (2010) · Advances in Health Sciences Education · doi:10.1007/s10459-010-9250-7 · Cited by 206
Self-citationThe wheel of competency assessment: Presenting quality criteria for competency assessment programs
Baartman, L. K. J., Bastiaens, T. J., Kirschner, P. A., & van der Vleuten, C. P. M. (2006) · Studies In Educational Evaluation · doi:10.1016/j.stueduc.2006.04.006 · Cited by 203
Self-citationGood judgments do not require complex cognition
Marewski, J. N., Gaissmaier, W., & Gigerenzer, G. (2009) · Cognitive Processing · doi:10.1007/s10339-009-0337-0 · Cited by 202
The New Handbook of Teacher Evaluation: Assessing Elementary and Secondary School Teachers
Heining‐Boynton, A. L., Millman, J., & Darling‐Hammond, L. (1992) · Modern Language Journal · doi:10.2307/329910 · Cited by 195
Engel, B. T. (1967) · Archives of General Psychiatry · doi:10.1001/archpsyc.1967.01730230058008 · Cited by 185
On the generality of specificity
Eva, K. W. (2003) · Medical Education · doi:10.1046/j.1365-2923.2003.01563.x · Cited by 182
Dannefer, E. F., & Henson, L. C. (2007) · Academic Medicine · doi:10.1097/acm.0b013e31803ead30 · Cited by 181
The Psychology of Experts An Alternative View
Shanteau, J. (2007) · Springer eBooks · doi:10.1007/978-0-585-34290-0_2 · Cited by 177
Pitfalls in the pursuit of objectivity: issues of validity, efficiency and acceptability
Norman, G. R., van der Vleuten, C. P. M., & de Graaff, E. (1991) · Medical Education · doi:10.1111/j.1365-2923.1991.tb00037.x · Cited by 172
The Quality of In-house Medical School Examinations
Józefowicz, R. F., Koeppen, B. M., Case, S. M., Galbraith, R. M., Swanson, D. B., & Glew, R. H. (2002) · Academic Medicine · doi:10.1097/00001888-200202000-00016 · Cited by 164
The mechanism of impact of summative assessment on medical students’ learning
Cilliers, F. J., Schuwirth, L., Adendorff, H. J., Herman, N., & van der Vleuten, C. P. M. (2010) · Advances in Health Sciences Education · doi:10.1007/s10459-010-9232-9 · Cited by 164
Self-citation“Directed” self-assessment: Practice and feedback within a social context
Sargeant, J., Mann, K., van der Vleuten, C., & Metsemakers, J. (2008) · Journal of Continuing Education in the Health Professions · doi:10.1002/chp.155 · Cited by 158
van Merriënboer, J. J. G., & Sluijsmans, D. M. A. (2009) · Educational Psychology Review · doi:10.1007/s10648-008-9092-5 · Cited by 156
A model of the pre-assessment learning effects of summative assessment in medical education
Cilliers, F. J., Schuwirth, L., Herman, N., Adendorff, H. J., & van der Vleuten, C. P. M. (2011) · Advances in Health Sciences Education · doi:10.1007/s10459-011-9292-5 · Cited by 150
Self-citationThe Schoolteacher's Portfolio: An Essay on Possibilities
Bird, T. (1990) · SAGE Publications Ltd eBooks · doi:10.4135/9781412986250.n15 · Cited by 142
A new framework for designing programmes of assessment
Dijkstra, J., van der Vleuten, C. P. M., & Schuwirth, L. (2009) · Advances in Health Sciences Education · doi:10.1007/s10459-009-9205-z · Cited by 133
Self-citationMedical education and the maintenance of incompetence
Hodges, B. D. (2006) · Medical Teacher · doi:10.1080/01421590601102964 · Cited by 129
Expertise and Decision Support
Powell, P. (1993) · Journal of the Operational Research Society · doi:10.1057/jors.1993.123 · Cited by 112
Developing professional judgment
Coles, C. R. (2002) · Journal of Continuing Education in the Health Professions · doi:10.1002/chp.1340220102 · Cited by 108
Sluijsmans, D. M. A., Brand‐Gruwel, S., van Merriënboer, J. J. G., & Bastiaens, T. J. (2002) · Studies In Educational Evaluation · doi:10.1016/s0191-491x(03)90003-4 · Cited by 99
When enough is enough: a conceptual basis for fair and defensible practice performance assessment
Schuwirth, L., Southgate, L. J., Page, G. G., Paget, N. S., Lescop, J., Lew, S. R., Wade, W. B., & Maldonado, M. B. (2002) · Medical Education · doi:10.1046/j.1365-2923.2002.01313.x · Cited by 95
Self-citationCross institutional collaboration in assessment: a case on progress testing
van der Vleuten, C. P. M., Schuwirth, L., Muijtjens, A. M. M., Thoben, A. J. N. M., Cohen‐Schotanus, J., & van der Vleuten, C. P. M. (2004) · Medical Teacher · doi:10.1080/01421590400016464 · Cited by 79
Self-citationDETERMINING THE QUALITY OF COMPETENCE ASSESSMENT PROGRAMS: A SELF-EVALUATION PROCEDURE
Baartman, L. K. J., Prins, F. J., Kirschner, P. A., & van der Vleuten, C. P. M. (2007) · Studies In Educational Evaluation · doi:10.1016/j.stueduc.2007.07.004 · Cited by 75
Self-citationEmbo, M. P. C., Driessen, E. W., Valcke, M., & van der Vleuten, C. P. M. (2010) · Medical Teacher · doi:10.3109/0142159x.2010.490281 · Cited by 63
Self-citationKadri, H. M. A., Al-Moamary, M. S., & van der Vleuten, C. P. M. (2009) · BMC Research Notes · doi:10.1186/1756-0500-2-263 · Cited by 50
Research in assessment: Consensus statement and recommendations from the Ottawa 2010 Conference
Schuwirth, L., Colliver, J. A., Gruppen, L. D., Kreiter, C. D., Mennin, S. P., Onishi, H., Pangaro, L. N., Ringsted, C. V., Swanson, D. B., van der Vleuten, C. P. M., et al. (2011) · Medical Teacher · doi:10.3109/0142159x.2011.551558 · Cited by 49
Self-citationThe Education and Training of Future Physicians
Cavalcanti, R. B., & Detsky, A. S. (2011) · JAMA · doi:10.1001/jama.2011.1232 · Cited by 44
Korthagen, F. A. J., & Wubbels, T. (2001) · Routledge eBooks · doi:10.4324/9781410600523-8 · Cited by 40
Flexible electronic feedback using the virtues of progress testing
Muijtjens, A. M. M., Timmermans, I., Donkers, J. H. H. L. M., Peperkamp, R., Medema, H., Cohen‐Schotanus, J., Thoben, A. J. N. M., Wenink, A. C. G., & van der Vleuten, C. P. M. (2010) · Medical Teacher · doi:10.3109/0142159x.2010.486058 · Cited by 35
van Merriënboer, J. J. G., Kirschner, P. A., & Frèrejean, J. (2024) · Routledge eBooks · doi:10.4324/9781003322481 · Cited by 33
Quality assurance in test construction: The approach of a multidisciplinary central test committee
Verhoeven, B. H., Verwijnen, G. M., Scherpbier, A. J. J. A., Schuwirth, L., & van der Vleuten, C. P. M. (1999) · Cited by 19
Self-citationThe wheel of competency assessment
Baartman, L. K. J., Bastiaens, T. J., Kirschner, P. A., & van der Vleuten, C. P. M. (2005) · Utrecht University Repository (Utrecht University) · Cited by 6
Self-citationClinical and statistical prediction after 50 years: a dedication to Paul Meehl
Morera, O. F., & Dawes, R. M. (2006) · Journal of Behavioral Decision Making · doi:10.1002/bdm.538 · Cited by 5
Flagged first, then by citations
Research works printed but not linked: 6
- Cavalcanti RB, 2011, JAMA, 306, 993
- Cilliers FJ, 2011, Adv Health Sci Educ Theory Pract
- Dijkstra J, Galbraith R, Hodges B, McAvoy P, McCrorie P, Southgate L, Van der Vleuten C, Wass V, Schuwirth L. under editorial review. Fit-for-purpose guidelines for designing programmes of assessment
- Harden RM, 1984, Med Teach, 18, 284
- Van Merriënboer JJG, 1997, Training complex cognitive skills
- Verhoeven BH, 1999, Educ Health, 12, 49
Cited by
Sorted by citations
Boscardin, C. K., Gin, B., Golde, P. B., & Hauer, K. E. (2024) · Academic Medicine · doi:10.1097/acm.0000000000005439 · Cited by 620
Core principles of assessment in competency-based medical education.
Lockyer, J., Carraccio, C., Chan, M.-K., Hart, D., Smee, S., Touchie, C., Holmboe, E. S., Frank, J. R., & Collaborators, O. B. O. T. I. (2017) · Medical Teacher · doi:10.1080/0142159x.2017.1315082 · Cited by 493
Assessment, feedback and the alchemy of learning
Watling, C., & Ginsburg, S. (2018) · Medical Education · doi:10.1111/medu.13645 · Cited by 461
Implementation of competency‐based medical education: are we addressing the concerns and challenges?
Hawkins, R. E., Welcher, C. M., Holmboe, E. S., Kirk, L. M., Norcini, J. J., Simons, K. B., & Skochelak, S. E. (2015) · Medical Education · doi:10.1111/medu.12831 · Cited by 409
Clinical Reasoning Assessment Methods: A Scoping Review and Practical Guidance
Daniel, M. M., Rencic, J. J., Durning, S. J., Holmboe, E. S., Santen, S. A., Lang, V. J., Ratcliffe, T. A., Gordon, D., Heist, B. S., Lubarsky, S., et al. (2019) · Academic Medicine · doi:10.1097/acm.0000000000002618 · Cited by 358
Metadata
- Authors
- van der Vleuten, C. P. M., Schuwirth, L. W. T., Driessen, E. W., Dijkstra, J., Tigelaar, D., Baartman, L. K. J., & van Tartwijk, J.
- Affiliations
- Flinders University (AU)
- Maastricht University (NL)
- Leiden University (NL)
- University of Applied Sciences Utrecht (NL)
- Venue
- Medical Teacher
- Publisher
- Informa UK Limited
- Published
- 2012-02-25
- Volume
- 34
- Issue
- 3
- Pages
- 205-214
- ISSN
- 0142-159X (ISSN-L)
- 1466-187X
- OpenAlex
- W2163467994
- Cited by
- 892
- References
- 69
- Citations by year
- 2026: 140 · 2025: 62 · 2024: 70 · 2023: 74 · 2022: 72 · 2021: 71 · 2020: 59 · 2019: 66 · 2018: 69 · 2017: 56 · 2016: 50 · 2015: 32 · 2014: 40 · 2013: 21 · 2012: 9
- FWCI
- 31.53
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Medical Teacher (publishedVersion)
- PubMed (publishedVersion)
- Research Publications (Maastricht University) (publishedVersion)
- Radboud Repository (Radboud University) (submittedVersion)
- Research Publications (Maastricht University) (submittedVersion)
- Topics
- Social Sciences › Education › Innovations in Medical Education
- Social Sciences › Education › Student Assessment and Feedback
- Social Sciences › Public Administration › Evaluation and Performance Assessment
- Keywords
- programmatic assessment, assessment for learning, student progress, evaluation principles, expert opinion, bias mitigation, sampling, feedback, medical education
- MeSH terms
- Decision Making, Educational Measurement, Humans, Program Evaluation, Models, Educational
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
