The DOI matches its Crossref record; 35 of the 36 references it cites were found in a registry or an index.
Programmatic assessment of competency-based workplace learning: when theory meets practice
Open accessAbstract
BACKGROUND: In competency-based medical education emphasis has shifted towards outcomes, capabilities, and learner-centeredness. Together with a focus on sustained evidence of professional competence this calls for new methods of teaching and assessment. Recently, medical educators advocated the use of a holistic, programmatic approach towards assessment. Besides maximum facilitation of learning it should improve the validity and reliability of measurements and documentation of competence development. We explored how, in a competency-based curriculum, current theories on programmatic assessment interacted with educational practice. METHODS: In a development study including evaluation, we investigated the implementation of a theory-based programme of assessment. Between April 2011 and May 2012 quantitative evaluation data were collected and used to guide group interviews that explored the experiences of students and clinical supervisors with the assessment programme. We coded the transcripts and emerging topics were organised into a list of lessons learned. RESULTS: The programme mainly focuses on the integration of learning and assessment by motivating and supporting students to seek and accumulate feedback. The assessment instruments were aligned to cover predefined competencies to enable aggregation of information in a structured and meaningful way. Assessments that were designed as formative learning experiences were increasingly perceived as summative by students. Peer feedback was experienced as a valuable method for formative feedback. Social interaction and external guidance seemed to be of crucial importance to scaffold self-directed learning. Aggregating data from individual assessments into a holistic portfolio judgement required expertise and extensive training and supervision of judges. CONCLUSIONS: A programme of assessment with low-stakes assessments providing simultaneously formative feedback and input for summative decisions proved not easy to implement. Careful preparation and guidance of the implementation process was crucial. Assessment for learning requires meaningful feedback with each assessment. Special attention should be paid to the quality of feedback at individual assessment moments. Comprehensive attention for faculty development and training for students is essential for the successful implementation of an assessment programme.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Evidence is now emerging from qualitative evaluations of programmatic assessment curricula which raises questions about the mixing of formative and summative assessment.”
From Guidelines: The do’s, don’ts and don’t knows of feedback for clinical education. (2015) · cites this paper as [69]
In the citing paper
“although it has also met some difficulties in implementation [69]”
From Guidelines: The do’s, don’ts and don’t knows of feedback for clinical education. (2015) · cites this paper as [69]
In the citing paper
“As Bok et al. showed, durable teacher-learner relationships also prompt learners to seek feedback more readily [69].”
From Guidelines: The do’s, don’ts and don’t knows of feedback for clinical education. (2015) · cites this paper as [69]
In the citing paper
“Institutions can create opportunities for longitudinal teacher-learner relationships to flourish, such as extended placements [68–70].”
From Guidelines: The do’s, don’ts and don’t knows of feedback for clinical education. (2015) · cites this paper as [69]
In the citing paper
“As a result, the formative assessments did not play the powerful assessment-for-learning role they are meant to have in a curriculum based on programmatic assessment [69].”
From Guidelines: The do’s, don’ts and don’t knows of feedback for clinical education. (2015) · cites this paper as [69]
In the citing paper
“A qualitative study with clinical undergraduate veterinary students and their supervisors highlighted that both struggled with formative assessments that are used as ‘data points’ for a final summative judgment.”
From Guidelines: The do’s, don’ts and don’t knows of feedback for clinical education. (2015) · cites this paper as [69]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Defining and Assessing Professional Competence
Epstein, R. M. (2002) · JAMA · doi:10.1001/jama.287.2.226 · Cited by 3,114
Competency-based medical education: Theory to practice.
Frank, J. R., Snell, L. S., Cate, O. T., Holmboe, E. S., Carraccio, C., Swing, S. R., Harris, P., Glasgow, N. J., Campbell, C., Dath, D., et al. (2010) · Medical Teacher · doi:10.3109/0142159x.2010.501190 · Cited by 2,883
Design Research: Theoretical and Methodological Issues
Collins, A., Joseph, D., & Bielaczyc, K. (2004) · Journal of the Learning Sciences · doi:10.1207/s15327809jls1301_2 · Cited by 1,946
Assessing professional competence: from methods to programmes
Van Der Vleuten, C. P. M., & Schuwirth, L. W. T. (2005) · Medical Education · doi:10.1111/j.1365-2929.2005.02094.x · Cited by 1,268
The assessment of professional competence: Developments, research and practical implications
Van Der Vleuten, C. P. M. (1996) · Advances in Health Sciences Education · doi:10.1007/bf00596229 · Cited by 1,247
Workplace-based assessment as an educational tool: AMEE Guide No. 31
Norcini, J., & Burch, V. (2007) · Medical Teacher · doi:10.1080/01421590701775453 · Cited by 984
Carraccio, C. L., Wolfsthal, S. D., Englander, R., Ferentz, K. S., & Martin, C. (2002) · Academic Medicine · doi:10.1097/00001888-200205000-00003 · Cited by 979
Programmatic assessment: From assessment of learning to assessment for learning
Schuwirth, L. W. T., & Van der Vleuten, C. P. M. (2011) · Medical Teacher · doi:10.3109/0142159x.2011.565828 · Cited by 902
A model for programmatic assessment fit for purpose.
van der Vleuten, C. P. M., Schuwirth, L. W. T., Driessen, E. W., Dijkstra, J., Tigelaar, D., Baartman, L. K. J., & van Tartwijk, J. (2012) · Medical Teacher · doi:10.3109/0142159x.2012.652239 · Cited by 892
Peer teaching in medical education: twelve reasons to move from theory to practice
ten Cate, O., & Durning, S. J. (2007) · Medical Teacher · doi:10.1080/01421590701606799 · Cited by 729
Tools for Direct Observation and Assessment of Clinical Skills of Medical Trainees
Kogan, J. R., Holmboe, E. S., & Hauer, K. E. (2009) · JAMA · doi:10.1001/jama.2009.1365 · Cited by 613
Qualitative methods in research on healthcare quality
Pope, C. J. (2002) · BMJ Quality & Safety · doi:10.1136/qhc.11.2.148 · Cited by 602
Portfolios in medical education: why do they meet with mixed success? A systematic review
Driessen, E. W., van Tartwijk, J., van der Vleuten, C. P. M., & Wass, V. (2007) · Medical Education · doi:10.1111/j.1365-2923.2007.02944.x · Cited by 354
From principles to practice: An embedded assessment system
Wilson, M., & Sloane, K. (2000) · Applied Measurement in Education · doi:10.1207/s15324818ame1302_4 · Cited by 333
Assessment in Health Professions Education
Downing, S. M., & Yudkowsky, R. (2009) · Routledge eBooks · doi:10.4324/9780203880135 · Cited by 328
Assessment in Health Professions Education
(2019) · Routledge eBooks · doi:10.4324/9781315166902 · Cited by 296
Peer assisted learning: a planning and implementation framework: AMEE Guide no. 30
Ross, M., & Cameron, H. S. (2007) · Medical Teacher · doi:10.1080/01421590701665886 · Cited by 294
Workplace-based assessment as an educational tool
Norcini, J. J., Burch, V. C., & Gibbs, T. J. (2007) · Medical Entomology and Zoology · Cited by 266
Driessen, E. W., van der Vleuten, C. P. M., Schuwirth, L., van Tartwijk, J., & Vermunt, J. D. H. M. (2005) · Medical Education · doi:10.1111/j.1365-2929.2004.02059.x · Cited by 207
Dannefer, E. F., & Henson, L. C. (2007) · Academic Medicine · doi:10.1097/acm.0b013e31803ead30 · Cited by 181
van Merriënboer, J. J. G., & Sluijsmans, D. M. A. (2009) · Educational Psychology Review · doi:10.1007/s10648-008-9092-5 · Cited by 156
Clarifying students’ feedback‐seeking behaviour in clinical clerkships
Bok, H. G. J., Teunissen, P. W., Spruijt, A., Fokkema, J. P. I., van Beukelen, P., Jaarsma, D. A. D. C., & van der Vleuten, C. P. M. (2013) · Medical Education · doi:10.1111/medu.12054 · Cited by 155
Self-citationA new framework for designing programmes of assessment
Dijkstra, J., van der Vleuten, C. P. M., & Schuwirth, L. (2009) · Advances in Health Sciences Education · doi:10.1007/s10459-009-9205-z · Cited by 133
A closer look at cueing effects in multiple‐choice questions
Schuwirth, L., van der Vleuten, C. P. M., & Donkers, J. H. H. L. M. (1996) · Medical Education · doi:10.1111/j.1365-2923.1996.tb00716.x · Cited by 133
The process of feedback in workplace‐based assessment: organisation, delivery, continuity
Pelgrim, E. A. M., Kramer, A. W. M., Mokkink, H. G. A., & van der Vleuten, C. P. M. (2012) · Medical Education · doi:10.1111/j.1365-2923.2012.04266.x · Cited by 128
Development and Validation of a Competency Framework for Veterinarians
Bok, H. G. J., Jaarsma, D. A. D. C., Teunissen, P. W., van der Vleuten, C. P. M., & van Beukelen, P. (2011) · Journal of Veterinary Medical Education · doi:10.3138/jvme.38.3.262 · Cited by 115
Self-citationLarge‐scale Portfolio Assessments in the US: evidence pertaining to the quality of measurement
Koretz, D. M. (1998) · Assessment in Education Principles Policy and Practice · doi:10.1080/0969595980050302 · Cited by 95
When enough is enough: a conceptual basis for fair and defensible practice performance assessment
Schuwirth, L., Southgate, L. J., Page, G. G., Paget, N. S., Lescop, J., Lew, S. R., Wade, W. B., & Maldonado, M. B. (2002) · Medical Education · doi:10.1046/j.1365-2923.2002.01313.x · Cited by 95
Prescott, L. E., Norcini, J. J., McKinlay, P., & Rennie, J. S. (2002) · Medical Education · doi:10.1046/j.1365-2923.2002.01099.x · Cited by 74
Curriculum development in medical education: From acronyms to dynamism
Prideaux, D. J. (2007) · Teaching and Teacher Education · doi:10.1016/j.tate.2006.12.017 · Cited by 44
Ringsted, C., Henriksen, A. H., Skaarup, A. M., & Van der Vleuten, C. P. M. (2004) · Medical Education · doi:10.1111/j.1365-2929.2004.01841.x · Cited by 41
Martinez, M. E., & Lipson, J. I. (1989) · Educational leadership · Cited by 32
Ringsted, C. V., Østergaard, D., & Scherpbier, A. J. J. A. (2003) · Medical Teacher · doi:10.1080/0142159021000061431 · Cited by 28
Implementing feedback cards in core clerkships
Kogan, J. R., & Shea, J. A. (2008) · Medical Education · doi:10.1111/j.1365-2923.2008.03158.x · Cited by 26
Ringsted, C. V., Skaarup, A. M., Henriksen, A., & Davis, D. (2006) · Medical Teacher · doi:10.1080/01421590500237721 · Cited by 21
Research works printed but not linked: 1
- Pritchard WR: Report of the Pew National Veterinary Education Program. Future directions for veterinary medicine. 1988, Durham, NC: Duke University
Cited by
Sorted by citations
Assessment, feedback and the alchemy of learning
Watling, C., & Ginsburg, S. (2018) · Medical Education · doi:10.1111/medu.13645 · Cited by 461
Guidelines: The do’s, don’ts and don’t knows of feedback for clinical education.
Lefroy, J., Watling, C., Teunissen, P. W., & Brand, P. (2015) · Perspectives on Medical Education · doi:10.1007/s40037-015-0231-7 · Cited by 347
Self-citationTwelve Tips for programmatic assessment
Van Der Vleuten, C. P. M., Schuwirth, L. W. T., Driessen, E. W., Govaerts, M. J. B., & Heeneman, S. (2015) · Medical Teacher · doi:10.3109/0142159x.2014.973388 · Cited by 333
Rekman, J. F., Gofton, W. T., Dudek, N. L., Gofton, T., & Hamstra, S. J. (2015) · Academic Medicine · doi:10.1097/acm.0000000000001045 · Cited by 238
Educational design research: Portraying, conducting, and enhancing productive scholarship
McKenney, S. E., & Reeves, T. C. (2020) · Medical Education · doi:10.1111/medu.14280 · Cited by 328
Metadata
- Authors
- Bok, H. G., Teunissen, P. W., Favier, R. P., Rietbroek, N. J., Theyse, L. F., Brommer, H., Haarhuis, J. C., van Beukelen, P., van der Vleuten, C. P., & Jaarsma, D. A.
- Affiliations
- Utrecht University (NL)
- Maastricht University (NL)
- University of Amsterdam (NL)
- Corresponding author
- Bok, Harold GJ
- Venue
- BMC Medical Education
- Published
- 2013-09-11
- Volume
- 13
- Issue
- 1
- Pages
- 123-123
- ISSN
- 1472-6920 (ISSN-L)
- PMC
- PMC3851012
- OpenAlex
- W2153422278
- Cited by
- 303
- References
- 36
- Citations by year
- 2026: 20 · 2025: 14 · 2024: 24 · 2023: 31 · 2022: 36 · 2021: 24 · 2020: 29 · 2019: 29 · 2018: 25 · 2017: 26 · 2016: 25 · 2015: 13 · 2014: 5
- FWCI
- 14.62
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- BMC Medical Education (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Pure Amsterdam UMC (publishedVersion)
- Amsterdam UMC (VU Amsterdam) - Institutional Repository (publishedVersion)
- Data Archiving and Networked Services (DANS) (submittedVersion)
- Research Publications (Maastricht University) (submittedVersion, cc-by)
- Radboud Repository (Radboud University) (submittedVersion)
- Radboud Repository (Radboud University) (submittedVersion, other-oa)
- Research Publications (Maastricht University) (submittedVersion, other-oa)
- Funders
- Flinders University
- Topics
- Social Sciences › Education › Innovations in Medical Education
- Social Sciences › Education › Student Assessment and Feedback
- Social Sciences › Business, Management and Accounting › Organizational Behavior and Human Resource Management › Competency Development and Evaluation
- Keywords
- competency-based medical education, programmatic assessment, workplace-based assessment, formative assessment, summative assessment, feedback, peer feedback, self-directed learning, portfolio assessment, low-stakes assessment, assessment implementation, faculty development, clinical supervisors, medical students
- MeSH terms
- Competency-Based Education, Curriculum, Education, Medical, Education, Veterinary, Educational Measurement, Humans, Netherlands, Surveys and Questionnaires, Schools, Veterinary
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
