Challenges in multisource feedback: intended and unintended outcomes
Abstract
Context Multisource feedback (MSF) is a type of formative assessment intended to guide learning and performance change. However, in earlier research, some doctors questioned its validity and did not use it for improvement, raising questions about its consequential validity (i.e. its ability to produce intended outcomes related to learning and change). The purpose of this qualitative study was to increase understanding of the consequential validity of MSF by exploring how doctors used their feedback and the conditions influencing this use. Methods We used interviews with open‐ended questions. We purposefully recruited volunteer participants from 2 groups of family doctors who participated in a pilot assessment of MSF: those who received high ( n = 25) and those who received average/lower ( n = 44) scores. Results Respondents included 12 in the higher‐ and 16 in the average/lower‐scoring groups. Fifteen interpreted their feedback as positive (i.e. confirming current practice) and did not make changes. Thirteen interpreted feedback as negative in 1 or more domains (i.e. not confirming their practice and indicating need for change). Seven reported making changes. The most common changes were in patient and team communication; the least common were in clinical competence. Positive influences upon change included receiving specific feedback consistent with other sources of feedback from credible reviewers who were able to observe the subjects. These reviewers were most frequently patients. Discussion Findings suggest circumstances that may contribute to low consequential validity of MSF for doctors. Implications for practice include enhancing procedural credibility by ensuring reviewers' ability to observe respective behaviours, enhancing feedback usefulness by increasing its specificity, and considering the use of more objective measures of clinical competence.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Credible feedback is well-informed, typically by direct observation of the task or event, and comes from a trustworthy source.
“Credible feedback is well-informed, typically by direct observation of the task or event, and it comes from a trustworthy source.”
From Guidelines: The do’s, don’ts and don’t knows of feedback for clinical education. (2015) · cites this paper as [9]
Trainees must perceive feedback as credible in order for it to be influential.
“Do recognize that trainees must perceive feedback as credible in order for it to be influential.”
From Guidelines: The do’s, don’ts and don’t knows of feedback for clinical education. (2015) · cites this paper as [9]
Cites
Kluger, A. N., & DeNisi, A. (1996) · Psychological Bulletin · doi:10.1037/0033-2909.119.2.254 · Cited by 5,882 · see citing works
(2021) · Edward Elgar Publishing eBooks · doi:10.4337/9781788976954.00014 · Cited by 4,030 · see citing works
Douglas, O. B., & Holland, B. F. (2015) · MacMillan Co eBooks · doi:10.1037/14910-020 · Cited by 2,283 · see citing works
Assessing professional competence: from methods to programmes
Van Der Vleuten, C. P. M., & Schuwirth, L. W. T. (2005) · Medical Education · doi:10.1111/j.1365-2929.2005.02094.x · Cited by 1,268 · see citing works
The assessment of professional competence: Developments, research and practical implications
Van Der Vleuten, C. P. M. (1996) · Advances in Health Sciences Education · doi:10.1007/bf00596229 · Cited by 1,247 · see citing works
Enhancing Learning Through Self-assessment
Boud, D. J. (2013) · Routledge eBooks · doi:10.4324/9781315041520 · Cited by 914 · see citing works
Enhancing learning through self-assessment
Vella, F. (1996) · Biochemical Education · doi:10.1016/0307-4412(96)82523-4 · Cited by 671 · see citing works
Understanding the influence of emotions and reflection upon multi-source feedback acceptance and use
Sargeant, J., Mann, K., Sinclair, D., Van der Vleuten, C., & Metsemakers, J. (2008) · Advances in Health Sciences Education · doi:10.1007/s10459-006-9039-x · Cited by 274 · see citing works
Self-citationAshworth, P. D. (2000) · Estudios pedagógicos · doi:10.4067/s0718-07052000000100007 · Cited by 204 · see citing works
Assessment of physician performance in Alberta: the physician achievement review.
Hall, W. C., Violato, C., Lewkonia, R. M., Lockyer, J., Fidler, H., Toews, J., Jennett, P., Donoff, M. G., & Moores, D. (1999) · Europe PMC (PubMed Central) · Cited by 191 · see citing works
Sargeant, J., Mann, K., & Ferrier, S. (2005) · Medical Education · doi:10.1111/j.1365-2929.2005.02124.x · Cited by 174 · see citing works
Self-citationMultisource feedback: a method of assessing surgical practice
Violato, C., Lockyer, J., & Fidler, H. (2003) · BMJ · doi:10.1136/bmj.326.7388.546 · Cited by 162 · see citing works
Rodgers, K. G., & Manifold, C. A. (2002) · Academic Emergency Medicine · doi:10.1197/aemj.9.11.1300 · Cited by 152 · see citing works
Use of SPRAT for peer review of paediatricians in training
Archer, J. C., Norcini, J. J., & Davies, H. A. (2005) · BMJ · doi:10.1136/bmj.38447.610451.8f · Cited by 140 · see citing works
The Value of Patient and Peer Ratings in Recertification
Lipner, R. S., Blank, L. L., Leas, B. F., & Fortna, G. S. (2002) · Academic Medicine · doi:10.1097/00001888-200210001-00021 · Cited by 132 · see citing works
Effects of source status and message favorability on acceptance of personality feedback.
Halperin, K. M., Snyder, C. R. R., Shenkel, R. J., & Houston, B. K. (1976) · Journal of Applied Psychology · doi:10.1037/0021-9010.61.1.85 · Cited by 125 · see citing works
Review of instruments for peer assessment of physicians
Evans, R., Elwyn, G. J., & Edwards, A. (2004) · BMJ · doi:10.1136/bmj.328.7450.1240 · Cited by 125 · see citing works
Changing physiciansʼ practices
Fidler, H., Lockyer, J., Toews, J., & Violato, C. (1999) · Academic Medicine · doi:10.1097/00001888-199906000-00019 · Cited by 125 · see citing works
Gielen, S., Dochy, F. J. R. C., & Dierick, S. (2006) · Kluwer Academic Publishers eBooks · doi:10.1007/0-306-48125-1_3 · Cited by 120 · see citing works
The framing of relative performance feedback: Seeing the glass as half empty or half full.
McFarland, C., & Miller, D. T. (1994) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.66.6.1061 · Cited by 77 · see citing works
Sargeant, J. M., Mann, K. V., Ferrier, S. N., Langille, D. B., Muirhead, P. D., Hayes, V. M., & Sinclair, D. E. (2003) · Academic Medicine · doi:10.1097/00001888-200310001-00014 · Cited by 68 · see citing works
Self-citationBeneficial and harmful effects of augmented feedback on physiciansʼ clinical-teaching performances
Litzelman, D. K., Stratos, G. A., Marriott, D. J., et al. (1998) · Academic Medicine · doi:10.1097/00001888-199803000-00022 · Cited by 67 · see citing works
Stone, D. L., & Stone, E. F. (1985) · Organizational Behavior and Human Decision Processes · doi:10.1016/0749-5978(85)90011-1 · Cited by 67 · see citing works
Effect of self-concept on group member reception of positive and negative feedback.
Morran, D. K., & Stockton, R. A. (1980) · Journal of Counseling Psychology · doi:10.1037/0022-0167.27.3.260 · Cited by 47 · see citing works
Karl, K. A. (1998) · Academy of Management Perspectives · doi:10.5465/ame.1998.1334026 · Cited by 47 · see citing works
Maximizing the Value of 360‐degree Feedback
(2001) · International Journal of Manpower · doi:10.1108/ijm.2001.22.7.664.4 · Cited by 46 · see citing works
Kernis, M. H., & Johnson, E. K. (1990) · Journal of Research in Personality · doi:10.1016/0092-6566(90)90019-3 · Cited by 36 · see citing works
Optimising new modes of assessment: in search of qualities and standards
Schuwirth, L. (2004) · Tijdschrift voor Medisch Onderwijs · doi:10.1007/bf03056663 · Cited by 21 · see citing works
EDUCATION: Mandatory practice self‐appraisal: moving towards outcomes based continuing education
Donen, N. (1999) · Journal of Evaluation in Clinical Practice · doi:10.1046/j.1365-2753.1999.00187.x · Cited by 11 · see citing works
Clinical audit--linking continuing medical education (CME) and practice assessment (PA).
Piterman, L., & Nelson, M. (1997) · PubMed · Cited by 6 · see citing works
Ciok, A. E. (2002) · Academic Medicine · doi:10.1097/00001888-200206000-00012 · Cited by 1 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Guidelines: The do’s, don’ts and don’t knows of feedback for clinical education.
Lefroy, J., Watling, C., Teunissen, P. W., & Brand, P. (2015) · Perspectives on Medical Education · doi:10.1007/s40037-015-0231-7 · Cited by 347 · see citing works
The processes and dimensions of informed self-assessment: A conceptual model.
Sargeant, J., Armson, H., Chesluk, B., Dornan, T., Eva, K., Holmboe, E., Lockyer, J., Loney, E., Mann, K., & van der Vleuten, C. (2010) · Academic Medicine · doi:10.1097/acm.0b013e3181d85a4e · Cited by 318 · see citing works
Self-citationSargeant, J., Lockyer, J., Mann, K., Holmboe, E., Silver, I., Armson, H., Driessen, E., MacLeod, T., Yen, W., Ross, K., et al. (2015) · Academic Medicine · doi:10.1097/acm.0000000000000809 · Cited by 306 · see citing works
Self-citationReflection: a link between receiving and using assessment feedback
Sargeant, J. M., Mann, K. V., van der Vleuten, C. P., & Metsemakers, J. F. (2009) · Advances in Health Sciences Education · doi:10.1007/s10459-008-9124-4 · Cited by 296 · see citing works
Self-citationLearning from clinical work: the roles of learning cues and credibility judgements
Watling, C., Driessen, E., van der Vleuten, C. P. M., & Lingard, L. (2012) · Medical Education · doi:10.1111/j.1365-2923.2011.04126.x · Cited by 261 · see citing works
“Directed” self-assessment: Practice and feedback within a social context
Sargeant, J., Mann, K., van der Vleuten, C., & Metsemakers, J. (2008) · Journal of Continuing Education in the Health Professions · doi:10.1002/chp.155 · Cited by 158 · see citing works
Self-citationPeer feedback as an aid to learning – What do we want? Feedback. When do we want it? Now!
Cushing, A., Abbott, S., Lothian, D., Hall, A., & Westwood, O. M. R. (2011) · Medical Teacher · doi:10.3109/0142159x.2011.542522 · Cited by 121 · see citing works
- Authors
- Sargeant, J., Mann, K., Sinclair, D., Van der Vleuten, C., & Metsemakers, J.
- Affiliations
- Dalhousie University (CA)
- Maastricht University (NL)
- Corresponding author
- Sargeant, Joan
- ORCID
- Sargeant, Joan
- Venue
- Medical Education
- Publisher
- Wiley
- Published
- 2007-05-18
- Volume
- 41
- Issue
- 6
- Pages
- 583-591
- ISSN
- 0308-0110 (ISSN-L)
- 1365-2923
- OpenAlex
- W1965011604
- Cited by
- 151
- References
- 31
- Citations by year
- 2026: 3 · 2025: 5 · 2024: 2 · 2023: 3 · 2022: 9 · 2021: 9 · 2020: 6 · 2019: 10 · 2018: 14 · 2017: 10 · 2016: 9 · 2015: 9 · 2014: 11 · 2013: 11 · 2012: 5
- FWCI
- 8.27
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Medical Education (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Innovations in Medical Education
- Social Sciences › Education › Student Assessment and Feedback
- Health Sciences › Medicine › Family Practice › Clinical Reasoning and Diagnostic Skills
- Keywords
- multisource feedback, consequential validity, formative assessment, general practitioners, physician performance feedback, feedback interpretation, clinical competence, reviewer credibility, observational assessment, feedback specificity
- MeSH terms
- Alberta, Clinical Competence, Education, Medical, Continuing, Family Practice, Female, Humans, Male, Nova Scotia, Physicians, Family, Pilot Projects, Surveys and Questionnaires, Feedback, Psychological
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
