Validity: on the meaningful interpretation of assessment data

Downing, Steven M

doi:10.1046/j.1365-2923.2003.01594.x

Cited by 1,434Innovations in Medical EducationPsychometric Methodologies and TestingHealth Education and Validation

Validity: on the meaningful interpretation of assessment data

Downing, Steven M·2003·Medical Education

DOI 10.1046/j.1365-2923.2003.01594.x

Open this paper in CoChat — free account

Abstract

Context All assessments in medical education require evidence of validity to be interpreted meaningfully. In contemporary usage, all validity is construct validity, which requires multiple sources of evidence; construct validity is the whole of validity, but has multiple facets. Five sources – content, response process, internal structure, relationship to other variables and consequences – are noted by the Standards for Educational and Psychological Testing as fruitful areas to seek validity evidence. Purpose The purpose of this article is to discuss construct validity in the context of medical education and to summarize, through example, some typical sources of validity evidence for a written and a performance examination. Summary Assessments are not valid or invalid; rather, the scores or outcomes of assessments have more or less evidence to support (or refute) a specific interpretation (such as passing or failing a course). Validity is approached as hypothesis and uses theory, logic and the scientific method to collect and assemble data to support or fail to support the proposed score interpretations, at a given point in time. Data and logic are assembled into arguments – pro and con – for some specific interpretation of assessment data. Examples of types of validity evidence, data and information from each source are discussed in the context of a high‐stakes written and performance examination in medical education. Conclusion All assessments require evidence of the reasonableness of the proposed interpretation, as test data in education have little or no intrinsic meaning. The constructs purported to be measured by our assessments are important to students, faculty, administrators, patients and society and require solid scientific evidence of their meaning.

Integrity signals

Registry

Crossref · 10.1046/j.1365-2923.2003.01594.x

Cites

32 of 32 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

The currently accepted understanding of validity places an emphasis on construct validity as the 'whole' of the validity argument.
“The currently accepted understanding of validity places an emphasis on construct validity as the ''whole'' of the validity argument (Downing 2003).”
From Measuring physician cognitive load: validity evidence for a physiologic and a psychometric tool (2017) · cites this paper as Downing 2003

Convergent and discriminant validation by the multitrait-multimethod matrix.
Campbell, D. T., & Fiske, D. W. (1959) · Psychological Bulletin · doi:10.1037/h0046016 · Cited by 17,116 · see citing works
Construct validity in psychological tests.
Cronbach, L. J., & Meehl, P. E. (1955) · Psychological Bulletin · doi:10.1037/h0040957 · Cited by 11,195 · see citing works
Validity of psychological assessment: Validation of inferences from persons' responses and performances as scientific inquiry into score meaning.
Messick, S. (1995) · American Psychologist · doi:10.1037/0003-066x.50.9.741 · Cited by 4,198 · see citing works
Educational measurement, 3rd ed.
Linn, R. L. (1989) · Cited by 2,858 · see citing works
Handbook of Modern Item Response Theory
van der Linden, W. J., & Hambleton, R. K. (1997) · Springer eBooks · doi:10.1007/978-1-4757-2691-6 · Cited by 2,331 · see citing works
Educational measurement.
Douglas, O. B., & Holland, B. F. (2015) · MacMillan Co eBooks · doi:10.1037/14910-020 · Cited by 2,283 · see citing works
Differential Item Functioning.
Gudgeon, A. C., Holland, P. W., & Wainer, H. (1994) · Journal of the Royal Statistical Society Series D (The Statistician) · doi:10.2307/2348590 · Cited by 2,103 · see citing works
Item Response Theory
Hambleton, R. K., & Swaminathan, H. (1985) · doi:10.1007/978-94-017-1988-9 · Cited by 1,777 · see citing works
Objective Tests as Instruments of Psychological Theory
Loevinger, J. (1957) · Psychological Reports · doi:10.2466/pr0.1957.3.3.635 · Cited by 1,476 · see citing works
Generalizability Theory
Brennan, R. L. (2001) · doi:10.1007/978-1-4757-3456-0 · Cited by 1,121 · see citing works
A Review of Multiple-Choice Item-Writing Guidelines for Classroom Assessment
Haladyna, T. M., Downing, S. M., & Rodriguez, M. C. (2002) · Applied Measurement in Education · doi:10.1207/s15324818ame1503_5 · Cited by 973 · see citing works
Self-citation
Review of Educational Measurement.
Langmuir, C. R. (1952) · Journal of Educational Psychology · doi:10.1037/h0051392 · Cited by 906 · see citing works
Current Concerns in Validity Theory
Kane, M. T. (2001) · Journal of Educational Measurement · doi:10.1111/j.1745-3984.2001.tb01130.x · Cited by 753 · see citing works
Standards of validity and the validity of standards in performance asessment
Messick, S. (1995) · Educational Measurement Issues and Practice · doi:10.1111/j.1745-3992.1995.tb00881.x · Cited by 697 · see citing works
Argument-Based Approach to Validity
Kane, M. T. (2022) · The SAGE Encyclopedia of Research Design · doi:10.4135/9781071812082.n28 · Cited by 557 · see citing works
The effect of assessments and examinations on the learning of medical students
Newble, D. I., & Jaeger, K. (1983) · Medical Education · doi:10.1111/j.1365-2923.1983.tb00657.x · Cited by 488 · see citing works
The Argument-Based Approach to Validation
Kane, M. T. (2013) · School Psychology Review · doi:10.1080/02796015.2013.12087465 · Cited by 361 · see citing works
Setting standards on educational tests
Norcini, J. J. (2003) · Medical Education · doi:10.1046/j.1365-2923.2003.01495.x · Cited by 315 · see citing works
Validating Measures of Performance
Kane, M. T., Crooks, T. J., & Cohen, A. S. (1999) · Educational Measurement Issues and Practice · doi:10.1111/j.1745-3992.1999.tb00010.x · Cited by 298 · see citing works
THE PSYCHOLOGY OF EDUCATIONAL MEASUREMENT
Messick, S. (1984) · Journal of Educational Measurement · doi:10.1111/j.1745-3984.1984.tb01030.x · Cited by 243 · see citing works
Assessing health professionals
Crossley, J., Humphris, G. M., & Jolly, B. C. (2002) · Medical Education · doi:10.1046/j.1365-2923.2002.01294.x · Cited by 209 · see citing works
Item Response Theory: Brief History, Common Models, and Extensions
van der Linden, W. J., & Hambleton, R. K. (1997) · Springer eBooks · doi:10.1007/978-1-4757-2691-6_1 · Cited by 199 · see citing works
Standards for Educational and Psychological Testing
(2008) · Encyclopedia of Special Education · doi:10.1002/9780470373699.speced1992 · Cited by 199 · see citing works
Simulation in clinical learning
Bradley, P., & Postlethwaite, K. (2003) · Medical Education · doi:10.1046/j.1365-2923.37.s1.1.x · Cited by 169 · see citing works
Generalisability: a key to unlock professional assessment
Crossley, J., Davies, H., Humphris, G., & Jolly, B. (2002) · Medical Education · doi:10.1046/j.1365-2923.2002.01320.x · Cited by 169 · see citing works
Item response theory: applications of modern test theory in medical education
Downing, S. M. (2003) · Medical Education · doi:10.1046/j.1365-2923.2003.01587.x · Cited by 148 · see citing works
Self-citation
Assessing Differential Item Functioning in Performance Assessment: Review and Recommendations
Penfield, R. D., & Lam, T. C. M. (2000) · Educational Measurement Issues and Practice · doi:10.1111/j.1745-3992.2000.tb00033.x · Cited by 138 · see citing works
A Practitioner's Guide to Computation and Interpretation of Reliability Indices for Mastery Tests
Subkoviak, M. J. (1988) · Journal of Educational Measurement · doi:10.1111/j.1745-3984.1988.tb00290.x · Cited by 109 · see citing works
Validating Interpretive Arguments for Licensure and Certification Examinations
Kane, M. T. (1994) · Evaluation & the Health Professions · doi:10.1177/016327879401700202 · Cited by 81 · see citing works
Teaching and Testing to Develop Fluid Abilities
Lohman, D. F. (1993) · Educational Researcher · doi:10.3102/0013189x022007012 · Cited by 81 · see citing works
Quality Assurance Methods for Performance-Based Assessments
Boulet, J. R., McKinley, D. W., Whelan, G. P., & Hambleton, R. K. (2003) · Advances in Health Sciences Education · doi:10.1023/a:1022639521218 · Cited by 74 · see citing works
An argument-based approach to validity.
Kane, M. T. (1992) · Psychological Bulletin · doi:10.1037//0033-2909.112.3.527 · Cited by 66 · see citing works

Flagged first, then by citations

1 of the 33 references couldn’t be matched to a known paper.

Sorted by citations

Current Concepts in Validity and Reliability for Psychometric Instruments: Theory and Application
Cook, D. A., & Beckman, T. J. (2006) · The American Journal of Medicine · doi:10.1016/j.amjmed.2005.10.036 · Cited by 1,745 · see citing works
A critical review of simulation‐based medical education research: 2003–2009
McGaghie, W. C., Issenberg, S. B., Petrusa, E. R., & Scalese, R. J. (2010) · Medical Education · doi:10.1111/j.1365-2923.2009.03547.x · Cited by 1,733 · see citing works
Assessing professional competence: from methods to programmes
Van Der Vleuten, C. P. M., & Schuwirth, L. W. T. (2005) · Medical Education · doi:10.1111/j.1365-2929.2005.02094.x · Cited by 1,268 · see citing works
Association Between Funding and Quality of Published Medical Education Research
Reed, D. A., Cook, D. A., Beckman, T. J., Levine, R. B., Kern, D. E., & Wright, S. M. (2007) · JAMA · doi:10.1001/jama.298.9.1002 · Cited by 961 · see citing works
Patient-reported outcomes: A new era in clinical research
Deshpande, P., Rajan, S., Sudeepthi, B., & Abdul Nazir, C. (2011) · Perspectives in Clinical Research · doi:10.4103/2229-3485.86879 · Cited by 811 · see citing works
Reliability: on the reproducibility of assessment data
Downing, S. M. (2004) · Medical Education · doi:10.1111/j.1365-2929.2004.01932.x · Cited by 721 · see citing works
Self-citation
Reliability, Internal Consistency, and Validity of Data Obtained With the Functional Gait Assessment
Wrisley, D. M., Marchetti, G. F., Kuharsky, D. K., & Whitney, S. L. (2004) · Physical Therapy · doi:10.1093/ptj/84.10.906 · Cited by 650 · see citing works
A contemporary approach to validity arguments: a practical guide to K ane's framework
Cook, D. A., Brydges, R., Ginsburg, S., & Hatala, R. A. (2015) · Medical Education · doi:10.1111/medu.12678 · Cited by 644 · see citing works
Tools for Direct Observation and Assessment of Clinical Skills of Medical Trainees
Kogan, J. R., Holmboe, E. S., & Hauer, K. E. (2009) · JAMA · doi:10.1001/jama.2009.1365 · Cited by 602 · see citing works
The Objective Structured Clinical Examination (OSCE): AMEE Guide No. 81. Part I: An historical and theoretical perspective
Khan, K. Z., Ramachandran, S., Gaunt, K., & Pushkar, P. (2013) · Medical Teacher · doi:10.3109/0142159x.2013.818634 · Cited by 490 · see citing works
Entrustment Decision Making in Clinical Training
ten Cate, O., Hart, D., Ankel, F., Busari, J., Englander, R., Glasgow, N., Holmboe, E., Iobst, W., Lovell, E., Snell, L. S., et al. (2016) · Academic Medicine · doi:10.1097/acm.0000000000001044 · Cited by 478 · see citing works
A Primer on the Validity of Assessment Instruments
Sullivan, G. M. (2011) · Journal of Graduate Medical Education · doi:10.4300/jgme-d-11-00075.1 · Cited by 446 · see citing works
Instruments for Evaluating Education in Evidence-Based Practice
Shaneyfelt, T. M., Baum, K. D., Bell, D. S., Feldstein, D., Houston, T. K., Kaatz, S. S., Whelan, C. T., & Green, M. (2006) · JAMA · doi:10.1001/jama.296.9.1116 · Cited by 422 · see citing works
Resuscitation Education Science: Educational Strategies to Improve Outcomes From Cardiac Arrest: A Scientific Statement From the American Heart Association
Cheng, A., Nadkarni, V., Mancini, M. B., Hunt, E. A., Sinz, E. H., Merchant, R. M., Donoghue, A. J., Duff, J. P., Eppich, W., Auerbach, M. A., et al. (2018) · Circulation · doi:10.1161/cir.0000000000000583 · Cited by 430 · see citing works
Anatomical dissection as a teaching method in medical school: a review of the evidence
Winkelmann, A. (2006) · Medical Education · doi:10.1111/j.1365-2929.2006.02625.x · Cited by 412 · see citing works
Patient Outcomes in Simulation-Based Medical Education: A Systematic Review
Zendejas, B., Brydges, R., Wang, A. T., & Cook, D. A. (2013) · Journal of General Internal Medicine · doi:10.1007/s11606-012-2264-5 · Cited by 384 · see citing works
Validation of educational assessments: a primer for simulation and beyond
Cook, D. A., & Hatala, R. A. (2016) · Advances in Simulation · doi:10.1186/s41077-016-0033-y · Cited by 382 · see citing works
Learn, See, Practice, Prove, Do, Maintain
Sawyer, T., White, M. L., Zaveri, P. P., Chang, T. P., Ades, A. M., French, H. M., Anderson, J. M., Auerbach, M. A., Johnston, L., & Kessler, D. O. (2015) · Academic Medicine · doi:10.1097/acm.0000000000000734 · Cited by 384 · see citing works
Evaluating the impact of simulation on translational patient outcomes
McGaghie, W. C., Draycott, T. J., Dunn, W. F., Lopez, C. M., & Stefanidis, D. (2011) · Simulation in Healthcare The Journal of the Society for Simulation in Healthcare · doi:10.1097/sih.0b013e318222fde9 · Cited by 336 · see citing works
A pilot study using high-fidelity simulation to formally evaluate performance in the resuscitation of critically ill patients: The University of Ottawa Critical Care Medicine, High-Fidelity Simulation, and Crisis Resource Management I Study
Kim, J., Neilipovitz, D., Cardinal, P., Chiu, M., & Clinch, J. (2006) · Critical Care Medicine · doi:10.1097/01.ccm.0000229877.45125.cc · Cited by 323 · see citing works
What counts as validity evidence? Examples and prevalence in a systematic review of simulation-based assessment
Cook, D. A., Zendejas, B., Hamstra, S. J., Hatala, R. A., & Brydges, R. (2013) · Advances in Health Sciences Education · doi:10.1007/s10459-013-9458-4 · Cited by 318 · see citing works
Predictive Validity Evidence for Medical Education Research Study Quality Instrument Scores: Quality of Submissions to JGIM’s Medical Education Special Issue
Reed, D. A., Beckman, T. J., Wright, S. M., Levine, R. B., Kern, D. E., & Cook, D. A. (2008) · Journal of General Internal Medicine · doi:10.1007/s11606-008-0664-3 · Cited by 284 · see citing works
Validity threats: overcoming interference with proposed interpretations of assessment data
Downing, S. M., & Haladyna, T. M. (2004) · Medical Education · doi:10.1046/j.1365-2923.2004.01777.x · Cited by 284 · see citing works
Self-citation
Cross-comparison of MRCGP & MRCP(UK) in a database linkage study of 2,284 candidates taking both examinations: assessment of validity and differential performance by ethnicity
Wakeford, R. E., Denney, M., Ludka-Stempien, K., Dacre, J. E., & McManus, I. C. (2015) · BMC Medical Education · doi:10.1186/s12909-014-0281-2 · Cited by 282 · see citing works
Technology-Enhanced Simulation to Assess Health Professionals
Cook, D. A., Brydges, R., Zendejas, B., Hamstra, S. J., & Hatala, R. A. (2013) · Academic Medicine · doi:10.1097/acm.0b013e31828ffdcf · Cited by 269 · see citing works
Linking Simulation-Based Educational Assessments and Patient-Related Outcomes
Brydges, R., Hatala, R. A., Zendejas, B., Erwin, P. J., & Cook, D. A. (2014) · Academic Medicine · doi:10.1097/acm.0000000000000549 · Cited by 253 · see citing works
The Movement Assessment Battery for Children—Second Edition (MABC-2): A Review and Critique
Brown, T., & LALOR, A. F. (2009) · Physical & Occupational Therapy In Pediatrics · doi:10.1080/01942630802574908 · Cited by 247 · see citing works
In BMI we trust: reframing the body mass index as a measure of health
Gutin, I. (2017) · Social Theory & Health · doi:10.1057/s41285-017-0055-0 · Cited by 229 · see citing works
Learning Curves in Health Professions Education
Pusic, M. V., Boutis, K., Hatala, R. A., & Cook, D. A. (2015) · Academic Medicine · doi:10.1097/acm.0000000000000681 · Cited by 205 · see citing works
Are United States Medical Licensing Exam Step 1 and 2 Scores Valid Measures for Postgraduate Medical Residency Selection Decisions?
McGaghie, W. C., Cohen, E. R., & Wayne, D. B. (2010) · Academic Medicine · doi:10.1097/acm.0b013e3181ffacdb · Cited by 201 · see citing works
What is the validity evidence for assessments of clinical teaching?
Beckman, T. J., Cook, D. A., & Mandrekar, J. N. (2005) · Journal of General Internal Medicine · doi:10.1111/j.1525-1497.2005.0258.x · Cited by 196 · see citing works
Educational Milestone Development in the First 7 Specialties to Enter the Next Accreditation System
Swing, S. R., Beeson, M. S., Carraccio, C. L., Coburn, M. C., Iobst, W. F., Selden, N. R. W., Stern, P. J., & Vydareny, K. H. (2013) · Journal of Graduate Medical Education · doi:10.4300/jgme-05-01-33 · Cited by 193 · see citing works
Assessing Trainees and Making Entrustment Decisions: On the Nature and Use of Entrustment-Supervision Scales
ten Cate, O., Schwartz, A., & Chen, H. C. (2020) · Academic Medicine · doi:10.1097/acm.0000000000003427 · Cited by 186 · see citing works
Towards a program of assessment for health professionals: from training into practice
Eva, K. W., Bordage, G., Campbell, C. B., Galbraith, R. M., Ginsburg, S., Holmboe, E. S., & Regehr, G. (2015) · Advances in Health Sciences Education · doi:10.1007/s10459-015-9653-6 · Cited by 181 · see citing works
Using clinical simulation to study how to improve quality and safety in healthcare
Lamé, G., & Dixon‐Woods, M. (2018) · BMJ Simulation & Technology Enhanced Learning · doi:10.1136/bmjstel-2018-000370 · Cited by 180 · see citing works
Simulation-based Assessment in Anesthesiology
Boulet, J. R., Murray, D. J., & Warner, D. S. (2010) · Anesthesiology · doi:10.1097/aln.0b013e3181cea265 · Cited by 178 · see citing works
Performance in assessment: Consensus statement and recommendations from the Ottawa conference
Boursicot, K. A. M., Etheridge, L., Setna, Z., Sturrock, A. M., Ker, J., Smee, S. M., & Elango, S. (2011) · Medical Teacher · doi:10.3109/0142159x.2011.565831 · Cited by 174 · see citing works
Script concordance testing: a review of published validity evidence
Lubarsky, S., Charlin, B., Cook, D. A., Chalk, C., & van der Vleuten, C. P. M. (2011) · Medical Education · doi:10.1111/j.1365-2923.2010.03863.x · Cited by 171 · see citing works
Twelve tips for evaluating educational programs
Cook, D. A. (2010) · Medical Teacher · doi:10.3109/01421590903480121 · Cited by 167 · see citing works
The Lesbian, Gay, Bisexual, and Transgender Development of Clinical Skills Scale (LGBT-DOCSS): Establishing a New Interdisciplinary Self-Assessment for Health Providers
Bidell, M. P. (2017) · Journal of Homosexuality · doi:10.1080/00918369.2017.1321389 · Cited by 167 · see citing works
Direct Observation in Medical Education: A Review of the Literature and Evidence for Validity
Fromme, H. B., Karani, R., & Downing, S. M. (2009) · Mount Sinai Journal of Medicine A Journal of Translational and Personalized Medicine · doi:10.1002/msj.20123 · Cited by 159 · see citing works
Self-citation
Technical Skills Assessment Toolbox
Ghaderi, I., Manji, F., Park, Y. S., Juul, D., Ott, M. C., Harris, I., & Farrell, T. M. (2014) · Annals of Surgery · doi:10.1097/sla.0000000000000520 · Cited by 157 · see citing works
A Comparison of Global Rating Scale and Checklist Scores in the Validation of an Evaluation Tool to Assess Performance in the Resuscitation of Critically Ill Patients During Simulated Emergencies (Abbreviated as “CRM Simulator Study IB”)
Kim, J., Neilipovitz, D., Cardinal, P., & Chiu, M. (2009) · Simulation in Healthcare The Journal of the Society for Simulation in Healthcare · doi:10.1097/sih.0b013e3181880472 · Cited by 156 · see citing works
How reliable are assessments of clinical teaching?
Beckman, T. J., Ghosh, A. K., Cook, D. A., Erwin, P. J., & Mandrekar, J. N. (2004) · Journal of General Internal Medicine · doi:10.1111/j.1525-1497.2004.40066.x · Cited by 156 · see citing works
Assessing the Reliability, Validity, and Use of the Lasater Clinical Judgment Rubric: Three Approaches
Adamson, K. A., Gubrud-Howe, P., Sideras, S. A., & Lasater, K. (2011) · Journal of Nursing Education · doi:10.3928/01484834-20111130-03 · Cited by 155 · see citing works
Assessing the Quality of Clinical Teachers
Fluit, C. R. M. G., Bolhuis, S. M., Grol, R. P. T. M., Laan, R., & Wensing, M. (2010) · Journal of General Internal Medicine · doi:10.1007/s11606-010-1458-y · Cited by 151 · see citing works
The Pediatrics Milestones: Conceptual Framework, Guiding Principles, and Approach to Development
Hicks, P. J., Schumacher, D. J., Benson, B. J., Burke, A. E., Englander, R., Guralnick, S., Ludwig, S., & Carraccio, C. L. (2010) · Journal of Graduate Medical Education · doi:10.4300/jgme-d-10-00126.1 · Cited by 149 · see citing works
Consequences Validity Evidence: Evaluating the Impact of Educational Assessments
Cook, D. A., & Lineberry, M. C. (2016) · Academic Medicine · doi:10.1097/acm.0000000000001114 · Cited by 143 · see citing works
Guidelines for Online Assessment in Emergency Remote Teaching during the COVID-19 Pandemic
Rahim, A. F. A. (2020) · Education in Medicine Journal · doi:10.21315/eimj2020.12.2.6 · Cited by 141 · see citing works
Improvements in Teamwork During Neonatal Resuscitation After Interprofessional TeamSTEPPS Training
Sawyer, T., Laubach, V., Hudak, J. J., Yamamura, K., & Pocrnich, A. (2013) · Neonatal Network The Journal of Neonatal Nursing · doi:10.1891/0730-0832.32.1.26 · Cited by 140 · see citing works
How to construct and implement script concordance tests: insights from a systematic review
Dory, V., Gagnon, R. J., Vanpee, D., & Charlin, B. (2012) · Medical Education · doi:10.1111/j.1365-2923.2011.04211.x · Cited by 139 · see citing works
Systematic review of instruments for measuring nurses' knowledge, skills and attitudes for evidence‐based practice
Leung, K., Trevena, L. J., & Waters, D. (2014) · Journal of Advanced Nursing · doi:10.1111/jan.12454 · Cited by 137 · see citing works
Measuring cognitive load: performance, mental effort and simulation task complexity
Haji, F. A., Rojas, D., Childs, R. A., de Ribaupierre, S., & Dubrowski, A. (2015) · Medical Education · doi:10.1111/medu.12773 · Cited by 136 · see citing works
Should essays and other “open-ended”-type questions retain a place in written summative assessment in clinical medicine?
Hift, R. (2014) · BMC Medical Education · doi:10.1186/s12909-014-0249-2 · Cited by 136 · see citing works
The Effect of Technical Performance on Patient Outcomes in Surgery
Fecső, A. B., Szász, P., Kerezov, G., & Grantcharov, T. (2016) · Annals of Surgery · doi:10.1097/sla.0000000000001959 · Cited by 130 · see citing works
Surgical simulation: Current practices and future perspectives for technical skills training
Bjerrum, F., Thomsen, A. S. S., Nayahangan, L. J., & Konge, L. (2018) · Medical Teacher · doi:10.1080/0142159x.2018.1472754 · Cited by 130 · see citing works
Simulator training for endobronchial ultrasound: a randomised controlled trial
Konge, L., Clementsen, P. F., Ringsted, C. V., Minddal, V., Larsen, K. R., & Annema, J. T. (2015) · European Respiratory Journal · doi:10.1183/13993003.02352-2015 · Cited by 128 · see citing works
The development and evaluation of a Professional Self Identity Questionnaire to measure evolving professional self-identity in health and social care students
Crossley, J., & Vivekananda‐Schmidt, P. (2009) · Medical Teacher · doi:10.3109/01421590903193547 · Cited by 126 · see citing works
Validity evidence for the Fundamentals of Laparoscopic Surgery (FLS) program as an assessment tool: a systematic review
Zendejas, B., Ruparel, R. K., & Cook, D. A. (2015) · Surgical Endoscopy · doi:10.1007/s00464-015-4233-7 · Cited by 126 · see citing works
The Effectiveness of Video-Assisted Debriefing Versus Oral Debriefing Alone at Improving Neonatal Resuscitation Performance
Sawyer, T., Sierocka-Castaneda, A., Chan, D. S., Berg, B. W., Lustik, M., & Thompson, M. W. (2012) · Simulation in Healthcare The Journal of the Society for Simulation in Healthcare · doi:10.1097/sih.0b013e3182578eae · Cited by 124 · see citing works
Best practices in summative assessment
Kibble, J. D. (2017) · AJP Advances in Physiology Education · doi:10.1152/advan.00116.2016 · Cited by 136 · see citing works
Factors affecting the utility of the multiple mini-interview in selecting candidates for graduate-entry medical school
Roberts, C., Walton, M., Rothnie, I., Crossley, J., Lyon, P., Kumar, K., & Tiller, D. (2008) · Medical Education · doi:10.1111/j.1365-2923.2008.03018.x · Cited by 121 · see citing works
Assessment of learning with multiple-choice questions
Brady, A. (2005) · Nurse Education in Practice · doi:10.1016/j.nepr.2004.12.005 · Cited by 123 · see citing works
The Top-Cited Articles in Medical Education
Azer, S. A. (2015) · Academic Medicine · doi:10.1097/acm.0000000000000780 · Cited by 126 · see citing works
Twelve tips for implementing tools for direct observation of medical trainees’ clinical skills during patient encounters
Hauer, K. E., Holmboe, E. S., & Kogan, J. R. (2011) · Medical Teacher · doi:10.3109/0142159x.2010.507710 · Cited by 118 · see citing works
The Mayo Colonoscopy Skills Assessment Tool: validation of a unique instrument to assess colonoscopy skills in trainees
Sedlack, R. E. (2010) · Gastrointestinal Endoscopy · doi:10.1016/j.gie.2010.09.001 · Cited by 117 · see citing works
Deliberate Practice Using Simulation Improves Neonatal Resuscitation Performance
Sawyer, T., Sierocka-Castaneda, A., Chan, D. S., Berg, B. W., Lustik, M., & Thompson, M. W. (2011) · Simulation in Healthcare The Journal of the Society for Simulation in Healthcare · doi:10.1097/sih.0b013e31822b1307 · Cited by 116 · see citing works
Epidural Injections for Spinal Pain
Bicket, M. C., Gupta, A., Brown, C. H., & Cohen, S. P. (2013) · Anesthesiology · doi:10.1097/aln.0b013e31829c2ddd · Cited by 116 · see citing works
Lung ultrasound training: a systematic review of published literature in clinical lung ultrasound training
Pietersen, P. I., Madsen, K. R., Graumann, O., Konge, L., Nielsen, B. U., & Laursen, C. B. (2018) · The Ultrasound Journal · doi:10.1186/s13089-018-0103-6 · Cited by 115 · see citing works
The Hispanic Stress Inventory Version 2: Improving the assessment of acculturation stress.
Cervantes, R. C., Fisher, D. G., Padilla, A. M., & Napper, L. E. (2015) · Psychological Assessment · doi:10.1037/pas0000200 · Cited by 115 · see citing works
A hypothesis‐driven physical examination learning and assessment procedure for medical students: initial validity evidence
Yudkowsky, R., Otaki, J., Lowenstein, T., Riddle, J. M., Nishigori, H., & Bordage, G. (2009) · Medical Education · doi:10.1111/j.1365-2923.2009.03379.x · Cited by 112 · see citing works
Learning curves in surgery: variables, analysis and applications
Valsamis, E. M., Chouari, T. A. M., O'Dowd-Booth, C. J., Rogers, B. A., & Ricketts, D. (2018) · Postgraduate Medical Journal · doi:10.1136/postgradmedj-2018-135880 · Cited by 111 · see citing works
The Richmond Agitation-Sedation Scale modified for palliative care inpatients (RASS-PAL): a pilot study exploring validity and feasibility in clinical practice
Bush, S. H., Grassau, P. A., Yarmo, M. N., Zhang, T., Zinkie, S. J., & Pereira, J. L. (2014) · BMC Palliative Care · doi:10.1186/1472-684x-13-17 · Cited by 110 · see citing works
Update on Simulation-Based Surgical Training and Assessment in Ophthalmology
Thomsen, A. S. S., Subhi, Y., Kiilgaard, J. F., la Cour, M., & Konge, L. (2015) · Ophthalmology · doi:10.1016/j.ophtha.2015.02.028 · Cited by 109 · see citing works
Addressing the Interprofessional Collaboration Competencies of the Association of American Medical Colleges: A Systematic Review of Assessment Instruments in Undergraduate Medical Education
Havyer, R. D. A., Nelson, D. R., Wingo, M. T., Comfere, N. I., Halvorsen, A. J., McDonald, F. S., & Reed, D. A. (2015) · Academic Medicine · doi:10.1097/acm.0000000000001053 · Cited by 109 · see citing works
Developing scholarly projects in education: A primer for medical teachers
Beckman, T. J., & Cook, D. A. (2007) · Medical Teacher · doi:10.1080/01421590701291469 · Cited by 107 · see citing works
Medical student and junior doctors’ tolerance of ambiguity: development of a new scale
Hancock, J., Roberts, M. J., Monrouxe, L. V., & Mattick, K. L. (2014) · Advances in Health Sciences Education · doi:10.1007/s10459-014-9510-z · Cited by 106 · see citing works
Accreditation of Medical Education Programs: Moving From Student Outcomes to Continuous Quality Improvement Measures
Blouin, D., & Tekian, A. S. (2017) · Academic Medicine · doi:10.1097/acm.0000000000001835 · Cited by 106 · see citing works
A systematic review of the psychometric properties of self-report research utilization measures used in healthcare
Squires, J. E., Estabrooks, C. A., O'Rourke, H. M., Gustavsson, P., Newburn-Cook, C. V., & Wallin, L. (2011) · Implementation Science · doi:10.1186/1748-5908-6-83 · Cited by 101 · see citing works
Standards for talking and thinking about validity.
Newton, P. E., & Shaw, S. D. (2013) · Psychological Methods · doi:10.1037/a0032969 · Cited by 105 · see citing works
Teamwork Assessment in Internal Medicine: A Systematic Review of Validity Evidence and Outcomes
Havyer, R. D. A., Wingo, M. T., Comfere, N. I., Nelson, D. R., Halvorsen, A. J., McDonald, F. S., & Reed, D. A. (2013) · Journal of General Internal Medicine · doi:10.1007/s11606-013-2686-8 · Cited by 105 · see citing works
Examining Reliability and Validity of an Online Score (ALiEM AIR) for Rating Free Open Access Medical Education Resources
Chan, T. M., Grock, A., Paddock, M. T., Kulasegaram, K. M., Yarris, L. M., & Lin, M. (2016) · Annals of Emergency Medicine · doi:10.1016/j.annemergmed.2016.02.018 · Cited by 105 · see citing works
Comparing the Maslach Burnout Inventory to Other Well-Being Instruments in Emergency Medicine Residents
Williamson, K., Lank, P. M., Cheema, N., Hartman, N., Lovell, E. O., & Emergency Medicine Education Research Alliance (EMERA) (2018) · Journal of Graduate Medical Education · doi:10.4300/jgme-d-18-00155.1 · Cited by 105 · see citing works
Endobronchial Ultrasound Skills and Tasks Assessment Tool: Assessing the Validity Evidence for a Test of Endobronchial Ultrasound-guided Transbronchial Needle Aspiration Operator Skill
Davoudi, M., Colt, H. G., Osann, K., Lamb, C., & Mullon, J. J. (2012) · American Journal of Respiratory and Critical Care Medicine · doi:10.1164/rccm.201111-1968oc · Cited by 104 · see citing works
Discrepancies between school grades and test scores at individual and school level: effects of gender and family background
Lekholm, A. K., & Cliffordson, C. (2008) · Educational Research and Evaluation · doi:10.1080/13803610801956663 · Cited by 105 · see citing works
Competitiveness Reconceptualized: Psychometric Development of the Competitiveness Orientation Measure as a Unified Measure of Trait Competitiveness
Newby, J., & Klein, R. G. (2014) · The Psychological Record · doi:10.1007/s40732-014-0083-2 · Cited by 99 · see citing works
Validity of Level of Supervision Scales for Assessing Pediatric Fellows on the Common Pediatric Subspecialty Entrustable Professional Activities
Mink, R. B., Schwartz, A., Herman, B. E., Turner, D. A., Curran, M. L., Myers, A., Hsu, D. C., Kesselheim, J. C., & Carraccio, C. L. (2017) · Academic Medicine · doi:10.1097/acm.0000000000001820 · Cited by 101 · see citing works
From test validity to construct validity … and back?
Colliver, J. A., Conlee, M. J., & Verhulst, S. J. (2012) · Medical Education · doi:10.1111/j.1365-2923.2011.04194.x · Cited by 96 · see citing works
Enhancing students’ learning in problem based learning: validation of a self-assessment scale for active learning and critical thinking
Khoiriyah, U., Roberts, C., Jorm, C. M., & van der Vleuten, C. P. M. (2015) · BMC Medical Education · doi:10.1186/s12909-015-0422-2 · Cited by 101 · see citing works
mini-PAT (Peer Assessment Tool): A Valid Component of a National Assessment Programme in the UK?
Archer, J. C., Norcini, J. J., Southgate, L. J., Heard, S., & Davies, H. A. (2006) · Advances in Health Sciences Education · doi:10.1007/s10459-006-9033-3 · Cited by 96 · see citing works
Using Social Media to Improve Continuing Medical Education: A Survey of Course Participants
Wang, A. T., Sandhu, N. P., Wittich, C. M., Mandrekar, J. N., & Beckman, T. J. (2012) · Mayo Clinic Proceedings · doi:10.1016/j.mayocp.2012.07.024 · Cited by 97 · see citing works
Measuring physician cognitive load: validity evidence for a physiologic and a psychometric tool
Szulewski, A., Gegenfurtner, A., Howes, D. W., Sivilotti, M. L. A., & van Merriënboer, J. J. G. (2017) · Advances in Health Sciences Education · doi:10.1007/s10459-016-9725-2 · Cited by 94 · see citing works
Summative Assessment in Medicine: The Promise of Simulation for High‐stakes Evaluation
Boulet, J. R. (2008) · Academic Emergency Medicine · doi:10.1111/j.1553-2712.2008.00228.x · Cited by 94 · see citing works
Assessing the patient safety competencies of healthcare professionals: a systematic review
Okuyama, A., Martowirono, K., & Bijnen, B. (2011) · BMJ Quality & Safety · doi:10.1136/bmjqs-2011-000148 · Cited by 95 · see citing works
Construct validity of a surgical simulator as a valid model for capsulorhexis training
Privett, B., Greenlee, E. C., Rogers, G. M., & Oetting, T. A. (2010) · Journal of Cataract & Refractive Surgery · doi:10.1016/j.jcrs.2010.05.020 · Cited by 93 · see citing works
Construct validity of anterior segment anti-tremor and forceps surgical simulator training modules
Mahr, M. A., & Hodge, D. O. (2008) · Journal of Cataract & Refractive Surgery · doi:10.1016/j.jcrs.2008.02.015 · Cited by 92 · see citing works
Performance assessment: Consensus statement and recommendations from the 2020 Ottawa Conference
Boursicot, K. A. M., Kemp, S., WILKINSON, T. J., Findyartini, A., Canning, C., Cilliers, F. J., & Fuller, R. (2020) · Medical Teacher · doi:10.1080/0142159x.2020.1830052 · Cited by 96 · see citing works
Validity: one word with a plurality of meanings
St‐Onge, C., Young, M. E., Eva, K. W., & Hodges, B. D. (2016) · Advances in Health Sciences Education · doi:10.1007/s10459-016-9716-3 · Cited by 91 · see citing works
Multiple choice questions: a literature review on the optimal number of options.
Vyas, R., & Supe, A. N. (2008) · PubMed · Cited by 90 · see citing works
A call for the utilization of consensus standards in the surgical education literature
Korndorffer, J. R., Kasten, S. J., & Downing, S. M. (2009) · The American Journal of Surgery · doi:10.1016/j.amjsurg.2009.08.018 · Cited by 89 · see citing works
Self-citation

Authors: Downing, S. M.
Affiliations: University of Illinois Chicago (US)
Corresponding author: Downing, Steven M
Venue: Medical Education
Publisher: Wiley
Published: 2003-08-27
Volume: 37
Issue: 9
Pages: 830-837
ISSN: 0308-0110 (ISSN-L)
1365-2923
DOI: 10.1046/j.1365-2923.2003.01594.x
OpenAlex: W2161254436
Cited by: 1,434
References: 33
Citations by year: 2026: 74 · 2025: 65 · 2024: 74 · 2023: 71 · 2022: 92 · 2021: 94 · 2020: 110 · 2019: 91 · 2018: 94 · 2017: 89 · 2016: 99 · 2015: 96 · 2014: 69 · 2013: 54 · 2012: 48
FWCI: 12.19
Type: journal article
Language: en
Open access: closed
Licence: onlinelibrary.wiley.com/termsAndConditions#vor (vor)
Locations: Medical Education (publishedVersion)
PubMed (publishedVersion)
Topics: Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Innovations in Medical Education
Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
Social Sciences › Education › Health Education and Validation
Keywords: construct validity, validity evidence, medical education assessment, written examinations, high-stakes testing, score interpretation, content validity, response processes, internal structure, Standards for Educational and Psychological Testing
MeSH terms: Data Interpretation, Statistical, Education, Medical, Educational Measurement, Humans, Reproducibility of Results

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Validity: on the meaningful interpretation of assessment data

Abstract

Integrity signals

Claims the paper makes

Claims other papers make about this one

Cites

Cited by

Related

Record

About these checks