A BRIEF INTRODUCTION TO EVIDENCE‐CENTERED DESIGN
Open accessThe DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index.
Abstract
Evidence‐centered assessment design (ECD) is an approach to constructing educational assessments in terms of evidentiary arguments. This paper provides an introduction to the basic ideas of ECD, as well as some of the terminology and models that have been developed to implement the approach. In particular, it presents the high‐level models of the Conceptual Assessment Framework (CAF) and the Four‐process Delivery Architecture for assessment delivery systems. Special attention is given to the role of probability‐based reasoning in accumulating evidence across task performances, in terms of belief about unobservable variables that characterize the knowledge, skills, and/or abilities of students. This is the role traditionally associated with psychometric models, such as those of item response theory (IRT) and latent class models. To unify the ideas and to provide a foundation for extending probability‐based reasoning in assessment applications more broadly, however, a more general expression in terms of graphical models is indicated. This brief overview of evidence‐centered design provides the reader with a feel for where and how graphical models fit into the larger enterprise of educational and psychological assessment. A simple example based on familiar large‐scale standardized tests such as the Graduate Record Examinations® (GRE®) is used to fix ideas.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Gelman, A., Carlin, J. B., Stern, H. S., & Rubin, D. B. (2003) · doi:10.1201/9780429258480 · Cited by 27,276
Educational measurement, 3rd ed.
Linn, R. L. (1989) · Cited by 2,883
Moses, L. E., & Lindquist, E. F. (1951) · Journal of the American Statistical Association · doi:10.2307/2280411 · Cited by 1,661
The interplay of evidence and consequences in the validation of performance assessments
Messick, S. (1994) · Educational Researcher · doi:10.3102/0013189x023002013 · Cited by 1,315
Molecular diversity of arbuscular mycorrhizal fungi colonising arable crops
Daniell, T. (2001) · FEMS Microbiology Ecology · doi:10.1016/s0168-6496(01)00134-9 · Cited by 990
An argument-based approach to validity.
Kane, M. T. (1992) · Psychological Bulletin · doi:10.1037/0033-2909.112.3.527 · Cited by 901
Focus Article: On the Structure of Educational Assessments
Mislevy, R. J., Steinberg, L. S., & Almond, R. G. (2003) · Measurement: Interdisciplinary Research & Perspective · doi:10.1207/s15366359mea0101_02 · Cited by 802
Self-citationConstruct validity: Construct representation versus nomothetic span.
Whitely, S. E. (1983) · Psychological Bulletin · doi:10.1037//0033-2909.93.1.179 · Cited by 604
Argument-Based Approach to Validity
(2022) · The SAGE Encyclopedia of Research Design · doi:10.4135/9781071812082.n28 · Cited by 556
The Evidential Foundations of Probabilistic Reasoning.
FH, & Schum, D. A. (1995) · Journal of the American Statistical Association · doi:10.2307/2291371 · Cited by 506
Cognitively Diagnostic Assessment
(2012) · Routledge eBooks · doi:10.4324/9780203052969 · Cited by 490
CONSTRUCT VALIDITY: CONSTRUCT REPRESENTATION VERSUS NOMOTHETIC SPAN
Embretson, S. E. (1983) · Cited by 322
Construct validation after thirty years.
Cronbach, L. J. (1989) · Cited by 277
Design and analysis in task-based language assessment
Mislevy, R. J., Steinberg, L. S., & Almond, R. G. (2002) · Language Testing · doi:10.1191/0265532202lt241oa · Cited by 218
Self-citationEVIDENCE AND INFERENCE IN EDUCATIONAL ASSESSMENT1,2
Mislevy, R. J. (1995) · ETS Research Report Series · doi:10.1002/j.2333-8504.1995.tb01643.x · Cited by 197
Self-citationThe evidential foundations of probabilistic reasoning
(1995) · Choice Reviews Online · doi:10.5860/choice.32-2700 · Cited by 174
Evidence and Inference in Educational Assessment
Mislevy, R. J. (1994) · Psychometrika · doi:10.1007/bf02294388 · Cited by 165
Self-citationThe role of probability-based inference in an intelligent tutoring system
Mislevy, R. J., & Gitomer, D. H. (1996) · User Modeling and User-Adapted Interaction · doi:10.1007/bf01126112 · Cited by 163
Self-citationEnhancing the Design and Delivery of Assessment Systems: A Four-Process Architecture
Almond, R. G., Steinberg, L. S., Mislevy, R. J., & Hoffmann, T. (2002) · Open Access Journals at BC (Boston College) · Cited by 144
Self-citationGRAPHICAL MODELS AND COMPUTERIZED ADAPTIVE TESTING
Almond, R. G., & Mislevy, R. J. (1998) · ETS Research Report Series · doi:10.1002/j.2333-8504.1998.tb01753.x · Cited by 122
Self-citationBayes nets in educational assessment: Where the numbers come fromPreprint
Mislevy, R. J., Almond, R. G., Yan, D., & Steinberg, L. S. (1999) · arXiv (Cornell University) · Cited by 118
Self-citationMaking Sense of Data From Complex Assessments
Mislevy, R. J., Steinberg, L. S., Breyer, F., Almond, R. G., & Johnson, L. A. (2002) · Applied Measurement in Education · doi:10.1207/s15324818ame1504_03 · Cited by 109
Self-citationPROBABILITY‐BASED INFERENCE IN COGNITIVE DIAGNOSIS
Mislevy, R. J. (1994) · ETS Research Report Series · doi:10.1002/j.2333-8504.1994.tb01576.x · Cited by 85
Self-citationOn the Roles of Task Model Variables in Assessment Design
Kyllonen, P. C. (2013) · Routledge eBooks · doi:10.4324/9781410602145-12 · Cited by 66
A cognitive task analysis with implications for designing simulation-based performance assessment
Mislevy, R. J., Steinberg, L. S., Breyer, F., Almond, R. G., & Johnson, L. A. (1999) · Computers in Human Behavior · doi:10.1016/s0747-5632(99)00027-8 · Cited by 66
Self-citationPsychometric Principles in Student Assessment
Mislevy, R. J., Wilson, M. R., Ercikan, K., & Chudowsky, N. (2003) · Springer eBooks · doi:10.1007/978-94-010-0309-4_31 · Cited by 54
Self-citationIntelligent tutoring and assessment built on an understanding of a technical problem-solving task
Steinberg, L. S., & Gitomer, D. H. (1996) · Instructional Science · doi:10.1007/bf00119978 · Cited by 38
Leverage Points for Improving Educational Assessment
Mislevy, R. J., Steinberg, L. S., Almond, R. G., Haertel, G. D., & Penuel, W. R. (2000) · Cited by 35
Self-citationSpecifying and refining a measurement model for a simulation-based assessment
Levy, R., & Mislevy, R. J. (2004) · Cited by 10
Self-citationSteinberg, L. S., Mislevy, R. J., Almond, R. G., Baird, A. B., Cahallan, C., DiBello, L. V., Şentürk, D., Yan, D., Chernick, H., & Kindfield, A. C. H. (2003) · Cited by 9
Self-citationArgument Substance and Argument Structure in Educational Assessment
Mislevy, R. J. (2003) · Cited by 7
Self-citationA Framework for Reusing Assessment Components
Almond, R. G., Steinberg, L. S., & Mislevy, R. J. (2003) · Springer eBooks · doi:10.1007/978-4-431-66996-8_31 · Cited by 6
Self-citationCreating a Complex Measurement Model Using Evidence Centered Design.
Williamson, D. M., Bauer, M., Steinberg, L. S., Mislevy, R. J., & Behrens, J. T. (2003) · American Educational Research Association Annual Meeting · Cited by 5
Self-citation
Cited by
Educational Data Mining and Learning Analytics
Baker, R. S., & Inventado, P. S. (2014) · Learning Analytics · doi:10.1007/978-1-4614-3305-7_4 · Cited by 1,029
Measuring progressions: Assessment structures underlying a learning progression
Wilson, M. (2009) · Journal of Research in Science Teaching · doi:10.1002/tea.20318 · Cited by 329
Characterizing College Science Assessments: The Three-Dimensional Learning Assessment Protocol
Laverty, J. T., Underwood, S. M., Matz, R. L., Posey, L. A., Carmel, J. H., Caballero, M. D., Fata-Hartley, C. L., Ebert-May, D., Jardeleza, S. E., & Cooper, M. M. (2016) · PLOS ONE · doi:10.1371/journal.pone.0162333 · Cited by 262
“One approach to assessment as an evidentiary argument is Evidence-Centered Design [47].”
BackgroundBang, H. J., Li, L., & Flynn, K. (2023) · Early Childhood Education Journal · doi:10.1007/s10643-022-01332-3 · Cited by 91
Culturally Responsive Assessment: Provisional Principles
Walker, M. E., Olivera‐Aguilar, M., Lehman, B., Laitusis, C., Guzman‐Orth, D., & Gholson, M. (2023) · ETS Research Report Series · doi:10.1002/ets2.12374 · Cited by 62
“…goal is to create culturally responsive assessments that emphasize new or enhanced processes or procedures and to incorporate these enhanced processes in principled assessment design approaches (e.g., evidence-centered design; Mislevy et al., 2003) across all stages of test design and development.”
Metadata
- Authors
- Mislevy, R. J., Almond, R. G., & Lukas, J. F.
- Venue
- ETS Research Report Series
- Publisher
- Wiley
- Published
- 2003-06
- Volume
- 2003
- Issue
- 1
- ISSN
- 2330-8516 (ISSN-L)
- OpenAlex
- W1987868810
- Cited by
- 473
- References
- 33
- FWCI
- 2.90
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.