Cognitive foundations for science assessment design: Knowing what students know about evolution
The DOI matches its Crossref record; 13 of the 13 references it cites were found in a registry or an index.
Abstract
To improve assessments of academic achievement, test developers have been urged to use an “assessment triangle” that starts with research‐based models of cognition and learning [NRC (2001)Knowing what students know: The science and design of educational assessment. Washington, DC: National Academy Press]. This approach has been successful in designing high‐quality reading and math assessments, but less progress has been made for assessments in content‐rich sciences such as biology. To rectify this situation, we applied the “assessment triangle” to design and evaluate new items for an instrument (ACORNS, Assessing Contextual Reasoning about Natural Selection) that had been proposed to assess students' use of natural selection to explain evolutionary change. Design and scoring of items was explicitly guided by a cognitive model that reflected four psychological principles: with development of expertise, (1) core concepts facilitate long‐term recall, (2) causally‐central features become weighted more strongly in explaining phenomena, (3) normative ideas co‐exist but increasingly outcompete naive ideas in reasoning, and (4) knowledge becomes more abstract and less specific to the learning situation. We conducted an evaluation study with 320 students to examine whether scores from our new ACORNS items could detect gradations of expertise, provide insight into thinking about evolutionary change, and predict teachers' assessments of student achievement. Findings were consistent with our cognitive model, and ACORNS was revealing about undergraduates' thinking about evolutionary change. Results indicated that (1) causally‐central concepts of evolution by natural selection typically co‐existed and competed with the presence of naïve ideas in all students' explanations, with naïve ideas being especially prevalent in low‐performers' explanations; (2) causally‐central concepts were elicited most frequently when students were asked to explain evolution of animals and familiar plants, with influence of superficial features being strongest for low‐performers; and (3) ACORNS scores accurately predicted students' later achievement in a college‐level evolution course. Together, findings illustrate usefulness of cognitive models in designing instruments intended to capture students' developing expertise. © 2012 Wiley Periodicals, Inc. J Res Sci Teach 49: 744–777, 2012
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Levels of processing: A framework for memory research
Craik, F. I. M., & Lockhart, R. S. (1972) · Journal of Verbal Learning and Verbal Behavior · doi:10.1016/s0022-5371(72)80001-x · Cited by 9,566 · cited 1×
“The first finding is that knowledge that is acquired with understanding (e.g., through ‘‘elaborative rehearsal’’; Craik & Lockhart, 1972) is retained better and transferred better than that which is acquired by rote (Katona, 1940).”
BackgroundCategorization and Representation of Physics Problems by Experts and Novices*
Chi, M. T. H., Feltovich, P. J., & Glaser, R. (1981) · Cognitive Science · doi:10.1207/s15516709cog0502_2 · Cited by 5,248 · cited 3×
“An important recent advance in assessment of evolution has been the finding that the knowledge and naı̈ve ideas that students employ depends very greatly on the specific contexts on which they are assessed (Nehm & Ha, 2011; see also Chi et al., 1981 for a classic study in physics).”
BackgroundNothing in Biology Makes Sense except in the Light of Evolution
Dobzhansky, T. (1973) · The American Biology Teacher · doi:10.2307/4444260 · Cited by 3,219
Schema induction and analogical transfer
Gick, M. L., & Holyoak, K. J. (1983) · Cognitive Psychology · doi:10.1016/0010-0285(83)90002-6 · Cited by 3,010 · cited 1×
“…generalizations overly rely on immediately perceptible and salient features while experts are able to use abstract information that has been most reliable in their past experience (Barnett & Ceci, 2002; Gentner & Markman, 1997; Gick & Holyoak, 1983; Newell & Simon, 1972; Opfer & Bulloch, 2007).”
BackgroundContextual prerequisites for understanding: Some investigations of comprehension and recall
Bransford, J. D., & Johnson, M. K. (1972) · Journal of Verbal Learning and Verbal Behavior · doi:10.1016/s0022-5371(72)80006-9 · Cited by 2,131
When and where do we apply what we learn?: A taxonomy for far transfer
Barnett, S. M., & Ceci, S. J. (2002) · Psychological Bulletin · doi:10.1037/0033-2909.128.4.612 · Cited by 2,076 · cited 1×
“…generalizations overly rely on immediately perceptible and salient features while experts are able to use abstract information that has been most reliable in their past experience (Barnett & Ceci, 2002; Gentner & Markman, 1997; Gick & Holyoak, 1983; Newell & Simon, 1972; Opfer & Bulloch, 2007).”
BackgroundInquiry, modeling, and metacognition: making science accessible to all students
White, B. Y., & Frederiksen, J. R. (1998) · Cognition and Instruction · doi:10.1207/s1532690xci1601_2 · Cited by 1,394 · cited 1×
“This strategy has proven highly useful in designing assessments of mathematics and physics (e.g., Hunt & Minstrell, 1994; Marshall, 1995; White & Frederiksen, 1998), but it has never been applied to improve the validity of evolution assessment instruments.”
MethodPsychological essentialism in children
GELMAN, S. (2004) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2004.07.001 · Cited by 492 · cited 2×
“…biological reasoning: (1) essentialist biases, the tendency to treat category members as having an enduring, heritable, ‘‘true nature’’ (Evans, 2001; Gelman, 2004; Shtulman, 2006; Shtulman & Schulz, 2008); (2) teleological biases, the tendency to treat features and actions of category…”
BackgroundThe Human Function Compunction: Teleological explanation in adults
Kelemen, D., & Rosset, E. (2009) · Cognition · doi:10.1016/j.cognition.2009.01.001 · Cited by 480 · cited 1×
“…& Schulz, 2008); (2) teleological biases, the tendency to treat features and actions of category members as existing for some purpose (Kelemen, 2003; Kelemen & Rosset, 2009; Opfer, 2002; Opfer & Gelman, 2001); and (3) intentionality biases, the tendency to assume that events are directed…”
BackgroundThe Coexistence of Natural and Supernatural Explanations Across Cultures and Development
Legare, C. H., Evans, E. M., Rosengren, K. S., & Harris, P. L. (2012) · Child Development · doi:10.1111/j.1467-8624.2012.01743.x · Cited by 332
Cited by
Pelletreau, K. N., Knight, J. K., Lemons, P. P., McCourt, J. S., Merrill, J. E., Nehm, R. H., Prevost, L. B., Urban-Lurain, M., & Smith, M. K. (2018) · CBE—Life Sciences Education · doi:10.1187/cbe.17-12-0260 · Cited by 66
Self-citationNehm, R. H. (2019) · Disciplinary and Interdisciplinary Science Education Research · doi:10.1186/s43031-019-0017-6 · Cited by 47
Self-citation
Metadata
- Authors
- Opfer, J. E., Nehm, R. H., & Ha, M.
- Venue
- Journal of Research in Science Teaching
- Publisher
- Wiley
- Published
- 2012-07-11
- Volume
- 49
- Issue
- 6
- Pages
- 744-777
- ISSN
- 0022-4308 (ISSN-L)
- 1098-2736
- OpenAlex
- W1976199158
- Cited by
- 195
- References
- 13
- FWCI
- 58.66
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.