Validating performance assessments: measures that may help to evaluate students’ expertise in ‘doing science’
The DOI matches its Crossref record; 50 of the 50 references it cites were found in a registry or an index.
Abstract
Background: Several different measures have been proposed to solve persistent validity problems, such as high task-sampling variability, in the assessment of students’ expertise in ‘doing science’. Such measures include working with a-priori progression models, using standardised item shells and rating manuals, augmenting the number of tasks per student and comparing different measurement methods.Purpose: The impact of these measures on instrument validity is examined here under three different aspects: structural validity, generalisability and external validity.Sample: Performance assessments were administered to 418 students (187 girls, ages 12–16) in grades 7, 8 and 9 in the 2 lowest school performance tracks in (lower) secondary school in the Swiss canton of Zurich.Design and methods: Students worked with printed test sheets on which they were asked to report the outcomes of their investigations. In addition to the written protocols, direct observations and interviews were used as measurement methods. Evidence of the instruments’ validity was reported by using different reliability and generalisability coefficients and by comparing our results to those found in literature.Results: An a-priori progression model was successfully used to improve the instrument’s structural validity. The use of a standardised item shell and rating manual ensured reliable rating of the written protocols (.79 ≤ p0 ≤ .98; .56 ≤ κ ≤ .97). Augmenting the number of tasks per student did not solve the challenge of reducing task-sampling variability. The observed performance differed from the performance assessed via the written protocols.Conclusions: Students’ performance in doing science can be reliably assessed with instruments that show good generalisability coefficients (ρ2 = 0.72 in this case). Even after implementing the different measures, task-sampling variability remains high (σˆpt2=47.2%). More elaborate studies that focus on the substantive aspect of validity must be conducted to understand why students’ expertise as shown in written protocols differs so markedly from their observed performance.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Current tools available for the identification of these errors primarily consist of rating schemes, paper-pencil tests, or on-the-fly observations which are employed by teachers or students themselves to evaluate student performance (Hild et al., 2019; Lehtinen et al., 2022).”
From Assessing student errors in experimentation using artificial intelligence and large language models: A comparative study with human raters (2023) · cites this paper as Hild et al., 2019
Citations
Cites
Motivational and self-regulated learning components of classroom academic performance.
Pintrich, P. R., & De Groot, E. V. (1990) · Journal of Educational Psychology · doi:10.1037/0022-0663.82.1.33 · Cited by 8,332
The interplay of evidence and consequences in the validation of performance assessments
Messick, S. (1994) · Educational Researcher · doi:10.3102/0013189x023002013 · Cited by 1,315 · cited 1×
“Based on the unitary view of construct validity postulated by Messick (1989, 1994, 1996), the validity of PAs may be defined as ‘the evaluative judgement of the degree to which empirical evidence and theoretical rationales support the adequacy and appropriateness of interpretations and actions…”
BackgroundDeBoer, G. E. (2000) · Journal of Research in Science Teaching · doi:10.1002/1098-2736(200008)37:6<582::aid-tea5>3.0.co;2-l · Cited by 1,285
Marsh, H. W., Trautwein, U., Lüdtke, O., Köller, O., & Baumert, J. (2005) · Child Development · doi:10.1111/j.1467-8624.2005.00853.x · Cited by 1,198 · cited 2×
“Subject- and topic-related situational interest, intrinsic and extrinsic motivation (Fechner 2009), self-concept in natural science (Marsh et al. 2005), and the expectancy of self-efficacy (Pintrich and De Groot 1990) may positively influence the cognitive outcome of a PA.”
BackgroundReview of Educational Measurement.
Langmuir, C. R. (1952) · Journal of Educational Psychology · doi:10.1037/h0051392 · Cited by 903
THEORY OF GENERALIZABILITY: A LIBERALIZATION OF RELIABILITY THEORY†
Cronbach, L. J., Rajaratnam, N., & Gleser, G. C. (1963) · British Journal of Statistical Psychology · doi:10.1111/j.2044-8317.1963.tb00206.x · Cited by 737 · cited 1×
“Different sources of measurement errors (‘facets’) may be investigated within generalisability (G) studies (Brennan 1996; Cronbach, Rajaratnam, and Gleser 1963).”
BackgroundStudents' transition from an engineering model to a science model of experimentation
Schauble, L., Klopfer, L. E., & Raghavan, K. (1991) · Journal of Research in Science Teaching · doi:10.1002/tea.3660280910 · Cited by 369
HOFSTEIN, A. (2004) · Chem. Educ. Res. Pract. · doi:10.1039/b4rp90027h · Cited by 315
Validating Measures of Performance
Kane, M., Crooks, T., & Cohen, A. (1999) · Educational Measurement: Issues and Practice · doi:10.1111/j.1745-3992.1999.tb00010.x · Cited by 299
What is the purpose of this experiment? Or can students learn something from doing experiments?
Hart, C., Mulhall, P., Berry, A., Loughran, J., & Gunstone, R. (2000) · Journal of Research in Science Teaching · doi:10.1002/1098-2736(200009)37:7<655::aid-tea3>3.0.co;2-e · Cited by 277
Research works printed but not linked: 14
- Commons M. L., 2008, Journal of Applied Measurement, 9, 182
- Erickson G., 1994, The Content of Science, 80
- Haertel E. H., 1996, Technical Issues in Large-Scale Performance Assessment, 59
- Harmon M., 1997, Performance Assessment in IEA’s Third International Mathematics and Science Study
- Hild P., 2015, In Heterogenität Und Diversität - Vielfalt der Voraussetzungen im naturwissenschaftlichen Unterricht, edited by S. Bernholt. Gesellschaft für Didaktik der Chemie und Physik, Jahrestagung in Bremen 2014, 145
- Hild P., 2018, Qualitätsvoller Chemie- und Physikunterricht - normative und empirische Dimensionen, edited by C. Maurer. Gesellschaft für Didaktik der Chemie und Physik, Jahrestagung in Regensburg, 2017, 348
- Labudde P., 2012, Making It Tangible. Learning Outcomes in Science Education, 235
- Messick S., 1989, Educational Measurement, 13
- Messick S., 1996, Technical Issues in Large-Scale Performance Assessment
- Metzger S, 2014, In E-proceedings of The ESERA 2013 conference in Nicosia
- Millar R., 1996, Progression in Learning
- Miller M. D., 1998, Generalizability of Performance-Based Assessments
- Solano-Flores, G. 1994. “A Logical Model for the Development of Science Performance Assessments.” Dissertation, University of Santa Barbara.
- Solano-Flores, G., R. J. Shavelson, S. E. Schultz, and E. W. Wiley. 1997.On the Development and Scoring of Classification and Observation Science Performance Assessments. California: University of Los Angeles. National Center for Research on Evaluation, Standards, and Student Testing. Report 458.
Cited by
Bewersdorff, A., Seßler, K., Baur, A., Kasneci, E., & Nerdel, C. (2023) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2023.100177 · Cited by 76
“Current tools available for the identification of these errors primarily consist of rating schemes, paper-pencil tests, or on-the-fly observations which are employed by teachers or students themselves to evaluate student performance (Hild et al., 2019; Lehtinen et al., 2022).”
BackgroundMethodCavelti, M. F. (2024) · Logos Verlag Berlin eBooks · doi:10.30819/5829 · Cited by 18
Hild, P., Buff, A., Gut, C., & Parchmann, I. (2020) · Zeitschrift für Didaktik der Naturwissenschaften · doi:10.1007/s40573-020-00109-8 · Cited by 12
Self-citationReith, M., & Nehring, A. (2024) · International Journal of Science Education · doi:10.1080/09500693.2024.2394708 · Cited by 10
Reith, M., & Nehring, A. (2022) · Journal of Chemical Education · doi:10.1021/acs.jchemed.2c00340 · Cited by 8
Metadata
- Authors
- Hild, P., Gut, C., & Brückmann, M.
- Affiliations
- Zurich University of Teacher Education (CH)
- Carl von Ossietzky Universität Oldenburg (DE)
- Corresponding author
- Hild, Pitt
- Venue
- Research in Science & Technological Education
- Publisher
- Informa UK Limited
- Published
- 2018-12-28
- Volume
- 37
- Issue
- 4
- Pages
- 419-445
- ISSN
- 0263-5143 (ISSN-L)
- 1470-1138
- OpenAlex
- W2906826987
- Cited by
- 9
- References
- 49
- Citations by year
- 2025: 3 · 2024: 1 · 2023: 2 · 2022: 1 · 2020: 1 · 2019: 1
- FWCI
- 1.62
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Research in Science & Technological Education (publishedVersion)
- Topics
- Social Sciences › Education › Science Education and Pedagogy
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Education › Educational Assessment and Pedagogy
- Keywords
- performance evaluation, assessment validity, structural validity, generalizability, junior high school students, science education assessment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Validating performance assessments: measures that may help to evaluate students’ expertise in ‘doing science’. https://app.cochat.ai/papers/10.1080/02635143.2018.1552851
- The paper
- Hild, P., Gut, C., & Brückmann, M. (2019). Validating performance assessments: measures that may help to evaluate students’ expertise in ‘doing science’. Research in Science & Technological Education. https://doi.org/10.1080/02635143.2018.1552851
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.