CoChat

Validating performance assessments: measures that may help to evaluate students’ expertise in ‘doing science’

Hild, Pitt, Gut, Christoph, Brückmann, Maja · 2019 · Research in Science & Technological Education

The DOI matches its Crossref record; 50 of the 50 references it cites were found in a registry or an index.

Abstract

Background: Several different measures have been proposed to solve persistent validity problems, such as high task-sampling variability, in the assessment of students’ expertise in ‘doing science’. Such measures include working with a-priori progression models, using standardised item shells and rating manuals, augmenting the number of tasks per student and comparing different measurement methods.Purpose: The impact of these measures on instrument validity is examined here under three different aspects: structural validity, generalisability and external validity.Sample: Performance assessments were administered to 418 students (187 girls, ages 12–16) in grades 7, 8 and 9 in the 2 lowest school performance tracks in (lower) secondary school in the Swiss canton of Zurich.Design and methods: Students worked with printed test sheets on which they were asked to report the outcomes of their investigations. In addition to the written protocols, direct observations and interviews were used as measurement methods. Evidence of the instruments’ validity was reported by using different reliability and generalisability coefficients and by comparing our results to those found in literature.Results: An a-priori progression model was successfully used to improve the instrument’s structural validity. The use of a standardised item shell and rating manual ensured reliable rating of the written protocols (.79 ≤ p0 ≤ .98; .56 ≤ κ ≤ .97). Augmenting the number of tasks per student did not solve the challenge of reducing task-sampling variability. The observed performance differed from the performance assessed via the written protocols.Conclusions: Students’ performance in doing science can be reliably assessed with instruments that show good generalisability coefficients (ρ2 = 0.72 in this case). Even after implementing the different measures, task-sampling variability remains high (σˆpt2=47.2%). More elaborate studies that focus on the substantive aspect of validity must be conducted to understand why students’ expertise as shown in written protocols differs so markedly from their observed performance.

Overview

Integrity signals

Registry

Crossref · 10.1080/02635143.2018.1552851

Cites

50 of 50 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Show all 50 references

Research works printed but not linked: 14

  1. Commons M. L., 2008, Journal of Applied Measurement, 9, 182
  2. Erickson G., 1994, The Content of Science, 80
  3. Haertel E. H., 1996, Technical Issues in Large-Scale Performance Assessment, 59
  4. Harmon M., 1997, Performance Assessment in IEA’s Third International Mathematics and Science Study
  5. Hild P., 2015, In Heterogenität Und Diversität - Vielfalt der Voraussetzungen im naturwissenschaftlichen Unterricht, edited by S. Bernholt. Gesellschaft für Didaktik der Chemie und Physik, Jahrestagung in Bremen 2014, 145
  6. Hild P., 2018, Qualitätsvoller Chemie- und Physikunterricht - normative und empirische Dimensionen, edited by C. Maurer. Gesellschaft für Didaktik der Chemie und Physik, Jahrestagung in Regensburg, 2017, 348
  7. Labudde P., 2012, Making It Tangible. Learning Outcomes in Science Education, 235
  8. Messick S., 1989, Educational Measurement, 13
  9. Messick S., 1996, Technical Issues in Large-Scale Performance Assessment
  10. Metzger S, 2014, In E-proceedings of The ESERA 2013 conference in Nicosia
  11. Millar R., 1996, Progression in Learning
  12. Miller M. D., 1998, Generalizability of Performance-Based Assessments
  13. Solano-Flores, G. 1994. “A Logical Model for the Development of Science Performance Assessments.” Dissertation, University of Santa Barbara.
  14. Solano-Flores, G., R. J. Shavelson, S. E. Schultz, and E. W. Wiley. 1997.On the Development and Scoring of Classification and Observation Science Performance Assessments. California: University of Los Angeles. National Center for Research on Evaluation, Standards, and Student Testing. Report 458.

Cited by

See all 13

Metadata

Authors
Hild, P., Gut, C., & Brückmann, M.
Affiliations
  • Zurich University of Teacher Education (CH)
  • Carl von Ossietzky Universität Oldenburg (DE)
Corresponding author
Hild, Pitt
Venue
Research in Science & Technological Education
Published
2018-12-28
Volume
37
Issue
4
Pages
419-445
ISSN
  • 0263-5143 (ISSN-L)
  • 1470-1138
OpenAlex
W2906826987
Cited by
9
References
49
Citations by year
2025: 3 · 2024: 1 · 2023: 2 · 2022: 1 · 2020: 1 · 2019: 1
FWCI
1.62
Type
journal article
Language
en
Open access
closed
Locations
Research in Science & Technological Education (publishedVersion)
Topics
  • Social Sciences › Education › Science Education and Pedagogy
  • Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
  • Social Sciences › Education › Educational Assessment and Pedagogy
Keywords
performance evaluation, assessment validity, structural validity, generalizability, junior high school students, science education assessment

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Validating performance assessments: measures that may help to evaluate students’ expertise in ‘doing science’. https://app.cochat.ai/papers/10.1080/02635143.2018.1552851
The paper
Hild, P., Gut, C., & Brückmann, M. (2019). Validating performance assessments: measures that may help to evaluate students’ expertise in ‘doing science’. Research in Science & Technological Education. https://doi.org/10.1080/02635143.2018.1552851

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.