Effective rating scale development for speaking tests: Performance decision trees
Open accessThe DOI matches its Crossref record; 3 of the 3 references it cites were found in a registry or an index.
Abstract
Rating scale design and development for testing speaking is generally conducted using one of two approaches: the measurement-driven approach or the performance data-driven approach. The measurement-driven approach prioritizes the ordering of descriptors onto a single scale. Meaning is derived from the scaling methodology and the agreement of trained judges as to the place of any descriptor on the scale. The performance data-driven approach, on the other hand, places primary value upon observations of language performance, and attempts to describe performance in sufficient detail to generate descriptors that bear a direct relationship with the original observations of language use. Meaning is derived from the link between performance and description. We argue that measurement-driven approaches generate impoverished descriptions of communication, while performance data-driven approaches have the potential to provide richer descriptions that offer sounder inferences from score meaning to performance in specified domains. With reference to original data and the literature on travel service encounters, we devise a new scoring instrument, a Performance Decision Tree (PDT). This instrument prioritizes what we term ‘performance effect’ by explicitly valuing and incorporating performance data from a specific communicative context. We argue that this avoids the reification of ordered scale descriptors which we find in measurement-driven scale construction for speaking tests.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
THEORETICAL BASES OF COMMUNICATIVE APPROACHES TO SECOND LANGUAGE TEACHING AND TESTING
CANALE, M., & SWAIN, M. (1980) · Applied Linguistics · doi:10.1093/applin/i.1.1 · Cited by 5,044
Theoretical Bases of Communicative Approaches to Second Language Teaching and Testing
CANALE, M. (1980) · Applied Linguistics · doi:10.1093/applin/1.1.1 · Cited by 4,021
Customer Switching Behavior in Service Industries: An Exploratory Study
Keaveney, S. M. (1995) · Journal of Marketing · doi:10.2307/1252074 · Cited by 2,905 · cited 1×
“It is now widely accepted that failure to provide satisfaction in fleeting encounters with customers causes them to switch to other service providers (Keaveney, 1995).”
Background
Cited by
Sato, M. (2014) · System · doi:10.1016/j.system.2014.05.004 · Cited by 37
Constructing a Scale to Assess L2 Written Speech Act Performance: WDCT and E-mail Tasks
Chen, Y.-S., & Liu, J. (2016) · Language Assessment Quarterly · doi:10.1080/15434303.2016.1213844 · Cited by 20
“The present study used learner performance data in the investigation of target language proficiency at the pragmatic level, which “makes the inferences drawn from the scores sounder” (Fulcher et al., 2011, p. 6).”
BackgroundMethod
Metadata
- Authors
- Fulcher, G., Davidson, F., & Kemp, J.
- Venue
- Language Testing
- Publisher
- SAGE Publications
- Published
- 2010-04-07
- Volume
- 28
- Issue
- 1
- Pages
- 5-29
- ISSN
- 0265-5322 (ISSN-L)
- 1477-0946
- OpenAlex
- W2103074295
- Cited by
- 154
- References
- 3
- FWCI
- 56.56
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.