CoChat

Evaluating Individual Students' Perceptions of Instructional Quality: An Investigation of their Factor Structure, Measurement Invariance, and Relations to Educational Outcomes

Open access
Scherer, Ronny, Nilsen, Trude, Jansen, Malte · 2016 · Frontiers in Psychology
Cited by 126

The DOI matches its Crossref record; 17 of the 17 references it cites were found in a registry or an index.

Abstract

Students' perceptions of instructional quality are among the most important criteria for evaluating teaching effectiveness. The present study evaluates different latent variable modeling approaches (confirmatory factor analysis, exploratory structural equation modeling, and bifactor modeling), which are used to describe these individual perceptions with respect to their factor structure, measurement invariance, and the relations to selected educational outcomes (achievement, self-concept, and motivation in mathematics). On the basis of the Programme for International Student Assessment (PISA) 2012 large-scale data sets of Australia, Canada, and the USA (N = 26,746 students), we find support for the distinction between three factors of individual students' perceptions and full measurement invariance across countries for all modeling approaches. In this regard, bifactor exploratory structural equation modeling outperformed alternative approaches with respect to model fit. Our findings reveal significant relations to the educational outcomes. This study synthesizes different modeling approaches of individual students' perceptions of instructional quality and provides insights into the nature of these perceptions from an individual differences perspective. Implications for the measurement and modeling of individually perceived instructional quality are discussed.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.3389/fpsyg.2016.00110

Cites

17 of 17 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 68

  1. Asparouhov, 2005, Sampling weights in latent variable modeling, Struct. Eq. Model., 12, 411, 10.1207/s15328007sem1203_4 · DOI · Paper page
  2. Asparouhov, 2009, Exploratory structural equation modeling, Struct. Eq. Model., 16, 397, 10.1080/10705510903008204 · DOI · Paper page
  3. Asparouhov, 2010, Weighted Least Squares Estimation with Missing Data
  4. Beauducel, 2006, On the performance of maximum likelihood versus means and variance adjusted weighted least square estimation in confirmatory factor analysis, Struct. Eq. Model., 13, 186, 10.1207/s15328007sem1302_2 · DOI · Paper page
  5. Blazar, 2015, Effective teaching in elementary mathematics: identifying classroom practices that support student achievement, Econ. Educ. Rev., 48, 16, 10.1016/j.econedurev.2015.05.005 · DOI
  6. Bulle, 2011, Comparing OECD educational models through the prism of PISA, Comp. Educ., 47, 503, 10.1080/03050068.2011.555117 · DOI
  7. Charalambous, 2014, Using generic and content-specific teaching practices in teacher evaluation: an exploratory study of teachers' perceptions, Teach. Teacher Educ., 41, 22, 10.1016/j.tate.2014.03.001 · DOI
  8. Chen, 2012, Modeling general and specific variance in multifaceted constructs: a comparison of the bifactor model to other approaches, J. Pers., 80, 219, 10.1111/j.1467-6494.2011.00739.x · DOI · Paper page
  9. Chung, 2015, Bifactor approach to modeling multi-dimensionality of physical self-perception profile, Measurem. Phys. Educat. Exercise Sci., 20, 1, 10.1080/1091367X.2015.1081594 · DOI
  10. Creemers, 2008, The Dynamics of Educational Effectiveness
  11. Desa, 2014, Evaluating Measurement Invariance of TALIS 2013 Complex Scales, 10.1787/5jz2kbbvlb7k-en · DOI
  12. Dietrich, 2015, Teacher support and its influence on students' intrinsic value and effort: dimensional comparison effects across subjects, Learn. Instruct., 39, 45, 10.1016/j.learninstruc.2015.05.007 · DOI · Paper page
  13. Dolan, 2009, Testing measurement invariance in the target rotated multigroup exploratory factor model, Struct. Eq. Model., 16, 295, 10.1080/10705510902751416 · DOI · Paper page
  14. Enders, 2010, Applied Missing Data Analysis
  15. Fauth, 2014, Student ratings of teaching quality in primary school: dimensions and prediction of student outcomes, Learn. Instruct., 29, 1, 10.1016/j.learninstruc.2013.07.001 · DOI · Paper page
  16. Greenwald, 1997, Validity concerns and usefulness of student ratings of instruction, Am. Psychol., 52, 1182, 10.1037/0003-066X.52.11.1182 · DOI · Paper page
  17. Grossman, 2013, The relationship between measures of instructional practice in middle school english language arts and teachers' value-added scores, Am. J. Educat., 119, 445, 10.1086/669901 · DOI · Paper page
  18. Guenole, 2014, The consequence of ignoring measurement invariance for path coefficients in structural equation models, Front. Psychol., 5, 10.3389/fpsyg.2014.00980 · DOI
  19. Gustafsson, 2010, Unidimensionality and interpretability of psychological instruments, 97
  20. Hattie, 2009, Visible Learning—A Synthesis of Meta-Analyses in Education
  21. He, 2013, Methodological issues in cross-cultural studies in educational psychology, 39
  22. Jak, 2014, Measurement bias in multilevel data, Struct. Equat. Model., 21, 31, 10.1080/10705511.2014.856694 · DOI · Paper page
  23. Jonassen, 2011, Supporting problem solving in PBL, Interdiscipl. J. Problem Based Learn., 5, 95, 10.7771/1541-5015.1256 · DOI · Paper page
  24. Klieme, 2013, The role of large-scale assessments in research on educational effectiveness and school development, 115, 10.1007/978-94-007-4629-9_7 · DOI
  25. Klieme, 2009, The Pythagoras study: Investigating effects of teaching and learning in Swiss and German mathematics classrooms, 137
  26. Kristen, 2005, Longitudinal Studies for Education Reports: European and North American Examples (Education Reform Volume 10)
  27. Kuenzi, 2009, Assembling fragments into a lens: a review, critique, and proposed research agenda for the organizational work climate literature, J. Manage., 35, 634, 10.1177/0149206308330559 · DOI · Paper page
  28. Kunter, 2007, Effective classroom management and the development of subject-related interest, Learn. Instruct., 17, 494, 10.1016/j.learninstruc.2007.09.002 · DOI · Paper page
  29. Kunter, 2008, Students' and mathematics teachers' perceptions of teacher enthusiasm and instruction, Learn. Instruct., 18, 468, 10.1016/j.learninstruc.2008.06.008 · DOI · Paper page
  30. Lazarides, 2012, Instructional quality and attitudes toward mathematics: do self-concept and interest differ across students' patterns of perceived instructional quality in mathematics classrooms?, Child Dev. Res., 2012, 813920, 10.1155/2012/813920 · DOI
  31. Little, 2013, Longitudinal Structural Equation Modeling
  32. Lipowsky, 2009, Quality of geometry instruction and its short-term impact on students' understanding of the Pythagorean Theorem, Learn. Instruct., 19, 527, 10.1016/j.learninstruc.2008.11.001 · DOI · Paper page
  33. Lüdtke, 2009, Assessing the impact of learning environments: how to use student ratings of classroom or school characteristics in multilevel modeling, Contemp. Educ. Psychol., 34, 77, 10.1016/j.cedpsych.2008.12.001 · DOI · Paper page
  34. Marsh, 1986, Verbal and math self-concepts: an internal/external frame of reference model, Am. Educ. Res. J., 23, 129, 10.3102/00028312023001129 · doi:10.3102/00028312023001129 · Paper page
  35. Marsh, 2013a, Factorial, convergent, and discriminant validity of TIMSS math and science motivation measures: a comparison of Arab and Anglo-Saxon countries, J. Educ. Psychol., 105, 108, 10.1037/a0029907 · DOI · Paper page
  36. Marsh, 1997, Making students' evaluations of teaching effectiveness effective, Am. Psychol., 52, 1187, 10.1037/0003-066X.52.11.1187 · doi:10.1037/0003-066x.52.11.1187 · Paper page
  37. Marsh, 2005, Goodness of fit evaluation in structural equation modeling, 275
  38. Marsh, 2012, Classroom climate and contextual effects: conceptual and methodological issues in the evaluation of group-level effects, Educ. Psychol., 47, 106, 10.1080/00461520.2012.670488 · DOI · Paper page
  39. Marsh, 2014, Exploratory structural equation modeling: an integration of the best features of exploratory and confirmatory factor analysis, Annu. Rev. Clin. Psychol., 10, 85, 10.1146/annurev-clinpsy-032813-153700 · DOI · Paper page
  40. Marsh, 2009, Exploratory structural equation modeling, integrating CFA and EFA: application to students' evaluations of university teaching, Struct. Equat. Model., 16, 439, 10.1080/10705510903008220 · DOI · Paper page
  41. Marsh, 2013b, Measurement invariance of big-five factors over the life span: ESEM tests of gender, age, plasticity, maturity, and la dolce vita effects, Dev. Psychol., 49, 1194, 10.1037/a0026913 · DOI · Paper page
  42. Martin, 2013, Effective schools in reading, mathematics, and science at fourth grade, 109
  43. McCoach, 2013, Instrument Development in the Affective Domain, 10.1007/978-1-4614-7135-6 · DOI · Paper page
  44. MET Project, 2013, Ensuring Fair and Reliable Measures of Effective Teaching (Policy and Practice Brief)
  45. Morin, 2016, A bifactor exploratory structural equation modeling framework for the identification of distinct sources of construct-relevant psychometric multidimensionality, Struct. Equat. Model., 23, 116, 10.1080/10705511.2014.961800 · DOI · Paper page
  46. Morin, 2014, Doubly latent multilevel analyses of classroom climate: an illustration, J. Experim. Educat., 82, 143, 10.1080/00220973.2013.769412 · DOI · Paper page
  47. Murray, 1938, Explorations in Personality
  48. Muthén, 1998–2014, Mplus 7.3 [Computer Software]
  49. OECD, 2013a, PISA 2012 Results: Ready to Learn: Students' Engagement, Drive and Self-Beliefs (Volume III), 10.1787/9789264201170-en · DOI · Paper page
  50. OECD, 2013b, PISA 2012 Assessment and Analytical Framework, 10.1787/9789264190511-en · DOI · Paper page
  51. OECD, 2014a, PISA 2012 Results: What Students Know and Can Do, 10.1787/6341a959-en · DOI · Paper page
  52. OECD, 2014b, PISA 2012 Technical Report, 10.1787/6341a959-en · DOI · Paper page
  53. Parker, 2003, Relationships between psychological climate perceptions and work outcomes: a meta-analytic review, J. Organ. Behav., 24, 389, 10.1002/job.198 · DOI · Paper page
  54. Parker, 2014, Predicting career aspirations and university majors from academic ability and self-concept, 224, 10.1017/CBO9781139128933.015 · DOI
  55. Perera, 2015, Construct validity of the social provisions scale: a bifactor exploratory structural equation modeling approach, Assessment, 10.1177/1073191115589344 · DOI
  56. Roche, 2002, Teaching self-concept in higher education, 179, 10.1007/978-94-010-0593-7_9 · DOI
  57. Satorra, 2010, Ensuring positiveness of the scaled difference chi-square test statistic, Psychometrika, 75, 243, 10.1007/s11336-009-9135-y · DOI · Paper page
  58. Scherer, 2015, Student assessment of teaching as a source of information about aspects of teaching quality in multiple subject domains: an application of multilevel bifactor structural equation modeling, Front. Psychol., 6, 10.3389/fpsyg.2015.01550 · DOI
  59. Seidel, 2007, Teaching effectiveness research in the past decade: the role of theory and research design in disentangling meta-analysis results, Rev. Educ. Res., 77, 454, 10.3102/0034654307310317 · DOI · Paper page
  60. Stenling, 2015, Using bifactor exploratory structural equation modeling to examine global and specific factors in measures of sports coaches' interpersonal styles, Front. Psychol., 6, 10.3389/fpsyg.2015.01303 · DOI · Paper page
  61. Stern, 1970, People in Context. Measuring Person–Environment Congruence in Education and Industry
  62. van de Schoot, 2013, Facing off with Scylla and Charybdis: a comparison of scalar, partial, and the novel possibility of approximate measurement invariance, Front. Psychol., 4, 10.3389/fpsyg.2013.00700 · DOI · Paper page
  63. Van Tartwijk, 2011, The neglected role of classroom management in teacher education, Teach. Educat., 22, 109, 10.1080/10476210.2001.567836 · doi:10.1080/10476210.2001.567836
  64. von Davier, 2009, What are plausible values and why are they useful?, 9
  65. Wagner, 2013, Construct validity of student perceptions of instructional quality is high, but not perfect: dimensionality and generalizability of domain-independent assessments, Learn. Instruct., 28, 1, 10.1016/j.learninstruc.2013.03.003 · DOI · Paper page
  66. Wiesner, 2013, Exploratory structural equation modeling, bifactor models, and standard confirmatory factor analysis models: application to the BASC-2 Behavioral and Emotional Screening System Teacher Form, J. Sch. Psychol., 51, 751, 10.1016/j.jsp.2013.09.001 · DOI
  67. Wu, 2005, The role of plausible values in large-scale surveys, Studies Educat. Eval., 31, 114, 10.1016/j.stueduc.2005.05.005 · DOI · Paper page
  68. Yang, 2011, Coefficient alpha: a reliability coefficient for the 21st century?, J. Psychoeduc. Assess., 29, 377, 10.1177/0734282911406668 · DOI · Paper page

Cited by

See all 126

Metadata

Authors
Scherer, R., Nilsen, T., & Jansen, M.
Venue
Frontiers in Psychology
Published
2016-02-08
Volume
7
Pages
110-110
ISSN
1664-1078 (ISSN-L)
OpenAlex
W2280568119
Cited by
126
References
17
FWCI
8.39
Type
journal article
Language
en
Open access
gold

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.