Do College Student Surveys Have Any Validity?

“Social desirability bias, or providing survey responses that make the respondent look favorable to others, has been extensively documented in the field of survey methodology (see, for example, DeMaio, 1984; Nederhof, 1985), but has been little studied in the area of college student survey response.”

“…relies on students’ selfreported assessment of academic engagement, a factor that some have identified as problematic due to inaccuracy (Bowman 2011; Porter 2009); however, Bowman (2010) found that first-generation students tend to be more accurate in self-reports than other students, which could…”

“However, it is important to acknowledge that grades, especially self-reported GPAs, as a measure of learning have been called into question and also deemed unreliable (Porter, 2011).”

“While the validity of student surveys in general and NSSE specifically has been challenged (Porter, 2011), leaders of NSSE and its counter-part, the Community College Survey of Student Engagement, offered a point-by-point response to the critique (McCormick & McClenney, 2012), and 2-and 4-year…”

“Porter ( 2011 ) also recommends the use of time diary methods as an alternative to using surveys to measure student behavior.”

“This problem potentially adds an additional layer to the questions currently being asked about the validity of student surveys used in higher education (Porter 2011).”

“Third, Porter (2001) questioned validity of a typical college survey concluding it has minimal validity; NSSE and other college student surveys cannot withstand scrutiny in his opinion.”

“77”

“Recently, several studies have raised questions about the reliability and validity of NSSE and other surveys (Campbell and Cabrera 2011; DiRamio and Shannon 2010; Gordon et al. 2008; Korzekwa and Marley 2010; LaNasa et al. 2009; Lee 2010; Nora et al. 2011; Porter 2011; Porter et al. 2011).”

“One criticism of survey-based measures of learning outcomes is that they are self-reported by students rather than derived from more objective measures of student performance, such as tests or direct observations of student performances (see, for example, [48]).”

“As indicated by the title of his article (“Do College Student Surveys Have Any Validity?”), Porter (2011) raises a fundamental challenge to all surveys of college students and, by extension, to the considerable portion of the higher education scholarly literature that is based on survey data, as he notes (p.”

“To test these predictions, I use the Wabash National Study, a unique longitudinal study from 2006 to 2010 of students at 19 colleges and universities.”

“Statement of the Problem As argued by Porter (2011), the evidence that inferences about institutional effectiveness based on the results of surveys of student engagement is not strong.”

“However, self-report measures have also received criticism from the educational research community (e.g., Bowman, 2010; Campbell & Cabrera, 2011; Kuncel, Crede, & Thomas, 2005; Porter, 2011).”

“Moreover, there are also validity issues with the NSSE (Pike 2006; Porter 2010).”

“Although there are many observational studies in the literature that reveal higher exam scores of students in unproctored versus proctored environments, not all of these studies attribute their findings to cheating behaviors (Prince et al. (2009); Carstairs and Myors, (2009).”

“The current research also relied on students’ self-reports of their high school experiences, and it is reasonable to assume that these self-reports do not represent precise point estimates of course taking and involvement (Porter, 2011).”

“While these studies reliably capture largescale self-reports of students’ behavior over the past year, there are limitations in the ability of these surveys to measure students’ lived experience in college (Porter, 2011).”

“Respondents may have overestimated their engagement in pro-social behaviors such as volunteerism—typically the case in questionnaires (Porter, 2011).”

“When students wish to provide an answer in the survey, but cannot retrieve information, they resort to intelligent guessing, often based on what they think should be happening or to make them look favourable to others (Porter 2011).”

“Previous discussions have focused on self-reporting learning gains and attitudes, suggesting that the former is not effective, whereas the latter may be best assessed through self-reports (Chan, 2009; Porter, 2011).”

“Other large-scale instruments such as the National Survey of Student Engagement have also been condemned for failing to meet the basic standards for reliability and validity (Porter, 2011).”

“Many college student surveys lack validity as students may have difficulty accurately recalling and reporting the frequency of events (Porter, 2011) and information about their behavior and attitudes (Bowman & Seifert, 2011).”

“Research has demonstrated that college students generally have an imperfect ability to report learning and behavior (Porter, 2011, 2013).”

“In addition, even though student selfreported items are valuable in understanding college student experiences and outcomes, there are increasing debates about the validity of selfreported surveys of college students (Porter, 2011).”

“, how much time has passed since the experience) and accuracy of summation (e.g., what degree of specificity is required in counting quantity; Gonyea, 2005; Herzog & Bowman, 2011; Porter, 2011).”

“Although the reliability and validity of the NSSE has been challenged due to its low internal consistency for individual subscales and lack of theoretical background for the formation of them (see Porter, 2011), this reliability result with one latent variable avoids the concern.”

“Despite recent discussions regarding the limitations of CCSSE data (Dowd, Sawatzky, & Korn, 2011; Porter, 2011) these data represent the most comprehensive representation of FSI of community college students currently available.”

“While using self-reported meassures is very common in educational studies, there are some limitations as well, such as response bias, missing data, or accuracy (e.g., Porter, 2011).”

“However, survey fatigue among students (Klemenčič & Chirikov, 2015; Porter, 2004) can lead to careless or inaccurate responses and questionable or ‘minimal validity’ (Porter, 2011, p. 45).”

“However, there are ongoing arguments that criticize the validity and the reliability of self-report measures and especially their use in higher education research, and they call for alternative, rigorous measures to assess students’ attitudes and behaviors (e.g., Bowman, 2010a, 2011; Porter, 2011).”

“…in asynchronous settings requires a more robust methodological approach given the increasing importance and difference of students’ experiences in that setting (Porter 2011; Kim & Bateman 2010; Clayson 2009; Beuschel et al. 2003; Picciano 2002; Merisotis & Phipps 1999; Jaffee 1997; Marsh 1987).”

“The CCSSE has been critiqued for its use of self­reported measures (Porter, 2011).”

“Although self-reported data in general and NSSE data specifically have received some recent criticism (Bowman, 2011; Campbell & Cabrera, 2011; Porter, 2011), scholarly debate on the topic is far from conclusive, with several scholars (e.g., Ewell, McClenney, & McCormick, 2011; Kuh, Hu, & Vesper,…”

“and rightfully scrutinized regarding accuracy (Bowman & Seifert, 2011; Herzog & Bowman, 2011; Pascarella, 2001; Porter, 2011).”

“Although this tendency is seldom addressed in the literature (Clayson 2013; Marsh and Roche 2000; Porter 2011), it appears to be pervasive.”

“Last, we are not unaware of the limitations of quantitative survey data from nationally representative samples (see, for example, Porter, 2011).”

“one of the historical concerns about the NSSE and its derivatives has been that results are reported using ‘benchmarks’ – averaged scores for groups of items – that have limited empirical support (Porter 2011).”

“Based on this research, the concerns of self-report instruments are applicable for contexts in which respondents evaluate objective learning gains like conceptual understanding (Porter, 2011) and less so for the items assessing student perceptions of experiences, like the items in our survey.”

“Some have questioned the validity of the instruments used and the legitimacy of viewing the benchmark indicators as easy-to-implement proxies for learning gain (Porter, 2011).”

“…the proper nouns used in questionnaire, like risk, value, volatility, exchange rate, deposit insurance, transaction or inspection motives for the holding of money, are fundamental terms for economics (a subject for most first-year college students), so the table was later deleted (Porter, 2011).”

“We took care to https://doi.org/10.20429/ijsotl.2017.110213 design our instrument in a way that avoided “educational jargon” (Porter, 2011, p. 53) that may confuse respondents The fall 2012 pilot included a survey and interviews of first-year writers at State U alone, which helped us to test and…”

“Concerns regarding student self-reports (Bowman, 2010, 2011) and survey research with undergraduate students generally (see Porter, 2011, 2013) are well documented within the field of higher education, further challenging measurement validity in an undergraduate survey-based context.”

“While self-reported data offers the advantage of gaining quick access to participants from a large population [13], we acknowledge that the accuracy of self-reporting has been debated in research literature [14][15].”

“Porter (2011) discusses the difficulties of obtaining reliable student reports of income, concluding that the question of whose income is relevant (the student’s only or the student’s family’s), the common lack of knowledge by the student about the family’s income, and the poor recollection or…”

“…for assessing student engagement (Olivas, 2011)—specifically, questions concerning the ability of students to accurately respond to survey questions (Porter, 2011), the survey’s consideration of “intercultural effort” (Dowd, Sawatzky, & Korn, 2011), and the CCSSE’s benchmarks of effective…”

“In terms of data collection, prior research has critiqued students’ self-reporting of their educational experiences as a way to understand teaching practices (Porter 2011; Campbell 2015).”

“Third, student respondents may interpret the meaning of the NSSE questionnaire items differently, and they may not encode and report accurately on their previous behavior and events (Campbell & Cabrera, 2011; Porter, 2011).”

“Doubt also exists regarding the psychometric properties of the five benchmarks of effective educational practice (Esquivel 2011; Lutz and Culver 2010; Porter 2011).”

“Porter 33 called this the “correlation fallacy in validity research”.”

“The NSSAP was developed within the guidance of the classical measurement model, which purports that scale items are comparable indicators and indirect measurements of an underlying, latent variable (Deville’s, 2012). For the NSSAP, each competency and subscale corresponds to a unique latent variable (e.g., advising an…”

“As suggested by Porter (2011), surveys of undergraduate students have become one of the largest and most frequently used data sources and play increasingly important role in evaluating institutional performance.”

“Although criticisms continue to indict the validity and utility of studying student perceptions of learning, especially through the use of self-report measures (Bowman, 2013; Bowman & Seifert, 2011; Porter, 2011), we offer that student perceptions are important dimensions of learning—adhering closely to the argument a…”

“…studied, how many papers they wrote, and how often they experienced certain cognitively challenging instructional practices; students may be challenged to accurately report these behaviors, especially when the asked to recall behaviors over a prolonged period of time (Porter, 2011; Ross, 1989).”

“This is a plausible explanation, since the early studies on motives to teach have shown that the subject is prone to socially desirable responses – see e.g. Porter and Freeman (1986) and Porter (2011).”

“Data collection does not suffer from possible recall biases, which has been noted as a limitation of survey responses (Porter, 2011, 2013).”

“As with other surveys, there is a potential for measurement error due to social desirability of responses in various categories (Fowler, 2008; Porter, 2011).”

“Survey methodology used in this study relies on self-reported data for all measures, and the possibility of social desirability, poor understanding of survey questions, and inaccurate retrieval of past events might undermine the validity of survey data (Porter, 2011).”

“Limitations Despite their widespread use, some scholars have raised issues about the validity of self-reports regarding engagement behaviors and gains (e.g., Bowman, 2010; Campbell & Cabrera, 2011; McCormick & McClenney, 2012; Porter, 2011, 2013).”

“Porter (2011) describes the multiple conundrums involved in the survey research response process: When confronted with a question, respon dents must understand the individual terms and their combined meaning with the result that they can identify the information sought.”

“A number of researchers such as Buckley (2014) and Porter (2011) have argued that instruments like the NSSE measure student engagement only by proxy to any direct and genuine measure of student development.”

“Social desirability problems that are pointed out in student surveys (Porter, 2011) would likely be the same for either vague quantifiers or open-ended answers, as higher (lower) on the scale or higher”