Do College Student Surveys Have Any Validity?

“Social desirability bias, or providing survey responses that make the respondent look favorable to others, has been extensively documented in the field of survey methodology (see, for example, DeMaio, 1984; Nederhof, 1985), but has been little studied in the area of college student survey response.”

“Cohen’s rule of thumb for effect sizes is now recognized as rigid and, for the most part, useful only for power analyses (McCartney & Rosenthal, 2000); researchers instead rely on the research context to determine the substantive relevance of an effect size.”

“…psychiatric diagnosis in first-degree relatives as assessed by 2 Minimum age was set at 24 to limit the over-representation of university students – a demographic highly prevalent on the Prolific platform (Prolific, n.d.). and frequently associated with biases (Porter, 2011).”

“While there is no known method of accurately measuring these constructs directly, self-reported measures may suffer from internal differences and distorted memories of the survey taker, thus limiting their validity (Porter, 2011).”

“Broadly, survey research is criticized for reliance on self-reported student data (Porter, 2011), lack of convergence between survey data and objective learning measures (Bowman, 2011), and the color-blind framing of these tools (Dowd et al., 2011).”

“A potential limitation of our study is reliance on self-reported learning, which has been questioned for its validity (Porter, 2011).”

“Surveys assume students can accurately recall and report their learning behaviors and interactions, yet research has shown this is not always the case [14].”

“In particular, bias in self-evaluation (Koriat & Bjork, 2005), inaccuracy in judging one’s learning status and outcomes, and difficulty in estimating learning progress over time (Porter, 2011) are common challenges.”

“…individuals who are worried about their performance may provide a more precise assessment of their competence when compared to those who are not (57) because respondents may actually treat the questions in the tool as attitudinal rather than factual, which they have done in clinical practice (58).”

“While self ‐ report is appropriate for determining perceptions, it is not free from threats to validity (Porter 2011).”

“Porter (2011) offered a critique of prevailing approaches for studying college students via four observations about the survey research literature.”

“Porter (2011) in his paper makes a great point on how many institutional databases have limits, hence, college student surveys have become one of our largest and most often used data sources.”

“Porter (2011) in the paper makes a great point on how many institutional databases have limits; hence, college student surveys have become one of our largest and most often used data sources.”

“However, the lack of difference between these two measures should be interpreted with caution (Porter, 2011) since this analysis has not investigated the independent (or common) bias or other error in these estimates.”

“It has been presented that [61] using data derived from student self-reported approaches may yield little in terms of information upon which to base improvement efforts.”

“Porter (2011) highlights that even simple elements, such as “had serious conversations with students”, can be interpreted differently.”

“conceptual domain of the NSSE; 2) College students must estimate their behaviors, creating the potential for errors; 3) The dimensional structure of the NSSE has not been replicated by other researchers and lacks reliability; and 4) Instrument scales are largely uncorrelated with objective external measures of student…”

“Selfreported achievement, as utilised by the AUSSE survey, may somewhat inflate the responses (Hagel et al., 2012; Porter, 2011).”