A Meta-analysis of University STEM Summer Bridge Program Effectiveness

Probably not a substantial claim: may not pass S3 (load-bearing).

First-semester GPA was a moderate predictor of whether students ultimately received a STEM degree in a sample of 1925 college students.

“In a sample of 1925 college students, first-semester GPA was a moderate predictor of whether students ultimately received a STEM degree (Crisp et al., 2009).”

• cites Crisp et al., 2009 sourceSupported

May combine two claims.

The log-odds ratio creates greater symmetry of the distribution of the outcome measures and centers it on 0, making the data more amenable to analyses.

“The log-odds ratio creates greater symmetry of the distribution of the outcome measures and centers it on 0 (Sterne et al., 2001), which makes the data more amenable to analyses.”

• cites Sterne et al., 2001 sourcePartly supported

  The “Funnel plots” section says that plotting odds ratios on a logarithmic scale places equal effects in opposite directions equidistant from the null value of 1 (zero on the log scale). It does not say that the transformation makes the distribution of outcome measures more symmetric or, for that reason, more amenable to analysis. (LLM, conf=0.84)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

The metafor package is a frequently used statistical package to fit fixed-, mixed-, and random-effects models to meta-analyses.

“All analyses were conducted in R statistical software using the metafor package, a frequently used statistical package to fit fixed-, mixed-, and random-effects models to meta-analyses (Viechtbauer, 2010).”

• cites Viechtbauer, 2010 sourcePartly supported

  The Conclusions (p. 42) state that metafor’s rma() function fits fixed- and random/mixed-effects models for meta-analysis in R. The Introduction (p. 2) also describes its fixed- and random-effects functions. Our copy does not establish that the package is frequently used. (LLM, conf=0.86)

  Experimental — accuracy not yet measured

May combine two claims.

The Meyerhoff program is extremely successful, and including it in the meta-analysis would only strengthen the findings regarding STEM bridge program effectiveness.

“Notably, the Meyerhoff program (see Maton et al., 2012) is extremely successful, and including it in our meta-analysis would only strengthen the findings regarding STEM bridge program effectiveness.”

• cites Maton et al., 2012 sourceSupported

May combine two claims.

The Meyerhoff Scholars Program is a comprehensive STEM program that far surpasses an intervention with a STEM bridge program as its primary element.

“The Meyerhoff program is a comprehensive STEM program that far surpasses an intervention with a STEM bridge program as its primary element (providing intensive, ongoing support for participants throughout all 4 years of university).”

• cites Maton et al., 2012 sourceSupported

May combine two claims.

Other studies used non-bridge underrepresented minority STEM students, or all other STEM students, as control group students.

“Other studies used non-bridge underrepresented minority STEM students, or in the broadest cases, all other STEM students, as control group students (e.g., Kopec and Blair, 2014).”

• cites Kopec and Blair, 2014Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Five programs used all other STEM or engineering students as a control, seven used a matched sample based on high school preparation, standardized test scores and/or demographic background, three used more specific STEM demographic groups, and one used all other students enrolled in precalculus.

“Five programs used all other STEM or engineering students as a control, seven programs used some sort of matched sample based on high school preparation, standardized tests scores and/or demographic background, three used more specific STEM demographic groups (two of underrepresented minority STEM students and one of female STEM students), and one program used all other students enrolled in precal”

May combine two claims.

The main effect of bridge program participation on first-year overall GPA was statistically and practically significant.

“The main effect of bridge program participation on GPA was statistically and practically significant, supporting hypothesis 1.”

• AssumptionThe authors state this as an assumption: Heterogeneity is a reasonable assumption in the current meta-analysis.
• AssumptionRests on an assumption: The standard deviations missing from studies that reported first-year overall GPA but not SD are similar enough to the SDs in the other studies to be imputed by their weighted average.

Probably not a substantial claim: may not pass S3 (load-bearing).

Eleven of the 16 programs incorporated some sort of tutoring arrangement through peers or the university's tutoring center.

“Eleven of the 16 programs incorporated some sort of tutoring arrangement through peers or the university’s tutoring center”

May combine two claims.

Of the studies that reported retention, only four were published in peer-reviewed journals, limiting the ability to find evidence of upward bias.

“However, of the studies that reported retention, only four were published in peer-reviewed journals, limiting our ability to find evidence of upward bias.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

In four programs, students paid some amount to attend; in the remaining 12 programs, the program covered all costs and in some cases provided stipends.

“In four of these programs, students paid some amount to attend. In the remaining 12 programs, the program covered all costs (and in some cases provided stipends).”

Probably not a substantial claim: may not pass S3 (load-bearing).

The first-year retention base rates in these studies were moderately high (weighted average retention rate across both groups was 76.1%).

“the first-year retention base rates in these studies were moderately high (the weighted average retention rate across both groups was 76.1%)”

May combine two claims.

The combination of services provided varied by program, as did the timing when students were offered these services (during or after the summer bridge program).

“although the combination of services provided varied by program, as did the timing when students were offered these services (i.e., during or after the summer bridge program).”

Probably not a substantial claim: may not pass S3 (load-bearing).

Ten programs described some sort of faculty or industry professional mentoring arrangement during the summer or afterward.

“Ten programs described some sort of faculty or industry professional mentoring arrangement during the summer or afterward”

Probably not a substantial claim: may not pass S3 (load-bearing).

The 16 studies in this analysis yielded 25 different samples.

“The 16 studies in this analysis yielded 25 different samples.”

May combine two claims.

Of the studies that reported GPA, only three were published in peer-reviewed journals, limiting interpretability of the publication-bias result.

“however, of the studies that reported GPA, only three were published in peer-reviewed journals, meaning interpretability of this result is limited.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Nine programs provided students with research opportunities during the summer or afterward.

“Nine programs provided students with research opportunities during the summer or afterward.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Of these studies, there were 13 first-year overall GPA effects and 19 first-year university retention effects.

“Of these studies, there were 13 first-year overall GPA effects and 19 first-year university retention effects”

Classified as probably not a substantial claim.

In total, 16 studies comprising 25 samples were used in the meta-analysis.

“In total, 16 studies comprising 25 samples were used in the meta-analysis.”

May combine two claims.

At least 11 effects would be necessary to conduct a meta-analysis to exceed a statistical power of at least 0.70 (and 13 to exceed 0.80) to detect the meta-analytic mean.

“we estimated that at least 11 effects would be necessary to conduct a meta-analysis to exceed a statistical power of at least 0.70 (and 13 to exceed 0.80) to detect the meta-analytic mean (see Borenstein et al., 2011).”

• cites Borenstein et al., 2011Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Underrepresented minority students are more likely than White or Asian-American students to be first in their immediate families to attend college and are less likely to have economic or social support at home.

“Underrepresented minority students are also more likely than White or Asian-American students to be the first in their immediate families to attend college and are less likely to have the economic or social support at home compared with later-generation college students (Jack, 2019).”

• cites Jack, 2019 sourceNot checked — no full text of the source

May combine two claims.

Limited empirical research on summer bridge programs is available, and much of it is descriptive accounts, qualitative results, and literature reviews rather than systematic and quantitative evaluations of bridge program success.

“limited empirical research is available, and much of this research is in the form of highly descriptive accounts, qualitative results, and literature reviews (Sablan, 2014; Kitchen et al., 2018) rather than systematic and quantitative evaluations of bridge program success (Gullatt and Jan, 2003).”

• cites Sablan, 2014 sourceNot checked — no full text of the source
• cites Kitchen et al., 2018 sourceNot checked — no full text of the source
• cites Gullatt and Jan, 2003Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

Higher first-year GPA predicted whether students would be retained in engineering into their second year in a study of 137 freshman engineering students.

“In a study of 137 freshman engineering students, higher first-year GPA predicted whether students would be retained in engineering into their second year in the program (Burtner, 2004).”

• cites Burtner, 2004Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

First-year GPA predicted STEM retention after three semesters in an analysis of approximately 1200 STEM first-generation college students.

“in an analysis of ∼1200 STEM first-generation college students, Dika and D’Amico (2016) found that first-year GPA predicted STEM retention after three semesters.”

• cites Dika and D’Amico (2016) sourceNot checked — no full text of the source

Classified as probably not a substantial claim.

Almost a third of matriculating STEM students leave STEM by the end of their first year in college.

“Research suggests that almost a third of matriculating STEM students leave STEM by the end of their first year in college (National Science Board, 2018).”

• cites National Science Board, 2018Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

Time-series designs, in which data are collected at multiple time points pre and post intervention, can increase the strength of conclusions drawn about the impact of bridge program participation.

“time-series designs, in which data are collected at multiple time points pre and post intervention in order to see the impact of the intervention beyond underlying group trends (Grimshaw et al., 2000) can increase the strength of conclusions drawn about the impact of bridge program participation.”

• cites Grimshaw et al., 2000 sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S4 (based).

The wide array of bridge program studies meta-analyzed are representative of what is available in the literature.

“We made every attempt to be comprehensive, and we can say with some confidence that the wide array of bridge program studies we meta-analyzed are representative of what is available in the literature.”

Probably not a substantial claim: may not pass S3 (load-bearing).

A retrospective power analysis using d = 0.34 found the GPA analysis was appropriately powered (P = 0.99) to detect differences of this magnitude.

“A retrospective power analysis using this effect size (d = 0.34) found that this analysis was appropriately powered (P = 0.99) to detect differences of this magnitude (Harrer et al., 2019).”

• cites Harrer et al., 2019Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

There is likely selection bias associated with the quasi-experimental approaches used in the included studies, and bridge participants may differ from nonparticipants in important ways that cannot be controlled.

“That is, there is likely selection bias associated with the quasi-experimental approaches used in the studies included in this meta-analysis, and students who participate in bridge programs may differ from those who do not in some important ways that we cannot control.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Traditional liberal arts colleges tend to not offer professional, vocational, or applied majors, including STEM majors such as engineering.

“Although traditional liberal arts colleges tend to not offer professional, vocational, or applied majors (including STEM majors such as engineering; Roche, 2010)”

• cites Roche, 2010 sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

University-specific gateway courses and class performance may be more straightforward for administrators to track, but these outcomes are among the least generalizable to other universities, which have different professors, class syllabi, and student populations.

“University-specific gateway courses and class performance may be more straightforward for administrators to track, but these outcomes are among the least generalizable to other universities, which have different professors, class syllabi, and student populations.”

May combine two claims.

The studies included in this meta-analysis did not systematically control for student motivation, self-efficacy, interest in science, or other variables that might influence performance through random assignment, so the effect of bridge performance on GPA and retention cannot be isolated.

“One caveat, however, is that we cannot isolate the effect of bridge performance on student GPA and retention, because the studies included in this meta-analysis did not systematically control for student motivation, self-efficacy, interest in science, or other variables that might influence performance through random assignment.”

Probably not a substantial claim: may not pass S3 (load-bearing).

In cases of discrepancy between the two research assistants (which occurred in three out of 16 cases), B.C.B. made the final determination on the appropriate coding.

“In cases of discrepancy between the two research assistants (which occurred in three out of 16 cases), B.C.B. made the final determination on the appropriate coding.”

Classified as probably not a substantial claim. May combine two claims.

The small number of studies published in peer-reviewed journals limited the implications of the findings, meaning further exploration of publication bias in STEM bridge program research is necessary.

“However, we were limited in the implications of our findings by the small number of studies published in peer-reviewed journals, meaning that further exploration of the extent of publication bias in STEM bridge program research is necessary.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Randomized experimental designs are generally the gold standard for experimental research.

“although randomized experimental designs are generally the “gold standard” for experimental research (Rogers and Révész, 2019)”

• cites Rogers and Révész, 2019Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Programs were counted as including an element only if it was explicitly stated in the publication, but it is conceivable that programs contained elements not described therein.

“We only counted programs as including an element if it was explicitly stated in the publication, but it is conceivable that programs contained elements not described therein.”

May combine two claims.

Because bridge program participation impacted retention, which college retention models generally regard as the result of academic performance, the results provide evidence of a longer-term impact of the bridge program beyond just increasing GPAs.

“The fact that bridge program participation impacted students’ retention, which college retention models generally regard as the result of academic performance (Tinto, 1999), provides evidence of a longer-term impact of the bridge program beyond just increasing GPAs.”

• cites Tinto, 1999 sourceNot checked — no full text of the source

May combine two claims.

Kulik et al. found in their meta-analysis of college programs for high-risk students that only 60 (less than 12%) of the 504 articles identified met their inclusion criteria, with a substantial portion failing to provide results for control groups or lacking appropriate control groups.

“Kulik et al. (1983), who found in their meta-analysis of college programs for high-risk students that only 60 (less than 12%) of the 504 articles the authors identified met their inclusion criteria, with a substantial portion failing to provide results for control groups or lacking appropriate control groups.”

• cites Kulik et al. (1983) sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing).

Interventions such as mandatory learning communities for transfer students might reduce attrition when students transfer to a 4-year college or university.

“Interventions such as mandatory learning communities for transfer students might reduce attrition when students transfer to a 4-year college or university (e.g., Scott et al., 2017).”

• cites Scott et al., 2017 sourceNot checked — no full text of the source

Classified as probably not a substantial claim. May combine two claims.

Two research assistants independently coded the sample size of the bridge and control groups; the overall first-year GPA of each group, the first-year university retention rate of each group, or both; and whether the study was published in a peer-reviewed journal.

“Two research assistants independently coded the sample size of the bridge and control groups; the overall first-year GPA of each group, the first-year university retention rate of each group, or both; and whether the study was published in a peer-reviewed journal.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Published studies reported a ratio of supported to unsupported hypotheses more than twice as high as dissertations, presumably because peer-reviewed publications are more likely to report significant results.

“This trend aligns with findings such as those in O’Boyle et al.’s (2017)’s management review, which found that published studies reported a ratio of supported to unsupported hypotheses that was more than twice as high as those in dissertations, presumably because peer-reviewed publications are more likely to report significant results.”

• cites O’Boyle et al.’s (2017) sourceNot checked — no full text of the source

May combine two claims.

Many of the bridge groups were relatively small (median size 75 students), meaning some caution should be used in extrapolating the retention findings.

“many of the bridge groups were relatively small (the median size was 75 students), meaning that some caution should be used in extrapolating these findings.”

Probably not a substantial claim: may not pass S3 (load-bearing).

About 20% of a nationally representative sample of college students entering a 4-year institution as STEM majors in 2003 dropped out of college rather than switching to a non-STEM major.

“∼20% of a nationally representative sample of college students entering a 4-year institution as STEM majors in 2003 dropped out of college rather than switching to a non-STEM major (Chen, 2013).”

• cites Chen, 2013Not checked — no full text of the source

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Students usually opt into bridge programs, making random assignment impossible and selection bias likely.

“students usually opt into bridge programs, making random assignment impossible and selection bias likely.”

Probably not a substantial claim: may not pass S4 (based).

This is the first systematic quantitative review of the effectiveness of STEM bridge programs.

“To our knowledge, this is the first systematic quantitative review of the effectiveness of STEM bridge programs.”

May combine two claims.

Matched sampling attempts to overcome confounding when initial group differences are not controlled for and provides an approach close to true experimental randomization.

“matched sampling attempts to overcome the confounding that may occur when initial group differences are not controlled for (Campbell and Stanley, 1967) and provides an approach that is close to true experimental randomization (Stuart and Rubin, 2008)”

• cites Campbell and Stanley, 1967Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Stuart and Rubin, 2008Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.