Methodological Choices When Assessing Summer Bridge Programs in STEM Majors: A Scoping Review

This imbalance toward nonexperimental designs aligns with previous attempts to systematize the body of evidence.

“This imbalance toward nonexperimental designs aligns with previous attempts to systematize the body of evidence (Bradford et al., 2021).”

• cites Bradford et al., 2021 sourceSupported

Lack of evidence could lead to erroneous conclusions due to inadequate comparisons.

“Finally, this lack of evidence could lead to erroneous conclusions due to inadequate comparisons (Slavin, 2020;Odeleye & Santiago, 2019;Baron, 2004).”

• cites Slavin, 2020 sourceNot checked — no full text of the source
• cites Odeleye & Santiago, 2019 sourceSupported
• cites Baron, 2004Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Three studies considered 10 or more cohorts (Murphy et al., 2010; Oppenheimer et al., 2020; Walter & Wade, 2024).

“Notably, one study considered 6 cohorts of students (Evans, 2014) and three studies considered 10 or more cohorts (Murphy et al., 2010;Oppenheimer et al., 2020;Walter & Wade, 2024).”

• cites Murphy et al., 2010 sourceSupported
• cites Oppenheimer et al., 2020 sourceNot checked — no full text of the source
• cites Walter & Wade, 2024 sourceSupported

These results provide limited evidence supporting a causal interpretation of SBPs' impacts on academic performance.

“As research implications, these results provide limited evidence supporting a causal interpretation of SBPs' impacts on academic performance (Loeb et al., 2017;Odeleye & Santiago, 2019;Baron, 2004;Thompson, 2006).”

• cites Loeb et al., 2017Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Odeleye & Santiago, 2019 sourcePartly supported

  The Discussion reports evidence that summer bridge programs improve academic readiness, but it also says many studies lack matched control groups, making it impossible to determine whether the programs truly affected academic performance. The review supports caution about causal claims, not the assertion that its findings establish even limited causal evidence. (LLM, conf=0.82)

  Experimental — accuracy not yet measured
• cites Baron, 2004Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• AssumptionRests on an assumption: Experimental design and matching/control of background variables constitute higher methodological rigor for assessing SBP effects.

Kornreich-Leshem et al. (2015) considered SAT Math score, high school GPA, gender, ethnicity, and major as covariates to be matched between treatment and control groups.

“in their multivariate matching method, Kornreich-Leshem et al. (2015) considered SAT Math score, high school GPA, gender, ethnicity, and major as the covariates to be matched between treatment and control groups”

• cites Kornreich-Leshem et al. (2015) sourceSupported

Wischusen and Wischusen (2007) and Jackson et al. (2024) considered high school GPA, ACT or SAT scores, major, and gender.

“Wischusen and Wischusen (2007) and Jackson et al. (2024) also considered high school GPA, ACT or SAT scores, major, and gender.”

• cites Wischusen and Wischusen (2007) sourceSupported
• cites Jackson et al. (2024) sourceNot checked — no full text of the source

Kornreich-Leshem et al. (2015) found no impact on GPA.

“while the other found no impact on GPA (Kornreich-Leshem et al., 2015).”

• cites Kornreich-Leshem et al., 2015 sourceSupported

Descriptive studies analyzed academic performance by comparing experimental and control groups using traditional descriptive methods.

“Those studies analyzed the academic performance of the SBP participants by comparing the experimental and control groups using traditional descriptive methods.”

Other studies used different techniques to match a student who did not participate with a participant who is similar in terms of background variables.

“Other studies used different techniques to match a student who did not participate with a participant who is similar in terms of background variables.”

Eight programs were restricted to specific subgroups, such as female students, particular ethnic groups, socioeconomically disadvantaged students, or first-generation students.

“Seven programs were open to all incoming students within the targeted major, and eight programs were restricted to specific subgroups, such as female students (e.g., Fletcher et al., 2001), particular ethnic groups (e.g., Pritchard et al., 2016), socioeconomically disadvantaged students (e.g., Oppenheimer et al., 2020), or firstgeneration students (e.g., Tomasko et al., 2016).”

Only one program required participation as a condition for college admission.

“Regarding program voluntariness, we identified only one program that required participation as a condition for college admission (Cançado et al., 2018;J. Reisel et al., 2011J. Reisel et al., , 2012J. Reisel et al., , 2014)).”

Studies considering confounding variables used previous academic performance as a covariate or controlled variable.

“Those considering confounding variables used previous academic performance as a covariate or variable to be controlled in their analysis.”

There is a lack of studies that used an experimental or observational approach while also controlling for participants' background variables.

“We found a lack of studies that used an experimental or observational approach while also controlling for participants' background variables.”

Most studies reported on a single cohort (n = 10).

“Regarding the cohorts, most studies reported on a single cohort (n = 10).”

Two of the highest-rigor programs focused on Engineering design.

“For instance, two programs focused on Engineering design”

Of the 21 observational studies, three used a matching technique to obtain two balanced groups because they did not use randomization.

“Of the 21 studies, three used a matching technique to obtain two balanced groups because they did not use randomization to allocate participants to the groups (Kornreich-Leshem et al., 2015;Wischusen & Wischusen, 2007;Jackson et al., 2024).”

There is a slight tendency towards retention or graduation rates in observational studies that did not control for confounding variables.

“whereas there is a slight tendency towards retention or graduation rates in the observational studies that did not control for confounding variables.”

ANOVA or T-test was used for numerical outcomes such as GPA.

“ANOVA or T-test was used for numerical outcomes, such as GPA (e.g., Steadman et al., 2014)”

One study considered 6 cohorts of students (Evans, 2014).

“Notably, one study considered 6 cohorts of students (Evans, 2014) and three studies considered 10 or more cohorts (Murphy et al., 2010;Oppenheimer et al., 2020;Walter & Wade, 2024).”

• cites Evans, 2014Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Four of the programs with academic readiness criteria targeted high-performing students.

“four of these programs targeted high-performing students (Jefferson et al., 2013;Steadman et al., 2014;Wheatland, 2002).”

Two SBPs targeted health science majors: biomedical basic science and nursing.

“two targeted health science majors: biomedical basic science (Oppenheimer et al., 2020) and nursing (Pritchard et al., 2016)”

Due to the study design, there is a clear imbalance in the subjects considered in the treatment and control groups.

“Also, due to the study design, there is a clear imbalance in the subjects considered in the treatment and control groups.”

For studies that reported the number of subjects in the control group, 20 studies had more than 100 participants.

“For those studies that reported the number of subjects in the control group, 20 studies had more than 100 participants.”

Chi-squared test was used for proportions such as graduation rate.

“whereas the Chi-squared test was used for proportions, such as graduation rate (e.g., Tomasko et al., 2016).”

Wischusen et al. (2010) considered students in the subsequent course in the fall and those on the program's waiting list as controls.

“Wischusen et al. (2010) considered students who were in the subsequent course in the fall, as well as those who remained on the program's waiting list.”

• cites Wischusen et al. (2010) sourceNot checked — no full text of the source

Academic performance was often measured using external instruments such as SAT scores, ACT scores, or high school GPA.

“Academic performance was often measured using external instruments such as SAT scores (e.g., Doerr et al., 2014), American College Testing (ACT) scores (e.g., Cançado et al., 2018), or high school GPA (e.g., Chingos et al., 2017).”

Studies that used matching techniques included a rather equal proportion of participants and non-participants.

“As expected, the studies that used matching techniques (Kornreich-Leshem et al., 2015;Jackson et al., 2024;Wischusen & Wischusen, 2007) included a rather equal proportion of participants and non-participants.”

For numerical variables, mean (and sometimes standard deviation) and median statistics were used to describe and compare groups.

“For numerical variables, such as GPA or exam scores, the mean (and sometimes the standard deviation) and median statistics were used to describe and compare the two groups (e.g., Pritchard et al., 2016).”

One SBP focused on biology students.

“one focused on biology students (Wischusen et al., 2010;Wischusen & Wischusen, 2007)”

Nine SBPs included STEM majors.

“Nine SBPs included STEM majors, two were oriented to SEM (Science, Engineering and Math) (Jura & Gerhardt, 2022;Wheatland, 2002), two targeted health science majors: biomedical basic science (Oppenheimer et al., 2020) and nursing (Pritchard et al., 2016), one focused on biology students (Wischusen et al., 2010;Wischusen & Wischusen, 2007), two were open to any student requiring an introductory”

An equal number of SBPs (n = 13) lasted one or two weeks and five or six weeks.

“Program duration ranged from five days to ten weeks, with an equal number of SBPs (n = 13) lasting one or two weeks and five or six weeks.”

Most analytical papers assessed the outcome of interest while controlling for one or more variables.

“Within the analytical papers, most studies assessed the outcome of interest while controlling for one or more variables.”

Program duration ranged from five days to ten weeks.

“Program duration ranged from five days to ten weeks, with an equal number of SBPs (n = 13) lasting one or two weeks and five or six weeks.”

Only two studies randomly assigned participants to the experimental or control group.

“There is almost no experimental study design in the analytical studies, as only two studies randomly assigned participants to the experimental or control group (Dockter et al., 2017;Chingos et al., 2017).”

13 studies had more than 100 participants in the treatment group.

“Of the studies considered, 13 had more than 100 participants in the treatment group.”

This review examined 37 studies that assessed the academic outcomes of SBPs intended to support first-year incoming students of STEM majors.

“This review examined 37 studies that assessed the academic outcomes of SBPs intended to support first-year incoming students of STEM majors.”

10 studies' inferential analyses did not involve any controlling variables.

“However, we identified 10 studies whose inferential analyses did not involve any controlling variables.”

Jura and Gerhardt (2022) considered three groups: all students taking subsequent courses in fall and spring, borderline cases for SBP based on readiness exam, and high scorers placed in a higher math course.

“Jura and Gerhardt (2022) considered three groups: the first group included all students taking the subsequent courses in the fall and spring terms, the second consisted of students who were borderline cases for SBP based on the readiness exam, and the third included students who scored high in the readiness exam and were placed in a higher math course.”

• cites Jura and Gerhardt (2022) sourceNot checked — no full text of the source

For studies that did not control for confounding variables, the analytical approach primarily involved statistical tests comparing means or proportions between groups.

“For studies that did not control for confounding variables, the analytical approach primarily involved statistical tests comparing means or proportions between groups.”

There were 23 analytical studies.

“There were 23 analytical studies, most of which used an observational design (n = 21).”

Schmid et al. (2012) controlled for students' self-reported experiences and prior knowledge about the subject taught in the SBP.

“In contrast, in the study by Schmid et al. (2012), the variable controlled was students' self-reported experiences and prior knowledge about the subject to be taught in the SBP”

• cites Schmid et al. (2012) sourceNot checked — no full text of the source

Most analytical studies used an observational design (n = 21).

“There were 23 analytical studies, most of which used an observational design (n = 21).”

This review revealed a wide variety of SBP implementations, as programs across institutions differ in duration, target population, ultimate goals, and program structure.

“For practice implications, this review revealed a wide variety of SBP implementations, as programs across institutions differ in duration, target population, ultimate goals, and program structure.”

Seven of those studies had more than 1000 subjects in the aggregated control groups.

“Seven of those had more than 1000 subjects in the aggregated control groups.”

Studies measuring academic performance usually collected data at the end of the first semester or year.

“These studies usually collected data at the end of the first semester or year.”

Two of the 16 studies reporting retention or graduation rates reported both retention and graduation.

“Of the 16 studies reporting retention or graduation rates, 15 focused on retention; two reported both (Cançado et al., 2018;Oppenheimer et al., 2020); and only one assessed graduation (Murphy et al., 2010).”

Seven programs were open to all incoming students within the targeted major.

“Seven programs were open to all incoming students within the targeted major, and eight programs were restricted to specific subgroups, such as female students (e.g., Fletcher et al., 2001), particular ethnic groups (e.g., Pritchard et al., 2016), socioeconomically disadvantaged students (e.g., Oppenheimer et al., 2020), or firstgeneration students (e.g., Tomasko et al., 2016).”

All other analytical studies were classified as observational (or non-experimental).

“All other analytical studies were classified as observational (or non-experimental).”

More than twenty selected studies did not control for any variables in their analysis.

“More than twenty selected studies did not control for any variables in their analysis.”

Most programs included math classes (n = 20).

“Consequently, most programs included math classes (n = 20).”

The academic evaluation relies on graduation or retention rates, GPA, and course performance, as previous research also pointed out.

“We also found that the academic evaluation relies on graduation or retention rates, GPA and course performance, as previous research also pointed out.”

Two SBPs were open to any student requiring an introductory chemistry course.

“two were open to any student requiring an introductory chemistry course (Schmid et al., 2012;Eitemüller & Habig, 2020)”

Studies assessing graduation rates collected data longitudinally, from 3 to 6 years after SBP participation.

“Accordingly, the studies assessing graduation rates collected data longitudinally, from 3 to 6 years after SBP participation; the rest of the studies collected data at the end of the fall semester (e.g., Steadman et al., 2014), at the end of the first year (e.g., Pritchard et al., 2016), or after four years (e.g., Tomasko et al., 2016).”

Oppenheimer et al. (2020) considered two control groups: students accepted but did not attend, and students who applied but were not selected.

“Oppenheimer et al. (2020) considered two groups as controls to compare to the treatment; one was students who were accepted into the program but did not attend, and the second was the group of students who applied but were not selected.”

• cites Oppenheimer et al. (2020) sourceNot checked — no full text of the source

Four programs based their eligibility on enrollment in a single course during the first or second semester, such as Calculus I or Linear Algebra or introductory Biology Science.

“Finally, four programs based their eligibility on enrollment in a single course during the first or second semester, such as Calculus I or Linear Algebra (Zorn et al., 2011) or introductory Biology Science (Wischusen et al., 2010;Wischusen & Wischusen, 2007).”

The evidence reported primarily addresses short-term effects of SBPs.

“The evidence reported primarily addresses short-term effects of SBPs, with the most common outcome being performance in subsequent courses related to summer content (n = 21), followed by retention or graduation rates (n = 20) and GPA (n = 12).”

On average, there are 10 times as many subjects in the control group as in the treatment group.

“On average, there are 10 times as many subjects in the control group as in the treatment group.”

Only one of the 16 studies reporting retention or graduation rates assessed graduation.

“Of the 16 studies reporting retention or graduation rates, 15 focused on retention; two reported both (Cançado et al., 2018;Oppenheimer et al., 2020); and only one assessed graduation (Murphy et al., 2010).”

Only four programs outside the US met the inclusion criteria: two in Germany, one in Australia, and one in Italy.

“Although we identified several programs across different regions in the screening phase, only four met the inclusion criteria: two in Germany (Eitemüller & Habig, 2020;Zorn et al., 2011), one in Australia (Schmid et al., 2012), and one in Italy (Bozzi et al., 2019).”

The evidence on SBP outcomes was mainly analytical, i.e., observational or experimental.

“Considering study design, the evidence was mainly analytical, i.e., observational or experimental.”

14 SBP studies were classified as descriptive.

“We classified 14 SBP studies as descriptive.”

The only two studies that used an experimental design were fully online (Dockter et al., 2017; Chingos et al., 2017).

“For this review, the only two studies that used an experimental design were fully online (Dockter et al., 2017;Chingos et al., 2017).”

• cites Dockter et al., 2017 sourceNot checked — no full text of the source
• cites Chingos et al., 2017 sourceNot checked — no full text of the source

Of the 16 studies reporting retention or graduation rates, 15 focused on retention.

“Of the 16 studies reporting retention or graduation rates, 15 focused on retention; two reported both (Cançado et al., 2018;Oppenheimer et al., 2020); and only one assessed graduation (Murphy et al., 2010).”

The eligibility criteria for participation varied significantly across programs.

“The eligibility criteria for participation varied significantly across programs.”

Gleason et al. (2010) compared the treatment group with students with similar math placement scores.

“Finally, Gleason et al. (2010) compared the treatment group with students with similar math placement scores.”

• cites Gleason et al. (2010)Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Ultimately, 37 reports were extracted (33 from the database search and 4 from the supplementary search).

“Ultimately, 37 reports were extracted (33 from the database search and 4 from the supplementary search).”

Among two experimental studies on online SBPs, Chingos et al. (2017) reported that SBP did not improve math skills.

“Similarly, among two experimental studies on online SBPs, one reported that SBP did not improve math skills (Chingos et al., 2017)”

• cites Chingos et al., 2017 sourceNot checked — no full text of the source

Observational studies with controlling variables provide evidence for all three outcomes equally.

“The observational studies with controlling variables provide evidence for all three outcomes equally”

For the first outcome, the program's evaluation was based on students' performance in the corresponding course during the first term or year.

“Regarding the first outcome which focuses on subsequent course performance, the program's evaluation was based on students' performance in the corresponding course during the first term or year.”

Most programs (n = 18) set criteria related to academic readiness, such as high school grades, standardized entrance tests, or placement tests.

“Most programs (n = 18) set criteria related to academic readiness, such as high school grades (e.g., Jackson et al., 2024), standardized entrance tests (e.g., Lecocke et al., 2019), or placement tests (e.g., Chingos et al., 2017); four of these programs targeted high-performing students (Jefferson et al., 2013;Steadman et al., 2014;Wheatland, 2002).”

Some studies, including experimental ones, considered confounding variables that could explain students' academic performance besides SBP participation.

“Some studies (including the experimental ones) also considered confounding variables that could explain students' academic performance besides participation in the SBP.”

The remaining studies reporting retention or graduation rates collected data at the end of the fall semester, at the end of the first year, or after four years.

“Accordingly, the studies assessing graduation rates collected data longitudinally, from 3 to 6 years after SBP participation; the rest of the studies collected data at the end of the fall semester (e.g., Steadman et al., 2014), at the end of the first year (e.g., Pritchard et al., 2016), or after four years (e.g., Tomasko et al., 2016).”

The five studies with the highest methodological rigor (the two experimental designs and three observational designs with matching techniques) are quite diverse in terms of implementation and effectiveness.

“we note that the five studies with the highest methodological rigor, the two experimental designs and three observational designs with matching techniques to balance the participants' background variables, are quite diverse in terms of implementation and effectiveness.”

• AssumptionRests on an assumption: Experimental design and matching/control of background variables constitute higher methodological rigor for assessing SBP effects.

Three studies addressed the limitation of lack of randomization by controlling for confounding variables.

“Finally, three studies addressed the limitation of the lack of randomization in the groups by controlling for confounding variables”

Variances in SBP implementation, such as format, social component, and duration, differ significantly across institutions depending on their goals.

“Variances in implementation, such as format, social component, and duration, differ significantly across institutions depending on their goals.”

The literature shows little consistency across definitions of SBPs.

“However, the literature shows little consistency across definitions of SBPs (Sablan, 2014) because each program is tailored to an HEI's specific goals and constraints, making each SBP unique.”

• cites Sablan, 2014 sourceNot checked — no full text of the source

Characteristics of SBP participation, administration, funding, and curriculum can impact program evaluations.

“Those characteristics can impact program evaluations (Baron, 2004;Calcagno & Long, 2009).”

• cites Baron, 2004Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Calcagno & Long, 2009Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Little attention has been given to program duration and the time allocated to specific activities in SBPs.

“Finally, little attention has been given to program duration and the time allocated to specific activities (Sablan, 2014).”

• cites Sablan, 2014 sourceNot checked — no full text of the source

Some studies considered students who applied to the program but were not accepted, thereby reducing selection bias surrounding these programs.

“Some studies considered students who applied to the program but were not accepted, thereby reducing selection bias surrounding these programs.”

Other outcomes, such as STEM identity, motivation, and self-efficacy, are missing from this review.

“Other outcomes, such as STEM identity, motivation, and self-efficacy are missing and could provide new insights into how SBPs impact other aspects of students' trajectories.”

A study by Pain (2016) found a decrease in math and English course grades after remediation became voluntary.

“For example, a study conducted by Pain (2016) explored the relationship between the voluntary nature of remediation programs and students' grades and completion rates, finding a decrease in math and English course grades after remediation became voluntary.”

• cites Pain (2016) sourceNot checked — no full text of the source

This review highlights the limitations of the evidence to guide policy and practice.

“Thus, this review highlights the limitations of the evidence to guide policy and practice.”

An RCT conducted by Barnett et al. (2012) in eight HEIs found that the programs did not affect the average number of credits attempted or earned.

“A Randomized Control Trial (RCT) conducted by Barnett et al. (2012) in eight HEIs found that the programs did not affect the average number of credits attempted or earned.”

• cites Barnett et al. (2012)Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

The review found a lack of studies that use an experimental or non-experimental approach and control for participants' background variables.

“Our findings indicate a lack of studies that use an experimental or non-experimental approach and control for participants' background variables.”

Differences in the implementation of SBPs may pose challenges to evaluating them as a single, uniform intervention.

“In this section, we outline some differences in the implementation of SBPs may pose challenges to evaluating them as a single, uniform intervention.”

The same RCT improved completion rates for first-year college-level courses in math and writing.

“However, they did improve completion rates for first-year college-level courses in math and writing, but no evidence suggested an effect on persistence.”

• cites Barnett et al. (2012)Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Most SBPs focus on mathematics and English because these subjects are foundational across most activities in HEIs.

“Most SBPs focus on mathematics and English because these subjects are foundational across most activities in HEIs (Henry et al., 2014;Peters, 2023).”

• cites Henry et al., 2014Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Peters, 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Previous reviews and meta-analyses have identified several issues in evaluating the effectiveness of SBPs.

“Regarding methodological aspects, previous reviews and meta-analyses have identified several issues in evaluating the effectiveness of SBPs.”

Students' motivation to complete the program changes when completion is tied to a grade or placement in an advanced course.

“Finally, students' motivation to complete the program changes when completion is tied to a grade or placement in an advanced course (Chamberlin et al., 2023;Demszky et al., 2024;Rodríguez Rincón et al., 2024).”

• cites Chamberlin et al., 2023 sourceNot checked — no full text of the source
• cites Demszky et al., 2024 sourceNot checked — no full text of the source
• cites Rodríguez Rincón et al., 2024 sourceNot checked — no full text of the source

The same RCT found no evidence suggesting an effect on persistence.

“However, they did improve completion rates for first-year college-level courses in math and writing, but no evidence suggested an effect on persistence.”

• cites Barnett et al. (2012)Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

The What Works Clearinghouse report found only one study that met the group design standards.

“Similarly, the report by What Works Clearinghouse only found one study that met the group design standards (U.S.Department of Education, Institute of Education Sciences, What Works Clearinghouse, 2016).”

• cites U.S.Department of Education, Institute of Education Sciences, What Works Clearinghouse, 2016Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

There is a possibility of publication bias.

“Nevertheless, there is a possibility of publication bias.”