Exploring the Impact of Worked Examples in a Novice Programming Environment

Classified as probably not a substantial claim. May combine two claims.

Novices have difficulty in understanding the examples and finding relevant blocks to finish the programming task.

“For example, they found that novices have difficulty in understanding the examples and finding relevant blocks to finish the programming task.”

• cites [6] sourceSupported

May combine two claims.

Students with worked examples completed more tasks within a fixed time period, but not significantly more.

“Overall, students with WEs completed more tasks within a fixed time period, but not significantly more.”

• Also stated in: § 1 INTRODUCTION AND BACKGROUND · plain

  “Students with WEs on a problem completed more objectives on that problem, but this difference was not significant.”

May combine two claims.

For Daisy Design, the median student in the WE group (E1) took 9.5 minutes to complete their next objective, while the median student in the non-WE group (E2) took 0 minutes.

“For Daisy Design, we see that the median student in the WE-group (E1) took 9.5 minutes to complete their next objective, while the median student in the non-WE group (E2) took 0 minutes (they accomplished the next objective at the same time as the WE objectives).”

• AssumptionThe authors state this as an assumption: Students who completed the WE objectives the slowest would have completed additional objectives no quicker than the other students in their group.

May combine two claims.

On Spiral Polygon, it took the WE group much longer to accomplish one additional objective (Q1 = 12.0m) compared to the non-WE group (Q1 = 2.6m), but this gap quickly reduced for the second objective and reversed by the third.

“Similarly, it takes the WE group much longer to accomplish one additional objective (Q1 = 12.0m) compared to the non-WE group (Q1 = 2.6m), but this gap quickly reduces for the second objective, and even reverses by the third (where not even 25% of non-WE of students had completed the third objective by 39 minutes).”

• AssumptionThe authors state this as an assumption: Students who completed the WE objectives the slowest would have completed additional objectives no quicker than the other students in their group.

May combine two claims.

There were no significant main or interaction effects on the number of help requests (F2,28=2.15, p=.14, partial η2=.13), indicating worked examples likely did not impact the number of help requests students made, nor did the problem, despite increasing levels of problem difficulty.

“We found no significant main or interaction effects (F 2, 28 = 2.15, p = .14,partial η 2 = .13),indicating that WEs likely did not impact the number of help requests students made, and nor did the problem, despite increasing levels of problem difficulty.”

May combine two claims.

There was no statistically significant interaction between group and test time (F2,28=0.13, p=.88, partial η2=.009), indicating students in both groups had similar learning across the study.

“Additionally, there was no statistically significant interaction between group and test time (F 2, 28 = 0.13, p = .88,partial η 2 = .009),indicating that students in both groups had similar learning across the study.”

• AssumptionRests on an assumption: The pre-test and post-tests are sensitive enough to detect changes in students' learning.

The quote only partly states this claim.

The next two objectives also took the WE group longer on Daisy Design.

“The next two objectives also take the WE group longer.”

• AssumptionThe authors state this as an assumption: Students who completed the WE objectives the slowest would have completed additional objectives no quicker than the other students in their group.

May combine two claims.

We found significant differences between the groups' intrinsic cognitive load for problem 2 but not for problem 1.

“We found significant differences between the groups' intrinsic cognitive load for problem 2 but not for problem 1.”

May combine two claims.

There were statistically significant improvements for all students from pre-test to post-test2 (t(15)=3.05, p<.01, d=.30) and post-test1 to post-test2 (t(15)=2.49, p<.05, d=.24), but not from pre-test to post-test1 (t(15)=-0.86, p=.40, d=.08).

“We found statistically significant improvements for all students from both pre-test to post-test2 (t (15) = 3.05, p < .01,d = .30)and post-test1 to post-test2 (t (15) = 2.49, p < .05,d = .24),but not from pre-test to post-test1 (t (15) = -0.86,p = .40,d = .08).”

• AssumptionRests on an assumption: The pre-test and post-tests are sensitive enough to detect changes in students' learning.

May combine two claims.

The difference in time to complete additional objectives never exceeded 15 minutes on Daisy Design and 10 minutes on Spiral Polygon, which is still far less time than the WEs saved the median student on each problem (19 and 24 minutes, respectively).

“In addition, the difference never exceeded 15 minutes on Daisy Design, and 10 minutes on Spiral Polygon, which is still far less time than the WEs saved the median student on each problem (19 and 24 minutes, respectively).”

• AssumptionThe authors state this as an assumption: Students who completed the WE objectives the slowest would have completed additional objectives no quicker than the other students in their group.

May combine two claims.

Some students finished the WE objectives slowly, or not at all, and ran out of time before they could complete additional objectives.

“Some students finished the WE objectives slowly, or not at all, and ran out of time before they could complete additional objectives.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

For Spiral Polygon, no additional objectives were completed by more than half of students, so only the Q1 time could be computed.

“For Spiral Polygon, no additional objectives were completed by more than half of students, so we can only compute the Q1 time.”

• AssumptionThe authors state this as an assumption: Students who completed the WE objectives the slowest would have completed additional objectives no quicker than the other students in their group.

May combine two claims.

The differences between groups in objectives completed are most apparent at the beginning of the curves, when students are completing the WE objectives, though the same trend holds even if the WE objectives are not included.

“These differences between groups is most apparent at the beginning of the curves, when students are completing the WE objectives, though the same trends holds, even if we do not include the WE objectives.”

May combine two claims.

There was no significant result for extraneous load or germane load.

“We did not find any significant result for extraneous load or germane load.”

May combine two claims.

On both Daisy Design and Spiral Polygon, the group with WEs (E1 and E2, respectively) had more objectives completed at all times throughout the 45 minute work period, while on Brick Wall (with no WEs) there is little difference.

“On both Daisy Design and Spiral Polygon, the group with WEs (E1 and E2, respectively), had more objectives completed at all times throughout the 45 minute work period, while on Brick Wall (with no WEs) there is little difference.”

May combine two claims.

On Daisy Design, the WE group (E1) ended with a median 6 objectives complete (67%), compared to 5 objectives (56%) for the non-WE group (E2), and the difference is not significant (W = 56.0; p = 0.382).

“On Daisy Design, the WE group (E1) end with a median 6 objectives complete (67%), compared to 5 objectives (56%) for the non-WE group (E2), and the difference is not significant (W = 56.0;p = 0.382).”

May combine two claims.

On Spiral Polygon, the WE group (E2) ended with a median 2 objectives complete (40%), compared to 1 objective (20%) for the non-WE group (E1), and the difference is not significant (W = 31.5; p = 0.270).

“On Spiral Polygon, the WE group (E2) ends with a median 2 objectives complete (40%), compared to 1 objective (20%) for the non-WE group (E1), and the difference is not significant (W = 31.5;p = 0.270).”

May combine two claims.

E2 students' intrinsic load on problem 2 was significantly lower than that on problem 3 (t(7)=-4.52, p<.01, d=1.04), but no such results were found for E1.

“Similarly, we found E2 students' intrinsic load on problem 2 was significantly lower than that on problem 3 (t (7) = -4.52,p < .01,d = 1.04), but no such results were found for E1.”

May combine two claims.

For intrinsic load, there was no significant main effect of group (F1,14=0.10, p=.76, partial η2=.007) or problem (F2,28=1.78, p=.19, partial η2=.11), but there was a statistically significant interaction between group and problem on intrinsic load scores (F2,28=4.65, p<.05, partial η2=.25).

“For the intrinsic load, there was not a significant main effect of group (F 1, 14 = 0.10, p = .76,partial η 2 = .007)or problem (F 2, 28 = 1.78, p = .19,partial η 2 = .11)factors, but there was a statistically significant interaction between group and problem on the intrinsic load scores (F 2, 28 = 4.65, p < .05,partial η 2 = .25).”

May combine two claims.

E1 students' intrinsic load on problem 1 was significantly lower than on problem 2 (t(7)=-3.51, p<.01, d=0.83), but no such results were found for E2.

“We investigated this further with post hoc t-tests and found that E1 students' intrinsic load on problem 1 was significantly lower than on problem 2 (t (7) = -3.51,p < .01,d = 0.83), but no such results were found for E2.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Patitsas et al. found that students in the comparing and contrasting condition showed greater learning gains for procedural knowledge and flexibility than students in the worked example condition.

“They found that students in the comparing and contrasting condition showed greater learning gains for procedural knowledge and flexibility than students in the WE condition [13].”

• cites [13] sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

Extraneous load should be minimized to maximize germane load and thus learning.

“extraneous load should be minimized to maximize germane load and thus learning.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Empirical studies have shown that worked examples are more beneficial to novice learners compared with traditional problem solving in learning algebra.

“Empirical studies have shown that WEs are more beneficial to novice learners compared with traditional problem solving in learning algebra [20].”

• cites [20]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Liu et al.'s work shows that worked examples are beneficial to undergraduate students in the first few problems they solve but less beneficial for later problems in the students' homework set.

“In an intelligent tutoring system for teaching and practicing propositional logic problems in discrete math, Liu et al. 's [9] work shows that WEs are beneficial to undergraduate students in the first few problems they solve but less beneficial for later problems in the students' homework set.”

• cites [9] sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing).

Sweller and Cooper found that students who were presented with worked example problems solved similar problems quicker and with fewer errors than students who solved the same problems themselves.

“Sweller and Cooper found that students who were presented with WE problems solved similar problems quicker and with fewer errors than students who solved the same problems themselves [20].”

• cites [20]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Trafton and Reiser's study suggests interleaving worked examples with similar practice problems to maximize students learning gains, compared to blocking worked examples with problems, or solving equivalent problems by writing LISP programs.

“Trafton and Reiser's study suggests interleaving WEs with similar practice problems to maximize students learning gains, compared to blocking WEs with problems, or solving equivalent problems by writing LISP programs [22].”

• cites [22]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim.

Cognitive Load Theory describes the relationship between mental effort and learning.

“Our work is based on Cognitive Load Theory (CLT), which describes the relationship between mental effort and learning [21].”

• cites [21]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Along with worked examples, self-explanation prompts help students foster greater understanding and improve students' problem solving performance.

“A review of self-explanation shows that along with WEs, self-explanation prompts help students foster greater understanding [16] and improves students' problem solving performance [12].”

• cites [16] sourceNot checked — no full text of the source
• cites [12] sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing).

Pirolli and Anderson found that three novices relied heavily on examples while writing new programs.

“Pirolli and Anderson investigated how three novices learned recursive functions and found that they relied heavily on examples while writing new programs [14].”

• cites [14]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Worked examples (WEs) have been studied in a variety of domains, and results consistently show that replacing some or all practice problems with WEs results in higher learning efficiency, where students take less time to learn the same amount of content.

“WEs have been studied in a variety of domains [9,10,20], and the results consistently show that replacing some or all practice problems with WEs results in higher learning efficiency, where students take less time to learn the same amount of content.”

• cites [9] sourceNot checked — no full text of the source
• cites [10] sourceNot checked — no full text of the source
• cites [20]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

McLaren et al. found that students in the worked example condition performed similarly in the post-test but were more efficient in finishing intervention problems and with less cognitive load, compared to students in other conditions including problem solving, erroneous worked examples, and tutored problem solving.

“They found that students in the WE condition performed similarly in the post-test but were more efficient in finishing intervention problems and with less cognitive load, compared to students in other conditions including problem solving, erroneous worked examples, and tutored problem solving [10].”

• cites [10] sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing).

Ericson et al. found that interleaving worked examples with Parsons problems can improve programming learning efficiency compared to worked examples paired with writing code or fixing code.

“Recently, Ericson et al. found that interleaving WEs with Parsons problems can improve programming learning efficiency compared to WEs paired with writing code or fixing code [3].”

• cites [3] sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing).

One way WEs may increase learning efficiency is by reducing students' cognitive load, or mental effort when learning.

“One way WEs may increase learning efficiency is by reducing students' cognitive load, or mental effort when learning [18].”

• cites [18]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Only Trafton and Reiser directly evaluate the effectiveness of worked examples by comparing them to equivalent problem solving tasks, but their results may not apply to novice programming environment.

“However, to our knowledge, only Trafton and Reiser directly evaluate the effectiveness of WEs by comparing them to equivalent problem solving tasks, but their results may not apply to novice programming environment.”

• cites [22]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim. May combine two claims.

WEs may have an effect on students' intrinsic cognitive load, but further research is needed on how and why they have this effect.

“We found that WEs may have an effect on students' intrinsic cognitive load, but further research is needed how and why they have this effect.”

• AssumptionThe authors state this as an assumption: Intrinsic cognitive load is inherent in the learning task.
• AssumptionThe authors state this as an assumption: Extraneous cognitive load is the result of the instructional design.

Classified as probably not a substantial claim.

The pre-test, and isomorphic post-test1 and post-test2, were adapted from the block-based version of the Commutative Assessments, with 16 multiple-choice questions on variables, loops, procedures, and conditionals.

“The pre-test, and isomorphic post-test1 and post-test2, were adapted from the block-based version of the Commutative Assessments [25], with 16 multiple-choice questions on variables, loops, procedures, and conditionals.”

• cites [25] sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing).

Some students were removed from the analysis because they lost their work due to errors in Snap!, leaving 9, 10, and 10 students in E1 and 10, 9, and 10 students in E2 on the three problems, respectively.

“On each problem we removed some students from our analysis who lost their work due to errors in Snap!, leaving 9, 10, and 10 students in E1 and 10, 9, and 10 students in E2 on the three problems, respectively.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

The self-selected summer campers may be more motivated to learn, but may have taken the assessments less seriously than other student populations.

“The self-selected summer campers and may be more motivated to learn, but may have taken the assessments less seriously than other student populations.”

Classified as probably not a substantial claim.

Morrison et al. showed that students who self-explained subgoal labels perform better than those who were not given subgoal labels.

“Morrison et al. showed that students who self-explained subgoal labels perform better than those who were not given subgoal labels [12].”

• cites [12] sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing).

It is not surprising that there was no significant difference in learning between groups.

“Since most students in both groups completed the core objectives, it is not surprising that there was no significant difference in their learning.”

May combine two claims.

Paired with the other results, the PCH is an overall improvement to Snap!, which students generally prefer over purely independent programming.

“Paired with the other results presented here, this indicates that the PCH is an overall improvement to Snap!, which students generally prefer over purely independent programming.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Two researchers reached an inter-rater reliability between 0.744 and 0.979 for the first 15 student submissions (5 in each problem).

“The two researchers reached an inter-rater reliability between 0.744 and 0.979 for first 15 student submissions (5 in each problem).”

May combine two claims.

Previous WE studies found that students learned more efficiently with WEs, but they did not learn more, since both groups were exposed to the same learning content.

“This is consistent with previous WE studies [10,17], where students learned more efficiently with WEs, but they did not learn more, since both groups were exposed to the same learning content.”

• cites [10] sourceNot checked — no full text of the source
• cites [17] sourceNot checked — no full text of the source

May combine two claims.

WEs may have impacted students' overall performance, but the difference was not significant.

“This suggests that WEs may have impacted students' overall performance, but the difference was not significant.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Students' desire for independence is a common reason that they avoid help when programming, and overcoming this barrier can be very challenging.

“Students' desire for independence is a common reason that they avoid help when programming, and overcoming this barrier can be very challenging [15].”

• cites [15] sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing).

One interpretation of these results is that students experienced reduced intrinsic cognitive load when using worked examples.

“One interpretation of these results is that students experienced reduced intrinsic CL when using WEs.”

May combine two claims.

The time saved by WEs is longer than the time students need to complete additional objectives, which allowed students who received WEs to complete more objectives overall on both problems 1 and 2, although this difference was not significant due to the low sample size.

“However, the time saved is longer than the time students need to complete additional objectives, which allowed students who received WEs to complete more objectives overall on both problems 1 and 2, although this difference was not significant due to the low sample size.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Participants' learning gains may also have come from repeated exposure to tests.

“Participants' learning gains may also have come from repeated exposure to tests.”

Probably not a substantial claim: may not pass S3 (load-bearing).

For analysis of pre- and post-tests, one additional student from Group E1 who did not fill in post-test1 was excluded, leaving 8 students in each group.

“For our analysis of pre-and post-tests, we exclude one additional student from Group E1 who did not fill in post-test1, leaving 8 students in each group.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

After excluding these students, there are 9 students in Group E1 and 8 students in Group E2.

“After excluding these students, there are 9 students in Group E1 and 8 students in Group E2.”

May combine two claims.

This helps to explain why the WE group accomplished more objectives on both problems, but the difference was not significant.

“This helps to explain why the WE group accomplished more objectives on both problems, but the difference was not significant.”

• AssumptionThe authors state this as an assumption: Students who completed the WE objectives the slowest would have completed additional objectives no quicker than the other students in their group.

May combine two claims.

The two groups experienced different levels of intrinsic cognitive load on different problems, which may be due to the worked examples.

“This indicates that the two groups experienced different levels of intrinsic CL on different problems, which may be due to the WEs.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The CS Cognitive Load Component Survey (CS CLCS) was used as the cognitive load instrument.

“We used the CS Cognitive Load Component Survey (CS CLCS) [11] as our cognitive load instrument.”

• cites [11] sourceNot checked — no full text of the source

Classified as probably not a substantial claim. May combine two claims.

Students were randomly assigned to groups (E1 receiving worked examples on problem 1, E2 with worked examples on problem 2), with matched pairs according to pre-test scores.

“In Step 1, students took the pre-test and the pre-survey, and were randomly assigned to groups (E1 receiving WEs on problem 1, E2 with WEs on problem 2), with matched pairs according to pre-test scores.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

All of the programming problems involved drawing patterns, and the design of WEs may not generalize to problems that lack visual output.

“All of the programming problems involved drawing patterns, and our design of WEs may not generalize to problems that lack visual output.”

May combine two claims.

WEs allowed all students to finish the WE objectives, compared to only 80% and 60% for non-WE students.

“Based on these results, we can say the WEs saved students lots of time (19-24 minutes on a 45 minute problem), and allowed all students to finish the WE objectives (compared to only 80% and 60% for non-WE students).”

May combine two claims.

Our analysis of programming efficiency suggests that initially, WEs save students considerable time in completing programming objectives (up to half of the time allotted), but that students take longer to complete later objectives.

“Our analysis of programming efficiency suggests that initially, WEs save students considerable time in completing programming objectives (up to half of the time allotted), but that students take longer to complete later objectives.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Five students were excluded from survey and assessment analysis because they were not present for the whole study (2) or encountered errors in Snap! that caused them to lose significant programming progress (3).

“However, five students were excluded from survey and assessment analysis because they were not present for the whole study (2) or encountered errors in Snap! that caused them to lose significant programming progress (3) .”

Classified as probably not a substantial claim.

Camp instructors followed a tutoring protocol to ensure that each student received the same quality and quantity of help.

“Camp instructors followed a tutoring protocol to ensure that each student received the same quality and quantity of help, using an audio recorder to capture the dialogue of these help episodes, and record who they helped, the help duration (time), and the type of help (e.g.high-level planning, programming, debugging, interface related).”

Classified as probably not a substantial claim. May combine two claims.

The study used a matched pairs, switching replication design to investigate the impact of worked examples on problem 1, and whether or not it matters which problem receives worked example support.

“The study used a matched pairs, switching replication design as shown in Table 1 to investigate the impact of WEs on problem 1, and whether or not it matters which problem receives WE support.”

May combine two claims.

Programming WEs may improve students' programming efficiency in the short term, but students do require additional time to process WEs before they can construct their own code.

“Our results also suggest that programming WEs may improve students' programming efficiency in the short term, but that students do require additional time to process WEs before they can construct their own code.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Another interpretation is that students experienced higher intrinsic cognitive load on problem solving that followed a worked example, which made it seem harder by contrast.

“However, another interpretation is that students experienced higher intrinsic CL on problem solving that followed a WE, which made it seem harder by contrast.”

May combine two claims.

After completing the WE-supported objectives, students with WEs initially worked less efficiently, especially since non-WE students could accomplish additional objectives while completing the WE objectives.

“From this, we conclude that, after completing the WE-supported objectives, students with WEs initially worked less efficiently, especially since non-WE students could accomplish additional objectives while completing the WE objectives (e.g.putting code into a procedure).”

• AssumptionThe authors state this as an assumption: Students who completed the WE objectives the slowest would have completed additional objectives no quicker than the other students in their group.

Classified as probably not a substantial claim. May combine two claims.

This is an exploratory study with a small sample size, which makes it difficult to rule out possible differences between groups.

“This is an exploratory study with a small sample size, which makes it difficult to rule out possible differences between groups.”