Using ChatGPT is easy, using it effectively is tough? A mixed methods study on K-12 students’ perceptions, interaction patterns, and support for learning with generative AI chatbots

May combine two claims.

Participants' requests were usually fulfilled by the chatbot, so they usually experienced the interaction as successful.

“The participants' requests were usually fulfilled by the chatbot, so they usually experienced the interaction as successful.”

May combine two claims.

Participants in the experimental group asked for an outline or had the chatbot make a table, which was not asked in the control group.

“For example, participants in the EG asked for an outline or had the chatbot make a table, which was not asked in the control group.”

May combine two claims. The quote only partly states this claim.

Adaptation guidance did not increase extraneous cognitive load; there was no significant difference between groups.

“without experiencing increased cognitive load”

• Also stated in: § RQ3 and RQ4: hypothesis tests · plain

  “Learners report similar levels of extraneous load in the experimental group (M = 3.0) and in the control group (M = 2.77).”

May combine two claims.

The experimental group used more prompts overall, especially more follow-up questions and adaptation prompts, while the control group showed more off-topic conversation.

“The experimental group used more prompts overall, especially more follow-up questions and adaptation prompts, while the control group showed more off-topic conversation which signals a difference in engagement between the two groups.”

May combine two claims.

Many chatbot interactions were a series of questions covering general aspects of the topic, with a few adaptation prompts in most conversations and single follow-up questions in many.

“Many chatbot interactions were a series of questions covering the general aspects of the topic (i.e., explanation, examples, causes, consequences, and remedies of cognitive biases), with a few adaptation prompts in most conversations and single follow-up questions in many.”

May combine two claims.

Overall, the experimental group had more interactions with the chatbot and less off-topic conversation.

“Overall, they had more interactions with the chatbot and less offtopic conversation.”

May combine two claims.

There is no significant difference between the conditions regarding several items on satisfaction, despite differences in use of adaptation prompts.

“Second, despite differences in use of adaptation prompts, there is no significant difference regarding several items on satisfaction.”

May combine two claims.

Complex, interactive SRL learning activities, as suggested by Chiu (2024), are not present in the control group and to some extent in the group with adaptation guidance.

“Complex, interactive SRL learning activities, as suggested by Chiu (2024), are not present in the control group and to some extent in the group with adaptation guidance.”

• cites Chiu (2024) sourceNot checked yet

  Not checked in the free run.

May combine two claims.

On average, participants did not prompt the chatbot as if it was a search engine, but they did not engage with it like a human interlocutor either.

“In terms of overall interaction style, it is notable that, on average, the participants did not prompt the chatbot as if it was a search engine (i.e., with abstracted search terms), but they did not engage with it like a human interlocutor either.”

May combine two claims.

The experimental group used more follow-up questions and more general questions when self-written questions and suggested questions were added up.

“The experimental group used more follow-up questions and more general questions when self-written questions and suggested questions were added up for the EG.”

May combine two claims.

18% of the adaptation prompts and 19% of the questions prompted by the students were suggested by the chatbot.

“18% of the adaptation prompts and 19% of the questions prompted by the students were suggested by the chatbot, so this feature was used considerably often but students also did not overly rely on it.”

May combine two claims.

In both groups, students felt they always knew what to ask next and that they always received the response they needed.

“In both groups, students felt they always knew what to ask next (CG: M = 4.4; EG: M = 4.57) and that they always received the response they needed (CG: M = 5.16; EG: M = 5.17).”

The quote only partly states this claim.

Knowledge gain did not differ significantly between the experimental and control groups.

“both groups showed similar knowledge gains”

• Also stated in: § RQ3 and RQ4: hypothesis tests · plain

  “The mean difference between pre-and post-test scores is similar between the experimental group (M = 2.43) and the control group (M = 2.52).”

May combine two claims.

Descriptively, the experimental group rated their germane load as higher (M = 5.2) than the control group (M = 4.9), but the difference is not significant.

“However, this interpretation is not clearly reflected in the ratings of germane load: Descriptively, the EG rated their germane load as higher (M = 5.2) than the control group (M = 4.9), but the difference is not significant.”

May combine two claims.

Students judged their ability to use chatbots for learning as high, despite using them only a few times a month and having had teacher instruction on chatbots only once.

“First, students judged their ability to use chatbots for learning as high, even though they used them only a few times a month and they had had teacher instruction on chatbots only once.”

May combine two claims.

There were no moderation effects found for domain knowledge, chatbot experience, or SRL skills.

“There were no moderation effects found for either of these aspects.”

May combine two claims.

Irrespective of the level of prior knowledge on domain knowledge, chatbot experience, and SRL skills, students used significantly more adaptation prompts and did not differ significantly in terms of learning gains and extraneous load.

“That is, irrespective of the level of prior knowledge on these three aspects, the students used significantly more adaptation prompts and did not differ significantly in terms of learning gains and extraneous load.”

May combine two claims.

Students generally perceived the chatbots favorably and reported high levels of satisfaction and ease of use, regardless of whether they received adaptation guidance.

“First, students generally perceived the chatbots favorably and reported high levels of satisfaction and ease of use, regardless of whether they received adaptation guidance.”

May combine two claims.

The automatically generated prompt suggestions were used, but not overused, and were perceived as useful.

“More specifically, the automatically generated prompt suggestions were used, but not overused, and were perceived as useful.”

May combine two claims.

Students who received adaptation guidance used a greater variety and a significantly higher number of adaptation prompts, asked more follow-up questions, and engaged in less off-topic conversation.

“Students who received adaptation guidance used a greater variety and a significantly higher number of adaptation prompts, asked more follow-up questions, and engaged in less off-topic conversation.”

May combine two claims.

Prior domain knowledge and prior chatbot experience were low overall and showed little variance, which was possibly a reason for the lack of moderation effect.

“Prior domain knowledge and prior chatbot experience were low overall and showed little variance, which was possibly a reason for the lack of moderation effect.”

May combine two claims.

Hypotheses 4.1 through 4.3 are not supported by the data for extraneous load and knowledge gain and could not be tested for adaptation prompts.

“Therefore, hypotheses 4.1 through 4.3 are not supported by the data for extraneous load and knowledge gain and could not be tested for adaptation prompts.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Tasks requiring the learner to search among multiple solutions induce extraneous load.

“Tasks requiring the learner to search among multiple solutions induce extraneous load, i.e., load induced by processing information that is irrelevant to the task (Sweller, 2011).”

• cites Sweller, 2011Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable).

Students need tailored support to effectively use chatbots for self-regulated learning.

“Abdelhalim (2024) found that students with high metacognitive awareness used ChatGPT more skillfully for research than those with lower awareness, suggesting students need tailored support to effectively use chatbots for self-regulated learning.”

• cites Abdelhalim, 2024 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Learners perceive chatbots as easy to use, which is an indication that they underutilize beneficial capabilities and possibly invest too little mental effort.

“The fact that learners perceive chatbots as easy to use is an indication that they underutilize these beneficial capabilities and possibly also invest too little mental effort.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Deciding among adaptation options can induce cognitive load.

“Deciding among these adaptation options can induce cognitive load.”

May combine two claims.

Given that students in the experimental group had only received a very short instruction on adaptation prompts and prompt suggestions, this small intervention produced a large effect.

“Conversely, given that the students in the experimental group had only received a very short instruction on adaptation prompts and prompt suggestions, this small intervention produced a large effect.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Changing the overall length to shorter and the language to simple might have been undesirable for students because it would reflect badly on their performance, and because students might have felt they missed out on important information.

“Changing the overall length to”

May combine two claims.

Even though students reported high confidence in their skill to effectively use chatbots for learning, they do not make use of the basic capabilities when unassisted.

“This indicates that even though the students reported high confidence in their skill to effectively use chatbots for learning, they do not make use of the basic capabilities when unassisted.”

May combine two claims.

The larger number of adaptations might have benefitted one aspect of learning but impaired another.

“The larger number of adaptations might have benefitted one aspect of learning but impaired another: Berthold et al. (2011) found evidence that explanation prompts can be double-edged in that they increase conceptual knowledge and detailedness of explanations, while they reduce procedural knowledge and number of tasks performed.”

• cites Berthold et al. (2011) sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

106 of the 108 datasets were complete and included in the sample.

“106 of the 108 datasets were complete and included in the sample.”

May combine two claims.

Easily implementable interventions can enhance students' use of genAI chatbot capabilities, potentially improving their self-regulated learning experiences.

“These findings suggest that easily implementable interventions can enhance students' use of genAI chatbot capabilities, potentially improving their self-regulated learning experiences.”

• AssumptionRests on an assumption: Using more adaptation prompts improves students' self-regulated learning.
• AssumptionRests on an assumption: The adaptation guidance intervention is easily implementable in real educational settings.

Classified as probably not a substantial claim. May combine two claims.

The second rater coded all chats with the coding scheme, and intercoder agreement was weighted kappa κ = 0.94 for adaptation prompts and κ = 0.91 for questions.

“The second rater coded all chats with this coding scheme.In cases of a discrepancy greater than 1, both raters discussed the case and reached an agreement.The intercoder agreement was then calculated as the weighted kappa for adaptation prompts (κ = 0.94) and questions (κ = 0.91).”

Classified as probably not a substantial claim.

Knowledge was assessed through open-ended pre-and post-tests scored by two raters with interrater agreement κ = 0.70-0.79 for pre-test and κ = 0.74-0.88 for post-test.

“Knowledge was assessed through open-ended pre-and post-tests on cognitive biases, scored for breadth, depth, and precision by two raters.”

• Also stated in: § Instruments · plain

  “interrater agreement reached κ = 0.70 for breadth, κ = 0.73 for depth and κ = 0.79 for precision for the pre-test, and κ = 0.88 for breadth, κ = 0.83 for depth and κ = 0.74 for precision for the post-test.”

May combine two claims.

Extensive 'prompt training' might not be necessary, but targeted, short instruction prior to a task can be effective.

“This shows that extensive”

May combine two claims.

In addition to being supportive, the instruction, and maybe also the supportive interface features, raised the task difficulty: Students in the control group did not perceive the task demand of adapting the chatbot responses to their needs.

“In addition to being supportive, the instruction, and maybe also the supportive interface features, raised the task difficulty: Students in the control group did not perceive the task demand of adapting the chatbot responses to their needs.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Self-regulated learning can cause cognitive load due to constant decision-making.

“From this perspective, self-regulated learning can cause cognitive load due to constant decision-making.”

May combine two claims.

Because students did not perceive the versatility the chatbot affords, they had a smaller range of possible next prompts and were not overwhelmed by the plentitude of interaction possibilities.

“Because the students did not perceive the versatility the chatbot affords, they had a smaller range of possible next prompts and were not overwhelmed by the plentitude of interaction possibilities.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

There are aspects of chatbots that can increase cognitive load, such as the numerous indirect affordances and lack of SRL support, while others, like the intuitive interface, may reduce it.

“Therefore, there are aspects of chatbots that can increase cognitive load, such as the numerous indirect affordances and lack of SRL support, while others, like the intuitive interface, may reduce it.”

May combine two claims.

The experimental group was more engaged in the task and the conversation with the chatbot.

“This suggests that the experimental group was more engaged in the task and the conversation with the chatbot.”

Probably not a substantial claim: may not pass S3 (load-bearing).

This study can be considered a true experimental study despite group-level randomization.

“On average, there were four students per group, so while randomization did not take place at the individual level but at the level of a small group, this study can be considered a true experimental study in contrast to quasi-experimental studies where whole classes are randomized.”

Classified as probably not a substantial claim.

Cognitive load was assessed using the 8-item instrument by Klepsch et al. (2017).

“Cognitive load was assessed using the 8-item instrument by Klepsch et al. (2017), which covers all three aspects of cognitive load.”

• cites Klepsch et al. (2017) sourceNot checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

Each group of up to seven participants was randomly assigned to one of the two conditions.

“Each group of up to seven participants was randomly assigned to one of the two conditions.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Chatbot experience was assessed with three items with a Cronbach's alpha of 0.68.

“Chatbot experience was assessed with three items on the frequency of chatbot use at home and school, the scale adapted from Thompson (2013), with a Cronbach's alpha of 0.68.”

• cites Thompson (2013) sourceNot checked yet

  Not checked in the free run.

Classified as probably not a substantial claim. May combine two claims.

As chatting with a chatbot can be perceived as familiar, overlearned, and lifelike, learners might not perceive the manifold indirect affordances of the chatbot, underutilize them, and invest too little effort.

“As chatting with a chatbot can similarly be perceived as familiar, overlearned, and lifelike, a similar phenomenon might occur: Learners might not perceive the manifold indirect affordances of the chatbot, underutilize them, and invest too little effort, as suggested in a recent study by Stadler et al. (2024).”

• cites Stadler et al., 2024 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Cognitive load increases with the number of mental processes performed simultaneously.

“Cognitive load increases with the number of mental processes performed simultaneously (Sweller, 2010).”

• cites Sweller, 2010 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Learners might quickly adapt to the chatbot's affordance and enter the habit of learning from texts.

“The learners might quickly adapt to this affordance and enter the habit of learning from texts.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Using generative AI as a tool for complex tasks without shortcuts might add a further layer of complexity and thereby cognitive load.

“However, using generative AI as a tool for complex tasks without shortcuts might add a further layer of complexity and thereby cognitive load.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

In this study, the adapted chatbot responses might have led to more readability but also to less challenging content which led to shallower learning.

“Similarly, in this study, the adapted chatbot responses might have led to more readability but also to less challenging content which led to shallower learning.”

May combine two claims.

This perception of the chatbot as 'easy' suggests an underinvestment of mental effort and, combined with students' moderate prior experience, a potential overestimation of their ability to effectively use these tools for learning.

“This perception of the chatbot as”

• cites Salomon, 1984 sourceNot checked yet

  Not checked in the free run.

• cites Stadler et al., 2024 sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Since no standard for effective learning with chatbots was provided in this study, students may have set a low bar, a common pattern in self-regulated learning.

“Since no standard for effective learning with chatbots was provided in this study, students may have set a low bar, a common pattern in selfregulated learning.”

May combine two claims. The quote only partly states this claim.

Using the full range of chatbot capabilities in self-regulated learning might still be 'tough', but this can be effectively alleviated by targeted support.

“Using the full range of chatbot capabilities in self-regulated learning might still be”

May combine two claims.

While the findings show immediate benefits in terms of increased use of adaptation prompts, one cannot draw conclusions about how this might impact learning outcomes or self-regulated learning skills over extended periods of use and in other contexts.

“While the findings show immediate benefits in terms of increased use of adaptation prompts, one cannot draw conclusions about how this might impact learning outcomes or self-regulated learning skills over extended periods of use and in other contexts.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

In everyday use, students would likely combine the chatbot with web search and other tools, possibly to differing degrees.

“In everyday use, students would likely combine the chatbot with web search and other tools, possibly to differing degrees.”

May combine two claims.

As there is no significant difference between the groups in intrinsic or germane cognitive load, the demand characteristics effect, if present, is not very pronounced.

“However, as there is no significant difference between the groups in intrinsic or germane cognitive load, this effect, if present, is not very pronounced, i.e., the experimental group did not perceive the task with adaptation guidance as substantially more complex and did not invest substantially more effort.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

Possibly, there is an aspect of self-disclosure when students opt for shorter answers in simple language.

“Possibly, there is an aspect of self-disclosure when students opt for shorter answers in simple language.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The intervention in this study does not require significant resources, and therefore even small effects would be of interest so that a larger sample would have been desirable.

“The intervention in this study, however, does not require significant resources and, therefore, even small effects would be of interest so that a larger sample would have been desirable.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The experimental group was not just given hints on how to use adaptation prompts but also that it is beneficial to use them.

“The experimental group was not just given hints on how to use adaptation prompts but also that it is beneficial to use them.”

May combine two claims.

The feature to change language level and length was perceived as useful and could be more relevant for learners with lower language skills.

“Still, the feature was perceived as useful and could be more relevant for learners with lower language skills.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based). May combine two claims.

Alternatively, students might have felt they missed out on important information if they had simplified all the chatbot's answers and therefore preferred targeted adaptations to individual messages.

“Alternatively, students might have felt they missed out on important information if they had simplified all the chatbot's answers and therefore preferred targeted adaptations to individual messages.”