Fostering AI Literacy in Undergraduates: A ChatGPT Workshop Case Study

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Automation facilitated by large language models may disrupt labor markets, posing challenges to employment and exacerbating income inequality globally.

“Additionally, the automation facilitated by large language models may disrupt labor markets, posing challenges to employment and exacerbating income inequality globally (Frank et al. 2019).”

• cites Frank et al. 2019 sourcePartly supported

  The Abstract says AI and automation may disrupt labor markets and replace some workers’ work, and it notes concern about employment in advanced and developing economies worldwide. It does not specifically address large language models. It describes automation as occurring amid growing inequality, while “Barrier: Places in Isolation” discusses possible changes in income distribution; neither establishes that this automation would exacerbate income inequality globally. (LLM, conf=0.88)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Brin et al. (2023) found that ChatGPT and GPT-4 showed decent accuracy in soft skill questions, with increased accuracy with GPT-4.

“While the findings showed decent accuracy in soft skill questions with ChatGPT and increased accuracy with GPT-4, the researchers point out that although the findings show the empathetic nature of AI, further research is needed using more ethical scenarios.”

• cites Brin et al. (2023) sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Students appreciated ChatGPT's ability to provide personalized guidance, facilitate understanding of difficult concepts, or assist with writing tasks.

“Students appreciated Chat-GPT's ability to provide personalized guidance, facilitate understanding of difficult concepts, or assist with writing tasks (Budhiraja et al. 2024;Ding et al. 2023;Li 2023;Mahapatra 2024;Malik et al. 2023;Yan 2023).”

• cites Budhiraja et al. 2024 sourceSupported
• cites Ding et al. 2023 sourcePartly supported

  The Conclusion and survey tables (page unavailable) report that students used ChatGPT as a virtual tutor for physics questions and rated its usefulness for learning, which supports the general point that they valued its learning assistance. The paper does not report that these students appreciated personalized guidance or help with writing tasks; its Introduction and AI in education section discuss those uses as possibilities or findings from other work. (LLM, conf=0.82)

  Experimental — accuracy not yet measured
• cites Li 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

Kamoun et al. (2024) found that a significant portion of students exhibited a lack of understanding or dismissive attitudes toward using ChatGPT for plagiarism.

“Notably, Kamoun et al. (2024) found that a significant portion of students exhibited a lack of understanding or dismissive attitudes toward using ChatGPT for plagiarism.”

• cites Kamoun et al. (2024) sourceSupported

Classified as probably not a substantial claim. May combine two claims.

Generative AI literacy involves critically evaluating the appropriateness of AI use in different contexts, understanding societal impacts, and recognizing potential risks and limitations.

“It involves critically evaluating the appropriateness of AI use in different contexts, understanding the societal impacts of AI (such as intellectual property rights, equity of access, and environmental concerns), and recognizing the potential risks and limitations of these technologies (Zhao et al. 2024).”

• cites Zhao et al. 2024 sourceSupported

May combine two claims.

Polyportis (2024) observed that as students grew more accustomed to ChatGPT, their feelings of emotional creepiness diminished, but their trust levels decreased, leading to decreased usage.

“Polyportis (2024) observes that as students grew more accustomed to ChatGPT, their feelings of emotional creepiness (i.e., the unsettling emotions or discomfort that individuals may encounter when interacting with AI systems that mimic human emotions or intentions) diminished, but their trust levels decreased, leading to decreased usage.”

• cites Polyportis (2024) sourcePartly supported

  The Results (§3.3) show that emotional creepiness, trust, and ChatGPT usage all declined between T1 and T2; the Discussion (§4.1) associates reduced creepiness with greater familiarity. Changes in trust positively predicted changes in usage, but the paper reports an association, not proof that declining trust caused the overall drop in usage. It also finds that reduced creepiness predicted increased, rather than decreased, usage. (LLM, conf=0.85)

  Experimental — accuracy not yet measured

May combine two claims.

Ding et al. (2023) found that higher trust in ChatGPT did not necessarily correspond with receiving more accurate answers but did lead to more positive perceptions and increased usage intentions.

“For example, Ding et al. (2023) found that higher trust in ChatGPT did not necessarily correspond with receiving more accurate answers but did lead to more positive perceptions and increased usage intentions.”

• cites Ding et al. (2023) sourcePartly supported

  In Analyses and results, answer accuracy did not differ significantly between the Trust and Partial Trust groups, and the Trust group rated ChatGPT significantly easier to use. Its higher future-use intention was only borderline significant (p = 0.059). The study reports associations between trust and perceptions, not evidence that trust caused more positive perceptions or increased usage intentions. (LLM, conf=0.88)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

ChatGPT's efficiency and speed, productivity-boosting capabilities, and 24/7 availability were seen as significant advantages.

“Chat-GPT's efficiency and speed, productivity-boosting capabilities, and 24/7 availability were also seen as significant advantages (Rogers et al. 2024;Šedlbauer et al. 2024;Zhao et al. 2024).”

• cites Rogers et al. 2024 sourceNot checked — no full text of the source
• cites Šedlbauer et al. 2024 sourcePartly supported

  The Conclusions report that students found ChatGPT fast and more efficient than a web search. The paper also reports perceived productivity benefits from another study, but our copy does not show that its own students identified productivity gains or 24/7 availability as advantages. (LLM, conf=0.85)

  Experimental — accuracy not yet measured
• cites Zhao et al. 2024 sourcePartly supported

  The Findings (page not provided) report that participants felt more efficient and productive using ChatGPT, and that it sped up writing tasks and saved time. The Findings also mention immediate responses, but neither they nor the Discussion establish that participants saw 24/7 availability as an advantage. (LLM, conf=0.86)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

AI literacy encompasses not only technical knowledge but also the ability to critically evaluate the ethical implications and societal impacts of AI systems.

“AI literacy encompasses not only technical knowledge but also the ability to critically evaluate the ethical implications and societal impacts of AI systems (Carolus et al. 2023;Laupichler et al. 2023;Long and Magerko 2020;Ng et al. 2024;Touretzky et al. 2019;Wang et al. 2023).”

• cites Carolus et al. 2023 sourceSupported
• cites Laupichler et al. 2023 sourceNot checked — no full text of the source
• cites Long and Magerko 2020 sourceNot checked — no full text of the source

May combine two claims.

The assessment revealed common areas where students' prompts could be improved, including lack of specificity in the desired output format, insufficient information about the target audience, absence of examples or sample outputs, ambiguity in the scope and depth of the explanation, and lack of iterative refinement.

“The assessment reveals common areas where students' prompts could be improved, including • Lack of specificity in the desired output format • Insufficient information about the target audience • Absence of examples or sample outputs • Ambiguity in the scope and depth of the explanation • Lack of iterative refinement”

• AssumptionRests on an assumption: The author-developed rubric validly measures students' prompt creation skills.

May combine two claims.

In the Padlet reflection, students expressed concerns about information inaccuracies, privacy, authorship (especially of their own work), locating accurate sources, and how the technology can take away from a person's creativity and originality.

“For the first question, students expressed concerns about information inaccuracies, privacy, authorship (especially of their own work), locating accurate sources, and how the technology can take away from a person's creativity and originality.”

• Also stated in: § Discussion · plain

  “The Padlet reflection activity revealed students' concerns about information inaccuracies, privacy, authorship, and potential impacts on creativity, echoing ethical issues discussed in the literature.”

May combine two claims.

Of the 36 students who responded to the true/false poll, 53% incorrectly voted that ChatGPT is trained via large volumes of internet material without human involvement, while 47% correctly voted false.

“Of the thirty-six students who responded, 53 percent of participants incorrectly voted that the answer was true, while 47 percent correctly voted that the answer was false (see table 3).”

May combine two claims.

When asked about limitations of ChatGPT in achieving their goal/task, students mentioned lack of specificity or understanding of the prompt, concerns about accuracy or reliability of information, lack of depth or clarity in provided information, and misunderstandings or need for further prompting; some students reported no limitations or issues.

“When asked about the limitations of ChatGPT in achieving their goal/ task, students mentioned several issues: • Lack of specificity or understanding of the prompt • Concerns about the accuracy or reliability of information • Lack of depth or clarity in the provided information • Misunderstandings or the need for further prompting Some students reported no limitations or issues.”

May combine two claims.

In the pre-survey, 49% of students disagreed or strongly disagreed with attributing humanlike qualities to ChatGPT, while 43% remained neutral.

“A significant portion of students (49 percent) disagreed or strongly disagreed with attributing humanlike qualities to ChatGPT, while 43 percent remained neutral.”

May combine two claims.

Students performed best on the 'Avoid ambiguity' and 'Clarity' criteria, with mean scores of 2.43 and 2.33, respectively; they struggled most with 'Examples' and 'Tinkering', scoring means of 1.48 and 1.00, respectively.

“Students performed best on the”

• AssumptionRests on an assumption: The author-developed rubric validly measures students' prompt creation skills.

Classified as probably not a substantial claim.

Faculty reported that students found the workshop's tone conflicting, as it presented both ethical concerns and practical usage guidance.

“Faculty reported that students found the workshop's tone conflicting, as it presented both ethical concerns and practical usage guidance.”

Probably not a substantial claim: may not pass S3 (load-bearing).

In the mid-workshop poll (n=38), 18% of students (n=7) identified exploitation of invisible labor to build AI as the biggest ethical threat.

“Exploitation of invisible labor to | 18% (n = 7)”

Probably not a substantial claim: may not pass S3 (load-bearing).

In the mid-workshop poll (n=38), 3% of students (n=1) identified political advertising and AI as the biggest ethical threat.

“Political advertising and AI | 3% (n = 1)”

Probably not a substantial claim: may not pass S3 (load-bearing).

In the CREATE rubric assessment (n=21), the mean score for Tinker was 1.00.

“Tinker | 21 | 0 | 0 | 1.00”

Probably not a substantial claim: may not pass S3 (load-bearing).

In the CREATE rubric assessment (n=21), the mean score for Avoid ambiguity was 2.43.

“Avoid ambiguity | 2 | 8 | 11 | 2.43”

Probably not a substantial claim: may not pass S3 (load-bearing).

In the CREATE rubric assessment (n=21), the mean score for Relevant information was 1.90.

“Relevant information | 9 | 5 | 7 | 1.90”

Probably not a substantial claim: may not pass S3 (load-bearing).

In the CREATE rubric assessment (n=21), the Evaluate criterion was not applicable (N/A).

“Evaluate | N/A | N/A | N/A | N/A”

Probably not a substantial claim: may not pass S3 (load-bearing).

In the mid-workshop poll (n=38), 18% of students (n=7) identified privacy violations of AI as the biggest ethical threat.

“Privacy violations of AI | 18% (n = 7)”

Probably not a substantial claim: may not pass S3 (load-bearing).

In the CREATE rubric assessment (n=21), the mean score for Examples was 1.48.

“Examples | 16 | 0 | 5 | 1.48”

Probably not a substantial claim: may not pass S3 (load-bearing).

In the CREATE rubric assessment (n=21), the mean score for Clarity was 2.33.

“Clarity | 2 | 10 | 9 | 2.33”

May combine two claims.

The majority of students were unable to provide an alternative example of a large language model besides ChatGPT, indicating a limited awareness of other large language models beyond ChatGPT.

“The majority of students were unable to provide an alternative example, indicating a limited awareness of other large language models beyond ChatGPT.”

Probably not a substantial claim: may not pass S3 (load-bearing).

In the mid-workshop poll (n=43), 14% of students (n=6) reported never having used ChatGPT.

“I have never used it. | 14% (n = 6)”

Probably not a substantial claim: may not pass S3 (load-bearing).

In the mid-workshop poll (n=38), 5% of students (n=2) identified environmental damage caused by AI as the biggest ethical threat.

“Environmental damage caused by AI | 5% (n = 2)”

Probably not a substantial claim: may not pass S3 (load-bearing).

In the pre-survey (n=37), 49% of students (n=18) strongly disagreed or disagreed that they tend to anthropomorphize or attribute humanlike qualities to ChatGPT.

“I tend to anthropomorphize or | 49% (n = 18) | 43% (n = 16) | 8% (n = 3)”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Educating the public on AI literacy helps people learn about ethical considerations and make informed choices about AI use.

“Educating the public on AI literacy helps people learn about these ethical considerations and make informed choices about AI use.”

Probably not a substantial claim: may not pass S3 (load-bearing).

There is a risk of personally identifiable information being included in data scraped from online sources and subsequently extracted from trained language models.

“There is a risk of personally identifiable information being included in the data scraped from online sources and subsequently extracted from the trained language models.”

• cites Stanford University 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

This extraction is a form of digital colonialism that has implications for the environment.

“This is a form of digital colonialism that has implications for the environment.”

• cites Schelenz 2022 sourceNot checked — no full text of the source

May combine two claims.

Language models can reproduce copyrighted text verbatim from their training data, posing intellectual property risks if such material is included in datasets sourced from the internet.

“Furthermore, language models can reproduce copyrighted text verbatim from their training data, posing intellectual property risks if such material is included in datasets sourced from the internet (Stanford University 2024).”

• cites Stanford University 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

Labeling violent data in AI systems has a detrimental impact on workers' physical and mental health because they are exposed to this information nine hours every day.

“This type of work in labeling violent data in AI systems has a detrimental impact on workers' physical and mental health because they are exposed to this information nine hours every day (Furze, n.d.;Perrigo 2023).”

• cites Furze, n.d.Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Perrigo 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Users may be unaware of the quality of information they receive, as AI can appear and sound humanlike, potentially misleading users into trusting the information provided without proper verification.

“A related concern is whether users are aware of the quality of information they receive, as AI can appear and sound humanlike, potentially misleading users into trusting the information provided without proper verification (Balmer 2023).”

• cites Balmer 2023 sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

AI systems require vast amounts of training data, often scraped from the internet without permission, raising concerns about privacy and intellectual property rights.

“According to the Artificial Intelligence Index Report (Stanford University 2024), AI systems require vast amounts of training data, often scraped from the internet without permission, raising concerns about both privacy and intellectual property rights.”

• cites Stanford University 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

Generative fluency without evaluative checks raises concerns about false and incorrect output from LLMs, called hallucinations.

“Furthermore, the generative fluency without evaluative checks that typically characterizes content generated by LLM queries raises concerns about false and incorrect output from them, called hallucinations (Furze, n.d.).”

• cites Furze, n.d.Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Extraction causes political and societal upheavals that can trigger violations of human rights, starvation, and genocide.

“This extraction causes political and societal upheavals that can trigger violations of human rights, starvation, and genocide (Schelenz 2022).”

• cites Schelenz 2022 sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

AI literacy empowers students to take agency over the technology and actively participate in shaping its development and application.

“Furthermore, AI literacy empowers students to take agency over the technology and actively participate in shaping its development and application.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The prompt engineering segment incorporated best practices identified in the literature review, such as the CREATE framework and the importance of clarity, relevance, examples, specifying AI role, iteration, and evaluation in crafting effective prompts.

“The prompt engineering segment incorporated best practices identified in the literature review, such as the CREATE framework (Barrett 2023) and the importance of clarity, relevance, examples, specifying AI role, iteration, and evaluation in crafting effective prompts (Giray 2023;Gruber, n.d.;Jacobs and Fisher 2023;Lo 2023;Mollick 2023;Ryan 2024).”

• cites Barrett 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Giray 2023 sourceNot checked — no full text of the source
• cites Gruber, n.d.Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

Several studies found that students perceived ChatGPT as a valuable tool for enhancing their learning experiences.

“Several studies found that students perceived ChatGPT as a valuable tool for enhancing their learning experiences.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

Significant interest from faculty and staff suggests a need for AI literacy training across the entire campus community.

“Significant interest from faculty and staff suggests a need for AI literacy training across the entire campus community.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The workshop's emphasis on ethical considerations aligned with the growing recognition of the importance of fostering a critical understanding of AI's capabilities and limitations.

“The workshop's emphasis on ethical considerations aligned with the growing recognition of the importance of fostering a critical understanding of AI's capabilities and limitations (Damiano et al. 2024;Das and Madhusudan 2024;Nam 2023;Rogers et al. 2024;Yan 2023).”

• cites Damiano et al. 2024 sourceNot checked — no full text of the source
• cites Das and Madhusudan 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Nam 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Students recognize the potential utility of ChatGPT in their learning process but maintain a pragmatic view of its role.

“These findings suggest that while students recognize the potential utility of ChatGPT in their learning process, they maintain a pragmatic view of its role.”

Probably not a substantial claim: may not pass S3 (load-bearing).

AI literacy is distinct from digital literacy, focusing on the advancing technologies powering digital platforms rather than just evaluating and using them.

“It is distinct from digital literacy, focusing on the advancing technologies powering digital platforms rather than just evaluating and using them (Wheatley and Hervieux 2022).”

• cites Wheatley and Hervieux 2022Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

While the workshop boosted overall confidence, there is a need for further clarity and emphasis on the technical aspects of ChatGPT's operation.

“This suggests that while the workshop boosted overall confidence, there is a need for further clarity and emphasis on the technical aspects of ChatGPT's operation.”

May combine two claims.

Academic librarians possess unique expertise in information ethics, privacy, and intellectual freedom, which is highly relevant to AI literacy education.

“Academic librarians possess unique expertise in information ethics, privacy, and intellectual freedom, which is highly relevant to AI literacy education (Michalak 2023).”

• cites Michalak 2023 sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing).

University libraries play a crucial role in providing AI literacy education to students, guided by principles set forth by the Association of Research Libraries (2024) for the ethical and transparent use of AI.

“University libraries play a crucial role in providing AI literacy education to students, guided by principles set forth by the Association of Research Libraries (2024) for the ethical and transparent use of AI.”

• cites Association of Research Libraries (2024)Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S2 (checkable).

The persistent misconception that ChatGPT searches the internet indicates a need for clearer explanation of the technology's functioning in future workshops.

“This persistent misconception indicates a need for clearer explanation of the technology's functioning in future workshops.”

Probably not a substantial claim: may not pass S2 (checkable).

The CREATE framework may have been too complex for some participants.

“The CREATE framework may have been too complex for some participants.”

Classified as probably not a substantial claim.

Students who possess AI literacy skills are better equipped to understand the potential biases and limitations of AI technology, enabling them to approach it with a critical lens.

“Students who possess AI literacy skills are better equipped to understand the potential biases and limitations of AI technology, enabling them to approach it with a critical lens.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The experimental nature of the assessment tools, particularly the CREATE rubric, requires further validation to ensure their reliability and validity across different settings.

“Furthermore, the experimental nature of our assessment tools, particularly the CREATE rubric, requires further validation to ensure their reliability and validity across different settings.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Reliance on self-reported data may introduce bias into the results.

“From a methodological standpoint, our reliance on self-reported data may introduce bias into our results.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The five-point Likert scale allowed for a nuanced assessment of students' attitudes, capturing degrees of agreement or disagreement rather than simple yes/no responses.

“This five-point scale allowed for a nuanced assessment of students' attitudes, capturing degrees of agreement or disagreement rather than simple yes/no responses.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The CREATE rubric scoring is based on a sample size of twenty-one, while the follow-up survey questions have a sample size of twenty-eight.

“Therefore, the CREATE rubric scoring is based on a sample size of twenty-one, while the follow-up survey questions have a sample size of twenty-eight.”

• AssumptionRests on an assumption: The sample of one sixty-minute workshop with forty-three first-year students is representative enough to generalize to undergraduate students.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

BabyGPT initially produced nonsensical output but gradually learned letters, words, and coherent sentences, demonstrating how LLMs build conversational abilities.

“BabyGPT initially produced nonsensical output but gradually learned letters, words, and coherent sentences, demonstrating how LLMs build conversational abilities.”

• cites Bhatia's (2023)Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

Applying a postphenomenological lens to our results reveals the complex human-technology relationship in ChatGPT literacy.

“Applying a postphenomenological lens to our results reveals the complex human-technology relationship in ChatGPT literacy.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The short-term nature of the assessment may not capture long-term impacts or behavioral changes resulting from the workshop.

“Additionally, the short-term nature of our assessment may not capture long-term impacts or behavioral changes resulting from the workshop.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Our findings align with several key themes identified in the literature review, particularly in the areas of user studies with students on ChatGPT.

“Our findings align with several key themes identified in the literature review, particularly in the areas of user studies with students on ChatGPT.”

Classified as probably not a substantial claim.

The assessment data discussed in the following sections is drawn from one of the sixty-minute workshops that had forty-three first-year student participants.

“The assessment data discussed in the following sections is drawn from one of the sixty-minute workshops that had forty-three first-year student participants.”

• AssumptionRests on an assumption: The sample of one sixty-minute workshop with forty-three first-year students is representative enough to generalize to undergraduate students.

May combine two claims.

The study's sample was limited to undergraduate students at a single private university in California, which constrains the generalizability of the findings to other educational contexts.

“Our study's sample was limited to undergraduate students at a single private university in California, which constrains the generalizability of our findings to other educational contexts.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The difference in sample sizes reflects varying levels of completion within the Prompt Crafting worksheet activity.

“This difference in sample sizes reflects the varying levels of completion within the Prompt Crafting worksheet activity.”

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

The findings contribute to the growing body of knowledge on AI literacy education in university libraries, emphasizing the importance of incorporating ethical considerations and critical reflection on the human-technology relationship.

“Our findings contribute to the growing body of knowledge on AI literacy education in university libraries, emphasizing the importance of incorporating ethical considerations and critical reflection on the humantechnology relationship.”