CoChat
Sign Up
Cited by 46Innovative Teaching and Learning MethodsIntelligent Tutoring Systems and Adaptive LearningEducational Technology in Learning

Designing learning technology collaboratively: analysis of a chatbot co-design

Open access
Durall Gazulla, Eva, Martins, Ludmila, Fernández-Ferrer, Maite·2023·Education and Information Technologies
DOI 10.1007/s10639-022-11162-wPMC PMC9226288
Open this paper in CoChat — free accountOpen PDF

Abstract

Collaborative design approaches have been increasingly adopted in the design of learning technologies since they contribute to develop pedagogically inclusive and appropriate learning designs. Despite the positive reception of collaborative design strategies in technology-enhanced learning, little attention has been dedicated to analyzing the challenges faced in design processes using a collaborative approach. In this paper, we disclose the collaborative design of a chatbot for self-regulated learning in higher education using an action research approach. We analyze the design process of EDUguia chatbot, which includes diverse evidence from questionnaires and workshops with students and lecturers, as well as intermediary design objects. Based on the qualitative analysis, we identify several challenges that are transversal to the co-design work, as well as specific to the design phases. We critically reflect on the strategies deployed to overcome these challenges and how they relate to decision-making processes, highlighting the need to make stakeholders’ tacit knowledge explicit, cultivate trust-building and support democratic decision-making in technology design processes. We believe that the recommendations we present in this paper contribute to developing best practices in the collaborative design of chatbots for the self-regulation of learning, as well as learning technology in general.

AI snapshot

Written from the paper’s full text.

TL;DR

This action research study analyzes the collaborative design of EDUguia, a chatbot for self-regulated learning in higher education, involving 12 students, 7 lecturers, and 22 researchers/lecturers. It identifies transversal and phase-specific collaboration challenges and strategies, and argues for distinguishing “big” decisions requiring consensus from “small” decisions needing stakeholder feedback.

Key findings

  • The co-design process surfaced two transversal challenges: addressing diverse needs while ensuring relevance, and sustaining stakeholders’ continuous involvement; strategies included assessing how critical a need was, supporting negotiation, identifying key input moments, and scheduling short micro-feedback.
  • Phase-specific challenges were identified for Understanding (unveiling tacit assumptions, interpreting meanings, ensuring technological literacy), Defining (making diverse needs explicit, supporting discussion of alternatives, moving from abstract to concrete), and Shaping (translating research into practice, reaching agreements); strategies included shared vocabularies, debriefing, visual examples, prototyping, intermediary objects, and voting.
  • The EDUguia design requirements were use across disciplines, theory-based, immediate feedback, easy to understand, brief and smooth interactions, adaptation to existing technology habits, accessible and inclusive, limited personal data collection, and data transparency (Table 5).
  • The paper distinguishes big decisions made by consensus (e.g., use across disciplines, theory-based, immediate feedback, adaptation to existing technology habits, accessible and inclusive, limited personal data collection, data transparency) from small decisions where students’ views were prioritized via voting (ease of understanding, brief and smooth interactions); pilot tests were planned in eight courses.

Methods

  • Action research cycles of planning, observing, and reflecting across three design phases: Understanding, Defining, and Shaping; qualitative thematic analysis followed Braun and Clarke’s phases with three researchers involved.
  • Participants were 12 undergraduate students from six bachelor degrees, 7 lecturers from those degrees, and 22 researchers/lecturers from two Spanish universities with diverse backgrounds.
  • Data came from pre-workshop questionnaires, two online co-design workshops, post-workshop questionnaires, project communications such as internal notes and meeting minutes, and intermediate design objects including chatbot drafts, design requirements, style guide, flowchart, and script change log.

Limitations

  • Stated: the study is limited in the number of participants and technologies examined; phase boundaries are blurry and non-linear; time and resource constraints meant not all stakeholders could be involved in all decisions; trust-building with students was limited to online workshops; thorough examination of participants’ expressions was only possible when analyzing session documentation.
  • Evident from the text: this is a single-case action research study without comparative or outcome evaluation, and the paper reports early feedback and planned pilot tests rather than final learning outcomes.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1007/s10639-022-11162-w

Cites

68 of 68 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 52 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    The challenges for supporting understanding related to unveiling tacit assumptions.

    “Such challenges related to unveiling tacit assumptions, interpreting meanings, and ensuring a certain technological literacy among stakeholders.”
    Quote in the paperSubstantial claim
  • Findingown data

    The challenges for supporting understanding related to ensuring a certain technological literacy among stakeholders.

    “Such challenges related to unveiling tacit assumptions, interpreting meanings, and ensuring a certain technological literacy among stakeholders.”
    Quote in the paperNot a substantial claim
  • Findingown data

    Challenges for prioritizing among diverse needs relate to aspects such as assessing the impact that addressing a particular need might have for the rest of the stakeholders.

    “The challenges for prioritizing among diverse needs relate to various aspects such as assessing the impact that addressing a particular need might have for the rest of the stakeholders.”
    Quote in the paperNot a substantial claim
  • Findingown data

    The analysis of discussions with students and lecturers during co-design workshops, and other stakeholders' wishes and needs, informed the definition of the key requirements for the chatbot design.

    “The analysis of the discussions with students and lecturers taking place during the co-design workshops, as well as of other stakeholders’ wishes and needs informed the definition of the key requirements for the chatbot design.”
    Quote in the paperSubstantial claim
  • Findingown data

    During the early stages of the EDUguia design (Understanding), several challenges for supporting stakeholders' mutual understanding and therefore collaboration were noticed.

    “During the early stages of the EDUguia design (Understanding), we noticed several challenges for supporting the stakeholders’ mutual understanding and therefore, collaboration.”
    Quote in the paperSubstantial claim
  • May combine two claims. 33
  • Classified as probably not a substantial claim. 43

  • May combine two claims.

    Findingnone given

    The intermediary objects were the basis for the design of the EDUguia chatbot functional prototypes, which were shared for feedback and revised accordingly.

    “The intermediary objects were the basis for the design of the EDUguia chatbot functional prototypes, which were shared for feedback and revised accordingly.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    In the lecturers' view, the chatbot provided valuable resources, and the tool interface and language were friendly and easy to understand.

    “In the lecturers’ view, the chatbot provided valuable resources, and the tool interface and language were friendly and easy to understand.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The strategy of encouraging stakeholders to face conflicts by pointing at opposing needs and dilemmas helped channel the conversation to hot topics, but moving away from dualistic thinking was hard.

    “While this strategy helped to channel the conversation to address hot topics, moving away from dualistic thinking was hard.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingreasoning

    The EDUguia chatbot was meant to be integrated in a learning environment and was informed by pedagogical theory.

    “The EDUguia chatbot was meant to be integrated in a learning environment and thus, it was informed by pedagogical theory.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The key requirements for the EDUguia chatbot design were: use across disciplines, theory-based, immediate feedback, easy to understand, brief and smooth interactions, adaptation to existing technology habits, accessible and inclusive, limited personal data collection, and data transparency.

    “Specifically, the chatbot design requirements consisted in: use across disciplines, theory-based, immediate feedback, easy to understand, brief and smooth interactions, adaptation to existing technology habits, accessible and inclusive, limited personal data collection, data transparency (Table 5).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    While listening to diverse stakeholders' needs is important, sometimes it is not possible to address all of them in the design solution.

    “While we acknowledge the importance of listening to diverse stakeholders’ needs during the design process, sometimes it is not possible to address all of them in the design solution.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Although the challenge of stakeholders' availability refers to logistics, it was a significant obstacle for stakeholders' participation throughout the project.

    “Although this challenge refers to logistics, it was a significant obstacle for the stakeholders' participation throughout the project.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingnone given

    In the EDUguia design process, shaping was grounded on a solid understanding of the socio-cultural context and informed by a careful definition of the stakeholders' diverse needs and wishes.

    “In the EDUguia design process, shaping was grounded on a solid understanding of the socio-cultural context and informed by a careful definition of the stakeholders’ diverse needs and wishes.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The key challenges that arose during the shaping phase are related to translating research into practice and reaching agreements.

    “The key challenges that arose during this phase are related to translating research into practice and reaching agreements.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Building on lecturers' existing practices and expertise was a valuable strategy to move from the abstract to the concrete, enabling lecturers to develop ownership of the chatbot and envision how it would integrate in their teaching.

    “Building on lecturers’ existing practices and expertise was a valuable strategy to move from the abstract to the concrete, enabling lecturers develop ownership of the chatbot and envision how this new tool would integrate in their teaching.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The development of intermediary documents such as rubrics and a chatbot style guide supported communication with researchers while helping to shape the chatbot design.

    “The development of intermediary documents to guide the design such as rubrics and a chatbot style guide supported communication with researchers while helping to shape the chatbot design.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Some students' comments confirmed the need to schedule synchronous sessions to discuss their experiences using the chatbot for supporting self-regulation, and to revise the chatbot contents based on the students' background and field of knowledge.

    “Some of the students’ comments confirmed the need to schedule synchronous sessions to discuss their experiences using the chatbot for supporting self-regulation, and to revise the chatbot contents based on the students’ background and field of knowledge.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    After shaping the chatbot into a functional prototype, lecturers were positive about its potential for supporting students' learning process, improving their performance and abilities to manage time, reflect, and ask for support.

    “After the shaping of the chatbot into a functional prototype, lecturers were positive about the chatbot’s potential for supporting the students' learning process, improving their performance and their abilities to manage time, reflect and ask for support to manage their academic activity.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning

    The decisions made throughout the collaborative design process led to the EDUguia chatbot, a conversational interface for supporting self-regulation of learning in student-led academic tasks, targeting higher education students from all fields of knowledge and meant as an aid for developing learning to learn skills.

    “The decisions made throughout the collaborative design process have led to the EDUguia chatbot, which is a conversational interface for supporting the self-regulation of learning in student-led academic tasks. The tool targets higher education students from all fields of knowledge and is meant as an aid for developing learning to learn skills.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingreasoning

    The differentiation between three moments (Understanding, Defining, and Shaping) helped analyze and reflect on the challenges, but the boundaries between phases are blurry and they cannot be understood as separate, linear tasks.

    “While the differentiation between three moments (Understanding, Defining, and Shaping) helped us to analyze and reflect on the challenges, it is worth noting that the boundaries between phases are blurry, and it is not possible to understand them as separate, linear tasks.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    Because of time and resource constraints, the authors could not involve all stakeholders in all decisions as they would have wished.

    “Because of time and resource constraints, we could not involve all stakeholders in all decisions, as we would have wished.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingcitation

    Small decisions in the EDUguia design were Ease of understanding and Brief and smooth interactions.

    “Small decisions in the EDUguia design were Ease of understanding and Brief and smooth interactions.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingreasoning

    Considering that the EDUguia chatbot was addressed to students, the authors prioritized students' views for small decisions, which were made by asking them to vote between different options.

    “Considering that the EDUguia chatbot was addressed to students, we prioritized their views for these decisions, which were made by asking them to vote between different options.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingreasoning

    In the EDUguia design, undertaking a thorough examination of participants' expressions was very challenging during the design process, and the authors only could do it when analyzing the session's documentation.

    “However, undertaking such a thorough examination of the participants’ expressions was very challenging during the design process, and we only could do it when analyzing the session's documentation.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    In the design of EDUguia, the big decisions stemmed from the requirements identified through the co-design workshops were Use across disciplines, Theory-based, Immediate feedback, Adaptation to existing technology habits, Accessible and inclusive, Limited personal data collection and Data transparency, and these decisions were made by consensus.

    “In the design of EDUguia, the big decisions stemmed from the requirements identified through the co-design workshops were Use across disciplines, Theory-based, Immediate feedback, Adaptation to existing technology habits, Accessible and inclusive, Limited personal data collection and Data transparency. These decisions were made by consensus.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning

    Trust-building actions with students were limited to the online co-design workshops, in which informal conversations were harder than in face-to-face interactions.

    “Our trust-building actions with students were limited to the online co-design workshops, in which informal conversations were harder than in face-to-face interactions.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Most common uses of chatbots in education contexts have focused on supporting teaching and learning, administrative tasks, student assessment, and research and development.

    “To date, most common uses of chatbots in education contexts have focused on supporting teaching and learning, administrative tasks, student assessment, and research and development (Okonkwo & Ade-Ibijola, 2021).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Okonkwo & Ade-Ibijola, 2021 source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Premisecitation

    In technology-enhanced learning, involving students during the design process is beneficial for avoiding feelings of alienation, especially when exploring emerging technologies such as automatic data collection systems and chatbots.

    “In Technology-Enhanced Learning (TEL), involving students during the design process has been also considered beneficial for avoiding feelings of alienation, especially when exploring the potential of emerging technologies for teaching and learning such as automatic data collection systems and chatbots (Durall Gazulla et al., 2020; Durall & Kapros, 2020).”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites Durall Gazulla et al., 2020

      Not checked in the free run.

    • cites Durall & Kapros, 2020

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Premisecitation

    There have been calls for involving education stakeholders in chatbot design processes, avoiding top-down approaches.

    “In particular, there have been calls for involving education stakeholders in chatbot design processes, avoiding top-down approaches (Durall Gazulla et al., 2020; Durall & Kapros, 2020; Fernández-Ferrer et al., 2021).”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites Durall Gazulla et al., 2020

      Not checked in the free run.

    • cites Durall & Kapros, 2020

      Not checked in the free run.

    • cites Fernández-Ferrer et al., 2021

      Not checked in the free run.

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    Collaborative design processes lead to solutions that are more flexible and solid in use, accessible to a greater number of people, and more adaptable to changing situations over time.

    “From a technology design perspective, it has been claimed that collaborative design processes lead to solutions that are more flexible and solid in use, accessible to a greater number of people, and more adaptable to changing situations over time (Gros & Durall, 2020).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Gros & Durall, 2020

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Trust-building and supporting joint negotiation are important aspects to consider in collaborative design processes.

    “For this to happen, trust-building and supporting joint negotiation have been also recognised as important aspects to consider in collaborative design processes (Robinson & Simonsen, 2012; Stelzle et al., 2017).”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites Robinson & Simonsen, 2012

      Not checked in the free run.

    • cites Stelzle et al., 2017 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    In the educational field, chatbots are increasingly adopted because they support immediate and personalized feedback.

    “In the educational field, chatbots are increasingly adopted since they support immediate and personalized feedback (Kumar, 2021; Okonkwo & Ade-Ibijola, 2021; Winkler & Söllner, 2018).”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites Kumar, 2021 source

      Not checked in the free run.

    • cites Okonkwo & Ade-Ibijola, 2021 source

      Not checked in the free run.

    • cites Winkler & Söllner, 2018

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Premisecitation

    Democratization of educational technology design processes is a promising strategy to develop high-quality teaching and learning.

    “The democratization of educational technology design processes has been considered a promising strategy to develop high-quality teaching and learning (Fleischmann, 2015).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Fleischmann, 2015

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable).

    Premisecitation

    Designing technology collaboratively implies generating collective processes in which all stakeholders involved have the possibility of influencing and controlling the process and the solutions generated.

    “Designing technology in a collaborative way implies generating collective processes in which all the stakeholders involved have the possibility of influencing and controlling the process and the solutions generated through it (Ehn, 1988).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Ehn, 1988

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable).

    Premisecitation

    For collaborative design to be successful, it is key to establish horizontal relationships of mutual learning in which stakeholders' knowledge and contributions are valued.

    “For collaborative design to be successful, it is key to establish horizontal relationships of mutual learning in which stakeholders’ knowledge and contributions are valued (Druin, 2014).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Druin, 2014 source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Premisecitation

    Collaborative design approaches have been increasingly adopted in learning technology design because they contribute to pedagogically inclusive and appropriate learning designs.

    “Collaborative design approaches have been increasingly adopted in the design of learning technologies since they contribute to develop pedagogically inclusive and appropriate learning designs.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Villatoro & de Benito, 2021

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    In human-computer interaction, action research has been considered a valuable approach to identifying issues of practice and providing insight on needs and challenges faced by practitioners.

    “In human–computer interaction, action research has been considered a valuable approach to identifying issues of practice, providing insight on the needs and challenges faced by practitioners (Ogunyemi et al., 2019).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Ogunyemi et al., 2019 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

    Premisecitation

    Navigating stakeholders' diverse and even opposing needs and creating consensus is not easy and requires participants to creatively explore their differences through shared discovery.

    “Navigating stakeholders’ diverse and even opposing needs and creating consensus is not easy and requires participants to creatively explore their differences through shared discovery (Atlee, 2003; Sanoff, 2021).”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites Atlee, 2003

      Not checked in the free run.

    • cites Sanoff, 2021

      Not checked in the free run.

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationown data

    The chatbot resources were expected to have effects on cognitive components (memory, attention, problem solving) and especially metacognitive and affective dimensions of learning.

    “These resources were expected to have effects on the cognitive components (memory, attention, problem solving) and, especially, the metacognitive (understanding of the learning process itself and thought processes) and affective (emotions, feelings, etc.) dimensions of learning.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Design practice and action research are both oriented towards change and transformation to improve a particular situation through the interplay of action and research moments in successive cycles.

    “As several scholars have noted, design practice and action research are both oriented towards change and transformation to improve a particular situation through the interplay of action and research moments in successive cycles (Blackler et al., 2018; Swann, 2002).”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites Blackler et al., 2018

      Not checked in the free run.

    • cites Swann, 2002 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Some strategies to reveal tacit knowledge consist of conversation analysis of users' dialogue or asking participants to express their ideas by doing and enacting.

    “As participatory design scholars have noted, some of these strategies might consist, for instance, of the conversation analysis of users’ dialogue (Luck, 2003) or asking participants to express their ideas by doing and enacting (Akach et al., 2021; Sanders et al., 2010; Spinuzzi, 2005).”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites Luck, 2003 source

      Not checked in the free run.

    • cites Akach et al., 2021

      Not checked in the free run.

    • cites Sanders et al., 2010

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    Making stakeholders' tacit assumptions explicit in collaborative design is key to ensure that the chatbot design supports the goals and needs of those who are expected to use the tool.

    “While the reasons why these ideas remain unspoken are diverse, making the stakeholders’ tacit assumptions explicit in collaborative design is key to ensure that the chatbot design supports the goals and needs of those who are expected to use the tool.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Both action and design research acknowledge the complex interrelations between humans and their context and thus embrace uncertainty and messiness as part of the process.

    “In addition, both action and design research acknowledge the complex interrelations between humans and their context and thus embrace uncertainty and messiness as part of the process (Blackler et al., 2018).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Blackler et al., 2018

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    The design process is often described as messy, and such messiness has an impact on collaboration since it makes communication and tracking decisions harder.

    “As designers, we were responsible for leading a process that has been often described as messy (Akama, 2009; Blackler et al., 2018; Sanders & Stappers, 2008). Such messiness has an impact on collaboration since it makes communication and tracking decisions harder.”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites Akama, 2009

      Not checked in the free run.

    • cites Blackler et al., 2018

      Not checked in the free run.

    • cites Sanders & Stappers, 2008 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Recent studies have shown that the use of conversational agents could be useful to support students' self-regulated learning in online environments.

    “Recent studies have shown that the use of conversational agents could be useful to support students’ self-regulated learning in online environments (Scheu & Benke, 2022; Song & Kim, 2021).”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites Scheu & Benke, 2022

      Not checked in the free run.

    • cites Song & Kim, 2021 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductcitation

    In this study, three researchers were involved in the thematic analysis phases identified by Braun and Clarke (2006), including familiarizing with the data, generating initial codes, searching for themes, reviewing themes, defining and naming themes, and producing a report.

    “In this study, three researchers have been involved in the phases identified by Braun and Clarke (2006) which include familiarizing with the data, generating initial codes, searching for themes, reviewing themes, defining, and naming themes, and producing a report based on the results.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Braun & Clarke, 2006 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Design phases are characterized by moving from a fuzzy phase (Understanding) to progressively narrowing the design space (Defining) until arriving at a crisp phase (Shaping) in an iterative fashion.

    “As is common in design processes, design phases are characterized for moving from a fuzzy phase (here referred as Understanding) to progressively narrowing the design space (Defining) until arriving at a crisp phase (Shaping) in an iterative fashion (Design Council, 2007; Tschimmel, 2012; Wolniak, 2017), as it is presented in Fig. 2.”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites Design Council, 2007

      Not checked in the free run.

    • cites Tschimmel, 2012

      Not checked in the free run.

    • cites Wolniak, 2017

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Supporting mutual understanding between different stakeholders has traditionally been one of the key missions of participatory processes.

    “Traditionally, supporting mutual understanding between different stakeholders has been one of the key missions of participatory processes (Sanoff, 2011).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Sanoff, 2011 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Researchers have pointed out the potential of chatbot technologies for supporting students to self-regulate their learning activity.

    “Researchers have pointed out the potential of chatbot technologies for supporting students to self-regulate their learning activity (Fernández-Ferrer et al., 2021; Scheu & Benke, 2022).”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites Fernández-Ferrer et al., 2021

      Not checked in the free run.

    • cites Scheu & Benke, 2022

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    It is critical to find strategies that help reveal stakeholders' tacit knowledge.

    “For this to happen, it is critical to find strategies that help reveal stakeholders’ tacit knowledge.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

    Interpretationown data

    The feedback can be taken as a sign of the collaborative design worth for aligning the tool with the expected beneficiaries' needs and values, as well as to open the design process.

    “we consider this feedback can be taken as a sign of the collaborative design worth for aligning the tool with the expected beneficiaries’ needs and values, as well as to open the design process.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    Based on the authors' experience in technology-enhanced learning, the identification of specific strategies that support mutual understanding about learning and technology among education stakeholders would contribute to making learning technology design processes more open and democratic.

    “Based on our experience in technology-enhanced learning, we believe that the identification of specific strategies that contribute to support mutual understanding about learning and technology among education stakeholders would contribute to making learning technology design processes more open and democratic.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The circumstance that thorough examination of participants' expressions is challenging during the design process and only possible when analyzing documentation is common in research and innovation projects.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Interpretationnone given

    While mutual understanding is key in collaborative decision making, it does not impede conflicts from occurring.

    “While mutual understanding is key in collaborative decision making, it does not impede conflicts from occurring.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Building trust has been pointed out to support equitable collaborations and power sharing.

    “Among these mechanisms, building trust has been pointed out to support equitable collaborations and power sharing (Clarke et al., 2021; Hillgrenet al., 2011; Pirinen, 2016; Warwick, 2017).”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites Clarke et al., 2021 source

      Not checked in the free run.

    • cites Hillgrenet al., 2011

      Not checked in the free run.

    • cites Pirinen, 2016

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    For small decisions, asking for feedback at certain points might be enough.

    “In contrast, for small decisions we consider that asking for feedback at certain points might be enough.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Big design decisions relate to values and concepts which reflect in the visions guiding the project, as well as in the ways about how such visions are implemented.

    “According to Bratteteig and Wagner, big design decisions relate to values and concepts which reflect in the visions guiding the project, as well as in the ways about how such visions are implemented (2012).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Bratteteig & Wagner, 2012

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

    Interpretationreasoning

    Big design decisions require consensus and thus the decision stream needs to stop until an agreement is reached.

    “From our perspective, big design decisions require consensus and thus the decision stream needs to stop until an agreement is reached.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationnone given

    The level of analysis required for unveiling implicit meanings takes time and a certain distance to critically reflect on it.

    “While these strategies have been widely adopted in participatory and co-design processes, it is worth noting that the level of analysis required for unveiling implicit meanings takes time and a certain distance to critically reflect on it.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Users are frequently involved in small decisions regarding formal aspects to ensure the final solution is familiar and easy to make sense of.

    “As Bratteteig and Wagner note (2012), users are frequently involved in small decisions regarding formal aspects to ensure the final solution is familiar and easy to make sense of.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Bratteteig & Wagner, 2012

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Premisecitation

    Developing shared vocabularies between designers and stakeholders can help to share meanings and create mutual understanding.

    “Developing shared vocabularies between designers and stakeholders can help to share meanings and create mutual understanding (Luck, 2003; Martin-Hammond et al., 2018).”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites Luck, 2003 source

      Not checked in the free run.

    • cites Martin-Hammond et al., 2018

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationnone given

    The differentiation between big and small decisions is relevant because not all decisions have the same importance.

    “We consider the differentiation between big and small decisions relevant, because not all decisions have the same importance.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    The differentiation between big and small decisions is valuable to assess when consensus is needed.

    “The differentiation between big and small decisions is also valuable to assess when we need to ensure a consensus is reached.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationcitation

    The creation of an intermediate material such as the EDUguia style guide reflects the idea that design decisions are interrelated in diverse ways.

    “The creation of an intermediate material such as the EDUguia style guide reflects the idea that design decisions are interrelated in diverse ways (Bratteteig & Wagner, 2012).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Bratteteig & Wagner, 2012

      Not checked in the free run.

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given

    The study is limited in terms of the number of participants and the technologies examined, but the implications for practice outlined are valuable for designers and researchers developing learning technologies such as chatbots for education contexts.

    “While the study presented in this paper is limited in terms of the number of participants and the technologies examined, we believe the implications for practice outlined in this study are valuable for designers and researchers developing learning technologies such as chatbots for education contexts.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    Design decisions can be classified as big decisions and small decisions, decisions internal to the project and related to the outside world, and decisions that might be called non-decisions.

    “Following Bratteteig and Wagner, design decisions can be classified “big decisions and small decisions, decisions internal to the project and related to the outside world, and decisions that might be called non-decisions.” (Bratteteig & Wagner, 2012, p.41).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Bratteteig & Wagner, 2012

      Not checked in the free run.

  • Classified as probably not a substantial claim. May combine two claims.

    Premisenone given

    Interpersonal relations have been considered key for trust-building, and there have been calls for supporting informal exchanges and spending time with stakeholders beyond project-focused activities.

    “Interpersonal relations have been considered key for trust-building and there have been calls for supporting informal exchanges and spending time with stakeholders, beyond the project-focused activities.”
    Quote in the paperSubstantial claim
61 of 61

  • The paper relies on this without support.

    Little attention has been dedicated to analyzing the challenges faced in design processes using a collaborative approach.

    “Despite the positive reception of collaborative design strategies in technology-enhanced learning, little attention has been dedicated to analyzing the challenges faced in design processes using a collaborative approach.”

  • The paper relies on this without support.

    There has been no debate around the diverse challenges that might hamper the collaborative design of learning technologies such as chatbots.

    “To the best of our knowledge, there has been no debate around the diverse challenges that might hamper the collaborative design of learning technologies such as chatbots.”

  • The paper relies on this without support.

    None of the existing studies have focused on the challenges faced in collaborative design processes in higher education and how these challenges affect decision-making processes.

    “However, to the best of our knowledge, none of these studies have focused on the challenges faced in the collaborative design processes in higher education and how these challenges affect decision-making processes.”

  • The paper relies on this without support.

    The circumstance that thorough examination of participants' expressions is challenging during the design process and only possible when analyzing documentation is common in research and innovation projects.

    “Considering this circumstance to be quite common in research and innovation projects”

  • The paper relies on this without support.

    Time and resource constraints preventing the involvement of all stakeholders in all decisions is a common situation in design.

    “Since this may be a common situation in design”
5 of 5

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The inductive themes of navigating complexity, unpredictability, serendipity, creativity, and adaptability in the cGPT projects illustrate a dynamic, dialogic learning process.

    “The inductive themes of navigating complexity, unpredictability, serendipity, creativity , and adaptability in the cGPT projects illustrate a dynamic, dialogic learning process (Durall Gazulla et al., 2023).”

    From Co-creating custom GPTs: an autoethnographic study of undergraduate students as partners in generative AI innovation (2026) · cites this paper as Durall Gazulla et al., 2023

    The Background section, “Collaborative design of learning technology” (page unavailable), describes learning as a process built over time in which interaction and dialogue play a fundamental role. The Methodology and Discussion also describe uncertainty and creative negotiation in the paper’s EDUguia chatbot co-design. Our copy does not report cGPT projects or identify the listed concepts as inductive themes of those projects. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

Cites

Flagged first, then by citations

Research works printed but not linked: 7

  1. Atlee, T. (2003). The Tao of Democracy: Using Co-Intelligence to Create a World that Works for All. The Writers Collective.
  2. Bonsignore, E., Ahn, J., Clegg, T., Guha, M. L., Hourcade, J. P., Yip, J. C., & Druin, A. (2013). Embedding Participatory Design into Designs for Learning: An Untapped Interdisciplinary Resource?. In N. Rummel, M. Kapur, M.J. Nathan, S. Puntambekar (Eds.), Proceedings of 10th International Conference on Computer-Supported Collaborative Learning (pp. 549–556). https://doi.org/10.22318/cscl2013.1.549 · DOI
  3. Calle, M., Narváez, E., & Maldonado-Mahauad, J. (2020). Proposal for the Design and Implementation of Miranda: A Chatbot-Type Recommender for Supporting Self-Regulated Learning in Online Environments. Retrieved June 6, 2022, from: http://ceur-ws.org/Vol-3059/paper2.pdf
  4. Council of the European Union. (2018). Council Recommendation of 22 May 2018 on Key Competences for Lifelong Learning. Official Journal of the European Union. Accessed 6 March 2022. Retrieved from: https://eurlex.europa.eu/legalcontent/EN/TXT/?uri=CELEX:32018H0604(01)
  5. Design Council. (2007). Eleven lessons: Managing design in eleven global companies-desk research report. Design Council.
  6. Druin, A. (2014). Inclusive ownership of participatory learning. Instructional Science, 42(1), 123–126. https://doi.org/10.1007/s11251-014-9339-0 · DOI · Paper page
  7. Ehn, P. (1988). Work-Oriented Design of Computer Artefacts. Lawrence Erlbaum Associates.

Cited by

Sorted by citations

Related

Authors
Durall Gazulla, E., Martins, L., & Fernández-Ferrer, M.
Affiliations
  • Universitat de Barcelona (ES)
  • Universitat Oberta de Catalunya (ES)
ORCID
Venue
Education and Information Technologies
Publisher
Springer Science and Business Media LLC
Published
2022-06-24
Volume
28
Issue
1
Pages
109-134
ISSN
  • 1360-2357 (ISSN-L)
  • 1573-7608
DOI
10.1007/s10639-022-11162-w
PMC
PMC9226288
OpenAlex
W4283392104
Cited by
46
References
68
Citations by year
2026: 8 · 2025: 10 · 2024: 14 · 2023: 14
FWCI
14.46
Type
journal article
Language
en
Update notices
correction (2022-10-04)
Open access
Licence
Locations
  • Education and Information Technologies (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
Funders
  • Ministerio de Ciencia, Innovación y Universidades (PID2019-104285GB-I00)
  • University of Oulu including Oulu University Hospital
Topics
  • Social Sciences › Psychology › Developmental and Educational Psychology › Innovative Teaching and Learning Methods
  • Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
  • Physical Sciences › Computer Science › Information Systems › Educational Technology in Learning
Keywords
co-design, chatbots, educational technology design, student self-regulation, higher education, design challenges, democratic decision-making, trust building, technology-enhanced learning

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-10.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-10.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review