Involving Users to Improve the Collaborative Logical Framework
Open accessThe DOI matches its Crossref record; 51 of the 53 references it cites were found in a registry or an index.
Abstract
In order to support collaboration in web-based learning, there is a need for an intelligent support that facilitates its management during the design, development, and analysis of the collaborative learning experience and supports both students and instructors. At aDeNu research group we have proposed the Collaborative Logical Framework (CLF) to create effective scenarios that support learning through interaction, exploration, discussion, and collaborative knowledge construction. This approach draws on artificial intelligence techniques to support and foster an effective involvement of students to collaborate. At the same time, the instructors’ workload is reduced as some of their tasks—especially those related to the monitoring of the students behavior—are automated. After introducing the CLF approach, in this paper, we present two formative evaluations with users carried out to improve the design of this collaborative tool and thus enrich the personalized support provided. In the first one, we analyze, following the layered evaluation approach, the results of an observational study with 56 participants. In the second one, we tested the infrastructure to gather emotional data when carrying out another observational study with 17 participants.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures




Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Three more articles mention pedagogical models on which the instruction is based: two use collaboration as a method of student work [21,38] and the third uses argument maps [39].”
From Artificial Intelligence for Student Assessment: A Systematic Review (2021) · cites this paper as [21]
The cited source says what the claim attributes to it.
In the citing paper
“Three other studies [21,38,39] also hint at their pedagogical rationale, which is mainly based on collaboration.”
From Artificial Intelligence for Student Assessment: A Systematic Review (2021) · cites this paper as [21]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“As for the papers that focus on formative assessment, it is clear that the idea of using AI for this task lies in the help that this technology can provide to teachers when they have a large number of students [41], or based on the idea that learning improves the more immediate the feedback given to students [21].”
From Artificial Intelligence for Student Assessment: A Systematic Review (2021) · cites this paper as [21]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Development and validation of brief measures of positive and negative affect: The PANAS scales.
Watson, D., Clark, L. A., & Tellegen, A. (1988) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.54.6.1063 · Cited by 32,433 · cited 1×
“…such as the Big Five Inventory (BFI) [61], the General Self-Efficacy Scale (GSE) [62], and the Positive and Negative Affect Schedule (PANAS) [63], (2) subjective reports such as explicit emotional reports written in natural language by the participants and the Self Assessment Manikin (SAM)…”
BackgroundAxelrod, R., & Hamilton, W. D. (1981) · Science · doi:10.1126/science.7466396 · Cited by 20,065
Russell, J. A. (1980) · Journal of Personality and Social Psychology · doi:10.1037/h0077714 · Cited by 16,334 · cited 1×
“…measuring two distinct dimensions of emotions (valence and arousal) by means of graphic representations of mood in the form of manikins, based on the circumplex model of affect [44] to evaluate the influence of an emotion representation tool in different collaborative work environments [45].”
BackgroundBreiman, L. (1996) · Machine Learning · doi:10.1007/bf00058655 · Cited by 16,325 · cited 1×
“…tree algorithms are a standard technique to classify instances and we did not use other techniques that can offer similar results (e.g., bagging [54]), because the purpose was to obtain metrics with explicit relationships between the dataset statistical indicators and student collaboration…”
MethodMeasuring emotion: The self-assessment manikin and the semantic differential
Bradley, M. M., & Lang, P. J. (1994) · Journal of Behavior Therapy and Experimental Psychiatry · doi:10.1016/0005-7916(94)90063-9 · Cited by 9,538 · cited 2×
“…Five Inventory (BFI) [61], the General Self-Efficacy Scale (GSE) [62], and the Positive and Negative Affect Schedule (PANAS) [63], (2) subjective reports such as explicit emotional reports written in natural language by the participants and the Self Assessment Manikin (SAM) [43] graphical scale.”
BackgroundMethodSUS: A 'Quick and Dirty' Usability Scale
(1996) · Usability Evaluation In Industry · doi:10.1201/9781498710411-35 · Cited by 8,989
(2005) · CTIT technical reports series · Cited by 5,307
(2000) · Elsevier eBooks · doi:10.1016/b978-0-12-109890-2.x5027-6 · Cited by 4,409 · cited 1×
“…(4) variety of source data and inference methods involved [24], (5) difficulties involved in evaluating collaborative learning processes [25], (6) self-regulation role in managing social and collaborative activities [26], and especially (7) lack of consensus on how to model collaboration [27].”
BackgroundContext and consciousness: activity theory and human-computer interaction
(1996) · Choice Reviews Online · doi:10.5860/choice.33-5756 · Cited by 2,511
Educational Data Mining: A review of the state of the art
Romero, C., & Ventura, S. (2010) · IEEE Transactions on Systems Man and Cybernetics Part C (Applications and Reviews) · doi:10.1109/tsmcc.2010.2053532 · Cited by 2,251 · cited 2×
“In order to describe the management of the intelligent support, as implemented in the CLF [3], we follow the distinctions suggested elsewhere [24] and analyze thus the source of data used (data acquisition), the inference method (modeling the collaborative behavior), and the adaptation processes.”
BackgroundMethodDavis, B. G. (1993) · Cited by 1,141
Belbin, R. M. (2012) · Routledge eBooks · doi:10.4324/9780080963242 · Cited by 734 · cited 1×
“On the other hand, there are benefits related to the achievement of teamwork skills and functional knowledge [19], where team member interacts with another so as to facilitate the progress of the team as a whole [20].”
BackgroundSoller, A., Martínez, A., Jermann, P., & Muehlenbrock, M. (2005) · International Journal of Artificial Intelligence in Education · doi:10.3233/irg-2005-15(4)02 · Cited by 587
Findings from observational studies of collaborative work
Tang, J. C. (1991) · International Journal of Man-Machine Studies · doi:10.1016/0020-7373(91)90039-a · Cited by 586 · cited 1×
“Another approach comes from the usability field and involves observing users carrying out the tasks to understand their behavior [42].”
BackgroundCooperation and the Use of Technology
Johnson, D. W., & Johnson, R. T. (2013) · doi:10.4324/9781410609519-41 · Cited by 518
Identifying emotional states using keystroke dynamics
Epp, C., Lippold, M., & Mandryk, R. L. (2011) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1978942.1979046 · Cited by 364 · cited 1×
“Enriching the settings with sensors such as eye-trackers and pressure detection on the chair can provide further insight for the real activity of the user, but might turn the session into a more intrusive experience [60].”
BackgroundA rating scheme for assessing the quality of computer-supported collaboration processes
Meier, A., Spada, H., & Rummel, N. (2007) · International Journal of Computer-Supported Collaborative Learning · doi:10.1007/s11412-006-9005-x · Cited by 363 · cited 1×
“In this case, inference methods can be diverse, and consider expert’s analysis [34], comparison with a preexisting model [29], interaction analysis, mainly statistically oriented [25, 30, 35, 36], and machine learning techniques [29, 31, 37], among others.”
MethodLearning technologies: Affective and social issues in computer-supported collaborative learning
Jones, A., & Issroff, K. (2005) · Computers & Education · doi:10.1016/j.compedu.2004.04.004 · Cited by 309 · cited 2×
“However, up to our knowledge, little attention has been paid on understanding the role of affective and social factors when learning collaboratively and there is still a need for further development of methodological approaches [12].”
BackgroundJärvenoja, H., & Järvelä, S. (2009) · British Journal of Educational Psychology · doi:10.1348/000709909x402811 · Cited by 304 · cited 1×
“Given that emotions can emerge in collaboration scenarios [11] and can be very motivating and rewarding for learners [12], socioemotional states can also be considered to overcome problems encountered by distant collaborators [9].”
BackgroundUsing data mining as a strategy for assessing asynchronous discussion forums
Dringus, L. P., & Ellis, T. (2005) · Computers & Education · doi:10.1016/j.compedu.2004.05.003 · Cited by 241 · cited 1×
“A typical source of data in collaboration scenarios is interactions in forums as they can be used to infer the level of activity or frequency of use [33].”
Backgroundda Gama, J. M. P., & Gaber, M. M. (2007) · Springer eBooks · doi:10.1007/3-540-73679-4 · Cited by 235
Adaptive and Intelligent Systems for Collaborative Learning Support: A Review of the Field
Magnisalis, I., Demetriadis, S., & Karakostas, A. (2011) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2011.2 · Cited by 231 · cited 3×
“In this context, a review of the literature of intelligent systems for collaborative learning support [9] agrees that data mining techniques can be applied to support an automated process of analysis, while a rule-based approach can be used to deliver adaptation to the learners.”
BackgroundMethodUser-centered evaluation of adaptive and adaptable systems: a literature review
VAN VELSEN, L., VAN DER GEEST, T., KLAASSEN, R., & STEEHOUDER, M. (2008) · The Knowledge Engineering Review · doi:10.1017/s0269888908001379 · Cited by 174
Solimeno, A., Mebane, M., Tomai, M., & Francescato, D. (2007) · Computers & Education · doi:10.1016/j.compedu.2007.04.003 · Cited by 172 · cited 1×
“Moreover, personality traits and emotions also play a key role in social and collaborative scenarios [46].”
BackgroundMethodological challenges for collaborative learning research
Strijbos, J.-W., & Fischer, F. (2007) · Learning and Instruction · doi:10.1016/j.learninstruc.2007.03.004 · Cited by 158 · cited 1×
“A major conclusion from all these issues is that there is lack of methodology and standards to analyze collaboration [28].”
BackgroundA layered framework for evaluating on-line collaborative learning interactions
Daradoumis, Τ., Martínez‐Monés, A., & Xhafa, F. (2006) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2006.02.001 · Cited by 147 · cited 2×
“…[23], (4) variety of source data and inference methods involved [24], (5) difficulties involved in evaluating collaborative learning processes [25], (6) self-regulation role in managing social and collaborative activities [26], and especially (7) lack of consensus on how to model…”
BackgroundMethodLayered evaluation of interactive adaptive systems: framework and formative methods
Paramythis, A., Weibelzahl, S., & Masthoff, J. (2010) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-010-9082-4 · Cited by 145 · cited 1×
“It defines five layers, corresponding to the main stages of adaptation: (1) collection of input data, (2) interpretation of the collected data, (3) modeling of the current state of the world, (4) deciding upon adaptation, and (5) applying adaptation.”
BackgroundRomero, C., González, P., Ventura, S., del Jesús, M. J., & Herrera, F. (2007) · Expert Systems with Applications · doi:10.1016/j.eswa.2007.11.026 · Cited by 111 · cited 1×
“To this end, we followed the available evidence on the use of Expectation-Maximization (EM) algorithm, where the clustering task can be viewed as a maximum likelihood estimation problem and the goal is to find themodel structure (number of clusters) that best fits the data [56].”
MethodStudying participation networks in collaboration using mixed methods
Martínez‐Monés, A., Dimitriadis, Y. A., Gómez‐Sánchez, E., Avi, B. R., Abellán, I. M. J., & Marcos, J. A. (2006) · International Journal of Computer-Supported Collaborative Learning · doi:10.1007/s11412-006-8705-6 · Cited by 102 · cited 1×
“In this case, inference methods can be diverse, and consider expert’s analysis [34], comparison with a preexisting model [29], interaction analysis, mainly statistically oriented [25, 30, 35, 36], and machine learning techniques [29, 31, 37], among others.”
MethodD'Mello, S., & Kory, J. (2012) · Proceedings of the 14th ACM international conference on Multimodal interaction · doi:10.1145/2388676.2388686 · Cited by 97 · cited 1×
“To design this study, we took into account recent works in affective computing that suggest that using a multimodal approach that combines different emotional data sources can improve the results obtained by using a single data source [66].”
Anaya, A. R., & Boticario, J. G. (2010) · Expert Systems with Applications · doi:10.1016/j.eswa.2010.05.010 · Cited by 82
Self-citationSantos, O. C., & Boticario, J. G. (2013) · Expert Systems · doi:10.1111/exsy.12041 · Cited by 63
Self-citationMetacognition and Open Learner Models
Susan Bull, A., & Kay, J. (2008) · Cited by 62
Supporting Collaborative Learning With An Intelligent Web-Based System
Israel, J., & Aiken, R. M. (2007) · International Journal of Artificial Intelligence in Education · doi:10.3233/irg-2007-17(1)02 · Cited by 61 · cited 1×
“The latter can also promote effective collaborative learning while accomplishing the task [10].”
BackgroundANALYZING STUDENT PERFORMANCE IN DISTANCE LEARNING WITH GENETIC ALGORITHMS AND DECISION TREES
Kalles, D., & Pierrakeas, C. J. (2006) · Applied Artificial Intelligence · doi:10.1080/08839510600844946 · Cited by 53 · cited 1×
“These algorithms have been used to analyze student performance [53] because they provided a logical tree that explicitly related performance to quantitative statistical indicators.”
MethodNo One Way: Teaching and Learning in Higher Education
Cranton, P. A. (1998) · Cited by 43
Content-free collaborative learning modeling using data mining
Anaya, A. R., & Boticario, J. G. (2011) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-010-9095-z · Cited by 40
Self-citationSelf-regulation and computer based learning
Steffens, K. (2001) · Anuario de Psicología/The UB Journal of Psychology · Cited by 36
Emotions Detection from Math Exercises by Combining Several Data Sources
Santos, O. C., Salmerón-Majadas, S., & Boticario, J. G. (2013) · Lecture notes in computer science · doi:10.1007/978-3-642-39112-5_102 · Cited by 33 · cited 2×
“On-going works from our side are showing results in the same direction [67].”
MethodSelf-citationIncorporating Affect into Educational Design Patterns and Frameworks
Calvo, R. A. (2009) · IEEE International Conference on Advanced Learning Technologies (ICALT) · doi:10.1109/icalt.2009.88 · Cited by 29 · cited 1×
“Data mining techniques are also useful for emotional information detection in collaborative scenarios [13].”
BackgroundBratitsis, T., Dimitracopoulou, A., Martínez‐Monés, A., Marcos, J. A., & Dimitriadis, Y. A. (2008) · IEEE International Conference on Advanced Learning Technologies (ICALT) · doi:10.1109/icalt.2008.261 · Cited by 26 · cited 2×
“In this case, inference methods can be diverse, and consider expert’s analysis [34], comparison with a preexisting model [29], interaction analysis, mainly statistically oriented [25, 30, 35, 36], and machine learning techniques [29, 31, 37], among others.”
BackgroundMethodAn Approach for an Affective Educational Recommendation Model
Santos, O. C., Boticario, J. G., & Riesco, Á. M. (2014) · Springer eBooks · doi:10.1007/978-1-4939-0530-0_6 · Cited by 24 · cited 1×
“In particular, based on our experience in developing educational recommender systems those improved indicators will serve to develop affective educational recommendations [70].”
BackgroundSelf-citationVirtual learning, personality, and learning styles.
Santo, S. A. (2001) · Cited by 18
Challenges for Inclusive Affective Detection in Educational Scenarios
Santos, O. C., Ascaso, A. R., Boticario, J. G., Salmerón-Majadas, S., Quirós, P., & Cabestrero, R. (2013) · Lecture notes in computer science · doi:10.1007/978-3-642-39188-0_61 · Cited by 18 · cited 1×
“Except for the SUS, the infrastructure prepared is similar to the one used in a related individual activity also carried out at the 2012 MSW [65].”
MethodSelf-citationSantos, O. C., & Boticario, J. G. (2004) · Informática educativa: nuevos retos, 2004, ISBN 8477236534 · Cited by 17
Self-citationAnalyzing User Behavior Distribution on E-Learning Platform with Techniques of Clustering
Teng, C.-W., Lin, C.-S., Cheng, S.-Y., & Heh, J. (2004) · Society for Information Technology & Teacher Education International Conference · Cited by 14
Sonderegger, A., Lalanne, D., Bergholz, L., Ringeval, F., & Sauer, J. (2013) · Lecture notes in computer science · doi:10.1007/978-3-642-40480-1_30 · Cited by 11 · cited 1×
“…measuring two distinct dimensions of emotions (valence and arousal) by means of graphic representations of mood in the form of manikins, based on the circumplex model of affect [44] to evaluate the influence of an emotion representation tool in different collaborative work environments [45].”
BackgroundMarcos-García, J. A., Martínez‐Monés, A., Dimitriadis, Y. A., Martínez, R. A., Requies, I. R., & Avi, B. R. (2009) · Studies in computational intelligence · doi:10.1007/978-3-642-04001-6_9 · Cited by 10
A DOMAIN-INDEPENDENT, TRANSFERABLE AND TIMELY ANALYSIS APPROACH TO ASSESS STUDENT COLLABORATION
Anaya, A. R., & Boticario, J. G. (2013) · International Journal of Artificial Intelligence Tools · doi:10.1142/s0218213013500206 · Cited by 10 · cited 2×
“For this, and based on relatedworks andour previous experience [52], decision trees algorithms were used to infer the relationship between the quantitative statistical indicators and student collaboration.”
MethodSelf-citation25 Extending Collaborative Learning Modeling with Emotional Information
Santos, O. C., Boticario, J. G., Cabestrero, R., & Quirós, P. (2013) · Cited by 3
Self-citationBayón, A., Santos, O. C., Couchet, J., & Boticario, J. G. (2011) · Studies in computational intelligence · doi:10.1007/978-3-642-19814-4_9 · Cited by 1 · cited 4×
“Details about the evaluation results are provided elsewhere [3].”
BackgroundMethodInfluentialSelf-citation
Research works printed but not linked: 7
- 2005, International Journal of Artificial Intelligence in Education, 15, 261
- 2007, International Journal of Artificial Intelligence in Education Archive, 17, 3
- 1999, Introduction, what do you mean by collaborative learning?
- 1996, Paradigms shift and instructional technology
- 2003, United States Distance Learning Association Journal, 17, 19
- 2001, Anuarion de Psicología, 32, 77
- 1996, Machine Learning, 24, 123
Cited by
Artificial Intelligence for Student Assessment: A Systematic Review
González-Calatayud, V., Prendes-Espinosa, P., & Roig-Vila, R. (2021) · Applied Sciences · doi:10.3390/app11125467 · Cited by 503
“In the case of Santos and Boticario [21], they are based on formative assessment through collaboration using a system called dotLRN.”
BackgroundMethodInfluentialMartínez-Comesaña, M., Rigueira, X., Larrañaga, A., Martínez, J. M., Ocarranza-Prado, I., & Kreibel, D. (2023) · Revista de Psicodidáctica · doi:10.1016/j.psicod.2023.06.001 · Cited by 194
Martínez-Comesaña, M., Rigueira-Díaz, X., Larrañaga-Janeiro, A., Martínez-Torres, J., Ocarranza-Prado, I., & Kreibel, D. (2023) · Revista de Psicodidáctica (English ed.) · doi:10.1016/j.psicoe.2023.06.002 · Cited by 79
Saneiro, M., Santos, O. C., Salmeron-Majadas, S., & Boticario, J. G. (2014) · The Scientific World Journal · doi:10.1155/2014/484873 · Cited by 91
“The other one followed a collaborative approach [48].”
BackgroundSelf-citationSantos, O. C., Boticario, J. G., & Pérez-Marín, D. (2013) · Science of Computer Programming · doi:10.1016/j.scico.2013.12.004 · Cited by 94
“Moreover, affective issues in collaborative learning scenarios are also being addressed [145].”
BackgroundSelf-citation
Metadata
- Authors
- Santos, O. C., & Boticario, J. G.
- Affiliations
- Universidad Nacional de Educación a Distancia (ES)
- aDeNu Research Group, Computer Science School, UNED, Calle Juan del Rosal 16, 28040 Madrid, Spain
- Corresponding author
- Santos, Olga C.
- Venue
- The Scientific World Journal
- Publisher
- Wiley
- Published
- 2014
- Volume
- 2014
- Pages
- 1-15
- ISSN
- 1537-744X (ISSN-L)
- 2356-6140
- PMC
- PMC3921980
- OpenAlex
- W2058924303
- Cited by
- 25
- References
- 53
- Citations by year
- 2026: 1 · 2025: 3 · 2024: 2 · 2023: 6 · 2022: 1 · 2021: 1 · 2020: 1 · 2016: 5 · 2015: 2 · 2014: 2
- FWCI
- 2.99
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- The Scientific World JOURNAL (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Ministerio de Economía y Competitividad
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
- Social Sciences › Psychology › Developmental and Educational Psychology › Innovative Teaching and Learning Methods
- Physical Sciences › Computer Science › Computer Science Applications › Online Learning and Analytics
- Keywords
- online collaborative learning, user participation, collaborative knowledge construction, teacher workload reduction, student behavior monitoring, personalized learning support, formative assessment, observational studies
- MeSH terms
- Adolescent, Adult, Aged, Artificial Intelligence, Computer-Assisted Instruction, Computer Communication Networks, Cooperative Behavior, Humans, Middle Aged, Software, Education, Distance
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.