Involving Users to Improve the Collaborative Logical Framework
Open accessThe DOI matches its Crossref record; 51 of the 53 references it cites were found in a registry or an index.
Abstract
In order to support collaboration in web-based learning, there is a need for an intelligent support that facilitates its management during the design, development, and analysis of the collaborative learning experience and supports both students and instructors. At aDeNu research group we have proposed the Collaborative Logical Framework (CLF) to create effective scenarios that support learning through interaction, exploration, discussion, and collaborative knowledge construction. This approach draws on artificial intelligence techniques to support and foster an effective involvement of students to collaborate. At the same time, the instructors’ workload is reduced as some of their tasks—especially those related to the monitoring of the students behavior—are automated. After introducing the CLF approach, in this paper, we present two formative evaluations with users carried out to improve the design of this collaborative tool and thus enrich the personalized support provided. In the first one, we analyze, following the layered evaluation approach, the results of an observational study with 56 participants. In the second one, we tested the infrastructure to gather emotional data when carrying out another observational study with 17 participants.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures




Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Three more articles mention pedagogical models on which the instruction is based: two use collaboration as a method of student work [21,38] and the third uses argument maps [39].”
From Artificial Intelligence for Student Assessment: A Systematic Review (2021) · cites this paper as [21]
The cited source says what the claim attributes to it.
In the citing paper
“Three other studies [21,38,39] also hint at their pedagogical rationale, which is mainly based on collaboration.”
From Artificial Intelligence for Student Assessment: A Systematic Review (2021) · cites this paper as [21]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“As for the papers that focus on formative assessment, it is clear that the idea of using AI for this task lies in the help that this technology can provide to teachers when they have a large number of students [41], or based on the idea that learning improves the more immediate the feedback given to students [21].”
From Artificial Intelligence for Student Assessment: A Systematic Review (2021) · cites this paper as [21]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Development and validation of brief measures of positive and negative affect: The PANAS scales.
Watson, D., Clark, L. A., & Tellegen, A. (1988) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.54.6.1063 · Cited by 32,433 · cited 1×
“…such as the Big Five Inventory (BFI) [61], the General Self-Efficacy Scale (GSE) [62], and the Positive and Negative Affect Schedule (PANAS) [63], (2) subjective reports such as explicit emotional reports written in natural language by the participants and the Self Assessment Manikin (SAM)…”
BackgroundAxelrod, R., & Hamilton, W. D. (1981) · Science · doi:10.1126/science.7466396 · Cited by 20,065
Russell, J. A. (1980) · Journal of Personality and Social Psychology · doi:10.1037/h0077714 · Cited by 16,334 · cited 1×
“…measuring two distinct dimensions of emotions (valence and arousal) by means of graphic representations of mood in the form of manikins, based on the circumplex model of affect [44] to evaluate the influence of an emotion representation tool in different collaborative work environments [45].”
BackgroundBreiman, L. (1996) · Machine Learning · doi:10.1007/bf00058655 · Cited by 16,325 · cited 1×
“…tree algorithms are a standard technique to classify instances and we did not use other techniques that can offer similar results (e.g., bagging [54]), because the purpose was to obtain metrics with explicit relationships between the dataset statistical indicators and student collaboration…”
MethodMeasuring emotion: The self-assessment manikin and the semantic differential
Bradley, M. M., & Lang, P. J. (1994) · Journal of Behavior Therapy and Experimental Psychiatry · doi:10.1016/0005-7916(94)90063-9 · Cited by 9,538 · cited 2×
“…Five Inventory (BFI) [61], the General Self-Efficacy Scale (GSE) [62], and the Positive and Negative Affect Schedule (PANAS) [63], (2) subjective reports such as explicit emotional reports written in natural language by the participants and the Self Assessment Manikin (SAM) [43] graphical scale.”
BackgroundMethodSUS: A 'Quick and Dirty' Usability Scale
(1996) · Usability Evaluation In Industry · doi:10.1201/9781498710411-35 · Cited by 8,989
(2005) · CTIT technical reports series · Cited by 5,307
(2000) · Elsevier eBooks · doi:10.1016/b978-0-12-109890-2.x5027-6 · Cited by 4,409 · cited 1×
“…(4) variety of source data and inference methods involved [24], (5) difficulties involved in evaluating collaborative learning processes [25], (6) self-regulation role in managing social and collaborative activities [26], and especially (7) lack of consensus on how to model collaboration [27].”
BackgroundContext and consciousness: activity theory and human-computer interaction
(1996) · Choice Reviews Online · doi:10.5860/choice.33-5756 · Cited by 2,511
Educational Data Mining: A review of the state of the art
Romero, C., & Ventura, S. (2010) · IEEE Transactions on Systems Man and Cybernetics Part C (Applications and Reviews) · doi:10.1109/tsmcc.2010.2053532 · Cited by 2,251 · cited 2×
“In order to describe the management of the intelligent support, as implemented in the CLF [3], we follow the distinctions suggested elsewhere [24] and analyze thus the source of data used (data acquisition), the inference method (modeling the collaborative behavior), and the adaptation processes.”
BackgroundMethod
Research works printed but not linked: 14
- Santos OC Rodríguez A Gaudioso E Boticario JG Helping the instructor to manage a collaborative task in a web-based learning environment Proceedings of the International Conference on Artificial Intelligence in Education (AIED ’03) 2003 153 162
- NORAD The Logical Framework Approach, Handbook for Objectives-Oriented Planning 1999 4th edition
- Santos OC Boticario JG Raffenne E Pastor R Why using dotLRN? UNED use cases Proceedings of the International Conference Free/Libre/Open Source Systems (FLOSS ’07) 2007 195 212
- Dillenbourg P Dillenbourg P Introduction, what do you mean by collaborative learning? Collaborative Learning: Cognitive and Computational Approaches 1999 Oxford, UK Elsevier Science 1 19
- Koschmann T Koschmann T Paradigms shift and instructional technology CSCL: Theory and Practice of an Emerging Paradigm 1996 Mahwah, NJ, USA Lawrence Erlbaum Associates 1 23
- Winter M McCalla GI An analysis of group performance in terms of the functional knowledge and teamwork skills of group members Proceedings of the Workshop on User and Group Models for Web-Based Adaptive Collaborative Environments at International Conference on User Modeling (UM ’03) 2003
- Bratitsis T Dimitracopoulou A Indicators for measuring quality in asynchronous discussion forae Proceedings of the 12th International Workshop on Groupware (CRIWG ’06) 2006 54 61
- Redondo MA Bravo C Ortega M Applying fuzzy logic to analyze collaborative learning experiences in an e-learning environment United States Distance Learning Association Journal 2003 17 2 19 28
- Talavera L Gaudioso E Mining student data to characterize similar behavior groups in instructured collaboration spaces Proceedings of the Workshop on Artificial Intelligence in CSCL at 16th European Conference on Artificial Intelligence (ECAI ’04) 2004 17 23
- Hong W Spinning your course into a web classroom—advantages and challenges Proceedings of the International Conference on Engineering Education August 2001 6 10
- Perera D Kay J Yacef K Koprinska I Mining learners’ traces from an online collaboration tool Proceedings of the Workshop Educational Data Mining at 13th International Conference of Artificial Intelligence in Education (AIED ’07) 2007
- Macarthur V Conlan O Using psychometric approaches in the modeling of abstract cognitive skills for personalization Proceedings of the Worshop Lifelong User Modelling at Conference on User Modeling, Adaptation, and Personalization (UMAP ’09) 2009
- Couchet J Santos OC Raffenne E Boticario JG The tracking and auditing module for the OpenACS framework Proceedings of the 7th OpenACS/LRN Conference 2008 Valencia, Spain
- Raffenne E Santos OC Granado J Couchet J J Boticario G Integrating OpenACS/dotLRN with an open and accessible service architecture framework Proceedings of the 8th OpenACS/LRN Conference 2009
Cited by
Artificial Intelligence for Student Assessment: A Systematic Review
González-Calatayud, V., Prendes-Espinosa, P., & Roig-Vila, R. (2021) · Applied Sciences · doi:10.3390/app11125467 · Cited by 503
“In the case of Santos and Boticario [21], they are based on formative assessment through collaboration using a system called dotLRN.”
BackgroundMethodInfluentialMartínez-Comesaña, M., Rigueira, X., Larrañaga, A., Martínez, J. M., Ocarranza-Prado, I., & Kreibel, D. (2023) · Revista de Psicodidáctica · doi:10.1016/j.psicod.2023.06.001 · Cited by 194
Martínez-Comesaña, M., Rigueira-Díaz, X., Larrañaga-Janeiro, A., Martínez-Torres, J., Ocarranza-Prado, I., & Kreibel, D. (2023) · Revista de Psicodidáctica (English ed.) · doi:10.1016/j.psicoe.2023.06.002 · Cited by 79
Saneiro, M., Santos, O. C., Salmeron-Majadas, S., & Boticario, J. G. (2014) · The Scientific World Journal · doi:10.1155/2014/484873 · Cited by 91
“The other one followed a collaborative approach [48].”
BackgroundSelf-citationSantos, O. C., Boticario, J. G., & Pérez-Marín, D. (2013) · Science of Computer Programming · doi:10.1016/j.scico.2013.12.004 · Cited by 94
“Moreover, affective issues in collaborative learning scenarios are also being addressed [145].”
BackgroundSelf-citationMahmud, B. U., Hong, G. Y., & Fong, B. (2022) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3542698 · Cited by 99
Emotions and Personality in Adaptive e-Learning Systems: An Affective Computing Perspective
Santos, O. C. (2016) · Human-computer interaction series · doi:10.1007/978-3-319-31413-6_13 · Cited by 66
Self-citationDesigning a model of sustainable education based on artificial intelligence in higher education
Bagherimajd, K., & Khajedad, K. (2025) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2025.100439 · Cited by 50
Exploring the use and impact of artificial intelligence in higher education in Africa
Pasipamire, N., Chigwada, J. P., & Maturure, R. (2025) · Journal of Pedagogical Sociology and Psychology · doi:10.33902/jpsp.202532046 · Cited by 25
A Taxonomy of AI-Based Assessment Educational Technologies
Hammad, M., Al-Refai, M., Musallam, W., Musleh, S., & Faouri, E. (2024) · International Conference on Information and Communication Systems (ICICS) · doi:10.1109/icics63486.2024.10638295 · Cited by 13
“For example, Santos and Boticario [6] proposed Collaborative Logical Framework (CLF) for creating effective collaboration learning environment based on AI to increase learners’ engagements and teaching strategies.”
BackgroundCruz, L. R. O., Polyzos, N., Fernandes, Â. S. F., Petropoulos, S. A., Gioia, F. D., Días, M. I., Pinela, J., Kostić, M., Sokóvić, M., Ferreira, I. C. F. R., et al. (2020) · Agronomy · doi:10.3390/agronomy10111824 · Cited by 10
One-Year Progress in Ophthalmic Education: Annual Review
Mayorga, E., Golnik, K. C., & Palis, A. G. (2015) · Asia-Pacific Journal of Ophthalmology · doi:10.1097/apo.0000000000000162 · Cited by 9
“The development of a standardized online course must be done on a common support system where its design and analysis can be managed.(65)”
BackgroundExplorando el rol transformador de la inteligencia artificial en la evaluación formativa educativa
Bautista, J. C. N. (2025) · Revista Tribunal · doi:10.59659/revistatribunal.v5i10.134 · Cited by 7
Pérez-Marín, D., Hijón‐Neira, R., & Santacruz-Valencia, L. P. (2016) · Behaviour and Information Technology · doi:10.1080/0144929x.2016.1171398 · Cited by 7
“It also follows the principles of the Collaborative Logical Framework (CLF) to create effective scenarios to support learning through interaction, exploration, discussion, and collaborative knowledge construction (Santos and Boticario 2014).”
Identifying recommendation opportunities for computer‐supported collaborative environments
Lobo, J. L., Santos, O. C., Boticario, J. G., & Ser, J. D. (2016) · Expert Systems · doi:10.1111/exsy.12159 · Cited by 7
“Forum was considered as a communication tool because it is widely used in collaborative environments (Anaya and Boticario, 2011) and because it is the main communication tool in our current and previous collaborative experiences (Anaya et al., 2013; Santos & Boticario, 2014).”
BackgroundMethodInfluentialSelf-citationArtificial Intelligence as a Harbinger of Engagement and Collaboration
Ouariach, S., Ouariach, F. Z., & Khaldi, M. (2025) · Advances in educational technologies and instructional design book series · doi:10.4018/979-8-3373-2262-9.ch006 · Cited by 6
Collaborative logical framework: An e-learning assesment tool in .LRN platform
Vega-Alvarado, J., Francesa-Alfaro, A., Chacón, M. P., & Garita, C. (2016) · Latin American Conference on Learning Technologies (LACLO) · doi:10.1109/laclo.2016.7751748 · Cited by 5
“Según estudios hechos por el periódico El Financiero1, en Costa Rica cada vez más personas ingresan a navegar desde celulares, tabletas, poniendo en evidencia la importancia de implementar un diseño responsive a los usuarios del MLC. En estudios anteriores realizados en Madrid [4] donde se probó el MLC en los lab…”
BackgroundInfluentialHigh Resistance Sintered Pond Ash/Kaolin (PA/CC) Ceramics
Panigrahi, M., Ganguly, R. I., & Dash, R. R. (2023) · Wiley eBooks · doi:10.1002/9781394200405.ch7 · Cited by 3
Lobo, J. L., Santos, O. C., & Boticario, J. G. (2014) · Cited by 2
Self-citationOptimising AI writing assessment using feedback and knowledge graph integration
Zhang, C. (2025) · PeerJ Computer Science · doi:10.7717/peerj-cs.2893 · Cited by 4
“Because it uses AI to facilitate group projects, the Santos & Boticario (2014) experience stands apart.”
High Resistance Sintered Fly Ash/Kaolin (FA/CC) Ceramics
Panigrahi, M., Ganguly, R. I., & Dash, R. R. (2023) · Wiley eBooks · doi:10.1002/9781394200405.ch4 · Cited by 2
Modeling Learner Information wihting an Integrated Model on Standard-based Representations.
Chacón, M. P., Santos, O. C., & Boticario, J. G. (2015) · International Conference on User Modeling, Adaptation, and Personalization · Cited by 2
Self-citationAffective and Behavioral Assessment for Adaptive Intelligent Tutoring Systems.
Marco-Giménez, L., Arevalillo‐Herráez, M., Ferri, F. J., Moreno-Picot, S., Boticario, J. G., Santos, O. C., Salmerón-Majadas, S., Saneiro, M., Uria-Rivas, R., Arnau, D., et al. (2016) · Cited by 5
Self-citationReport on Smart Education in Montenegro
Melović, B. (2023) · Lecture notes in educational technology · doi:10.1007/978-981-19-7319-2_9 · Cited by 1
High Resistance Sintered Fly Ash (FA) Ceramics
Panigrahi, M., Ganguly, R. I., & Dash, R. R. (2023) · Wiley eBooks · doi:10.1002/9781394200405.ch3
High Resistant HCFC Slag/Fly Ash Hybrid Ceramics
Panigrahi, M., Ganguly, R. I., & Dash, R. R. (2024) · Wiley eBooks · doi:10.1002/9781394231287.ch9
Hendra, A., & Supeno, H. (2024) · TEMATIK · doi:10.38204/tematik.v11i1.1841
Panigrahi, M., Ganguly, R. I., Sapkal, S., Dash, R. R., & Panda, H. S. (2025) · Journal of Materials Science Materials in Electronics · doi:10.1007/s10854-025-15463-x
Enhancing Learning and Evaluation: AI as a Catalyst for Sustainable Education
Sehgal, S., Tomar, R. V., Mehta, D., & Kumar, S. (2026) · Revista de Administração Contemporânea · doi:10.1590/1982-7849rac2026250118.en
A Study of Possible AI Applications within Childhood Educations
Kubde, B. R., & Dekate, M. V. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-27160-0_6
Díaz, C. B., & Calatayud, V. G. (2026) · Revista Internacional de Lenguas Extranjeras / International Journal of Foreign Languages · doi:10.17345/rile26.4376
Metadata
- Authors
- Santos, O. C., & Boticario, J. G.
- Affiliations
- Universidad Nacional de Educación a Distancia (ES)
- aDeNu Research Group, Computer Science School, UNED, Calle Juan del Rosal 16, 28040 Madrid, Spain
- Corresponding author
- Santos, Olga C.
- Venue
- The Scientific World Journal
- Publisher
- Wiley
- Published
- 2014
- Volume
- 2014
- Pages
- 1-15
- ISSN
- 1537-744X (ISSN-L)
- 2356-6140
- PMC
- PMC3921980
- OpenAlex
- W2058924303
- Cited by
- 25
- References
- 53
- Citations by year
- 2026: 1 · 2025: 3 · 2024: 2 · 2023: 6 · 2022: 1 · 2021: 1 · 2020: 1 · 2016: 5 · 2015: 2 · 2014: 2
- FWCI
- 2.99
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- The Scientific World JOURNAL (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Ministerio de Economía y Competitividad
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
- Social Sciences › Psychology › Developmental and Educational Psychology › Innovative Teaching and Learning Methods
- Physical Sciences › Computer Science › Computer Science Applications › Online Learning and Analytics
- Keywords
- online collaborative learning, user participation, collaborative knowledge construction, teacher workload reduction, student behavior monitoring, personalized learning support, formative assessment, observational studies
- MeSH terms
- Adolescent, Adult, Aged, Artificial Intelligence, Computer-Assisted Instruction, Computer Communication Networks, Cooperative Behavior, Humans, Middle Aged, Software, Education, Distance
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.