A Framework for Identifying Sequences of Interactions That Cause Usability Problems in Collaborative Systems
Open accessThe DOI matches its Crossref record; 39 of the 39 references it cites were found in a registry or an index.
Abstract
Collaborative systems support shared spaces, where groups of users exchange interactions. In order to ensure the usability of these systems, an intuitive interactions’ organization and that each user has awareness information to know the activity of others are necessary. Usability laboratories allow evaluators to verify these requirements. However, laboratory usability evaluations can be problematic for reproducing mobile and ubiquitous contexts, as they restrict the place and time in which the user interacts with the system. This paper presents a framework for building software support that it collects human–machine interactions in mobile and ubiquitous contexts and outputs an assessment of the system’s usability. This framework is constructed through learning that is based on neural networks, identifying sequences of interactions related to usability problems when users carry out collaborative activities. The paper includes a case study that puts the framework into action during the development process of a smartphone application that supports collaborative sport betting.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures






Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Likewise, the rapid evolution of digital devices and their integration with artificial intelligence systems demands renewed approaches to visual interaction that adapt dynamically to users' contexts and needs [8,9].”
From The Impact of User Interface and Experience (UI/UX) Design on Visual Ergonomics: A Technical Approach for Reducing Human Error in Industrial Settings (2026) · cites this paper as [8]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Existing studies are distributed across multiple disciplines, including human-computer interaction, industrial engineering, cognitive science, and design research, often focusing on isolated interface components rather than integrated ergonomic strategies [6-8].”
From The Impact of User Interface and Experience (UI/UX) Design on Visual Ergonomics: A Technical Approach for Reducing Human Error in Industrial Settings (2026) · cites this paper as [8]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In contrast, studies [6,8,16,28,29] address intermediate or non-specialized users, whether university students, mobile users or digital collaborators.”
From The Impact of User Interface and Experience (UI/UX) Design on Visual Ergonomics: A Technical Approach for Reducing Human Error in Industrial Settings (2026) · cites this paper as [8]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Articles such as [6,8,16,26,29] reveal that the absence of a clear visual structure, feedback, or functional coding leads to misunderstandings, omissions, and navigation errors.”
From The Impact of User Interface and Experience (UI/UX) Design on Visual Ergonomics: A Technical Approach for Reducing Human Error in Industrial Settings (2026) · cites this paper as [8]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Hybrid or collaborative environments demand participatory designs and explicit role indicators to maintain visual coherence between remote and in-person users [5,6,8,22].”
From The Impact of User Interface and Experience (UI/UX) Design on Visual Ergonomics: A Technical Approach for Reducing Human Error in Industrial Settings (2026) · cites this paper as [8]
Experimental — accuracy not yet measured
4 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · cited 1×
“Adam [42] has been used as an optimizer and MAE (Mean Average Error) as loss function.”
MethodBarron, B. (2003) · Journal of the Learning Sciences · doi:10.1207/s15327809jls1203_1 · Cited by 1,446 · cited 1×
“This classification is based on a dual model of the process of interaction [39], in which a set of users can participate.”
BackgroundAbdeljaber, O., Avci, O., Kiranyaz, S., Gabbouj, M., & Inman, D. J. (2017) · Journal of Sound and Vibration · doi:10.1016/j.jsv.2016.10.043 · Cited by 1,278 · cited 1×
“Abdeljaber et al. [37] propose an adaptative 1D-CNN to detect structural damage by analyzing the vibration of the materials.”
BackgroundA Descriptive Framework of Workspace Awareness for Real-Time Groupware
Gutwin, C., & Greenberg, S. (2002) · Computer Supported Cooperative Work (CSCW) · doi:10.1023/a:1021271517844 · Cited by 1,185 · cited 1×
“Awareness information usually provides information of shared workspace state and social interactions between users [26].”
BackgroundThe state of the art in automating usability evaluation of user interfaces
Ivory, M. Y., & Hearst, M. A. (2001) · ACM Computing Surveys · doi:10.1145/503112.503114 · Cited by 875 · cited 2×
“A solution for this is to use software support that collects human–machine interactions and outputs information on the user behaviour and produces an assessment of the system’s usability [3].”
MethodModel-Based Design and Evaluation of Interactive Applications
Paternò, F. (2000) · Applied Computing · doi:10.1007/978-1-4471-0445-2 · Cited by 818
A specifier's introduction to formal methods
Wing, J. M. (1990) · Computer · doi:10.1109/2.58215 · Cited by 736 · cited 1×
“These methods are a set of well-defined mathematical languages and techniques for the specification, modelling, and verification of systems [15].”
Method(2007) · CRC Press eBooks · doi:10.1201/9781410615862 · Cited by 665
Measuring usability: preference vs. performance
Nielsen, J., & Levy, J. (1994) · Communications of the ACM · doi:10.1145/175276.175282 · Cited by 458 · cited 1×
“In most cases, there is a correlation between the subjective metrics that assess user satisfaction and objective metrics of the user performance that can be calculated from log repositories, according to Nielsen and Levy [8].”
BackgroundHuman activity recognition from accelerometer data using Convolutional Neural Network
Lee, S.-M., Yoon, S. M., & Cho, H. (2017) · 2017 IEEE International Conference on Big Data and Smart Computing (BigComp) · doi:10.1109/bigcomp.2017.7881728 · Cited by 331 · cited 2×
“Many recent examples have taken advantage of 1D Convolutional Neural Networks to undertake data analysis similar to those proposed, obtaining excellent results [36–38].”
BackgroundMethodPinelle, D., Gutwin, C., & Greenberg, S. (2003) · ACM Transactions on Computer-Human Interaction · doi:10.1145/966930.966932 · Cited by 310 · cited 1×
“Pinelle et al. [29] have proposed a specific task analysis technique for collaborative systems, called Collaboration Usability Analysis (CUA).”
MethodNew techniques for usability evaluation of mobile systems
Kjeldskov, J., & Stage, J. (2004) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2003.11.001 · Cited by 301 · cited 1×
“For this reason, Kjeldskov and Stage [11] describe a framework that studies the impact of physical motion on the usability of mobile systems.”
BackgroundEffects of awareness support on groupware usability
Gutwin, C., & Greenberg, S. (1998) · Proceedings of the SIGCHI conference on Human factors in computing systems - CHI '98 · doi:10.1145/274644.274713 · Cited by 216 · cited 1×
“One of the first usability studies of groupware systems found how awareness mechanisms had a clear influence on usability [25].”
BackgroundUsability evaluation of mobile applications using ISO 9241 and ISO 25062 standards
Moumane, K., Idri, A., & Abran, A. (2016) · SpringerPlus · doi:10.1186/s40064-016-2171-z · Cited by 163 · cited 2×
“Moumane et al. [12] propose a framework for calculating the usability metrics in mobile environments while using ISO 9241 [13] and ISO 25062 [14] standards as references.”
MethodA machine learning-based usability evaluation method for eLearning systems
Oztekin, A., Delen, D., Türkyılmaz, A., & Zaim, S. (2013) · Decision Support Systems · doi:10.1016/j.dss.2013.05.003 · Cited by 161 · cited 1×
“…these works present decision support systems to help evaluators identify critical points in the analyzed systems [34]; others incorporate machine learning algorithms, which are based on neural networks, to improve the evaluation process through the analysis of the interactions with the system [35].”
BackgroundAutomatic support for usability evaluation
Lecerof, A., & Paterno, F. (1998) · IEEE Transactions on Software Engineering · doi:10.1109/32.729686 · Cited by 147 · cited 1×
“Lecerof and Paternò [21] propose an evaluation method while using the CTT notation that quantifies the difficulties of performing the task model activities (activities that are never performed, failures to follow the sequence of activities, etc.).”
MethodAlzheimer’s Disease stage identification using deep learning models
Bringas, S., Salomón, S., Duque, R., Lage, C., & Montaña, J. L. (2020) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2020.103514 · Cited by 109 · cited 1×
“[36] presents a 1D-CNN to process data from accelerometers to predict Alzheimer’s disease stage.”
MethodSelf-citationAdvances in Web Mining and Web Usage Analysis
Mobasher, B., Nasraoui, O., Masand, B. M., & Liu, B. (2006) · Lecture notes in computer science · doi:10.1007/11899402 · Cited by 105
A license to kill – Improving UCSD in Agile development
Larusdottir, M. K., Gulliksen, J., & Cajander, Å. (2016) · Journal of Systems and Software · doi:10.1016/j.jss.2016.01.024 · Cited by 79 · cited 1×
“In order to obtain an evaluation of requirements related to the interaction with the system, the users can experiment with the usage of the system and indicate their degree of satisfaction [2].”
BackgroundMartinie, C., Palanque, P. A., Ragosta, M., & Fahssi, R. (2013) · European Conference on Cognitive Ergonomics · doi:10.1145/2501907.2501954 · Cited by 67 · cited 1×
“Martinie et al. [23] enriched the HAMSTER task modeling notation [19] to introduce usability-related concepts, like the complexity or information workload that is associated with a task.”
BackgroundDescriptive theory of awareness for groupware development
Collazos, C. A., Gutiérrez, F. L., Gallardo, J., Ortega, M., Fardoun, H. M., & Molina, A. I. (2019) · Journal of Ambient Intelligence and Humanized Computing · doi:10.1007/s12652-018-1165-9 · Cited by 65 · cited 1×
“In order to avoid these usability problems, the collaborative user interfaces should show awareness information that enables the users to know the activities of others [30].”
BackgroundJohn, B. E., Vera, A. H., Matessa, M., Freed, M. C., & Remington, R. W. (2002) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/503376.503404 · Cited by 61 · cited 1×
“Several notations, like CTT (Concur Task Trees) [16], GOMS (Goals, Operators, Methods, and Selection Rules) [17], EOFM (Enhanced Operator Function Model) [18], and HAMSTER [19] have been proposed to specify the task models.”
MethodAwareness Supporting Technologies used in Collaborative Systems: A Systematic Literature Review
Lopez, G., & Guerrero, L. A. (2017) · Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing · doi:10.1145/2998181.2998281 · Cited by 51 · cited 1×
“A traditional taxonomy that classifies the collaborative systems into synchronous and asynchronous has been used to understand the mechanisms to show the information of others [31].”
BackgroundA decision support system for usability evaluation of web-based information systems
Oztekin, A. (2010) · Expert Systems with Applications · doi:10.1016/j.eswa.2010.07.151 · Cited by 48 · cited 1×
“Some of these works present decision support systems to help evaluators identify critical points in the analyzed systems [34]; others incorporate machine learning algorithms, which are based on neural networks, to improve the evaluation process through the analysis of the interactions with the…”
BackgroundA design methodology for user-centered innovation in the software development area
Zaina, L. A. M., & Álvaro, A. (2015) · Journal of Systems and Software · doi:10.1016/j.jss.2015.08.029 · Cited by 41 · cited 1×
“Current software methodologies promote user participation to evaluate the prototypes generated in each iteration of the life cycle development [1].”
MethodHuman-Centered and Error-Resilient Systems Development
Bogdan, C., Gulliksen, J., Sauer, S., Forbrig, P., Winckler, M. A. A., Johnson, C., Palanque, P. A., Bernhaupt, R., & Kis, F. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-44902-9 · Cited by 28
Enhanced operator function model: A generic human task behavior modeling language
Bolton, M. L., & Bass, E. J. (2009) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/icsmc.2009.5346067 · Cited by 27 · cited 2×
“In a similar way, the EOFM [18] notation is used for evaluation purposes to identify user behaviors that had not been expected.”
MethodTowards an XML-Based Representation of Collaborative Action
Martínez‐Monés, A., Fuente, P. D. L., & Dimitriadis, Y. A. (2003) · Springer eBooks · doi:10.1007/978-94-017-0195-2_46 · Cited by 24
Usage-Based Automatic Detection of Usability Smells
Harms, P., & Grabowski, J. (2014) · Lecture notes in computer science · doi:10.1007/978-3-662-44811-3_13 · Cited by 23 · cited 1×
“Harms and Grabowski [10] present a proposal that processes log files, generates task trees with the user actions, and identifies usability smells.”
BackgroundBernhaupt, R., Palanque, P. A., Manciet, F., & Martinie, C. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-44902-9_5 · Cited by 19 · cited 1×
“The task models can be updated with the results of these evaluations [22].”
A Measurement and Structural Model for Usability Evaluation of Shared Workspace Groupware
Berkman, M. İ., Karahoca, D., & Karahoca, A. (2017) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2017.1326578 · Cited by 19 · cited 1×
“Berkman et al. [27] propose a measurement model for assessing the usability of shared workspace systems, which include variables that measure quality of awareness of collaborative interactions.”
BackgroundModel-driven web usage analysis for the evaluation of web application quality
Fraternali, P., Lanzi, P. L., Matera, M., & Maurino, A. (2004) · Virtual Community of Pathological Anatomy (University of Castilla La Mancha) · doi:10.5555/2011143.2011147 · Cited by 18
Geszten, D., Hámornik, B. P., & Hercegfi, K. (2018) · IEEE International Conference on Cognitive Infocommunications · doi:10.1109/coginfocom.2018.8639865 · Cited by 14 · cited 2×
“Geszten et al. [28] described a method to study, in usability laboratories, the relation between awareness mechanisms and usability problems.”
MethodTask Model-Based Usability Evaluation for Smart Environments
Propp, S., Buchholz, G., & Forbrig, P. (2008) · Lecture notes in computer science · doi:10.1007/978-3-540-85992-5_3 · Cited by 13 · cited 1×
“[20] plan usability evaluations that include test cases in which the users should perform all of the activities in the task model.”
BackgroundTîrnȃucȃ, C., Duque, R., & Montaña, J. L. (2017) · Sensors · doi:10.3390/s17071669 · Cited by 6 · cited 1×
“[40] to visualize through graphs those sequences of interactions performed by users that would be associated with poor usability levels according to the prediction of the neural network.”
MethodSelf-citationAn Approach for Asynchronous Awareness Support in Collaborative Non-Linear Storytelling
Schumann, J., Buttler, T., & Lukosch, S. G. (2012) · Computer Supported Cooperative Work (CSCW) · doi:10.1007/s10606-012-9162-6 · Cited by 5 · cited 1×
“However, asynchronous collaborative systems do not provide real-time information on the actions of other users, although they should describe the progressive modifications of shared spaces over a long period of time [33].”
BackgroundJohn, B. E., Vera, A. H., Matessa, M., Freed, M., & Remington, R. W. (2002) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/503403.503404 · Cited by 4
Study of the Effect of Awareness on Synchronous Collaborative Problem-Solving
Chounta, I., & Avouris, N. M. (2010) · International Conference on Intelligent Networking and Collaborative Systems (INCoS) · doi:10.1109/incos.2010.75 · Cited by 2 · cited 1×
“Thus, Chounta and Avouris [32] point out that synchronous collaboration requires that user interfaces show real-time information of the interactions of other users.”
BackgroundHAMSTERS : un environnement d'édition et de simulation de modèles de tâches (Démonstration)
Fahssi, R., Martinie, C., & Palanque, P. A. (2014) · Open Archive Toulouse Archive Ouverte (University of Toulouse) · Cited by 2
Research works printed but not linked: 4
- Madan, A.; Dubey, S.K. Usability Evaluation Methods: A Literature Review. Int. J. Eng. Sci. Technol. 2012, 4, 590-599.
- Preece, J.; Benyon, D.; University, O. A Guide to Usability: Human Factors in Computing, 1st ed.; Addison-Wesley Longman Publishing Co., Inc.: Boston, MA, USA, 1993.
- Bogdan, C., Gulliksen, J., Sauer, S., Forbrig, P., Winckler, M., Johnson, C., Palanque, P., Bernhaupt, R., Kis, F., Eds.; Springer International Publishing: Cham, Switzerland, 2016; pp. 56-72.
- Russell, S.; Norvig, P. Artificial Intelligence: A Modern Approach, 3rd ed.; Prentice Hall: Upper Saddle River, NJ, USA, 2010.
Cited by
Team usability testing: development and validation of a groupware usability evaluation method
Geszten, D., Hámornik, B. P., & Hercegfi, K. (2024) · Cognition Technology & Work · doi:10.1007/s10111-024-00759-5 · Cited by 9
“Besides, several attempts were made to automatize the data collection (Grigera et al. 2021) or the data analysis (Bringas et al. 2021) parts of groupware evaluation.”
Vizcarra, A., Quiroz, G., & Cornejo, J. (2026) · Designs · doi:10.3390/designs10010008 · Cited by 3
Tycho: Facilitation Support for Groupware User Tests
Grigera, J., Gardey, J. C., Fernández, A., & Garrido, A. (2021) · Interaction design & architecture(s)/ID&A Interaction design & architecture(s) · doi:10.55612/s-5002-049-003 · Cited by 1
Córdova, K. A., Santos, M. T. S., Pimentel, C., & Gasparini, I. (2022) · doi:10.5753/sbsc.2022.19475 · Cited by 1
Metadata
- Authors
- Bringas, S., Duque, R., Nieto-Reyes, A., Tîrnăucă, C., & Montaña, J. L.
- Affiliations
- Fundación Centro Tecnológico de Componentes CTC, 39011 Santander, Spain
- Universidad de Cantabria (ES)
- Department of Mathematics, Statistics and Computer Science, Universidad de Cantabria, 39005 Santander, Spain
- Corresponding author
- Bringas, Santos
- Venue
- Electronics
- Publisher
- MDPI AG
- Published
- 2021-02-05
- Volume
- 10
- Issue
- 4
- Pages
- 388
- ISSN
- 2079-9292 (ISSN-L)
- OpenAlex
- W3128612482
- Cited by
- 3
- References
- 39
- Citations by year
- 2026: 1 · 2024: 1 · 2022: 1
- FWCI
- 0.31
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Electronics (publishedVersion, cc-by)
- UCrea (University of Cantabria) (publishedVersion, other-oa)
- LA Referencia (Red Federada de Repositorios Institucionales de Publicaciones Científicas) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Ministerio de Ciencia, Innovación y Universidades (MTM2017-86061-C2-2-P)
- Universidad de Cantabria
- Topics
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Human-Computer Interaction › Innovative Human-Technology Interaction
- Keywords
- collaborative systems, usability problems, interaction sequences, ubiquitous computing, artificial neural networks, user awareness, shared spaces, mobile application
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.