The DOI matches its Crossref record; 200 of the 201 references it cites were found in a registry or an index.
Layered evaluation of interactive adaptive systems: framework and formative methods
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“This is a well known issue in evaluating interactive adaptive systems (c.f. Paramythis et al., 2010).”
From Affective learning: Improving engagement and enhancing learning with affect-aware feedback (2017) · cites this paper as Paramythis et al., 2010
Citations
Cites
Data Mining: Practical Machine Learning Tools and Techniques
Witten, I. H., Frank, E., & Hall, M. A. (2011) · Elsevier eBooks · doi:10.1016/c2009-0-19715-5 · Cited by 25,543
Focus Groups: A Practical Guide for Applied Research
McQuarrie, E. F., & Krueger, R. A. (1989) · Journal of Marketing Research · doi:10.2307/3172912 · Cited by 15,489
A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection
Kohavi, R. (1995) · Zenodo · doi:10.5281/zenodo.19712698 · Cited by 10,909
I.—COMPUTING MACHINERY AND INTELLIGENCE
TURING, A. M. (1950) · Mind · doi:10.1093/mind/lix.236.433 · Cited by 10,290 · cited 1×
“Note that this resembles a Turing test (Turing, 1950), in that we could say that our system performs well, if participants judge it as well as they judge human performance.”
Background(1993) · Cited by 9,879
Witten, I. H. (2008) · Cited by 9,143
Focus Groups: A Practical Guide for Applied Research
Billson, J. M. (1989) · Human Biology · Cited by 8,560
Ogata, K., & Brewer, J. W. (1971) · Journal of Dynamic Systems Measurement and Control · doi:10.1115/1.3426465 · Cited by 6,380
Computing Machinery and Intelligence (1950)
Turing, A. M. (2004) · doi:10.1093/oso/9780198250791.003.0017 · Cited by 6,120
Evaluating collaborative filtering recommender systems
Herlocker, J. L., Konstan, J. A., Terveen, L. G., & Riedl, J. T. (2004) · ACM Transactions on Information Systems · doi:10.1145/963770.963772 · Cited by 5,798
Flagged first, then by citations
1 of the 201 references couldn’t be matched to a known paper.
Research works printed but not linked: 10
- Albrecht D., Zuckerman I.: Statistical and probabilistic methods for user modeling. Special issue User Model. User-Adap. Inter. 17(1–2), 1–215 (2007)
- Dey, A.K., Abowd, G.D.: Towards a better understanding of context and context-awareness. In: Workshop on the What, Who, Where, When, and How of Context-Awareness at CHI 2000, pp. 304–307. The Hague, The Netherlands (2000)
- Hoppe, H., Tauber, M., Ziegler, J.: A survey of models and formal description methods in HCI with example applications. In: ESPRIT Project, vol. 385 (1986)
- Jameson A.: Systems that Adapt to their Users: An Integrative Perspective. Saarland University, Saarbrücken (2001)
- Ohene-Djan, J.: Ownership transfer via personalisation as a value-adding strategy for web-based education. In: Workshop on Adaptive Systems for Web-Based Education at AH2002, pp. 27–41. Málaga, Spain (2002)
- O’Malley, C.E., Draper, S.W., Riley, M.S.: Constructive interaction: A method for studying Human–Computer-human interaction. In: 1st International Conference on Human–Computer Interaction, pp. 269–274. Honolulu, HI (1984)
- Stary, C., Totter, A.: How to integrate concepts of the design and the evaluation of adaptable and adaptive user interfaces. In: 3rd ERCIM Workshop on User Interfaces for All, pp. 68–75. Obernai, France (1997)
- Tobar, C.M.: Yet another evaluation framework. In: Workshop on Empirical Evaluation of Adaptive Systems at UM2003, pp. 15–24. Johnstown, PA (2003)
- van Barneveld, J., van Setten, M.: Involving users in the design of user interfaces for TV recommender systems. In: 3rd Workshop on Personalization in Future TV at UM03, Johnstown, PA (2003)
- Weibelzahl, S.: Evaluation of adaptive systems. PhD Thesis, University of Trier, Germany (2003)
Cited by
Sorted by citations
Items 101–144 of 144
Towards a Multi-Modal Methodology for User-Centred Evaluation of Adaptive Systems
Nussbaumer, A., Steiner, C., & Conlan, O. (2019) · ACM Conference on User Modeling, Adaptation and Personalization (UMAP) · doi:10.1145/3314183.3323681 · Cited by 3
“For example, different variants of layered evaluation have been proposed, where an adaptive system is decomposed into its main functions and each of them is evaluated individually [1][5].”
BackgroundDebackere, F., Clavel, C., Rören, A., Rannou, F., Nguyen, C., Tran, V.-T., Messai, Y., & Martin, J. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713224 · Cited by 3
“Finally, in the case of adaptive systems, the diversity of adaptation methods employed makes it difficult to compare and evaluate them [49].”
A Multi-Criteria Decision Support Framework for Interactive Adaptive Systems Evaluation
Dhouib, A., Trabelsi, A., Kolski, C., & Néji, M. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-60042-0_42 · Cited by 2
User Expectations Regarding Design Dimensions of Adapative Assistance Systems
Heinz-Jakobs, M., Oestreich, H., Wrede, S., & Röcker, C. (2022) · Conference on Human System Interactions · doi:10.1109/hsi55341.2022.9869509 · Cited by 2
“Other research is investigating the effects of adaptive systems and describes methods for methodically correct evaluations [17].”
MethodVrochidis, A., Tsita, C., Dimitriou, N., Krinidis, S., Panagiotidis, S., Parcharidis, S., Tzovaras, D. K., & Chatzis, V. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-48057-7_17 · Cited by 2
Adaptive hypermedia systems analysis approach by means of the GAF framework
Knutov, E., Bra, P. P. D., & Pechenizkiy, M. (2012) · TU/e Research Portal · Cited by 2
Layered Evaluation of a Personalized Interaction Approach
Augstein, M., & Neumayr, T. (2017) · ACM Conference on User Modeling, Adaptation and Personalization (UMAP) · doi:10.1145/3099023.3099043 · Cited by 2
“section summarizes the results of a study conducted to evaluate the PI process described before. study follows the LE framework for adaptive systems proposed by Paramythis et al. [9].”
BackgroundMethodInfluentialTowards the Layered Evaluation of Interactive Adaptive Systems using ELECTRE TRI Method
Dhouib, A., Trabelsi, A., Kolski, C., & Néji, M. (2017) · International Conference on Software Technologies (ICSOFT) · doi:10.5220/0006437901630170 · Cited by 2
“It is essential to choose the appropriate evaluation methods for individual layers in particular evaluation constraints (Paramythis et al., 2010).”
BackgroundMethodInfluentialA User Modeling Shared Challenge Proposal
Conlan, O., Fraser, K., Kelly, L., & Yousuf, B. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-28577-7_13 · Cited by 2
Developing a Patient-Therapist-Robot User Model for Motivation in Neurorehabilitation Therapies
Bundea, A. (2020) · ACM Conference on User Modeling, Adaptation and Personalization (UMAP) · doi:10.1145/3340631.3398679 · Cited by 2
“The system is evaluated based on frameworks [21], observing the personalization of the system in several parts.”
MethodDhouib, A., Kolski, C., & Néji, M. (2021) · Universal Access in the Information Society · doi:10.1007/s10209-021-00850-y · Cited by 3
“The focus group method is particularly suitable during the specification and design stages of the development process [43].”
BackgroundMethodInfluentialImplementing High-Resolution Adaptivity in Game-Based Learning
Berger, F., & Müller, W. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-33466-5_4 · Cited by 2
Qualitative Evaluation of an Adaptive Exercise Selection Algorithm
Okpo, J. A., Masthoff, J., & Dennis, M. (2021) · ACM Conference on User Modeling, Adaptation and Personalization (UMAP) · doi:10.1145/3450614.3462240 · Cited by 2
“Evaluating an exercise selection algorithm is complicated as there is no standardized method for evaluating these kinds of adaptive algorithms [Conlan et al. 2017], though researchers have argued for layered evaluation [Paramythis et al. 2010].”
MethodSelf-citationCecilia, M. R., & di Orio, F. (2016) · Advances in intelligent systems and computing · doi:10.1007/978-3-319-40165-2_12 · Cited by 1
User Modeling in Human-Centered AI
Herder, E., & Masthoff, J. (2025) · Human-computer interaction series · doi:10.1007/978-3-031-61375-3_6 · Cited by 1
Self-citationNadj, M., Maedche, A., Riedl, R., Seitz, J., & Schultz, T. (2026) · Electronic Markets · doi:10.1007/s12525-026-00895-y · Cited by 1
“This body of work proposes layered evaluation frameworks and formative approaches that enable scholars and practitioners to assess adaptive systems (Paramythis et al., 2010).”
InfluentialThe 1st Release of the TERENCE Learner GUI: The User-Based Usability Evaluation
Cecilia, M. R., Mascio, T. D., & Melonio, A. (2013) · Advances in intelligent systems and computing · doi:10.1007/978-3-319-00554-6_1 · Cited by 1
An Evaluation Framework for Assessing and Optimizing Multimodal Fusion Engines Performance
Feiteira, P., & Duarte, C. (2012) · ... International Conference on Complex, Intelligent and Software Intensive Systems · doi:10.1109/cisis.2012.165 · Cited by 1
“To ensure an efficient evaluation of multimodal interfaces, several authors [6], [7] propose a “divide and conquer” method, which consists on adopting a step-by-step process, of evaluating each component on its own and base later evaluations on the previous ones.”
MethodTowards Semantic Recommendation of Biodiversity Datasets based on Linked Open Data
Löffler, F., Sateli, B., Witte, R., & König‐Ries, B. (2014) · Cited by 1
ISO/IEC 25040 Adaptation for Interactive Adaptive Systems Evaluation
Dhouib, A., Kolski, C., Trabelsi, A., & Néji, M. (2021) · Conference on Human System Interactions · doi:10.1109/hsi52170.2021.9538630 · Cited by 1
Bundea, A. (2021) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3397482.3450712 · Cited by 1
“• User Model Validation [8] and POC of the adaptive system • Testing and validation of the system in a study with participants in rehabilitation therapies and validation by means of post-session analysis by medically trained personnel”
MethodAdeniran, A., & Masthoff, J. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-78270-2_5 · Cited by 1
Self-citationEpp, C. D. (2019) · Springer eBooks · doi:10.1007/978-981-13-2766-7_92 · Cited by 1
Pascual, M. G. (2013) · doi:10.4995/thesis/10251/31660 · Cited by 1
“defined a layered evaluation of adaptive systems and presented an evaluation criterion in terms of adaptation phases (Paramythis et al., 2010), and Paterno et al.”
BackgroundMethodRobotic - Cognitive Adaptive System for Teaching and Learning (R-CASTLE)
Tozadore, D. C. (2020) · doi:10.11606/t.55.2020.tde-31082020-093935 · Cited by 1
Evaluating Adaptive Work-Integrated Learning Systems: From the Lab to the Field
Kump, B., Ley, T., & Lindstaedt, S. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-23985-4_36
“Two main approaches exist to reduce complexity: (a) component-wise evaluation of adaptive systems [2] [3] [4], and (b) evaluation along the development cycles of (adaptive) software systems [4] [3].”
BackgroundSantos, O. C. (2013) · Redalyc (Universidad Autónoma del Estado de México)
Koidl, K., Levacher, K., Conlan, O., & Steichen, B. (2016)
Evaluating Adaptive Systems and Applications is often Nonsense.
Bra, P. M. D. (2016) · TU/e Research Portal
Extending the Evaluation Paradigm for Personalisation
Dowell, J. D., & Constantinides, M. (2019) · ACM Conference on User Modeling, Adaptation and Personalization (UMAP) · doi:10.1145/3314183.3323682
Empirical Evaluation of User Modeling Systems
Chin, D. N. (2019) · ACM Conference on User Modeling, Adaptation and Personalization (UMAP) · doi:10.1145/3314183.3340265
“In addition, this tutorial will cover the specific-to-UM topic of layered evaluation [2], [4], which recommends splitting the evaluation of the entire UM system into more manageable evaluations of components.”
BackgroundKoidl, K. (2014) · Trinity's Access to Research Output (TARA) (Trinity College Dublin)
Examining the Educational Effectiveness of an Adaptive Web-Based Learning Tutorial
Inan, F. A., Ari, F., Flores, R., Zaier, A., & Arslan‐Ari, I. (2021) · International journal on e-learning · doi:10.70725/069874dukmcd
Gisatie: A User Interface Adaptation Life-Cycle
López‐Jaquero, V., Motti, V. G., Montero, F., & López, P. G. (2021) · International Joural of User-System Interaction · doi:10.37789/ijusi.2021.14.1.1
“The authors proposed a framework serving as a guide for layered evaluation of adaptive interactive systems (Paramythis et al., 2010).”
BackgroundWorkshop: ABIS 2012: 19th International Workshop on Adaptivity and User Modeling
Herder, E., Augstein, M., & Heckmann, D. (2012) · R. Oldenbourg Verlag eBooks · doi:10.1524/9783486719918.69
Adaptive Context-Aware Planning Support for Students with Autism
Cromjongh, R. (2023) · ACM Conference on User Modeling, Adaptation and Personalization (UMAP) · doi:10.1145/3565472.3595610
“Finally, to evaluate the separate components of our design, I will use the layered evaluation framework [23] to determine the influence of different parts of the adaptive system, and long-term studies to measure the long-term impact of the design on the lives of participants.”
MethodAl-Abri, A., Al-Khanjari, Z. A. S., & Jamoussi, Y. (2024) · Jurnal Kejuruteraan · doi:10.17576/jkukm-2024-36(4)-36
User Testing of a System Prototype for Adaptive English Learning using Perturbation Learner Model
Dũng, N. V., Sulaiman, S. B., Fadzil, M., Do, V. D., Hassan, M. F., & Iskandar, B. S. (2013) · ISICO 2013
Zhou, Q., Soto, J. P. V., Bai, Z., Parent, M., Todi, K., Jonker, T. R., & Velloso, E. (2026) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2026.3667949
“…of software tools also make it difficult to predict user expectations for adaptive placement of AR interfaces based on those factors [55], or to evaluate adaptive systems (that aim to account for these factors) by enumerating different contextual changes that may affect the adaptation outcome [40].”
Oonk, A. (2026) · ACM Conference on User Modeling, Adaptation and Personalization (UMAP) · doi:10.1145/3774935.3803061
“We will iteratively create and test layers of the prototype [25].”
Representing Uncertainty in Open Learner Models: How Well Do People Interpret It?
Epp, C. D., Gorgun, G., Riley, J., Abdullah, Y. A., & Sisson-Pau, K. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-29770-9_14
Evaluation of Human–AI Teamwork
Lindgren, H., Fenech, K., & Prada, R. (2026) · Springer eBooks · doi:10.1007/978-3-031-61050-9_18-1
Adaptation based on Learning Style and Knowledge Level in E-Learning Systems
(2016) · University of Birmingham Institutional Research Archive (University of Birmingham) · doi:10.13140/rg.2.1.2724.1849 · Cited by 20
Metadata
- Authors
- Paramythis, A., Weibelzahl, S., & Masthoff, J.
- Affiliations
- Johannes Kepler University of Linz (AT)
- National College of Ireland (IE)
- University of Aberdeen (GB)
- Corresponding author
- Paramythis, Alexandros
- Weibelzahl, Stephan
- Masthoff, Judith
- Venue
- User Modeling and User-Adapted Interaction
- Published
- 2010-11-28
- Volume
- 20
- Issue
- 5
- Pages
- 383-453
- ISSN
- 0924-1868 (ISSN-L)
- 1573-1391
- OpenAlex
- W2032468592
- Cited by
- 145
- References
- 201
- Citations by year
- 2026: 5 · 2025: 4 · 2024: 5 · 2023: 5 · 2022: 6 · 2021: 18 · 2020: 6 · 2019: 10 · 2018: 7 · 2017: 9 · 2016: 14 · 2015: 10 · 2014: 12 · 2013: 12 · 2012: 17
- FWCI
- 21.28
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- www.springer.com/tdm (tdm)
- Locations
- User Modeling and User-Adapted Interaction (publishedVersion)
- Aberdeen University Research Archive (Aberdeen University) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
- Physical Sciences › Computer Science › Software › Advanced Software Engineering Methodologies
- Keywords
- adaptive systems, interactive systems, development life cycle, user modeling
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
