Layered evaluation of interactive adaptive systems: framework and formative methods

“Note that this resembles a Turing test (Turing, 1950), in that we could say that our system performs well, if participants judge it as well as they judge human performance.”

“This method is called layered evaluation in Paramythis et al. (2010).”

“We used the layered evaluation framework, which advocates that adaptation needs to be decomposed and assessed in layers in order to be evaluated effectively (Paramythis et al. 2010).”

“As for all other aspects of personalization research, the system and user evaluation would benefit from a common effort and the creation of a shared understanding of which new systems should be evaluated, and how (Weibelzahl, 2002; Paramythis et al., 2010).”

“…J paper is cited as indicated in Table 3, this has mostly been as a reference to the existence of the framework (e.g. Brusilovsky et al., 2011; Paramythis et al., 2010), as a detailed review and/or as giving examples of open learner models (e.g. Hsiao & Brusilovsky, 2012; Kump et al., 2012),…”

“Evaluation of adaptive systems needs to involve end-users and has to be specifically designed to address all aspects of adaptivity (Paramythis et al, 2010; van Velsen et al., 2008).”

“This is a well known issue in evaluating interactive adaptive systems (c.f. Paramythis et al., 2010).”

“TORMES combines user centred design methods with data mining techniques and extends the design cycle of interactive systems as defined by ISO 9241–210 (ISO 2010) with the life cycle of e-learning (Van Rosmalen et al. 2004) and the layered evaluation of adaptive systems (Paramythis et al. 2010).”

“However, some generic frameworks for interactive adaptive systems have being defined following a layered evaluation approach, where adaptation is broken down into its constituents (called layers) and each of these layers is evaluated separately [113].”

“…approach), there are other methodologies that impinge on the design process, namely the e-learning life cycle of personalised educational systems proposed in (Van Rosmalen et al., 2004) and the layered evaluation of the corresponding adaptation features as compiled in (Paramythis et al., 2010).”

“Similarly Paramythis, Weibelzahl and Masthoff identify scrutability as one of the criteria on which to evaluate personalised systems [Paramythis et al. 2010].”

“A comprehensive model for Interactive Adaptive Systems (IAS), taking into account all the previous layered evaluation models, was proposed by Paramythis et al. (2010), according to which the main levels of adaptation are collection of input data, interpretation of the collected data, modeling of…”

“To implement such a personalized system, Paramythis et al. (2010) proposed a blueprint with five different layers collecting input, interpreting input data, modeling the current state of the world, deciding which adaptation to apply, and applying that adaptation.”

“Uncertainty can be introduced from a variety of sources or during any level of information processing [66].”

“…design, prototypes Preliminary Early prototype evaluations, evaluation before implementation Van Velsen et al., 2008; Gena, 2005; Masthoff, 2006 Paramythis et al. (2010) have shown how the traditional layered evaluation methods need to be tailored to suit the particular requirements of…”

“Many previous studies on the evaluation of adaptive systems suggested that it is important to evaluate different blocks separately in order to identify the problems in the adaptive systems (Brusilovsky et al. 2001; Paramythis et al. 2010).”

“Building on previous studies, which found that friends of a user consistently provided better recommendations than a recommender system [22], this experiment adopts the User-as-Wizard method [23, 24].”

“In the literature, these are also often referred to as the semantic, syntactic, and lexical levels according to the three levels of interaction in an interface (e.g. Stephanidis et al. (2012), Paramythis et al. (2010)).”

“The last experiment evaluates the online usage Paramythis et al. (2010).”

“As Paramythis et al. (2010) describe, each layer of an interactive adaptive system should carefully be designed and evaluated.”

“Within the RWUM framework, there is a need to mix qualitative with quantitative assessments in a layered , where it is assumed that she is within a location until a sensor in another area is triggeredevaluation approach (Paramythis et al. 2010) even more than in a traditional UM context.”

“To design an intelligent system which adapts to personality, we can utilize the User as Wizard method [9].”

“Proposal for layered evaluation of adaptive systems [32] includes evaluation of data collection, but does not address specifically its impact on subsequent steps.”

“, [11,31,36], and software evaluation [12,27,59,60,66]).”

“In this context, several research initiatives have been proposed tomodel the context of use, to design and generate the mobile user interface (Gena and Weibelzahl 2007; Paramythis et al. 2010).”

“It defines five layers, corresponding to the main stages of adaptation: (1) collection of input data, (2) interpretation of the collected data, (3) modeling of the current state of the world, (4) deciding upon adaptation, and (5) applying adaptation.”

“A proposal for layered evaluation of adaptive systems (Paramythis et al. 2010) includes the evaluation of data collection but does not address specifically its impact on subsequent steps.”

“[15] further elaborated their decomposition by proposing five layers (or modules): (i) interaction monitoring, (ii) interpretation and interface, (iii) modeling, (iv) personalization decision making, and (v) applying personalizations.”

“This approach is referred to as the ‘User as Wizard’ method (Masthoff, 2006; Paramythis et al., 2010).2 To simply provide a short sentence stating “John is a CFR and is feeling lonely” however is unlikely to elicit a strong enough feeling within participants so that they would empathize with John…”

“We used the layered evaluation proposed by Paramythis et al.[7], which suggest that for effective adaptation, the process needs to be decomposed and evaluated in layers.”

“Finally, in Paramythis et al. [2010], the authors identify the following five main layers of adaptation: i) collection of input data; ii) interpretation of the collected data; iii) modeling of the current state of the “world”; iv) deciding upon adaptation; and v) applying (or instan-tiating)…”

“Layered evaluation [20, 1] has been proposed as a basic framework for evaluation of adaptive systems.”

“…and rewards [35,38,53], personalized prompts [28], personalized arguments [29,54-56], tailored choice of techniques [46, 57-60], adapting to culture [61,62], personalized / tailored persuasion for healthy eating [32, 63-65], ethics of persuasive technology [66], and evaluation methods [67,68].”

“llenges of the ROLE framework and its implementation. For our distributed infrastructure and the diverse test beds we adjusted the postulates of the Layered Evaluation Framework for adaptive systems (Paramythis et al., 2010). We applied a multi-method evaluation approach, meaning that different evaluation instruments…”

“Learning gain Learning effectiveness is a fundamental factor that should be measured when evaluating adaptive e-learning systems [53].”

“Described in [106] the layered evaluation provides a description of the system functionality and helps to solve many related problems.”

“The background knowledge for adaptation decisions is usually derived from implicit interaction cues [15].”

“Paramythis et al [13] proposed the concept of layered evaluation of adaptation, with several evaluation criteria regarding the quality, appropriateness and necessity with respect to the collection of input data, the interpretation of the data, the way it is represented in a user model, how it is used for deciding upon…”

“[17] the evaluation methods can be assigned to one or more of the following five evaluation layers:”

“Adaptive systems dynamically adjust to the user and/or their context over time [125].”

“As in the previous layer, however, it might be dif-ficult for the experts to assess only the presentation decisions, if they do not think the list of recommendations is appropriate (Paramythis et al. 2010).”

“However, the lack of standards among AH designs has hindered the adoption of AH systems (Paramythis, Weibelzahl, & Masthoff, 2010).”

“We used a layered evaluation approach [18], focusing on the decision of an adaptation and how it was applied (in contrast to which data was collected or how it was analyzed).”

“The methodological approach of the CALO Test was a kind of layered evaluation [14,64].”

“The factors considered in the decision making process, are related to: temporal, financial and human resources, location, evaluation phases, type of IAS, types of adaptation, adaptation layers, complexity of task, and user characteristics (i.e. number, expertise with the IAS and age).”

“Authors could collect some strategies of already known frameworks of recommender system evaluation [59], as well as from adaptive systems evaluation [60].”

“Furthermore, interactive adaptive systems have formative methods from which can be analyzed the digital work settings (Paramythis et al. 2010).”

“Paramythis et al. (2010) further elaborated their decomposition by proposing five layers (or modules): (i) interaction monitoring, (ii) interpretation and interface, (iii) modelling, (iv) adaptation decision making, and (v) applying adaptations.”

“Both layers should be evaluated separately in order to effectively interpret the evaluation results (Paramythis et al., 2010).”

“Finally, to ascertain the quality and validity of the UTP module, it should be evaluated using the “layered evaluation” approach [80], which allows input, processing, and output procedures to be evaluated separately.”

“Furthermore, studies involving real users are fraught with challenges: they are typically costly, time-consuming, and plagued by difficulties in participant recruitment [13].”

“grouped together the main approaches and suggested the following main layers of adaptation: (i) collection of input data; (ii) interpretation of the collected data; (iii) modeling the current state of the “world”; (iv) deciding upon adaptation; and (v) applying (or instantiating) adaptation [1].”

“As a result, the recommendation results are inclined to catering to users’ needs [4].”

“user perception, motivation and satisfaction, complementary approaches to evaluation of adaptive systems are emerging suggesting the using of usability evaluation methods such as usability testing and heuristic evaluation [20; 24].”

“web personalization and have been highlighted throughout different publications [45][47].”

“Despite an increasing awareness of the importance of evaluation, current frameworks for evaluating adaptive learning systems and user-adaptive systems, in general, can be considered not yet fully elaborated [3][5].”

“In 2010, Paramythis et al. (2010) proposed another layered evaluation framework, suggesting the decomposition of adaptation process into five layers including: (1) collection of input data in which data about user interaction are collected, (2) interpretation of the collected data which validate…”

“[35], this paper evaluates the personalisation process that provides media presentation and remedial content.”

“How to model and evaluate user experience taking into account user psychological and cognitive properties is a problem [1, 3, 7].”

“When evaluating a student model in isolation, we should provide multiple sources of validity evidence [18, 25] — this is analogical to the common advice in psychometrics.”

“As stated in [11], the adaptation requires (in our case) to: collect input data (the customer’s answers), interpret data (the interpretation of the customer’s utterances), model the current state of the world (the update of the beliefs), decide upon adaptation (the selection of the offers) and apply adaptation (the go…”

“…in the literature, interactive systems are studied from di ff erent perspectives, from performance and usability evaluation (e.g. Norman 2002; Paramythis, Weibelzahl, and Mastho ff 2010; Ras-togi and Singh 2016) to decision making (e.g. Cheung and Lee 2010; Jameson et al. 2014) and…”

“In addition to RQ 5, the evaluation of personalization is challenging and requires a layered evaluation approach [33].”

“Methodology for user-centered evaluation of C evaluation goes hand-in-hand with the H involving users as early as possible in the des it is vital in discovering what and how to interactive system [7].”