CoChat
Sign Up
Cited by 89Mobile Health and mHealth ApplicationsDigital Mental Health InterventionsInnovative Human-Technology Interaction

Opening the Black Box of Electronic Health: Collecting, Analyzing, and Interpreting Log Data

Open access
Sieverink, Floor, Kelders, Saskia, Poel, Mannes, van Gemert-Pijnen, Lisette·2017·JMIR Research Protocols
DOI 10.2196/resprot.6452PMC PMC5565791
Open this paper in CoChat — free accountOpen PDF

Abstract

In electronic health (eHealth) research, limited insight has been obtained on process outcomes or how the use of technology has contributed to the users' ability to have a healthier life, improved well-being, or activate new attitudes in their daily tasks. As a result, eHealth is often perceived as a black box. To open this black box of eHealth, methodologies must extend beyond the classic effect evaluations. The analyses of log data (anonymous records of real-time actions performed by each user) can provide continuous and objective insights into the actual usage of the technology. However, the possibilities of log data in eHealth research have not been exploited to their fullest extent. The aim of this paper is to describe how log data can be used to improve the evaluation and understand the use of eHealth technology with a broader approach than only descriptive statistics. This paper serves as a starting point for using log data analysis in eHealth research. Here, we describe what log data is and provide an overview of research questions to evaluate the system, the context, the users of a technology, as well as the underpinning theoretical constructs. We also explain the requirements for log data, the starting points for the data preparation, and methods for data collection. Finally, we describe methods for data analysis and draw a conclusion regarding the importance of the results for both scientific and practical applications. The analysis of log data can be of great value for opening the black box of eHealth. A deliberate log data analysis can give new insights into how the usage of the technology contributes to found effects and can thereby help to improve the persuasiveness and effectiveness of eHealth technology and the underpinning behavioral models.

AI snapshot

Written from the paper’s full text.

TL;DR

This methodological paper argues that analyzing log data—anonymous records of real-time user actions—can help open the “black box” of eHealth by providing continuous, objective insight into actual usage beyond classic effect evaluations. It offers practical guidance on formulating research questions, collecting and preparing log data, and using descriptive, pathway, predictive, and machine-learning analyses.

Key findings

  • Log data can reveal actual usage patterns and help explain outcomes, but eHealth log analyses have mainly used descriptive statistics such as number of logins, time spent, and frequency of use, often assuming a dose-response relationship without accounting for user goals.
  • Broader methods include pathway analysis, Markov modeling, supervised and unsupervised machine learning, and market-basket analysis; these can predict dropout/adherence, identify usage profiles, and show which elements are used together.
  • Examples cited: users of a personal health record followed the global menu structure when first exploring it, and most were likely to drop out when they visited the education section as a first step after first login; in “Living to the Full,” early non-adherers used fewer and shorter sessions than late non-adherers and adherers, and late non-adherers had a shorter total duration of sessions than adherers.
  • In a fictional data-transformation example, 2 out of 3 users opened the monitoring function but only 1 actually added a value; 2 out of 3 opened their mailbox and sent a message, while User 3 opened the mailbox twice but sent a message only once.

Methods

  • Design: Methodological/guidance paper, not an empirical study; it describes steps for using log data in formative and summative eHealth evaluation.
  • Data or sample: No primary dataset; it uses illustrative examples from prior log data research, including a personal health record for patients with chronic conditions, the “Living to the Full” depression intervention, and a diet support site, plus fictional log data examples.
  • Main analysis: Recommends starting with descriptive frequencies, then pathway analyses, predictive modeling, Markov modeling, and machine-learning algorithms (supervised, unsupervised, market-basket) via Weka; data preparation translates raw records into variables such as sessions, activities, sequences, and time spent.

Limitations

  • Stated: Log data analyses do not always indicate why certain usage patterns occur, so a mixed-methods approach is needed; using log data may require extra effort from researchers, developers, and database managers; predictions are difficult with small groups such as those in an RCT, and not all research questions can be answered with such data.
  • Stated: Privacy and de-anonymization concerns remain, and it is hard to truly anonymize log data; definitions of adherence are often not well defined or evidence-based.
  • Evident: The paper is conceptual and does not empirically test the proposed framework; its examples are explicitly described as illustrative and not an exhaustive overview.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.2196/resprot.6452

Cites

42 of 42 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 60 cited sources checked · 0 supported · 0 flagged

  • Premisecitation

    Log data analyses in eHealth research have mainly focused on descriptive statistics such as number of logins, time spent, and frequency of use of different elements by all users as a group.

    “Log data analyses in eHealth research have mainly focused on descriptive statistics, such as the number of logins, time spent, and the frequency of use of the different elements by all users as a group [18-20].”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites [18] source

      Not checked in the free run.

    • cites [19] Kim, E, 2007, AMIA Annu Symp Proc

      Not checked in the free run.

    • cites [20] source

      Not checked in the free run.

  • Premisecitation

    Users' adoption and long-term use of eHealth technologies remains lower than expected.

    “While such technologies can be effective [1,2], one of the main problems is that users’ adoption and long-term use remains lower than expected [3-5].”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites [3] source

      Not checked in the free run.

    • cites [4] source

      Not checked in the free run.

    • cites [5] source

      Not checked in the free run.

  • Interpretationreasoning

    RCTs cannot provide insight into process outcomes or how use of different components of eHealth technology contributed to healthier living, improved well-being, or daily tasks.

    “Therefore, no insight can be obtained on process outcomes or how the use of the different components of the technology has contributed to healthier living, improved well-being, or a user’s ability to conduct daily tasks [7,9,10].”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites [7] source

      Not checked in the free run.

    • cites [9] source

      Not checked in the free run.

    • cites [10] source

      Not checked in the free run.

  • Premisecitation

    The possibilities of log data in eHealth research have not been exploited to their fullest extent.

    “However, the possibilities of log data in eHealth research have not been exploited to their fullest extent.”
    • Also stated in: § Introduction · plain
      “There is evidence that the use of log data can be of added value to the more traditional approaches of eHealth evaluation, but its possibilities have not yet been exploited to their fullest extent or have mostly been described on a conceptual level [7-9].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [7] source

      Not checked in the free run.

    • cites [8] source

      Not checked in the free run.

    • cites [9] source

      Not checked in the free run.

  • Interpretationreasoning

    eHealth is often perceived as a black box.

    “As a result, eHealth is often perceived as a black box.”
    • Also stated in: § Introduction · boosted
      “This particular lack of insight is known as the “Black Box Phenomenon” [2,10,11].”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites [2] source

      Not checked in the free run.

    • cites [10] source

      Not checked in the free run.

    • cites [11] van Gemert-Pijnen, JE, 2013, Improving eHealth

      Not checked in the free run.

  • May combine two claims. 19
  • Classified as probably not a substantial claim. 33

  • May combine two claims.

    Premisecitation

    eHealth technologies often consist of multiple components that may interact in reaching a certain effect and that people can use in many different ways.

    “Furthermore, to conform to the complexity of behavior change, eHealth technologies often consist of multiple components that may interact in reaching a certain effect and that people can use in many different ways in terms of the elements they use as well as the frequency, time, and place of use [7,8].”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [7] source

      Not checked in the free run.

    • cites [8] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    When analyzing 100 usage sessions, 10 usage sessions of 10 different users provide more generalizeable information than 50 usage sessions of 2 different users.

    “For example, when analyzing 100 usage sessions, 10 usage sessions of 10 different users provide more generalizeable information than 50 usage sessions of 2 different users.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given

    Information regarding user identity, date and time of action, and identification of action is essential to identify the user, logins, and usage patterns within and between logins.

    “Information regarding the user identity, date and time of the action, and an identification of the action is essential to identify the user, logins, and the usage patterns within and between logins.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given

    Client-side logging can provide valuable information regarding usage of technology, containing actions that do not require server requests.

    “Besides server-side logging, client-side logging can also provide valuable information regarding the usage of the technology. Client-side information contains actions that do not require server requests, such as scrolling up and down the screen, moving the pointer, and clicking and filling out a text field.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based). May combine two claims.

    Interpretationnone given

    The amount of data needed depends on the complexity of the research question and can only be determined empirically.

    “The amount of data needed depends on the complexity of the research question and can only be determined empirically.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    A single user action can lead to multiple server requests leading to multiple Web addresses in the log data file, making it harder to identify single user actions.

    “On the other side, a single user action (eg, clicking a button to add a health measurement) can lead to multiple server requests leading to multiple Web addresses in the log data file. This can make it harder to identify single user actions.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    RCTs treat technology as a fixed entity for all participants throughout the intervention period and provide insight only at fixed time points, treating technologies as a singular entity.

    “Although RCTs provide valuable insight into the effectiveness of an intervention, fundamental to this methodology is to have the technology as a fixed entity for all participants throughout the entire intervention period.”
    • Also stated in: § Introduction · plain
      “However, RCTs only provide insight into outcomes at fixed time points and treat technologies as a singular entity.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [6] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisenone given

    eHealth technology has evolved from Web- or telephone-based systems into unobtrusive and pervasive systems embedded into users' daily lives.

    “However, eHealth technology has evolved from Web- or telephone-based systems that required users’ active involvement into unobtrusive and pervasive systems that are embedded into users’ daily lives.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationcitation

    Log data analysis has the potential to explain outcomes of traditional methods by gaining insight into mediating mechanisms and to identify unexpected effects of a technology.

    “Such analyses are a promising approach to explain the outcomes of the more traditional methods, such as RCTs, by gaining insight into the mediating mechanisms that contribute to the found effects [8,9] and have also the potential to identify unexpected effects of a technology.”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [8] source

      Not checked in the free run.

    • cites [9] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationreasoning

    eHealth technology is characterized by constant evolution, and apps or interventions often become obsolete by the time RCT results are available.

    “In contrast, (eHealth) technology can be characterized by its constant evolution and, consequently, apps or interventions often become obsolete by the time the results of the RCT are available.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    A deliberate log data analysis can give new insights into how usage of technology contributes to found effects and can help improve the persuasiveness and effectiveness of eHealth technology and the underpinning behavioral models.

    “A deliberate log data analysis can give new insights into how the usage of the technology contributes to found effects and can thereby help to improve the persuasiveness and effectiveness of eHealth technology and the underpinning behavioral models.”
    • Also stated in: § Composing Research Questions · plain
      “In turn, this information can be used to increase the effectiveness, persuasiveness, and the long-term usage of the technology.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationcitation

    Longer and more frequent exposure to an eHealth technology might indicate how well the system fits users' needs, but can also signify unfocused and/or non-strategic use or an inefficient system.

    “For example, a longer and more frequent exposure to an eHealth technology might indicate how well the system fits the users’ needs, but it can also signify unfocused and/or non-strategic use or an inefficient system [9,21].”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [9] source

      Not checked in the free run.

    • cites [21] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationreasoning

    The fact that many users did not open mindfulness exercises might indicate that the concept of mindfulness is insufficiently operationalized in the intervention and that the found effects are an underestimation of the attainable effects.

    “This might be an indicator that the concept of mindfulness is insufficiently operationalized in the intervention and that the found effects are an underestimation of the attainable effects.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Late non-adherers had a shorter total duration of sessions than adherers.

    “Furthermore, late non-adherers had a shorter total duration of sessions than the adherers.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [21] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given

    This result does not necessarily mean that the same participants who did not participate in the research dropped out from using the technology.

    “this result does not necessarily mean that the same participants who did not participate in the research dropped out from using the technology.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Participants often find it time consuming and labor intensive to complete questionnaires at different time points or to participate in an interview or focus group, resulting in dropouts from the research study.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Early non-adherers used fewer and shorter sessions than late non-adherers and adherers.

    “A 1-way analyses of variance (ANOVA) and chi-square tests showed that early non-adherers used fewer and shorter sessions than late non-adherers and adherers.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [21] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Interpretationreasoning

    Using log data in research might require an extra effort from researchers, developers, database managers, etc.

    “using log data in research might require an extra effort from researchers, developers, database managers, etc.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given

    Log data can be used to show healthcare providers how their responsiveness to client messages influences a client's adherence to the therapy.

    “log data can be used to show healthcare providers how their responsiveness to client messages influences a client’s adherence to the therapy.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given

    It is possible that a user picks up activities where he left off when returning within 30 minutes after the last action.

    “It is possible that a user picks up the activities where he left off when returning within 30 minutes after the last action.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationreasoning

    Preparing data before analysis is vital since typical log data consist of tens of thousands records.

    “Preparing the data before analysis is vital since, for the most part, typical log data consist of tens of thousands records.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Market-basket analysis allows researchers to ask what elements of the technology are often used together.

    “Market-basket analysis allows researchers to ask what elements of the technology are often used together [36].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [36] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    By combining different data sources, new variables do not have to contain, by definition, information from the same data file.

    “By combining different data sources, new variables do not have to contain, by definition, information from the same data file.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationreasoning

    Not all research questions can be answered with log data from small groups.

    “not all research questions can be answered with this data”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    Supervised learning predicts adherence and effects from early use patterns, which enables early intervention for users at risk.

    “Supervised learning predicts adherence and effects from early use patterns, which enables early intervention for users at risk [31,33,34].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [31] source

      Not checked in the free run.

    • cites [33] Han, J, 2011, Data Mining: Concepts and Techniques

      Not checked in the free run.

    • cites [34] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given

    Behavior change theories and behavior models often form the basis for the content and structure of eHealth technologies.

    “Behavior change theories and behavior models often form the basis for the content and the structure of eHealth technologies.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationreasoning

    Log data can provide insight into what the most effective usage patterns are and for whom to experience improvements in self-efficacy.

    “Or, when one of the goals of a technology is to improve self-efficacy, for example, log data can provide insight into what the most effective usage patterns are and for whom to experience any improvements in the self-efficacy.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given

    Log data can be used to assess the theories that underpin a technology.

    “log data can be used to assess the theories that underpin a technology.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    Unsupervised learning determines what usage profiles appear from the log data and if this data can be matched to a certain group of participants.

    “Unsupervised learning determines what usage profiles appear from the log data and if this data can be matched to a certain group of participants [33,35].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [33] Han, J, 2011, Data Mining: Concepts and Techniques

      Not checked in the free run.

    • cites [35] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

    Interpretationnone given

    Answers to the research questions can provide input for improving the look and feel and architecture of the technology as well as the fit between the technology, the user, and the context.

    “The answers to the research questions can provide input for improving the look and feel and architecture of the technology as well as the fit between the technology, the user, and the context.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationreasoning

    This information can be used to increase the effectiveness, persuasiveness, and long-term usage of the technology.

    “In turn, this information can be used to increase the effectiveness, persuasiveness, and the long-term usage of the technology.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    Mere login data does not reveal adherence, but assigning a substantiated threshold value does.

    “mere login data (eg, the more logins, the better the adherence is) does not reveal adherence, but assigning a substantiated threshold value does.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationnone given

    In the early operationalization phase, log data has the potential to evaluate use of the system and assess what people do or do not do with a technology.

    “In the early operationalization phase, for example, log data has the potential to evaluate the use of the system and to assess what people do (or not do) with a technology.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationreasoning

    The analysis of log data can be of great value for scientists and designers as well as caregivers and policy makers in their research into the black box of eHealth technology.

    “The analysis of log data can be of great value for scientists and designers as well as caregivers and policy makers in their research into the black box of eHealth technology.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Log data is always available and easy to collect, without requiring any extra effort from the participants.

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given

    A deliberate analysis of log data can provide insight into the usage of the technology by all users as a group as well as by individual users.

    “A deliberate analysis of log data can provide insight into the usage of the technology by all users as a group as well as by individual users”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Log data is always available and easy to collect, without requiring any extra effort from the participants.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The Waikato Environment for Knowledge Analysis (Weka) tool is a relatively accessible and easy to use software package for applying machine learning algorithms for data mining tasks.

    “the Waikato Environment for Knowledge Analysis (Weka) tool is a relatively accessible and easy to use software package for applying machine learning algorithms for data mining tasks [32].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [32] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given

    Log data can be used to assess the uptake of the technology.

    “Log data can be used to assess the uptake of the technology.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The 6th lesson is a hard lesson for participants because of the focus on observing themselves and learning new skills to accept suffering.

    “It was revealed by counselors giving the course that this is indeed a hard lesson for participants because of the focus on observing themselves and learning new skills to accept suffering [21].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [21] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Freyne et al found that uploading a profile picture on a diet support site in the first week resulted in higher return rates.

    “In another example, Freyne et al [25] found that uploading a profile picture on a diet support site in the first week resulted in higher return rates.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [25] FreyneJSaundersIBrindalEBerkovskySSmithGFactors associated with persistent participation in an online diet interventionCHI '12 Extended Abstracts on Human Factors in Computing Systems2012CHI Conference on Human Factors in Computing Systems2012 May 05-12Austin, Texas237580 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    In a previous study, users of a personal health record followed the global menu structure when exploring the PHR for the first time, and most users were likely to drop out when they visited the education section as a first step after the first login.

    “For example, in a previous study we found that users of a personal health record (PHR), an electronic application consisting of different tools for monitoring and coaching patients with chronic conditions to support self-management [22], followed the global menu structure when exploring the PHR for the first time. Furthermore, most users were likely to drop out when they visited the education sect”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [23] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationreasoning

    Critical moments for dropping out can be identified, as well as profiles for both users and usage.

    “Critical moments for dropping out can be identified, as well as profiles for both users and usage.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    Exercising for at least 4 times a week for 6 weeks was the minimum activity to establish an exercise habit.

    “Kaushal and Rhodes discovered that exercising for at least 4 times a week for 6 weeks was the minimum activity to establish an exercise habit [47].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [47] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    An effect evaluation indicated that a Web-based technology based on acceptance and commitment therapy might help users reduce depressive complaints.

    “An effect evaluation indicated that a Web-based technology based on ACT might help users reduce depressive complaints [26].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [26] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    It is difficult to make predictions based on the usage data of relatively small groups of users (eg, in a RCT).

    “it is difficult to make predictions based on the usage data of relatively small groups of users (eg, in a RCT)”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    By using log data in addition to questionnaires, researchers have more than one method to collect data and are no longer dependent on having a majority of the participants complete questionnaires or participate in interviews or focus groups.

    “By using log data in addition to questionnaires, researchers have more than one method to collect data and are no longer dependent on having a majority of the participants complete questionnaires or participate in interviews or focus groups.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Participants often find it time consuming and labor intensive to complete questionnaires at different time points or to participate in an interview or focus group, resulting in dropouts from the research study.
44 of 44

  • The paper relies on this without support.

    Log data represents actual and continuous usage behavior and not subjective, recalled behavior.

    “An important benefit of log data is that it represents the actual and continuous usage behavior and not subjective, recalled behavior.”

  • The paper relies on this without support.

    The use of log data mainly focuses on the interaction with Web-based technologies.

    “The use of log data mainly focuses on the interaction with Web-based technologies.”

  • The paper relies on this without support.

    Informed consent depends upon whether log data includes or needs to be combined with personal data.

    “Informed consent depends upon whether log data includes or needs to be combined with personal data.”

  • The paper relies on this without support.

    It remains undecided whether log data in itself is personal data.

    “Currently, it remains undecided whether log data in itself is personal data, as usage data does not always, per se, contain information that can be traced back to individual users.”

  • The paper relies on this without support.

    Participants often find it time consuming and labor intensive to complete questionnaires at different time points or to participate in an interview or focus group, resulting in dropouts from the research study.

    “A common problem in (eHealth) research is that participants often find it time consuming and labor intensive to complete questionnaires at different time points or to participate in an interview or focus group, resulting in dropouts from the research study.”

  • The paper relies on this without support.

    Until now, modifications of technologies have often been made after an evaluation period.

    “Until now, modifications of technologies have often been made after an evaluation period.”

  • The paper relies on this without support.

    Log data is always available and easy to collect, without requiring any extra effort from the participants.

    “An advantage of log data is that it is always available and easy to collect, without requiring any extra effort from the participants.”
7 of 7

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Cites

Flagged first, then by citations

Research works printed but not linked: 6

  1. Oinas-Kukkonen, H, 2009, CAIS, 24, 28
  2. van Gemert-Pijnen, JE, 2013, Improving eHealth
  3. Kim, E, 2007, AMIA Annu Symp Proc, 409
  4. AkkersdijkSMKeldersSMBraakman-JansenLMAvan Gemert-PijnenJEUsing Markov chains to analyze paths through a personal health recordPersuasive Technology XII (Adjunct Proceedings)2017Persuasive Technology XII2017 April 03-06Amsterdam
  5. Kaptein, M, 2012, JIM, 26, 176
  6. KapteinMLacroixJSainiPIndividual Differences in Persuadability in the Health Promotion DomainPERSUASIVE 20102010Proceedings of the 5th International Conference on Persuasive Technology2010 Jun 07-10Copenhagen, Denmark94105

Cited by

Sorted by citations

Related

Authors
Sieverink, F., Kelders, S., Poel, M., & van Gemert-Pijnen, L.
Affiliations
  • University of Twente (NL)
  • North-West University (ZA)
Corresponding author
Sieverink, Floor
ORCID
Venue
JMIR Research Protocols
Publisher
JMIR Publications Inc.
Published
2017-08-07
Volume
6
Issue
8
Pages
e156
ISSN
1929-0748 (ISSN-L)
DOI
10.2196/resprot.6452
PMC
PMC5565791
OpenAlex
W2744373335
Cited by
88
References
42
Citations by year
2026: 9 · 2025: 7 · 2024: 9 · 2023: 5 · 2022: 8 · 2021: 13 · 2020: 13 · 2019: 15 · 2018: 7 · 2017: 2
FWCI
16.40
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • JMIR Research Protocols (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • University of Twente Research Information (publishedVersion, cc-by)
  • University of Twente Research Information (submittedVersion, cc-by)
Topics
  • Health Sciences › Health Professions › General Health Professions › Mobile Health and mHealth Applications
  • Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
  • Physical Sciences › Computer Science › Human-Computer Interaction › Innovative Human-Technology Interaction
Keywords
e-health, log analysis, eHealth evaluation, process outcomes, behavioral models, data preparation, data collection, user behavior

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-10.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-10.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review