Analyzing Engagement in a Web-Based Intervention Platform Through Visualizing Log-Data
Open accessAbstract
BACKGROUND: Engagement has emerged as a significant cross-cutting concern within the development of Web-based interventions. There have been calls to institute a more rigorous approach to the design of Web-based interventions, to increase both the quantity and quality of engagement. One approach would be to use log-data to better understand the process of engagement and patterns of use. However, an important challenge lies in organizing log-data for productive analysis. OBJECTIVE: Our aim was to conduct an initial exploration of the use of visualizations of log-data to enhance understanding of engagement with Web-based interventions. METHODS: We applied exploratory sequential data analysis to highlight sequential aspects of the log data, such as time or module number, to provide insights into engagement. After applying a number of processing steps, a range of visualizations were generated from the log-data. We then examined the usefulness of these visualizations for understanding the engagement of individual users and the engagement of cohorts of users. The visualizations created are illustrated with two datasets drawn from studies using the SilverCloud Platform: (1) a small, detailed dataset with interviews (n=19) and (2) a large dataset (n=326) with 44,838 logged events. RESULTS: We present four exploratory visualizations of user engagement with a Web-based intervention, including Navigation Graph, Stripe Graph, Start-Finish Graph, and Next Action Heat Map. The first represents individual usage and the last three, specific aspects of cohort usage. We provide examples of each with a discussion of salient features. CONCLUSIONS: Log-data analysis through data visualization is an alternative way of exploring user engagement with Web-based interventions, which can yield different insights than more commonly used summative measures. We describe how understanding the process of engagement through visualizations can support the development and evaluation of Web-based interventions. Specifically, we show how visualizations can (1) allow inspection of content or feature usage in a temporal relationship to the overall program at different levels of granularity, (2) detect different patterns of use to consider personalization in the design process, (3) detect usability issues, (4) enable exploratory analysis to support the design of statistical queries to summarize the data, (5) provide new opportunities for real-time evaluation, and (6) examine assumptions about interactivity that underlie many summative measures in this field.
AI snapshot
Written from the paper’s full text.
TL;DR
This paper explores visualizations of log-data as a way to understand engagement with Web-based interventions beyond summative adherence measures. Using two SilverCloud datasets, it presents four exploratory visualizations—Navigation Graph, Stripe Graph, Start–Finish Graph, and Next Action Heat Map—to reveal temporal and individual patterns of use.
Key findings
- Four visualizations were developed: Navigation Graph (individual usage), Stripe Graph (cohort temporal use), Start–Finish Graph (session start/end pages), and Next Action Heat Map (cohort navigation linearity).
- The visualizations showed diverse individual trajectories: Janet progressed relatively linearly and repeated content, while Robert was app-focused, jumped across modules, and viewed reviews 28 times although only 8 reviews were written.
- Cohort patterns challenged assumptions behind summative metrics: many users used the intervention in bursts, used it beyond the formal 8-week period, and did not complete modules in single sittings, suggesting module completion may not accurately reflect engagement.
- The paper reports using a small dataset with interviews (n=19) and a large dataset (n=326) with 44,838 logged events.
Methods
- Exploratory sequential data analysis was applied to log-data to highlight sequential/temporal aspects such as time or module number.
- Data came from two studies using the SilverCloud platform: a small pilot in a UK primary-care mental health setting with 19 participants and interview data, and a larger university-setting dataset with 326 participants and 44,838 logged events.
- Log-data included user ID, timestamp, and page URL; URLs were transformed into meaningful labels, sessions were segmented using a 60-minute inactivity threshold, and visualizations were generated with freely available programs.
Limitations
- Stated: the visualizations cannot capture offline or downloaded use, such as Robert listening to downloaded mindfulness exercises daily, and without interviews the authors would not know users’ motivations or meanings.
- Stated: artifacts can appear because apps such as Mindfulness could be reached from multiple locations without separate URLs, making some navigation jumps hard to interpret.
- Evident: the work is an initial exploration using examples from one platform, the authors state these are not “the ultimate set” of visualizations, and they note further work is needed to determine best visualizations for real-time evaluation.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
0 of 29 cited sources checked · 0 supported · 0 flagged
- Findingown data
In Figure 7, extreme cases of app-focused and content-focused usage are visible.
“In Figure 7, we can see extreme cases of app-focused and content-focused usage showing the importance of preference in how people engage.”
Quote in the paperNot a substantial claim - Findingown data
Janet and Robert had very different patterns of use.
“Comparing the visualizations immediately shows that these two users had very different patterns of use.”
Quote in the paperNot a substantial claim - Findingown data
People were doing the interactive exercises in modules 1, 2, and 3 consistently.
“The red dots that come at the beginning of modules 1, 2, and 3 in the top row, indicate that people were doing the interactive exercises in the modules consistently.”
Quote in the paperNot a substantial claim - Findingown data
Adherence is often not consistent but comes in bursts.
“the adherence is often not consistent but comes in bursts”
Quote in the paperSubstantial claim - Findingown data
Many users did not start at the beginning of one module and stop at the end.
“many users did not start at the beginning of one module and stop at the end”
Quote in the paperSubstantial claim
- Findingown data
The gaps corresponded to optional content where multiple different examples were available in a carousel to illustrate the same point.
“the gaps corresponded to optional content where multiple different examples were available in a carousel to illustrate the same point”
Quote in the paperNot a substantial claim - Findingown data
Preferences for certain types of interactivity are highly specific to the individual.
“preferences for certain types of interactivity are highly specific to the individual”
Quote in the paperSubstantial claim - Findingown data
Individual patterns of use are diverse in Figures 5-7.
“It is striking how diverse individual patterns of use are in Figures 5-7.”
Quote in the paperSubstantial claim - Findingown data
The review and optional modules, 6 and 7 respectively, were not done as frequently.
“The review and optional modules, 6 and 7 respectively, were also not done as frequently.”
Quote in the paperNot a substantial claim - Findingreasoning
No usability problems were identified for this intervention.
“we did not identify any usability problems for this intervention”
Quote in the paperNot a substantial claim - Findingown data
More people seemed to stop at intermediate points between modules 4 and 5 than modules 2 and 3.
“more people seemed to stop at intermediate points between modules 4 and 5 than modules 2 and 3”
Quote in the paperNot a substantial claim - Findingown data
Engagement wanes at the end of the intervention.
“engagement wanes at the end of the intervention”
Quote in the paperSubstantial claim - Findingown data
Many people used the intervention in bursts sometimes with long periods in between.
“many people used the intervention in bursts sometimes with long periods in between”
Quote in the paperSubstantial claim - Findingown data
There is significant usage beyond the formally supported 8-week period.
“It is also interesting to note the significant level of usage beyond the formally supported 8-week period.”
Quote in the paperSubstantial claim - Findingown data
Many people did not complete module 0-the introduction to the intervention.
“Many people did not complete module 0—the introduction to the intervention.”
Quote in the paperNot a substantial claim - Premisecitation
Log-data analysis provides a way to capture patterns of use.
“Log-data analysis, similar to that used in the analysis of websites in other domains [27], provides a way to capture patterns of use.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [27] RoddenKHutchinsonHFuXMeasuring the user experience on a large scale: user-centered metrics for web applications2010SIGCHI Conference on Human Factors in Computing SystemsApril 10-15, 2010Atlanta, GAAssociation for Computing Machinery23952398 source
Not checked in the free run.
- Premisecitation
A substantial amount of variance seen in adherence between studies can be explained by the characteristics of the Web-based intervention, that is, interactive technological elements.
“Rather, a substantial amount of variance seen in adherence between studies can be explained by the characteristics of the Web-based intervention, that is, interactive technological elements [26].”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [26] source
Not checked in the free run.
- Interpretationreasoning
Summative measures do not capture the temporal elements of engagement needed to provide insight into the design of Web-based interventions.
“Such summative measures do not capture the temporal elements of engagement needed to provide insight into the design of Web-based interventions.”
Quote in the paperSubstantial claim - Interpretationreasoning
We cannot capture such elements of intervention use in the log-data.
“We cannot capture such elements of intervention use in the log-data.”
Quote in the paperSubstantial claim- Rests on an assumption: Some of the apps, such as the Mindfulness exercises, could be run in the browser or downloaded.
- Interpretationown data
The top two examples illustrate that content review is an important element of usage.
“The top two examples illustrate that content review is an important element of usage.”
Quote in the paperSubstantial claim - Premisecitation
High rates of attrition have kept Web-based interventions from reaching their potential.
“However, obstinately high rates of attrition have kept them from reaching their potential [7,8].”
Quote in the paperSubstantial claim0 of 2 sources checked - Interpretationown data
People's usage may not be as straightforward as they describe.
“These examples indicate that people’s usage may not be as straightforward as they describe.”
Quote in the paperSubstantial claim - Study design and conductreasoning
The combination of outcome data, log-data, and interview data for 19 participants allows exploration of engagement patterns of individuals in depth.
“The combination of data allows us to explore engagement patterns of individuals in depth.”
Quote in the paperSubstantial claim - Premisecitation
Previous research has focused on the summative measurement of engagement, such as level of adherence or intervention exposure.
“Previous research has focused on the summative measurement of engagement, such as level of adherence [12,13] or intervention exposure [14-16].”
Quote in the paperSubstantial claim0 of 5 sources checked - Interpretationown data
The visualizations in Figure 7 make apparent very different pattern of use trajectories that would be hard to detect through manual inspection of the data.
“The visualizations in Figure 7 make apparent very different pattern of use trajectories that would be hard to detect through manual inspection of the data.”
Quote in the paperSubstantial claim - Interpretationown data
There are some perspectives that the visualization cannot capture.
“There are some perspectives that the visualization cannot capture.”
Quote in the paperSubstantial claim - Premisecitation
There are no precedents for representation of continuous data in log-data analysis using exploratory sequential data analysis.
“While this approach has been applied to log-data analysis for session events grouped into a small number of distinct categories [32], there are no precedents for representation of continuous data.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [32] source
Not checked in the free run.
- Interpretationown data
Not all users move through the content in the linear order planned but find their own pathways when allowed.
“This difference emphasizes that not all users move through the content in the linear order planned but find their own pathways when allowed.”
Quote in the paperSubstantial claim - Interpretationown data
The visualizations detect different patterns of use to consider personalization in the design process.
“detect different patterns of use to consider personalization in the design process”
Quote in the paperSubstantial claim - Interpretationown data
Usage visualizations offer a different perspective on interactivity.
“Usage visualizations also offer a different perspective on interactivity.”
Quote in the paperNot a substantial claim - Interpretationcitation
Log-data visualization can, with other methods, add to the richness of understanding engagement needed to support the iterative design approach for complex interventions.
“Log-data visualization can, with other methods, add to the richness of understanding engagement needed to support the iterative design approach for complex interventions [40].”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [40] Campbell, M, 2000, BMJ source
Not checked in the free run.
- Interpretationown data
Module completion may not be an accurate reflection of engagement.
“The Navigation Graphs and the Start–Finish Graphs both suggest that module completion may not be an accurate reflection of engagement either.”
Quote in the paperSubstantial claim - Interpretationown data
The visualizations presented here question the implicit assumption of linear usage that underpins summative adherence metrics such as intervention completion or module completion.
“The visualizations presented here question the implicit assumption of linear usage that underpins summative adherence metrics such as intervention completion or module completion.”
Quote in the paperSubstantial claim- Rests on an assumption: Summative adherence metrics assume linear usage.
- Interpretationreasoning
The visualizations examine assumptions about interactivity that underlie many summative measures in this field.
“examine assumptions about interactivity that underlie many summative measures in this field”
Quote in the paperSubstantial claim - Interpretationcitation
Log-data visualization can offer a representation to support the elicitation of experience through qualitative interviewing.
“it can offer a representation to support the elicitation of experience through qualitative interviewing [38]”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [38] ProsserJLoxleyAIntroducing Visual Methods20082014-10-21UKNational Centre for Research Methodshttp://eprints.ncrm.ac.uk/420/6TUCPI5pK
Not checked in the free run.
- Interpretationreasoning
The visualizations can help detect usability issues that may mask the underlying effectiveness of the intervention.
“The visualization can also help detect usability issues that may mask the underlying effectiveness of the intervention.”
Quote in the paperSubstantial claim - Interpretationnone given
The visualizations allow inspection of content or feature usage in a temporal relationship to the overall program at different levels of granularity.
“allow inspection of content or feature usage in a temporal relationship to the overall program at different levels of granularity”
Quote in the paperSubstantial claim
- May combine two claims. 14
- Classified as probably not a substantial claim. 41
May combine two claims.
Findingown dataFigures 5 and 6 show contrasting preferences to a lesser extreme with clear routines emerging quickly.
“Figures 5 and 6 also show these contrasting preferences to a lesser extreme with clear routines emerging quickly.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataJanet progressed in a more or less linear manner through the content, while Robert jumped around.
“Janet progressed in a more or less linear manner through the content, while Robert jumped around.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataJanet focused on content and Robert on apps.
“Janet focused on content and Robert on apps.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataSome users took several sessions to complete a module, while others did several in one sitting.
“Some took several sessions to complete a module, while others did several in one sitting.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataIn Figure 10, there was the strongest page-by-page usage in module 1, with a similar trend to a lesser extent in modules 2 and 3.
“there was the strongest page-by-page usage in module 1, the first content module, with a similar trend to a lesser extent in modules 2 and 3”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
Findingnone givenPeople return repeatedly to content they find relevant to them.
“knowing that people return repeatedly to content they find relevant to them”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingown dataJanet consistently used the diary at the beginning or ends of most sessions.
“Janet consistently used the diary at the beginning or ends of most sessions for example.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataPeople were taking advantage of the multiple routes in each module, focusing on content that was of most interest and moving back and forth from apps.
“people were taking advantage of the multiple routes in each module, focusing on content that was of most interest and moving back and forth from apps”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingown dataThe Navigation Graph illustrates the back and forth nature of content intake and interactive activities.
“The Navigation Graph, for example, illustrates the back and forth nature of content intake and interactive activities.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingnone givenAt the time of application development, it was not considered that apps accessed from multiple locations should have separate URLs for ease of tracking interaction.
“At the time of application development, it was not considered that apps accessed from multiple locations should have separate URLs for ease of tracking interaction.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataPeople used the multiple navigation pathways while still finding the proposed navigation useful.
“people used the multiple navigation pathways while still finding the proposed navigation useful”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Interpretationnone givenThe Start-Finish Visualization depicts the shape of users' sessions across a cohort, showing the starting and finishing content page of each session.
“Start–Finish Visualization depicts the shape of users’ sessions across a cohort, showing the starting and finishing content page of each session.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S2 (checkable).
Interpretationown dataThese visualizations provide an interesting contrast with feedback garnered from interviews.
“These visualizations provide an interesting contrast with feedback garnered from interviews.”
Quote in the paperNot a substantial claimMay combine two claims.
Interpretationown dataThese examples stand in contrast to enforced linear use in many Web-based interventions and suggests that some summative measures, such as module completion, may not always be appropriate.
“These examples stand in contrast to enforced linear use in many Web-based interventions and suggests that some summative measures, such as module completion, may not always be appropriate.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S2 (checkable).
InterpretationreasoningIt is essential to understand the temporal process of engagement or patterns of use.
“These two findings taken together suggest that it is essential to understand the temporal process of engagement or patterns of use.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim.
PremisecitationThere is not a linear relationship between usage (either adherence or exposure) and outcome.
“This is echoed in recent work that has demonstrated that there is not a linear relationship between usage (either adherence or exposure) and outcome [25].”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [25] source
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
Interpretationnone givenThe Next Action Heat Map Visualization aggregates the navigation path through the intervention.
“Finally, Next Action Heat Map Visualization aggregates the navigation path through the intervention.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Interpretationnone givenThe Stripe Graph Visualization illustrates the temporal pattern of use over the intervention period for individual users and is compact enough to be stacked, allowing usage across a cohort to be examined and compared.
“Stripe Graph Visualization illustrates the temporal pattern of use over the intervention period for individual users but is compact enough to be stacked, allowing usage across a cohort to be examined and compared.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S2 (checkable).
InterpretationreasoningThe Navigation visualization captures a substantial amount of information about an individual's pattern of use.
“The Navigation visualization captures a substantial amount of information about an individual’s pattern of use.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim.
PremisecitationThere have been calls to develop a more scientific approach to the design and evaluation of Web-based interventions, including the analysis of engagement.
“As the corpus of research studies grows, there have been calls to develop a more scientific approach to the design and evaluation of Web-based interventions: a methodology that includes the analysis of engagement.”
Quote in the paperSubstantial claim0 of 3 sources checkedProbably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).
PremisecitationExploratory sequential data analysis involves the successive transformation of raw data sequences until the product enables statements to be made that answer research or design questions.
“Exploratory sequential data analysis involves the successive transformation of raw data sequences until the product enables statements to be made that answer research or design questions [31].”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [31] source
Not checked in the free run.
May combine two claims.
Interpretationnone givenWithout interviewing users, we would not know that Janet treated the intervention like a course to be completed and Robert saw it as his sanctuary to help him through sleepless nights.
“Without interviewing users, we would not know that Janet treated the intervention like a course to be completed and Robert saw it as his sanctuary to help him through sleepless nights.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S2 (checkable), S4 (based).
Interpretationnone givenThe challenge of using log-data to understand the process of engagement lies in organizing it for productive analysis.
“The challenge of using log-data to understand the process of engagement lies in organizing it for productive analysis.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
PremisecitationInformation visualization can be used either to explore or communicate a set of data.
“Information visualization can be used either to explore or communicate a set of data [30].”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [30] Tukey, J, 1977, Exploratory data analysis
Not checked in the free run.
May combine two claims.
Study design and conductown dataThe second dataset contains 326 participants and over 44,838 logged events, which allows exploration of issues surrounding visualization of a larger cohort.
“The second dataset contains 326 participants, and over 44,838 logged events, which allows us to explore the issues surrounding visualization of a larger cohort.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Interpretationnone givenThe Navigation Graph Visualization illustrates the temporal process of engagement by an individual.
“The first, Navigation Graph Visualization, illustrates the temporal process of engagement by an individual.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductreasoningThe 60-minute inactivity threshold was set to make allowance for users watching videos repeatedly or taking time to compose longer textual entries for certain exercises.
“The threshold has been set to make allowance for users watching videos repeatedly or taking time to compose longer textual entries for certain exercises.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S2 (checkable).
Interpretationown dataThis would not be obvious without the visualization.
“This would not be obvious without the visualization.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductreasoningAs session time is not a feature in these visualizations, the impact of the threshold is only in the segmentation of the data into different sessions.
“As session time is not a feature in these visualizations, the impact is only in the segmentation of the data into different sessions.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationLog-data has been used to illustrate changes in frequency of feature usage over time and staged feature usage.
“Log-data has been used, for example, to illustrate changes in frequency of feature usage over time (eg, feedback messages) as well as staged feature usage (eg, starting a mindfulness exercise, but not downloading it) [28].”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [28] source
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationAttempts to focus on interactive technological elements of an intervention have reported positive correlations between interactive feature inclusion and outcomes.
“Attempts to focus more directly on interactive technological elements of an intervention that may facilitate engagement have reported positive correlations between interactive feature inclusion and outcomes [21-24].”
Quote in the paperNot a substantial claim0 of 4 sources checkedProbably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationThe SilverCloud platform can be used to deliver a range of programs (eg, depression and anxiety).
“The sample platform can be used to deliver a range of programs (eg, depression [36] and anxiety [37]).”
Quote in the paperNot a substantial claim0 of 2 sources checkedProbably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationResearchers have identified factors that correlate with adherence, including patient characteristics, context of delivery, aspects of delivery, and characteristics of the intervention.
“Researchers have also identified factors that correlate with adherence, including patient characteristics [17] (eg, gender), the context of delivery [18] (eg, classroom), aspects of the delivery [19] (eg, therapist support), and characteristics of the intervention itself [20] (eg, tailored content).”
Quote in the paperNot a substantial claim0 of 3 sources checkedProbably not a substantial claim: may not pass S2 (checkable), S4 (based).
Interpretationnone givenThe visualizations provide new opportunities for real-time evaluation.
“provide new opportunities for real-time evaluation”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
Interpretationnone givenIf many people are stopping mid-way through a module in the Start-Finish Graph, it suggests that people are losing interest at this point.
“If many people are stopping mid-way through a module in the Start–Finish Graph, it suggests that people are losing interest at this point.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S2 (checkable).
InterpretationreasoningIt is unlikely for the research community to settle on a single set of visualizations for all interventions.
“it is unlikely for the research community to settle on a single set of visualizations for all interventions”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S2 (checkable).
Interpretationown dataMeasures of adherence calculated based on weekly usage should not be considered in isolation.
“suggesting that measures of adherence calculated based on weekly usage should not be considered in isolation”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S2 (checkable).
InterpretationreasoningEngagement visualizations can allow continual assessment of an intervention as it is changed to incorporate new technologies or adapted to new settings.
“Engagement visualizations can allow continual assessment of an intervention as it is changed to incorporate new technologies or adapted to new settings.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim.
Interpretationnone givenThe visualizations offer an alternative view on user engagement with a Web-based intervention than more commonly used summative measures.
“We offer these as an alternative view on user engagement with a Web-based intervention than more commonly used summative measures.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).
InterpretationreasoningLog-data visualization is likely to be most useful in conjunction with other types of data.
“Log-data visualization is likely to be most useful in conjunction with other types of data.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S2 (checkable).
InterpretationreasoningA shared toolset within the community will require some degree of standardization, for example through a common format for log-data from online interventions.
“A shared toolset within the community, which allows exploratory and not just confirmatory analysis of data, will require some degree of standardization, for example through a common format for log-data from online interventions.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
InterpretationreasoningBlue dots (low frequency usage) at random points can happen because the Mindfulness app can be reached from a number of locations without any indication in the log-data.
“it can also happen because the Mindfulness app can be reached from a number of locations without any indication in the log-data”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S2 (checkable). May combine two claims.
InterpretationreasoningVisualization will need to some extent be specific to the design intent of the Web-based intervention as it is best suited to exploratory analysis.
“visualization will need to some extent be specific to the design intent of the Web-based intervention as it is best suited to exploratory analysis”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S2 (checkable).
Interpretationown dataA search for the 'best' interactivity may be the wrong line of inquiry.
“This finding indicates that a search for the “best” interactivity may be the wrong line of inquiry.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S4 (based).
Interpretationnone givenThe visualizations detect usability or content issues.
“detect usability or content issues”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S2 (checkable).
InterpretationreasoningSummative clinical trials alone are unlikely to provide the flexibility needed to address the continuing ongoing refinement of Web-based interventions.
“Summative clinical trials alone are unlikely to provide the flexibility needed to address the continuing ongoing refinement of Web-based interventions.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S2 (checkable).
Interpretationnone givenThe four visualizations provide a novel way to interrogate the patterns of use through log-data of individual usage and specific aspects of cohort usage.
“These four visualizations provide a novel way to interrogate the patterns of use through log-data of individual usage and specific aspects of cohort usage.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
InterpretationcitationLog-data visualization could be used as a method of exploratory analysis before the creation and calculation of engagement statistics.
“It could also be used as a method of exploratory analysis before the creation and calculation of engagement statistics, a technique currently being developed by Kelders & Gemert-Pijnen [39].”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [39] source
Not checked in the free run.
Probably not a substantial claim: may not pass S2 (checkable).
InterpretationreasoningCommon fields such as event identifier (URL), timestamp, and user identifier are clearly required.
“Common fields such as event identifier (URL), timestamp, and user identifier are clearly required”
Quote in the paperNot a substantial claimMay combine two claims.
InterpretationreasoningLarge breaks in the Next Action Heat Map indicate that many people are following routes other than proposed, highlighting a potential usability or content issue.
“large breaks in the Next Action Heat Map indicate that many people are following routes other than proposed, highlighting a potential usability or content issue”
Quote in the paperSubstantial claimClassified as probably not a substantial claim.
Interpretationnone givenThe visualizations enable exploratory analysis to support the design of statistical queries to summarize the data.
“enable exploratory analysis to support the design of statistical queries to summarize the data”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S2 (checkable).
InterpretationreasoningLog-data visualization provides new opportunities for evaluation.
“Log-data visualization also provides new opportunities for evaluation.”
Quote in the paperNot a substantial claim
The paper relies on this without support.
SilverCloud is specifically designed to improve engagement through personalization, interactive exercises, guidance and support, and social features.
“It is specifically designed to improve engagement through the following design strategies:”
The paper relies on this without support.
Some of the apps, such as the Mindfulness exercises, could be run in the browser or downloaded.
“A further issue is that some of the apps, such as the Mindfulness exercises, could be run in the browser or downloaded.”
The paper relies on this without support.
Most adherence metrics assume consistency in usage.
“the consistency assumed in most adherence metrics”
The paper relies on this without support.
Most studies tend to focus on the formally defined usage period associated with their post-intervention outcome measures.
“Most studies tend to focus on the formally defined usage period associated with their post-intervention outcome measures.”
The paper relies on this without support.
Many interventions measure their usage through module completion.
“many interventions measure their usage through module completion”
The paper relies on this without support.
Summative adherence metrics assume linear usage.
“the implicit assumption of linear usage that underpins summative adherence metrics such as intervention completion or module completion”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Freely available visualization tools are useful for detecting usability or content issues, enabling researchers to edit content for future Web-based interventions.
“detecting usability or content issues, and thereby enable researchers to edit content for use in future Web-based interventions”
From A Visualization Tool to Analyse Usage of Web-Based Interventions: The Example of Positive Online Weight Reduction (POWeR) (2015) · cites this paper as [15]
Freely available visualization tools are useful for performing exploratory analysis to support the design of statistical queries about whether use of particular pages is related to benefit.
“performing exploratory analysis to support the design of statistical queries to summarize data regarding whether use of particular pages is related to benefit [15].”
From A Visualization Tool to Analyse Usage of Web-Based Interventions: The Example of Positive Online Weight Reduction (POWeR) (2015) · cites this paper as [15]
Freely available visualization tools are useful for detecting patterns of usage and how they vary across individuals/groups.
“Freely available visualization tools have been developed and argued to be useful for detecting patterns of usage and how they vary across individuals/groups”
From A Visualization Tool to Analyse Usage of Web-Based Interventions: The Example of Positive Online Weight Reduction (POWeR) (2015) · cites this paper as [15]
Visualizations use aspects of exploratory sequencing techniques to summarize and plot participant usage of every intervention component over time.
“By contrast, visualizations use aspects of exploratory sequencing techniques to summarize and plot the participant’s usage of every intervention component over time [15].”
From A Visualization Tool to Analyse Usage of Web-Based Interventions: The Example of Positive Online Weight Reduction (POWeR) (2015) · cites this paper as [15]
The visualization tool complements the work of Morrison and Doherty [15] by enabling an in-depth analysis of all participants' usage of every intervention component within one sequence plot.
“The visualization tool complements the work of Morrison and Doherty [15] by enabling an in-depth analysis of all participants’ usage of EVERY intervention component within one sequence plot.”
From A Visualization Tool to Analyse Usage of Web-Based Interventions: The Example of Positive Online Weight Reduction (POWeR) (2015) · cites this paper as [15]
The toolkit developed by Morrison and Doherty [15] may be particularly beneficial for individual-level analyses or group-level analyses of nontunneled interventions that do not have a clear start and end point.
“The toolkit developed by Morrison and Doherty[15] may be particularly beneficial for individual-level analyses or group-level analyses of nontunneled interventions that do not have a clear start and end point.”
From A Visualization Tool to Analyse Usage of Web-Based Interventions: The Example of Positive Online Weight Reduction (POWeR) (2015) · cites this paper as [15]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
A gap exceeding 60 minutes remains in the same session
MinorTable 1 assigns both events to session 4, although they are 3 hours 52 minutes apart. Under the stated 60-minute inactivity threshold, the 16:41 event should start a new session.
“| 8 | 08/03/2012 12:49 | journal/1537/delete/ | Journal | 4 | | 8 | 08/03/2012 16:41 | | Home | 4 |”
“The threshold is configurable, and for the examples presented here, a value of 60 minutes is used.”
Cites
Readings in Information Visualization: Using Vision to Think
Card, S. K., Mackinlay, J. D., & Shneiderman, B. (1999) · Cited by 3,959 · see citing works
Framework for design and evaluation of complex interventions to improve health
Campbell, M. (2000) · BMJ · doi:10.1136/bmj.321.7262.694 · Cited by 3,474 · see citing works
Eysenbach, G. (2005) · Journal of Medical Internet Research · doi:10.2196/jmir.7.1.e11 · Cited by 2,524 · see citing works
Bulmer, M. (1978) · Sociology · doi:10.1177/003803857801200331 · Cited by 1,438 · see citing works
Persuasive System Design Does Matter: a Systematic Review of Adherence to Web-based Interventions
Kelders, S. M., Kok, R. N., Ossebaard, H. C., & Van Gemert-Pijnen, J. E. (2012) · Journal of Medical Internet Research · doi:10.2196/jmir.2104 · Cited by 1,308 · see citing works
Barak, A., Hen, L., Boniel-Nissim, M., & Shapira, N. (2008) · Journal of Technology in Human Services · doi:10.1080/15228830802094429 · Cited by 1,283 · see citing works
Adherence in Internet Interventions for Anxiety and Depression
Christensen, H., Griffiths, K. M., & Farrer, L. (2009) · Journal of Medical Internet Research · doi:10.2196/jmir.1194 · Cited by 1,133 · see citing works
A Systematic Review of the Impact of Adherence on the Effectiveness of e-Therapies
Donkin, L., Christensen, H., Naismith, S. L., Neal, B., Hickie, I. B., & Glozier, N. (2011) · Journal of Medical Internet Research · doi:10.2196/jmir.1772 · Cited by 876 · see citing works
Griffiths, F., Lindenmeyer, A., Powell, J., Lowe, P., & Thorogood, M. (2006) · Journal of Medical Internet Research · doi:10.2196/jmir.8.2.e10 · Cited by 820 · see citing works
The Delivery of Public Health Interventions via the Internet: Actualizing Their Potential
Bennett, G. G., & Glasgow, R. E. (2009) · Annual Review of Public Health · doi:10.1146/annurev.publhealth.031308.100235 · Cited by 800 · see citing works
Interactive Health Communication Applications for people with chronic disease
Murray, E., Burns, J., See Tai, S., Lai, R., & Nazareth, I. (2005) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd004274.pub4 · Cited by 722 · see citing works
Newman, M. G., Szkodny, L. E., Llera, S. J., & Przeworski, A. (2011) · Clinical Psychology Review · doi:10.1016/j.cpr.2010.09.008 · Cited by 480 · see citing works
Brouwer, W., Kroeze, W., Crutzen, R., de Nooijer, J., de Vries, N. K., Brug, J., & Oenema, A. (2011) · Journal of Medical Internet Research · doi:10.2196/jmir.1639 · Cited by 455 · see citing works
Internet-based physical activity interventions: a systematic review of the literature.
van den Berg, M. H., Schoones, J. W., & Vliet Vlieland, T. P. (2007) · Journal of Medical Internet Research · doi:10.2196/jmir.9.3.e26 · Cited by 377 · see citing works
Griffiths, K. M., Farrer, L., & Christensen, H. (2010) · Medical Journal of Australia · doi:10.5694/j.1326-5377.2010.tb03685.x · Cited by 303 · see citing works
Web-Based Smoking-Cessation Programs
Strecher, V. J., McClure, J. B., Alexander, G. L., Chakraborty, B., Nair, V. N., Konkel, J. M., Greene, S. M., Collins, L. M., Carlier, C. C., Wiese, C. J., et al. (2008) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2007.12.024 · Cited by 266 · see citing works
Strecher, V., McClure, J., Alexander, G., Chakraborty, B., Nair, V., Konkel, J., Greene, S., Couper, M., Carlier, C., Wiese, C., et al. (2008) · Journal of Medical Internet Research · doi:10.2196/jmir.1002 · Cited by 264 · see citing works
Engagement and retention: measuring breadth and depth of participant use of an online intervention
Couper, M. P., Alexander, G. L., Zhang, N., Little, R. J., Maddy, N., Nowak, M. A., McClure, J. B., Calvi, J. J., Rolnick, S. J., Stopponi, M. A., et al. (2010) · Journal of Medical Internet Research · doi:10.2196/jmir.1430 · Cited by 244 · see citing works
Measuring the user experience on a large scale: user-centered metrics for web applications
Rodden, K., Hutchinson, H., & Fu, X. (2010) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1753326.1753687 · Cited by 233 · see citing works
Engagement in a Diabetes Self-management Website: Usage Patterns and Generalizability of Program Use
Glasgow, R. E., Christiansen, S. M., Kurz, D., King, D. K., Woolley, T., Faber, A. J., Estabrooks, P. A., Strycker, L., Toobert, D., & Dickman, J. (2011) · Journal of Medical Internet Research · doi:10.2196/jmir.1391 · Cited by 219 · see citing works
Donkin, L., Hickie, I. B., Christensen, H., Naismith, S. L., Neal, B., Cockayne, N. L., & Glozier, N. (2013) · Journal of Medical Internet Research · doi:10.2196/jmir.2771 · Cited by 193 · see citing works
Methodological issues in research on web-based behavioral interventions
Danaher, B. G., & Seeley, J. R. (2009) · Annals of Behavioral Medicine · doi:10.1007/s12160-009-9129-0 · Cited by 193 · see citing works
A review of computer and Internet-based interventions for smoking behavior
Walters, S. T., Wright, J. A., & Shegog, R. (2005) · Addictive Behaviors · doi:10.1016/j.addbeh.2005.05.002 · Cited by 191 · see citing works
Neil, A. L., Batterham, P., Christensen, H., Bennett, K., & Griffiths, K. M. (2009) · Journal of Medical Internet Research · doi:10.2196/jmir.1050 · Cited by 152 · cited 1× · see citing works
“Researchers have also identified factors that correlate with adherence, including patient characteristics [17] (eg, gender), the context of delivery [18] (eg, classroom), aspects of the delivery [19] (eg, therapist support), and characteristics of the intervention itself [20] (eg, tailored content).”
Defining Participant Exposure Measures in Web-Based Health Behavior Change Programs
Danaher, B. G., Boles, S. M., Akers, L., Gordon, J. S., & Severson, H. H. (2006) · Journal of Medical Internet Research · doi:10.2196/jmir.8.3.e15 · Cited by 145 · cited 1× · see citing works
“Previous research has focused on the summative measurement of engagement, such as level of adherence [12,13] or intervention exposure [14-16].”
The Science of Internet Interventions
Ritterband, L. M., & Tate, D. F. (2009) · Annals of Behavioral Medicine · doi:10.1007/s12160-009-9132-5 · Cited by 111 · see citing works
The Ins and Outs of an Online Bipolar Education Program: A Study of Program Attrition
Nicholas, J., Proudfoot, J., Parker, G., Gillis, I., Burckhardt, R., Manicavasagar, V., & Smith, M. (2010) · Journal of Medical Internet Research · doi:10.2196/jmir.1450 · Cited by 97 · see citing works
Van Gemert-Pijnen, J. E., Kelders, S. M., & Bohlmeijer, E. T. (2014) · Journal of Medical Internet Research · doi:10.2196/jmir.2991 · Cited by 84 · cited 2× · see citing works
“Log-data has been used, for example, to illustrate changes in frequency of feature usage over time (eg, feedback messages) as well as staged feature usage (eg, starting a mindfulness exercise, but not downloading it) [28].”
An, L. C., Schillo, B. A., Saul, J. E., Wendling, A. H., Klatt, C. M., Berg, C. J., Ahulwalia, J. S., Kavanaugh, A. M., Christenson, M., & Luxenberg, M. G. (2008) · Journal of Medical Internet Research · doi:10.2196/jmir.1018 · Cited by 70 · see citing works
Session Viewer: Visual Exploratory Analysis of Web Session Logs
Lam, H., Russell, D., Tang, D., & Munzner, T. (2007) · 2007 IEEE Symposium on Visual Analytics Science and Technology · doi:10.1109/vast.2007.4389008 · Cited by 62 · see citing works
Sharry, J., Davidson, R., McLoughlin, O., & Doherty, G. (2013) · Journal of Medical Internet Research · doi:10.2196/jmir.2248 · Cited by 55 · see citing works
Self-citationRitterband, L. M., Cox, D. J., Gordon, T. L., Borowitz, S. M., Kovatchev, B. P., Walker, L. S., & Sutphen, J. L. (2006) · Children's Health Care · doi:10.1207/s15326888chc3501_5 · Cited by 39 · cited 1× · see citing works
“Attempts to focus more directly on interactive technological elements of an intervention that may facilitate engagement have reported positive correlations between interactive feature inclusion and outcomes [21-24].”
Richards, D., Timulak, L., Doherty, G., Sharry, J., Colla, A., Joyce, C., & Hayes, C. (2014) · BMC Psychiatry · doi:10.1186/1471-244x-14-147 · Cited by 35 · see citing works
Self-citationUsing Log-Data as a Starting Point to Make eHealth More Persuasive
Kelders, S. M., & van Gemert-Pijnen, J. E. W. C. (2013) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-37157-8_13 · Cited by 24 · see citing works
Richards, D., Timulak, L., Doherty, G., Sharry, J., McLoughlin, O., Rashleigh, C., Colla, A., & Joyce, C. (2014) · Trials · doi:10.1186/1745-6215-15-145 · Cited by 16 · cited 1× · see citing works
“The sample platform can be used to deliver a range of programs (eg, depression [36] and anxiety [37]).”
Self-citation
Flagged first, then by citations
Research works printed but not linked: 5
- Griffiths, KM, 2010, Med J Aust, 192, S4
- 10.2196/jmir.1194 · DOI · Paper page
- 10.1080/07370024.1994.9667208 · DOI
- ShneidermanBThe eyes have it: A task by data type taxonomy for information visualizationsSymp Vis Lang (IEEE )1996Boulder, CO336343
- Morrison, C, 2015, The Cognitive Behavioural Therapist, in press
Cited by
Sorted by citations
Perski, O., Blandford, A., West, R., & Michie, S. (2017) · Translational Behavioral Medicine · doi:10.1007/s13142-016-0453-1 · Cited by 1,193 · see citing works
The promise of machine learning in predicting treatment outcomes in psychiatry
Chekroud, A. M., Bondar, J., Delgadillo, J., Doherty, G., Wasil, A., Fokkema, M., Cohen, Z., Belgrave, D., DeRubeis, R., Iniesta, R., et al. (2021) · World Psychiatry · doi:10.1002/wps.20882 · Cited by 531 · see citing works
Self-citationThieme, A., Belgrave, D., & Doherty, G. (2020) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3398069 · Cited by 406 · see citing works
Self-citationFoundations of online learning: Challenges and opportunities
Greenhow, C. M., Graham, C. R., & Koehler, M. J. (2022) · Educational Psychologist · doi:10.1080/00461520.2022.2090364 · Cited by 289 · see citing works
Gamification and Adherence to Web-Based Mental Health Interventions: A Systematic Review
Brown, M., O'Neill, N., van Woerden, H., Eslambolchilar, P., Jones, M., & John, A. (2016) · JMIR Mental Health · doi:10.2196/mental.5710 · Cited by 178 · see citing works
Hermes, E. D., Lyon, A. R., Schueller, S. M., & Glass, J. E. (2019) · Journal of Medical Internet Research · doi:10.2196/11752 · Cited by 149 · see citing works
Pham, Q., Graham, G., Carrion, C., Morita, P. P., Seto, E., Stinson, J. N., & Cafazzo, J. A. (2019) · JMIR mHealth and uHealth · doi:10.2196/11941 · Cited by 131 · see citing works
Enrique, A., Palacios, J. E., Ryan, H., & Richards, D. (2019) · Journal of Medical Internet Research · doi:10.2196/12775 · Cited by 81 · see citing works
Health Technologies 'In the Wild'
Rennick‐Egglestone, S., Knowles, S. E., Toms, G. R., Bee, P., Lovell, K., & Bower, P. (2016) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858128 · Cited by 68 · see citing works
Miller, S., Ainsworth, B., Yardley, L., Milton, A., Weal, M. J., Smith, P. W. F., & Morrison, L. G. (2018) · Journal of Medical Internet Research · doi:10.2196/10966 · Cited by 66 · see citing works
Chen, A. T., Wu, S., Tomasino, K. N., Lattie, E. G., & Mohr, D. C. (2019) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2019.103187 · Cited by 58 · see citing works
“Previous research has observed that a critical challenge to the analysis of digital interventions is the organization of log data for analysis[25].”
BackgroundComparing usage of a web and app stress management intervention: An observational study
Morrison, L. G., Geraghty, A. W. A., Lloyd, S., Goodman, N., Michaelides, D. T., Hargood, C., Weal, M., & Yardley, L. (2018) · Internet Interventions · doi:10.1016/j.invent.2018.03.006 · Cited by 57 · see citing works
Lederman, R., Gleeson, J., Wadley, G., D’alfonso, S., Rice, S., Santesteban-Echarri, O., & Alvarez-Jimenez, M. (2019) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3301421 · Cited by 46 · see citing works
Understanding users’ perspectives on mobile apps for anxiety management
Balaskas, A., Schueller, S. M., Cox, A. L., & Doherty, G. J. (2022) · Frontiers in Digital Health · doi:10.3389/fdgth.2022.854263 · Cited by 49 · see citing works
Self-citationAddressing engagement in technology-based behavioural HIV interventions through paradata metrics
Bauermeister, J. A., Golinkoff, J., Muessig, K. E., Horvath, K. J., & Hightow‐Weidman, L. (2017) · Current Opinion in HIV and AIDS · doi:10.1097/coh.0000000000000396 · Cited by 46 · see citing works
Milward, J., Deluca, P., Drummond, C., & Kimergård, A. (2018) · JMIR mHealth and uHealth · doi:10.2196/11692 · Cited by 38 · see citing works
Ainsworth, B., Steele, M., Stuart, B., Joseph, J., Miller, S., Morrison, L., Little, P., & Yardley, L. (2017) · Annals of Behavioral Medicine · doi:10.1007/s12160-016-9866-9 · Cited by 41 · see citing works
Mulvenna, M. D., Bond, R., Delaney, J., Dawoodbhoy, F. M., Boger, J. N., Potts, C., & Turkington, R. (2021) · Philosophy & Technology · doi:10.1007/s13347-021-00445-8 · Cited by 43 · see citing works
Kemmeren, L. L., van Schaik, A., Smit, J. H., Ruwaard, J., Rocha, A., Henriques, M., Ebert, D. D., Titzler, I., Hazo, J.-B., Dorsey, M., et al. (2019) · JMIR Mental Health · doi:10.2196/12707 · Cited by 36 · see citing works
Bell, L., Garnett, C. V., Qian, T., Perski, O., Williamson, E., & POTTS, H. W. W. (2020) · Journal of Medical Internet Research · doi:10.2196/23369 · Cited by 33 · see citing works
Arden-Close, E. J., Smith, E., Bradbury, K., Morrison, L., Dennison, L., Michaelides, D., & Yardley, L. (2015) · JMIR Human Factors · doi:10.2196/humanfactors.4310 · Cited by 32 · see citing works
Dobashi, K., Ho, C. P., Fulford, C. P., Grace Lin, M.-F., & Higa, C. (2022) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2022.100105 · Cited by 35 · see citing works
Initial Field Trial of a Coach-Supported Web-Based Depression Treatment
Schueller, S. M., & Mohr, D. C. (2015) · International Conference on Pervasive Computing Technologies for Healthcare · doi:10.4108/icst.pervasivehealth.2015.260115 · Cited by 25 · see citing works
Richmond, H., Hall, A. M., Hansen, Z., Williamson, E. M., Davies, D., & Lamb, S. E. (2016) · BMC Medical Education · doi:10.1186/s12909-016-0683-4 · Cited by 24 · see citing works
Lim, B. Y., Kay, J., & Liu, W. (2019) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3328928 · Cited by 24 · see citing works
Morrison, C. P., D’Souza, M., Huckvale, K., Dorn, J. F., Burggraaff, J., Kamm, C. P., Steinheimer, S., Kontschieder, P., Criminisi, A., Uitdehaag, B. M. J., et al. (2015) · JMIR Human Factors · doi:10.2196/humanfactors.4129 · Cited by 22 · see citing works
Self-citationScarpa, M. P., Prilletensky, I., McMahon, A., Myers, N. D., Prilleltensky, O., Lee, S., Pfeiffer, K. A., Bateman, A. G., & Brincks, A. M. (2021) · Frontiers in Computer Science · doi:10.3389/fcomp.2021.690389 · Cited by 20 · see citing works
Jin, S., Kim, B., & Han, K. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713732 · Cited by 15 · see citing works
Examining young adults daily perspectives on usage of anxiety apps: A user study
Balaskas, A., Schueller, S. M., Cox, A. L., Rashleigh, C., & Doherty, G. J. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000185 · Cited by 14 · see citing works
Self-citationJawad, D., Cheng, H., Wen, L. M., Rissel, C. E., Baur, L. A., Mihrshahi, S., & Taki, S. (2022) · Journal of Medical Internet Research · doi:10.2196/38641 · Cited by 13 · see citing works
Meyer, D., Jayawardana, M., Muir, S. D., Ho, D. Y.-T., & Sackett, O. (2018) · Journal of Medical Internet Research · doi:10.2196/jmir.9058 · Cited by 13 · see citing works
The Application of User Event Log Data for Mental Health and Wellbeing Analysis
Turkington, R., Mulvenna, M. D., Bond, R., O.Neill, S., & Armour, C. (2018) · Electronic workshops in computing · doi:10.14236/ewic/hci2018.4 · Cited by 12 · see citing works
Mihuta, M. E., & Green, H. J. (2017) · Supportive Care in Cancer · doi:10.1007/s00520-017-3855-9 · Cited by 11 · see citing works
Usher‐Smith, J. A., Winther, L. R., Shefer, G., Šilarova, B., Payne, R., & Griffin, S. J. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.7697 · Cited by 11 · see citing works
Linehan, C., & Kirman, B. J. (2017) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3027063.3052753 · Cited by 10 · see citing works
Bond, R., Moorhead, A., Mulvenna, M. D., O’Neill, S. M., Potts, C., & Murphy, N. (2019) · Behaviour and Information Technology · doi:10.1080/0144929x.2019.1648553 · Cited by 10 · see citing works
Supporting Self-Care of Adolescents with Nut Allergy Through Video and Mobile Educational Tools
Davidson, N., Vines, J., Bartindale, T., Sutton, S. J., Green, D., Comber, R., Balaam, M., Olivier, P. L., & Vance, G. H. S. (2017) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025680 · Cited by 9 · see citing works
Stassen, G., Grieben, C., Froböse, I., & Schaller, A. (2020) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph17072180 · Cited by 10 · see citing works
Myneni, S., Sridharan, V., Cobb, N. K., & Cohen, T. (2018) · Journal of Participatory Medicine · doi:10.2196/jopm.9745 · Cited by 8 · see citing works
Petersen, G. B., Nielsen, J. H., Olsen, J. V., & Kok, R. N. (2019) · doi:10.31234/osf.io/3ewz4 · Cited by 8 · see citing works
Mi, R. Z., Yang, E. F., Tahk, A. M., Tarfa, A., Cotter, L. M., Lu, L., Yang, S., Gustafson, D. H., Westergaard, R. P., & Shah, D. V. (2024) · Journal of Medical Internet Research · doi:10.2196/57774 · Cited by 7 · see citing works
Zhang, Z., Fang, H., & Wang, H. (2015) · IEEE International Conference on e-Health Networking, Applications and Services (Healthcom) · doi:10.1109/healthcom.2015.7454549 · Cited by 7 · see citing works
Teacher Engagement Interventions through ITEI Apps
Sasmoko, S., Indrianti, Y., Widhoyoko, S. A., Udjaja, Y., & Tanurwijaya, A. (2018) · International Conference on Cyber and IT Service Management (CITSM) · doi:10.1109/citsm.2018.8674049 · Cited by 6 · see citing works
Mulvenna, M. D., Bond, R., Grigorash, A., O’Neill, S. M., & Ryan, A. A. (2018) · Ulster University Research Portal (Ulster University) · Cited by 6 · see citing works
Usher‐Smith, J. A., Masson, G., Mills, K., Sharp, S. J., Sutton, S. R., Klein, W. M. P., & Griffin, S. J. (2018) · BMC Public Health · doi:10.1186/s12889-018-5712-2 · Cited by 6 · see citing works
Wang, Y., Dabbs, A. D., Thomas, T. H., Campbell, G. B., & Donovan, H. S. (2024) · JMIR Formative Research · doi:10.2196/57529 · Cited by 4 · see citing works
Mapping User Trajectories to Examine Behavior and Outcomes in Digital Health Intervention Data
Chen, A., Chang, J. H., Hallinan, S., & Mohr, D. C. (2019) · Workshop on Visual Analytics in Healthcare (VAHC) · doi:10.1109/vahc47919.2019.8945038 · Cited by 4 · see citing works
Student Engagement Awareness in an Asynchronous E-Learning Environment
Wakjira, A., & Bhattacharya, S. (2023) · International Journal of Technology-Enabled Student Support Services · doi:10.4018/ijtesss.316211 · Cited by 5 · see citing works
Gao, P., Guan, L., Liu, Y., Liu, F., Yu, W., Li, X., Liu, S., Lu, Y., Li, H., & Xiang, H. (2020) · Global Health Journal · doi:10.1016/j.glohj.2020.01.003 · Cited by 3 · see citing works
Bernstein, E. E., Daniel, K. E., Miyares, P. E., Hoeppner, S. S., Bentley, K. H., Snorrason, Í., Fisher, L. B., Greenberg, J. L., Weingarden, H. M., Harrison, O., et al. (2025) · JMIR Mental Health · doi:10.2196/63497 · Cited by 3 · see citing works
Learning based visual engagement and self-efficacy
Dhamija, S. (2017) · International Conference on Affective Computing and Intelligent Interaction and workshops · doi:10.1109/acii.2017.8273659 · Cited by 3 · see citing works
What is the State of the Art on UX Data Visualization? A Systematic Mapping of the Literature
do Amaral, L. K., Macedo, M. P., & Zaina, L. A. M. (2025) · Journal on Interactive Systems · doi:10.5753/jis.2025.4487 · Cited by 2 · see citing works
Sync Ratio and Cluster Heat Map for Visualizing Student Engagement
Dobashi, K. (2023) · Big data management · doi:10.1007/978-981-99-0026-8_7 · Cited by 1 · see citing works
The SupportNet Website: A Social Media Self-Care System
Yeager, C. M., Bock, J., & Benight, C. C. (2016) · Palgrave Macmillan US eBooks · doi:10.1057/978-1-349-95103-1_3
Learning Visual Engagement for Trauma Recovery
Dhamija, S., & Boult, T. E. (2018) · IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW) · doi:10.1109/wacvw.2018.00016
A scoping review of the use of log data for evaluating mobile Apps (Preprint)Preprint
van Schalkwyk, A., Grobbelaar, S. S. S., & Vermeulen, E. (2021) · doi:10.2196/preprints.31407
Jawad, D., Cheng, H., Wen, L. M., Rissel, C. E., Baur, L. A., Mihrshahi, S., & Taki, S. (2022) · doi:10.2196/preprints.38641
A MOBILE HEALTH DESIGN PROCESS TO IMPROVE COMMUNITY HEALTH WORKERS' ENGAGEMENT WITH MOBILE HEALTH
Ikwunne, T. A., & Wall, L. H. A. P. J. (2023) · IADIS International Conferences Interfaces and Human Computer Interaction; Game and Entertainment Technologies; and Computer Graphics, Visualization, Computer Vision and Image Processing (IHCI/GET/CGVCVIP) · doi:10.33965/mccsis2023_202304l005
Visualização de dados sobre UX: um estudo preliminar da literatura
do Amaral, L. K., Macedo, M. P., & Zaina, L. A. M. (2023) · Simpósio Brasileiro de Fatores Humanos em Sistemas Computacionais (IHC) · doi:10.5753/ihc_estendido.2023.233479
Jung, G., Lim, H., Han, K., Kim, H., & Lee, U. (2025) · JMIR Human Factors · doi:10.2196/64967 · Cited by 1 · see citing works
Measuring User Engagement in Open Data Platforms
Satrio, B., Kleiman, F., & Janssen, M. (2026) · Lecture notes in business information processing · doi:10.1007/978-3-032-31564-9_6
- Authors
- Morrison, C., & Doherty, G.
- Affiliations
- University of Cambridge (GB)
- Trinity College Dublin (IE)
- Corresponding author
- Morrison, Cecily
- Venue
- Journal of Medical Internet Research
- Publisher
- JMIR Publications Inc.
- Published
- 2014-11-13
- Volume
- 16
- Issue
- 11
- Pages
- e252
- ISSN
- 1438-8871 (ISSN-L)
- PMC
- PMC4260085
- OpenAlex
- W2085586906
- Semantic Scholar
- 48dd78431abb8eb8413669bcf67132b628a04973
- Cited by
- 62
- References
- 35
- Citations by year
- 2026: 1 · 2025: 4 · 2024: 2 · 2023: 5 · 2022: 5 · 2021: 5 · 2020: 4 · 2019: 9 · 2018: 11 · 2017: 6 · 2016: 6 · 2015: 4
- FWCI
- 8.26
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- Journal of Medical Internet Research (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- National Institute for Health and Care Research
- Science Foundation Ireland (10/CE/i1855, 12/CE/I2267)
- Topics
- Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Data Visualization and Analytics
- Physical Sciences › Computer Science › Computer Science Applications › Online Learning and Analytics
- Keywords
- user engagement, online intervention, sequential data analysis, navigation graph, usage patterns, cohort analysis, personalization, usability problems, real-time assessment, interactivity
- MeSH terms
- Computer Graphics, Decision Support Techniques, Humans, Patient Care Management, Therapeutics, Internet
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-10.
Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-10.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
