Screenomics: a new approach for observing and studying individuals’ digital lives.

“After multiple iterations, we obtained a random forest with 600 trees (Hastie, Tibshirani, & Friedman, 2001; Liaw & Wiener, 2002) that had an out-of-bag error rate of only 6.8%, and used that model to propagate the app labels to the rest of the data.”

“, following particular YouTube channels) that might facilitate modeling of food-related behavior through vicarious engagement with similar others (Bandura, 1986).”

“Specifically, the screenomes were analyzed using a grounded theory approach following the “constant comparison” process outlined by Glaser and Strauss (Glaser, 1992; Glaser & Strauss, 1967; Strauss & Corbin, 1998).”

“For example, our values, experiences, interests, beliefs, and, importantly for this study, cultural background and social and professional identities, inevitably shape how we approached and analyzed the data (Berger, 2015).”

“…for dashboards that provide summary information from intensive longitudinal data streams (such as might be obtained from monitors and wearables), and interpretations that might eventually become part of practitioner’s regular workflow (Nahum-Shani et al., 2017; Shneiderman et al., 2016).”

“From a methodological perspective, the uniqueness of individuals’ media use implies a shift toward person-specific idiographic approaches (see Molenaar, 2004), or at least that such analyses should supplement the typical engagement with group-level averaging.”

“Inferences range from statements about how adolescents from different eras use media differently (Twenge, 2017) to gender differences in social media dependence (Pea et al., 2012) to multitasking effects that differ by cognitive abilities (Ophir, Nass, & Wagner, 2009).”

“This distinction is important given prior findings that passive consumption of social media is associated with lower subjective well-being and greater envy, both across persons and within persons (Krasnova, Wenninger, Widjaja, & Buxmann, 2013; Sagioglou & Greitemeyer, 2014; Verduyn et al., 2015).”

“Inferences range from statements about how adolescents from different eras use media differently (Twenge, 2017) to gender differences in social”

“…smart-phone devices, though, has not been documented, in part because people’s self-reports of use are inaccurate (correlations between reported and actual use, r s < .50, agreement between reported and actual use, κ s < .20; Boase & Ling, 2013; Gold, Rauscher, & Zhu, 2015; Mireku et al., 2018).”

“Complementing bioecological models of development that opened discovery of the complex interplay of the micro-and macro-processes that drive development (Bronfenbrenner & Morris, 2006), these new data open up a new realm—the dynamic digital environment—that is already influencing individual development, health, and we…”

“Developmental science has long championed and prioritized analysis of intraindividual change and dynamic processes, both of which require longitudinal data (Baltes & Nesselroade, 1979; Molenaar, 2004; Ram & Gerstorf, 2009).”

“This distinction is important given prior findings that passive consumption of social media is associated with lower subjective well-being and greater envy, both across persons and within persons (Krasnova, Wenninger, Widjaja, & Buxmann, 2013; Sagioglou & Greitemeyer, 2014; Verduyn et al., 2015).”

“Inferences range from statements about how adolescents from different eras use media differently (Twenge, 2017) to gender differences in social media dependence (Pea et al., 2012) to multitasking effects that differ by cognitive abilities (Ophir, Nass, & Wagner, 2009).”

“Digitalization and representation of adolescents’ developmental tasks onto personal screens provides new opportunity for observing how those tasks manifest in daily life (Lundby, 2014; Subrahmanyam & Smahel, 2011).”

“…the research on media and adolescents is focused on identifying between-person variations in exposures and their correlates in large samples of adolescents (e.g., higher frequency of digital media use is associated with greater symptoms of attention-deficit/hyperactivity disorder, Ra et al., 2018).”

“…salient, relevant, and consequential, and how detailed knowledge about food-related media exposure might influence adolescent’s daily food choices and later health (e.g., Blumberg et al., 2019; Borgogna et al., 2015; Folkvord & Van’t Riet, 2018; Vaterlaus, Patten, Roche, & Young, 2015).”

“One may zoom in or out across time scales to examine specific behaviors of finite length, cyclic or acyclic patterns of behavior, or long-term trends in behavior that match the times scale relevant to theoretical questions (Aigner, Miksch, Muller, Schumann, & Tominski, 2008).”

“Digitalization and representation of adolescents’ developmental tasks onto personal screens provides new opportunity for observing how those tasks manifest in daily life (Lundby, 2014; Subrahmanyam & Smahel, 2011).”

“Adolescence is known for its variability (e.g., storm and stress, see Hollenstein & Lougheed, 2013).”

“The resulting sequence of screenshots (and associated information) constitute the individual’s “ screenome ”—the structure of which can then be examined to inform knowledge of when and how individuals engage with their fast-moving and dynamic digital environment (Reeves et al., 2019).”

“For example, George, Russell, Piontak, and Odgers (2018) followed 151 adolescents for a month, asking them each evening to estimate the amount of time they spent using social media, the Internet, texting, and the number of text messages they sent that day.”

“…in particular, is a developmental period where children and parents are renegotiating their autonomy and monitoring of many behaviors, including screen time and social media sharing (e.g., how adolescents and parents attempt to regulate each others’ social media content; Shin & Kang, 2016).”

“Certainly, many of the same legal and ethical issues surrounding collection and use of individuals’ genomes and other -omes will apply here as well (Caulfield & McGuire, 2012).”

“…salient, relevant, and consequential, and how detailed knowledge about food-related media exposure might influence adolescent’s daily food choices and later health (e.g., Blumberg et al., 2019; Borgogna et al., 2015; Folkvord & Van’t Riet, 2018; Vaterlaus, Patten, Roche, & Young, 2015).”

“The velocity measure (formally, the first derivative of the image complexity time-series) indicates how quickly the content is changing (e.g., by analogy, how quickly a video shot pans across a scene) and is known to influence viewers’ motivations and choices (Wang, Lang, & Busemeyer, 2011).”

“…salient, relevant, and consequential, and how detailed knowledge about food-related media exposure might influence adolescent’s daily food choices and later health (e.g., Blumberg et al., 2019; Borgogna et al., 2015; Folkvord & Van’t Riet, 2018; Vaterlaus, Patten, Roche, & Young, 2015).”

“Following previous research (Böhmer, Hecht, Schöning, Krüger, & Bauer, 2011; Murnane et al., 2016), apps were further categorized by type using the developer-specified category associated with the app in the official Android, Google Play marketplace.”

“Some of the participants had participated in a prior study of weight gain prevention in Latino children, a population with high prevalence of obesity (Robinson et al., 2013).”

“…smart-phone devices, though, has not been documented, in part because people’s self-reports of use are inaccurate (correlations between reported and actual use, r s < .50, agreement between reported and actual use, κ s < .20; Boase & Ling, 2013; Gold, Rauscher, & Zhu, 2015; Mireku et al., 2018).”

“…salient, relevant, and consequential, and how detailed knowledge about food-related media exposure might influence adolescent’s daily food choices and later health (e.g., Blumberg et al., 2019; Borgogna et al., 2015; Folkvord & Van’t Riet, 2018; Vaterlaus, Patten, Roche, & Young, 2015).”

“With similar intent, Rich, Bickham, and Shrier (2015) developed a multimodal method for measuring youth media exposure.”

“…smart-phone devices, though, has not been documented, in part because people’s self-reports of use are inaccurate (correlations between reported and actual use, r s < .50, agreement between reported and actual use, κ s < .20; Boase & Ling, 2013; Gold, Rauscher, & Zhu, 2015; Mireku et al., 2018).”

“Text is extracted from each screenshot using Optical Character Recognition methods (Chiatti et al., 2017) and is quantified in a variety of ways that inform us about media behavior and content.”

“Important studies have been started, 70 , 71 and the Adolescent Brain Cognitive Devel - opment Study (ABCD study) should be mentioned.”

“…[90], future studies could understand the relationship between inter-individual heterogeneity in mental health trajectories and MP/WD messaging patterns and online usage, supported by new technologies such as screenomics, the machine-learning assisted categorisation of images and text [91, 92].”

“Although both the mobile sensing and the Screenomics approaches hold great promise to capture moment-to-moment involvement with social media, DDPs come with four additional advantages.”

“This, too, requires that we gather fine-grained and objective data, for instance, through the use of aforementioned APIs or screen recordings (Reeves et al., 2019; Ram et al., 2020).”

“In brief, Screenomics is made possible by a new data collection and analysis framework for capturing, storing, visualizing, and analyzing the full record of what individuals view and interact with on smartphone and laptop screens over time – the screenome (Reeves et al., 2019; Ram et al., 2020).”

“For example, language information can be directly collected via voice data from microphones (Mehl, 2017), keystrokes logged by custom keyboards (Buschek, Bisinger, & Alt, 2018), or extracted from screenshots of smartphone content (Ram et al., 2020; Reeves et al., 2019).”

“Furthermore, participants can install software on their computers or portable devices that make it possible to record chronological data, as done in the Screeno-mics Project (Ram et al., 2020).”

“This approach has already been utilized to examine different research questions such as task switching, how incidental exposure of news drives intentional information-seeking, within- and between-person differences in engaging with media content, and adolescents’ digital lives (Brinberg et al., 2021; Ram et al., 2020;…”

“Specifically, future research can capture on-platform behavior and content consumption on social media platforms using novel screenshot techniques [e.g., 54 ].”

“Recently, a new method has also evolved (ie, screenomics [103]) that captures the digital behavior of a person on their smartphone passively via multiple screenshots.”

“New technologies have emerged that allow researchers to objectively and passively collect smartphone data from participants (see Ram et al., 2020).”

“, [36,64,73]) demonstrate that variations in individual adolescents and the specific ways they interact with technology matter [35].”

“For example, a screenomics approach, which provides a fine-grained record of an individual’s digital experience by taking screenshots (Ram et al. , 2020), and geolocation tracking, which has been used to show a positive relationship of dopaminergic RSFC with positive affect and variation in…”

“The researchers adopted a screenomics analysis approach (Ram et al., 2020) to analyze the learning materials and students ’ work.”

“Notably, the associations we observed were stronger for network efficiency than purely content-based measures such as information diversity, suggesting a need for additional analyses of intensive longitudinal data of day- to-day digital experiences ( 113 , 114 ).”

“In an example of rich detailed data collection, Ram et al. [36] captured smartphone screen shots from 4 adolescents every 5 s the smartphone was activated over several months (some 500,000 images).”

“Following methods used in prior Screenomics research (Brinberg et al., 2021; Ram et al., 2020), we developed informative data visualizations for each timescale to: describe the digital interaction patterns that manifest speci fi c dimensions of connectedness and independence; illustrate and examine…”

“For example, researchers are exploring ways to achieve a more granular view of teens’ social media interactions in real-time using log data [71] and screen capture methods [73].”

“New tools to track people’s screen use on mobile devices rely on background apps that record smartphone use [10,11] or obtain intermittent automatic screenshots of the mobile device to record how it is being used over a specific period [12].”

“Along with information about people’s interactions with their phones, it can help researchers reconstruct both moment-tomoment phone use and wider usage patterns [58, 74, 76, 98].”

“New approaches, such as screenomics, will be valuable for sepa-rating the effects of content from the duration of exposures from different apps (Ram et al., 2020; Reeves et al., 2020).”

“Given the overall speedy rate at which individuals ’ screens go on and o ff (Brinberg et al., 2021; Ram et al., 2020), it was usual and normal to have multiple short sessions in the 10-minute periods surrounding ad exposures.”

“Current evidence suggests that self-reporting by adolescents on such complex screen use habits is unreliable, with many possibly unaware of their engagement in such behaviours [5, 29].”

“The current study employs Screenomics – intensive, real-time data captured from participants ’ smartphones, including text-based content across all platforms and applications that an individual uses – to explore a less intrusive, lower-burden method for assessing suicide risk.”

“One of the most recently available methods in performing digital phenotyping is screenomics [18,19], wherein participants passively provide temporally intensive records of their psychological and social life as it manifests on the screens of their digital devices [19].”

“— Cumulative entropy per hour : We followed prior research to compute the entropy [43, 49].”

“Similar increments in the precision of assessing media use have paid off in the study of other media effects; for example, exposure to political information [95], advertising [94], and food content in digital media [30].”

“Screenomes consist of sequential high-frequency screenshots taken every 5 seconds, capturing individuals’ day-to-day digital experiences and facilitating analysis of their engagement with the digital environment [82].”

“We employed a speci fi c form of digital phenotyping, screenomics 21 – 23 , where smartphone screens were automatically captured every fi ve seconds during active phone use.”

“70 Youth themselves remark on the need to address these complexities simultaneously.”

“27 Textual content was summarized using Linguistic Inquiry Word Count (LIWC), 37 the NRC-Canada Emotion Lexicon, 38 and the sentimentr package in R. 39–41 Visual content was summarized using image entropy and the number of faces appearing on the screen.”

“Thus, future research should aim to collect data on the contents people view on social media to better capture their experience [49,50].”

“The fine-grained temporal scale and the frequency of sampling possible through screenomics provides opportunities to examine social interactions through digital media by quantifying relationship dynamics at any number of time-scales (Ram et al., 2020).”

“…temporal features of the screenome, including perceptual features like color entropy, high-level features surrounding topics like food, politics, or shopping, and even more general embeddings that support captioning and verbal description of images (Brinberg et al., 2021; Ram et al., 2020, 2023).”

“Approaches that capture second-by-second images of individual users ’ exposures, such as the Screenomics approach [51] or that ask participants to use screen-sharing software to share their device screens with researchers during shorter periods [38,41], would likely provide more accurate estimates…”

“…while in use, offers a richer semantic representation by incorporating incoming and outgoing communication, consumed content (ie, what a person reads/watches/is exposed to), and the visual and structural context of that content [25], aligned with a more fine-grained semantic sensing approach.”

“Advances in screen data collection now enable continuous observation of everything that appears on users’ screens as they go about their daily lives (e.g., [52, 70]), offering unparalleled opportunities for understanding human behaviors on and beyond screens.”

“There are also assessment tools to track the screen use of a person on a mobile device used only by that person [13,15,16].”

“The resulting sequences of screenshots provide comprehensive records of device use that can be parsed, evaluated, and used to study all aspects of digital life (Ram et al 2020, Reeves et al 2020).”

“— Cumulative entropy per hour : We followed prior research to compute the entropy [53, 59, 85].”

“Perhaps most visibly, studies drawing on the ‘screenomics’ method (Ram et al. 2020; Reeves et al. 2021) have observed frequent and abundant swinging between different apps (e.g. Ram et al. 2020).”

“For example, Ram et al. (2020) took a screenshot of participants’ smart phones every 5 s for between 34 and 105 days, which corresponds to half a million to one and half million-time points.”

“For instance, Screenomics (Ram et al., 2019), data collected through the automatic capture of screenshots on participant’s mobile devices taken while in use, could provide more objective data about information acquisition in a digital space.”

“Adolescents in modern society are digital natives whose digital life has a substantial impact on most aspects of their lives, including education, peer relationships, and hobbies [2].”

“This precludes the measurement of behaviors including multitasking where more than 1 device is used simultaneously [26,27].”

“Many research questions remain unexplored simply because we lack information on what users are actually seeing and doing on their screens (Faust et al., 2024; Ohme et al., 2024; Ram et al., 2020; Reeves et al., 2021; Yee et al., 2023).”

“…studies have adopted similar intelligent monitoring technologies to investigate the specific content of social media used by adolescents, the application of such technologies in exploring the screen exposure of preschool children remains underdeveloped and requires further in-depth research [49].”

“Screenomics [118] provides fine-grained, content-level capture via frequent screenshots [119], but at the cost of substantial intrusiveness, privacy concerns, and limited scalability for long-term use [117,120, 121].”

“Recent tools also record digital content directly, including the Screenomics framework [19], which captures periodic screenshots to study engagement, and the screen text sensor [24], which logs on-screen text in real time to create detailed portraits of users’ digital lives.”

“However, emerging research leveraging longitudinal studies or intensive ecological momentary analyses of individuals’ real-time responses to media suggests that these relationships are often non-linear [59, 77].”

“Here, various memory extensions have been proposed, either by recording visited files’ content, as in Microsoft’s seminal ”Stuff I’ve Seen” system [11], or by capturing screenshots, recognizing text in them with OCR, and indexing this text to facilitate retrieval [9, 29].”

“Individuals may spend similar amounts of time using the same media applications while encountering very different content, leading to substantially different experiential exposures (Ram et al., 2020).”