Guidelines for wrist-worn consumer wearable assessment of heart rate in biobehavioral research

In the paper

“a second study found that the accuracy of HR measurement was highest during running51”

What the sources say

• Disputed

  The Methods (Experimental Procedure; page unavailable) state that heart rate was measured only at rest; during running, the study measured distance and energy consumption. The Results (Accuracy of the Same Indicators Under Different States of Activity; page unavailable) identify distance—not heart rate—as most accurate during running. (LLM, conf=0.90)

  Checked by a reasoning model — may be wrong.

  cites [51] Xie, J. et al. Evaluating the validity of current mainstream wearable devices in fitness tracking under various physical activities: comparative study. JMIR Mhealth Uhealth 6, e94 (2018). sourceExperimental — accuracy not yet measured

In the paper

“For example, researchers compared the accuracy of ambulatory ECG, Apple Watch 3, and Fitbit Charge 2 data within an individual across 24 h and found that while devices slightly underestimated HR when values were aggregated within and across different daily activities, which indicated high HR accuracy”

What the sources say

• Partly supported

  Split into 2 atoms — 1 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

  cites [38] Nelson, B. W. & Allen, N. B. Accuracy of consumer wearable heart rate measurement during an ecologically valid 24-hour period: intraindividual validation study. JMIR mHealth uHealth 7, e10828 (2019). sourceExperimental — accuracy not yet measured

In the paper

“Other research found that across four consumer wearables and two research-grade devices all were “reasonably accurate” during rest or during prolonged HR that was elevated and that differences in accuracy tended to arise during changes in activity50.”

What the sources say

• Supported

  The Results (Study summary) identify six devices: four consumer-grade and two research-grade. The Abstract states that the wearables were reasonably accurate at rest and at prolonged elevated heart rate, but differed in how they responded to changes in activity. (LLM, conf=0.95)

  cites [50] Bent, B., Goldstein, B. A., Kibbe, W. A., & Dunn, J. P. Investigating sources of inaccuracy in wearable optical heart rate sensors. npj Digital Med. 3, https://doi.org/10.1038/s41746-020-0226-6 (2020). source

In the paper

“Between 2008 and 2018 there was a 580% increase in total articles published that contained “wearable” and “heart rate”, indicating that this is a burgeoning area of research that may benefit from standardization of procedures to increase generalizability and clinical utility of research findings.”

In the paper

“Researchers have increasingly begun to use consumer wearables or wrist-worn smartwatches and fitness monitors for measurement of cardiovascular psychophysiological processes related to mental and physical health outcomes.”

In the paper

“These devices have strong appeal because they allow for continuous, scalable, unobtrusive, and ecologically valid data collection of cardiac activity in “big data” studies.”

In the paper

“HR is predominantly influenced by the coordination of the sympathetic and parasympathetic branches of the autonomic nervous system, which can be modified by biopsychosocial factors including physical and relational stress, diet, physical fitness, medication use, and substance use1; thus, offering a direct and quantifiable connection to the stress diathesis model of health.”

What the sources say

• Not checked yetcites [1] Cacioppo J. T., Tassinary L. G., & Berntson G. G. (eds) Handbook of Psychophysiology 4th edn. (Cambridge University Press, 2017). source

In the paper

“Second, collection of HR data in these environments lacks temporal resolution within and across days as HR is usually collected during a short recording within a single day, which has precluded research from comprehensively investigating the transition from acute to chronic stress22.”

What the sources say

• Not checked yetcites [22] Rohleder, N. Stress and inflammation—the need to address the gap in the transition between acute and chronic stress effects. Psychoneuroendocrinology https://doi.org/10.1016/j.psyneuen.2019.02.021 (2019). source

In the paper

“(2) inability to collect temporally detailed and longitudinal HR measurements both within and across time (e.g., hours, days, weeks, months)”

In the paper

“Researchers are increasingly using consumer wearables or wrist-worn smartwatches and fitness monitors for the digital measurement of cardiovascular health and psychophysiological stress as these devices allow for continuous, scalable, and unobtrusive, “big data” collection of overall cardiac activity in real-world conditions with large samples25–30.”

What the sources say

• Not checked yetcites [25] Bradshaw, B. et al. Influenza surveillance using wearable mobile health devices. In International Society for Disease Surveillance Conference 2019 (2019).
• Not checked yetcites [26] Natarajan, A., Pantelopoulos, A., Emir-Farinas, H. & Natarajan, P. Heart rate variability with photoplethysmography in 8 million individuals: results and scaling relations with age, gender, and time of day. Preprint at https://www.biorxiv.org/content/10.1101/772285v1 (2019). source
• Not checked yetcites [27] Perez, M. V. et al. Large-scale assessment of a smartwatch to identify atrial fibrillation. N. Engl. J. Med. 381, 1909–1917 (2019). source
• Not checked yetcites [28] Sim, I. Mobile devices and health. N. Engl. J. Med. 381, 956–968 (2019). source
• Not checked yetcites [29] Tison, G. H. et al. Passive detection of atrial fibrillation using a commercially available smartwatch. JAMA Cardiol. 3, 409 (2018). source
• Not checked yetcites [30] Tison, G. H. et al. Cardiovascular Risk Stratification Using Off-the-Shelf Wearables and a Multi-Task Deep Learning Algorithm. In:; 2018. source

In the paper

“The latter of these two is the predominant method for data collected with wrist-worn wearable devices, although some more recent wrist-worn wearable devices, such as the Apple Watch Series 4 and 5, have allowed for the collection of HR via electrical impulses on par with a single-lead electrode.”

In the paper

“In other words, research has been historically limited to collecting data from medical patients and research participants in novel settings that might not generalize to their everyday lives, which creates the potential for inaccurate baselines and therefore inaccurate measurement of HR dynamics (i.e., reactivity/recovery).”

In the paper

“(3) prohibitive financial cost of devices for everyday use.”

In the paper

“Furthermore, PPG can be used to index heart rate variability (HRV), blood pressure, oxygen saturation, and cardiac output24, but as of yet have not be widely rolled out in consumer-use devices and therefore data collection other than HR will not be discussed further.”

What the sources say

• Not checked yetcites [24] Elgendi, M. On the analysis of fingertip photoplethysmogram signals. Curr. Cardiol. Rev. 8, 14–25 (2012). source

In the paper

“Until recently, the quantification and measurement of HR has been relegated to medical and research settings, which has largely precluded the observation of HR during patient and participant daily life, thereby limiting the generalizability of findings to real-world living conditions.”

In the paper

“In contrast, red LEDs, which are more commonly used in medical settings, are not absorbed as well by the skin allowing the light to transmit deeper and allowing for detection of multiple biological measures, but it has a lower signal-to-noise ratio and is more susceptible to motion artifacts23.”

What the sources say

• Not checked yetcites [23] Mahloko, L. & Adebesin, F. in Responsible Design, Implementation and Use of Information and Communication Technology. Lecture Notes in Computer Science (eds Hattingh, M., et al.) Vol. 12067, 96–107 (Springer International Publishing, Cham, 2020). source

In the paper

“For example, green LED light is a shorter wave-length and has been shown to be convenient, because it has good signal-to-noise ratio and is resistant to motion artifacts, but is limited by the amount of light that can pass through tissue, especially darker skin tones.”

In the paper

“Therefore, this novel electrocardiogram (ECG) technology only allows for the discrete, rather than continuous, collection of data, which precludes the ability for temporally detailed longitudinal ECG collection at this time.”

In the paper

“In other words, HR may be a transdiagnostic biomarker with clinical utility for psychology, psychiatry, and medicine to improve nosology and increase identification of those at risk for the onset of psychiatric and physical health disorders.”

In the paper

“The foundation of HR research has, until recently, been largely constrained due to (1) lack of ecological validity (i.e., the degree to which a laboratory or medical environment represents actual real-world conditions)”

In the paper

“Currently, wrist-worn wearables provide an opportunity to examine HR in real-world environments, across longer periods of time, and with high resolution at a low cost, by using an optical sensor (photoplethysmography; PPG), that allows these devices to collect volumetric changes in blood profusion (i.e., pulse rate) that can serve as a surrogate for HR”

In the paper

“Recent information on the reproducibility and replicability crisis has swept the sciences and as such these issues are very important for digital health and mHealth research, which includes wrist-worn HR accuracy and any studies that utilize wearable HR data, as this new field will allow for large data samples31 with various sensors that will increase the ease for p-hacking.”

What the sources say

• Not checked yetcites [31] Stupple, A., Singerman, D., & Celi, L. A. The reproducibility crisis in the age of digital medicine. npjDigital Med. 2, https://doi.org/10.1038/s41746-019-0079-z (2019). source

In the paper

“yet questions remain as to the accuracy and validity of data collected from wearables35,36.”

What the sources say

• Not checked yetcites [35] Evenson, K. R., Goto, M. M., & Furberg, R. D. Systematic review of the validity and reliability of consumer-wearable activity trackers. Int. J. Behav. Nutr. Phys. Activity. 12, https://doi.org/10.1186/s12966-015-0314-1 (2015). source
• Not checked yetcites [36] Peake, J. M., Kerr, G., & Sullivan, J. P. A critical review of consumer wearables, mobile applications, and equipment for providing biofeedback, monitoring stress, and sleep in physically active populations. Front. Physiol. 9, https://doi.org/10.3389/fphys.2018.00743 (2018). source

In the paper

“Although some studies have attempted to address wearable HR accuracy using the gold-standard reference method of ECG, which directly measures HR via electrical impulses from the heart and is used for clinical accuracy38–44”

What the sources say

• Not checked yetcites [38] Nelson, B. W. & Allen, N. B. Accuracy of consumer wearable heart rate measurement during an ecologically valid 24-hour period: intraindividual validation study. JMIR mHealth uHealth 7, e10828 (2019). source
• Not checked yetcites [39] Boudreaux, B. D. et al. Validity of wearable activity monitors during cycling and resistance exercise. Med. Sci. Sports Exerc. 50, 624–633 (2018). source
• Not checked yetcites [40] de Zambotti, M. et al. Measures of sleep and cardiac functioning during sleep using a multi-sensory commercially-available wristband in adolescents. Physiol. Behav. 158, 143–149 (2016). source
• Not checked yetcites [41] Kroll, R. R., Boyd, J. G. & Maslove, D. M. Accuracy of a wrist-worn wearable device for monitoring heart rates in hospital inpatients: a prospective observational study. J. Med. Internet Res. 18, e253 (2016). source
• Not checked yetcites [42] Shcherbina, A. et al. Accuracy in wrist-worn, sensor-based measurements of heart rate and energy expenditure in a diverse cohort. J. Personalized Med. 7, 3 (2017). source
• Not checked yetcites [43] Wallen, M. P., Gomersall, S. R., Keating, S. E., Wisløff, U., & Coombes, J. S. Accuracy of heart rate watches: implications for weight management. PLoS ONE 11, https://doi.org/10.1371/journal.pone.0154420 (2016). source
• Not checked yetcites [44] Wang, R. et al. Accuracy of wrist-worn heart rate monitors. JAMA Cardiol. 2, 104–106 (2017). source

In the paper

“This finding indicated that while overall, summary statistics after outlier detection and data cleaning may be very accurate for research purposes, any single observation in real-time may have a large degree of error, which could be significant for moment-to-moment observations in medical settings.”

In the paper

“Furthermore, some wearable devices, such as the Apple Watch 4 and 5, have gained FDA 510(k) class II clearance for the ECG feature and ability to detect arrhythmias49”

What the sources say

• Not checked yetcites [49] FDA. Electrocardiographsoftware for Over-the-Counter Use. https://www.accessdata.fda.gov/cdrh_docs/pdf18/DEN180044.pdf (FDA, 2018).

In the paper

“Furthermore, recent research has indicated that consumer wearables, such as the Apple Watch and Fitbit are more accurate than research-grade wearables, which are much more expensive yet provide the benefit of raw data50.”

What the sources say

• Not checked yetcites [50] Bent, B., Goldstein, B. A., Kibbe, W. A., & Dunn, J. P. Investigating sources of inaccuracy in wearable optical heart rate sensors. npj Digital Med. 3, https://doi.org/10.1038/s41746-020-0226-6 (2020). source

In the paper

“Therefore, many researchers have begun to push for increased focus on reproducible and open practices in science31,53–55.”

What the sources say

• Not checked yetcites [31] Stupple, A., Singerman, D., & Celi, L. A. The reproducibility crisis in the age of digital medicine. npjDigital Med. 2, https://doi.org/10.1038/s41746-019-0079-z (2019). source
• Not checked yetcites [53] Bavel, J. J. V., Mende-siedlecki, P., Brady, W. J. & Reinero, D. A. Contextual sensitivity in scientific reproducibility. Proc. Natl Acad. Sci. USA 113, 6454–6459 (2016). source
• Not checked yetcites [54] Munafò, M. R. et al. A manifesto for reproducible science. Nat. Hum. Behav. 1, https://doi.org/10.1038/s41562-016-0021 (2017). source
• Not checked yetcites [55] Tackett, J. L., Brandes, C. M., King, K. M. & Markon, K. E. Psychology’s replication crisis and clinical psychological science. Annu. Rev. Clin. Psychol. 15, 579–604 (2019). source

In the paper

“These findings have led some researchers to conclude that wearables detect HR with acceptable accuracy in both laboratory and real-world settings for research, but importantly not medical settings38,42 as more research needs to be done.”

What the sources say

• Not checked yetcites [38] Nelson, B. W. & Allen, N. B. Accuracy of consumer wearable heart rate measurement during an ecologically valid 24-hour period: intraindividual validation study. JMIR mHealth uHealth 7, e10828 (2019). source
• Not checked yetcites [42] Shcherbina, A. et al. Accuracy in wrist-worn, sensor-based measurements of heart rate and energy expenditure in a diverse cohort. J. Personalized Med. 7, 3 (2017). source

In the paper

“lower mean difference standard deviation43, and higher agreement with ECG than Fitbit devices39,44.”

What the sources say

• Not checked yetcites [39] Boudreaux, B. D. et al. Validity of wearable activity monitors during cycling and resistance exercise. Med. Sci. Sports Exerc. 50, 624–633 (2018). source
• Not checked yetcites [43] Wallen, M. P., Gomersall, S. R., Keating, S. E., Wisløff, U., & Coombes, J. S. Accuracy of heart rate watches: implications for weight management. PLoS ONE 11, https://doi.org/10.1371/journal.pone.0154420 (2016). source
• Not checked yetcites [44] Wang, R. et al. Accuracy of wrist-worn heart rate monitors. JAMA Cardiol. 2, 104–106 (2017). source

In the paper

“Hardware differences between devices determine HR measurement second to second, which has large implications on overall interpretability of the data, such that sampling rate can introduces researcher degrees of freedom for calculating bpm.”

In the paper

“For long-term field studies, information about participant adherence and wear-time should be considered and reported since these issues can affect data quality.”

In the paper

“For example, outliers in linear models may strongly impact linear estimators, and some studies may therefore adopt a modeling strategy wherein values are winsorized (e.g., extreme values can be truncated at the 2.5th and 97.5th quantiles56).”

What the sources say

• Not checked yetcites [56] Hastings, C. Jr., Mosteller, F., Tukey, J. W. & Winsor, C. P. Low moments for small samples: a comparative study of order statistics. Ann. Math. Stat. 18, 413–426 (1947). source

In the paper

“For example, Fitbit logs HR data at 1 s intervals during exercise tracking and 5 s intervals at other times, whereas the Apple Watch samples HR data every 10 min, except during workouts, which are continuous.”

In the paper

“There is a strong need to customize the methods of data cleaning to the device being used and the models being adopted, rather than adopting a universal pre-processing pipeline due to device-specific differences and based on the study research questions.”

In the paper

“There are many researcher degrees of freedom when it comes to decisions related to study protocol design and data collection, cleaning, and processing that are important to note as standardizing this process may help ameliorate some barriers and inconsistencies in the literature.”

In the paper

“The practice of completely removing persons from studies due to proportions of missing data (i.e. listwise deletion) is known to bias estimation and adversely impact the standard error of point estimates across modeling strategies58.”

What the sources say

• Not checked yetcites [58] Graham, J. W. Missing data analysis: making it work in the real world. Annu. Rev. Psychol. 60, 549–576 (2009). source

In the paper

“However, if a study were interested in HR arrythmias using wearable assessments (such as atrial fibrillation), then these extreme values may be of vital interest and should not be removed or altered27.”

What the sources say

• Not checked yetcites [27] Perez, M. V. et al. Large-scale assessment of a smartwatch to identify atrial fibrillation. N. Engl. J. Med. 381, 1909–1917 (2019). source

In the paper

“In addition, the methods researchers select to clean and process raw data prior to analyses can also affect results and conclusions and should be reported in detail.”

In the paper

“Larger wrist circumference has been shown to be associated with reduced HR accuracy42, although these outcomes have not been found in all studies44,46.”

What the sources say

• Not checked yetcites [42] Shcherbina, A. et al. Accuracy in wrist-worn, sensor-based measurements of heart rate and energy expenditure in a diverse cohort. J. Personalized Med. 7, 3 (2017). source
• Not checked yetcites [44] Wang, R. et al. Accuracy of wrist-worn heart rate monitors. JAMA Cardiol. 2, 104–106 (2017). source
• Not checked yetcites [46] Gillinov, S. et al. Variable accuracy of wearable heart rate monitors during aerobic exercise. Med. Sci. Sports Exerc. 49, 1697–1703 (2017). source

In the paper

“Moreover, for some, but not all devices will report missing HR data with zero values and these values should be removed and filled in with missing data rather than winsorized (which would otherwise substantially bias parameter estimates unless the degree of missing HR points was trivial).”

In the paper

“One issue reported by multiple users on the online Fitbit Community Forum is occasional logging of HR values (often higher and more variable than typical, e.g., 100–150 bpm) when the devices are not being worn, suggesting that supplemental non-wear-time diaries may be required for some devices or protocols.”

In the paper

“Relatedly, some devices, such as the Apple Watch can take multiple HR measurements within a minute allowing researchers to use various calculations, such as the mean or median within this minute to calculate bpm, which may alter wearable data values.”

In the paper

“some research showing that absolute error during activity is 30% higher than rest50”

What the sources say

• Not checked yetcites [50] Bent, B., Goldstein, B. A., Kibbe, W. A., & Dunn, J. P. Investigating sources of inaccuracy in wearable optical heart rate sensors. npj Digital Med. 3, https://doi.org/10.1038/s41746-020-0226-6 (2020). source

In the paper

“the last of which likely will not be an issue for mHealth research as large samples sizes will be relatively easy to collect even in samples with low base rates in the population.”

In the paper

“Currently, the vast majority of wrist-worn wearable studies incorporating HR measurements do not report or control for participant race and ethnicity, which undermines the ability to look at potentially important group differences in health outcomes and therefore decreases the ability of results to generalize to large segments of the population.”

In the paper

“Evidence is starting to accumulate on the importance of skin tone when it comes to accuracy and generalizability of wrist-worn wearable studies with darker skin tone (and tattoos) being found to be associated with reduced wrist-worn HR accuracy as they absorb more green light67, yet this is not found in all studies43,50.”

What the sources say

• Not checked yetcites [67] Weiler, D. T., Villajuan, S. O., Edkins, L., Cleary, S. & Saleem, J. J. Wearable heart rate monitor technology accuracy in research: a comparative study between PPG and ECG technology. Proc. Hum. Factors Ergonomics Soc. Annu. Meet. 61, 1292–1296 (2017). source
• Not checked yetcites [43] Wallen, M. P., Gomersall, S. R., Keating, S. E., Wisløff, U., & Coombes, J. S. Accuracy of heart rate watches: implications for weight management. PLoS ONE 11, https://doi.org/10.1371/journal.pone.0154420 (2016). source
• Not checked yetcites [50] Bent, B., Goldstein, B. A., Kibbe, W. A., & Dunn, J. P. Investigating sources of inaccuracy in wearable optical heart rate sensors. npj Digital Med. 3, https://doi.org/10.1038/s41746-020-0226-6 (2020). source

In the paper

“wearables that use red light will likely be more effective on darker skin tones as it is not absorbed by melanin, which may allow for more accurate HR measurements23.”

What the sources say

• Not checked yetcites [23] Mahloko, L. & Adebesin, F. in Responsible Design, Implementation and Use of Information and Communication Technology. Lecture Notes in Computer Science (eds Hattingh, M., et al.) Vol. 12067, 96–107 (Springer International Publishing, Cham, 2020). source

In the paper

“Higher body mass index, which likely covaries with larger wrist circumference as described above, has been shown to be associated with reduced HR accuracy42,62, although again these outcomes have not been found in all studies44,46.”

What the sources say

• Not checked yetcites [42] Shcherbina, A. et al. Accuracy in wrist-worn, sensor-based measurements of heart rate and energy expenditure in a diverse cohort. J. Personalized Med. 7, 3 (2017). source
• Not checked yetcites [62] Menghini, L. et al. Stressing the accuracy: wrist-worn wearable sensor validation over different conditions. Psychophysiology. 56, https://doi.org/10.1111/psyp.13441 (2019). source
• Not checked yetcites [44] Wang, R. et al. Accuracy of wrist-worn heart rate monitors. JAMA Cardiol. 2, 104–106 (2017). source
• Not checked yetcites [46] Gillinov, S. et al. Variable accuracy of wearable heart rate monitors during aerobic exercise. Med. Sci. Sports Exerc. 49, 1697–1703 (2017). source

In the paper

“some research indicates that hair follicle density and sweat can influence device measurement69.”

What the sources say

• Not checked yetcites [69] Cosoli, G., Spinsante, S. & Scalise, L. Wrist-worn and chest-strap wearable devices: systematic review on accuracy and metrological characteristics. Measurement 159, 107789 (2020). source

In the paper

“The importance of this point cannot be overstated as historically, science and medicine have paid less attention to recruiting minority groups68, which has resulted in blunted health gains in these populations”

What the sources say

• Not checked yetcites [68] Hall, G. C. N., Yip, T. & Zárate, M. A. On becoming multicultural in a monocultural research world: a conceptual approach to studying ethnocultural diversity. Am. Psychol. 71, 40–51 (2016). source

In the paper

“Furthermore, wearables that use red light will likely be more effective on darker skin tones as it is not absorbed by melanin, which may allow for more”

In the paper

“these devices can be used to supplement medical-grade devices and continuously collect HR data on research subjects when risk for acute cardiac events is not of immediate concern.”

In the paper

“This has the potential to improve the study of cardiovascular health in medicine and psychophysiological stress in psychiatry and psychology.”

In the paper

“This has the potential to advance research into cardiovascular health and psychophysiological stress in order to better understand how HR influences overall physical and psychological health.”

In the paper

“one strength of PPG green light is that it is less susceptible to motion artifacts, whereas PPG red light is more susceptible to motion artifacts”

What the sources say

• Not checked yetcites [23] Mahloko, L. & Adebesin, F. in Responsible Design, Implementation and Use of Information and Communication Technology. Lecture Notes in Computer Science (eds Hattingh, M., et al.) Vol. 12067, 96–107 (Springer International Publishing, Cham, 2020). source

In the paper

“although some researchers have worked with industry partners to derive measurements of HRV26.”

What the sources say

• Not checked yetcites [26] Natarajan, A., Pantelopoulos, A., Emir-Farinas, H. & Natarajan, P. Heart rate variability with photoplethysmography in 8 million individuals: results and scaling relations with age, gender, and time of day. Preprint at https://www.biorxiv.org/content/10.1101/772285v1 (2019). source

In the paper

“Therefore, it is possible that activity intensity may be less important to device accuracy than the degree of erratic wrist movements and corresponding position of the wearable device during an activity74, which cause motion artifacts and may or may not co-occur with”

What the sources say

• Not checked yetcites [74] Parak, J. & Korhonen, I. Evaluation of wearable consumer heart rate monitors based on photopletysmography. Conf. Proc. IEEE Eng. Med. Biol. Soc. 3670-3673 2014, 3670–3673 (2014).

In the paper

“some studies finding that accuracy of HR measurement was comparable across resting baseline or vigorous activity45”

What the sources say

• Not checked yetcites [45] Dooley, E. E., Golaszewski, N. M. & Bartholomew, J. B. Estimating accuracy at exercise intensities: a comparative study of self-monitoring heart rate and physical activity wearable devices. JMIR mHealth uHealth 5, e34 (2017). source

In the paper

“Utilizing these guidelines and referencing them in methods sections has the potential to decrease non-standard reporting, increase reproducibility and replicability, while also leading to greater generalizability of results, which may increase the clinical utility of findings in this field.”