CoChat
Sign Up
Cited by 114

Keeping Pace with Wearables: A Living Umbrella Review of Systematic Reviews Evaluating the Accuracy of Consumer Wearable Technologies in Health Measurement

Doherty, Cailbhe, Baldwin, Maximus, Keogh, Alison, Caulfield, Brian, Argent, Rob·2024·Sports Medicine
DOI 10.1007/s40279-024-02077-2PMC PMC11560992
Open this paper in CoChat — free account

Abstract

Background: Consumer wearable technologies have become ubiquitous, with clinical and non-clinical populations leveraging a variety of devices to quantify various aspects of health and wellness. However, the accuracy with which these devices measure biometric outcomes such as heart rate, sleep and physical activity remains unclear. Objective: To conduct a ‘living’ (i.e. ongoing) evaluation of the accuracy of consumer wearable technologies in measuring various physiological outcomes. Methods: A systematic search of the literature was conducted in the following scientific databases: MEDLINE via PubMed, Embase, Cinahl and SPORTDiscus via EBSCO. The inclusion criteria required systematic reviews or meta-analyses that evaluated the validation of consumer wearable devices against accepted reference standards. In addition to publication details, review protocol, device specifics and a summary of the authors’ results, we extracted data on mean absolute percentage error (MAPE), pooled absolute bias, intraclass correlation coefficients (ICCs) and mean absolute differences. Results: Of 904 identified studies through the initial search, 24 systematic reviews met our inclusion criteria; these systematic reviews included 249 non-duplicate validation studies of consumer wearable devices involving 430,465 participants (43% female). Of the commercially available wearable devices released to date, approximately 11% have been validated for at least one biometric outcome. However, because a typical device can measure a multitude of biometric outcomes, the number of validation studies conducted represents just 3.5% of the total needed for a comprehensive evaluation of these devices. For heart rate, wearables showed a mean bias of ± 3%. In arrhythmia detection, wearables exhibited a pooled sensitivity and specificity of 100% and 95%, respectively. For aerobic capacity, wearables significantly overestimated VO 2max by ± 15.24% during resting tests and ± 9.83% during exercise tests. Physical activity intensity measurements had a mean absolute error ranging from 29 to 80%, depending on the intensity of the activity being undertaken. Wearables mostly underestimated step counts (mean absolute percentage errors ranging from − 9 to 12%) and energy expenditure (mean bias = − 3 kcal per minute, or − 3%, with error ranging from − 21.27 to 14.76%). For blood oxygen saturation, wearables showed a mean absolute difference of up to 2.0%. Sleep measurement showed a tendency to overestimate total sleep time (mean absolute percentage error typically > 10%). Conclusions: While consumer wearables show promise in health monitoring, a conclusive assessment of their accuracy is impeded by pervasive heterogeneity in research outcomes and methodologies. There is a need for standardised validation protocols and collaborative industry partnerships to enhance the reliability and practical applicability of wearable technology assessments. Prospero ID: CRD42023402703.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1007/s40279-024-02077-2

Cites

12 of 12 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Structural factors such as vendor lock-in and 'choice legacy' reduce users' ability to manage or transfer their data.

    “Structural factors such as vendor lock-in and 'choice legacy'-where users become dependent on proprietary software ecosystems-further reduce users' ability to manage or transfer their data 4,33 .”

    From Privacy in consumer wearable technologies: a living systematic analysis of data policies across leading manufacturers (2025) · cites this paper as [4]

    The Conclusion calls for data privacy and user autonomy, but does not describe vendor lock-in, ‘choice legacy,’ or limits on users’ ability to manage or transfer data. Searches for those terms and for data portability and interoperability found no passage addressing the claimed mechanism; no page numbers were provided in our copy. (LLM, conf=0.75)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 46

  1. [1]Lupton D. The quantified self: a sociology of self-tracking. Sociol Health Illn. 2016;39:1557–71. 29071731 · DOI
  2. [2]Piwek L, Ellis DA, Andrews S, Joinson A. The rise of consumer health wearables: promises and barriers. PLoS Med. 2016;13(2): e1001953. 26836780 PMC4737495 · DOI
  3. [5]Carpenter A, Frontera A. Smart-watches: a potential challenger to the implantable loop recorder? Europace. 2016;18(6):791–3. 26847074 · DOI · Paper page
  4. [6]Jia Y, Wang W, Wen D, Liang L, Gao L, Lei J. Perceived user preferences and usability evaluation of mainstream wearable devices for health monitoring. PeerJ. 2018;6: e5350. 30065893 PMC6064199 · DOI · Paper page
  5. [8]Radin JM, Wineinger NE, Topol EJ, Steinhubl SR. Harnessing wearable device data to improve state-level real-time surveillance of influenza-like illness in the USA: a population-based study. Lancet Digit Health. 2020;2(2):e85–93. 33334565 PMC8048388 · DOI · Paper page
  6. [10]Kimura N, Aso Y, Yabuuchi K, Ishibashi M, Hori D, Sasaki Y, et al. Association of modifiable lifestyle factors with cortical amyloid burden and cerebral glucose metabolism in older adults with mild cognitive impairment. JAMA Netw Open. 2020;3(6): e205719. 32515796 PMC7284299 · DOI
  7. [16]Keogh A, Argent R, Doherty C, Duignan C, Fennelly O, Purcell C, et al. Breaking down the digital fortress: the unseen challenges in healthcare technology—lessons learned from 10 years of research. Sensors. 2024;24(12):3780. 38931564 PMC11207951 · DOI · Paper page
  8. [17]Huhn S, Axt M, Gunga H-C, Maggioni MA, Munga S, Obor D, et al. The impact of wearable technologies in health research: scoping review. JMIR Mhealth Uhealth. 2022;10(1): e34384. 35076409 PMC8826148 · DOI · Paper page
  9. [19]Gualtieri L, Rosenbluth S, Phillips J. Can a free wearable activity tracker change behavior? The impact of trackers on adults in a physician-led wellness group. JMIR Res Protoc. 2016;5(4): e237. 27903490 PMC5156822 · DOI
  10. [20]Montoye AHK, Mitrzyk JR, Molesky MJ. Comparative accuracy of a wrist-worn activity tracker and a smart shirt for physical activity assessment. Meas Phys Educ Exerc Sci. 2017;21(4):201–11. · DOI
  11. [21]Evenson KR, Goto MM, Furberg RD. Systematic review of the validity and reliability of consumer-wearable activity trackers. Int J Behav Nutr Phys Activity. 2015;12(1):159. PMC4683756 26684758 · DOI · Paper page
  12. [24]Ash GI, Stults-Kolehmainen M, Busa MA, Gaffey AE, Angeloudis K, Muniz-Pardos B, et al. Establishing a global standard for wearable devices in sport and exercise medicine: perspectives from academic and industry stakeholders. Sports Med. 2021;51(11):2237–50. 34468950 PMC8666971 · DOI · Paper page
  13. [25]Argent R, Hetherington-Rauth M, Stang J, Tarp J, Ortega FB, Molina-Garcia P, et al. Recommendations for determining the validity of consumer wearables and smartphones for the estimation of energy expenditure: expert statement and checklist of the INTERLIVE network. Sports Med. 2022;52(8):1817–32. 35260991 PMC9325806 · DOI
  14. [26]Johnston W, Judice PB, Molina García P, Mühlen JM, Lykke Skovgaard E, Stang J, et al. Recommendations for determining the validity of consumer wearable and smartphone step count: expert statement and checklist of the INTERLIVE network. Br J Sports Med. 2021;55(14):780–93. 33361276 PMC8273687 · DOI
  15. [27]Mühlen JM, Stang J, Lykke Skovgaard E, Judice PB, Molina-Garcia P, Johnston W, et al. Recommendations for determining the validity of consumer wearable heart rate devices: expert statement and checklist of the INTERLIVE Network. Br J Sports Med. 2021;55(14):767–79. 33397674 PMC8273688 · DOI
  16. [28]Düking P, Stammel C, Sperlich B, Sutehall S, Muniz-Pardos B, Lima G, et al. Necessary steps to accelerate the integration of wearable sensors into recreation and competitive sports. Curr Sports Med Rep. 2018;17(6):178–82. 29889145 · DOI
  17. [29]Simmonds M, Elliott JH, Synnot A, Turner T. Living systematic reviews. Methods Mol Biol. 2022;2345:121–34. 34550587 · DOI
  18. [30]Cochrane. Living systematic reviews. 2023 [cited; https://community.cochrane.org/review-development/resources/living-systematic-reviews#:~:text=Living%20Evidence%20Network-,What%20is%20a%20living%20systematic%20review%3F,evidence%20as%20it%20becomes%20available.
  19. [31]Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Br Med J (Online). 2009;339(7716):332–6. PMC3090117 21603045
  20. [32]Drucker AM, Fleming P, Chan AW. Research techniques made simple: assessing risk of bias in systematic reviews. J Invest Dermatol. 2016;136(11):e109–14. 27772550 · DOI
  21. [33]Belani S, Wahood W, Hardigan P, Placzek AN, Ely S. Accuracy of detecting atrial fibrillation: a systematic review and meta-analysis of wrist-worn wearable technology. Cureus. 2021;13(12): e20362. 35036196 PMC8752409
  22. [34]Board EM, Ispoglou T, Ingle L. Validity of telemetric-derived measures of heart rate variability: a systematic review. J Exerc Physiol Online. 2016;19(6):64–84.
  23. [35]Byrne J, Lynch S, Shipp A, Tran B, Mohan S, Reindel K. Investigating the accuracy of wheelchair push counts measured by fitness watches: a systematic review. Cureus. 2023;15(9): e45322. 37849605 PMC10577091
  24. [36]Chevance G, Golaszewski NM, Tipton E, Hekler EB, Buman M, Welk GJ, et al. Accuracy and precision of energy expenditure, heart rate, and steps measured by combined-sensing fitbits against reference measures: systematic review and meta-analysis. JMIR Mhealth Uhealth. 2022;10(4): e35626. 35416777 PMC9047731 · DOI
  25. [37]Evenson KR, Goto MM, Furberg RD. Systematic review of the validity and reliability of consumer-wearable activity trackers. Int J Behav Nutr Phys Act. 2015;12:1–22. 26684758 PMC4683756 · DOI · Paper page
  26. [39]Georgiou K, Larentzakis AV, Khamis NN, Alsuhaibani GI, Alaska YA, Giallafos EJ. Can wearable devices accurately measure heart rate variability? A systematic review. Folia Med. 2018;60(1):7–20. 29668452
  27. [40]Germini F, Noronha N, Debono VB, Philip BA, Pete D, Navarro T, et al. Accuracy and acceptability of wrist-wearable activity-tracking devices: systematic review of the literature. J Med Internet Res. 2022;24(1): e30791. 35060915 PMC8817215 · DOI
  28. [41]Giebel GD, Gissel C. Accuracy of mHealth devices for atrial fibrillation screening: systematic review. JMIR Mhealth Uhealth. 2019;7(6): e13641. 31199337 PMC6598422 · DOI · Paper page
  29. [42]Haghayegh S, Khoshnevis S, Smolensky MH, Diller KR, Castriotta RJ. Accuracy of wristband fitbit models in assessing sleep: systematic review and meta-analysis. J Med Internet Res. 2019;21(11): e16273. 31778122 PMC6908975 · DOI · Paper page
  30. [43]Henriksen A, Johansson J, Hartvigsen G, Grimsgaard S, Hopstock L. Measuring physical activity using triaxial wrist worn polar activity trackers: a systematic review. Int J Exerc Sci. 2020;13(4):438–54. 32509122 PMC7241625
  31. [44]Hermans ANL, Gawalko M, Dohmen L, van der Velden RMJ, Betz K, Duncker D, et al. Mobile health solutions for atrial fibrillation detection and management: a systematic review. Clin Res Cardiol. 2022;111(5):479–91. 34549333 PMC8454991 · DOI
  32. [45]Irwin C, Gary R. Systematic review of Fitbit Charge 2 validation studies for exercise tracking. Transl J Am Coll Sports Med. 2022;7(4):1–7. 36711436 PMC9881599
  33. [46]Kenyon A, McEvoy M, Sprod J, Maher C. Validity of pedometers in people with physical disabilities: a systematic review. Arch Phys Med Rehabil. 2013;94(6):1161–70. 23201318 · DOI
  34. [47]Koerber D, Khan S, Shamsheri T, Kirubarajan A, Mehta S. Accuracy of heart rate measurement with wrist-worn wearable devices in various skin tones: a systematic review. J Racial Ethnic Health Disparities. 2022;10:2676–84. PMC9662769 36376641 · DOI · Paper page
  35. [48]Leung W, Case L, Sung MC, Jung J. A meta-analysis of Fitbit devices: same company, different models, different validity evidence. J Med Eng Technol. 2022;46(2):102–15. 34881682 · DOI
  36. [49]Lopez Perales CR, Van Spall HGC, Maeda S, Jimenez A, Laţcu DG, Milman A, et al. Mobile health applications for the detection of atrial fibrillation: a systematic review. Europace. 2021;23(1):11–28. 33043358 PMC7842109 · DOI · Paper page
  37. [50]Molina-Garcia P, Notbohm HL, Schumann M, Argent R, Hetherington-Rauth M, Stang J, et al. Validity of estimating the maximal oxygen consumption by consumer wearables: a systematic review with meta-analysis and expert statement of the INTERLIVE network. Sports Med. 2022;52(7):1577–97. 35072942 PMC9213394 · DOI
  38. [51]Nazarian S, Lam K, Darzi A, Ashrafian H. Diagnostic accuracy of smartwatches for the detection of cardiac arrhythmia: systematic review and meta-analysis. J Med Internet Res. 2021;23(8): e28974. 34448706 PMC8433941 · DOI
  39. [52]O’Driscoll R, Turicchi J, Beaulieu K, Scott S, Matu J, Deighton K, et al. How well do activity monitors estimate energy expenditure? A systematic review and meta-analysis of the validity of current technologies. Br J Sports Med. 2020;54(6):332–40. 30194221 · DOI
  40. [53]Schyvens AM, Van Oost NC, Aerts JM, Masci F, Peters B, Neven A, et al. Accuracy of Fitbit Charge 4, Garmin Vivosmart 4, and WHOOP versus polysomnography: systematic review. JMIR Mhealth Uhealth. 2024;27(12): e52192. PMC11004611 38557808 · DOI
  41. [54]Windisch P, Schröder C, Förster R, Cihoric N, Zwahlen DR. Accuracy of the Apple Watch oxygen saturation measurement in adults: a systematic review. Cureus. 2023;15(2): e35355. 36974257 PMC10039641
  42. [56]Kokkinos P, Faselis C, Samuel IBH, Pittaras A, Doumas M, Murphy R, et al. Cardiorespiratory fitness and mortality risk across the spectra of age, race, and sex. J Am Coll Cardiol. 2022;80(6):598–609. 35926933 · DOI · Paper page
  43. [58]Spaccarotella C, Polimeni A, Mancuso C, Pelaia G, Esposito G, Indolfi C. Assessment of non-invasive measurements of oxygen saturation and heart rate with an Apple Smartwatch: comparison with a standard pulse oximeter. J Clin Med. 2022;11(6):1467. 35329793 PMC8951323 · DOI · Paper page
  44. [60]Denny JC, Rutter JL, Goldstein DB, Philippakis A, Smoller JW, Jenkins G, et al. The “all of us” research program. N Engl J Med. 2019;381(7):668–76. 31412182 PMC8291101 · DOI · Paper page
  45. [61]Stamatakis E, Ahmadi MN, Gill JMR, Thøgersen-Ntoumani C, Gibala MJ, Doherty A, et al. Association of wearable device-measured vigorous intermittent lifestyle physical activity with mortality. Nat Med. 2022;28(12):2521–9. 36482104 PMC9800274 · DOI · Paper page
  46. [68]Belbasis L, Bellou V, Ioannidis JPA. Conducting umbrella reviews. BMJ Med. 2022;1(1): e000071. 36936579 PMC9951359 · DOI · Paper page

Cited by

Sorted by citations

Authors
Doherty, C., Baldwin, M., Keogh, A., Caulfield, B., & Argent, R.
Venue
Sports Medicine
Publisher
Springer Science and Business Media LLC
Published
2024-07-30
Volume
54
Issue
11
Pages
2907-2926
ISSN
  • 0112-1642
  • 1179-2035
DOI
10.1007/s40279-024-02077-2
PMC
PMC11560992
Cited by
114
References
12
Language
en
Licence
Funders
  • Science Foundation Ireland (22/NCF/FD/10949)
  • University College Dublin

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review