CoChat
Sign Up
Cited by 715Physical Activity and HealthMobile Health and mHealth ApplicationsContext-Aware Activity Recognition Systems

Reliability and Validity of Commercially Available Wearable Devices for Measuring Steps, Energy Expenditure, and Heart Rate: Systematic Review

Open access
Fuller, Daniel, Colwell, Emily, Low, Jonathan, Orychock, Kassia, Tobin, Melissa Ann, Simango, Bo, Buote, Richard, Van Heerden, Desiree +4 more·2020·JMIR mHealth and uHealth
DOI 10.2196/18694PMC PMC7509623

Published version of a preprint

Open this paper in CoChat — free accountOpen PDF

Abstract

Background: Consumer-wearable activity trackers are small electronic devices that record fitness and health-related measures. Objective: The purpose of this systematic review was to examine the validity and reliability of commercial wearables in measuring step count, heart rate, and energy expenditure. Methods: We identified devices to be included in the review. Database searches were conducted in PubMed, Embase, and SPORTDiscus, and only articles published in the English language up to May 2019 were considered. Studies were excluded if they did not identify the device used and if they did not examine the validity or reliability of the device. Studies involving the general population and all special populations were included. We operationalized validity as criterion validity (as compared with other measures) and construct validity (degree to which the device is measuring what it claims). Reliability measures focused on intradevice and interdevice reliability. Results: We included 158 publications examining nine different commercial wearable device brands. Fitbit was by far the most studied brand. In laboratory-based settings, Fitbit, Apple Watch, and Samsung appeared to measure steps accurately. Heart rate measurement was more variable, with Apple Watch and Garmin being the most accurate and Fitbit tending toward underestimation. For energy expenditure, no brand was accurate. We also examined validity between devices within a specific brand. Conclusions: Commercial wearable devices are accurate for measuring steps and heart rate in laboratory-based settings, but this varies by the manufacturer and device type. Devices are constantly being upgraded and redesigned to new models, suggesting the need for more current reviews and research.

AI snapshot

Written from the paper’s full text.

TL;DR

This systematic review of 158 publications found that commercial wearables are generally accurate for step count and heart rate in controlled settings, but not for energy expenditure. Accuracy varied by brand, device, and setting, while interdevice reliability was generally very strong and intradevice reliability data were limited mainly to step count.

Key findings

  • The review included 158 publications, 169 studies/substudies, 5,934 participants, and 1,838 comparisons; Fitbit was most studied (144 studies, 12 models), followed by Garmin (42 studies, 13 models), Apple (28 studies, 2 models), Polar (15 studies, 6 models), Misfit (13 studies, 2 models), Withings (12 studies, 2 models), Samsung (8 studies, 4 models), Mio (6 studies, 2 models), and Xiaomi (2 studies, 2 models).
  • For step count in controlled settings, 45.2% (364/805) of comparisons were within ±3% measurement error, with an overall tendency to underestimate (mean −9%, median −2%); in free-living settings, 42% (29/69) were within ±10%, with slight overestimation (mean 5%, median 6%).
  • For heart rate in controlled settings, 56.5% (100/177) of comparisons were within ±3%, with a slight underestimation tendency (median −1%); Apple Watch was within ±3% 71% (35/49) of the time, Fitbit 51% (36/71), and Garmin 49% (23/47). For energy expenditure in controlled settings, only 9.2% (28/305) were within ±3%, and no brand was within ±3% more than 13% of the time.
  • Reliability analyses included 19 studies and 166 comparisons; 153/166 comparisons reported interdevice reliability. Interdevice correlations were mostly very strong for step count (35 very strong, 3 strong) and energy expenditure (75% of 16 reported correlations very strong), while intradevice step-count reliability was variable (mean correlation coefficient 0.58).

Methods

  • Systematic review reported per PRISMA; searched PubMed, Embase, and SPORTDiscus through May 28, 2019, limited to English-language articles published from 2000 onward, including consumer wearable brands Apple, Empatica, Fitbit, Garmin, Jawbone, Mio, Misfit, Polar, Samsung, UnderArmour, Withings, and Xiaomi.
  • From 34,890 unique citations, 21,083 remained after duplicate removal, 542 underwent full-text screening, and 158 publications were included; these yielded 169 studies/substudies, with 168 examining validity and 19 examining reliability, and 1,838 comparisons (1,672 validity, 166 reliability).
  • Because of heterogeneous testing conditions and outcomes, meta-analysis was not conducted; the authors used narrative synthesis, criterion validity via correlation and group percentage difference, acceptable limits of ±3% in controlled settings and ±10% in free-living settings, and modified COSMIN-based risk-of-bias assessment.

Limitations

  • Stated by the authors: the consumer wearable market is volatile (e.g., Google purchased Fitbit for US $2.3 billion on November 1, 2019), many included devices are outdated or discontinued, and restricting to English-language studies may introduce bias.
  • Stated/evident: meta-analyses were not possible because of wide ranges in testing conditions and reported outcomes; reliability studies were fewer than validity studies, and between-brand or between-device reliability comparisons were not possible because of small sample sizes; no intradevice reliability data were reported for heart rate or energy expenditure.
  • Evident from the text: 82.8% of studies (140/169; 1,640/1,838 comparisons) were rated fair or poor for sample size (<50 participants), with only 7 excellent (≥100 participants) and 22 good (50-99 participants) studies; many tested devices are no longer available, and no brand stood out as a “gold standard” or had complete assessment across all measures.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.2196/18694

Cites

179 of 179 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 39 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    Fitbit was the most studied brand in the review.

    “Fitbit was by far the most studied brand.”
    • Also stated in: § Study and Participant Characteristics · plain
      “Fitbit consumer-grade wearables were examined most frequently (144 studies examining 12 models), followed by Garmin (42 studies, 13 models), Apple (28 studies, 2 models), Polar (15 studies, 6 models), Misfit (13 studies, 2 models), Withings (12 studies, 2 models), Samsung (8 studies, 4 models), Mio (6 studies, 2 models), and Xiaomi (2 studies, 2 models)”
    Quote in the paperNot a substantial claim
  • Findingnone given

    In laboratory-based settings, Fitbit, Apple Watch, and Samsung appeared to measure steps accurately.

    “In laboratory-based settings, Fitbit, Apple Watch, and Samsung appeared to measure steps accurately.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The criterion measures used in the included studies (manual counting, accelerometry, electrocardiography, Polar chest straps, pulse oximetry, direct and indirect calorimetry) provide valid reference standards for step count, heart rate, and energy expenditure.
  • Findingown data

    Those three comparisons noted underestimation of heart rate that varied depending on activity intensity.

    “and noted underestimation of heart rate that varied depending on activity intensity”
    Quote in the paperNot a substantial claim
  • Findingown data

    The correlation coefficient was moderate in one free-living heart rate comparison examining estimation during high-intensity activity.

    “and moderate in one comparison examining estimation during high-intensity activity.”
    Quote in the paperNot a substantial claim
  • Findingown data

    Despite similar ±3% measurement error rates, Fitbit appeared to underestimate heart rate more than Apple Watch and Garmin.

    “Despite similar ±3% measurement error rates, Fitbit appeared to underestimate heart rate more than Apple Watch and Garmin.”
    Quote in the paperNot a substantial claim
  • May combine two claims. 41
  • Classified as probably not a substantial claim. 22
  • The quote only partly states this claim. 1

  • May combine two claims.

    Findingown data

    The systematic review included 158 publications examining nine different commercial wearable device brands.

    “We included 158 publications examining nine different commercial wearable device brands.”
    • Also stated in: § Results · plain
      “As a result, a total of 158 publications were included in this systematic review (Figure 1) [14].”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Heart rate measurement was more variable across devices, with Apple Watch and Garmin being the most accurate and Fitbit tending toward underestimation.

    “Heart rate measurement was more variable, with Apple Watch and Garmin being the most accurate and Fitbit tending toward underestimation.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The criterion measures used in the included studies (manual counting, accelerometry, electrocardiography, Polar chest straps, pulse oximetry, direct and indirect calorimetry) provide valid reference standards for step count, heart rate, and energy expenditure.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). The quote only partly states this claim.

    Findingown data

    All of those three comparisons reported MAPE values within 10% measurement error.

    “but all reported that MAPE values fell within 10% measurement error.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    For energy expenditure in controlled settings, group measurement error was reported or calculable for 305 of 312 comparisons; 9.2% within ±3%, 54.1% below −3%, 36.7% above 3%.

    “Group measurement error was reported or calculable for 305 of the 312 comparisons, regardless of the criterion measure. Of these, 9.2% (n=28) were within ±3% measurement error, 54.1% (n=165) were below −3% measurement error, and 36.7% (n=112) were above 3% measurement error.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The criterion measures used in the included studies (manual counting, accelerometry, electrocardiography, Polar chest straps, pulse oximetry, direct and indirect calorimetry) provide valid reference standards for step count, heart rate, and energy expenditure.

  • May combine two claims.

    Findingown data

    Fitbit devices tended to provide inaccurate measures compared with the criterion, underestimating 48.4% (76/157) and overestimating 39.5% (62/157) of the time.

    “Fitbit devices tended to provide inaccurate measures compared with the criterion, underestimating 48.4% (76/157) of the time and overestimating 39.5% (62/157) of the time, despite the boxplot in Figure 5 showing a reasonable median value for accuracy.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The criterion measures used in the included studies (manual counting, accelerometry, electrocardiography, Polar chest straps, pulse oximetry, direct and indirect calorimetry) provide valid reference standards for step count, heart rate, and energy expenditure.

  • May combine two claims.

    Findingown data

    Group measurement error was reported or calculable for one of the five free-living heart rate comparisons, with the Fitbit Charge HR falling within ±10% measurement error in healthy children.

    “Group measurement error was reported or calculable for one of the five comparisons, with the Fitbit Charge HR falling within ±10% measurement error in the study examining healthy children.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Among the remaining step count comparisons, 11 of 15 reported MAPE.

    “Among the remaining comparisons, 11 of 15 reported MAPE”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    For step count in controlled settings, group measurement error was reported or calculable for 805 of 979 comparisons; 45.2% were within ±3%, 42.7% below −3%, and 12.1% above 3%.

    “Group measurement error was reported or calculable for 805 of the 979 comparisons, regardless of the criterion measure. Of these, 45.2% (n=364) were within ±3% measurement error, 42.7% (n=344) were below −3% measurement error, and 12.1% (n=97) were above 3% measurement error.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The criterion measures used in the included studies (manual counting, accelerometry, electrocardiography, Polar chest straps, pulse oximetry, direct and indirect calorimetry) provide valid reference standards for step count, heart rate, and energy expenditure.

  • May combine two claims.

    Findingown data

    Studies showed a tendency to underestimate energy expenditure and to provide inaccurate measures of energy expenditure compared with the criterion.

    “Studies showed a tendency to underestimate energy expenditure and to provide inaccurate measures of energy expenditure compared with the criterion.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The criterion measures used in the included studies (manual counting, accelerometry, electrocardiography, Polar chest straps, pulse oximetry, direct and indirect calorimetry) provide valid reference standards for step count, heart rate, and energy expenditure.

  • May combine two claims.

    Findingown data

    Apple wearables overestimated energy expenditure 58% (18/31) of the time and Polar 69% (9/13) of the time.

    “Conversely, Apple wearables overestimated energy expenditure 58% (18/31) of the time and Polar wearables overestimated energy expenditure 69% (9/13) of the time.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The criterion measures used in the included studies (manual counting, accelerometry, electrocardiography, Polar chest straps, pulse oximetry, direct and indirect calorimetry) provide valid reference standards for step count, heart rate, and energy expenditure.

  • May combine two claims.

    Findingown data

    Garmin wearables underestimated energy expenditure 69% (37/51) of the time and Withings 74% (34/46) of the time.

    “Underestimation of energy expenditure (less than −3%) was observed in Garmin wearables 69% (37/51) of the time and in Withings wearables 74% (34/46) of the time.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The criterion measures used in the included studies (manual counting, accelerometry, electrocardiography, Polar chest straps, pulse oximetry, direct and indirect calorimetry) provide valid reference standards for step count, heart rate, and energy expenditure.

  • May combine two claims.

    Findingown data

    Withings and Misfit wearables consistently underestimated step count, and Apple and Samsung had less measurement variability than other brands.

    “Withings and Misfit wearables consistently underestimated step count, and Apple and Samsung had less measurement variability than other brands.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The criterion measures used in the included studies (manual counting, accelerometry, electrocardiography, Polar chest straps, pulse oximetry, direct and indirect calorimetry) provide valid reference standards for step count, heart rate, and energy expenditure.

  • May combine two claims.

    Findingown data

    For heart rate in controlled settings, group measurement error was reported or calculable for 177 of 266 comparisons; 56.5% within ±3%, 24.9% below −3%, 18.6% above 3%.

    “Group measurement error was reported or calculable for 177 of 266 comparisons, regardless of the criterion measure. Of these, 56.5% (n=100) were within ±3% measurement error, 24.9% (n=44) were below −3% measurement error, and 18.6% (n=33) were above 3% measurement error.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The criterion measures used in the included studies (manual counting, accelerometry, electrocardiography, Polar chest straps, pulse oximetry, direct and indirect calorimetry) provide valid reference standards for step count, heart rate, and energy expenditure.

  • May combine two claims.

    Findingown data

    Apple Watch was within ±3% 71% (35/49) of the time, Fitbit 51% (36/71), and Garmin 49% (23/47).

    “Apple Watch was within ±3% 71% (35/49) of the time, while Fitbit wearables were within ±3% 51% (36/71) of the time and Garmin wearables were within ±3% 49% (23/47) of the time.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The criterion measures used in the included studies (manual counting, accelerometry, electrocardiography, Polar chest straps, pulse oximetry, direct and indirect calorimetry) provide valid reference standards for step count, heart rate, and energy expenditure.

  • May combine two claims.

    Findingown data

    Within brands, devices appeared to vary, with Fitbit Classic tending to overestimate steps, while Fitbit Charge tending to underestimate steps.

    “Within brands, devices appeared to vary, with Fitbit Classic tending to overestimate steps, while Fitbit Charge tending to underestimate steps”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Five studies (50 comparisons) reported interdevice reliability for energy expenditure, with analyzed sample sizes ranging from 13 to 29.

    “Five studies (50 comparisons) reported on interdevice reliability [85,113,116,117,166], with analyzed sample sizes ranging from 13 [117] to 29 [113].”
    Quote in the paperNot a substantial claim0 of 5 sources checked
    • cites [85] source

      Not checked in the free run.

    • cites [113] source

      Not checked in the free run.

    • cites [116] source

      Not checked in the free run.

    • cites [117] source

      Not checked in the free run.

    • cites [166] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    All five interdevice energy expenditure reliability studies recruited healthy adults (mean age 26.3 years, SD 3.9 years).

    “All five studies recruited healthy adults (mean age 26.3 years, SD 3.9 years).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Intradevice reliability correlations for step count were very weak (n=1), weak (n=2), moderate (n=5), strong (n=2), and very strong (n=3).

    “Intradevice reliability correlations were very weak (n=1), weak (n=2), moderate (n=5), strong (n=2), and very strong (n=3).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingcitation

    Similar standard typical error levels were seen in maximum heart rate measured during a single incremental maximal oxygen uptake test and heart rate taken from the highest 30-s mean heart rate, with somewhat lower correlation coefficients [24].

    “Similar standard typical error levels were seen in maximum heart rate measured during a single incremental maximal oxygen uptake test performed on a treadmill and heart rate taken from the highest 30-s mean heart rate, with somewhat lower correlation coefficients [24].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [24] source

      Not checked in the free run.

  • May combine two claims.

    Findingreasoning

    On average, only Fitbit measured energy expenditure to within acceptable limits, but there was wide variation around the estimate.

    “On average, only Fitbit measured energy expenditure to within acceptable limits, but there was wide variation around the estimate.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The review included nine brands and 45 devices, with the number of comparisons ranging from 201 for the Fitbit Zip to one for the Garmin Forerunner 405CX and the Polar M600.

    “Our review included nine brands and 45 devices with the number of comparisons ranging from 201 for the Fitbit Zip to one for the Garmin Forerunner 405CX and the Polar M600.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingcitation

    During the same tasks, Garmin Vivosmart HR+ showed good reliability during all tasks and had narrower limits of agreement than Fitbit.

    “During the same tasks, Garmin Vivosmart HR+ showed good reliability during all tasks and had narrower limits of agreement than Fitbit.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [58] source

      Not checked in the free run.

  • May combine two claims.

    Findingcitation

    Apple Watch showed very good interdevice reliability at 5-s epochs during treadmill bouts at 4, 7, and 10 km/h, with reliability increasing and standard typical error decreasing with increasing pace [26].

    “Apple Watch showed very good interdevice reliability at 5-s epochs during treadmill bouts at 4, 7, and 10 km/h, with reliability increasing and standard typical error decreasing with increasing pace [26].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [26] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation

    Xiaomi data were not analyzed in a single indirect calorimetry study owing to the lack of data [53].

    “Xiaomi data were not analyzed in a single indirect calorimetry study owing to the lack of data [53].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [53] source

      Not checked in the free run.

  • May combine two claims.

    Findingown data

    Energy expenditure estimates varied widely with less than 10% of estimates falling within acceptable limits in controlled settings.

    “Energy expenditure estimates varied widely with less than 10% of estimates falling within acceptable limits in controlled settings.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    For step count, in controlled laboratory settings, a higher proportion of devices showed accuracy, and this was within a tighter limit of acceptable accuracy compared with free-living conditions.

    “For step count, our review showed that in controlled laboratory settings, a higher proportion of devices showed accuracy, and this was within a tighter limit of acceptable accuracy compared with free-living conditions.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingcitation

    In healthy older adults, Fitbit Charge 2 showed good reliability during treadmill and overground bouts and poor reliability during hand movement tasks such as dusting [58].

    “In the examination of interdevice reliability in healthy older adults, Fitbit Charge 2 showed good reliability during treadmill and overground bouts and poor reliability during hand movement tasks such as dusting [58].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [58] source

      Not checked in the free run.

  • May combine two claims.

    Findingcitation

    Three studies (23 comparisons) examined interdevice reliability for heart rate, with analyzed sample sizes ranging from 13 to 21.

    “Three studies (23 comparisons) examined interdevice reliability for heart rate [24,26,58], with analyzed sample sizes ranging from 13 [24] to 21 [26].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [24] source

      Not checked in the free run.

    • cites [26] source

      Not checked in the free run.

    • cites [58] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Correlation coefficients were reported for 16 of the 50 interdevice energy expenditure reliability comparisons.

    “Correlation coefficients were reported for 16 of 50 comparisons.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingnone given

    Fitbit was studied the most and Xiaomi and Mio were studied the least.

    “Among the studies included, Fitbit was studied the most and Xiaomi and Mio were studied the least.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingnone given

    Interdevice reliabilities for steps, heart rate, and energy expenditure were all very strong.

    “Interdevice reliabilities for steps, heart rate, and energy expenditure were all very strong.”
    • Also stated in: § Conclusion · boosted
      “Interdevice reliabilities for steps, heart rate, and calories were all very strong.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    This systematic review of 158 publications included assessments of consumer wearable devices from nine brands (Apple Inc, Fitbit, Garmin, Mio, Misfit, Polar, Samsung, Withings, and Xiaomi).

    “This systematic review of 158 publications included assessments of consumer wearable devices from nine brands (Apple Inc, Fitbit, Garmin, Mio, Misfit, Polar, Samsung, Withings, and Xiaomi), with a focus on the reliability and validity of the devices in measuring heart rate, energy expenditure, and step count.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingnone given

    Energy expenditure estimates varied by model, with the Fitbit Classic underestimating the value considerably and Fitbit Charge HR overestimating the value.

    “Energy expenditure estimates also varied by model, with the Fitbit Classic underestimating the value considerably and Fitbit Charge HR overestimating the value.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The results showed considerable variability within the same device for step count for Fitbit Charge HR, Fitbit Surge, Fitbit Zip, and Garmin Vivofit, with five, five, one, and two comparisons, respectively.

    “The results showed considerable variability within the same device for step count for Fitbit Charge HR, Fitbit Surge, Fitbit Zip, and Garmin Vivofit, with five, five, one, and two comparisons, respectively.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    The four extreme outliers for step count involved measurement during sedentary and light physical activity in a single study with fewer than 40 participants and were likely inflated by limited steps accumulated.

    “The four extreme outliers for step count involved measurement during sedentary and light physical activity in a single study with fewer than 40 participants [20] and were likely inflated by the limited number of steps accumulated during those bouts.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    Commercial wearable devices are accurate for measuring steps and heart rate in laboratory-based settings, but this varies by the manufacturer and device type.

    “Commercial wearable devices are accurate for measuring steps and heart rate in laboratory-based settings, but this varies by the manufacturer and device type.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The commercial wearable devices included in the review (nine brands) are representative of commercial wearable devices generally.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The review was not registered with PROSPERO.

    “The review was not registered with PROSPERO.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductown data

    Full-text documents were independently assessed for inclusion by at least two authors, with conflicts resolved by discussion and consensus.

    “Following the title and abstract screening, full-text documents of the selected studies were searched and retrieved and were independently assessed for inclusion by at least two authors (EC, JL, and DF). Any conflicts were resolved by discussion and consensus.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductcitation

    The systematic review was conducted and reported according to PRISMA guidelines.

    “This systematic review was conducted and reported according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines [14].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [14] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductcitation

    The MEDLINE search strategy was developed by a librarian and peer reviewed by a second librarian according to PRESS 2015.

    “A librarian (KR) developed the MEDLINE search strategy, which was peer reviewed by a second librarian according to the Peer Review of Electronic Search Strategies (PRESS) 2015 Guideline Statement [15].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [15] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Fitbit wearables were highly correlated with criterion measures of step count during laboratory-based assessment and had consistently high interdevice reliability for both step count and energy expenditure.

    “Fitbit wearables were highly correlated with criterion measures of step count during laboratory-based assessment and had consistently high interdevice reliability for both step count and energy expenditure [13].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [13] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Bunn et al noted tendencies of wearables to underestimate energy expenditure, heart rate, and step count.

    “In a recent review of 10 articles, Bunn et al [12] noted tendencies of wearables to underestimate energy expenditure, heart rate, and step count.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [12] Bunn, JA, 2018, Int J Exerc Sci

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Fitbit wearables provide accurate measures only in limited circumstances.

    “However, this review found that these devices tended to underestimate energy expenditure, which is consistent with a separate review of Fitbit accuracy [14] indicating that Fitbit wearables provide accurate measures only in limited circumstances.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [14] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    There are possible interactions between the number and size of studies and device wear location that may influence brand comparisons.

    “There are possible interactions between the number and size of studies and device wear location that may influence the brand comparisons.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductcitation

    Risk of bias and quality of each study were assessed using a modification of the modified COSMIN validation subscale and an a priori modified COSMIN reliability subscale.

    “using a modification of the modified Consensus-Based Standards for the Selection of Health Status Measurement Instruments (COSMIN) validation subscale used by Feehan et al [13] and an a priori modified COSMIN reliability subscale (Multimedia Appendix 2) to assess the quality and risk of bias of each study (EC and DF)”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [13] source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Research has shown variation in step count among devices, and the applicability of metrics may vary by device brand and device type.

    “However, research has shown variation in step count among devices, and the applicability of these metrics may vary by device brand and device type [6].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [6] source

      Not checked in the free run.

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    Acceptable limits of percentage difference were ±3% in controlled settings and ±10% in free-living settings.

    “Our interpretation of measurement accuracy was focused on acceptable limits of percentage difference of ±3% in controlled settings and percentage difference of ±10% in free-living settings, as outlined in previous work [13].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [13] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    Details from each reviewer were compared, and inconsistencies were resolved through consensus before compiling results.

    “Details from each reviewer were compared, and inconsistencies were resolved through consensus before compiling the results (EC and DF).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data

    Titles and abstracts were independently assessed by at least two authors, with conflicts resolved by a third author or consensus.

    “The titles and abstracts of the studies included from the initial database search were independently assessed by at least two authors from the team. Conflicts arising during any step of the screening for inclusion/exclusion were resolved by a third author or by consensus.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    The variability observed could be attributed to differences in the number of comparisons for each device and in wear locations of the devices.

    “however, the variability observed could be attributed to differences in the number of comparisons for each device and in wear locations of the devices.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    The bias assessment showed no apparent bias toward studies of different sample sizes.

    “Our bias assessment showed no apparent bias toward studies of different sample sizes.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    There is a strong overrepresentation of studies with 20 participants.

    “However, there is a strong overrepresentation of studies with 20 participants.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductcitation

    There is some potential for bias when including only English language studies in systematic reviews.

    “There is some potential for bias when including only English language studies in systematic reviews.”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [183] source

      Not checked in the free run.

    • cites [184] source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Studies have shown that the effect of English-language restriction may be small in general but may be difficult to measure for an individual systematic review [183,184].

    “However, studies have shown that the effect may be small in general but may be difficult to measure for an individual systematic review [183,184].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [183] source

      Not checked in the free run.

    • cites [184] source

      Not checked in the free run.

54 of 54

  • The paper relies on this without support.

    The criterion measures used in the included studies (manual counting, accelerometry, electrocardiography, Polar chest straps, pulse oximetry, direct and indirect calorimetry) provide valid reference standards for step count, heart rate, and energy expenditure.

    “Criterion validity: comparing the devices to a criterion measure of steps, heart rate, or energy expenditure.”
    “compared with reference standard criterion measures of manual counting [32,34-38,42,46,47,50-53,57,58,72,80-84,88-102,109, 114-125,138-141,144-147,149-153,158-161,165,169-171,173] and accelerometry [20,60,64-66,85,103,109,126-128,148, 154,164]”
    “compared with reference standard criterion measures, including electrocardiography [22,23,38-40,43,44,54,61,62,67-70,142,162,176], Polar brand chest straps [20,21,24-28,58,63,71,163], and pulse oximetry [66]”

  • The paper relies on this without support.

    The commercial wearable devices included in the review (nine brands) are representative of commercial wearable devices generally.

    “We included 158 publications examining nine different commercial wearable device brands.”
    “Commercial wearable devices are accurate for measuring steps and heart rate in laboratory-based settings, but this varies by the manufacturer and device type.”

  • The paper relies on this without support.

    Research is equivocal about whether commercial wearable devices are valid and reliable methods for estimating steps, heart rate, and energy expenditure.

    “Despite their popularity, research is equivocal about whether commercial wearable devices are valid and reliable methods for estimating metrics associated with physical activity including steps, heart rate, and energy expenditure.”

  • The paper relies on this without support.

    Accelerometry is a valid reference standard criterion measure for step count in free-living settings.

    “compared with the reference standard criterion measure of accelerometry [33,45,48,49, 56,59,60,64,74-76,89,96,101,106-112,120,131-136, 149,154-156,159,167,168,172-175] in free-living settings”

  • The paper relies on this without support.

    A Polar brand chest strap is a valid reference standard criterion measure for heart rate in free-living settings.

    “compared with the reference standard criterion measure of a Polar brand chest strap in free-living settings [75,77,78].”

  • The paper relies on this without support.

    Doubly labeled water and accelerometry are valid criterion measures for energy expenditure in free-living settings.

    “compared with the criterion measures of doubly labeled water [104] and accelerometry [29,49,79,101,131,172,174,175].”
6 of 6

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The study was unable to account for daily step variations between different types of Fitbit models, seasonal differences, and the COVID-19 pandemic because device data were not available at the time of analyses and data were date-shifted to protect participant privacy.

    “We were not able to account for daily step variations between different types of Fitbit models14 and seasonal differences15 as well as the occurrence of the COVID-19 pandemic because device data were not available at the time of analyses and data were date-shifted to protect privacy of participants.”

    From Association of step counts over time with the risk of chronic disease in the All of Us Research Program (2022) · cites this paper as [14]

  • In recent years, the quality and accuracy of wearables have improved.

    “In recent years, the quality and accuracy of wearables have improved [7,8], resulting in more clinically approved certifications [9].”

    From The impact of wearable technologies in health research: scoping review (2022) · cites this paper as [7]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Discussion reverses the direction of free-living step-count error

    Major

    The discussion says devices tended to underestimate steps in free-living studies, but the free-living results report more overestimates than underestimates and describe an overall tendency toward overestimation.

  • Garmin energy-expenditure percentage does not match its fraction

    Minor

    The reported 37 of 51 Garmin comparisons is 72.55%, not 69%.

    Reported 69 · recomputed 72.54902

  • MAPE categories exceed the number said to report MAPE

    Minor

    The six comparisons below 10% error and nine above it total 15, although only 11 comparisons are said to have reported MAPE.

    Reported 11 · recomputed 15

  • Mean comparisons per reliability study is understated

    Minor

    The stated 166 reliability comparisons across 19 reliability studies give a mean of about 8.74, which rounds to 9 rather than the reported 8.

    Reported 8 · recomputed 8.736842

  • Withings model count conflicts with Table 1

    Minor

    The study-characteristics summary says two Withings models were examined, while Table 1 lists three: Pulse, Pulse O2, and Pulse Ox.

5 of 5

Cites

Flagged first, then by citations

1 of the 180 references couldn’t be matched to a known paper.

Research works printed but not linked: 14

  1. CostelloKGartner Says Worldwide Wearable Device Sales to Grow 26 Percent in 2019Gartner, Inc20182020-08-17https://www.gartner.com/en/newsroom/press-releases/2018-11-29-gartner-says-worldwide-wearable-device-sales-to-grow-
  2. Bunn, JA, 2018, Int J Exerc Sci, 11, 503
  3. Swinscow, T, 2009, Statistics at Square One
  4. Bunn, J, 2019, Int J Exerc Sci, 12, 374
  5. Sears, T, 2017, International Journal of Exercise Science, 10, 764
  6. Husted, HM, 2017, Int J Exerc Sci, 10, 146
  7. Keating, XD, 2012, ICHPER-SD Journal of Research, 9, 28
  8. Dondzila, C, 2018, International Journal of Exercise Science, 11, 575
  9. Jo, E, 2016, J Sports Sci Med, 15, 540
  10. Brazendale, K, 2019, Int J Exerc Sci, 12, 471
  11. Montes, J, 2017, Journal Exercise Physiology Online, 20, 134
  12. Stackpool, C, 2014, J Fit Res, 3, 32
  13. Gerrior, S, 2006, Prev Chronic Dis, 3, A129
  14. RallsJFitbit to Be Acquired by GoogleFitbit20192019-11-01https://investor.fitbit.com/press/press-releases/press-release-details/2019/Fitbit-to-Be-Acquired-by-Google/default.aspx

Cited by

Sorted by citations

Related

Authors
Fuller, D., Colwell, E., Low, J., Orychock, K., Tobin, M. A., Simango, B., Buote, R., Van Heerden, D., Luan, H., Cullen, K., Slade, L., & Taylor, N. G. A.
Affiliations
  • Memorial University of Newfoundland (CA)
  • University of Oregon (US)
  • Dalhousie University (CA)
Corresponding author
Fuller, Daniel
ORCID
Venue
JMIR mHealth and uHealth
Publisher
JMIR Publications Inc.
Published
2020-09-08
Volume
8
Issue
9
Pages
e18694
ISSN
2291-5222 (ISSN-L)
DOI
10.2196/18694
PMC
PMC7509623
OpenAlex
W3037734338
Cited by
715
References
180
Citations by year
2026: 130 · 2025: 155 · 2024: 153 · 2023: 116 · 2022: 101 · 2021: 55 · 2020: 5
FWCI
8.82
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • JMIR mhealth and uhealth (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Memorial University Research Repository (Memorial University) (acceptedVersion)
  • HARVEST (University of Saskatchewan) (submittedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • Memorial University Research Repository (Memorial University) (submittedVersion)
Topics
  • Health Sciences › Medicine › Physiology › Physical Activity and Health
  • Health Sciences › Health Professions › General Health Professions › Mobile Health and mHealth Applications
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Context-Aware Activity Recognition Systems
Keywords
wearable devices, wearable activity trackers, step counting, heart rate measurement, energy expenditure, device validity, device reliability, Fitbit, Apple Watch, Garmin, consumer health technology, inter-device agreement
MeSH terms
Fitness Trackers, Wearable Electronic Devices, Energy Metabolism, Heart Rate, Humans, Reproducibility of Results

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-09.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-09.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review