CoChat
Sign Up
Cited by 36Non-Invasive Vital Sign MonitoringCardiovascular and exercise physiologyHemodynamic Monitoring and Therapy

Investigating the accuracy of blood oxygen saturation measurements in common consumer smartwatches

Open access
Jiang, Yihang, Spies, Connor, Magin, Justin, Bhosai, Satasuk Joy, Snyder, Laurie, Dunn, Jessilyn·2023·PLOS Digital Health
DOI 10.1371/journal.pdig.0000296PMC PMC10337940
Open this paper in CoChat — free accountOpen PDF

Abstract

Blood oxygen saturation (SpO2) is an important measurement for monitoring patients with acute and chronic conditions that are associated with low blood oxygen levels. While smartwatches may provide a new method for continuous and unobtrusive SpO2 monitoring, it is necessary to understand their accuracy and limitations to ensure that they are used in a fit-for-purpose manner. To determine whether the accuracy of and ability to take SpO2 measurements from consumer smartwatches is different by device type and/or by skin tone, our study recruited patients aged 18–85 years old, with and without chronic pulmonary disease, who were able to provide informed consent. The mean absolute error (MAE), mean directional error (MDE) and root mean squared error (RMSE) were used to evaluate the accuracy of the smartwatches as compared to a clinical grade pulse oximeter. The percent of data unobtainable due to inability of the smartwatch to record SpO2 (missingness) was used to evaluate the measurability of SpO2 from the smartwatches. Skin tones were quantified based on the Fitzpatrick (FP) scale and Individual Typology Angle (ITA), a continuous measure of skin tone. A total of 49 individuals (18 female) were enrolled and completed the study. Using a clinical-grade pulse oximeter as the reference standard, there were statistically significant differences in accuracy between devices, with Apple Watch Series 7 having measurements closest to the reference standard (MAE = 2.2%, MDE = -0.4%, RMSE = 2.9%) and the Garmin Venu 2s having measurements farthest from the reference standard (MAE = 5.8%, MDE = 5.5%, RMSE = 6.7%). There were also significant differences in measurability across devices, with the highest data presence from the Apple Watch Series 7 (88.9% of attempted measurements were successful) and the highest data missingness from the Withings ScanWatch (only 69.5% of attempted measurements were successful). The MAE, RMSE and missingness did not vary significantly across FP skin tone groups, however, there may be a relationship between FP skin tone and MDE (intercept = 0.04, beta coefficient = 0.47, p = 0.04). No statistically significant difference was found between skin tone as measured by ITA and MAE, MDE, RMSE or missingness.

AI snapshot

Written from the paper’s full text.

TL;DR

In 49 adults with and without pulmonary/cardiovascular disease, four consumer smartwatches showed significant differences in SpO2 accuracy and missingness compared with a clinical-grade pulse oximeter. Apple Watch Series 7 was closest to the reference standard, while Garmin Venu 2s was farthest; Fitzpatrick skin tone showed no significant differences for MAE, RMSE, or missingness, though MDE may relate to FP skin tone.

Key findings

  • Across 588 attempted SpO2 measurements, 451 were successful. Apple Watch Series 7 had the best accuracy (MAE = 2.2%, MDE = -0.4%, RMSE = 2.9%) and lowest missingness (11%); Garmin Venu 2s had the worst accuracy (MAE = 5.8%, MDE = 5.5%, RMSE = 6.7%). Withings ScanWatch had the highest missingness (31%), followed by Garmin Fenix 6 Pro (28%).
  • Device differences were statistically significant: Garmin Venu 2s had significantly higher MAE and RMSE than all other devices; for MDE, every pairwise comparison differed significantly except Garmin Fenix 6 vs Withings ScanWatch. Apple Watch Series 7 had significantly lower missingness than Garmin Fenix 6 and Withings ScanWatch.
  • By Fitzpatrick skin tone groups II–VI, MAE was 4.2%, 3.3%, 3.1%, 4.1%, 3.0%; MDE was 2.4%, 1.4%, 1.3%, 3.6%, 2.2%; RMSE was 5.2%, 4.0%, 3.5%, 5.1%, 3.9%; missingness was 18.4%, 24.4%, 25.7%, 17.7%, 16.7%. No statistically significant differences were found across FP groups for MAE, MDE, RMSE, or missingness after Bonferroni correction.
  • An ordinary least squares model found MDE may have a relationship with FP skin tone groups (intercept = 0.04, beta coefficient = 0.47, p = 0.04), but no statistically significant relationship was found using the continuous ITA skin tone measure.

Methods

  • Prospective study at Duke University Health System; 49 participants aged 18–85 with and without respiratory disease completed the study (31 male, 18 female; median age 64, range 38–76; median BMI 28.8 kg/m²; 34.7% Black, 65.3% White). Most had pulmonary disease (42), three had cardiovascular disease, two had both, and two had no disease-causing hypoxemia.
  • Four consumer smartwatches—Apple Watch Series 7, Garmin Venu 2s, Garmin Fenix 6 Pro, and Withings ScanWatch—were compared against the Masimo MightySat Rx clinical-grade finger pulse oximeter. Measurements were taken at rest, with three measurements per device and reference per round and randomized wrist placement; skin tone was measured by Fitzpatrick scale and Delfin SkinColorCatch ITA.
  • Accuracy and usability metrics were MAE, MDE, missingness, and RMSE. Two-sided paired t-tests with Bonferroni correction compared devices; two-sided unpaired t-tests with Bonferroni correction compared skin tone groups; ordinary least squares regression assessed relationships between skin tone and performance metrics.

Limitations

  • The paper states that arterial blood gas was not used, so racial bias may exist in the reference gold standard; the Fitzpatrick scale is not an ideal objective skin tone measure and has suboptimal inter-rater reliability; unmeasured conditions such as anemia or hemoglobinopathies could have affected readings; only three participants had average reference SpO2 below 90%, limiting evaluation of low ranges; and 11 of 49 participants (22%) were missing ITA values, reducing robustness of those findings.
  • The text makes evident that the sample was small (49 participants), drawn from a single tertiary referral center, and mostly composed of patients with pulmonary disease, which limits generalizability.
  • The text makes evident that measurements were taken at rest only, so performance during motion, low perfusion, or other real-world conditions was not evaluated.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1371/journal.pdig.0000296

Cites

29 of 29 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

11 of 27 cited sources checked · 4 supported · 2 flagged · accuracy not yet measured

  • Premisecitation

    Evidence of the performance of consumer smartwatch pulse oximetry devices remains limited.

    “However, evidence of the performance of these devices remains limited, and disclaimers may be overlooked by consumers (S1 Text) [9].”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [9] B Bent, 2020, Wearables in the SARS-CoV-2 Pandemic: What Are They Good for?, JMIR Mhealth Uhealth. source
  • Premisecitation

    Racial bias can lead to undetected hidden hypoxemia.

    “For example, racial bias can lead to undetected hidden hypoxemia.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [17] MW Sjoding, 2020, Racial Bias in Pulse Oximetry Measurement, N Engl J Med source
  • Premisecitation

    In patients with anemia and sickle cell anemia, variability in the relationship between SpO2 and true arterial saturation measurements is not clinically significant.

    “although there is some variability in the relationship between SpO2 and the true arterial saturation measurements, this was not a clinically significant difference”
    Quote in the paperNot a substantial claim1 of 4 sources checked
    • cites [28] FO Ortiz, 1999, Accuracy of Pulse Oximetry in Sickle Cell Disease, Am J Respir Crit Care Med source
    • cites [29] RK Fitzgerald, 2001, Pulse oximetry in sickle cell anemia, Critical Care Medicine source
    • cites [30] GD Perkins, 2003, Do changes in pulse oximeter oxygen saturation predict equivalent changes in arterial oxygen saturation?, Critical Care source

      The Discussion says anaemia affects the relationship between SpO2 and SaO2, but that the clinical significance of this effect is small. It also says no patients were known to have sickle cell anaemia or trait, so the finding cannot be extended to that group. The paper further cautions that changes in SpO2 do not reliably predict equivalent changes in SaO2 overall. (LLM, conf=0.88)

      Experimental — accuracy not yet measured
    • cites [31] J Hinkelbein, 2004, Accuracy of pulse oximetry in a case of severe anemia from gastrointestinal hemorrhage: A-167. European Journal of Anaesthesiology |, EJA source
  • Premisecitation

    In prior smartwatch testing, accuracy and data missingness varied substantially by device type.

    “Further, the accuracy and data missingness varied substantially by device type.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [14] B Bent, 2020, Investigating sources of inaccuracy in wearable optical heart rate sensors, npj Digit Med source
  • Findingown data

    The Withings ScanWatch had the highest data missingness, with only 69.5% of attempted measurements successful.

    “and the highest data missingness from the Withings ScanWatch (only 69.5% of attempted measurements were successful)”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Missing smartwatch SpO2 measurements are due to the device's inability to record SpO2 rather than other causes such as user error.
  • May combine two claims. 30
  • Classified as probably not a substantial claim. 31

  • May combine two claims.

    Premisecitation

    The few existing studies do not investigate the effects of individual characteristics such as skin tone, compare across multiple devices, thoroughly examine performance in low ranges, or explore patterns of error and bias or missingness.

    “The few studies that do exist do not investigate the effects of individual characteristics (e.g., skin tone), compare across multiple devices or thoroughly examine performance in low ranges, or explore patterns of error and bias (e.g., consistent over- or underestimation of measurements) or missingness [10–13].”
    Quote in the paperSubstantial claim3 of 4 sources checked
    • cites [10] C Hahnen, 2020, Accuracy of Vital Signs Measurements by a Smartwatch and a Portable Health Device: Validation Study., JMIR mHealth and uHealth. source

      The Methods (Data and Statistical Analysis; page not provided) describe comparisons with reference values, Bland–Altman error plots, and assessment of abnormal ranges. The Discussion (Principal Findings; page not provided) explicitly compares two devices, reports detection of low heart rates, and quantifies failed measurements. The paper does not appear to examine skin tone, but it contradicts several of the sentence’s other claims about what these studies do not investigate. (LLM, conf=0.90)

      Checked by a reasoning model — may be wrong.

      Experimental — accuracy not yet measured
    • cites [11] C Spaccarotella, 2022, Assessment of Non-Invasive Measurements of Oxygen Saturation and Heart Rate with an Apple Smartwatch: Comparison with a Standard Pulse Oximeter, JCM source

      The Discussion (section 4) says all participants were white and few had oxygen saturation below 90%, supporting the skin-tone and low-range limitations. But the study compared an Apple Watch with a standard pulse oximeter (Methods, section 2), quantified measurement bias (Results, section 3), and reported cases where measurements could not be obtained (Results, section 3). Those parts of the citing sentence overstate what this study omitted. (LLM, conf=0.87)

      Experimental — accuracy not yet measured
    • cites [12] LZ Pipek, 2021, Comparison of SpO2 and heart rate values on Apple Watch and conventional commercial oximeters devices in patients with lung disease, Sci Rep source

      The Methods (Data collection; no page supplied) say the study recorded skin color and measured SpO2 with an Apple Watch and two commercial oximeters. The Methods (Statistical analysis) and Results also report an analysis of measurement bias. These passages contradict the claim that this study did not examine individual characteristics, compare multiple devices, or explore bias; they do not establish whether it thoroughly examined low ranges or missingness. (LLM, conf=0.90)

      Checked by a reasoning model — may be wrong.

      Experimental — accuracy not yet measured
    • cites [13] C Pätz, 2023, Accuracy of the Apple Watch Oxygen Saturation Measurement in Adults and Children with Congenital Heart Disease, Pediatr Cardiol source

  • May combine two claims.

    Premisecitation

    A prior study of commercial smartwatches found no difference in heart rate accuracy based on skin tone, but during physical activity the error increased by 30%.

    “we recently tested the accuracy of heart rate measurements from commercial smartwatches and found that, while there was no difference in accuracy based on skin tone, during physical activity the error increased by 30%[14].”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [14] B Bent, 2020, Investigating sources of inaccuracy in wearable optical heart rate sensors, npj Digit Med source

  • May combine two claims.

    Premisecitation

    The Masimo MightySat Rx is a clinical-grade finger pulse oximeter reference standard currently not thought to be affected by racial bias, with a reported 2% error measurements without motion.

    “a clinical-grade finger pulse oximeter reference standard currently not thought to be affected by racial bias[20], with a reported 2% error measurements without motion”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites [20] SJ Barker, 2023, Racial effects on Masimo pulse oximetry: a laboratory study., J Clin Monit Comput source

      The Discussion reports no clinically significant racial difference in accuracy for Masimo SET pulse oximeters with RD SET sensors in healthy Black and White volunteers. It does not identify or test the MightySat Rx, so the finding cannot establish that this particular finger oximeter is unaffected by racial bias. The Results report a combined root-mean-square error of 1.38%, not a 2% no-motion error; no page numbers are supplied in our copy. (LLM, conf=0.87)

      Experimental — accuracy not yet measured
    • cites [21] 510(k) Premarket Notification. [cited 29 Aug 2022]. Available: https://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfpmn/pmn.cfm?ID=K181956

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Premisecitation

    A large study including 1333 white patients and 276 black patients indicated that the incidence of hypoxemia can be 3 times higher among patients self-reported as Black versus White because oxygen saturation readings of black patients tend to be overestimated at a low SpO2 range.

    “Results from a large study [17] that includes 1333 white patients and 276 black patients indicated that the incidence of hypoxemia can be 3 times higher among patients self-reported as Black vs White because the oxygen saturation readings of black patients tend to be overestimated at a low SpO2 range.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [17] MW Sjoding, 2020, Racial Bias in Pulse Oximetry Measurement, N Engl J Med source

      Split into 2 atoms — 1 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Premisecitation

    Pulse oximetry should remain an important diagnostic tool in patients with anemia and sickle cell anemia.

    “pulse oximetry should remain an important diagnostic tool in these patients”
    Quote in the paperNot a substantial claim1 of 4 sources checked
    • cites [28] FO Ortiz, 1999, Accuracy of Pulse Oximetry in Sickle Cell Disease, Am J Respir Crit Care Med source
    • cites [29] RK Fitzgerald, 2001, Pulse oximetry in sickle cell anemia, Critical Care Medicine source
    • cites [30] GD Perkins, 2003, Do changes in pulse oximeter oxygen saturation predict equivalent changes in arterial oxygen saturation?, Critical Care source

      The Conclusion calls pulse oximetry a valuable tool in ICU care and reports that anaemia did not alter measurement precision to a clinically important extent. But the Discussion says no patients were known to have sickle cell anaemia or trait, so this study does not support extending that finding specifically to those patients. (LLM, conf=0.88)

      Experimental — accuracy not yet measured
    • cites [31] J Hinkelbein, 2004, Accuracy of pulse oximetry in a case of severe anemia from gastrointestinal hemorrhage: A-167. European Journal of Anaesthesiology |, EJA source

  • May combine two claims.

    Findingown data

    MAE, RMSE and missingness did not vary significantly across Fitzpatrick skin tone groups.

    “The MAE, RMSE and missingness did not vary significantly across FP skin tone groups”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The Fitzpatrick scale and Individual Typology Angle validly measure skin tone for the purpose of assessing differences in smartwatch SpO2 accuracy.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    Participants ranged from Type II to Type VI on the Fitzpatrick skin tone scale, with an average of 10 participants in each category.

    “Participants ranged from Type II to Type VI on the Fitzpatrick skin tone scale, with an average of 10 participants in each category.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The Fitzpatrick scale and Individual Typology Angle validly measure skin tone for the purpose of assessing differences in smartwatch SpO2 accuracy.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    On the ITA skin tone scale, participants ranged from -56° to 53°, with a median value of 21°.

    “On the ITA skin tone scale, participants ranged from -56° to 53°, with a median value of 21°.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The Fitzpatrick scale and Individual Typology Angle validly measure skin tone for the purpose of assessing differences in smartwatch SpO2 accuracy.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    The RMSE closest to zero was for group FP IV and the MDE farthest from zero was for FP II.

    “The RMSE closest to zero was for group FP IV and the MDE farthest from zero was for FP II.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Of the total number of expected measurements, 31% of Withings ScanWatch values and 28% of Garmin Fenix 6 Pro values were blank measurements.

    “Of the total number of expected measurements, 31% and 28% of the values for the Withings ScanWatch and the Garmin Fenix 6 Pro, respectively, were blank measurements.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The Apple Watch Series 7 had the lowest MAE and the Garmin Venu 2 had the highest MAE.

    “The Apple Watch Series 7 had the lowest MAE and the Garmin Venu 2 had the highest MAE.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    For MDE, every pairwise comparison showed a significant difference except for the Garmin Fenix 6 compared to the Withings ScanWatch.

    “For MDE, every pairwise comparison showed a significant difference except for the Garmin Fenix 6 compared to the Withings ScanWatch”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Neither MAE, MDE, RMSE, nor missingness demonstrated an apparent relationship to skin tone.

    “Neither MAE, MDE, RMSE, nor missingness demonstrated an apparent relationship to skin tone.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The Fitzpatrick scale is a valid measure of skin tone for classifying participants.

  • May combine two claims.

    Findingown data

    The Masimo MightySat Rx SpO2 measurements ranged from 82.0% to 100.0%, with an average of 95.8% and a median of 97.0%.

    “The Masimo MightySat Rx SpO2 measurements ranged from 82.0% to 100.0%, with an average of 95.8% and a median of 97.0% (S1 Table).”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The Masimo MightySat Rx clinical-grade pulse oximeter provides an accurate reference standard for SpO2 measurements.
    • Rests on an assumption: A 2% error range around the reference device's SpO2 reading is an appropriate threshold for classifying smartwatch measurements as within, over, or under the reference accuracy range.

  • May combine two claims.

    Findingown data

    The Apple Watch Series 7 had the MDE closest to zero and the Garmin Venu 2 had the MDE farthest from zero.

    “The Apple Watch Series 7 had the MDE closest to zero and the Garmin Venu 2 had the MDE farthest from zero.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    The MDE closest to zero was for group FP VI and the MDE farthest from zero was for FP V.

    “The MDE closest to zero was for group FP VI and the MDE farthest from zero was for FP V.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Average SpO2 values from the Apple Watch Series 7, Garmin Venu 2s, Garmin Fenix 6 Pro, and Withings ScanWatch devices were 96.3%, 90.4%, 94.4%, and 94.3%, respectively.

    “Average SpO2 values from the Apple Watch Series 7, Garmin Venu 2s, Garmin Fenix 6 Pro, and Withings ScanWatch devices were 96.3%, 90.4%, 94.4%, and 94.3%, respectively.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The Masimo MightySat Rx clinical-grade pulse oximeter provides an accurate reference standard for SpO2 measurements.

  • May combine two claims.

    Findingown data

    The Apple Watch Series 7 had the lowest RMSE and Garmin Venu 2 had the highest RMSE.

    “The Apple Watch Series 7 had the lowest RMSE and Garmin Venu 2 had the highest RMSE.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The Apple Watch Series 7 had the highest percentage of overestimated measurements (17.4%), meaning that the watch reported a higher blood oxygen saturation than the clinical-grade oximeter value plus a 2% error.

    “and also had the highest percentage of overestimated measurements (17.4%), meaning that the watch reported a higher blood oxygen saturation than the clinical-grade oximeter value plus a 2% error.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The Masimo MightySat Rx clinical-grade pulse oximeter provides an accurate reference standard for SpO2 measurements.
    • Rests on an assumption: A 2% error range around the reference device's SpO2 reading is an appropriate threshold for classifying smartwatch measurements as within, over, or under the reference accuracy range.

  • May combine two claims.

    Findingown data

    Missingness was 11% for Apple Watch, 14% for Garmin Venu 2s, 28% for Garmin Fenix 6 Pro, and 31% for Withings ScanWatch.

    “The missingness of the Apple Watch, Garmin Venu 2s, Garmin Fenix 6 Pro, and Withings device data ranged from 11%, 14%, 28%, and 31%, respectively”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The FP skin tone of participants ranged from group II to group VI.

    “The FP skin tone of participants ranged from group II to group VI”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    FP IV had the lowest MAE and FP II had the highest MAE.

    “Notably, FP IV had the lowest MAE and FP II had the highest MAE.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    There was a strong correspondence between skin tone values assessed by the Fitzpatrick scale and ITA, with a Spearman correlation coefficient of 0.928.

    “There was a strong correspondence between the skin tone values assessed by these two measurement modalities, with a Spearman correlation coefficient between the FP skin tone group and the objective ITA measurement value of 0.928 (S1 Fig).”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The Fitzpatrick scale and Individual Typology Angle validly measure skin tone for the purpose of assessing differences in smartwatch SpO2 accuracy.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    The ITA and Fitzpatrick measurements demonstrated a strong correlation.

    “While the ITA and Fitzpatrick measurements demonstrated a strong correlation”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The Apple Watch Series 7 had significantly lower missingness compared with both the Garmin Fenix 6 and the Withings ScanWatch.

    “The Apple Watch Series 7 had significantly lower missingness compared with both the Garmin Fenix 6 and the Withings ScanWatch”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    A total of 49 individuals (18 female) were enrolled in and completed the study.

    “A total of 49 individuals (18 female) were enrolled and completed the study.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductown data

    Statistical comparisons across skin tone groups used two-sided unpaired t-tests with Bonferroni multiple hypothesis correction, with a threshold of 0.005.

    “we used two-sided, unpaired t-tests with Bonferroni multiple hypothesis correction (Table 2). The Bonferroni threshold for comparison of MAE, MDE, RMSE and missingness was 0.005.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductown data

    Statistical comparisons among devices used two-sided paired t-tests with Bonferroni multiple hypothesis correction, with a threshold of 0.0083.

    “we used two-sided, paired t-tests with Bonferroni multiple hypothesis correction. The Bonferroni threshold for comparison of MAE, MDE, RMSE, and missingness was 0.0083.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    This study was conducted under the approval of the Duke University Health System (DUHS) Institutional Review Board and all participants signed informed consent before participating.

    “This study was conducted under the approval of the Duke University Health System (DUHS) Institutional Review Board and all participants signed informed consent before participating”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Continuous ITA values are difficult to employ as compared with ordinal Fitzpatrick measurements for the statistical analyses presented in the study.

    “the continuous ITA values are difficult to employ as compared with the ordinal Fitzpatrick measurements for the statistical analyses presented herein”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationreasoning

    Overestimated measurements are potentially dangerous if a person decides not to seek care as a result of considering the inaccurate measurement.

    “Overestimated measurements are potentially dangerous if a person decides not to seek care as a result of considering the inaccurate measurement.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductown data

    An ordinary least squares model was fit to test the relationship between skin tone (FP or ITA) and MAE, MDE, RMSE, or missingness.

    “we fit an ordinary least squares model”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    Four representative commercial wearables with high market share were explored.

    “Here, we explored four representative commercial wearables with high market share [22,23].”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [22] 510(k) Premarket Notification. [cited 3 Mar 2022]. Available: https://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfpmn/pmn.cfm?ID=K201456

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [23] 2020, Apple, Garmin, and Huawei continue to top the smartwatch market in

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data

    Overall, 588 blood oxygen saturation measurements were attempted, 451 were made successfully, with an average of 2.3 measurements per device per patient.

    “Overall, 588 blood oxygen saturation measurements were attempted, 451 were made successfully, with an average of 2.3 measurements/device/patient.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Missing smartwatch SpO2 measurements are due to the device's inability to record SpO2 rather than other causes such as user error.

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    The smartwatches were placed on separate and randomized wrists.

    “first with the Apple Watch Series 7 and Garmin Venu 2s on separate and randomized wrists, followed by the Garmin Fenix 6 Pro and Withings ScanWatch on separate and randomized wrists.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Interpretationown data

    The missingness values indicate the proportion of time in this study when the device was unable to generate measurements.

    “which indicates the proportion of time in this study when the device was unable to generate measurements”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Missing measurements reflect the device's inability to generate a measurement rather than other factors such as user error or improper placement.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Many digital health devices make claims that meet FDA standards, but this does not always mean that they have been rigorously validated; it could mean that a device is not harmful, but not necessarily accurate or backed by sufficient clinical data.

    “Many digital health devices make claims that meet FDA standards, but this does not always mean that they have been rigorously validated. Simply, it could mean that a device is not harmful, but not necessarily accurate or backed by sufficient clinical data [25].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [25] 2020

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    The Fitzpatrick scale was developed based on self-reported erythema sensitivity and the ability of Caucasians to tan, and has been commonly used to categorize skin tones into six categories.

    “The Fitzpatrick scale was developed based on self-reported erythema sensitivity and the ability of Caucasians to tan, and has been commonly used to categorize skin tones into six categories [26].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [26] BCK Ly, 2020, Research Techniques Made Simple: Cutaneous Colorimetry: A Reliable Technique for Objective Skin Color Measurement, Journal of Investigative Dermatology

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    A handheld colorimeter was used to measure skin tones in a standardized manner.

    “We also utilized a handheld colorimeter to measure skin tones in a standardized manner.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The Fitzpatrick scale and Individual Typology Angle validly measure skin tone for the purpose of assessing differences in smartwatch SpO2 accuracy.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductreasoning

    Although exclusion criteria accounted for many conditions, others such as anemia or hemoglobinopathies could have been present in these patients and could have affected the readings.

    “Although our exclusion criteria accounted for many of these conditions, others, such as anemia or hemoglobinopathies, could have been present in these patients and could have affected the readings.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductreasoning

    Because the study did not use arterial blood gas to calculate oxygen saturation, racial bias may exist in the reference gold standard, which can affect the accuracy of subjects with dark skin tones.

    “Since we didn’t use arterial blood gas to calculate oxygen saturation, it is possible that racial bias exists in our reference gold standard, which can affect the accuracy of subjects with dark skin tones.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given

    Skin tones were quantified using the Fitzpatrick scale and Individual Typology Angle, a continuous measure of skin tone.

    “Skin tones were quantified based on the Fitzpatrick (FP) scale and Individual Typology Angle (ITA), a continuous measure of skin tone.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The Fitzpatrick scale and Individual Typology Angle validly measure skin tone for the purpose of assessing differences in smartwatch SpO2 accuracy.

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Interpretationown data

    Directionality of errors may serve as a useful metric for comparing across different smartwatch-based oxygen saturation measurement technologies.

    “This indicates that directionality of errors may serve as a useful metric for comparing across different smartwatch-based oxygen saturation measurement technologies.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    Of the 49 total participants, 11 were missing skin tone measurements on the ITA scale because the device used to measure skin tone objectively was not yet available.

    “Of the 49 total participants, 11 were missing skin tone measurements on the ITA scale because we did not yet have the device used to measure skin tone objectively (S4 Table).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given

    This is the first study to explore the accuracy of blood oxygen saturation measurements from consumer smartwatches as well as patterns of missingness, error, and bias in patients likely to experience intermittent oxygen desaturations.

    “To our knowledge, this is the first study to explore the accuracy of blood oxygen saturation measurements from consumer smartwatches as well as patterns of missingness, error, and bias in patients likely to experience intermittent oxygen desaturations.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    If only MAE was reported, the Apple Watch, Garmin Fenix Pro 6, and Withings ScanWatch would appear to have similar performance; however, when MDE is reported, there is a significant difference between these devices.

    “if only MAE was reported, we would conclude that the Apple Watch, Garmin Fenix Pro 6, and Withings ScanWatch all have similar performance. However, when MDE is reported, we find that there is a significant difference between these devices.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    A clinical-grade pulse oximeter was used as the reference standard for SpO2 measurements.

    “Using a clinical-grade pulse oximeter as the reference standard”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The Masimo MightySat Rx clinical-grade pulse oximeter provides an accurate reference standard for SpO2 measurements.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Pulse oximeters underperform under circumstances of low perfusion.

    “it is known that pulse oximeters underperform under circumstances of low perfusion [33]”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [33] A Louie, 2018, Four Types of Pulse Oximeters Accurately Detect Hypoxia during Low Perfusion and Motion, Anesthesiology source

  • Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

    Interpretationreasoning

    When SpO2 values are truly lower, a patient who tests positive for COVID-19 might delay seeking care as a result of overestimated measurements, which could be dangerous practically.

    “This could be dangerous practically because when SpO2 values are truly lower, a patient who tests positive for COVID-19 might delay seeking care as a result of overestimated measurements.”
    Quote in the paperNot a substantial claim
49 of 49

  • The paper relies on this without support.

    The Masimo MightySat Rx clinical-grade pulse oximeter provides an accurate reference standard for SpO2 measurements.

    “The Masimo MightySat Rx SpO2 measurements ranged from 82.0% to 100.0%, with an average of 95.8% and a median of 97.0% (S1 Table).”

  • The paper relies on this without support.

    A 2% error range around the reference device's SpO2 reading is an appropriate threshold for classifying smartwatch measurements as within, over, or under the reference accuracy range.

    “plus a 2% error”

  • The paper relies on this without support.

    Missing smartwatch SpO2 measurements are due to the device's inability to record SpO2 rather than other causes such as user error.

    “The percent of data unobtainable due to inability of the smartwatch to record SpO2 (missingness) was used to evaluate the measurability of SpO2 from the smartwatches.”

  • The paper relies on this without support.

    The Fitzpatrick scale and Individual Typology Angle validly measure skin tone for the purpose of assessing differences in smartwatch SpO2 accuracy.

    “Skin tones were quantified based on the Fitzpatrick (FP) scale and Individual Typology Angle (ITA), a continuous measure of skin tone.”

  • The paper relies on this without support.

    The Fitzpatrick scale is a valid measure of skin tone for classifying participants.

    “We measured skin tone initially by the Fitzpatrick scale (FP)”

  • The paper relies on this without support.

    The Delfin ITA scale is a valid measure of skin tone.

    “patients’ skin tone was measured on the ITA scale using the Delfin SkinColorCatch”

  • The paper relies on this without support.

    Missing measurements reflect the device's inability to generate a measurement rather than other factors such as user error or improper placement.

    “If the device still failed to produce a measurement, no value was recorded for this specific time point and it was labeled as a missing (unobtainable) observation.”

  • The paper relies on this without support.

    Pulse oximetry can be an effective tool to detect the risk of deterioration and promote patient safety.

    “Pulse oximetry can be an effective tool to detect the risk of deterioration and promote patient safety.”

  • The paper relies on this without support.

    The advent of smartwatch-based oxygen saturation monitoring tools opens up new opportunities for remote and continuous patient monitoring.

    “The advent of smartwatch-based oxygen saturation monitoring tools opens up new opportunities for remote and continuous patient monitoring”

  • The paper relies on this without support.

    The 90% threshold is important clinically because most people have a normal resting oxygen level over 90% and rarely go below this range unless an acute event is ongoing.

    “The 90% threshold is important clinically because most people have a normal resting oxygen level over 90% and rarely go below this range unless an acute event is ongoing.”
10 of 10

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • FP VI is incorrectly identified as having MDE closest to zero

    Minor

    All listed MDEs are positive; FP IV has the smallest, at 1.3%, compared with 2.2% for FP VI.

  • Table 2 is cited for skin-tone comparisons but reports device comparisons

    Minor

    The text describes Table 2 as supporting unpaired comparisons across skin-tone groups, whereas its caption and columns identify paired comparisons between devices.

  • FP IV is incorrectly identified as having the lowest MAE

    Minor

    The listed MAEs put FP VI at 3.0%, below FP IV at 3.1%.

3 of 3

Cites

Flagged first, then by citations

Research works printed but not linked: 3

  1. 2022, Smartwatch Market to Grasp the Value of USD 64,480.18 Million by 2030, Size, Share, Key Growth Drivers, Trends, Technology Advancement and Competitive Landscape
  2. 2020, Apple, Garmin, and Huawei continue to top the smartwatch market in
  3. 2020, FDA

Cited by

Sorted by citations

Related

Authors
Jiang, Y., Spies, C., Magin, J., Bhosai, S. J., Snyder, L., & Dunn, J.
Affiliations
  • Duke University (US)
  • University of North Carolina at Chapel Hill (US)
Corresponding author
Dunn, Jessilyn
ORCID
Venue
PLOS Digital Health
Publisher
Public Library of Science (PLoS)
Published
2023-07-12
Volume
2
Issue
7
Pages
e0000296
ISSN
2767-3170 (ISSN-L)
DOI
10.1371/journal.pdig.0000296
PMC
PMC10337940
OpenAlex
W4384025158
Semantic Scholar
0a945a146adac12209f82f59981edd79635726a8
Cited by
36
References
28
Citations by year
2026: 8 · 2025: 17 · 2024: 9 · 2023: 2
FWCI
2.64
Type
journal article
Language
en
Open access
Licence
Locations
  • PLOS Digital Health (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Carolina Digital Repository (University of North Carolina at Chapel Hill) (submittedVersion)
  • PubMed Central (submittedVersion, cc-by)
  • UNC Libraries
Funders
  • AstraZeneca
  • National Institute of Diabetes and Digestive and Kidney Diseases (R01DK133531)
Topics
  • Physical Sciences › Engineering › Biomedical Engineering › Non-Invasive Vital Sign Monitoring
  • Health Sciences › Medicine › Complementary and alternative medicine › Cardiovascular and exercise physiology
  • Health Sciences › Medicine › Surgery › Hemodynamic Monitoring and Therapy
Keywords
oxygen saturation, wearable devices, pulse oximetry, skin tone, Fitzpatrick skin type, Individual Typology Angle, chronic lung disease

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-17.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-17.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review