Reliability of Scoring Respiratory Disturbance Indices and Sleep Staging
Abstract
STUDY OBJECTIVES: Unattended, home-based polysomnography (PSG) is increasingly used in both research and clinical settings as an alternative to traditional laboratory-based studies, although the reliability of the scoring of these studies has not been described. The purpose of this study is to describe the reliability of the PSG scoring in the Sleep Heart Health Study (SHHS), a multicenter study of the relation between sleep-disordered breathing measured by unattended, in-home PSG using a portable sleep monitor, and cardiovascular outcomes. DESIGN: The reliability of SHHS scorers was evaluated based on 20 randomly selected studies per scorer, assessing both interscorer and intrascorer reliability. RESULTS: Both inter- and intrascorer comparisons on epoch-by-epoch sleep staging showed excellent reliability (kappa statistics >0.80), with stage 1 having the greatest discrepancies in scoring and stage 3/4 being the most reliably discriminated. The arousal index (number of arousals per hour of sleep) was moderately reliable, with an intraclass correlation (ICC) of 0.54. The scorers were highly reliable on various respiratory disturbance indices (RDIs), which incorporate an associated oxygen desaturation in the definition of respiratory events (2% to 5%) with or without the additional use of associated EEG arousal in the definition of respiratory events (ICC>0.90). When RDI was defined without considering oxygen desaturation or arousals to define respiratory events, the RDI was moderately reliable (ICC=0.74). The additional use of associated EEG arousals, but not oxygen desaturation, in defining respiratory events did little to increase the reliability of the RDI measure (ICC=0.77). CONCLUSIONS: The SHHS achieved a high degree of intrascorer and interscorer reliability for the scoring of sleep stage and RDI in unattended in-home PSG studies.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the unattended PSGs in the Sleep Heart Health Study, events were more reliably scored when desaturation criteria alone were used in comparison to inclusion of the arousal criteria.
“For example, in the unattended PSGs in the SHHS, events were more reliably scored when desaturation criteria alone were used in comparison to inclusion of the arousal criteria.23”
From International Consensus Statement on Obstructive Sleep Apnea (2023) · cites this paper as [23]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
A Coefficient of Agreement for Nominal Scales
Cohen, J. (1960) · Educational and Psychological Measurement · doi:10.1177/001316446002000104 · Cited by 42,398 · see citing works
Intraclass correlations: Uses in assessing rater reliability.
Shrout, P. E., & Fleiss, J. L. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.2.420 · Cited by 23,433 · see citing works
Statistical Methods for Rates and Proportions.
Everitt, B. S., & Fleiss, J. L. (1981) · Biometrics · doi:10.2307/2530193 · Cited by 9,034 · see citing works
Hobson, J. A. (1969) · Electroencephalography and Clinical Neurophysiology · doi:10.1016/0013-4694(69)90021-2 · Cited by 5,215 · see citing works
The Sleep Heart Health Study: Design, Rationale, and Methods
(1997) · Sleep · doi:10.1093/sleep/20.12.1077 · Cited by 1,331 · see citing works
Sample size requirements for reliability studies
Donner, A. P., & Eliasziw, M. (1987) · Statistics in Medicine · doi:10.1002/sim.4780060404 · Cited by 693 · see citing works
Methods for Obtaining and Analyzing Unattended Polysomnography Data for a Multicenter Study
Redline, S. S., SANDERS, M. H., Lind, B. K., Quan, S. F., Iber, C., Gottlieb, D. J., BONEKAT, W. H., Rapoport, D. M., Smith, P. L., & Kiley, J. P. (1998) · SLEEP · doi:10.1093/sleep/21.7.759 · Cited by 534 · see citing works
Self-citationSchaltenbrand, N., Lengelle, R., Toussaint, M., Luthringer, R., Carelli, G., Jacqmin, A., Lainey, E., Muzet, A. G., & Macher, J. P. (1996) · SLEEP · doi:10.1093/sleep/19.1.26 · Cited by 157 · see citing works
Moser, N. J., Phillips, B. A., Berry, D. T. R., & Harbison, L. (1994) · CHEST Journal · doi:10.1378/chest.105.2.426 · Cited by 142 · see citing works
Interobserver Variability in Recognizing Arousal in Respiratory Sleep Disorders
Drinnan, M. J., Murray, A., Griffiths, C. J., & Gibson, G. J. (1998) · American Journal of Respiratory and Critical Care Medicine · doi:10.1164/ajrccm.158.2.9705035 · Cited by 106 · see citing works
Automated sleep scoring: a comparative reliability study of two algorithms
Stanus, E., Lacroix, B., Kerkhofs, M., & MENDLEWICZ, J. (1987) · Electroencephalography and Clinical Neurophysiology · doi:10.1016/0013-4694(87)90214-8 · Cited by 57 · see citing works
Measurement error in visually scored electrophysiological data: respiration during sleep
Bliwise, D., Bliwise, N. G., Kraemer, H. C., & Dement, W. (1984) · Journal of Neuroscience Methods · doi:10.1016/0165-0270(84)90047-5 · Cited by 52 · see citing works
Inter-Rater Reliability for Scoring Periodic Leg Movements in Sleep
Bliwise, D. L., Keenan, S. A., Bumburg, D., et al. (1991) · SLEEP · doi:10.1093/sleep/14.3.249 · Cited by 21 · see citing works
Karacan, I., Orr, W. C., Roth, T., et al. (1978) · Psychophysiology · doi:10.1111/j.1469-8986.1978.tb01358.x · Cited by 18 · see citing works
Flagged first, then by citations
Research works printed but not linked: 2
- [4]4. The Atlas Task Force. EEG arousals: scoring rules and examples. Sleep 1992;15:173-84.
- [18]18. Lord S, Sawyer B, Pond D, et al. Interrater reliability of computer- assisted scoring of breathing during sleep [see comments]. Sleep 12(6):550-8, 1989. Comment in: Sleep 1991;14(1):89-90.
Cited by
Sorted by citations
Berry, R. B., Budhiraja, R., Gottlieb, D. J., Gozal, D., Iber, C., Kapur, V. K., Marcus, C. L., Mehra, R., Parthasarathy, S., Quan, S. F., et al. (2012) · Journal of Clinical Sleep Medicine · doi:10.5664/jcsm.2172 · Cited by 5,302 · see citing works
Self-citationNieto, F. J. (2000) · JAMA · doi:10.1001/jama.283.14.1829 · Cited by 3,369 · see citing works
Self-citationSleep-disordered Breathing and Cardiovascular Disease
SHAHAR, E., WHITNEY, C. W., REDLINE, S., LEE, E. T., NEWMAN, A. B., JAVIER NIETO, F., O'CONNOR, G. T., BOLAND, L. L., SCHWARTZ, J. E., & SAMET, J. M. (2001) · American Journal of Respiratory and Critical Care Medicine · doi:10.1164/ajrccm.163.1.2001008 · Cited by 2,901 · see citing works
Self-citationGottlieb, D. J., Yenokyan, G., Newman, A. B., O'Connor, G. T., Punjabi, N. M., Quan, S. F., Redline, S., Resnick, H. E., Tong, E. K., Diener-West, M., et al. (2010) · Circulation · doi:10.1161/circulationaha.109.901801 · Cited by 1,551 · see citing works
Self-citationYoung, T. (2002) · Archives of Internal Medicine · doi:10.1001/archinte.162.8.893 · Cited by 1,304 · see citing works
Association of Nocturnal Arrhythmias with Sleep-disordered Breathing
Mehra, R., Benjamin, E. J., Shahar, E., Gottlieb, D. J., Nawabit, R., Kirchner, H. L., Sahadevan, J., & Redline, S. (2006) · American Journal of Respiratory and Critical Care Medicine · doi:10.1164/rccm.200509-1442oc · Cited by 1,262 · see citing works
Self-citationPunjabi, N. M. (2004) · American Journal of Epidemiology · doi:10.1093/aje/kwh261 · Cited by 1,107 · see citing works
The visual scoring of sleep in adults
Silber, M. H., Ancoli-Israel, S., Bonnet, M. H., Chokroverty, S., Grigg-Damberger, M. M., Hirshkowitz, M., Kapen, S., Keenan, S. A., Kryger, M. H., Penzel, T., et al. (2007) · Journal of Clinical Sleep Medicine · doi:10.5664/jcsm.26814 · Cited by 1,064 · see citing works
Racial/Ethnic Differences in Sleep Disturbances: The Multi-Ethnic Study of Atherosclerosis (MESA)
Chen, X., Wang, R., Zee, P., Lutsey, P. L., Javaheri, S., Alcántara, C., Jackson, C. L., Williams, M. A., & Redline, S. (2015) · SLEEP · doi:10.5665/sleep.4732 · Cited by 922 · see citing works
Self-citationThe effects of age, sex, ethnicity, and sleep-disordered breathing on sleep architecture
Redline, S., Kirchner, H. L., Quan, S. F., Gottlieb, D. J., Kapur, V., & Newman, A. (2004) · Archives of Internal Medicine · doi:10.1001/archinte.164.4.406 · Cited by 732 · see citing works
Self-citationThe new AASM criteria for scoring hypopneas: impact on the apnea hypopnea index
Ruehland, W. R., Rochford, P. D., O’Donoghue, F. J., Pierce, R. J., Singh, P., & Thornton, A. T. (2009) · SLEEP · doi:10.1093/sleep/32.2.150 · Cited by 696 · see citing works
Diabetes and Sleep Disturbances
Resnick, H. E., Redline, S., Shahar, E., Gilpin, A., Newman, A., Walter, R., Ewy, G. A., Howard, B. V., & Punjabi, N. M. (2003) · Diabetes Care · doi:10.2337/diacare.26.3.702 · Cited by 504 · see citing works
Self-citationSleep and Sleep-disordered Breathing in Adults with Predominantly Mild Obstructive Airway Disease
Sanders, M. H., Newman, A. B., Haggerty, C. L., Redline, S., Lebowitz, M., Samet, J., O'Connor, G. T., Punjabi, N. M., & Shahar, E. (2003) · American Journal of Respiratory and Critical Care Medicine · doi:10.1164/rccm.2203046 · Cited by 408 · see citing works
Self-citationInternational Consensus Statement on Obstructive Sleep Apnea
Chang, J. L., Goldberg, A. N., Alt, J. A., Mohammed, A., Ashbrook, L., Auckley, D., Ayappa, I., Bakhtiar, H., Barrera, J. E., Bartley, B. L., et al. (2023) · International Forum of Allergy & Rhinology · doi:10.1002/alr.23079 · Cited by 350 · see citing works
Self-citationRelationship between reported and measured sleep times: the sleep heart health study (SHHS)
Silva, G. E., Goodwin, J. L., Sherrill, D. L., Arnold, J. L., Bootzin, R. R., Smith, T., Walsleben, J. A., Baldwin, C. M., & Quan, S. F. (2007) · Journal of Clinical Sleep Medicine · doi:10.5664/jcsm.26974 · Cited by 295 · see citing works
Mehra, R., Stone, K. L., Varosy, P. D., Hoffman, A. R., Marcus, G. M., Blackwell, T., Ibrahim, O. A., Salem, R., & Redline, S. (2009) · Archives of Internal Medicine · doi:10.1001/archinternmed.2009.138 · Cited by 269 · see citing works
Self-citationEffects of Varying Approaches for Identifying Respiratory Disturbances on Sleep Apnea Assessment
REDLINE, S., KAPUR, V. K., SANDERS, M. H., QUAN, S. F., GOTTLIEB, D. J., RAPOPORT, D. M., BONEKAT, W. H., SMITH, P. L., KILEY, J. P., & IBER, C. (2000) · American Journal of Respiratory and Critical Care Medicine · doi:10.1164/ajrccm.161.2.9904031 · Cited by 225 · see citing works
Self-citationRoca, G. Q., Redline, S., Claggett, B., Bello, N., Ballantyne, C. M., Solomon, S. D., & Shah, A. M. (2015) · Circulation · doi:10.1161/circulationaha.115.016985 · Cited by 182 · see citing works
Self-citationGenetic Associations with Obstructive Sleep Apnea Traits in Hispanic/Latino Americans
Cade, B. E., Chen, H., Stilp, A. M., Gleason, K. J., Sofer, T., Ancoli-Israel, S., Arens, R., Bell, G. I., Below, J. E., Bjonnes, A. C., et al. (2016) · American Journal of Respiratory and Critical Care Medicine · doi:10.1164/rccm.201512-2431oc · Cited by 134 · see citing works
Self-citationNakayama-Ashida, Y., Takegami, M., Chin, K., Sumi, K., Nakamura, T., Takahashi, K.-I., Wakamura, T., Horita, S., Oka, Y., Minami, I., et al. (2008) · SLEEP · doi:10.1093/sleep/31.3.419 · Cited by 119 · see citing works
Sleepiness, Quality of Life, and Sleep Maintenance in REM versus non-REM Sleep-disordered Breathing
Chami, H. A., Baldwin, C. M., Silverman, A., Zhang, Y., Rapoport, D., Punjabi, N. M., & Gottlieb, D. J. (2010) · American Journal of Respiratory and Critical Care Medicine · doi:10.1164/rccm.200908-1304oc · Cited by 107 · see citing works
Self-citationObstructive sleep apnea and quality of life: comparison of the SAQLI, FOSQ, and SF-36 questionnaires
University of Arizona, T., Silva, G., Goodwin, J., Vana, K., & Quan, S. (2016) · Southwest Journal of Pulmonary and Critical Care · doi:10.13175/swjpcc082-16 · Cited by 95 · see citing works
Khan, A., Harrison, S. L., Kezirian, E. J., Ancoli-Israel, S., O’Hearn, D., Orwoll, E., Redline, S., Ensrud, K., Stone, K. L., & for the Osteoporotic Fractures in Men (MrOS) Study Research Group (2013) · Journal of Clinical Sleep Medicine · doi:10.5664/jcsm.2474 · Cited by 51 · see citing works
Self-citationProbabilistic sleep architecture models in patients with and without sleep apnea
BIANCHI, M. T., EISEMAN, N. A., CASH, S. S., MIETUS, J., PENG, C., & THOMAS, R. J. (2012) · Journal of Sleep Research · doi:10.1111/j.1365-2869.2011.00937.x · Cited by 44 · see citing works
Cade, B. E., Chen, H., Stilp, A. M., Louie, T., Ancoli-Israel, S., Arens, R., Barfield, R., Below, J. E., Cai, J., Conomos, M. P., et al. (2019) · PLOS Genetics · doi:10.1371/journal.pgen.1007739 · Cited by 39 · see citing works
Self-citationBudhiraja, R., Javaheri, S., Parthasarathy, S., Berry, R. B., & Quan, S. F. (2019) · Journal of Clinical Sleep Medicine · doi:10.5664/jcsm.7916 · Cited by 33 · see citing works
Newman, A. B. (2005) · Archives of Internal Medicine · doi:10.1001/archinte.165.20.2408 · Cited by 5 · see citing works
- Authors
- Whitney, C. W., Gottlieb, D. J., Redline, S., Norman, R. G., Dodge, R. R., Shahar, E., Surovec, S., & Nieto, F. J.
- Affiliations
- University of Washington (US)
- Boston University (US)
- Case Western Reserve University (US)
- NYU Langone Health (US)
- University of Arizona (US)
- University of Minnesota (US)
- Johns Hopkins University (US)
- Corresponding author
- Whitney, Coralyn W.
- Venue
- Sleep
- Publisher
- Oxford University Press (OUP)
- Published
- 1998-10
- Volume
- 21
- Issue
- 7
- Pages
- 749-757
- ISSN
- 0161-8105 (ISSN-L)
- 1550-9109
- OpenAlex
- W187647308
- Semantic Scholar
- 39128048443d1d57281002fbebbaa7e554611ffc
- Cited by
- 327
- References
- 14
- Citations by year
- 2026: 5 · 2025: 9 · 2024: 15 · 2023: 16 · 2022: 10 · 2021: 28 · 2020: 22 · 2019: 14 · 2018: 11 · 2017: 12 · 2016: 6 · 2015: 13 · 2014: 4 · 2013: 11 · 2012: 13
- FWCI
- 3.07
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- SLEEP (publishedVersion)
- PubMed (publishedVersion)
- Funders
- University of Minnesota (U01HL53941, U01HL53940, U01HL539 31, U01HL53934, U01HL53937, U01HL53916)
- University of Washington (U01HL53941, U01HL53940, U01HL53916, U01HL53931, U01HL53934)
- Johns Hopkins University (U01HL53937, U01HL53916, U01HL53941, U01HL53931, U01HL53940, U01HL53934)
- University of Pittsburgh
- University of Oklahoma
- Case Western Reserve University (U01HL53934, U01HL53931, U01HL53916, U01HL53940, U01HL53941)
- Strong
- York University
- National Institutes of Health (5u01hl053940-05)
- University of California, Davis (U01HL53941, U01HL53931, U01HL53940, U01HL53916, U01HL53934)
- National Heart, Lung, and Blood Institute (U01 HL 53940, U01HL053941, U01 HL53934, U01 HL 53941, U01 HL53916, U01 HL53931, U0HL53938, U01-HL53937)
- Topics
- Health Sciences › Medicine › Physiology › Obstructive Sleep Apnea Research
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Sleep and related disorders
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Keywords
- sleep stage classification, respiratory disturbance index, sleep-disordered breathing, arousal index, oxygen desaturation, EEG arousal, Sleep Heart Health Study
- MeSH terms
- Humans, Research Design, Sleep Apnea Syndromes, Sleep Stages, Reproducibility of Results, Polysomnography
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
