Disk scrubbing in large archival storage systems
The DOI matches its Crossref record; 17 of the 17 references it cites were found in a registry or an index.
Abstract
Large archival storage systems experience long periods of idleness broken up by rare data accesses. In such systems, disks may remain powered off for long periods of time. These systems can lose data for a variety of reasons, including failures at both the device level and the block level. To deal with these failures, we must detect them early enough to be able to use the redundancy built into the storage system. We propose a process called "disk scrubbing" in a system in which drives are periodically accessed to detect drive failure. By scrubbing all of the data stored on all of the disks, we can detect block failures and compensate for them by rebuilding the affected blocks. Our research shows how the scheduling of disk scrubbing affects overall system reliability, and that "opportunistic" scrubbing, in which the system scrubs disks only when they are powered on for other reasons, performs very well without the need to power on disks solely to check them.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Existing archival storage solutions, such as magnetic tapes and hard disk drives, suffer from limited media lifespans that render them unsuitable for long-term data retention1–3.”
From Laser writing in glass for dense, fast and efficient archival data storage (2026) · cites this paper as [3]
In the citing paper
“Most digital archive systems rely on media that degrade within a few years1–3.”
From Laser writing in glass for dense, fast and efficient archival data storage (2026) · cites this paper as [3]
Citations
Cites
Efficient randomized pattern-matching algorithms
Karp, R. M., & Rabin, M. O. (1987) · IBM Journal of Research and Development · doi:10.1147/rd.312.0249 · Cited by 1,269
A tutorial on Reed-Solomon coding for fault-tolerance in RAID-like systems
Plank, J. S. (1997) · Software: Practice and Experience · doi:10.1002/(sici)1097-024x(199709)27:9<995::aid-spe111>3.0.co;2-6 · Cited by 525 · cited 1×
“We use the “algebraic signature” proposed by Litwin and Schwarz [8, 9] that has all the properties, but for property 5, which it only has for Reed-Solomon codes [10, 11], a convolutional array code, or a code using XOR to calculate parity as in Hellerstein, et al.”
MethodMassive Arrays of Idle Disks for Storage Archives
Colarelli, D., & Grunwald, D. C. (2002) · Cited by 339
Some applications of Rabin’s fingerprinting method
Broder, A. Z. (1993) · Sequences II · doi:10.1007/978-1-4613-9323-8_11 · Cited by 204
Massive Arrays of Idle Disks For Storage Archives
Colarelli, D., & Grunwald, D. (2002) · ACM/IEEE SC 2002 Conference (SC'02) · doi:10.1109/sc.2002.10058 · Cited by 198 · cited 4×
“Colarelli and Grunwald [2] called such a system a Massive Array of mainly Idle Disks (MAID).”
BackgroundInfluentialReliability mechanisms for very large storage systems
Xin, Q., Miller, E. L., Schwarz, T., Long, D. D. E., Brandt, S. A., & Litwin, W. (2003) · 20th IEEE/11th NASA Goddard Conference on Mass Storage Systems and Technologies, 2003. (MSST 2003). Proceedings. · doi:10.1109/mass.2003.1194851 · Cited by 190 · cited 1×
“Previous research [15] discusses the advantages and disadvantages of using complete disks or smaller r-blocks for reliability and storage system operations.”
BackgroundSelf-citationMore than an interface: scsi vs. ata
Anderson, D., Dykes, J., & Riedel, E. (2003) · File and Storage Technologies · Cited by 144
Honicky, R. J., & Miller, E. L. (2004) · IEEE International Parallel and Distributed Processing Symposium (IPDPS) · doi:10.1109/ipdps.2004.1303042 · Cited by 144 · cited 1×
“Data is distributed across 10,000 disk drives evenly using 10 GB reliability blocks; the placement algorithm [5] guarantees that the original data and its replicas or parity blocks will never be put on the same drive.”
MethodSelf-citationCoding techniques for handling failures in large disk arrays
Hellerstein, L., Gibson, G. A., Karp, R. M., Katz, R. H., & Patterson, D. A. (1994) · Algorithmica · doi:10.1007/bf01185210 · Cited by 143 · cited 1×
“[4].”
Evaluation of distributed recovery in large-scale storage systems
Xin, Q., Miller, E. L., & Schwarz, S. J. T. (2004) · doi:10.1109/hpdc.2004.12 · Cited by 75
Self-citation
Research works printed but not linked: 4
- 0, Self-Monitoring Analysis and Reporting Technology (SMART) Disk Drive Monitoring Tools
- 2004, Seagate, Estimating Drive Reliability in Desktop Computers and Consumer Electronic Systems
- 10.1002/(sici)1097-024x(199709)27:9<995::aid-spe111>3.3.co;2-y · doi:10.1002/(sici)1097-024x(199709)27:9<995::aid-spe111>3.3.co;2-y
- litwin, 2003, Disk backup through algebraic signatures in scalable and distributed data structures, Proceedings of the 5th Workshop on Distributed Data and Structures (WDAS 2003)
Cited by
Availability in globally distributed storage systems
Ford, D. A., Labelle, F., Popovici, F. I., Stokely, M., Truong, V., Barroso, L. A., Grimes, C., & Quinlan, S. (2010) · doi:10.5555/1924943.1924948 · Cited by 539
An analysis of latent sector errors in disk drives
Bairavasundaram, L. N., Goodson, G. R., Pasupathy, S., & Schindler, J. (2007) · Proceedings of the 2007 ACM SIGMETRICS international conference on Measurement and modeling of computer systems · doi:10.1145/1254882.1254917 · Cited by 296
“Similarly, understanding the incidence of latent sector errors can aid in choosing the most effective execution schedule of proactive reading and data verification, called disk scrubbing [18].”
BackgroundPrabhakaran, V., Bairavasundaram, L. N., Agrawal, N., Gunawi, H. S., Arpaci-Dusseau, A. C., & Arpaci-Dusseau, R. H. (2005) · ACM SIGOPS Operating Systems Review · doi:10.1145/1095809.1095830 · Cited by 189
A fresh look at the reliability of long-term digital storage
Baker, M., Shah, M., Rosenthal, D. S. H., Roussopoulos, M., Maniatis, P., Giuli, T., & Bungale, P. (2006) · Proceedings of the 1st ACM SIGOPS/EuroSys European Conference on Computer Systems 2006 · doi:10.1145/1217935.1217957 · Cited by 162
Pergamum: replacing tape with energy efficient, reliable, disk-based archival storage
Storer, M. W., Greenan, K. M., Miller, E. L., & Voruganti, K. (2008) · Cited by 140
Self-citationUnderstanding latent sector errors and how to protect against them
Schroeder, B., Damouras, S., & Gill, P. (2010) · ACM Transactions on Storage · doi:10.1145/1837915.1837917 · Cited by 131
“As a result, existing work typically relies on hypothetical assumptions, such as LSEs that follow a Poisson process [2, 7, 10, 17].”
BackgroundMethodInfluentialSafeStore: a durable and practical storage system
Kotla, R., Alvisi, L., & Dahlin, M. (2007) · Cited by 116
File systems unfit as distributed storage backends: lessons from 10 years of Ceph evolution
Aghayev, A., Weil, S. A., Kuchnik, M., Nelson, M., Ganger, G. R., & Amvrosiadis, G. (2019) · ACM SIGOPS Symposium on Operating Systems Principles (SOSP) · doi:10.1145/3341301.3359656 · Cited by 104
Enhanced Reliability Modeling of RAID Storage Systems
Elerath, J. G., & PECHT, M. G. (2007) · International Conference on Dependable Systems & Networks/Proceedings · doi:10.1109/dsn.2007.41 · Cited by 99
“Data scrubbing [10], the remedy for latent defects, is also incorporated in the model.”
BackgroundInfluentialEnd-to-end data integrity for file systems: a ZFS case study
Zhang, Y., Rajimwale, A., Arpaci-Dusseau, A. C., & Arpaci-Dusseau, R. H. (2010) · File and Storage Technologies · doi:10.5555/1855511.1855514 · Cited by 91
WorkOut: I/O Workload Outsourcing for Boosting RAID Reconstruction Performance
Wu, S., Jiang, H., Feng, D., Tian, L., & Mao, B. (n.d.) · Cited by 89
Wear unleveling: improving NAND flash lifetime by balancing page endurance
Jimenez, X., Novo, D., & Ienne, P. (2014) · File and Storage Technologies · doi:10.5555/2591305.2591311 · Cited by 88
Parity lost and parity regained
Krioukov, A., Bairavasundaram, L. N., Goodson, G. R., Srinivasan, K., Thelen, R., Arpaci-Dusseau, A. C., & Arpaci-Dusseau, R. H. (2008) · Cited by 80
RAIDShield: Characterizing, Monitoring, and Proactively Protecting Against Disk Failures
Ma, A., Traylor, R., Douglis, F., Chamness, M., Lu, G., Sawyer, D., Chandra, S., & Hsu, W. (2015) · ACM Transactions on Storage · doi:10.1145/2820615 · Cited by 78
“Schwarz, et al. [2004] propose disk scrubbing to proactively detect latent sector errors.”
BackgroundDeduplication in SSDs: Model and quantitative analysis
Kim, J.-H., Lee, C., Lee, S. Y., Son, I., Choi, J., Yoon, S., Lee, H.-U., Kang, S., Won, Y., & Cha, J. (2012) · Digest of papers - IEEE Symposium on Mass Storage Systems · doi:10.1109/msst.2012.6232379 · Cited by 72
STAIR Codes: A General Family of Erasure Codes for Tolerating Device and Sector Failures
Li, M., & Lee, P. P. C. (2014) · ACM Transactions on Storage · doi:10.1145/2658991 · Cited by 74
“One scheme is scrubbing [24, 36, 38], which proactively scans all disks and recovers any spotted sector failure using the underlying erasure codes.”
Providing High Reliability in a Minimum Redundancy Archival Storage System
Bhagwat, D., Pollack, K. T., Long, D. D. E., Schwarz, T. J. E., Miller, E. L., & Paris, J.-F. (2006) · Proceedings - International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems · doi:10.1109/mascots.2006.42 · Cited by 70
“The surplus storage space we save by using interfile compression can be used to implement proactive policies for ensuring reliability [31], verifying the data integrity [27], and developing recovery strategies [32] for large scale storage systems.”
MethodSelf-citationRAIDShield: Characterizing, Monitoring, and Proactively Protecting Against Disk Failures
Ma, A., Douglis, F., Lu, G., Sawyer, D., Emc Corporation, Douglis, F., Lu, G., Ch, S., & Hsu, W. (2015) · Cited by 69
Higher reliability redundant disk arrays: Organization, operation, and coding
Thomasian, A., & Blaum, M. (2009) · ACM Transactions on Storage · doi:10.1145/1629075.1629076 · Cited by 63
“Media failures can be dealt with by disk scrubbing, which systematically reads disk sectors and reconstructs and relocates unreadable blocks [Kari 1997; Schwarz et al. 2004; Iliadis et al. 2008].”
BackgroundDisk scrubbing versus intra-disk redundancy for high-reliability raid storage systems
Iliadis, I., Haas, R., Hu, X.-Y., & Eleftheriou, E. S. (2008) · ACM SIGMETRICS International Conference on Measurement and Modeling of Computer Systems (SIGMETRICS) · doi:10.1145/1375457.1375485 · Cited by 59
A Highly Accurate Method for Assessing Reliability of Redundant Arrays of Inexpensive Disks (RAID)
Elerath, J. G., & PECHT, M. G. (2008) · IEEE Transactions on Computers · doi:10.1109/tc.2008.163 · Cited by 59
“Most often, the system reliability and number of DDFs, leading to a loss of data, are estimated using the MTTDL [1], [2], [7], [8], [9], [10], [11], a rather straightforward expression that assumes the time-to-failure and time-torestore distributions are exponentially distributed with constant failure and restoration…”
BackgroundMethodInfluentialHard Disk Drives: The Good, the Bad and the Ugly!
Elerath, J. G. (2007) · Queue · doi:10.1145/1317394.1317403 · Cited by 44
Efficient management of idleness in storage systems
Mi, N., Riska, A., Zhang, Q., Smirni, E., & Riedel, E. (2009) · ACM Transactions on Storage · doi:10.1145/1534912.1534913 · Cited by 41
Estimation of deduplication ratios in large data sets
Harnik, D., Margalit, O., Naor, D., Sotnikov, D., & Vernik, G. (2012) · Digest of papers - IEEE Symposium on Mass Storage Systems · doi:10.1109/msst.2012.6232381 · Cited by 39
“This data was used in the chunk based tests only, and not in the full-file deduplication case, since it only contains 13 large files.”
MethodAdding aggressive error correction to a high-performance compressing flash file system
Kang, Y., & Miller, E. L. (2009) · ACM International Conference on Embedded Software (EMSOFT) · doi:10.1145/1629335.1629376 · Cited by 32
Self-citationProtecting against rare event failures in archival systems
Wildani, A., Schwarz, T. J. E., Miller, E. L., & Long, D. D. E. (2009) · Proceedings - International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems · doi:10.1109/mascot.2009.5366825 · Cited by 30
Self-citationMinimizing Retention Induced Refresh Through Exploiting Process Variation of Flash Memory
Di, Y., Shi, L., Gao, C., Li, Q., Xue, C. J., & Wu, K. (2018) · IEEE Transactions on Computers · doi:10.1109/tc.2018.2858771 · Cited by 29
“types of refresh schemes widely studied in previous works, fixed-period refresh [16], [17], [18], [19] and demand based refresh [21], [22], [23], [24].”
BackgroundParity Cloud Service: A Privacy-Protected Personal Data Recovery Service
Song, C.-W., Park, S., Kim, D., & Kang, S. (2011) · IEEE International Conference on Trust, Security and Privacy in Computing and Communications (TrustCom) · doi:10.1109/trustcom.2011.107 · Cited by 29
“In [11], they argue that the actual mean-time-to-disk-failure is 3/2 times the MTTF.”
BackgroundAnalyzing the effects of disk-pointer corruption
Bairavasundaram, L. N., Rungta, M., Agrawa, N., Arpaci-Dusseau, A. C., Arpaci-Dusseau, R. H., & Swift, M. M. (2008) · International Conference on Dependable Systems & Networks/Proceedings · doi:10.1109/dsn.2008.4630121 · Cited by 29
Reliability and power-efficiency in erasure-coded storage systems
Greenan, K. M. (2009) · Cited by 29
Proactive Data Migration for Improved Storage Availability in Large-Scale Data Centers
Wu, S., Jiang, H., & Mao, B. (2014) · IEEE Transactions on Computers · doi:10.1109/tc.2014.2366734 · Cited by 28
“reconstruction [12], [28], [34], [38], [41], RAID re-synchronization [7], [42], disk scrubbing [15], [24], [31] and RAID reshape [21].”
Design of an exact data deduplication cluster
Kaiser, J., Meister, D., Brinkmann, A., & Effert, S. (2012) · Digest of papers - IEEE Symposium on Mass Storage Systems · doi:10.1109/msst.2012.6232380 · Cited by 30
Opportunistic storage maintenance
Amvrosiadis, G., Brown, A. D., & Goel, A. (2015) · ACM SIGOPS Symposium on Operating Systems Principles (SOSP) · doi:10.1145/2815400.2815424 · Cited by 26
“To protect against data loss due to silent data corruption [10, 11, 37], commercial storage systems rely on scrubbing [44, 52].”
BackgroundA spin-up saved is energy earned: achieving power-efficient, erasure-coded storage
Greenan, K. M., Long, D. D. E., Miller, E. L., Schwarz, S. J. T., & Wylie, J. J. (2008) · Cited by 26
Self-citationLaser writing in glass for dense, fast and efficient archival data storage
Microsoft Research Project Silica Team, Allison, J., Anderson, P., Aranas, E., Assaf, Y., Black, R., Caballero, M., Canakci, B., Chattaway, J. A., Chatzieleftheriou, A., et al. (2026) · Nature · doi:10.1038/s41586-025-10042-w · Cited by 25
Restrained utilization of idleness for transparent scheduling of background tasks
Mi, N., Riska, A., Li, X., Smirni, E., & Riedel, E. (2009) · ACM SIGMETRICS International Conference on Measurement and Modeling of Computer Systems (SIGMETRICS) · doi:10.1145/1555349.1555373 · Cited by 25
“These background activities aim at improving system performance and availability [22, 10, 23, 1, 2, 14, 19].”
BackgroundProject Silica: Towards Sustainable Cloud Archival Storage in Glass
Anderson, P., Aranas, E. B., Assaf, Y., Behrendt, R., Black, R., Caballero, M., Cameron, P., Canakci, B., Carvalho, T. D., Chatzieleftheriou, A., et al. (2023) · ACM SIGOPS Symposium on Operating Systems Principles (SOSP) · doi:10.1145/3600006.3613208 · Cited by 30
“Since bit rot occurs in both over time, storage systems use scrubbing [5, 38, 39] to guarantee data integrity.”
Exploiting Process Variation for Retention Induced Refresh Minimization on Flash Memory
Di, Y., Shi, L., Wu, K., & Xue, C. J. (2016) · Design, Automation, and Test in Europe · doi:10.3850/9783981537079_0343 · Cited by 23
“Currently, refresh optimization techniques have been studied to reduce the system impact, similar to the research works in DRAM, Phase Change Memory (PCM) and many other storage memories [13][14][15].”
MethodEnhancing data availability in disk drives through background activities
Mi, N., Riska, A., Smirni, E., & Riedel, E. (2008) · International Conference on Dependable Systems & Networks/Proceedings · doi:10.1109/dsn.2008.4630120 · Cited by 23
“Disk scrubbing is an error detection technique that detects latent sector errors via background media scans [19].”
BackgroundImproving Availability of RAID-Structured Storage Systems by Workload Outsourcing
Wu, S., Jiang, H., Feng, D., Tian, L., & Mao, B. (2010) · IEEE Transactions on Computers · doi:10.1109/tc.2010.206 · Cited by 23
“Many approaches have been proposed to improve the performance of the low-priority background tasks, such as RAID reconstruction [4], [14], [16], [27], [48], RAID resynchronization [9], and disk scrubbing [20], [35], [43].”
BackgroundMethodIncremental redundancy to reduce data retention errors in flash-based SSDs
Park, H., Kim, J., Choi, J., Lee, D., & Noh, S. H. (2015) · Digest of papers - IEEE Symposium on Mass Storage Systems · doi:10.1109/msst.2015.7208279 · Cited by 22
“To resolve these data retention errors, the FTL inside an SSD periodically refreshes data by performing data scrubbing operations [29].”
MethodPerformability Modeling for RAID Storage Systems by Markov Regenerative Process
Machida, F., Xia, R., & Trivedi, K. S. (2015) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2015.2502240 · Cited by 24
“The practical approach to cope with LSEs is disk scrubbing that continually scans the disk in order to detect LSEs proactively [21], [22].”
MethodRiska, A., & Riedel, E. (2008) · Cited by 21
Practical scrubbing: Getting to the bad sector at the right time
Amvrosiadis, G., Oprea, A., & Schroeder, B. (2012) · IEEE/IFIP International Conference on Dependable Systems and Networks (DSN 2012) · doi:10.1109/dsn.2012.6263919 · Cited by 20
“Other proposed algorithms focus on actions taken once an LSE is detected [10], [11], or scrubbing in archival systems with limited uptime [12].”
BackgroundFfsck: The Fast File-System Checker
Ma, A., Dragga, C., Arpaci-Dusseau, A. C., Arpaci-Dusseau, R. H., & McKusick, M. K. (2014) · ACM Transactions on Storage · doi:10.1145/2560011 · Cited by 19
“Checksums [6, 9, 40, 42] and scrubbing [37] can detect hardware failures and repair corrupted blocks with copies [29].”
BackgroundA Modeling Framework for Reliability of Erasure Codes in SSD Arrays
Kishani, M., Ahmadian, S., & Asadi, H. (2019) · IEEE Transactions on Computers · doi:10.1109/tc.2019.2962691 · Cited by 18
“However, in EXP BS and EXP BSWO states, bad symbol can be detected by read error or scrubbing [69] and be removed by reconstructing the data from parity of other devices, that moves the array to the operational state.”
BackgroundA Call for Research on Storage Emissions
Mcallister, S., Kazhamiaka, F., Berger, D. S., Fonseca, R., Frost, K., Ogus, A., Sah, M., Bianchini, R., Amvrosiadis, G., Beckmann, N., et al. (2024) · ACM SIGEnergy Energy Informatics Review · doi:10.1145/3727200.3727211 · Cited by 17
“These policies often do not work for common maintenance tasks that stream through large amounts of data sequentially, such as scrubbing [33, 117], and are hard to change in cloud environments where much of the work is not under the cloud provider’s control .”
Write amplification due to ECC on flash memory or leave those bit errors alone
Moon, S., & Reddy, A. L. N. (2012) · Digest of papers - IEEE Symposium on Mass Storage Systems · doi:10.1109/msst.2012.6232375 · Cited by 17
“Many studies have investigated the bit error failure behavior of SLC and MLC.”
BackgroundPACEMAKER: Avoiding HeART attacks in storage clusters with disk-adaptive redundancyPreprint
Kadekodi, S., Maturana, F., Subramanya, S. J., Yang, J., Rashmi, K. V., & Ganger, G. R. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.08191 · Cited by 23
Highly reliable two-dimensional RAID arrays for archival storage
Paris, J.-F., Schwarz, S. J. T., Amer, A., & Long, D. D. E. (2012) · IEEE International Performance, Computing, and Communications Conference · doi:10.1109/pccc.2012.6407774 · Cited by 16
Self-citationA simple equation for estimating reliability of an N+1 redundant array of independent disks (RAID)
Elerath, J. G. (2009) · International Conference on Dependable Systems & Networks/Proceedings · doi:10.1109/dsn.2009.5270303 · Cited by 16
“Markov process models have been used as an alternative to the MTTDL calculation and can include any number of additional states corresponding to conditions such as degraded operation or latent defects [17, 18, 19].”
MethodaHDFS: An Erasure-Coded Data Archival System for Hadoop Clusters
Chen, Y., Zhou, Y. X., Taneja, S., Qin, X., & Huang, J. (2017) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2017.2706686 · Cited by 16
Effect of Latent Errors on the Reliability of Data Storage Systems
Venkatesan, V., & Iliadis, I. (2013) · Proceedings - International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems · doi:10.1109/mascots.2013.38 · Cited by 16
Using device diversity to protect data against batch-correlated disk failures
Pâris, J.-F., & Long, D. D. E. (2006) · Workshop On Storage Security And Survivability · doi:10.1145/1179559.1179568 · Cited by 15
“This problem is especially acute in archival storage system, as most data will be infrequently accessed [1, 9].”
BackgroundSelf-citationProject Silica: Towards Sustainable Cloud Archival Storage in Glass
Anderson, P., Aranas, E., Assaf, Y., Behrendt, R., Black, R., Caballero, M., Cameron, P., Canakci, B., Chatzieleftheriou, A., Clarke, R., et al. (2025) · ACM Transactions on Storage · doi:10.1145/3708996 · Cited by 13
“Since bit rot occurs in both over time, storage systems use scrubbing [5, 48, 49] to guarantee data integrity.”
A HIERARCHICAL MARKOV RELIABILITY MODEL FOR DATA STORAGE SYSTEMS WITH MEDIA SELF-RECOVERY
Jin, T., Xing, L., & Yu, Y. (2011) · International Journal of Reliability Quality and Safety Engineering · doi:10.1142/s0218539311004019 · Cited by 12
RESAR: Reliable Storage at Exabyte Scale
Schwarz, S. J. T., Amer, A., Kroeger, T. M., Miller, E. L., Long, D. D. E., & Paris, J.-F. (2016) · Proceedings - International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems · doi:10.1109/mascots.2016.13 · Cited by 12
“This greater failure tolerance can be achieved by using stronger error-correcting codes, where each data block is covered by more than two parity blocks, but one can also use mechanisms such as disk scrubbing [26] and disk health monitoring [15] for disk-based storage systems to detect failures before they can do harm.”
BackgroundSelf-citationReliability of Disk Arrays with Double Parity
Schwarz, S. J. T., Long, D. D. E., & Paris, J.-F. (2013) · IEEE Pacific Rim International Symposium on Dependable Computing (PRDC) · doi:10.1109/prdc.2013.20 · Cited by 12
Self-citationA fresh look at the reliability of long-term digital storage
Baker, M. G., Shah, M. A., Rosenthal, D. S. H., Roussopoulos, M., Maniatis, P., Giuli, T. J., & Bungale, P. P. (2006) · ACM SIGOPS Operating Systems Review · doi:10.1145/1218063.1217957 · Cited by 12
Disaster recovery codes: increasing reliability with large-stripe erasure correcting codes
Greenan, K. M., Miller, E. L., Schwarz, T., & Long, D. D. E. (2007) · Workshop On Storage Security And Survivability · doi:10.1145/1314313.1314322 · Cited by 12
“[10] propose and analyze a mechanism, called disk scrubbing, used to actively check data integrity in large storage systems.”
BackgroundSelf-citationCharacteristics, impact, and tolerance of partial disk failures
Arpaci-Dusseau, A. C., Arpaci-Dusseau, R. H., & Bairavasundaram, L. N. (2008) · Cited by 11
Reliability challenges for storing exabytes
Amer, A., Long, D. D. E., & Schwarz, S. J. T. (2014) · 2012 International Conference on Computing, Networking and Communications (ICNC) · doi:10.1109/iccnc.2014.6785458 · Cited by 11
“Disk scrubbing can be used to detect latent errors before they can cause a data recovery operation to fail [17], whereas intra-disk redundancy masks latent errors completely [9], and idle read-after-write can detect latent write errors shortly after they occur [12].”
BackgroundSelf-citationUnderstanding data survivability in archival storage systems
Li, Y., Miller, E. L., & Long, D. D. E. (2012) · doi:10.1145/2367589.2367605 · Cited by 10
“Another application is adjusting the scrubbing interval [26] for various devices and systems to optimize DOS.”
BackgroundSelf-citationA data-assisted reliability model for carrier-assisted cold data storage systems
Arslan, Ş. S., Peng, J., & Goker, T. (2019) · Reliability Engineering & System Safety · doi:10.1016/j.ress.2019.106708 · Cited by 10
Evaluating the Impact of Irrecoverable Read Errors on Disk Array Reliability
Paris, J.-F., Amer, A., Long, D. D. E., & Schwarz, T. (2009) · IEEE Pacific Rim International Symposium on Dependable Computing (PRDC) · doi:10.1109/prdc.2009.66 · Cited by 10
“This recovery could be the result of a failed read access or a periodic scrubbing of the whole disk [16].”
BackgroundMethodInfluentialSelf-citationImproving Disk Array Reliability Through Expedited Scrubbing
Pâris, J.-F., Schwarz, T., Amer, A., & Long, D. D. E. (2010) · IEEE International Conference on Networking, Architecture and Storage (NAS) · doi:10.1109/nas.2010.37 · Cited by 9
“When a disk fails, a repair process is immediately initiated for that disk.”
BackgroundSelf-citationEvaluation of Applied Intra-disk Redundancy Schemes to Improve Single Disk Reliability
Grawinkel, M., Schäfer, T., Brinkmann, A., Hagemeyer, J., & Porrmann, M. (2011) · Proceedings - International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems · doi:10.1109/mascots.2011.13 · Cited by 10
“disk scrubbing to proactively find corrupt data [7], [8].”
BackgroundOnline availability upgrades for parity-based RAIDs through supplementary parity augmentations
Tian, L., Cao, Q., Jiang, H., Feng, D., Xie, C. M., & Xin, Q. (2011) · ACM Transactions on Storage · doi:10.1145/1970338.1970341 · Cited by 9
“In addition to these solutions, disk scrubbing [Schwarz et al. 2004] and Interleaved Parity Check (IPC) [Dholakia et al. 2008; Iliadis et al. 2008] are proposed to detect or tolerate latent sector errors.”
InfluentialSelf-citationDependability Analysis of Virtual Memory Systems
Bairavasundaram, L. N., Arpaci-Dusseau, A. C., & Arpaci-Dusseau, R. H. (2006) · International Conference on Dependable Systems & Networks/Proceedings · doi:10.1109/dsn.2006.26 · Cited by 9
“This technique is used in RAID systems to weed out potential bad blocks – the process is called “disk scrubbing” [20, 27].”
BackgroundMethodElerath, J. G. (2007) · University Libraries (University of Maryland) · Cited by 9
Modeling the Impact of Disk Scrubbing on Storage System
Liu, J., Zhou, K., Wang, Z., Pang, L., & Feng, D. (2010) · Journal of Computers · doi:10.4304/jcp.5.11.1629-1637 · Cited by 8
Dynamically Quantifying and Improving the Reliability of Distributed Storage Systems
Bachwani, R., Gryz, L., Bianchini, R., & Dubnicki, C. (2008) · IEEE Symposium on Reliable Distributed Systems (SRDS) · doi:10.1109/srds.2008.36 · Cited by 8
“Several previous papers have studied the reliability of large-scale storage systems, e.g. [3, 8, 12, 13, 14, 17, 18, 21].”
BackgroundGao, Y., Meister, D., & Brinkmann, A. (2010) · IEEE International Conference on Networking, Architecture and Storage (NAS) · doi:10.1109/nas.2010.11 · Cited by 7
“The detection of those failed sectors could be the result of a failed read access, a write access (by reading the old data) or a periodic scrubbing of disk [6], [15].”
BackgroundMethodUsing Storage Class Memory for Archives with {DAWN}, a Durable Array of Wimpy Nodes
Adams, I. F., Miller, E. L., & Rosenthal, D. S. H. (2011) · Cited by 7
Self-citationThe Reliability Model of a Distributed Data Storage in Case of Explicit and Latent Disk Faults
Ivanichkina, L., & Neporada, A. (2015) · Proceedings of the Institute for System Programming of RAS · doi:10.15514/ispras-2015-27(6)-16 · Cited by 7
Redundancy and Aging of Efficient Multidimensional MDS Parity-Protected Distributed Storage Systems
Arslan, Ş. S. (2014) · IEEE Transactions on Device and Materials Reliability · doi:10.1109/tdmr.2013.2293491 · Cited by 7
“The times between successive system failures are reported to be relatively larger than what conventional models suggest, even though each component disk hazard rate is increasing [11].”
BackgroundWan, L., Wang, F., Oral, S., Tiwari, D., Vazhkudai, S. S., & Cao, Q. (2015) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1145/2807591.2807615 · Cited by 6
“This observation indicates that in reality the failure rate of disk drives could be neither constant nor monotonically increasing or decreasing, which differs from what is usually assumed by many existing studies [7, 8, 11, 28].”
ResultDecoding STAR code for tolerating simultaneous disk failure and silent errors
Luo, J., Huang, C., & Xu, L. (2010) · International Conference on Dependable Systems & Networks/Proceedings · doi:10.1109/dsn.2010.5544948 · Cited by 7
On the Feasibility of Data Loss Insurance for Personal Cloud Storage.
Ma, X. (2014) · USENIX conference on Hot topics in storage and file systems · Cited by 6
Reliability Assurance of RAID Storage Systems for a Wide Range of Latent Sector Errors
Iliadis, I., & Hu, X.-Y. (2008) · IEEE International Conference on Networking, Architecture and Storage (NAS) · doi:10.1109/nas.2008.20 · Cited by 6
“Techniques such as disk scrubbing [13, 14] and intradisk redundancy [5, 6] have been proposed to enhance the reliability of RAID systems.”
MethodReliability modeling of RAID storage systems with latent errors
Iliadis, I. (2009) · Proceedings - International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems · doi:10.1109/mascot.2009.5366195 · Cited by 6
“A model for assessing the effect of disk scrubbing was presented in [6].”
BackgroundMethodEvaluating the Performability of Systems with Background Jobs
Zhang, Q., Mi, N., Smirni, E., Riska, A., & Riedel, E. (2006) · International Conference on Dependable Systems & Networks/Proceedings · doi:10.1109/dsn.2006.33 · Cited by 6
“Given that a system operates in low utilization, a myriad of approaches have been proposed aiming at utilizing idle time to improve performance [4, 3], fault tolerance [1], and reliability [17].”
BackgroundInfluentialPCS: a parity-based personal data recovery service in cloud
Jung, H., Park, Y., Song, C.-W., & Kang, S. (2017) · Cluster Computing · doi:10.1007/s10586-017-0805-8 · Cited by 5
“In [15], they argue that the actual mean-time-to-disk-failure is 1.5 times the MTTF.”
BackgroundInfluentialUnderstanding and coping with failures in large-scale storage systems
Xin, Q., & Miller, E. L. (2005) · Cited by 5
Self-citationPartial Disk Failures: Using Software to Analyze Physical Damage
Hai, H., & Shin, K. G. (2007) · Digest of papers - IEEE Symposium on Mass Storage Systems · doi:10.1109/msst.2007.4367973 · Cited by 5
Maintaining and checking parity in highly available Scalable Distributed Data Structures
Cieslicki, D., Schaeckeler, S., & Schwarz, T. (2009) · Journal of Systems and Software · doi:10.1016/j.jss.2009.10.013 · Cited by 4
“For example, Xin et al. (2004) show the importance of fast recovery for data availability in a very large storage system.”
BackgroundMethodInfluentialSelf-citationWarding off the dangers of data corruption with amulet
Borisov, N., Babu, S., Mandagere, N., & Uttamchandani, S. (2011) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/1989323.1989353 · Cited by 4
“Despite these mechanisms, recent literature [14] as well as plenty of anecdotal evidence show that problems due to corruption happen, and more frequently than expected [3, 22].”
BackgroundOn the impact of disk scrubbing on energy savings
Wang, G., Butt, A. R., & Gniady, C. (2008) · International Conference on Cluster Computing · Cited by 4
Implicit effect of decoding time on fault tolerance in erasure coded Cloud Storage systems
Safaei, B., Miremadi, S. G., & Chamazcoti, S. A. (2016) · International Computer Science and Engineering Conference · doi:10.1109/icsec.2016.7859937 · Cited by 4
“In [23], the BMTTF is calculated as 3×105 hours.”
MethodPreventing Silent Data Corruptions from Propagating During Data Reconstruction
Li, M., & Shu, J. (2010) · IEEE Transactions on Computers · doi:10.1109/tc.2010.36 · Cited by 4
“[50] uses signatures (which have some of the characteristics of hashes, a type of checksum) together with disk scrubbing to detect some forms of silent data corruptions.”
BackgroundXbft: byzantine fault tolerance with high performance, low cost, and aggressive fault isolation
Dahlin, M. D., & Kotla, R. (2008) · Cited by 4
Hardened Paxos through Consistency Validation
Barbieri, R. R., & Vieira, G. M. D. (2015) · Brazilian Symposium on Computing System Engineering (SBESC) · doi:10.1109/sbesc.2015.10 · Cited by 4
“1) Data and state redundancy [10], [12], [17], [18]: each process variable or stored data has a duplicate which can be used for validation and backup; 2) Checksums and hashes [10], [12], [18], [17]: generating a checksum or hash of data and attaching it to the protocol messages prior to transmitting across the network…”
MethodEffects of Data Scrubbing on Reliability in Storage Systems
Ryu, J., & Park, C. (2009) · IEICE Transactions on Information and Systems · doi:10.1587/transinf.e92.d.1639 · Cited by 3
“[2] recognized the data loss possibility due to the latent sector errors in the disk drives forming a MAID (Massive Array of mainly Idle Disks) system and proposed a disk scrubbing technique.”
BackgroundExamining Energy Use in Heterogeneous Archival Storage Systems
Adams, I. F., Miller, E. L., & Storer, M. W. (2010) · Proceedings - International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems · doi:10.1109/mascots.2010.38 · Cited by 3
“First, frequent spin-ups and spin-downs may physically impact the reliability of hardware when done at high frequency, but at relatively low rates such as that shown (effectively 2-3 times a day for most of our policies) its impact is minimal [23].”
BackgroundSelf-citationLi, J., Wu, J., & Ma, Z. (2015) · Tsinghua Science & Technology · doi:10.1109/tst.2015.7128935 · Cited by 3
Litwin, W., Mokadem, R., & Schwarz, T. (2007) · Springer eBooks · doi:10.1007/978-3-540-71661-7_14 · Cited by 3
“Recent work attempts to adapt popular string search algorithm to work on the encoded data appeared therefore recently for selected compression schemes, e.g., [ NT04 ].”
BackgroundSelf-citationTowards reliable storage systems
Arpaci-Dusseau, A. C., Arpaci-Dusseau, R. H., & Gunawi, H. S. (2009) · Cited by 2
Optimizing RAID for long term data archives
Klein, H., & Keller, J. (2010) · IEEE International Symposium on Parallel & Distributed Processing, Workshops and Phd Forum · doi:10.1109/ipdpsw.2010.5470870 · Cited by 2
“Optimizing RAID for Long Term Data Archives Henning Klein Fujitsu Technology Solutions GmbH Bürgermeister-Ulrich-Strasse 100”
MethodProject Silica: sustainable cloud archival storage in glass
Stefanovici, I. A., Anderson, P., Aranas, E. B., Black, R., Cabellero, M., Canakci, B., Chatzieleftheriou, A., Clegg, J., Cletheroe, D., Cooper, B., et al. (2024) · Frontiers in Ultrafast Optics: Biomedical, Scientific, and Industrial Applications · doi:10.1117/12.3010515 · Cited by 2
Incorporating solid state drives into distributed storage systems
Brandt, S., & Wacha, R. (2012) · Cited by 2
Metadata
- Authors
- Schwarz, T. J. E., Xin, Q., Miller, E. L., Long, D. D. E., Hospodor, A., & Ng, S.
- Affiliations
- Santa Clara University (US)
- University of California, Santa Cruz (US)
- Venue
- The IEEE Computer Society's 12th Annual International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunications Systems, 2004. (MASCOTS 2004). Proceedings.
- Publisher
- IEEE
- Published
- 2004-11-08
- Pages
- 409-418
- ISSN
- 1526-7539 (ISSN-L)
- OpenAlex
- W2120396387
- Cited by
- 128
- References
- 17
- Citations by year
- 2026: 2 · 2024: 4 · 2023: 1 · 2022: 1 · 2020: 2 · 2019: 3 · 2018: 2 · 2017: 2 · 2016: 5 · 2015: 10 · 2014: 9 · 2013: 3 · 2012: 12
- FWCI
- 3.01
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- Proceedings - International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems (publishedVersion)
- eScholarship (California Digital Library) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Networks and Communications › Advanced Data Storage Technologies
- Physical Sciences › Computer Science › Computer Networks and Communications › Distributed systems and fault tolerance
- Physical Sciences › Engineering › Safety, Risk, Reliability and Quality › Reliability and Maintenance Optimization
- Keywords
- data redundancy, disk scheduling, storage reliability, data loss prevention
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.