Disk scrubbing in large archival storage systems
The DOI matches its Crossref record; 17 of the 17 references it cites were found in a registry or an index.
Abstract
Large archival storage systems experience long periods of idleness broken up by rare data accesses. In such systems, disks may remain powered off for long periods of time. These systems can lose data for a variety of reasons, including failures at both the device level and the block level. To deal with these failures, we must detect them early enough to be able to use the redundancy built into the storage system. We propose a process called "disk scrubbing" in a system in which drives are periodically accessed to detect drive failure. By scrubbing all of the data stored on all of the disks, we can detect block failures and compensate for them by rebuilding the affected blocks. Our research shows how the scheduling of disk scrubbing affects overall system reliability, and that "opportunistic" scrubbing, in which the system scrubs disks only when they are powered on for other reasons, performs very well without the need to power on disks solely to check them.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Existing archival storage solutions, such as magnetic tapes and hard disk drives, suffer from limited media lifespans that render them unsuitable for long-term data retention1–3.”
From Laser writing in glass for dense, fast and efficient archival data storage (2026) · cites this paper as [3]
In the citing paper
“Most digital archive systems rely on media that degrade within a few years1–3.”
From Laser writing in glass for dense, fast and efficient archival data storage (2026) · cites this paper as [3]
Citations
Cites
Efficient randomized pattern-matching algorithms
Karp, R. M., & Rabin, M. O. (1987) · IBM Journal of Research and Development · doi:10.1147/rd.312.0249 · Cited by 1,269
A tutorial on Reed-Solomon coding for fault-tolerance in RAID-like systems
Plank, J. S. (1997) · Software: Practice and Experience · doi:10.1002/(sici)1097-024x(199709)27:9<995::aid-spe111>3.0.co;2-6 · Cited by 525 · cited 1×
“We use the “algebraic signature” proposed by Litwin and Schwarz [8, 9] that has all the properties, but for property 5, which it only has for Reed-Solomon codes [10, 11], a convolutional array code, or a code using XOR to calculate parity as in Hellerstein, et al.”
MethodMassive Arrays of Idle Disks for Storage Archives
Colarelli, D., & Grunwald, D. C. (2002) · Cited by 339
Some applications of Rabin’s fingerprinting method
Broder, A. Z. (1993) · Sequences II · doi:10.1007/978-1-4613-9323-8_11 · Cited by 204
Massive Arrays of Idle Disks For Storage Archives
Colarelli, D., & Grunwald, D. (2002) · ACM/IEEE SC 2002 Conference (SC'02) · doi:10.1109/sc.2002.10058 · Cited by 198 · cited 4×
“Colarelli and Grunwald [2] called such a system a Massive Array of mainly Idle Disks (MAID).”
BackgroundInfluentialReliability mechanisms for very large storage systems
Xin, Q., Miller, E. L., Schwarz, T., Long, D. D. E., Brandt, S. A., & Litwin, W. (2003) · 20th IEEE/11th NASA Goddard Conference on Mass Storage Systems and Technologies, 2003. (MSST 2003). Proceedings. · doi:10.1109/mass.2003.1194851 · Cited by 190 · cited 1×
“Previous research [15] discusses the advantages and disadvantages of using complete disks or smaller r-blocks for reliability and storage system operations.”
BackgroundSelf-citationMore than an interface: scsi vs. ata
Anderson, D., Dykes, J., & Riedel, E. (2003) · File and Storage Technologies · Cited by 144
Honicky, R. J., & Miller, E. L. (2004) · IEEE International Parallel and Distributed Processing Symposium (IPDPS) · doi:10.1109/ipdps.2004.1303042 · Cited by 144 · cited 1×
“Data is distributed across 10,000 disk drives evenly using 10 GB reliability blocks; the placement algorithm [5] guarantees that the original data and its replicas or parity blocks will never be put on the same drive.”
MethodSelf-citationCoding techniques for handling failures in large disk arrays
Hellerstein, L., Gibson, G. A., Karp, R. M., Katz, R. H., & Patterson, D. A. (1994) · Algorithmica · doi:10.1007/bf01185210 · Cited by 143 · cited 1×
“[4].”
Evaluation of distributed recovery in large-scale storage systems
Xin, Q., Miller, E. L., & Schwarz, S. J. T. (2004) · doi:10.1109/hpdc.2004.12 · Cited by 75
Self-citation
Research works printed but not linked: 4
- 0, Self-Monitoring Analysis and Reporting Technology (SMART) Disk Drive Monitoring Tools
- 2004, Seagate, Estimating Drive Reliability in Desktop Computers and Consumer Electronic Systems
- 10.1002/(sici)1097-024x(199709)27:9<995::aid-spe111>3.3.co;2-y · doi:10.1002/(sici)1097-024x(199709)27:9<995::aid-spe111>3.3.co;2-y
- litwin, 2003, Disk backup through algebraic signatures in scalable and distributed data structures, Proceedings of the 5th Workshop on Distributed Data and Structures (WDAS 2003)
Cited by
Items 101–131 of 131
POTSHARDS: Secure Long-Term Archival Storage Without Encryption
Storer, M. W., Greenan, K. M., & Miller, E. L. (2006) · Cited by 2
Self-citationA Novel Cost-Effective Disk Scrubbing Scheme
Liu, J., Zhou, K., Pang, L., Wang, Z., Deng, Y., & Feng, D. (2009) · International Joint Conference on INC, IMS and IDC (NCM) · doi:10.1109/ncm.2009.26 · Cited by 2
SCAN: An Efficient Decoding Algorithm for RAID-6 Codes
Luo, J., & Xu, L. (2011) · IEEE International Symposium on Network Computing and Applications (NCA) · doi:10.1109/nca.2011.20 · Cited by 2
“proposed a technique called disk scrubbing [7], which runs a background process and proactively scans disk sectors masking failed ones.”
MethodOn the Possibility of Small, Service-Free Disk Based Storage Systems
Jehan-Francois, P., & Schwarz, T. (2008) · International Conference on Availability, Reliability and Security (ARES) · doi:10.1109/ares.2008.98 · Cited by 2
Self-citationProtecting Single Shingled Write Drives Against Latent Sector Failures
Schwarz, T. (2018) · ACM International Systems and Storage Conference (SYSTOR) · doi:10.1145/3211890.3211893 · Cited by 2
“Scrubbing [2, 15, 20] periodically reads all sectors in a disk and thereby detects latent sector failures, whose data then can be recovered.”
BackgroundSelf-citationLimiting trust in the storage stack
Bairavasundaram, L. N., Rungta, M., Arpaci-Dusseau, A. C., & Arpaci-Dusseau, R. H. (2006) · Workshop On Storage Security And Survivability · doi:10.1145/1179559.1179569 · Cited by 2
“In addition, the mechanical components of the disk can cause media scratches to occur resulting in latent sector errors thereby rendering the data inaccessible [7, 16].”
BackgroundAchieving high data reliability at low scrubbing cost via failure-aware scrubbing
Jiang, T., Huang, P. S., & Zhou, K. (2020) · Journal of Parallel and Distributed Computing · doi:10.1016/j.jpdc.2020.05.007 · Cited by 1
Research on Reliability Design of Data Storage for Embedded System
Liu, C., & Fang, H. (2012) · Physics Procedia · doi:10.1016/j.phpro.2012.03.253 · Cited by 1
Scrubbing in Storage Virtualization Platform for Long-Term Backup Application
Ma, A., Yin, Y., Na, W., Meng, X., Bu, Q., & Xu, L. X. (2009) · International Conference on Availability, Reliability and Security (ARES) · doi:10.1109/ares.2009.87 · Cited by 1
“Previous study proposes two methods to deal with the latent sector error: scrubbing [12] and intra-disk redundancy [13].”
BackgroundMethodInfluentialEfficient Management of System Idleness
Mi, N., Riska, A., Zhang, Q., Smirni, E., & Riedel, E. (2008) · Cited by 1
Schwarz, T., Amer, A., & Paris, J.-F. (2015) · IEEE Pacific Rim International Symposium on Dependable Computing (PRDC) · doi:10.1109/prdc.2015.16 · Cited by 1
Self-citationAn Efficient and Parallel Bad Block Checker for Parallelism of Storage Devices
Han, J., Zhu, G., Lee, E., & Son, Y. (2020) · IEEE International Conference on Autonomic Computing and Self-Organizing Systems Companion (ACSOS-C) · doi:10.1109/acsos-c51401.2020.00022 · Cited by 1
“[28] propose the disk scrubbing to periodically access the entire storage to detect bad blocks.”
BackgroundDesigning Reliable High-Performance Storage Systems for HPC Environments
Hindman, L. S. (2011) · ScholarWorks (Boise State University) · Cited by 1
Thermal Management and Data Archiving in Data Centers
Chen, Y. (2016) · Cited by 1
Kadekodi, S. (2021) · Figshare · doi:10.1184/r1/14461768 · Cited by 2
Arquivamento digital a longo prazo baseado em seleção de repositórios em redes peer-to-peer
Vignatti, T. (2009) · LA Referencia (Red Federada de Repositorios Institucionales de Publicaciones Científicas)
Hyfs: design and implementation of a reliable file system
Xu, L., & Luo, J. (2011)
Paired-replicas with constant repair time: Loss functions and memorylessness
Deolalikar, V. (2013) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata.2013.6691657
“[8] study the same problem as [14], but in the enterprise setting where disks are mostly powered up.”
BackgroundPaluska, J. M., & Ward, S. (2010) · IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops) · doi:10.1109/percomw.2010.5470593
“Penalties prescribed by a chit are a proxy for the value of data; risks to the data may come from anywhere in the hardware/software stack.”
BackgroundSecure, energy-efficient, evolvable, long-term archival storage
Miller, E. L., & Storer, M. W. (2009)
Self-citationDistributed Detecting and Real-Time Advertising System of Equipment Failure Based on MANET
Qin, X., & Yi, T. (2012) · Advances in intelligent and soft computing · doi:10.1007/978-3-642-27945-4_55
“GPRS (General Packet Radio Service) is developed based on existing GSM network, a data communications network that wireless packet-switched technologies to provide end to end wireless IP connectivity [6].”
BackgroundDaci, G., & Bezhani, A. (2012)
Zhang, Y. (n.d.) · Cited by 1
Ullah, M. S., & McGough, A. S. (2020) · International Conference on Computing Advancements (ICCA) · doi:10.1145/3377049.3377091
“Alternatively, [14] initiated the proposal for disk scrubbing for early disk failure detection in one of his researches which is a distinct work then all we considered in our DDS extension.”
BackgroundMethodWang, X., Wang, Q., Zhang, B., He, J., & Huo, Z. (2022) · IEICE Electronics Express · doi:10.1587/elex.19.20220198
A Reliability Framework for Proactive-Tolerance Reed-Solomon Storage Systems*
Li, J., Zhou, Z., & Ding, J. (2024) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/smc54092.2024.10831258
Impact of Data Organization on Distributed Storage System
Gryz, L. (2012)
Parity Resynchronization using a Block-level Journaling for Software RAID
Baek, S. H., & Park, K.-W. (2015) · Information Systems · doi:10.1016/j.is.2015.05.004
“Research topics on RAID include access method [2,3], reconstruction [4–7], scrubbing [8–10], scaling [11–15], data layout [16–21], erasure code [22–26], and resynchronization [27–29].”
MethodA temporal logic of robustness
McCabe-Dansted, J. C. (2011) · UWA Profiles and Research Repository (University of Western Australia)
Reliability Analysis for Multi-cloud Auditing Systems
Yan, M., Sun, Y., & Wang, G. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-6837-5_28
Metadata
- Authors
- Schwarz, T. J. E., Xin, Q., Miller, E. L., Long, D. D. E., Hospodor, A., & Ng, S.
- Affiliations
- Santa Clara University (US)
- University of California, Santa Cruz (US)
- Venue
- The IEEE Computer Society's 12th Annual International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunications Systems, 2004. (MASCOTS 2004). Proceedings.
- Publisher
- IEEE
- Published
- 2004-11-08
- Pages
- 409-418
- ISSN
- 1526-7539 (ISSN-L)
- OpenAlex
- W2120396387
- Cited by
- 128
- References
- 17
- Citations by year
- 2026: 2 · 2024: 4 · 2023: 1 · 2022: 1 · 2020: 2 · 2019: 3 · 2018: 2 · 2017: 2 · 2016: 5 · 2015: 10 · 2014: 9 · 2013: 3 · 2012: 12
- FWCI
- 3.01
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- Proceedings - International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems (publishedVersion)
- eScholarship (California Digital Library) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Networks and Communications › Advanced Data Storage Technologies
- Physical Sciences › Computer Science › Computer Networks and Communications › Distributed systems and fault tolerance
- Physical Sciences › Engineering › Safety, Risk, Reliability and Quality › Reliability and Maintenance Optimization
- Keywords
- data redundancy, disk scheduling, storage reliability, data loss prevention
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.