Disk scrubbing in large archival storage systems
The DOI matches its Crossref record; 17 of the 17 references it cites were found in a registry or an index.
Abstract
Large archival storage systems experience long periods of idleness broken up by rare data accesses. In such systems, disks may remain powered off for long periods of time. These systems can lose data for a variety of reasons, including failures at both the device level and the block level. To deal with these failures, we must detect them early enough to be able to use the redundancy built into the storage system. We propose a process called "disk scrubbing" in a system in which drives are periodically accessed to detect drive failure. By scrubbing all of the data stored on all of the disks, we can detect block failures and compensate for them by rebuilding the affected blocks. Our research shows how the scheduling of disk scrubbing affects overall system reliability, and that "opportunistic" scrubbing, in which the system scrubs disks only when they are powered on for other reasons, performs very well without the need to power on disks solely to check them.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Existing archival storage solutions, such as magnetic tapes and hard disk drives, suffer from limited media lifespans that render them unsuitable for long-term data retention1–3.”
From Laser writing in glass for dense, fast and efficient archival data storage (2026) · cites this paper as [3]
In the citing paper
“Most digital archive systems rely on media that degrade within a few years1–3.”
From Laser writing in glass for dense, fast and efficient archival data storage (2026) · cites this paper as [3]
Citations
Cites
Efficient randomized pattern-matching algorithms
Karp, R. M., & Rabin, M. O. (1987) · IBM Journal of Research and Development · doi:10.1147/rd.312.0249 · Cited by 1,269
A tutorial on Reed-Solomon coding for fault-tolerance in RAID-like systems
Plank, J. S. (1997) · Software: Practice and Experience · doi:10.1002/(sici)1097-024x(199709)27:9<995::aid-spe111>3.0.co;2-6 · Cited by 525 · cited 1×
“We use the “algebraic signature” proposed by Litwin and Schwarz [8, 9] that has all the properties, but for property 5, which it only has for Reed-Solomon codes [10, 11], a convolutional array code, or a code using XOR to calculate parity as in Hellerstein, et al.”
MethodMassive Arrays of Idle Disks for Storage Archives
Colarelli, D., & Grunwald, D. C. (2002) · Cited by 339
Some applications of Rabin’s fingerprinting method
Broder, A. Z. (1993) · Sequences II · doi:10.1007/978-1-4613-9323-8_11 · Cited by 204
Massive Arrays of Idle Disks For Storage Archives
Colarelli, D., & Grunwald, D. (2002) · ACM/IEEE SC 2002 Conference (SC'02) · doi:10.1109/sc.2002.10058 · Cited by 198 · cited 4×
“Colarelli and Grunwald [2] called such a system a Massive Array of mainly Idle Disks (MAID).”
BackgroundInfluentialReliability mechanisms for very large storage systems
Xin, Q., Miller, E. L., Schwarz, T., Long, D. D. E., Brandt, S. A., & Litwin, W. (2003) · 20th IEEE/11th NASA Goddard Conference on Mass Storage Systems and Technologies, 2003. (MSST 2003). Proceedings. · doi:10.1109/mass.2003.1194851 · Cited by 190 · cited 1×
“Previous research [15] discusses the advantages and disadvantages of using complete disks or smaller r-blocks for reliability and storage system operations.”
BackgroundSelf-citationMore than an interface: scsi vs. ata
Anderson, D., Dykes, J., & Riedel, E. (2003) · File and Storage Technologies · Cited by 144
Coding techniques for handling failures in large disk arrays
Hellerstein, L., Gibson, G. A., Karp, R. M., Katz, R. H., & Patterson, D. A. (1994) · Algorithmica · doi:10.1007/bf01185210 · Cited by 143 · cited 1×
“[4].”
Honicky, R. J., & Miller, E. L. (2004) · IEEE International Parallel and Distributed Processing Symposium (IPDPS) · doi:10.1109/ipdps.2004.1303042 · Cited by 128 · cited 1×
“Data is distributed across 10,000 disk drives evenly using 10 GB reliability blocks; the placement algorithm [5] guarantees that the original data and its replicas or parity blocks will never be put on the same drive.”
MethodSelf-citationEvaluation of distributed recovery in large-scale storage systems
Xin, Q., Miller, E. L., & Schwarz, S. J. T. (2004) · doi:10.1109/hpdc.2004.12 · Cited by 75
Self-citationSuel, T., Noel, P., & Trendafilov, D. (2004) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde.2004.1319992 · Cited by 58 · cited 1×
“Property 3 is related to the notion of composable hash functions [14].”
BackgroundAlgebraic signatures for scalable distributed data structures
Litwin, W., & Schwarz, T. J. E. (2004) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde.2004.1320015 · Cited by 52 · cited 1×
“This step validates the contents of the s-block, whereas disk scrubbing only insures that we can read the individual disk blocks that comprise an s-block.”
BackgroundSelf-citationLatent Sector Faults and Reliability of Disk Arrays
Kari, H. H. (2005) · Aaltodoc (Aalto University) · Cited by 38
A Parity Structure for Large Remotely Located Replicated Data Files
Metzner (1983) · IEEE Transactions on Computers · doi:10.1109/tc.1983.1676310 · Cited by 35
Disk drive reliability case study: dependence upon head fly-height and quantity of heads
Elerath, J. G., & Shah, S. (2003) · Annual Reliability and Maintainability Symposium (RAMS) · doi:10.1109/rams.2003.1182057 · Cited by 20 · cited 1×
“For example, if a particle left over from the manufacturing process remains within the disk drive, then this particle will be periodical ly caught between a head and a surface, scratching the surface and damaging the head [3].”
BackgroundVerification of Parity Data in Large Scale Storage Systems.
Schwarz, T. (2004) · Parallel and Distributed Processing Techniques and Applications · Cited by 11
Self-citationOn the Methods to Detect Sector Faults of a Disk Subsystem
Kari, H. H., Saikkonen, H., & Lombardi, F. (1993) · Modeling, Analysis, and Simulation On Computer and Telecommunication Systems · Cited by 1
Research works printed but not linked: 4
- 0, Self-Monitoring Analysis and Reporting Technology (SMART) Disk Drive Monitoring Tools
- 2004, Seagate, Estimating Drive Reliability in Desktop Computers and Consumer Electronic Systems
- 10.1002/(sici)1097-024x(199709)27:9<995::aid-spe111>3.3.co;2-y · doi:10.1002/(sici)1097-024x(199709)27:9<995::aid-spe111>3.3.co;2-y
- litwin, 2003, Disk backup through algebraic signatures in scalable and distributed data structures, Proceedings of the 5th Workshop on Distributed Data and Structures (WDAS 2003)
Cited by
Availability in globally distributed storage systems
Ford, D. A., Labelle, F., Popovici, F. I., Stokely, M., Truong, V., Barroso, L. A., Grimes, C., & Quinlan, S. (2010) · doi:10.5555/1924943.1924948 · Cited by 539
An analysis of latent sector errors in disk drives
Bairavasundaram, L. N., Goodson, G. R., Pasupathy, S., & Schindler, J. (2007) · Proceedings of the 2007 ACM SIGMETRICS international conference on Measurement and modeling of computer systems · doi:10.1145/1254882.1254917 · Cited by 296
“Similarly, understanding the incidence of latent sector errors can aid in choosing the most effective execution schedule of proactive reading and data verification, called disk scrubbing [18].”
BackgroundPrabhakaran, V., Bairavasundaram, L. N., Agrawal, N., Gunawi, H. S., Arpaci-Dusseau, A. C., & Arpaci-Dusseau, R. H. (2005) · ACM SIGOPS Operating Systems Review · doi:10.1145/1095809.1095830 · Cited by 189
A fresh look at the reliability of long-term digital storage
Baker, M., Shah, M., Rosenthal, D. S. H., Roussopoulos, M., Maniatis, P., Giuli, T., & Bungale, P. (2006) · Proceedings of the 1st ACM SIGOPS/EuroSys European Conference on Computer Systems 2006 · doi:10.1145/1217935.1217957 · Cited by 162
Pergamum: replacing tape with energy efficient, reliable, disk-based archival storage
Storer, M. W., Greenan, K. M., Miller, E. L., & Voruganti, K. (2008) · Cited by 140
Self-citation
Metadata
- Authors
- Schwarz, T. J. E., Xin, Q., Miller, E. L., Long, D. D. E., Hospodor, A., & Ng, S.
- Affiliations
- Santa Clara University (US)
- University of California, Santa Cruz (US)
- Venue
- The IEEE Computer Society's 12th Annual International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunications Systems, 2004. (MASCOTS 2004). Proceedings.
- Publisher
- IEEE
- Published
- 2004-11-08
- Pages
- 409-418
- ISSN
- 1526-7539 (ISSN-L)
- OpenAlex
- W2120396387
- Cited by
- 128
- References
- 17
- Citations by year
- 2026: 2 · 2024: 4 · 2023: 1 · 2022: 1 · 2020: 2 · 2019: 3 · 2018: 2 · 2017: 2 · 2016: 5 · 2015: 10 · 2014: 9 · 2013: 3 · 2012: 12
- FWCI
- 3.01
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- Proceedings - International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems (publishedVersion)
- eScholarship (California Digital Library) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Networks and Communications › Advanced Data Storage Technologies
- Physical Sciences › Computer Science › Computer Networks and Communications › Distributed systems and fault tolerance
- Physical Sciences › Engineering › Safety, Risk, Reliability and Quality › Reliability and Maintenance Optimization
- Keywords
- data redundancy, disk scheduling, storage reliability, data loss prevention
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.