Disk scrubbing in large archival storage systems
The DOI matches its Crossref record; 17 of the 17 references it cites were found in a registry or an index.
Abstract
Large archival storage systems experience long periods of idleness broken up by rare data accesses. In such systems, disks may remain powered off for long periods of time. These systems can lose data for a variety of reasons, including failures at both the device level and the block level. To deal with these failures, we must detect them early enough to be able to use the redundancy built into the storage system. We propose a process called "disk scrubbing" in a system in which drives are periodically accessed to detect drive failure. By scrubbing all of the data stored on all of the disks, we can detect block failures and compensate for them by rebuilding the affected blocks. Our research shows how the scheduling of disk scrubbing affects overall system reliability, and that "opportunistic" scrubbing, in which the system scrubs disks only when they are powered on for other reasons, performs very well without the need to power on disks solely to check them.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Existing archival storage solutions, such as magnetic tapes and hard disk drives, suffer from limited media lifespans that render them unsuitable for long-term data retention1–3.”
From Laser writing in glass for dense, fast and efficient archival data storage (2026) · cites this paper as [3]
In the citing paper
“Most digital archive systems rely on media that degrade within a few years1–3.”
From Laser writing in glass for dense, fast and efficient archival data storage (2026) · cites this paper as [3]
Citations
Cites
Efficient randomized pattern-matching algorithms
Karp, R. M., & Rabin, M. O. (1987) · IBM Journal of Research and Development · doi:10.1147/rd.312.0249 · Cited by 1,269
A tutorial on Reed-Solomon coding for fault-tolerance in RAID-like systems
Plank, J. S. (1997) · Software: Practice and Experience · doi:10.1002/(sici)1097-024x(199709)27:9<995::aid-spe111>3.0.co;2-6 · Cited by 525 · cited 1×
“We use the “algebraic signature” proposed by Litwin and Schwarz [8, 9] that has all the properties, but for property 5, which it only has for Reed-Solomon codes [10, 11], a convolutional array code, or a code using XOR to calculate parity as in Hellerstein, et al.”
MethodMassive Arrays of Idle Disks for Storage Archives
Colarelli, D., & Grunwald, D. C. (2002) · Cited by 339
Some applications of Rabin’s fingerprinting method
Broder, A. Z. (1993) · Sequences II · doi:10.1007/978-1-4613-9323-8_11 · Cited by 204
Massive Arrays of Idle Disks For Storage Archives
Colarelli, D., & Grunwald, D. (2002) · ACM/IEEE SC 2002 Conference (SC'02) · doi:10.1109/sc.2002.10058 · Cited by 198 · cited 4×
“Colarelli and Grunwald [2] called such a system a Massive Array of mainly Idle Disks (MAID).”
BackgroundInfluentialReliability mechanisms for very large storage systems
Xin, Q., Miller, E. L., Schwarz, T., Long, D. D. E., Brandt, S. A., & Litwin, W. (2003) · 20th IEEE/11th NASA Goddard Conference on Mass Storage Systems and Technologies, 2003. (MSST 2003). Proceedings. · doi:10.1109/mass.2003.1194851 · Cited by 190 · cited 1×
“Previous research [15] discusses the advantages and disadvantages of using complete disks or smaller r-blocks for reliability and storage system operations.”
BackgroundSelf-citationMore than an interface: scsi vs. ata
Anderson, D., Dykes, J., & Riedel, E. (2003) · File and Storage Technologies · Cited by 144
Coding techniques for handling failures in large disk arrays
Hellerstein, L., Gibson, G. A., Karp, R. M., Katz, R. H., & Patterson, D. A. (1994) · Algorithmica · doi:10.1007/bf01185210 · Cited by 143 · cited 1×
“[4].”
Honicky, R. J., & Miller, E. L. (2004) · IEEE International Parallel and Distributed Processing Symposium (IPDPS) · doi:10.1109/ipdps.2004.1303042 · Cited by 128 · cited 1×
“Data is distributed across 10,000 disk drives evenly using 10 GB reliability blocks; the placement algorithm [5] guarantees that the original data and its replicas or parity blocks will never be put on the same drive.”
MethodSelf-citationEvaluation of distributed recovery in large-scale storage systems
Xin, Q., Miller, E. L., & Schwarz, S. J. T. (2004) · doi:10.1109/hpdc.2004.12 · Cited by 75
Self-citation
Research works printed but not linked: 4
- 0, Self-Monitoring Analysis and Reporting Technology (SMART) Disk Drive Monitoring Tools
- 2004, Seagate, Estimating Drive Reliability in Desktop Computers and Consumer Electronic Systems
- 10.1002/(sici)1097-024x(199709)27:9<995::aid-spe111>3.3.co;2-y · doi:10.1002/(sici)1097-024x(199709)27:9<995::aid-spe111>3.3.co;2-y
- litwin, 2003, Disk backup through algebraic signatures in scalable and distributed data structures, Proceedings of the 5th Workshop on Distributed Data and Structures (WDAS 2003)
Cited by
Availability in globally distributed storage systems
Ford, D. A., Labelle, F., Popovici, F. I., Stokely, M., Truong, V., Barroso, L. A., Grimes, C., & Quinlan, S. (2010) · doi:10.5555/1924943.1924948 · Cited by 539
An analysis of latent sector errors in disk drives
Bairavasundaram, L. N., Goodson, G. R., Pasupathy, S., & Schindler, J. (2007) · Proceedings of the 2007 ACM SIGMETRICS international conference on Measurement and modeling of computer systems · doi:10.1145/1254882.1254917 · Cited by 296
“Similarly, understanding the incidence of latent sector errors can aid in choosing the most effective execution schedule of proactive reading and data verification, called disk scrubbing [18].”
BackgroundPrabhakaran, V., Bairavasundaram, L. N., Agrawal, N., Gunawi, H. S., Arpaci-Dusseau, A. C., & Arpaci-Dusseau, R. H. (2005) · ACM SIGOPS Operating Systems Review · doi:10.1145/1095809.1095830 · Cited by 189
A fresh look at the reliability of long-term digital storage
Baker, M., Shah, M., Rosenthal, D. S. H., Roussopoulos, M., Maniatis, P., Giuli, T., & Bungale, P. (2006) · Proceedings of the 1st ACM SIGOPS/EuroSys European Conference on Computer Systems 2006 · doi:10.1145/1217935.1217957 · Cited by 162
Pergamum: replacing tape with energy efficient, reliable, disk-based archival storage
Storer, M. W., Greenan, K. M., Miller, E. L., & Voruganti, K. (2008) · Cited by 140
Self-citation
Metadata
- Authors
- Schwarz, T. J. E., Xin, Q., Miller, E. L., Long, D. D. E., Hospodor, A., & Ng, S.
- Affiliations
- Santa Clara University (US)
- University of California, Santa Cruz (US)
- Venue
- The IEEE Computer Society's 12th Annual International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunications Systems, 2004. (MASCOTS 2004). Proceedings.
- Publisher
- IEEE
- Published
- 2004-11-08
- Pages
- 409-418
- ISSN
- 1526-7539 (ISSN-L)
- OpenAlex
- W2120396387
- Cited by
- 128
- References
- 17
- Citations by year
- 2026: 2 · 2024: 4 · 2023: 1 · 2022: 1 · 2020: 2 · 2019: 3 · 2018: 2 · 2017: 2 · 2016: 5 · 2015: 10 · 2014: 9 · 2013: 3 · 2012: 12
- FWCI
- 3.01
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- Proceedings - International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems (publishedVersion)
- eScholarship (California Digital Library) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Networks and Communications › Advanced Data Storage Technologies
- Physical Sciences › Computer Science › Computer Networks and Communications › Distributed systems and fault tolerance
- Physical Sciences › Engineering › Safety, Risk, Reliability and Quality › Reliability and Maintenance Optimization
- Keywords
- data redundancy, disk scheduling, storage reliability, data loss prevention
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Disk scrubbing in large archival storage systems. https://app.cochat.ai/papers/10.1109/mascot.2004.1348296
- The paper
- Schwarz, T. J. E., Xin, Q., Miller, E. L., Long, D. D. E., Hospodor, A., & Ng, S. (2004). Disk scrubbing in large archival storage systems. The IEEE Computer Society's 12th Annual International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunications Systems, 2004. (MASCOTS 2004). Proceedings. https://doi.org/10.1109/mascot.2004.1348296
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.