The DOI matches its Crossref record; 26 of the 26 references it cites were found in a registry or an index.
MillWheel: fault-tolerant stream processing at internet scale
Abstract
MillWheel is a framework for building low-latency data-processing applications that is widely used at Google. Users specify a directed computation graph and application code for individual nodes, and the system manages persistent state and the continuous flow of records, all within the envelope of the framework's fault-tolerance guarantees. This paper describes MillWheel's programming model as well as its implementation. The case study of a continuous anomaly detector in use at Google serves to motivate how many of MillWheel's features are used. MillWheel's programming model provides a notion of logical time, making it simple to write time-based aggregations. MillWheel was designed from the outset with fault tolerance and scalability in mind. In practice, we find that MillWheel's unique combination of scalability, fault tolerance, and a versatile programming model lends itself to a wide variety of problems at Google.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
MapReduce: simplified data processing on large clusters
Dean, J., & Ghemawat, S. (2008) · Communications of the ACM · doi:10.1145/1327452.1327492 · Cited by 18,476
Time, clocks, and the ordering of events in a distributed system
Lamport, L. (1978) · Communications of the ACM · doi:10.1145/359545.359563 · Cited by 8,488
Resilient distributed datasets: a fault-tolerant abstraction for in-memory cluster computing
Zaharia, M., Chowdhury, M., Das, T., Dave, A., Ma, J. H., McCauley, M., Franklin, M. J. M., Shenker, S. J., & Stoica, I. G. (2012) · UC Berkeley · Cited by 4,891
Bigtable: A Distributed Storage System for Structured Data
Chang, F., Dean, J., Ghemawat, S., Hsieh, W. C., Wallach, D. A., Burrows, M., Chandra, T., Fikes, A., & Gruber, R. E. (2008) · ACM Transactions on Computer Systems · doi:10.1145/1365815.1365816 · Cited by 3,460
Models and issues in data stream systems
Babcock, B., Babu, S., Datar, M., Motwani, R., & Widom, J. (2002) · Proceedings of the twenty-first ACM SIGMOD-SIGACT-SIGART symposium on Principles of database systems · doi:10.1145/543613.543615 · Cited by 2,495
Jefferson, D. R. (1985) · ACM Transactions on Programming Languages and Systems · doi:10.1145/3916.3988 · Cited by 2,397
Aurora: a new model and architecture for data stream management
Abadi, D. J., Carney, D., �etintemel, U., Cherniack, M., Convey, C., Lee, S., Stonebraker, M., Tatbul, N., & Zdonik, S. (2003) · The VLDB Journal The International Journal on Very Large Data Bases · doi:10.1007/s00778-003-0095-z · Cited by 1,495
The design of the borealis stream processing engine
Abadi, D. J., Ahmad, Y., Bałazińska, M., Cherniack, M., Hwang, J.-H., Lindner, W., Maskey, A. S., Rasin, A., Ryvkina, E., Tatbul, N., et al. (2005) · Cited by 1,255
S4: Distributed Stream Computing Platform
Neumeyer, L., Robbins, B., Nair, A., & Kesari, A. (2010) · 2010 IEEE International Conference on Data Mining Workshops · doi:10.1109/icdmw.2010.172 · Cited by 900
Spanner: Google's globally-distributed database
Corbett, J. C., Dean, J. B., Epstein, M., Fikes, A., Frost, C., Furman, J. J., Ghemawat, S., Gubarev, A., Heiser, C., Hochschild, P., et al. (2012) · doi:10.5555/2387880.2387905 · Cited by 807
TelegraphCQ: continuous dataflow processing
Chandrasekaran, S., Cooper, O., Deshpande, A., Franklin, M. J., Hellerstein, J. M., Hong, W., Krishnamurthy, S., Madden, S. R., Reiss, F., & Shah, M. A. (2003) · Proceedings of the 2003 ACM SIGMOD international conference on Management of data · doi:10.1145/872757.872857 · Cited by 744
Condie, T., Conway, N., Alvaro, P., Hellerstein, J. M., Elmeleegy, K., & Sears, R. (2010) · doi:10.5555/1855711.1855732 · Cited by 673
The 8 requirements of real-time stream processing
Stonebraker, M., Çetintemel, U., & Zdonik, S. (2005) · ACM SIGMOD Record · doi:10.1145/1107499.1107504 · Cited by 640
Discretized streams: an efficient and fault-tolerant model for stream processing on large clusters
Zaharia, M., Das, T., Li, H., Shenker, S. J., & Stoica, I. G. (2012) · USENIX conference on Hot Topics in Cloud Ccomputing · Cited by 522
Large-scale incremental processing using distributed transactions and notifications
Peng, D. N., & Dabek, F. (2010) · doi:10.5555/1924943.1924961 · Cited by 402
Query Processing, Resource Management, and Approximation ina Data Stream Management System
Motwani, R., Widom, J., Arasu, A., Babcock, B., Babu, S., Datar, M., Manku, G. S., Olston, C., Rosenstein, J., & Varma, R. (2002) · Cited by 342
Exploiting punctuation semantics in continuous data streams
Tucker, P. A., Maier, D., Sheard, T., & Fegaras, L. (2003) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2003.1198390 · Cited by 308
Flexible time management in data stream systems
Srivastava, U., & Widom, J. (2004) · Proceedings of the twenty-third ACM SIGMOD-SIGACT-SIGART symposium on Principles of database systems · doi:10.1145/1055558.1055596 · Cited by 275
High-Availability Algorithms for Distributed Stream Processing
Hwang, J.-H., Balazinska, M., Rasin, A., Cetintemel, U., Stonebraker, M., & Zdonik, S. (2005) · 21st International Conference on Data Engineering (ICDE'05) · doi:10.1109/icde.2005.72 · Cited by 261
CIEL: a universal execution engine for distributed data-flow computing
Murray, D. G., Schwarzkopf, M., Smowton, C., Smith, S. M., Madhavapeddy, A., & Hand, S. (2011) · doi:10.5555/1972457.1972470 · Cited by 225
Out-of-order processing: a new architecture for high-performance stream systems
Li, J., Tufte, K., Shkapenyuk, V., Papadimos, V., Johnson, T., & Maier, D. (2008) · Proceedings of the VLDB Endowment · doi:10.14778/1453856.1453890 · Cited by 177
Stateful bulk processing for incremental analytics
Logothetis, D., Olston, C., Reed, B. C., Webb, K. C., & Yocum, K. (2010) · ACM Symposium on Cloud Computing (SoCC) · doi:10.1145/1807128.1807138 · Cited by 172
Fault-tolerant stream processing using a distributed, replicated file system
Kwon, Y., Bałazińska, M., & Greenberg, A. G. (2008) · Proceedings of the VLDB Endowment · doi:10.14778/1453856.1453920 · Cited by 90
A heartbeat mechanism and its application in gigascope
Johnson, T. J., Muthukrishnan, S., Shkapenyuk, V., & Spatscheck, O. (2005) · Cited by 67
Centrifuge: integrated lease management and partitioning for cloud services
Adya, A., Dunagan, J., & Wolman, A. (2010) · Networked Systems Design and Implementation · doi:10.5555/1855711.1855712 · Cited by 43
Sonora: A Platform for Continuous Mobile-Cloud Computing
Chen, X., Beschastnikh, I., Li, Z., Yang, F., Qian, Z., Zhou, L., Shen, G., & Shen, J. (2012) · Cited by 27
Cited by
Large-scale cluster management at Google with Borg
Verma, A., Pedrosa, L., Korupolu, M., Oppenheimer, D., Tune, E., & Wilkes, J. (2015) · Proceedings of the Tenth European Conference on Computer Systems · doi:10.1145/2741948.2741964 · Cited by 1,468
(2015) · KTH Publication Database DiVA (KTH Royal Institute of Technology) · Cited by 1,144
Discretized streams: fault-tolerant streaming computation at scale
Zaharia, M., Das, T., Li, H., Hunter, T., Shenker, S., & Stoica, I. (2013) · Proceedings of the Twenty-Fourth ACM Symposium on Operating Systems Principles · doi:10.1145/2517349.2522737 · Cited by 982
Toshniwal, A., Taneja, S., Shukla, A., Ramasamy, K., Patel, J. M., Kulkarni, S., Jackson, J., Gade, K., Fu, M., Donham, J., et al. (2014) · Proceedings of the 2014 ACM SIGMOD International Conference on Management of Data · doi:10.1145/2588555.2595641 · Cited by 956
Akidau, T., Bradshaw, R., Chambers, C., Chernyak, S., Fernández-Moctezuma, R. J., Lax, R., McVeety, S., Mills, D., Perry, F., Schmidt, E., et al. (2015) · Proceedings of the VLDB Endowment · doi:10.14778/2824032.2824076 · Cited by 750
Self-citation
Metadata
- Authors
- Akidau, T., Balikov, A., Bekiroğlu, K., Chernyak, S., Haberman, J., Lax, R., McVeety, S., Mills, D., Nordstrom, P., & Whittle, S.
- Venue
- Proceedings of the VLDB Endowment
- Published
- 2013-08
- Volume
- 6
- Issue
- 11
- Pages
- 1033-1044
- ISSN
- 2150-8097 (ISSN-L)
- OpenAlex
- W2153972927
- Cited by
- 536
- References
- 26
- FWCI
- 65.64
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.