The DOI matches its Crossref record.
Twitter Heron: Stream Processing at Scale
Open accessAbstract
Storm has long served as the main platform for real-time analytics at Twitter. However, as the scale of data being processed in real-time at Twitter has increased, along with an increase in the diversity and the number of use cases, many limitations of Storm have become apparent. We need a system that scales better, has better debug-ability, has better performance, and is easier to manage -- all while working in a shared cluster infrastructure. We considered various alternatives to meet these needs, and in the end concluded that we needed to build a new real-time stream data processing system. This paper presents the design and implementation of this new system, called Heron. Heron is now the de facto stream data processing engine inside Twitter, and in this paper we also share our experiences from running Heron in production. In this paper, we also provide empirical evidence demonstrating the efficiency and scalability of Heron.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Apache Spark, Apache Storm, and Apache Heron are considered heavyweight frameworks and fit less into the context of microservices.
“Apache Spark (Zaharia et al., 2016), Apache Storm (Toshniwal et al., 2014), and the successor of the latter, Apache Heron (Kulkarni et al., 2015), are considered heavyweight frameworks and, thus, fit less into the context of microservices (Bellemare, 2020).”
From Benchmarking scalability of stream processing frameworks deployed as microservices in the cloud (2024) · cites this paper as Kulkarni et al., 2015
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
No references could be resolved yet.
Research works printed but not linked: 14
- 10.14778/2536222.2536229 · DOI
- 10.1109/ICDE.2011.5767878 · DOI
- 10.1145/2463676.2465272 · DOI
- Arvind Arasu , Brian Babcock , Shivnath Babu , Mayur Datar , Keith Ito , Rajeev Motwani , Itaru Nishizawa , Utkarsh Srivastava , Dilys Thomas , Rohit Varma , Jennifer Widom: STREAM: The Stanford Stream Data Manager. IEEE Data Eng. Bull. 26 ( 1 ): 19 -- 26 ( 2003 ) Arvind Arasu, Brian Babcock, Shivnath Babu, Mayur Datar, Keith Ito, Rajeev Motwani, Itaru Nishizawa, Utkarsh Srivastava, Dilys Thomas, Rohit Varma, Jennifer Widom: STREAM: The Stanford Stream Data Manager. IEEE Data Eng. Bull. 26(1): 19--26 (2003)
- 10.1007/s00778-004-0133-5 · DOI
- 10.14778/2733004.2733016 · DOI
- Minos N. Garofalakis Johannes Gehrke: Querying and Mining Data Streams: You Only Get One Look. VLDB 2002 Minos N. Garofalakis Johannes Gehrke: Querying and Mining Data Streams: You Only Get One Look. VLDB 2002 · DOI
- Benjamin Hindman , Andy Konwinski , Matei Zaharia , Ali Ghodsi , Anthony D. Joseph , Randy H. Katz , Scott Shenker , Ion Stoica : Mesos: A Platform for Fine-Grained Resource Sharing in the Data Center . NSDI 2011 Benjamin Hindman, Andy Konwinski, Matei Zaharia, Ali Ghodsi, Anthony D. Joseph, Randy H. Katz, Scott Shenker, Ion Stoica: Mesos: A Platform for Fine-Grained Resource Sharing in the Data Center. NSDI 2011
- Jay Kreps , Neha Narkhede , and Jun Rao . Kafka : a distributed messaging system for log processing . SIGMOD Workshop on Networking Meets Databases , 2011 . Jay Kreps, Neha Narkhede, and Jun Rao. Kafka: a distributed messaging system for log processing. SIGMOD Workshop on Networking Meets Databases, 2011.
- 10.1145/2320765.2320789 · DOI
- Sankar Subramanian , Srikanth Bellamkonda , Hua-Gang Li , Vince Liang , Lei Sheng , Wayne Smith , James Terry , Tsae-Feng Yu , Andrew Witkowski : Continuous Queries in Oracle. VLDB 2007 : 1173--1184 Sankar Subramanian, Srikanth Bellamkonda, Hua-Gang Li, Vince Liang, Lei Sheng, Wayne Smith, James Terry, Tsae-Feng Yu, Andrew Witkowski: Continuous Queries in Oracle. VLDB 2007: 1173--1184
- 10.1145/2588555.2595641 · DOI · Paper page
- Vinod Kumar Vavilapalli , Arun C. Murthy , Chris Douglas , Sharad Agarwal , Mahadev Konar , Robert Evans , Thomas Graves , Jason Lowe , Hitesh Shah , Siddharth Seth , Bikas Saha , Carlo Curino , Owen O'Malley, Sanjay Radia, Benjamin Reed , Eric Baldeschwieler : Apache Hadoop YARN: yet another resource negotiator. SoCC 2013 : 5 Vinod Kumar Vavilapalli, Arun C. Murthy, Chris Douglas, Sharad Agarwal, Mahadev Konar, Robert Evans, Thomas Graves, Jason Lowe, Hitesh Shah, Siddharth Seth, Bikas Saha, Carlo Curino, Owen O'Malley, Sanjay Radia, Benjamin Reed, Eric Baldeschwieler: Apache Hadoop YARN: yet another resource negotiator. SoCC 2013: 5
- ZeroMQ : http://zeromq.org. Retrieved December 1, 2014 . ZeroMQ: http://zeromq.org. Retrieved December 1, 2014.
Cited by
Sorted by citations
Benchmarking scalability of stream processing frameworks deployed as microservices in the cloud
Henning, S., & Hasselbring, W. (2024) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111879 · Cited by 85
Metadata
- Authors
- Kulkarni, S., Bhagat, N., Fu, M., Kedigehalli, V., Kellogg, C., Mittal, S., Patel, J. M., Ramasamy, K., & Taneja, S.
- Affiliations
- Twitter (United States) (US)
- University of Wisconsin–Madison (US)
- ORCID
- Patel, Jignesh M.
- Venue
- Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data
- Publisher
- ACM
- Published
- 2015-05-27
- Pages
- 239-250
- ISSN
- 0730-8078 (ISSN-L)
- OpenAlex
- W2008503861
- Cited by
- 570
- References
- 0
- Citations by year
- 2026: 12 · 2025: 15 · 2024: 36 · 2023: 34 · 2022: 21 · 2021: 65 · 2020: 70 · 2019: 83 · 2018: 85 · 2017: 78 · 2016: 64 · 2015: 6
- FWCI
- 56.48
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- ACM SIGMOD International Conference on Management of Data (SIGMOD) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Information Systems › Advanced Database Systems and Queries
- Physical Sciences › Computer Science › Information Systems › Cloud Computing and Resource Management
- Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
- Keywords
- herons, Apache Storm, real-time stream processing, scalability, fault tolerance, distributed systems, Twitter, production deployment, system design
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
