CoChat
Sign Up
Cited by 37Advanced Database Systems and QueriesDistributed systems and fault toleranceCloud Computing and Resource Management

The DOI matches its DataCite record.

Kafka, Samza and the Unix Philosophy of Distributed Data

Open access
Kleppmann, M, Kreps, J·2015·IEEE
DOI 10.17863/cam.33349
Open this paper in CoChat — free accountOpen PDF

Abstract

Apache Kafka is a scalable message broker, and Apache Samza is a stream processing framework built upon Kafka. They are widely used as infrastructure for implementing personalized online services and real-time predictive analytics. Besides providing high throughput and low latency, Kafka and Samza are designed with operational robustness and long-term maintenance of applications in mind. In this paper we explain the reasoning behind the design of Kafka and Samza, which allow complex applications to be built by composing a small number of simple primitives – replicated logs and stream operators. We draw parallels between the design of Kafka and Samza, batch processing pipelines, database architecture, and the design philosophy of Unix.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.17863/cam.33349

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Apache Samza is sometimes considered a predecessor of Kafka Streams.

    “Although still maintained, Samza is sometimes considered a predecessor of Kafka Streams (Kleppmann and Kreps, 2015).”

    From Benchmarking scalability of stream processing frameworks deployed as microservices in the cloud (2024) · cites this paper as Kleppmann and Kreps, 2015

    Section 1.4 says Samza was already in use while Kafka Streams was being added, and describes their stream-processing capabilities as similar. This supports the chronological comparison, but the paper does not say Samza was considered a predecessor of Kafka Streams. (LLM, conf=0.78)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

No references could be resolved yet.

Research works printed but not linked: 23

  1. Tyler Akidau, Robert Bradshaw, Craig Chambers, Slava Chernyak, et al. The dataflow model: A practical approach to balancing correctness, latency, and cost in massive-scale, unbounded, out-of-order data processing. Proceedings of the VLDB Endowment, 8(12):1792-1803, August 2015. doi:10.14778/2824032.2824076. · DOI · Paper page
  2. Shivnath Babu and Herodotos Herodotou. Massively parallel databases and MapReduce systems. Foundations and Trends in Databases, 5(1):1-104, November 2013. doi:10.1561/1900000036. · DOI
  3. Mahesh Balakrishnan, Dahlia Malkhi, Ted Wobber, Ming Wu, et al. Tango: Distributed data structures over a shared log. In 24th ACM Symposium on Operating Systems Principles (SOSP), pages 325-340, November 2013. doi:10.1145/2517349.2522732. · DOI
  4. Raul Castro Fernandez, Peter Pietzuch, Jay Kreps, Neha Narkhede, et al. Liquid: Unifying nearline and offline big data integration. In 7th Biennial Conference on Innovative Data Systems Research (CIDR), January 2015.
  5. Shirshanka Das, Chavdar Botev, Kapil Surlaker, Bhaskar Ghosh, et al. All aboard the Databus! In 3rd ACM Symposium on Cloud Computing (SoCC), October 2012.
  6. Jeffrey Dean and Sanjay Ghemawat. MapReduce: Simplified data processing on large clusters. In 6th USENIX Symposium on Operating System Design and Implementation (OSDI), December 2004.
  7. Ken Goodhope, Joel Koshy, Jay Kreps, Neha Narkhede, et al. Building LinkedIn's real-time activity data pipeline. Bulletin of the IEEE Computer Society Technical Committee on Data Engineering, 35(2):33-45, June 2012.
  8. Pat Helland. Immutability changes everything. In 7th Biennial Conference on Innovative Data Systems Research (CIDR), January 2015.
  9. Jay Kreps. I Heart Logs. O'Reilly Media, September 2014. ISBN 978-1-4919-0932-4.
  10. Jay Kreps, Neha Narkhede, and Jun Rao. Kafka: a distributed messaging system for log processing. In 6th Interna- tional Workshop on Networking Meets Databases (NetDB), June 2011.
  11. M D McIlroy, E N Pinson, and B A Tague. UNIX time-sharing system: Foreword. The Bell System Technical Journal, 57(6):1899-1904, July 1978.
  12. Patrick O'Neil, Edward Cheng, Dieter Gawlick, and Elizabeth O'Neil. The log-structured merge-tree (LSM-Tree). Acta Informatica, 33(4):351-385, June 1996. doi:10.1007/s002360050048. · DOI
  13. Rob Pike and Brian W Kernighan. Program design in the UNIX environment. AT&T Bell Laboratories Technical Journal, 63(8):1595-1605, October 1984. doi:10.1002/j.1538-7305.1984.tb00055.x. · DOI
  14. Azarias Reda, Yubin Park, Mitul Tiwari, Christian Posse, and Sam Shah. Metaphor: A system for related search recommendations. In 21st ACM International Conference on Information and Knowledge Management (CIKM), October 2012.
  15. Dennis M Ritchie and Ken Thompson. The UNIX time-sharing system. Communications of the ACM, 17(7), July 1974. doi:10.1145/361011.361061. · DOI · Paper page
  16. Mendel Rosenblum and John K Ousterhout. The design and implementation of a log-structured file system. ACM Transactions on Computer Systems (TOCS), 10(1):26-52, February 1992. doi:10.1145/146941.146943. · DOI
  17. Fred B Schneider. Implementing fault-tolerant services using the state machine approach: A tutorial. ACM Comput- ing Surveys, 22(4):299-319, December 1990.
  18. Margo Seltzer. Beyond relational databases. Communications of the ACM, 51(7):52-58, July 2008. doi:10.1145/1364782.1364797. · DOI
  19. Michael Stonebraker and Ugur C ¸etintemel. "One size fits all": An idea whose time has come and gone. In 21st International Conference on Data Engineering (ICDE), April 2005.
  20. Roshan Sumbaly, Jay Kreps, and Sam Shah. The "Big Data" ecosystem at LinkedIn. In ACM International Confer- ence on Management of Data (SIGMOD), July 2013.
  21. Guozhang Wang, Joel Koshy, Sriram Subramanian, Kartik Paramasivam, et al. Building a replicated log- ging system with Apache Kafka. Proceedings of the VLDB Endowment, 8(12):1654-1655, August 2015. doi:10.14778/2824032.2824063. · DOI
  22. Lili Wu, Sam Shah, Sean Choi, Mitul Tiwari, and Christian Posse. The browsemaps: Collaborative filtering at LinkedIn. In 6th Workshop on Recommender Systems and the Social Web (RSWeb), October 2014.
  23. Matei Zaharia, Mosharaf Chowdhury, Tathagata Das, Ankur Dave, et al. Resilient distributed datasets: A fault- tolerant abstraction for in-memory cluster computing. In 9th USENIX Symposium on Networked Systems Design and Implementation (NSDI), April 2012.

Cited by

Sorted by citations

Metadata

Authors
Kleppmann, M., & Kreps, J.
ORCID
Kleppmann, M
Venue
IEEE
Publisher
IEEE
Published
2015-12-01
Volume
38
Pages
4-14
DOI
10.17863/cam.33349
OpenAlex
W2576742135
Cited by
37
References
0
Citations by year
2025: 2 · 2024: 3 · 2023: 5 · 2021: 5 · 2020: 4 · 2019: 6 · 2018: 4 · 2017: 7 · 2016: 1
FWCI
6.64
Type
journal article
Language
en
Open access
Locations
  • Apollo (University of Cambridge) (submittedVersion)
  • Apollo
  • IEEE Data(base) Engineering Bulletin
Topics
  • Physical Sciences › Computer Science › Information Systems › Advanced Database Systems and Queries
  • Physical Sciences › Computer Science › Computer Networks and Communications › Distributed systems and fault tolerance
  • Physical Sciences › Computer Science › Information Systems › Cloud Computing and Resource Management
Keywords
Apache Kafka, message brokers, stream processing, Unix philosophy, distributed data, stream operators, database architecture, software design principles, real-time analytics

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review