The DOI matches its Crossref record.
Towards Observability Data Management at Scale
Abstract
Observability has been gaining importance as a key capability in today's large-scale software systems and services. Motivated by current experience in industry exemplified by Slack and as a call to arms for database research, this paper outlines the challenges and opportunities involved in designing and building Observability Data Management Systems (ODMSs) to handle this emerging workload at scale.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“This is an OT system characteristic, not a pipeline failure, and must be subtracted before threshold comparison:”
From Data Observability Framework for Mission-Critical Industrial ETL Pipelines: Anomaly Detection, SLA Modeling, and Proactive Failure Identification (2026) · cites this paper as [9]
In the citing paper
“PI interface reconnection suppression polls PI Web API interface status [9] at 5-minute intervals, with a 30-minute suppression window following confirmed reconnection.”
From Data Observability Framework for Mission-Critical Industrial ETL Pipelines: Anomaly Detection, SLA Modeling, and Proactive Failure Identification (2026) · cites this paper as [9]
Citations
Cites
No references could be resolved yet.
Research works printed but not linked: 21
- R. H. Arpaci-Dusseau et al. 2018 . Cloud-Native File Systems. In USENIX Conference on Hot Topics in Cloud Computing (HotCloud). R. H. Arpaci-Dusseau et al. 2018. Cloud-Native File Systems. In USENIX Conference on Hot Topics in Cloud Computing (HotCloud).
- C. Chan etal 2020. Debugging Incidents in Google's Distributed Systems. ACM Queue 18 2 (2020). C. Chan et al. 2020. Debugging Incidents in Google's Distributed Systems. ACM Queue 18 2 (2020). · DOI
- 10.1145/2814710.2814713 · DOI
- C. Gormley 2015 . Elasticsearch: The Definitive Guide . O'Reilly Media . C. Gormley et al. 2015. Elasticsearch: The Definitive Guide. O'Reilly Media.
- N. Narkhede 2017 . Kafka: The Definitive Guide Real-Time Data and Stream Processing at Scale . O'Reilly Media . N. Narkhede et al. 2017. Kafka: The Definitive Guide Real-Time Data and Stream Processing at Scale. O'Reilly Media.
- 10.1145/13677.22723 · DOI
- 10.1145/3132747.3132749 · DOI
- 10.1145/359545.359563 · DOI · Paper page
- 10.1145/2815400.2815415 · DOI
- S. Niedermaier etal 2019. On Observability and Monitoring of Distributed Systems -- An Industry Interview Study. In ICSOC. S. Niedermaier et al. 2019. On Observability and Monitoring of Distributed Systems -- An Industry Interview Study. In ICSOC. · DOI
- F. Özcan et al. 2017 . Hybrid Transactional/Analytical Processing: A Survey. In ACM SIGMOD Conference. 1771--1775 . F. Özcan et al. 2017. Hybrid Transactional/Analytical Processing: A Survey. In ACM SIGMOD Conference. 1771--1775.
- 10.1145/2814710.2814719 · DOI
- 10.1145/3377391.3377400 · DOI
- J. Rodrigues 2017 . Sieve: Actionable Insights from Monitored Metrics in Distributed Systems . In ACM Middleware Conference. 14-- 27 . J. Rodrigues et al. 2017. Sieve: Actionable Insights from Monitored Metrics in Distributed Systems. In ACM Middleware Conference. 14-- 27.
- R. Sethi 2019 . Presto: SQL on Everything . In IEEE ICDE. R. Sethi et al. 2019. Presto: SQL on Everything. In IEEE ICDE.
- Y. Shkuro . 2019. Mastering Distributed Tracing: Analyzing Performance in Microservices and Complex Systems . Packt Publishing . Y. Shkuro. 2019. Mastering Distributed Tracing: Analyzing Performance in Microservices and Complex Systems. Packt Publishing.
- B. H. Sigelman 2010 . Dapper: A Large-Scale Distributed Systems Tracing Infrastructure. Technical Report. Google , Inc . B. H. Sigelman et al. 2010. Dapper: A Large-Scale Distributed Systems Tracing Infrastructure. Technical Report. Google, Inc.
- C. Sridharan . 2018. Distributed Systems Observability: A Guide to Building Robust Systems . O'Reilly Media . C. Sridharan. 2018. Distributed Systems Observability: A Guide to Building Robust Systems. O'Reilly Media.
- D. Vohra. 2016. Apache Parquet. In Practical Hadoop Ecosystem: A Definitive Guide to Hadoop-Related Frameworks and Tools. 325--335. D. Vohra. 2016. Apache Parquet. In Practical Hadoop Ecosystem: A Definitive Guide to Hadoop-Related Frameworks and Tools. 325--335. · DOI
- A. Wiedemann etal 2019. The DevOps Phenomenon. ACM Queue 17 2 (2019). A. Wiedemann et al. 2019. The DevOps Phenomenon. ACM Queue 17 2 (2019). · DOI
- M. Zaharia 2010 . Spark: Cluster Computing withWorking Sets. In USENIX Conference on Hot Topics in Cloud Computing (HotCloud). M. Zaharia et al. 2010. Spark: Cluster Computing withWorking Sets. In USENIX Conference on Hot Topics in Cloud Computing (HotCloud).
Cited by
Narayanan, S., S, M., & Zephan, P. (2024) · ICST Transactions on Scalable Information Systems · doi:10.4108/eetsis.5065 · Cited by 8
“Overall, it provides a valuable survey of the current state-of-the-art in observability data management and sets the stage for future research in this important area [10].”
Sojitra, E. (2026) · American International Journal of Computer Science and Technology · doi:10.63282/3117-5481/aijcst-v8i3p101
Metadata
- Authors
- Karumuri, S., Solleza, F., Zdonik, S., & Tatbul, N.
- Affiliations
- Slack Incorporated (United States) (US)
- Brown University (US)
- Intel (United States) (US)
- Venue
- ACM SIGMOD Record
- Published
- 2021-03-08
- Volume
- 49
- Issue
- 4
- Pages
- 18-23
- ISSN
- 0163-5808 (ISSN-L)
- OpenAlex
- W3137837084
- Cited by
- 48
- References
- 0
- Citations by year
- 2026: 7 · 2025: 6 · 2024: 15 · 2023: 8 · 2022: 11 · 2021: 1
- FWCI
- 3.93
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- ACM SIGMOD Record (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Networks and Communications › Software System Performance and Reliability
- Physical Sciences › Computer Science › Information Systems › Advanced Database Systems and Queries
- Physical Sciences › Computer Science › Information Systems › Cloud Computing and Resource Management
- Keywords
- observability, large software systems, monitoring data, database management systems, scalability, distributed systems
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review