The DOI matches its Crossref record; its reference list is not available.
Real-Time Machine Learning: The Missing Pieces
Abstract
Machine learning applications are increasingly deployed not only to serve predictions using static models, but also as tightly-integrated components of feedback loops involving dynamic, real-time decision making. These applications pose a new set of requirements, none of which are difficult to achieve in isolation, but the combination of which creates a challenge for existing distributed execution frameworks: computation with millisecond latency at high throughput, adaptive construction of arbitrary task graphs, and execution of heterogeneous kernels over diverse sets of resources. We assert that a new distributed execution framework is needed for such ML applications and propose a candidate approach with a proof-of-concept architecture that achieves a 63x performance improvement over a state-of-the-art execution framework for a representative application.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Early calls for real-time ML systems [29] and distributed caches [30] foreshadowed stateful serving but predate the KV-cache regime; LLM serving's per-request, bursty KV state is a different regime.
“Early calls for realtime ML systems [29] and distributed caches [30] foreshadowed stateful serving but predate the KV-cache regime; LLM serving's per-request, bursty KV state is a different regime.”
From Janus: Joint Prefill/Decode Disaggregation with KV-Cache-Aware Multi-Cloud Routing for Edge-Adjacent LLM Serving (2026) · cites this paper as [29]
Citations
No citation lists for this paper
No citation lists for this paper yet — checked again daily.
References 23 · Cited by 49
Research works printed but not linked: 18
- Abadi Martín, 2016, TensorFlow: A System for Large-Scale Machine Learning. In 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16)
- Dario Amodei Rishita Anubhai Eric Battenberg Carl Case Jared Casper Bryan Catanzaro Jingdong Chen Mike Chrzanowski Adam Coates Greg Diamos etal 2015. Deep speech 2: End-to-end speech recognition in english and mandarin. arXiv preprint arXiv:1512.02595 (2015). Dario Amodei Rishita Anubhai Eric Battenberg Carl Case Jared Casper Bryan Catanzaro Jingdong Chen Mike Chrzanowski Adam Coates Greg Diamos et al. 2015. Deep speech 2: End-to-end speech recognition in english and mandarin. arXiv preprint arXiv:1512.02595 (2015). · arXiv
- 10.1145/800228.806932 · DOI
- 10.1145/2038916.2038932 · DOI
- Chen Tianqi, 2016, MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems. In NIPS Workshop on Machine Learning Systems (LearningSys'16)
- Coates Adam, 2013, Proceedings of The 30th International Conference on Machine Learning. 1337--1345
- Daniel Crankshaw Peter Bailis Joseph E Gonzalez Haoyuan Li Zhao Zhang Michael J Franklin Ali Ghodsi and Michael I Jordan. 2014. The missing piece in complex analytics: Low latency scalable model management and serving with Velox. arXiv preprint arXiv:1409.3809 (2014). Daniel Crankshaw Peter Bailis Joseph E Gonzalez Haoyuan Li Zhao Zhang Michael J Franklin Ali Ghodsi and Michael I Jordan. 2014. The missing piece in complex analytics: Low latency scalable model management and serving with Velox. arXiv preprint arXiv:1409.3809 (2014). · arXiv
- 10.1145/1327452.1327492 · DOI · Paper page
- Duan Yan, 2016, Proceedings of the 33rd International Conference on Machine Learning (ICML).
- Gabriel Edgar, Proceedings, 11th European PVM/MPI Users' Group Meeting
- 10.1145/1272996.1273005 · DOI
- 10.1109/JPROC.2014.2371999 · DOI · Paper page
- 10.1145/2517349.2522738 · DOI
- Murray Derek G., 2011, Proceedings of the 8th USENIX Conference on Networked Systems Design and Implementation (NSDI'11)
- 10.25080/Majora-7b98e3ed-013 · DOI
- Sanfilippo Salvatore, 2009, Redis: An open source, in-memory data structure store. https://redis.io/.
- 10.1145/1531793.1531805 · DOI
- 10.1145/2934664 · DOI · Paper page
Metadata
- Authors
- Nishihara, R., Moritz, P., Wang, S., Tumanov, A., Paul, W., Schleier-Smith, J., Liaw, R., Niknami, M., Jordan, M. I., & Stoica, I.
- Affiliations
- University of California, Berkeley (US)
- Venue
- Proceedings of the 16th Workshop on Hot Topics in Operating Systems
- Publisher
- ACM
- Published
- 2017-05-07
- Pages
- 106-110
- OpenAlex
- W2598842133
- Cited by
- 49
- References
- 23
- Citations by year
- 2026: 1 · 2025: 3 · 2024: 4 · 2023: 7 · 2022: 3 · 2021: 12 · 2020: 6 · 2019: 8 · 2018: 4 · 2017: 1
- FWCI
- 3.69
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- Workshop on Hot Topics in Operating Systems (publishedVersion)
- Funders
- Ant Financial Services Group
- Topics
- Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
- Physical Sciences › Computer Science › Information Systems › Cloud Computing and Resource Management
- Physical Sciences › Computer Science › Computer Networks and Communications › Distributed and Parallel Computing Systems
- Keywords
- real-time machine learning, feedback loops, resource management, latency, throughput, real-time decision making
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
