Building Continuous Integration Services for Machine Learning
The DOI matches its Crossref record; 16 of the 16 references it cites were found in a registry or an index.
Abstract
Continuous integration (CI) has been a de facto standard for building industrial-strength software. Yet, there is little attention towards applying CI to the development of machine learning (ML) applications until the very recent effort on the theoretical side. In this paper, we take a step forward to bring the theory into practice.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“This is a repetitive task which, again, requires a high level of automation [66], [θ].”
From Machine Learning Operations (MLOps): Overview, Definition, and Architecture (2023) · cites this paper as [66]
Citations
Cites
Searching for exotic particles in high-energy physics with deep learning
Baldi, P., Sadowski, P., & Whiteson, D. (2014) · Nature Communications · doi:10.1038/ncomms5308 · Cited by 1,164 · cited 1×
“We take a real dataset that represents a classification task regarding the presence of Higgs bosons in a physical process [10].”
MethodContinuous Integration: Improving Software Quality and Reducing Risk
Duvall, P., Matyas, S., & Glover, A. (2007) · Cited by 510
The reusable holdout: Preserving validity in adaptive data analysis
Dwork, C., Feldman, V., Hardt, M., Pitassi, T., Reingold, O., & Roth, A. (2015) · Science · doi:10.1126/science.aaa9375 · Cited by 307 · cited 1×
“Recent progress on adaptive analysis [13] reveals the fact that the fidelity of the test dataset may fade away when it being accessed again and again.”
BackgroundSchawinski, K., Zhang, C., Zhang, H., Fowler, L., & Santhanam, G. K. (2017) · Monthly Notices of the Royal Astronomical Society: Letters · doi:10.1093/mnrasl/slx008 · Cited by 246
Self-citationUsing transfer learning to detect galaxy mergers
Ackermann, S., Schawinski, K., Zhang, C., Weigel, A. K., & Turp, M. D. (2018) · Monthly Notices of the Royal Astronomical Society · doi:10.1093/mnras/sty1398 · Cited by 137 · cited 1×
“In our experience in supporting a range of industrial and academic users [9, 14, 22–25], it is not uncommon for users of modern ML platforms to suffer from this problem, as illustrated in Figure 1(a), that the gap between the estimated test/validation accuracy and true test accuracy increases…”
BackgroundSelf-citationNorthstar: an interactive data science system
Kraska, T. (2018) · Proceedings of the VLDB Endowment · doi:10.14778/3229863.3240493 · Cited by 106 · cited 1×
“…from major cloud service providers such as Amazon SageMaker [1], Microsoft Azure Machine Learning [6], and Google Cloud AutoML [5], as well as ones from academic institutions such as the Northstar system developed at MIT [16], and the ease.ml service [15, 19, 27] by ETH Zurich, among others.”
BackgroundThe Ladder: A Reliable Leaderboard for Machine Learning Competitions
Blum, A., & Hardt, M. (2015) · arXiv · doi:10.48550/arxiv.1502.04585 · Cited by 92 · cited 2×
“As was shown in [20], by using an idea similar to the Ladder mechanism [11], we can, however, apply the union bound over all 2 possible (binary) sequences that represent the whole space of possible test outcomes from N submitted models:”
BackgroundMethodEase.ml: Towards Multi-tenant Resource Sharing for Machine Learning WorkloadsPreprint
Li, T., Zhong, J., Liu, J., Wu, W., & Zhang, C. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1708.07308 · Cited by 33 · cited 1×
“ml service [15, 19, 27] by ETH Zurich, among others.”
BackgroundSelf-citationStark, D., Launet, B., Schawinski, K., Zhang, C., Koss, M., Turp, M. D., Sartori, L. F., Zhang, H., Chen, Y., & Weigel, A. K. (2018) · Monthly Notices of the Royal Astronomical Society · doi:10.1093/mnras/sty764 · Cited by 24
Self-citationExploring galaxy evolution with generative models
Schawinski, K., Turp, M. D., & Zhang, C. (2018) · Astronomy and Astrophysics · doi:10.1051/0004-6361/201833800 · Cited by 24
Self-citationExploring galaxy evolution with generative models
Schawinski, K., Turp, M. D., & Zhang, C. (2018) · Springer Link (Chiba Institute of Technology) · doi:10.1051/0004-6361/201833800/pdf · Cited by 23
Self-citationPatient Risk Assessment and Warning Symptom Detection Using Deep Attention-Based Neural Networks
Girardi, I., Ji, P., Nguyen, A.-P., Hollenstein, N., Ivankay, A., Kuhn, L., Marchiori, C., & Zhang, C. (2018) · International Workshop on Health Text Mining and Information Analysis (LOUHI) · doi:10.18653/v1/w18-5616 · Cited by 21 · cited 1×
“In our experience in supporting a range of industrial and academic users [9, 14, 22–25], it is not uncommon for users of modern ML platforms to suffer from this problem, as illustrated in Figure 1(a), that the gap between the estimated test/validation accuracy and true test accuracy increases…”
BackgroundSelf-citationSu, M., Zhang, H., Schawinski, K., Zhang, C., & Cianfrocco, M. A. (2018) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/256792 · Cited by 19 · cited 1×
“In our experience in supporting a range of industrial and academic users [9, 14, 22–25], it is not uncommon for users of modern ML platforms to suffer from this problem, as illustrated in Figure 1(a), that the gap between the estimated test/validation accuracy and true test accuracy increases…”
BackgroundSelf-citationEase.ml in Action: Towards Multi-tenant Declarative Learning Services
Karlas, B., Liu, J., Wu, W., & Zhang, C. (2018) · Proceedings of the VLDB Endowment · doi:10.14778/3229863.3236258 · Cited by 12 · cited 1×
“…from major cloud service providers such as Amazon SageMaker [1], Microsoft Azure Machine Learning [6], and Google Cloud AutoML [5], as well as ones from academic institutions such as the Northstar system developed at MIT [16], and the ease.ml service [15, 19, 27] by ETH Zurich, among others.”
BackgroundSelf-citationEase.ml/ci and Ease.ml/meter in action: towards data management for statistical generalization
Renggli, C., Hubis, F. A., Karlaš, B., Schawinski, K., Wu, W., & Zhang, C. (2019) · Proceedings of the VLDB Endowment · doi:10.14778/3352063.3352110 · Cited by 10 · cited 1×
“[20, 21] is the first work along this line, as far as we know.”
BackgroundSelf-citationAn Overreaction to the Broken Machine Learning Abstraction
Zhang, C., Wu, W., & Li, T. (2017) · Workshop on Human-In-the-Loop Data Analytics (HILDA) · doi:10.1145/3077257.3077265 · Cited by 7 · cited 1×
“ml service [15, 19, 27] by ETH Zurich, among others.”
BackgroundSelf-citation
Research works printed but not linked: 1
- C. Renggli Continuous integration of machine learning models with ease.ml/ci: Towards a rigorous yet practical treatment . In SysML , 2019 . C. Renggli et al. Continuous integration of machine learning models with ease.ml/ci: Towards a rigorous yet practical treatment. In SysML, 2019.
Cited by
Machine Learning Operations (MLOps): Overview, Definition, and Architecture
Kreuzberger, D., Kühl, N., & Hirschl, S. (2023) · IEEE Access · doi:10.1109/access.2023.3262138 · Cited by 766
“This is a repetitive task which, again, requires a high level of automation [18] [θ].”
BackgroundAdvances, challenges and opportunities in creating data for trustworthy AI
Liang, W., Tadesse, G. A., Ho, D., Fei-Fei, L., Zaharia, M., Zhang, C., & Zou, J. (2022) · Nature Machine Intelligence · doi:10.1038/s42256-022-00516-1 · Cited by 559
“But real-world AI users generally need to update data-sets and models continuously 85 .”
BackgroundSelf-citationSteidl, M., Felderer, M., & Ramler, R. (2023) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111615 · Cited by 126
The art and practice of data science pipelines
Biswas, S., Wardat, M., & Rajan, H. (2022) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3510003.3510057 · Cited by 128
“Today, continuous integration and deployment (CI/CD) tools are widely used in traditional software lifecycle to automate compilation, building, and testing [29, 41].”
BackgroundMethodTransitioning from MLOps to LLMOps: Navigating the Unique Challenges of Large Language Models
Pahune, S. A., & Akhtar, Z. (2025) · Information · doi:10.3390/info16020087 · Cited by 116
Metadata
- Authors
- Karlaš, B., Interlandi, M., Renggli, C., Wu, W., Zhang, C., Mukunthu Iyappan Babu, D., Edwards, J., Lauren, C., Xu, A., & Weimer, M.
- Affiliations
- ETH Zurich (CH)
- Microsoft (United States) (US)
- ETH Zürich, Zürich, Switzerland
- Microsoft, Redmond, WA, USA
- Venue
- Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining
- Publisher
- ACM
- Published
- 2020-08-20
- Pages
- 2407-2415
- OpenAlex
- W3035597669
- Cited by
- 30
- References
- 16
- Citations by year
- 2026: 1 · 2025: 5 · 2024: 4 · 2023: 6 · 2022: 8 · 2021: 5 · 2020: 1
- FWCI
- 3.79
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Keywords
- continuous integration, machine learning applications
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.