Building Continuous Integration Services for Machine Learning
The DOI matches its Crossref record; 16 of the 16 references it cites were found in a registry or an index.
Abstract
Continuous integration (CI) has been a de facto standard for building industrial-strength software. Yet, there is little attention towards applying CI to the development of machine learning (ML) applications until the very recent effort on the theoretical side. In this paper, we take a step forward to bring the theory into practice.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“This is a repetitive task which, again, requires a high level of automation [66], [θ].”
From Machine Learning Operations (MLOps): Overview, Definition, and Architecture (2023) · cites this paper as [66]
Citations
Cites
Searching for exotic particles in high-energy physics with deep learning
Baldi, P., Sadowski, P., & Whiteson, D. (2014) · Nature Communications · doi:10.1038/ncomms5308 · Cited by 1,164 · cited 1×
“We take a real dataset that represents a classification task regarding the presence of Higgs bosons in a physical process [10].”
MethodContinuous Integration: Improving Software Quality and Reducing Risk
Duvall, P., Matyas, S., & Glover, A. (2007) · Cited by 510
The reusable holdout: Preserving validity in adaptive data analysis
Dwork, C., Feldman, V., Hardt, M., Pitassi, T., Reingold, O., & Roth, A. (2015) · Science · doi:10.1126/science.aaa9375 · Cited by 307 · cited 1×
“Recent progress on adaptive analysis [13] reveals the fact that the fidelity of the test dataset may fade away when it being accessed again and again.”
BackgroundSchawinski, K., Zhang, C., Zhang, H., Fowler, L., & Santhanam, G. K. (2017) · Monthly Notices of the Royal Astronomical Society: Letters · doi:10.1093/mnrasl/slx008 · Cited by 246
Self-citationUsing transfer learning to detect galaxy mergers
Ackermann, S., Schawinski, K., Zhang, C., Weigel, A. K., & Turp, M. D. (2018) · Monthly Notices of the Royal Astronomical Society · doi:10.1093/mnras/sty1398 · Cited by 137 · cited 1×
“In our experience in supporting a range of industrial and academic users [9, 14, 22–25], it is not uncommon for users of modern ML platforms to suffer from this problem, as illustrated in Figure 1(a), that the gap between the estimated test/validation accuracy and true test accuracy increases…”
BackgroundSelf-citationNorthstar: an interactive data science system
Kraska, T. (2018) · Proceedings of the VLDB Endowment · doi:10.14778/3229863.3240493 · Cited by 106 · cited 1×
“…from major cloud service providers such as Amazon SageMaker [1], Microsoft Azure Machine Learning [6], and Google Cloud AutoML [5], as well as ones from academic institutions such as the Northstar system developed at MIT [16], and the ease.ml service [15, 19, 27] by ETH Zurich, among others.”
BackgroundThe Ladder: A Reliable Leaderboard for Machine Learning Competitions
Blum, A., & Hardt, M. (2015) · arXiv · doi:10.48550/arxiv.1502.04585 · Cited by 92 · cited 2×
“As was shown in [20], by using an idea similar to the Ladder mechanism [11], we can, however, apply the union bound over all 2 possible (binary) sequences that represent the whole space of possible test outcomes from N submitted models:”
BackgroundMethodEase.ml: Towards Multi-tenant Resource Sharing for Machine Learning WorkloadsPreprint
Li, T., Zhong, J., Liu, J., Wu, W., & Zhang, C. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1708.07308 · Cited by 33 · cited 1×
“ml service [15, 19, 27] by ETH Zurich, among others.”
BackgroundSelf-citationStark, D., Launet, B., Schawinski, K., Zhang, C., Koss, M., Turp, M. D., Sartori, L. F., Zhang, H., Chen, Y., & Weigel, A. K. (2018) · Monthly Notices of the Royal Astronomical Society · doi:10.1093/mnras/sty764 · Cited by 24
Self-citationExploring galaxy evolution with generative models
Schawinski, K., Turp, M. D., & Zhang, C. (2018) · Astronomy and Astrophysics · doi:10.1051/0004-6361/201833800 · Cited by 24
Self-citation
Research works printed but not linked: 1
- C. Renggli Continuous integration of machine learning models with ease.ml/ci: Towards a rigorous yet practical treatment . In SysML , 2019 . C. Renggli et al. Continuous integration of machine learning models with ease.ml/ci: Towards a rigorous yet practical treatment. In SysML, 2019.
Cited by
Machine Learning Operations (MLOps): Overview, Definition, and Architecture
Kreuzberger, D., Kühl, N., & Hirschl, S. (2023) · IEEE Access · doi:10.1109/access.2023.3262138 · Cited by 766
“This is a repetitive task which, again, requires a high level of automation [18] [θ].”
BackgroundAdvances, challenges and opportunities in creating data for trustworthy AI
Liang, W., Tadesse, G. A., Ho, D., Fei-Fei, L., Zaharia, M., Zhang, C., & Zou, J. (2022) · Nature Machine Intelligence · doi:10.1038/s42256-022-00516-1 · Cited by 559
“But real-world AI users generally need to update data-sets and models continuously 85 .”
BackgroundSelf-citationSteidl, M., Felderer, M., & Ramler, R. (2023) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111615 · Cited by 126
The art and practice of data science pipelines
Biswas, S., Wardat, M., & Rajan, H. (2022) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3510003.3510057 · Cited by 128
“Today, continuous integration and deployment (CI/CD) tools are widely used in traditional software lifecycle to automate compilation, building, and testing [29, 41].”
BackgroundMethodTransitioning from MLOps to LLMOps: Navigating the Unique Challenges of Large Language Models
Pahune, S. A., & Akhtar, Z. (2025) · Information · doi:10.3390/info16020087 · Cited by 116
EasyFL: A Low-Code Federated Learning Platform for Dummies
Zhuang, W., Gan, X., Wen, Y., & Zhang, S. (2022) · IEEE Internet of Things Journal · doi:10.1109/jiot.2022.3143842 · Cited by 81
Filippou, K., Aifantis, G., Papakostas, G. A., & Tsekouras, G. E. (2023) · Information · doi:10.3390/info14040232 · Cited by 51
“…to create automated testing tools for quality assessment and deploy software automation tools to manage the rollout of the above code changes [35,36], with the ultimate purpose being the automatic production of software applications in a continuous and iterative manner, utilizing less time and…”
MethodSAGA: A Scalable Framework for Optimizing Data Cleaning Pipelines for Machine Learning Applications
Siddiqi, S., Kern, R., & Böehm, M. (2023) · Proceedings of the ACM on Management of Data · doi:10.1145/3617338 · Cited by 48
“…distributed computation on the 1+6 node cluster (32+192 virtual cores), flattened, distributed model assertions [51], validation via unit tests in Deequ [88] and ease.ml/ci [ 52, 84], as well as tests for model robustness against data corruptions in Jenga [91] and expected prediction quality [90].”
MethodA Comprehensive Review on Scaling Machine Learning Workflows Using Cloud Technologies and DevOps
Ramesh, G., Pai, T. V., Birau, R., Poojary, K. K., Abhay, Shingad, A. R., Sowjanya, N., Popescu, V., Mitroi, A. T., Chireac, R. M. N., et al. (2025) · IEEE Access · doi:10.1109/access.2025.3599281 · Cited by 40
“Furthermore, cloud capabilities are improved by edge computing and multi-access edge techniques, which lower latency and improve in real-time processing for mission-critical applications [10], [68].”
An empirical study on ML DevOps adoption trends, efforts, and benefits analysis
Rzig, D. E., Hassan, F., & Kessentini, M. (2022) · Information and Software Technology · doi:10.1016/j.infsof.2022.107037 · Cited by 31
“Based on observations by Karlaš et al. [29], Renggli et al. [16], Lwakatare et al. [11, 66], Amershi et al. [67], and Arpteg et al. [68], we attribute the lower adoption of DevOps by ML Applied projects to the differences between traditional software projects and ML projects, and a lack of DevOps…”
Bernardo, J. H., da Costa, D. A., de Medeiros, S. Q., & Kulesza, U. (2024) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/3643991.3644915 · Cited by 29
“Furthermore, recent efforts aimed to create dedicated CI services for the ML field [21], considering its intricacies (i.e., traditional testing may overfit ML models).”
BackgroundMaterialization and Reuse Optimizations for Production Data Science Pipelines
Derakhshan, B., Mahdiraji, A. R., Kaoudi, Z., Rabl, T., & Markl, V. (2022) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/3514221.3526186 · Cited by 22
“Vertex for P1 only Vertex for P2 only Vertex for P1 and P2 S3 S2 S1 S4 Figure 5: The process of materialization DAG construction. algorithm removes 𝑝2 from all of its ancestors, i.e., ⟨Var-f, [0, 10)⟩, ⟨Scale-t, [0, 10)⟩, ⟨Scale-f, [0, 10)⟩, and ⟨FG-t, [0, 10)⟩.”
BackgroundMethodInfluentialCharacterizing the Usage of CI Tools in ML Projects
Rzig, D. E., Hassan, F., Bansal, C., & Nagappan, N. (2022) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1145/3544902.3546237 · Cited by 26
“Applying this same approach to ML projects by evaluating their results on the same testing set repeatedly can cause problems with their accuracy [40].”
InfluentialMLOps critical success factors - A systematic literature review
Mehmood, Y., Sabahat, N., & Ijaz, M. A. (2024) · VFAST Transactions on Software Engineering · doi:10.21015/vtse.v12i1.1747 · Cited by 20
Ease. ML: A Lifecycle Management System for Machine Learning
Aguilar, L., Dao, D., Gan, S., Gürel, N. M., Hollenstein, N., Jiang, J., Karlaš, B., Lemmin, T., Li, T., Li, Y., et al. (2021) · Repository for Publications and Research Data (ETH Zurich) · doi:10.3929/ethz-b-000458916 · Cited by 24
Self-citationEnabling Collaborative Data Science Development with the Ballet Framework
Smith, M. J., Cito, J., Lu, K., & Veeramachaneni, K. K. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3479575 · Cited by 13
“Some code may introduce errors or decrease the performance of the ML model [46, 48, 69, 81].”
BackgroundInferBench: Understanding Deep Learning Inference Serving with an Automatic Benchmarking SystemPreprint
Zhang, H., Huang, Y., Wen, Y., Yin, J., & Guan, K. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.02327 · Cited by 20
Automation and Development Effort in Continuous AI Development: A Practitioners’ Survey
Steidl, M., Golendukhina, V., Felderer, M., & Ramler, R. (2023) · Euromicro Conference on Software Engineering and Advanced Applications (SEAA) · doi:10.1109/seaa60479.2023.00027 · Cited by 9
Ease.ML: A Lifecycle Management System for MLDev and MLOps
Aguilar, L., Dao, D., Gan, S., Gürel, N. M., Hollenstein, N., Jiang, J., Karlaš, B., Lemmin, T., Tian, L., Li, Y., et al. (2021) · Conference on Innovative Data Systems Research · Cited by 11
Self-citationFan, Y., Shibahara, T., Ohsita, Y., Chiba, D., Akiyama, M., & Murata, M. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-85987-9_6 · Cited by 4
Git Workflow for Active Learning: A Development Methodology Proposal for Data-Centric AI Projects
Stieler, F., & Bauer, B. (2023) · International Conference on Evaluation of Novel Approaches to Software Engineering · doi:10.5220/0011988400003464 · Cited by 3
Siddiqi, S., Phani, A., Kern, R., & Böehm, M. (2026) · ACM Transactions on Database Systems · doi:10.1145/3771766 · Cited by 4
“Recent work includes model assertions [55], validation via unit tests in Deequ [98] and ease.ml/ci [57, 93], as well as tests for model robustness against data corruptions A Scalable Framework for Optimizing Data Cleaning Pipelines for ML Applications 8:27 in Jenga [101], a variety of realistic…”
Improving Automotive Aftermarket Forecasting with MLOps
Sobral, P., Teixeira, R., Marques, R., Figueiredo, N., Antunes, M., & Gomes, D. (2024) · International Conference on Future Internet of Things and Cloud (FiCloud) · doi:10.1109/ficloud62933.2024.00065 · Cited by 2
“MLOps borrows many valuable concepts from DevOps and adapts them to accommodate the dynamic nature of data and ML models [13], [14].”
BackgroundSharma, K., Sharma, P., & Sharma, P. (2026) · Indian Heart Journal · doi:10.1016/j.ihj.2026.06.002 · Cited by 4
GradeML: Towards Holistic Performance Analysis for Machine Learning Workflows
Hegeman, T., Jansen, M., Iosup, A., & Trivedi, A. (2021) · ACM/SPEC International Conference on Performance Engineering (ICPE) · doi:10.1145/3447545.3451185 · Cited by 2
“the bigger Continuous Integration/Continuous Development (CI/CD) trend in ML workflows [28].”
BackgroundZhang, H. (2022) · DR-NTU (Nanyang Technological University) · doi:10.32657/10356/160371 · Cited by 1
Toward a generic federated learning platform optimized for computer vision applications
Zhuang, W. (2022) · DR-NTU (Nanyang Technological University) · doi:10.32657/10356/164712
Teguh, A. M., Slamet, J., Saputro, H., Sungkono, K. R., & Sarno, R. (2024) · International Conference on Technology Innovation and Its Applications (ICTIIA) · doi:10.1109/ictiia61827.2024.10761914
“Additionally, machine learning can aid in defect identification, which is crucial for effective construction quality management, thus improving project outcomes [22].”
الجمل, ز. ش. ح. ا., الفنجري, ل., & Fahmy, H. (2025) · النشرة المعلوماتية في الحاسبات والمعلومات · doi:10.21608/fcihib.2025.261807.1105
Exploiting the Availability-Continuity Trade-off in Imperfect Retraining of Machine Learning Systems
Wang, Z., & Machida, F. (2025) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre66568.2025.00048
“Additionally, models must be retrained on new data due to dataset shifts, further emphasizing the need for automation [29].”
Metadata
- Authors
- Karlaš, B., Interlandi, M., Renggli, C., Wu, W., Zhang, C., Mukunthu Iyappan Babu, D., Edwards, J., Lauren, C., Xu, A., & Weimer, M.
- Affiliations
- ETH Zurich (CH)
- Microsoft (United States) (US)
- ETH Zürich, Zürich, Switzerland
- Microsoft, Redmond, WA, USA
- Venue
- Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining
- Publisher
- ACM
- Published
- 2020-08-20
- Pages
- 2407-2415
- OpenAlex
- W3035597669
- Cited by
- 30
- References
- 16
- Citations by year
- 2026: 1 · 2025: 5 · 2024: 4 · 2023: 6 · 2022: 8 · 2021: 5 · 2020: 1
- FWCI
- 3.79
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Keywords
- continuous integration, machine learning applications
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.