Collective knowledge: organizing research projects as a database of reusable components and portable workflows with common interfaces
Open accessAbstract
This article provides the motivation and overview of the Collective Knowledge Framework (CK or cKnowledge). The CK concept is to decompose research projects into reusable components that encapsulate research artifacts and provide unified application programming interfaces (APIs), command-line interfaces (CLIs), meta descriptions and common automation actions for related artifacts. The CK framework is used to organize and manage research projects as a database of such components. Inspired by the USB ‘plug and play’ approach for hardware, CK also helps to assemble portable workflows that can automatically plug in compatible components from different users and vendors (models, datasets, frameworks, compilers, tools). Such workflows can build and run algorithms on different platforms and environments in a unified way using the customizable CK program pipeline with software detection plugins and the automatic installation of missing packages. This article presents a number of industrial projects in which the modular CK approach was successfully validated in order to automate benchmarking, auto-tuning and co-design of efficient software and hardware for machine learning and artificial intelligence in terms of speed, accuracy, energy, size and various costs. The CK framework also helped to automate the artifact evaluation process at several computer science conferences as well as to make it easier to reproduce, compare and reuse research techniques from published papers, deploy them in production, and automatically adapt them to continuously changing datasets, models and systems. The long-term goal is to accelerate innovation by connecting researchers and practitioners to share and reuse all their knowledge, best practices, artifacts, workflows and experimental results in a common, portable and reproducible format at cKnowledge.io . This article is part of the theme issue ‘Reliability and reproducibility in computational science: implementing verification, validation and uncertainty quantification in silico ’.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Versioning of data, model, and code is required to ensure robustness and reproducibility.
“as well as versioning of data, model, and code to ensure robustness and reproducibility [7], [32], [39].”
From Machine Learning Operations (MLOps): Overview, Definition, and Architecture (2023) · cites this paper as [39]
In §6 (p. 13), the paper says a lack of versioning made CK workflows difficult to keep stable and bug-free, and describes a platform intended to version and test components and workflows. That supports the general link between versioning and reliability, but it does not state that versioning each of data, models, and code is required to ensure robustness and reproducibility. (LLM, conf=0.82)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The FAIR Guiding Principles for scientific data management and stewardship
Wilkinson, M. D., Dumontier, M., Aalbersberg, I. J., Appleton, G., Axton, M., Baak, A., Blomberg, N., Boiten, J.-W., da Silva Santos, L. B., Bourne, P. E., et al. (2016) · Scientific Data · doi:10.1038/sdata.2016.18 · Cited by 18,423 · see citing works
Docker: lightweight Linux containers for consistent development and deployment
Merkel, D. (2014) · Linux journal · Cited by 3,277 · see citing works
Wolstencroft, K., Haines, R., Fellows, D. K., Williams, A. R., Withers, D., Owen, S., Soiland‐Reyes, S., Dunlop, I., Nenadić, A., Fisher, P., et al. (2013) · Nucleic Acids Research · doi:10.1093/nar/gkt328 · Cited by 671 · see citing works
Gamblin, T., LeGendre, M., Collette, M., Lee, G. L., Moody, A. T., de Supinski, B. R., & Futral, S. (2015) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1145/2807591.2807623 · Cited by 286 · see citing works
Milepost GCC: Machine Learning Enabled Self-tuning Compiler
Fursin, G., Kashnikov, Y., Memon, A. W., et al. (2011) · International Journal of Parallel Programming · doi:10.1007/s10766-010-0161-2 · Cited by 238 · see citing works
Self-citationCommon Workflow Language, v1.0
Amstutz, P., Crusoe, M. R., Tijanić, N., et al. (2016) · eScholarship (California Digital Library) · doi:10.6084/m9.figshare.3115156.v2 · Cited by 210 · see citing works
Common Workflow Language, draft 3
Amstutz, P., Andeer, R., Chapman, B., Chilton, J., Crusoe, M. R., Guimerà, R. V., Hernandez, G. C., Ivkovic, S., Kartashov, A., Kern, J., et al. (2016) · eScholarship (California Digital Library) · doi:10.6084/m9.figshare.3115156 · Cited by 204 · see citing works
Extreme scale multi-physics simulations of the tsunamigenic 2004 sumatra megathrust earthquake
Uphoff, C., Rettenberger, S., Bader, M., et al. (2017) · 2012 International Conference for High Performance Computing, Networking, Storage and Analysis · doi:10.1145/3126908.3126948 · Cited by 103 · see citing works
A Modular Benchmarking Infrastructure for High-Performance and Reproducible Deep Learning
Ben‐Nun, T., Besta, M., Huber, S., Ziogas, A. N., Peter, D., & Hoefler, T. (2019) · IEEE International Parallel and Distributed Processing Symposium (IPDPS) · doi:10.1109/ipdps.2019.00018 · Cited by 74 · see citing works
Producing wrong data without doing anything obviously wrong!
Mytkowicz, T., Diwan, A., Hauswirth, M., et al. (2009) · ACM SIGPLAN Notices · doi:10.1145/1508284.1508275 · Cited by 68 · see citing works
EasyBuild: Building Software with Ease
Hoste, K., Timmerman, J., Georges, A., et al. (2012) · doi:10.1109/sc.companion.2012.81 · Cited by 61 · see citing works
The Popper Convention: Making Reproducible Systems Evaluation Practical
Jimenez, I., Sevilla, M. A., Watkins, N. W., et al. (2017) · IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW) · doi:10.1109/ipdpsw.2017.157 · Cited by 58 · see citing works
XSP: Across-Stack Profiling and Analysis of Machine Learning Models on GPUs
Li, C., Dakkak, A., Xiong, J., et al. (2020) · Proceedings - IEEE International Parallel and Distributed Processing Symposium · doi:10.1109/ipdps47924.2020.00042 · Cited by 30 · see citing works
Community-driven reviewing and validation of publications
Fursin, G., & Dubach, C. (2014) · doi:10.1145/2618137.2618142 · Cited by 12 · see citing works
Self-citationArtifact Evaluation for Publications (Dagstuhl Perspectives Workshop 15452)
Childers, B. R., Fursin, G., Krishnamurthi, S., et al. (2016) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/dagrep.5.11.29 · Cited by 6 · see citing works
Self-citationEnabling reproducible ML and Systems research: the good, the bad, and the ugly
Fursin, G. (2020) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.4005773 · Cited by 5 · see citing works
Self-citationEnabling reproducible ML and Systems research: the good, the bad, and the ugly
Fursin, G. (2020) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.4005587 · Cited by 5 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 4
- [1][1] ACM Pilot demo 'Collective Knowledge: packaging and sharing'. https://youtu.be/DIkZxraTmGM . -
- [21][21] LPIRC: low-power image recognition challenge. https://rebootingcomputing.ieee.org/lpirc . -
- [53][53] Vijay Janapa Reddi, Christine Cheng, David Kanter, Peter Mattson, Guenther Schmuelling, Carole-Jean Wu, Brian Anderson, Maximilien Breughe, Mark Charlebois, William Chou, Ramesh Chukka, Cody Coleman, Sam Davis, Pan Deng, Greg Diamos, Jared Duke, Dave Fick, J. Scott Gardner, Itay Hubara, Sachin Idgunji, Thomas B. Jablin, Jeff Jiao, Tom St. John, Pankaj Kanwar, David Lee, Jeffery Liao, Anton Lokhmotov, Francisco Massa, Peng Meng, Paulius Micikevicius, Colin Osborne, Gennady Pekhimenko, Arun Tejusve Raghunath Rajan, Dilip Sequeira, Ashish Sirasao, Fei Sun, Hanlin Tang, Michael Thomson, Frank Wei, Ephrem Wu, Lingjie Xu, Koichi Yamada, Bing Yu, George Yuan, Aaron Zhong, Peizhao Zhang, and Yuchen Zhou. MLPerf Inference Benchmark. https:// arxiv.org/abs/1911.02549 , 2019. - · arXiv
- [57][57] Matei Zaharia, Andrew Chen, Aaron Davidson, Ali Ghodsi, Sue Ann Hong, Andy Konwinski, Siddharth Murching, Tomas Nykodym, Paul Ogilvie, Mani Parkhe, Fen Xie, and Corey Zumar. Accelerating the Machine Learning Lifecycle with MLflow. IEEE Data Eng. Bull. , 41:39-45, 2018.
Cited by
Sorted by citations
Machine Learning Operations (MLOps): Overview, Definition, and Architecture
Kreuzberger, D., Kühl, N., & Hirschl, S. (2023) · IEEE Access · doi:10.1109/access.2023.3262138 · Cited by 766 · see citing works
“As a foundational infrastructure layer, a scalable and distributed model serving infrastructure is recommended [7,11,25,40,45,46]”
Background
- Authors
- Fursin, G.
- Affiliations
- cTuning foundation and cKnowledge SAS
- Corresponding author
- Fursin, Grigori
- ORCID
- Fursin, Grigori
- Venue
- Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences
- Publisher
- The Royal Society
- Published
- 2021-03-29
- Volume
- 379
- Issue
- 2197
- Pages
- 20200211-20200211
- ISSN
- 1364-503X (ISSN-L)
- 1471-2962
- OpenAlex
- W3096672856
- Cited by
- 33
- References
- 17
- Citations by year
- 2026: 1 · 2025: 5 · 2024: 4 · 2023: 4 · 2022: 10 · 2021: 9
- FWCI
- 5.76
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- Philosophical Transactions of the Royal Society A Mathematical Physical and Engineering Sciences (publishedVersion)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
- Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Keywords
- reusable components, auto-tuning, hardware-software co-design, reproducibility
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
