TFX: A TensorFlow-Based Production-Scale Machine Learning Platform
Open accessAbstract
Creating and maintaining a platform for reliably producing and deploying machine learning models requires careful orchestration of many components---a learner for generating models based on training data, modules for analyzing and validating both data as well as models, and finally infrastructure for serving models in production. This becomes particularly challenging when data changes over time and fresh models need to be produced continuously. Unfortunately, such orchestration is often done ad hoc using glue code and custom scripts developed by individual teams for specific use cases, leading to duplicated effort and fragile systems with high technical debt.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Serving-related challenges are some of the ones most often reported in the literature, usually from a performance perspective.
“Serving-related challenges are some of the ones most often reported in the literature [5-7, 9, 11, 12, 16, 20, 28, 34, 44, 53, 54], usually from a performance perspective.”
From Operationalizing machine learning models: a systematic literature review (2022) · cites this paper as [5]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Pan, S. J., & Yang, Q. (2010) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2009.191 · Cited by 23,387 · see citing works
TensorFlow: A system for large-scale machine learningPreprint
Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., et al. (2016) · arXiv (Cornell University) · doi:10.5555/3026877.3026899 · Cited by 16,369 · see citing works
Self-citationHow transferable are features in deep neural networks?
Yosinski, J., Clune, J., Bengio, Y., & Lipson, H. (2014) · arXiv · doi:10.48550/arxiv.1411.1792 · Cited by 6,061 · see citing works
Wide & Deep Learning for Recommender Systems
Cheng, H.-T., Koç, L., Harmsen, J., Shaked, T., Chandra, T., Aradhye, H., Anderson, G., Corrado, G. S., Chai, W., Ispir, M., et al. (2016) · Workshop on Deep Learning for Recommender Systems (DLRS) · doi:10.1145/2988450.2988454 · Cited by 3,547 · see citing works
Self-citationDeep Neural Networks for YouTube Recommendations
Covington, P., Adams, J., & Sargin, E. (2016) · Proceedings of the 10th ACM Conference on Recommender Systems · doi:10.1145/2959100.2959190 · Cited by 3,539 · see citing works
MLlib: Machine Learning in Apache SparkPreprint
Meng, X., Bradley, J. K., Yavuz, B., Sparks, E., Venkataraman, S., Liu, D., Freeman, J., Tsai, D., Amde, M., Owen, S., et al. (2015) · Journal of machine learning research · doi:10.48550/arxiv.1505.06807 · Cited by 1,898 · see citing works
(2015) · neural information processing systems · Cited by 912 · see citing works
McMahan, H. B., Holt, G., Sculley, D., Young, M., Ebner, D., Grady, J., Nie, L., Phillips, T., Davydov, E. V., Golovin, D., et al. (2013) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/2487575.2488200 · Cited by 870 · see citing works
Identifying and attacking the saddle point problem in high-dimensional non-convex optimization
Dauphin, Y., Pascanu, R., Gulcehre, C., Cho, K., Ganguli, S., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1406.2572 · Cited by 827 · see citing works
HyperLogLog: the analysis of a near-optimal cardinality estimation algorithm
Flajolet, P., Fusy, É., Gandouet, O., & Meunier, F. (2007) · Discrete Mathematics & Theoretical Computer Science · doi:10.46298/dmtcs.3545 · Cited by 709 · see citing works
A survey of open source tools for machine learning with big data in the Hadoop ecosystem
Landset, S., Khoshgoftaar, T. M., Richter, A. N., et al. (2015) · Journal Of Big Data · doi:10.1186/s40537-015-0032-1 · Cited by 438 · see citing works
Selection and sorting with limited storage
Munro, J. I., & Paterson, M. (1980) · Theoretical Computer Science · doi:10.1016/0304-3975(80)90061-4 · Cited by 426 · see citing works
MLbase: A Distributed Machine-learning System
Kraska, T., Talwalkar, A., & Duchi, J. C. (2013) · Cited by 310 · see citing works
ActiveClean: interactive data cleaning for statistical modeling
Krishnan, S., Wang, J., Wu, E., Franklin, M. J., & Goldberg, K. (2016) · Proceedings of the VLDB Endowment · doi:10.14778/2994509.2994514 · Cited by 264 · see citing works
Large-scale machine learning at twitter
Lin, J., & Kolcz, A. (2012) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/2213836.2213958 · Cited by 221 · see citing works
Vartak, M., Subramanyam, H., Lee, W.-E., Viswanathan, S., Husnoo, S., Madden, S. R., & Zaharia, M. (2016) · Workshop on Human-In-the-Loop Data Analytics (HILDA) · doi:10.1145/2939502.2939516 · Cited by 157 · see citing works
Click-through Prediction for Advertising in Twitter Timeline
Li, C., Lu, Y., Mei, Q., Wang, D., & Pandey, S. (2015) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/2783258.2788582 · Cited by 117 · see citing works
North Atlantic Right Whale Contact Call DetectionPreprint
Abousleiman, R., Qu, G., & Rawashdeh, O. A. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1304.7851 · Cited by 4 · see citing works
KeystoneML: Optimizing Pipelines for Large-Scale Advanced AnalyticsPreprint
Sparks, E., Venkataraman, S., Kaftan, T., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.09451 · Cited by 4 · see citing works
Flagged first, then by citations
Research works printed but not linked: 3
- Manasi Vartak, Harihar Subramanyam, Wei-En Lee, Srinidhi Viswanathan, Saadiyah Husnoo, Samuel Madden, and Matei Zaharia. 2016. ModelDB: a system for machine learning model management. In HILDA@SIGMOD. 14.
- Cassandra Xia, Clemens Mewald, D. Sculley, David So- ergel, George Roumpos, Heng-Tze Cheng, Illia Polosukhin, Jamie Alexander Smith, Jianwei Xie, Lichan Hong, Martin Wicke, Mustafa Ispir, Philip Daniel Tucker, Yuan Tang, and Zakaria Haque. 2017. Train and Distribute: Managing Simplicity vs. Flex- ibility in High-Level Machine Learning Frameworks. KDD (under review).
- Martin Zinkevich. 2016. Rules of Machine Learning. In NIPS Workshop on Reliable Machine Learning. Invited Talk.
Cited by
Sorted by citations
Machine Learning Testing: Survey, Landscapes and Horizons
Zhang, J. M., Harman, M., Ma, L., & Liu, Y. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2962027 · Cited by 897 · see citing works
The ML test score: A rubric for ML production readiness and technical debt reduction
Breck, E., Cai, S., Nielsen, E., Salib, M., & Sculley, D. (2017) · 2017 IEEE International Conference on Big Data (Big Data) · doi:10.1109/bigdata.2017.8258038 · Cited by 310 · see citing works
Self-citationHutchinson, B., Smart, A., Hanna, A., Denton, R., Greer, C., Kjartansson, O., Barnes, P., & Mitchell, M. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445918 · Cited by 279 · see citing works
Lwakatare, L. E., Raj, A., Crnkovic, I., Bosch, J., & Olsson, H. H. (2020) · Information and Software Technology · doi:10.1016/j.infsof.2020.106368 · Cited by 241 · see citing works
Roselli, D., Matthews, J., & Talagala, N. (2019) · Companion Proceedings of The 2019 World Wide Web Conference · doi:10.1145/3308560.3317590 · Cited by 237 · see citing works
Data Lifecycle Challenges in Production Machine Learning: A Survey
Polyzotis, N., Roy, S., Whang, S. E., & Zinkevich, M. (2018) · ACM SIGMOD Record · doi:10.1145/3299887.3299891 · Cited by 214 · see citing works
Self-citationHow does Machine Learning Change Software Development Practices?
Wan, Z., Xia, X., Lo, D., & Murphy, G. C. (2020) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2937083 · Cited by 181 · see citing works
Steidl, M., Felderer, M., & Ramler, R. (2023) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111615 · Cited by 126 · see citing works
Developments in MLflow: A System to Accelerate the Machine Learning Lifecycle
Chen, A., Chow, A., Davidson, A., DCunha, A., Ghodsi, A., Hong, S. A., Konwinski, A., Mewald, C., Murching, S., Nykodym, T., et al. (2020) · Proceedings of the Fourth International Workshop on Data Management for End-to-End Machine Learning · doi:10.1145/3399579.3399867 · Cited by 112 · see citing works
Self-citationDemystifying MLOps and Presenting a Recipe for the Selection of Open-Source Tools
Ruf, P., Madan, M., Reich, C., & Ould-Abdeslam, D. (2021) · Applied Sciences · doi:10.3390/app11198861 · Cited by 112 · see citing works
ModelOps: Cloud-Based Lifecycle Management for Reliable and Trusted AI
Hummer, W., Muthusamy, V., Rausch, T., Dube, P., El Maghraoui, K., Murthi, A., & Oum, P. (2019) · 2019 IEEE International Conference on Cloud Engineering (IC2E) · doi:10.1109/ic2e.2019.00025 · Cited by 105 · see citing works
Adoption and Effects of Software Engineering Best Practices in Machine Learning
Serban, A., van der Blom, K., Hoos, H., & Visser, J. (2020) · Proceedings of the 14th ACM / IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM) · doi:10.1145/3382494.3410681 · Cited by 127 · see citing works
Towards MLOps: A Case Study of ML Pipeline Platform
Zhou, Y., Yu, Y., & Ding, B. (2020) · 2020 International Conference on Artificial Intelligence and Computer Engineering (ICAICE) · doi:10.1109/icaice51518.2020.00102 · Cited by 103 · see citing works
Slice Finder: Automated Data Slicing for Model Validation
Chung, Y., Kraska, T., Polyzotis, N., Tae, K. H., & Whang, S. E. (2019) · 2019 IEEE 35th International Conference on Data Engineering (ICDE) · doi:10.1109/icde.2019.00139 · Cited by 83 · see citing works
Self-citationEdge Intelligence: The Convergence of Humans, Things, and AI
Rausch, T., & Dustdar, S. (2019) · 2019 IEEE International Conference on Cloud Engineering (IC2E) · doi:10.1109/ic2e.2019.00022 · Cited by 82 · see citing works
Artificial intelligence bias in medical system designs: a systematic review
Kumar, A., Aelgani, V., Vohra, R., Gupta, S. K., Bhagawati, M., Paul, S., Saba, L., Suri, N., Khanna, N. N., Laird, J. R., et al. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-023-16029-x · Cited by 65 · see citing works
Data Cleaning for Accurate, Fair, and Robust Models: A Big Data - AI Integration Approach
Tae, K. H., Roh, Y., Oh, Y. H., Kim, H., & Whang, S. E. (2019) · Proceedings of the 3rd International Workshop on Data Management for End-to-End Machine Learning · doi:10.1145/3329486.3329493 · Cited by 62 · see citing works
Self-citationGranlund, T., Stirbu, V., & Mikkonen, T. (2021) · SN Computer Science · doi:10.1007/s42979-021-00726-1 · Cited by 53 · see citing works
Parity models: erasure-coded resilience for prediction serving systems
Kosaian, J., Rashmi, K. V., & Venkataraman, S. (2019) · Proceedings of the 27th ACM Symposium on Operating Systems Principles · doi:10.1145/3341301.3359654 · Cited by 48 · see citing works
Towards Trustworthy Machine Learning in Production: An Overview of the Robustness in MLOps Approach
Bayram, F., & Ahmed, B. S. (2025) · ACM Computing Surveys · doi:10.1145/3708497 · Cited by 48 · see citing works
Benchmarking Machine Learning Solutions in Production
Cardoso Silva, L., Rezende Zagatti, F., Silva Sette, B., Nildaimon dos Santos Silva, L., Lucredio, D., Furtado Silva, D., & de Medeiros Caseli, H. (2020) · 2020 19th IEEE International Conference on Machine Learning and Applications (ICMLA) · doi:10.1109/icmla51294.2020.00104 · Cited by 33 · see citing works
Operationalizing machine learning models: a systematic literature review
Kolltveit, A. B., & Li, J. (2022) · Proceedings of the 1st Workshop on Software Engineering for Responsible AI · doi:10.1145/3526073.3527584 · Cited by 34 · see citing works
Migrating a Recommendation System to Cloud Using ML Workflow
Chahal, D., Ojha, R., Choudhury, S. R., & Nambiar, M. (2020) · Companion of the ACM/SPEC International Conference on Performance Engineering · doi:10.1145/3375555.3384423 · Cited by 10 · see citing works
- Authors
- Baylor, D., Breck, E., Cheng, H.-T., Fiedel, N., Foo, C. Y., Haque, Z., Haykal, S., Ispir, M., Jain, V., Koc, L., Koo, C. Y., Lew, L., Mewald, C., Modi, A. N., Polyzotis, N., Ramesh, S., Roy, S., Whang, S. E., Wicke, M., Wilkiewicz, J., Zhang, X., & Zinkevich, M.
- Affiliations
- Google (United States) (US)
- Google Inc., Mountain View, CA, USA
- Venue
- Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining
- Publisher
- ACM
- Published
- 2017-08-13
- Pages
- 1387-1395
- OpenAlex
- W2743948853
- Cited by
- 350
- References
- 19
- Citations by year
- 2026: 45 · 2025: 33 · 2024: 20 · 2023: 28 · 2022: 32 · 2021: 56 · 2020: 48 · 2019: 63 · 2018: 20 · 2017: 1 · 2016: 4
- FWCI
- 27.93
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
- Physical Sciences › Computer Science › Computer Networks and Communications › Software System Performance and Reliability
- Keywords
- model deployment, machine learning pipelines, data validation, model validation, data drift, technical debt, MLOps
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
