ML Reproducibility Challenge 2022
Open accessAbstract
Editorial
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Of the 44 papers submitted to the 2022 Machine Learning Reproducibility Challenge, 28 attempted to reproduce papers where both data and code were fully available.
“Of the 44 papers submitted to the challenge, 28 attempted to reproduce papers where both data and code were fully available.”
From CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmark (2024) · cites this paper as Sinha et al., 2023
The Challenge section reports 74 reproducibility-report submissions, not 44. The following section says 45 reports were accepted from those 74 submissions; neither section reports that 28 attempted to reproduce papers with both data and code fully available. (LLM, conf=0.88)
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Still being checked.
Cited by
Sorted by citations
MixupExplainer: Generalizing Explanations for Graph Neural Networks with Data Augmentation
Zhang, J., Luo, D., & Wei, H. (2023) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3580305.3599435 · Cited by 26 · see citing works
FAIR AI models in high energy physics
Duarte, J. M., Li, H., Roy, A., Zhu, R., Huerta, E. A., Díaz, D., Harris, P. C., Kansal, R., Katz, D. S., Kavoori, I. H., et al. (2023) · Machine Learning Science and Technology · doi:10.1088/2632-2153/ad12e3 · Cited by 13 · see citing works
Reproducibility in Computational Linguistics: Is Source Code Enough?
Arvan, M., Pina, L., & Parde, N. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.150 · Cited by 12 · see citing works
Reproduce, Replicate, Reevaluate. The Long but Safe Way to Extend Machine Learning Methods
Werner, L. S., Layaïda, N., Genevès, P., Euzenat, J., & Graux, D. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i14.29515 · Cited by 7 · see citing works
MLXP: A framework for conducting replicable experiments in Python
Arbel, M., & Zouaoui, A. (2024) · ACM Conference on Reproducibility and Replicability (ACM REP) · doi:10.1145/3641525.3663648 · Cited by 5 · see citing works
Laying Foundations to Quantify the “Effort of Reproducibility”
Akella, A. P., Koop, D., & Alhoori, H. (2023) · ACM/IEEE Joint Conference on Digital Libraries · doi:10.1109/jcdl57899.2023.00018 · Cited by 5 · see citing works
Siegel, Z. S., Kapoor, S., Nadgir, N., Stroebl, B., & Narayanan, A. (2024) · arXiv · doi:10.48550/arxiv.2409.11363 · Cited by 4 · see citing works
Advancing white balance correction through deep feature statistics and feature distribution matching
Kınlı, F., Özcan, B., & Kıraç, F. (2025) · Journal of Visual Communication and Image Representation · doi:10.1016/j.jvcir.2025.104412 · Cited by 4 · see citing works
Coca-Castro, A., Fouilloux, A., Lourenço, R. B., McDonald, A. T., Rao, Y. D., & Hosking, J. S. (2025) · Environmental Data Science · doi:10.1017/eds.2024.35 · Cited by 3 · see citing works
Wang, Y., Wang, C., Mou, L.-H., & Jiang, J. (2025) · JACS Au · doi:10.1021/jacsau.5c00764 · Cited by 3 · see citing works
Cheng, K., Si, Y., Zhou, H., & Tahir, R. (2022) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3503161.3548260 · Cited by 1 · see citing works
(2024) · Environmental Data Science · doi:10.1017/eds.2024.35.pr4
- Authors
- Sinha, K., Bleeker, M., Bhargav, S., Forde, J. Z., Raparthy, S. C., Dodge, J., Pineau, J., & Stojnic, R.
- ORCID
- Venue
- Zenodo
- Publisher
- Zenodo
- Published
- 2023-07-31
- OpenAlex
- W4385404463
- Cited by
- 10
- References
- 0
- Citations by year
- 2025: 3 · 2024: 2 · 2023: 3 · 2022: 2
- FWCI
- 2.06
- Type
- journal article
- Language
- en
- Open access
- green
- doi.org
- Licence
- Locations
- Zenodo (CERN European Organization for Nuclear Research) (cc-by)
- Topics
- Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
- Keywords
- reproducibility, machine learning, reproducibility challenge
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
