Eliminating the Variability of Cross-Validation Results with LIBLINEAR due to Randomization and Parallelization
Open accessAbstract
Cross-validation is the gold standard for evaluating machine learning algorithms or finetuning their parameters.The results of this technique, however, are not always reproducible and may depend on the computing platform and the number of parallel threads, especially if the underlying learning algorithm uses a pseudo-random number generator (PRNG).This paper gives a recipe for solving these reproducibility problems and applies it to LIBLINEAR 1 , a popular software library that implements randomized learning algorithms based on support vector machines 2 .The proposed approach solves these problems by using a cross-platform PRNG and by making the PRNG state private in each thread.The cross-validation results obtained with the modified version of LIBLINEAR are the same across platforms.Furthermore, the parallelized cross-validation results are no longer affected by random fluctuations arising from the sharing of the PRNG state by parallel threads.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Running an algorithm on different sets of hardware has been shown to affect the reproducibility of algorithmic results.
“Reproducibility: Running an algorithm on different sets of hardware has been shown to affect the reproducibility of algorithmic results (Gundersen and Kjensmo, 2018; Sukhoy and Stoytchev, 2019).”
From Towards the Systematic Reporting of the Energy and Carbon Footprints of Machine Learning (2020) · cites this paper as Sukhoy and Stoytchev, 2019
The Introduction (p. 1) says cross-validation results can depend on the computing platform, and the paper reports reproducibility across platforms and compilers after modifying LIBLINEAR. It identifies operating systems, compilers, and runtime libraries as platform differences; it does not specifically show that different hardware affects results, and its finding concerns cross-validation with randomized learning algorithms rather than algorithms generally. (LLM, conf=0.84)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
The citation lists haven’t been fetched yet
Cited by 1
- Authors
- Sukhoy, V., & Stoytchev, A.
- Venue
- Zenodo
- Publisher
- Zenodo
- Published
- 2019-11-04
- OpenAlex
- W3209506400
- Cited by
- 1
- References
- 0
- Citations by year
- 2020: 1
- FWCI
- 0.11
- Type
- journal article
- Language
- en
- Open access
- green
- zenodo.org
- Licence
- Locations
- Zenodo (CERN European Organization for Nuclear Research) (publishedVersion, cc-by)
- Zenodo (CERN European Organization for Nuclear Research) (submittedVersion, cc-by)
- Zenodo (CERN European Organization for Nuclear Research) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Inference
- Keywords
- cross-validation, randomization, parallelization, reproducibility, model selection, support vector machine
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
