The DOI matches its Crossref record; 1 of the 1 references it cites were found in a registry or an index.
E-values: Calibration, combination and applications
Open accessAbstract
Multiple testing of a single hypothesis and testing multiple hypotheses are usually done in terms of p-values. In this paper, we replace p-values with their natural competitor, e-values, which are closely related to betting, Bayes factors and likelihood ratios. We demonstrate that e-values are often mathematically more tractable; in particular, in multiple testing of a single hypothesis, e-values can be merged simply by averaging them. This allows us to develop efficient procedures using e-values for testing multiple hypotheses.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The paper's sequential tools are conformal test martingales.
“our sequential tools are conformal test martingales (Vovk, 2021)”
From The Limits of Query Monitoring against Distributed Model Extraction (2026) · cites this paper as Vovk, 2021
The “Testing with martingales” section explains how sequential e-values yield test supermartingales and, under an equality condition, test martingales. Our copy does not identify these as conformal test martingales, so it supports the martingale connection but not the conformal qualification. No page number is available in the indexed passages. (LLM, conf=0.82)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
An improved Bonferroni procedure for multiple tests of significance
SIMES, R. J. (1986) · Biometrika · doi:10.1093/biomet/73.3.751 · Cited by 2,396
Flagged first, then by citations
Research works printed but not linked: 21
- Shafer, G., Shen, A., Vereshchagin, N. and Vovk, V. (2011). Test martingales, Bayes factors and p-values. Statist. Sci. 26 84–101. · DOI
- Dobriban, E. (2020). Fast closed testing for exchangeable local tests. Biometrika 107 761–768. · DOI
- Goeman, J. J. and Solari, A. (2011). Multiple testing for exploratory research. Statist. Sci. 26 584–597. · DOI
- Marcus, R., Peritz, E. and Gabriel, K. R. (1976). On closed testing procedures with special reference to ordered analysis of variance. Biometrika 63 655–660. · DOI
- Shaffer, J. P. (1986). Modified sequentially rejective multiple test procedures. J. Amer. Statist. Assoc. 81 826–831. · DOI
- Benjamin, D. J. and Berger, J. O. (2019). Three recommendations for improving the use of p-values. Amer. Statist. 73 186–191. · DOI
- Duan, B., Ramdas, A., Balakrishnan, S. and Wasserman, L. (2019). Interactive martingale tests for the global null. Technical report. Available at arXiv:1909.07339 [stat.ME]. · arXiv
- Embrechts, P., Wang, B. and Wang, R. (2015). Aggregation-robustness and model uncertainty of regulatory risk measures. Finance Stoch. 19 763–790. · DOI
- Fisher, R. A. (1932). Statistical Methods for Research Workers, 4th ed. Oliver and Boyd, Edinburgh. Section 21.1 on combining independent p-values first appears in this edition and is present in all subsequent editions.
- Grünwald, P., de Heide, R. and Koolen, W. M. (2020). Safe testing. Technical report. Available at arXiv:1906.07801 [math.ST]. · arXiv
- Hardy, G. H., Littlewood, J. E. and Pólya, G. (1952). Inequalities, 2nd ed. Cambridge Univ. Press, Cambridge.
- Howard, S. R., Ramdas, A., McAuliffe, J. and Sekhon, J. (2020). Time-uniform, nonparametric, nonasymptotic confidence sequences. Technical report. Available at arXiv:1810.08240 [math.ST]. Ann. Statist. To appear. · DOI · Paper page
- Jeffreys, H. (1961). Theory of Probability, 3rd ed. Clarendon Press, Oxford.
- Rüger, B. (1978). Das maximale Signifikanzniveau des Tests: “Lehne H0 ab, wenn k unter n gegebenen Tests zur Ablehnung führen”. Metrika 25 171–178. · DOI
- Shafer, G. (2019). The language of betting as a strategy for statistical and scientific communication. Technical report. Available at arXiv:1903.06991 [math.ST]. J. R. Stat. Soc., A. To appear. · arXiv
- Shafer, G. and Vovk, V. (2019). Game-Theoretic Foundations for Probability and Finance. Wiley, Hoboken, NJ. · DOI
- Vovk, V. G. (1993). A logic of probability, with application to the foundations of statistics. J. Roy. Statist. Soc. Ser. B 55 317–351. With discussion and a reply by the author. · DOI
- Vovk, V. and Wang, R. (2020). Combining p-values via averaging. Technical report. Available at arXiv:1212.4966 [math.ST]. Biometrika. To appear. https://doi.org/10.1093/biomet/asaa027. · DOI
- Vovk, V. and Wang, R. (2021). Supplement to “E-values: Calibration, combination, and applications.” https://doi.org/10.1214/20-AOS2020SUPP · DOI
- Wald, A. (1950). Statistical Decision Functions. Wiley, New York, NY.
- Wright, S. P. (1992). Adjusted p-values for simultaneous inference. Biometrics 48 1005–1013. · DOI · Paper page
Cited by
Sorted by citations
The Limits of Query Monitoring against Distributed Model Extraction
Khilar, S. C. (2026) · arXiv
“Our sequential tools are e-values and conformal test martingales (Vovk & Wang, 2021; Vovk, 2021; Vovk et al., 2021; Ramdas et al., 2023).”
Metadata
- Authors
- Vovk, V., & Wang, R.
- Affiliations
- University of London (GB)
- University of Waterloo (CA)
- Venue
- The Annals of Statistics
- Publisher
- Institute of Mathematical Statistics
- Published
- 2021-06-01
- Volume
- 49
- Issue
- 3
- Pages
- 1736-1754
- ISSN
- 0090-5364 (ISSN-L)
- 2168-8966
- OpenAlex
- W3087003407
- Cited by
- 172
- References
- 1
- Citations by year
- 2026: 44 · 2025: 35 · 2024: 27 · 2023: 28 · 2022: 19 · 2021: 14
- FWCI
- 15.95
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- The Annals of Statistics (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Natural Sciences and Engineering Research Council of Canada (rgpin-2018-03823, RGPAS‐2018‐522590, RGPAS-2018, RGPIN 2018)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Inference
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Keywords
- E-value, multiple comparisons, p-values, calibration, betting, Bayes factor, likelihood ratios
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
