CoChat
Sign Up
Cited by 384Natural Language Processing TechniquesTopic ModelingComputational and Text Analysis Methods

The Hitchhiker’s Guide to Testing Statistical Significance in Natural Language Processing

Open access
Dror, Rotem, Baumer, Gili, Shlomov, Segev, Reichart, Roi·2018·Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
DOI 10.18653/v1/p18-1128
Open this paper in CoChat — free accountOpen PDF

Abstract

Statistical significance testing is a standard statistical tool designed to ensure that experimental results are not coincidental.In this opinion/theoretical paper we discuss the role of statistical significance testing in Natural Language Processing (NLP) research.We establish the fundamental concepts of significance testing and discuss the specific aspects of NLP tasks, experimental setups and evaluation measures that affect the choice of significance tests in NLP research.Based on this discussion, we propose a simple practical protocol for statistical significance test selection in NLP setups and accompany this protocol with a brief survey of the most relevant tests.We then survey recent empirical papers published in ACL and TACL during 2017 and show that while our community assigns great value to experimental results, statistical significance testing is often ignored or misused.We conclude with a brief discussion of open issues that should be properly addressed so that this important tool can be applied in NLP research in a statistically sound manner 1 .

AI snapshot

Written from the paper’s full text.

TL;DR

This opinion/theoretical paper explains how to choose statistical significance tests for NLP and proposes a simple decision-tree protocol. A survey of ACL 2017 and TACL 2017 papers finds that significance testing is often ignored or misused.

Key findings

  • In ACL 2017, of 180 experimental long papers, only 63 reported significance testing; 21 of those did not name the test, and 6 of the 42 that named a test used the wrong one.
  • In TACL 2017, of 33 experimental papers, 15 did not include significance testing; all papers that reported significance and named the test used a valid test.
  • Among papers needing replicability analysis, only 3 of 110 ACL papers and 4 of 19 TACL papers corrected for multiplicity.
  • The paper proposes a decision tree: use parametric tests if the test statistic distribution is known; otherwise use sampling-based non-parametric tests for small data or sampling-free non-parametric tests for large data.

Methods

  • The paper is an opinion/theoretical guide that reviews significance testing concepts, NLP evaluation measures, and prominent parametric and non-parametric tests.
  • It surveys long papers from ACL 2017 (196 total, 180 experimental) and TACL 2017 Volume 5 Issue 1 (37 total, 33 experimental), focusing on empirical papers with at least one method comparison.
  • The main analysis is descriptive counting of evaluation measures, significance test use, test names, wrong tests, and multiplicity corrections.

Limitations

  • The paper states open issues: most significance tests assume independent observations, but NLP datasets often contain dependent examples, and the nature of these dependencies is hard to characterize or quantify.
  • The paper states that reporting significance under cross-validation is challenging because fold test sets are not independent and may overlap; it proposes using replicability analysis with the K Bonferroni estimator but calls for further statistical investigation.
  • Evident from the text: the empirical survey is limited to ACL 2017 long papers and TACL 2017, so the reported rates may not generalize to other venues or years.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.18653/v1/p18-1128

Cites

43 of 43 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

6 of 28 cited sources checked · 2 supported · 0 flagged · accuracy not yet measured

  • Premisereasoning

    McNemar's test is appropriate for binary classification tasks.

    “This test is appropriate for binary classification tasks, and has been indeed used in such NLP works (e.g.sentiment classificaiton, (Blitzer et al., 2006;Ziser and Reichart, 2017)).”
    Quote in the paperNot a substantial claim2 of 2 sources checked
    • cites Blitzer et al., 2006 source

      The Empirical Results section (Figures 5–6; page not available) shows that Blitzer et al. used McNemar’s test to compare NLP systems. However, those results concern part-of-speech tagging, not sentiment classification, and the paper does not state that the test is appropriate for binary classification tasks. The binary classification problems in §3.2 are pivot predictors, not the task for which the reported McNemar tests were used. (LLM, conf=0.88)

      Experimental — accuracy not yet measured
    • cites Ziser and Reichart, 2017 source
  • Interpretationcitation

    A statistically sound way to test for significance under cross-validation is to calculate the p-value for each fold separately and then perform replicability analysis for dependent datasets with K Bonferroni.

    “One statistically sound way to test for significance when a cross-validation protocol is employed is hence to calculate the pvalue for each fold separately, and then to perform replicability analysis for dependent datasets with K Bonf erroni .”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Dror et al., 2017 source

      Section 4.1 recommends Bonferroni-based partial conjunction testing for dependent datasets, and the Framework Summary recommends reporting kBonferroni when independence cannot be assumed. Our copy does not establish that cross-validation folds should each receive a separate p-value or that folds should be treated as the datasets in this framework (searched: cross-validation, folds). (LLM, conf=0.85)

      Experimental — accuracy not yet measured
  • Premisecitation

    The Wilcoxon signed-rank test is applicable for most NLP setups.

    “The test is actually applicable for most NLP setups and it has been used widely (e.g.(Søgaard et al., 2014;Søgaard, 2013;Yang and Mitchell, 2017)) due to its improved power compared to the sign test variants.”
    Quote in the paperNot a substantial claim2 of 3 sources checked
    • cites Søgaard et al., 2014 source

      The Introduction says Wilcoxon’s test is used for comparisons across data sets, and Section 2 says rank-based tests are sometimes used in NLP, especially with few observations. This supports its use in some NLP setups, but not its applicability to most setups or widespread use. Section 2 does not compare its power with sign-test variants; it instead describes rank-based tests as relatively weak compared with bootstrap testing. (LLM, conf=0.88)

      Experimental — accuracy not yet measured
    • cites Søgaard, 2013

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Yang and Mitchell, 2017 source

      Section 4.1.1 (Implementation Details; page unavailable) reports using a Wilcoxon rank-sum test to compare NLP models, so the paper provides an example of a Wilcoxon test in an NLP setup. It does not report using the signed-rank test or establish that signed-rank testing applies to most NLP setups; it also does not compare its power with sign-test variants. (LLM, conf=0.84)

      Experimental — accuracy not yet measured
  • Findingnone given

    Seven of the 12 most common evaluation measures in ACL/TACL 2017 (Accuracy, Precision, Recall, F-score, Pearson and Spearman correlations, Perplexity) are not specific to NLP.

    “Notice that seven of these measures: Accuracy, Precision, Recall, F-score, Pearson and Spearman correlations and Perplexity, are not specific to NLP.”
    Quote in the paperNot a substantial claim
  • Findingown data

    The most commonly used evaluation measures in long ACL and TACL 2017 papers are the 12 measures in Table 1.

    “In § 4 we analyze the (long) ACL and TACL 2017 papers, and observe that the most commonly used evaluation measures are the 12 measures that appear in Table 1.”
    Quote in the paperNot a substantial claim
  • Classified as probably not a substantial claim. 29
  • May combine two claims. 17

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Dror et al. (2017) proposes a statistical significance framework for multiple comparisons performed with dependent test sets, using the K Bonferroni estimator for the number of datasets with significant effect.

    “This paper proposes a statistical significance framework for multiple comparisons performed with dependent test sets, using the K Bonf erroni estimator for the number of datasets with significant effect.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Dror et al., 2017 source

  • May combine two claims.

    Findingown data

    A large portion of the surveyed long ACL and TACL 2017 papers do not test the significance of their results or use incorrect tests for this purpose.

    “Particularly, a large portion of the surveyed papers do not test the significance of their results, or use incorrect tests for this purpose.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    In the surveyed long ACL and TACL 2017 papers, statistical significance testing is often ignored or misused.

    “We then survey recent empirical papers published in ACL and TACL during 2017 and show that while our community assigns great value to experimental results, statistical significance testing is often ignored or misused.”
    • Also stated in: § 1 Introduction · plain
      “Our analysis reveals that there is still a room for improvement in the way statistical significance is used in papers published in our top-tier publication venues.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given

    In a considerable number of papers, the word significant was used as a synonym for words such as important, considerable, meaningful, substantial, major, and notable.

    “We noticed that in a considerable number of papers this word was used as a synonym for words such as important, considerable, meaningful, substantial, major, notable etc.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    In ACL 2017 and TACL 2017, the variety of different NLP tasks is large: 36 tasks in ACL 2017 and 15 tasks in TACL.

    “In both conference and journal papers, the variety of different NLP tasks is quite large: 36 tasks in ACL 2017 and 15 tasks in TACL.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    In ACL 2017, 23 papers performed cross validation; in TACL 2017, 5 papers performed cross validation.

    “| # papers that perform | 23 | 5 |”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given

    It is customary in NLP papers to create a number of random splits of the experimental corpus into train, development and test portions.

    “To increase the validity of reported results, it is customary in NLP papers to create a number of random splits of the experimental corpus into train, development and test portions (see Table 2).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    Some papers report the fraction of folds for which one algorithm was better than the others.

    “Some papers also report the fraction of the folds for which one algorithm was better than the others.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    In ACL 2017, 24 different evaluation measures were used; in TACL 2017, 19 different evaluation measures were used.

    “| # different evaluation | 24 | 19 |”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationreasoning

    The sign test has limited practical implications because it only checks whether algorithm A is better than B and ignores the extent of the difference.

    “The sign test has limited practical implications since it only checks if algorithm A is better than B and ignores the extent of the difference.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisereasoning

    BLEU, ROUGE and METEOR are designed for tasks where there are several possible outputs, a characteristic property of several NLP tasks.

    “While UAS and LAS are in fact accuracy measures, BLEU, ROUGE and METEOR are designed for tasks where there are several possible outputs -a characteristic property of several NLP tasks.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisenone given

    All 12 most common evaluation measures return a real number, either in [0,1] or in R.

    “All 12 measures return a real number, either in [0, 1] or in R.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given

    The Central Limit Theorem establishes that when independent random variables are added, their properly normalized sum tends toward a normal distribution even if the original variables themselves are not normally distributed.

    “CLT establishes that, in most situations, when independent random variables are added, their properly normalized sum tends toward a normal distribution even if the original variables themselves are not normally distributed.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisereasoning

    The other five most common evaluation measures (BLEU, ROUGE, METEOR, UAS, LAS) are unique measures developed for NLP applications.

    “The other five measures: BLEU (Papineni et al., 2002), ROUGE (Lin, 2004), METEOR (Banerjee and Lavie, 2005), UAS and LAS (Kübler et al., 2009), are unique measures that were developed for NLP applications.”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites Papineni et al., 2002

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Lin, 2004

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Banerjee and Lavie, 2005

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Premisecitation

    The paired Student's t-test assesses whether the population means of two sets of measurements differ from each other, and is based on the assumption that both samples come from a normal distribution.

    “This test assesses whether the population means of two sets of measurements differ from each other, and is based on the assumption that both samples come from a normal distribution (Fisher, 1937).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Fisher, 1937

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    BLEU and METEOR are standard evaluation measures for machine translation, ROUGE for extractive summarization, and UAS and LAS for dependency parsing.

    “BLEU and METEOR are standard evaluation measures for machine translation, ROUGE for extractive summarization, and UAS and LAS for dependency parsing.”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites Papineni et al., 2002

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Lin, 2004

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Banerjee and Lavie, 2005

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Premisereasoning

    Non-parametric tests may be less powerful than parametric counterparts, but they do not make unjustified assumptions and are statistically sound even when the test statistic distribution is unknown.

    “While non-parametric tests may be less powerful than their parametric counterparts, they do not make unjustified assumptions and are hence statistically sound even when the test statistic distribution is unknown.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Premisereasoning

    If the distribution of the test statistic is unknown, non-parametric tests should be used.

    “If the distribution is unknown then any assumption made by a test may lead to erroneous conclusions and hence we should rely on non-parametric tests that do not make any such assumption.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Premisereasoning

    If one algorithm is consistently better than another on test sets, it should also be better on future, currently unknown, datasets.

    “reasoning that if one algorithm is consistently better than the other, hopefully with a sufficiently large margin, then it should also be better on future, currently unknown, datasets.”
    Quote in the paperNot a substantial claim
    • The authors state this as an assumption: The paper's model of an NLP paper is that it presents a new algorithm and hypothesizes that this algorithm is better than a previous strong algorithm serving as the baseline.

  • May combine two claims.

    Premisereasoning

    Parametric tests are more statistically powerful and less computationally intensive compared to non-parametric counterparts.

    “These tests are more statistically powerful and less computationally intensive compared to their non-parametric counterparts.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Premisenone given

    The probability of falsely concluding that one algorithm is better than another should be very small.

    “we have to make sure that the probability of falsely concluding that one algorithm is better than the other is very small.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The WSJ Penn Treebank corpus consists of 2,499 articles from a three year Wall Street Journal collection of 98,732 stories.

    “the popular WSJ Penn Treebank corpus (Marcus et al., 1993) consists of 2,499 articles from a three year Wall Street Journal (WSJ) collection of 98,732 stories.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Marcus et al., 1993

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Premisereasoning

    Sampling-based tests consider actual values of evaluation measures and are not restricted to higher order properties such as ranks, hence their statistical power is higher.

    “These tests consider the actual values of the evaluation measures and are not restricted to higher order properties (e.g.ranks) of the observed values -their statistical power is hence higher.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    The impact of dependence between test examples on results is hard to quantify.

    “The impact of this phenomenon on our results is hard to quantify, partly because it is hard to quantify the nature of the dependence between test set examples in NLP datasets.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Approximate randomization can be used for test sets of more than 20 points.

    “To overcome this, approximate randomization can be used, as was done, e.g., by Yeh (2000) for test sets of more than 20 points.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Yeh (2000) source

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisereasoning

    For very large datasets, sampling-free tests are less powerful but computationally feasible.

    “The other alternative for very large datasets are sampling-free tests that are less powerful but are computationally feasible.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given

    The experimental differences between two algorithms might be coincidental.

    “Yet, the experimental differences might be coincidental.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given

    The distribution of NLP evaluation measures is of great importance to statistical significance testing.

    “The distribution of these measures is of great importance to statistical significance testing.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisenone given

    The final reported result in cross-validation is typically the average of the test set results across the splits.

    “The final reported result is typically the average of the test set results across the splits.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationnone given

    The nature of dependencies among NLP data points is hard to characterize, let alone quantify.

    “Unfortunately, the nature of these dependencies is hard to characterize, let alone to quantify.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based). May combine two claims.

    Premisenone given

    Statistical significance testing of each individual comparison is the basic building block of replicability analysis, and its accurate performance is a pre-condition for any multiple dataset analysis.

    “Statistical significance testing of each individual comparison is the basic building block of replicability analysis -its accurate performance is a pre-condition for any multiple dataset analysis.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisenone given

    Sampling-free tests trade statistical power for efficiency.

    “As noted above, sampling-free tests trade statistical power for efficiency.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    Some sentences in the WSJ Penn Treebank come from the same article, were written by the same author, or were reviewed before publication by the same editor.

    “Obviously, some of the sentences included in the corpus come from the same article, were written by the same author or were reviewed before publication by the same editor.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given

    The exact permutation test requires generation of all 2^n data permutations, where n is the number of points in the dataset.

    “the exact permutation test, for example, requires the generation of all 2 n data permutations (where n is the number of points in the dataset).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Premisenone given

    The Student's t-test assumes the data is i.i.d. and the test statistics are normally distributed.

    “the Student's t-test that assumes the data is i.i.d and that the test statistics are normally distributed.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    The higher computational demand of some non-parametric tests is the result of their sampling process.

    “while the higher computational demand of some non-parametric tests is the result of their sampling process.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationreasoning

    The stronger statistical power of parametric tests stems from the stronger parametric assumptions they make.

    “The stronger statistical power of parametric tests stems from the stronger, parametric assumptions they make”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Premisenone given

    Sampling-based methods are computationally intensive and can be intractable for large datasets, even with modern computing power.

    “Clearly, Sampling-based methods are computationally intensive and can be intractable for large datasets, even with modern computing power.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    The test sets of different folds in cross-validation are not independent and their content is likely to overlap.

    “Particularly, the test sets of the different folds are obviously not independent -their content is even likely to overlap.”
    Quote in the paperNot a substantial claim
39 of 39

  • The authors state this as an assumption.

    The paper's model of an NLP paper is that it presents a new algorithm and hypothesizes that this algorithm is better than a previous strong algorithm serving as the baseline.

    “our (somewhat simplified) model of an NLP paper is one that presents a new algorithm and makes the hypothesis that this algorithm is better than a previous strong algorithm, which serves as the baseline.”

  • The authors state this as an assumption.

    Higher values of the evaluation measure are better.

    “Without loss of generality, we assume that higher values of the measure are better.”

  • The paper relies on this without support.

    In practice, the t-test is often applied with evaluation measures such as accuracy, UAS and LAS, which compute the mean number of correct predictions per input example.

    “In practice, t-test is often applied with evaluation measures such as accuracy, UAS and LAS, that compute the mean number of correct predictions per input example.”

  • The paper relies on this without support.

    The sample of ACL 2017 and TACL 2017 papers is representative of recent top venue NLP papers.

    “We analyzed the long papers from the proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL17, (Barzilay and Kan, 2017)), a total of 196 papers, and the papers from the Transactions of the Association of Computational Linguistics journal (TACL17), Volume 5, Issue 1, a total of 37 papers.”

  • The authors state this as an assumption.

    The TACL review process is more strict and of higher quality than the ACL review process.

    “In TACL, where the review process is presumably more strict and of higher quality”
5 of 5

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The standard statistical test for comparing classifiers on paired data is McNemar's test.

    “The standard statistical test for comparing classifiers on paired data is McNemar's test (Dietterich, 1998; Dror et al., 2018), which uses the numbers of items where the models disagree”

    From With Little Power Comes Great Responsibility (2020) · cites this paper as Dror et al., 2018

    Section 3.2.2 says McNemar’s test is designed for paired nominal observations and is appropriate for binary classification. The claim omits that restriction: the same passage identifies Cochran’s Q as the generalization for multiclass classification. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • Pointwise 95% percentile bootstrap intervals are used for each paired mean difference, resampling 944 questions on ASQA and 271 papers on QASPER with replacement.

    “We use pointwise 95% percentile bootstrap intervals for each paired mean difference (Dror et al., 2018), resampling 944 questions on ASQA and 271 papers on QASPER with replacement.”

    From The Attribution-Compression Frontier in Retrieval-Augmented Generation (2026) · cites this paper as Dror et al., 2018

    Section 3.2.2 (page unavailable) describes paired bootstrap sampling with replacement to approximate a statistic’s sampling distribution. It discusses p-values, not pointwise 95% percentile intervals for paired mean differences. The ASQA and QASPER sample sizes describe the citing authors’ procedure, rather than a finding attributed to Dror et al. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

  • The study evaluated statistical robustness by generating 10,000 bootstrap-resampled versions of the test set and computing standard deviations of all performance metrics.

    “We evaluate the statistical robustness of our results by generating 10,000 bootstrap-resampled versions of the test set (Dror et al., 2018) and computing the standard deviation of all performance metrics.”

    From Fact or Fiction: Verifying Scientific Claims (2020) · cites this paper as Dror et al., 2018

  • For Judge-EM, McNemar's test was used, appropriate for paired nominal data [8].

    “For Judge-EM, we used McNemar's test , which is appropriate for paired nominal data [8].”

    From Replace, Don't Expand: Mitigating Context Dilution in Multi-Hop RAG via Fixed-Budget Evidence Assembly (2025) · cites this paper as [8]

Cites

Flagged first, then by citations

Research works printed but not linked: 2

  1. Alexander Rush, Roi Reichart, Michael Collins, and Amir Globerson. 2012. Improved parsing and pos tagging using inter-sentence consistency constraints. In Proceedings of EMNLP-CoNLL.
  2. Anders Søgaard. 2013. Estimating effect size across datasets. In Proceedings of NAACL-HLT.

Cited by

Sorted by citations

Related

Authors
Dror, R., Baumer, G., Shlomov, S., & Reichart, R.
Affiliations
Technion – Israel Institute of Technology (IL)
ORCID
Venue
Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Publisher
Association for Computational Linguistics
Published
2018
Pages
1383-1392
DOI
10.18653/v1/p18-1128
OpenAlex
W2798935874
Semantic Scholar
d10df96b3fb0ab5c6b1d0cc22c7400d0acccc3cc
Cited by
384
References
43
Citations by year
2026: 59 · 2025: 27 · 2024: 23 · 2023: 56 · 2022: 50 · 2021: 78 · 2020: 55 · 2019: 33 · 2018: 3
FWCI
17.19
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
Annual Meeting of the Association for Computational Linguistics (ACL) (publishedVersion, cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Social Sciences › General Social Sciences › Computational and Text Analysis Methods
Keywords
statistical significance testing, hypothesis testing, evaluation metrics, experimental design, misuse of statistics, reproducibility

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-22.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-22.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review