NUTMEG: Separating Signal From Noise in Annotator Disagreement
Open accessAbstract
NLP models often rely on human-labeled data for training and evaluation.Many approaches crowdsource this data from a large number of annotators with varying skills, backgrounds, and motivations, resulting in conflicting annotations.These conflicts have traditionally been resolved by aggregation methods that assume disagreements are errors.Recent work has argued that for many tasks annotators may have genuine disagreements and that variation should be treated as signal rather than noise.However, few models separate signal and noise in annotator disagreement.In this work, we introduce NUTMEG, a new Bayesian model that incorporates information about annotator backgrounds to remove noisy annotations from human-labeled training data while preserving systematic disagreements.Using synthetic and real-world data, we show that NUTMEG is more effective at recovering ground-truth from annotations with systematic disagreement than traditional aggregation methods, and we demonstrate that downstream models trained on NUT-MEG-aggregated data significantly outperform models trained on data from traditionally aggregation methods.We provide further analysis characterizing how differences in subpopulation sizes, rates of disagreement, and rates of spam affect the performance of our model.Our results highlight the importance of accounting for both annotator competence and systematic disagreements when training on human-labeled data.
AI snapshot
Written from the paper’s full text.
TL;DR
NUTMEG is a Bayesian annotation model that estimates annotator competence while inferring separate ground-truth labels per annotator subpopulation, aiming to remove spam while preserving systematic disagreement. Across synthetic and POPQUORN experiments, it recovers minority subpopulation labels better than traditional aggregation and improves downstream politeness modeling, though not offensiveness.
Key findings
- In synthetic data, NUTMEG correctly identifies true labels for both majority and minority subpopulations as systematic disagreement increases, while other methods remain accurate for the majority but become increasingly inaccurate for the minority. NUTMEG’s average Pearson correlation between estimated and true annotator competence is 0.81, versus 0.58 for MACE.
- As spam rates increase from 0 to 25%, NUTMEG shows an average 4.22% drop in minority accuracy; it can differentiate genuine disagreement from spam, though its disagreement-rate estimates diverge at high and low extremes.
- With a subpopulation proportion of 0.3 (45 annotators), NUTMEG requires 5 annotations to achieve 92% accuracy; for a proportion of 0.1 (15 annotators), it would need more than 15 annotations per item to achieve the same performance.
- On POPQUORN politeness, models trained on NUTMEG-aggregated labels outperform both traditional aggregation and disaggregated-annotation baselines (p < 0.05) for all subpopulation splits except race; on offensiveness, there are no significant differences, but NUTMEG does as well as or better than no aggregation.
Methods
- NUTMEG extends item-response models: for each item and subpopulation it samples a true subpopulation label; each annotator either uses that label or spams according to a Bernoulli competence parameter, with spam labels drawn from an annotator-specific multinomial. It is fit with Variational-Bayes training using symmetric Beta(0.5) priors on competence and symmetric Dirichlet priors on spam behavior.
- Synthetic experiments use 150 annotators, 500 items, two labels, and an 80/20 majority/minority split, varying divisiveness from 0 to 1 and global spam from 0 to 0.25; comparisons include majority vote, Dawid & Skene, MACE, LFC, and BCC. A second synthetic experiment fixes spam at 0.1 and divisiveness at 0.2, varying minority subpopulation size from 10% to 50% and annotations per item from 3 to 15.
- Real-data experiments use POPQUORN offensiveness and politeness annotations, binarizing 1–5 Likert ratings at ≥3 and removing subpopulations with ≤5% of annotations. Multi-task ModernBERT models with per-subpopulation heads are trained on NUTMEG outputs and evaluated with Jensen-Shannon divergence against test-set subpopulation label distributions.
Limitations
- The paper states NUTMEG is not a universal fix for noisy annotations and does not always improve downstream performance, as in the offensiveness task; other sources of label variation may dominate, most NLP still uses single-label models, and NUTMEG treats truth variables as nominal.
- The paper states experiments used known, discrete subpopulation labels and did not evaluate inferred clustering for subpopulations, and that real-world evaluation is limited to the POPQUORN dataset. It also notes that imputing unobserved subpopulation labels assumes observed labels from different subpopulations are independent.
- Evident from the text: synthetic results rely on simplified assumptions such as two subpopulations, binary labels, and a known spam process, and the real-data evaluation covers only two tasks from one dataset with demographic splits, so generalization to other tasks, label types, and subpopulation definitions is not established.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
2 of 24 cited sources checked · 0 supported · 0 flagged · accuracy not yet measured
- Findingown data
NUTMEG is effective at assessing annotator competence.
“We further find that NUTMEG is effective at as-sessing annotator competence.”
Quote in the paperSubstantial claim - Findingown data
NUTMEG can effectively differentiate genuine disagreement from spam annotations.
“We find that NUTMEG can effectively differentiate genuine disagreement from spam annotations, as shown in Figure 5.”
Quote in the paperSubstantial claim - Findingown data
NUTMEG does not always improve performance.
“Though we consider these compelling use cases for NUTMEG, we also find that it does not always improve performance.”
Quote in the paperSubstantial claim- Rests on an assumption: Annotator demographics (gender, age, race, and education) capture relevant subpopulation-level variation in judgments for the tasks studied.
- Findingown data
Among the aggregation steps, NUTMEG does as well as or better than no-aggregation for offensiveness detection.
“However, among the aggregation steps NUTMEG does as well as or better than no-aggregation.”
Quote in the paperSubstantial claim- Rests on an assumption: Annotator demographics (gender, age, race, and education) capture relevant subpopulation-level variation in judgments for the tasks studied.
- Findingown data
Through real-world experiments, NUTMEG's ability to distinguish systematic disagreement can be combined with Learning from Disagreement models to improve performance on subjective classification tasks.
“Then, through real-world experiments, we demonstrate that NUTMEG's ability to distinguish systematic disagreement can be combined with Learning from Disagreement models to improve performance on subjective classification tasks.”
Quote in the paperSubstantial claim- Rests on an assumption: Annotator demographics (gender, age, race, and education) capture relevant subpopulation-level variation in judgments for the tasks studied.
- Findingown data
NUTMEG's overall accuracy is still high regardless of how often the subpopulation disagrees.
“While a 25% spam rate is likely on the high side, NUTMEG's overall accuracy is still high regardless of how often the subpopulation disagrees.”
Quote in the paperSubstantial claim - Findingown data
As spam rates increase from 0 to 25%, there is an average 4.22% drop in minority accuracy.
“Further, as spam rates increase from 0 to 25%, we see an average 4.22% drop in minority accuracy.”
Quote in the paperSubstantial claim - Findingown data
As the size of a subpopulation in a dataset gets smaller, NUTMEG requires a much larger number of annotations per item to maintain the same level of accuracy.
“We find that as the size of a subpopulation in a dataset gets smaller, NUTMEG requires a much larger number of annotations per item to maintain the same level of accuracy (Figure 6).”
Quote in the paperSubstantial claim - Findingown data
NUTMEG mitigates spam annotations.
“Our experiments on synthetic and real-world data demonstrate that NUTMEG effectively recovers distinct subpopulation labels, mitigates spam annotations, and improves the performance of Learning from Disagreement models.”
Quote in the paperSubstantial claim- Rests on an assumption: Annotator demographics (gender, age, race, and education) capture relevant subpopulation-level variation in judgments for the tasks studied.
- Findingown data
NUTMEG effectively recovers distinct subpopulation labels.
“Our experiments on synthetic and real-world data demonstrate that NUTMEG effectively recovers distinct subpopulation labels, mitigates spam annotations, and improves the performance of Learning from Disagreement models.”
Quote in the paperSubstantial claim- Rests on an assumption: Annotator demographics (gender, age, race, and education) capture relevant subpopulation-level variation in judgments for the tasks studied.
- Findingown data
In synthetic experiments, NUTMEG is highly effective at distinguishing genuine systematic disagreement from noise in annotations.
“In synthetic experiments, we show that NUTMEG is highly effective at distinguishing genuine systematic disagreement from noise in annotations.”
Quote in the paperSubstantial claim- Rests on an assumption: Annotator demographics (gender, age, race, and education) capture relevant subpopulation-level variation in judgments for the tasks studied.
- Findingnone given
In the offensiveness task, NUTMEG does not hurt performance compared to other methods.
“Though, we note that in this task, NUTMEG does not hurt performance compared to other methods.”
Quote in the paperNot a substantial claim- Rests on an assumption: Annotator demographics (gender, age, race, and education) capture relevant subpopulation-level variation in judgments for the tasks studied.
- Findingown data
The majority of examples in the offensiveness data exhibit high subjectivity.
“After a manual review of the data, we found that the majority of examples exhibit high subjectivity.”
Quote in the paperNot a substantial claim - Findingnone given
Other methods are accurate for recovering the majority's true label but increasingly inaccurate for the minority subpopulation's true label as the rate of divisiveness increases.
“In contrast, while the other methods are accurate for recovering the majority's true label, they are increasingly inaccurate for the minority subpopulation's true label as the rate of divisiveness increases (as expected).”
Quote in the paperSubstantial claim - Findingown data
As the rate of systematic disagreement increases, NUTMEG correctly identifies the true label for both the majority and minority subpopulations, despite the minority having four times less available data.
“We find that as the rate of systematic disagreement increases, NUTMEG correctly identifies the true label for both the majority and minority subpopulations (Figure 4), despite the minority having four times less available data.”
Quote in the paperSubstantial claim - Findingown data
Downstream models trained on NUTMEG-aggregated data significantly outperform models trained on data from traditional aggregation methods.
“and we demonstrate that downstream models trained on NUT-MEG-aggregated data significantly outperform models trained on data from traditionally aggregation methods.”
Quote in the paperSubstantial claim - Findingown data
For politeness detection, models trained on NUTMEG outperform both models trained on traditionally aggregated annotations and models trained on disaggregated annotations for all subpopulation splits except race (p < 0.05).
“We find that for politeness detection, models trained on NUTMEG outperform (p < 0.05) both models trained on traditionally aggregated annotations and models trained on disaggregated annotations for all subpopulation splits except race (Figure 7, left).”
Quote in the paperSubstantial claim- Rests on an assumption: Annotator demographics (gender, age, race, and education) capture relevant subpopulation-level variation in judgments for the tasks studied.
- Findingown data
Any type of aggregation is helpful to reduce noise.
“Our results show that any type of aggregation is helpful to reduce noise, but by jointly modeling subpopulation-preference with annotator competence, NUTMEG is better able to match the true label distributions.”
Quote in the paperNot a substantial claim- Rests on an assumption: Annotator demographics (gender, age, race, and education) capture relevant subpopulation-level variation in judgments for the tasks studied.
- Findingown data
For offensiveness detection, there are no significant differences between models trained with different aggregation methods or no aggregation.
“We find that for offensiveness detection (Figure 7, right), there are no significant differences between models trained with different aggregation methods or no aggregation.”
Quote in the paperNot a substantial claim- Rests on an assumption: Annotator demographics (gender, age, race, and education) capture relevant subpopulation-level variation in judgments for the tasks studied.
- Findingreasoning
In the offensiveness detection task, there are difficult cases that cannot be explained by the model.
“For example, in the offensiveness detection task, we often encounter difficult cases that cannot be explained by the model.”
Quote in the paperNot a substantial claim - Findingown data
The average Pearson's correlation between NUTMEG's estimate of annotator competence θ_j and the annotator's true competence across all runs is 0.81.
“The average Pearson's correlation between NUTMEG's estimate of annotator competence θ j and the annotator's true competence across all runs is 0.81.”
Quote in the paperSubstantial claim - Findingown data
NUTMEG improves the performance of Learning from Disagreement models.
“Our experiments on synthetic and real-world data demonstrate that NUTMEG effectively recovers distinct subpopulation labels, mitigates spam annotations, and improves the performance of Learning from Disagreement models.”
Quote in the paperSubstantial claim- Rests on an assumption: Annotator demographics (gender, age, race, and education) capture relevant subpopulation-level variation in judgments for the tasks studied.
- Findingown data
MACE's average correlation between estimated and true annotator competence is 0.58.
“By comparison, MACE's average correlation is 0.58.”
Quote in the paperSubstantial claim - Findingreasoning
NUTMEG doesn't always improve downstream model performance in the Offensiveness task.
“Indeed, we note that NUTMEG doesn't always improve downstream model performance (in the Offensiveness task) suggesting that even when demographic labels are present and modeled, other sources of label variation in the data may more strongly influence performance; indeed, past work in NLP has found a mixed trend, where for some datasets, demographics explain little variation (Orlikowski et al., 202”
Quote in the paperNot a substantial claim- Rests on an assumption: Annotator demographics (gender, age, race, and education) capture relevant subpopulation-level variation in judgments for the tasks studied.
- Findingown data
These correlations do not significantly differ for members of the majority or minority subpopulations.
“These correlations do not significantly differ for members of the majority or minority subpopulations”
Quote in the paperSubstantial claim - Findingown data
NUTMEG makes these improvements without reducing its ability to estimate the majority opinion.
“Importantly, NUTMEG makes these improvements without reducing its ability to estimate the majority opinion.”
Quote in the paperSubstantial claim - Findingown data
NUTMEG is more effective at recovering ground-truth from annotations with systematic disagreement than traditional aggregation methods.
“Using synthetic and real-world data, we show that NUTMEG is more effective at recovering ground-truth from annotations with systematic disagreement than traditional aggregation methods”
Quote in the paperSubstantial claim - Findingown data
With a subpopulation proportion of 0.3 (45 annotators), NUTMEG only requires 5 annotations to achieve 92% accuracy.
“With a subpopulation proportion of 0.3 (equivalent to 45 annotators), NUTMEG only requires 5 annotations to achieve 92% accuracy”
Quote in the paperSubstantial claim - Findingown data
In experiments with real data labeled with demographics, by first reducing noise with NUTMEG, subpopulation labels can be used with learning from disagreement models to make more accurate predictions.
“Finally, in experiments with real data labeled with demographics, we show that by first reducing noise with NUTMEG, we can use subpopulation labels with learning from disagreement models to make more accurate predictions.”
Quote in the paperSubstantial claim - Findingown data
For a subpopulation proportion of 0.1 (15 annotators), NUTMEG would need more than 15 annotations per item to achieve the same performance as a subpopulation proportion of 0.3 with 5 annotations.
“but for a subpopulation proportion of 0.1 (equivalent to 15 annotators), NUT-MEG would need more than 15 annotations per item to achieve the same performance.”
Quote in the paperSubstantial claim - Interpretationnone given
NUTMEG is a Bayesian model that incorporates information about annotator backgrounds to remove noisy annotations from human-labeled training data while preserving systematic disagreements.
“In this work, we introduce NUTMEG, a new Bayesian model that incorporates information about annotator backgrounds to remove noisy annotations from human-labeled training data while preserving systematic disagreements.”
Quote in the paperSubstantial claim - Interpretationown data
Learning from more-accurate aggregations that both reduce noise and highlight systematic disagreement enables models to better predict the full label distribution for each item in the test set.
“By learning from more-accurate aggregations that both reduce noise and highlight systematic disagreement, models are better able to predict the full label distribution for each item in the test set.”
Quote in the paperSubstantial claim- Rests on an assumption: Annotator demographics (gender, age, race, and education) capture relevant subpopulation-level variation in judgments for the tasks studied.
- Premisecitation
Previous work has shown that in many subjective NLP tasks, like detecting offensiveness, politeness, and toxicity, annotator disagreements are correlated with backgrounds and social variables such as gender, race, age, and region.
“Previous work has shown that in many subjective NLP tasks, like detecting offensiveness, politeness, and toxicity, annotator disagreements are correlated with backgrounds and social variables such as gender, race, age, and region (Larimore et al., 2021;Sap et al., 2022;Pei and Jurgens, 2023;Wan et al., 2023;Mostafazadeh Davani et al., 2024).”
Quote in the paperSubstantial claim0 of 3 sources checked- cites Larimore et al., 2021
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites Sap et al., 2022
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites Pei and Jurgens, 2023
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Interpretationreasoning
NUTMEG can use annotator data to better separate signal from noise in annotator disagreement.
“NUTMEG can use this data to better separate signal from noise in annotator disagreement.”
Quote in the paperSubstantial claim - Premisecitation
For many tasks there can be genuine disagreements between annotators.
“however, for many tasks there can be genuine disagreements between annotators (Plank et al., 2014).”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Plank et al., 2014 source
- Interpretationown data
Accounting for relevant subpopulation-level variation when predicting annotator competence improves performance.
“It also demonstrates that accounting for relevant subpopulation-level variation when predicting annotator competence improves performance.”
Quote in the paperSubstantial claim- Rests on an assumption: Annotator demographics (gender, age, race, and education) capture relevant subpopulation-level variation in judgments for the tasks studied.
- Interpretationreasoning
NUTMEG is not a universal panacea for noisy annotations.
“Thus, while the method can potentially improve quality, it is not a universal panacea for noisy annotations.”
Quote in the paperNot a substantial claim- Rests on an assumption: Most real data contains label variation beyond that due to subpopulations.
- Interpretationown data
For the politeness task, there is likely meaningful variation that can be modeled.
“Yet, the gap between MACE and NUTMEG highlights that there is additional benefit to modeling subpopulation-variation-and that for this politeness task, there is likely meaningful variation that can be modeled.”
Quote in the paperSubstantial claim- Rests on an assumption: Annotator demographics (gender, age, race, and education) capture relevant subpopulation-level variation in judgments for the tasks studied.
- Interpretationreasoning
Annotator groups who systematically disagree with the majority label are likely lost in aggregation.
“As a result, annotator groups who systematically disagree with the majority label are likely lost in aggregation.”
Quote in the paperSubstantial claim - Interpretationown data
Even when demographic labels are present and modeled, other sources of label variation in the data may more strongly influence performance.
“Indeed, we note that NUTMEG doesn't always improve downstream model performance (in the Offensiveness task) suggesting that even when demographic labels are present and modeled, other sources of label variation in the data may more strongly influence performance; indeed, past work in NLP has found a mixed trend, where for some datasets, demographics explain little variation (Orlikowski et al., 202”
Quote in the paperNot a substantial claim- Rests on an assumption: Annotator demographics (gender, age, race, and education) capture relevant subpopulation-level variation in judgments for the tasks studied.
- Interpretationreasoning
The divergence at extreme disagreement rates is caused by the increased effect that contrary evidence has at extreme rates of disagreement.
“We believe that this divergence is caused by the increased effect that contrary evidence has at extreme rates of disagreement.”
Quote in the paperSubstantial claim - Interpretationreasoning
NUTMEG will be most effective when applied to tasks where there is a prior belief that subpopulations differ in their judgments or attitudes.
“From these findings, we conclude that NUTMEG will be most effective when applied to tasks where there is a prior belief that subpopulations differ in their judgments or attitudes.”
Quote in the paperSubstantial claim- Rests on an assumption: Annotator demographics (gender, age, race, and education) capture relevant subpopulation-level variation in judgments for the tasks studied.
- Interpretationown data
By jointly modeling subpopulation-preference with annotator competence, NUTMEG is better able to match the true label distributions.
“Our results show that any type of aggregation is helpful to reduce noise, but by jointly modeling subpopulation-preference with annotator competence, NUTMEG is better able to match the true label distributions.”
Quote in the paperSubstantial claim- Rests on an assumption: Annotator demographics (gender, age, race, and education) capture relevant subpopulation-level variation in judgments for the tasks studied.
- Rests on an assumption: The full, disaggregated label distribution of a subpopulation in the test set represents the true label distribution for that subpopulation.
- Interpretationnone given
NUTMEG produces estimated true labels for each subpopulation.
“NUTMEG produces estimated true labels for each subpopulation and therefore, for predictive modeling, we adopt the learning from disagreement setting where a classifier is trained on multiple labels per annotation.”
Quote in the paperSubstantial claim - Interpretationreasoning
There is no systematic variation by subpopulation in the offensiveness data.
“We interpret this trend as an example where there is no systematic variation by subpopulation in the data.”
Quote in the paperNot a substantial claim- Rests on an assumption: Annotator demographics (gender, age, race, and education) capture relevant subpopulation-level variation in judgments for the tasks studied.
- Premisenone given
These methods rely on the assumption that disagreements between annotators indicate errors or a lack of competence.
“These methods rely on the assumption that disagreements between annotators indicate errors or a lack of competence; however, for many tasks there can be genuine disagreements between annotators (Plank et al., 2014).”
Quote in the paperSubstantial claim - Interpretationown data
Accounting for both annotator competence and systematic disagreements is important when training on human-labeled data.
“Our results highlight the importance of accounting for both annotator competence and systematic disagreements when training on human-labeled data.”
Quote in the paperSubstantial claim - Study design and conductnone given
The set of annotators in the training and validation sets is entirely separate from the set of annotators in the test set.
“Note that the set of annotators in the training and validation sets is entirely separate from the set of annotators in the test set, and we score using the label distribution of the full, disaggregated labels in the test set.”
Quote in the paperSubstantial claim - Interpretationown data
There is additional benefit to modeling subpopulation variation for the politeness task.
“Yet, the gap between MACE and NUTMEG highlights that there is additional benefit to modeling subpopulation-variation-and that for this politeness task, there is likely meaningful variation that can be modeled.”
Quote in the paperSubstantial claim- Rests on an assumption: Annotator demographics (gender, age, race, and education) capture relevant subpopulation-level variation in judgments for the tasks studied.
- Premisecitation
Recent work has established that annotators from specific backgrounds may systematically differ in their judgments on an item, particularly for subjective tasks.
“However, recent work has established that annotators from specific backgrounds may systematically differ in their judgments on an item, particularly for subjective tasks (Larimore et al., 2021;Sap et al., 2022;Pei and Jurgens, 2023;Wan et al., 2023;Mostafazadeh Davani et al., 2024).”
- Also stated in: § Modeling Annotator Backgrounds · boosted
“An individual's background (e.g., demographics, occupation) is known to systematically influence their annotation behavior, leading to disagreements in labeling (Lerner et al., 2024;Pei and Jurgens, 2023).”
Quote in the paperSubstantial claim0 of 3 sources checked- cites Larimore et al., 2021
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites Sap et al., 2022
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites Pei and Jurgens, 2023
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Also stated in: § Modeling Annotator Backgrounds · boosted
- Interpretationreasoning
By preserving meaningful disagreement, NUTMEG provides a more nuanced understanding of annotation data, particularly in subjective NLP tasks.
“By preserving meaningful disagreement, NUTMEG provides a more nuanced understanding of annotation data, particularly in subjective NLP tasks.”
Quote in the paperSubstantial claim- Rests on an assumption: Annotator demographics (gender, age, race, and education) capture relevant subpopulation-level variation in judgments for the tasks studied.
- Interpretationreasoning
NUTMEG's ability to model annotator variation contributes to more representative NLP models.
“Its data efficiency makes it well-suited for crowdsourcing settings, and its ability to model annotator variation contributes to more representative NLP models.”
Quote in the paperSubstantial claim- Rests on an assumption: Annotator demographics (gender, age, race, and education) capture relevant subpopulation-level variation in judgments for the tasks studied.
- Premisecitation
NUTMEG does not necessarily require additional data collection because annotators can be clustered into subpopulations based on annotator behaviors.
“We note that though we choose to evaluate on data using annotator demographics to split subpopulations, NUTMEG does not necessarily require additional data collection as recent work has demonstrated the efficacy of clustering annotators into subpopulations based on annotator behaviors (Vitsakis et al., 2024).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Vitsakis et al., 2024
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Rests on an assumption: Annotator demographics (gender, age, race, and education) capture relevant subpopulation-level variation in judgments for the tasks studied.
- Interpretationown data
Using traditional item-response models can lead a practitioner to erroneously conclude that they have low-quality annotators when in fact they have high-quality annotators and systematic disagreement.
“which illustrates how using traditional item-response models can lead a practitioner to erroneously conclude that they have low-quality annotators when in fact they have high-quality annotators and systematic disagreement.”
Quote in the paperSubstantial claim - Interpretationown data
NUTMEG can be accurately deployed even when divisive items are relatively rare.
“As experimenters may not know how often a particular group might disagree, this performance trend suggests NUTMEG can be accurately deployed even when divisive items are relatively rare.”
Quote in the paperSubstantial claim
- Classified as probably not a substantial claim. 18
- May combine two claims. 12
Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
PremisecitationPast work in NLP has found a mixed trend: for some datasets, demographics explain little variation, while for others, demographics explain substantial variation.
“indeed, past work in NLP has found a mixed trend, where for some datasets, demographics explain little variation (Orlikowski et al., 2023), while for others, demographics explain substantial variation (Larimore et al., 2021;Sap et al., 2022;Pei and Jurgens, 2023;Wan et al., 2023).”
Quote in the paperNot a substantial claim1 of 3 sources checked- cites Orlikowski et al., 2023 source
The Results (§5) find no statistically significant predictive gain from adding sociodemographic information in the dataset studied, supporting the cited clause’s general direction. But the paper does not measure how much label variation demographics explain; its finding is narrower and specific to adding group-specific layers to multi-annotator models. (LLM, conf=0.82)
Experimental — accuracy not yet measured - cites Larimore et al., 2021
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites Sap et al., 2022
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim: may not pass S2 (checkable).
InterpretationcitationSubpopulation modeling is a promising direction for improving the representation and evaluation for the diverse stakeholders of NLP models.
“However, our results show subpopulation modeling is a promising direction for improving the representation and evaluation for the diverse stakeholders of NLP models (Cabitza et al., 2023).”
Quote in the paperNot a substantial claim1 of 1 sources checked- cites Cabitza et al., 2023 source
The Motivations and Background section advocates modeling diverse human perspectives, and Related work describes clustering raters by affinity in an NLP hate-speech corpus. The paper also argues for evaluation that accounts for differing annotations. It supports the broader direction, but does not specifically identify subpopulation modeling as a way to improve both representation and evaluation for NLP stakeholders. The citing authors’ phrase “our results show” describes their own findings, not this source’s. (LLM, conf=0.78)
Experimental — accuracy not yet measured
- Rests on an assumption: Annotator demographics (gender, age, race, and education) capture relevant subpopulation-level variation in judgments for the tasks studied.
May combine two claims.
FindingreasoningAs the rate of disagreement approaches the high and low extremes, NUTMEG's estimate of the rate of disagreement diverges from the true rate; at high rates it underestimates and at low rates it overestimates.
“However, as the rate of disagreement approaches the high and low extremes, NUTMEG's estimate of the rate of disagreement diverges from the true rate.At especially high rates of disagreement, the model underestimates and at especially low rates of disagreement, the model overestimates.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataNUTMEG's accuracy for the minority subpopulation is lower than for the majority, primarily due to data sparsity.
“Note that while NUTMEG is able to recover most of the true labels, the accuracy for the minority subpopulation is still lower.This gap is primarily due to data sparsity.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataNUTMEG is effective even for small numbers of annotations per subpopulation, making it readily amenable to use with crowdsourcing.
“(ii) NUTMEG is effective even for small numbers of annotations per subpopulation, making it readily amenable to use with crowdsourcing.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataAggregation generally helps, with even majority voting often reducing noise in the data.
“Our results show aggregation generally helps, with even the naive but commonly-used majority voting often reducing noise in the data.”
Quote in the paperSubstantial claim- Rests on an assumption: Annotator demographics (gender, age, race, and education) capture relevant subpopulation-level variation in judgments for the tasks studied.
May combine two claims.
Findingown dataIn experiments on synthetic data, NUTMEG can accurately recover distinct labels for each subpopulation when they differ, while still recognizing when annotators are spamming.
“In experiments on synthetic data, we demonstrate that (i) NUTMEG can accurately recover distinct labels for each subpopulation when they differ, while still recognizing when annotators are spammingfoot_1”
Quote in the paperSubstantial claimMay combine two claims.
InterpretationcitationNUTMEG does not require that subpopulations be derived from additional annotator data; it can be combined with any unsupervised method for grouping annotators based on behavior.
“However, we note that NUTMEG does not require that subpopulations be derived from additional annotator data, like those collected from questionnaires.Instead, it can be combined with any unsupervised method for grouping annotators based on behavior (e.g., Vitsakis et al., 2024), and the subpopulations can be derived from the resultant inferred groups.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Vitsakis et al., 2024
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductreasoningWe have not evaluated the strategy of clustering annotators to create inferred subpopulations.
“While we note that recent work has pointed to the ability to cluster annotators to create inferred subpopulations (Vitsakis et al., 2024), we have not evaluated that strategy here in favor of first demonstrating that the method works with known demographics.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Vitsakis et al., 2024
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
May combine two claims.
InterpretationreasoningNUTMEG is at least not introducing additional noise, unlike majority vote which generally has higher JSD.
“This neutral behavior highlights it is at least not introducing additional noise, unlike majority vote which generally has higher JSD.”
Quote in the paperSubstantial claim- Rests on an assumption: Annotator demographics (gender, age, race, and education) capture relevant subpopulation-level variation in judgments for the tasks studied.
May combine two claims.
Interpretationown dataLearning directly from disaggregated annotations can introduce noise, and NUTMEG's noise reduction improves performance on downstream tasks.
“This trend supports our hypothesis that learning directly from disaggregated annotations can introduce noise and NUTMEG's noise reduction improves performance on downstream tasks.”
Quote in the paperSubstantial claim- Rests on an assumption: Annotator demographics (gender, age, race, and education) capture relevant subpopulation-level variation in judgments for the tasks studied.
May combine two claims.
Interpretationnone givenNUTMEG estimates annotator competence and infers per-subpopulation labels for each item.
“Our approach, NUTMEG (Nuanced Understanding of annoTation by MultiplE Groups) estimates annotator competence and infers persubpopulation labels for each item.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S2 (checkable).
Interpretationown dataIntentional data collection methods are important when representing the opinions of small subpopulations.
“This result shows the importance of intentional data collection methods when representing the opinions of small subpopulations.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationNUTMEG most closely resembles MACE, which attempts to simultaneously learn the spamming rates of annotators and the item's likely label.
“NUTMEG most closely resembles MACE (Hovy et al., 2013) which attempts to simultaneously learn the spamming rates of annotators and the item's likely label.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Hovy et al., 2013
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationIn real scenarios, annotators may disagree for a variety of reasons, not all of which relate to a subpopulation identity.
“In real scenarios, annotators may disagree for a variety of reasons (Sandri et al., 2023), not all of which relate to a subpopulation identity.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Sandri et al., 2023 source
Checked before — its earlier verdict is held back until a new check runs.
Probably not a substantial claim: may not pass S4 (based).
Premisenone givenNUTMEG requires that annotators be associated with a particular subpopulation.
“NUTMEG requires that annotators be associated with a particular subpopulation.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductreasoningOur experimental design limits the potential confounding influence of the clustering model on the potential benefits of NUTMEG.
“Our experimental design limits the potential confounding influence of the clustering model on the potential benefits of NUTMEG.”
Quote in the paperNot a substantial claimMay combine two claims.
InterpretationreasoningCurrent models of annotator competence treat consensus as correct and disagreement as error, and as a result, valid label disagreements by minority subpopulations risk being omitted.
“This relationship is particularly important because current models of annotator competence treat consensus as correct-and disagreement as error-and as a result, valid label disagreements by minority subpopulations risk being omitted.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductnone givenOur experiments used a limited number of real-world datasets to demonstrate effectiveness.
“Finally, our experiments used a limited number of real-world datasets to demonstrate effectiveness.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
PremisecitationPrior work has identified differences in annotator labels as label noise and attempted to reduce it prior to model training.
“Prior work has identified differences in annotator labels as labelnoise and attempted to reduce it prior to model training (Dawid and Skene, 1979).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Dawid and Skene, 1979 source
Probably not a substantial claim: may not pass S4 (based).
Premisenone givenOne key assumption of prior Bayesian models is that there exists a single correct label for each item agreed upon by annotators.
“One key assumption of prior Bayesian models is that there exists a single correct label for each item agreed upon by annotators.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationMany methods use unsupervised probabilistic models of annotator behavior to identify incorrect labels (spam) and estimate annotator competence.
“Many of these methods use unsupervised probabilistic models of annotator behavior to identify incorrect labels, also known as spam, and estimate annotator competence (Whitehill et al., 2009;Liu et al., 2012;Hovy et al., 2013;Paun et al., 2018).”
Quote in the paperNot a substantial claim0 of 3 sources checked- cites Whitehill et al., 2009
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites Liu et al., 2012
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites Hovy et al., 2013
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationMultiple works have noted meaningful variation by race and gender.
“Multiple works have noted meaningful variation by race and gender (Larimore et al., 2021;Sap et al., 2022;Pei and Jurgens, 2023;Wan et al., 2023).”
Quote in the paperNot a substantial claim0 of 3 sources checked- cites Larimore et al., 2021
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites Sap et al., 2022
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites Pei and Jurgens, 2023
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationConservative annotators are less likely to rate anti-Black language as toxic.
“for example, showing that conservative annotators are less likely to rate anti-Black language as toxic (Sap et al., 2022).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Sap et al., 2022
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Classified as probably not a substantial claim.
Interpretationnone givenNUTMEG's data efficiency makes it well-suited for crowdsourcing settings.
“Its data efficiency makes it well-suited for crowdsourcing settings, and its ability to model annotator variation contributes to more representative NLP models.”
Quote in the paperSubstantial claimClassified as probably not a substantial claim.
Study design and conductnone givenOur experiments with synthetic and real data used a known, discrete subpopulation label for each annotator.
“Our experiments with synthetic and real data used a known, discrete subpopulation label for each annotator.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductreasoningAll random seeds in Python were set to 42 to ensure reproducibility.
“To ensure reproducibility, we set all random seeds in Python to be 42.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
InterpretationreasoningNUTMEG could potentially increase the collection of personal data.
“As a result, NUTMEG could potentially increase the collection of personal data, which needs to be responsibly stored.”
Quote in the paperNot a substantial claim
The authors state this as an assumption.
In NUTMEG, there exists a single correct label for each subpopulation that is always given by an annotator in that subpopulation when they try to.
“In NUTMEG, we relax this assumption so that there exists a single correct label for each subpopulation that is always given by an annotator in that subpopulation when they try to.”
The authors state this as an assumption.
Observed labels from different subpopulations are independent.
“For this process, we make the simplifying assumption that observed labels from different subpopulations are independent.”
The paper relies on this without support.
Annotator demographics (gender, age, race, and education) capture relevant subpopulation-level variation in judgments for the tasks studied.
“we choose to evaluate on data using annotator demographics to split subpopulations”
“Our experiments with synthetic and real data used a known, discrete subpopulation label for each annotator.”
The paper relies on this without support.
Most real data contains label variation beyond that due to subpopulations.
“most real data contains label variation beyond that due to subpopulations”
The paper relies on this without support.
The full, disaggregated label distribution of a subpopulation in the test set represents the true label distribution for that subpopulation.
“we score using the label distribution of the full, disaggregated labels in the test set”
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Per-annotator minimum exceeds the stated average
MinorThe paper says every annotator labels more than 20 items, but gives an average of 16.67 items per annotator. An average cannot be below the minimum for every annotator.
“each annotator labels more than 20 items”
“average of 16.67 items per annotator”
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 34
- Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. 2019. Optuna: A Next- generation Hyperparameter Optimization Framework. In Proceedings of the 25th ACM SIGKDD Interna- tional Conference on Knowledge Discovery & Data Mining, KDD '19, pages 2623-2631, New York, NY, USA. Association for Computing Machinery. · DOI
- Lora Aroyo and Chris Welty. 2014. The three sides of crowdtruth. Human Computation, 1(1). · DOI · Paper page
- Mélanie Bernhardt, Daniel C. Castro, Ryutaro Tanno, Anton Schwaighofer, Kerem C. Tezcan, Miguel Mon- teiro, Shruthi Bannur, Matthew P. Lungren, Aditya Nori, Ben Glocker, Javier Alvarez-Valle, and Ozan Oktay. 2022. Active label cleaning for improved dataset quality under resource constraints. Nature Communications, 13(1):1161. Publisher: Nature Publishing Group. · DOI
- Federico Cabitza, Andrea Campagner, and Valerio Basile. 2023. Toward a perspectivist turn in ground truthing for predictive computing. Proceedings of the AAAI Conference on Artificial Intelligence, 37(6):6860-6868. · DOI · Paper page
- A. P. Dawid and A. M. Skene. 1979. Maximum Likeli- hood Estimation of Observer Error-Rates Using the EM Algorithm. Journal of the Royal Statistical Soci- ety. Series C (Applied Statistics), 28(1):20-28. Pub- lisher: [Royal Statistical Society, Oxford University Press]. · DOI · Paper page
- Eve Fleisig, Rediet Abebe, and Dan Klein. 2023. When the majority is wrong: Modeling annotator disagree- ment for subjective tasks. In Proceedings of the 2023 Conference on Empirical Methods in Natural Lan- guage Processing, pages 6715-6726, Singapore. As- sociation for Computational Linguistics. · DOI
- Eve Fleisig, Su Lin Blodgett, Dan Klein, and Zeerak Talat. 2024. The perspectivist paradigm shift: As- sumptions and challenges of capturing human labels. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 2279-2292, Mexico City, Mexico. Association for Computational Lin- guistics. · DOI
- Tommaso Fornaciari, Alexandra Uma, Silviu Paun, Bar- bara Plank, Dirk Hovy, and Massimo Poesio. 2021. Beyond black & white: Leveraging annotator dis- agreement via soft-label multi-task learning. In Pro- ceedings of the 2021 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2591-2597, Online. Association for Computational Linguistics. · DOI
- Mitchell L. Gordon, Michelle S. Lam, Joon Sung Park, Kayur Patel, Jeff Hancock, Tatsunori Hashimoto, and Michael S. Bernstein. 2022. Jury Learning: Integrat- ing Dissenting Voices into Machine Learning Models. In CHI Conference on Human Factors in Computing Systems, pages 1-19, New Orleans LA USA. ACM. · DOI
- Dirk Hovy, Taylor Berg-Kirkpatrick, Ashish Vaswani, and Eduard Hovy. 2013. Learning whom to trust with MACE. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1120-1130, Atlanta, Georgia. Association for Computational Linguistics.
- Srikanth Jagabathula, Lakshminarayanan Subramanian, and Ashwin Venkataraman. 2017. Identifying unreli- able and adversarial workers in crowdsourced label- ing tasks. Journal of Machine Learning Research, 18(93):1-67.
- Hyun-Chul Kim and Zoubin Ghahramani. 2012. Bayesian classifier combination. In Artificial Intelli- gence and Statistics, pages 619-627.
- PMLR.
- Savannah Larimore, Ian Kennedy, Breon Haskett, and Alina Arseniev-Koehler. 2021. Reconsidering anno- tator disagreement about racist language: Noise or signal? In Proceedings of the Ninth International Workshop on Natural Language Processing for So- cial Media, pages 81-90, Online. Association for Computational Linguistics. · DOI
- Qiang Liu, Jian Peng, and Alexander T Ihler. 2012. Vari- ational Inference for Crowdsourcing. In Advances in Neural Information Processing Systems, volume 25.
- Curran Associates, Inc. Soda Marem Lo, Valerio Basile, et al. 2023. Hierar- chical clustering of label-based annotator represen- tations for mining perspectives. In CEUR WORK- SHOP PROCEEDINGS, volume 3494, pages 1-10. CEUR-WS.
- Negar Mokhberian, Myrl Marmarelis, Frederic Hopp, Valerio Basile, Fred Morstatter, and Kristina Lerman. 2024. Capturing perspectives of crowdsourced anno- tators in subjective learning tasks. In Proceedings of the 2024 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 7337-7349, Mexico City, Mexico. As- sociation for Computational Linguistics. · DOI
- Aida Mostafazadeh Davani, Mark Diaz, Dylan K Baker, and Vinodkumar Prabhakaran. 2024. D3CODE: Dis- entangling disagreements in data across cultures on offensiveness detection and evaluation. In Proceed- ings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 18511-18526, Miami, Florida, USA. Association for Computational Linguistics. · DOI
- Aida Mostafazadeh Davani, Mark Díaz, and Vinodku- mar Prabhakaran. 2022. Dealing with disagreements: Looking beyond the majority vote in subjective an- notations. Transactions of the Association for Com- putational Linguistics, 10:92-110. · DOI
- Stefanie Nowak and Stefan Rüger. 2010. How reliable are annotations via crowdsourcing: a study about inter-annotator agreement for multi-label image an- notation. In Proceedings of the International Confer- ence on Multimedia Information Retrieval, MIR '10, page 557-566, New York, NY, USA. Association for Computing Machinery. · DOI
- Silviu Paun, Bob Carpenter, Jon Chamberlain, Dirk Hovy, Udo Kruschwitz, and Massimo Poesio. 2018. Comparing Bayesian models of annotation. Transac- tions of the Association for Computational Linguis- tics, 6:571-585. · DOI
- Jiaxin Pei and David Jurgens. 2023. When do annota- tor demographics matter? measuring the influence of annotator demographics with the POPQUORN dataset. In Proceedings of the 17th Linguistic Annota- tion Workshop (LAW-XVII), pages 252-265, Toronto, Canada. Association for Computational Linguistics. · DOI
- Barbara Plank. 2022. The "problem" of human label variation: On ground truth in data, modeling and evaluation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Process- ing, pages 10671-10682, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. · DOI · Paper page
- Barbara Plank, Dirk Hovy, and Anders Søgaard. 2014. Linguistically debatable or just plain wrong? In Proceedings of the 52nd Annual Meeting of the As- sociation for Computational Linguistics (Volume 2: Short Papers), pages 507-511, Baltimore, Maryland. Association for Computational Linguistics. · DOI · Paper page
- Maarten Sap, Swabha Swayamdipta, Laura Vianna, Xuhui Zhou, Yejin Choi, and Noah A. Smith. 2022. Annotators with attitudes: How annotator beliefs and identities bias toxic language detection. In Pro- ceedings of the 2022 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5884-5906, Seattle, United States. Association for Computational Linguistics. · DOI
- Pia Sommerauer, Antske Fokkens, and Piek Vossen. 2020. Would you describe a leopard as yellow? eval- uating crowd-annotations with justified and informa- tive disagreement. In Proceedings of the 28th Inter- national Conference on Computational Linguistics, pages 4798-4809, Barcelona, Spain (Online). Inter- national Committee on Computational Linguistics. · DOI
- Alexandra N. Uma, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, and Massimo Poesio. 2021. Learning from Disagreement: A Survey. Journal of Artificial Intelligence Research, 72:1385- 1470. · DOI
- Nikolas Vitsakis, Amit Parekh, and Ioannis Konstas. 2024. Voices in a crowd: Searching for clusters of unique perspectives. In Proceedings of the 2024 Con- ference on Empirical Methods in Natural Language Processing, pages 12517-12539, Miami, Florida, USA. Association for Computational Linguistics. · DOI
- Ruyuan Wan, Jaehyung Kim, and Dongyeop Kang. 2023. Everyone's Voice Matters: Quantifying An- notation Disagreement Using Demographic Informa- tion. Proceedings of the AAAI Conference on Arti- ficial Intelligence, 37(12):14523-14530. Number: 12. 2886 · DOI
- Xinpeng Wang and Barbara Plank. 2023. ACTOR: Ac- tive learning with annotator-specific classification heads to embrace human label variation. In Proceed- ings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2046-2052, Singapore. Association for Computational Linguis- tics. · DOI
- Leon Weber-Genzel, Siyao Peng, Marie-Catherine De Marneffe, and Barbara Plank. 2024. VariErr NLI: Separating annotation error from human label varia- tion. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pages 2256-2269, Bangkok, Thailand. Association for Computational Linguistics. · DOI
- Tharindu Cyril Weerasooriya, Alexander Ororbia, Raj Bhensadadia, Ashiqur KhudaBukhsh, and Christo- pher Homan. 2023. Disagreement matters: Preserv- ing label diversity by jointly modeling item and an- notator label distributions with DisCo. In Findings of the Association for Computational Linguistics: ACL 2023, pages 4679-4695, Toronto, Canada. Associa- tion for Computational Linguistics. · DOI
- Jacob Whitehill, Ting-fan Wu, Jacob Bergsma, Javier Movellan, and Paul Ruvolo. 2009. Whose Vote Should Count More: Optimal Integration of Labels from Labelers of Unknown Expertise. In Advances in Neural Information Processing Systems, volume 22.
- Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Remi Louf, Morgan Funtow- icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Trans- formers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics. · DOI · Paper page
- Authors
- Ivey, J., Gauch, S., & Jurgens, D.
- Affiliations
- Johns Hopkins University (US)
- University of Arkansas at Fayetteville (US)
- University of Michigan (US)
- Venue
- Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing
- Published
- 2025
- Pages
- 2874-2887
- OpenAlex
- W4416037083
- Cited by
- 0
- References
- 0
- FWCI
- 0.00
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- Journal of the Arkansas Academy of Science (submittedVersion, cc-by-nd)
- arXiv (Cornell University) (cc-by)
- Funders
- National Science Foundation (2143529, 1946391)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Keywords
- annotator disagreement, noisy annotations, Bayesian model, annotation aggregation, nutmeg
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-29.
Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-29.
Cited sources are checked by gpt-6-sol.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
