CoChat
Sign Up
PreprintCited by 2Hate Speech and Cyberbullying DetectionSentiment Analysis and Opinion MiningTopic Modeling

Benchmark on Peer Review Toxic Detection: A Challenging Task with a New Dataset

Open access
Luo, Man, Peterson, Bradley, Gan, Rafael, Ramalingame, Hari, Gangrade, Navya, Dimarogona, Ariadne, Banerjee, Imon, Howard, Phillip·2025·arXiv
DOI 10.48550/arxiv.2502.01676arXiv 2502.01676
Open this paper in CoChat — free accountOpen PDF

Abstract

Peer review is crucial for advancing and improving science through constructive criticism. However, toxic feedback can discourage authors and hinder scientific progress. This work explores an important but underexplored area: detecting toxicity in peer reviews. We first define toxicity in peer reviews across four distinct categories and curate a dataset of peer reviews from the OpenReview platform, annotated by human experts according to these definitions. Leveraging this dataset, we benchmark a variety of models, including a dedicated toxicity detection model, a sentiment analysis model, several open-source large language models (LLMs), and two closed-source LLMs. Our experiments explore the impact of different prompt granularities, from coarse to fine-grained instructions, on model performance. Notably, state-of-the-art LLMs like GPT-4 exhibit low alignment with human judgments under simple prompts but achieve improved alignment with detailed instructions. Moreover, the model's confidence score is a good indicator of better alignment with human judgments. For example, GPT-4 achieves a Cohen's Kappa score of 0.56 with human judgments, which increases to 0.63 when using only predictions with a confidence score higher than 95%. Overall, our dataset and benchmarks underscore the need for continued research to enhance toxicity detection capabilities of LLMs. By addressing this issue, our work aims to contribute to a healthy and responsible environment for constructive academic discourse and scientific collaboration.

AI snapshot

Written from the paper’s full text.

TL;DR

This paper introduces an expert-annotated dataset of 313 peer-review sentences from OpenReview for detecting toxicity, defined across four categories. It benchmarks toxicity, sentiment, and LLM detectors, finding GPT-4 best aligns with human judgments under detailed prompts (Cohen’s Kappa 0.56, rising to 0.63 with >95% confidence).

Key findings

  • The final testing set contains 313 sentences: 182 non-toxic and 131 toxic, drawn from 50,108 OpenReview reviews across 47 conferences/workshops; the final test reviews come from 6 conferences/workshops. The most common toxic subcategory is “lack of constructive feedback,” followed by “Excessive Negativity,” “Emotive Comments,” and “Personal Attack.”
  • The Perspective API performs poorly: mean predicted toxicity probability is 0.03 and max is 0.32; its maximum Cohen’s Kappa is approximately 0.25 within thresholds [0.006, 0.06]. The sentiment model SA-2 reaches Cohen’s Kappa 0.12, better than most open-source LLMs.
  • With simple prompts, Qwen-7B has the best open-source alignment (Cohen’s Kappa 0.16), while GPT-4 has 0.06. With detailed instructions, GPT-4 reaches Cohen’s Kappa 0.56 and GPT-3.5 reaches 0.28; with a toxicity-summary prompt, GPT-4 drops to 0.35. Using only GPT-4 predictions with confidence >95% raises alignment to 0.63; at 100% confidence alignment is perfect but only 3 sentences remain.
  • GPT-3.5 revisions of toxic sentences were preferred by human evaluators 80% of the time (28/35 revisions).

Methods

  • Data collection and annotation: OpenReview API reviews from 2018–2023 yielded 50,108 reviews across 47 conferences/workshops; 1,495 were randomly sampled for annotation. Five undergraduate annotators identified potential toxic sentences, three senior researchers verified them, and a two-phase process (50 sentences, then 300) retained only sentences with unanimous agreement, producing 313 test sentences.
  • Benchmark: binary toxic/non-toxic classification. Models included the Perspective API, a sentiment classifier (SA-1 and SA-2), open-source LLMs (Gemma-7B, Qwen-7B, Mistral-7B, LLaMA3-8B, LLaMA3-70B), and closed-source GPT-3.5 and GPT-4. Prompts varied from simple to detailed to summary; metrics were precision, recall, F1, accuracy, and Cohen’s Kappa. GPT-4 confidence scores were also analyzed, and GPT-3.5 was tested for toxic-sentence revision.

Limitations

  • Stated: the toxicity definition may not be the most precise and can raise different opinions; toxicity is subjective, so annotations can suffer from inconsistency; only sentences with unanimous annotator agreement were included.
  • Stated: the task was formulated as binary classification for simplicity, with more fine-grained classification left as future work.
  • Evident: the final test set is small (313 sentences); the 100% confidence threshold leaves only 3 sentences; the revision evaluation is small (35 sentences); and open-source models failed to follow detailed instructions, limiting their comparison under that prompt.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2502.01676

Cites

0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

8 of 14 cited sources checked · 4 supported · 0 flagged · accuracy not yet measured

  • Premisecitation

    Toxic reviews may hinder scientific progress.

    “This, in turn, may hinder scientific progress (Beaumont 2019;Gerwing et al. 2020).”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites Beaumont 2019 source
    • cites Gerwing et al. 2020 source
  • Premisecitation

    Most efforts in toxicity detection focus on social media comments.

    “Most efforts in the domain of toxicity detection are focusing on social media comments (Cheng et al. 2022;Hosseinmardi et al. 2015;Ghosh et al. 2021;Bensalem, Rosso, and Zitouni 2024).”
    Quote in the paperSubstantial claim3 of 3 sources checked
    • cites Cheng et al. 2022 source

      The Abstract links toxicity detection to abusive comments associated with social media and says models have been developed to detect toxic comments online. It does not establish that most toxicity-detection efforts focus on social media comments; searches for that broader characterization found no further support in our copy. (LLM, conf=0.78)

      Experimental — accuracy not yet measured
    • cites Hosseinmardi et al. 2015 source

      The Introduction (p. 1) says prior cyberbullying and cyberaggression studies largely used text analysis of online comments. That supports the comment-focused part of the claim, but the paper does not establish that most toxicity-detection efforts generally focus on social media comments. (LLM, conf=0.84)

      Experimental — accuracy not yet measured
    • cites Ghosh et al. 2021 source

      The Introduction says most prior work and datasets focus on conversational text, and Related Work says most hate-speech detection work is text-based. The paper discusses social networks and cites social-media studies, but it does not establish that most efforts specifically focus on social media comments. (LLM, conf=0.84)

      Experimental — accuracy not yet measured
  • Findingown data

    The paper develops the first benchmark dataset for detecting toxic sentences in peer-review context.

    “We develop the first benchmark dataset for detecting toxic sentences in peer-review context.”
    Quote in the paperSubstantial claim
  • Findingown data

    When given a detailed prompt, all open-source models did not generate intended answers (e.g., 0/1 or toxic/non-toxic).

    “However, when given such detailed prompt to the open-source models, all models do not generate an intended answers (e.g.0/1 or toxic/non toxic).”
    Quote in the paperSubstantial claim
  • Findingown data

    The distribution of decision categories for toxic sentences reveals a shift toward more reject decisions.

    “Figure 2 shows the distribution of decision categories specifically for toxic sentences, revealing a shift toward more reject decisions.”
    Quote in the paperNot a substantial claim
  • Classified as probably not a substantial claim. 26
  • The quote only partly states this claim. 1
  • May combine two claims. 20

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    58% of 1,106 survey participants reported having received an unprofessional review.

    “The prevalence of this issue is underscored by a survey in which 58% of 1,106 participants reported having received an unprofessional review (Silbiger and Stubler 2019).”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Silbiger and Stubler 2019 source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Toxic reviews can severely impact authors, causing feelings of discouragement, anger, or even depression.

    “Such reviews, characterized by harsh and unconstructive criticism, rude comments, and unprofessionalism, can severely impact authors, causing feelings of discouragement, anger, or even depression (Baron 1988;Mavrogenis, Quaile, and Scarlat 2020).”
    Quote in the paperNot a substantial claim1 of 2 sources checked
    • cites Baron 1988 source
    • cites Mavrogenis, Quaile, and Scarlat 2020 source

      In “The rude peer-review” (p. 2), the paper says rude reviews can be discouraging, make authors “give up in despair,” and cause psychological distress and self-doubt. This supports a serious emotional impact, but the paper does not specifically report anger or depression. (LLM, conf=0.86)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The Perspective API has been widely used in previous studies on automatic toxicity detection.

    “The Perspective API has been widely used in previous studies on automatic toxicity detection (Deshpande et al. 2023;Faal, Schmitt, and Yu 2023;Wen et al. 2023;Howard et al. 2024)”
    Quote in the paperNot a substantial claim2 of 3 sources checked
    • cites Deshpande et al. 2023 source
    • cites Faal, Schmitt, and Yu 2023

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Wen et al. 2023 source

  • Probably not a substantial claim: may not pass S3 (load-bearing). The quote only partly states this claim.

    Findingown data

    In some cases, the model revision changes the meaning of the original sentence.

    “Original, the model revision change the meaning of the original sentences.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The maximum Cohen's Kappa score for the toxicity detection model is approximately 0.25 and falls within the threshold range [0.006, 0.06].

    “The maximum Cohen's Kappa score, approximately 0.25, falls within the range of [0.006, 0.06].”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning

    A higher confidence threshold means fewer sentences are judged; the number of sentences selected for the thresholds shown in the figure are 313, 300, 300, 258, 248, 53, 3.

    “However, a higher threshold also means less sentences are being judged, the number of sentences being selected for the threshold shown in the figure are: 313, 300, 300, 258, 248, 53, 3.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Closed-source models GPT-3.5 and GPT-4 can follow the detailed instruction and achieve much better alignment compared to the previous simple prompt.

    “On the other hand, the close-source models, GPT-3.5 and GPT-4 can follow the instruction and achieve much better alignment compared to the previous simple prompt as shown in the first block performance in Table 3.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The SA-2 result is better than most open-source LLMs, which have much larger parameters.

    “While the result does not reach a substantial alignment with human, it is even better than most of the open source LLMs which are at much larger parameters.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Findingreasoning

    The paper defined toxicity and identified four key toxic categories.

    “Lacking a comprehensive guideline, we first defined toxicity and identified four key toxic categories.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: There is a lack of a comprehensive guideline for toxicity in peer review.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The paper randomly sampled 1,495 reviews for annotation.

    “Among the total collection, we randomly sample 1,495 for the annotations”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Open-source models struggled to align with human judgments, highlighting the challenge of detecting toxicity in peer reviews.

    “Our results suggest that Open-source models struggled to align with human judgments, highlighting the challenge of detecting toxicity in peer reviews.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Both GPT-3.5 and GPT-4 performance decreases significantly when using the toxicity summary prompt compared to the detailed instruction prompt.

    “Both model performance decreases significantly, this showcase the importance of the detailed instruction for detecting the toxicity in the peer-review context.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    On average, each review consisted of 1,553 characters.

    “and on average, each review consisted of 1,553 characters.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Closed-source models like GPT-3.5 and GPT-4 showed much better alignment, suggesting their potential with careful use.

    “Conversely, closed-source models like GPT-3.5 and GPT-4 showed much better alignment, suggesting their potential with careful use.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The toxicity detection models predicted very low probability of being toxic for all sentences in the testing set, with mean 0.03 and maximum 0.32.

    “The toxicity detection models have predict very low probability of being toxic for all the sentences: the mean value cross the entire testing set is 0.03 and the max value is only 0.32.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    There are three examples where GPT-4 is 100% confident in its answer and all are aligned with human judgement.

    “Here we show the three examples where GPT-4 is 100% confidence its answer and all of them are aligned with human judgement.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Detailed instructions significantly improve GPT-4's performance and alignment with human judgments.

    “discovering that detailed instructions significantly improve the model's performance and alignment with human judgments.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given

    A toxic sentence could belong to multiple categories.

    “Note that a toxic sentence could belong to multiple categories.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingnone given

    While constructiveness does not improve, the annotator reports that the revision is more polite and thus more preferable.

    “While the constructiveness does not improve, the annotator still reports that the revision is more polite and thus more preferable.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The model's confidence scores had min/max/mean values of 70%, 89%, and 100%.

    “The min/max/mean values of the confidence is 70%, 89%, and 100%, this shows that the model is quite confidence with its answer in most of the cases.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    GPT-3.5 is generally competent at revising toxic sentences while maintaining the original critiques.

    “We find that the revisions are favorable 80% of the time (28/35 revisions) suggesting that the model is generally competent.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Study design and conductnone given

    The dataset follows rigorous human annotation to ensure its reliability.

    “This dataset follows rigorous human annotation to ensure its reliability.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given

    The annotation procedure was conducted in two phases, each consisting of independent judgement and discussion among annotators.

    “the annotation procedure is conducted into two phases, and each phase consist of an independent judgement and a discussion among the annotators.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductown data

    In the initial round of 50 sentences, the Cohen's Kappa scores among the three annotators were 0.74, 0.52, and 0.34, with a Fleiss' Kappa score of 0.55.

    “In the initial round of 50 sentences, the Cohen's Kappa scores among the three annotators were 0.74, 0.52, and 0.34, with a Fleiss' Kappa score of 0.55.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    Only sentences with consenting from all annotators are included in the testing set.

    “Eventually only sentences with consenting from all annotators are included in our testing set.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    Many journals have guidelines and codes of conduct for peer-reviewers, but unacceptable behaviour are often difficult to be found on journal websites.

    “Many journals have guidelines and codes of conduct for peer-reviewers, but the unacceptable behaviour are often difficult to be found on journal websites”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites foot_5

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Study design and conductown data

    In the larger set of 300 sentences, the initial Cohen's Kappa score was 0.60, which increased to 0.92 after discussion.

    “In this larger set, the initial Cohen's Kappa score was 0.60, which significantly increased to 0.92 after discussion.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationown data

    The model is quite confident with its answer in most cases.

    “The min/max/mean values of the confidence is 70%, 89%, and 100%, this shows that the model is quite confidence with its answer in most of the cases.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    The paper's toxicity definition may raise different opinions among researchers due to lack of formal definition and few literature.

    “We acknowledge that due to the lack of formal definition of toxicity and few literature are available, our toxicity definition in §3.2 can raise different opinion among researchers.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationown data

    The paper's dataset and benchmarks underscore the need for continued research to enhance toxicity detection capabilities of LLMs.

    “our dataset and benchmarks underscore the need for continued research to enhance toxicity detection capabilities of LLMs.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    Due to the subjective nature of toxicity, annotations can suffer from inconsistency.

    “due to the subject nature of toxicity, the annotations can suffer from inconsistency”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Premisecitation

    Using emotive or sarcastic language is often the hardest for authors to cope with.

    “Using emotive or sarcastic language is often the hardest for authors to cope with”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites foot_7

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Study design and conductown data

    After discussing disagreements, the Fleiss' Kappa score increased to 0.63, and the highest Cohen's Kappa score between two annotators rose to 0.83.

    “After discussing the sentences with disagreements, the Fleiss' Kappa score increased to 0.63, and the highest Cohen's Kappa score between two annotators rose to 0.83.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationown data

    There is a potential possibility of using LLMs to detoxify peer reviews.

    “indicating a potential possibility of using LLMs to detoxify peer reviews.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    Potentially toxic peer-review sentences are generally assigned very low toxicity probabilities, indicating that the semantic interpretation of toxicity in peer review differs significantly from that in the general domain on which the toxicity model was trained.

    “This suggests that potentially toxic peer-review sentences are generally assigned very low toxicity probabilities, indicating that the semantic interpretation of toxicity in peer review differs significantly from that in the general domain on which the toxicity model was trained.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Usually a toxic peer-review does not contain overtly hostile or abusive language, which is often a defining characteristic of toxic reviews in the general domain.

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    It is almost not possible to make a comment more constructive without reading the paper.

    “On the other hand, it is almost not possible to make a comment more constructive without reading the paper, therefore, we suggest that for this type of toxicity, rather than ask the model to rewrite the sentence, the model should remind the reviewer to give more constructive feedback.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Five undergraduate students majoring in computer science initially identified toxic sentences.

    “Initially, five undergraduate students majoring in computer science identified toxic sentences.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given

    No previous work has provided a comprehensive definition of toxic sentences in peer review.

    “We define four characteristic of toxic sentences in peerreview context as no previous work have provided comprehensive definition.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    ACL Rolling Review is an exception, where a list of lazy review behavior have been mentioned.

    “ACL Rolling Review is an exception, where a list of lazy review behavior have been mentioned”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites foot_6

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The final testing set excludes data points where annotators did not reach agreement.

    “We exclude those data point in our final testing set”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductown data

    The two annotators with the highest agreement were selected to annotate the larger set of 300 sentences.

    “based on Cohen's Kappa score, the two annotators with the highest agreement were selected to annotate the larger set of 300 sentences.”
    Quote in the paperSubstantial claim
41 of 41

  • The paper relies on this without support.

    The reviews collected from OpenReview are representative of peer reviews in general.

    “To collect paper reviews, we utilized the OpenReview API, starting with a comprehensive list of all venues between 2018 and 2023, including conferences and workshops.”

  • The paper relies on this without support.

    Human annotations are a valid ground truth for toxicity in peer reviews.

    “To ensure the reliability of the dataset, the annotation procedure is conducted into two phases, and each phase consist of an independent judgement and a discussion among the annotators.”

  • The paper relies on this without support.

    Usually a toxic peer-review does not contain overtly hostile or abusive language, which is often a defining characteristic of toxic reviews in the general domain.

    “It is because usually a toxic peer-review does not contain overtly hostile or abusive language, which is often a defining characteristic of toxic reviews in general domain.”

  • The paper relies on this without support.

    The paper is the first to explore toxicity in peer review.

    “Our work is the first to explore toxicity in peer review.”

  • The paper relies on this without support.

    There is a lack of a comprehensive guideline for toxicity in peer review.

    “Lacking a comprehensive guideline, we first defined toxicity and identified four key toxic categories.”

  • The paper relies on this without support.

    Toxic peer reviews are negative in nature.

    “As toxic peer reviews are also negative in nature, our task can be viewed as related to the problem of sentiment analysis (Socher et al. 2013).”
6 of 6

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Number of review-decision categories differs

    Minor

    The Table 1 caption says the final testing-set reviews are categorized into five groups, whereas Figure 1 describes six decision categories for the same testing data.

  • Confidence maximum is lower than a reported confidence value

    Minor

    The stated minimum/maximum/mean sequence assigns 89% as the maximum and 100% as the mean. The paper also reports predictions with 100% confidence, contradicting that maximum.

2 of 2

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 1

  1. Rosso, P.; and Zitouni, H. 2024. Toxic lan- guage detection: A systematic review of Arabic datasets. Ex- pert Systems, e13551.

Cited by

Sorted by citations

Related

Authors
Luo, M., Peterson, B., Gan, R., Ramalingame, H., Gangrade, N., Dimarogona, A., Banerjee, I., & Howard, P.
Affiliations
  • Intel (United States) (US)
  • Arizona State University (US)
  • Mayo Clinic (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2025
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2502.01676
arXiv
2502.01676
OpenAlex
W4407184773
Cited by
1
References
0
Citations by year
2026: 1
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Hate Speech and Cyberbullying Detection
  • Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
toxicity detection, annotated dataset, large language models, human judgment alignment, sentiment analysis, academic discourse

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-29.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-29.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review