Cited by 43Neuroendocrine regulation and behaviorOlfactory and Sensory Function StudiesAction Observation and Synchronization

Null results of oxytocin and vasopressin administration across a range of social cognitive and behavioral paradigms: Evidence from a randomized controlled trial

Open access
Tabak, Benjamin A., Teed, Adam R., Castle, Elizabeth, Dutcher, Janine M., Meyer, Meghan L., Bryan, Ronnie, Irwin, Michael R., Lieberman, Matthew D. +1 more·2019·Psychoneuroendocrinology

Abstract

Research examining oxytocin and vasopressin in humans has the potential to elucidate neurobiological mechanisms underlying human sociality that have been previously unknown or not well characterized. A primary goal of this work is to increase our knowledge about neurodevelopmental and psychiatric disorders characterized by impairments in social cognition. However, years of research highlighting wide-ranging effects of, in particular, intranasal oxytocin administration have been tempered as the fields of psychology, neuroscience, and other disciplines have been addressing concerns over the reproducibility and validity of research findings. We present a series of behavioral tasks that were conducted using a randomized, double-blind, placebo controlled, between-subjects design, in which our research group found no main effects of oxytocin and vasopressin on a host of social outcomes. In addition to null hypothesis significance testing, we implemented equivalence testing and Bayesian hypothesis testing to examine the sensitivity of our findings. These analyses indicated that 47–83% of our results (depending on the method of post-hoc analysis) had enough sensitivity to detect the absence of a main effect. Our results add to evidence that intranasal oxytocin may have a more limited direct effect on human social processes than initially assumed and suggest that the direct effects of intranasal vasopressin may be similarly limited. Randomized controlled trial registration: .

AI snapshot

Written from the paper’s full text.

TL;DR

In a randomized, double-blind, placebo-controlled between-subjects trial, intranasal oxytocin and vasopressin showed no significant main effects across six social cognitive and behavioral tasks and 18 dependent variables. Equivalence and Bayesian analyses indicated that 47–83% of null results had enough sensitivity to support the absence of a main effect, though many tests remained underpowered.

Key findings

  • Null-hypothesis significance testing found no main effects of oxytocin or vasopressin versus placebo on empathic concern, social versus non-social working memory, deception detection, interpersonal distance, hypothetical bystander intervention, or written reflections on interpersonal interactions (18 dependent variables; 36 comparisons).
  • Equivalence testing showed that, with Bonferroni correction, 47.22% (17/36) of null results were due to statistical equivalence; with Holm correction, 72.22% (26/36) were due to statistical equivalence. For oxytocin vs. placebo, 10/18 (55.56%) were equivalent under Bonferroni and 15/18 (83.33%) under Holm; for vasopressin vs. placebo, 7/18 (38.89%) and 11/18 (61.11%), respectively.
  • Bayesian hypothesis testing found no support for the alternative hypothesis. With the r=0.707 JASP prior, 25/36 tests (69.44%) reached BF=3; with Jeffreys’ r=1 prior, 30/36 tests (83.33%) reached BF=3. The largest BF using the .707 Cauchy prior was 4.27.
  • The average smallest effect size detectable across all 36 main-effect tests was d=0.67, ranging from d=0.62 to d=0.8.

Methods

  • Randomized, double-blind, placebo-controlled, between-subjects design with 125 UCLA undergraduates (90 female, 35 male; age range 18–31 years, mean age=20.88, SD=2.71) assigned to intranasal oxytocin (n=42), vasopressin (n=42), or placebo (n=41). Oxytocin dose was 24 IU; vasopressin dose was 20 IU; tasks began approximately 40 minutes after administration and were completed in randomized order.
  • Six behavioral tasks produced 18 dependent variables. Analyses focused on main effects of oxytocin vs. placebo and vasopressin vs. placebo (36 comparisons), using null-hypothesis significance testing, equivalence testing (TOSTER/TOST with bounds based on the smallest detectable effect at 80% power), and Bayesian t-tests (JASP; Jeffreys’ r=1 and r=0.707 priors).
  • The study sought more than 34 participants per condition to achieve >80% power, based on prior moderate-to-large vasopressin effects (e.g., d=0.7) on social cognitive processes.

Limitations

  • Stated: relatively small sample size; predominantly female sample (approximately 3:1); healthy non-clinical sample; only one dose of oxytocin and vasopressin; varied timing of tasks relative to doses and incubation period; post-hoc null results were not preregistered and may be subject to researcher bias (e.g., reverse p-hacking); two tasks were late additions not added to the preregistration.
  • Stated/evident: 17–53% of analyses lacked sensitivity to detect statistical equivalence; smallest detectable effects were rather large (average d=0.67); Bayesian evidence was modest (largest BF=4.27 with the .707 prior), so some conclusions rest on low-end moderate evidence.
  • Evident: the authors chose not to conduct an internal meta-analysis; findings are limited to main effects, not interactions, and to the specific tasks, doses, and timing used.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1016/j.psyneuen.2019.04.019

Cites

38 of 38 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 15 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    The study found no main effects of oxytocin and vasopressin on social outcomes across the behavioral tasks.

    “our research group found no main effects of oxytocin and vasopressin on a host of social outcomes.”
    • Also stated in: § Introduction · plain
      “In the Supplementary Material, we describe the results of analyses using NHST, which showed null results for both drugs vs. PLA across all dependent variables.”
    Quote in the paperSubstantial claim
  • Findingown data

    Equivalence testing and Bayesian hypothesis testing indicated that 47-83% of the results, depending on the post-hoc method, had enough sensitivity to detect the absence of a main effect.

    “These analyses indicated that 47–83% of our results (depending on the method of post-hoc analysis) had enough sensitivity to detect the absence of a main effect.”
    Quote in the paperSubstantial claim
  • Findingown data

    After equivalence testing and Bayesian hypothesis testing on all tests, approximately 47-83% of the analyses provided evidence for the null model relative to the alternative model.

    “After conducting equivalence testing and Bayesian hypothesis testing on all tests, we found that approximately 47–83% of our analyses provided evidence for the null model relative to the alternative model.”
    Quote in the paperSubstantial claim
  • Findingown data

    Using the r=0.707 JASP prior, 25 tests (69.44%) reached BF=3, indicating at least three times more evidence for the null hypothesis relative to the alternative.

    “Using the r=0.707 JASP prior, 25 tests (69.44%) reached BF=3, which is typically considered “moderate” evidence (i.e., nearly all the tests provided at least three times more evidence for the null hypothesis relative to the alternative hypothesis).”
    Quote in the paperSubstantial claim
  • Findingown data

    The largest BF using a .707 Cauchy was 4.27, indicating the null model was only 4.27 times more favored than the alternative model in the data.

    “Specifically, the largest BF using a .707 Cauchy was 4.27, which suggests the null model was only 4.27 times more favored than the alternative model in our data, demonstrating modest evidence.”
    Quote in the paperSubstantial claim
  • May combine two claims. 19
  • The quote only partly states this claim. 1
  • Classified as probably not a substantial claim. 16
  • May combine two claims.

    Findingown data

    The average smallest effect size detectable across all 36 main effects tests conducted in the study, regardless of the neuropeptide, was d=0.67 and ranged from d=0.62 to d=0.8.

    “The average smallest effect size detectable across all 36 main effects tests conducted in the study, regardless of the neuropeptide, was d=0.67 and ranged from d=0.62 to d=0.8.”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Findingown data

    Under Holm correction, OT vs. PLA resulted in 15 of 18 (83.33%) null results sensitive enough to reveal statistical equivalence, whereas AVP vs. PLA resulted in 11 of 18 (61.11%).

    “Using Holm correction for multiple testing, this changed to 15 of 18 (83.33%) for OT vs. PLA and 11 of 18 (61.11%) AVP vs. PLA.”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Findingown data

    Using Holm correction for multiple testing, 72.22% (26 of 36) of all null results were due to statistical equivalence, leaving 27.78% of analyses lacking sensitivity.

    “Using a less conservative Holm correction for multiple testing, 72.22%, or 26 of 36 of all null results were due to statistical equivalence which resulted in only 27.78% of analyses that lacked sensitivity.”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Findingown data

    Under Bonferroni correction, OT vs. PLA resulted in 10 of 18 (55.56%) null results sensitive enough to reveal statistical equivalence, whereas AVP vs. PLA resulted in 7 of 18 (38.89%).

    “These results show that OT vs. PLA resulted in 10 of 18 (55.56%) null results that had the sensitivity to reveal statistical equivalence, whereas AVP vs. PLA resulted in 7 of 18 (38.89%) null results demonstrating sensitivity.”
    Quote in the paperSubstantial claim
  • May combine two claims. The quote only partly states this claim.

    Findingown data

    In this randomized controlled trial, across all experimental tasks, main effects of all dependent variables were non-significant using NHST, with no effects of OT or AVP (vs. PLA) on empathic concern, social versus non-social working memory, deception detection, interpersonal distance, hypothetical bystander intervention, or reflections of interpersonal interactions.

    “Using NHST, no main effects were found for all analyses.”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Findingown data

    Using Holm correction, 72.22% of the equivalence tests were significant, a rate nearly identical to the 69.44% of tests above BF=3 using the r=0.707 default prior in JASP.

    “For instance, using Holm correction instead, another of the more conservative correction methods, shows 72.22% of the equivalence test to be significant, a rate nearly identical to the 69.44% of tests above BF=3 using the r=0.707 default prior in JASP.”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Findingown data

    The present study's average smallest detectable effect size (d=0.67) is similar to that reported by Quintana (2018) for the Lane et al. (2015) study set (d=0.75), but the present data had a lower proportion of insensitive tests (52.78% vs 73.5%).

    “Our equivalence testing results are similar to those reported by Quintana (2018) for the Lane et al. (2015) study set when comparing the average smallest detectable effect size (d=0.67 in the present study and d=0.75 for Lane et al. (2015), but our data did have a lower proportion of tests deemed to be insensitive (i.e. mean differences were not significantly different from zero but statistically”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites Quintana (2018)

      Not checked in the free run.

    • cites Lane et al. (2015) source

      Not checked in the free run.

  • May combine two claims.

    Findingown data

    While 47-83% of the analyses had adequate sensitivity to detect statistical equivalence, 17-53% lacked the sensitivity necessary to have confidence in the findings.

    “Indeed, while 47–83% of our analyses had adequate sensitivity to detect statistical equivalence, 17–53% of our analyses lacked the sensitivity necessary to have confidence in our findings.”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Findingown data

    Results provide evidence for a lack of main effects of intranasal AVP on certain social processes, but these results had less sensitivity relative to OT.

    “In addition, results also provide evidence for a lack of main effects of intranasal AVP (for which there is considerably less published evidence) on certain social processes; however, our analyses determined that these results had less sensitivity relative to OT.”
    Quote in the paperSubstantial claim
  • Classified as probably not a substantial claim.

    Study design and conductcitation

    To achieve >80% power, the study sought to recruit more than 34 participants per condition, based on previous studies showing moderate to large main effects of AVP (e.g., d=0.7) on social cognitive processes.

    “to achieve > 80% power, we sought to recruit more than 34 participants in each condition (i.e., OT, AVP, and PLA) based on previous studies showing moderate to large main effects of AVP (e.g., d=0.7) on social cognitive processes (Uzefovsky et al., 2012).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Uzefovsky et al., 2012

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Study design and conductown data

    Participants were randomly assigned to receive intranasal OT, AVP, or PLA.

    “participants included 125 undergraduate students from the University of California, Los Angeles (90 female; 35 male, age range=18–31 years, Mean age=20.88, SD=2.71) who were randomly assigned to receive intranasal OT (n=42; 30 female, 12 male), AVP (n=42; 30 female, 12 male), or PLA (n=41; 30 female, 11 male).”
    • Also stated in: § Procedure · plain
      “a third-party research coordinator unrelated to the study used an online random number generator (www.random.org) to randomly assign participants to the OT, AVP, or PLA condition (blocked on gender) and communicated this information to the UCLA pharmacy.”
    Quote in the paperSubstantial claim
  • Classified as probably not a substantial claim.

    Premisecitation

    None of the between-subjects studies in two meta-analyses were sufficiently powered to report the detected effects at 80% statistical power.

    “the authors of two of these meta-analyses (Leppanen et al., 2017; 2018) point out that none of the studies using a between-subjects design were sufficiently powered to report the detected effects at 80% statistical power.”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites Leppanen et al., 2017 source

      Not checked in the free run.

    • cites Leppanen et al., 2018 source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Study design and conductown data

    The smallest effect size detectable with 80% power was calculated for each study using the pwr package in R.

    “The smallest effect size detectable with 80% power was calculated using the pwr package in R for each study as shown in Table 1.”
    Quote in the paperSubstantial claim
  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data

    State positive and negative affect and state anxiety did not differ pre- and post-administration.

    “State positive and negative affect, along with state anxiety was measured and did not differ pre and post-administration (see Supplemental Materials for further details).”
    Quote in the paperNot a substantial claim
  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    All tasks were presented in randomized order to minimize potential order effects.

    “All tasks were presented in randomized order to minimize potential order effects.”
    Quote in the paperNot a substantial claim
  • Classified as probably not a substantial claim.

    Premisecitation

    Intranasal oxytocin was initially assumed to have wide-ranging effects on human social processes.

    “Indeed, these neuropeptides, and in particular OT, have generated tremendous enthusiasm for their potential translational influence on human social cognition and behavior (Insel, 2016).”
    • Also stated in: § Abstract · plain
      “years of research highlighting wide-ranging effects of, in particular, intranasal oxytocin administration have been tempered as the fields of psychology, neuroscience, and other disciplines have been addressing concerns over the reproducibility and validity of research findings.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Insel, 2016 source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Premisecitation

    Quintana (2018) reanalyzed Lane et al.'s data and found that 73.5% of the tests did not result from statistical equivalence between tested conditions, but rather from insensitivity due to lack of power.

    “Quintana (2018) reanalyzed their data and applied equivalence testing which showed that 73.5% of the tests did not result from statistical equivalence between tested conditions, but rather from insensitivity due to lack of power.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Quintana (2018)

      Not checked in the free run.

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductown data

    Allocation was concealed: a third-party coordinator assigned participants and a UCLA pharmacist prepared the drug or placebo with no indication on the label.

    “a third-party research coordinator unrelated to the study used an online random number generator (www.random.org) to randomly assign participants to the OT, AVP, or PLA condition (blocked on gender) and communicated this information to the UCLA pharmacy.”
    • Also stated in: § Procedure · plain
      “A UCLA pharmacist prepared the drug or PLA for each participant with no indication on the label as to its contents (to maintain the blind).”
    Quote in the paperSubstantial claim
  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given

    Both NHST and equivalence tests were corrected for multiple comparisons across the set of 18 unique variables in separate analyses for OT vs. PLA and AVP vs. PLA.

    “Both NHST and equivalence tests were corrected for multiple comparisons across the set of 18 unique variables in separate analyses for OT vs. PLA and AVP vs. PLA.”
    Quote in the paperSubstantial claim
  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    Study personnel and research nurses were blind to the drug condition.

    “Study personnel and research nurses were blind to the drug condition (there were participants in each condition in each group session).”
    Quote in the paperNot a substantial claim
  • May combine two claims.

    Premisecitation

    Recent meta-analyses in healthy samples have found a positive effect of OT on recognition of basic emotions, increased expression of positive emotions, and increased physiological startle responses to threat.

    “Recent meta-analyses in healthy samples have found a positive effect of OT on the recognition of basic emotions (Hedge’s g = 13 [0.02, 0.24]), increased expression of positive emotions (Hedge’s g = 0.25 [0.04. 0.47]; Leppanen et al., 2017), and increased physiological startle responses to threat (Hedge’s g = 0.3 [0.07. 0.53]; Leppanen et al., 2018).”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites Leppanen et al., 2017 source

      Not checked in the free run.

    • cites Leppanen et al., 2018 source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Recent meta-analyses have shown no meta-analytic effects of OT on theory of mind, expression of negative emotions, and attentional or behavioral responses toward threat in healthy samples.

    “Nonetheless, recent meta-analyses have shown no meta-analytic effects of OT on a variety of social outcomes in healthy samples including: theory of mind (Leppanen et al., 2017), the expression of negative emotions (Leppanen et al., 2017), and attentional or behavioral responses toward threat (Leppanen et al., 2018).”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites Leppanen et al., 2017 source

      Not checked in the free run.

    • cites Leppanen et al., 2018 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The present results reflect varied timing of the tasks relative to the doses and the specific incubation period selected.

    “In addition, the present results reflect varied timing of the tasks relative to the doses and the specific incubation period selected.”
    Quote in the paperNot a substantial claim
  • Classified as probably not a substantial claim.

    Interpretationreasoning

    Many published studies finding main effects of either drug may actually lack the sensitivity to detect the effect.

    “Conversely, many published studies finding main effects of either drug may actually lack the sensitivity to detect the effect.”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Study design and conductnone given

    The study had limitations including a relatively small sample size, a predominantly female sample (approximately 3:1), a healthy non-clinical sample, and the use of only one dose of OT and AVP.

    “Limitations include a relatively small sample size, a predominantly female sample (a ratio of approximately 3:1), the inclusion of a healthy (non-clinical) sample, and the use of only one dose of OT and AVP.”
    Quote in the paperNot a substantial claim
  • May combine two claims.

    Interpretationreasoning

    The 69.44% rate may be the high end of what is reasonable, while the Bonferroni corrected equivalence tests figure of 47.22% may represent the conservative end of the number of tests sensitive enough to reveal statistical equivalence.

    “In that context, the 69.44% rate may be the high end of what is reasonable, while the Bonferroni corrected equivalence tests figure of 47.22% may represent the conservative end of the number of tests sensitive enough to reveal statistical equivalence.”
    Quote in the paperSubstantial claim
  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    The original preregistration listed four of the six tasks; the final two tasks (bystander intervention and written reflections) were late additions and were not added to the preregistration.

    “Last, it should be noted that our original preregistration for the study () listed four of the six tasks described in the present manuscript. The final two tasks (i.e., the bystander intervention task and the task assessing written reflections of a supportive interaction or a conflict with a close other) were late additions to the protocol and, unfortunately, were not added to the preregistration”
    Quote in the paperNot a substantial claim
  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductreasoning

    All tasks were conducted in a randomized order during the same session with the same participants, which helps minimize outside factors that could influence the results.

    “In addition, the fact that all tasks were conducted in a randomized order during the same session with the same participants helps to minimize outside factors that could influence the results.”
    Quote in the paperNot a substantial claim
  • May combine two claims.

    Interpretationown data

    The advantage of the BF metric as an indicator of the relative degree of evidence for a model means that the majority of reported results offered some support for the null hypothesis.

    “In either case, the advantage of the BF metric as an indicator of the relative degree of evidence for a model, rather than the discrete decision between statistical equivalence and data insensitivity obtained via p-values means that the majority of reported results offered some support for the null hypothesis.”
    Quote in the paperSubstantial claim
  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Post-hoc null results that are not preregistered may be subject to researcher bias (e.g., reverse p-hacking).

    “It should be noted that post-hoc null results that are not preregistered (as in the case of the present manuscript) may be subject to researcher bias (e.g. reverse p-hacking; see Chuard et al., 2019).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Chuard et al., 2019

      Not checked in the free run.

30 of 30
  • The paper relies on this without support.

    The true effect size in this area of research is likely smaller than d=0.67.

    “If the “true” effect is smaller than .67, which is likely for this area of research, then an equivalence test would not be sensitive enough to reject the presence of this effect.”
1 of 1

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Effect sizes of studies using intranasal oxytocin seem to be smaller than initially assumed, reflected in unsuccessful attempts to replicate earlier findings of oxytocin effects on trust and mind-reading in healthy populations.

    “Generally, effect sizes of studies using intranasal oxytocin seem to be smaller than initially assumed99, a finding that is also reflected in unsuccessful attempts to replicate earlier findings of oxytocin effects on trust100,101 and mind-reading in healthy populations102.”

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Reported effect size falls outside its confidence interval

    Minor

    The reported Hedges' g of 13 is outside its stated confidence interval of 0.02 to 0.24. The point estimate and interval cannot both be correct as printed.

1 of 1

Cites

Flagged first, then by citations

Cited by

Sorted by citations

Authors
Tabak, B. A., Teed, A. R., Castle, E., Dutcher, J. M., Meyer, M. L., Bryan, R., Irwin, M. R., Lieberman, M. D., & Eisenberger, N. I.
Affiliations
  • Southern Methodist University (US)
  • University of California, Los Angeles (US)
  • Carnegie Mellon University (US)
  • Dartmouth College (US)
  • Los Angeles Trade Technical College (US)
Corresponding author
Tabak, Benjamin A.
Venue
Psychoneuroendocrinology
Publisher
Elsevier BV
Published
2019-09
Volume
107
Pages
124-132
ISSN
0306-4530 (ISSN-L)
OpenAlex
W2940659074
Cited by
43
References
38
Citations by year
2026: 2 · 2025: 3 · 2024: 4 · 2023: 2 · 2022: 8 · 2021: 12 · 2020: 10 · 2019: 2
FWCI
10.05
Type
journal article
Language
en
Open access
Locations
  • Psychoneuroendocrinology (publishedVersion)
  • PubMed (publishedVersion)
  • PubMed Central (acceptedVersion)
  • eScholarship (California Digital Library) (submittedVersion)
Funders
  • University of California, Los Angeles
  • National Institutes of Health
Topics
  • Social Sciences › Psychology › Social Psychology › Neuroendocrine regulation and behavior
  • Life Sciences › Neuroscience › Sensory Systems › Olfactory and Sensory Function Studies
  • Social Sciences › Psychology › Social Psychology › Action Observation and Synchronization
Keywords
oxytocin, vasopressin, intranasal administration, social cognition, social behavior, negative results, equivalence testing, Bayesian hypothesis testing, reproducibility, neurodevelopmental disorders, mental illness, placebo effect
MeSH terms
Negative Results, Administration, Intranasal, Adolescent, Adult, Bayes Theorem, Cognition, Double-Blind Method, Female, Humans, Male, Oxytocin, Social Behavior, Vasopressins, Reproducibility of Results, Placebo Effect, Young Adult, Social Skills

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-09.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-09.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review