CoChat
Sign Up
Cited by 40Natural Language Processing TechniquesDiscourse Analysis in Language StudiesTopic Modeling

Detecting Causal Language Use in Science Findings

Open access
Yu, Bei, Li, Yingya, Wang, Jun·2019·Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)
DOI 10.18653/v1/d19-1473
Open this paper in CoChat — free accountOpen PDF

Abstract

Bei Yu, Yingya Li, Jun Wang. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.

AI snapshot

Written from the paper’s full text.

TL;DR

The paper builds a BERT/BioBERT classifier to detect causal language in biomedical research conclusions and applies it to about 38,000 PubMed observational studies. It finds that direct causal language is common and varies by authors’ country and language background, challenging a shared global consensus on causal language use.

Key findings

  • BioBERT performed best among four models, achieving 0.901 accuracy and 0.881 macro-F1 on the annotated corpus; BERT followed with 0.889 accuracy and 0.874 macro-F1.
  • In 28,217 English observational-study conclusions, 21.7% used direct causal language exclusively and 32.4% used some direct causal language; 53.7% used only correlational language and 8.1% only conditional causal language.
  • Causal language ratios differed by country and language: Germany had the highest country ratio of causal language use (42.1% causal, 29.5% causal only), while the US had the lowest (27.7% causal, 17.4% causal only).
  • No obvious increase in causal language use was observed from 2013 to 2019, unlike prior findings of an increase.

Methods

  • The authors annotated 3,126 PubMed conclusion sentences, with 3,061 single-class sentences, using four relation categories: no relationship, correlational, conditional causal, and direct causal; inter-coder Cohen’s Kappa was 0.98.
  • They trained and evaluated Linear SVM, BiRNN, BERT, and BioBERT via 5-fold stratified cross-validation; BioBERT used the same BERT code with a BioBERT pretrained model.
  • They applied the best model to 37,746 observational studies with structured abstracts and conclusion subsections of four or fewer sentences, totaling 72,565 conclusion sentences, then analyzed ratios by relation type, year, language, and author country.

Limitations

  • The paper states that PubMed observational-study labels were introduced in 2014, so the dataset covers only seven years with partial data from 2013 and 2019; it also notes that a larger dataset might help build a stronger model.
  • The paper states that not all language and country results are comparable because some languages are used in multiple countries and some countries did not publish many papers.
  • The text makes evident additional constraints: only structured abstracts with conclusions of four or fewer sentences were used, studies with no mentioned relationship were excluded from further analysis, and error analysis showed confusion from causal markers in no-relationship sentences and less common noun-based causal markers.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.18653/v1/d19-1473

Cites

59 of 59 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 30 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    PMC embedding did better than GloVe.

    “We found that the PMC embedding did better than GloVe, and their combination performed the best.”
    Quote in the paperNot a substantial claim
  • Findingown data

    The conditional causal category only covers 7% of the data in the training set.

    “BioBERT achieved 0.881 macro-averaged F1 score with fairly balanced performance on each category except the conditional causal one, which only covers 7% data in the training set.”
    Quote in the paperNot a substantial claim
  • Findingown data

    BioBERT's performance was fairly balanced on each category except the conditional causal one.

    “BioBERT achieved 0.881 macro-averaged F1 score with fairly balanced performance on each category except the conditional causal one, which only covers 7% data in the training set.”
    Quote in the paperNot a substantial claim
  • Findingown data

    The most common disagreement between machine prediction and human annotation is between the categories of direct causal and no relationship.

    “Error analysis of misclassified cases shows the most common disagreement between machine prediction and human annotation is between the categories of direct causal and no relationship, even though the two categories are not semanti- cally close in our dataset.”
    Quote in the paperNot a substantial claim
  • Findingown data

    The combination of PMC embedding and GloVe performed best.

    “We found that the PMC embedding did better than GloVe, and their combination performed the best.”
    Quote in the paperSubstantial claim
  • The quote only partly states this claim. 3
  • Classified as probably not a substantial claim. 17
  • May combine two claims. 7

  • The quote only partly states this claim.

    Findingown data

    BioBERT achieved a macro-F1 of 0.88.

    “achieving an accuracy of 0.90 and a macro-F1 of 0.88.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Of the corpus, 65 sentences had more than one type of relations.

    “The corpus contains 3,126 annotated sentences, of which 3,061 sentences contained only one type of relations, and the remaining 65 sentences had more than one type of relations.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The conclusion subsection of a structured PubMed abstract represents the research conclusions or findings of the study.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    The two categories direct causal and no relationship are not semantically close in the dataset.

    “even though the two categories are not semanti- cally close in our dataset.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The corpus contains 3,126 annotated sentences.

    “The corpus contains 3,126 annotated sentences, of which 3,061 sentences contained only one type of relations, and the remaining 65 sentences had more than one type of relations.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The conclusion subsection of a structured PubMed abstract represents the research conclusions or findings of the study.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    Performance on count and tf-idf vectorizations were similar.

    “Performance on count and tf-idf vectorizations were similar.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Adding tri-grams did not improve model performance.

    “Adding tri-grams did not improve model performance.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given

    Cased BERT-Base performs better than Uncased BERT-Base.

    “We used the Pytorch version of BERT with its pre-trained model (Cased BERT-Base, which performs better than the Uncased one)”
    Quote in the paperNot a substantial claim

  • The quote only partly states this claim.

    Findingown data

    BioBERT achieved an accuracy of 0.90.

    “achieving an accuracy of 0.90 and a macro-F1 of 0.88.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Of the corpus, 3,061 sentences contained only one type of relations.

    “The corpus contains 3,126 annotated sentences, of which 3,061 sentences contained only one type of relations, and the remaining 65 sentences had more than one type of relations.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The conclusion subsection of a structured PubMed abstract represents the research conclusions or findings of the study.

  • May combine two claims.

    Findingown data

    BioBERT performed the best by all measures, followed by BERT, BiRNN and then LinearSVM.

    “The result in Table 4 shows that BioBERT performed the best by all measures, followed by BERT, BiRNN and then LinearSVM.”
    Quote in the paperSubstantial claim

  • The quote only partly states this claim.

    Findingown data

    Among the 28,217 English publications, 83.5% contained only one type of relation in the conclusion.

    “It shows that 83.5% of studies contained only one type of relation in the conclusion”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: PubMed labels correctly identify observational studies.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    Most of these studies contain only one sentence in the conclusion subsection.

    “Most of these studies contain only one sentence in the conclusion subsection.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: PubMed labels correctly identify observational studies.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Only 33% of the publications consist of one conclusion sentence.

    “only 33% of the publications consist of one conclusion sentence.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: PubMed labels correctly identify observational studies.

  • May combine two claims.

    Premisecitation

    Manual approaches for extracting causal relations often require an exhaustive list of linguistic cues, which costs a significant amount of human effort and its generalizability may be limited by the small sample size.

    “However, such manual approaches often require an exhaustive list of linguistic cues, which costs a significant amount of human effort and its generalizability may be limited by the small sample size.”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites Garcia et al., 1997

      Not checked in the free run.

    • cites Cofield et al., 2010

      Not checked in the free run.

    • cites Lazarus et al., 2015

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Premisecitation

    Sumner et al. (2014) proposed a fine-grained categorization of causal relations with seven levels.

    “To capture the different levels of certainty in research findings, Sumner et al. (2014) proposed a more fine-grained categorization with seven levels: no mentioned relationship, statement of no relationship, statement of correlation, ambiguous statement of relationship, conditional statement of causation, statement of can, and statement of causation.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Sumner et al., 2014

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    There is a lack of consistent teaching of strategies for thinking about probability and causality.

    “There is also a lack of consistent teaching of strategies for thinking about probability and causality (Stanovich, 2010).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Stanovich, 2010

      Not checked in the free run.

  • May combine two claims.

    Study design and conductown data

    The overall Cohen's Kappa agreement for inter-coder reliability was 0.98, indicating near-perfect inter-coder agreement.

    “The overall Cohen's Kappa agreement (Cohen, 1960) was 0.98, indicating a near-perfect inter-coder agreement (McHugh, 2012).”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites Cohen, 1960

      Not checked in the free run.

    • cites McHugh, 2012

      Not checked in the free run.

    • Rests on an assumption: The conclusion subsection of a structured PubMed abstract represents the research conclusions or findings of the study.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Li et al. (2017) used SVM to develop a four-category classifier for causal relations in research findings with a 0.718 F1-score.

    “To this end, Li et al. (2017) adopted a simplified taxonomy from (Sumner et al., 2014) for causal relations in research findings, and used SVM to develop a fourcategory classifier with a 0.718 F1-score, suggesting room for further performance improvement.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Li et al., 2017

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Study design and conductown data

    The four learning approaches were evaluated on the annotated corpus via 5-fold stratified cross-validation.

    “To compare the performance of the above four learning approaches, we evaluated them on our annotated corpus via 5-fold stratified cross-validation.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    BioBERT has shown to outperform BERT on three representative biomedical text mining tasks.

    “BioBERT has shown to outperform BERT on three representative biomedical text mining tasks (Lee et al., 2019).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Lee et al., 2019

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    A number of linguistic studies have found that expressing and interpreting epistemic modality is not an easy task for learners of English as a second language.

    “For example, a number of linguistic studies have found that expressing and interpreting epistemic modality is not an easy task for learners of English as a second language (Holmes, 1982).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Holmes, 1982

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Observational studies (non-intervention) are designed for testing association/correlation between variables, while intervention studies, such as clinical trials, are for testing causal relations.

    “In scientific research, observational studies (non-intervention) are designed for testing association/correlation between variables, while intervention studies, such as clinical trials, are for testing causal relations (Buhse et al., 2018).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Buhse et al., 2018

      Not checked in the free run.

  • May combine two claims.

    Premisereasoning

    Prior studies on identifying inappropriate use of causal language relied on manual content analysis, which is not scalable for examining a large volume of science publications.

    “Prior studies on identifying inappropriate use of causal language relied on manual content analysis, which is not scalable for examining a large volume of science publications.”
    • Also stated in: § 2.4 Methods for Identifying Causal Relations · hedged
      “However, such manual approaches often require an exhaustive list of linguistic cues, which costs a significant amount of human effort and its generalizability may be limited by the small sample size.”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites Cofield et al., 2010

      Not checked in the free run.

    • cites Lazarus et al., 2015

      Not checked in the free run.

    • cites Chiu et al., 2017

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Premisecitation

    Making valid causal inferences from observational data is a challenging and risky process.

    “Making valid causal inferences from observational data is a challenging and risky process (Hernán and Robins, 2006).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Hernán and Robins, 2006

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S4 (based).

    Premisecitation

    Prior studies in linguistics have found that German, French, and Dutch translations tend to use stronger claims than the original English science publications.

    “Prior studies in linguistics, as described in sections 1 and 2, have found that German, French, and Dutch translations tend to use stronger claims than the original English science publications.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductreasoning

    Not all results in Table 6 and Table 7 are comparable because some languages are used in multiple countries and some countries did not publish many papers.

    “Because some languages are used in multiple countries and some countries did not publish many papers, not all results in Table 6 and Table 7 are comparable.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductreasoning

    The PubMed labels of observational studies were introduced in 2014, so the dataset covers only seven years with partial data from 2013 and 2019.

    “One limitation of our study is that the PubMed labels of observational studies were introduced in 2014 (although some 2013 publications were an-notated too); therefore, our dataset covers only seven years with partial data from 2013 and 2019.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: PubMed labels correctly identify observational studies.
27 of 27

  • The paper relies on this without support.

    The conclusion subsection of a structured PubMed abstract represents the research conclusions or findings of the study.

    “For the convenience of locating research findings, we used the structured abstracts only.”

  • The paper relies on this without support.

    The five selected health topics are representative of health research for causal language use.

    “First, to account for the vocabulary variation among different health issues, five common health topics -nutrition, diabetes, obesity, breast cancer, and cholesterol-were selected.”

  • The paper relies on this without support.

    The ratio of causal language use in abstracts reflects the ratio in full-text research conclusions.

    “In this study we chose to analyze the abstracts instead of full-text articles because the PubMed abstracts are openly available to the public while many full-text articles are behind a paywall.”

  • The paper relies on this without support.

    PubMed labels correctly identify observational studies.

    “the PubMed labels of observational studies were introduced in 2014”

  • The authors state this as an assumption.

    The English abstracts of non-English publications reflect the authors' native language causal language use.

    “because these authors mainly used their native languages for science communication.”

  • The paper relies on this without support.

    PubMed affiliation metadata accurately identifies authors' countries.

    “For each publication, each author's country was extracted from the PubMed affiliation metadata.”

  • The paper relies on this without support.

    The four country-language pairs Germany-German, Italy-Italian, France-French, and China-Chinese correspond well to each other.

    “we were able to identify four country-language pairs that correspond well to each other: Germany-German, Italy-Italian, France-French, and China-Chinese.”

  • The paper relies on this without support.

    PubMed language metadata accurately identifies the language of publications.

    “we used the PubMed metadata”
8 of 8

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • These results align with other evidence of causal overclaiming in psychology and biomedical science.

    “which align with other evidence of causal overclaiming in psychology 51 and biomedical science 35”

    From Quantifying the prevalence and impact of overreaching causal claims in social science (2026) · cites this paper as [35]

  • Alternative approaches to identifying causal language (keyword searches and a causal language model from prior research) found even greater prevalence rates than the primary approach.

    “we conducted two alternative approaches to identifying causal language, including keyword searches and a causal language model developed in prior research 35 and found even greater prevalence rates”

    From Quantifying the prevalence and impact of overreaching causal claims in social science (2026) · cites this paper as [35]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Most single-relation studies cannot have one-sentence conclusions

    Minor

    The paper says 83.5% of the 28,217 English studies contain only one relation type, but only 33% of the larger set of 29,410 studies have one-sentence conclusions. Even if every one-sentence study belonged to that English single-relation group, fewer than half of the group could have a one-sentence conclusion.

1 of 1

Cites

Flagged first, then by citations

Research works printed but not linked: 5

  1. Isabelle Boutron and Philippe Ravaud. 2018. Misrep- resentation and distortion of research in biomedi- cal literature. Proceedings of the National Academy of Sciences of the United States of America, 115(11):2613-2619.
  2. Kellia Chiu, Quinn Grundy, and Lisa Bero. 2017. spinin published biomedical literature: A method- ological systematic review. PLoS biology, 15(9):e2002173.
  3. Miguel A Hernán and James M Robins. 2006. In- struments for causal inference: an epidemiologist's dream? Epidemiology, pages 360-372.
  4. Svenja Kranich. 2009. Epistemic modality in english popular scientific texts and their german transla- tions. Trans-kom, 2(1):26-41.
  5. Claudiu Mihȃilȃ and Sophia Ananiadou. 2013. Recog- nising discourse causality triggers in the biomedi- cal domain. Journal of bioinformatics and compu- tational biology, 11(06):1343008.

Cited by

Sorted by citations

Related

Authors
Yu, B., Li, Y., & Wang, J.
Affiliations
Syracuse University (US)
Corresponding author
Wang, Jun
ORCID
Yu, Bei
Venue
Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)
Publisher
Association for Computational Linguistics
Published
2019
Pages
4663-4673
DOI
10.18653/v1/d19-1473
OpenAlex
W2971182981
Semantic Scholar
df9319a0661951cff977b77931a9f923392b58a8
Cited by
40
References
58
Citations by year
2026: 4 · 2025: 6 · 2024: 7 · 2023: 5 · 2022: 4 · 2021: 9 · 2020: 4 · 2019: 1
FWCI
1.68
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
Funders
  • Nvidia
  • Syracuse University
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Social Sciences › Arts and Humanities › Literature and Literary Theory › Discourse Analysis in Language Studies
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
causal language, causal inference, scientific texts

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-09.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-09.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review