CoChat
Sign Up
Cited by 251Climate variability and modelsClimate Change and Environmental ImpactClimate Change Communication and Perception

Machine-learning-based evidence and attribution mapping of 100,000 climate impact studies

Open access
Callaghan, Max, Schleussner, Carl-Friedrich, Nath, Shruti, Lejeune, Quentin, Knutson, Thomas R., Reichstein, Markus, Hansen, Gerrit, Theokritoff, Emily +11 more·2021·Nature Climate Change
DOI 10.1038/s41558-021-01168-6
Open this paper in CoChat — free accountOpen PDF

Abstract

Increasing evidence suggests that climate change impacts are already observed around the world. Global environmental assessments face challenges to appraise the growing literature. Here we use the language model BERT to identify and classify studies on observed climate impacts, producing a comprehensive machine-learning-assisted evidence map. We estimate that 102,160 (64,958–164,274) publications document a broad range of observed impacts. By combining our spatially resolved database with grid-cell-level human-attributable changes in temperature and precipitation, we infer that attributable anthropogenic impacts may be occurring across 80% of the world’s land area, where 85% of the population reside. Our results reveal a substantial ‘attribution gap’ as robust levels of evidence for potentially attributable impacts are twice as prevalent in high-income than in low-income countries. While gaps remain on confidently attributabing climate impacts at the regional and sectoral level, this database illustrates the potential current impact of anthropogenic climate change across the globe. Evidence is growing on the impacts of climate change on human and natural systems. A two-step attribution approach—machine-learning-assisted literature review coupled with grid-cell-level temperature and precipitation—allows comprehensive mapping of the evidence on impacts and tentative attribution to anthropogenic influence.

AI snapshot

Written from the paper’s full text.

TL;DR

The paper uses BERT-based machine learning to map over 100,000 studies of observed climate impacts and combines them with grid-cell detection/attribution of temperature and precipitation trends. It finds potential attributable impacts across 80% of global land area and 85% of the population, while revealing a substantial attribution gap in evidence density between high- and low-income countries.

Key findings

  • The authors estimate 100,724 (62,950–162,838) publications document observed climate impacts; the search returned 601,677 unique documents, of which 102,160 (64,386–164,274) were estimated relevant.
  • Attributable temperature and/or precipitation trends cover 80% of global land area excluding Antarctica, hosting 85% of the world’s population; 48% of land area (74% of population) has robust or high evidence of impacts co-located with attributable trends.
  • An “attribution gap” is reported: 23% of low-income-country population lives in areas with low impact evidence despite at least partially attributable temperature/precipitation trends, versus 3% in high-income countries; ≥5 studies per grid cell with attributable impacts is 1.76 times as prevalent by population in high-income countries (88%) as low-income countries (50%), and ≥20 studies is more than 4 times as prevalent (81% vs 17%).
  • Classifier performance: binary inclusion average F1 0.71 and ROC AUC 0.92; impact-type prediction macro F1 0.84; climate-driver prediction F1 0.79.

Methods

  • Searched Web of Science and Scopus with a validated query, deduplicated to 601,677 records; human reviewers screened 2,373 documents and included/coded 1,125, plus 351 documents extracted from IPCC AR5 WGII Tables 18.5–18.9; 732 randomly drawn documents were used for validation.
  • Fine-tuned DistilBERT classifiers to predict relevance, impact category, and climate driver from titles/abstracts, using nested cross-validation and bootstrapping for uncertainty; compared against SVM.
  • Extracted geographic entities with the Mordecai geoparser, matched studies to 2.5° grid cells, and combined them with grid-cell detection/attribution of 1951–2018 temperature and 1951–2016 precipitation trends using HadCRUT4, GPCC, and 10 CMIP6 models.

Limitations

  • Stated: the database is “vast, but neither complete nor perfect”; the query excludes some relevant studies, machine-learning classification adds compounding uncertainties, coding used abstracts only, no critical appraisal of study quality was performed, and the two-step attribution applies only to the subset of studies driven by temperature or precipitation.
  • Stated: the automated two-step attribution cannot verify that every cited temperature or precipitation trend matches the sign, magnitude, or time period of trends attributed to human influence, a greater problem for precipitation; cascading causal chains are not covered except where studies address each part.
  • Evident from the text: the approach is explicitly “no substitute for careful assessment by experts,” and the authors note possible expansion to non-English evidence, indicating language coverage as a limitation.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1038/s41558-021-01168-6

Cites

55 of 55 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 33 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    Robust levels of evidence for attributable impacts is twice as prevalent in high income than low income countries.

    “Our results reveal a substantial 'attribution gap' as robust levels of evidence for attributable impacts is twice as prevalent in high income than low income countries.”
    Quote in the paperSubstantial claim
  • Findingown data

    The paper estimates that 100,724 (62,950-162,838) publications document a broad range of observed impacts.

    “We estimate that 100,724 (62,950-162,838) publications document a broad range of observed impacts.”
    Quote in the paperSubstantial claim
  • Findingown data

    The prediction of climate driver is achieved with an average F1 score of 0.79.

    “The prediction of impact type is achieved with an average macro F1 score of 0.84 while the prediction of climate driver is achieved with an average F1 score of 0.79”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The training data of 2,629 abstracts is representative of the broader literature on climate impacts.
  • Findingown data

    The prediction of impact type is achieved with an average macro F1 score of 0.84.

    “The prediction of impact type is achieved with an average macro F1 score of 0.84 while the prediction of climate driver is achieved with an average F1 score of 0.79”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The training data of 2,629 abstracts is representative of the broader literature on climate impacts.
  • Findingown data

    46,442 (34,473-87,861) articles have been published in the sixth assessment cycle so far.

    “46,442 (34,473-87,861) articles have been published in the sixth assessment cycle so far; this represents more than twice the number of studies published during the AR5 period.”
    Quote in the paperNot a substantial claim
  • May combine two claims. 33
  • Classified as probably not a substantial claim. 29
  • The quote only partly states this claim. 1

  • May combine two claims.

    Findingown data

    Although the number of relevant studies in North America, Asia, and Europe is much higher than in South America, Africa, and Oceania, there is a large body of relevant studies available on all continents.

    “Although the number of relevant studies in North America, Asia, and Europe is much higher than in South America, Africa, and Oceania, there is a large body of relevant studies available on all continents (fig 1.c).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Adjusted for population, the number of papers focusing on Oceania far exceeds the size of the literature devoted to other continents, with Africa and Asia receiving the least attention per million inhabitants.

    “Adjusted for population (Supplementary Fig. 1), the number of papers focusing on Oceania far exceeds the size of the literature devoted to other continents, with Africa and Asia receiving the least attention per million inhabitants.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingreasoning

    This mismatch problem is greater for studies driven by precipitation, where both wetting and drying trends occur with greater temporal variation, though these make up the minority of partially attributed studies and grid cells.

    “This is a greater problem for studies driven by precipitation, where both wetting and drying trends occur with greater temporal variation, though these make up the minority of partially attributed studies and grid cells.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The binary inclusion classifier achieves an average F1 score of 0.71 and ROC AUC score of 0.92.

    “we employ a nested cross-validation approach to hyperparameter tuning, model selection and classifier evaluation, and find that our binary inclusion classifier achieves an average F1 score of 0.71, and ROC AUC score of 0.92.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The training data of 2,629 abstracts is representative of the broader literature on climate impacts.

  • May combine two claims.

    Findingown data

    The majority of land grid cells show attributable warming trends, with exceptions where trends cannot be robustly distinguished from internal variability or where there is insufficient data to establish trends.

    “The majority of land grid cells show attributable warming trends, with exceptions where trends cannot be robustly distinguished from internal variability (white cells, category 0) or where there is insufficient data to establish trends (grey cells).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    For example, many studies refer to the impacts of temperature in the state of Western Australia, but of the 40 grid cells in the state, an attributable temperature trend can be demonstrated for 22 cells.

    “For example, many studies refer to the impacts of temperature in the state of Western Australia, but of the 40 grid cells in the state, an attributable temperature trend can be demonstrated for 22 cells.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    These constitute almost all grid cells in Western Europe, North America, South and East Asia, and there are parts of all continents which have similar pockets of substantial preliminary evidence.

    “These constitute almost all grid cells in Western Europe, North America, South and East Asia, and there are parts of all continents which have similar pockets of substantial preliminary evidence.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Parts of Western Africa, South-east, Western and Northern Asia contain several light red grid cells where there is evidence to suggest that the climate (temperature and/or precipitation) has changed because of human influence, but there is little evidence on how this may be impacting human and natural systems.

    “Parts of Western Africa, South-east, Western and Northern Asia contain several light red grid cells where there is evidence to suggest that the climate (temperature and/or precipitation) has changed because of human influence, but there is little evidence on how this may be impacting human and natural systems.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The lightest grey cells (17% of land area) describe areas where we do not detect anthropogenic influence on regional temperature or precipitation and find few publications about the impacts of temperature or precipitation on human and natural systems in those areas.

    “The lightest grey cells (17% of land area) describe areas where we do not detect anthropogenic influence on regional temperature or precipitation and find few publications about the impacts of temperature or precipitation on human and natural systems in those areas.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The number of weighted studies per grid cell is a valid proxy for the level of evidence of impacts on human and natural systems.

  • May combine two claims.

    Findingown data

    Research on human and managed systems makes up 12% of all research globally, but only 10% of research in Europe, compared to 19% in Africa.

    “For example, research on human and managed systems makes up 12% of all research globally, but only 10% of research in Europe, compared to 19% in Africa.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The number of weighted studies per grid cell is a valid proxy for the level of evidence of impacts on human and natural systems.

  • May combine two claims.

    Findingown data

    For 33% of global land area (hosting 11% of global population), although there is evidence that long-term trends in precipitation and temperature are attributable at least in part to human influence, there is relatively little evidence in the existing literature about how these trends impact human and natural systems.

    “However, for 33% of global land area (hosting 11% of global population), although there is evidence that long-term trends in precipitation and temperature are attributable at least in part to human influence, there is relatively little evidence in the existing literature about how these trends impact human and natural systems (Fig. 2c lightest purple shading).”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The number of weighted studies per grid cell is a valid proxy for the level of evidence of impacts on human and natural systems.

  • May combine two claims.

    Findingown data

    For 16 of the remaining cells a lack of data means that a detectable trend cannot be established, and for the remaining 2 cells, no attributable trend can be established.

    “For 16 of the remaining cells a lack of data means that a detectable trend cannot be established, and for the remaining 2 cells, no attributable trend can be established.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation

    The implicit inclusion and coding decisions for 351 documents were extracted from Tables 18.5-18.9 in IPCC AR5 WGII.

    “The implicit inclusion and coding decisions for a further 351 documents were extracted from Tables 18.5-18.9 in the contribution of Working Group II to the Fifth Assessment Report of the IPCC 1 .”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [1] Cramer, W. et al. in Climate Change 2014: Impacts, Adaptation, and Vulnerability (eds Field, C. B. et al.) 979–1037 (Cambridge Univ. Press, 2014).

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Over 600,000 documents were retrieved from bibliographic databases using a query.

    “Over 600,000 documents were retrieved from bibliographic databases using a query.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    For impact categories, DistilBERT outperforms SVM, achieving a macro-averaged F1 score of 0.84 and a macro-averaged ROC AUC score of 0.95.

    “For the prediction of impact categories, DistilBERT outperforms SVM, achieving a macro-averaged F1 score of 0.84 and a macro-averaged ROC AUC score of 0.95 (Extended Fig. 4.).”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Documents selected using active learning are broadly representative of the documents to which the multilabel classifiers are applied.

  • May combine two claims.

    Findingown data

    DistilBERT clearly outperformed SVM across all metrics, achieving an average F1 score of 0.71, and an average ROC AUC score of 0.92.

    “DistilBERT clearly outperformed SVM across all metrics, achieving an average F1 score of 0.71, and an average ROC AUC score of 0.92 (Extended Fig. 3).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    Web of Science returned 411,194 documents, while Scopus returned 476,778 documents.

    “Web of Science returned 411,194 documents, while Scopus returned 476,778 documents.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    A total of 2,373 documents were screened, of which 1,125 were included as relevant and coded for impacts and drivers.

    “A total of 2,373 documents were screened by members of the author team using the NACSOS platform 48 , of which 1,125 were included as relevant and coded for impacts and drivers.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [48] Callaghan, M., Müller-Hansen, F., Hilaire, J. & Lee, Y. T. NACSOS: NLP assisted classification, synthesis and online screening. Zenodo https://doi.org/10.5281/zenodo.4121526 (2020). source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    2,373 documents were screened for relevance and coded for impact type and driver by human reviewers.

    “2,373 of these documents were screened for relevance and coded for impact type and driver by human reviewers.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    11 studies refer to impacts predicted to be driven by temperature trends in Sudan, where Sudan is the smallest geographical entity mentioned.

    “Extended Figures 9c. and d. show 11 studies which refer to impacts predicted to be driven by temperature trends in Sudan, where Sudan is the smallest geographical entity mentioned.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Unadjusted results (without variability adjustment) are broadly similar to adjusted results but show more category -4 (unexplained trends of incorrect sign) over Africa.

    “Results without this variability adjustment (not shown) are broadly similar but show more category -4 (unexplained trends of incorrect sign) over Africa, where internal low-frequency variability appears to be underestimated in models according to this analysis;”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). The quote only partly states this claim.

    Findingown data

    The global density map of climate impact evidence includes N=77,785 weighted studies per grid cell for all evidence on climate impacts.

    “Map colouring denotes the number of weighted studies per grid cell for all evidence on climate impacts (N=77,785).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    These effects are emerging in a range of different systems and at different scales, covering a broad range of research fields from glaciology to agricultural science, and marine biology to migration and conflict research.

    “These effects are emerging in a range of different systems and at different scales, covering a broad range of research fields from glaciology to agricultural science, and marine biology to migration and conflict research 2 .”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [2] IPCC Climate Change 2014: Impacts, Adaptation, and Vulnerability (eds Field, C. B. et al.) (Cambridge Univ. Press, 2014).

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    The authors establish a new paradigm for assessing the impacts of climate change across human and natural systems.

    “In doing so, we establish a new paradigm for assessing the impacts of climate change across human and natural systems.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductown data

    The search used two large bibliographic databases (Web of Science and Scopus) with an inclusive and transparent search method to systematically identify the literature on climate impacts.

    “We searched two large bibliographic databases (Web of Science and Scopus) using an inclusive and transparent search method to systematically identify the literature on climate impacts.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    The spatially resolved approach allows for a systematic provision of regional to local, sector-specific climate impact information to local or regional experts and adaptation practitioners.

    “Our spatially resolved approach allows for a systematic provision of regional to local, sector-specific climate impact information to local or regional experts and adaptation practitioners.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    These evidence gaps constitute significant blind spots in understanding of climate impacts, and in some cases understanding of attributable anthropogenic influence on regional precipitation and/or temperature.

    “These evidence gaps constitute significant blind spots in understanding of climate impacts, and in some cases understanding of attributable anthropogenic influence on regional precipitation and/or temperature.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    Some of the spatial features can be explained by the geographical characteristics.

    “Some of the spatial features can be explained by the geographical characteristics.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    Using machine learning to scale up evidence synthesis allows mapping 100,000 studies of climate impacts, providing a comprehensive picture of the evidence base.

    “Using machine learning to scale up evidence synthesis allows us to map 100,000 studies of climate impacts, providing a comprehensive picture of the evidence base.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

    Interpretationreasoning

    This offers perspectives for a novel climate service supporting the uptake of scientific information in local contexts and providing relevant information for adaptation action.

    “This offers perspectives for a novel climate service supporting the uptake of scientific information in local contexts and providing relevant information for adaptation action.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

    Interpretationreasoning

    Bringing together these two lines of evidence on climate change and climate impacts provides a new bridge between the climate science community and the impacts, adaptation, and vulnerabilities communities, and highlights the synergistic nature of their approaches.

    “Bringing together these two lines of evidence on climate change and climate impacts provides a new bridge between the climate science community and the impacts, adaptation, and vulnerabilities communities, and highlights the synergistic nature of their approaches.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    This does not rule out the possibility that some trends in precipitation or temperature have occurred in these regions that have been driven, at least in part, by human influence on the climate.

    “This does not rule out the possibility that some trends in precipitation or temperature have occurred in these regions that have been driven, at least in part, by human influence on the climate.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    The automated nature of the assessment allows for continuous updating of the database, creating a 'living' evidence map.

    “Furthermore, the automated nature of the assessment allows for continuous updating of the database, creating a 'living' evidence map that can also be improved and extended by incorporating additional sources of relevant publications (e.g.non-English speaking evidence, or”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    Wide confidence intervals here reflect the compound uncertainty deriving from classification of relevance, impact and driver.

    “Wide confidence intervals here reflect the compound uncertainty deriving from classification of relevance, impact and driver.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    The query was validated against a set of records known to be relevant.

    “To validate the query, we tested this against a set of records known to be relevant.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given

    Machine learning classifiers were trained to predict relevance of documents using titles and abstracts, and evaluated using nested cross-validation.

    “Machine learning classifiers were trained to predict relevance of documents using the titles and abstracts, and evaluated using nested cross-validation.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The two-step attribution process is only applied for the subset of papers which provide evidence on impacts driven by temperature and precipitation.

    “The two-step attribution process is also only applied for the subset of papers which provide evidence on impacts driven by temperature and precipitation.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductown data

    For each model, the potential effect of internal variability was assessed by computing trends of the length being investigated in 50 random samples of the corresponding piControl runs.

    “For each model, we then assessed the potential effect of internal variability by computing trends of the length being investigated in 50 random samples of the corresponding piControl runs from each model.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductreasoning

    Because documents selected using active learning are unlikely to be representative of the full set, 732 documents drawn at random were screened for validation.

    “Because the documents selected using the methods described above are unlikely to be representative of the full set of documents returned by the query, we also screened 732 documents drawn at random which we used for validation.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    The query was built to return all references in the tables that specifically referred to the role of climate change.

    “After extracting these references, we built a query that would return all of the references in the tables that specifically referred to the role of climate change (rather than of counterfactual explanations for impacts).”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    The impact area studied and documented driver are expected to be covered in the abstract.

    “Given the relative simplicity of the type of information we extract (focusing on the impact area studied and the documented driver), we expect them to be covered in the abstract, which provides the condensed summary of the study's findings.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    Our detection results for precipitation will tend to be conservative, while the ability of All-Forcing to be consistent with observations will tend to be liberal, because the modeled spread is relatively wide.

    “So our detection results for precipitation will tend to be conservative, while conversely, the ability of All-Forcing to be consistent with observations will tend to be liberal, because the modeled spread is relatively wide.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductcitation

    The analysis provides a test of the ability of the corresponding ALL simulations to reproduce the regional trends in annual mean temperature and precipitation against observational data.

    “The analysis provides a test of the ability of the corresponding ALL simulations to reproduce the regional trends in annual mean temperature and precipitation against observational data 56 .”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [56] Eyring, V. et al. Overview of the Coupled Model Intercomparison Project Phase 6 (CMIP6) experimental design and organization. Geosci. Model Dev. 9, 1937–1958 (2016). source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Premisecitation

    Identifying a robust climate change driver in systems subject to other anthropogenic interference requires careful assessment of other socio-economic factors.

    “In the case of systems subject to other anthropogenic interference such as the global biosphere, managed systems such as agriculture, or human systems themselves, identifying a robust climate change driver requires careful assessment of other socio-economic factors 41,42 , adding additional levels of complexity 43 .”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [41] Shepherd, T. G. Storyline approach to the construction of regional climate change information. Proc. R. Soc. A 475, 20190013 (2019). source

      Not checked in the free run.

    • cites [42] Rosenzweig, C. & Neofotis, P. Detection and attribution of anthropogenic climate change impacts. Wiley Interdiscip. Rev. Clim. Change 4, 121–150 (2013). source

      Not checked in the free run.

    • cites [43] Mengel, M., Treu, S., Lange, S. & Frieler, K. ATTRICI 1.1—counterfactual climate for impact attribution. Geosci. Model Dev. https://doi.org/10.5194/gmd-14-5269-2021 (2021). source

      Not checked in the free run.

  • May combine two claims.

    Interpretationreasoning

    The database will incorrectly exclude some relevant documents and contain some documents that have been incorrectly included or incorrectly coded.

    “The database we assemble will also incorrectly exclude some relevant documents and contain some documents that have been incorrectly included or incorrectly coded”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductown data

    Observed trends in precipitation and temperature were collected for 2.5 and 5 degree grid cells and compared with climate models to assess whether observed trends were detectable and at least partially attributable to human influence.

    “Observed trends in precipitation and temperature were collected for 2.5 and 5 degree grid cells and compared with climate models to assess whether observed trends were detectable (i.e., unusual compared with natural variability, and in the same direction as simulated by historical forcing climate model simulations) and at least partially attributable to human influence on the climate, as discussed”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    The best performing classifier was fit with all labelled documents using bootstrapping to make predictions with confidence intervals for the relevance of the remaining documents.

    “The best performing classifier was then fit with all labelled documents using bootstrapping to make predictions with confidence intervals for the relevance of the remaining documents.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    Applying classifiers to noisy full texts would greatly increase the risks of false positives.

    “Applying classifiers to noisy full texts which contain contextual information and related research as well as the results and topic of a study would greatly increase the risks of false positives.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductown data

    Nested cross-validation was used to optimize parameter settings and evaluate performance of SVM and DistilBERT.

    “We use a nested cross-validation procedure (Extended Fig. 2) to optimize parameter settings and evaluate the performance of a support vector machine (SVM) classifier 50 as well as a pre-trained DistilBERT model fine-tuned with our labelled dataset 28 .”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [50] Callaghan, M. Machine learning-based evidence and attribution mapping of 100,000 climate impact studies - code. Zenodo https://doi.org/10.5281/ZENODO.5327409 (2021). source

      Not checked in the free run.

    • cites [28] Sanh, V., Debut, L., Chaumond, J. & Wolf, T. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. Preprint at https://arxiv.org/abs/1910.01108 (2020).

      Not checked in the free run.

  • May combine two claims.

    Study design and conductreasoning

    Documents were coded only at the abstract level, and only the abstracts were used as inputs to classifiers.

    “Further, documents were coded only at the abstract level, and only the abstracts were used as inputs to our classifiers.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given

    Multilabel classifiers were trained using the remaining labelled relevant documents, and assessed using cross-validation.

    “Multilabel classifiers were then trained using the remaining labelled relevant documents, and assessed in a similar fashion using cross-validation.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Documents selected using active learning are broadly representative of the documents to which the multilabel classifiers are applied.

  • May combine two claims.

    Interpretationreasoning

    The selection and categorisation of studies was achieved using machine learning, meaning results are subject to additional uncertainties, which compound for each level of classification.

    “The selection and categorisation of studies was achieved using machine learning, meaning that results are subject to additional uncertainties, which compound for each level of classification.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    With the termination of the 'global warming hiatus' around 2014, the additional recent years have been adding to an ongoing strengthening warming signal and leading to even greater assessed area with detectable anthropogenic warming.

    “With the termination of the 'global warming hiatus' around 2014, the additional recent years have been adding to an ongoing strengthening warming signal and leading to even greater assessed area with detectable anthropogenic warming.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationcitation

    Changes in category -/+2 are the clearest case of changes that are at least partially attributable to anthropogenic forcing, according to the CMIP6 ensemble.

    “This is clearest case of changes that are at least partially attributable to anthropogenic forcing, according to the CMIP6 ensemble.”
    Quote in the paperNot a substantial claim
54 of 54

  • The paper relies on this without support.

    The IPCC AR5 WGII tables 18.5-18.9 provide a comprehensive benchmark for the climate impact literature.

    “We assessed comprehensiveness by ensuring that our search string returned all references from tables 18.5-18.9 in AR5 WGII, which deal with the detection and attribution of climate impacts.”

  • The paper relies on this without support.

    The number of weighted studies per grid cell is a valid proxy for the level of evidence of impacts on human and natural systems.

    “We characterise areas with fewer than 5 weighted studies per grid cell as displaying low evidence, areas with between 5 and 20 weighted studies as robust evidence, and areas with more than 20 weighted studies as high evidence.”

  • The paper relies on this without support.

    BERT and similar models are able to capture the context-dependent meanings of texts.

    “Such models are able to some extent able to capture the context-dependent meanings of texts.”

  • The paper relies on this without support.

    The training data of 2,629 abstracts is representative of the broader literature on climate impacts.

    “Using training data assembled by collaboratively screening and coding 2,629 abstracts, we use supervised machine learning, fine-tuning the smaller and faster BERT variant DistilBERT 28 , to classify, also based on the abstract text, documents relevant to understanding the observed impacts of climate change in general, and to predict the human or natural systems for which they document impacts (i.e”

  • The paper relies on this without support.

    Documents selected using active learning are broadly representative of the documents to which the multilabel classifiers are applied.

    “Because documents selected for screening using the active learning process are broadly representative of the documents to which the multilabel classifiers are applied, all documents selected in this manner are also used for validation.”
5 of 5

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Deduplicated document count differs by ten

    Minor

    The results and search-methods sections give different counts for the unique documents returned by the query after deduplication: 601,677 and 601,667.

  • Temperature example assigns a precipitation proportion

    Minor

    The example derives the 20/27 proportion from cells with attributable warming for a temperature-driven study, but calls the resulting value a precipitation trend proportion. The next sentence treats it as a temperature-trend proportion.

  • Partial-attribution threshold differs between text and figure caption

    Minor

    The text defines partially attributable studies using at least 1% of grid cells, whereas the Figure 3 caption defines them using more than 0%. Those thresholds differ for studies referring to more than 100 grid cells.

  • Category ±4 trends are both distinguishable and non-detectable

    Minor

    The methods assign category −4 or +4 when an observed trend is significantly different from the natural-forcing distribution, but later call those categories examples of non-detectable trends.

4 of 4

Cites

Flagged first, then by citations

Research works printed but not linked: 3

  1. Cramer, W. et al. in Climate Change 2014: Impacts, Adaptation, and Vulnerability (eds Field, C. B. et al.) 979–1037 (Cambridge Univ. Press, 2014).
  2. IPCC Climate Change 2014: Impacts, Adaptation, and Vulnerability (eds Field, C. B. et al.) (Cambridge Univ. Press, 2014).
  3. Cochrane Handbook for Systematic Reviews of Interventions (John Wiley & Sons, 2019).

Cited by

Sorted by citations

Related

Authors
Callaghan, M., Schleussner, C.-F., Nath, S., Lejeune, Q., Knutson, T. R., Reichstein, M., Hansen, G., Theokritoff, E., Andrijevic, M., Brecha, R. J., Hegarty, M., Jones, C., Lee, K., Lucas, A., van Maanen, N., Menke, I., Pfleiderer, P., Yesil, B., & Minx, J. C.
Affiliations
  • University of Leeds (GB)
  • Humboldt-Universität zu Berlin (DE)
  • ETH Zurich (CH)
  • Climate Analytics (DE)
  • NOAA Geophysical Fluid Dynamics Laboratory (US)
  • Max Planck Institute for Biogeochemistry (DE)
  • Robert Bosch Stiftung (DE)
  • University of Dayton (US)
Corresponding author
Callaghan, Max
ORCID
Venue
Nature Climate Change
Publisher
Springer Science and Business Media LLC
Published
2021-10-11
Volume
11
Issue
11
Pages
966-972
ISSN
  • 1758-678X (ISSN-L)
  • 1758-6798
DOI
10.1038/s41558-021-01168-6
OpenAlex
W3188562849
Semantic Scholar
9592f1f9ad9e61447cfa0332df8e76270ba2e6c1
Cited by
251
References
55
Citations by year
2026: 36 · 2025: 68 · 2024: 49 · 2023: 63 · 2022: 34 · 2021: 1
FWCI
15.07
Type
journal article
Language
en
Open access
Licence
Locations
  • Nature Climate Change (publishedVersion)
  • NOAA Institutional Repository (submittedVersion, public-domain)
  • NOAA Institutional Repository (submittedVersion)
  • RePEc: Research Papers in Economics (submittedVersion)
  • Publication Database PIK (Potsdam Institute for Climate Impact Research (PIK)) (submittedVersion)
Funders
Heinrich Böll Stiftung
Topics
  • Physical Sciences › Environmental Science › Global and Planetary Change › Climate variability and models
  • Physical Sciences › Environmental Science › Global and Planetary Change › Climate Change and Environmental Impact
  • Social Sciences › Sociology and Political Science › Climate Change Communication and Perception
Keywords
anthropogenic climate change attribution, evidence mapping, literature review automation, climate impact studies, temperature and precipitation change, global environmental assessment

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-10.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-10.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review