CoChat
Sign Up
Cited by 1scientometrics and bibliometrics researchComputational and Text Analysis MethodsResearch, Science, and Academia

Can Large Language Models Evaluate Grant Proposal Quality? Revisiting the Wennerås and Wold Peer Review Data

Open access
Sandström, Ulf, Thelwall, Mike·2026·Journal of Data and Information Science
DOI 10.1515/jdis-2026-0048
Open this paper in CoChat — free account

Abstract

Purpose: Despite the importance of peer review for grant funding decisions, academics are often reluctant to conduct it. This can lead to long delays between submission and the final decision as well as the risk of substandard reviews from busy or non-specialist scholars. At least one funder now uses Large Language Models (LLMs) to reduce the reviewing burden but the accuracy of LLMs for scoring grant proposals needs to be assessed. Design/methodology/approach: This article compares scores from a range of medium sized open-weight LLMs with peer review scores for a well-researched dataset, 142 Swedish Medical Council post-doctoral fellowship applications from 1994. Findings: Whilst the LLM scores correlate moderately between each other (mean Spearman correlation: 0.34), they correlated weakly but positively and mostly statistically significantly with the average expert scores (mean Spearman correlation: 0.22). The highest rank correlation between expert scores and LLMs was 0.33 for Gemma 3 27 b based on proposal titles and summaries without their main texts, which is about half (56 %) of the correlation between reviewers. Research limitations: The small sample size, old funding call and heterogeneous evaluation criteria all undermine the robustness of the analysis. Practical implications: Despite the ability of LLMs to score grant proposals being quantitatively weaker than that of experts, at least in this special case, they may have role in application triage or tie-breaking. Originality/value: This is the first assessment of the value of LLM scores for funding proposals.

AI snapshot

Written from the paper’s full text.

TL;DR

This study tests whether medium-sized open-weights LLMs can score Swedish MRC postdoctoral fellowship applications from 1994 in ways that align with expert peer-review scores. It finds weak but mostly positive and statistically significant rank correlations between LLM and expert scores, weaker than agreement among expert panels.

Key findings

  • LLM scores correlated weakly but positively and mostly statistically significantly with average expert scores: mean Spearman correlation 0.22; the highest rank correlation was 0.33 for Gemma 3 27b using proposal titles and summaries without main texts, about half (56%) of the correlation between reviewers.
  • LLMs correlated moderately with each other: mean Spearman correlation 0.34, higher than their average correlation with reviewer scores (0.22).
  • Panel-score agreement was higher than LLM–expert agreement where sample sizes allowed: General 0.63 (0.26, 0.83), n=21; Mobility 0.83 (-1.00, 1.00), n=4; Priority 0.35 (-0.54, 0.94), n=9; All 0.62 (0.32, 0.81), n=26.
  • Title-and-summary versus full-text input was not clearly superior overall; averaging across models improved correlations slightly, but differences were mostly not statistically significant.

Methods

  • The study used 142 Swedish Medical Research Council postdoctoral fellowship applications from 1994, scanned and OCR-processed with Tesseract, translated from Swedish to English with Gemma 3 27b, redacted for personal information, and prepared as title+summary and title+summary+main-text documents.
  • Expert peer-review scores were 0–4 on research question, methodology, and applicant competence; applications were assessed within programs General (n=119), Priority Areas (n=19), and Mobility (n=24), with up to two panels (PK1 and PK2) per program, aggregated within program by the mean where both were available.
  • Six open-weights LLMs were tested: Gemma 3 27b, Magistral Small 24b, Llama Scout 17b, Qwen3 32b, DeepSeek-R1 32b, and DeepSeek-R1 72b; each proposal was submitted five times per LLM and averaged, and Spearman rank correlations with bootstrapped 95% confidence intervals were used.

Limitations

  • The paper states that the data are old, sample sizes are too small to analyze panels separately, evaluation criteria are heterogeneous, LLMs did not account for host environment quality/reputation, outcomes may partly be in training data, relevance of 1994 criteria and processes today is unclear, and translation may have influenced scoring.
  • The paper also states that averaging scores across programs assumes commensurability of evaluations under different mandates, which is not necessarily true.
  • Evident from the text: Llama Scout is missing from the full-text set because it produced few valid scores, and the Mobility and Priority panel samples are very small.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1515/jdis-2026-0048

Cites

39 of 40 references verified0 citation errors · 1 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

18 of 35 cited sources checked · 8 supported · 1 flagged · accuracy not yet measured

  • Premisecitation
    § 1 Introduction · p. 1

    LLMs have shown a moderately good ability to estimate the research quality of other academic documents, such as research environment statements, conference papers and books.

    “and other academic documents, such as research environment statements (Kousha et al., 2025), conference papers (Thelwall & Yaghi, 2025a) and books (Thelwall & Cox, 2025)”
    Quote in the paperSubstantial claim1 of 3 sources checked
    • cites Kousha et al., 2025

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Thelwall & Yaghi, 2025a

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Thelwall & Cox, 2025 source

      The ‘Comparison with citations’ section reports a moderate positive correlation between ChatGPT book-quality scores and citation counts. It also says the evidence for either as a quality indicator is not strong and that both may be weak indicators. Thus, the source supports some ability to estimate book quality, but ‘moderately good’ overstates its conclusion. The other document types have separate citations and are not judged here. (LLM, conf=0.86)

      Experimental — accuracy not yet measured
  • Premisecitation
    § 2.1 Inter-reviewer reliability for grant peer review · p. 4

    Analyses of NIH, NSF, and European funding programmes report correlations between reviewers that often fall in the range of 0.2-0.4, depending on field and design.

    “analyses of NIH, NSF, and European funding programmes report correlations between reviewers that often fall in the range of 0.2 -0.4, depending on field and design (Bornmann et al., 2010; Li & Agha, 2015).”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites Bornmann et al., 2010 source

      The Results (Comparison of Average Effects; page unavailable) report reliability estimates of .23–.34 for journal manuscript reviews, not reviewer correlations for funding programmes; the paper says the correlations were squared to obtain those estimates. A search for NIH, NSF, European funding programmes, and grant-review correlations found no passage reporting the claimed .2–.4 range for those programmes. (LLM, conf=0.76)

      Experimental — accuracy not yet measured
    • cites Li & Agha, 2015 source
  • Findingown data
    § Can Large Language Models Evaluate Grant Proposal Quality? Revisiting the Wennerås and Wold Peer Review Data · p. 1

    The highest LLM-expert correlation is about half (56%) of the correlation between reviewers.

    “which is about half (56%) of the correlation”
    Quote in the paperSubstantial claim
  • Findingown data
    § Can Large Language Models Evaluate Grant Proposal Quality? Revisiting the Wennerås and Wold Peer Review Data · p. 1

    LLM scores correlate moderately with each other, with a mean Spearman correlation of 0.34.

    “Whilst the LLM scores correlate moderately between each other (mean Spearman correlation: 0.34)”
    Quote in the paperNot a substantial claim
  • Findingown data
    § Can Large Language Models Evaluate Grant Proposal Quality? Revisiting the Wennerås and Wold Peer Review Data · p. 1

    The highest rank correlation between expert scores and LLMs was 0.33 for Gemma 3 27b based on proposal titles and summaries without their main texts.

    “The highest rank correlation between expert scores and LLMs was 0.33 for Gemma 3 27b based on proposal titles and summaries without their main texts”
    Quote in the paperNot a substantial claim
  • Classified as probably not a substantial claim. 39
  • May combine two claims. 21

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 2.2 LLMs for research evaluation · p. 4

    Methods to detect LLM-authored reviews have been developed.

    “leading to the development of methods to detect LLMauthored reviews (Liu et al., 2025; Ryu et al., 2025).”
    Quote in the paperNot a substantial claim2 of 2 sources checked
    • cites Liu et al., 2025 source
    • cites Ryu et al., 2025 source

      Section 3.3 says the proposed method uses LLMs to filter out misinformed review points. The Conclusion likewise describes ReviewScore as detecting misinformed points, not determining whether an LLM authored a review. The claim confuses using LLMs for detection with detecting LLM authorship. (LLM, conf=0.86)

      Checked by a reasoning model — may be wrong.

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § 1 Introduction · p. 1

    La Caixa employs large language models to triage proposals, with the lowest scoring proposals being sent to human experts for quick reviews.

    “La Caixa employs Large Language Models (LLMs) to triage proposals, with the lowest scoring proposals being sent to human experts for quick reviews.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Carbonell Cortés et al., 2024 source

      Section 4.2 says three NLP models flag proposals unanimously classified in the bottom score class. Page e49-8 says two human eligibility reviewers assess every flagged proposal, and Section 5.2 describes their evaluation as superficial, supporting the quick-review characterization. The paper identifies the deployed models as BioLinkBERT-Base and BioELECTRA-Base variants, but does not identify them as large language models; its discussion of LLMs in Section 1.2 does not establish that La Caixa uses them for this process. (LLM, conf=0.85)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 1 Introduction · p. 1

    One funder now uses artificial intelligence for reviewing.

    “one now uses it for reviewing.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Carbonell Cortés et al., 2024 source

  • Classified as probably not a substantial claim.

    Premisecitation
    § 1 Introduction · p. 1

    LLMs have shown a moderately good ability to estimate the research quality of published academic journal articles.

    “LLMs have shown a moderately good ability to estimate the research quality of published academic journal articles (Thelwall, 2025; Thelwall & Yang, 2025)”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites Thelwall, 2025

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Thelwall & Yang, 2025 source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 1 Introduction · p. 1

    After La Caixa's triage, proposals judged weak by all three (LLM and two reviewers) are rejected without a full review.

    “After this, proposals judged weak by all three (LLM and two reviewers) are rejected without a full review”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Carbonell Cortés et al., 2024 source

      The paper says proposals flagged by the models are checked by two eligibility reviewers; if either reviewer has reasonable doubt, the proposal is sent to peer review (e49-8). This supports removal without full peer review when the reviewers agree. However, §4.2 says the automated decision requires agreement among three NLP models, not one LLM, so the claim misstates who the three decision-makers are. (LLM, conf=0.88)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 2.2 LLMs for research evaluation · p. 4

    A range of cloud-based and open weights LLMs have an ability to score published journal articles for research quality in the sense of giving scores that correlate positively and statistically significantly with expert review scores.

    “A range of cloud-based and open weights LLMs have an ability to score published journal articles for research quality in the sense of giving scores that correlate positively and statistically significantly with expert review scores (Thelwall, 2025; Thelwall & Mohammadi, 2026; Thelwall & Yang, 2025).”
    Quote in the paperNot a substantial claim2 of 3 sources checked
    • cites Thelwall, 2025

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Thelwall & Mohammadi, 2026 source

      The Introduction reports correlations between cloud-based LLM scores and expert judgement, and the RQ1 results say medium-sized models achieved similar correlations; the Conclusions find research-quality scoring ability across open-weights models. However, the study’s main expert-review benchmark is a departmental-average proxy, while its article-level scores were assigned by a single author who says he is not an expert in the six fields (Data). The cited passages do not establish that the reported correlations with expert review scores were statistically significant. (LLM, conf=0.85)

      Experimental — accuracy not yet measured
    • cites Thelwall & Yang, 2025 source

      Section 1.2 says ChatGPT quality scores correlate with a research-quality indicator and reports a similar, weaker ability for Gemini. Sections 3.1 and 3.6 clarify that this paper tests ChatGPT against departmental average expert scores as a proxy, not individual article review scores. The cited paper does not establish the claim for open-weight LLMs or state that the reported correlations are statistically significant. (LLM, conf=0.86)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Premisenone given
    § 1 Introduction · p. 1

    Some previous studies have used AI to assess funding applications, but none seem to have produced results good enough to be adopted by funders.

    “While some previous studies have used AI to assess funding applications (e.g., Sikimić, & Radovanović, 2022 ), none seem to have produced results that were good enough to be adopted by funders”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Sikimić, & Radovanović, 2022 source

      The Introduction describes a study using machine learning to predict project efficiency from data available in proposals, so the paper is an example of AI research relevant to funding-application assessment. But it analyzes records of completed Fermilab experiments, rather than reporting that AI assessed submitted funding applications. The sentence’s broader conclusion about adoption by funders is not specifically attributed to this citation. (LLM, conf=0.82)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 1 Introduction · p. 2

    Previous research has suggested that medium-sized open weights LLMs work nearly as well as larger cloud-based models such as ChatGPT for academic evaluations.

    “Previous research has suggested that these LLMs work nearly as well as larger cloud-based models such as ChatGPT for academic evaluations (Thelwall & Mohammadi, 2026).”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Thelwall & Mohammadi, 2026 source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 1 Introduction · p. 1

    La Caixa's use of LLMs has led to a small reduction in the number of reviews needed.

    “This has led to a small reduction in the number of reviews needed (Carbonell Cortés et al., 2024).”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Carbonell Cortés et al., 2024 source

      Section 4.1 (p. e49-8) reports that the AI-assisted process removed 63 of 493 proposals from peer review in the HR23 call, supporting a reduction in reviews needed. But Section 4.2 identifies the prescreening tools as three specific NLP models, not as LLMs, so the claim's description of the technology is not established by this source. (LLM, conf=0.83)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § 2.2 LLMs for research evaluation · p. 4

    Smaller open weights LLMs seem to be powerful enough for the task of LLM review support, allowing more security than when using cloud-based versions.

    “Smaller open weights LLMs seem to be powerful enough for this task (Sadallah et al., 2025), allowing more security than when using cloud-based versions.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Sadallah et al., 2025 source

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § 3.1.2 Peer review scores · p. 6

    Panel deliberations seemed to also incorporate the host research environment, but this was not an explicit criterion so was not included.

    “Panel deliberations seemed to also incorporate the host research environment (Sandström, & Riis, 2025), but this was not an explicit criterion so was not included.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Sandström, & Riis, 2025 source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 2.2 LLMs for research evaluation · p. 4

    Of those accepting the feedback at ICLR, 27% subsequently altered their reviews.

    “Of those accepting the feedback, 27% subsequently altered their reviews (Thakkar et al., 2025)”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Thakkar et al., 2025 source

      Section 3.1 (p. 7) reports that 5,031 of 18,946 reviews that received feedback were updated—26.6%, reasonably rounded to 27%. But the denominator is reviewers who received feedback, not reviewers who accepted it; Section 2.1 (p. 4) says reviewers could ignore the feedback. (LLM, conf=0.84)

      Experimental — accuracy not yet measured

  • Classified as probably not a substantial claim.

    Premisecitation
    § 3.2.3 Request processing · p. 8

    Previous research has shown that averaging multiple iterations increases correlations with human judgments.

    “Previous research has shown that averaging multiple iterations increases correlations with human judgments (e.g., Thelwall & Yang, 2025 )”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Thelwall & Yang, 2025 source

  • May combine two claims.

    Premisecitation
    § 3.3 Analysis · p. 8

    Despite LLM scores being essentially mostly wrong, the rankings are still often useful.

    “Despite these scores being essentially mostly wrong, the rankings are still often useful (Thelwall & Mohammadi, 2026).”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Thelwall & Mohammadi, 2026 source

  • May combine two claims.

    Findingown data
    § Can Large Language Models Evaluate Grant Proposal Quality? Revisiting the Wennerås and Wold Peer Review Data · p. 1

    LLM scores correlate weakly but positively and mostly statistically significantly with average expert scores, with a mean Spearman correlation of 0.22.

    “they correlated weakly but positively and mostly statistically significantly with the average expert scores (mean Spearman correlation: 0.22).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 6 Conclusions · p. 14

    Averaging the scores from multiple models and both types of input seems to give the best results, with correlations that are 56% of the correlations between the expert panels.

    “Moreover, averaging the scores from multiple models and both types of input (i.e., titles and summaries with and without main texts) seems to give the best results, with correlations that are 56% of the correlations between the expert panels.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: Averaging scores across programs assumes commensurability of evaluations produced under different mandates.
    • Rests on an assumption: In the Swedish MRC review process, the original score set by each reviewer to a proposal was not changed.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data
    § 4 Results · p. 8

    Averaging across models seems to improve the correlation, although the difference is small and not statistically significant.

    “Moreover, averaging across models seems to improve the correlation, although the difference is small and not statistically significant in any case.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 4 Results · p. 12

    For overall scores, the average correlation between LLMs was 0.34 whereas their average correlation with reviewer scores was 0.22.

    “For example, for the overall scores, the average correlation between LLMs was 0.34 whereas their average correlation with the reviewer scores was 0.22”
    Quote in the paperSubstantial claim
    • Rests on an assumption: In the Swedish MRC review process, the original score set by each reviewer to a proposal was not changed.

  • May combine two claims.

    Findingown data
    § 4 Results · p. 8

    The LLMs had weak but usually positive and statistically significant positive correlations with mean reviewer scores for all dimensions and overall, both for the title and summary and the title, summary and main text.

    “The LLMs had weak but usually positive and statistically significant positive correlations with mean reviewer scores for all dimensions and overall, both for the title and summary and the title, summary and main text (Figures 2-4).”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: Averaging scores across programs assumes commensurability of evaluations produced under different mandates.
    • Rests on an assumption: In the Swedish MRC review process, the original score set by each reviewer to a proposal was not changed.

  • May combine two claims.

    Findingown data
    § 4 Results · p. 8

    Neither the title and summary nor the full text are clearly superior overall: it does not seem to matter which is fed to any of the LLMs.

    “Neither the title and summary nor the full text are clearly superior overall: it does not seem to matter which is fed to any of the LLMs.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 2.1 Inter-reviewer reliability for grant peer review · p. 4

    Low inter-reviewer agreement is not primarily a consequence of reviewer incompetence but reflects the inherently multidimensional and judgement-based nature of grant assessment.

    “More recent evidence suggests that low inter-reviewer agreement is not primarily a consequence of reviewer incompetence but rather reflects the inherently multidimensional and judgement-based nature of grant assessment.”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites Langfeldt, 2006 source
    • cites Fogelholm et al., 2012 source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 1 Introduction · p. 1

    Funders already use artificial intelligence to help select reviewers.

    “Funders already use Artificial Intelligence (AI) to help select reviewers (NewmanGriffis et al., 2025)”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites NewmanGriffis et al., 2025

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Premisereasoning
    § 2.1 Inter-reviewer reliability for grant peer review · p. 4

    Variability between reviewers is often systematic rather than random, with different reviewers emphasising different aspects of proposal quality.

    “As a result, variability between reviewers is often systematic rather than random, with different reviewers emphasising different aspects of proposal quality.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given
    § 5 Discussion · p. 13

    The data is old.

    “The main limitations are that the data is old”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given
    § 5 Discussion · p. 13

    The sample sizes are too small to analyse the panels separately, which would be the most robust approach.

    “the sample sizes are too small to analyse the panels separately, which would be the most robust approach.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Interpretationnone given
    § 5 Discussion · p. 13

    The LLMs would not have taken into account host environment quality and reputation, and the human reviewers probably did.

    “In addition, the LLMs would not have taken into account host environment quality and reputation, and the human reviewers probably did.”
    Quote in the paperNot a substantial claim
    • The authors state this as an assumption: Averaging scores across programs assumes commensurability of evaluations produced under different mandates.
    • Rests on an assumption: In the Swedish MRC review process, the original score set by each reviewer to a proposal was not changed.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 2.2 LLMs for research evaluation · p. 4

    The International Conference on Learning Representations offers LLM suggestions to reviewers after completing their initial reviews.

    “The International Conference on Learning Representations (ICLR) is a venue that offers LLM suggestions to reviewers after completing their initial reviews.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 2.2 LLMs for research evaluation · p. 4

    LLMs can make suggestions about unpublished manuscripts from their full text as part of the peer review process.

    “LLMs can make suggestions about unpublished manuscripts from their full text as part of the peer review process (Liang et al., 2024)”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Liang et al., 2024

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data
    § 3.1.1 Applications · p. 5

    The text extraction process using Tesseract worked generally well but with some mistakes and many errors with chemical formulae.

    “This process worked generally well but with some mistakes and many errors with chemical formulae.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisenone given
    § 2.2 LLMs for research evaluation · p. 4

    Currently, the formal use of LLMs for peer review is supported only by a few computer science conferences and one publisher.

    “Currently, the formal use of LLMs for peer review is supported only by a few computer science conferences and one publisher”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 2.2 LLMs for research evaluation · p. 4

    Reviewers often elicit LLM help even when explicitly told not to.

    “reviewers often elicit LLM help even when explicitly told not to (Naddaf, 2026)”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Naddaf, 2026

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given
    § 5 Discussion · p. 13

    The translation step for the Swedish applications may have influenced the scoring.

    “Moreover, the translation step for the Swedish applications may have influenced the scoring.”
    Quote in the paperNot a substantial claim
    • The authors state this as an assumption: Averaging scores across programs assumes commensurability of evaluations produced under different mandates.
    • Rests on an assumption: In the Swedish MRC review process, the original score set by each reviewer to a proposal was not changed.

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning
    § 3.1.1 Applications · p. 6

    The omission of figures is probably less critical than for publications because the figures usually offered visual explanations of the text rather than research results.

    “This omission is probably less critical than for publications because the figures usually offered visual explanations of the text rather than research results.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data
    § 6 Conclusions · p. 14

    Gemma 3 27b is possibly the best model available at the start of 2026, although not all have been tested and differences between models were not statistically significant.

    “with Gemma 3 27b being possibly the best model available at the start of 2026, although not all have been tested and differences between models were not statistically significant.”
    Quote in the paperNot a substantial claim
    • The authors state this as an assumption: Averaging scores across programs assumes commensurability of evaluations produced under different mandates.
    • Rests on an assumption: In the Swedish MRC review process, the original score set by each reviewer to a proposal was not changed.

  • May combine two claims.

    Study design and conductreasoning
    § 3.1.2 Peer review scores · p. 6

    Because panel evaluations are nested within programs and programs differ in mandate and evaluative criteria, there is no uniquely correct way to aggregate panel scores into a single applicant-level measure.

    “Because panel evaluations are nested within programs and because programs differ in mandate and evaluative criteria, there is no uniquely correct way to aggregate these scores into a single applicant-level measure.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data
    § 4 Results · p. 9

    Llama Scout is missing from the titles, summaries and main texts set because it produced few valid scores.

    “Llama Scout is missing from the titles, summaries and main texts set because it produced few valid scores.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationreasoning
    § 5 Discussion · p. 14

    The age of the proposals might have made the task more difficult.

    “Alternatively, the age of the proposals might have made the task more difficult.”
    Quote in the paperNot a substantial claim
    • The authors state this as an assumption: Averaging scores across programs assumes commensurability of evaluations produced under different mandates.
    • Rests on an assumption: In the Swedish MRC review process, the original score set by each reviewer to a proposal was not changed.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationcitation
    § 2.2 LLMs for research evaluation · p. 4

    LLM suggestions seemed to be useful in the ICLR context.

    “so they seemed to be useful in this context.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § 2.2 LLMs for research evaluation · p. 4

    LLM scores correlate positively with expert scores despite LLMs being fed with article titles and abstracts only, which is insufficient to evaluate an article.

    “This correlation occurs despite the LLMs being fed with article titles and abstracts only, which is insufficient to evaluate an article.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given
    § 3.1.1 Applications · p. 5

    All translated documents were checked by a bilingual native speaker of Swedish for translation accuracy.

    “Nevertheless, all translated documents were checked by a bilingual native speaker of Swedish (Ulf) for translation accuracy.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 2.2 LLMs for research evaluation · p. 4

    LLMs can summarise reviewers' comments to support editors or authors.

    “and can summarise reviewers ' comments to support editors or authors (Du et al., 2024).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Du et al., 2024

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

    Interpretationreasoning
    § 5 Discussion · p. 14

    The proposal evaluation is highly person-centred, for fellowships, and for a relatively small group, such as Swedish medical researchers, the human reviewers could have taken more tacit and background knowledge into account when evaluating the individuals.

    “Another explanation is that the proposal evaluation is highly person-centred, for fellowships, and for a relatively small group, such as Swedish medical researchers, the human reviewers could have taken more tacit and background knowledge into account when evaluating the individuals.”
    Quote in the paperNot a substantial claim
    • The authors state this as an assumption: Averaging scores across programs assumes commensurability of evaluations produced under different mandates.
    • Rests on an assumption: In the Swedish MRC review process, the original score set by each reviewer to a proposal was not changed.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning
    § 5 Discussion · p. 13

    The outcomes of the funding process are partly known and may be in the training data, in the sense that funded researchers may acknowledge their funding on their CVs and in their publications.

    “Another important limitation that may have increased the correlation is that the outcomes of the funding process is partly known and may be in the training data, in the sense that funded researchers may acknowledge their funding on their CVs and in their publications.”
    Quote in the paperNot a substantial claim
    • The authors state this as an assumption: Averaging scores across programs assumes commensurability of evaluations produced under different mandates.
    • Rests on an assumption: In the Swedish MRC review process, the original score set by each reviewer to a proposal was not changed.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 2.2 LLMs for research evaluation · p. 4

    The strength of the correlation between LLM scores and expert review scores varies between weak and moderate, depending on the field, with stronger correlations for health, life and physical sciences.

    “The strength of the correlation varies between weak and moderate, depending on the field, with stronger correlations for health, life and physical sciences (Thelwall & Yaghi, 2025b).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Thelwall & Yaghi, 2025b

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Interpretationown data
    § 6 Conclusions · p. 14

    The results show for the first time, albeit on a limited scale, that LLMs have a weak but statistically significant ability to score grant proposals in a way that aligns with human scores in rank, although not in magnitude.

    “The results show for the first time, albeit on a limited scale, that LLMs have a weak but statistically significant ability to score grant proposals in a way that aligns with human scores in rank, although not in magnitude.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: Averaging scores across programs assumes commensurability of evaluations produced under different mandates.
    • Rests on an assumption: In the Swedish MRC review process, the original score set by each reviewer to a proposal was not changed.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning
    § 5 Discussion · p. 13

    The study has very many limitations, most of which are likely to have reduced the strength of the LLM correlations with expert judgment relative to a perfect test.

    “This study has very many limitations, most of which are likely to have reduced the strength of the LLM correlations with expert judgment relative to a perfect test.”
    Quote in the paperNot a substantial claim
    • The authors state this as an assumption: Averaging scores across programs assumes commensurability of evaluations produced under different mandates.
    • Rests on an assumption: In the Swedish MRC review process, the original score set by each reviewer to a proposal was not changed.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data
    § 3.1.2 Peer review scores · p. 6

    It was not possible to analyse the proposals separately within panels due to small sample sizes.

    “although the latter was not possible due to small sample sizes.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductown data
    § 3.1.1 Applications · p. 5

    All proposals were individually corrected for text extraction anomalies by a native speaker.

    “All proposals were individually corrected for these anomalies by a native speaker (Ulf for Swedish, Mike for English).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationreasoning
    § 2.2 LLMs for research evaluation · p. 4

    LLM scores for journal articles are just guesses, despite correlating positively with expert scores.

    “Thus, the LLM scores are just guesses, despite correlating positively with expert scores.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given
    § 3.1.1 Applications · p. 5

    Gemma 3 27b seemed to perform the translation of Swedish proposals into English flawlessly.

    “Translation is a core task for LLMs, and Gemma 3 27b seemed to perform it flawlessly.”
    Quote in the paperNot a substantial claim
50 of 50

  • The paper relies on this without support.

    Expert peer-review scores are a valid criterion for grant proposal quality.

    “The current study assesses the ability of LLMs to rate funding proposals in a way that aligns with expert scores.”

  • The paper relies on this without support.

    Findings from the 1994 Swedish MRC fellowship applications are suggestive about how well LLMs will perform in other grant review cases.

    “getting evidence for this case will be suggestive about how well LLMs will perform in other cases.”

  • The paper relies on this without support.

    The 1994 expert scores remain a meaningful benchmark for evaluating current LLM performance.

    “repurposes the same historical material to address a different but complementary question: whether contemporary Large Language Models can approximate expert peer-review scores when applied to real grant proposals.”

  • The paper relies on this without support.

    LLM scores based on titles and summaries are comparable to scores based on full proposals for grant quality evaluation.

    “Although it is more logical to assess a full proposal than just its summary, LLMs have been shown to work as well on summaries as with full texts for academic texts so both approaches were used.”

  • The authors state this as an assumption.

    Averaging scores across programs assumes commensurability of evaluations produced under different mandates.

    “averaging scores across programs implicitly assumes commensurability of evaluations produced under different mandates”

  • The paper relies on this without support.

    In the Swedish MRC review process, the original score set by each reviewer to a proposal was not changed.

    “The original score set by each reviewer to a proposal was not changed in the process.”
6 of 6

Cites

Flagged first, then by citations

1 of the 39 references couldn’t be matched to a known paper.

Cited by

Sorted by citations

Authors
Sandström, U., & Thelwall, M.
Affiliations
  • KTH Royal Institute of Technology (SE)
  • University of Sheffield (GB)
ORCID
Venue
Journal of Data and Information Science
Publisher
Walter de Gruyter GmbH
Published
2026-09-21
ISSN
  • 2096-157X (ISSN-L)
  • 2543-683X
DOI
10.1515/jdis-2026-0048
OpenAlex
W7137828848
Cited by
1
References
39
Citations by year
2026: 1
FWCI
5.15
Type
journal article
Language
en
Open access
Licence
Locations
  • Journal of Data and Information Science (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • White Rose Research Online (University of Leeds, The University of Sheffield, University of York) (submittedVersion, other-oa)
  • arXiv (Cornell University) (cc-by)
Funders
ESRC Metascience (UKRI1486)
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › scientometrics and bibliometrics research
  • Social Sciences › General Social Sciences › Computational and Text Analysis Methods
  • Social Sciences › Decision Sciences › Information Systems and Management › Research, Science, and Academia
Keywords
large language models, grant proposal evaluation, peer review, research funding, open-weight LLMs

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-23.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-23.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review