CoChat
Sign Up
Cited by 17Topic ModelingSemantic Web and OntologiesAdvanced Text Analysis Techniques

The DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index; 2 of the 11 sources its claims cite were checked for support.

Evaluating Large Language Models for Structured Science Summarization in the Open Research Knowledge Graph

Open access
Nechakhin, Vladyslav, D’Souza, Jennifer, Eger, Steffen·2024·Information
DOI 10.3390/info15060328
Open this paper in CoChat — free accountOpen PDF

Abstract

Structured science summaries or research contributions using properties or dimensions beyond traditional keywords enhance science findability. Current methods, such as those used by the Open Research Knowledge Graph (ORKG), involve manually curating properties to describe research papers’ contributions in a structured manner, but this is labor-intensive and inconsistent among human domain-expert curators. We propose using Large Language Models (LLMs) to automatically suggest these properties. However, it is essential to assess the readiness of LLMs like GPT-3.5, Llama 2, and Mistral for this task before their application. Our study performs a comprehensive comparative analysis between the ORKG’s manually curated properties and those generated by the aforementioned state-of-the-art LLMs. We evaluate LLM performance from four unique perspectives: semantic alignment with and deviation from ORKG properties, fine-grained property mapping accuracy, SciNCL embedding-based cosine similarity, and expert surveys comparing manual annotations with LLM outputs. These evaluations occur within a multidisciplinary science setting. Overall, LLMs show potential as recommendation systems for structuring science, but further fine-tuning is recommended to improve their alignment with scientific tasks and mimicry of human expertise.

Overview

AI snapshot

Written from the paper’s full text.

TL;DR

The paper evaluates GPT-3.5, Llama 2, and Mistral for automatically suggesting ORKG research dimensions from research problems, comparing them with expert-curated properties across a multidisciplinary dataset. It finds moderate alignment and low exact mapping, but high embedding similarity, suggesting LLMs can assist as recommendation systems but need domain fine-tuning.

Key findings

  • Semantic alignment was moderate: average alignment 2.9/5 and deviation 4.1/5, normalized to 41.2% alignment and 58.8% deviation.
  • Fine-grained mapping was low: average 0.33 mappings between ORKG properties and LLM dimensions; average ORKG property count was 4.73 versus average GPT dimension count of 8.
  • Embedding similarity was high: SciNCL cosine similarity was 0.84 for GPT-3.5, 0.79 for Mistral, and 0.76 for Llama 2; SPECTER gave 0.86, 0.76, and 0.70, and SciBERT gave 0.84, 0.66, and 0.57, respectively.
  • In the expert survey (23 responses), 36.3% of LLM-generated dimensions were considered highly relevant, 60.9% would not change existing ORKG annotations, 65.2% found LLM suggestions helpful, and alignment was rated 2.65/5 on average.

Methods

  • The study compares manually curated ORKG properties with research dimensions generated by three LLMs: GPT-3.5, Llama 2, and Mistral, using a zero-shot prompt after testing zero-shot, few-shot, and chain-of-thought prompting.
  • The gold-standard dataset contains 103 ORKG Comparisons, 1,317 papers, 35 research fields, and over 150 distinct research problems; comparisons were selected if they had at least 3 properties and at least 5 contributions.
  • Evaluation used four perspectives: GPT-3.5 semantic alignment/deviation scoring, GPT-3.5 fine-grained property-to-dimension mapping counts, SciNCL/SciBERT/SPECTER embedding-based cosine similarity, and a human expert survey comparing ORKG annotations with GPT-3.5 outputs.

Limitations

  • Stated in the paper: the LLMs were trained on broad text rather than exclusively scientific literature; hardware limitations prevented testing larger models; and this is a first empirical investigation with a preliminary paradigm.
  • Evident from the text: only three models were tested, only zero-shot prompting was used in the final evaluation, GPT-3.5 served as the evaluator for alignment/deviation and mapping, and the human survey had 23 responses, which limits generalizability.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.3390/info15060328

Cites

33 of 33 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

2 of 11 cited sources checked · 1 supported · 0 flagged · accuracy not yet measured

  • Findingown data

    There is a noticeable gap in the mapping of dimensions generated by LLMs versus those annotated by domain experts.

    “However, there is a noticeable gap in the mapping of dimensions generated by LLMs versus those annotated by domain experts, highlighting the need for fine-tuning LLMs on domain-specific datasets to reduce this disparity.”
    Quote in the paperSubstantial claim
  • Findingown data

    The average number of mappings between paper properties and research field dimensions was 0.33.

    “The average number of mappings was found to be 0.33, indicating a low number of mappings between paper properties and research field dimensions.”
    • Also stated in: § 5. Conclusions · hedged
      “Second, our property and research dimension mapping analysis indicated a low number of mappings between paper properties and research dimensions.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: GPT-3.5's counts of similar values between lists are a valid measure of property-dimension mappings.
  • Findingreasoning

    The evaluation dataset comprises 103 ORKG Comparisons.

    “Upon the application of these criteria, the resulting dataset comprised 103 ORKG Comparisons.”
    Quote in the paperSubstantial claim
  • Findingown data

    The average alignment between paper properties and research field dimensions was 2.9 out of 5.

    “The average alignment between paper properties and research field dimensions was found to be 2.9 out of 5, indicating a moderate level of alignment.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: GPT-3.5's alignment and deviation ratings are reliable and valid measures of semantic alignment and deviation between the lists.
  • Findingreasoning

    When normalized, the proportional values translate to 41.2% alignment and 58.8% deviation.

    “When normalized, the proportional values translate to 41.2% alignment and 58.8% deviation (Figure 6).”
    Quote in the paperSubstantial claim
    • Rests on an assumption: GPT-3.5's alignment and deviation ratings are reliable and valid measures of semantic alignment and deviation between the lists.
  • Probably not a substantial claim. 25
  • It may state two separate claims as one. 20
  • The quote only partly states this claim. 1

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Premisecitation

    SciNCL is a language model designed to capture the semantic similarity between scientific concepts by leveraging pretrained BERT embeddings.

    “More recently, Ostendorff et al. introduced SciNCL [8], a language model designed to capture the semantic similarity between scientific concepts by leveraging pretrained BERT embeddings.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [8] Ostendorff, M., Rethmeier, N., Augenstein, I., Gipp, B., and Rehm, G. (2022). Neighborhood contrastive learning for scientific document representations with citation embeddings. arXiv. source

      The Contributions and Model Training and Implementation sections describe SciNCL as a model for scientific document representations initialized with SciBERT weights. The claim shifts its target from similarity between papers to similarity between scientific concepts, and omits the citation-graph-based sampling central to the method. Page numbers are not available in our copy. (LLM, conf=0.88)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Premisecitation

    GPT-3 models contain from 125 million parameters for the smallest model to 175 billion for the largest.

    “In comparison, its predecessor GPT-3 models come in different sizes and contain from 125 million parameters for the smallest model to 175 billion for the largest [10].”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [10] Brown, 2020, Language models are few-shot learners, Adv. Neural Inf. Process. Syst. source

  • It may state two separate claims as one. The quote only partly states this claim.

    Findingown data

    LLM-generated suggestions were deemed useful in facilitating the creation of comparisons, identifying relevant properties, providing a starting point for further refinement, providing a basis for improvement, and aiding in including additional properties.

    “The respondents noted that such LLM-based dimension suggestions could serve as a useful starting point, provide a basis for improvement, and aid in including additional properties.”
    • Also stated in: § 4.4. Human Assessment Survey · plain
      “For instance, the suggestions were deemed useful in facilitating the creation of comparisons, identifying relevant properties, and providing a starting point for further refinement.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The manually curated ORKG properties are an appropriate ground truth for evaluating LLM-generated research dimensions.
    • Rests on an assumption: The survey respondents' ratings are a valid measure of domain-expert assessment of relevance, alignment, and utility.

  • Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

    Findingown data

    Respondents emphasized the importance of aligning LLMs based on the domain-expert property names while allowing descriptions to be generated, ensuring relevance across different research domains and capturing specific details like measurement values and units.

    “For the final question on the release of such an LLM-based feature, the respondents emphasized the importance of aligning LLMs based on the domain-expert property names while allowing descriptions to be generated, ensuring relevance across different research domains and capturing specific details like measurement values and units.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The manually curated ORKG properties are an appropriate ground truth for evaluating LLM-generated research dimensions.
    • Rests on an assumption: The survey respondents' ratings are a valid measure of domain-expert assessment of relevance, alignment, and utility.

  • Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

    Findingown data

    We received 23 total responses to our survey, corresponding to 23 different papers.

    “At the end, we received 23 total responses to our survey, corresponding to 23 different papers.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The survey respondents' ratings are a valid measure of domain-expert assessment of relevance, alignment, and utility.

  • It may state two separate claims as one.

    Findingown data

    Using SPECTER embeddings, the cosine similarity between the ORKG properties and the LLM-generated dimensions was 0.86 for GPT-3.5, 0.70 for Llama 2, and 0.76 for Mistral.

    “Specifically, using SPECTER embeddings, the cosine similarity between the ORKG properties and the LLM-generated dimensions was 0.86 for GPT-3.5, 0.70 for Llama 2, and 0.76 for Mistral.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The manually curated ORKG properties are an appropriate ground truth for evaluating LLM-generated research dimensions.
    • Rests on an assumption: Cosine similarity values of approximately 0.76–0.86 indicate strong semantic alignment or overlap between ORKG properties and LLM-generated dimensions.

  • Probably not a substantial claim.

    Findingown data

    Human experts noted they were not ready to change their existing property annotations based on the LLM-generated dimensions.

    “In the survey, the human experts noted that while they were not ready to change their existing property annotations based on the LLM-generated dimensions, they highlighted the utility of the auto-LLM recommendation service at the time of creating the structured summary descriptions.”
    Quote in the paperSubstantial claim

  • It may state two separate claims as one.

    Findingown data

    GPT-3.5 demonstrates the highest similarity to the ORKG properties, outperforming both Llama 2 and Mistral.

    “The embedding-based evaluations indicate that GPT-3.5 demonstrates the highest similarity to the ORKG properties, outperforming both Llama 2 and Mistral.”
    • Also stated in: § 5. Conclusions · plain
      “Third, our embedding-based evaluations showed that GPT-3.5 achieved the highest semantic similarity between ORKG properties and LLM-generated research dimensions, outperforming Mistral and Llama 2 in that order.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The manually curated ORKG properties are an appropriate ground truth for evaluating LLM-generated research dimensions.
    • Rests on an assumption: Cosine similarity values of approximately 0.76–0.86 indicate strong semantic alignment or overlap between ORKG properties and LLM-generated dimensions.

  • It may state two separate claims as one.

    Findingown data

    With SciBERT embeddings, the cosine similarity values were 0.84 for GPT-3.5, 0.57 for Llama 2, and 0.66 for Mistral.

    “With SciBERT embeddings, the cosine similarity values were 0.84 for GPT-3.5, 0.57 for Llama 2, and 0.66 for Mistral (Figure 8).”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The manually curated ORKG properties are an appropriate ground truth for evaluating LLM-generated research dimensions.
    • Rests on an assumption: Cosine similarity values of approximately 0.76–0.86 indicate strong semantic alignment or overlap between ORKG properties and LLM-generated dimensions.

  • It may state two separate claims as one.

    Findingown data

    The cosine similarity between ORKG properties and LLM-generated dimensions reached 0.84 for GPT-3.5, 0.79 for Mistral, and 0.76 for Llama 2 using SciNCL embeddings.

    “The results indicate a high degree of semantic similarity, with the cosine similarity between ORKG properties and the LLM-generated dimensions reaching 0.84 for GPT-3.5, 0.79 for Mistral, and 0.76 for Llama 2.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The manually curated ORKG properties are an appropriate ground truth for evaluating LLM-generated research dimensions.
    • Rests on an assumption: Cosine similarity values of approximately 0.76–0.86 indicate strong semantic alignment or overlap between ORKG properties and LLM-generated dimensions.

  • Probably not a substantial claim. It may state two separate claims as one.

    Findingown data

    Concerns regarding the specificity and alignment of the generated properties with research goals were noted, suggesting areas for further refinement.

    “However, concerns regarding the specificity and alignment of the generated properties with research goals were noted, suggesting areas for further refinement.”
    • Also stated in: § 5. Conclusions · hedged
      “Concerns regarding specificity and alignment with research goals were noted, suggesting areas for improvement.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The manually curated ORKG properties are an appropriate ground truth for evaluating LLM-generated research dimensions.
    • Rests on an assumption: The survey respondents' ratings are a valid measure of domain-expert assessment of relevance, alignment, and utility.

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Findingown data

    Participants raised concerns about the specificity of generated dimensions potentially diverging from the actual goal of the paper.

    “As such, participants raised concerns about the specificity of generated dimensions potentially diverging from the actual goal of the paper.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The manually curated ORKG properties are an appropriate ground truth for evaluating LLM-generated research dimensions.
    • Rests on an assumption: The survey respondents' ratings are a valid measure of domain-expert assessment of relevance, alignment, and utility.

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Findingown data

    Human experts highlighted the utility of the auto-LLM recommendation service at the time of creating structured summary descriptions.

    “In the survey, the human experts noted that while they were not ready to change their existing property annotations based on the LLM-generated dimensions, they highlighted the utility of the auto-LLM recommendation service at the time of creating the structured summary descriptions.”
    Quote in the paperNot a substantial claim

  • It may state two separate claims as one.

    Interpretationreasoning

    LLMs have the potential to serve as valuable tools for automated research metadata creation and the retrieval of related work.

    “This finding highlights the potential of LLMs to serve as valuable tools for automated research metadata creation and the retrieval of related work.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The manually curated ORKG properties are an appropriate ground truth for evaluating LLM-generated research dimensions.
    • Rests on an assumption: Cosine similarity values of approximately 0.76–0.86 indicate strong semantic alignment or overlap between ORKG properties and LLM-generated dimensions.

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Premisecitation

    GPT-3.5's number of parameters is not publicly disclosed.

    “Its number of parameters is not publicly disclosed [26].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [26] (2024, April 23). Introducing ChatGPT. Available online: https://openai.com/blog/chatgpt.

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Premisecitation

    Llama 2 is available free of charge for both research and commercial purposes.

    “Importantly, Llama 2 is available free of charge for both research and commercial purposes, with multiple parameter configurations available, including a 13-billion-parameter option.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [4] Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., and Bhosale, S. (2023). Llama 2: Open foundation and fine-tuned chat models. arXiv.

  • It may state two separate claims as one.

    Premisereasoning

    Common ORKG properties like 'research problem', 'method', 'data source', and others provide valuable information about a specific paper, but they cannot be used to comprehensively describe a research field as a whole.

    “Common ORKG properties like”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Premisecitation

    Llama 2 has a 4k context window.

    “Notably, Mistral has an 8k context window, which allows for a more complete understanding of context compared to the 4k context window of Llama 2 [28].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [28] Thakkar, H., and Manimaran, A. (2023, January 1–8). Comprehensive Examination of Instruction-Based Language Models: A Comparative Analysis of Mistral-7B and Llama-2-7B. Proceedings of the 2023 International Conference on Emerging Research in Computational Science (ICERCS), Coimbatore, India. source

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Premisecitation

    Mistral has a parameter count of 7.3 billion.

    “With a parameter count of 7.3 billion, Mistral demonstrates competitive performance on various benchmarks, often outperforming Llama 2 despite its smaller size [5].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [5] Jiang, A.Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D.S., Casas, D.d.l., Bressand, F., Lengyel, G., Lample, G., and Saulnier, L. (2023). Mistral 7B. arXiv.

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • It may state two separate claims as one.

    Interpretationown data

    There is some alignment between paper properties and research field dimensions, but there is also a significant amount of deviation, highlighting the difference between the concepts of structured papers' properties and research dimensions.

    “These results imply that, while there is some alignment between paper properties and research field dimensions, there is also a significant amount of deviation, highlighting the difference between the concepts of structured papers' properties and research dimensions.”
    • Also stated in: § 5. Conclusions · plain
      “First, our semantic alignment and deviation assessments revealed a moderate level of alignment between manually curated ORKG properties and LLM-generated research dimensions, accompanied by a higher degree of deviation.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: GPT-3.5's alignment and deviation ratings are reliable and valid measures of semantic alignment and deviation between the lists.

  • It may state two separate claims as one.

    Interpretationown data

    LLMs can generate a more diverse set of research dimensions than the ORKG properties, but with a lower degree of similarity.

    “These results suggest that LLMs can generate a more diverse set of research dimensions than the ORKG properties, but with a lower degree of similarity.”
    • Also stated in: § 5. Conclusions · plain
      “While LLMs can generate a more diverse set of research dimensions than ORKG properties, the degree of similarity is lower, highlighting the challenges in aligning LLM-generated dimensions with human-expert-curated properties.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: GPT-3.5's counts of similar values between lists are a valid measure of property-dimension mappings.

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Premisecitation

    Llama 2 features twice the context length of its predecessor, Llama 1.

    “Featuring twice the context length of its predecessor, Llama 1 [27], Llama 2 offers researchers and practitioners a versatile tool for working with NLP.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [27] Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., and Azhar, F. (2023). Llama: Open and efficient foundation language models. arXiv.

  • Probably not a substantial claim.

    Study design and conductnone given

    Due to hardware limitations, we were unable to test larger LLM models, restricting our evaluation to small and medium-sized models.

    “Secondly, due to hardware limitations, we were unable to test larger LLM models, restricting our evaluation to small and medium-sized models.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

    Premisecitation

    Llama 2 has multiple parameter configurations available, including a 13-billion-parameter option.

    “Importantly, Llama 2 is available free of charge for both research and commercial purposes, with multiple parameter configurations available, including a 13-billion-parameter option.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [4] Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., and Bhosale, S. (2023). Llama 2: Open foundation and fine-tuned chat models. arXiv.

  • It may state two separate claims as one.

    Interpretationown data

    The LLM-generated dimensions exhibit a strong correlation with the manually curated ORKG properties, signifying a substantial semantic overlap between the two.

    “These values suggest that the LLM-generated dimensions exhibit a strong correlation with the manually curated ORKG properties, signifying a substantial semantic overlap between the two.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The manually curated ORKG properties are an appropriate ground truth for evaluating LLM-generated research dimensions.
    • Rests on an assumption: Cosine similarity values of approximately 0.76–0.86 indicate strong semantic alignment or overlap between ORKG properties and LLM-generated dimensions.

  • Probably not a substantial claim. It may be too vague to check against evidence. It may state two separate claims as one.

    Interpretationreasoning

    While LLMs show promise as tools for automated research metadata creation and the retrieval of related work, further development is necessary to enhance their accuracy and relevance in this domain.

    “While LLMs show promise as tools for automated research metadata creation and the retrieval of related work, further development is necessary to enhance their accuracy and relevance in this domain.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The manually curated ORKG properties are an appropriate ground truth for evaluating LLM-generated research dimensions.
    • Rests on an assumption: Cosine similarity values of approximately 0.76–0.86 indicate strong semantic alignment or overlap between ORKG properties and LLM-generated dimensions.
    • Rests on an assumption: The survey respondents' ratings are a valid measure of domain-expert assessment of relevance, alignment, and utility.

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Premisecitation

    Mistral has an 8k context window.

    “Notably, Mistral has an 8k context window, which allows for a more complete understanding of context compared to the 4k context window of Llama 2 [28].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [28] Thakkar, H., and Manimaran, A. (2023, January 1–8). Comprehensive Examination of Instruction-Based Language Models: A Comparative Analysis of Mistral-7B and Llama-2-7B. Proceedings of the 2023 International Conference on Emerging Research in Computational Science (ICERCS), Coimbatore, India. source

  • It may state two separate claims as one.

    Interpretationown data

    There is some similarity between LLM-generated research dimensions and original ORKG structured contribution properties, but there is room for further alignment.

    “This indicates that, while there is some similarity between the two, there is room for further alignment.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The manually curated ORKG properties are an appropriate ground truth for evaluating LLM-generated research dimensions.
    • Rests on an assumption: The survey respondents' ratings are a valid measure of domain-expert assessment of relevance, alignment, and utility.

  • Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.

    Interpretationreasoning

    These concerns highlight the need for LLMs to better capture the nuances of research goals and objectives in order to generate more targeted and relevant suggestions.

    “These concerns highlight the need for LLMs to better capture the nuances of research goals and objectives in order to generate more targeted and relevant suggestions.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The manually curated ORKG properties are an appropriate ground truth for evaluating LLM-generated research dimensions.
    • Rests on an assumption: The survey respondents' ratings are a valid measure of domain-expert assessment of relevance, alignment, and utility.

  • Probably not a substantial claim.

    Premisenone given

    Research dimensions refer to the shared properties of a research question, rather than focus on an individual paper.

    “In contrast, research dimensions refer to the shared properties of a research question, rather than focus on an individual paper.”
    Quote in the paperSubstantial claim

  • It may state two separate claims as one.

    Interpretationown data

    Although the generated properties did not perfectly align with the original contributions, they still provided valuable suggestions that authors found potentially helpful in various aspects of creating structured contributions for the ORKG.

    “Although the generated properties did not perfectly align with the original contributions, they still provided valuable suggestions that authors found potentially helpful in various aspects of creating structured contributions for the ORKG.”
    • Also stated in: § 5. Conclusions · hedged
      “While the suggestions provided by LLMs were deemed potentially helpful in various aspects of creating structured contributions, concerns regarding specificity and alignment with research goals were noted, suggesting areas for improvement.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The manually curated ORKG properties are an appropriate ground truth for evaluating LLM-generated research dimensions.
    • Rests on an assumption: The survey respondents' ratings are a valid measure of domain-expert assessment of relevance, alignment, and utility.

  • It may state two separate claims as one.

    Interpretationown data

    While LLMs demonstrate some capacity to capture semantic similarities, there are notable differences between the concepts of structured paper properties and research dimensions.

    “While LLMs demonstrate some capacity to capture semantic similarities, there are notable differences between the concepts of structured paper properties and research dimensions.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: GPT-3.5's alignment and deviation ratings are reliable and valid measures of semantic alignment and deviation between the lists.
    • Rests on an assumption: The manually curated ORKG properties are an appropriate ground truth for evaluating LLM-generated research dimensions.

  • Probably not a substantial claim.

    Study design and conductreasoning

    The convergence of opposing alignment and deviation measures would provide strong evidence for the validity of the evaluation results.

    “The convergence of these opposing measures would provide strong evidence for the validity of our evaluation results.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: GPT-3.5's alignment and deviation ratings are reliable and valid measures of semantic alignment and deviation between the lists.

  • Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.

    Study design and conductreasoning

    The mapping count provides a more fine-grained insight into the relationships between the properties and dimensions.

    “The mapping count provides a more fine-grained insight into the relationships between the properties and dimensions.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: GPT-3.5's counts of similar values between lists are a valid measure of property-dimension mappings.

  • It may state two separate claims as one.

    Study design and conductown data

    Additional evaluations using SciBERT and SPECTER embeddings provided similar results, reinforcing the reliability of the conclusions.

    “To further validate our findings, we extended our evaluations using SciBERT and SPECTER embeddings in addition to SciNCL.These additional embeddings provided similar results, reinforcing the reliability of our conclusions.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The manually curated ORKG properties are an appropriate ground truth for evaluating LLM-generated research dimensions.
    • Rests on an assumption: Cosine similarity values of approximately 0.76–0.86 indicate strong semantic alignment or overlap between ORKG properties and LLM-generated dimensions.

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Premisecitation

    Mistral often outperforms Llama 2 despite its smaller size.

    “With a parameter count of 7.3 billion, Mistral demonstrates competitive performance on various benchmarks, often outperforming Llama 2 despite its smaller size [5].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [5] Jiang, A.Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D.S., Casas, D.d.l., Bressand, F., Lengyel, G., Lample, G., and Saulnier, L. (2023). Mistral 7B. arXiv.

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.

    Study design and conductnone given

    The LLMs used in this research are trained on a wide range of text, not exclusively the scientific literature, which may affect their ability to accurately generate research dimensions.

    “Firstly, the LLMs used in this research are trained on a wide range of text, not exclusively the scientific literature, which may affect their ability to accurately generate research dimensions.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. It may be too vague to check against evidence.

    Study design and conductnone given

    By combining alignment and deviation evaluations, we can gain a more nuanced understanding of the relationship between ORKG properties and LLM-generated research dimensions.

    “By combining these two evaluations, we can gain a more nuanced understanding of the relationship between the ORKG properties and LLM-generated research dimensions.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: GPT-3.5's alignment and deviation ratings are reliable and valid measures of semantic alignment and deviation between the lists.

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Premisecitation

    SciBERT was the first language model tailored for scientific texts.

    “SciBERT, introduced by Beltagy et al. [16], was the first language model tailored for scientific texts.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [16] Beltagy, I., Lo, K., and Cohan, A. (2019). SciBERT: A pretrained language model for scientific text. arXiv. source
39 of 39

  • The paper relies on this without support.

    Manual annotation of ORKG properties is a time-consuming process.

    “Specifically, the manual extraction of these salient properties of research or research comparison properties presents two significant challenges: (1) manual annotation is a time-consuming process, and (2) it introduces inconsistencies among human annotators, potentially leading to variations in interpretation and annotation.”

  • The paper relies on this without support.

    Manual annotation introduces inconsistencies among human annotators.

    “Specifically, the manual extraction of these salient properties of research or research comparison properties presents two significant challenges: (1) manual annotation is a time-consuming process, and (2) it introduces inconsistencies among human annotators, potentially leading to variations in interpretation and annotation.”

  • The paper relies on this without support.

    The predominant method in the ORKG for creating structured paper descriptions or research comparisons is manually performed by domain experts.

    “Notably, however, the predominant method in the ORKG for creating structured paper descriptions or research comparisons is manually performed by domain experts.”

  • The paper relies on this without support.

    ORKG properties are generically applicable across various contributions on the same problem, making structured paper descriptions comparable.

    “Furthermore, a distinguishing characteristic of the properties is that they are also generically applicable across various contributions on the same problem, thus making the structured paper descriptions comparable.”

  • The paper relies on this without support.

    An LLM's performance on a particular task is highly dependent on the quality of the prompt.

    “An LLM's performance on a particular task is highly dependent on the quality of the prompt.”

  • The paper relies on this without support.

    The three LLMs have comparable parameter counts.

    “To test the automated generation of research dimensions, we tested and compared the outputs of three different state-of-the-art LLMs with comparable parameter counts, namely, GPT-3.5, Llama 2, and Mistral.”

  • The paper relies on this without support.

    GPT-3.5's alignment and deviation ratings are reliable and valid measures of semantic alignment and deviation between the lists.

    “To further validate the accuracy of our alignment scores, we leveraged GPT-3.5 as an evaluator again but this time to generate semantic deviation scores.”
    “The convergence of these opposing measures would provide strong evidence for the validity of our evaluation results.”

  • The paper relies on this without support.

    The manually curated ORKG properties are an appropriate ground truth for evaluating LLM-generated research dimensions.

    “We conducted a survey to evaluate the utility of LLM-generated dimensions in the context of domain-expert-annotated ORKG properties.”
    “the similarity between ORKG properties and LLM-generated research dimensions.”

  • The paper relies on this without support.

    Cosine similarity values of approximately 0.76–0.86 indicate strong semantic alignment or overlap between ORKG properties and LLM-generated dimensions.

    “The results indicate a high degree of semantic similarity, with the cosine similarity between ORKG properties and the LLM-generated dimensions reaching 0.84 for GPT-3.5, 0.79 for Mistral, and 0.76 for Llama 2. These values suggest that the LLM-generated dimensions exhibit a strong correlation with the manually curated ORKG properties, signifying a substantial semantic overlap between the two.”

  • The paper relies on this without support.

    The survey respondents' ratings are a valid measure of domain-expert assessment of relevance, alignment, and utility.

    “The survey was designed to solicit the impressions of domain experts when shown their original annotated properties versus the research dimensions generated by GPT-3.5.”
    “At the end, we received 23 total responses to our survey, corresponding to 23 different papers.”

  • The paper relies on this without support.

    GPT-3.5's counts of similar values between lists are a valid measure of property-dimension mappings.

    “we instructed GPT-3.5 to identify the number of properties that exhibit similarity to individual research dimensions.”
    “The mapping count provides a more fine-grained insight into the relationships between the properties and dimensions.”
11 of 11

Citations

Cites

Flagged first, then by citations

Research works printed but not linked: 4

  1. Jiang, A.Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D.S., Casas, D.d.l., Bressand, F., Lengyel, G., Lample, G., and Saulnier, L. (2023). Mistral 7B. arXiv.
  2. Antu, S.A., Chen, H., and Richards, C.K. (2023, January 7). Using LLM (Large Language Model) to Improve Efficiency in Literature Review for Undergraduate Research. Proceedings of the Workshop on Empowering Education with LLMs-the Next-Gen Interface and Content Generation, Tokyo, Japan.
  3. Latif, E., Fang, L., Ma, P., and Zhai, X. (2023). Knowledge distillation of llm for education. arXiv.
  4. Radford, 2019, Language models are unsupervised multitask learners, OpenAI Blog, 1, 9

Cited by

Sorted by citations

Related

Metadata

Authors
Nechakhin, V., D’Souza, J., & Eger, S.
Affiliations
  • Leibniz University Hannover (DE)
  • Technische Informationsbibliothek (TIB) (DE)
  • University of Mannheim (DE)
Corresponding author
  • Nechakhin, Vladyslav
  • D’Souza, Jennifer
ORCID
Venue
Information
Publisher
MDPI AG
Published
2024-06-05
Volume
15
Issue
6
Pages
328
ISSN
2078-2489 (ISSN-L)
DOI
10.3390/info15060328
OpenAlex
W4399358184
Semantic Scholar
cddb18d8b29501e8b8398cc507998e001fda1fe0
Cited by
17
References
16
Citations by year
2026: 8 · 2025: 9
FWCI
1.84
Type
journal article
Language
en
Open access
Licence
Locations
  • Information (publishedVersion, cc-by)
  • MADOC (University of Mannheim) (acceptedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
  • German BMBF project SCINEXT (01lS22070, 819536, 460234259)
  • European Research Council for ScienceGRAPH (01lS22070, 819536, 460234259)
  • German DFG for NFDI4DataScience (01lS22070, 819536, 460234259)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Semantic Web and Ontologies
  • Physical Sciences › Computer Science › Artificial Intelligence › Advanced Text Analysis Techniques
Keywords
large language models, Open Research Knowledge Graph, LLM evaluation, cosine similarity, GPT-3.5, Llama 2, Mistral

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-11-09, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-11-09 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review