CoChat
Sign Up
Cited by 5Semantic Web and OntologiesBiomedical Text Mining and OntologiesScientific Computing and Data Management

The DOI matches its Crossref record; 51 of the 51 references it cites were found in a registry or an index; 21 of the 49 sources its claims cite were checked for support.

Provenance-driven nanopublications: representing source lineage and trust networks for multi-source assertions

Open access
Menotti, Laura, Marchesin, Stefano, Giachelle, Fabio, Silvello, Gianmaria·2025·International Journal on Digital Libraries
DOI 10.1007/s00799-025-00431-x
Open this paper in CoChat — free accountOpen PDF

Abstract

Nanopublishing is a paradigm enabling the representation of scientific claims in a distinctive, identifiable, citable, and reusable format, i.e., as a named graph. This approach can be applied to sentences extracted from scientific publications or triples within a Knowledge Base (KB). This way, one can track the provenance of assertions derived from a specific publication or database. However, nanopublications do not natively support multi-source scientific claims generated by aggregating different bodies of knowledge. This work extends the nanopublication model with knowledge provenance , capturing provenance information for assertions derived by an aggregation algorithm or a truth discovery process , e.g., an information extraction system aggregating several sources of knowledge to populate a Knowledge Base (KB). In these cases, provenance information cannot be attributed to a single source, but it is the result of an ensemble of evidence, that can comprehend supporting and conflicting pieces of evidence and truth values. Knowledge provenance is represented as a named graph following the PROV-K ontology, developed for the case. To show how knowledge provenance applies to a real-world scenario, we serialized gene expression-cancer associations generated by the Collaborative Oriented Relation Extraction (CORE) System. To demonstrate the value of trust relationships, we present a use case leveraging an existing scientific KB to construct a trust network employing three Large Language Model (LLM) agents. We analyzed the ability of LLMs to evaluate trustworthiness, exploiting techniques from KB accuracy estimation. We published 197, 511 assertions generated by the CORE system in the form of extended nanopublications with knowledge provenance. PROV-K also defines trust relationships between agents or between an agent and a proposition. Starting from these assertions, we leveraged external agents – namely, multiple LLMs – to assess their trusted truth value. Based on these values, we defined trust relationships between the agents and the facts, yielding an exemplar trust network comprising over 45,000 facts and four agents. The knowledge provenance graph allows the tracking of provenance for each piece of evidence contributing to the support or refutation of an assertion. To capture the semantics of the newly presented graph, we define the PROV-K ontology, designed to represent provenance information for multi-source assertions. The two use cases serve as a template to show how to serialize extended nanopublications and showcase the trust relationships’ capabilities.

Overview

AI snapshot

Written from the paper’s full text.

TL;DR

The paper extends the nanopublication model with an optional knowledge provenance graph and the PROV-K ontology to represent multi-source assertions, including supporting/conflicting evidence, reliability, and trust. It serializes 197,511 CoreKB gene expression-cancer associations as extended nanopublications and builds a trust network over 45,649 facts using three LLM agents plus a crowd agent.

Key findings

  • The extended nanopublication adds a fourth named graph, knowledge provenance, which is backward-compatible and optional; PROV-K extends PROV-O and models propositions, information sources, trust relationships, and truth values, including `supportedBy`, `conflictingWith`, `assignedCertaintyDegree`, and `certaintyDegree`.
  • 197,511 CoreKB GCSs were published as extended nanopublications: 156,172 reliable facts and 41,339 unreliable due to contrasting evidence. Gene class distribution: Biomarker 107,830; Oncogene 35,821; Tumor Suppressor Gene 12,521; Contrasting Evidence 41,339.
  • The trust network covers 45,649 reliable GCSs assessed by DeepSeek V3, GPT-4o mini, Meta Llama 3.1 405B-Instruct, and a crowd agent. Accuracy estimates were DeepSeek V3 0.75 [0.68, 0.80], GPT-4o mini 0.58 [0.51, 0.65], and Meta Llama 3.1 405B 0.76 [0.69, 0.81].
  • Trusted GCS counts were Llama 3.1 405B 9,793; Crowd Agent 9,124; DeepSeek V3 6,455; GPT-4o mini 2,875. Trust relationship frequency: 1,333 GCSs trusted by all four agents, 4,473 by three, 3,409 by two, 2,678 by one, and 33,756 by none.

Methods

  • Design: the paper proposes an extended nanopublication model with a knowledge provenance graph and defines the PROV-K ontology, then applies both in two use cases: serializing CoreKB facts and building a trust network.
  • Data: CoreKB contains 231,099 GCSs generated by CORE; after filtering out unreliable facts due to insufficient evidence, 197,511 GCSs were serialized. The trust use case uses a subset of 45,649 reliable GCSs.
  • Analysis: facts were serialized in TriG using an extended Python `nanopub` package. For trust, three LLMs received zero-shot prompts with the fact, supporting/conflicting sentence counts, and up to 15 sentences per side; a Simple Random Sample of 200 GCSs was manually annotated, accuracy was estimated with sample proportion and 95% credible intervals via aHPD, and a crowd agent aggregated LLM scores weighted by estimated accuracy. Trust relationships were defined using the third quartile of each agent’s trusted truth value distribution.

Limitations

  • Stated in the paper: LLMs overestimate factual correctness, struggle with isolated statements and less popular entities, can introduce bias from prompt phrasing/input/order, and are not yet suitable substitutes for human annotators. Budget constraints limited trusted truth values to 45,649 GCSs, and prompts truncate supporting/conflicting sentences to the first 15 when there are more.
  • Stated in the paper: only reliable GCSs were considered for trusted truth values, and LLM parameters were set empirically. The crowd aggregation formula multiplies probabilities, skewing scores low and capping the crowd agent’s maximum trusted truth value at 0.7.
  • Evident from the text: the trust network is presented as a synthetic/exemplar use case with three LLMs plus a crowd agent, and the manual evaluation relies on a sample of 200 GCSs rather than full human annotation.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1007/s00799-025-00431-x

Cites

51 of 51 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

21 of 49 cited sources checked · 11 supported · 1 flagged · accuracy not yet measured

  • Premisereasoning

    Aggregating evidence enables the CORE system to formulate a fact that reflects the most likely scientific consensus, thereby increasing its reliability.

    “Aggregating this evidence enables the system to formulate a fact that reflects the most likely scientific consensus, thereby increasing its reliability.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [9] Marchesin, S., Menotti, L., Giachelle, F., Silvello, G., Alonso, O.: Building a large gene expression-cancer knowledge base with limited human annotations. Database J. Biol. Databases Curation 2023 (2023) https://doi.org/10.1093/database/baad061 source

      The CORE system > Overview explains that CORE aggregates sentences about the same entities and assigns higher probabilities to aspect values that recur consistently. Reliability testing says it checks for sufficient, noncontradictory evidence before tagging facts as reliable. But that section also cautions that the resulting gene classes are not definitive; the paper does not establish that a tagged fact reflects the broader scientific consensus. (LLM, conf=0.84)

      Experimental — accuracy not yet measured
  • Premisereasoning

    The aHPD algorithm ensures reliability by dynamically selecting the most precise interval among candidates based on the annotated sample.

    “In this way, the aHPD algorithm ensures reliability by dynamically selecting the most precise interval among the candidates based on the annotated sample.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [44] Marchesin, S., Silvello, G.: Credible Intervals for Knowledge Graph Accuracy Estimation. Proc. ACM Manag. Data (SIGMOD) 3(3) (2025) https://doi.org/10.1145/3725279 source
  • Premisecitation

    LLMs struggle to represent accurately and reason over isolated and single statements.

    “LLMs struggle to represent accurately and reason over isolated and single statements [49].”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [49] Mruthyunjaya, V., Pezeshkpour, P., Hruschka, E., Bhutani, N.: Rethinking Language Models as Symbolic Knowledge Graphs (2023). https://arxiv.org/abs/2308.13676 source

      The Introduction reports low performance on the paper’s proposed benchmarks, and the Conclusion says language models do not fully capture knowledge-graph attributes. But the Introduction also says retrieving independent triples may be relatively simpler than capturing topological and semantic attributes. The paper therefore supports limitations in knowledge representation and reasoning, not the specific claim that models struggle with isolated, single statements. (LLM, conf=0.82)

      Experimental — accuracy not yet measured
  • Findingown data

    The authors present a use case leveraging an existing scientific KB to construct a trust network employing three LLM agents.

    “we construct a trust network comprising a subset of 45, 649 GCSs generated by CORE and three Large Language Model (LLM) agents.”
    Quote in the paperSubstantial claim
  • Findingown data

    The authors analyzed the ability of LLMs to evaluate trustworthiness, exploiting techniques from KB accuracy estimation.

    “We analyzed the ability of LLMs to evaluate trustworthiness, exploiting techniques from KB accuracy estimation.”
    • Also stated in: § 1 Introduction · plain
      “Trust relationships are first derived from external agents, namely LLMs, and their accuracy is evaluated (Subsection 5.1).”
    Quote in the paperSubstantial claim
  • Probably not a substantial claim. 66
  • It may state two separate claims as one. 35

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Premisecitation

    CoreKB stores over 230,000 gene expression-cancer associations automatically generated by the CORE system.

    “CoreKB stores over 230, 000 gene expression-cancer associations, automatically generated by the Collaborative Oriented Relation Extraction (CORE) system [9].”
    Quote in the paperNot a substantial claim2 of 2 sources checked
    • cites [8] Giachelle, F., Marchesin, S., Silvello, G., Alonso, O.: Searching for reliable facts over a medical knowledge base. In: Proc. of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2023), pp. 3205–3209. ACM, New York, NY, USA (2023). https://doi.org/10.1145/3539618.3591822 source

      The Introduction says CoreKB provides access to a knowledge base with more than 230,000 gene expression–cancer facts. Section 2.1 says automated methods generate facts during deployment, but also describes manual annotation during bootstrapping; the Introduction calls extraction semi-automatic. Our copy does not identify the construction system as “Collaborative Oriented Relation Extraction.” (LLM, conf=0.86)

      Experimental — accuracy not yet measured
    • cites [9] Marchesin, S., Menotti, L., Giachelle, F., Silvello, G., Alonso, O.: Building a large gene expression-cancer knowledge base with limited human annotations. Database J. Biol. Databases Curation 2023 (2023) https://doi.org/10.1093/database/baad061 source

      The Search platform section says COREKB provides access to the knowledge base generated by CORE. But Table 4 in Implementation and experiments > Knowledge base creation reports 153,016 facts in its largest version, KB3—not over 230,000 gene expression–cancer associations. (LLM, conf=0.88)

      Checked by a reasoning model — may be wrong.

      Experimental — accuracy not yet measured

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Premisecitation

    There are more than 10M nanopublications publicly accessible worldwide.

    “Overall, there are more than 10 M nanopublications publicly accessible worldwide [21].”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [21] Kuhn, T., Meroño-Peñuela, A., Malic, A., Poelen, J.H., Hurlbert, A.H., Ortiz, E.C., Furlong, L.I., Queralt-Rosinach, N., Chichester, C., Banda, J.M., Willighagen, E.L., Ehrhart, F., Evelo, C.T.A., Malas, T.B., Dumontier, M.: Nanopublications: A Growing Resource of Provenance-Centric Scientific Linked Data. In: Proc. of the 14th IEEE International Conference on e-Science (e-Science 2018), pp. 83–92. IEEE Computer Society, Washington, DC, USA (2018). https://doi.org/10.1109/eScience.2018.00024 source

      The indexed copy’s Abstract says that more than 10 million nanopublications have been published, but the audit instructions exclude the abstract as evidence. Searches for “10 million” and public accessibility found no eligible passage in the paper’s body establishing how many were publicly accessible worldwide. (LLM, conf=0.69)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Premisecitation

    CORE is a Knowledge Base Construction (KBC) system based on the combination of ML-based models and domain-expert feedback.

    “CORE is a Knowledge Base Construction (KBC) system based on the combination of ML-based models and domain-expert feedback [9].”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [9] Marchesin, S., Menotti, L., Giachelle, F., Silvello, G., Alonso, O.: Building a large gene expression-cancer knowledge base with limited human annotations. Database J. Biol. Databases Curation 2023 (2023) https://doi.org/10.1093/database/baad061 source

  • Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

    Premisecitation

    CORE harvests text from the literature, identifies sentences containing pairs of relevant entities, and extracts fine-grained aspects from them to generate gene expression-cancer associations that can be published as facts-defined as Gene-Cancer Status (GCS).

    “CORE harvests text from the literature, identifies sentences containing pairs of relevant entities, and extracts fine-grained aspects from them to generate gene expression-cancer associations that can be published as facts -defined as Gene-Cancer Status (GCS).”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [9] Marchesin, S., Menotti, L., Giachelle, F., Silvello, G., Alonso, O.: Building a large gene expression-cancer knowledge base with limited human annotations. Database J. Biol. Databases Curation 2023 (2023) https://doi.org/10.1093/database/baad061 source

      The CORE system > Overview (page unavailable) says CORE collects scientific literature, identifies sentences containing relevant entity pairs, and extracts aspects to produce facts for a knowledge base. Manual annotation and Relation extraction describe the gene expression–cancer aspects and how retained sentences populate that knowledge base. Our copy does not define those facts as “Gene-Cancer Status (GCS)”; it instead defines GCC as gene–cancer context and CCS as change of cancer status. (LLM, conf=0.88)

      Experimental — accuracy not yet measured

  • It may state two separate claims as one.

    Interpretationcitation

    The PROV-K ontology is built upon the PROV Ontology (PROV-O) and is grounded in the literature defining knowledge provenance.

    “The PROV-K ontology is built upon the PROV Ontology (PROV-O) and is grounded in the literature defining knowledge provenance [11][12][13].”
    Quote in the paperSubstantial claim1 of 3 sources checked
    • cites [11] Fox, M.S., Huang, J.: An Ontology for Static Knowledge Provenance. In: Proc. of the Knowledge Sharing in the Integrated Enterprise - Interoperability Strategies for the Enterprise Architect, 2004 International Conference on Enterprise Integration and Modelling Technology (ICEIMT 2004), The 7th International Conference on Design of Information Infrastructure Systems for Manufacturing, (DIISM 2004). IFIP, vol. 183, pp. 203–213. Springer, Heidelberg, Germany (2004). https://doi.org/10.1007/0-387-29766-9_17 source
    • cites [12] Huang, J., Fox, M.S.: Dynamic Knowledge Provenance. In: Proc. of the Business Agents and Semantic Web Workshop, pp. 372–387. National Research Council of Canada, Canada (2004). http://www.eil.utoronto.ca/wp-content/uploads/km/papers/huang-nrc04.pdf source
    • cites [13] Huang, J., Fox, M.S.: Uncertainty in Knowledge Provenance. In: Proc. of The Semantic Web: Research and Applications, First European Semantic Web Symposium (ESWS 2004). Lecture Notes in Computer Science, vol. 3053, pp. 372–387. Springer, Heidelberg, Germany (2004). https://doi.org/10.1007/978-3-540-25956-5_26 source

  • It may state two separate claims as one.

    Interpretationown data

    The trust network graph is available in Zenodo and can be used for analytics and to assess the trustworthiness of each GCS, as evaluated by external agents not involved in its creation.

    “This graph is available in Zenodo [16] and can be used for analytics and to assess the trustworthiness of each GCS, as evaluated by external agents not involved in its creation.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [16] Menotti, L., Marchesin, S., Giachelle, F., Silvello, G.: CoreKB Trust Network. Zenodo (2025). https://doi.org/10.5281/zenodo.15748152 source

  • Probably not a substantial claim.

    Premisecitation

    Nanopublications enable representation of scientific claims in a distinctive, identifiable, citable, and reusable format as named graphs.

    “Nanopublishing is a paradigm enabling the representation of scientific claims in a distinctive, identifiable, citable, and reusable format, i.e., as a named graph.”
    • Also stated in: § 2.1 Related work · plain
      “The nanopublication model uses Semantic Web technologies to represent scientific claims in a distinctive, identifiable, citable, and reusable format.”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites [3] Groth, P., Gibson, A., Velterop, J.: The anatomy of a nanopublication. Inf. Serv. Use 30(1–2), 51–56 (2010). https://doi.org/10.3233/ISU-2010-0613 source

      The Core model (section 2) describes nanopublications as identifiable and citable, and A realization as Named Graphs (section 3) supports representing them with Named Graphs. The claim’s singular “as a named graph” is imprecise: section 3 says the simplest nanopublication has two Named Graphs, one for the statement and one for its annotations. (LLM, conf=0.86)

      Experimental — accuracy not yet measured
    • cites [4] Fabris, E., Kuhn, T., Silvello, G.: A Framework for Citing Nanopublications. In: Proc. of the Digital Libraries for Open Knowledge - 23rd International Conference on Theory and Practice of Digital Libraries, TPDL 2019, Oslo, Norway, September 9-12, 2019. Lecture Notes in Computer Science, vol. 11799, pp. 70–83. Springer, Heidelberg, Germany (2019). https://doi.org/10.1007/978-3-030-30760-8_6 source

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Premisecitation

    The nanopublication model has been used to represent statements from different fields, especially in the life science domain.

    “The nanopublication model has been used to represent statements from different fields, especially in the life science domain.”
    Quote in the paperNot a substantial claim2 of 3 sources checked
    • cites [5] Chichester, C., Gaudet, P., Karch, O., Groth, P., Lane, L., Bairoch, A., Mons, B., Loizou, A.: Querying neXtProt nanopublications and their value for insights on sequence variants and tissue expression. J. Web Semant. 29, 3–11 (2014). https://doi.org/10.1016/j.websem.2014.05.001 source
    • cites [6] Waagmeester, A., Kutmon, M., Riutta, A., Miller, R.A., Willighagen, E.L., Evelo, C.T.A., Pico, A.R.: Using the Semantic Web for Rapid Integration of WikiPathways with Other Biological Online Data Resources. PLoS Comput. Biol. 12(6) (2016) https://doi.org/10.1371/journal.pcbi.1004989 source

      The Introduction describes representing WikiPathways content as triple-based statements about biological entities, and Materials and Methods describes RDF guidelines for biomedical data. Neither passage addresses the nanopublication model or its use across different fields; searches for “nanopublication” and related terms found no passage that does. (LLM, conf=0.76)

      Experimental — accuracy not yet measured
    • cites [7] Queralt-Rosinach, N., Kuhn, T., Chichester, C., Dumontier, M., Sanz, F., Furlong, L.I.: Publishing disgenet as nanopublications. Semantic Web 7(5), 519–528 (2016). https://doi.org/10.3233/SW-150189 source

      The Abstract and Introduction (page not provided) describe nanopublications as a model for scientific statements and document their use for DisGeNET’s biomedical gene–disease associations. The Applications section also mentions nanopublications for other life-science datasets. Our copy does not establish that the model had been used to represent statements across different fields beyond life sciences. (LLM, conf=0.83)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

    Premisecitation

    These intervals operate directly in the probabilistic space, thereby avoiding common interpretational issues associated with confidence intervals, and are well suited for one-shot settings.

    “These intervals operate directly in the probabilistic space, thereby avoiding the common interpretational issues associated with confidence intervals[47], emerging as well suited solutions for one-shot settings[44].”
    Quote in the paperNot a substantial claim2 of 2 sources checked
    • cites [47] Morey, R.D., Hoekstra, R., Rouder, J.N., Lee, M.D., Wagenmakers, E.J.: The fallacy of placing confidence in confidence intervals. Psychon. Bull. Rev. 23, 103–123 (2016). https://doi.org/10.3758/s13423-015-0947-8 source
    • cites [44] Marchesin, S., Silvello, G.: Credible Intervals for Knowledge Graph Accuracy Estimation. Proc. ACM Manag. Data (SIGMOD) 3(3) (2025) https://doi.org/10.1145/3725279 source

  • Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

    Premisecitation

    CoreKB contains information about 23,879 genes and 11,530 diseases for a total of more than 230,000 fine-grained facts supported by 1,037,845 sentences from 251,038 research articles.

    “CoreKB contains information about 23, 879 genes and 11, 530 diseases for a total of more than 230, 000 fine-grained facts supported by 1, 037, 845 sentences from 251, 038 research articles.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [8] Giachelle, F., Marchesin, S., Silvello, G., Alonso, O.: Searching for reliable facts over a medical knowledge base. In: Proc. of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2023), pp. 3205–3209. ACM, New York, NY, USA (2023). https://doi.org/10.1145/3539618.3591822 source

  • It may state two separate claims as one.

    Findingown data

    The paper serialized an RDF graph representing the trust network of 45,649 reliable GCSs assessed by LLMs and published it in Zenodo [16].

    “As a result, we serialize an RDF graph representing the trust network of the 45, 649 reliable GCSs assessed by LLMs and publish it in Zenodo [16].”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [16] Menotti, L., Marchesin, S., Giachelle, F., Silvello, G.: CoreKB Trust Network. Zenodo (2025). https://doi.org/10.5281/zenodo.15748152 source

  • Probably not a substantial claim.

    Premisecitation

    A CoreKB fact may be deemed unreliable if it fails one of the two reliability criteria defined by the CORE system: one sufficiency criterion (with two conditions) and one consistency criterion.

    “A CoreKB fact may be deemed unreliable if it fails one of the two reliability criteria defined by the CORE system: one sufficiency criterion (with two conditions) and one consistency criterion.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [9] Marchesin, S., Menotti, L., Giachelle, F., Silvello, G., Alonso, O.: Building a large gene expression-cancer knowledge base with limited human annotations. Database J. Biol. Databases Curation 2023 (2023) https://doi.org/10.1093/database/baad061 source

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Premisecitation

    A related work in fact verification has similarly employed multiple LLMs to evaluate the same factual statement, leveraging entropy-based measures as indicators of cross-model agreement and overall factual consistency.

    “A related work in fact verification has similarly employed multiple LLMs to evaluate the same factual statement, leveraging entropy-based measures as indicators of cross-model agreement and overall factual consistency [54].”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [54] Shami, F., Marchesin, S., Silvello, G.: Fact verification in knowledge graphs using llms. In: Proc. of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2025) (2025). https://doi.org/10.1145/3726302.3730142 source

      Section 3 (FactCheck; page unavailable) says four LLMs evaluate the correctness of the same knowledge-graph triple and combine their predictions by majority vote. It discusses model consistency when resolving ties, but does not describe entropy-based measures of cross-model agreement or factual consistency. (LLM, conf=0.88)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Premisecitation

    Each GCS is linked to the sentences supporting the fact and those conflicting with it.

    “In addition, each GCS is linked to the sentences supporting the fact, i.e., identifying the same gene class, and those conflicting with it.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [8] Giachelle, F., Marchesin, S., Silvello, G., Alonso, O.: Searching for reliable facts over a medical knowledge base. In: Proc. of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2023), pp. 3205–3209. ACM, New York, NY, USA (2023). https://doi.org/10.1145/3539618.3591822 source

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Premisecitation

    For each fact, CORE aggregates the probabilities of CGE, CCS, and GCI to assign a probability to three mutually exclusive gene classes: oncogene, tumor suppressor gene, and biomarker.

    “For each fact, CORE aggregates the probabilities of these three aspects to assign a probability to three mutually exclusive gene classes: oncogene, tumor suppressor gene, and biomarker.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [9] Marchesin, S., Menotti, L., Giachelle, F., Silvello, G., Alonso, O.: Building a large gene expression-cancer knowledge base with limited human annotations. Database J. Biol. Databases Curation 2023 (2023) https://doi.org/10.1093/database/baad061 source

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Findingcitation

    The authors published the serialized nanopublications in Zenodo for everyone to download in bulk.

    “We published the serialized nanopublications in Zenodo for everyone to download in bulk [15].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [15] Giachelle, F., Marchesin, S., Menotti, L., Silvello, G.: CORE Extended Nanopublications. Zenodo (2023). https://doi.org/10.5281/zenodo.10277210 source

      Checked before — its earlier verdict is held back until a new check runs.

  • Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

    Findingcitation

    The PROV-K ontology is available in Zenodo and its complete documentation is available at https://prov-k.dei.unipd.it/ontology/.

    “The PROV-K ontology is available in Zenodo [14], and its complete documentation is available at: https://prov-k.dei.unipd.it/ontology/.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [14] Menotti, L., Marchesin, S., Silvello, G.: The PROV-K Ontology for tracking provenance of multi-sourced assertions. Zenodo (2025). https://doi.org/10.5281/zenodo.15187371 source

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Findingown data

    The authors integrated the extended nanopublications into the CoreKB platform to enable their visualization and download.

    “Additionally, we integrated the extended nanopublications into the CoreKB platform to enable their visualization and download.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Findingnone given

    The authors released the source code for building the extended nanopublications as a template for future applications on different resources.

    “We released the source code for building the extended nanopublications as a template for future applications on different resources.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim.

    Findingown data

    Starting from the published assertions, the authors leveraged external agents-multiple LLMs-to assess their trusted truth value.

    “Starting from these assertions, we leveraged external agents -namely, multiple LLMs -to assess their trusted truth value.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim.

    Findingown data

    Trusted truth values were obtained from three LLMs for a subset of 45,649 GCSs.

    “Due to budget constraints, the trusted truth values are obtained from the three LLMs for a subset of 45, 649 GCSs.”
    Quote in the paperSubstantial claim

  • It may state two separate claims as one.

    Findingown data

    In the example GCS 'ANP32A is a tumor suppressor gene for malignant neoplasm of pancreas', DeepSeek V3 and GPT-4o mini both assign a probability of 0.1, while Meta Llama 3.1 405B-Instruct assigns 0.8.

    “For instance, in the example shown in Fig. 1 representing the fact”
    Quote in the paperNot a substantial claim

  • It may state two separate claims as one.

    Findingown data

    The first two GCSs in Table 3 have a high trusted truth value assigned by all agents, hence can be identified as reliable.

    “The first two GCSs have a high trusted truth value assigned by all agents, hence one can identify them as reliable.”
    Quote in the paperNot a substantial claim

  • It may state two separate claims as one.

    Findingown data

    DeepSeek V3 and Meta Llama 3.1 405B are more reliable than GPT-4o mini, with accuracy estimates around 0.75 versus 0.58.

    “The results indicate that DeepSeek V3 and Meta Llama 3.1 405B are more reliable, with accuracy estimates around 0.75, compared to GPT-4o mini, which yields a lower estimate of 0.58.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The manual correctness labels assigned to the 200 sampled GCSs are accurate.

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Findingown data

    For the last two GCSs in Table 3, all LLMs agree to a low truth score.

    “For the last two GCSs, all LLMs agree to a low truth score.”
    Quote in the paperNot a substantial claim

  • It may state two separate claims as one.

    Findingreasoning

    The authors published 197,511 GCSs from CoreKB as extended nanopublications, comprising 156,172 reliable facts and 41,339 unreliable ones due to contrasting evidence.

    “As a result, we published 197, 511 GCSs from CoreKB as extended nanopublications, accounting for 156, 172 reliable facts and 41, 339 unreliable ones due to contrasting evidence.”
    Quote in the paperNot a substantial claim

  • It may state two separate claims as one.

    Findingown data

    GCS ...846a66 has a high trusted truth value assigned by GPT-4o mini, DeepSeek V3 assigns 0.5 (borderline), and Llama 3.1 405B-Instruct gives a low score.

    “In particular, GCS . ..846a66 has a high trusted truth value assigned by GPT-4o mini, DeepSeek V3 assigns 0.5, which can be interpreted as borderline, while Llama 3.1 405B-Instruct gives a low score.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper may give no reason why it holds.

    Findingnone given

    The maximum trusted truth value for all LLMs is 1.0.

    “In this scenario, the maximum trusted truth value for all LLMs is 1.0.”
    Quote in the paperNot a substantial claim

  • It may state two separate claims as one.

    Findingown data

    1,333 GCSs are trusted by all LLMs, 4,473 GCSs are trusted by three LLMs, 3,409 GCSs are trusted by two LLMs, and 2,678 GCSs are trusted by one LLM.

    “As we can see1, 333GCSs are trusted by all LLMs, 4, 473 GCSs are trusted by three LLMs, 3, 409 GCSs are trusted by two LLMs, while 2, 678 GCSs are trusted by one LLM.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

    Findingcitation

    The PROV-K ontology is published in Zenodo [14] and its documentation is available at the given URL.

    “The PROV-K ontology is published in Zenodo [14] and the ontology documentation is available at: https://prov-k.dei. unipd.it/ontology/.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [14] Menotti, L., Marchesin, S., Silvello, G.: The PROV-K Ontology for tracking provenance of multi-sourced assertions. Zenodo (2025). https://doi.org/10.5281/zenodo.15187371 source

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Findingown data

    In the example of GCS 49b1e46252f16378e25e2407ee8ab17b, only Meta Llama 3.1 405-Instruct trusts the GCS.

    “In this example, only Meta Llama 3.1 405-Instruct trusts the GCS.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Findingown data

    Some GCSs have as many as 2,588 supporting sentences.

    “Although the average number of supporting sentences per GCS is 2.3, some GCSs have as many as 2, 588 supporting sentences.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Findingown data

    The average number of supporting sentences per GCS is 2.3.

    “Although the average number of supporting sentences per GCS is 2.3, some GCSs have as many as 2, 588 supporting sentences.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

    Findingown data

    Manual evaluation identifies the example GCS as highly reliable, so only the Llama output is marked as correct.

    “Manual evaluation identifies the GCS as highly reliable, so only the Llama output is marked as correct.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The manual correctness labels assigned to the 200 sampled GCSs are accurate.

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Findingown data

    The source code to build nanopublications following the extended model and to represent facts in CoreKB as nanopublications is published in the GitHub repository at the given URL.

    “The source code to build nanopublications following the extended model and to represent facts in CoreKB as nanopublications is published in the GitHub repository at: https://github.com/ mntlra/knowledgeProvenance.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The authors may not put it forward as true.

    Findingown data

    Under the quartile threshold, the majority of the GCSs were not trusted by any LLM.

    “Under the quartile threshold, we expected that the majority of the GCSs were not trusted by any LLM.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Findingcitation

    The trust network consisting of trust relationships between external agents and CoreKB facts enriched with trusted truth values is published in Zenodo [16].

    “The trust network consisting of trust relationships between external agents and CoreKB facts enriched with trusted truth values is published in Zenodo [16].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [16] Menotti, L., Marchesin, S., Giachelle, F., Silvello, G.: CoreKB Trust Network. Zenodo (2025). https://doi.org/10.5281/zenodo.15748152 source

      Checked before — its earlier verdict is held back until a new check runs.

  • It may state two separate claims as one.

    Findingown data

    GPT-4o mini assigns a trusted truth value of 0.85 to 23,196 GCSs, accounting for 51% of the entire dataset.

    “Notably, GPT-4o mini assigns a trusted truth value of 0.85 to 23,196 GCSs, accounting for 51% of the entire dataset.”
    Quote in the paperNot a substantial claim

  • It may state two separate claims as one.

    Findingown data

    The CoreKB facts serialized as extended nanopublications include 107,830 biomarker, 35,821 oncogene, 12,521 tumor suppressor gene, and 41,339 contrasting evidence nanopublications.

    “| Class | # of Nanopublications | | --- | --- | | Biomarker | 107,830 | | Oncogene | 35,821 | | Tumor Suppressor Gene | 12,521 | | Contrasting Evidence | 41,339 |”
    Quote in the paperSubstantial claim

  • It may state two separate claims as one.

    Findingreasoning

    The resulting thresholds for establishing trust relationships are 0.52 for the crowd agent, 0.7 for DeepSeek V3, 0.85 for GPT-4o mini, and 0.7 for Meta Llama 3.1 405-Instruct.

    “In practice, the resulting thresholds for establishing trust relationships are: 0.52 for the crowd agent, 0.7 for DeepSeek V3, 0.85 for GPT-4o mini, and 0.7 for Meta Llama 3.1 405-Instruct.”
    Quote in the paperNot a substantial claim

  • It may state two separate claims as one.

    Findingown data

    The Crowd agent trusts 9,124 GCSs, DeepSeek V3 trusts 6,455 GCSs, and GPT-4o mini trusts 2,875 GCSs.

    “The Crowd agent follows with 9, 124 trusted GCSs, DeepSeek V3 trusts 6, 455 GCSs, and GPT-4o mini only trusts 2, 875 GCSs.”
    Quote in the paperSubstantial claim

  • It may state two separate claims as one.

    Findingown data

    Both GPT-4o mini and DeepSeek V3 assign a very low reliability score to GCS ...0ef38b, while Llama 3.1 405B-Instruct assigns a score of 0.9.

    “On the other hand, both GPT-4o mini and DeepSeek V3 assigns a very low reliability score to GCS . ..0ef38b while Llama 3.1 405B-Instruct is almost certain the fact is true and assigns a score of 0.9.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Findingcitation

    All extended nanopublications derived from the facts in CoreKB are available in Zenodo [15].

    “All extended nanopublications derived from the facts in CoreKB are available in Zenodo [15].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [15] Giachelle, F., Marchesin, S., Menotti, L., Silvello, G.: CORE Extended Nanopublications. Zenodo (2023). https://doi.org/10.5281/zenodo.10277210 source

      Checked before — its earlier verdict is held back until a new check runs.

  • Probably not a substantial claim. It may be too vague to check against evidence.

    Interpretationreasoning

    It is necessary to extend the nanopublication model to track knowledge provenance-the provenance of the scientific fact itself, as inferred from multiple supporting assertions.

    “Therefore, it is necessary to extend the nanopublication model to track the knowledge provenance -the provenance of the scientific fact itself, as inferred from multiple supporting assertions.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

    Interpretationreasoning

    The two use cases serve as a template to show how to serialize extended nanopublications and showcase the trust relationships' capabilities.

    “The two use cases serve as a template to show how to serialize extended nanopublications and showcase the trust relationships' capabilities.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.

    Premisenone given

    To the authors' knowledge, nanopublications are the only data model that supports publishing at the granularity of individual statements with such detailed context, enabling provenance traceability, interoperability, and precise citation.

    “To the best of our knowledge, nanopublications are the only data model that supports publishing at the granularity of individual statements with such detailed context, enabling provenance traceability, interoperability, and precise citation.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim.

    Interpretationreasoning

    Knowledge provenance is represented as a named graph following the PROV-K ontology.

    “Knowledge provenance is represented as a named graph following the PROV-K ontology, developed for the case.”
    • Also stated in: § 1 Introduction · plain
      “the knowledge provenance is modeled as an additional named graph referring to the newly developed PROV-K ontology to represent the provenance information of web assertions derived from multiple sources of evidence.”
    Quote in the paperSubstantial claim

  • It may state two separate claims as one.

    Interpretationreasoning

    The extension remains fully compatible with the standard nanopublication model while enhancing it to support multi-source evidence aggregation and traceability.

    “This extension remains fully compatible with the standard nanopublication model while enhancing it to support multi-source evidence aggregation and traceability.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim.

    Premisereasoning

    A GCS is not based on a single claim from a single publication, but is synthesized from multiple sources, potentially containing both supporting and refuting evidence.

    “A GCS is not based on a single claim from a single publication, but rather synthesized from multiple sources, potentially containing both supporting and refuting evidence.”
    Quote in the paperSubstantial claim

  • It may state two separate claims as one.

    Premisereasoning

    PROV-O models provenance as in the provenance graph of the nanopublication model; therefore, it lacks representations for supporting and contradicting evidence and reliability scores.

    “While extensive in scope, the PROV-O models provenance as in the provenance graph of the nanopublication model; therefore, it lacks the representations for supporting and contradicting evidence and reliability scores.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim.

    Interpretationnone given

    PROV-K defines trust relationships between agents or between an agent and a proposition.

    “PROV-K also defines trust relationships between agents or between an agent and a proposition.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim.

    Interpretationreasoning

    The knowledge provenance graph represents the different sources that support or conflict with the given assertion.

    “This novel graph represents the different sources that support or conflict with the given assertion.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

    Premisecitation

    Little work has been done towards Knowledge Provenance, and it mostly focuses on providing a taxonomy of four levels of provenance based on the certainty degree of each assertion.

    “Little work has been done towards this end, and it mostly focuses on providing a taxonomy of four levels of provenance based on the certainty degree of each assertion [30].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [30] Fox, M.S., Huang, J.: Knowledge Provenance. In: Proc. of the Advances in Artificial Intelligence, 17th Conference of the Canadian Society for Computational Studies of Intelligence, (Canadian AI 2004). Lecture Notes in Computer Science, vol. 3060, pp. 517–523. Springer, Heidelberg, Germany (2004). https://doi.org/10.1007/978-3-540-24840-8_47 source

  • Probably not a substantial claim.

    Premisenone given

    A nanopublication's structure is purposefully designed to support the representation of one assertion from a single evidence source.

    “This structure is purposefully designed to support the representation of one assertion from a single evidence source.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim.

    Interpretationreasoning

    The authors define the PROV-K ontology, designed to represent provenance information for multi-source assertions.

    “To capture the semantics of the newly presented graph, we define the PROV-K ontology, designed to represent provenance information for multi-source assertions.”
    • Also stated in: § 1 Introduction · plain
      “The PROV-K ontology, representing provenance for multi-sourced assertions and describing their uncertainty”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim.

    Premisecitation

    Knowledge Provenance has been proposed to annotate the reliability of information extracted from web sources based on who created the assertion, how much the creator can be trusted, and what the information depends on.

    “Knowledge Provenance (KP) has been proposed to create an approach to annotate the reliability of information extracted from web sources based on who created the assertion, how much the creator can be trusted, and what the information depends on.”
    Quote in the paperSubstantial claim0 of 4 sources checked
    • cites [12] Huang, J., Fox, M.S.: Dynamic Knowledge Provenance. In: Proc. of the Business Agents and Semantic Web Workshop, pp. 372–387. National Research Council of Canada, Canada (2004). http://www.eil.utoronto.ca/wp-content/uploads/km/papers/huang-nrc04.pdf source
    • cites [13] Huang, J., Fox, M.S.: Uncertainty in Knowledge Provenance. In: Proc. of The Semantic Web: Research and Applications, First European Semantic Web Symposium (ESWS 2004). Lecture Notes in Computer Science, vol. 3053, pp. 372–387. Springer, Heidelberg, Germany (2004). https://doi.org/10.1007/978-3-540-25956-5_26 source
    • cites [30] Fox, M.S., Huang, J.: Knowledge Provenance. In: Proc. of the Advances in Artificial Intelligence, 17th Conference of the Canadian Society for Computational Studies of Intelligence, (Canadian AI 2004). Lecture Notes in Computer Science, vol. 3060, pp. 517–523. Springer, Heidelberg, Germany (2004). https://doi.org/10.1007/978-3-540-24840-8_47 source
    • cites [31] Huang, J., Fox, M.S.: Trust Judgment in Knowledge Provenance. In: Proc. of the 16th International Workshop on Database and Expert Systems Applications (DEXA 2005), pp. 524–528. IEEE Computer Society, Washington, DC, USA (2005). https://doi.org/10.1109/DEXA.2005.193 source

  • It may state two separate claims as one.

    Premisereasoning

    The standard nanopublication provenance graph identifies only the origin of a represented fact, such as a publication and extraction metadata, and cannot express that an assertion is derived from multiple, potentially conflicting sources.

    “Indeed, the provenance graph in a nanopublication specifies only that a given assertion originates from a particular publication, including metadata such as the extraction method, the publisher, venue, and year of publication.”
    • Also stated in: § 1 Introduction · plain
      “A standard nanopublication can only represent the fact as originating from a single source and lacks the capacity to express that the assertion is derived from multiple, potentially conflicting, sources.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim.

    Premisecitation

    These associations are derived by mining scientific literature and aggregating evidence from multiple documents to produce comprehensive, trustworthy scientific facts.

    “These associations are derived by mining scientific literature and aggregating evidence from multiple documents to produce comprehensive, trustworthy scientific facts.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [9] Marchesin, S., Menotti, L., Giachelle, F., Silvello, G., Alonso, O.: Building a large gene expression-cancer knowledge base with limited human annotations. Database J. Biol. Databases Curation 2023 (2023) https://doi.org/10.1093/database/baad061 source

      Checked before — its earlier verdict is held back until a new check runs.

  • Probably not a substantial claim.

    Interpretationreasoning

    One can include details about the reliability of each assertion, considering both the source of information and the extraction process.

    “Additionally, one can include details about the reliability of each assertion, considering both the source of information and the extraction process.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim.

    Study design and conductown data

    A sample of 200 GCSs was drawn using simple random sampling and manually annotated for correctness of LLM-generated truth values.

    “Following the above evaluation procedure, we draw a sample of 200 GCSs using SRS and manually annotate the correctness of the LLM-generated truth values.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The manual correctness labels assigned to the 200 sampled GCSs are accurate.

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Interpretationcitation

    Employing LLMs can introduce various forms of bias into their outputs.

    “In addition, employing LLMs can introduce various forms of bias into their outputs.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.

    Premisenone given

    Both the sample-based estimator and the aHPD algorithm operate independently of the population size.

    “It is worth noting that both the sample-based estimator and the aHPD algorithm operate independently of the population size.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Premisenone given

    The PROV-K ontology is designed to be extensible, allowing for the integration of more complex decision processes or use-case-specific trust relationships.

    “Note that the PROV-K ontology is designed to be extensible, allowing for the integration of more complex decision processes or use-case-specific trust relationships.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. It may state two separate claims as one.

    Interpretationreasoning

    The network can be queried to derive meaningful statistics, and the trust structure can be further analyzed to uncover additional insights.

    “The network can be queried to derive meaningful statistics, and the trust structure can be further analyzed to uncover additional insights.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim. It may be too vague to check against evidence.

    Interpretationreasoning

    Pinpointing these uncertain regions can enhance knowledge graph refinement by directing attention to areas that require further verification, thereby reducing the need for exhaustive manual annotation across the entire graph.

    “Pinpointing these uncertain regions can enhance knowledge graph refinement by directing attention to areas that require further verification, thereby reducing the need for exhaustive manual annotation across the entire graph.”
    Quote in the paperNot a substantial claim

  • It may state two separate claims as one.

    Interpretationreasoning

    By examining the underlying trust structure, one can identify assertions that are broadly accepted as true or offer a deeper understanding of how trust propagates through indirect relationships.

    “By examining the underlying trust structure, one can identify assertions that are broadly accepted as true or offer a deeper understanding of how trust propagates through indirect relationships.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim.

    Premisenone given

    The PROV-K ontology is a general resource designed to track provenance information for aggregated sources of evidence beyond the knowledge provenance graph of the nanopublication model.

    “the PROV-K ontology is a general resource designed to track provenance information for aggregated sources of evidence beyond the knowledge provenance graph of the nanopublication model.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Interpretationown data

    The trust network can be explored through SPARQL queries to provide useful network statistics.

    “This graph can be explored through SPARQL queries to provide useful network statistics.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Study design and conductnone given

    The same prompt was used for all LLMs.

    “The same prompt is used for all LLMs”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim.

    Interpretationown data

    The paper presented an additional use case for the PROV-K ontology by showcasing the role of trust relationships.

    “Finally, we presented an additional use case for the PROV-K ontology by showcasing the role of trust relationships.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim.

    Interpretationreasoning

    The paper proposed a novel component for the nanopublication model: the knowledge provenance graph.

    “To address this challenge, we proposed a novel component for the nanopublication model: the knowledge provenance graph.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Interpretationreasoning

    The PROV-K ontology has been designed as a more general resource capable of modeling provenance for aggregated sources of evidence beyond the nanopublication context.

    “While the primary use of PROV-K is to support the serialization of the knowledge provenance graph within nanopublications, the ontology has been designed as a more general resource capable of modeling provenance for aggregated sources of evidence beyond the nanopublication context.”
    Quote in the paperNot a substantial claim

  • It may state two separate claims as one.

    Interpretationreasoning

    By leveraging the structure of the trust network, one can identify facts that are widely accepted as well as contentious facts.

    “By leveraging the structure of the trust network, one can identify facts that are widely accepted -indicating a high degree of trust for all agentsas well as contentious facts, where there is significant disagreement among agents regarding their truth value.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Premisecitation

    The consistency test checks whether the difference between the probabilities of the fact being classified with the two gene classes with the highest likelihood is bigger than a threshold value β set to 0.4.

    “The consistency test instead checks whether the difference between the probabilities of the fact being classified with the two gene classes with the highest likelihood is bigger than a threshold value β set to 0.4.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [9] Marchesin, S., Menotti, L., Giachelle, F., Silvello, G., Alonso, O.: Building a large gene expression-cancer knowledge base with limited human annotations. Database J. Biol. Databases Curation 2023 (2023) https://doi.org/10.1093/database/baad061 source

      Checked before — its earlier verdict is held back until a new check runs.

  • Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.

    Premisenone given

    The formula ttv(LLM i )•acc(LLM i ) provides an entropy-inspired estimate of the reliability of the given LLM.

    “The formula ttv(LLM i )•acc(LLM i ) provides an entropy-inspired estimate of the reliability of the given LLM.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim.

    Interpretationnone given

    This work highlights the limitations of the standard nanopublication model when used to represent multi-source assertions supported by multiple, and possibly conflicting, pieces of evidence.

    “This work highlights the limitations of the standard nanopublication model when used to represent multi-source assertions supported by multiple, and possibly conflicting, pieces of evidence.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds. It may state two separate claims as one.

    Interpretationreasoning

    The overlap across all four LLMs is naturally limited, and universally trusted GCSs likely represent clear, unambiguous content.

    “Since the overlap across all four LLMs is naturally limited, and these universally trusted GCSs likely represent clear, unambiguous content.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Premisecitation

    Each GCS comprises information about the gene and disease involved, identified by NCBI Gene IDs and UMLS CUIs, together with the assigned gene class.

    “Each GCS comprises information about the gene and disease involved, which are identified by the National Center for Biotechnology Information (NCBI) Gene IDs and the Unified Medical Language System (UMLS) Concept Unique Identifiers (CUIs) respectively, together with the assigned gene class.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [8] Giachelle, F., Marchesin, S., Silvello, G., Alonso, O.: Searching for reliable facts over a medical knowledge base. In: Proc. of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2023), pp. 3205–3209. ACM, New York, NY, USA (2023). https://doi.org/10.1145/3539618.3591822 source

      Checked before — its earlier verdict is held back until a new check runs.

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Premisecitation

    LLM performance degrades when dealing with less popular entities (torso and tail entities).

    “Furthermore, their performance degrades when dealing with less popular entities, commonly referred to as torso and tail entities [50].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [50] Sun, K., Xu, Y., Zha, H., Liu, Y., Dong, X.L.: Head-to-tail: How knowledgeable are large language models (LLMs)? A.K.A. will LLMs replace knowledge graphs? In: Duh, K., Gomez, H., Bethard, S. (eds.) Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 311–325. Association for Computational Linguistics, Mexico City, Mexico (2024). https://doi.org/10.18653/v1/2024.naacl-long.18 . https://aclanthology.org/2024.naacl-long.18/ source

      Checked before — its earlier verdict is held back until a new check runs.

  • It may state two separate claims as one.

    Interpretationreasoning

    The sample represents a good approximation of the population, and the estimated accuracy reflects what would be obtained by evaluating the full dataset.

    “In other words, the sample represents a good approximation of the population, and the estimated accuracy reflects what would be obtained by evaluating the full dataset.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim.

    Premisereasoning

    The aggregation is weighted by the estimated accuracy of each LLM, ensuring that models with higher reliability exert greater influence on the final score.

    “The aggregation is weighted by the estimated accuracy of each LLM, ensuring that models with higher reliability exert greater influence on the final score.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim.

    Study design and conductnone given

    A sample of 200 GCSs was manually annotated to evaluate each LLM.

    “A sample of 200 GCSs is manually annotated to evaluate each LLM.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.

    Interpretationnone given

    The trust network can represent entropy-based fact assessments that evolve over time, which is crucial for capturing how trust in facts fluctuates with new evidence, shifting consensus, or changes in source credibility.

    “The trust network can represent entropy-based fact assessments that evolve over time, which is crucial for capturing how trust in facts fluctuates with new evidence, shifting consensus, or changes in source credibility.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim.

    Premisecitation

    The PROV-K ontology is an extension of PROV-O to represent provenance information of assertions derived from multiple sources using an aggregation algorithm.

    “To this end, we define the PROV-K ontology [14], an extension of PROV-O to represent provenance information of assertions derived from multiple sources using an aggregation algorithm.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [14] Menotti, L., Marchesin, S., Silvello, G.: The PROV-K Ontology for tracking provenance of multi-sourced assertions. Zenodo (2025). https://doi.org/10.5281/zenodo.15187371 source

  • It may state two separate claims as one.

    Interpretationown data

    The presented experiments provide additional evidence that LLMs overestimate factual correctness and are not yet suitable substitutes for human annotators.

    “The presented experiments align with prior research, providing additional evidence that LLMs overestimate factual correctness and are not yet suitable substitutes for human annotators.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The manual correctness labels assigned to the 200 sampled GCSs are accurate.
    • Rests on an assumption: The trusted truth values assigned by the LLMs can be interpreted as probabilities that the facts are true.
    • Rests on an assumption: The three selected LLMs are representative of LLMs for assigning trusted truth values to CoreKB facts.

  • It may state two separate claims as one.

    Interpretationreasoning

    The evaluation procedure remains valid and scalable even for large KBs.

    “Thus, the evaluation procedure remains valid and scalable even for large KBs, making the approach particularly efficient at scale.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Premisecitation

    The ordering of candidate responses has been shown to influence the model's predictions.

    “Notably, also the ordering of candidate responses has been shown to influence the model's predictions [53].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [53] Wang, P., Li, L., Chen, L., Cai, Z., Zhu, D., Lin, B., Cao, Y., Kong, L., Liu, Q., Liu, T., Sui, Z.: Large language models are not fair evaluators. In: Ku, L. W., Martins, A., Srikumar, V. (eds.) Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 9440–9450. Association for Computational Linguistics, Bangkok, Thailand (2024). https://doi.org/10.18653/v1/2024.acl-long.511 source

  • It may state two separate claims as one.

    Premisecitation

    PROV-K follows the theoretical guidelines provided by the Static KP ontology while integrating some elements from the Dynamic KP, such as timestamps for truth values and trust relationships.

    “The PROV-K follows the theoretical guidelines provided by the Static KP ontology [11] while integrating some elements from the Dynamic KP, such as timestamps for truth values and trust relationships [12].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [11] Fox, M.S., Huang, J.: An Ontology for Static Knowledge Provenance. In: Proc. of the Knowledge Sharing in the Integrated Enterprise - Interoperability Strategies for the Enterprise Architect, 2004 International Conference on Enterprise Integration and Modelling Technology (ICEIMT 2004), The 7th International Conference on Design of Information Infrastructure Systems for Manufacturing, (DIISM 2004). IFIP, vol. 183, pp. 203–213. Springer, Heidelberg, Germany (2004). https://doi.org/10.1007/0-387-29766-9_17 source

      Checked before — its earlier verdict is held back until a new check runs.

    • cites [12] Huang, J., Fox, M.S.: Dynamic Knowledge Provenance. In: Proc. of the Business Agents and Semantic Web Workshop, pp. 372–387. National Research Council of Canada, Canada (2004). http://www.eil.utoronto.ca/wp-content/uploads/km/papers/huang-nrc04.pdf source

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Premisecitation

    PROV-K includes the concepts of assigned certainty degree and certainty degree, originally defined in the Uncertainty-oriented KP.

    “We also include the concepts of assigned certainty degree and certainty degree, originally defined in the Uncertainty-oriented KP [13].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [13] Huang, J., Fox, M.S.: Uncertainty in Knowledge Provenance. In: Proc. of The Semantic Web: Research and Applications, First European Semantic Web Symposium (ESWS 2004). Lecture Notes in Computer Science, vol. 3053, pp. 372–387. Springer, Heidelberg, Germany (2004). https://doi.org/10.1007/978-3-540-25956-5_26 source
89 of 89

  • The paper relies on this without support.

    The manual correctness labels assigned to the 200 sampled GCSs are accurate.

    “Following the above evaluation procedure, we draw a sample of 200 GCSs using SRS and manually annotate the correctness of the LLM-generated truth values.”

  • The paper relies on this without support.

    The trusted truth values assigned by the LLMs can be interpreted as probabilities that the facts are true.

    “Different LLMs are employed as external agents to assign trusted truth values to each GCS, where the value represents the probability that a given fact is considered true by the respective LLM.”

  • The paper relies on this without support.

    The three selected LLMs are representative of LLMs for assigning trusted truth values to CoreKB facts.

    “We selected these three models primarily due to their availability and compatibility with our computational resources, which facilitated the integration into our experimental setup and enabled an efficient and reliable testing process.”

  • The paper relies on this without support.

    The manually annotated sample of 200 GCSs is representative of the 45,649 GCSs for estimating LLM accuracy.

    “A sample of 200 GCSs is manually annotated to evaluate each LLM.”
4 of 4

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Trust-relationship counts are described as LLM counts

    Minor

    The prose describes the frequency categories as numbers of LLMs trusting a GCS. The network has only three LLMs; its fourth agent is the crowd agent, so Table 5's four-relationship category cannot mean four LLMs, and its other categories need not equal the number of trusting LLMs.

1 of 1

Citations

Cites

Flagged first, then by citations

Research works printed but not linked: 3

  1. Giachelle, F., Marchesin, S., Menotti, L., Silvello, G.: Extending Nanopublications with Knowledge Provenance for Multi-Source Scientific Assertions. In: Proc. of the 21st Conference on Information and Research Science Connecting to Digital and Library Science (IRCDL 2025). CEUR-WS Proceedings, vol. 3937. CEUR-WS.org, Aachen, Germany (2025). https://ceur-ws.org/Vol-3937/paper10.pdf
  2. Huang, J., Fox, M.S.: Uncertainty in Knowledge Provenance. In: Proc. of The Semantic Web: Research and Applications, First European Semantic Web Symposium (ESWS 2004). Lecture Notes in Computer Science, vol. 3053, pp. 372–387. Springer, Heidelberg, Germany (2004). https://doi.org/10.1007/978-3-540-25956-5_26 · DOI · Paper page
  3. Cochran, W.G.: Sampling Techniques, 3rd edn. John Wiley & Sons, New York, NY, USA (1977)

Cited by

Sorted by citations

Related

Metadata

Authors
Menotti, L., Marchesin, S., Giachelle, F., & Silvello, G.
Affiliations
University of Padua (IT)
Corresponding author
Menotti, Laura
ORCID
Venue
International Journal on Digital Libraries
Publisher
Springer Science and Business Media LLC
Published
2025-10-24
Volume
26
Issue
4
ISSN
  • 1432-1300 (ISSN-L)
  • 1432-5012
DOI
10.1007/s00799-025-00431-x
OpenAlex
W4415527021
Cited by
5
References
46
Citations by year
2026: 4 · 2025: 1
FWCI
1.29
Type
journal article
Language
en
Open access
Licence
Locations
  • International Journal on Digital Libraries (publishedVersion, cc-by)
  • Padua Research Archive (University of Padova) (submittedVersion, cc-by)
Funders
HORIZON EUROPE Framework Programme (GA 101137074)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Semantic Web and Ontologies
  • Health Sciences › Medicine › Health Informatics › Biomedical Text Mining and Ontologies
  • Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
Keywords
nanopublications, knowledge provenance, trust networks, truth discovery, LLM agents, core system, named graphs, scientific claims

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-11-09, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-11-09 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review