CoChat
Sign Up
Cited by 0Topic ModelingBiomedical Text Mining and OntologiesAdvanced Text Analysis Techniques

Diagnosing the limits of large language models for structured scientific evidence extraction

Open access
Tan, Zhiyin, D’Souza, Jennifer, Auer, Sören·2026·International Journal on Digital Libraries
DOI 10.1007/s00799-026-00449-9
Open this paper in CoChat — free accountOpen PDF

Abstract

Scientific evidence is predominantly communicated through unstructured articles, while many downstream workflows require structured representations of study populations, variables, methods, and quantitative findings. Large language models (LLMs) offer a promising route toward automating this transformation, yet their reliability in constructing schema-constrained evidence records from parsed scientific full texts remains insufficiently understood. Evidence synthesis is a representative use case in which systematic reviews and meta-analyses require accurate, structured study information before quantitative integration can begin. We present a schema-grounded, tiered evaluation framework for structured scientific evidence extraction that increases task difficulty from single-entity identification to relational tuples and corpus-level statistical tasks. We evaluate frontier commercial and open-weight models across five scientific domains under a stricter multi-level assessment protocol, complemented by human validation of model-based judging and process-level inspection of intermediate reasoning for statistical tasks. Results reveal a clear pattern: models perform substantially better on low-complexity extraction tasks than on tasks requiring correct bindings among entities, numerical values, and methodological attributes. Intermediate-step analysis further indicates error amplification under exact downstream aggregation, where small upstream errors often prevent correct final outputs. Scaling model size yields uneven gains and does not remove these weaknesses. These findings show that recent advances have not resolved the fundamental challenges of structured scientific evidence extraction. We provide a reusable query suite and diagnostic evaluation framework to support future fine-grained benchmarks and more rigorous assessment of models for evidence-intensive scientific workflows.

AI snapshot

Written from the paper’s full text.

TL;DR

This paper presents a schema-grounded, tiered evaluation of large language models for structured scientific evidence extraction across five domains and two input regimes, finding that performance drops sharply from single-field extraction to multi-field binding and exact corpus-level aggregation. The authors conclude that current models remain closer to competent evidence retrieval than to reliable end-to-end structured reasoning.

Key findings

  • Single-field extraction is strongest: O1.x reaches 65.5 F1 and M1.x 64.0, while multi-field binding drops to O2.x 42.7 and M2.x 42.5 under medium matching.
  • Exact downstream statistical answers are much harder: OC Final Answer falls to 39.0 and MC Final Answer to 2.3, despite higher upstream scores such as OC Evidences 75.9, OC Final List 80.3, MC Evidences 50.7, and MC Final List 51.0.
  • Per-paper decomposition helps complex extraction: M2.x rises from 38.1 under Global to 48.3 under Per-paper, MC Evidences from 45.1 to 57.4, and MC Final List from 45.6 to 57.4; however, final answers do not consistently improve, with OC Final Answer at 45.0 Global vs 33.3 Per-paper and MC Final Answer at 2.2 vs 2.7.
  • Human validation of the LLM judge found high inter-annotator agreement (93.0%; Cohen’s κ = 0.906) and judge agreement with human consensus on 85% of cases (Cohen’s κ = 0.80).

Methods

  • Five domain-specific corpora of primary empirical association studies spanning engineering, health, agricultural, environmental, and social sciences, comprising 52 full-text articles total (9–11 per domain); gold annotations were created manually by a single expert annotator using a written codebook.
  • Five LLMs were evaluated: GPT-5.4, Gemma 4 31B-it, Qwen3.5-122B-A10B-FP8, Qwen3.5-9B, and MiniMax-M2.5, in text-only mode on MinerU markdown, temperature 0.0, without retrieval, tools, or code execution; two input regimes were compared: Global long-context and Per-paper decomposed.
  • The tiered query suite progressed from object-centric O1/O2 and method-centric M1/M2 extraction to corpus-level OC/MC statistical tasks; outputs were scored with tuple-level precision, recall, and F1 under strict, medium, and lenient matching, with an LLM judge adjudicating matches; the benchmark required approximately 20,956 LLM calls.

Limitations

  • The paper states that gold annotations came from a single expert annotator, carrying risks of annotation fatigue and idiosyncratic normalization choices; the human validation study validates the LLM judge, not the gold data, and is not a substitute for independent dual annotation.
  • The paper states that the five corpora comprise only 52 papers, a size driven by the Global regime’s context-window constraints, so the study is positioned as a diagnostic evaluation of failure modes rather than an exhaustive benchmark.
  • The paper states that models were evaluated bare, without retrieval, tools, or agentic orchestration, so results are a lower bound on composite systems; additionally, GPT-5.4 was run once due to API cost, so its run-to-run variance is not characterized.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1007/s00799-026-00449-9

Cites

57 of 57 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

30 of 41 cited sources checked · 17 supported · 0 flagged · accuracy not yet measured

  • Premisecitation

    Studies on compositional generalization show that models often struggle to combine familiar components into systematically correct novel structures.

    “Studies on compositional generalization show that models often struggle to combine familiar components into systematically correct novel structures [11,47,48].”
    Quote in the paperSubstantial claim3 of 3 sources checked
    • cites [11] Dziri, N., Lu, X., Sclar, M., Li, X.L., Jiang, L., Lin, B.Y., Welleck, S., West, P., Bhagavatula, C., Le Bras, R., Hwang, J., Sanyal, S., Ren, X., Ettinger, A., Harchaoui, Z., Choi, Y.: Faith and fate: limits of transformers on compositionality. In: Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., Levine, S. (eds.) Advances in Neural Information Processing Systems, vol. 36, pp. 70293–70332. Curran Associates Inc, Red Hook (2023) source
    • cites [47] Zhao, J., Tong, J., Mou, Y., Zhang, M., Zhang, Q., Huang, X.: Exploring the compositional deficiency of large language models in mathematical reasoning through trap problems. In: Al-Onaizan, Y., Bansal, M., Chen, Y.-N. (eds.) Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 16361–16376. Association for Computational Linguistics, Miami (2024). https://doi.org/10.18653/v1/2024.emnlp-main.915 source
    • cites [48] Shojaee, P., Mirzadeh, I., Alizadeh, K., Horton, M., Bengio, S., Farajtabar, M.: The illusion of thinking: understanding the strengths and limitations of reasoning models via the lens of problem complexity. In: The Thirty-ninth Annual Conference on Neural Information Processing Systems (2025). https://doi.org/10.48550/arXiv.2506.06941. NeurIPS 2025 source
  • Premisecitation

    In systematic reviews and related workflows, extracting study-level information such as populations, variables, sample sizes, measurement units, statistical methods, and reported effects often requires careful manual reading and verification across many articles.

    “In systematic reviews and related workflows, extracting study-level information such as populations, variables, sample sizes, measurement units, statistical methods, and reported effects often requires careful manual reading and verification across many articles [6].”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [6] Tsafnat, G., Glasziou, P., Choong, M.K., Dunn, A., Galgani, F., Coiera, E.: Systematic review automation technologies. Syst. Rev. 3(1), 74 (2014) source
  • Premisecitation

    Numerical reasoning introduces further difficulties, especially when values must be compared, computed, or correctly grounded to specific entities and variables.

    “Numerical reasoning introduces further difficulties, especially when values must be compared, computed, or correctly grounded to specific entities and variables [16-18, 51, 52].”
    Quote in the paperSubstantial claim5 of 5 sources checked
    • cites [16] Akhtar, M., Shankarampeta, A., Gupta, V., Patil, A., Cocarascu, O., Simperl, E.: Exploring the numerical reasoning capabilities of language models: a comprehensive analysis on tabular data. In: Bouamor, H., Pino, J., Bali, K. (eds.) Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 15391–15405. Association for Computational Linguistics, Singapore (2023). https://doi.org/10.18653/v1/2023.findings-emnlp.1028 source
    • cites [17] Ahmed, K., Osama, M., Sharif, O., Hossain, E., Hoque, M.M.: BenNumEval: A benchmark to assess LLMs’ numerical reasoning capabilities in Bengali. In: Che, W., Nabende, J., Shutova, E., Pilehvar, M.T. (eds.) Findings of the Association for Computational Linguistics: ACL 2025, pp. 17782–17799. Association for Computational Linguistics, Vienna (2025). https://doi.org/10.18653/v1/2025.findings-acl.915 source
    • cites [18] Oyesanmi, F., Olukanmi, P.O.: How good are large language models at arithmetic reasoning in low-resource language settings?—a study on yorùbá numerical probes with minimal contamination. Appl. Sci. 15(8), 4459 (2025) source

      Section 4.1 reports difficulties with computations and with representing Yorùbá numerals accurately; it says low numerical-representation accuracy affected the models’ final computations. The passages do not address comparing values or grounding values to specific entities and variables, so those parts of the citing sentence are not supported by this source. (LLM, conf=0.83)

      Experimental — accuracy not yet measured
    • cites [51] Boye, J., Moell, B.: Large Language Models and Mathematical Reasoning Failures. arXiv:2502.11574 (2025) source
    • cites [52] Yan, Y., Lu, Y., Xu, R., Lan, Z.: Do large language models truly grasp addition? A rule-focused diagnostic using two-integer arithmetic (2025) source

      The Abstract and Conclusion report difficulties with two-integer addition, supporting the claim about computing values. The Limitations section says the synthetic tasks strip away contextual cues; the paper does not establish difficulties with comparing values or grounding them to specific entities and variables. (LLM, conf=0.87)

      Experimental — accuracy not yet measured
  • Premisecitation

    Long-context evaluations report degradation when relevant evidence is distributed across extended inputs, including reduced attention to information located in the middle of long contexts.

    “Long-context evaluations similarly report degradation when relevant evidence is distributed across extended inputs, including reduced attention to information located in the middle of long contexts [15].”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [15] Liu, N.F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., Liang, P.: Lost in the middle: How language models use long contexts. Trans. Assoc. Comput. Linguist. 12, 157–173 (2024). https://doi.org/10.1162/tacl_a_00638 source
  • Premisecitation

    Multi-document and multimodal scientific reasoning benchmarks report substantial gaps between current systems and expert performance when evidence must be integrated across papers or modalities.

    “Multi-document and multimodal scientific reasoning benchmarks further report substantial gaps between current systems and expert performance when evidence must be integrated across papers or modalities [31][32][33].”
    Quote in the paperSubstantial claim3 of 3 sources checked
    • cites [31] Li, C., Shangguan, Z., Zhao, Y., Li, D., Liu, Y., Cohan, A.: M3SciQA: a multi-modal multi-document scientific QA benchmark for evaluating foundation models. In: Al-Onaizan, Y., Bansal, M., Chen, Y.-N. (eds.) Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 15419–15446. Association for Computational Linguistics, Miami, Florida (2024). https://doi.org/10.18653/v1/2024.findings-emnlp.904 source
    • cites [32] Park, J., Pyeon, S., Kim, J., Cabal, R.C., Ding, Y., Han, S.C.: DocHop-QA: Towards Multi-Hop Reasoning over Multimodal Document Collections (2025) source
    • cites [33] Tan, Z., Duan, C.: Multi-disciplinary dataset discovery from citation-verified literature contexts. In: Proceedings of the 2025 ACM/IEEE Joint Conference on Digital Libraries (JCDL), pp. 109–118. IEEE Computer Society, Los Alamitos (2025). https://doi.org/10.1109/JCDL67857.2025.00022 source

      The Results and Analysis section compares dataset-discovery systems on recall and other discovery metrics, while the Expert Evaluation Protocol assesses the usefulness of datasets they surface. Searches for multi-document or multimodal scientific reasoning benchmarks and gaps between systems and experts found no passage addressing that claim in our copy; page numbers were unavailable. (LLM, conf=0.77)

      Experimental — accuracy not yet measured
  • Classified as probably not a substantial claim. 36
  • May combine two claims. 55

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Recent systems have shown strong general-purpose language capabilities and encouraging results in scientific text mining, information extraction, and structured generation.

    “Recent systems have shown strong general-purpose language capabilities and encouraging results in scientific text mining, information extraction, and structured generation [7][8][9][10].”
    Quote in the paperNot a substantial claim4 of 4 sources checked
    • cites [7] Shamsabadi, M., D’Souza, J., Auer, S.: Large language models for scientific information extraction: An empirical study for virology. In: Findings of the Association for Computational Linguistics: EACL 2024, pp. 374–392 (2024) source
    • cites [8] Dagdelen, J., Dunn, A., Lee, S., Walker, N., Rosen, A.S., Ceder, G., Persson, K.A., Jain, A.: Structured information extraction from scientific text with large language models. Nat. Commun. 15(1), 1418 (2024) source
    • cites [9] Lu, W., Luu, R.K., Buehler, M.J.: Fine-tuning large language models for domain adaptation: exploration of training strategies, scaling, model merging and synergistic capabilities. nPJ Comput. Mater. 11, 84 (2025). https://doi.org/10.1038/s41524-025-01564-y source
    • cites [10] D’Souza, J., Laubach, Z., Al Mustafa, T., Zarrieß, S., Frühstückl, R., Illari, P.: Mining for species, locations, habitats, and ecosystems from scientific papers in invasion biology: A large-scale exploratory study with large language models. In: Proceedings of the 1st Workshop on Ecology, Environment, and Natural Language Processing (NLP4Ecology2025), pp. 16–23 (2025) source

      The Introduction describes using general-purpose LLMs without domain-specific fine-tuning for scientific information extraction and reports structured information extracted from invasion-biology abstracts. It characterizes the results as showing both promise and limitations, but does not establish strong general-purpose language capabilities beyond this task. (LLM, conf=0.82)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Scholarly knowledge platforms increasingly rely on explicit entities and relations to support comparison, retrieval, and reuse.

    “Scholarly knowledge platforms increasingly rely on explicit entities and relations to support comparison, retrieval, and reuse [3][4][5].”
    Quote in the paperNot a substantial claim2 of 3 sources checked
    • cites [3] Auer, S., D’Souza, J., Farfar, K.E., Jaradeh, M.Y., Jiomekong, A., Karras, O., Oelen, A., Snyder, L., Stocker, M., Vogt, L.: Open research knowledge graph: a large-scale neuro-symbolic knowledge organization system. In: Handbook on Neurosymbolic AI and Knowledge Graphs, pp. 385–420. IOS Press, Amsterdam (2025) source
    • cites [4] Ammar, W., Groeneveld, D., Bhagavatula, C., Beltagy, I., Crawford, M., Downey, D., Dunkelberger, J., Elgohary, A., Feldman, S., Ha, V., Kinney, R., Kohlmeier, S., Lo, K., Murray, T., Ooi, H.-H., Peters, M., Power, J., Skjonsberg, S., Wang, L.L., Wilhelm, C., Yuan, Z., Zuylen, M., Etzioni, O.: Construction of the literature graph in semantic scholar. In: Bangalore, S., Chu-Carroll, J., Li, Y. (eds.) Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 3 (Industry Papers), pp. 84–91. Association for source

      The Introduction (page unavailable) describes Semantic Scholar’s literature graph as structured information extracted from scientific documents and says it can improve academic search and answer queries. The entity–entity relations section also describes explicit relations in that graph. The paper supports retrieval through one platform, but does not establish that scholarly platforms increasingly rely on this approach or that it supports comparison and reuse. (LLM, conf=0.84)

      Experimental — accuracy not yet measured
    • cites [5] Dessí, D., Osborne, F., Buscaldi, D.: Cs-kg 2.0: a large-scale knowledge graph of computer science. Sci. Data 12, 964 (2025). https://doi.org/10.1038/s41597-025-05200-8 source

  • Classified as probably not a substantial claim.

    Premisecitation

    In systematic reviews and meta-analyses, information extraction is a central step in which reviewers convert study reports into structured records.

    “In systematic reviews and meta-analyses, information extraction, e.g., data or evidence abstraction, is a central step in which reviewers convert study reports into structured records [1,34].”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites [1] Cochrane Handbook for Systematic Reviews of Interventions. Wiley (2019). https://doi.org/10.1002/9781119536604 source
    • cites [34] Schmidt, L., Finnerty Mutlu, A.N., Elmore, R., : Data extraction methods for systematic review (semi)automation: Update of a living systematic review. F1000Research 10, 401 (2025) https://doi.org/10.12688/f1000research.51117.3 . Version 3; peer review: 3 approved source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Other work reports failures in reasoning fidelity and intermediate consistency, where fluent outputs may mask unstable internal reasoning processes.

    “Other work reports failures in reasoning fidelity and intermediate consistency, where fluent outputs may mask unstable internal reasoning processes [49,50].”
    Quote in the paperNot a substantial claim1 of 2 sources checked
    • cites [49] Song, P., Han, P., Goodman, N.: A survey on large language model reasoning failures. In: 2nd AI for Math Workshop @ ICML 2025 (2025)

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [50] Mahowald, K., Ivanova, A.A., Blank, I.A., Kanwisher, N., Tenenbaum, J.B., Fedorenko, E.: Dissociating language and thought in large language models. Trends Cogn. Sci. 28(6), 517–540 (2024). https://doi.org/10.1016/j.tics.2024.01.011 source

      The Formal Reasoning section reports failures on multi-step reasoning tasks and says that generating intermediate computation steps does not make results foolproof. The language-thought conflation section also warns that coherent text can be mistaken for evidence of reasoning ability. Neither passage establishes failures of reasoning fidelity or intermediate consistency, or describes unstable internal reasoning processes; page numbers are unavailable in our copy. (LLM, conf=0.85)

      Experimental — accuracy not yet measured

  • Classified as probably not a substantial claim.

    Premisecitation

    Prior work reports position-dependent degradation in long-context models, where information distant from the query is used less effectively.

    “First, prior work reports position-dependent degradation in long-context models, where information distant from the query is used less effectively [15].”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [15] Liu, N.F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., Liang, P.: Lost in the middle: How language models use long contexts. Trans. Assoc. Comput. Linguist. 12, 157–173 (2024). https://doi.org/10.1162/tacl_a_00638 source

      The Introduction and §2.3 report position-dependent degradation, particularly for relevant information in the middle of a long context. But §4.2 says the experiments place the query after the data, while §2.3 finds performance is often highest for information at the beginning or end. The paper therefore does not support a general claim that information farther from the query is used less effectively. (LLM, conf=0.88)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Document-level benchmarks such as SciREX emphasize that important scientific entities and relations are often distributed across sections and must be assembled into coherent, structured records.

    “Document-level benchmarks such as SciREX emphasize that important scientific entities and relations are often distributed across sections and must be assembled into coherent, structured records [28].”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [28] Jain, S., Zuylen, M., Hajishirzi, H., Beltagy, I.: SciREX: a challenge dataset for document-level information extraction. In: Jurafsky, D., Chai, J., Schluter, N., Tetreault, J. (eds.) Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7506–7516. Association for Computational Linguistics (2020). https://doi.org/10.18653/v1/2020.acl-main.670 source

  • May combine two claims.

    Premisereasoning

    Limitations in compositionality, entity binding, long-context evidence use, and numerical grounding are especially relevant, but are rarely examined within one unified setting.

    “Limitations in compositionality, entity binding, long-context evidence use, and numerical grounding are therefore especially relevant, but are rarely examined within one unified setting [11][12][13][14][15][16][17][18].”
    Quote in the paperSubstantial claim6 of 8 sources checked
    • cites [11] Dziri, N., Lu, X., Sclar, M., Li, X.L., Jiang, L., Lin, B.Y., Welleck, S., West, P., Bhagavatula, C., Le Bras, R., Hwang, J., Sanyal, S., Ren, X., Ettinger, A., Harchaoui, Z., Choi, Y.: Faith and fate: limits of transformers on compositionality. In: Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., Levine, S. (eds.) Advances in Neural Information Processing Systems, vol. 36, pp. 70293–70332. Curran Associates Inc, Red Hook (2023) source

      The Introduction and Discussion describe transformer limitations in compositional reasoning, including multiplication and logic-grid puzzles. The paper does not establish the broader claim about long-context evidence use or that these limitations are rarely examined together in one unified setting; its multiplication task also does not by itself establish a limitation in numerical grounding. (LLM, conf=0.84)

      Experimental — accuracy not yet measured
    • cites [12] Ismayilzada, M., Circi, D., Sälevä, J., Sirin, H., Köksal, A., Dhingra, B., Bosselut, A., Ataman, D., Plas, L.V.D.: Evaluating morphological compositional generalization in large language models. In: Chiruzzo, L., Ritter, A., Wang, L. (eds.) Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 1270–1305. Association for Computational Linguistics, Albuquerque (2025). https://doi.org/10.18653/v1/2025.naacl-long.59 source

      The Introduction reports limitations in LLMs’ morphological compositional generalization, and §2.1 describes compositional generalization as challenging. The paper does not address entity binding, long-context evidence use, or numerical grounding, and it does not establish that these four limitations are rarely examined in one unified setting. Page numbers are not available in our copy. (LLM, conf=0.87)

      Experimental — accuracy not yet measured
    • cites [13] Feng, J., Steinhardt, J.: How do language models bind entities in context? In: The Twelfth International Conference on Learning Representations (2024)

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [14] Gur-Arieh, Y., Geva, M., Geiger, A.: Mixing mechanisms: how language models retrieve bound entities in-context (2025)

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [15] Liu, N.F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., Liang, P.: Lost in the middle: How language models use long contexts. Trans. Assoc. Comput. Linguist. 12, 157–173 (2024). https://doi.org/10.1162/tacl_a_00638 source
    • cites [16] Akhtar, M., Shankarampeta, A., Gupta, V., Patil, A., Cocarascu, O., Simperl, E.: Exploring the numerical reasoning capabilities of language models: a comprehensive analysis on tabular data. In: Bouamor, H., Pino, J., Bali, K. (eds.) Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 15391–15405. Association for Computational Linguistics, Singapore (2023). https://doi.org/10.18653/v1/2023.findings-emnlp.1028 source

      The Introduction says earlier benchmarks focused on a few numerical-reasoning types, and this paper evaluates a wider range within one framework. That supports the numerical-grounding and limited-prior-coverage parts of the claim. The cited passages do not establish that compositionality, entity binding, and long-context evidence use are rarely examined together with numerical grounding. (LLM, conf=0.86)

      Experimental — accuracy not yet measured
    • cites [17] Ahmed, K., Osama, M., Sharif, O., Hossain, E., Hoque, M.M.: BenNumEval: A benchmark to assess LLMs’ numerical reasoning capabilities in Bengali. In: Che, W., Nabende, J., Shutova, E., Pilehvar, M.T. (eds.) Findings of the Association for Computational Linguistics: ACL 2025, pp. 17782–17799. Association for Computational Linguistics, Vienna (2025). https://doi.org/10.18653/v1/2025.findings-acl.915 source

      The Introduction describes LLM difficulties with multistep numerical reasoning, and the Error Analysis reports calculation errors, supporting the numerical-grounding part of the claim. The cited paper does not establish limitations in entity binding or long-context evidence use, or show that all four limitations are rarely examined in one unified setting. The indexed passages provide section labels but no page numbers. (LLM, conf=0.85)

      Experimental — accuracy not yet measured
    • cites [18] Oyesanmi, F., Olukanmi, P.O.: How good are large language models at arithmetic reasoning in low-resource language settings?—a study on yorùbá numerical probes with minimal contamination. Appl. Sci. 15(8), 4459 (2025) source

      Sections 3.1 and 4.1 examine combined arithmetic operations, step-by-step reasoning, and accurate representation of Yorùbá numerals, supporting the compositional-reasoning and numerical-grounding parts of the claim. The paper does not establish limitations in entity binding or long-context evidence use, or say that these limitations are rarely examined together. (LLM, conf=0.86)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Findingown data

    Qwen 3.5-9B on M2.x under Per-paper reaches 57.1 precision but only 39.6 recall.

    “For example, Qwen 3.5-9B on M2.x under Per-paper reaches 57.1 precision but only 39.6 recall”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    GPT-5.4 usually maintains smaller precision-recall gaps, even on harder tasks, whereas several open-source models show stronger one-sided weaknesses in selected settings.

    “GPT-5.4 usually maintains smaller precision-recall gaps, even on harder tasks, whereas several open-source models show stronger one-sided weaknesses in selected settings.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    M1.x increases from 56.1 to 64.0 to 66.9, while M2.x rises from 35.8 to 42.5 to 47.2 under strict, medium, and lenient scoring.

    “M1.x increases from 56.1 to 64.0 to 66.9, while M2.x rises from 35.8 to 42.5 to 47.2.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Object-centric multi-step tasks show the same tendency: OC Evidences improves from 72.9 to 79.8, and OC Final List from 78.7 to 83.1.

    “Object-centric multi-step tasks show the same tendency: OC Evidences improves from 72.9 to 79.8, and OC Final List from 78.7 to 83.1.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Qwen 3.5-9B improves on M2.x from 20.3 to 42.6, and Qwen 3.5-122B improves on OC Final List from 79.5 to 84.9.

    “Qwen 3.5-9B improves on M2.x from 20.3 to 42.6, and Qwen 3.5-122B improves on OC Final List from 79.5 to 84.9.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    For O2.x, average F1 rises from 20.7 under strict scoring to 42.7 under medium scoring, and further to 71.3 under lenient scoring.

    “For O2.x, average F1 rises from 20.7 under strict scoring to 42.7 under medium scoring, and further to 71.3 under lenient scoring.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Simple field extraction is relatively balanced; method-field extraction more often overgenerates; object and relation sets more often underextract; and Per-paper frequently trades higher recall for lower precision on the hardest tasks.

    “Simple field extraction is relatively balanced; method-field extraction more often overgenerates; object and relation sets more often underextract; and Perpaper frequently trades higher recall for lower precision on the hardest tasks.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Similar and even larger gains appear for MC Evidences (45.1 to 57.4) and MC Final List (45.6 to 57.4).

    “Similar and even larger gains appear for MC Evidences (45.1 to 57.4) and MC Final List (45.6 to 57.4).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The benchmark requires approximately 21,000 LLM calls (20,956 in total).

    “Across all settings, the benchmark requires approximately 21,000 LLM calls (20,956 in total, consisting of 1,690 Global calls, 17,576 Per-paper document-level calls, and 1,690 Per-paper aggregation calls, accumulated over all models, domains, queries, and runs).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Although Per-paper often improves Final List quality, the gains are much smaller or even reversed at the Final Answer stage.

    “Although Perpaper often improves Final List quality, the gains are much smaller or even reversed at the Final Answer stage.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Findingnone given

    In the Global regime, all papers within a domain are concatenated into a single input of approximately 107k-176k tokens, depending on the domain.

    “For the Global regime, all papers within a domain are concatenated into a single input of approximately 107k-176k tokens, depending on the domain.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Scaling model size yields uneven gains and does not remove these weaknesses.

    “Scaling model size yields uneven gains and does not remove these weaknesses.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    There is substantial cross-domain variation in both study scale and the structural complexity of reported evidence.

    “These statistics indicate substantial cross-domain variation in both study scale and the structural complexity of reported evidence.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Under Global processing, this pattern appears consistently (GPT-5.4:69.9/78.6; Qwen 3.5-122B: 66.4/71.9; MiniMax-M2.5: 61.8/69.2), and often persists under Per-paper.

    “Under Global processing, this pattern appears consistently (GPT-5.4:69.9 / 78.6; Qwen 3.5-122B: 66.4 / 71.9; MiniMax-M2.5: 61.8 / 69.2), and often persists under Per-paper.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    In O1.x under the Global regime, several models remain near parity (GPT-5.4:70.9/71.3; Gemma 4: 68.2/68.2; Qwen 3.5-122B: 76.0/76.4), and Per-paper shows a similar pattern for most systems.

    “In O1.x under the Global regime, several models remain near parity (GPT-5.4:70.9 / 71.3; Gemma 4: 68.2 / 68.2; Qwen 3.5-122B: 76.0 / 76.4), and Per-paper shows a similar pattern for most systems.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    In the object-centric series, OC Evidences reaches 75.9, and OC Final List 80.3, but OC Final Answer falls to 39.0.

    “In the object-centric series, OC Evidences reaches 75.9, and OC Final List 80.3, but OC Final Answer falls to 39.0.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    MC Evidences rises from 47.3 to 50.7 to 53.5, while MC Final List increases from 47.0 to 51.0 to 52.8.

    “MC Evidences rises from 47.3 to 50.7 to 53.5, while MC Final List increases from 47.0 to 51.0 to 52.8.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Performance generally declines from simpler extraction tasks (O1.x, M1.x) to more compositional settings (O2.x, M2.x), while Final Answer stages, especially MC, remain substantially harder than upstream evidence or list-construction steps.

    “Performance generally declines from simpler extraction tasks (O1.x,M1.x) to more compositional settings (O2.x,M2.x), while Final Answer stages, especially MC, remain substantially harder than upstream evidence or list-construction steps”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Decomposed per-paper processing is most beneficial for structurally demanding extraction tasks, while global long-context prompting remains competitive for simpler extraction and some exact downstream answers.

    “Decomposed per-paper processing is most beneficial for structurally demanding extraction tasks, while global long-context prompting remains competitive for simpler extraction and some exact downstream answers”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    OC Evidences increases from 71.6 to 75.9 to 77.0, and OC Final List changes only slightly from 78.3 to 80.3.

    “OC Evidences increases from 71.6 to 75.9 to 77.0, and OC Final List changes only slightly from 78.3 to 80.3.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Object-centric tasks are consistently easier than their method-centric counterparts, and the gap widens with complexity.

    “Object-centric tasks are consistently easier than their method-centric counterparts, and the gap widens with complexity.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    On direct extraction, O1.x slightly exceeds M1.x (65.5 vs. 64.0) and O2.x matches M2.x.

    “On direct extraction, O1.x slightly exceeds M1.x (65.5 vs. 64.0)and O2.x matches M2.x.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Single-field extraction scores highest: O1.x reaches 65.5 F1 and M1.x reaches 64.0.

    “Single-field extraction scores highest: O1.x reaches 65.5 F 1 and M1.x reaches 64.0”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning

    OC Final Answer remains 39.0 under all three criteria, and MC Final Answer remains 2.3 throughout.

    “OC Final Answer remains 39.0 under all three criteria, and MC Final Answer remains 2.3 throughout.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Individual papers average approximately 12-16k tokens.

    “Individual papers average approximately 12-16k tokens (Table 5).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Binding multiple fields into tuples causes a large drop: O2.x falls to 42.7 and the more relational M2.x to 42.5.

    “Binding multiple fields into tuples causes a large drop: O2.x falls to 42.7 and the more relational M2.x to 42.5.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    For Qwen 3.5-9B, the average variance is somewhat lower under Global than under Per-paper processing, whereas Gemma 4 and Qwen 3.5-122B show nearly identical variance under both regimes.

    “For Qwen 3.5-9B, the average variance is somewhat lower under Global than under Per-paper processing, whereas Gemma 4 and Qwen 3.5-122B show nearly identical variance under both regimes.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Format compliance scales sharply with model size within a family: the 9B Qwen variant exhibits 65% broken per-paper outputs, whereas the 122B variant exhibits 42%; Gemma 4 (31B) is fully compliant.

    “Format compliance scales sharply with model size within a family: the 9B Qwen variant exhibits 65% broken per-paper outputs, whereas the 122B variant exhibits 42%; Gemma 4 (31B) is fully compliant.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Qwen 3.5-122B shows somewhat larger fluctuations, but remains broadly reliable.

    “Qwen 3.5-122B shows somewhat larger fluctuations, but remains broadly reliable.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Most settings remain stable, while a few model-task combinations show substantial run-to-run variance.

    “Most settings remain stable, while a few model-task combinations show substantial run-to-run variance”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    The query suite is agnostic to the system under evaluation and can be applied unchanged to agentic pipelines in future work.

    “The query suite itself is agnostic to the system under evaluation and can be applied unchanged to agentic pipelines in future work.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationreasoning

    The Per-paper regime provides a decomposition baseline that reduces context length and makes the provenance of extracted evidence explicit by construction.

    “This provides a decomposition baseline that reduces context length and makes the provenance of extracted evidence explicit by construction.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    All domain inputs fit within each model's context window.

    “All domain inputs fit within each model's context window (Gemma 4 31B-it: 262k tokens; Qwen3.5-122B-A10B-FP8:262k tokens; Qwen3.5-9B:262k tokens; MiniMax-M2.5: 196k tokens; GPT-5.4:1,050k tokens).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationreasoning

    Agentic and retrieval-augmented systems are composed of the same underlying model calls, so the intrinsic reliability of schema-constrained extraction bounds and diagnoses the behavior of any framework built on top of it.

    “First, agentic and retrieval-augmented systems are composed of the same underlying model calls, so the intrinsic reliability of schemaconstrained extraction bounds and diagnoses the behavior of any framework built on top of it.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given

    Many evaluations focus on end-task outputs or narrowly scoped extraction targets, leaving the reliability of schema-constrained, multi-field scientific evidence records less well understood.

    “However, many evaluations focus on end-task outputs or narrowly scoped extraction targets, leaving the reliability of schema-constrained, multi-field scientific evidence records less well understood.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductown data

    On the subset with human consensus, the LLM judge agreed with the consensus label on 85% of cases (Cohen's κ = 0.80), suggesting strong alignment with human semantic judgment.

    “On the subset with human consensus, the LLM judge agreed with the consensus label on 85% of cases (Cohen's κ = 0.80), suggesting strong alignment with human semantic judgment.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    Scoring the three output stages separately helps distinguish whether failures arise during evidence extraction, intermediate consolidation, or final answer generation.

    “Scoring these stages separately helps distinguish whether failures arise during evidence extraction, intermediate consolidation, or final answer generation.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    Per-paper processing simplifies attribution because each extraction call sees only one source document.

    “Second, per-paper processing simplifies attribution because each extraction call sees only one source document.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductreasoning

    The two external annotators labeled prediction-gold match verdicts, not source articles, and their work does not constitute an independent second annotation of the gold-standard data itself.

    “The two external annotators labeled prediction-gold match verdicts, not source articles, and their work therefore does not constitute an independent second annotation of the goldstandard data itself (see Sect. 6).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    These tasks require aligning multiple fields within a study before any corpus-level synthesis occurs, so processing one paper at a time may better preserve local structural constraints.

    “These tasks require aligning multiple fields within a study before any corpus-level synthesis occurs, so processing one paper at a time may better preserve local structural constraints.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    Per-paper recovers more target items, but also introduces more spurious candidates.

    “Per-paper therefore recovers more target items, but also introduces more spurious candidates.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Prior work has documented the difficulty and error-proneness of manual extraction procedures, including higher error rates under single-extractor workflows.

    “Prior work has also documented the difficulty and error-proneness of manual extraction procedures, including higher error rates under single-extractor workflows [35].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [35] Buscemi, N., Hartling, L., Vandermeer, B., Tjosvold, L., Klassen, T.P.: Single data extraction generated more errors than double data extraction in systematic reviews. J. Clin. Epidemiol. 59(7), 697–703 (2006). https://doi.org/10.1016/j.jclinepi.2005.11.010 source

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    Because all models share the same query suites and scoring protocol, the comparison isolates the effect of structural demands on outputs.

    “Because all models share the same query suites and scoring protocol, the comparison isolates the effect of structural demands on outputs.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    Both regimes may improve intermediate F1, but their downstream numerical consequences differ.

    “Both regimes may improve intermediate F1, but their downstream numerical consequences differ.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given

    High lexical overlap does not guarantee semantic identity, as superficially similar strings may denote different constructs.

    “Conversely, high lexical overlap does not guarantee semantic identity, as superficially similar strings may denote different constructs.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

    Interpretationown data

    Together, these results clarify the current limits of automated scientific evidence structuring and motivate more rigorous next-generation evaluations.

    “Together, these results clarify the current limits of automated scientific evidence structuring and motivate more rigorous next-generation evaluations.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    Missing items tend to underestimate counts or omit required relations, whereas redundant items inflate counts or contaminate candidate sets.

    “Missing items tend to underestimate counts or omit required relations, whereas redundant items inflate counts or contaminate candidate sets.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    The Global regime directly evaluates long-context evidence extraction over multi-document inputs.

    “This setting directly evaluates long-context evidence extraction over multi-document inputs.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductown data

    Human inter-annotator agreement was high (93.0%; Cohen's κ = 0.906), indicating that the two independent annotators applied the four-way labeling scheme consistently, i.e., the matching criteria are well-defined enough to be reproduced across annotators.

    “Human inter-annotator agreement was high (93.0%;Cohen's κ = 0.906), indicating that the two independent annotators applied the four-way labeling scheme consistently, i.e., the matching criteria are well-defined enough to be reproduced across annotators.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Every paper received a second verification pass in which the annotator re-checked all recorded tuples against the source text.

    “every paper received a second verification pass in which the annotator re-checked all recorded tuples against the source text.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    Open-weight models are evaluated with three independent runs to measure run-to-run stability.

    “To measure runto-run stability, open-weight models are evaluated with three independent runs; GPT-5.4 is run once due to API cost constraints.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    The Per-paper regime already implements the most common orchestration primitive, namely task decomposition followed by aggregation, and the observed failures on exact corpus-level answers largely persist under it, indicating that these limitations are not removed by simple pipelining alone.

    “Third, our Per-paper regime already implements the most common orchestration primitive, namely task decomposition followed by aggregation, and the observed failures on exact corpus-level answers largely persist under it (Sect.5.4), indicating that these limitations are not removed by simple pipelining alone.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    GPT-5.4 is run once due to API cost constraints.

    “GPT-5.4 is run once due to API cost constraints.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductown data

    The human validation study was conducted on a stratified sample of 100 prediction-gold pairs, balanced across four verdicts, with two external annotators blind to the judge's decision and each other.

    “We stratified-sampled 100 prediction-gold pairs from the full pool of judge decisions, balanced across the four possible verdicts (exact, over_detail, less_detail, no_match).Two external annotators with graduate training in computational linguistics, blind to the judge's decision and to each other, independently labeled all sampled pairs using the same scheme.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    These systems are usually accurate when they output an item, yet still fail to recover the full gold set.

    “These systems are usually accurate when they output an item, yet still fail to recover the full gold set.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    Some normalization decisions in the annotation remain judgment-sensitive.

    “however, some normalization decisions remain judgment-sensitive.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    The aggregation strategy gives equal weight to each paper regardless of how many tuples it contains, preventing overall scores from being dominated by a small number of tuple-rich documents.

    “This aggregation strategy gives equal weight to each paper regardless of how many tuples it contains, preventing overall scores from being dominated by a small number of tuple-rich documents.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    Single-annotator annotation is a limitation of the study.

    “We therefore treat single-annotator annotation as a limitation of the study and discuss the associated risks in Sect.6.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    Models identify meaningful evidence and build partially correct candidate sets, but cannot deliver unique final values that require complete coverage, stable de-duplication, and exact aggregation.

    “Models identify meaningful evidence and build partially correct candidate sets, but cannot deliver unique final values that require complete coverage, stable de-duplication, and exact aggregation.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    A scaffolding-free setup yields a reproducible, framework-agnostic baseline whose comparisons remain interpretable as orchestration strategies evolve.

    “Second, a scaffolding-free setup yields a reproducible, frameworkagnostic baseline whose comparisons remain interpretable as orchestration strategies evolve.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    The validation study assesses the reliability of the LLM judge as an evaluation instrument only.

    “We note that this validation study assesses the reliability of the LLM judge as an evaluation instrument only.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    The paper-level constraint ensures that extracted evidence is credited only when assigned to the correct document.

    “This paper-level constraint ensures that extracted evidence is credited only when assigned to the correct document.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationreasoning

    Results reflect intrinsic extraction and reasoning ability because no retrieval tools or code execution are allowed.

    “No retrieval tools or code execution are allowed, so results reflect intrinsic extraction and reasoning ability.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductreasoning

    A single-annotator design carries two risks that we cannot fully rule out, namely annotation fatigue over a large number of tuples and idiosyncratic normalization choices where reporting conventions differ across disciplines.

    “Although the task is predominantly factual extraction, and every paper received a second verification pass, a single-annotator design carries two risks that we cannot fully rule out, namely annotation fatigue over a large number of tuples and idiosyncratic normalization choices where reporting conventions differ across disciplines.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    Several open-source models benefit more from local document structure and may be more vulnerable to interference in large joint contexts.

    “These results suggest that several opensource models benefit more from local document structure and may be more vulnerable to interference in large joint contexts.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductown data

    Scores in the main-text result tables were computed on predictions after deterministic re-parsing and an LLM-based rescue stage.

    “The main-text result tables report scores computed on predictions after our deterministic re-parsing and LLM-based rescue stage.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    Processing one article at a time often improves intermediate extraction quality, especially on structurally complex tasks, but these gains transfer only weakly to final analytical correctness.

    “Processing one article at a time often improves intermediate extraction quality, especially on structurally complex tasks, but these gains transfer only weakly to final analytical correctness.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    Variance does not map directly onto task difficulty; it can also reflect model-specific generation behavior under certain output formats.

    “Variance, therefore, does not map directly onto task difficulty; it can also reflect model-specific generation behavior under certain output formats.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    Per-paper often raises the quality of intermediate evidence sets, especially for open-source models, yet it does not by itself solve the harder problem of producing exact corpus-level analytical answers.

    “It often raises the quality of intermediate evidence sets, especially for open-source models, yet it does not by itself solve the harder problem of producing exact corpus-level analytical answers.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    Absolute scores should be read with the single-annotator caveat in mind, whereas the relative comparisons across models, task tiers, and input regimes, which share the same gold standard, are less affected by any systematic annotator bias.

    “Absolute scores should therefore be read with this caveat in mind, whereas the relative comparisons across models, task tiers, and input regimes, which share the same gold standard, are less affected by any systematic annotator bias.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductreasoning

    The human validation study in Sect.3.3 demonstrates that the matching criteria can be applied consistently by independent annotators, but it validates the LLM judge, not the gold data, and it is not a substitute for independent dual annotation of the corpus.

    “The human validation study in Sect.3.3 demonstrates that the matching criteria can be applied consistently by independent annotators, but it validates the LLM judge, not the gold data, and it is not a substitute for independent dual annotation of the corpus.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    The study is positioned as a diagnostic evaluation of failure modes rather than an exhaustive benchmark.

    “The study is accordingly positioned as a diagnostic evaluation of failure modes rather than an exhaustive benchmark.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data

    GPT-5.4 was run once due to API cost, so its run-to-run variance is not characterized.

    “In addition, GPT-5.4 was run once due to API cost, so its run-to-run variance is not characterized.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationreasoning

    The Per-paper regime introduces additional stages, including repeated document-level extraction and a separate aggregation step, each of which can add variability.

    “A plausible explanation is that the Perpaper regime introduces additional stages, including repeated document-level extraction and a separate aggregation step, each of which can add variability.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationnone given

    The query suite and staged evaluation protocol introduced here provide a reusable framework for diagnosing such failures and benchmarking future systems.

    “The query suite and staged evaluation protocol introduced here provide a reusable framework for diagnosing such failures and benchmarking future systems.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

    Interpretationreasoning

    Results should be interpreted as a lower bound on what composite systems may achieve, and as a diagnosis of the component-level weaknesses such systems must address.

    “Results should therefore be interpreted as a lower bound on what composite systems may achieve, and as a diagnosis of the component-level weaknesses such systems must address.”
    Quote in the paperNot a substantial claim
83 of 83

  • The paper relies on this without support.

    Scientific evidence is predominantly communicated through unstructured articles, while many downstream workflows require structured representations of study populations, variables, methods, and quantitative findings.

    “Scientific evidence is predominantly communicated through unstructured articles, while many downstream workflows require structured representations of study populations, variables, methods, and quantitative findings.”

  • The paper relies on this without support.

    The reliability of LLMs in constructing schema-constrained evidence records from parsed scientific full texts remains insufficiently understood.

    “yet their reliability in constructing schema-constrained evidence records from parsed scientific full texts remains insufficiently understood.”

  • The paper relies on this without support.

    Current capabilities remain insufficiently understood for the forms of extraction that matter most in practice.

    “However, current capabilities remain insufficiently understood for the forms of extraction that matter most in practice.”

  • The paper relies on this without support.

    The extraction of real scientific evidence often requires assembling multiple interdependent fields into coherent, structured records.

    “Yet the extraction of real scientific evidence often requires assembling multiple interdependent fields into coherent, structured records.”

  • The paper relies on this without support.

    Across these settings, a recurring bottleneck is converting free-form scientific prose into reliable, structured evidence records.

    “Across these settings, a recurring bottleneck is converting free-form scientific prose into reliable, structured evidence records.”

  • The paper relies on this without support.

    This transformation remains highly labor-intensive.

    “This transformation remains highly labor-intensive.”

  • The paper relies on this without support.

    Beyond evidence synthesis, building structured scientific resources typically depends on expensive annotation pipelines, handcrafted rules, or narrowly scoped extraction systems.

    “Beyond evidence synthesis, building structured scientific resources typically depends on expensive annotation pipelines, handcrafted rules, or narrowly scoped extraction systems.”

  • The paper relies on this without support.

    As scientific publishing continues to grow, scalable methods for structured extraction of scientific evidence are increasingly needed.

    “As scientific publishing continues to grow, scalable methods for structured extraction of scientific evidence are increasingly needed.”

  • The paper relies on this without support.

    Domain databases and monitoring systems depend on normalized study descriptors and results that can be queried computationally.

    “Domain databases and monitoring systems likewise depend on normalized study descriptors and results that can be queried computationally.”

  • The paper relies on this without support.

    Many existing evaluations focus on isolated entities, short-context question answering, or domain-specific benchmarks with relatively local targets.

    “Many existing evaluations focus on isolated entities, short-context question answering, or domain-specific benchmarks with relatively local targets.”

  • The paper relies on this without support.

    The same requirements for structured extraction also arise in scientific databases, scholarly knowledge graphs, automated literature analysis, and cross-paper comparison systems.

    “However, the same requirements also arise in scientific databases, scholarly knowledge graphs, automated literature analysis, and cross-paper comparison systems.”

  • The paper relies on this without support.

    Recovering complete study findings requires accurate entity binding, numerical grounding, and disambiguation of many-to-many relationships among variables across studies.

    “While reporting conventions differ by discipline, recovering complete study findings still requires accurate entity binding, numerical grounding, and disambiguation of many-to-many relationships among variables across studies.”

  • The paper relies on this without support.

    Recovering isolated fields mainly requires recognition and normalization; recovering higher-order tuples additionally requires that all fields originate from the correct study context and remain correctly bound within a coherent record.

    “Recovering isolated fields mainly requires recognition and normalization.Recovering higher-order tuples additionally requires that all fields originate from the correct study context and remain correctly bound within a coherent record.”

  • The paper relies on this without support.

    Mainstream open-weight models currently provide context windows of roughly 196k-262k tokens.

    “Mainstream open-weight models currently provide context windows of roughly 196k-262k tokens”

  • The paper relies on this without support.

    The corpus size follows from the design of the Global input regime, which requires all papers of a domain to be placed into a single prompt.

    “The corpus size is not arbitrary but follows from the design of the Global input regime, which requires all papers of a domain to be placed into a single prompt.”

  • The paper relies on this without support.

    Such studies are common across the behavioral, social, medical, environmental, and agricultural sciences, and they provide a suitable testbed for structured evidence extraction.

    “Such studies are common across the behavioral, social, medical, environmental, and agricultural sciences, and they provide a suitable testbed for structured evidence extraction.”

  • The paper relies on this without support.

    The corpora are intentionally diverse in study scale, methodological complexity, and reporting practices, providing a challenging testbed for structured evidence extraction and derived statistical analysis.

    “The corpora are intentionally diverse in study scale, methodological complexity, and reporting practices, providing a challenging testbed for structured evidence extraction and derived statistical analysis.”

  • The paper relies on this without support.

    The single-annotator gold standard is accurate enough that the reported comparisons reflect model performance.

    “Gold-standard annotations are created manually by a single expert annotator.”

  • The paper relies on this without support.

    The five scientific domains and the 52-paper corpus are representative of scientific literature for diagnosing LLM extraction limits.

    “The five corpora comprise 52 papers.”

  • The paper relies on this without support.

    The five evaluated LLMs are representative of contemporary LLMs.

    “We evaluate five LLMs spanning closed-source and open-weight families.”

  • The paper relies on this without support.

    The task is predominantly factual extraction.

    “Although the task is predominantly factual extraction”
21 of 21

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • The cited earlier conference paper is a different paper

    Minor

    The paper identifies reference [19] as its earlier IRCDL 2026 conference paper, but reference 19 is a paper on Yorùbá arithmetic reasoning. The IRCDL evidence-extraction paper appears as reference 20.

1 of 1

Cites

Flagged first, then by citations

Research works printed but not linked: 8

  1. Feng, J., Steinhardt, J.: How do language models bind entities in context? In: The Twelfth International Conference on Learning Representations (2024)
  2. Tan, Z., D’Souza, J.: Diagnosing structural failures in LLM-based evidence extraction for meta-analysis. In: IRCDL 2026 Information and Research Science Connecting to Digital and Library Science. CEUR workshop proceedings (2026). https://ceur-ws.org/Vol-4240/paper52.pdf
  3. Oelen, A., Jaradeh, M.Y., Farfar, K.E., Stocker, M., Auer, S.: Comparing research contributions in a scholarly knowledge graph. In: SciKnow 2019, Third International Workshop on Capturing Scientific Knowledge: Proceedings of the Third International Workshop on Capturing Scientific Knowledge, Co-located with the 10th International Conference on Knowledge Capture (K-CAP 2019), vol. 2526, pp. 21–26 (2019). Aachen: RWTH Aachen
  4. Pham, D., Ho, X., Ha, Q.T., Aizawa, A.: Solving label variation in scientific information extraction via multi-task learning. In: Huang, C.-R., Harada, Y., Kim, J.-B., Chen, S., Hsu, Y.-Y., Chersoni, E., A, P., Zeng, W.H., Peng, B., Li, Y., Li, J. (eds.) Proceedings of the 37th Pacific Asia Conference on Language, Information and Computation, pp. 243–256. Association for Computational Linguistics, Hong Kong (2023)
  5. Wallace, B.C., Kuiper, J., Sharma, A., Zhu, M.B., Marshall, I.J.: Extracting PICO sentences from clinical trial reports using supervised distant supervision. J. Mach. Learn. Res. 17(132), 1–25 (2016)
  6. Song, P., Han, P., Goodman, N.: A survey on large language model reasoning failures. In: 2nd AI for Math Workshop @ ICML 2025 (2025)
  7. Zhou, D., Schärli, N., Hou, L., Wei, J., Scales, N., Wang, X., Schuurmans, D., Cui, C., Bousquet, O., Le, Q.V., Chi, E.H.: Least-to-most prompting enables complex reasoning in large language models. In: The Eleventh International Conference on Learning Representations, (2023)
  8. Khot, T., Trivedi, H., Finlayson, M., Fu, Y., Richardson, K., Clark, P., Sabharwal, A.: Decomposed prompting: a modular approach for solving complex tasks. In: The Eleventh International Conference on Learning Representations (2023)

Cited by

Still being checked.

Related

Authors
Tan, Z., D’Souza, J., & Auer, S.
Affiliations
  • Leibniz University Hannover (DE)
  • Technische Informationsbibliothek (TIB) (DE)
Corresponding author
Tan, Zhiyin
ORCID
Venue
International Journal on Digital Libraries
Publisher
Springer Science and Business Media LLC
Published
2026-10-28
Volume
27
Issue
4
ISSN
  • 1432-1300 (ISSN-L)
  • 1432-5012
DOI
10.1007/s00799-026-00449-9
OpenAlex
W7224517687
Cited by
0
References
49
FWCI
0.00
Type
journal article
Language
en
Open access
Licence
Locations
International Journal on Digital Libraries (publishedVersion, cc-by)
Funders
Niedersächsisches Ministerium für Wissenschaft und Kultur (ZN4219)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Biomedical Text Mining and Ontologies
  • Physical Sciences › Computer Science › Artificial Intelligence › Advanced Text Analysis Techniques
Keywords
large language models, systematic reviews, meta-analysis, entity linking, model size scaling, benchmarking

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-11-03.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-11-03.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review