CoChat

The SciQA Scientific Question Answering Benchmark for Scholarly Knowledge

Auer, Sören, Barone, Dante A. C., Bartz, Cassiano, Cortes, Eduardo G., Jaradeh, Mohamad Yaser, Karras, Oliver, Koubarakis, Manolis, Mouromtsev, Dmitry, Pliukhin, Dmitrii, Radyush, Daniil, Shilin, Ivan, Stocker, Markus, Tsalapati, Eleni · 2023 · Scientific Reports
Cited by 45

The DOI matches its Crossref record; 10 of the 10 references it cites were found in a registry or an index.

Abstract

Knowledge graphs have gained increasing popularity in the last decade in science and technology. However, knowledge graphs are currently relatively simple to moderate semantic structures that are mainly a collection of factual statements. Question answering (QA) benchmarks and systems were so far mainly geared towards encyclopedic knowledge graphs such as DBpedia and Wikidata. We present SciQA a scientific QA benchmark for scholarly knowledge. The benchmark leverages the Open Research Knowledge Graph (ORKG) which includes almost 170,000 resources describing research contributions of almost 15,000 scholarly articles from 709 research fields. Following a bottom-up methodology, we first manually developed a set of 100 complex questions that can be answered using this knowledge graph. Furthermore, we devised eight question templates with which we automatically generated further 2465 questions, that can also be answered with the ORKG. The questions cover a range of research fields and question types and are translated into corresponding SPARQL queries over the ORKG. Based on two preliminary evaluations, we show that the resulting SciQA benchmark represents a challenging task for next-generation QA systems. This task is part of the open competitions at the 22nd International Semantic Web Conference 2023 as the Scholarly Question Answering over Linked Data (QALD) Challenge.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1038/s41598-023-33607-z

Cites

10 of 10 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 10 references

Research works printed but not linked: 40

  1. Chakraborty, N. et al. Introduction to neural network-based question answering over knowledge graphs. Wiley Interdiscip. Rev. Data Min. Knowl. Discov. 11 (2021). · DOI
  2. Diefenbach, D., López, V., Singh, K. D. & Maret, P. Core techniques of question answering systems over knowledge bases: A survey. Knowl. Inf. Syst. 55, 529–569 (2018). · DOI
  3. Jaradeh, M. Y. et al. Open research knowledge graph: next generation infrastructure for semantic scholarly knowledge. K-CAP, 243–246 (2019). · DOI · Paper page
  4. Stocker, M. et al. SKG4EOSC—Scholarly knowledge graphs for EOSC: Establishing a backbone of knowledge graphs for FAIR scholarly information in EOSC. Res. Ideas Outcomes 8, e83789 (2022). · DOI
  5. Kim, D. et al. Knowledge boosting: A graph-based integration approach with multi-omics data and genomic knowledge for cancer clinical outcome prediction. J. Am. Med. Inform. Assoc. 22, 109–120 (2015). · DOI · Paper page
  6. Stocker, M. et al. FAIR scientific information with the open research knowledge graph. FAIR Connecthttps://doi.org/10.3233/FC-221513 (2023). · DOI · Paper page
  7. Budde, L. et al. Investigation of the material combination 20mncr5 and x45crsi9-3 in the tailored forming of shafts with bearing seats. Product. Eng. 16, 661–671 (2022). · doi:10.1007/s11740-022-01119-w
  8. Karras, O. Investigation of the material combination 20mncr5 and x45crsi9-3 in the tailored forming of shafts with bearing seats. https://doi.org/10.48366/R288295 (2023). · doi:10.48366/r288295
  9. Jaradeh, M. Y., Stocker, M. & Auer, S. Question answering on scholarly knowledge graphs. TPDL.19–32 (2020). · DOI
  10. Leiter, C. et al. Chatgpt: A meta-analysis after 2.5 months. https://doi.org/10.48550/ARXIV.2302.13795 (2023). · DOI
  11. Saikh, T., Ghosal, T., Mittal, A., Ekbal, A. & Bhattacharyya, P. Scienceqa: A novel resource for question answering on scholarly articles. Int. J. Digital Libraries 23, 289–301. https://doi.org/10.1007/s00799-022-00329-y (2022). · DOI
  12. Cortes, E. & Karras, O. Question answering over linked data benchmark comparison. https://doi.org/10.48366/R161787 (2022). · doi:10.48366/r161787
  13. Berant, J., Chou, A., Frostig, R. & Liang, P. Semantic parsing on freebase from question-answer pairs. EMNLP. 1533–1544 (2013).
  14. Yih, W.-T., Richardson, M., Meek, C., Chang, M.-W. & Suh, J. The value of semantic parse labeling for knowledge base question answering. ACL. https://doi.org/10.18653/v1/P16-2033 (2016). · DOI · Paper page
  15. Bordes, A., Usunier, N., Chopra, S. & Weston, J. Large-scale simple question answering with memory networks. CoRR. abs/1506.02075 (2015). · arXiv
  16. Dubey, M., Banerjee, D., Abdelkawi, A. & Lehmann, J. LC-QuAD 2.0: A large dataset for complex question answering over Wikidata and DBpedia. ISWC. 69–78 (2019). · DOI · Paper page
  17. Karras, O., Groen, E. C., Khan, J. A. & Auer, S. Researcher or crowd member? Why not both! The open research knowledge graph for applying and communicating CrowdRE research. in 2021 IEEE 29th International Requirements Engineering Conference Workshops (REW). https://doi.org/10.1109/REW53955.2021.00056 (2021). · DOI
  18. Oelen, A. Semantic representations of scholarly communication. https://doi.org/10.48366/R8364 (2022). · doi:10.48366/r8364
  19. Lehnert, W. A conceptual theory of question answering. in Readings in Natural Language Processing (Morgan Kaufmann, 1986).
  20. Li, X. & Roth, D. Learning question classifiers. ACL. (2002). · DOI · Paper page
  21. Riloff, E. & Thelen, M. A rule-based question answering system for reading comprehension tests. in ANLP/NAACL Workshop on Reading comprehension tests as Evaluation for Computer-based Language Understanding Systems (2000). · DOI · Paper page
  22. Leidner, J. L. Question answering over unstructured data without domain restrictions. arXiv preprint cs/0207058 (2002). · arXiv
  23. Lopez, V., Unger, C., Cimiano, P. & Motta, E. Evaluating question answering over linked data. Web Semantics. 21, 3–13 (2013). · DOI · Paper page
  24. Cimiano, P. & Minock, M. Natural language interfaces: What is the problem? A data-driven quantitative analysis. in Int. Conf. on Appl. of Natural Lang. to Inf, Systems (Springer, 2009). · DOI
  25. Moldovan, D. et al. The structure and performance of an open-domain question answering system. ACL 563–570 (2000). · DOI · Paper page
  26. Ferret, O. et al. Finding an answer based on the recognition of the question focus. TREC. (2001). · doi:10.6028/nist.sp.500-250.qa-limsi
  27. Mikhailian, A., Dalmas, T. & Pinchuk, R. Learning foci for question answering over topic maps. ACL-IJCNLP 325–328 (2009). · doi:10.3115/1667583.1667684
  28. Bonifati, A., Martens, W. & Timm, T. An analytical study of large SPARQL query logs. VLDB J. 29, 655–679 (2020). · doi:10.1007/s00778-019-00558-9
  29. Kullmann, F. et al. Comparison of Studies on Germany’s Energy Supply in 2050 (Tech. Rep Technoökonomische Systemanalyse, 2021).
  30. Kullmann, F. et al. Comparison of studies on Germany’s energy supply in 2050. https://doi.org/10.48366/R153801 (2021). · doi:10.48366/r153801
  31. Marín, M. A. Genetic variability (COI variation) in studies large sampled (>1000 sequences). https://doi.org/10.48366/R149849 (2022). · doi:10.48366/r149849
  32. Jin, Q., Dhingra, B., Liu, Z., Cohen, W. W. & Lu, X. Pubmedqa: A dataset for biomedical research question answering. https://doi.org/10.48550/ARXIV.1909.06146 (2019). · DOI
  33. Jaradeh, M. Y., Singh, K., Stocker, M., Both, A. & Auer, S. Information extraction pipelines for knowledge graphs. Knowl. Inform. Syst.https://doi.org/10.1007/s10115-022-01826-x (2023). · DOI
  34. Grangel-González, I. et al. An rdf-based approach for implementing industry 4.0 components with administration shells. In 21st IEEE International Conference on Emerging Technologies and Factory Automation, ETFA 2016, Berlin, Germany, September 6-9, 2016, 1–8. https://doi.org/10.1109/ETFA.2016.7733503 (IEEE, 2016). · DOI
  35. Fischer, F. et al. Stack Overflow Considered Harmful? The Impact of Copy &Paste on Android Application Security (2017). · DOI · Paper page
  36. Prxžulj, N., Corneil, D. G. & Jurisica, I. Modeling interactome: Scale-free or geometric?. Bioinformatics 20, 3508–3515. https://doi.org/10.1093/bioinformatics/bth436 (2004). · DOI · Paper page
  37. Milo, R. et al. Network motifs: Simple building blocks of complex networks. Science 298, 824–827. https://doi.org/10.1126/science.298.5594.824 (2002). · DOI · Paper page
  38. Breit, A. et al. Combining machine learning and semantic web: A systematic mapping study. ACM Comput. Surv.https://doi.org/10.1145/3586163 (2023). · DOI
  39. Vogt, L., D’Souza, J., Stocker, M. & Auer, S. Toward representing research contributions in scholarly knowledge graphs using knowledge graph cells. JCDLhttps://doi.org/10.1145/3383583.3398530 (2020). · DOI
  40. Haris, M., Farfar, K. E., Stocker, M. & Auer, S. Federating scholarly infrastructures with GraphQL. ICADLhttps://doi.org/10.1007/978-3-030-91669-5_24 (2021). · DOI

Cited by

See all 45

Metadata

Authors
Auer, S., Barone, D. A. C., Bartz, C., Cortes, E. G., Jaradeh, M. Y., Karras, O., Koubarakis, M., Mouromtsev, D., Pliukhin, D., Radyush, D., Shilin, I., Stocker, M., & Tsalapati, E.
Venue
Scientific Reports
Published
2023-05-04
Volume
13
Issue
1
ISSN
2045-2322
Cited by
45
References
10
Language
en
Licence
Funders
  • European Research Council (819536)
  • German Federal Ministry of Education and Research (01DD20003)
  • European Unions Horizon 2020 research and innovation programme (101032307)
  • Coordenacao de Aperfeicoamento de 386 Pessoal de Nivel Superior - Brasil (001)
  • TIB - Leibniz Informationszentrum für Technik und Naturwissenschaften

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: The SciQA Scientific Question Answering Benchmark for Scholarly Knowledge. https://app.cochat.ai/papers/10.1038/s41598-023-33607-z
The paper
Auer, S., Barone, D. A. C., Bartz, C., Cortes, E. G., Jaradeh, M. Y., Karras, O., Koubarakis, M., Mouromtsev, D., Pliukhin, D., Radyush, D., Shilin, I., Stocker, M., & Tsalapati, E. (2023). The SciQA Scientific Question Answering Benchmark for Scholarly Knowledge. Scientific Reports. https://doi.org/10.1038/s41598-023-33607-z

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.