The SciQA Scientific Question Answering Benchmark for Scholarly Knowledge
The DOI matches its Crossref record; 10 of the 10 references it cites were found in a registry or an index.
Abstract
Knowledge graphs have gained increasing popularity in the last decade in science and technology. However, knowledge graphs are currently relatively simple to moderate semantic structures that are mainly a collection of factual statements. Question answering (QA) benchmarks and systems were so far mainly geared towards encyclopedic knowledge graphs such as DBpedia and Wikidata. We present SciQA a scientific QA benchmark for scholarly knowledge. The benchmark leverages the Open Research Knowledge Graph (ORKG) which includes almost 170,000 resources describing research contributions of almost 15,000 scholarly articles from 709 research fields. Following a bottom-up methodology, we first manually developed a set of 100 complex questions that can be answered using this knowledge graph. Furthermore, we devised eight question templates with which we automatically generated further 2465 questions, that can also be answered with the ORKG. The questions cover a range of research fields and question types and are translated into corresponding SPARQL queries over the ORKG. Based on two preliminary evaluations, we show that the resulting SciQA benchmark represents a challenging task for next-generation QA systems. This task is part of the open competitions at the 22nd International Semantic Web Conference 2023 as the Scholarly Question Answering over Linked Data (QALD) Challenge.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Evaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436
LaMDA: Language Models for Dialog Applications
Thoppilan, R., De Freitas, D., Hall, J., Shazeer, N., Kulshreshtha, A., Cheng, H.-T., Jin, A., Bos, T., Baker, L., Du, Y., et al. (2022) · arXiv · doi:10.48550/arxiv.2201.08239 · Cited by 778
Galactica: A Large Language Model for Science
Taylor, R., Kardas, M., Cucurull, G., Scialom, T., Hartshorn, A., Saravia, E., Poulton, A., Kerkez, V., & Stojnic, R. (2022) · arXiv · doi:10.48550/arxiv.2211.09085 · Cited by 605
The Web as a Knowledge-Base for Answering Complex Questions
Talmor, A., & Berant, J. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1059 · Cited by 520 · cited 1×
“This approach is inspired by the procedures of related works that also combined the use of manual question creation and automated question generation 16,18–20 .”
MethodLC-QuAD: A Corpus for Complex Question Answering over Knowledge Graphs
Trivedi, P., Maheshwari, G., Dubey, M., & Lehmann, J. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-68204-4_22 · Cited by 261
MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answering
Pal, A., Umapathi, L. K., & Sankarasubbu, M. (2022) · arXiv · doi:10.48550/arxiv.2203.14371 · Cited by 157
Improving alignment of dialogue agents via targeted human judgements
Glaese, A., McAleese, N., Trębacz, M., Aslanides, J., Firoiu, V., Ewalds, T., Rauh, M., Weidinger, L., Chadwick, M., Thacker, P., et al. (2022) · arXiv · doi:10.48550/arxiv.2209.14375 · Cited by 134
Generate FAIR Literature Surveys with Scholarly Knowledge Graphs
Oelen, A., Jaradeh, M. Y., Stocker, M., & Auer, S. (2020) · Proceedings of the ACM/IEEE Joint Conference on Digital Libraries in 2020 · doi:10.1145/3383583.3398520 · Cited by 52
Self-citation
Research works printed but not linked: 40
- Chakraborty, N. et al. Introduction to neural network-based question answering over knowledge graphs. Wiley Interdiscip. Rev. Data Min. Knowl. Discov. 11 (2021). · DOI
- Diefenbach, D., López, V., Singh, K. D. & Maret, P. Core techniques of question answering systems over knowledge bases: A survey. Knowl. Inf. Syst. 55, 529–569 (2018). · DOI
- Jaradeh, M. Y. et al. Open research knowledge graph: next generation infrastructure for semantic scholarly knowledge. K-CAP, 243–246 (2019). · DOI · Paper page
- Stocker, M. et al. SKG4EOSC—Scholarly knowledge graphs for EOSC: Establishing a backbone of knowledge graphs for FAIR scholarly information in EOSC. Res. Ideas Outcomes 8, e83789 (2022). · DOI
- Kim, D. et al. Knowledge boosting: A graph-based integration approach with multi-omics data and genomic knowledge for cancer clinical outcome prediction. J. Am. Med. Inform. Assoc. 22, 109–120 (2015). · DOI · Paper page
- Stocker, M. et al. FAIR scientific information with the open research knowledge graph. FAIR Connecthttps://doi.org/10.3233/FC-221513 (2023). · DOI · Paper page
- Budde, L. et al. Investigation of the material combination 20mncr5 and x45crsi9-3 in the tailored forming of shafts with bearing seats. Product. Eng. 16, 661–671 (2022). · doi:10.1007/s11740-022-01119-w
- Karras, O. Investigation of the material combination 20mncr5 and x45crsi9-3 in the tailored forming of shafts with bearing seats. https://doi.org/10.48366/R288295 (2023). · doi:10.48366/r288295
- Jaradeh, M. Y., Stocker, M. & Auer, S. Question answering on scholarly knowledge graphs. TPDL.19–32 (2020). · DOI
- Leiter, C. et al. Chatgpt: A meta-analysis after 2.5 months. https://doi.org/10.48550/ARXIV.2302.13795 (2023). · DOI
- Saikh, T., Ghosal, T., Mittal, A., Ekbal, A. & Bhattacharyya, P. Scienceqa: A novel resource for question answering on scholarly articles. Int. J. Digital Libraries 23, 289–301. https://doi.org/10.1007/s00799-022-00329-y (2022). · DOI
- Cortes, E. & Karras, O. Question answering over linked data benchmark comparison. https://doi.org/10.48366/R161787 (2022). · doi:10.48366/r161787
- Berant, J., Chou, A., Frostig, R. & Liang, P. Semantic parsing on freebase from question-answer pairs. EMNLP. 1533–1544 (2013).
- Yih, W.-T., Richardson, M., Meek, C., Chang, M.-W. & Suh, J. The value of semantic parse labeling for knowledge base question answering. ACL. https://doi.org/10.18653/v1/P16-2033 (2016). · DOI · Paper page
- Bordes, A., Usunier, N., Chopra, S. & Weston, J. Large-scale simple question answering with memory networks. CoRR. abs/1506.02075 (2015). · arXiv
- Dubey, M., Banerjee, D., Abdelkawi, A. & Lehmann, J. LC-QuAD 2.0: A large dataset for complex question answering over Wikidata and DBpedia. ISWC. 69–78 (2019). · DOI · Paper page
- Karras, O., Groen, E. C., Khan, J. A. & Auer, S. Researcher or crowd member? Why not both! The open research knowledge graph for applying and communicating CrowdRE research. in 2021 IEEE 29th International Requirements Engineering Conference Workshops (REW). https://doi.org/10.1109/REW53955.2021.00056 (2021). · DOI
- Oelen, A. Semantic representations of scholarly communication. https://doi.org/10.48366/R8364 (2022). · doi:10.48366/r8364
- Lehnert, W. A conceptual theory of question answering. in Readings in Natural Language Processing (Morgan Kaufmann, 1986).
- Li, X. & Roth, D. Learning question classifiers. ACL. (2002). · DOI · Paper page
- Riloff, E. & Thelen, M. A rule-based question answering system for reading comprehension tests. in ANLP/NAACL Workshop on Reading comprehension tests as Evaluation for Computer-based Language Understanding Systems (2000). · DOI · Paper page
- Leidner, J. L. Question answering over unstructured data without domain restrictions. arXiv preprint cs/0207058 (2002). · arXiv
- Lopez, V., Unger, C., Cimiano, P. & Motta, E. Evaluating question answering over linked data. Web Semantics. 21, 3–13 (2013). · DOI · Paper page
- Cimiano, P. & Minock, M. Natural language interfaces: What is the problem? A data-driven quantitative analysis. in Int. Conf. on Appl. of Natural Lang. to Inf, Systems (Springer, 2009). · DOI
- Moldovan, D. et al. The structure and performance of an open-domain question answering system. ACL 563–570 (2000). · DOI · Paper page
- Ferret, O. et al. Finding an answer based on the recognition of the question focus. TREC. (2001). · doi:10.6028/nist.sp.500-250.qa-limsi
- Mikhailian, A., Dalmas, T. & Pinchuk, R. Learning foci for question answering over topic maps. ACL-IJCNLP 325–328 (2009). · doi:10.3115/1667583.1667684
- Bonifati, A., Martens, W. & Timm, T. An analytical study of large SPARQL query logs. VLDB J. 29, 655–679 (2020). · doi:10.1007/s00778-019-00558-9
- Kullmann, F. et al. Comparison of Studies on Germany’s Energy Supply in 2050 (Tech. Rep Technoökonomische Systemanalyse, 2021).
- Kullmann, F. et al. Comparison of studies on Germany’s energy supply in 2050. https://doi.org/10.48366/R153801 (2021). · doi:10.48366/r153801
- Marín, M. A. Genetic variability (COI variation) in studies large sampled (>1000 sequences). https://doi.org/10.48366/R149849 (2022). · doi:10.48366/r149849
- Jin, Q., Dhingra, B., Liu, Z., Cohen, W. W. & Lu, X. Pubmedqa: A dataset for biomedical research question answering. https://doi.org/10.48550/ARXIV.1909.06146 (2019). · DOI
- Jaradeh, M. Y., Singh, K., Stocker, M., Both, A. & Auer, S. Information extraction pipelines for knowledge graphs. Knowl. Inform. Syst.https://doi.org/10.1007/s10115-022-01826-x (2023). · DOI
- Grangel-González, I. et al. An rdf-based approach for implementing industry 4.0 components with administration shells. In 21st IEEE International Conference on Emerging Technologies and Factory Automation, ETFA 2016, Berlin, Germany, September 6-9, 2016, 1–8. https://doi.org/10.1109/ETFA.2016.7733503 (IEEE, 2016). · DOI
- Fischer, F. et al. Stack Overflow Considered Harmful? The Impact of Copy &Paste on Android Application Security (2017). · DOI · Paper page
- Prxžulj, N., Corneil, D. G. & Jurisica, I. Modeling interactome: Scale-free or geometric?. Bioinformatics 20, 3508–3515. https://doi.org/10.1093/bioinformatics/bth436 (2004). · DOI · Paper page
- Milo, R. et al. Network motifs: Simple building blocks of complex networks. Science 298, 824–827. https://doi.org/10.1126/science.298.5594.824 (2002). · DOI · Paper page
- Breit, A. et al. Combining machine learning and semantic web: A systematic mapping study. ACM Comput. Surv.https://doi.org/10.1145/3586163 (2023). · DOI
- Vogt, L., D’Souza, J., Stocker, M. & Auer, S. Toward representing research contributions in scholarly knowledge graphs using knowledge graph cells. JCDLhttps://doi.org/10.1145/3383583.3398530 (2020). · DOI
- Haris, M., Farfar, K. E., Stocker, M. & Auer, S. Federating scholarly infrastructures with GraphQL. ICADLhttps://doi.org/10.1007/978-3-030-91669-5_24 (2021). · DOI
Cited by
A review of large language models and autonomous agents in chemistry
Ramos, M. C., Collison, C. J., & White, A. D. (2025) · Chemical Science · doi:10.1039/d4sc03921a · Cited by 274
PeerQA: A Scientific Question Answering Dataset from Peer Reviews
Baumgärtner, T., Briscoe, T., & Gurevych, I. (2025) · Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) · doi:10.18653/v1/2025.naacl-long.22 · Cited by 39
Can GPT-4 Replicate Empirical Software Engineering Research?
Liang, J. T., Badea, C., Bird, C., DeLine, R., Ford, D., Forsgren, N., & Zimmermann, T. (2024) · Proceedings of the ACM on Software Engineering · doi:10.1145/3660767 · Cited by 19
“The selection criteria for the research papers were: • Has a quantitative empirical analysis on software engineering data .”
BackgroundGated Linear Attention Transformers with Hardware-Efficient Training
Yang, S., Wang, B., Shen, Y., Panda, R., & Kim, Y. (2023) · arXiv · doi:10.48550/arxiv.2312.06635 · Cited by 10
“10 Here GLA adopts the”
LimGen: Probing the LLMs for Generating Suggestive Limitations of Research Papers
Faizullah, A. R. B. M., Urlana, A., & Mishra, R. (2024) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-70344-7_7 · Cited by 6
Metadata
- Authors
- Auer, S., Barone, D. A. C., Bartz, C., Cortes, E. G., Jaradeh, M. Y., Karras, O., Koubarakis, M., Mouromtsev, D., Pliukhin, D., Radyush, D., Shilin, I., Stocker, M., & Tsalapati, E.
- Venue
- Scientific Reports
- Published
- 2023-05-04
- Volume
- 13
- Issue
- 1
- ISSN
- 2045-2322
- Cited by
- 45
- References
- 10
- Language
- en
- Licence
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Funders
- European Research Council (819536)
- German Federal Ministry of Education and Research (01DD20003)
- European Unions Horizon 2020 research and innovation programme (101032307)
- Coordenacao de Aperfeicoamento de 386 Pessoal de Nivel Superior - Brasil (001)
- TIB - Leibniz Informationszentrum für Technik und Naturwissenschaften
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: The SciQA Scientific Question Answering Benchmark for Scholarly Knowledge. https://app.cochat.ai/papers/10.1038/s41598-023-33607-z
- The paper
- Auer, S., Barone, D. A. C., Bartz, C., Cortes, E. G., Jaradeh, M. Y., Karras, O., Koubarakis, M., Mouromtsev, D., Pliukhin, D., Radyush, D., Shilin, I., Stocker, M., & Tsalapati, E. (2023). The SciQA Scientific Question Answering Benchmark for Scholarly Knowledge. Scientific Reports. https://doi.org/10.1038/s41598-023-33607-z
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.