DBLP-QuAD: A Question Answering Dataset over the DBLP Scholarly Knowledge Graph

“In the future, we aim to conduct a more comprehensive evaluation using standardised benchmarks[11][12] and benchmarking frameworks[2] to assess the system’s precision, usability, and overall performance across a wider range of queries.”

“Unlike that, the Scholarly QALD Challenge organized at the ISWC 2023 comes up with two new Scholarly QA data sets, namely SciQA (Scientific QA) [18] and DBLP-QUAD [4]; and provides Codalab [19] as a competition platform.”

“In the past decade, tremendous efforts [1, 2, 3, 4] have been devoted to organizing scholarly data in the form of knowledge graphs (KGs), which store heterogeneous relational information in an integrated and extensible manner, supporting querying with complex semantics.”

“Hence, our platform achieves significant advancements in dialogue benchmark quality.”

“Our evaluation uses KGQA datasets derived from heterogeneous KGs, such as Freebase [11], DBpedia [20] and DBLP [21].”

“…subgraphs from KG, uses reasoning to extract entities as answers (Wang et al., 2021), or uses semantic parsing or LLMs in a zero or few shot setting to transform questions into SQL or SPARQL queries to get answers from the KG (Banerjee et al., 2023; Taffa and Usbeck, 2023; Avila et al., 2024).”

“The DBLP computer science bibliography is an example of such an effort that provides open bibliographic information on major computer science journals and proceedings [2].”

“Based on DBLP-QuAD, Wang et al. developed NLQxform [28], a transformer-based model that translates scholarly questions into executable SPARQL queries.”

“…effectively scientometric analyses (Salatino et al., 2020, 2023), intelligent systems for exploring the literature (Angioni et al., 2021), and, increasingly, conversational systems (Meloni et al., 2023) and question-answering methods (Auer et al., 2023; Lehmann et al., 2024; Banerjee et al., 2023).”

“This is exemplified by the introduction of DBLP-QuAD, a dataset specifically designed for question-answering over the DBLP KG[13].”

“Unlike JarvisQA Jaradeh et al. (2020), which relies on converting tabular data into text, and DBLP-QuAD Banerjee et al. (2023) requiring extensive training for new schemas.”

“These improvements increase the DBLP usability in KG Question Answering (KGQA) 3 [4].”

“For finding optimal parameters for the different components of the entity linker pipeline, we randomly selected a set of 100 questions from the test set of the DBLP_QuAD dataset [10].”

“We fine-tuned the adapted BART model using the training set of DBLP-QuAD [3] dataset, which consists of 7,000 pairs of questions and SPARQL queries over the DBLP KG.”

“The scholarly-QALD dataset is the second iteration of Question Answering dataset for scholarly data, preceded by DBLP-QuAD dataset[3].”

“Table 2 shows exact match accuracies across DBLP-QuAD question types.”

“Traditional KGQA datasets, however, are either manually curated—making them costly and time-intensive [9]—or template-based, which often limits their diversity and generalizability [1].”

“DBLP-QuAD [18] provides one-hop and multi-hop QA benchmarks derived from the DBLP scholarly graph, constructed using query templates and paraphrasing.”

“We start with 98 SPARQL and question templates, adapted from prior work [5].”

“A 100-query benchmark was created by adapting data from the DBLP-QuAD dataset [27].”

“Additional scientific Q&A datasets fall outside our research scope, such as DBLP-QuAD (Baner-jee et al., 2023) for bibliographic queries and datasets designed to extract factual knowledge from scientific literature for knowledge graph population (Auer et al., 2023; Yan et al., 2024).”

“These resources also underpin SPARQL benchmarking, question answering, and recommendation systems [3,43,54].”

“DBLP-QuAD is a public question–SPARQL benchmark; we randomly sampled 100 questions from its test split [3].”

“We adapt (a subset of) the definitions provided by Banerjee et al. (2023), with some alterations.”

“…following the setup of the challenge: We used DBLP KG and 7,000/1,000 training/validation questions (with query and answer annotations) from DBLP-QuAD [2] to develop NLQxform-UI, including fine-tuning the BART model for the first step and constructing the template base for the third step.”