CoChat
Sign Up
Cited by 102Topic ModelingNatural Language Processing TechniquesInformation Retrieval and Search Behavior

Why Reinvent the Wheel: Let's Build Question Answering Systems Together

Open access
Singh, Kuldeep, Lange, Christoph, Vidal, Maria Esther, Lehmann, Jens, Auer, Sören, Radhakrishna, Arun Sethupat, Both, Andreas, Shekarpour, Saeedeh +5 more·2018·Proceedings of the 2018 World Wide Web Conference on World Wide Web - WWW '18
DOI 10.1145/3178876.3186023
Open this paper in CoChat — free accountOpen PDF

Abstract

Modern question answering (QA) systems need to flexibly integrate a number of components specialised to fulfil specific tasks in a QA pipeline. Key QA tasks include Named Entity Recognition and Disambiguation, Relation Extraction, and Query Building. Since a number of different software components exist that implement different strategies for each of these tasks, it is a major challenge to select and combine the most suitable components into a QA system, given the characteristics of a question. We study this optimisation problem and train classifiers, which take features of a question as input and have the goal of optimising the selection of QA components based on those features. We then devise a greedy algorithm to identify the pipelines that include the suitable components and can effectively answer the given question. We implement this model within Frankenstein, a QA framework able to select QA components and compose QA pipelines. We evaluate the effectiveness of the pipelines generated by Frankenstein using the QALD and LC-QuAD benchmarks. These results not only suggest that Frankenstein precisely solves the QA optimisation problem but also enables the automatic composition of optimised QA pipelines, which outperform the static Baseline QA pipeline. Thanks to this flexible and fully automated pipeline generation process, new QA components can be easily included in Frankenstein, thus improving the performance of the generated pipelines.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3178876.3186023

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Entity linking can affect relation linking (RL).

    “Recent studies (Singh et al., 2018) have shown that entity linking can affect RL.”

    From Implicit Relation Linking for Question Answering over Knowledge Graph (2022) · cites this paper as Singh et al., 2018

    The Motivating Example (Section 2; page unavailable) places named entity disambiguation before relation linking, and Section 3.1 describes both tasks as transformations of a question’s representation. This supports a possible connection, but the cited passages do not report that changing entity-linking results affects relation-linking performance. (LLM, conf=0.76)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 31

  1. Sören Auer, Christian Bizer, Georgi Kobilarov, Jens Lehmann, Richard Cyganiak, and Zachary G. Ives. 2007. DBpedia: A Nucleus for a Web of Open Data. In The Semantic Web, 6th International Semantic Web Conference, 2nd Asian Semantic Web Conference, ISWC 2007 + ASWC 2007, Busan, Korea, November 11-15, 2007.
  2. Rainer Berbner, Michael Spahn, Nicolas Repp, Oliver Heckmann, and Ralf Stein- metz. 2006. Heuristics for QoS-aware Web Service Composition. In 2006 IEEE In- ternational Conference on Web Services (ICWS 2006), 18-22 September 2006, Chicago, Illinois, USA. 72-82. https://doi.org/10.1109/ICWS.2006.69 · DOI
  3. Phil Blunsom, Krystle Kocik, and James R. Curran. 2006. Question classification with log-linear models. In SIGIR 2006: Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Seattle, Washington, USA, August 6-11, 2006. ACM, 615-616.
  4. Andreas Both, Dennis Diefenbach, Kuldeep Singh, Saeedeh Shekarpour, Didier Cherix, and Christoph Lange. 2016. Qanary -A Methodology for Vocabulary- Driven Open Question Answering Systems. In The Semantic Web. Latest Advances and New Domains -13th International Conference, ESWC 2016, Heraklion, Crete, Greece, May 29 -June 2, 2016, Proceedings. Springer, 625-641.
  5. Andreas Both, Axel-Cyrille Ngonga Ngomo, Ricardo Usbeck, Denis Lukovnikov, Christiane Lemke, and Maximilian Speicher. 2014. A service-oriented search framework for full text, geospatial and semantic search. In Proceedings of the 10th International Conference on Semantic Systems, SEMANTICS 2014, Leipzig, Germany, September 4-5, 2014. ACM, 65-72.
  6. Dennis Diefenbach, Kuldeep Singh, Andreas Both, Didier Cherix, Christoph Lange, and Sören Auer. 2017. The Qanary Ecosystem: Getting New Insights by Composing Question Answering Pipelines. In Web Engineering -17th International Conference, ICWE 2017, Rome, Italy, June 5-8, 2017, Proceedings. Springer, 171-189.
  7. Milan Dojchinovski and Tomas Kliegr. 2013. Entityclassifier.eu: Real-time Classi- fication of Entities in Text with Wikipedia. In Proceedings of the European Confer- ence on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECMLPKDD'13). Springer-Verlag, 654-658.
  8. Paolo Ferragina and Ugo Scaiella. 2010. TAGME: on-the-fly annotation of short text fragments (by wikipedia entities). In Proceedings of the 19th ACM Conference on Information and Knowledge Management, CIKM 2010, Toronto, Ontario, Canada, October 26-30, 2010. 1625-1628. https://doi.org/10.1145/1871437.1871689 · DOI
  9. Óscar Ferrández, Christian Spurk, Milen Kouylekov, Iustin Dornescu, Sergio Ferrández, Matteo Negri, Rubén Izquierdo, David Tomás, Constantin Orasan, Guenter Neumann, Bernardo Magnini, and José Luis Vicedo González. 2011. The QALL-ME Framework: A specifiable-domain multilingual Question Answering architecture. J. Web Sem. 9, 2 (2011), 137-145.
  10. Jenny Rose Finkel, Trond Grenager, and Christopher D. Manning. 2005. Incor- porating Non-local Information into Information Extraction Systems by Gibbs Sampling. In ACL 2005, 43rd Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference, 25-30 June 2005, University of Michigan, USA. The Association for Computer Linguistics, 363-370.
  11. Johannes Hoffart, Mohamed Amir Yosef, Ilaria Bordino, Hagen Fürstenau, Manfred Pinkal, Marc Spaniol, Bilyana Taneva, Stefan Thater, and Gerhard Weikum. 2011. Robust Disambiguation of Named Entities in Text. In Pro- ceedings of the 2011 Conference on Empirical Methods in Natural Language Pro- cessing, EMNLP 2011, 27-31 July 2011, John McIntyre Conference Centre, Edin- burgh, UK, A meeting of SIGDAT, a Special Interest Group of the ACL. 782-792. http://www.aclweb.org/anthology/D11-1072
  12. Zhiheng Huang, Marcus Thint, and Asli Çelikyilmaz. 2009. Investigation of Question Classifier in Question Answering. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, EMNLP 2009, 6-7 August 2009, Singapore, A meeting of SIGDAT, a Special Interest Group of the ACL.
  13. Zhiheng Huang, Marcus Thint, and Zengchang Qin. 2008. Question Classification using Head Words and their Hypernyms. In 2008 Conference on Empirical Methods in Natural Language Processing, EMNLP 2008, Proceedings of the Conference, 25-27 October 2008, Honolulu, Hawaii, USA, A meeting of SIGDAT, a Special Interest Group of the ACL. ACL, 927-936.
  14. Atanas Kiryakov, Borislav Popov, Ivan Terziev, Dimitar Manov, and Damyan Ognyanoff. 2004. Semantic annotation, indexing, and retrieval. J. Web Sem. (2004), 49-79. https://doi.org/10.1016/j.websem.2004.07.005 · DOI
  15. Edgard Marx, Ricardo Usbeck, Axel-Cyrille Ngonga Ngomo, Konrad Höffner, Jens Lehmann, and Sören Auer. 2014. Towards an open question answering architecture. In Proceedings of the 10th International Conference on Semantic Systems, SEMANTICS 2014, Leipzig, Germany, September 4-5, 2014. ACM, 57-60. https://doi.org/10.1145/2660517.2660519 · DOI
  16. Pablo N. Mendes, Max Jakob, Andrés García-Silva, and Christian Bizer. 2011. DBpedia spotlight: shedding light on the web of documents. In Proceedings the 7th International Conference on Semantic Systems, I-SEMANTICS 2011, Graz, Austria, September 7-9, 2011. ACM, 1-8. https://doi.org/10.1145/2063518.2063519 · DOI
  17. Andrea Moro, Alessandro Raganato, and Roberto Navigli. 2014. Entity Linking meets Word Sense Disambiguation: a Unified Approach. TACL 2 (2014), 231-244. https://tacl2013.cs.columbia.edu/ojs/index.php/tacl/article/view/291
  18. Jun-Ping Ng and Min-Yen Kan. 2015. QANUS: An Open-source Question- Answering Platform. CoRR abs/1501.00311 (2015). http://arxiv.org/abs/1501.00311 · arXiv
  19. Muhammad Saleem, Samaneh Nazari Dastjerdi, Ricardo Usbeck, and Axel- Cyrille Ngonga Ngomo. 2017. Question Answering Over Linked Data: What is Difficult to Answer? What Affects the F scores?. In Joint Proceedings of BLINK2017: 2nd International Workshop on Benchmarking Linked Data and NLIWoD3: Nat- ural Language Interfaces for the Web of Data co-located with 16th International Semantic Web Conference (ISWC 2017), Vienna, Austria, October 21st -to -22nd, 2017. CEUR-WS.org. http://ceur-ws.org/Vol-1932/paper-02.pdf
  20. Saeedeh Shekarpour, Edgard Marx, Axel-Cyrille Ngonga Ngomo, and Sören Auer. 2015. SINA: Semantic interpretation of user queries for question answering on interlinked data. J. Web Sem. 30 (2015), 39-51. https://doi.org/10.1016/j.websem. 2014.06.002 · DOI
  21. Kuldeep Singh, Andreas Both, Dennis Diefenbach, and Saeedeh Shekarpour. 2016. Towards a Message-Driven Vocabulary for Promoting the Interoperability of Question Answering Systems. In Tenth IEEE International Conference on Semantic Computing, ICSC 2016, Laguna Hills, CA, USA, February 4-6, 2016. IEEE Computer Society, 386-389. https://doi.org/10.1109/ICSC.2016.59 · DOI
  22. Kuldeep Singh, Andreas Both, Dennis Diefenbach, Saeedeh Shekarpour, Didier Cherix, and Christoph Lange. 2016. Qanary -The Fast Track to Creating a Question Answering System with Linked Data Technology. In The Semantic Web -ESWC 2016 Satellite Events, Heraklion, Crete, Greece, May 29 -June 2, 2016, Revised Selected Papers. 183-188. https://doi.org/10.1007/978-3-319-47602-5_36 · DOI
  23. Kuldeep Singh, Ioanna Lytra, Maria-Esther Vidal, Dharmen Punjani, Harsh Thakkar, Christoph Lange, and Sören Auer. 2017. QAestro -Semantic-Based Composition of Question Answering Pipelines. In Database and Expert Sys- tems Applications -28th International Conference, DEXA 2017, Lyon, France, Au- gust 28-31, 2017, Proceedings, Part I. Springer, 19-34. https://doi.org/10.1007/ 978-3-319-64468-4_2 · DOI
  24. Kuldeep Singh, Isaiah Onando Mulang, Ioanna Lytra, Mohamad Yaser Jaradeh, Ahmad Sakor, Maria-Esther Vidal, Christoph Lange, and Sören Auer. 2017. Cap- turing Knowledge in Semantically-typed Relational Patterns to Enhance Relation Linking. In Proceedings of the Knowledge Capture Conference, K-CAP 2017, Austin, TX, USA, December 4-6, 2017. 31:1-31:8. https://doi.org/10.1145/3148011.3148031 · DOI · Paper page
  25. Priyansh Trivedi, Gaurav Maheshwari, Mohnish Dubey, and Jens Lehmann. 2017. LC-QuAD: A Corpus for Complex Question Answering over Knowledge Graphs. In The Semantic Web -ISWC 2017 -16th International Semantic Web Conference, Vienna, Austria, October 21-25, 2017, Proceedings, Part II. Springer, 210-218. https: //doi.org/10.1007/978-3-319-68204-4_22 · DOI · Paper page
  26. Christina Unger, Lorenz Bühmann, Jens Lehmann, Axel-Cyrille Ngonga Ngomo, Daniel Gerber, and Philipp Cimiano. 2012. Template-based question answering over RDF data. In Proceedings of the 21st World Wide Web Conference 2012, WWW 2012, Lyon, France, April 16-20, 2012. ACM, 639-648. https://doi.org/10.1145/ 2187836.2187923 · DOI
  27. Christina Unger, Corina Forascu, Vanessa López, Axel-Cyrille Ngonga Ngomo, Elena Cabrio, Philipp Cimiano, and Sebastian Walter. 2015. Question Answering over Linked Data (QALD-5). In Working Notes of CLEF 2015 -Conference and Labs of the Evaluation forum, Toulouse, France, September 8-11, 2015. CEUR-WS.org. http://ceur-ws.org/Vol-1391/173-CR.pdf
  28. Ricardo Usbeck, Michael Hoffmann, Michael Röder, Jens Lehmann, and Axel- Cyrille Ngonga Ngomo. 2017. Using Multi-Label Classification for Improved Question Answering. CoRR (submitted) (2017). https://arxiv.org/abs/1710.08634 · arXiv
  29. Ricardo Usbeck, Axel-Cyrille Ngonga Ngomo, Michael Röder, Daniel Gerber, Sandro Athaide Coelho, Sören Auer, and Andreas Both. 2014. AGDISTIS -Graph- Based Disambiguation of Named Entities Using Linked Data. In The Semantic Web -ISWC 2014. Springer, 457-471. https://doi.org/10.1007/978-3-319-11964-9_29 · DOI
  30. Ricardo Usbeck, Michael Röder, Michael Hoffmann, Felix Conrads, Jonathan Huthmann, Axel-Cyrille Ngonga Ngomo, Christian Demmler, and Christina Unger. [n. d.]. Benchmarking Question Answering Systems. Semantic Web Journal (to be published) ([n. d.]). http://www.semantic-web-journal.net/content/ benchmarking-question-answering-systems
  31. Zi Yang, Elmer Garduño, Yan Fang, Avner Maiberg, Collin McCormack, and Eric Nyberg. 2013. Building optimal information systems automatically: configuration space exploration for biomedical information systems. In 22nd ACM International Conference on Information and Knowledge Management, CIKM'13, San Francisco, USA. ACM, 1421-1430.

Cited by

Sorted by citations

Authors
Singh, K., Lange, C., Vidal, M. E., Lehmann, J., Auer, S., Radhakrishna, A. S., Both, A., Shekarpour, S., Lytra, I., Usbeck, R., Vyas, A., Khikmatullaev, A., & Punjani, D.
Affiliations
  • University of Bonn (DE)
  • Leibniz Information Centre for Science and Technology University Library & Fraunhofer IAIS, Hannover, Germany
  • Leibniz University Hannover (DE)
  • University of Minnesota (US)
  • DATEV (Germany) (DE)
  • University of Dayton (US)
  • Paderborn University (DE)
  • National and Kapodistrian University of Athens (GR)
ORCID
Venue
Proceedings of the 2018 World Wide Web Conference on World Wide Web - WWW '18
Publisher
ACM Press
Published
2018
Pages
1247-1256
DOI
10.1145/3178876.3186023
OpenAlex
W2788460763
Cited by
102
References
0
Citations by year
2026: 2 · 2025: 4 · 2024: 2 · 2023: 28 · 2022: 13 · 2021: 16 · 2020: 13 · 2019: 16 · 2018: 8
FWCI
5.39
Type
conference paper
Language
en
Open access
Licence
Locations
  • International World Wide Web Conference (WWW) (publishedVersion, cc-by)
  • Fraunhofer-Publica (Fraunhofer-Gesellschaft) (submittedVersion, other-oa)
  • Fraunhofer-Publica (Fraunhofer-Gesellschaft) (submittedVersion, cc-by)
  • Zenodo (CERN European Organization for Nuclear Research) (submittedVersion, other-oa)
Funders
  • EU H2020 R&I programme for the Marie Skodowska-Curie action (WDAqua (GA No 642795))
  • EU H2020 R&I (HOBBIT (GA688227))
  • Eurostars project (QAMEL (E!9725))
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
Keywords
component selection, Frankenstein, named entity recognition, entity disambiguation, relation extraction, greedy algorithm, LC-QuAD

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review