The DOI matches its Crossref record; 51 of the 52 references it cites were found in a registry or an index; none of the 4 sources its claims cite is checked for support yet.
Generate FAIR Literature Surveys with Scholarly Knowledge Graphs
Open accessAbstract
Reviewing scientific literature is a cumbersome, time consuming but crucial activity in research. Leveraging a scholarly knowledge graph, we present a methodology and a system for comparing scholarly literature, in particular research contributions describing the addressed problem, utilized materials, employed methods and yielded results. The system can be used by researchers to quickly get familiar with existing work in a specific research domain (e.g., a concrete research question or hypothesis). Additionally, it can be used to publish literature surveys following the FAIR Data Principles. The methodology to create a research contribution comparison consists of multiple tasks, specifically: (a) finding similar contributions, (b) aligning contribution descriptions, (c) visualizing and finally (d) publishing the comparison. The methodology is implemented within the Open Research Knowledge Graph (ORKG), a scholarly infrastructure that enables researchers to collaboratively describe, find and compare research contributions. We evaluate the implementation using data extracted from published review articles. The evaluation also addresses the FAIRness of comparisons published with the ORKG.
Overview
CoChat Summary
TL;DR
The paper presents a methodology and ORKG-based system for generating and publishing FAIR literature surveys by comparing structured research contributions from a scholarly knowledge graph. Evaluation shows original survey tables can be recreated without information loss, published comparisons rank high in FAIRness, and the ORKG alignment approach outperforms a naive comparison method.
Key findings
- Using data imported from four review papers (12 tables, 169 papers, 3,750 statements), the ORKG comparison service could recreate the same tabular views as originally published, with no information loss reported for the imported tables.
- Published comparisons rank high in FAIRness: the service provides unique identifiers, machine- and user-generated metadata, HTTP access without authentication, separate metadata storage, RDF/CSV exports, and a CC BY-SA license, though interoperability and reusability still require user effort.
- The ORKG approach outperforms a naive property-alignment approach; for 8 compared contributions, Table 3 reports 14,421 seconds for the naive approach versus 0.0204 seconds for ORKG.
Methods
- Design: a five-step comparison methodology implemented in the Open Research Knowledge Graph (ORKG): finding comparison candidates, selecting related statements, aligning contribution descriptions, visualizing comparisons, and publishing FAIR comparisons.
- Data/sample: the ORKG was populated by importing tabular survey data from four review papers, yielding 12 tables, 169 reviewed papers, and 3,750 statements; evaluation compared original tables with ORKG-generated tables, assessed FAIR sub-principles, and benchmarked performance.
- Main analysis: similar contributions are found with TF-IDF; related statements are selected transitively to depth δ = 5; properties are aligned using pre-trained fastText embeddings with similarity threshold τ ≥ 0.9; visualization displays properties shared by at least α ≥ 2 contributions by default; performance compared naive versus ORKG alignment for 2–8 contributions with on average ten properties per paper.
Limitations
- Stated in the paper: the ORKG does not yet contain sufficient contribution descriptions for the comparison functionality to be practically useful for researchers, and more contribution data is needed to evaluate components such as finding related papers or aligning similar properties.
- Stated in the paper: the information representation evaluation does not give insight into usefulness or usability; FAIR is not a standard and its principles are permissive and not prescriptive, so implementation and evaluation are subject to interpretation; interoperability and reusability can be improved, users are responsible for correct metadata, and there is currently no community standard to describe comparison data.
- Evident from the text: the performance evaluation is limited to eight papers because the naive approach does not scale, and the imported evaluation data comes from only four review papers and 12 tables; no user study is reported.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures






Verification
Claims the paper makes
0 Valid · 13 Not valid · 3 Not checked · 63 Own results and interpretations
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Data is (partially) described using the ORKG core ontology and other ontologies we use to canonicalize the representation of relevant information content types.”
§ Generate FAIR Literature Surveys with Scholarly Knowledge Graphs > 6.2 FAIR data evaluation
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In the naive approach, the”
§ Generate FAIR Literature Surveys with Scholarly Knowledge Graphs > 6.3 Performance evaluation
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This methodology has been used to populate the ORKG with comparable paper data.”
§ Generate FAIR Literature Surveys with Scholarly Knowledge Graphs > 5.1 Leverage legacy review paper tables
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Otherwise, findability, accessibility and to some extent also interoperability are largely handled by the service.”
§ Generate FAIR Literature Surveys with Scholarly Knowledge Graphs > 6.2 FAIR data evaluation
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The evaluation of the FAIR principles shows that comparisons published with our service rank high in FAIRness, which can be even further increased with some effort from users.”
§ Generate FAIR Literature Surveys with Scholarly Knowledge Graphs > 6.2 FAIR data evaluation
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
21 more claims
Create a free account to see all 79 claims and every source verdict.
- It may state two separate claims as one. 17
- Probably not a substantial claim. 49
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The Open Research Knowledge Graph (ORKG) (Jaradeh, et al., 2019) addresses this problem by representing scholarly knowledge in a machine actionable, structured, and semantic manner, and providing services for comparing scholarly knowledge (Oelen et al., 2020) in tabular and other visual forms (Wiens et al., 2020) as well as other types of services for efficient reuse of scholarly knowledge.”
From Persistent Identification and Interlinking of FAIR Scholarly Knowledge (2022) · cites this paper as Oelen et al., 2020
Citations
Cites
The FAIR Guiding Principles for scientific data management and stewardship
Wilkinson, M. D., Dumontier, M., Aalbersberg, I. J., Appleton, G., Axton, M., Baak, A., Blomberg, N., Boiten, J.-W., da Silva Santos, L. B., Bourne, P. E., et al. (2016) · Scientific Data · doi:10.1038/sdata.2016.18 · Cited by 18,423
Binary Codes capable of currecting deletions, insertions, and reversals
Levenshtein, V. I. (1965) · Soviet physics. Doklady · Cited by 14,807
Enriching Word Vectors with Subword Information
Bojanowski, P., Grave, E., Joulin, A., & Mikolov, T. (2017) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00051 · Cited by 9,645
Analyzing the Past to Prepare for the Future: Writing a Literature Review
Webster, J., & Watson, R. T. (2002) · MIS Quarterly · doi:10.2307/4132319 · Cited by 7,449
Educational Research: An Introduction
Bennett, N., Borg, W. R., & Gall, M. D. (1984) · British Journal of Educational Studies · doi:10.2307/3121583 · Cited by 5,595
Writing Integrative Literature Reviews: Guidelines and Examples
Torraco, R. J. (2005) · Human Resource Development Review · doi:10.1177/1534484305278283 · Cited by 3,534
(1961) · Journal of the Optical Society of America · doi:10.1364/josa.51.000248 · Cited by 2,288
Mitzias, P., Riga, M., Waddington, S. N., Kontopoulos, E., Μεδίτσκος, Γ., Laurenson, P., & Kompatsiaris, I. Y. (2015) · Cited by 1,240
Ontology Matching: State of the Art and Future Challenges
Shvaiko, P., & Euzenat, J. (2013) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2011.253 · Cited by 1,148
String Comparator Metrics and Enhanced Decision Rules in the Fellegi-Sunter Model of Record Linkage.
Winkler, W. E. (1990) · Cited by 787
A Guide to Writing the Dissertation Literature Review
Randolph, J. (2020) · ScholarWorks@UMassAmherst (University of Massachusetts Amherst) · doi:10.7275/b0az-8t74 · Cited by 722
How to write a literature review paper?
Wee, B. V., & Banister, D. (2016) · Transport Reviews · doi:10.1080/01441647.2015.1065456 · Cited by 578
Kohl, C., McIntosh, E. J., Unger, S., Haddaway, N. R., Kecke, S., Schiemann, J., & Wilhelm, R. (2018) · Environmental Evidence · doi:10.1186/s13750-018-0115-5 · Cited by 471
Shedding Light on the Dark Data in the Long Tail of Science
Heidorn, P. B. (2008) · Library trends · doi:10.1353/lib.0.0036 · Cited by 470 · cited 1×
“structured) information than what is presented in the published table. However, the additional information is not published. It is “dark data” which is not stored or indexed and likely lost over time [12]. Furthermore, published tables are not machine actionable. Their overall low FAIRness hinders reusability of the p…”
BackgroundMons, B., Neylon, C., Velterop, J., Dumontier, M., da Silva Santos, L. O. B., & Wilkinson, M. D. (2017) · Information Services and Use · doi:10.3233/isu-170824 · Cited by 417
Core techniques of question answering systems over knowledge bases: a survey
Diefenbach, D., López, V., Singh, K., & Maret, P. (2017) · Knowledge and Information Systems · doi:10.1007/s10115-017-1100-y · Cited by 406
How to Write a Literature Review
Denney, A. S., & Tewksbury, R. (2013) · Journal of Criminal Justice Education · doi:10.1080/10511253.2012.730617 · Cited by 364
The FAIR guiding principles for data stewardship: fair enough?
Boeckhout, M., Zielhuis, G. A., & Bredenoord, A. L. (2018) · European Journal of Human Genetics · doi:10.1038/s41431-018-0160-0 · Cited by 290
FaBiO and CiTO: Ontologies for describing bibliographic resources and citations
Peroni, S., & Shotton, D. (2012) · Journal of Web Semantics · doi:10.1016/j.websem.2012.08.001 · Cited by 212 · cited 1×
“Semantic Publishing and Referencing (SPAR)1 is a collection of ontologies that can be used to describe scholarly publishing and referencing of documents [5, 9, 26, 27].”
BackgroundOpen Research Knowledge Graph: Next Generation Infrastructure for Semantic Scholarly Knowledge
Jaradeh, M. Y., Oelen, A., Farfar, K. E., Prinz, M., D'Souza, J., Kismihók, G., Stocker, M., & Auer, S. (2019) · Proceedings of the 10th International Conference on Knowledge Capture · doi:10.1145/3360901.3364435 · Cited by 212
Self-citationAdomavičius, G., & Kwon, Y. (2011) · ACM Conference on Recommender Systems (RecSys) · Cited by 209
Lasda Bergman, E. M. (2012) · The Journal of Academic Librarianship · doi:10.1016/j.acalib.2012.08.002 · Cited by 208
Conducting a Literature Review — The Example of Sustainability in Supply Chains
Seuring, S. A., Müller, M., Westhaus, M., & Morana, R. (2005) · Physica-Verlag eBooks · doi:10.1007/3-7908-1636-1_7 · Cited by 195
Proceedings of the 33rd Annual ACM Symposium on Applied Computing
van Roosmalen, J., Vranken, H. P. E., van Eekelen, R. N., Haddad, H. M., Wainwright, R. L., & Chbeir, R. (2018) · doi:10.1145/3167132 · Cited by 154
Achieving human and machine accessibility of cited data in scholarly publications
Starr, J., Castro, E., Crosas, M., Dumontier, M., Downs, R. R., Duerr, R. E., Laurel, H., Haendel, M. A., Herman, I., Hodson, S., et al. (2015) · PeerJ Computer Science · doi:10.7717/peerj-cs.1 · Cited by 144
Exploration and Visualization in the Web of Big Linked Data: A Survey of the State of the Art
Bikakis, N., & Sellis, T. (2016) · Swinburne figshare (Swinburne University of Technology) · doi:10.25916/sut.26223617 · Cited by 96
Peroni, S., & Shotton, D. M. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-00668-6_8 · Cited by 94
Decentralized provenance-aware publishing with nanopublications
Kuhn, T., Chichester, C., Krauthammer, M., Queralt-Rosinach, N., Verborgh, R., Giannakopoulos, G., Ngonga Ngomo, A.-C., Viglianti, R., & Dumontier, M. (2016) · PeerJ Computer Science · doi:10.7717/peerj-cs.78 · Cited by 90
McIntosh, E. J., Chapman, S., Kearney, S. G., Williams, B. A., Althor, G., Thorn, J., Pressey, R. L., McKinnon, M., & Grenyer, R. (2018) · Environmental Evidence · doi:10.1186/s13750-018-0134-2 · Cited by 87
The Document Components Ontology (DoCO)
Constantin, A., Peroni, S., Pettifer, S. R., Shotton, D. M., & Vitali, F. (2016) · Semantic Web · doi:10.3233/sw-150177 · Cited by 86
SERIMI - Resource Description Similarity, RDF Instance Matching and InterlinkingPreprint
Araújo, S., Hidders, J., Schwabe, D., & de Vries, A. P. (2011) · arXiv (Cornell University) · doi:10.48550/arxiv.1107.1104 · Cited by 75
Towards a Knowledge Graph Representing Research Findings by Semantifying Survey Articles
Fathalla, S., Vahdati, S., Auer, S., & Lange, C. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-67008-9_25 · Cited by 71
Self-citationA survey of author name disambiguation techniques: 2010–2016
Hussain, I., & Asghar, S. (2017) · The Knowledge Engineering Review · doi:10.1017/s0269888917000182 · Cited by 67
Digital Object Identifier (DOI®) System
(2017) · CRC Press eBooks · doi:10.1081/e-elis4-120044418 · Cited by 64
Publishing FAIR Data: An Exemplar Methodology Utilizing PHI-Base
Rodríguez-Iglesias, A., Rodríguez‐González, A., Irvine, A. G., Sesma, A., Urban, M., Hammond-Kosack, K. E., & Wilkinson, M. D. (2016) · Frontiers in Plant Science · doi:10.3389/fpls.2016.00641 · Cited by 52
Digital Object Identifier (DOI®) System
Paskin, N. (2009) · CRC Press eBooks · doi:10.1081/e-elis3-120044418 · Cited by 42
Text Summarization with Automatic Keyword Extraction in Telugu e-Newspapers
Naidu, R., Bharti, S. K., Babu, K. S., & Mohapatra, R. K. (2017) · Smart innovation, systems and technologies · doi:10.1007/978-981-10-5544-7_54 · Cited by 36
Sateli, B., & Witte, R. (2015) · PeerJ Computer Science · doi:10.7717/peerj-cs.37 · Cited by 35
Designing Scientific Knowledge Infrastructures: The Contribution of Epistemology
Hars, A. (2001) · Information Systems Frontiers · doi:10.1023/a:1011401704862 · Cited by 30
Ziemba, P., Jankowski, J., & Wątróbski, J. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-67077-5_56 · Cited by 28
The Publishing Workflow Ontology (PWO)
Aldo Gangemi, A., Peroni, S., Shotton, D. M., & Vitali, F. (2016) · Semantic Web · doi:10.3233/sw-160230 · Cited by 27
Open Research Knowledge Graph: A System Walkthrough
Jaradeh, M. Y., Oelen, A., Prinz, M., Stocker, M., & Auer, S. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-30760-8_31 · Cited by 27
Self-citationA review of ontologies for describing scholarly and scientific documents.
Ruiz‐Iniesta, A., & Corcho, Ó. (2014) · Cited by 24
Comparing Pretrained Multilingual Word Embeddings on an Ontology Alignment Task
Gromann, D., & Declerck, T. (2018) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/595h4gjdxwfr · Cited by 24
Entity Comparison in RDF Graphs
Petrova, A., Sherkhonov, E., Grau, B. C., & Horrocks, I. R. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-68288-4_31 · Cited by 21
Measuring structural similarity between RDF graphs
Maillot, P., & Bobed, C. (2018) · ACM Symposium on Applied Computing · doi:10.1145/3167132.3167342 · Cited by 18
Semantic Representation of Scientific Publications
Vahdati, S., Fathalla, S., Auer, S., Lange, C., & Vidal, M.-E. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-30760-8_37 · Cited by 16
Self-citationComparing research contributions in a scholarly knowledge graph
Oelen, A., Jaradeh, M. Y., Farfar, K. E., Stocker, M., & Auer, S. (2019) · Institutional Repository of Leibniz Universität Hannover (Leibniz Universität Hannover) · doi:10.15488/9388 · Cited by 14
Self-citationHow to learn professional competencies via blogs
Medina, C. P., & Ramon, M. R. R. (2015) · The New Educational Review · doi:10.15804/tner.2015.42.4.03 · Cited by 13
Westbrook, D. L. (2010) · Oxford University Press eBooks · doi:10.1093/med:psych/9780199561308.003.0018 · Cited by 7
Winkler, W. E. (1992) · Springer eBooks · doi:10.1007/978-1-4612-2856-1_101 · Cited by 4
Cited by
Machine Knowledge: Creation and Curation of Comprehensive Knowledge Bases
Weikum, G., Dong, X. L., Razniewski, S., & Suchanek, F. M. (2021) · Foundations and Trends in Databases · doi:10.1561/1900000064 · Cited by 202
Improving Access to Scientific Literature with Knowledge Graphs
Auer, S., Oelen, A., Haris, M., Stocker, M., D’Souza, J., Farfar, K. E., Vogt, L., Prinz, M., Wiens, V., & Jaradeh, M. Y. (2020) · Bibliothek Forschung und Praxis · doi:10.1515/bfp-2020-2042 · Cited by 82
Self-citationScholarly knowledge graphs through structuring scholarly communication: a review
Verma, S., Bhatia, R., Harit, S., & Batish, S. (2023) · Complex & Intelligent Systems · doi:10.1007/s40747-022-00806-6 · Cited by 78
Azher, I. A., Seethi, V. D. R., Akella, A. P., & Alhoori, H. (2024) · ACM/IEEE Joint Conference on Digital Libraries · doi:10.1145/3677389.3702605 · Cited by 73
Brack, A., Hoppe, A., Stocker, M., Auer, S., & Ewerth, R. (2021) · International Journal on Digital Libraries · doi:10.1007/s00799-021-00306-x · Cited by 37
Self-citation
Metadata
- Authors
- Oelen, A., Jaradeh, M. Y., Stocker, M., & Auer, S.
- Affiliations
- Leibniz University Hannover (DE)
- Technische Informationsbibliothek (TIB) (DE)
- Venue
- Proceedings of the ACM/IEEE Joint Conference on Digital Libraries in 2020
- Publisher
- ACM
- Published
- 2020-08
- Pages
- 97-106
- ISSN
- 1552-5996 (ISSN-L)
- OpenAlex
- W3033889636
- Cited by
- 52
- References
- 52
- Citations by year
- 2026: 3 · 2025: 7 · 2024: 11 · 2023: 7 · 2022: 9 · 2021: 10 · 2020: 5
- FWCI
- 2.05
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- ACM/IEEE Joint Conference on Digital Libraries (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by-sa)
- RENATE (submittedVersion, cc-by-nc-nd)
- arXiv (Cornell University)
- RENATE
- Funders
- H2020 European Research Council (819536)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Semantic Web and Ontologies
- Social Sciences › Arts and Humanities › Literature and Literary Theory › Digital Humanities and Scholarship
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Keywords
- scholarly knowledge graphs, literature survey, FAIR principles, Open Research Knowledge Graph, similarity search, knowledge graph alignment, literature visualization, fairness evaluation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-11-18, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-18 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.