Quality assessment for Linked Data: A Survey: A systematic literature review and conceptual framework
Open accessAbstract
The development and standardization of Semantic Web technologies has resulted in an unprecedented volume of data being published on the Web as Linked Data (LD). However, we observe widely varying data quality ranging from extensively curated datasets to crowdsourced and extracted data of relatively low quality. In this article, we present the results of a systematic review of approaches for assessing the quality of LD. We gather existing approaches and analyze them qualitatively. In particular, we unify and formalize commonly used terminologies across papers related to data quality and provide a comprehensive list of 18 quality dimensions and 69 metrics. Additionally, we qualitatively analyze the 30 core approaches and 12 tools using a set of attributes. The aim of this article is to provide researchers and data curators a comprehensive understanding of existing work, thereby encouraging further experimentation and development of new approaches focused towards data quality, specifically for LD.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Preferred reporting items for systematic reviews and meta-analyses: the PRISMA Statement.
(2009) · PubMed Central · Cited by 44,859 · see citing works
(2004) · Cited by 5,198 · see citing works
Beyond Accuracy: What Data Quality Means to Data Consumers
Wang, R. Y., & Strong, D. M. (1996) · Journal of Management Information Systems · doi:10.1080/07421222.1996.11518099 · Cited by 4,672 · see citing works
DBpedia - A large-scale, multilingual knowledge base extracted from Wikipedia
Lehmann, J., Isele, R., Jakob, M., Jentzsch, A., Kontokostas, D., Mendes, P. N., Hellmann, S., Morsey, M., van Kleef, P., Auer, S., et al. (2015) · Semantic Web · doi:10.3233/sw-140134 · Cited by 3,737 · see citing works
Self-citationLinked Data: Evolving the Web into a Global Data Space
Heath, T., & Bizer, C. (2011) · Synthesis lectures on the semantic web · doi:10.2200/s00334ed1v01y201102wbe001 · Cited by 2,029 · see citing works
Bleiholder, J., & Naumann, F. (2009) · ACM Computing Surveys · doi:10.1145/1456650.1456651 · Cited by 1,874 · see citing works
Pipino, L. L., Lee, Y. W., & Wang, R. Y. (2002) · Communications of the ACM · doi:10.1145/505248.506010 · Cited by 1,724 · see citing works
AIMQ: a methodology for information quality assessment
Lee, Y. W., Strong, D. M., Kahn, B. K., & Wang, R. Y. (2002) · Information & Management · doi:10.1016/s0378-7206(02)00043-5 · Cited by 1,611 · see citing works
Anchoring Data Quality Dimensions in Ontological Foundations
Wand, Y., & Wang, R. Y. (1996) · Communications of the ACM · doi:10.1145/240455.240479 · Cited by 1,479 · see citing works
Methodologies for data quality assessment and improvement
Batini, C., Cappiello, C., Francalanci, C., & Maurino, A. (2009) · ACM Computing Surveys · doi:10.1145/1541880.1541883 · Cited by 1,351 · see citing works
Self-citationA product perspective on total data quality management
Wang, R. Y. (1998) · Communications of the ACM · doi:10.1145/269012.269022 · Cited by 911 · see citing works
Data Quality For The Information Age
Redman, T. C., & Godfrey, A. B. (1997) · Cited by 689 · see citing works
Data Quality : Concepts, Methodologies and Techniques
Batini, C., & Scannapieco, M. (2006) · doi:10.1007/3-540-33173-5 · Cited by 664 · see citing works
Fundamentals of Data Warehouses
Jarke, M., Maurizio Lenzerini, B., Vassiliou, Y., & Vassiliadis, P. (2003) · Springer eBooks · doi:10.1007/978-3-662-05153-5 · Cited by 450 · see citing works
Trust Networks on the Semantic Web
Golbeck, J., Parsia, B., & Hendler, J. (2003) · Lecture notes in computer science · doi:10.1007/978-3-540-45217-1_18 · Cited by 442 · see citing works
Developing a Framework for Assessing Information Quality on the World Wide Web
Knight, S.-A., & Burn, J. M. (2005) · Informing Science The International Journal of an Emerging Transdiscipline · doi:10.28945/493 · Cited by 355 · see citing works
When owl:sameAs Isn’t the Same: An Analysis of Identity in Linked Data
Halpin, H., Hayes, P. J., McCusker, J. P., et al. (2010) · Lecture notes in computer science · doi:10.1007/978-3-642-17746-0_20 · Cited by 306 · see citing works
A conceptual framework and belief-function approach to assessing overall information quality
Bovee, M. W., Srivastava, R. P., & Mak, B. L. (2003) · International Journal of Intelligent Systems · doi:10.1002/int.10074 · Cited by 290 · see citing works
Quality-Driven Query Answering for Integrated Information Systems
Naumann, F. (2002) · Lecture notes in computer science · doi:10.1007/3-540-45921-9 · Cited by 259 · see citing works
Test-driven evaluation of linked data quality
Kontokostas, D., Westphal, P., Auer, S., et al. (2014) · International World Wide Web Conference (WWW) · doi:10.1145/2566486.2568002 · Cited by 254 · see citing works
Self-citationMendes, P. N., Mühleisen, H., & Bizer, C. (2012) · doi:10.1145/2320765.2320803 · Cited by 236 · see citing works
Quality-driven information filtering using the WIQA policy framework
Bizer, C., & Cyganiak, R. (2008) · Journal of Web Semantics · doi:10.1016/j.websem.2008.02.005 · Cited by 219 · see citing works
An empirical survey of Linked Data conformance
Hogan, A., Umbrich, J., Harth, A., et al. (2012) · Journal of Web Semantics · doi:10.1016/j.websem.2012.02.001 · Cited by 186 · see citing works
Improving the Quality of Linked Data Using Statistical Distributions
Paulheim, H., & Bizer, C. (2014) · International Journal on Semantic Web and Information Systems · doi:10.4018/ijswis.2014040104 · Cited by 181 · see citing works
Hogan, A., Harth, A., Passant, A., et al. (2010) · ARAN (University of Galway Research Repository) (Ollscoil na Gaillimhe – University of Galway) · doi:10.13025/21354 · Cited by 166 · see citing works
Characterizing the Semantic Web on the Web
Ding, L., & Finin, T. (2006) · Lecture notes in computer science · doi:10.1007/11926078_18 · Cited by 156 · see citing works
Trusting Information Sources One Citizen at a Time
Gil, Y., & Ratnakar, V. (2002) · Lecture notes in computer science · doi:10.1007/3-540-48005-6_14 · Cited by 149 · see citing works
User-driven quality evaluation of DBpedia
Zaveri, A. J., Kontokostas, D., Sherif, M. A., Bühmann, L., Morsey, M., Auer, S., & Lehmann, J. (2013) · International Conference on Semantic Systems · doi:10.1145/2506182.2506195 · Cited by 140 · see citing works
Self-citationAssessing Linked Data Mappings Using Network Measures
Guéret, C., Groth, P. T., Stadler, C., et al. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-30284-8_13 · Cited by 132 · see citing works
Metrics for Evaluation of Ontology-based Information Extraction
Maynard, D., Peters, W., & Li, Y. (2006) · Cited by 128 · see citing works
Crowdsourcing Linked Data Quality Assessment
Acosta, M., Zaveri, A. J., Simperl, E. P. B., et al. (2013) · Lecture notes in computer science · doi:10.1007/978-3-642-41338-4_17 · Cited by 127 · see citing works
Self-citationUsing web data provenance for quality assessment
Hartig, O., & Zhao, J. (2009) · Lancaster EPrints (Lancaster University) · Cited by 123 · see citing works
Developing a Framework for Assessing Information Quality on the World Wide Web
Knight, S.-A., & Burn, J. M. (2005) · Informing Science and IT Education Conference · doi:10.28945/2854 · Cited by 123 · see citing works
Towards content trust of web resources
Gil, Y., & Artz, D. (2007) · Journal of Web Semantics · doi:10.1016/j.websem.2007.09.005 · Cited by 119 · see citing works
DBpedia and the live extraction of structured data from Wikipedia
Morsey, M., Lehmann, J., Auer, S., et al. (2012) · Data Technologies and Applications · doi:10.1108/00330331211221828 · Cited by 111 · see citing works
Self-citationCarroll, J. J. (2003) · Lecture notes in computer science · doi:10.1007/978-3-540-39718-2_24 · Cited by 99 · see citing works
Detecting Incorrect Numerical Data in DBpedia
Wienand, D., & Paulheim, H. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-07443-6_34 · Cited by 83 · see citing works
SWIQA – A SEMANTIC WEB INFORMATION QUALITY ASSESSMENT FRAMEWORK
Fürber, C., & Hepp, M. F. (2011) · European Conference on Information Systems · Cited by 78 · see citing works
Quality-Driven Information Filtering in the Context of Web-Based Information Systems
Bizer, C. (2007) · Universitätsbibliothek der FU Berlin Hochschulschriftenstelle u. Dokumentenserver · doi:10.17169/refubium-14260 · Cited by 77 · see citing works
Modeling and evaluation of trust with an extension in semantic web
Shekarpour, S., & Katebi, S. D. (2009) · Journal of Web Semantics · doi:10.1016/j.websem.2009.11.003 · Cited by 65 · see citing works
Robust and scalable Linked Data reasoning incorporating provenance and trust annotations
Bonatti, P. A., Hogan, A., Polleres, A., & Sauro, L. (2011) · Journal of Web Semantics · doi:10.1016/j.websem.2011.06.003 · Cited by 64 · see citing works
Profiling linked open data with ProLOD
Böhm, C., Naumann, F., Abedjan, Z., et al. (2010) · IEEE International Conference on Data Engineering Workshops (ICDEW) · doi:10.1109/icdew.2010.5452762 · Cited by 53 · see citing works
On the Diversity and Availability of Temporal Information in Linked Open Data
Rula, A., Palmonari, M., Harth, A., et al. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-35176-1_31 · Cited by 47 · see citing works
Self-citationQuality, trust, and utility of scientific data on the web
Gamble, M., & Goble, C. (2011) · ACM Web Science Conference (WebSci) · doi:10.1145/2527031.2527048 · Cited by 44 · see citing works
Using Trust and Provenance for Content Filtering on the Semantic Web
Golbeck, J., & Mannes, A. (2006) · Cited by 37 · see citing works
A framework for evaluating semantic metadata
Lei, Y., Uren, V. S., & Motta, E. (2007) · International Conference on Knowledge Capture · doi:10.1145/1298406.1298431 · Cited by 32 · see citing works
An ontology-based method for quality assessment of spatial data bases
Mostafavi, M. A., Edwards, G., & Jeansoulin, R. (2004) · HAL (Le Centre pour la Communication Scientifique Directe) · Cited by 30 · see citing works
Improving Curated Web-Data Quality with Structured Harvesting and Assessment
Feeney, K. C., O’Sullivan, D., Tai, W., & Brennan, R. (2014) · International Journal on Semantic Web and Information Systems · doi:10.4018/ijswis.2014040103 · Cited by 28 · see citing works
Rule-Based Trust Assessment on the Semantic Web
Jacobi, I., Kagal, L., & Khandelwal, A. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-22546-8_18 · Cited by 26 · see citing works
Analyzing Linked Data Quality with LiQuate
Edna Ruckhaus, A., Vidal, M.-E., Castillo, S., Burguillos, O., & Baldizán, O. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-11955-7_72 · Cited by 25 · see citing works
Hybrid Acquisition of Temporal Scopes for RDF Data
Rula, A., Palmonari, M., Ngomo, A.-C. N., et al. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-07443-6_33 · Cited by 23 · see citing works
Self-citationTracing where and who provenance in Linked Data: A calculus
Mariangiola Dezani-ciancaglini, A., Horne, R., & Vladimiro Sassone, C. (2012) · Theoretical Computer Science · doi:10.1016/j.tcs.2012.06.020 · Cited by 23 · see citing works
Methodology for assessment of linked data quality
Rula, A., & Zaveri, A. J. (2014) · BOA (University of Milano-Bicocca) · Cited by 20 · see citing works
Self-citationAssessing linkset quality for complementing third-party datasets
Albertoni, R., & Gómez-Pérez, A. (2013) · doi:10.1145/2457317.2457327 · Cited by 19 · see citing works
I18n of Semantic Web Applications
Auer, S., Weidl, M., Lehmann, J., et al. (2010) · Lecture notes in computer science · doi:10.1007/978-3-642-17749-1_1 · Cited by 18 · see citing works
Self-citationAnalyzing Linked Data Quality with LiQuate
Ruckhaus, E., Baldizán, O., & Vidal, M.-E. (2013) · Lecture notes in computer science · doi:10.1007/978-3-642-41033-8_80 · Cited by 17 · see citing works
Capturing the Age of Linked Open Data: Towards a Dataset-Independent Framework
Rula, A., Palmonari, M., & Maurino, A. (2012) · IEEE International Conference on Semantic Computing (ICSC) · doi:10.1109/icsc.2012.17 · Cited by 17 · see citing works
Self-citationDetecting Quality Problems in Semantic Metadata without the Presence of a Gold Standard.
Lei, Y., & Nikolov, A. (2007) · Cited by 12 · see citing works
Hypothesis generation and data quality assessment through association mining
Chen, P., & Garcia, W. (2010) · IEEE International Conference on Cognitive Informatics · doi:10.1109/coginf.2010.5599828 · Cited by 9 · see citing works
Facilitating Ontology Reuse Using User-Based Ontology Evaluation
Lewen, H. (2010) · Repository KITopen (Karlsruhe Institute of Technology) · doi:10.5445/ir/1000020951 · Cited by 1 · see citing works
Flagged first, then by citations
Research works printed but not linked: 8
- [35]35. Franz Baader, Calvanese Diageo, Deborah McGuinness, Daniele Nardi, and Peter Patel-Schneider, editors. The De- scription Logic Handbook. Cambridge, 2003.
- [40]40. Jens Bleiholder and Felix Naumann. Data fusion. ACM Com- puting Surveys (CSUR), 41(1):1, 2008.
- [58]58. Peter F. Patel-Schneider, Yue Pan, Pascal Hitzler, Peter Mika, Lei Zhang, Jeff Z. Pan, Ian Horrocks, and Birte Glimm, edi- tors, Proceedings of the 9th International Semantic Web Con- ference, ISWC, Lecture Notes in Computer Science, pages 53- 59. Springer Berlin Heidelberg, 2010.
- [59]59. Olaf Hartig. Trustworthiness of Data on the Web. In Di- eter Fensel and Elena Simperl, editors, Proceedings of the STI Berlin and CSW PhD Workshop, co-located with Xinnovations, Berlin, Germany, 2008.
- [66]66. Joseph M. Juran, Richard S. Bingham, and Frank M. Gryna. The Quality Control Handbook. Rainbow-Bridge, 3 edition, 1974.
- [73]73. David Kopcso Leo Pipino, Ricahrd Wang and William Rybold, editors. Information Quality, volume 1. Routledge, 2005.
- [83]83. Leo L. Pipino, Yang W. Lee, and Richarg Y. Wang. Data qual- ity assessment. Communications of the ACM, 45(4), 2002.
- [92]92. Denny Vrandecic. Ontology Evaluation. PhD thesis, Karl- sruher Instituts für Technologie (KIT), 2010.
Cited by
Sorted by citations
Survey of conversational agents in health
Montenegro, J. L. Z., da Costa, C. A., & da Rosa Righi, R. (2019) · Expert Systems with Applications · doi:10.1016/j.eswa.2019.03.054 · Cited by 322 · see citing works
A Survey of Data Quality Requirements That Matter in ML Development Pipelines
Priestley, M., O’donnell, F., & Simperl, E. (2023) · Journal of Data and Information Quality · doi:10.1145/3592616 · Cited by 86 · see citing works
Multicriteria Based Decision Making of DevOps Data Quality Assessment Challenges Using Fuzzy TOPSIS
Rafi, S., Yu, W., Akbar, M. A., Alsanad, A., & Gumaei, A. (2020) · IEEE Access · doi:10.1109/access.2020.2976803 · Cited by 37 · see citing works
Bayram, F., Ahmed, B. S., & Hallin, E. (2024) · Journal of Systems and Software · doi:10.1016/j.jss.2024.112184 · Cited by 16 · see citing works
El Morr, C., Maret, P., Muhlenbach, F., Dharmalingam, D., Tadesse, R., Creighton, A., Kundi, B., Buettgen, A., Mgwigwi, T., Dinca-Panaitescu, S., et al. (2021) · JMIR Formative Research · doi:10.2196/33335 · Cited by 12 · see citing works
- Authors
- Zaveri, A., Rula, A., Maurino, A., Pietrobon, R., Lehmann, J., & Auer, S.
- Affiliations
- Leipzig University (DE)
- University of Milano-Bicocca (IT)
- Duke University (US)
- University of Bonn (DE)
- Corresponding author
- Zaveri, Amrapali
- Venue
- Semantic Web
- Publisher
- SAGE Publications
- Published
- 2015-03-17
- Volume
- 7
- Issue
- 1
- Pages
- 63-93
- ISSN
- 1570-0844 (ISSN-L)
- 2210-4968
- OpenAlex
- W1870959433
- Cited by
- 596
- References
- 60
- Citations by year
- 2026: 24 · 2025: 23 · 2024: 33 · 2023: 54 · 2022: 47 · 2021: 61 · 2020: 76 · 2019: 66 · 2018: 70 · 2017: 73 · 2016: 46 · 2015: 21 · 2014: 2
- FWCI
- 68.51
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- other-oa
- Locations
- Semantic Web (publishedVersion, other-oa)
- BOA (University of Milano-Bicocca) (submittedVersion, other-oa)
- Research Publications (Maastricht University) (submittedVersion)
- Institutional Research Information System (Università degli Studi di Brescia) (submittedVersion)
- Fraunhofer-Publica (Fraunhofer-Gesellschaft) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Semantic Web and Ontologies
- Social Sciences › Decision Sciences › Management Science and Operations Research › Data Quality and Management
- Physical Sciences › Computer Science › Information Systems › Library Science and Information Systems
- Keywords
- linked data, data quality assessment, quality metrics, quality dimensions, quality terminology, quality management tools, crowdsourced data, data curation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
