On the measurability of information quality
Abstract
The notion of information quality (IQ) has been investigated extensively in recent years. Much of this research has been aimed at conceptualizing IQ and its underlying dimensions (e.g., accuracy, completeness) and at developing instruments for measuring these quality dimensions. However, less attention has been given to the measurability of IQ. The objective of this study is to explore the extent to which a set of IQ dimensions—accuracy, completeness, objectivity, and representation—lend themselves to reliable measurement. By reliable measurement, we refer to the degree to which independent assessors are able to agree when rating objects on these various dimensions. Our study reveals that multiple assessors tend to agree more on certain dimensions (e.g., accuracy) while finding it more difficult to agree on others (e.g., completeness). We argue that differences in measurability stem from properties inherent to the quality dimension (i.e., the availability of heuristics that make the assessment more tangible) as well as on assessors' reliance on these cues. Implications for theory and practice are discussed.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
High-quality information ensures that consumer reviews are perceived as genuine, unbiased, and helpful.
“Firstly, regarding trust in reviews, high-quality information ensures that consumer reviews are perceived as genuine, unbiased, and helpful.”
From Reviews, trust, and customer experience in online marketplaces: the case of Mercado Libre Colombia (2024) · cites this paper as Arazy and Kopak (2011)
Information quality is crucial in online reviews as it builds trust.
“Information quality is crucial in online reviews as it builds trust (Arazy and Kopak, 2011; Yaari etal., 2011; Huang etal., 2015).”
From Reviews, trust, and customer experience in online marketplaces: the case of Mercado Libre Colombia (2024) · cites this paper as Arazy and Kopak (2011)
Cites
Evaluating Structural Equation Models with Unobservable Variables and Measurement Error
Fornell, C., & Larcker, D. F. (1981) · Journal of Marketing Research · doi:10.2307/3151312 · Cited by 109,314 · see citing works
Evaluating structural equation models with unobservable variables and measurement error
Fornell, C., & Larcker, D. F. (1981) · Journal of Marketing Research · doi:10.1177/002224378101800104 · Cited by 109,190 · see citing works
The Measurement of Observer Agreement for Categorical Data
Landis, J. R., & Koch, G. G. (1977) · Biometrics · doi:10.2307/2529310 · Cited by 81,148 · see citing works
A Coefficient of Agreement for Nominal Scales
Cohen, J. (1960) · Educational and Psychological Measurement · doi:10.1177/001316446002000104 · Cited by 42,398 · see citing works
A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection
Kohavi, R. (1995) · Zenodo · doi:10.5281/zenodo.19712698 · Cited by 10,909 · see citing works
Assessing construct validity in organizational research
Bagozzi, R. P., Yi, Y., & Phillips, L. W. (1991) · Administrative Science Quarterly · doi:10.2307/2393203 · Cited by 6,621 · see citing works
Beyond Accuracy: What Data Quality Means to Data Consumers
Wang, R. Y., & Strong, D. M. (1996) · Journal of Management Information Systems · doi:10.1080/07421222.1996.11518099 · Cited by 4,672 · see citing works
The kappa statistic in reliability studies: use, interpretation, and sample size requirements
Sim, J., & Wright, C. C. (2005) · Physical Therapy · doi:10.1093/ptj/85.3.257 · Cited by 4,093 · see citing works
Statistical Methods for Rates and Proportions, 2nd ed
GROVE, W. A. (1981) · American Journal of Psychiatry · doi:10.1176/ajp.138.12.1644-a · Cited by 3,756 · see citing works
Answers to 20 questions about interrater reliability and interrater agreement
LeBreton, J. M., & Senter, J. L. (2008) · Organizational Research Methods · doi:10.1177/1094428106296642 · Cited by 3,608 · cited 1× · see citing works
“Despite the importance the interrater reliability (LeBreton & Senter, 2008; Moskal, 2000), to date very little is known of the extent to which existing scales of IQ lend themselves to consistent assessments by independent judges.”
BackgroundFleiss, J. L., & Cohen, J. (1973) · Educational and Psychological Measurement · doi:10.1177/001316447303300309 · Cited by 3,354 · see citing works
Validation Guidelines for IS Positivist Research
Straub, D., & Gefen, D. (2004) · Communications of the Association for Information Systems · doi:10.17705/1cais.01324 · Cited by 2,797 · see citing works
Internet encyclopaedias go head to head
Giles, J. (2005) · Nature · doi:10.1038/438900a · Cited by 1,978 · see citing works
AIMQ: a methodology for information quality assessment
Lee, Y. W., Strong, D. M., Kahn, B. K., & Wang, R. Y. (2002) · Information & Management · doi:10.1016/s0378-7206(02)00043-5 · Cited by 1,611 · see citing works
Stemler, S. E. (2020) · Scholarworks (University of Massachusetts Amherst) · doi:10.7275/96jp-xz07 · Cited by 1,025 · see citing works
Flanagin, A. J., & Metzger, M. J. (2007) · New Media & Society · doi:10.1177/1461444807075015 · Cited by 817 · see citing works
Hilligoss, B., & Rieh, S. Y. (2008) · Information Processing & Management · doi:10.1016/j.ipm.2007.10.001 · Cited by 611 · see citing works
Nov, O. (2007) · Communications of the ACM · doi:10.1145/1297797.1297798 · Cited by 593 · see citing works
Klein, K. J., Conn, A. B., Smith, D. B., & Sorra, J. (2001) · Journal of Applied Psychology · doi:10.1037/0021-9010.86.1.3 · Cited by 511 · see citing works
Value-Added Processes in Information Systems
Murphy, P., & Taylor, R. S. (1987) · Journal of Education for Library and Information Science · doi:10.2307/40323680 · Cited by 439 · see citing works
Intraclass Correlation and the Analysis of Variance
Trese, T., & Haggard, E. A. (1958) · The American Catholic Sociological Review · doi:10.2307/3708305 · Cited by 403 · see citing works
Gagliardi, A. R. (2002) · BMJ · doi:10.1136/bmj.324.7337.569 · Cited by 378 · see citing works
Eppler, M. J. (2006) · Springer eBooks · doi:10.1007/3-540-32225-6 · Cited by 358 · see citing works
Developing a Framework for Assessing Information Quality on the World Wide Web
Knight, S.-A., & Burn, J. M. (2005) · Informing Science The International Journal of an Emerging Transdiscipline · doi:10.28945/493 · Cited by 355 · see citing works
Credibility: A multidisciplinary framework
Rieh, S. Y., & Danielson, D. R. (2007) · Annual Review of Information Science and Technology · doi:10.1002/aris.2007.1440410114 · Cited by 336 · see citing works
Scoring Rubrics: What, When and How?
Moskal, B. (2020) · ScholarWorks@UMassAmherst (University of Massachusetts Amherst) · doi:10.7275/a5vq-7q66 · Cited by 333 · see citing works
Information quality work organization in wikipedia
Stvilia, B., Twidale, M. B., Smith, L. C., & Gasser, L. (2008) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.20813 · Cited by 233 · see citing works
An empirical examination of Wikipedia's credibility
Chesney, T. (2006) · First Monday · doi:10.5210/fm.v11i11.1413 · Cited by 230 · see citing works
Denning, P. J., Horning, J. J., Parnas, D. L., & Weinstein, L. (2005) · Communications of the ACM · doi:10.1145/1101779.1101804 · Cited by 212 · see citing works
The perceived credibility of personal Web page information as influenced by the sex of the source
Flanagin, A. J., & Metzger, M. J. (2003) · Computers in Human Behavior · doi:10.1016/s0747-5632(03)00021-9 · Cited by 208 · see citing works
Rains, S. A., & Donnerstein, C. (2009) · Computers in Human Behavior · doi:10.1016/j.chb.2008.11.005 · Cited by 183 · see citing works
How and why do college students use Wikipedia?
Lim, S. (2009) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.21142 · Cited by 178 · see citing works
Toward an epistemology of<i>Wikipedia</i>
Fallis, D. (2008) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.20870 · Cited by 149 · see citing works
Perceptions of credibility of scholarly information on the web
Liu, Z. (2003) · Information Processing & Management · doi:10.1016/s0306-4573(03)00064-5 · Cited by 140 · see citing works
Kittur, A., Ed, H., & Suh, B. (2009) · ACM SIGCHI Conference on Human Factors in Computing Systems (CHI) · doi:10.1145/1518701.1518930 · Cited by 124 · see citing works
Intraclass Correlation and the Analysis of Variance.
Sitgreaves, R., & Haggard, E. A. (1960) · Journal of the American Statistical Association · doi:10.2307/2281758 · Cited by 116 · see citing works
Wong, S. (2007) · Journal of Organizational Behavior · doi:10.1002/job.490 · Cited by 111 · see citing works
Oakleaf, M. (2009) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.21030 · Cited by 96 · see citing works
Experts vs. Online Consumers: A Comparative Credibility Study of Health and Finance Web Sites
Stanford, J., Tauber, E. R., & Marable, L. (2002) · Cited by 88 · see citing works
The assessment of the information quality with the aid of multiple criteria analysis
Michnik, J., & Lo, M. (2007) · European Journal of Operational Research · doi:10.1016/j.ejor.2007.11.017 · Cited by 82 · see citing works
Research in Low Back Pain: Time to Stop Seeking the Elusive “Magic Bullet”
Delitto, A. (2005) · Physical Therapy · doi:10.1093/ptj/85.3.206 · Cited by 57 · see citing works
Improving Wikipedia's accuracy: Is edit age a solution?
Luyt, B., Tay, A., Thian, L. H., & Hong, C. K. (2007) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.20755 · Cited by 30 · see citing works
The Democratization of Information? Wikipedia as a Reference Resource
Wallace, D. P., & Fleet, C. V. (2005) · Reference & User Services Quarterly · Cited by 25 · see citing works
Automated classification and retrieval of reusable software components
Yao, H., Etzkorn, L. H., & Bhada, S. V. (2008) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.20775 · Cited by 9 · see citing works
Conceptual analysis of parallel corpus collected from the Web
Li, K. W., & Yang, C. C. (2006) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.20326 · Cited by 8 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
A study of factors that contribute to online review helpfulness
Huang, A. H., Chen, K., Yen, D. C., & Tran, T. P. (2015) · Computers in Human Behavior · doi:10.1016/j.chb.2015.01.010 · Cited by 364 · see citing works
Web credibility assessment: Conceptualization, operationalization, variability, and models
Choi, W., & Stvilia, B. (2015) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.23543 · Cited by 168 · see citing works
Information quality assessment of community generated content: a user study of Wikipedia
Yaari, E., Baruchson-Arbib, S., & Bar-Ilan, J. (2011) · Journal of Information Science · doi:10.1177/0165551511416065 · Cited by 83 · see citing works
Reviews, trust, and customer experience in online marketplaces: the case of Mercado Libre Colombia
Peña-García, N., Losada-Otálora, M., Auza, D. P., & Cruz, M. P. (2024) · Frontiers in Communication · doi:10.3389/fcomm.2024.1460321 · Cited by 38 · see citing works
- Authors
- Arazy, O., & Kopak, R.
- Affiliations
- University of Alberta (CA)
- University of British Columbia (CA)
- Corresponding author
- Arazy, Ofer
- Kopak, Rick
- ORCID
- Arazy, Ofer
- Venue
- Journal of the American Society for Information Science and Technology
- Publisher
- Wiley
- Published
- 2010-11-04
- Volume
- 62
- Issue
- 1
- Pages
- 89-99
- ISSN
- 1532-2882 (ISSN-L)
- OpenAlex
- W2105089148
- Cited by
- 154
- References
- 45
- Citations by year
- 2026: 7 · 2025: 6 · 2024: 8 · 2023: 5 · 2022: 16 · 2021: 12 · 2020: 6 · 2019: 12 · 2018: 8 · 2017: 13 · 2016: 12 · 2015: 19 · 2014: 11 · 2013: 9 · 2012: 5
- FWCI
- 8.28
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of the American Society for Information Science and Technology (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Management Science and Operations Research › Data Quality and Management
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
- Keywords
- information quality, measurability, reliability statistics, interobserver agreement, accuracy, completeness, objectivity, representation, quality dimensions, heuristic cues, information quality assessment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
