The DOI matches its Crossref record; 5 of the 5 references it cites were found in a registry or an index.
Reliability in content analysis: some common misconceptions and recommendations.
Open accessAbstract
In a recent article published in this journal, Lombard, Snyder-Duch, and Bracken (2002) surveyed 200 content analyses for their reporting of reliability tests; compared the virtues and drawbacks of five popular reliability measures; and proposed guidelines and standards for their use. Their discussion revealed that numerous misconceptions circulate in the content analysis literature regarding how these measures behave and can aid or deceive content analysts in their effort to ensure the reliability of their data. This paper proposes three conditions for statistical measures to serve as indices of the reliability of data and examines the mathematical structure and the behavior of the five coefficients discussed by the authors, plus two others. It compares common beliefs about these coefficients with what they actually do and concludes with alternative recommendations for testing reliability in content analysis and similar data-making efforts.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“For the 6 moderator variables, Krippendorff’s alpha ranged from 0.81 to 0.99, and therefore above the acceptable level (Krippendorff, 2004).”
From The Power of Feedback Revisited: A Meta-Analysis of Educational Feedback Research (2020) · cites this paper as Krippendorff, 2004
The cited source says what the claim attributes to it.
In the citing paper
“Krippendorff's (2004) criterion of significance is therefore satisfied here too.”
From Weighted Krippendorff's alpha is a more reliable metrics for multi-coders ordinal annotations: experimental studies on emotion, opinion and coreference annotation (2014) · cites this paper as Krippendorff's (2004)
The cited source says what the claim attributes to it.
In the citing paper
“Krippendorff's α α α α (Krippendorff, 2004) considers chance independently from coders like Scott's π, but data reliability is estimated depending on disagreement instead of agreement:”
From Weighted Krippendorff's alpha is a more reliable metrics for multi-coders ordinal annotations: experimental studies on emotion, opinion and coreference annotation (2014) · cites this paper as Krippendorff, 2004
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“The criterion of data significance -at least five chance agreements per category -proposed by (Krippendorff, 2004) is greatly satisfied for the valence annotation (3 categories).”
From Weighted Krippendorff's alpha is a more reliable metrics for multi-coders ordinal annotations: experimental studies on emotion, opinion and coreference annotation (2014) · cites this paper as Krippendorff, 2004
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 13
- Bennett, E. M., Alpert, R., & Goldstein, A. C. (1954). Communications through limited response questioning. Public Opinion Quarterly, 18, 303-308.
- Craig, R. T. (1981). Generalization of Scott"s index of intercoder agreement. Public Opinion Quarterly, 45, 260-264.
- Goodman, L. A, & Kruskal, W. H. (1954). Measures of association for cross classifications. Journal of the American Statistical Association, 49, 732-764.
- Holsti, O. R. (1969). Content analysis for the social sciences and humanities. Reading, MA: Addison- Wesley.
- Hughes, M. A., & Garrett, D. E. (1990). Intercoder reliability estimation -Approaches in marketing: A generalizability theory framework for quantitative data. Journal of Marketing Research, 27, 185- 195.
- Krippendorff, K. (1970). Bivariate agreement coefficients for reliability data. In E. R. Borgatta & G. W. Bohrnstedt (Eds.), Sociological methodology 1970 (pp. 139-150). San Francisco, CA: Jossey Bass.
- Krippendorff, K. (2004). Content analysis: An introduction to its methodology. Second Edition. Thousand Oaks, CA: Sage.
- Maxwell, A. E. (1970). Comparing the classification of subjects by two independent judges. British Journal of Psychiatry, 116, 651-655.
- Osgood, C. E. (1959). The representational model and relevant research. In I. de Sola Pool (Ed.), Trends in content analysis (pp. 33-88). Urbana: University of Illinois Press.
- Potter, W. J., & Levine-Donnerstein, D. (1999). Rethinking validity and reliability in content analysis. Journal of Applied Communication Research, 27, 258-284.
- Riffe, D., & Freitag, A. (1996, August). Twenty-five years of content analyses in Journalism & Mass Communication Quarterly. Paper presented to the annual convention of the Association for Education in Journalism and Mass Communication, Anaheim, CA, cited in D. Riffe, S. Lacy, & F. G. Fico (1998). Analyzing media messages. Mahwah, NJ: Erlbaum.
- Scott, W. A. (1955). Reliability of content analysis: The case of nominal scale coding. Public Opinion Quarterly, 19, 321-325.
- Zwick, R. (1988). Another look at interrater agreement. Psychological Bulletin, 103, 347-387.
Cited by
Sorted by citations
Items 301–400 of 1,010
Antoine, J.-Y., Villaneau, J., & Lefeuvre, A. (2014) · Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics · doi:10.3115/v1/e14-1058 · Cited by 64
“Krippendorff‘s αααα (Krippendorff, 2004) considers chance independently from coders like Scott’s π , but data reliability is estimated depending on disagreement instead of agreement: where D o is the observed disagreement be-tween coders and D e is an estimation of the possible chance disagreement.”
BackgroundInfluentialBot-Adversarial Dialogue for Safe Conversational Agents
Xu, J., Ju, D., Li, M., Boureau, Y.-L., Weston, J., & Dinan, E. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.235 · Cited by 64
Are bank risk disclosures informative? Evidence from debt markets
Elamer, A. A., Ntim, C. G., Abdou, H. A., Owusu, A., Elmagrhi, M., & Ibrahim, A. E. A. (2020) · International Journal of Finance & Economics · doi:10.1002/ijfe.1849 · Cited by 66
Mayer, C., Ludwig, S., & Brandt, S. (2022) · Journal of Research on Technology in Education · doi:10.1080/15391523.2022.2142872 · Cited by 72
Rap Music Videos and African American Women's Body Image: The Moderating Role of Ethnic Identity
Zhang, Y., Dixon, T. L., & Conrad, K. M. (2009) · Journal of Communication · doi:10.1111/j.1460-2466.2009.01415.x · Cited by 72
Analyzing consumer online group buying motivations: An interpretive structural modeling approach
Xiao, L. (2018) · Telematics and Informatics · doi:10.1016/j.tele.2018.01.010 · Cited by 64
Gaither, B. M., Austin, L., & Collins, M. (2018) · The Journal of Public Interest Communications · doi:10.32473/jpic.v2.i2.p176 · Cited by 64
Adaui, C. R. L. (2020) · Sustainability · doi:10.3390/su12031135 · Cited by 73
Martin, V. Y., Christidis, L., Lloyd, D. J., & Pecl, G. T. (2016) · Journal of Science Communication · doi:10.22323/2.15020202 · Cited by 63
What Arguments against COVID-19 Vaccines Run on Facebook in Poland: Content Analysis of Comments
Wawrzuta, D., Jaworski, M., Gotlib, J., & Pańczyk, M. (2021) · Vaccines · doi:10.3390/vaccines9050481 · Cited by 62
Park, H., Jiang, S., Lee, O.-K. D., & Chang, Y. (2021) · Information Systems Frontiers · doi:10.1007/s10796-021-10207-8 · Cited by 65
Goal-Oriented Requirements Engineering: A Systematic Literature Map
Horkoff, J., Aydemir, F. B., Cardoso, E. C. S., Li, T., Maté, A., Paja, E., Salnitri, M., Mylopoulos, J., & Giorgini, P. (2016) · IEEE International Requirements Engineering Conference (RE) · doi:10.1109/re.2016.41 · Cited by 62
Deng, T., Ekachai, D. “., & Pokrywczynski, J. V. (2020) · Health Communication · doi:10.1080/10410236.2020.1859725 · Cited by 62
Esau, K., Fleuß, D., & Nienhaus, S.-M. (2020) · Policy & Internet · doi:10.1002/poi3.232 · Cited by 69
Reconceptualizing cybersecurity awareness capability in the data-driven digital economy
Akter, S., Uddin, M. R., Sajib, S., Lee, W. J., Michael, K., & Hossain, M. A. (2022) · Annals of Operations Research · doi:10.1007/s10479-022-04844-8 · Cited by 61
An empirical investigation and users' perceptions on intellectual capital reporting in banks
Khan, H. Z., & Ali, M. (2010) · Journal of Human Resource Costing & Accounting · doi:10.1108/14013381011039799 · Cited by 61
Quantifying long‐term impact of zoo and aquarium visits on biodiversity‐related learning outcomes
Jensen, E. A., Moss, A., & Gusset, M. (2017) · Zoo Biology · doi:10.1002/zoo.21372 · Cited by 61
Zaini, S. M., Sharma, U., Samkin, G., & Davey, H. (2019) · Accounting Forum · doi:10.1080/01559982.2019.1605874 · Cited by 61
Reposting “till Albert Einstein is TikTok famous”: The Memetic Construction of Science on TikTokPreprint
Zeng, J., Schäfer, M. S., & Allgaier, J. (2020) · doi:10.31219/osf.io/8tdvm · Cited by 61
Moon, A. C., Zotos, E. K., Finkenstaedt-Quinn, S. A., Gere, A. R., & Shultz, G. V. (2018) · Chemistry Education Research and Practice · doi:10.1039/c8rp00090e · Cited by 60
MALE AND FEMALE SOCIAL MEDIA INFLUENCERS: THE IMPACT OF GENDER ON EMERGING ADULTS
Lokithasan, K., Simon, S., Jasmin, N. Z., & Othman, N. (2019) · International Journal of Modern Trends in Social Sciences · doi:10.35631/ijmtss.29003 · Cited by 47
Investigating risk disclosures in Italian integrated reports
Guthrie, J., Rossi, F. M., Orelli, R. L., & Nicolò, G. (2020) · Meditari Accountancy Research · doi:10.1108/medar-10-2019-0596 · Cited by 60
The financial consequences of human capital disclosure as part of integrated reporting
Salvi, A., Raimo, N., Petruzzella, F., & Vitolla, F. (2021) · Journal of Intellectual Capital · doi:10.1108/jic-03-2021-0079 · Cited by 60
Data Collection Methods in Health Services Research
Bowles, K., Skinner, E. H., Mitchell, D., Haas, R., Ho, M., Salter, K., May, K., Markham, D., O’Brien, L., Plumb, S., et al. (2015) · Applied Clinical Informatics · doi:10.4338/aci-2014-10-ra-0097 · Cited by 60
Tsandilas, T. (2018) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3182168 · Cited by 60
Corner, E., & Gill, P. (2019) · Journal of Quantitative Criminology · doi:10.1007/s10940-019-09420-1 · Cited by 60
Borch, K., Munk, A. K., & Dahlgaard, V. (2020) · Energy Policy · doi:10.1016/j.enpol.2019.111223 · Cited by 60
The Evaluation of Complex Infrastructure Projects
Gerrits, L., & Verweij, S. (2018) · Edward Elgar Publishing eBooks · doi:10.4337/9781783478422 · Cited by 60
McLoughlin, L., & Southern, R. (2020) · The British Journal of Politics and International Relations · doi:10.1177/1369148120930594 · Cited by 59
Intercoder Reliability for Use in Qualitative Research and Evaluation
Coleman, M. L., Ragan, M., & Dari, T. (2024) · Measurement and Evaluation in Counseling and Development · doi:10.1080/07481756.2024.2303715 · Cited by 59
Analyzing Dataset Annotation Quality Management in the Wild
Klie, J.-C., de Castilho, R. E., & Gurevych, I. (2024) · Computational Linguistics · doi:10.1162/coli_a_00516 · Cited by 66
Guillet, B. D., Küçükusta, D., & Liu, L. (2015) · Journal of Travel & Tourism Marketing · doi:10.1080/10548408.2015.1064337 · Cited by 59
Giving and Making Sense About Change: The Back and Forth Between Leaders and Employees
Kraft, A., Sparr, J. L., & Peus, C. V. (2016) · Journal of Business and Psychology · doi:10.1007/s10869-016-9474-5 · Cited by 67
Building a Values‐Informed Mental Model for New Orleans Climate Risk Management
Bessette, D., Mayer, L. A., Cwik, B., Vezér, M. A., Keller, K. M., Lempert, R. J., & Tuana, N. (2017) · Risk Analysis · doi:10.1111/risa.12743 · Cited by 64
The political economy of economic reform in the Pacific
Duncan, R. (2010) · ANU Open Research (Australian National University) · Cited by 59
Theory In, Theory Out: The Uses of Social Theory in Machine Learning for Social Science
Radford, J., & Joseph, K. (2020) · Frontiers in Big Data · doi:10.3389/fdata.2020.00018 · Cited by 58
Do ownership structures affect risk disclosure in Islamic banks? International evidence
Grassa, R., Moumen, N., & Hussainey, K. (2020) · Journal of financial reporting & accounting · doi:10.1108/jfra-02-2020-0036 · Cited by 58
Milanesi, M., Kyrdoda, Y., & Runfola, A. (2022) · Journal of Global Fashion Marketing · doi:10.1080/20932685.2021.1998789 · Cited by 58
Mapping the socially responsible consumption gap research: Review and future research agenda
Falcão, D., & Roseira, C. (2022) · International Journal of Consumer Studies · doi:10.1111/ijcs.12803 · Cited by 58
Upper-level undergraduate chemistry students’ goals for their laboratory coursework
DeKorver, B. K., & Towns, M. H. (2016) · Journal of Research in Science Teaching · doi:10.1002/tea.21326 · Cited by 62
Reliability in content analysis: The case of semantic feature norms classification
Bolognesi, M., Pilgram, R., & van den Heerik, R. A. M. (2016) · Behavior Research Methods · doi:10.3758/s13428-016-0838-6 · Cited by 58
A Strategic Initiative for Successful Reverse Logistics Management in Retail Industry
Panigrahi, S., Kar, F. W., Fen, T. A., Hoe, L. K., & Wong, M. (2018) · Global Business Review · doi:10.1177/0972150918758096 · Cited by 58
Tao, S., & Kim, H. (2019) · Asia Pacific Journal of Tourism Research · doi:10.1080/10941665.2019.1591473 · Cited by 58
Spearing, L. A., Bakchan, A., Hamlet, L. C., Stephens, K. K., Kaminsky, J., & Faust, K. M. (2022) · Journal of Construction Engineering and Management · doi:10.1061/(asce)co.1943-7862.0002313 · Cited by 57
Plummer, R., Velaniškis, J., de Grosbois, D., Kreutzwiser, R. D., & de Loë, R. (2010) · Environmental Science & Policy · doi:10.1016/j.envsci.2010.05.004 · Cited by 57
The prominence of eco in ecotourism experiences: An analysis of post-purchase online reviews
Ruhanen, L. (2019) · Journal of Hospitality and Tourism Management · doi:10.1016/j.jhtm.2019.03.006 · Cited by 57
Overtourism in Iceland: Fantasy or Reality?
Sæþórsdóttir, A. D., Hall, C. M., & Wendt, M. (2020) · Sustainability · doi:10.3390/su12187375 · Cited by 57
Paek, H. (2005) · Asian Journal of Communication · doi:10.1080/01292980500118292 · Cited by 57
Examining the instructor-student collaborative partnership in an online learning community course
Ouyang, F., Chang, Y.-H., Scharber, C., Jiao, P., & Huang, T. (2020) · Instructional Science · doi:10.1007/s11251-020-09507-4 · Cited by 60
The future of marketing analytics in the sharing economy
Akter, S., Hani, U., Dwivedi, Y. K., & Sharma, A. (2022) · Industrial Marketing Management · doi:10.1016/j.indmarman.2022.04.008 · Cited by 61
Grimm, A., Hulse, L. M., Preiss, M., & Schmidt, S. (2013) · Disasters · doi:10.1111/disa.12034 · Cited by 61
Tower, Am I Cleared to Land?: Problematic Communication in Aviation Discourse
Howard, J. W. (2008) · Human Communication Research · doi:10.1111/j.1468-2958.2008.00325.x · Cited by 56
Comparative analysis of existing food safety culture evaluation systems
Jespersen, L., Griffiths, M. W., & Wallace, C. A. (2017) · Food Control · doi:10.1016/j.foodcont.2017.03.037 · Cited by 56
What drives risk disclosure in Islamic and conventional banks? An international comparison
Grassa, R., Moumen, N., & Hussainey, K. (2020) · International Journal of Finance & Economics · doi:10.1002/ijfe.2122 · Cited by 56
Donkor, A., Djajadikerta, H. G., & Roni, S. M. (2021) · International Journal of Auditing · doi:10.1111/ijau.12229 · Cited by 55
Kuhn, M., Gerlicher, A. M. V., & Lonsdorf, T. B. (2022) · Psychophysiology · doi:10.1111/psyp.14058 · Cited by 55
Bevilacqua, C., Calabrò, F., & Spina, L. D. (2020) · Smart innovation, systems and technologies · doi:10.1007/978-3-030-52869-0 · Cited by 55
Paige, S. R., Stellefson, M., Chaney, B. H., & Alber, J. M. (2015) · American Journal of Health Education · doi:10.1080/19325037.2015.1044586 · Cited by 55
‘New media’ research publication trends and outlets in communication, 1990-2006
Tomasello, T. K., Lee, Y., & Baer, A. P. (2009) · New Media & Society · doi:10.1177/1461444809342762 · Cited by 55
What secondary teachers think and do about student engagement in mathematics
Skilling, K., Bobis, J., Martin, A. J., Anderson, J., & Way, J. (2016) · Mathematics Education Research Journal · doi:10.1007/s13394-016-0179-x · Cited by 55
Consumers’ health-related perceptions of bread – Implications for labeling and health communication
Sandvik, P., Nydahl, M. C., Kihlberg, I., & Marklinder, I. (2017) · Appetite · doi:10.1016/j.appet.2017.11.092 · Cited by 55
The mediating effects of finance on the performance of hospital facilities management services
Amos, D., Au-Yong, C. P., & Musa, Z. N. (2020) · Journal of Building Engineering · doi:10.1016/j.jobe.2020.101899 · Cited by 54
Recognition of Haptic Interaction Patterns in Dyadic Joint Object Manipulation
Madan, C. E., Küçükyılmaz, A., Sezgin, T. M., & Başdoğan, Ç. (2014) · IEEE Transactions on Haptics · doi:10.1109/toh.2014.2384049 · Cited by 54
What do airline mission statements reveal about value and strategy?
Law, K. M. Y., & Breznik, K. (2018) · Journal of Air Transport Management · doi:10.1016/j.jairtraman.2018.04.015 · Cited by 54
Ivanov, B., Sims, J. D., Compton, J., Miller, C. H., Parker, K. A., Parker, J. L., Harrison, K. J., & Averbeck, J. M. (2015) · Western Journal of Communication · doi:10.1080/10570314.2014.943423 · Cited by 53
Functional Text Dimensions for the annotation of web corpora
Sharoff, S. (2018) · Corpora · doi:10.3366/cor.2018.0136 · Cited by 53
Arrigo, E., Vaio, A. D., Hassan, R., & Palladino, R. (2022) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2022.132151 · Cited by 53
Content Analysis of Video Games: Challenges and Potential Solutions
Schmierbach, M. (2009) · Communication Methods and Measures · doi:10.1080/19312450802458950 · Cited by 61
Developing and Assessing Intercoder Reliability in Studies of Group Interaction
Compton, D. R., Love, T. P., & Sell, J. A. (2012) · Sociological Methodology · doi:10.1177/0081175012444860 · Cited by 55
Analyzing consumer goal structure in online group buying: A means-end chain approach
Xiao, L., Guo, Z., & D’Ambra, J. D. (2017) · Information & Management · doi:10.1016/j.im.2017.03.001 · Cited by 53
Citation function, polarity and influence classification
Hernández-Álvarez, M., Gomez, J. M., & Martínez-Barco, P. (2017) · Natural Language Engineering · doi:10.1017/s1351324916000346 · Cited by 58
An Investigation Into Facebook “Liking” Behavior An Exploratory Study
Ozanne, M., Navas, A. M. C., Mattila, A. S., & Hoof, H. B. V. (2017) · Social Media + Society · doi:10.1177/2056305117706785 · Cited by 59
Wulff, P., Buschhüter, D., Westphal, A., Mientus, L., Nowak, A., & Borowski, A. (2022) · Journal of Science Education and Technology · doi:10.1007/s10956-022-09969-w · Cited by 64
Hadely, K. A., Power, E., & O’Halloran, R. (2014) · BMC Health Services Research · doi:10.1186/1472-6963-14-110 · Cited by 52
Business models for open government data
Magalhães, G., Roseira, C., & Manley, L. (2014) · International Conference on Theory and Practice of Electronic Governance · doi:10.1145/2691195.2691273 · Cited by 57
Rickard, M., Easterbrook, B., Kim, S., Farrokhyar, F., Stein, N., Arora, S., Belostotsky, V., DeMaria, J., Lorenzo, A. J., & Braga, L. H. (2016) · Journal of Pediatric Urology · doi:10.1016/j.jpurol.2016.09.005 · Cited by 52
"Our Privacy Needs to be Protected at All Costs"
Xia, H., Wang, Y., Huang, Y., & Shah, A. (2017) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3134748 · Cited by 52
Build community before the storm: The National Weather Service's social media engagement
Olson, M. K., Sutton, J. N., Vos, S. C., Prestley, R., Renshaw, S. L., & Butts, C. T. (2019) · Journal of Contingencies and Crisis Management · doi:10.1111/1468-5973.12267 · Cited by 52
Scherr, C. L., Ramesh, S., Marshall-Fricker, C., & Perera, M. A. (2019) · Frontiers in Genetics · doi:10.3389/fgene.2019.00548 · Cited by 52
Learner experience in artificial intelligence-scaffolded argumentation
Kim, M. K., Kim, N. J., & Heidari, A. (2022) · Assessment & Evaluation in Higher Education · doi:10.1080/02602938.2022.2042792 · Cited by 51
Leventon, J. (2014) · Journal of Environmental Planning and Management · doi:10.1080/09640568.2014.916611 · Cited by 52
Abdenour, J. (2017) · Journalism & Mass Communication Quarterly · doi:10.1177/1077699017733438 · Cited by 51
Piñeiro-Naval, V. (2020) · Communication & Society · doi:10.15581/003.33.3.1-15 · Cited by 51
Management accounting for a circular economy: current limits and avenue for a dialogic approach
Aureli, S., Foschi, E., & Paletta, A. (2023) · Accounting Auditing & Accountability Journal · doi:10.1108/aaaj-04-2022-5766 · Cited by 51
The role of online product reviews on information adoption of new product development professionals
Lee, K. Y., & Yang, S. (2015) · Internet Research · doi:10.1108/intr-11-2013-0238 · Cited by 51
A Blended Professional Development Program to Help a Teacher Learn to Provide One-to-One Scaffolding
Belland, B., Burdo, R., & Gu, J. (2015) · Journal of Science Teacher Education · doi:10.1007/s10972-015-9419-2 · Cited by 53
Jeon, Y. A., HALE, B. J., Knackmuhs, E. G., & Mackert, M. (2018) · Interactive Journal of Medical Research · doi:10.2196/ijmr.9182 · Cited by 58
System Dynamics Approaches and Collective Action for Community Health: An Integrative Review
Cilenti, D., Issel, L. M., Wells, R. S., Anderson, S., & Lich, K. H. (2019) · American Journal of Community Psychology · doi:10.1002/ajcp.12305 · Cited by 51
Civil Society and Its Institutional Context in CEE
Meyer, M., Moder, C. M., Neumayr, M., & Vandor, P. (2019) · VOLUNTAS International Journal of Voluntary and Nonprofit Organizations · doi:10.1007/s11266-019-00106-7 · Cited by 51
Skilling, K., Bobis, J., & Martin, A. J. (2020) · Mathematics Education Research Journal · doi:10.1007/s13394-020-00313-2 · Cited by 66
Using Multi-Sensory and Multi-Dimensional Immersive Virtual Reality in Participatory Planning
Meenar, M., & Kitson, J. (2020) · Urban Science · doi:10.3390/urbansci4030034 · Cited by 51
Purhonen, S., Heikkilä, R. L. S., Hazır, I. K., Lauronen, T., Rodríguez, C. J. F., & Gronow, J. (2018) · Routledge eBooks · doi:10.4324/9781315183404 · Cited by 51
AI In education: comparative perspectives from STEM and non-STEM instructors
Parviz, M. (2024) · Computers and Education Open · doi:10.1016/j.caeo.2024.100190 · Cited by 50
News Values on Instagram: A Comparative Study of International News
Al‐Rawi, A., Al-Musalli, A., & Fakida, A. (2021) · Journalism and Media · doi:10.3390/journalmedia2020018 · Cited by 50
Nekmahmud, M. (2022) · Journal of Foodservice Business Research · doi:10.1080/15378020.2022.2090802 · Cited by 55
Wesseling, J. H., & Meijerhof, N. (2023) · PLOS Sustainability and Transformation · doi:10.1371/journal.pstr.0000075 · Cited by 50
Atela, J. O., Quinn, C. H., Minang, P. A., Duguma, L. A., & Houdet, J. (2016) · Forest Policy and Economics · doi:10.1016/j.forpol.2016.01.003 · Cited by 50
TopicCheck: Interactive Alignment for Assessing Topic Model Stability
Chuang, J., Roberts, M. E., Stewart, B., Weiss, R., Tingley, D. H., Grimmer, J., & Heer, J. (2015) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.3115/v1/n15-1018 · Cited by 50
Training and interobserver reliability in qualitative scoring of skeletal samples
Wilczak, C. A., Mariotti, V. M., Pany‐Kucera, D., Villotte, S., & Henderson, C. (2016) · Journal of Archaeological Science Reports · doi:10.1016/j.jasrep.2016.11.033 · Cited by 50
Constructing Difference: Lego® Set Narratives Promote Stereotypic Gender Roles and Play
Reich, S. M., Black, R. W., & Foliaki, T. (2017) · Sex Roles · doi:10.1007/s11199-017-0868-2 · Cited by 62
Metadata
- Authors
- Krippendorff, K.
- Corresponding author
- Krippendorff, K.
- ORCID
- Krippendorff, K.
- Venue
- Human Communication Research
- Publisher
- Oxford University Press (OUP)
- Published
- 2004-07-01
- Volume
- 30
- Issue
- 3
- Pages
- 411-433
- ISSN
- 0360-3989 (ISSN-L)
- 1468-2958
- OpenAlex
- W4243989635
- Cited by
- 1,027
- References
- 0
- Citations by year
- 2026: 42 · 2025: 100 · 2024: 108 · 2023: 108 · 2022: 124 · 2021: 135 · 2020: 105 · 2019: 57 · 2018: 44 · 2017: 29 · 2016: 41 · 2015: 25 · 2014: 26 · 2013: 21 · 2012: 15
- FWCI
- 5.93
- Type
- journal article
- Language
- en
- Open access
- green
- repository.upenn.edu
- Locations
- Human Communication Research (publishedVersion)
- ScholarlyCommons (University of Pennsylvania) (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Keywords
- inter-rater reliability, reliability coefficients, interobserver agreement, reporting reliability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
