The DOI matches its Crossref record; 10 of the 10 references it cites were found in a registry or an index.
K-Alpha Calculator—Krippendorff’s Alpha Calculator: A user-friendly tool for computing Krippendorff’s Alpha inter-rater reliability coefficient
Open accessAbstract
Krippendorff's Alpha is a measure for assessing inter-rater reliability, facilitating triangulated evaluations among multiple raters. Indeed, Krippendorff's Alpha is key in validating the dependability of human assessments, thereby reinforcing the robustness of human-based choices in contexts where interpretive variability could otherwise undermine the research outcomes. Despite its versatility across various data types, the procedure for computing this coefficient might limit its applicability for researchers unfamiliar with specialised statistical software. Addressing this issue, this paper introduces the "K-Alpha Calculator"-Krippendorff's Alpha Calculator-a freely accessible, user-friendly, web application available at https://www.k-alpha.org designed for easy computation of Krippendorff's Alpha. By offering a web interface without any software dependency, the K-Alpha Calculator seeks to promote the broader integration of such a reliability metric in research processes. We envisage that the tool's contribution could support researchers in enhancing methodological rigour, fostering conclusions are rooted in triangulated interpretations and assessments. The K-Alpha Calculator is both a computational tool and an educational resource, encouraging more researchers to include reliability measures in their studies.•Development of the K-Alpha Calculator.The study centres on the introduction of a web-based application: the K-Alpha Calculator. The tool is developed to simplify the calculation of Krippendorff's Alpha without any required software dependency.•User-friendly interface for diverse data types.The K-Alpha Calculator is characterised by its accessibility and user-friendliness, designed to simplify the procedures for computing the Krippendorff's Alpha. Its utility extends to various data types, which aligns with the intrinsic versatility of this coefficient.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Inter-coder reliability was 0.81 based on Krippendorff’s Alpha [45,46], indicating a good level of agreement among coders.”
From Help-Seeking Situations Related to Visual Interactions on Mobile Platforms and Recommended Designs for Blind and Visually Impaired Users (2024) · cites this paper as [46]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A Coefficient of Agreement for Nominal Scales
Cohen, J. (1960) · Educational and Psychological Measurement · doi:10.1177/001316446002000104 · Cited by 42,398
Content Analysis: An Introduction to Its Methodology
Krippendorff, K. (2019) · doi:10.4135/9781071878781 · Cited by 15,448
Measuring nominal scale agreement among many raters.
Fleiss, J. L. (1971) · Psychological Bulletin · doi:10.1037/h0031619 · Cited by 8,733
Answering the Call for a Standard Reliability Measure for Coding Data
Hayes, A. F., & Krippendorff, K. (2007) · Communication Methods and Measures · doi:10.1080/19312450709336664 · Cited by 4,225
Intercoder Reliability in Qualitative Research: Debates and Practical Guidelines
O’Connor, C., & Joffe, H. (2020) · International Journal of Qualitative Methods · doi:10.1177/1609406919899220 · Cited by 3,871
Answers to 20 questions about interrater reliability and interrater agreement
LeBreton, J. M., & Senter, J. L. (2008) · Organizational Research Methods · doi:10.1177/1094428106296642 · Cited by 3,608
Zapf, A., Castell, S., Morawietz, L., & Karch, A. (2016) · BMC Medical Research Methodology · doi:10.1186/s12874-016-0200-9 · Cited by 377
Zhang, Y. E., Liu, G., Zhang, L., Xu, S., & Cheung, M. W. ‐. (2020) · Journal of Management · doi:10.1177/0149206320917195 · Cited by 212
Multiple‐Principal Demands and CEO Compliance in Emerging Market State‐Owned Enterprises
Apriliyanti, I. D., Dieleman, M. H., & Randøy, T. (2023) · Journal of Management Studies · doi:10.1111/joms.12977 · Cited by 36
KANOM: Stata module to estimate Krippendorff's alpha for nominal variables
Mitnik, P. A. (2016) · RePEc: Research Papers in Economics · Cited by 1
Research works printed but not linked: 2
- Freelon, 2010, ReCal: intercoder reliability calculation as a web service, Int. J. Internet Sci., 5, 20
- Freelon, 2013, ReCal OIR: ordinal, interval, and ratio intercoder reliability as a web service, Int. J. Internet Sci., 8, 10
Cited by
Sorted by citations
Marzi, G., Balzano, M., Caputo, A., & Pellegrini, M. M. (2024) · International Journal of Management Reviews · doi:10.1111/ijmr.12381 · Cited by 831
“Convergence assessments can be performed recurring to the computation of Krippendorff’s Alpha coefficient (Krippendorff, 2019), for example, using the K-Alpha Calculator (Marzi et al., 2024) available at https://www.k-alpha.org/.”
Self-citationInter-Coder Agreement in Qualitative Coding: Considerations for its Use
Halpin, S. N. (2024) · American Journal of Qualitative Research · doi:10.29333/ajqr/14887 · Cited by 149
SMEs and institutional theory: major inroads and opportunities ahead
Balzano, M., Marzi, G., & Turzo, T. (2024) · Management Decision · doi:10.1108/md-05-2023-0734 · Cited by 153
Self-citationAnalysis of LLMs for educational question classification and generation
Faraby, S. A., Romadhony, A., & Adiwijaya, A. (2024) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2024.100298 · Cited by 79
Cafforio, F., Franco, S., & Giannoccaro, I. (2025) · Business Strategy and the Environment · doi:10.1002/bse.4338 · Cited by 20
Reyes-Acosta, R. E., Domínguez-Báez, C.-A., Mendoza-González, R., & Martín, M. V. (2024) · International Journal of Information Security · doi:10.1007/s10207-024-00935-8 · Cited by 19
Digital marketing and sustainable innovation in SMEs through bibliometric and systematic review
Noer, M. Y., Chan, A., Tresna, P. W., & Purbasari, R. (2025) · Cogent Business & Management · doi:10.1080/23311975.2025.2548953 · Cited by 30
“…between the reviewers were resolved through discussion until a consensus was reached. krippendorff’s alpha was calculated to evaluate the reliability of the inclusion and exclusion decisions. it yielded a coefficient of 0.87, indicating a high level of inter-rater agreement (Marzi et al., 2024).”
InfluentialLeveraging Transfer Learning for Hate Speech Detection in Portuguese Social Media Posts
Ramos, G., Batista, F., Ribeiro, R., Fialho, P., Moro, S., Fonseca, A. J. F., Guerra, R., Carvalho, P., Marques, C. M. V. A., & Silva, C. (2024) · IEEE Access · doi:10.1109/access.2024.3430848 · Cited by 19
“Additionally, a subset of comments/tweets (825 for YouTube and 805 for Twitter) was assigned to all annotators to assess IAA and annotation reliability using Krippendorff’s alpha [58].”
MethodDomínguez‐Díaz, A., Goyanes, M., de‐Marcos, L., & Sánchez, V. P. P. (2024) · PeerJ Computer Science · doi:10.7717/peerj-cs.2378 · Cited by 15
Lee, D., Brown, M., Hammond, J., & Zakowski, M. I. (2024) · International Journal of Obstetric Anesthesia · doi:10.1016/j.ijoa.2024.104317 · Cited by 15
Lebepe, P., & Mathaba, T. N. D. (2025) · Journal of Economy and Technology · doi:10.1016/j.ject.2025.01.002 · Cited by 17
Lanfranchi, G., Cioli, A., Amanti, A., & Marinelli, L. (2025) · European Journal of Innovation Management · doi:10.1108/ejim-03-2024-0317 · Cited by 23
Khac, L. T. D., & Leyer, M. (2025) · Technology in Society · doi:10.1016/j.techsoc.2025.103064 · Cited by 17
Exploring Thematic Analysis in Qualitative Research
Qaissi, A. (2024) · Advances in data mining and database management book series · doi:10.4018/979-8-3693-8689-7.ch010 · Cited by 17
McDonnell, E. J., & Findlater, L. (2024) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/3663548.3675649 · Cited by 17
“After the second pass, authors achieved an average Krippendorf’s Alpha inter-rater reliability reliability score of 0.88 (range .45-1), indicating satisfactory agreement [99].”
BackgroundInfluentialHe, K., Yang, Q., Wu, D. C. W., & Zou, Y. (2025) · Journal of Travel & Tourism Marketing · doi:10.1080/10548408.2025.2464804 · Cited by 11
“There are more than a dozen different inter-rater reliability measures, such as the consistency ratio, intra-class correlation, and Cohen’s kappa, Krippendorff’s Alpha, etc (Cohen, 1960; Marzi et al., 2024).”
Nurses’ roles and responsibilities in suicide prevention: a scoping review
Riedel, A., Feinauer, S., Jacob, E., Haug, P. M., Klotz, K., & Heidenreich, T. (2025) · BMC Nursing · doi:10.1186/s12912-025-04009-5 · Cited by 16
“To assess inter-rater reliability, Krippendorff’s Alpha was applied [60], using the Krippendorff’s Alpha Calcula-tor developed by Marzi et al. [61].”
Characterization of agrammatism in Tagalog: Evidence from narrative spontaneous speech
Gerona, J., de Kok, D., Salis, C., Webster, J., & Jonkers, R. (2024) · Aphasiology · doi:10.1080/02687038.2024.2353164 · Cited by 10
“Krippendorff’s Alpha (Krippendorff, 2019) was employed to assess the interrater reliability of the analysis protocol and the data set was processed using a web-based K-Alpha calculator (Marzi et al., 2024).”
Zandbiglari, K., Kumar, S., Bilal, M., Goodin, A. J., & Rouhizadeh, M. (2024) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2024.104755 · Cited by 11
Employee experience: conceptualization, scale development, and validation
Yang, P., & Zhang, S. (2025) · Humanities and Social Sciences Communications · doi:10.1057/s41599-025-04926-5 · Cited by 10
“Speci fi cally, we calculated Krippendorff ’ s α coef fi cients by entering the scoring data provided by R3 into the web-based statistical package K-Alpha Calculator (Marzi et al., 2024).”
Xie, I., Choi, W., Wang, S., Lee, H. S., Hong, B. H., Wang, N.-C., & Cudjoe, E. K. (2024) · Journal of Imaging · doi:10.3390/jimaging10080205 · Cited by 8
“Inter-coder reliability was 0.81 based on Krippendor ff ’s Alpha [45,46], indicating a good level of agreement among coders.”
MethodInfluentialAt the Cybersecurity Frontier: Key Strategies and Persistent Challenges for Business Leaders
Balzano, M., & Marzi, G. (2024) · Strategic Change · doi:10.1002/jsc.2622 · Cited by 9
“Regarding our inclusion/exclusion criteria, to ensure a targeted and relevant analysis, we established specific boundary conditions that guided our selection of studies (Marzi et al. 2024).”
Self-citationLiu, J., Liu, T., Zhu, Y., & Yan, J. (2025) · Journal of Knowledge Management · doi:10.1108/jkm-01-2025-0058 · Cited by 9
Quantifying Psychological Sophistication of Malicious Emails
Longtchi, T. T., Rodriguez, R. M., Gwartney, K., Ear, E., Azari, D. P., Kelley, C. P., & Xu, S. (2024) · IEEE Access · doi:10.1109/access.2024.3514603 · Cited by 8
Wu, W., & Xiao, R. (2025) · Technovation · doi:10.1016/j.technovation.2025.103241 · Cited by 8
Abouzeid, E., Wassef, R., Jawwad, A., & Harris, P. (2025) · JMIR Medical Education · doi:10.2196/69521 · Cited by 12
“The Krippen-dorff alpha [23] was used to assess interrater reliability, using the K-Alpha Calculator [24].”
Tareke, T. G., Oo, T. Z., & Józsa, K. (2025) · International Journal of Educational Research Open · doi:10.1016/j.ijedro.2025.100510 · Cited by 21
Halliday, C., Berry, M. T., Shah, S. V., Soni, S., Saini, R., Vaughan, D., & Chaggar, R. S. (2025) · British Journal of Anaesthesia · doi:10.1016/j.bja.2025.11.007 · Cited by 8
Fridman, I., Boyles, D., Chheda, R., Baldwin-SoRelle, C., Smith, A. B., & Lafata, J. E. (2025) · JMIR Infodemiology · doi:10.2196/62703 · Cited by 9
“To distinguish posts with misinformation from other discussions, 2 reviewers (IF and CR) double-coded a randomly chosen subset of 1064 posts, achieving an acceptable interrater agreement of 0.68 measured with Krippendorff a [27].”
InfluentialCarr, A. L., Jenkins, A. M., Jonklaas, J., Gabriel, K., Miller, K., & Graves, K. D. (2025) · JMIR Formative Research · doi:10.2196/60588 · Cited by 7
“The final Krippendorf Cu-α=.88, indicating acceptable agreement [43].”
Ördek, B., Coatanéa, É., & Borgianni, Y. (2025) · Results in Engineering · doi:10.1016/j.rineng.2025.104418 · Cited by 7
Rahimi, A. R. (2025) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2025.100420 · Cited by 7
Interaction-based creative thinking research and its application in artistic creation
Wu, R., Fan, J., Tang, C., & Yang, X. (2024) · Thinking Skills and Creativity · doi:10.1016/j.tsc.2024.101750 · Cited by 7
Gallée, J., Cartwright, J., Henry, M. L., Mooney, A. R., Stark, B. C., Volkmer, A., Nakano, C. Y., Fredericksen, R. J., Domoto‐Reilly, K., & Crane, P. K. (2024) · PLoS ONE · doi:10.1371/journal.pone.0301652 · Cited by 6
Giannozzi, A., Balzano, M., Magrini, A., & Marzi, G. (2025) · The Journal of Technology Transfer · doi:10.1007/s10961-025-10205-x · Cited by 6
“To enhance the reliability of the coding procedures, we also relied on K-Alpha calculator, and the Krippendorff’s Alpha exhibited satisfactory levels of agreements among the coders (Marzi et al., 2024).”
Self-citationSusitha, E., Jayarathne, P. G. S. A., & Herath, H. (2025) · PLoS ONE · doi:10.1371/journal.pone.0325945 · Cited by 7
“Two trained researchers conducted the coding process, achieving Krippendorff’s alpha of 0.88, indicating a high level of intercoder reliability and reinforcing the robustness of the data analysis process [122].”
InfluentialJayalath, V., Barthakur, A., Dawson, S., Tingey-Holyoak, J. L., Crase, L., & Kovanović, V. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-98414-3_21 · Cited by 8
Philbin, C. A., & Sentance, S. (2025) · Conference on United Kingdom & Ireland Computing Education Research (UKICER) · doi:10.1145/3754508.3754513 · Cited by 6
Boen, H., Glenisson, L., Hallez, L., & Smits, T. (2025) · International Journal of Behavioral Nutrition and Physical Activity · doi:10.1186/s12966-025-01832-6 · Cited by 6
“They measured agreement using Krippendorff’s Alpha, following Hayes and Krippendorff’s [21] guidelines.”
Time is running out: How design thinking shapes team innovation under time constraints
Balzano, M., & Bortoluzzi, G. (2024) · European Management Journal · doi:10.1016/j.emj.2024.08.003 · Cited by 6
Self-citationInvestigating Developers' Preferences for Learning and Issue Resolution Resources in the ChatGPT Era
Tayeb, A. J., Alahmadi, M. D., Tajik, E., & Haiduc, S. (2024) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme58944.2024.00045 · Cited by 15
“This indicates a high level of inter-annotator agreement, underscoring the coding process’s reliability and the qualitative data’s integrity [32].”
Friedrich, L., Leutheuser, V., Mertel, M., & Voigt, K. (2025) · R&D Management · doi:10.1111/radm.12760 · Cited by 6
“After several iterations of the analysis process, the workshop resulted in four overarching third-order themes with an inter-coder reliability of 91%, according to Krippendorff's alpha (see Table A5) (Marzi et al. 2024).”
InfluentialBeher, J., Treml, E. A., & Wintle, B. A. (2025) · PLoS ONE · doi:10.1371/journal.pone.0326166 · Cited by 5
“Recommended thresholds for different agreement rates to pass a reliability test are given in the literature, for example, 60% for kappa [1] or 80% for Krippendorff’s alpha [38].”
Achar, J. C., Firman, J. W., Tran, C., Kim, D., Cronin, M., & Öberg, G. (2024) · Regulatory Toxicology and Pharmacology · doi:10.1016/j.yrtph.2024.105716 · Cited by 5
Tomás‐Miquel, J., Maldonado-Gómez, G., & Vicedo, J. C. (2025) · Benchmarking An International Journal · doi:10.1108/bij-12-2023-0907 · Cited by 5
Annotated tweet data of mixed Wolof-French for detecting obnoxious messages
Ndao, I., Dramé, K., Sambe, G., & Diallo, G. (2025) · Data in Brief · doi:10.1016/j.dib.2025.111500 · Cited by 5
“Krippendorff’s α is a measure of the annotation reliability of several annotators on categorical, ordinal or nominal data [ 21 ].”
Borrowing a mouth to speak? Foreign content creators in China’s national image building
Cho-Li, Q., Jin, Y., & Kiousis, S. K. (2025) · Place Branding and Public Diplomacy · doi:10.1057/s41254-025-00398-1 · Cited by 4
“While the value of 0.68 indicated moderate agreement, it is generally considered acceptable for exploratory studies (Lombard et al. 2002; Marzi et al. 2024).”
InfluentialKanis, T., Mann, J., & Stumpf-Wollersheim, J. (2026) · Strategy Science · doi:10.1287/stsc.2025.0442 · Cited by 4
CT Staging of Colon Cancer: Reproducibility and Pathology Association in a Multicenter Study
Shkurti, J., Lipman, K. B. W. G., Tissier, R., Chupetlovska, K., Pooch, E., Haan, M. C. V. D.-D., Pijl, M. E. J., Taveira, M. D. O., Huige, J. C. B. M., Gielens, M. P. M., et al. (2026) · Academic Radiology · doi:10.1016/j.acra.2026.02.008 · Cited by 4
Overcoming the Digital Dilemmas: How Do Enterprises Develop Digital Transformative Capability?
Gao, Y., Zhang, H., Ding, L., Zhang, S., & Zeng, C. (2025) · IEEE Transactions on Engineering Management · doi:10.1109/tem.2025.3558022 · Cited by 4
“Then, we input the rated data into the web-based statistical package K-alpha calculator [66].”
Aransyah, M. F., Hermanto, B., Muftiadi, A., & Oktadiana, H. (2025) · Cogent Social Sciences · doi:10.1080/23311886.2025.2548015 · Cited by 11
“…analysis of knowledge production in the mapping patterns, identifying emerging clusters and evaluating the thematic evolution of research over time (marzi et al., 2024; Van eck & Waltman, 2010). by focusing specifically on asia, the study examines scholarly output related to sustainable tourism,…”
Modified Clavien-Dindo Classification for Adverse Events in Otolaryngology–Head and Neck Surgery
Hidalgo, C. M., Martin, E. J., Eyassu, D. G., Hernandez-Herrera, G. A., Jenkins, S. M., Chweya, C. M., Sankar, G. B., Bayan, S. L., Carlson, M. L., Hamilton, G. S., et al. (2025) · JAMA Network Open · doi:10.1001/jamanetworkopen.2025.39761 · Cited by 4
Chen, H., Tsai, H., Chen, Y., Chiu, Y.-J., Kang, E. Y.-N., Hsu, Y.-T., Issenberg, S. B., & Wu, J.-C. (2025) · BMC Medical Education · doi:10.1186/s12909-025-06705-x · Cited by 4
“The management task performance is categorized as nominal data, and Krippendorff’s alpha was used assessing reliability with this type of data [28, 29].”
Personality-based pair programming: toward intrinsic motivation alignment in very small entities
Valový, M., & Buchalcevová, A. (2025) · PeerJ Computer Science · doi:10.7717/peerj-cs.2774 · Cited by 7
Spies, S., Nazarian, E., Annavarapu, S., Collini, P., L’Herminé, A. C., D’Hooghe, E., Kobos, J. W., Morcrette, G., Morini, M. A., Popov, S. D., et al. (2025) · Cancers · doi:10.3390/cancers17101693 · Cited by 4
Labeling of Cannabis Products From Licensed and Unlicensed Retailers in New York
Becker, T. D., Menzi, P. J., Olfson, M., Mosharova, P., LEVIN, F. R., & Sultan, R. S. (2025) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2025.108000 · Cited by 4
Arendsen, J. L., Brugman, B. C., van Koningsbruggen, G. M., & Fransen, M. L. (2025) · International Journal of Consumer Studies · doi:10.1111/ijcs.70056 · Cited by 4
“(Marzi et al. 2024).”
Arbuatti, A., Serafino, A. D., & Lucidi, P. (2024) · Animals · doi:10.3390/ani14233469 · Cited by 3
Ördek, B., & Borgianni, Y. (2025) · International Journal on Interactive Design and Manufacturing (IJIDeM) · doi:10.1007/s12008-025-02394-4 · Cited by 3
Visser, M., Jansen, M. G., Frankenmolen, N. L., Camerino, I., Brazil, I. A., Piai, V., Hendriks, M. P. H., Kessels, R. P. C., & Oosterman, J. M. (2025) · Aging Neuropsychology and Cognition · doi:10.1080/13825585.2025.2555436 · Cited by 4
“…were classified as elaborate ( N = 9), less elaborate ( N = 4), visual ( N = 3), and verbal ( N = 8), with inter-rater reliability scores of α k = .59 for elaborateness (82% agreement), and α k = .78 for modality (87% agreement), suggesting poor to moderate reliability (Marzi et al., 2024).”
Variable regional histomorphology of the ascending aorta: Implications for disease classification
Pryma, C. S., Amin, M. S., Leduc, C., Wu, C., Jenkins, S. M., Bois, M. C., & Maleszewski, J. J. (2025) · Cardiovascular Pathology · doi:10.1016/j.carpath.2025.107787 · Cited by 3
Stoner, A. M., & Knowles, A. J. (2025) · Critical Sociology · doi:10.1177/08969205251383421 · Cited by 3
“The resulting Krippendorff’s Alpha was 0.81, indicating substantial agreement and a reliable rating (Marzi et al., 2024).”
InfluentialBa, S., Lu, G., Mendoza, N. B., Yang, Y. P., Pan, Z., & Wang, Y. (2025) · Journal of Educational Computing Research · doi:10.1177/07356331251396357 · Cited by 3
“…traditional survey research approaches have inherent limitations in capturing learning complexities (Coughlan et al., 2009), advances in inter-rater reliability assessment (Marzi et al., 2024) and sophisticated analytical frameworks have enhanced our ability to evaluate LLM-mediated discussions.”
InfluentialMeisenbacher, S., Klymenko, A., & Matthes, F. (2025) · Proceedings of the 2025 Workshop on Human-Centered AI Privacy and Security · doi:10.1145/3733816.3760760 · Cited by 5
Community of inquiry in motion: Modeling inquiry dynamics with movement analysis (MOVA)
Ba, S., Swiecki, Z., Tan, Y., Lu, G., Shaffer, D. W., & Gašević, D. (2025) · Computers & Education · doi:10.1016/j.compedu.2025.105513 · Cited by 4
FORGE: An LLM-driven Framework for Large-Scale Smart Contract Vulnerability Dataset Construction
Chen, J., Shen, Y., Zhang, J., Li, Z., Grundy, J., SHAO, Z., Wang, Y., Wang, J., Chen, T., & Zheng, Z. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3744916.3764529 · Cited by 3
Momeni, P., Laverghetta, G., Ligatti, J., & Li, L. (2025) · Online Journal of Public Health Informatics · doi:10.2196/80660 · Cited by 3
“We acknowledge that the specific algorithms used for this demographic inference are proprietary to Brandwatch, and as such, the details of their methods are not publicly available [53].”
Wyss, R., & Chiru, M. (2026) · Environmental Politics · doi:10.1080/09644016.2025.2609431 · Cited by 3
Wray, R. E., Barton, S. A., & Hecht, E. E. (2026) · Applied Animal Behaviour Science · doi:10.1016/j.applanim.2026.106959 · Cited by 3
Joshi, A., McGoldrick, B., Mittal, N., Roy, S., Zeba, Z., Ofori, M. A., Dockery, S., Jha, N., Auerbach, J. M., Cadet, J., et al. (2026) · Frontiers in Public Health · doi:10.3389/fpubh.2026.1796810 · Cited by 3
Pommée, T., Shanks, M., Morsomme, D., Michel, S., & Verduyckt, I. A. (2024) · Journal of Voice · doi:10.1016/j.jvoice.2024.10.021 · Cited by 3
Koushiou, M., Manzoor, S., Jossif, A. P., & Ferreira, N. B. (2024) · Healthcare · doi:10.3390/healthcare12242594 · Cited by 3
Kruse, S. P., D’Souza, L., Young, J., & Tuncer, H. G. G. (2025) · Frontiers in Communication · doi:10.3389/fcomm.2025.1527164 · Cited by 4
A Large Scale Test Corpus for Semantic Table Search
Leventidis, A. S., Christensen, M. P., Lissandrini, M., Rocco, L. D., Hose, K., & Miller, R. J. (2024) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3626772.3657877 · Cited by 3
A comprehensive review on survivor syndrome in the workplace
George, T., & Maheshwari, B. (2024) · Benchmarking An International Journal · doi:10.1108/bij-12-2023-0890 · Cited by 3
Mental imagery shapes emotions in people's decisions related to risk taking
Smieja, J., Zaleśkiewicz, T., & Gąsiorowska, A. (2025) · Cognition · doi:10.1016/j.cognition.2025.106082 · Cited by 3
“We calculated the inter-rater reliability using Krippendorff ’ s alpha, yielding a value of α = 0.121, 95 % CI [0.097, 0.145], suggesting poor reliability (Marzi et al., 2024).”
InfluentialFighting Evaluation Inflation: Concentrated Datasets for Grammatical Error Correction
Starchenko, V., Kharlamova, D., Klykova, E. A., Shavrina, A., Миронович, С. А., Vinogradova, O., & Lyashevskaya, O. (2024) · Journal of language and Education · doi:10.17323/jle.2024.22272 · Cited by 2
Measuring Reflective Inquiry in Professional Learning Networks: A Conceptual Framework
Warmoes, A., Brown, C., Decabooter, I., & Consuegra, E. (2025) · Education Sciences · doi:10.3390/educsci15030333 · Cited by 4
“After ChatGPT coded the CSV file, intercoder reliability was measured using Krippendorff’s Alpha Calculator (Marzi et al., 2024), resulting in a Krippendorff’s Alpha of 0.503, which is considered too low.”
InfluentialSpeaking of Stigma: Piloting Language Training in Addiction Counseling Coursework
Grybush, A. L., Stoupas, G., & Clarke, P. B. (2025) · Counselor Education and Supervision · doi:10.1002/ceas.12344 · Cited by 2
Murphy, T. I., Armitage, J. A., Abel, L. A., van Wijngaarden, P., & Douglass, A. G. (2025) · Journal of Clinical Medicine · doi:10.3390/jcm14093046 · Cited by 2
“…added for each image for each of the five retinal arcades, arterioles and venules were annotated with the thicker vessel An example of this annotation is shown in Figure 5, with areas of overlapping annotations highlighted. the main branch at each bifurcation, as described by Nowroozzadeh et al.”
Okola, I., Ochieng, D. O., Miriti, E. K., & Ong'isa, G. O. (2025) · Results in Control and Optimization · doi:10.1016/j.rico.2025.100606 · Cited by 2
Simon, S., Sankaranarayanan, S., Tajik, E., Borchers, C., shahrokhian, B., Balzan, F., Strauß, S., Viswanathan, S. A., Ataş, A. H., Čarapina, M., et al. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-98420-4_5 · Cited by 2
Clark, S. A., Patterson, S., Duguid, I., Noar, S. M., Lazard, A. J., Thrasher, J. F., Goldstein, A. O., & Kowitt, S. D. (2025) · Nicotine & Tobacco Research · doi:10.1093/ntr/ntaf195 · Cited by 2
Precision Livestock Farming: YOLOv12-Based Automated Detection of Keel Bone Lesions in Laying Hens
Bergamasco, T., Ambrosi, A., Tregnaghi, V., Urbani, R., Nalesso, G., Menegon, F., Trocino, A., Pravato, M., Bordignon, F., Sparesato, S., et al. (2025) · Poultry · doi:10.3390/poultry4040043 · Cited by 2
“…criterion described in Section 2.1.2 resulted in the following inter-annotator reliability metrics: Fleiss’s Kappa of 0.63 (substantial agreement [20]), Krippen-dorff’s Alpha of 0.63 (insufficient reliability, not recommended for robust conclusions [21]), and the F1 scores summarized in Table 4.”
InfluentialJin, X., Huang, L., Tan, C., Xiao, S., Zhang, C., & Wang, Y. (2025) · Decision Support Systems · doi:10.1016/j.dss.2025.114558 · Cited by 2
Organisational function synergy: capabilities enabled by smart industrial products
PISSARDINI, P. E., Filho, M. G., Callefi, M. H. B. M., & Ganga, G. M. D. (2025) · Management Decision · doi:10.1108/md-12-2024-2887 · Cited by 4
Lobbying coalitions and their determinants: a systematic literature review
Pritoni, A., Bitonti, A., & Montalbano, G. (2025) · European Political Science Review · doi:10.1017/s1755773925100143 · Cited by 2
“The resulting K-Alpha is 0.889, which is generally considered a high level of intercoder agreement (Marzi et al. , 2024).”
InfluentialJiang, Y., Zheng, X., Chen, X., Xiao, Y., & Shen, M. (2025) · Humanities and Social Sciences Communications · doi:10.1057/s41599-025-06030-0 · Cited by 4
Hariyanto, S., Maulidiyah, F., Suyono, A., Sulistiyo, U., & Fehring, H. (2025) · Humanities and Social Sciences Communications · doi:10.1057/s41599-025-06331-4 · Cited by 2
“The K-Alpha was calculated using the K-Alpha calculator developed by Marzi et al. (2024), resulting in a K-alpha of 0.915 or 92% agreement.”
InfluentialHalck, M. L., Jensen, K. S., Adji, A. V., Albert, M., Norin, T., Rasmussen, M. H., & Bjørnvad, C. R. (2025) · Veterinary Record · doi:10.1002/vetr.6125 · Cited by 2
Verlinden, L., Michotte, A., Cools, W., Locy, H., Forsyth, R. G., Duerinck, J., Neyns, B., & Brock, S. (2026) · Apmis · doi:10.1111/apm.70143 · Cited by 2
“Krippendorff's alpha was 0.916 for interrater reliability and 0.953 for intrarater reliability, in line with satisfactory agreement [22].”
InfluentialChampion, C. P., Meuten, D. J., Rissi, D. R., Hoffmann, A. R., Tatiersky, L. J., Miller, A. D., Laufer-Amorim, R., Porcellato, I., Sills, S., Rätsep, E., et al. (2026) · Veterinary Pathology · doi:10.1177/03009858261465445 · Cited by 2
“18,35 The calculation was classified as ordinal, nominal, or ratio based on the question type (Table 1).”
InfluentialCollins, A. C., Bhattacharya, S., Oh, J. Y., Salzhauer, A. J., Taylor, C. T., Wolitzky-Taylor, K. B., Aupperle, R. L., Budney, A. J., & Jacobson, N. C. (2024) · JMIR Formative Research · doi:10.2196/54751 · Cited by 2
“Each theme or semantic domain demonstrated a satisfactory level of agreement above 0.80 regarding the suggested cutoffs for Krippendorff a [76].”
BackgroundInfluentialParry, S., Zabaglo, L. A., Wilkinson, D., & Dodson, A. R. (2025) · Diagnostic histopathology · doi:10.1016/j.mpdhp.2024.12.001 · Cited by 2
Shrivastava, T. P., Wagdhare, S., Saboo, B., & Goyal, R. K. (2025) · Digital Health · doi:10.1177/20552076251343700 · Cited by 2
Python’s evolution on Stack Overflow: An empirical analysis of topic trends
Hu, F., Xue, W., Zhou, S., Wang, Y., Jiang, B., Huang, Q., & Zhang, H. (2025) · Journal of Computer Languages · doi:10.1016/j.cola.2025.101340 · Cited by 2
LLMs replicate metaphor norms based on word co-occurrence but struggle with topic-vehicle mappings
Pissani, L., Jobanputra, M., & Demberg, V. (2026) · Frontiers in Language Sciences · doi:10.3389/flang.2026.1756514 · Cited by 2
Put at and kept from risk: The sexual risk dilemmas confronting marginalized youth
Bay‐Cheng, L. Y., & Sutton, E. R. (2024) · Journal of Adolescence · doi:10.1002/jad.12430 · Cited by 1
El Empleo de la Red Social Instagram como Plataforma de Difusión Periodística
Ventura-Salom, B., Bengoa, M. T., & Díez, L. G. (2024) · VISUAL REVIEW International Visual Culture Review / Revista Internacional de Cultura Visual · doi:10.62161/revvisual.v16.5381 · Cited by 1
This or that? Managerial responses to competing priorities in organizations
Mariano, S. (2025) · Management Decision · doi:10.1108/md-02-2024-0363 · Cited by 1
Metadata
- Authors
- Marzi, G., Balzano, M., & Marchiori, D.
- Affiliations
- IMT School for Advanced Studies Lucca (IT)
- Ca' Foscari University of Venice (IT)
- Corresponding author
- Marzi, Giacomo
- Venue
- MethodsX
- Publisher
- Elsevier BV
- Published
- 2024-06
- Volume
- 12
- Pages
- 102545
- ISSN
- 2215-0161 (ISSN-L)
- PMC
- PMC11636850
- OpenAlex
- W4390636334
- Cited by
- 292
- References
- 10
- Citations by year
- 2026: 143 · 2025: 113 · 2024: 36
- FWCI
- 80.91
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- MethodsX (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- ArTS Archivio della ricerca di Trieste (University of Trieste https://www.units.it/) (submittedVersion, cc-by)
- ARCA (Università Ca' Foscari Venezia) (submittedVersion, other-oa)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Multidisciplinary Science and Engineering Research
- Keywords
- Krippendorff's alpha, inter-rater reliability, web-based calculator, reliability analysis software
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
