Exploring the reliability of social and environmental disclosures content analysis
Abstract
This paper reports the results of an exploratory study of inter‐coder reliability of annual report social and environmental disclosures content analysis. Using the sentence‐based coding instruments and decision rules from Hackston and Milne (1996), this study reports the co‐agreement levels reached by three coders over five rounds of testing 49 annual reports. The study also provides a commentary on the implications of formal reliability analysis for past and future social and environmental disclosures content analyses, and the complexities of formal reliability measurement. The overall findings suggest that the coded output from inexperienced coders using the Hackston and Milne approach with little or no prior training can be relied on for aggregate total disclosures analysis. For more detailed sub‐category analysis, however, the findings suggest a period of training for the less experienced coders with at least 20 reports appears necessary before their coded output could be relied on.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The reliability of the content analysis was ensured by initially analyzing a 20% subset of textbook pages to check categorization appropriateness and by cross-checking overall findings after a few weeks for inter-coder reliability.
“To ensure the reliability of the content analysis, we followed two strategies. First, a small subset of textbook pages (20%) was initially analyzed to check the appropriateness of categorization as suggested by Weber [69] and Cohen [67]. Second, following Milne and Adler [100], overall findings were cross-checked after a few weeks of the primary content analysis to ensure inter-coder reliability.”
From Gender stereotypes and education: A comparative content analysis of Malaysian, Indonesian, Pakistani and Bangladeshi school textbooks (2018) · cites this paper as Milne and Adler [100]
Cites
Reliability of Nominal Data Based on Qualitative Judgments
Perreault, W. D., & Leigh, L. E. (1989) · Journal of Marketing Research · doi:10.2307/3172601 · Cited by 1,575 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Evidence in development of sustainability reporting: a case of a developing country
Amran, A., & Haniffa, R. (2011) · Business Strategy and the Environment · doi:10.1002/bse.672 · Cited by 301 · see citing works
Islam, K. M. M., & Asadullah, M. N. (2018) · PLOS ONE · doi:10.1371/journal.pone.0190807 · Cited by 196 · see citing works
Mion, G., & Loza Adaui, C. R. (2019) · Sustainability · doi:10.3390/su11174612 · Cited by 180 · see citing works
Nonfinancial Risk Disclosure: The “State of the Art” of Italian Companies
Leopizzi, R., Iazzi, A., Venturelli, A., & Principale, S. (2020) · Corporate Social Responsibility and Environmental Management · doi:10.1002/csr.1810 · Cited by 84 · see citing works
Framing of Climate Change Impacts and Use of Management Accounting Practices
Faculty of Business and Law, S. U. O. T., Kumarasiri, J., Jubb, C., & Centre for Transformative Innovation, F. O. B. A. L. (2018) · Asian Academy of Management Journal of Accounting and Finance · doi:10.21315/aamjaf2017.13.2.3 · Cited by 6 · see citing works
- Authors
- Milne, M. J., & Adler, R. W.
- Affiliations
- University of Otago (NZ)
- Venue
- Accounting, Auditing & Accountability Journal
- Publisher
- Emerald
- Published
- 1999-05-01
- Volume
- 12
- Issue
- 2
- Pages
- 237-256
- ISSN
- 0951-3574 (ISSN-L)
- OpenAlex
- W2013387701
- Cited by
- 1,425
- References
- 1
- Citations by year
- 2026: 55 · 2025: 45 · 2024: 56 · 2023: 68 · 2022: 60 · 2021: 77 · 2020: 97 · 2019: 68 · 2018: 74 · 2017: 84 · 2016: 85 · 2015: 97 · 2014: 73 · 2013: 77 · 2012: 73
- FWCI
- 24.23
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Accounting Auditing & Accountability Journal (publishedVersion)
- Topics
- Social Sciences › Business, Management and Accounting › Strategy and Management › Corporate Social Responsibility Reporting
- Social Sciences › Business, Management and Accounting › Accounting › Auditing, Earnings Management, Governance
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Keywords
- intercoder reliability, environmental and social disclosure, content analysis, coder training
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
