The DOI matches its Crossref record; 84 of the 88 references it cites were found in a registry or an index, and 1 are not checked yet.
Measuring Association Between Labels and Free-Text Rationales
Open accessAbstract
In interpretable NLP, we require faithful rationales that reflect the model's decision-making process for an explained instance.While prior work focuses on extractive rationales (a subset of the input words), we investigate their lessstudied counterpart: free-text natural language rationales.We demonstrate that pipelines, models for faithful rationalization on informationextraction style tasks, do not work as well on "reasoning" tasks requiring free-text rationales.We turn to models that jointly predict and rationalize, a class of widely used high-performance models for free-text rationalization.We investigate the extent to which the labels and rationales predicted by these models are associated, a necessary property of faithful explanation.Via two tests, robustness equivalence and feature importance agreement, we find that stateof-the-art T5-based joint models exhibit desirable properties for explaining commonsense question-answering and natural language inference, indicating their potential for producing faithful free-text rationales. 1
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“While largerscale evaluations are desirable, prior work (Rudin, 2019; Wiegreffe et al., 2021) has shown that even small samples, when selected for conceptual diversity and difficulty, yield robust insights into model behavior.”
From GIER: Gap-Driven Self-Refinement for Large Language Models (2025) · cites this paper as Wiegreffe et al., 2021
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
Transformers: State-of-the-Art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2020.emnlp-demos.6 · Cited by 8,741
Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank
Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., & Potts, C. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1170 · Cited by 6,878
Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
Simonyan, K., Vedaldi, A., & Zisserman, A. (2013) · arXiv · doi:10.48550/arxiv.1312.6034 · Cited by 5,370
Explanation in artificial intelligence: Insights from the social sciences
Miller, T. (2019) · Artificial Intelligence · doi:10.1016/j.artint.2018.07.007 · Cited by 5,302
A large annotated corpus for learning natural language inference
Bowman, S. R., Angeli, G., Potts, C., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1075 · Cited by 3,377
Towards A Rigorous Science of Interpretable Machine Learning
Doshi-Velez, F., & Kim, B. (2017) · arXiv · doi:10.48550/arxiv.1702.08608 · Cited by 3,121
Learning Important Features Through Propagating Activation Differences
Shrikumar, A., Greenside, P., & Kundaje, A. (2017) · arXiv · doi:10.48550/arxiv.1704.02685 · Cited by 2,761
Research works printed but not linked: 2
- Miruna-Adriana Clinciu, Arash Eshghi, and Helen Hastie. 2021. A study of automatic metrics for the evaluation of natural language explanations. In Pro- ceedings of the 16th Conference of the European Chapter of the Association for Computational Lin- guistics: Main Volume, pages 2376-2387, Online. Association for Computational Linguistics.
- Omar Zaidan and Jason Eisner. 2008. Modeling an- notators: A generative approach to learning from annotator rationales. In Proceedings of the 2008 Conference on Empirical Methods in Natural Lan- guage Processing, pages 31-40, Honolulu, Hawaii. Association for Computational Linguistics.
Cited by
Sorted by citations
Items 101–147 of 147
Hendriks, D., Spitzer, P., Kühl, N., & Satzger, G. (2026) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2026.3671872 · Cited by 3
On Advances in Text Generation from Images Beyond Captioning: A Case Study in Self-Rationalization
Palaskar, S., Bhagia, A., Bisk, Y., Metze, F., Black, A. W., & Marasović, A. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.194 · Cited by 4
Self-citationOn Measuring Faithfulness or Self-consistency of Natural Language Explanations
Parcalabescu, L., & Frank, A. (2023) · arXiv · doi:10.48550/arxiv.2311.07466 · Cited by 27
“…than relying on self-consistency tests that compare the outputs of models after modifying their inputs, we instead measure self-consistency by analysing how much a model’s input contributes to its answer prediction vs. generated explanation – similar to the rationale of Wiegreffe et al. (2021).”
InfluentialMulti-View Cooperative Learning with Invariant Rationale for Document-Level Relation Extraction
Lin, R., Fan, J., He, Y., Yang, Y., Li, J., & Guo, C. (2024) · Cognitive Computation · doi:10.1007/s12559-024-10322-z · Cited by 2
DoME: An Architecture for Domain Model Evolution at Runtime Using NLP
Gomes, A. M., & Maia, P. H. M. (2023) · Brazilian Symposium on Software Engineering · doi:10.1145/3613372.3613405 · Cited by 2
Black-Box Attacks on Image Activity Prediction and its Natural Language Explanations
Baia, A. E., Poggioni, V., & Cavallaro, A. (2023) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw60793.2023.00396 · Cited by 2
Few Shot Rationale Generation using Self-Training with Dual Teachers
Veerubhotla, A. S., Poddar, L., Yin, J., Szarvas, G., & Eswaran, S. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.297 · Cited by 3
Make Your Decision Convincing! A Unified Two-Stage Framework: Self-Attribution and Decision-Making
Du, Y., Zhao, S., Wang, H., Chen, Y., Bai, R., Qiang, Z., Cai, M., & Qin, B. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.78 · Cited by 2
You Can Do Better! If You Elaborate the Reason When Making PredictionPreprint
Li, D., Tao, J., Chen, Q., & Hu, B. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.14919 · Cited by 3
Lane, J. N., Boussioux, L., Ayoubi, C., Chen, Y. H., Lin, C., Spens, R., Wagh, P., & Wang, P.-H. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4914367 · Cited by 2
Alon, B., Zimerman, I., & Wolf, L. (2026) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2026.acl-long.300 · Cited by 1
Feature Attribution Explanations of Session-Based Recommendations
Bruun, S. B., Maistro, M., & Lioma, C. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-88711-6_4 · Cited by 1
Kuo, M.-T., Hsueh, C.-C., & Tsai, R. T. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i21.30470 · Cited by 1
A Roadmap on Developing a Taxonomy for Text Data Mining
Pokusajev, S., & Stefanovič, P. (2024) · IEEE Open Conference of Electrical, Electronic and Information Sciences (eStream) · doi:10.1109/estream61684.2024.10542606 · Cited by 1
Cross-Examiner: Evaluating Consistency of Large Language Model-Generated Explanations
Villa, D., Chang, M., Murugesan, K., Uceda‐Sosa, R., & Ramamurthy, K. N. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11462075 · Cited by 1
GIER: Gap-Driven Self-Refinement for Large Language Models
Dewri, R. (2025) · arXiv · doi:10.48550/arxiv.2509.00325
Diagnostics-Guided Explanation GenerationPreprint
Atanasova, P., Simonsen, J. G., Lioma, C., & Augenstein, I. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.03756
The model thinks what?! Interpreting deep NLP models with rationales and influence
Jain, S. (2022) · Digital Repository Service (Northeastern University) · doi:10.17760/d20467212
Explanation Graph Generation via Generative Pre-training over Synthetic Graphs
Cui, H., Li, S., Zhang, Y., & Shi, Q. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.629
Atanasova, P. (2024) · Springer eBooks · doi:10.1007/978-3-031-51518-7_1
From Information Overload to Intelligent Recommendation: The Power of Large Language Models
Sahni, D. K., Khurana, D., & Jain, R. (2024) · International Conference on Advancement in Electronics & Communication Engineering (AECE) · doi:10.1109/aece62803.2024.10911816
Towards Explainable Medical Machine Reading Comprehension With Rationale Generation
Li, D., He, S., Hu, B., & Chen, Q. (2025) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2025.3555105
Self-explaining Neural Network for Multi-criteria Sentiment Analysis
Hoang, X.-T., Nguyen, V.-L., Kitagawa, T., & Huynh, V. (2025) · Lecture notes in computer science · doi:10.1007/978-981-96-4606-7_8
Iterative Self-Improvement of Vision Language Models for Image Scoring and Self-Explanation
Tanji, N., & Yamasaki, T. (2025) · Proceedings - International Conference on Image Processing · doi:10.1109/icip55913.2025.11084643
Believing without Seeing: Quality Scores for Contextualizing Vision-Language Model Explanations
He, K., Srinivasan, T., Joshi, B., Ren, X. Y., Thomason, J., & Swayamdipta, S. (2026) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2026.acl-long.673
Developing trustworthy language models with consistent predictions
Jang, M. (2024) · Oxford University Research Archive (ORA) (University of Oxford) · doi:10.5287/ora-pm75v7jex
Interpretable and Controllable Language Models
Hase, P. (2024) · Carolina Digital Repository (University of North Carolina at Chapel Hill) · doi:10.17615/dvq0-fp08
Towards reliability and interactive debugging for large language models
Paranjape, B. (2024) · ResearchWorks at the University of Washington (University of Washington)
Ibrahim, A. F., Hassan, M., & Ali, Y. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.01.12.698948
MREF: Metric-Based Instance Re-Weighting for Rationale Enhancement
Huang, Y., Kuang, Z., Chiang, M.-F., & Lee, W. C. (2025) · Proceedings · doi:10.1109/icdm65498.2025.00042
GEESE: Generating and Evaluating Explanations for Semantic Entailment: A CALAMITA Challenge
Zaninello, A., & Magnini, B. (2024) · Open Research (University of Surrey)
The Power of Trust in Recommender Systems: Improving Outcomes and Reducing Noise
Sahni, D. K., Khurana, D., Nakul, & Rani, S. (2025) · Global AI Summit - International Conference on Artificial Intelligence and Emerging Technology (AI Summit) · doi:10.1109/aisummit66170.2025.11411632
Feedback-Integrated Short Answer Grading with Data Augmentation and Large Language Models
Bin, Z., Chen, Z., & Liu, S. (2025) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata66926.2025.11402058
Schleider, T. (2022) · theses.fr (ABES) · doi:10.70675/6f2b45eez5308z4992z9e4azd2a0ac72f19d
Nguyen, D. H. (2024) · theses.fr (ABES) · doi:10.70675/97b1b150zc009z4f96z9ec6z2790870387a0
Rank-ICL: Ranking-based In-context Learning for Search Result Explanation
Laksito, A. D., Alqarni, A. A., & Stevenson, M. (2026) · ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR) · doi:10.1145/3805713.3820420
Praschk, A., Fischill-Neudeck, V., Caspari, T., & Wiesinger, H. (2026) · Journal of Medical Systems · doi:10.1007/s10916-026-02449-3
Yoo, J., Kim, B., Yang, Y., & Jang, H. (2026) · IEEE Access · doi:10.1109/access.2026.3725783
GRACE: Gradient-Regulated Approach for Consistent Explanations
Basu, S. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-36039-7_33
Evidence-Consistent Generative Detection under Scenario-Level Distribution Shift
Kim, S., & Bak, J. (2026) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3799682.3840763
Kim, S., & Ko, B. (2026) · Applied Sciences · doi:10.3390/app16189190
Faithfulness in Black-Box LLM-Based Recommender Systems’ Explanations
Silva, Í. D. S., & Marinho, L. B. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-39892-5_45
Alderete, J. D., Betageri, S., Flynn, J., Greenhalgh, A., Rubin, J., & Annam, Y. V. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.33813
Better Answers, Weaker Evidence Control: Evidence Bypass in On-Policy DistillationPreprint
Xu, R., Zhao, Y., Chen, Z., Kang, X., & Cohan, A. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2611.01967
Metadata
- Authors
- Wiegreffe, S., Marasović, A., & Smith, N. A.
- Affiliations
- Georgia Institute of Technology (US)
- Venue
- Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing
- Published
- 2021
- Pages
- 10266-10284
- OpenAlex
- W3093530468
- Cited by
- 119
- References
- 88
- Citations by year
- 2026: 9 · 2025: 17 · 2024: 22 · 2023: 29 · 2022: 24 · 2021: 18
- FWCI
- 10.37
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Allen Institute for Artificial Intelligence
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- interpretability, commonsense question answering, natural language inference, T5
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review