CoChat
Sign Up
Cited by 75Explainable Artificial Intelligence (XAI)Topic ModelingAdversarial Robustness in Machine Learning

The DOI matches its Crossref record; 9 of the 9 references it cites were found in a registry or an index.

How do Decisions Emerge across Layers in Neural Models? Interpretation with Differentiable Masking

Open access
De Cao, Nicola, Schlichtkrull, Michael Sejr, Aziz, Wilker, Titov, Ivan·2020·Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)
DOI 10.18653/v1/2020.emnlp-main.262
Open this paper in CoChat — free accountOpen PDF

Abstract

Attribution methods assess the contribution of inputs to the model prediction.One way to do so is erasure: a subset of inputs is considered irrelevant if it can be removed without affecting the prediction.Though conceptually simple, erasure's objective is intractable and approximate search remains expensive with modern deep NLP models.Erasure is also susceptible to the hindsight bias: the fact that an input can be dropped does not mean that the model 'knows' it can be dropped.The resulting pruning is over-aggressive and does not reflect how the model arrives at the prediction.To deal with these challenges, we introduce Differentiable Masking.DIFFMASK learns to maskout subsets of the input while maintaining differentiability.The decision to include or disregard an input token is made with a simple model based on intermediate hidden layers of the analyzed model.First, this makes the approach efficient because we predict rather than search.Second, as with probing classifiers, this reveals what the network 'knows' at the corresponding layers.This lets us not only plot attribution heatmaps but also analyze how decisions are formed across network layers.We use DIFFMASK to study BERT models on sentiment classification and question answering.1Question: Where did the Broncos practice for the Super Bowl ?

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.18653/v1/2020.emnlp-main.262

Cites

9 of 9 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Other work learns masks over network components to remove irrelevant components.

    “Other work learns masks over network components to remove irrelevant components (De Cao et al., 2020;Csordás et al., 2021;Davies et al., 2023).”

    From Open Problems in Mechanistic Interpretability (2025) · cites this paper as De Cao et al., 2020

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Flagged first, then by citations

Research works printed but not linked: 52

  1. Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg. 2017. Fine-grained analy- sis of sentence embeddings using auxiliary predic- tion tasks. International Conference on Learning Representations.
  2. Betty van Aken, Benjamin Winter, Alexander Löser, and Felix A Gers. 2019. How Does BERT Answer Questions? A Layer-Wise Analysis of Transformer Representations. In Proceedings of the 28th ACM In- ternational Conference on Information and Knowl- edge Management, pages 1823-1832. · DOI
  3. David Baehrens, Timon Schroeter, Stefan Harmel- ing, Motoaki Kawanabe, Katja Hansen, and Klaus- Robert M Þller. 2010. How to explain individual classification decisions. Journal of Machine Learn- ing Research, 11(Jun):1803-1831.
  4. Jasmijn Bastings, Wilker Aziz, and Ivan Titov. 2019. Interpretable neural predictions with differentiable binary variables. In Proceedings of the 57th Annual Meeting of the Association for Computational Lin- guistics, pages 2963-2977, Florence, Italy. Associa- tion for Computational Linguistics. · DOI
  5. Yonatan Belinkov and James Glass. 2019. Analysis methods in neural language processing: A survey. Transactions of the Association for Computational Linguistics, 7:49-72. · DOI
  6. Stephen Boyd, Stephen P Boyd, and Lieven Vanden- berghe. 2004. Convex optimization. Cambridge uni- versity press. · DOI · Paper page
  7. Jianbo Chen, Le Song, Martin Wainwright, and Michael Jordan. 2018. Learning to explain: An information-theoretic perspective on model inter- pretation. In International Conference on Ma- chine Learning, volume 80 of Proceedings of Ma- chine Learning Research, pages 883-892, Stock- holmsmässan, Stockholm Sweden. PMLR.
  8. Kyunghyun Cho, Bart van Merriënboer, Caglar Gul- cehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Nat- ural Language Processing (EMNLP), pages 1724- 1734, Doha, Qatar. Association for Computational Linguistics. · DOI · Paper page
  9. Nicola De Cao, Wilker Aziz, and Ivan Titov. 2019. Question answering by reasoning across documents with graph convolutional networks. In Proceed- ings of the 2019 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2306-2317, Min- neapolis, Minnesota. Association for Computational Linguistics. · DOI · Paper page
  10. Shi Feng, Eric Wallace, Alvin Grissom II, Mohit Iyyer, Pedro Rodriguez, and Jordan Boyd-Graber. 2018. Pathologies of neural models make interpretations difficult. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3719-3728, Brussels, Belgium. Association for Computational Linguistics. · DOI
  11. Chaoyu Guan, Xiting Wang, Quanshi Zhang, Runjin Chen, Di He, and Xing Xie. 2019. Towards a deep and unified understanding of deep neural models in NLP. In International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 2454-2463, Long Beach, California, USA. PMLR.
  12. Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. 2018. Annotation artifacts in natural lan- guage inference data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 107-112, New Orleans, Louisiana. Associa- tion for Computational Linguistics. · DOI · Paper page
  13. Yaru Hao, Li Dong, Furu Wei, and Ke Xu. 2019. Visu- alizing and understanding the effectiveness of BERT. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNLP-IJCNLP), pages 4143- 4152, Hong Kong, China. Association for Computa- tional Linguistics. · DOI
  14. Kenneth Holstein, Jennifer Wortman Vaughan, Hal Daumé, Miro Dudik, and Hanna Wallach. 2019. Im- proving fairness in machine learning systems: What do industry practitioners need? In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, CHI '19, page 1-16, New York, NY, USA. Association for Computing Machinery. · DOI · Paper page
  15. Alon Jacovi and Yoav Goldberg. 2020. Towards faith- fully interpretable NLP systems: How should we de- fine and evaluate faithfulness? In Proceedings of the 58th Annual Meeting of the Association for Compu- tational Linguistics, pages 4198-4205, Online. As- sociation for Computational Linguistics. · DOI · Paper page
  16. Sarthak Jain and Byron C. Wallace. 2019. Attention is not Explanation. In Proceedings of the 2019 Con- ference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologies, Volume 1 (Long and Short Pa- pers), pages 3543-3556, Minneapolis, Minnesota. Association for Computational Linguistics. · DOI
  17. Eric Jang, Shixiang Gu, and Ben Poole. 2017. Cat- egorical reparameterization with Gumbel-Softmax. International Conference on Learning Representa- tions.
  18. Xisen Jin, Junyi Du, Zhongyu Wei, Xiangyang Xue, and Xiang Ren. 2020. Towards Hierarchical Im- portance Attribution: Explaining Compositional Se- mantics for Neural Sequence Models. International Conference on Learning Representations.
  19. Divyansh Kaushik and Zachary C. Lipton. 2018. How much reading does reading comprehension require? a critical investigation of popular benchmarks. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 5010-5015, Brussels, Belgium. Association for Computational Linguistics. · DOI
  20. Been Kim. 2015. Interactive and interpretable ma- chine learning models for human machine collabo- ration. Ph.D. thesis, Massachusetts Institute of Tech- nology.
  21. Diederik P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. International Conference on Learning Representations.
  22. Olga Kovaleva, Alexey Romanov, Anna Rogers, and Anna Rumshisky. 2019. Revealing the dark secrets of BERT. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-IJCNLP), pages 4365-4374, Hong Kong, China. Association for Computational Linguistics. · DOI
  23. Jiwei Li, Will Monroe, and Dan Jurafsky. 2016. Un- derstanding neural networks through representation erasure. arXiv:1612.08220. · arXiv
  24. Chris J Maddison, Andriy Mnih, and Yee Whye Teh. 2017. The concrete distribution: A continuous re- laxation of discrete random variables. International Conference on Learning Representations.
  25. Paul Michel, Omer Levy, and Graham Neubig. 2019. Are sixteen heads really better than one? In H. Wal- lach, H. Larochelle, A. Beygelzimer, F. d'Alché Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 14014- 14024.
  26. Curran Associates, Inc. W James Murdoch and Arthur Szlam. 2017. Automatic rule extraction from long short term memory net- works. International Conference on Learning Rep- resentations.
  27. Weili Nie, Yang Zhang, and Ankit Patel. 2018. A theoretical explanation for perplexing behaviors of backpropagation-based visualizations. In Interna- tional Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 3809-3818, Stockholmsmässan, Stockholm Sweden. PMLR.
  28. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natu- ral Language Processing, pages 2383-2392, Austin, Texas. Association for Computational Linguistics. · DOI · Paper page
  29. Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. 2014. Stochastic backpropagation and ap- proximate inference in deep generative models. In International Conference on Machine Learning, vol- ume 32 of Proceedings of Machine Learning Re- search, pages 1278-1286, Bejing, China. PMLR.
  30. Marco Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. "why should I trust you?": Explaining the pre- dictions of any classifier. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Demon- strations, pages 97-101, San Diego, California. As- sociation for Computational Linguistics. · DOI · Paper page
  31. Tim Rocktäschel, Edward Grefenstette, Karl Moritz Hermann, Tomáš Kočiskỳ, and Phil Blunsom. 2016. Reasoning about entailment with neural attention. International Conference on Learning Representa- tions (ICLR).
  32. Michael Sejr Schlichtkrull, Nicola De Cao, and Ivan Titov. 2020. Interpreting Graph Neural Net- works for NLP With Differentiable Edge Masking. arXiv:2010.00577. · arXiv
  33. Sofia Serrano and Noah A. Smith. 2019. Is attention interpretable? In Proceedings of the 57th Annual Meeting of the Association for Computational Lin- guistics, pages 2931-2951, Florence, Italy. Associa- tion for Computational Linguistics. · DOI · Paper page
  34. Lloyd S Shapley. 1953. A value for n-person games, volume 2, pages 307-317. Princeton University Press, Princeton. · DOI · Paper page
  35. Avanti Shrikumar, Peyton Greenside, and Anshul Kun- daje. 2017. Learning important features through propagating activation differences. In International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 3145-3153, International Convention Centre, Syd- ney, Australia. PMLR.
  36. Karen Simonyan, Andrea Vedaldi, and Andrew Zisser- man. 2014. Deep inside convolutional networks: Vi- sualising image classification models and saliency maps. Workshop at International Conference on Learning Representations.
  37. Chandan Singh, W. James Murdoch, and Bin Yu. 2019. Hierarchical interpretations for neural network pre- dictions. In International Conference on Learning Representations.
  38. Leon Sixt, Maximilian Granz, and Tim Landgraf. 2020. When Explanations Lie: Why Many Modified BP Attributions Fail. International Conference on Ma- chine Learning.
  39. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment tree- bank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1631-1642, Seattle, Washington, USA. Asso- ciation for Computational Linguistics.
  40. Pascal Sturmfels, Scott Lundberg, and Su-In Lee. 2020. Visualizing the impact of feature attribution base- lines. Distill. https://distill.pub/2020/attribution- baselines. · DOI
  41. Tony Sun, Andrew Gaut, Shirlyn Tang, Yuxin Huang, Mai ElSherief, Jieyu Zhao, Diba Mirza, Elizabeth Belding, Kai-Wei Chang, and William Yang Wang. 2019. Mitigating gender bias in natural language processing: Literature review. In Proceedings of the 57th Annual Meeting of the Association for Com- putational Linguistics, pages 1630-1640, Florence, Italy. Association for Computational Linguistics. · DOI · Paper page
  42. Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017. Axiomatic attribution for deep networks. In In- ternational Conference on Machine Learning, vol- ume 70 of Proceedings of Machine Learning Re- search, pages 3319-3328, International Convention Centre, Sydney, Australia. PMLR.
  43. Saeid Asgari Taghanaki, Mohammad Havaei, Tess Berthier, Francis Dutil, Lisa Di Jorio, Ghassan Hamarneh, and Yoshua Bengio. 2019. Infomask: Masked variational latent representation to localize chest disease. In Medical Image Computing and Computer Assisted Intervention -MICCAI 2019, pages 739-747, Cham. Springer International Pub- lishing. · DOI
  44. Tijmen Tieleman and Geoffrey Hinton. 2012. Lecture 6.5-RmsProp: Divide the gradient by a running av- erage of its recent magnitude. COURSERA: Neural networks for machine learning, 4(2):26-31.
  45. Shikhar Vashishth, Shyam Upadhyay, Gaurav Singh Tomar, and Manaal Faruqui. 2019. Attention inter- pretability across NLP tasks. arXiv:1909.11218. · arXiv
  46. Elena Voita, Rico Sennrich, and Ivan Titov. 2019a. The bottom-up evolution of representations in the trans- former: A study with machine translation and lan- guage modeling objectives. In Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4396-4406, Hong Kong, China. Association for Computational Linguistics. · DOI
  47. Elena Voita, David Talbot, Fedor Moiseev, Rico Sen- nrich, and Ivan Titov. 2019b. Analyzing multi-head self-attention: Specialized heads do the heavy lift- ing, the rest can be pruned. In Proceedings of the 57th Annual Meeting of the Association for Com- putational Linguistics, pages 5797-5808, Florence, Italy. Association for Computational Linguistics. · DOI · Paper page
  48. Ronald J. Williams. 1992. Simple statistical gradient- following algorithms for connectionist reinforce- ment learning. Machine Learning, 8(3):229-256. · DOI
  49. Thomas Wolf, L Debut, V Sanh, J Chaumond, C De- langue, A Moi, P Cistac, T Rault, R Louf, M Fun- towicz, et al. 2019. Huggingface's transform- ers: State-of-the-art natural language processing. arXiv:1910.03771. · arXiv
  50. Zhitao Ying, Dylan Bourgeois, Jiaxuan You, Marinka Zitnik, and Jure Leskovec. 2019. GNNExplainer: Generating Explanations for Graph Neural Net- works. In H. Wallach, H. Larochelle, A. Beygelz- imer, F. d'Alché Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 9244-9255.
  51. Michael Zhang, James Lucas, Jimmy Ba, and Geof- frey E Hinton. 2019. Lookahead optimizer: k steps forward, 1 step back. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché Buc, E. Fox, and R. Gar- nett, editors, Advances in Neural Information Pro- cessing Systems 32, pages 9597-9608.
  52. Luisa M Zintgraf, Taco S Cohen, Tameem Adel, and Max Welling. 2017. Visualizing deep neural network decisions: Prediction difference analysis. International Conference on Learning Representa- tions.

Cited by

Sorted by citations

Metadata

Authors
De Cao, N., Schlichtkrull, M. S., Aziz, W., & Titov, I.
Affiliations
University of Amsterdam (NL)
Venue
Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)
Publisher
Association for Computational Linguistics
Published
2020
Pages
3243-3255
DOI
10.18653/v1/2020.emnlp-main.262
OpenAlex
W3022176767
Cited by
75
References
9
Citations by year
2026: 5 · 2025: 9 · 2024: 6 · 2023: 18 · 2022: 12 · 2021: 20 · 2020: 5
FWCI
4.40
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
  • Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • UvA-DARE (University of Amsterdam) (submittedVersion, cc-by)
  • Queen Mary Research Online (Queen Mary University of London) (submittedVersion)
  • Data Archiving and Networked Services (DANS) (submittedVersion, other-oa)
  • arXiv (Cornell University)
Funders
  • European Commission (HORIZON2020, 825299)
  • Nederlandse Organisatie voor Wetenschappelijk Onderzoek (639.022.518)
  • H2020 European Research Council (678254)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
Keywords
input masking, BERT, sentiment classification, question answering, probing classifiers, hindsight bias

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review