Hidden Conflicts in Neural Networks and Their Implications for Explainability

Abstract

Artificial Neural Networks (ANNs) often represent conflicts between features, arising naturally during training as the network learns to integrate diverse and potentially disagreeing inputs to better predict the target variable. Despite their relevance to the ``reasoning'' processes of these models, the properties and implications of conflicts for understanding and explaining ANNs remain underexplored. In this paper, we develop a rigorous theory of conflicts in ANNs and demonstrate their impact on ANN explainability through two case studies. In the first case study, we use our theory of conflicts to inspire the design of a novel feature attribution method, which we call Conflict-Aware Feature-wise Explanations (CAFE). CAFE separates the positive and negative influences of features and biases, enabling more faithful explanations for models applied to tabular data. In the second case study, we take preliminary steps towards understanding the role of conflicts in out-of-distribution (OOD) scenarios. Through our experiments, we identify potentially useful connections between model conflicts and different kinds of distributional shifts in tabular and image data. Overall, our findings demonstrate the importance of accounting for conflicts in the development of more reliable explanation methods for AI systems, which are crucial for the beneficial use of these systems in the society.