Here, we will use the LRP method to illustrate the principle of determining which features in an input contribute most strongly to the output of a trained DNN model. In this example, the input to the network is an image.** We are interested in figuring out the relevance* of each image pixel to a given image class. This could be the final DNN classification decision for that image (“cat”) or any other class known to the model (e.g. “dog”, “cow”, “bee”, etc. if the model is trained to classify animals). The output relevance is considered a weighted sum of the relevancies of all the pixels of the input image. The relevance at the output node in the network graph is decomposed by iterative propagation backward through the network layers until the input layer. The trained model weights are used in this decomposition. Using this relevance propagation process, the final *relevance scores map *(or heatmap) is obtained. The final relevance map gives us an indication for the DNN’s hidden logic.

How to find your Artificial Intelligence explainer Relevance scores of the image pixels in respect to the class “cat” obtained via relevance propagation.Besides LRP, there are many other XAI methods, which use different ways of determining relevance. In order to determine which of the plethora of post-hoc XAI methods to include in DIANNA, we started by compiling an initial list of 18 XAI post-hoc feature-relevance methods following Arrieta et al. (2020):

We excluded all XAI methods without a working and well-maintained open-source implementation or proven user community. That resulted in a subset of the 13 methods highlighted above.

Evaluation criteria

The large variety of XAI approaches benefits the AI community by providing freedom in the search for explainability. However, it also increases the difficulty of choosing the most appropriate method for each use case. Every XAI approach has pros and cons. In order to select the best methods for our task, we needed a framework to evaluate these methods systematically. Fortunately, this topic has already been visited by the XAI researchers and there is an extensive list of objective criteria defined in the Explainability fact sheet paper [Sokol, K., & Flach, P. (2020)]. The authors look at an abundant set of XAI aspects and cluster the requirements towards an explainer into **5 groups **(or dimensions):

  • Functional (F1-F9)
  • Operational (O1-O10)
  • Usability (U1-U11)
  • Safety (S1-S4)
  • Validation (V1-V2)

For each dimension, there are specific criteria, e.g. the “Functional” group has 9 and in total there are 36 criteria. We will describe our choices next.

The selection

In our DIANNA project, we built a Python package that brings XAI to non (X)AI experts and various scientific domain researchers. The XAI methods to be included in the package had to have the capacity to work with different types of data, various AI architectures and we chose to focus on the model explanation. Most importantly, the resulting explanations had to be easy to interpret. Based on our goals, we have compiled a shortlist of 18 requirements from the 36 requirements in the XAI fact sheet. Using these relevant for our target users’ requirements, we evaluated the selected 13 methods listed in the XAI overview section.

Apart from the requirements on the list, we have also considered the similarity and the “popularity” of these methods, the quality of the existing implementations, and the difficulty of integrating them in DIANNA. The evaluation procedure can be summarized with the table shown below: (Deconvnet and GradCAM are filtered out since they can only work with Convolutional NNs and are therefore not included in the table.)

Finally, we narrowed down our list to 5 methods:

  • Layer-wise Relevance Propagation (LRP)
  • Local Interpretable Model-agnostic Explanations (LIME)
  • SHapley Additive exPlanations (SHAP)
  • PatternAttribution (PA)
  • Randomized input sampling for explanation of black-box models (RISE)

Within limited project time, we were able to include three of them in our DIANNA library: RISE, LIME & SHAP. A preview of the explanations that we obtained using our DIANNA library is shown below:

How to find your Artificial Intelligence explainer Explanation for “0” with RISE, LIME, and SHAP using DIANNA library for a model trained on binary MNIST data. Red indicates high or positive relevance, blue- low or negative.We could observe that the methods are complimentary to each other in respect to their output, which can be particularly useful for researchers seeking new insights to their data and problem. We also believe that simpler datasets, e.g. a binary (only 0s and 1s) MNIST should be used as “Hello world” example for the XAI (user) community!

Our library is open-source and it is ready to help you decrypt your neural network models. Wait, you want to ask which format do we support? TensorFlow? PyTorch? Don’t worry. We can work with all of them, as we support ONNX — the de-facto DNN format standard! Any contributions to our repository are very welcome (talk is cheap, show me your pull request! Want to give it a try? Check out the link below our logo!

How to find your Artificial Intelligence explainer https://github.com/dianna-ai/dianna

Acknowledgments

The work described in this blog has been performed by the members of the DIANNA team: Christiaan Meijer, Leon Oostrum, Yang Liu and Elena Ranguelova with the big help of Patrick Bos and Giulia Crocioni. Special thanks to Christiaan Meijer, Giulia Crocioni, Matthieu Laneuville, Pablo Rodríguez-Sánchez and Lieke de Boer for their valuable contributions. The DIANNA project is supported by Netherlands eScience Center and SURF.

References

[Arrieta et al. (2020)*] Arrieta, A. B., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., … & Herrera, F. (2020). Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI. Information fusion, 58, pp. 82–115.[Sokol, K., & Flach, P. (2020)*] Sokol, K., & Flach, P. (2020, January). Explainability fact sheets: a framework for systematic assessment of explainable approaches. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, pp. 56–67.