XAI Benchmarks

[the16thpythonist]

I started to implement XAI datasets and benchmarking. Currently there is only the base functionality, which uses a mock dataset, but I figured I rather create a PR early so that I don't deviate from the master branch too much.

I'll briefly explain what & how I implemented.

Benchmarks with train_visual_graph_datasets.py

I added a new training script train_visual_graph_datasets.py, which can be used to benchmark on special VisualGraphDatasets. The script can be executed without additional command line options:

python3 train_visual_graph_datasets.py

This will train the MEGAN model on the mock dataset.

I tried to follow the structure of the other train scripts as best as possible. This script also creates the appropriate files in the results folder. I also added the necessary lines in summary.py such that the results can be included into the results-README.md . I tested this workflow on a new Ubuntu VM as well and there it worked.

Visual Graph Datasets

The datasets that can be used with this training script have to be sub classes of the new base class kgcnn.data.visual_graph_dataset.VisualGraphDataset. The special thing about this kind of dataset is that they contain the entire graph representation of the elements rather than a reduced representation like SMILES for example. Additionally, these datasets also contain PNG files for each element which visualize the graph in some way. This is important to later on visualize the attributional explanations.

I decided to package the entire business logic behind these visual graph datasets in an independent package: https://github.com/awa59kst120df/visual_graph_datasets , which means that the VisualGraphDatasets class is relatively simple. The package for example wraps the functionality of downloading the datasets from a remote Nextcloud (BwSync&Share) file share, if they don't already exist. Once downloaded they are permanently stored in a folder within the users home directory.

Other Changes

I noticed and fixed some more bugs in the implementation of MEGAN, for example that the explanation sparsity regularization was completely missing until now.

Additionally I refactored existing code in a few minor places. For all the code I modified I added unittests in beforehand to make sure the functionality isn't broken.

For the Future

Datasets
Currently, the only dataset I added is a mock dataset of 100 graphs which mainly serves testing purposes. In the future I want to add the datasets I have been working with until now as well, which includes the RbMotifs dataset & the AqSolDB solubility dataset. Possibly also interesting is the datasets from https://papers.nips.cc/paper/2020/hash/417fbbf2e9d5a28a855a11894b2e795a-Abstract.html which present molecule datasets with ground truth explanations

Models
Currently, the only model which is able to work with these datasets is MEGAN, but I also want to extend to GNNExlainer and GradCAM at least since I already implemented them in my own code.

Metrics
The only explainability metric currently implemented is explanation accuracy AUC. There are other interesting XAI metrics however, sparsity and fidelity being the most interesting at first I think.

[PatReis]

I think I will refactor visual_graph_dataset into just visual_graph.