Thomas Fel

David Vigouroux

Rémi Cadène

Thomas Serre

How Good is your Explanation? Algorithmic Stability Measures to Assess the Quality of Explanations for Deep Neural Networks

whether they are published or not. The documents may come

Introduction

Machine learning approaches such as deep neural networks have become essential in multiple domains such as image classification, language processing and speech recognition. These approaches have achieved excellent classification accuracy -approaching human performance in specific domains [START_REF] Lecun | Deep learning[END_REF][START_REF] Serre | Deep learning: The good, the bad, and the ugly[END_REF]. However, one significant drawback associated with these deep networks is that it is difficult to interpret their decisions [START_REF] Lipton | The mythos of model interpretability[END_REF]. This problem constitutes a serious obstacle for the wide adoption of these systems for safety-critical applications.

Recently, several explainability methods have been pro-Email correspondance to: thomas fel@brown.edu Figure 1. The explanations of a predictor f can be trusted when they are algorithmically stable. It means that, on average, for any image x removed for the training set D, another predictor f |x trained with the same algorithm A produces a similar explanation for the same input image. This stability ensures that the explanations of correctly classified images of the same class will point out similar evidence of the class. For instance, the explanations for the class panda will all point out the black area around their eyes.

posed to help understand how predictors make a particular decision [START_REF] Sundararajan | Axiomatic attribution for deep networks[END_REF][START_REF] Smilkov | Smoothgrad: removing noise by adding noise[END_REF][START_REF] Vitali Petsiuk | Rise: Randomized input sampling for explanation of black-box models[END_REF][START_REF] Ramprasaath | Gradcam: Visual explanations from deep networks via gradientbased localization[END_REF]. These methods are also meant to improve predictors and more importantly to build trust with people affected by their predictions. Unfortunately, these methods have strong limitations. In particular, they are subject to confirmation bias: while some methods appear to offer useful explanations to a human experimenter, they turn out not to reflect the actual behaviour of the predictor [START_REF] Adebayo | Sanity checks for saliency maps[END_REF][START_REF] Ghorbani | Interpretation of neural networks is fragile[END_REF]. Instead of providing confidence in a system's decisions, these explanations can themselves be potentially erroneous and cannot be trusted.

In this work, we focus on evaluation methods that objectively assess the quality and trustworthiness of explanations. Numerous methods have been proposed [START_REF] Diogo | Machine learning interpretability: A survey on methods and metrics[END_REF][START_REF] Tintarev | A survey of explanations in recommender systems[END_REF][START_REF] Miller | Explanation in artificial intelligence: Insights from the social sciences[END_REF][START_REF] Robnik | Perturbationbased explanations of prediction models[END_REF][START_REF] Gilpin | Explaining explanations: An overview of interpretability of machine learning[END_REF][START_REF] Alvarez | Towards robust interpretability with self-explaining neural networks[END_REF]. They are often derived from axioms and properties that good explanations should possess. The most studied property is Fidelity. It is associated with a metric that evaluates an explanation method on its ability to reflect the internal decision process of a given neural network predictor. An important limitation of Fidelity is that it does not take into account the prediction capability of the predictor. For instance, an explainability method could have a high Fidelity given a random predictor. Thus, Fidelity is only a first step towards predictors that are explainable and can be trusted.

Instead, we propose to additionally take the prediction capability of the predictor into account through the Generalizability and Consistency of its explanations. A shown in Figure 1, we borrow these two concepts from the fields of generalization and algorithmic stability [START_REF] Bousquet | Stability and generalization[END_REF], and adapt them to the field of explainability. Intuitively, when some images of a certain class are well classified by a given predictor, trustworthy explanations should point out to the same evidence accross all images of the same category. For instance, the black region near the eyes of a panda. That is captured by the notion of Generalizability. Conversely, when images of the same category are misclassified, trustworthy explanations should ideally point out to image evidence which differs from that used for correctly classified images. That is captured well by the notion of Consistency. In practice, we estimate with a k-fold cross-training approach the metrics associated with different predictors, i.e., their average generalizability (MeGe) and their relative consistency (ReCo).

We provide an extensive experimental validation of our approach using different neural networks, common explainability methods and several image datasets. We compare the average generalizability (MeGe) and relative consistency (ReCo) measures against the leading Fidelity measure [START_REF] Bhatt | Evaluating and aggregating feature-based model explanations[END_REF][START_REF] Yeh | On the (in)fidelity and sensitivity for explanations[END_REF]. Importantly, we experimentally show that the Fidelity is not enough to ensure that a given pretrained model produces trustworthy explanations. Finally, we show that 1-Lipschitz networks obtain a higher average generalizability (MeGe) and relative consistency (ReCo) compared to common neural networks while reaching similar accuracies. This finding offers an interesting research track towards more explainable and trustworthy predictors. To summarize, our contributions include:

• Generalizability (MeGe) and Consistency (ReCo) practical measures to assess the quality and trustworthiness of explanations, • Extensive experimental validation of our approach on several neural networks, explainability methods, and image datasets (including ImageNet). • Empirical demonstration that 1-Lipschitz networks deliver explanations with higher MeGe and ReCo.

Related work

In this work, we focus on evaluating explanations provided by explainability methods, which give insight into how a given neural network architecture reaches a particular decision [START_REF] Doshi | Towards a rigorous science of interpretable machine learning[END_REF]. These explainability methods produce an influence score for each input dimension. In the case of image classification, these methods will produce heatmaps indicating the diagnosticity of individual image regions. Most of these explainability methods rely on backpropagating the gradient with respect to a given input image [START_REF] Zeiler | Adaptive deconvolutional networks for mid and high level feature learning[END_REF][START_REF] Simonyan | Deep inside convolutional networks: Visualising image classification models and saliency maps[END_REF][START_REF] Bach | On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation[END_REF][START_REF] Fong | Interpretable explanations of black boxes by meaningful perturbation[END_REF][START_REF] Shrikumar | Learning important features through propagating activation differences[END_REF][START_REF] Sundararajan | Axiomatic attribution for deep networks[END_REF][START_REF] Smilkov | Smoothgrad: removing noise by adding noise[END_REF][START_REF] Ramprasaath | Gradcam: Visual explanations from deep networks via gradientbased localization[END_REF][START_REF] Hartley | Swag: Superpixels weighted by average gradients for explanations of cnns[END_REF] or with respect to a perturbation of the input [START_REF] Matthew | Visualizing and understanding convolutional networks[END_REF][START_REF] Zhou | Object detectors emerge in deep scene cnns[END_REF][START_REF] Tulio Ribeiro | Explaining the predictions of any classifier[END_REF][START_REF] Li | Understanding neural networks through representation erasure[END_REF][START_REF] Zintgraf | Visualizing deep neural network decisions: Prediction difference analysis[END_REF][START_REF] Tulio Ribeiro | Anchors: High-precision model-agnostic explanations[END_REF][START_REF] Thomas | Look at the variance! efficient black-box explanations with sobolbased sensitivity analysis[END_REF].

Despite a wide range of explainability methods, assessing the quality and trustworthiness of these explanations is still an open problem. It is in part due to the difficulty of obtaining objective ground truths [START_REF] Samek | Evaluating the visualization of what a deep neural network has learned[END_REF][START_REF] Linsley | Learning what and where to attend[END_REF]. Several criteria have been proposed to evaluate the quality of explanations [START_REF] Tintarev | A survey of explanations in recommender systems[END_REF][START_REF] Miller | Explanation in artificial intelligence: Insights from the social sciences[END_REF][START_REF] Robnik | Perturbationbased explanations of prediction models[END_REF][START_REF] Gilpin | Explaining explanations: An overview of interpretability of machine learning[END_REF][START_REF] Alvarez | Towards robust interpretability with self-explaining neural networks[END_REF][START_REF] Diogo | Machine learning interpretability: A survey on methods and metrics[END_REF][START_REF] Ferrettini | Coalitional strategies for efficient individual prediction explanation[END_REF]. According to [START_REF] Diogo | Machine learning interpretability: A survey on methods and metrics[END_REF], the five major properties include: Fidelity, Stability, Comprehensibility, Generalizability and Consistency. Yet, properties such as Generalizability and Consistency do not come with a practical definition.

In order to measure these different properties, there are two main approaches currently used. The first subjective approach consists in putting the human at the heart of the process, either by explicitly asking for human feedback [START_REF] Ramprasaath | Gradcam: Visual explanations from deep networks via gradientbased localization[END_REF][START_REF] Tulio Ribeiro | Explaining the predictions of any classifier[END_REF][START_REF] Lundberg | A unified approach to interpreting model predictions[END_REF], or by indirectly measuring the performance of the human/classifier duo [START_REF] Lage | An evaluation of the human-interpretability of explanation[END_REF][START_REF] Chang | Reading tea leaves: How humans interpret topic models[END_REF][START_REF] Narayanan | How do humans un-derstand explanations from machine learning systems? an evaluation of the human-interpretability of explanation[END_REF][START_REF] Schmidt | Quantifying interpretability and trust in machine learning systems[END_REF]. Nevertheless, human intervention sometimes brings undesirable effects, including a possible confirmation bias [START_REF] Adebayo | Sanity checks for saliency maps[END_REF].

A second type of approaches has also started to emerge specifically for computer vision applications. The main idea is to build objective proxy tasks that a good explanation must be able to solve. These measures aim to evaluate explanations based on two properties: Fidelity and Stability. The first method to measure Fidelity was first proposed in [START_REF] Samek | Evaluating the visualization of what a deep neural network has learned[END_REF] based on estimating the drop in prediction score resulting from deleting pixels deemed important by an explanation method. To ensure that the drop in score does not come from a change in distribution, ROAR [START_REF] Hooker | A benchmark for interpretability methods in deep neural networks[END_REF] was proposed which re-train a classifier model between each deletion step. This boils down to measuring the correlation between the attributions for each pixel and the difference in the prediction score when they are modified and has been clearly formalized [START_REF] Yeh | On the (in)fidelity and sensitivity for explanations[END_REF][START_REF] Bhatt | Evaluating and aggregating feature-based model explanations[END_REF][START_REF] Rieger | Irof: a low resource evaluation metric for explanation methods[END_REF]. Nevertheless, it should be noted that the different fidelity metrics proposed requires defining a baseline state which might favor explainability methods that internally relies on the same baseline [START_REF] Sturmfels | Visualizing the impact of feature attribution baselines[END_REF].

Those Fidelity metrics are a first step toward trustworthy explanations: by making sure that we have faithful explanations, we can then look at other criteria to quantitatively measure these explanations. However, the different Fidelity metrics suffer from limitations demonstrated by [START_REF] Tomsett | Sanity checks for saliency metrics[END_REF] which proposes several sanity checks revealing their high variance, their sensitivity to the baseline and their low consistency.

Methods

Below, we briefly provide some motivation for the proposed MeGe and ReCo measures before describing a training procedure applicable to a large family of machine learning models in order to estimate these two values. One basic assumption for the proposed approach is borrowed from algorithmic stability and generalization: to be reliable, an explanation obtained for a specific predictor for a given image should be stable when the training dataset used to train the predictor is perturbed slightly as when, for instance, this particular image is added or removed from the dataset.

Notations

We consider a standard supervised learning setting where a datapoint is denoted z = (x, y) s.t. x ∈ X is an observation (e.g., X = R d ) and y ∈ Y is a class label (e.g., Y = R p ). The data set is denoted as D = {z 1 , ..., z m }, we designate V = {V 1 , ..., V k } the set of k disjoints subsets (folds) of size m/k at random where each V i ⊂ D. Throughout this work, we will assume k divides m for convenience. Let A be a deterministic learning algorithm that maps any number of data points onto a function f from X to Y. In particular, we consider the fold V i and the associated predictor

f i = A(V \ V i ).
An explanation method is a functional, denoted Φ, which, given a predictor f i and a datapoint x, assigns an importance score for each input dimension φ (i) x = Φ(f i , x). We define a distance d(•, •) over the explanations. Finally, the following Boolean connectives are used: ¬ denotes a negation, ∧ denotes a conjunction, and ⊕ denotes an exclusive or (XOR).

Motivation

We first consider Generalizability: we provide a definition, discuss the inherent difficulties associated with its measurement, and describe a method for estimating it. We then motivate the need for assessing the Consistency of an explanation and propose a measure. Definition 1 Generalizability A measure of how generalizable an explanation is, and the extent to which it truly reflects the underlying process by which the predictor makes a decision.

Intuitively, a representative explanation would be an explanation that holds for a large number of samples. To assess the number of samples that can be covered by a given explanation, it might be tempting to compute a distance between the explanations associated with those samples. However, because of the large variations in the appearance of objects that arise because of translation, scale, and 3D rotation in natural images, two explanations can be similar (i.e., close in pixel space) without necessarily reflecting a similar visual strategy used by the predictor (for instance, decisions could be driven by the same pixel locations -yet driven by different visual features). Conversely, two spatially distant explanations could be based on the same features that appear at different locations because of translation.

Our proposed solution to this problem is to only use distance measured between explanations for the same sample. This constraint leads us to consider the notion of algorithmic stability as a proxy for generalization: intuitively, given a predictor and a training data set, a good explanation for a decision made for a given data point should be robust to the addition or removal of that data point from the training set. One benefit of such a characteristic is that it can be evaluated based solely on a distance between explanations from the same sample.

In what follows, we will propose a relaxed version of the algorithmic stability -computationally more manageableapplied to the explanations using several predictors trained on different folds. It is important to note that the term algorithmic stability [START_REF] Bousquet | Stability and generalization[END_REF] is not related to the Stability of an explanation as defined in [START_REF] Bhatt | Evaluating and aggregating feature-based model explanations[END_REF].

Following this consideration, we will be looking at how well a predictor's explanations generalize from seen to unseen data points:

δ (i,j) x = d(φ i x , φ j x ) s.t. x ∈ V i , i = j. (1)
By making sure that x belongs to the fold V i , we measure the distance between two explanations, one of which comes from a predictor that was not fitted to the sample x. By computing these distances, we hope to characterize the Generalizability of the explanations.

Definition 2 Consistency

The extent to which different predictors trained on the same task do not exhibit logical contradictions.

A statement, or a set of statements, is said to be logically consistent when it has no logical contradictions. A logical contradiction occurs when both a statement and its negation are found to be true. In logic, a fundamental law -the law of non-contradiction -is that a statement and its negation cannot both be true simultaneously. Similarly, we measure the consistency between explanations by ensuring that contradictory predictions lead to different explanations.

Following this definition, if the same explanation gets associated with two contradictory predictions the explanation is said to be inconsistent. This means avoiding the case where for an observation x ∈ V i , two predictors f i , f j (where i = j), trained on the same task, give the same explanation but different predictions:

f i (x) = f j (x) =⇒ φ (i) x = φ (j) x (2)
Nevertheless, we have to define what it means for two explanations to be different. For this, we use a measure of dissimilarity between explanations and a threshold to judge whether the explanations are consistent or not. This threshold will be relative to the distance between explanations when predictions are not contrary. By measuring the rate of inconsistent explanations, we hope to capture the notion of Consistency for explanations.

k-Fold Cross-Training

We recall that our data set is divided into k-folds of the same size D = {V i } k i=0 , and that each predictor is trained through a learning algorithm f i = A(V \ V i ). We assume that the predictors exhibit comparable accuracies across folds. In our experiments, we ensure a similar accuracy on the test set.

We will now measure the distances between two explanations associated with these different predictors. To be more precise, we are really only interested in computing δ (i,j) x (see Eq. 1):, the distance between two explanations whereby one of the two predictors was not fitted on x. Otherwise, it may be trivial for two predictors that were trained on that sample to yield the same explanation -especially if overfitting occurs (see Fig. 2).

In the case where both predictors gave a correct prediction, a small distance between the two explanations suggests that the explanations receive support from several samples. In other words, the fact that explanations do not vary widely when adding or removing a particular sample or set of samples suggests good Generalizability. Alternatively, if the two predictors give contrary predictions, the corresponding explanations should be different. Indeed, the very notion of Consistency between explanations implies that the same explanation cannot account for two different outcomes.

We separate distances into two sets, S = when the predictors have made correct predictions s.t. it is desirable to have a small distance between explanations, S = when one of the predictors have given a wrong prediction s.t. it is desirable to have higher distances between the pairs of explanations. The case where both predictors give a bad prediction is ignored (for details, see the Alg. 1 in the appendix).

S = = {δ (i,j) x : f i (x) = y ∧ f j (x) = y} (3)
S = = {δ (i,j) x : f i (x) = y ⊕ f j (x) = y} (4)
∀(i, j) ∈ {1, ..., k} 2 s.t. i = j, ∀(x, y) ∈ V i

Mean generalizability : MeGe

From Def. 1, the distance between explanations arising from predictors trained on a dataset that contained vs. did not contain a given sample should be small. As those distances are contained in S = , one way to measure the Generalizability of explanations is to compute the average distance over S = .

As a reminder, the average of S = corresponds to the average change of explanation when the sample is removed from the training set. This change is related to the Generalizability of the explanation: the more representative an explanation is, the more it persists when we remove a point.

To ensure a high value for low distances, we define the MeGe measure as a similarity measure:

M eGe = 1 + 1 |S = | δ ∈ S = δ -1 (5)
Explanations with good Generalizability will therefore be associated with higher similarity scores between explanations (close to 1).

Relative consistency : ReCo

From Def. 2 and Eq. 2, explanations arising from different predictors are said to be consistent if they are close when the predictions agree with one another. As a reminder, the distance between explanations for the consistent predictions are represented by S = , and those associated with inconsistent predictions by S = . Visually, we seek to maximize the shift between the corresponding distributions for the sets S = and S = . Formally, we are looking for a distance value that separates S = and S = , e.g., such that all the lower distances belong to S = and the higher ones to S = . The clearer the separation, the more consistent the explanations are. In order to find this separation, we introduce ReCo, a statistical measure based on maximizing the balanced accuracy.

Where S = S = ∪ S = and γ ∈ S a fixed threshold value, we can define the true positive rate T P R as the rate for which distances below a threshold come from consistent predictions among all distances below the threshold T P R(γ) = |{δ∈S = :δ γ}| |{δ∈S : δ γ}| . In a similar way, T N R denotes the rate for which distances above a threshold come from opposite predictions among all the distances above the threshold T N R(γ) = |{δ∈S = :δ>γ}| |{δ∈S : δ>γ}| . Basing our measure on these rates allows us to assess the quality of these explanations independently of the accuracy of the predictor, we define ReCo as the maximal balanced accuracy:

ReCo = max γ∈S T P R(γ) + T N R(γ) -1, (6)
with a score of 1 indicating perfect consistency of the predictors' explanations, and a score of 0 indicating a complete inconsistency.

Experiments

We carried out three sets of experiments using a variety of neural network architectures and explanation methods. The first one consisted in ensuring the functioning and the reliability of the measures ReCo and MeGe via a simple sanity check done over a large number of predictors ( 175Figure 2. Application of the proposed procedure for 3 folds. Each predictor is trained on two of the 3 folds, e.g, f1 is trained on D \ V1. For a given sample x such that x ∈ V1, the explanations for each predictors are calculated (φ

(1)
x , φ

x , φ

x ). The distance between φ

(1)

x and the other two explanations φ

(2)
x , φ

x are computed. All distances for which predictions do not contradict each other are added to S = while the others are added to S = (note that this is the case for δ

(1,3) x since f1(x) = f3(x)).
in total). The second set of experiments consisted in highlighting a limitation of the fidelity measure -namely its independence with respect to the quality of the explanations. This underlines the need for new measures that are dedicated to explanations and not to methods. We developed these considerations in a dedicated section where we demonstrate an application to the selection of a method using the two new criteria MeGe and ReCo. Finally, in a third set of experiments, we showed quantitatively that some predictors are more interpretable: our analyses revealed that 1-Lipschitz neural networks yield explanations that are more coherent and representative.

Setup

For all experiments, we used 5 splits (k = 5), i.e., 5 predictors with comparable accuracy (±3%), which allows us to study the explanations in common training conditions (80% of the data are used for training and 20% for testing).

For ILSVRC 2012, our predictors are based on a ResNet-50 architecture [START_REF] He | Deep residual learning for image recognition[END_REF], and a ResNet-18 for the other datasets (see appendix E for details on each predictor).

Explanation methods

In order to produce the necessary explanations for the experiment, we used 7 methods of explanation. The methods selected are those commonly found in the literature in addition to one control method (Random).

The explanations methods chosen are as follow: Saliency (SA) [START_REF] Simonyan | Deep inside convolutional networks: Visualising image classification models and saliency maps[END_REF], Gradient Input (GI) [START_REF] Ancona | Towards better understanding of gradient-based attribution methods for deep neural networks[END_REF], Integrated Gradients (IG) [START_REF] Sundararajan | Axiomatic attribution for deep networks[END_REF], SmoothGrad (SG) [START_REF] Smilkov | Smoothgrad: removing noise by adding noise[END_REF], Grad-CAM (GC) [START_REF] Ramprasaath | Gradcam: Visual explanations from deep networks via gradientbased localization[END_REF], Grad-CAM++ (G+) [START_REF] Chattopadhay | Grad-cam++: Generalized gradient-based visual explanations for deep convolutional networks[END_REF] and RISE (RI) [START_REF] Vitali Petsiuk | Rise: Randomized input sampling for explanation of black-box models[END_REF]. Further information on these methods can be found in the appendix B.

Datasets We applied the procedure described above and evaluated the proposed measures for each of the degradations on 4 image classification datasets:

ILSVRC 2012 [START_REF] Deng | ImageNet: A Large-Scale Hierarchical Image Database[END_REF]: a subset of the ImageNet dataset from which we randomly selected 50 classes. The size of the images considered was 224 × 224. The reduced number of classes being sufficient to show that the metrics pass the test performed in 4.2 even in the case of high dimensional images.

CIFAR10 [START_REF] Krizhevsky | Learning multiple layers of features from tiny images[END_REF]: a low-resolution labeled datasets with 10 classes respectively, consisting of 60, 000 (32 × 32) color images.

EuroSAT [START_REF] Helber | Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification[END_REF]: a labeled dataset with 10 classes consisting of 27, 000 color images (64 × 64) from the Sentinel-2 satellite.

Fashion MNIST [START_REF] Xiao | Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms[END_REF]: a dataset containing 70, 000 lowresolution (28 × 28) grayscale images labeled in 10 categories.

Distance over explanations The procedure introduced in section 3.3 requires to define a distance between two explanations derived for the same sample. Since a feature attribution consists of ranking the features most sensitive to the predictor's decision, it seems natural to consider the Spearman rank correlation [START_REF] Spearman | The proof and measurement of association between two things[END_REF] to compare the similarity between explanations. Several authors have provided theoretical and experimental arguments in line with this choice [START_REF] Ghorbani | Interpretation of neural networks is fragile[END_REF][START_REF] Adebayo | Sanity checks for saliency maps[END_REF][START_REF] Tomsett | Sanity checks for saliency metrics[END_REF]. However, it is important to note that the problem of measuring similarity between explanations is still an open problem. We conduct two sanity checks: spatial correlation, and noise test on several candidates distances to ensure they could respond to the problem. The distances tested were built from: 1-Wasserstein distance (the Earth mover distance from [START_REF] Flamary | Pot python optimal transport library[END_REF]), Sørensen-Dice [START_REF] Dice | Measures of the amount of ecologic association between species[END_REF] coefficient, Spearman rank correlation, SSIM [START_REF] Zhou Wang | Image quality assessment: from error visibility to structural similarity[END_REF], and 1 and 2 norms. In line with prior work, we chose to use one minus the absolute value of the Spearman rank correlation (see F for more details).

Sanity check for explanation measures

Our first set of experiments aims to ensure that the propose metrics approximate the desired quantities by perform-Figure 3. MeGe and ReCo scores for predictors trained with no degradations (first point from the left), as well as for progressively randomized predictors and predictors trained with switched labels. For all the methods tested, the more the predictor is degraded, the more the Consistency and Generalizability scores drop, which means that the associated metrics pass the sanity check. Top ImageNet. Bottom Cifar-10.

ing a sanity check: on average, as the learning is degraded, we expect to see an overall increase in the number of specific and inconsistent explanations. To ensure that the metric captures these notions, we applied two different types of degradation on the predictors for each data set: weight randomizations and label shuffling.

• Randomizing the weights, inspired by [START_REF] Adebayo | Sanity checks for saliency maps[END_REF]. We gradually randomize 5%, 10% and 30% of the predictor layers by adding Gaussian noise. By degrading the weights learned by the network, we expect to find degraded explanations. • Shuffling of labels, inspired by [START_REF] Behnam Neyshabur | Exploring generalization in deep learning[END_REF][START_REF] Adebayo | Sanity checks for saliency maps[END_REF] the predictors are trained on a data set with 5%, 10% and 30% of bad labels. By artificially breaking the relationship between the labels, we expect the explanations to lose their consistency.

The MeGe measure encodes the Generalizability of the explanations, which is related to the ability of the predictor to derive general strategies. Thus, the degradation of the parameters of a predictor directly affects these strategies. Fig. 3 shows the correlation of the measures with the intensity of the degradation applied: MeGe and ReCo capture the degradation of the explanation and pass the sanity check.

We note that all the tested methods perform better than the random baseline (random). However, the drop in score, is not the same and some methods are more sensitive to predictor changes, such as Grad-CAM or RISE, in accordance with previous work [START_REF] Adebayo | Sanity checks for saliency maps[END_REF][START_REF] Sixt | When explanations lie: Why many modified bp attributions fail[END_REF]. It was subsequently observed that this sensitivity seems to translate into a better Fidelity score for the methods. Nevertheless, it should be noted that this sanity test is a necessary but not sufficient condition for a Generalizability and Consistency metric.

The implications of the fidelity metric

To mark the difference between the proposed measures and the Fidelity, we applied the µF measure from [START_REF] Bhatt | Evaluating and aggregating feature-based model explanations[END_REF] (see supplementary document C) to the normally trained predictors and those progressively degraded. We seek to verify that this property does not pass the sanity check: the fidelity measure is invariant to the performancee the predictor as well as to the quality of its explanations. For µF , the score obtained is averaged over 10, 000 test samples, with 0 for baseline. The size of the |S| subset is 15% of the image. 7) on ImageNet for normally trained ResNet-50 predictors (first point on the left) as well as for progressively randomized predictors and predictors trained with switched labels. Even a strong degradation of the predictor does not impact the Fidelity of the tested methods. Hence, the Fidelity is intended to ensure that the explanations correctly reflect the underlying strategies of the model, regardless of whether these strategies are general or consistent.

As shown in Fig. 4, predictor degradation does not impact the Fidelity metric on the methods tested. The Fidelity property is essential in a good explanation since it allows us to make sure that we are studying the strategies of the predictor. However, it is not sufficient: if the explanation reflects well the strategies of the predictor, the latter may use specific and inconsistent strategies. In that, the Fidelity measure is only a first step towards a good explanation.

Method selection criterion

The MeGe and ReCo measures can be used as additional criteria for choosing an explainability method. As a a good method should provide explanations that are as faithas possible and, if possible, and representative. Thus, the tested methods can be compared using the scores obtained for these measures. We note that these measures are complementary in that the fidelity score can be interpreted as a confidence bound on the other measures performed on the explanations.

Table 1 reports the Fidelity (µF ), Consistency (ReCo) and Generalizability (MeGe) scores obtained for the ResNet-50 predictors trained without degradation on ImageNet. We can exploit a selection criterion from the differences in scores. First of all, we notice that the two methods obtaining a good fidelity score are RISE and Grad-CAM, they reflect well the predictor functioning. Their high fidelity score acts as a confidence bound on the MeGe and ReCo metrics: by correctly transcribing the functioning of the predictor, we obtain at the same time the Generalizability and the Consistency of the explanations. This score can then be used as a criterion to separate RISE from Grad-CAM. In view of the differences between the MeGe and ReCo scores, RISE method seems preferable. Concerning the Generalizability score, it is important to note that two methods tested here involve the elementwise product of the explanation with the input: Integrated Gradients and Gradient Input. This operation could eliminate the attribution score on a part of the image, thus reducing the distance between the two explanations. The result is a better MeGe score which is in fact due to the dominance of input in the element-wise product.

It can be observed that the change of predictor has an effect on this ranking, and that a good method of explainability must be chosen according to a context: predictor and data set. However, even considering these effects, the experiments carried out suggest 3 methods that give faithful, representative and consistent explanations: Grad-CAM, Grad-CAM++ and RISE (for more results on Cifar-10, EuroSAT and Fashion MNIST, see appendix G).

Towards predictors with better explanations

In an attempt to find predictors that give better explanations, we extend the experience on the Cifar-10 dataset by adding a family of 1-Lipschitz networks. Indeed different works mention the Lipschitz constrained networks as particularly robust [START_REF] Usama | Towards robust neural networks with lipschitz continuity[END_REF][START_REF] Scaman | Lipschitz regularity of deep neural networks: analysis and efficient estimation[END_REF][START_REF] Pauli | Training robust neural networks using lipschitz bounds[END_REF][START_REF] Béthune | The many faces of 1-lipschitz neural networks[END_REF] and have good generalizability. As a reminder, a

f function is called L-Lipschitz, with L ∈ R + if |f (x 1 ) -f (x 2 )| ≤ L|x 1 -x 2 | For every pair (x 1 , x 2 ) ∈ X 2 .
The smallest of these L is called the Lipschitz constant of f . This constant certifies that the gradients of the function represented by the deep neural network are bounded (given a norm) and that this bound is known. This robustness certificate also comes with new generalization bounds that critically rely on the Lipschitz constant of the neural network [START_REF] Von | Distance-based classification with lipschitz functions[END_REF][START_REF] Behnam Neyshabur | Exploring generalization in deep learning[END_REF][START_REF] Bartlett | Spectrally-normalized margin bounds for neural networks[END_REF].

The predictors were trained using the Deel-Lip library [START_REF] Serrurier | Achieving robustness in classification using optimal transport with hinge regularization[END_REF]. All the predictors, including the 1-Lipschitz, have comparable accuracy (78 ± 4%). To our knowledge, no previous work has made the link between Lipschitz networks and the chosen explainability methods. The Fig. 5 shows the difference in S = and S = between ResNet and 1-Lipschitz predictors. In the left column, the results come from ResNet-18 predictors trained on Cifar-10 while the right column is dedicated to 1-Lipschitz predictors. We observe a clear improvement of the consistency and generalization of the explanations respectively as a result of better separation of the histograms and a smaller expectation of S = . SmoothGrad is the method that obtains the most consistent explanations as indicated in the table 3, in front of RISE and Saliency (more results in the supplementary material G Fig. 10). Concerning MeGe, the results reported in Table 2 show an improvement in the Generalizability of the explanations for the 1-Lipschitz predictors. Indeed, the Generalizability score has increased compared to the ResNet predictors for all tested methods, except Grad-CAM++. Like MeGe, the results in Table 3 show an improvement for the 1-Lipschitz predictors in the Consistency of the explanations for all the methods tested except for Grad-CAM++, reflecting the more marked separation between the two histograms of S = and S = in Fig. 5.

In general, the experiments carried out allow us to observe a clear improvement in the quality of explanations from the 1-Lipschitz predictor. These encouraging results show that there is a close link between the methods used and predictor architectures, as well as the usefulness of Lipschitz networks for explainability. Furthermore, underlines the fact that the search for new methods is not the only path to explainability: the search for predictors with better explanations is another under-exploited avenue.

Conclusion

We introduced a procedure to derive two new measures to characterize important properties of a good explanation: Generalizability and Consistency. We highlight the fact that Fidelity is intended to ensure that the explanations correctly reflect the underlying strategies of the model, regardless of whether these strategies are general or consistent. Finally, as a case in point, we presented a novel analysis using 1-Lipschitz networks. We used our measures to quantify the consistency of their explanations and showed that this class of networks gives much more stable and trustworthy explanations compared to standard neural networks.

We see the present work as constituting a necessary next step in characterizing good explanations -towards the quest for more explainable ML models.

B. Explanation methods

In the following section, the formulation of the different methods used is given. As a reminder, we focus on a classification model f : R d → R C where C is the number of classes. We assume f c (x) the logit score (before softmax) for class c. An explanation method provides an attribution φ ∈ R d for each input feature from a model and an input of interest. Each value then corresponds to the importance of this feature for the model results.

Saliency Map (SA) is a visualization techniques based on the gradient of a class score relative to the input, indicating in an infinitesimal neighborhood, which pixels must be modified to most affect the score of the class of interest.

Φ SA (x) = ∂f c (x) ∂x Gradient Input (GI) is based on the gradient of a class score relative to the input, element-wise with the input, it was introduced to improve the sharpness of the attribution maps. A theoretical analysis conducted by [START_REF] Ancona | Towards better understanding of gradient-based attribution methods for deep neural networks[END_REF] showed that Gradient Input is equivalent to -LRP and DeepLIFT methods under certain conditions: using a baseline of zero, and with all biases to zero.

Φ GI (x) = x ∂f c (x) ∂x
Integrated Gradients (IG) consists of summing the gradient values along the path from a baseline state to the current value. The baseline is defined by the user and often chosen to be zero. This integral can be approximated with a set of m points at regular intervals between the baseline and the point of interest. In order to approximate from a finite number of steps, we use a Trapezoidal rule and not a left-Riemann summation, which allows for more accurate results and improved performance (see [START_REF] Sotoudeh | Computing linear restrictions of neural networks[END_REF] for a comparison). The final result depends on both the choice of the baseline x 0 and the number of points to estimate the integral. In the context of these experiments, we use zero as the baseline and m = 60.

Φ IG (x) = (x -x 0 ) 1 0 ∂f c (x 0 + α(x -x 0 )) ∂x dα
SmoothGrad (SG) is also a gradient-based explanation method, which, as the name suggests, averages the gradient at several points corresponding to small perturbations (drawn i.i.d from a normal distribution of standard deviation σ) around the point of interest. The smoothing effect induced by the average help reducing the visual noise, and hence improve the explanations. In practice, Smoothgrad is obtained by averaging after sampling m points. In the context of these experiments, we took m = 60 and σ = 0.2 as suggested in the original paper.

Φ SG (x) = E ε ∼ N (0,Iσ 2 ) ∂f c (x + ε) ∂x
Grad-CAM (GC) can be used on Convolutional Neural Network (CNN), it uses the gradient and the feature maps A (k) of the last convolution layer. More precisely, to obtain the localization map for a class, we need to compute the weights α (k) c associated to each of the feature map activation A (k) , with k the number of filters and Z the number of features in each feature map we define α

(k) c = 1 Z i j ∂fc(x) ∂A (k) ij and Φ GC = max(0, k α (k) c A (k) )
Notice that the size of the explanation depends on the size (height, width) of the last feature map, a bilinear interpolation is performed in order to find the same dimensions as the input.

Grad-CAM++ (G+) is an extension of Grad-CAM combining the positive partial derivatives of feature maps of a convolutional layer with a weighted special class score. The weights α (k) c associated to each feature map is computed as follow :

α c k = i j [ ∂ 2 fc(x) (∂A (k) ij ) 2 2 ∂ 2 fc(x) (∂A (k) ij ) 2 + i j A (k) ij ∂ 3 fc(x) (∂A (k) ij ) 3
] RISE (RI) is a black-box method that consist of probing the model with randomly masked versions of the input image to deduce the importance of each pixel using the corresponding outputs. The binary masks m ∼ M are generated in a subspace of the input space, then upsampled with a bilinear interpolation (once upsampled the masks are no longer binary).

For ImageNet the number of masks was m = 4000, for all the other datasets m = 1000.

Φ RI (x) = 1 E(M)N N i=0 f c (x m i )m i

C. Fidelity

Various fidelity metrics have been proposed that essentially measure the correlation between input variables and the drop in score when these variables are set to a baseline state [START_REF] Samek | Evaluating the visualization of what a deep neural network has learned[END_REF][START_REF] Yeh | On the (in)fidelity and sensitivity for explanations[END_REF][START_REF] Rieger | Irof: a low resource evaluation metric for explanation methods[END_REF][START_REF] Vitali Petsiuk | Rise: Randomized input sampling for explanation of black-box models[END_REF]. In this work, we use µF from [START_REF] Bhatt | Evaluating and aggregating feature-based model explanations[END_REF]:

µF = Corr S⊆{1,...,d} |S|=k i∈S Φ(f , x) i , f (x) -f (x [xi= xi,i∈S] ) (7)
Where f is a predictor, Φ an explanation function, S a subset indices of x and x a baseline reference. The choice of a proper baseline is still an active area of research [START_REF] Sturmfels | Visualizing the impact of feature attribution baselines[END_REF].

D. Considered measures for ReCo

As mentioned in when introducing ReCo, one would be tempted to use directly a distance between distributions, we briefly explain why we did not make this choice. In addition, we detail an alternative measure, also based on balanced accuracy, which gives consistent results.

A first intuition to measure the shift between the S = and S = histograms would be to consider the usual measures, such as Kullback-Leibler (KL) divergence.

However, these distances are problematic in that the order of the distributions actually matters more than the distance between them, and these two measures can give a good score even when the explanations are inconsistent Similarly, considering the 1-Wasserstein measure, we could construct an inconsistent case by exploiting the invariance to the direction of transport. For these reasons, we have therefore chosen a [START_REF] Chang | Reading tea leaves: How humans interpret topic models[END_REF] classification measure, based on maximizing balanced accuracy. Nevertheless, one could also (observing similar results) use the area under the curve (AUC) of the balanced accuracy, such as :

ReCo AU C = 1 |S| γ∈S T P R(γ) + T N R(γ) -1

E. Models

As mentioned in the paper, the models used are all (with the exception of 1-Lipschitz networks) ResNet-18, with variations in size and number of filters used. Preserving the increase of filters at each depth by the original factor (x2), we took care to define for each dataset, a base filters value, as the number of filters for the first convolution layer. Another difference concerns the dropout rates used, indeed we had dropout to improve the performance of the tested models. Moreover, it should be remembered that there is no difference in architecture between the normally trained models and the degraded models.

We report here the architecture of the models for each of the datasets: Fashion-MNIST base filters 26, Dropout 0.4 (92%, ±1%) EuroSAT base filters 46, Dropout 0.25 (95%, ±1%) Cifar10 base filters 32, Dropout 0.25 (78%, ±4%) ImageNet ResNet50 (88%, ±3%)

E.1. Lipschitz models

The 1-Lipschitz models use spectral regularization on the Dense and Convolutions layers. The architecture is as described in Table 4.

E.2. Randomization test

For the randomisation of the model weights, we added noise drawn from a normal distribution ε ∼ N (0, 0.5) to each convolution layer, with the intensity of the degradation impacting on the number of parameters affected by this noise.

F. Distances tests F.1. Spatial correlation

The first test concerns the spatial distance between two areas of interest for an explanation. It is desired that the spatial distance between areas of interest be expressed by the distance used. As a results, two different but spatially close explanations should have a low distance. The test consists in generating several masks representing a point of interest, starting from a left corner of an image of size (32 x 32) and moving towards the right corner by interpolating 100 different masks. The distance between the first image and each interpolation is then measured (see Fig. 6). The different distances evaluated pass this sanity check, i.e. a monotonous growth of the distance, image of the spatial distance of the two points of interest.

F.2. Noise test

The second test concerns the progressive addition of noise. It is desired that the progressive addition of noise to an original image will affect the distance between the original noise-free image and the noisy image. Formally, with x the original image, and ε ∼ N (0, Iσ 2 ) an isotropic Gaussian noise, we wish the distance d to show a monotonic positive correlation corr(dist(x, x + ε), ε).

In order to validate this prerogative, a Gaussian noise with a progressive intensity σ is added to an original image, and the distance between each of the noisy images and the original image is measured. For each value of σ the operation is repeated 50 times.

Over the different distances tested, they all pass the sanity test : there is a monotonous positive correlation (as seen in Fig. 7). Although SSIM and 2 have a higher variance.

One will nevertheless note the instability of the Dice score in cases where the areas of interest have a low surface area, as well as a significant computation cost for the Wasserstein distance. For all these reasons, we chose to stay in line with previous work using the absolute value of Spearman rank correlation.

G. Additional results

Metrics

Figure 4 .

 4 Figure 4. Fidelity scores (Equation7) on ImageNet for normally trained ResNet-50 predictors (first point on the left) as well as for progressively randomized predictors and predictors trained with switched labels. Even a strong degradation of the predictor does not impact the Fidelity of the tested methods. Hence, the Fidelity is intended to ensure that the explanations correctly reflect the underlying strategies of the model, regardless of whether these strategies are general or consistent.

Figure 5 .

 5 Figure 5. Lipschitz predictors (right column) on Cifar10. As explained in this paper, a clear separation between the S = and S = histograms is a sign of consistent explanations.

  0.11 0.15 0.15 0.09 0.64 0.49 0.52 1-Lipschitz 0.60 0.90 0.81 0.50 0.67 0.24 0.84 Table3. ReCo scores obtained by 1-Lipschitz models and ResNet-18 models on Cifar10. Higher is better. For almost all methods, the Consistency of explanations increases significantly on 1-Lipschitz models.

2 Figure 6 .

 26 Figure 6. Distances with moving interest point. The first line shows the successive interpolations between the baseline image (left), and the target image (right). The second line shows the evolution of the distance between each interpolation and the baseline image.

2 Figure 7 .

 27 Figure 7. Distances with noisy images. The first line shows original noise-free image (left) and noisy copies computed by increasing σ. The second line shows the distances between each noisy image and the baseline image.

Figure 8 .

 8 Figure 8. Eurosat MeGe and ReCo scores for normally trained models (first point from the left), as well as for progressively randomized models and models trained with switched labels.

Figure 9 .

 9 Figure 9. Fashion-MNIST MeGe and ReCo scores for normally trained models (first point from the left), as well as for progressively randomized models and models trained with switched labels.

Figure 10 .

 10 Figure 10. S = and S = for ResNet (left column) and 1-Lipschitz models (right column) on Cifar10. As explained in this paper, a clear separation between the S = and S = histograms is a sign of consistent explanations.

Table 1 .

 1 Consistency, Generalizability and Fidelity score for ResNet-50 models on ImageNet. Higher is better. The first and second best results are respectively in bold and underlined.

	ImageNet SA	GI	IG	SG	GC	G+	RI
	µF	0.47 0.51 0.55 0.48 0.69 0.49 0.67
	MeGe	0.40 0.50 0.58 0.36 0.34 0.33 0.66
	ReCo	0.20 0.17 0.16 0.02 0.35 0.26 0.59

Table 2 .

 2 MeGe scores obtained by 1-Lipschitz models and ResNet-18 models on Cifar10. Higher is better. For almost all methods, the Generalizability of explanations increases significantly on 1-Lipschitz models.

	MeGe	IG	SG	SA	GI	GC	G+	RI
	ResNet-18 0.58 0.46 0.45 0.55 0.72 0.83 0.57
	1-Lipschitz 0.72 0.60 0.58 0.67 0.75 0.54 0.85

Table 4 .

 4 1-Lipschitz model architecture for Cifar10.

	Conv2D(48)
	PReLU
	AvgPooling2D((2, 2))
	Dropout(0.2)
	Conv2D(96)
	PReLU
	AvgPooling2D((2, 2))
	Dropout(0.2)
	Conv2D(96)
	AvgPooling2D((2, 2))
	Flatten
	Dense

Table 5 .

 5 .31 0.23 0.10 0.91 0.89 0.84 MeGe 0.58 0.46 0.45 0.55 0.72 0.82 0.56 ReCo 0.11 0.15 0.15 0.09 0.64 0.49 0.52 Fidelity, Consistency and Generalizability score for ResNet-18 models on Cifar10. Higher is better. The first and second best results are respectively in bold and underlined.

	IG	SG	SA	GI	GC	G+	RI
	µF 0.11 0Metrics IG SG	SA	GI	GC	G+	RI

MeGe 0.40 0.42 0.41 0.41 0.67 0.67 0.39 ReCo 0.31 0.18 0.18 0.23 0.59 0.64 0.34

Table 6 .

 6 Consistency and Generalizability score for ResNet-18 models on Eurosat. Higher is better. The first and second best results are respectively in bold and underlined.

	Metrics IG	SG	SA	GI	GC	G+	RI
	MeGe	0.90 0.36 0.30 0.90 0.77 0.84 0.52
	ReCo	0.37 0.13 0.10 0.37 0.52 0.32 0.37

Table 7 .

 7 Consistency and Generalizability score for ResNet-18 models on Fashion-MNIST. Higher is better. The first and second best results are respectively in bold and underlined.

* https://www.deel.ai/

Acknowledgement

This work was conducted as part of the DEEL project * . Funding was provided by ANR-3IA Artificial and Natural Intelligence Toulouse Institute (ANR-19-PI3A-0004). Additional funding to TS was provided by ONR (N00014-19-1-2029), NSF (IIS-1912280 and EAR-1925481), DARPA (D19AC00015), NIH/NINDS (R21 NS 112743). The computing hardware was supported in part by NIH Office of the Director grant S10OD025181. We thank Melanie Ducoffe and Mikael Capelle of the DEEL team for insightful comments which have helped improved the manuscript.

A. Method Details

x ← Φ(f i , x) end for end for for all i ∈ {1, . . . , k} do for all (x, y) ∈ V i do for all j ∈ {1, . . . , k | i = j} do // f j was trained on x, f i was not δ

x , φ

x ) if f i (x) = y and f j (x) = y then // both model are correct

} end if end for end for end for Return S = , S =


Pr Bernard Mignotte
email: bernard.mignotte@uvsq.fr

The mitochondrial TOM complex modulates bax-induced apoptosis in Drosophila

Bax is a pro-apoptotic member of the Bcl-2 family proteins involved in the release of apoptogenic factors from mitochondria to the cytosol. Recently, it has been shown both in mammals and yeast that Bax insertion in the mitochondrial outer membrane involves at least two distinct mechanisms, one of which uses the TOM complex. Here, we show that in Drosophila, heterozygous loss of function mutations of Tom22 or Tom70, two receptors of the TOM complex, attenuates bax-induced phenotypes in vivo. These results argue that the TOM complex may be used as a mitochondrial Bax receptor in Drosophila.

Introduction

In mammals, members of the Bcl-2 family play a key role in regulating mitochondrial outer membrane permeabilization during apoptosis (for review: [START_REF] Kroemer | Mitochondrial membrane permeabilization in cell death[END_REF][START_REF] Chipuk | How do BCL-2 proteins induce mitochondrial outer membrane permeabilization?[END_REF]). This permeabilization authorizes the release from the intermembrane space to the cytosol of apoptogenic factors such as cytochrome c. Once in the cytosol, cytochrome c binds Apaf-1, which leads to the recruitment of procaspase-9 in the presence of dATP to form the apoptosome. Other mitochondria released factors can trigger a caspase-independent cell death. This release of apoptogenic factors is mainly controlled by proteins of the Bcl-2 family, which comprises about twenty proteins with either pro-or anti-apoptotic activities. Some of these proteins can integrate into the mitochondrial outer membrane among which the pro-apoptotic protein Bax. In most cases, Bcl-2 family proteins do not constitutively reside at this membrane but are mainly located in the cytosol before cell death induction. Apoptotic signals often target these proteins to the mitochondrial outer membrane.

The TOM (translocase of the outer membrane) complex is constituted by seven subunits and is in charge of importing nuclearly encoded mitochondrial protein precursors in the mitochondria (for review: [START_REF] Bolender | Multiple pathways for sorting mitochondrial precursor proteins[END_REF][START_REF] Neupert | Translocation of proteins into mitochondria[END_REF]). Three components of this complex, Tom70, Tom20 and Tom22 act as receptors and interact via their large cytosolic domains with mitochondrial protein precursors. Tom40, Tom5, Tom6 and Tom7 constitute the channel enabling translocation of mitochondrial preproteins. Another protein, more recently identified, Tom34, facilitates import of some proteins by interacting with their mature portion instead of binding mitochondrial localization sequences [START_REF] Chewawiwat | Characterization of the novel mitochondrial protein import component, Tom34, in mammalian cells[END_REF][START_REF] Mukhopadhyay | Tom34 unlike Tom20 does not interact with the leader sequences of mitochondrial precursor proteins[END_REF]. It is mainly cytosolic but can be found interacting with the TOM complex. Tom34 has been proposed to be a molecular chaperone that keeps preproteins in an unfolded, import-competent state [START_REF] Mukhopadhyay | Tom34 unlike Tom20 does not interact with the leader sequences of mitochondrial precursor proteins[END_REF].

The mechanism by which proteins of the Bcl-2 family are addressed to and inserted in the outer membrane is still unclear but some recent data suggest that at least some of them could be inserted in the outer membrane like mitochondrial protein precursors, using the TOM complex. Tom20 interacts with two lysines on each side of the transmembrane domain of the carboxyterminal region of Bcl-2 [START_REF] Motz | Bcl-2 and porin follow different pathways of TOM-dependent insertion into the mitochondrial outer membrane[END_REF] and Tom70 facilitates Mcl-1 import [START_REF] Chou | An internal EELD domain facilitates mitochondrial targeting of Mcl-1 via a Tom70-dependent pathway[END_REF]. Regarding Bax, mitochondrial import has been shown to necessitate Tom22 [START_REF] Bellot | TOM22, a core component of the mitochondria outer membrane protein translocation pore, is a mitochondrial receptor for the proapoptotic protein Bax[END_REF], which interacts with the first helix of Bax in mammals. Furthermore, it has been shown recently that monomers, but not dimers, of Bax require the presence of both Tom22 and Tom40 to integrate into mitochondria [START_REF] Cartron | Bax inserts into the mitochondrial outer membrane by different mechanisms[END_REF]. Nonetheless, Bax mitochondrial localization can be TOM-independent in mammals, and this involves the Bax carboxyterminal region (helix 9). Furthermore, Bax is able to insert itself Colin et al.

4/14

without the TOM complex [START_REF] Sanjuan Szklarz | Preprotein transport machineries of yeast mitochondrial outer membrane are not required for Bax-induced release of intermembrane space proteins[END_REF] in isolated yeast mitochondria, even though this insertion seems to be facilitated when a functional complex is present [START_REF] Ott | The mitochondrial TOM complex is required for tBid/Bax-induced cytochrome c release[END_REF].

We have previously shown that murine bax is functional in Drosophila and induces an apoptosis associated to mitochondrial events [START_REF] Brun | reaper and bax initiate two different apoptotic pathways affecting mitochondria and antagonized by bcl-2 in Drosophila[END_REF][START_REF] Gaumer | Bcl-2 and Bax mammalian regulators of apoptosis are functional in Drosophila[END_REF]. Here we show that, in Drosophila, Bax is found both in the cytosol and in mitochondria. In order to test whether the TOM complex is involved in Bax mitochondrial insertion in Drosophila, we also tested the consequence of loss of function mutations affecting TOM complex components on bax-induced apoptosis. Our results indicate that the TOM complex is involved in Bax pro-apoptotic abilities, indicating that Bax mitochondrial localization in Drosophila is important for Bax activity and showing that Drosophila provides a good alternative model of moderate genetic complexity to study the mechanism of action of Bcl-2 family proteins.

Material and methods

Fly stocks

Transgenic Drosophila carrying the P[UAS bax] construct (UAS-bax flies) were generated in our group [START_REF] Gaumer | Bcl-2 and Bax mammalian regulators of apoptosis are functional in Drosophila[END_REF]. These flies express a full length wild type murin Bax cDNA. All mutants of the TOM complex were obtained from the Bloomington Stock Center. mge b10 , an EMS-induced allele null mutant, in which the Trp-94 codon (TGG) is converted to a stop codon (TGA), encodes a Mge protein truncated at the beginning of the transmembrane domain and has been described in [START_REF] Vaskova | Genetic analysis of the Drosophila 63F early puff. Characterization of mutations in E63-1 and maggie, a putative Tom22[END_REF]. The Tom7 BG02496 , Tom34 03692 and Tom70 KG06223 alleles are hypomorphic. None of these four Tom mutant flies display a wing phenotype in the absence of bax expression.

Preparation of mitochondrial and cytosolic fractions

UAS-bax flies were crossed to flies expressing the GAL4 protein under the control of regulatory sequences of the daughterless gene (da-GAL4). This driver allowed us to induce ubiquitous expression of the transgene during development of the fly. 150 embryos were collected, dechorionated in undiluted bleach and rinsed off in PBS1x. Embryos were then pelleted and washed once with PBS1x. Pellets were resuspended in 150 l of ice cold CLAMI buffer (250 mM sucrose, 70 mM KCl, 200 g/ml digitonin), homogenized on ice using a Wheaton glass homogenizer (until the suspension became milky) and incubated at 4°C for 5 min. Samples were then centrifuged at 1000g for 5 min, at 4°C. Supernatants (cytosolic fractions) were recovered and placed at -80°C, pellets were resuspended in 50l of IP buffer (30 mM Tris HCl pH 7.5, 150 mM NaCl, 2 mM EDTA, 2 mM EGTA, 0.2% Triton X-100, Colin et al.

5/14 0.3% NP-40, AEBSF SC ) and incubated for 10 min on ice. Samples were then centrifugated at 4°C for 10 min at 10,000g. Supernatants (mitochondrial fractions) were stored at -80°C.

Western-blot

Protein extracts corresponding to either 10 embryos (crude extract and cytoplasmic fractions) or 40 embryos (mitochondrial fractions) were loaded on an INVITROGEN NuPAGE TM 4-12% Bis-Tris Gels and run in MES running buffer (50mM MES, 50mM Tris base, 0.1% SDS, 1 mM EDTA, pH 7.3). Electrotransfer on PVDF membrane (Immobilon-P, MILLIPORE) was performed in transfer buffer (25 mM Bicine, 25 mM Bis-Tris, 1 mM EDTA, pH 7.2). Transfer efficiency was evaluated with Ponceau-S (0.3% Ponceau-S, 1% acetic acid).

Membrane was then saturated in non-fat dry milk for one hour before antibody staining and probed using anti-Bax 5B7 from Santa Cruz (1:500 dilution), anti-tubulin E7 from DSHB (1:1,000 dilution), anti-cytochrome c from Pharmingen (1:500 dilution) and anti F1 ATP synthase from Mitoscience (1:2000 dilution). HRP conjugate anti-mouse or anti-rabbit are from Jackson and used at 1:10,000 dilution. Immunodetection was performed by chemoluminescence (ECL, AMERSHAM). Signals were quantified using the Bio-Rad GS-800 calibrated densitometer and the Quantity One® software.

Phenotypic suppression test and Wilcoxon test

Bax-induced apoptosis in the wing disc, driven by vg-gal4, results in a wing phenotype (notches) [START_REF] Brun | reaper and bax initiate two different apoptotic pathways affecting mitochondria and antagonized by bcl-2 in Drosophila[END_REF][START_REF] Gaumer | Bcl-2 and Bax mammalian regulators of apoptosis are functional in Drosophila[END_REF]. However, expressivity of this phenotype displays variability. Thus, bax-expressing flies can be classified within three categories according to the strength of their wing phenotypes: strong, intermediate or weak [START_REF] Brun | reaper and bax initiate two different apoptotic pathways affecting mitochondria and antagonized by bcl-2 in Drosophila[END_REF]. Phenotypic suppression tests were performed as follow: vg>bax virgin flies were crossed at 18°C to either mutant flies to test or to control wild type flies. Each progeny of these crosses (expressing bax and heterozygous for a loss of function Tom gene) was classified according to the strength of its phenotype within one of the three phenotypic categories. A Wilcoxon test was then used to compare the distribution of phenotypes between the two genotypes (control vs mutant). Difference of distribution is considered significant for a value of  < 10 -3 . A positive Ws value indicates enhancement of the phenotype while a negative Ws shows suppression of the phenotype.

Results

In Drosophila, Bax is both cytosolic and mitochondrial

In mammals, Bax resides in the cytosol of living cells and induction of apoptosis induces its conformational change allowing its targeting to and insertion in the mitochondrial outer membrane. In order to further investigate Bax function in Drosophila, we performed a Similarly to what can be observed in mammalian cells, Bax is found both in the cytosolic and in the mitochondrial fraction (figure 1A). However, a quantification of the signal taking into account the amount of material loaded on the gel shows that 80 to 85% of Bax is found in the cytosolic fraction (figure 1B).

Release of cytochrome c during Drosophila cells apoptosis remains a matter of debate.

In these experiments, about 15% of the cytochrome c is detected in the cytosolic fraction. This amount is not significantly higher than the amount of F1 ATP synthase found in the cytosolic fraction in this type of experiment (figure 1B), which could suggest that the cytochrome c found in the cytosolic fraction is a contamination and thus that cytochrome c may not be released in the cytosol. However, the relatively low percentage of apoptotic cells in embryos at a given time does not allow to determine whether Bax-induced apoptosis of Drosophila cells is associated or not with a permeabilization of the mitochondrial outer membrane.

The TOM complex receptor Maggie is necessary for bax-induced cell death in

Drosophila

Given that Bax displays both cytoplasmic and mitochondrial localization, we decided to test whether the TOM complex is involved in mediating bax-induced cell death in Drosophila.

Hence, we chose loss of function mutants affecting components of the TOM complex to perform phenotypic suppression tests. We used vg-gal4 to drive murine bax expression in the wing imaginal discs. bax expression leads to apoptosis in the wing tissue that is not essential to fly survival, thus adult wings exhibit notches on their wing margins. These bax-expressing flies were crossed either to a control strain or to flies mutant for a TOM complex component.

Progenies of these crosses were classified according to their phenotype (figure 2). Statistical analysis of the difference in distribution between the two progenies were then tested (see materials and methods and [START_REF] Brun | reaper and bax initiate two different apoptotic pathways affecting mitochondria and antagonized by bcl-2 in Drosophila[END_REF]).

As the component of the TOM complex described as Bax receptor in mammals is Tom22, we first tested whether a loss of function allele of its homolog in Drosophila, maggie (mge), could impair bax-induced cell death. For this purpose, we chose the mge b10 null allele that expresses a truncated form of Mge which only possesses the cytosolic portion of the protein [START_REF] Vaskova | Genetic analysis of the Drosophila 63F early puff. Characterization of mutations in E63-1 and maggie, a putative Tom22[END_REF]. Since mge b10 is homozygous lethal, the analysis was performed in mge b10 heterozygous flies. A shift in phenotype distribution from about 60% of strong and about 40% of intermediate phenotypes in the control cross to 30% of strong and about 70% of intermediate phenotypes was observed when using the mge b10 allele (figure 3A). This decrease in strong phenotypes percentage and the enrichment in intermediate ones reflect a global suppression of Colin et al.

7/14

the bax-induced phenotype that is highly significant according to the Wilcoxon test (=6.15x10 -5 ). A mutation of mge impairs bax pro-apoptotic function, showing that Bax necessitates functional Mge to kill cells efficiently. We can thus hypothesize that Bax mitochondrial localization could be mediated by the TOM complex and could use Mge/Tom22 as a receptor.

The TOM complex receptor Tom70 participates in bax pro-apoptotic activity

To further investigate involvement of the TOM complex in bax killing properties in Drosophila, we tested whether a loss of function mutation of the TOM receptor tom70 could impair bax-induced cell death. We used the tom70 KG06223 allele to perform a phenotypic suppression test. In the control cross progeny, about 55% of phenotypes are classified as strong, 40% are intermediate and 5% are weak (figure 3B). In a tom70 KG06223 heterozygous background, less than 40% of phenotypes are strong while the frequency of weaker phenotypes increases from 5% to 16%. This shift toward weaker phenotypes reveals a significant ( = 3.73x10 -4 ) suppression of bax-induced cell death even though this suppression is less efficient than what is observed with a mge b10 allele. This result suggests that Tom70 may participate in Bax mitochondrial localization.

A Tom7 mutation does not affect bax-induced cell death

We also tested the involvement of Tom7, a protein which does not appear to be essential for the TOM complex function as a deletion of tom7 gene in N. crassa does not by itself display any phenotype [START_REF] Sherman | Functions of the small proteins in the TOM complex of Neurospora crasssa[END_REF]. We performed a phenotypic suppression test using the loss of function allele tom7 BG02496 . This mutation did not modulate any significant change in baxinduced phenotype distribution ( = 0.028). Indeed, we observed almost the same distribution between progenies of the control cross (which only expresses bax) and of the cross with the tom7 BG02496 mutant allele (figure 3C). Therefore, altering tom7 does not affect bax-induced cell death.

A tom34 mutation enhances bax-induced apoptosis

The last component of the TOM complex we have tested is Tom34. This accessory protein of the TOM complex normally resides in the cytosol. Nonetheless, Tom34 has been shown to interact with the TOM complex and facilitate translocation of some proteins through physical interaction probably acting as a molecular chaperone [START_REF] Chewawiwat | Characterization of the novel mitochondrial protein import component, Tom34, in mammalian cells[END_REF][START_REF] Mukhopadhyay | Tom34 unlike Tom20 does not interact with the leader sequences of mitochondrial precursor proteins[END_REF]. We used the tom34 03692 loss of function allele to perform a phenotypic suppression test of the bax-induced phenotype. Use of tom34 03692 induces a significant increase ( = 4.99x10 -7 ) from 60 to 90% of strong phenotypes (compared to control cross) and a decrease from 40 to 10% of intermediate phenotypes (figure 3D). This shift in phenotype distribution reflects an enhancement of the bax-induced phenotype Colin et al.

8/14

thus showing an increase of bax-induced cell death. This effect on cell death could be mediated through another cell death regulator as discussed later.

Discussion

Recent studies on mammalian or yeast systems have shown that the TOM complex is at least to some extent involved in Bax mitochondrial localization showing that the TOM complex should regulate bax-killing properties. We have shown here that in Drosophila, Bax is located mainly in the cytosol but also in mitochondria. Therefore, Bax localization in Drosophila has to be regulated and the mechanisms involved in this process could be conserved during evolution.

Using loss of function mutants affecting the TOM complex, we have shown that heterozygous tom22/mge and tom70 mutations suppress bax-induced apoptosis, strongly suggesting a role of the TOM complex in Bax mitochondrial localization and pro-apoptotic activity. However, the low level of mitochondrial Bax in the whole embryo does not allow determining quantitatively if the level of mitochondrial Bax decreases. Thus we can not exclude an indirect effect of these mutations on bax-induced cell death. Nevertheless, heterozygous tom22/mge and tom70 mutations used here do not display any other phenotype, which strongly suggests that the mitochondrial metabolism is not seriously affected in these flies. Furthermore, mutation of tom7 does not affect Bax toxicity, which is in agreement with the observation that Tom7 is not essential for the TOM complex function. More surprisingly, heterozygous loss of function of tom34 enhances Bax toxicity. One possibility is that Tom34 is involved in the mitochondrial targeting of an antiapoptotic component such as Buffy. This effect is therefore probably indirect.

An interesting perspective would be to study the role of the components of the TOM complex during apoptosis induced by Drosophila homologs of Bax. Indeed, Drosophila possesses two members of the Bcl-2 family: Debcl/Drob-1 which is pro-apoptotic, and Buffy which is anti-apoptotic. However, if the mitochondrion and proteins of the Bcl-2 family seem to have an important place in the Drosophila apoptotic cascade, their role(s) remains elusive (for review: [START_REF] Colin | Mitochondria, Bcl-2 family proteins and apoptosomes: of worms, flies and men[END_REF]). As it is known that mitochondrial localization of Debcl/Drob-1 is necessary for its pro-apoptotic function [START_REF] Igaki | Drob-1, a Drosophila member of the Bcl-2/CED-9 family that promotes cell death[END_REF], it would be very helpful to initiate the same kind of study on debcl-induced apoptosis.

Figures legends

  analyze Bax localization in Drosophila cells. Bax localization was studied in flies expressing bax under the control of the ubiquitous Da-Gal4 driver (figure1).

Figure 1 :

 1 Figure 1: Bax subcellular localization. Lysates of Drosophila embryos ubiquitously expressing bax were subjected to subcellular fractionation and analyzed by western blotting. (A) Immunoblotting was used to detect Bax, Tubulin, Cytochrome c and F1 ATPase. (B) Protein amounts were quantified (see materials and methods) and the average signals (+/-SEM) were corrected to match the same number of embryos. Bax is found both in the cytosolic fraction and in the mitochondrial fraction. T : total fraction, C : cytosolic fraction, M : mitochondrial fraction

Figure 2 :

 2 Figure 2: bax-induced phenotypes in the wing. Phenotypes induced by UAS-bax driven by vg-GAL4 are variable and were classified into three categories (B, C and D) according to their strength. (A) wildtype fly wing. (B) Weak phenotypes are characterized by notches scattered along the margin without significant change in the wing size. (C) Intermediate phenotypes correspond to wings that are reduced in size and an important loss of the margin. (D) Individuals with a strong phenotype display partially or completely absent wings. Adapted from [13].

Figure 3 :

 3 Figure 3: Wing phenotype distribution of vg>bax adult flies Drosophila expressing bax under the control of a vg-gal4 driver were crossed either to a control strain or to a mutant for a TOM complex component. Progenies of these crosses were classified according to their phenotype. Since there is no wild type fly, the phenotypes are classified within three categories (strong, intermediate or weak). Ordinate axis represents the percentage of individuals within each category. Control cross progeny is presented in pale-gray and progeny of the cross with mutant of the TOM complex in dark-gray. Genotypes of the mutant flies are: (A) +/vg-gal4, UAS-bax ; mge b10 /+ (B) P[SUPor-P]tom70 KG06223 /vg-gal4, UAS-bax (C) P{GT1]tom7 BG02496 / vg-gal4, UAS-bax (D) P{PZ,ry + }tom34 03692 ry 506 /vg-gal4, UAS-bax. Genotypes of the control flies are vg-gal4, UAS-bax/+. Results of the Wilcoxon tests are presented under each graph.

Acknowledgements

This work was supported in part by grants from the Association pour la Recherche contre le Cancer (#3819) and the Ligue Nationale contre le Cancer (Comité des Yvelines). We Colin et al. 9/14 thank Sébastien Gaumer for his critical reading of the manuscript and Yanick Risler for her technical assistance. Jessie Colin helds a fellowship from the Ministère de la Recherche et de l'Enseignement Supérieur. Jessie Colin and Julie Garibal were supported by the Ecole Pratique des Hautes Etudes.


Hydrogen on Unsupported Ruthenium Sulfide: Thermodesorption and 1H NMR Studies

The interaction of transition metal sul fi des with hydrogen is a complex process. Understanding this process is important due to the very large number of industrial pro cesses dealing with these systems. The main utilization of these catalysts concerns cata lytic hydrotreating which involves hydroge nation reactions and C-S, C-N, and C-0 bond cleavage reactions carried out in the presence of high partial pressures of hydro gen. The complexity of the interaction of hydrogen with metal sulfi des is related to its double role, as reactant and as modifi er of the concentration of active sites. As a matter of fact, the interaction of hydrogen with the surface of the catalyst provokes its partial reduction, leading to coordinatively unsatu rated metal ions which are directly involved in the adsorptive and catalytic properties (1).

As molybdenum sulfi de is the essential component of industrial hydrotreating cata lysts, its interaction with hydrogen has been the subject of many investigations. The re view of Moyes (2) and the recent paper of Hall and Komatsu (3) have shown that the amount of adsorbed hydrogen changes dras tically from one study to another. In order to compare easily this large number of litera ture data the amount of adsorbed hydrogen is often expressed using the nomenclature HxMoS 2

• Using the same preparation proce dure (decomposition of ammonium tetrathi omolybdate) the values of x vary between 0.012 and 0.37. This important variation of

x has probably to be ascribed to different catalysts pretreatment prior to adsorption measurements, leading to different number of vacancies. This hypothesis has been demonstrated by Jalowiecki et al. (4), who have reported that the activity and the hy drogen adsorption capacity of a molybde num-based catalyst is greatly dependent on the sulfur-to-molybdenum ratio. Concern ing the chemical nature of the adsorbed species, inelastic neutron scattering studies have evidenced that dissociated hydrogen resides on sulfur anions in the form of SH groups (5,6). This result has been recently confi rmed by solid NMR spectroscopy (3). Although these techniques have never shown the presence of adsorbed hydride type hydrogen, its presence on molybdenum or rhenium sulfides has been assumed for explaining the mechanisms of thiophene or hydrothiophene hydrodesulfurization and alkenes hydrogenation (7,8).

The relation between the sulfur-to-metal ratio and the catalytic activity was also stud ied for other sulfi des. We have recently re ported that the H2-D2 exchange activity of a model ruthenium sulfi de is drastically en hanced by sulfur removal (9). As ruthenium sulfi de is one of the most active chalcogen ide (13 times more active than molybdenum sulfi de in the hydrodesulfurization of thio phene and 10 times in the hydrogenation of biphenyl), its activity could be related to a large adsorption capacity or to the presence of adsorbed hydrogen species different from those already detected on lamellar sulfi des. To tackle this fundamental problem of hy drogen adsorption on ruthenium sulfi de, we have taken the fully sulfi ded state as starting point and evaluated the infl uence of progres sive desulfurization on the amount of ad sorbed hydrogen. The nature of the ad sorbed species was investigated using thermodesorption and 1H NMR.

Ruthenium sulfi de was prepared by pre- lated to H1 species follows a completely dif ferent trend, i.e., its amount increases with desulfurization, reaches a maximum for a = 0.3-0.4, and then decreases above.

1
This different behavior of H1 and H 2 to ward sulfur removal suggests that different adsorbing sites are required for hydrogen chemisorption. As the concentration of H 2 decreases when the desulfurization of the solid increases, this type of hydrogen does not seem to be adsorbed on anionic vacanc ies but in interaction with sulfur anions as SH groups. This species is probably the one observed in Heise 's INS work on ruthenium sulfide (10). By contrast, the amount of H1 hydrogen increases with desulfurization up to a-0.4. At this stage of the reduction, the surface of the catalyst contains a very low concentration of sulfur anions. Conse quently, these experimental data suggest that H1 is adsorbed on coordinatively unsat urated ruthenium ions. The comparison of these adsorption results with those obtained previously for the Hi-D 2 exchange reaction (also reported in Fig. 2) and the 1-butene hydrogenation reaction (9, 11) evidences a striking similarity in their variations with a. This strongly favours the hypothesis that H1 is the active species responsible for the hydrogenation reaction. For a > 0.5, the diminution of H1 adsorption can be ascribed to surface area collapsing and to the forma tion of a surface-poisoned ruthenium metal-lie phase inactive for hydrogen activation, as already proposed for the exchange reac tion (9,[START_REF] Lu | [END_REF](13)(14).

To obtain more information about the chemical nature of H1 and H 2 two samples were studies by solid 1H NMR spectros copy. Sample 1 was poorly desulfurized and presented both kinds of adsorbed hydrogen (a�0.15), whereas sample 2 contained mostly type-1 hydrogen (a = 0.35). Both magic-angle-spinning (MAS) spectra were recorded in sealed tubes on a Bruker MSL apparatus at 400 MHz at room temperature. A weak rotation speed (1 kHz) was enough to observe the signals (Fig. 3), the small widths of which characterize species mov ing during the time characteristic of the ex periment 00-9 sec).

Figure 3 shows the results of the 1H NMR. For both solids, the chemical shift relative to tetramethylsilane could be deduced with fair accuracy. Sample 1 exhibits two signals at -7.4 ppm and 5 .1 ppm. The value of the later is consistent with an acidic character of the adsorbed species, in agreement with the hypothesis of the formation of superfi cial SH groups (15). The other signal, -7.4 ppm (upfi eld), can be most likely ascribed to a hydride-type species (16)[START_REF] Holm | NMR of paramagnetic molecules. Principles and applica tions[END_REF][START_REF] De Menorval | Growth and Properties of Metal Clusters[END_REF][START_REF] De Menorval | [END_REF](20).

To sum up, the characterization of hydro gen adsorbed on partially desulfurized ru thenium sulfi de has evidenced two different species: one leading to the formation of SH groups and the other with hydridic character being chemisorbed on coordinatively unsat urated ruthenium cations. The latter, related to the hydrogenating properties of the cata lyst, probably explains the high activity of ruthenium sulfi de.

OFm. 2 .

 2 IJ..=: .... .. .+ =-• •� �-->:t::;:= �'!--����---"' >.iAmount of adsorbed H1 and H2 and Hi-D2 exchange activity as a function of the degree of reduc tion of the catalyst.

ppm - 7 . 4 Fm. 3 .

 743 Solid 1H NMR of adsorbed hydrogen. The degree of reduction is 0.15 for sample 1 and 0.35 for sample 2.