Adulam Jeyasothy

Thibault Laugel

Marie-Jeanne Lesot

Christophe Marsala

Marcin Detyniecki

A General Framework for Personalising Post Hoc Explanations through User Knowledge Integration

Keywords: eXplainable Artificial Intelligence, XAI, user knowledge, compatibility, counterfactual explanation, surrogate model explanations, local feature importance

The field of XAI aims at providing explanations about the behavior of AI methods to a user. In particular, local post-hoc interpretability approaches aim at generating explanations for a particular prediction of a trained machine learning model. It is generally recognized that such explanations should be adapted to each user: integrating user knowledge and taking into account the user specificity allows to provide personalized explanations and to improve the explanation understandability. Yet these elements appear to be rarely taken into account, and only in specific configurations. In this paper, we propose a general framework to allow this integration of user knowledge in post-hoc interpretability methods, relying on the addition of a compatibility term in the cost function. We instantiate the proposed formalization in two scenarios, varying in the explanation form they propose, in the case where the available user knowledge provides information about the data features. As a result, two new explainability methods are proposed, respectively named Knowledge Integration in Counterfactual Explanation (KICE) and Knowledge Integration in Surrogate Model (KISM). These methods are experimentally studied on several benchmark data sets to characterize the explanations they generate as compared to reference methods.

Introduction

The democratization of Machine Learning (ML) applications has led to a raise in the awareness of the potential issues these systems may have. The eXplainable Artificial Intelligence (XAI) domain [START_REF] Burkart | A Survey on the Explainability of Supervised Machine Learning[END_REF][START_REF] Linardatos | Explainable AI: A Review of Machine Learning Interpretability Methods[END_REF][START_REF] Molnar | Interpretable machine learning, A Guide for Making Black Box Models Explainable[END_REF]] focuses on answering questions a user may ask when faced with the result of such a ML model that constitutes a black box. Such questions can be expressed as "Why do I get this prediction?" or "What do I need to do to get the prediction I want?". Among the large variety of approaches proposed in this domain, post-hoc methods focus on explaining the predictions obtained from a given pretrained classifier. Different formats of explanations can be proposed, such as feature importance (e.g LIME [START_REF] Ribeiro | Why should I trust you?" Explaining the predictions of any classifier[END_REF] and SHAP [START_REF] Lundberg | A unified approach to interpreting model predictions[END_REF]) or counterfactual examples [START_REF] Wachter | Counterfactual Explanations without Opening the Black Box: Automated Decisions and the GDPR[END_REF] (e.g. Growing Spheres [START_REF] Laugel | Comparison-based Inverse Classification for Interpretability in Machine Learning[END_REF] and FACE [START_REF] Poyiadzi | FACE: Feasible and Actionable Counterfactual Explanations[END_REF]). Explanation methods also differ depending on the hypotheses they make regarding the information considered available: model agnostic and data agnostic approaches consider that no knowledge is available, neither on the ML model (e.g. on the form of the decision boundary), nor on data (e.g. on their distribution). This absence of knowledge has been shown to lead to numerous issues, such as their risk of generating meaningless explanations [START_REF] Laugel | The dangers of post-hoc interpretability: Unjustified counterfactual explanations[END_REF] or their lack of connection to the real-world actions required to change features [START_REF] Barocas | The hidden assumptions behind counterfactual explanations and principal reasons[END_REF]. As a result, they may then not be always understood by the user [START_REF] Rudin | Stop Explaining Black Box Machine Learning Models for High Stakes decisions and Use Interpretable Models Instead[END_REF]. To tackle these issues, as detailed in Section 2, some methods propose to enrich the considered input and integrate the user in the loop by taking into account user knowledge [START_REF] Mahajan | Preserving Causal Constraints in Counterfactual Explanations for Machine Learning Classifiers[END_REF][START_REF] Frye | Asymmetric Shapley values: incorporating causal knowledge into model-agnostic explainability[END_REF][START_REF] Ustun | Actionable Recourse in Linear Classification[END_REF], so the explanation is more understandable.

Focusing on the same issue, this paper takes over the general formalization we proposed in previous work [START_REF] Jeyasothy | Integrating prior knowledge in post-hoc explanations[END_REF]. It relies on the definition of a generic cost function that integrates the aforementioned user knowledge in the search for an explanation, independently of the explanation form and the knowledge representation. It consists in adding, to the traditional explanation-generation-optimization objective, a complementary compatibility term, taking as parameter the considered user knowledge. This general framework can be used to augment traditional post-hoc explanation methods of different types, e.g. counterfactual and surrogate model-based approaches, and make them compatible, or complementary depending on the considered context and objective, with knowledge of various forms.

The paper provides a discussion about the need for such a framework, as well as two instantiations of the framework, for different types of explanations, in a model and data agnostic paradigm. First, we focus on the case of counterfactual explanations that favor actions along the features that are known by the user. Then, we consider the case of surrogate models to reach the same objective. To solve these two instantiations, two new explainability approaches are proposed, respectively named Knowledge Integration in Counterfactual Explanation (KICE) and Knowledge Integration in Surrogate Model (KISM), and their relevance is studied empirically. Finally, the paper proposes an exploratory discussion for the case of user knowledge expressed in a different, enriched, form: beyond the set of features the user indicates he/she understands, it considers user knowledge represented in the form of a rule-based system. It discusses the interest and challenges of using it to augment counterfactual explanations.

The paper is structured as follows: Section 2 details the background of XAI and existing dedicated approaches that take into account specific user knowledge forms for generating specific explanation forms. Section 3 describes the general framework we propose for integrating user knowledge into interpretability methods. Its instantiations to the case of feature based knowledge, for explanations in the form of counterfactual examples and surrogate models, are respectively presented in sections 4 and 5. Section 6 describes the experimental study conducted on several benchmarks to characterize the explanations generated by these approaches, as compared to reference methods. Perspectives for the case where the user knowledge is expressed as a rule-based system are discussed in Section 7. Finally Section 8 concludes the paper.

Background

This section provides background information to which the paper refers: it first describes some post-hoc explanation methods; then, it discusses different possible forms the user knowledge can take, before presenting some existing approaches that integrate knowledge in explanation generation.

Post-hoc Explanations

There exist numerous post-hoc explanation methods, i.e. methods that take as input a trained classifier considered as a black box and that can only request it to label new data points (see e.g. [START_REF] Guidotti | A Survey of Methods for Explaining Black Box Models[END_REF] for a survey). Many of them rely on generating explanations as a solution to an optimization problem, for a well-defined cost function. Among those, we are particularly interested in counterfactual and surrogate approaches.

The first ones [START_REF] Verma | Counterfactual explanations for machine learning[END_REF][START_REF] Stepin | A survey of contrastive and counterfactual explanation generation methods for explainable artificial intelligence[END_REF][START_REF] Guidotti | Counterfactual explanations and how to find them: literature review and benchmarking[END_REF] aim at answering the question: "What do I have to change to get the prediction I want?", generating so-called contrastive explanations that oppose the predicted output to the desired one [START_REF] Miller | Explanation in artificial intelligence: Insights from the social sciences[END_REF]. They provide local explanations, i.e. explanations that depend on the instance the user asks a question about. The explanation is defined as the data point that is both predicted to belong to the desired class and as close as possible to the instance of interest: the difference between this generated data point and the reference instance indicates the required changes and as such constitutes the explanation. From the first propositions [START_REF] Wachter | Counterfactual Explanations without Opening the Black Box: Automated Decisions and the GDPR[END_REF][START_REF] Lash | Generalized Inverse Classification[END_REF], numerous variants have been proposed to take into account additional requirements, such as reducing the number of features to modify [START_REF] Laugel | Comparison-based Inverse Classification for Interpretability in Machine Learning[END_REF], improving the feasibility of the changes [START_REF] Poyiadzi | FACE: Feasible and Actionable Counterfactual Explanations[END_REF] or their realism [START_REF] Mothilal | Explaining Machine Learning Classifiers through Diverse Counterfactual Explanations[END_REF] to name a few (see e.g. [START_REF] Verma | Counterfactual explanations for machine learning[END_REF][START_REF] Stepin | A survey of contrastive and counterfactual explanation generation methods for explainable artificial intelligence[END_REF][START_REF] Guidotti | Counterfactual explanations and how to find them: literature review and benchmarking[END_REF] for recent surveys). Most approaches consist in optimizing, under constraint, a cost function that expresses the desired characteristics of the counterfactual example. The basic form of this cost function is defined as the distance between the considered instance and the explanation, e.g. the Euclidean distance, possibly combined with the l 0 norm [START_REF] Laugel | Comparison-based Inverse Classification for Interpretability in Machine Learning[END_REF][START_REF] Lash | Generalized Inverse Classification[END_REF] under the constraint its prediction equals the desired class.

On the other hand, surrogate-based explanations rather answer the question: "Why do I get this prediction?": they aim at capturing locally the behavior of the considered machine learning model, representing it by a local approximation performed by a simple interpretable model, such as decision trees, linear regressions or classification rules [START_REF] Ribeiro | Why should I trust you?" Explaining the predictions of any classifier[END_REF][START_REF] Peltola | Local Interpretable Model-agnostic Explanations of Bayesian Predictive Models via Kullback-Leibler Projections[END_REF][START_REF] Sokol | bLIMEy: Surrogate Prediction Explanations Beyond LIME[END_REF], that mimics its behavior. One popular surrogate model approach is LIME [START_REF] Ribeiro | Why should I trust you?" Explaining the predictions of any classifier[END_REF] that relies on a local linear approximation of the studied classifier: it optimizes the coefficients of a linear regression model and the number of non-zero coefficients to maximize the fidelity to the black box classifier and outputs the features with the highest absolute values of the coefficients. LIME bridges the gap to local feature importance explanation approaches that aim at extracting the locally most influential features.

Knowledge Representation

As discussed in the introduction, post-hoc explanations, as well as any explanation building method, can be enriched by the integration of available user knowledge, so as to improve their quality and intelligibility, in a human-in-the-loop paradigm. A preliminary question, before its integration in the explanation methods, concerns the formalism used to express this user knowledge. Many have been considered, with numerous variants in each case, one can cite among others: class prototypes [START_REF] Van Looveren | Interpretable Counterfactual Explanations Guided by Prototypes[END_REF] (representative instances of classes that satisfy the characteristics necessary to belong to a class), distribution of data [START_REF] Poyiadzi | FACE: Feasible and Actionable Counterfactual Explanations[END_REF] (real data set in the studied context) or feature-based knowledge. In the latter case, detailed below as the case on which this paper focuses, the user provides personal information according to which all features are not equivalent: some of them may be more important, or related to others, which should be exploited to formulate the explanations.

A first type of such feature-based user knowledge provides information on the individual features: the user can for instance indicate the so-called actionable features [START_REF] Ustun | Actionable Recourse in Linear Classification[END_REF], i.e. the features that can be modified, as opposed to the ones that must remain unchanged. This information may be in particular crucial in the case of explanation expressed as counterfactual examples, as a way to express preferences regarding the features that need to be changed to be predicted in a different class. For instance in a credit application scenario, a user may indicate that the budget is not actionable, whereas the duration of the credit reimbursement is. At a more refined level, knowledge may also indicate that some modifications can only apply in a given direction: common knowledge for instance indicates that age cannot be decreased [START_REF] Lash | Generalized Inverse Classification[END_REF]. The user can also give the importance of each feature. The knowledge is then a feature importance vector. This type of knowledge gives an idea of the impact of the features on the prediction according to the user. This information can be used in the case of feature importance explanations [START_REF] Zhao | Baylime: Bayesian local interpretable model-agnostic explanations[END_REF].

A second type of feature knowledge provides information on their interactions. It may be expressed through their covariation [START_REF] Mahajan | Preserving Causal Constraints in Counterfactual Explanations for Machine Learning Classifiers[END_REF][START_REF] Drescher | Toward rigorous use of expert knowledge in ecological research[END_REF] (e.g. "An increase in education level leads to an increase in age") or, more generally and more formally, through a causal graph that expresses functional dependencies between the feature values [START_REF] Mahajan | Preserving Causal Constraints in Counterfactual Explanations for Machine Learning Classifiers[END_REF][START_REF] Frye | Asymmetric Shapley values: incorporating causal knowledge into model-agnostic explainability[END_REF].

It can be underlined that knowledge can have different sources and it can be useful to distinguish between the notions of prior, expert or user knowledge: all three can be represented in the same formalism, but may have different impacts on the explanation. Prior knowledge can be considered as common knowledge, normally shared between various users and that needs to be taken into account generally. On the other hand, expert and user knowledge are more specific and are the ones that should lead to personalized explanations. The difference between these two types is that all users are not experts and that their own knowledge may be sometimes unreliable or imposing strong constraints that may endanger the existence of explanations satisfying them all. For instance a user may have too few actionable features for counterfactual examples to exist. This case of user knowledge is however the one this paper focuses on.

Integration of Knowledge

Beside the question of the form the available knowledge can take, the question of its exploitation and integration in the explanation must be considered. Several approaches have been proposed, to the best of our knowledge they are dedicated to specific combinations of the knowledge type and the explanation form they consider.

In the case where knowledge indicates actions which are possible or not and explanations take the form of counterfactual examples, Ustun et al. [START_REF] Ustun | Actionable Recourse in Linear Classification[END_REF] propose to generate actionable explanations. The set of actions is defined as a set of features where an action is feasible, i.e when values can be modified. This allows to avoid proposing inapplicable explanation such as "In order to get the credit, you need to decrease your age". To the classic choice of a distance-based cost function (see Section 3.2.1), Ustun et al. add a densitybased criterion which is the Cumulative Distribution Function. One of the limits of this approach is that it can only applied in the case where the classifier or regressor to explain is a linear models. The KICE method we propose in this paper (see Section 4) relies on the same form of explanation and user knowledge as Ustun et al. but it considers a model agnostic context.

For image data, Zhao et al. [START_REF] Zhao | Baylime: Bayesian local interpretable model-agnostic explanations[END_REF] consider the integration of feature weight knowledge in feature importance explanations: in this case the considered knowledge and explanation types are identical, the final explanation is a compromise between the two feature score vectors.

Considering the case of counterfactual explanations, Mahajan et al. [START_REF] Mahajan | Preserving Causal Constraints in Counterfactual Explanations for Machine Learning Classifiers[END_REF] focus on the case where knowledge is expressed as causal interactions between features. They propose to integrate the latter in the definition of a distance measure that quantifies the extent to which a candidate explanation satisfies the causal relationships. Unlike Ustun et al. [START_REF] Ustun | Actionable Recourse in Linear Classification[END_REF], they do not exclude candidates that do not satisfy these constraints, but penalize them, in a more flexible approach. This method makes it possible to avoid unrealistic explanations of the form "In order to get the credit, you need to increase your budget and to decrease your salary".

Frye et al. [START_REF] Frye | Asymmetric Shapley values: incorporating causal knowledge into model-agnostic explainability[END_REF] are also interested in the integration of causal relationships, in the case of explanations in the form of local feature importance vectors. They focus on the asymmetry brought by the causal link orientation and propose to exploit it by deriving weights to define personalized variants of the cost function.

These four approaches integrate different types of user knowledge into different forms of explanation. They consider the available knowledge as an additional constraint in the explanation generation. This constraint is represented in different forms: reduction of the search space, addition of penalty in the cost function or weight in the cost function. In the following section, we propose a general formalization for such methods integrating knowledge.

Proposed General Framework

This section describes the general framework for building explanations that takes into account user knowledge, independently of the type of explanation and the type of knowledge. After describing the notations used throughout the paper, the proposed model is described and then its generality is illustrated showing how it can be instantiated to represent a reference existing approach.

Notations

The considered objective is to explain the prediction of a machine learning model denoted f : X -→ Y where X and Y respectively denote the input and the output spaces. We consider a domain whose elements are described by d numerical features, i.e. X ⊆ R d , equipped with the Euclidean distance. The case of categorical features, not explicitly studied in this paper, requires to modify this distance. Focusing on the common case of explanation for classification tasks, we consider Y to be a discrete set, e.g. {0, 1} for binary classification or {c 1 , . . . , c k } for k classes. In the considered model and data agnostic paradigm, no further information is available about f nor about X except for the user knowledge. The latter is denoted E in its most general form, it will be instantiated in the following sections, Sections 4, 5 and 7.

E denotes the set of candidate explanations, independently of the their type: it may contain counterfactual examples or surrogate models. e ∈ E denotes a candidate explanation.

In the considered local explanation case, the explanation is required for a reference instance x ∈ X whose prediction f (x) needs to be explained. In the following x may be used as subscript of the elements depending on it.

Proposed Optimization Problem

Following the principle of most post-hoc explanation methods, briefly reminded in Section 2.1, the generic framework we proposed is formulated as an optimization problem of an enriched cost function: the latter adds, to the classical penalty term that assesses the quality of a candidate explanation with respect to the considered instance x and classifier f , a term that depends on the user knowledge E. It is formally defined as

argmin e∈E agg(penalty x (e, f ), incompatibility x (e, E)) (1)
where penalty x , incompatibility x and agg are three functions described in turn in the following. As discussed below, the exact expression of these three functions depends on the studied context: the motivations of the user, the type of explanation to be generated or the considered type of knowledge.

Penalty Function

As recalled in Section 2.1, most existing approaches to generate explanations minimize a cost function denoted here penalty x : it takes as argument a candidate explanation e and the classifier to be explained f , and it may depend on the studied instance x in the case of local explainers. The lower the penalty, the better the candidate explanation e.

In the case of counterfactual examples, penalty can be defined as the distance between the candidate and the considered instance, d(x, e), to which an arbitrary high value Z is added if the class predicted for e is the same1 as the one for x, i.e. f (e) = f (x): penalty x (e, f ) = d(x, e) + 1 f (e)=f (x) × Z. As a consequence, only candidates with a predicted class different from the one of x are considered. This is equivalent to restricting the set of explanations E to a local definition E x = {e ∈ E/f (e) ̸ = f (x)}. Regarding the distance, usual choices include the Euclidean distance [START_REF] Lash | Generalized Inverse Classification[END_REF], or the l 0 norm to incorporate a sparsity constraint [START_REF] Laugel | Comparison-based Inverse Classification for Interpretability in Machine Learning[END_REF] to name a few.

In the case of surrogate explainers, penalty can be defined as the local fidelity of the surrogate e to the classifier f in the neighborhood of the studied instance x.

Incompatibility Function

In order to integrate the user knowledge, we propose to add a function that measures the extent to which the candidate explanation is incompatible with this knowledge, so as to favor the explanations that are in agreement with the latter.

Indeed, in the case of non-expert users, the aim is to increase the intelligibility of the explanation, relying on a common language. For instance in a credit application scenario, an explanation for a non-expert customer asking how to get his/her credit application accepted needs to focus on common knowledge actions such as increasing his/her salary. On the other hand, in the case where the explanation is generated for a domain expert, the incompatibility can represent the level of fidelity of the explanation with respect to the expert's comprehension of the domain. The lower the incompatibility, the more fidel the explanation.

Detailed instantiations are discussed in the next sections, we illustrate here the principle with a basic case. For a given candidate explanation e, we denote A e the set of features is involves, independently of its form. For counterfactual example, it can for example contain the modified features, for local feature importance, it can be defined as the set of features with non-zero scores. Similarly, we denote A E the features involved in the user knowledge, independently of its form. An incompatibility function then aims at measuring the discrepancy between A e and A E . Decreasing incompatibility can then mean minimizing the number of features the explanation takes into account that are not part of the user knowledge:

incompatibility x (e, E) = Card(A e \ A E ) (2)
Note on a Different Semantics for Knowledge Integration. The above discussion considers that the explanation must be compatible with the user knowledge, which appears as a relevant choice that may contribute to increase the user's confidence in the machine learning models and his/her willingness to use it. However, we wish to discuss here an alternative, and actually contradictory, semantics for knowledge integration, that makes sense in a different scenario, for a different type of users with different objectives. Indeed, in some cases, a user may be interested in an explanation that is actually orthogonal to his/her knowledge: he/she may wish to get a complementary explanation, likely to provide him/her new information he/she did not dispose of. In other words, his/her aim in such a case is to acquire new, enriching, pieces of knowledge that should not be redundant with what he/she already knows. The desired explanation should then be not redundant with the knowledge he/she expresses, which is at cross-purposes of the knowledge integration principle discussed above.

In such a case, considering the example and notations given above, expressing that the explanation needs to be complementary to the knowledge, i.e. minimizing the redundancy, may for instance advocate for a function such as incompatibility x (e, E) = Card(A e ∩ A E )

Aggregation Function

The aggregation function combines the penalty and incompatibility functions. There exists an abundant literature on aggregation operators (see e.g. [START_REF]Aggregation Operators: New Trends and Applications[END_REF][START_REF] Grabisch | Aggregation Functions[END_REF]) which can be divided into several categories depending on their behavior and semantics. For illustration purposes, we focus on three of them, called conjunctive, disjunctive or compromise operators. A conjunctive operator, for example the minimum function, returns a high value only if all criteria to be aggregated, here both penalty and incompatibility, are high. A disjunctive operator, such as the maximum function, requires only at least one value to be high. Trade-off operators such as the weighted average allow for the compensation of low values by high values. Other categories include operators offering a so-called reinforcement behavior, according to which if all criteria take high values, they result in an even higher value.

The choice of the aggregation function can be made according to the user preferences, instead of necessarily considering the same function for all of them: one way to personalize the explanation to a user is to let him/her choose the aggregation he/she wants to perform.

Illustration of the Proposed General Framework

In order to illustrate the generality of the proposed framework, we show in this section how it can be instantiated to express the state-of-the-art approach proposed by Ustun et al. [START_REF] Ustun | Actionable Recourse in Linear Classification[END_REF], highlighting the definition of the three involved functions.

In Ustun et al. [START_REF] Ustun | Actionable Recourse in Linear Classification[END_REF] approach, the user knowledge E is local and depends on the instance of interest: it is denoted A(x) and defined as the set of allowed modifications, that can be applied to instance x. Its integration makes it possible to generate an actionable counterfactual explanation. The latter is defined as the solution to the following optimization problem [START_REF] Ustun | Actionable Recourse in Linear Classification[END_REF]:

η * = argmin η∈A(x) cost f ct(η, x) with f (x + η) ̸ = f (x)
cost f ct is defined as the distance between x and x + η. The generated explanation η * expresses the move between the instance of interest x and the closest instance in the opposite class x + η * .

We propose to rewrite this optimization problem as:

e * = argmin e∈Ex cost f ct(e -x, x) + 1 |x-e| / ∈A(x) × Z where E x = {x ′ ∈ X | f (x ′ ) ̸ = f (x)
} and Z is an arbitrarily large real number. Here e * is the closest instance in the opposite class, and it enables to retrieve η * as η * = e * -x. This expression, equivalent to the previous one, makes it possible to identify the penalty, incompatibility and agg functions, respectively defined as, for an arbitrary high value

Z ′ penalty x (e, f ) = cost f ct(e -x, x) + 1 f (e)=f (x) × Z ′ incompatibility = 1 |x-e| / ∈A(x) × Z agg(u, v) = u + v
Indeed, the penalty function equals cost f ct defined in [START_REF] Ustun | Actionable Recourse in Linear Classification[END_REF] with the introduction of a arbitrary high value Z ′ for candidates that are predicted to be in the same class as x as discussed in Section 3.2.1. The latter expresses the restriction of the search space E to E x . The incompatibility function represents the presence or absence of the modified feature in the user knowledge; it takes only two values 0 or Z. An incompatible counterfactual explanation thus has a very high cost function, resulting in only compatible counterfactuals being considered. Finally, aggregation is performed by a sum. However, as the incompatibility function is binary, only compatible counterfactuals are considered, so the resulting counterfactual is both compatible and of good quality. In this setting, the aggregation function has a conjunctive behavior.

Knowledge Integration in Counterfactual Explanation (KICE)

This section introduces a first instantiation of the general framework described in the previous section, leading to a new explanation method taking into account user knowledge in the case of counterfactual example explanations. It first describes the characteristics and motivations for the considered type of user knowledge, relevant both for counterfactual explanations and surrogate models considered in Section 5. It then describes in turn the proposed cost function and the algorithm optimizing it.

Characteristics of the Knowledge Type and Motivations

The knowledge we propose to consider takes the simple form of a set of features, E = {X i , i = 1 . . . m} with m ≤ d where d is the total number of features: within the framework discussed in Section 2.2, it corresponds to a case of individual feature-based user knowledge. It can be seen as similar to the case of actionable features, but its meaning differs: the information provided by the user does not provide indication about his/her ability to modify the associated values, but refers to a more basic level. As discussed in Section 3.2.2, two interpretations can be distinguished depending on whether the user is an expert or not, as illustrated in the following.

We consider the following fictitious example regarding a classification of vegetables: let us consider a sample predicted to be a carrot. We are interested in two candidate explanations e 1 and e 2 which explain why the model predicts that the sample is a carrot. The first explanation e 1 is: "The color of the sample is orange and the taste is sweet". The second explanation e 2 is: "The sample has a high rate of provitamine A and a high rate of saccharose".

We consider two users: a non-expert and an expert of the domain who express the same knowledge: E = {color, f lavor}. Even if the knowledge is the same, it represents two different semantics. For the non-expert user, this knowledge represents the characteristics that he knows about and that he agrees to find in an explanation in order to better understand it. A nonexpert user wants an explanation in his language so that it is understandable. He then prefers explanation e 1 because it contains only features presented in his knowledge. For the expert user, this knowledge represents the features whose impacts on the prediction are already known by him. He then wants additional information and an explanation that is not redundant with his knowledge. Thus, he wants a complementary explanation that does not contain the elements he already knows. He then prefers explanation e 2 because it has features that he has not expressed. In the following, we focus on the case of non-expert users who expect explanations in their language.

Proposed Cost Function: Instantiation of the Framework

This section describes the instantiation of the general cost function given in Eq. ( 1) for the case of explanation in the form of counterfactual examples

E x = {x ′ ∈ X | f (x ′ ) ̸ = f (x)},
user knowledge as a set of features and an explanation expressed in the user language. It describes in turn the penalty and incompatibility functions and the aggregation operator.

Penalty Function. For the penalty function, we propose to use the classical counterfactual function which is the Euclidean square distance:

penalty x (e, f ) =∥ x -e ∥ 2 (3)
with the restriction of the search space E to E x to guarantee the prediction associated with the counterfactual example differs from the one of the reference instance.

Incompatibility Function. As specified above, the objective is to propose a counterfactual explanation in agreement with the user knowledge, meaning that ideally, the counterfactual modifications must be only performed according to features appearing in E. However, for some instances, focusing only on a subset features raises the risk of never being able to meet the decision boundary of f , leading to no counterfactual explanation being generated. Therefore, we propose to relax this constraint by penalizing the modifications according to the features from E, i.e. features that are not present in the knowledge E. This allows, when the boundary cannot be found along the sole features of E, to make sure that a solution can still be found. Thus, we propose the incompatibility function that computes the Euclidean square distance only using absent features:

incompatibility x (e, E) =∥ x -e ∥ 2 E = i / ∈E (x i -e i ) 2 (4)
Minimizing this incompatibility avoids generating counterfactual explanations that greatly modify the unknown features.

Aggregation Function. In order to combine the penalty function with the incompatibility function, we propose to use a compromise operator, more precisely a weighted average:

agg(u, v) = u + λv (5)
where λ ∈ R + is a user-set hyper-parameter. The higher λ, the more intransigent the user is with respect to to his/her knowledge, i.e. the less open he/she is to features he/she may not understand. A high value of λ implies that the counterfactual example may be located further away from the reference instance, meaning the user may need to perform more modifications. On the other hand, these modifications only apply to features he/she understands the meaning of.

Global Function. We then obtain the following optimization problem:

Denoting E x = {x ′ ∈ X | f (x ′ ) ̸ = f (x)} and λ ∈ R + , e * = argmin e∈Ex cost x,E (e) (6)
with

cost x,E (e) = ∥ x -e ∥ 2 +λ ∥ x -e ∥ 2 E

Description of algorithm KICE

This section describes the algorithm we propose to solve the optimisation problem defined in Equation ( 6) named KICE for Knowledge Integration in Counterfactual Explanation. Its general principle is presented below before a more detailed description of its generation step.

Principle. KICE exploits the principle of iterative generation of instances proposed by the Growing Spheres algorithm [START_REF] Laugel | Comparison-based Inverse Classification for Interpretability in Machine Learning[END_REF]: given f the considered classifier, x the instance for which an explanation is requested, E the set of user known features and λ the weight of the incompatibility, KICE generates instances around x in increasingly larger spaces until it finds one predicted by f with a different class. The integration of E in the incompatibility term in the cost function calls for a non uniform generation: the space is distorted, with features being associated to different weights.

Indeed, for any ν, the equation cost x,E (e) = ν with cost presented in Equation ( 6) defines an ellipse with center x and radius ν 1+λ with respect to features in E and √ ν with respect to features in E. The spaces to be considered are thus not spherical layers, but ellipsoidal ones.

This principle is illustrated on Figure 1 that shows a toy 2D-data set, the classifier prediction with the colored regions and an instance of interest x. The considered user knowledge is defined as E = {X 2 }. As a consequence, it is less costly to modify feature X 2 than feature X 1 . Tentative countefactual examples are thus iteratively generated in the "vertical" ellipses until reaching an instance predicted in the other class e * . Uniform Generation in Ellipsoidal Layers. In order to generate candidates uniformly in the ellipsoidal layers, KICE relies on a modified version of the HLG algorithm [START_REF] Muller | A Note on a Method for Generating Points Uniformly on N-Dimensional Spheres[END_REF] presented in Algorithm 1. The latter generates instances uniformly in the spherical layer SL(x, a 0 , a 1 ) defined as the set of points at a distance greater than a 0 and less than a 1 from x. Modification of the algorithm HLG we propose relies on the random variable U = {u i } ∼ U([0, 1]) used to generate points according to a 0 + a 1 U 1 dim(X ) .

To modify HLG we propose to distinguish between features in E and in E: for the former, the same procedure as HLG is applied, considering a 0 + a 1 u i for i ∈ E. For the latter, the generation is weighted by 1/ √ 1 + λ, i.e. considers a 0 √ 1+λ + a 1 √ 1+λ u i for i ∈ E. KICE overview. Globally, the KICE algorithm covers the space by generating instances iteratively: in an initialisation step, n instances are generated in the ellipse of center x and radius ν 0 1+λ with respect to the features in E and √ ν 0 with respect to the features in E, where n and ν 0 are hyperparameters. If none of these instances belong to the opposite class, KICE generates instances in the ellipsoidal layer between ν 0 and ν 0 + ϵ where ϵ > 0 is a third hyperparameter. Then, at each iteration if none of generated instances belong to the opposite class, new instances are generated in a new layer with new base radius ν 0 ← ν 0 + ϵ.

Algorithm 1 Ellipsoïdal Layer Generation

Require: x, the center of the ellipsoïdal layer Require: E, the prior knowledge Require: a 0 and a 1 the bounds of the layer Require: n, number of desired points Ensure:

Z = {z i } i≤n Y = {y i } i≤n ∼ N (0, 1) Y ← Y ∥Y ∥ 2 U ← {u i } i≤n ∼ U([0, 1]) R ← a 0 + a 1 U 1/dim(X) W ← R T Y + x R ′ ← a 0 √ 1+λ + a 1 √ 1+λ U 1/dim(X) W ′ ← R ′T Y + x for i ← 1 to n do if i ∈ E then Z i ← W i else Z i ← W ′ i end if end for return Z

Knowledge Integration in Surrogate Models (KISM)

This section presents a second instantiation of the general framework described in Section 3: it considers the same type of user knowledge as KICE, as discussed in Section 4.1, but describes its integration to explanations in the form of linear surrogate models. This section first describes the proposed cost function and then presents the algorithm proposed to optimize it.

Proposed Cost Function: Instantiation of the Framework

This section discusses in turn the three components of the general cost function, penalty, incompatibility and aggregation function, using the same notations as in Section 4. E denotes the set of candidate surrogate models, for instance linear regression models in the case of LIME [START_REF] Ribeiro | Why should I trust you?" Explaining the predictions of any classifier[END_REF]. In this case, E = {g|∀z ∈ X , g(z) = d i=1 w i z i + w 0 } and the final explanation is a feature importance vector obtained from the w * associated to the chosen e * ∈ E. To simplify the notations, we do not note the link between the model g and his coefficients w, i.e. we use a simplified notation w i instead of w g i . The set E may also be a set of decision trees which are then considered as transparent models and thus directly define an interpretable explanation. For any e ∈ E, and for any data point z ∈ X , e(z) then denotes the prediction performed by the surrogate model e for z. For such surrogate approaches the black box classifier f to be explained is usually considered to output continuous probability scores that are later thresholded to get a prediction result.

Penalty Function. We propose to define the penalty using the classical surrogate cost function: it assesses the fidelity of e which respect to f , measured by their l 2 difference on a neighborhood Z x around the instance of interest x. The definition of the neighborhood Z x is a crucial and non trivial step of the processus, as a.g discussed by Laugel et al. [START_REF] Laugel | Defining locality for surrogates in post-hoc interpretablity[END_REF]. Formally,

penalty x (e, f ) = z∈Zx (f (z) -e(z)) 2 + Ω(e) (7)
The first term captures the local fidelity of the surrogate model to f and the second term Ω(e) captures the complexity of the candidate explanation e. The latter depends on the type of surrogate classifiers. In the classical case of linear regression models, the complexity can be the l 0 , l 1 of l 2 norms of the regression coefficients, depending on whether a simple, lasso or ridge regression is performed. In the following, we consider the case of the l 2 norm with weight α: Ω(e) = α d i=1 w 2 i . Incompatibility Function. In order to value the agreement with the user knowledge, we consider that the surrogate-model explanation can be associated with weights on the features, denoted w i : this is obviously the case for linear regression surrogates; when decision trees are considered, the depth of the occurrence of the feature allows to derive such weights.

We then interpret the user knowledge E as providing the most important features and wish to favor, when possible, the coefficients associated to these variables. For this purpose, we thus propose to penalize the coefficients associated to features that do not belong to E.

To match our choice to define the complexity Ω using a l 2 penalty, we propose to define the incompatibility function as:

incompatibility x (e, E) = i∈E w 2 i (8)
with w i the weights associated with the features.

Aggregation Function. As in Section 4.2, we propose to formulate the new explanation objective as an explicit trade-off between the quality and the compatibility of the explanation, through a weighted average.

Global Function. As a consequence, the proposed optimization problem is:

e * = argmin e∈E cost x,E (e) (9)
with cost x,E (e) = argmin

e∈E z∈Zx (f (z) -e(z)) 2 + α d i=1 w 2 i + λ i∈E w 2 i
with α and λ ∈ R + .

Algorithm Description

We propose to solve the optimization problem defined in Eq. 9 in the case of linear surrogate models, i.e. when E = {g|∀z ∈ X , g(z) = d i=1 w i z i + w 0 }. The solution of a typical ridge regression problem to minimize can be expressed as: W = (X t X + αI) -1 X t y where X denotes the matrix representing the neighborhood data, W the vector of w i , y the labels associated to X and I the identity matrix. Now the cost function defined in Eq. 9 belongs to this framework, with an additional λ to weight the features outside the user knowledge. Thus, denoting I E an identity matrix containing 1 only on the rows associated to unknown features, the solution to the considered optimization problem can be rewritten

(X t X + αI + λI E ) -1 X t y
where X denotes the data points generated in the neighborhood Z x and y the vector of their associated prediction by the classifier f to be explained. This solution makes explicit the integration of knowledge within the method. The integration of a new term does not change the complexity and the procedure to apply the algorithm.

Experiments

This section presents the experiments performed to study the explanations generated by the proposed methods KICE and KISM and to assess their relevance. Table 1: Values of the hyperparameters λ, ϵ, n and ν 0 of KICE chosen for the three datasets.

Experimental Protocol

Experiments are conducted on three classical benchmark tabular data sets: Half-Moons, Boston and Breast Cancer where the first one is a dataset generated with the package sklearn and the two last ones are UCI datasets. As a pre-treatment, the data are normalized and in the case of the Boston data set, the regression value is transformed into a binary class by discretization: the price is "expensive" if it is greater than $21, 000, and "cheap" otherwise. The data sets are split into train and test data subsets (80%-20%).

Within the considered post-hoc explanation framework, the classifier choice does not matter, we apply SVM with a Gaussian kernel, that achieves a high accuracy on the three data sets: 0.99, 0.98 and 0.93 respectively for Half-Moons, Boston and Breast cancer.

Explanations e * are then generated for each instance x of the test data set using different methods. First, python implementation of KICE and KISM, using the numpy, sklearn and lime libraries, are run, using the λ, ϵ, n and ν 0 parameters values shown in the Table 1. ϵ, n and ν 0 are chosen according to the dimension of the datasets and the number of features considered for E: the more features the dataset contains, the higher the value of the parameters. λ is chosen arbitrarily, in order to obtain interesting results, to have explanations that differ from the competitor ones, for the competitors described below.

The first competitor, generating the explanation denoted e ref , corresponds to the case where no knowledge is taken into account: it solves the reference optimization problem that only minimizes the penalty function. This competitor corresponds to Growing Spheres [START_REF] Laugel | Comparison-based Inverse Classification for Interpretability in Machine Learning[END_REF] in the case of counterfactual explanations, and LIME [START_REF] Ribeiro | Why should I trust you?" Explaining the predictions of any classifier[END_REF] in the case of surrogate models. This corresponds to an extreme case of aggregation of Eq. ( 6) where the incompatibility term is ignored, i.e. setting λ = 0.

In the case of counterfactual explanation, a second competitor is proposed by imposing to comply strictly with the knowledge under the constraint it is predicted in the opposite class. Its associated cost function thus equals the incompatibility function, a naive way of integrating knowledge in the explanation. In this case, we denote e user the counterfactual instance that solves the associated problem:

e user = argmin e∈E ∥ x -e ∥ 2 E
This corresponds to the other extreme case of aggregation of Equation ( 6), where the penalty term is ignored, i.e. λ arbitrarily large. We do not consider Ustun et al. as competitor although it shares common points with the proposed KICE because it is restricted to linear classifiers and cannot be applied to the SVM with Gaussian kernel considered here).

Finally, regarding user knowledge E, we consider that the user disposes of less features than the classifier. In order to build somehow realistic knowledge, we train a decision tree with low depth on the train data (more precisely the maximal depth is set to half the number of data features), the set of features E then contains the ones that occur in this tree.

KICE Experimental Study

This section presents the experiments conducted to evaluate the KICE algorithm: the purpose of these experiments is to show that the proposed method finds the expected counterfactual instance, i.e. it allows to achieve a trade-off between the quality and the compatibility of the explanation.

Illustrative Examples

First, we illustrate the behavior of the methods with the 2D Half-Moons data set, denoting X 0 and X 1 the two dimensions. Figure 2 shows the counterfactual examples obtained for three different instances x. The other points represent the training examples, the blue and red regions represent the predicted classes. The considered knowledge system contains a single feature, it is represented by the horizontal line:

E = {X 1 }.
As expected, we can observe that: (i) the counterfactual instance e ref is the closest point belonging to the opposite class; (ii) e user is further away than e ref and only modifies feature X 1 which is the knowledge feature; (iii) e * is a compromise between e ref and e user . It requires less modifications according to X 0 than e ref , hence it is more compatible. It is also closer to the studied instance than e user . In the graph, on the right, we notice that there is no e user . In this case, there is no counterfactual instance modifying only feature X 1 . The proposed KICE method is then useful because it allows to obtain a more compatible counterfactual than e ref .

Evaluation of the KICE Method

In this section, we further compare KICE with its competitors and study quantitatively the gain in terms of the proposed cost function.

We apply the three methods described in the previous section on the test set of the three data sets. Among the test set, for some instances no counterfactual example is found by the method that modifies only the features in E (e user ). This happens to the half-moons and breast cancer data sets, for respectively 20% and 11% of the instances. For the other instances, Table 2 shows the mean and standard deviation values of the penalty, incompatibility and cost functions for the three approaches.

We observe that, as expected, the proposed counterfactual examples have a penalty greater than that of e ref but lower than that of e user . Moreover, the incompatibility function is much smaller than that of e ref . Thus, the modifications on the unknown features decrease, which increases the understanding of the explanation. Finally, e * cost function is the lowest. We can notice that the standard deviations are high, which is due the fact that the instances are at different distances from the boundary.

Table 3 shows the execution times to obtain the three counterfactuals. As expected, for all three data sets, we notice that the time associated with e * is higher than that of e ref . Since only one feature is taken into account,

penalty x (e) incompatibility x (e, E) cost x,E (e)
Half moons Table 2: Quantitative evaluation of the three considered approaches on the three datasets for metrics: penalty defined in Eq. ( 3), incompatibility defined in Eq. ( 4) and cost defined in Eq. ( 6). Average and standard deviation are computed on all instances x of the test set for which all three approaches provide an explanation.

e ref 0.
e the e user counterfactuals are further away, and require more iterations to be identified, increasing the execution time. For the two datasets Boston and Breast Cancer, the time associated with e * is higher than the other two. The higher the dimension of the dataset, the longer the execution time. As in the evaluation of the metrics, we also note that the standard deviations are high as the studied instances are located in different areas of the data space.

To verify that KICE minimizes the cost function as opposed to the other two on all instances, Figure 3 shows the value of the cost function obtained by e * (that is defined as minimizing it) as compared to the value it takes for e ref (left) and e user (right), for each of the test instances of the Halfmoons dataset. The figures show that, as expected, all points are above the line y = x.

On the right hand graph, the points are more scattered but they remain above the line. We notice that the generated counterfactual instances are closer to the cost function of e ref than to e user . The user knowledge contains a single feature, it is difficult to get a close counterfactual instance only by modifying this feature. It is then necessary to move further away from the studied instance to get 100% compatible one. It is also possible to increase λ so that the points are less scattered.

KISM Experimental Study

This section describes the experiments conducted with the KISM method by presenting illustrative examples and by showing that the algorithm achieves its goal, i.e. it associates a more important weight to the features included in the user knowledge. For the experiments, we arbitrarily set their weight in the ridge regression to 1.

In order to make the numerical evaluation easier, through the study of the rank associated to the features present in the user knowledge, we restrict the latter to contain a single feature: for Half Moons, except for the illustrative examples given below, E = {X 1 }, for Boston, E = {X 7 } and for Breast Cancer, E = {X 3 }. The value of λ is set to 500, the LIME parameters are set to their default values.

Explanations e * are compared to one competitor that corresponds to LIME. This competitor solves the reference optimization problem that only minimizes the penalty function, the explanation is denoted e ref .

Illustrative Examples

First, we illustrate the behavior of the methods using the same 2D Half-Moons data set as before: for the instance x represented with a +, Figure 4 shows the decision boundaries (corresponding to a 0.5 probability) of the In both graphs, the classic surrogate model e ref shown in red, that does not take into account any user knowledge, provides the same result. On the left graph, we observe the weight according the feature X 0 is higher for e * (green line) than for e ref (red line). On the right graph, we observe the weight according to feature X 1 is higher for e * (green line) than for e ref (red line). The proposed KISM method is thus useful, as it effectively allows for an explanation to give more importance to known features than to unknown features.

Evaluation of the method

This section discusses the evaluation of the proposed method according to different criteria. First, for each obtained explanation, we rank the features Table 5: Results with the two considered approaches on the three data sets for metrics: penalty defined in Eq. ( 7), incompatibility defined in Eq. ( 8) and cost defined in Eq. [START_REF] Barocas | The hidden assumptions behind counterfactual explanations and principal reasons[END_REF] of the explanation in increasing order of absolute value of weights. Then, we study the position in this ranking of the user's feature, denoted E with a notation abuse below. Table 4 gives, in the first column, the average ranking (with standard deviation) over all instances of the test set in the explanations obtained without taking into account the knowledge (e ref , provided by LIME). In the second column, the same average ranking is shown for the proposed KISM approach. The third column gives the number of positions gained between the two methods. We observe that the studied feature moves upward in the ranking in average (between 0.09 for half-moons and 4.67 for the breast cancer data set) when it is taken into account by the method. This effect is more visible for the data sets with more features. Indeed, in the 2D Half-Moons case, only two rank values are possible, which makes it harder to modify the values. Altogether, this shows that the proposed KISM method helps increasing the importance given to the user feature. Second, we study the two methods according to the three criteria presented in Section 3: penalty, incompatibility and cost function. To do so, we apply LIME and the proposed method on all test instances of the different data sets. As expected e ref minimizes the penalty function and e * the incompatibility function. As compared to e ref , the proposed explanation e * does have a higher penalty, a lower incompatibility and altogether a lower cost function, meeting the desired behavior.

The proposed approach is based on the same procedure as LIME and only changes the type of considered model. Thus, the complexity of the proposed approach is the same as that of LIME, the time needed to obtain e ref and e * is the same.

Discussion on Knowledge Expressed as Rules

This section proposes to discuss the relevance and challenge of considering a richer type of user knowledge, beyond the case of a set of features, in the form of logical rules, and integrating it into counterfactual explanations. We name rKICE this rule Knowledge Integration in Counferfactual Explanation principle.

Characteristic of the Knowledge Type and Motivations

User knowledge expressed as a rule based system is defined as a set of logical implications that allow the user to make his/her own prediction for any instance. Formally, for any x ∈ X , a rule is defined as

d i=1 x i ∈ [v i inf , v i sup ] =⇒ class = y
with v i inf ∈ R and v i sup ∈ R the lower and upper bounds of the interval over the i-th feature, possibly with infinite values. Such a rule can also be written

i∈A R x i ∈ [v i inf , v i sup ] =⇒ class = y
where A R denotes the set of features whose associated interval is not R. A R then corresponds to the user knowledge considered in the previous section.

As compared to the knowledge type considered in the previous sections, defined as set of features, rule-based knowledge is much richer, as it provides two additional pieces of information: each involved feature is associated with an interval and the associated prediction of a class.

We consider that for any instance of interest x, a single rule of the user knowledge E is triggered. This rule could be denoted E(x), we use the notation and keep E, not making explicit this dependency. This hypothesis is not restrictive as several rules can be combined into a single one using the appropriate logical operators.

It seems reasonable to make the assumption that the user requests an explanation for instances for which the prediction of the classifier differs from the rule's decision.

Proposed Cost Function for rKICE

As in Sections 4 and 5, this section discusses the definition of the three components of the generic framework, the penalty, incompatibility and aggregation functions, for the integration of rule-based knowledge in counterfactual explanations.

Penalty Function. We propose to evaluate a candidate counterfactual example using the classical counterfactual penalty, as in Eq. ( 3):

penalty x (e) =∥ x -e ∥ 2
with the restriction of E to E x to guarantee the prediction associated with the counterfactual example differs from the one of the reference instance.

Incompatibility Function. In order to integrate the user knowledge in the explanation, we propose to impose that the counterfactual example belongs to the same region as defined by the user rule set, as the instance of interest: in other words the generated example needs to satisfy the premise of the rule triggered by the instance. Indeed, this premise describes a region where the user possesses some knowledge and makes sens to him/her.

Hence we propose to penalize counterfactual candidates whose values for the features involved in the rule premise do not satisfy the associated rule constraints. Formally the incompatibility function is thus defined as:

i∈A E (x i -e i ) 2 × 1 e i / ∈[v i inf ,v i sup ]
with A E the set of features present in the premise of the rule E.

Aggregation Function. As in the previous proposed instantiations of the generic framework, we propose to aggregate the two terms of penalty and incompatibility using a trade-off operator, the weighted average. and KICE (middle) modes. The user rule is "if X 1 < t, then class=blue".

Comparison of KICE and rKICE principles

This section analyzes the rKICE optimization problem defined in Eq. ( 10) and compares it to the KICE optimization problem defined in Eq. [START_REF] Wachter | Counterfactual Explanations without Opening the Black Box: Automated Decisions and the GDPR[END_REF]. The first difference between these two problems is that rKICE incompatibility function depends on the features present in the user knowledge, whereas, on the contrary, the KICE one depends on the features absent from the user defined set of features. The principle of KICE may be combined to that of rKICE, through a more complex incompatibility function additionally penalizing the use of features absent from the premise of the triggered rule. However, the KICE incompatibility term is motivated by the notion of features known to the user, i.e. that he/she may understand. Following these lines it could be relevant to define the rule induced set of features as the union of the sets of features involved in all the rules. This variety of possibilities, that are all instantiations of the proposed generic framework, shows that the definition of an incompatibility term for rich knowledge as expressed by a rule set is a complex task that requires a refined study, out of the scope of this paper.

Focusing on the basic definitions given in Eq. ( 10) and Eq. ( 6) for rKICE and KICE respectively, one can distinguish between two cases for a given candidate explanation e: in the case where it satisfies the triggered rule premise, the incompatibility term equals zero and the cost function reduces to the classical counterfactual example case. Otherwise, let us denote A ′ E the set of features involved in the triggered rule E whose interval constraint is not satisfied: the cost function is then of the form of the KICE one, for a user feature set defined as A ′ E . Figure 5 illustrates the combination of these two cases in a 2D toy data set, in the case where the user triggered rule is "if X 1 < t, then class=blue", where t is a threshold value as represented on the graph (this rule is chosen to be a not appropriate one with respect to the classification task, for illustrative purposes). For the given instance x predicted as pink for the considered black box, the candidate counterfactual examples are generated along spherical layers in the region that does satisfy the rule premise, i.e. below the line and along ellipsoidal layers, in a KICE mode, above the line. The resulting combined generation form for rKICE is shown on the right graph.

Conclusion

In this paper, we propose a general framework defining an optimization problem to integrate user knowledge in post-hoc model-agnostic explanations. Using this framework, we propose two methods integrating knowledge as a set of features, interpreted as the features the user understands and can thus be exploited to build the explanation: KICE for the case of counterfactual explanations and KISM for the case of linear surrogate model explanations. KICE generates counterfactual explanations in the knowledge language by minimizing modifications according to unknown features, Similarly, KISM generates a linear model approximating the classifier to be explained in the knowledge language by minimizing the weights of unknown features. The relevance of these approaches are illustrated and evaluated numerically on several benchmark data sets. The paper also proposes a discussion extending the principles to the case of richer user knowledge in the form of a set of logical rules, showing the challenges of defining the incompatibility function.

The opened directions for future works are numerous and first include experiments with real users, to assess their satisfaction of the offered enriched explanations. A crucial question is also that of collecting their knowledge to be integrated in the generated explanations, which probably calls for the design of dedicated and user-friendly interfaces. It seems that an interactive process, e.g. generating first explanations and allowing the users to react and ask questions regarding some involved features, might be a direction to explore, in line with the desirable interactive property of explanations [START_REF] Miller | Explanation in artificial intelligence: Insights from the social sciences[END_REF]. Ongoing works aim at developing the exploitation of user knowledge expressed as rule-based systems, among others the impact of rule fidelity with respect to the classifier to be explained. Other works aim at exploring and proposing instantiations of the proposed generic framework for other explanation and knowledge types.

Figure 1 :

 1 Figure 1: Illustration of KICE ellipsoidal layers generation step for a toy 2D data set: f is indirectly defined by the colored regions, user knowledge E is {X 2 }, x denotes the instance of interest, the ellipses the considered increasingly large spaces around x, e * the finally generated counterfactual.

Figure 2 :

 2 Figure 2: Examples of resulting e ref , e user and e * for three instances x (+ + +: x, ▲: e ref , ■: e user , •: e * )

Figure 3 :

 3 Figure 3: Cost functions cost as defined in Eq. (6) of e ref , e user and e * for the 80% of the Half-moons test set for which all three counterfactual instances are defined. (left) Cost functions associated to e ref and e * , (right) cost functions associated to e user and e * .

Figure 4 :

 4 Figure 4: Surrogate models for KISM e * and reference e ref for the same instance x and different user knowledge: left E = {X 0 }, right E = {X 1 } (+ + +: x, -: e ref , -: e * ) e ref e * Ranking gain Half-Moons 1.63 ± 0.48 1.55 ± 0.5 0.09 ± 0.28 Boston 6.08 ± 2.89 4.49 ± 2.74 1.58 ± 1.35 Breast Cancer 13.18 ± 5.19 8.51 ± 4.86 4.67 ± 2.46

  two surrogate models obtained by KISM, denoted e * and shown in green, and the reference e ref shown in red. On the left graph, the considered user knowledge is E = {X 0 } and on the right graph, it corresponds to the other feature E = {X 1 }.

Global Function.Figure 5 :

 5 Figure 5: Right: rKICE generation mode, as a combination of the Growing Sphere (left) and KICE (middle) modes. The user rule is "if X 1 < t, then class=blue".

Table 3 :

 3 Execution times of the three considered approaches to obtain e ref , e user and e * for the three datasets.

Table 4 :

 4 Average rank, with standard deviation, of the user defined feature, for the two considered methods and the three data sets, together with the ranking gain.

In the case where the user makes explicit the class he/she is interested in, denoted c * ∈ Y with c * ̸ = f (x), the additional term penalises candidates such that f (e) ̸ = c * and not only f (e) = f (x). In the following we keep to the simple case where c * is only required to differ from f (x).