Jiaoyan Chen

Freddy Lécué

Jeff Z Pan

Ian Horrocks

Huajun Chen

Knowledge-based Transfer Learning Explanation

Machine learning explanation can significantly boost machine learning's application in decision making, but the usability of current methods is limited in humancentric explanation, especially for transfer learning, an important machine learning branch that aims at utilizing knowledge from one learning domain (i.e., a pair of dataset and prediction task) to enhance prediction model training in another learning domain. In this paper, we propose an ontology-based approach for humancentric explanation of transfer learning. Three kinds of knowledge-based explanatory evidence, with different granularities, including general factors, particular narrators and core contexts are first proposed and then inferred with both local ontologies and external knowledge bases. The evaluation with US flight data and DBpedia has presented their confidence and availability in explaining the transferability of feature representation in flight departure delay forecasting.

Introduction

Prediction with machine learning (ML) has been increasingly applied in a variety of fields to assist humans in decision making. ML explanation work such as interpreting the prediction model or justifying the prediction result can significantly increase decision makers' confidence on the prediction and boost its application [START_REF] Biran | Explanation and justification in machine learning: A survey[END_REF], especially in making critical decisions like cancer diagnosis when people need to understand how and why the prediction is made.

Most ML explanation studies such as designing inherently interpretable models [START_REF] Wu | Beyond sparsity: Tree regularization of deep models for interpretability[END_REF]) and approximating a "black box" model with multiple "white box" models [START_REF] Tulio Ribeiro | Why should i trust you?: Explaining the predictions of any classifier[END_REF]) aim at users with ML expertise. The explanations lack background and common sense knowledge, thus are too hard to be understood by non-ML-experts, those common users without ML expertise such as doctors. There are only a limited number of human-centric ML explanation studies. Most of them adopt some corpus (e.g., Wikipedia articles [START_REF] Biran | Human-centric justification of machine learning predictions[END_REF]) or Link Data [START_REF] Tiddi | Dedalo: Looking for clusters explanations in a labyrinth of linked data[END_REF] to generate text to describe model components (e.g., effective ML features) or justify the prediction results

Accepted by International Conference on Principles of Knowledge Representation and Reasoning, 2018. (e.g., data clusters). They adopt background knowledge but are limited by expressivity, which in turn restricts the reasoning and inhibits rich explanations.

On the other hand, transfer learning which utilizes samples, features (i.e., representations of original data) or models of one learning domain (i.e., a pair of dataset and prediction task) to enhance prediction model training in another learning domain [START_REF] Sinno | A survey on transfer learning[END_REF] has been widely applied, especially in dealing with critical challenges like lacking training data. Its explanation aims at justifying the good or bad performance of the prediction model trained by a specific transfer learning algorithm with a specific parameter setting. Current work on transfer learning explanation such as analyzing the impact of a feature's specificity and generality on its transferability [START_REF] Yosinski | How transferable are features in deep neural networks? In Advances in neural information processing systems[END_REF]) aim at ML experts and represent the insights in a machine understandable way. It's hard for common users to understand why transfer from one learning domain contributes to an accurate prediction model (i.e., positive transfer) while transfer from another learning domain contributes to an inaccurate prediction model (i.e., negative transfer).

In this paper, we propose an ontology-based knowledge representation and reasoning framework for human-centric transfer learning explanation. It first models a learning domain in transfer learning, including the dataset and the prediction task, with expressive OWL (Web Ontology Language [START_REF] Bechhofer | OWL: Web ontology language[END_REF])) ontologies, and then complements the learning domain with the prediction task-related common sense knowledge using an efficient individual matching and external knowledge importing algorithm. The framework further uses a correlative reasoning algorithm to infer three kinds of explanatory evidence (i.e., general factors, particular narrators and core contexts) to explain a positive feature or a negative transfer from one learning domain to another. Some technical challenges such as feature transferability measurement and core context (entailment subset) searching are overcome.

As far as we know this is the first work to study humancentric transfer learning explanation and ontology-based ML explanation. It achieves confident and rich human understandable evidence for explaining both positive and negative transfers in predicting US flight delay, where the feature learned by a Convolutional Neural Network (CNN) is transferred. For example, we find that transferring between flights carried a big airline company is an evidence to explain positive transfers, while transferring between flights departing from the airport of SFO is an evidence to explain negative transfers (cf. Example 4 for more examples).

The remainder of the paper is organized as follows. The next section introduces the ML background with ontologies, and defines the problem of transfer learning explanation. Then we present the ontology-based framework and report the evaluation. In the final two sections, we review the related work and conclude the paper.

Background and Problem Definition

We use Description Logics (DL) based ontologies written in the W3C OWL 2 standard 1 , in particular the EL ++ [START_REF] Baader | Pushing the EL envelope[END_REF]) fragment of the OWL 2 EL profile. In this section, we first introduce EL ++ based ontology, then revisit the notions of learning domain, supervised learning and transfer learning with ontologies, and eventually define the problem of transfer learning explanation.

The EL ++ Description Logic

Given a signature Σ = (N C , N R , N I ), consisting of 3 disjoint sets of atomic concepts N C , atomic roles N R , and individuals N I , the top concept , the bottom concept ⊥, an atomic concept A, an individual a, an atomic role r, EL ++ concept expressions C and D can be composed with the following constructs:

| ⊥ | A | C D | ∃r.C | {a}
An EL ++ ontology is composed of a TBox T and an ABox A. The TBox T is a set of concept and role axioms. EL ++ supports General Concept Inclusion axioms (GCIs e.g., C D), Role Inclusion axioms (RIs e.g., r 1 r 2 , r 1 • r 2 s), where C, D are concept expressions, r 1 , r 2 , s are atomic roles. The ABox A is a set of class assertion axioms e.g., C(a), role assertion axioms e.g., r(a, b), individual equality and inequality axioms e.g., a = b, a = b, where C is a concept expression, r is an atomic roles and a, b are individuals. Entailment reasoning in EL ++ is PTime-Complete.

Learning with Ontology

In order to support ML, we need to specify the input and output. To this end, we introduce the notions of learning sample ontology (LSO) and target entailment. We use an LSO as an input for ML methods, and the truth of a target entailment as an output. A learning domain in ML equals to a combination of an LSO set (i.e., a dataset) and a target entailment (i.e., a prediction task). Definition 1. (Learning Sample Ontology (LSO)) A learning sample ontology O = T , A , S is an ontology T ,A annotated by property-value pairs S. Its ABox entailment closure {g|T ∪ A |= g} is denoted as G(O), where g represents an entailment.

The annotation S in Definition 1 acts as key dimensions to uniquely identify an input sample of ML methods. When the context is clear, sometimes we also use LSO to refer to its 6) and ABox axioms ( 7)-( 24), with some atomic concepts (e.g., Airport), defined concepts (e.g., DelayedDep), individuals(e.g. LAX) and roles (e.g., hasCarrier). With the above definitions, Definition 4 revisits the notion of within domain supervised learning task [START_REF] Mohri | Foundations of machine learning[END_REF] In the training LSO set, we assume the ABox axioms (observations) are complete. The target entailment is true if it is entailed by an LSO, and false otherwise. In the testing LSO set, we assume the ABox axioms are incomplete (some observations are missing or we predict before they are observed), and the truth of the target entailment is predicted by the model. Example 3. (Within Domain Supervised Learning) Given the learning domain D 0 in Example 2, we train the model M with LSOs before or at date t 0 (i.e., O ), and apply the model to predict the truth of DelayedDep(d) in each LSO after t 0 (i.e., O ).

Airport(JF K) (15) hasDes(d, JF K) (16) LAX = ori (17) DL = car (18) hasRecDep(d, d 1 ) (19) hasCarrier(d 1 , M U ) (20) hasRecDep(d, d 2 ) (21) hasCarrier(d 2 , AA) (22) DelayedDep(d) (23) HeavySnow(wea) (24)
To feed an ML algorithm, each LSO in both training LSO set and testing LSO set is encoded into a real value vector, denoted as x. We first transform it into (i) a value vector v with data properties by concatenating their numeric values and (ii) an entailment vector e by BOE embedding with a set of entailments entailed by domain LSOs (cf. Definition 5). Then we concatenate e and v as the real value vector: x = [e, v]. The target entailment g t in each training LSO is transformed into a binary existence variable, denoted as y. It's assigned 1 if g t ∈ G(O), and 0 otherwise. Definition 5. (Bag of Entailments) Given an entailment set {g i |i = 1, ..., n}, Bag of Entailment (BOE) is an ontology encoding method that represents an LSO (with ABox entailment closure G) to a vector e = (e 1 , e 2 , ..., e n ) where e i = 1 if g i ∈ G and 0 otherwise.

Explaining Transfer Learning

Definition 6 revisits the concepts of transfer learning, positive transfer and negative transfer [START_REF] Sinno | A survey on transfer learning[END_REF] In Definition 6, D α and D β are called source domain and target domain, respectively, while O β and O β are the training LSO set and testing LSO set in the target domain. In comparison with supervised learning within the target domain (Definition 4), transfer learning has the same settings except that it learns the model from not only the training LSO set but also the LSO set from a source domain.

The effect of F α→β , namely the transferability, can be measured by comparing the performance of M α→β and M β . In Definition 6, positive transfer and negative are defined for qualitative description. We also define a metric called Feature Transferability Index (FTI) (to be specified in (25) on page 4) for quantitative measurement. The higher the FTI metric, the higher the transferability.

Explaining transfer learning aims at justifying the good or bad performance of a model trained by a transfer learning algorithm with a specific parameter setting. It describes human understandable factors that influence the transferability. Definition 7 defines this problem as correlation-based transfer explanation. Definition 7. (Correlation-based Transfer Explanation) Given a transfer F α→β in Definition 6, correlation-based transfer explanation is a task of inferring a set of influential factors X such that each X in X is correlated with F T I and the absolute value of the correlation coefficient γ(F T I, X ) ≥ , where is a parameter in [0, 1]. An influential factor X is called an explanatory evidence.

In Definition 7, the confidence of an explanatory evidence for transferability explanation is proportional to its absolute coefficient value γ(F T I, X ) . We abuse the notion and note it as γ(X ) in the remainder of the paper.

Three kinds of explanatory evidence (cf. Example 4) are proposed, including • General factors which are those statistic indexes of ABox entailments that quantify the overall knowledge variance and invariance from the source learning domain to the target learning domain, • Particular narrators which are those particular ABox entailments that have a high impact on the transferability, • Core contexts which are those ABox entailment combinations that have a high impact on the transferability.

Method Transferability Measurement

In transfer learning, the training of the model for the target learning domain (i.e., M α→β in Definition 6) either directly integrates samples of the source learning domain or indirectly utilizes model parameters learned in the learning source domain [START_REF] Sinno | A survey on transfer learning[END_REF][START_REF] Weiss | A survey of transfer learning[END_REF]. In this study, we adopt the latter. Features learned by Convolutional Neural Networks (CNNs) i.e., learned parameters of hidden network layers, are transferred from the source learning domain to the target. As transferability measurement only depends on the performance of the model trained within the target learning domain (i.e, M β ) and the model trained with transfer (i.e., M α→β ), how transfer learning is implemented does not impact the generality of our explanation framework.

A CNN is stacked by convolutional (Conv) layers which learn the feature with data locality, and fully connected (FC) layers which learn the non-linear relationship between the input and output. As shown in Figure 2, we first train a CNN model within the source learning domain (i.e., M α ) using its LSO set (i.e., O α ) and target entailment (i.e., g t α ), then transfer the model's feature (i.e., parameters of the Conv layers) to a CNN model in the target learning domain (i.e., M α→β ) which has the same network architecture. We eventually fine-tune the parameters of the model in the target learning domain with its training LSO set (i.e., O β ) and target entailment (i.e., g t β ). It's called hard transfer if we only fine-tune the FC layers and soft transfer if we fine-tunes both FC layers and Conv layers.

Source Learning Domain Target Learning Domain

Step 2: Feature Transfer

Step 1: Train

Step 1: Fine-tune A feature's transferability depends on its specificity to the learning domain where it is trained and its generality [START_REF] Yosinski | How transferable are features in deep neural networks? In Advances in neural information processing systems[END_REF]. By comparing the performance of the model trained within the target learning domain with the models trained with hard transfer and soft transfer, the specificity and generality can be measured (cf. Definition 8). Definition 8. (Feature Specificity/Generality Index) Given transfer learning in Definition 6, let M h α→β and M s α→β be the models trained with hard transfer and soft transfer respectively, Feature Specificity Index (FSI) is defined as the performance drop of M h α→β over M β , while Feature Generality Index (FGI) is defined as the performance gain of M s α→β over M β , where the performance of all models are measured with the testing LSO set of the target learning domain.

We propose a comprehensive index called Feature Transferability Index (FTI) to measure the feature's transferability. It is proportional to its generality and inversely proportional to its specificity (cf. Property 1). The more the hard transfer or the soft transfer benefits the model in the target learning domain, the higher transferability the feature has. With FSI and FGI, we calculate FTI as follows:

F T I = ω1 • F GI -ω2 • F SI ω1 + ω2 (25)
where ω 1 and ω 2 are weight parameters in [0, 1] and are both set to 1 in the remainder of the paper if not specified. We denote the FTI value from learning domain D α to learning domain D β as f t (D α , D β ). Property 1. (Relation between FTI and FSI/FGI) FTI is proportional to FGI and inversely proportional to FSI. In Definition 9, we calculate the rate of LSOs that contain an entailment g. The entailments that appear in a large part of LSOs are frequent entailments. In Definition 10, r e (r i ) represents the rate of LSOs where the entailment subset G κ and the target entailment g t co-exist (co-inexist). The higher r e + r i , the more effective G κ in predicting the truth of g t , according to the theory and practice of correlation-based ML feature selection (Hall 1999). Knowledge Importing Workflow. For each learning domain, we match each of its root individuals with an entity of an external KB, and then import the concepts and roles of the entity. The workflow is shown in Algorithm 1.

External Knowledge

In Line 5 and 6, we use root individuals to match external entities by name matching. Using root individuals signifi-cantly saves computation and storage, as the non-root individuals take a large part but usually lead to external axioms that contribute little to the richness of explanatory evidence (cf. Evaluation for more details).

From Line 7 to Line 13, we (i) extract values of concepts and roles (i.e., K) of each matched entity, (ii) transform them into ABox axioms with the terminologies defined in TBox, (iii) check their consistence with local LSOs and the constraint axioms (i.e., C), and (iv) add them into the set of external axioms (i.e., A e ) (cf. Example 6). The consistency checking is to avoid errors caused by name matching (cf. Example 7). Line 14 eventually computes the entailment closure of the learning domain, denoted as G(D), together with the local LSOs and external axioms. Example 6. (External Axioms) The individual LAX in Example 5 is matched with the entity Los Angeles International Airport in DBPedia. The triples related to the concept ("rdf : type") and roles, e.g., "geo : lat" and "geo : long", "dbo : hubAirport", of the entity are extracted and transformed into external ABox axioms e.g., "hasLat(LAX, 38.94)". Example 7. (Consistency Checking) In Example 6, the individual LAX can be matched with the entity L.A.InternationalAirport (a song by Leanne Scott) in DBPedia by name matching. The constraint axiom "Location Song ⊥", the local axioms "Airport(LAX)" and "Airport Location", and the external axiom "Song(LAX)" suggest that the entity matching is incorrect.

The constraint axioms, which may contain concept expressions that are more expressive than DL EL ++ for the requirement of a specific prediction application, are an extension of the TBox of the learning domain, but are detached from the TBox to avoid increasing reasoning complexity in other steps.

Correlative Reasoning

We propose a method called correlative reasoning for calculating the explanatory evidence (i.e., general factors, particular narrators and core contexts) with entailment closures of the learning domains and the FTIs between learning domains. It is composed of two steps: evidence embedding and correlation analysis. Evidence Embedding. It represents an explanatory evidence by a real value, without losing the evidence's semantics in analyzing the feature transferability. Given an evidence X , we denote its embedding as f e (X ).

General factors are statistic indexes that measure the overall difference and similarity between two learning domains. Definition 11 defines the embedding approach for three gen- A particular narrator is one single entailment that (i) is shared by the source and target learning domains, and (ii) has positive or negative impact on the feature's transferability. Core context is an extension of particular narrator from one single entailment to an entailment set (combination). To simplify the representation, we regard a particular narrator as a one-element entailment set, and use evidence note X to denote the entailment set involved in a particular narrator or core context.

Definition 12 defines the embedding approach for particular narrators and core contexts. It transforms a specific entailment or an entailment set into a binary variable called DEC with the entailments' co-existence in the source and target learning domains considered. Different from BOE embedding in Definition 5, which transforms entailments of an LSO into one vector as ML input, evidence embedding transforms the entailment-based change from one learning domain to another into a real value for transferability analysis. Correlation Analysis. We analyze the correlation between a given explanatory evidence X and its impact on the transferability of a feature with a set of learning domains D, as shown in Algorithm 2. Line 6 traverses each pair of source and target learning domains, which correspond to one transfer. For particular narrators and core contexts, Line 7 and 8 skip the transfers whose source learning domains fail to entail all the entailments that are involved the evidence. Line 10 and 13 calculate the embedding of the evidence and the FTI value of the transfer, respectively. Line 14 calculates the Pearson Correlation Coefficient (Lee Rodgers and Nicewander 1988) and its p-value in a t-test with non-correlation hypothesis. The algorithm eventually returns the correlation coefficient (cf. Definition 7) and its p-value.

An evidence X is a valid explanatory evidence for the feature transferability if (i) its absolute value of correlation coefficient γ(X ) ≥ (cf. Definition 7) and (ii) the correlation analysis is significant (i.e., ρ(X ) ≤ 0.05).

In correlative reasoning, m(m -1) /2 times FTI calculation are totally needed, where m is the size of the given domain set D. Meanwhile, correlative reasoning costs m(m -1) /2 times evidence embedding calculation and one time correlation analysis calculation for each evidence. On the other hand, to compute a complete set of explanatory evidence, we need to traverse all the candidate evidence. The number of general factors is a constant, while the numbers of particular narrators and core contexts are n and 2 n -1 respectively, where n is the size of the entailment closure of the given learning domains (i.e., ∪ D∈D G(D)). Directly searching all the core contexts is impractical; thus we need some optimized methods of searching for core context .

v f := ∅ % Init. of FTI value array 6 foreach D k 1 ∈ D, D k 2 ∈ D such that D k 1 = D k 2 do 7 if (X is a particular narrator or a core context) and (X ⊆ G(D k 1 )) then 8 Continue 9 % Cal. evidence embedding 10 fe(X ) Def.11,Def.12 ←---------(X , G(D k 1 ), G(D k 2 )) 11 ve := [ve, fe(X )] 12 % Cal. FTI for the transfer from D k 1 to D k 2 13 ft(D k 1 , D k 2 ) (25),Def.8 ← ------(D k 1 , D k 2 ) 14 v f := [v f , ft(D k 1 , D k 2 )]
15 % Pearson correlation analysis and t-test 16 γ(X ), ρ(X ) = corr(ve, v f ) 17 return γ(X ),ρ(X );

Optimized Core Context Searching

A core context is composed of a subset of entailments of the given learning domain set (i.e., X ⊆ ∪ D∈D G(D)). Algorithm 3 presents our core context searching algorithm. It starts by traversing core contexts composed of two entailments (cf. Line 5 to 7), and then traverses core contexts with higher dimension by adding an entailment to the current core context (cf. Line 14 to 17). It adopts two approaches, EarlyStop and FastExtend to accelerate the search.

(X ), ρ(X ) ← CorrelativeReason(D, X ) 11 print X , γ(X ), ρ(X ) 12 if ! EarlyStop (D,X ) then 13
% Traverse core contexts with one more ent.

14 foreach g ∈ G0 such that g ∈ X do 15 if FastExtend(D, X , g) then 16 print X ∪ {g} , γ(X ), ρ(X ) 17 else 18 CoreContextSearch(D, X ∪ {g})
Early Stop. The function EarlyStop returns true if adding more entailments to a core context evidence will not lead to any valid core contexts, and false otherwise. According to Algorithm 2 and the principle of t-test, enough Evidence Domains (cf. Definition 13) are needed for significant correlation analysis of an evidence, while Property 2 shows that when a core context evidence is extended by an entailment, the number of its evidence domains decreases. Thus when the correlation analysis of the current evidence is insignificant (i.e., ρ(X ) > 0.05), we stop extending this evidence with more entailments. Definition 13. (Evidence Domains) Given a core context or particular narrator evidence X and a learning domain set D, the evidence domains of X , denoted as D(X ), are defined as {D ∈ D|X ⊆ G(D)}.

Property 2. (Monotonicity of Evidence Domains)

In Definition 13, for all the entailment g in ∪ D∈D G(D) and g ∈ X , we have D(X ∪ {g}) ⊆ D(X ). Fast Extend. The function FastExtend returns true if an entailment is a Synchronized Entailment (cf. Definition 14) of another entailment in the current core context evidence, and false otherwise. According to Lemma 1, the new core context evidence with the synchronized entailment added has the same impact on the feature transferability as the original one. FastExtend enables us to directly extend a core context evidence and avoid the calculation of evidence embedding and correlation analysis. Definition 14. (Synchronized Entailments) Given a learning domain set D, two entailments g 1 and g 2 are synchronized, denoted by g 1

D = g 2 , if for all the learning domain D in D, {g 1 , g 2 } ⊆ G(D) or {g 1 , g 2 } ∩ G(D) = ∅.
Example 10. (Synchronized Entailments) In our departure flights example, "locatedIn(LAX, LA)" and "serveCity(LAX, LA)" are synchronized entailments, w.r.t. the 92 learning domains used in our evaluation. Lemma 1. (Synchronized Evidence Extension) In Definition 14, given a core context or particular narrator evidence X , for all the entailment g in ∪ D∈D G(D) and g ∈ X , the new core context evidence X := X ∪ {g} has the same correlation analysis result as X , i.e., γ(X ) = γ(X ) and ρ(X ) = ρ(X ), if there is an entailment g 0 in X such that g

D = g 0 . Proof. g D = g 0 implies D({g}) = D({g 0 }) (Definition 13 and 14), while {g 0 } ⊆ X implies D(X ) ⊆ D({g 0 }) (Prop- erty 2), thus D(X ) ⊆ D({g}); X = {g} ∪ X implies D(X ) = D(X ) ∩ D({g}) (Definition 13); thus D(X ) = D(X ); thus X and X have the same FTI values (v f ) in Algorithm 2. Meanwhile, g 0 ∈ X and g D = g 0 imply f e (X ) = f e (X )
in Algorithm 2 (Definition 12 and 14); Thus D(X ) = D(X ) implies X and X have the same evidence embedding (v e ) in Algorithm 2. X and X have the same v e and v f imply γ(X ) = γ(X ) and ρ(X ) = ρ(X ).

Synchronized entailments are common especially when a large number of external axioms are imported. The time complexity of computing all the synchronized entailment pairs is O( n(n -1) /2), where n is the size of ∪ D∈D G(D). Meanwhile, with the transitivity property of synchronized entailment (cf. Property 3), we can merge two sets of synchronized entailments if an entailment of one set is synchronized with an entailment of another set, thus quickly calculating clusters of synchronized entailment. Property 3. (Transitivity of Synchronized Entailment) In Definition 14, (g

1 D = g 2 ) ∧ (g 2 D = g 3 ) → (g 1 D = g 3 ).
Heuristics can also be developed to approximately search the core contexts. For example, in extending a core context, we can either ignore entailments that are not particular narrators, or only add entailments that are semantically close to the core context (e.g., about the same carrier). They are left in our future work.

Evaluation

Experiment Setting. In the experiment, we predict whether a flight's departure will be delayed or not, with observations of recent and surrounding flights, as well as meteorology2 . The target entailment is set to DelayedDep(d) for all the learning domains, and carrier, origin airport and destination airport are used to identify a learning domain. 92 learning domains composed of 10 airports and 11 carriers in US are adopted. One learning domain has 1,880 to 9,500 LSOs extracted from 01/01/2010 to 07/01/2017. 8372 transfers are evaluated, where FTI is measured with Area Under ROC Curve, a widely used performance metric for the prediction model. In deciding a valid evidence, the coefficient threshold in Definition 7 is set to 0.1. We report results of (i) average number of root entailments, root individuals and external axioms per learning domain (Table 1), (ii) general factors (Figure 3), (iii) particular narrators (Figure 4) and (iv) core contexts (Figure 5), and at the same time analyze the impact of entailment reasoning and external knowledge importing on the explanation. External Knowledge. Table 1 presents that the size of root entailments (including root concept assertion entailments and role assertion entailments), root individuals and external axioms all decreases when the parameters (σ, κ, τ ) increase from P1 to P5. When (σ, κ, τ ) are set to P5, only 9.3% of the individuals are selected as root individuals, reducing external axioms from around 21, 000 to 615.

On the other hand, importing less external axioms by selecting root individuals does not harm the richness of explanatory evidence. Firstly, Figure 3 (page 8, more explanatoins below) reports that setting (σ, κ, τ ) to P4 (6271 external axioms imported) does not help infer more confident general factors than P5 (615 external axioms imported). In contrast, it reduces the confidence of general factors d new , d obs and d inv by (7.5%, 51.3%, 52.4%), when they are measured with external axioms alone. It means those additional external axioms in setting P4 bring more noise than effective information to general factors in explaining the transferability. Secondly, Figure 4 [Left] (page8) reports that the richness of particular narrators is kept from setting P4 to P5, as the total number decreases very little (e.g., positive entailments decrease from 833 to 828). "the origin airport of both source and target learning domains is in the east part of US" (e2) and (ii) "the carriers of both source and target learning domains are public companies" (e4). We can explain the negative transfer F (DL,ORD,LAX)→(B6,LAX,JF K) with descriptions like "the carriers of both source and target learning domains are small companies; it's hard to transfer a feature between two learning domains with small carriers" (e10).

- ) and 5.9% (62.5%) higher absolute coefficient than using local ABox entailments alone and using external axioms alone respectively. This verifies external knowledge's positive impact on the confidence of general factors. Meanwhile, Figure 4 [Right] shows that 44.4% of particular narrators use local ABox axioms only, while 61.1% and 38.9% of them involve external axioms and entailed entailments respectively. This verifies the positive impact of entailment reasoning and external knowledge on the quality of particular narrators.

Core Context. Figure 5 [Left] and [Middle] present that the core contexts composed of 2 to 4 entailments have much higher absolute coefficient than general factors and particular narrators. For example, the average coefficient of the top k% most positively correlated core contexts ranges from (0.18, 0.28, 0.33) to (0.35, 0.59, 0.78) when the dimension C is (2, 3, 4). They are more confident in explaining the transferability. For example, with the core context composed of locatedIn(des, CA), ListCar(car) and BigCar(car), whose coefficient is 0.35, we can explain the positive transfer F (DL,ORD,LAX)→(AA,ORD,SF O) more confidently by "The carrier of both source and target learning domain belongs to big and list airline companies, and their destination airports are both located in California".

Figure 5 [Right] reports that (19.9%, 11.6%, 4.8%) of all the (2, 3, 4)-dimension entailment subsets have significant correlation analysis with FTI (i.e., ρ(X ) < 0.05), while (13.6%, 1.8%, 0.2%) are valid core contexts (i.e., ρ(X ) < 0.05 and γ(X ) ≥ 0.1). On one hand, as the dimension increases, the percentage of valid core contexts significantly decreases. On the other hand, the fact that a very large part of the entailment subsets have insignificant correlation analysis verifies that EearlyStop in core context searching (Algorithm 3) is effective. For example, when the dimension of the current core context is 4, it avoids 95.2% of the traversing for core contexts with higher dimension. Discussion and Lessons. The evaluation presents the explanatory evidence's confidence and percentage (i.e., the probability of being available as evidence). For confidence, we have core contexts > general factors d new and d obs > entailment narrators > general factor d inv , while for percentage (of being available), we have general factors (100%) > entailment narrators (30.4% in average) > core contexts (15.6% in average). General factors can successfully explain any negative transfers, but fail to provide confident evidence for positive transfers. Core contexts, especially those with high dimensions, have very high confidence but the percentage decreases quickly as the dimension grows. For both high confidence and availability, all the three kinds of evidence need to be used together.

The evaluation also analyzes the positive impact of our techniques, including (i) root individual selection, which saves much computation but keeps high quality evidence, (ii) entailment reasoning and external axiom importing, which enrich the evidence and improve the percentage, and (iii) early stop strategy in core context searching, which significantly reduces unnecessary searching.

The explanations lead to insights of feature transfer for users without ML expertise, and in turn allow them to further improve a transfer learning approach with more optimized settings. For a specific target domain, the explanations can answer the question of what to transfer by comparing the evidence of different source learning domains. Meanwhile, we can infer explanatory evidence for different features such as different Conv layers of a CNN architecture. Thus for a specific pair of source and target learning domains, we can answer the question of when to transfer by selecting a feature that maximizes the positive evidence.

Related Work

ML explanation has been studied for years, mainly including model interpretation (i.e., understanding how decisions are made) and prediction justification (i.e., justifying why a particular decision is good) [START_REF] Biran | Explanation and justification in machine learning: A survey[END_REF]. In this section, we first overview the above two aspects, and then introduce the state-of-the-art in transfer learning explanation and human-centric explanation. Model Interpretation. Some ML models are inherently interpretable. One type is sparse linear models such as Supersparse Linear Integer Models [START_REF] Ustun | Supersparse linear integer models for optimized medical scoring systems[END_REF]. These models' variable coefficients can present how much each variable contributes to the decision. Another type is rule-based models such as sparse Decision Tree [START_REF] Wu | Beyond sparsity: Tree regularization of deep models for interpretability[END_REF] and Bayesian Rule Lists [START_REF] Letham | Interpretable classifiers using rules and bayesian analysis: Building a better stroke prediction model[END_REF]. They can explain the decision inference procedure with internal probabilities and rules.

To interpret black-box models, visualization techniques have been applied. For example, (Zeiler and Fergus 2014) visualized the hidden layer output of a CNN to understand the feature representation of data. For another example, [START_REF] Jakulin | Nomograms for visualizing support vector machines[END_REF] proposed the algorithm nomograms to visualize Support Vector Machines. Recent advances in model interpretation include (i) attention mechanism for weighting the importance of different input parts (Qin et al. 2017), (ii) reasoning-based consistent sample selection for stream learning [START_REF] Chen | Learning from ontology streams with semantic concept drift[END_REF]), (iii) data distribution summarization with prototypes and criticisms [START_REF] Kim | Examples are not enough, learn to criticize! criticism for interpretability[END_REF], etc. Prediction Justification. A specific prediction can be explained by evaluating the effect of each meaningful input variable [START_REF] Biran | Human-centric justification of machine learning predictions[END_REF]. It can be directly calculated in an interpretable model or estimated with input isolation strategies such as omitting a subset of input [START_REF] Robnik | Explaining classifications for individual instances[END_REF][START_REF] Martens | Explaining data-driven document classifications[END_REF]. For complex and black-box models, [START_REF] Baehrens | How to explain individual classification decisions[END_REF][START_REF] Tulio Ribeiro | Why should i trust you?: Explaining the predictions of any classifier[END_REF] proposed to approximate them by multiple linear models which have interpretable data representations and local fidelity.

Generating description text is another approach to justify predictions. Recent advances include (i) caption generation for visual decisions such as image classification [START_REF] Lisa | Generating visual explanations[END_REF], (ii) text description of effective ML features [START_REF] Biran | Human-centric justification of machine learning predictions[END_REF], etc. Transfer Learning Explanation. Current studies on transfer learning explanation mainly lie in transferability analysis. Problems like when and what to transfer have been investigated in both theory and practice [START_REF] Sinno | A survey on transfer learning[END_REF][START_REF] Weiss | A survey of transfer learning[END_REF]). Recent advances include (i) experimental quantification of the generality (transferable) and specificity (untransferable) of CNN feature [START_REF] Yosinski | How transferable are features in deep neural networks? In Advances in neural information processing systems[END_REF], (ii) theoretic justification of the relation between feature structure similarity and transferability (Liu et al. 2017), etc. These attempts of transferability analysis definitely benefit ML experts, but will fail to explain the learned model or justify the prediction to common people. The understanding to transferability is encoded in a machine understandable way (e.g., loss function) to enhance learning. The explanations are neither represented in a human understandable format nor enriched with common sense knowledge. To the best of our knowledge, there are currently no studies for humancentric transfer learning explanation. Human-centric ML Explanation. Human-centric ML explanation aims at interpreting learned models or justifying predictions with background or common sense knowledge in a human understandable way [START_REF] Biran | Human-centric justification of machine learning predictions[END_REF]. Most of the current studies are based on corpuses. [START_REF] Lisa | Generating visual explanations[END_REF]) utilized external corpuses to generate captions to explain image classification decisions, while [START_REF] Biran | Human-centric justification of machine learning predictions[END_REF] used Wikipedia articles to describe effective features of a ML model. Few studies utilize semantic data in human-centric explanation. [START_REF] Tiddi | Dedalo: Looking for clusters explanations in a labyrinth of linked data[END_REF] proposed a framework to traverse Linked Data and use graph path commonalities to explain data clusters.

The current studies incorporate external knowledge, but ignore expressive knowledge e.g., ontology and their reasoning capability. It lacks a general knowledge representation and reasoning framework to utilize local ontologies and external knowledge bases for human-centric ML explanation. This work bridges the above gap and is among the first to study human-centric transfer learning explanation.

Conclusion and Outlook

In this study, we address the problem of human-centric transfer learning explanation. Our ontology-based framework exploits the reasoning capability and external knowledge bases like DBpedia to infer different kinds of human understandable explanatory evidence, including general factors, particular narrators and core contexts. It allows users without ML expertise to have a good insight of positive / negative transfers, and to answer the questions of what and when to transfer for more optimized transfer learning settings. The quality of explanatory evidence, including the confidence and availability, and the effect of our methods, are evaluated with US flight departure delay prediction, where features learned by CNNs are transferred. In the future, we will exploit more efficient core context search algorithms and the impact of semantic expressivity and compilation [START_REF] Pan | Approximating OWL-DL Ontologies[END_REF][START_REF] Pan | Tractable approximate deduction for OWL[END_REF], with further experiments and evaluations.

  1 https://www.w3.org/TR/owl2-overview/ ontology T , A . By entailment reasoning with both TBox and ABox axioms, we get a complete set of ABox entailments i.e., G(O) for modeling the input sample. Example 1. (An LSO on Departure Flights ) Figure 1 displays some axiom examples of an LSO annotated by property-value pairs S := {dat : 01/01/2018, car : DL, ori : LAX, des : JF K}. The LSO corresponds to one ML input sample that is related to a flight departure from Los Angeles International Airport (LAX) to John F. Kennedy International Airport (JFK) on 01/01/2018, carried by Delta Air Lines (DL). The examples include some TBox axioms (1)-(

Figure 1 :

 1 Figure 1: Ontology Examples of An LSO on Departure Flights Definition 2 revisits the concept of learning domain in ML and defines target entailment. A learning domain is also annotated by property-value pairs (cf. Definition 3). Definition 2. (Learning Domain and Target Entailment) A learning domain D = O, g t consists of a set of LSOs O that share the same TBox T , and a target entailment g t whose truth in an LSO is to be predicted. Definition 3. (Learning Domain Annotation) The annotation property-value pairs of the learning domain D in Definition 2 are defined as S(D) = (∩ T ,A ,S ∈O S) ∪ {t e : g t }. Example 2. ( A Learning Domain on Departure Flights) Now we consider a learning domain D 0 = O 0 , g t 0 , where the target entailment g t 0 is DelayedDep(d) and O 0 contains the LSO in Example 1, as well as many similar LSOs with the same (carrier, origin airport and destination airport), but different dates. The domain annotation S(D 0 ) is {car : DL, ori : LAX, des : JF K, t e : DelayedDep(d)}.

Example 4 .

 4 (Explanatory Evidence) In US flight departure delay prediction, we consider three learning domains whose target entailments are all DelayedDep(d): D (DL,ORD,LAX) for Delta Airlines from ORD to LAX, D (B6,LAX,JF K) for JetBlue from LAX to JFK and D (AA,ORD,SF O) for American Airlines from ORD to SFO, as well as a negative transfer from D (DL,ORD,LAX) to D (B6,LAX,JF K) and a positive transfer from domain D (DL,ORD,LAX) to D (AA,ORD,SF O) . We explain the two transfers with (i) general factors e.g., the percentage of shared entailments between D (DL,ORD,LAX) and D (AA,ORD,SF O) (D (DL,ORD,LAX) and D (B6,LAX,JF K) ) is high (low), (ii) particular narrators e.g., entailment "locatedIn(ori, East)" plays a positive role in the transfer from D (DL,ORD,LAX) to D (AA,ORD,SF O) , and (iii) core contexts, e.g., the entailment set composed of "hasOri(dep, ORD)" and "locatedIn(des, CA)" has a high impact on the positive transfer from D (DL,ORD,LAX) to D (AA,ORD,SF O) .

Figure 2 :

 2 Figure 2: Transfer Learning with Convolutional Neural Networks.

  Each learning domain is extended with external knowledge from existent knowledge bases (KBs), such as DBPedia[START_REF] Auer | DBpedia: A nucleus for a web of open data[END_REF], for richer common sense knowledge about the prediction application. The extension includes two steps: (i) root individual selection, and (ii) external knowledge matching and importing. Root Entailment and Root Individual. We define those entailments that play an import role in predicting the truth of the target entailment as Root Entailments, denoted as G R . Root entailments include Frequent Entailments (cf. Definition 9) and Effective Entailments (cf. Definition 10). Those individuals that are involved in at least one root entailment are defined as Root Individuals, denoted as I R . Definition 9. (Frequent Entailment) Given a learning domain D = O, g t and its local entailment closureG(O) = ∪ O∈O G(O), g ∈ G(O) is a frequent entailment if |{O ∈ O|g ∈ G(O)}| /|O| ≥ σ,where |•| calculates the set cardinality, σ is a parameter in [0, 1]. Definition 10. (Effective Entailment) In Definition 9, a κ-element entailment subset G κ ⊆ G(O) is a set of effective entailments if r e + r i ≥ τ , where r e = O ∈ O|Gκ ∪ g t ⊆ G(O) /|O| and r i = O ∈ O|(Gκ ∪ g t ) ∩ G(O) = ∅ /|O| , κ ≥ 1 and τ ∈ [0, 1] are parameters.

  Example 5. (Root Individual Selection) We consider the learning domain D (B6,LAX,JF K) in Example 4, "hasOri(d, LAX)" is a frequent entailment as it appears in all the LSOs of the domain, while the entailment subset composed of "hasRecDep(d, d 2 )", "DelayedDep(d 2 )" and "hasCarrier(d 2 , AA)" are effective as they co-exist or co-inexist with the target entailment "DelayedDep(d)" in a large part of LSOs. The individuals d, d 2 , LAX and AA that are involved in the above entailments are root individuals.

Algorithm 1 :

 1 ExternalAxiomsImport D, I R , B, C 1 Input: (i) A learning domain D = g t , O with TBox T , (ii) Root individuals I R , (iii) An external KB B, (iv) Constraint axioms Tc 2 Result: G(D): Entailment closure of the learning domain 3 begin 4 Ae := ∅; % Init. of the external axiom set 5 foreach root individual i ∈ I R do 6 N i ← (B, i) % Match external entity by name. 7 foreach entity e ∈ N i do 8 V ← (B, e) % Extract concpepts and roles 9 K ← (V, T ) %Transform to external axioms 10 % Consistency checking 11 if O ∪ Tc ∪ K |= ⊥ for ∀O ∈ O then 12 Ae := Ae ∪ K 13 break % Adopt the first matched entity 14 G(D) ← ∪ O∈O G(O ∪ Ae) % Entailment reasoning 15 return G(D);

  eral factors: d new , d obs and d inv , which are directed domain change rates from the source learning domain to the target, with new, obsolete and invariant entailments respectively. Definition 11. (Entailment-based Domain Change Rates) Given source learning domain D α and target learning domain D β in transfer learning (Definition 6), the entailmentbased domain change rates from D α to D β are defined as: |{g|g∈G(D β ),g ∈G(Dα)}| |G(D β )| d obs = |{g|g∈G(Dα),g ∈G(D β )}| |G(Dα)| d inv = |{g|g∈G(Dα),g∈G(D β )}| |G(Dα)∪G(D β )| (26) where the operation |•| calculates set cardinality. Example 8. (Entailment-based Domain Change Rates) In the transfer from domain D (DL,ORD,LAX) (entailment closure size: 25180) to domain D (B6,LAX,JF K) (entailment closure size: 13412) in Example 4, the sizes of new, obsolete and invariant entailments are 11419, 23187 and 1193. Thus the domain change rates d new , d obs and d inv are calculated as

  Definition 12. (Directed Entailment Co-existence) Given source learning domain D α and target learning domain D β in transfer learning (Definition 6), Directed Entailment Co-existence (DEC) of an entailment set G ⊆ G(D α ) is 1 if G ⊆ G(D β ) and 0 otherwise. When |G| = 1, it calculates the DEC of a single entailment. Example 9. (Directed Entailment Co-existence) In the transfer from learning domain D (DL,ORD,LAX) to learning domain D (AA,ORD,SF O) in Example 4, the particular narrator of "hasOri(d, ORD)" is embedded into 1, while the core context composed of "hasOri(d, ORD)" and "hasCarrier(d, DL)" is embedded into 0.

Algorithm 2 :

 2 CorrelativeReason D, X 1 Input: (i) A set of learning domains D = {D k |k = 1, ..., m} and (ii) an explanatory evidence X 2 Result: Correlation coefficient γ(X ) and p-value ρ(X ) 3 begin 4 ve := ∅ % Init. of evidence value array 5

Algorithm 3 :%

 3 CoreContextSearch D, X 1 Input: (i) A learning domain set D, (ii) A candidate core context evidence X 2 Result: Records of (evidence, coefficient, p-value) 3 begin 4 G0 := ∪ D∈D G(D) % Cal. entailment closure 5 if X = ∅ then 6 Traverse core contexts with two entailments 7 foreach g1 ∈ G0, g2 ∈ G0 such that g1 = g2 do 8 CoreContextSearch(D, {g1, g2})

Figure 5 :

 5 Figure 5: [Left] ([Middle]) Average Correlation Coefficient of Top k% Most Positively (Negatively) Correlated Core Contexts, [Right] Percentage of Valid Core Contexts (i.e., ρ(X ) < 0.05 and γ(X ) ≥ 0.1) , with Dimension C = 2, 3, 4.

  Given a learning domain D = O, g t , whose LSOs O are divided into two disjoint sets O and O , a supervised learning task within D, denoted by L = D, O , O , M , is a task of learning a model M with O and g t to predict the truth of g t in each O in O . Here, O is called a training LSO set, while O is called a testing LSO set.

	. It reduces a prediction problem to a supervised learn-
	ing problem with steps of learning and predicting.
	Definition 4. (Within Domain Supervised Learning)

β and O β , transfer learning from D α to D β , denoted by F α→β is a task of learning a model M α→β from O α , g t α , O β and g t β to predict the truth of g t β in each LSO in O β . F α→β is defined as a positive transfer if M α→β outperforms M β which is learned within domain D β ac

  . cording to Definition 4, and a negative transfer otherwise.

	Definition 6. (Transfer Learning)
	Given two learning domains D α = O α , g t α and D β = O β , g t β , where the LSOs of domain D β are divided into
	two disjoint sets O

Table 1 :

 1 Average Number of Root Entailments, Root Individuals and External Axioms per Learning Domain. There are a high percentage of new and obsolete entailments from domain D (DL,ORD,LAX) to D (B6,LAX,JF K) ". On the other hand, there is positive correlation between general factor d inv and FTI (ρ(X ) < 0.05), which is opposite to d obs and d new , but the correlation is weak ( γ(X ) < 0.08). Therefore, sharing a large percentage of entailments is not a confident evidence to explain a positive feature transfer. General Factors d new , d obs and d inv Calculated with Different Knowledge Parts, and Parameter Settings of P4 and P5.

	TBox Axi.:	Concept Ast.	Role Ast.	Individual:	Ext. Axi.:
	541	Ent.: 1824	Ent.: 4528	1159	∼ 21 K
	Parameters	Root	Root Role	Root	External
	(σ, κ, τ )	Concept Ast.	Ast. Ent.	Individual	Axioms
		Ent.
	P1:(.90, 1, .40) 1105 (61%)	3805 (84%)	1103 (95%)	∼ 20 K
	P2:(.93, 1, .43)	990 (54%)	3459 (76%)	1080 (93%)	∼ 19 K
	P3:(.96, 1, .46)	540 (30%)	1816 (40%)	872 (75%)	∼ 16 K
	P4:(.99, 1, .49)	305 (17%)	980 (22%)	510 (44%)	6271
	P5:(.99, 2, .49) 157 (8.6%)	402 (8.9%)	108 (9.3%)	615
	General Factors. Figure 3 (Local ABox Ent. + Exter-
	nal Axioms (P5)) presents that general factors d obs and
	d new have a significant negative impact on the feature's
	transferability (γ(X ) < -0.2). Thus we can explain a
	negative transfer like F (DL,ORD,LAX)→(B6,LAX,JF K) with
	explanations like "

0.19 -0.14 -0.09 -0.04

		0.16		P4: <Positive: 833 (6.6% ) , Negative: 1340 (10.6%)>	Percentage (%)
				P5: <Positive: 828 (11.3% ) , Negative: 1329 (19.1%)>	of Narrators
		0.11									with (1) local
	Coefficient	0.01 0.06	e1	e2	e3	e4	e5	e6	e7	e8 e8: hasOri(dep, SFO) e9 e10 e11 e9: inCADep(dep) e10: SmallCarrier(car) e11: hasHub(car, des)	ABox axiom only, (2) External axioms, (3) 60.00 70.00 Entailed Ent.
					e5, 6, 7: P-value >= 0.05 Not Correlated		50.00
			e1: hasOri(dep, CLT)						40.00
			e2: locatedIn(ori, East)	e5: locatedIn(des, CA)
			e3: BigCarrier(car)		e6: hasCar(dep, VX)		30.00
			e4: ListedCarrier(car)	e7: serveFor(ori, NewYork)	1	2	3
		Figure 4: Examples and Statistics of Particular Narrators.
	Entailment Reasoning and External Knowledge. Figure 3
	(Local ABox Ent. Only vs. External Axioms (P5) vs. Local
	ABox Ent. + External Axioms (P5)) shows that combing lo-
	cal ABox entailments and external axioms for d obs (d new )
	achieves 178.9% (56.0%

Codes and data: https://github.com/ChenJiaoyan/X-TL

Acknowledgments

The work was partially funded by the research center SIR-IUS, the EPSRC project DBOnto, the EU Marie Curie K-Drive project (286348) and NSFC61673338.


Rory Mc Grath
email: rory.m.mc.grath@accenture.com

Luca Costabello
email: luca.costabello@accenture.com

Chan Le Van
email: chan.v.le.van@accenture.com

Paul Sweeney
email: paul.p.sweeney@accenture.com

Farbod Kamiab
email: farbod.kamiab@accenture.com

Zhao Shen
email: zhao.shen@accenture.com

Freddy Lécué
email: freddy.lecue@inria.fr

Interpretable Credit Application Predictions With Counterfactual Explanations

We predict credit applications with off-the-shelf, interchangeable black-box classifiers and we explain single predictions with counterfactual explanations. Counterfactual explanations expose the minimal changes required on the input data to obtain a different result e.g., approved vs rejected application. Despite their effectiveness, counterfactuals are mainly designed for changing an undesired outcome of a prediction i.e. loan rejected. Counterfactuals, however, can be difficult to interpret, especially when a high number of features are involved in the explanation. Our contribution is two-fold: i) we propose positive counterfactuals, i.e. we adapt counterfactual explanations to also explain accepted loan applications, and ii) we propose two weighting strategies to generate more interpretable counterfactuals. Experiments on the HELOC loan applications dataset show that our contribution outperforms the baseline counterfactual generation strategy, by leading to smaller and hence more interpretable counterfactuals.

Introduction

Explaining predictions of black box models is of uttermost importance in the domain of credit risk assessment Bruckner [2018]. The problem is even more prominent given the recent right to explanation introduced by the European General Data Protection Regulation [START_REF] Goodman | European union regulations on algorithmic decision-making and a "right to explanation[END_REF], and a must due to regulation in the financial domain. A common approach to explain black box predictions focuses on generating local approximations of decisions. If f is a machine learning model taking the features X and mapping them to targets Y , then the goal is to find a subdomain of the feature variables and over that domain approximate f ∼ g, where g is an interpretable and easy to understand function. There has been recent interest in model-agnostic methods of explainability. These methods look to create an explainer that should be able to explain any model treating the underlying model as a black box. Ribeiro et al. [2016a].

This paper focuses on Counterfactual Explanations [START_REF] Wachter | Counterfactual explanations without opening the black box: Automated decisions and the GDPR[END_REF], one of these modelagnostic methods. A counterfactual explanation may justify a rejected loan application as follows:

Your application was denied because your annual income is $30,000 and your current balance is $200. If your income had instead been $35,000 and your current balance had been $400 and all other values remained constant, your application would have been approved.

The explanation describes the required minimum change in inputs to flip the decision of the black box classifier. Note that the latter remains a black box: it is only through changing inputs and outputs that an explanation is obtained. to describe what it takes to flip the decision of a classifier, hence they poorly address the case in which the decision was satisfactory from an end user perspective (e.g. loan approved). Problems also arise when assessing the interpretability of counterfactuals: the generated counterfactuals often suggest to change a high number of features, therefore leading to less intelligible explanations. For example, counterfactuals generation strategies do not take into account the importance of the dataset features, thus underestimating or overestimating certain dimensions. This problem is of particular importance given that it has been showed that human short-term memory is unable to retain a large number of information units Vogel et al. [2001], [START_REF] George | The capacity of visual short-term memory is set both by visual information load and by number of objects[END_REF]. This remains as problematic in the finance domains of loan approval when data scientis and regulators are in the loop.

We predict loan applications with off-the-shelf, interchangeable black-box estimators, and we explain their predictions with counterfactual explanations. To overcome the aforementioned problems, we present the following contribution:

• Positive Counterfactuals: in case of a desired outcome, we interpret counterfactuals as a safety margin, i.e. a tolerance from the decision boundary. Such counterfactual explanations for positive predictions answer the question "How much was I accepted by?".

• Weighted Counterfactuals: inspired by Huysmans et al. [START_REF] Huysmans | An empirical evaluation of the comprehensibility of decision table, tree and rule based predictive models[END_REF], we use the size of explanations as a proxy to measure their interpretability. To obtain more compact (and hence more intelligible) counterfactuals we introduce weights in the their generation strategy. We propose two weighting strategies: one based on global feature importance, the other based on nearest neighbours.

We experiment on a credit application dataset and show that our weighted counterfactuals generation strategies lead to smaller counterfactuals (i.e. counterfactuals that suggest to change a smaller number of features), thus delivering more interpretable explanations.

Related Work

Local Interpretability A number of works focus on explaining single predictions of machine learning models, rather than the model as a whole. This task is also known as local interpretability. White box models come with local explanations by design: traditional transparent design approaches include decision trees and rule extraction [START_REF] Guidotti | A survey of methods for explaining black box models[END_REF], [START_REF] Molnar | Interpretable Machine Learning[END_REF]. In that respect some works such as [START_REF] Craven | Extracting tree-structured representations of trained networks[END_REF] Although they evaluate the predictive power of their models they do not attempt to explain their predictions, either locally or globally. To the best of our knowledge, no work in literature focuses on local interpretability for black box models applied to credit risk prediction.

Preliminaries: Counterfactual Explanations

A counterfactual explanation describes a generic causal situation in the form:

Score y was returned because variables X had values (x 1 , x 2 ...) associated with them. If X instead had values (x ′ 1 , x ′ 2 , ...), and all other variables had remained constant, score y ′ would have been returned.

Counterfactuals do not need to know the internal structure or state of model or system (e.g. neural network, logistic regression, support vector machine, etc). We treat f as a black box that takes the feature vector x and generates the outcome y, and we determine what is the closest x ′ to x that would change the outcome of the model from y to the desired y ′ (Figure 1). When generating counterfactuals it is assumed that the model f , the feature vector x and the desired output y ′ are provided. The challenge is finding x ′ , i.e. an hypothetical input vector which falls close to x but also for which f (x ′ ) falls sufficiently close to y ′ . Generating Counterfactuals. We generate counterfactual explanations by calculating the smallest possible change (∆X) that can be made to the input X, such that the outcome flips from y to y ′ . We generate counterfactuals by optimizing the following loss function L, as proposed by [START_REF] Wachter | Counterfactual explanations without opening the black box: Automated decisions and the GDPR[END_REF]:

L(x, x ′ , y ′ , λ) = λ( f (x ′ ) -y ′ ) 2 + d(x, x ′ ) (1) arg min x ′ max λ L(x, x ′ , y ′ , λ) (2)
where x is the actual input vector, x ′ is counterfactual vector, y ′ is the desired output state, f (...) is the trained model, λ is the balance weight. λ balances the counterfactual between obtaining the exact desired output and making the smallest possible changes to the input vector x. Larger values for λ favor counterfactuals x ′ which result in a f (x ′ ) that comes close to the desired output y ′ , while smaller values lead to counterfactuals x ′ that are very similar to x. The distance metric d(x, x ′ ) measures ∆x, i.e. the amount of change between x and x ′ . We use the Manhattan distance weighted feature-wise with the inverse median absolute deviation (MAD) 3. Such metric is robust to outliers, and introduces sparse solutions where most entries are zero [START_REF] Wachter | Counterfactual explanations without opening the black box: Automated decisions and the GDPR[END_REF]. Indeed, the ideal counterfactual is one in which only a small number of features change and the majority of them remain constant. The distance metric d(x, x ′ ) can be written as:

d(x, x ′ ) = p j=1 |x j -x ′ j | M AD j , (3)
M AD j = median i∈{1,...,n} x i,jmedian l∈{1,...,n} (x l,j ) .

(4)

To generate counterfactuals we adopt the iterative approach described in Algorithm 1. We optimize L with the Nelder-Mead algorithm, as suggested in [START_REF] Molnar | Interpretable Machine Learning[END_REF]. We constrain the optimisation with a tolerance ε s.

t | f (x ′ ) -y ′ | ≤ ε.
The value for ε depends on the problem space and is determined by the range and scale of y.

Step 3 iterates over λ until the ε constraint is satisfied. A check is performed for a value greater than ε as increasing λ will place more weight on obtaining an f (x ′ ) closer to the given desired output y ′ . Once an acceptable value for λ is obtained for the given x and y ′ a set of counterfactuals can be obtained by repeating steps 1 and 2 with the calculated λ. Note that we constrain the features manually, since the heuristic in Algorithm 1 and the adopted optimization algorithm are designed for unconstrained optimization.

Contribution

In this section we describe our two main contributions made towards the explainability of black box machine learning pipelines that predict credit decisions. Using positive counterfactuals we explain why a loan was accepted and provide details that help inform an individual when making future financial decisions. Next we present weighted counterfactuals that aim at personalizing the counterfactual recommendations that are provided to individuals that received an undesirable outcome (i.e. their loan was denied).

Counterfactuals for positive predictions

In order to explain why applications were accepted, we applied counterfactuals to the scenario where the individual received the desired outcome, i.e positive counterfactuals. Here instead of answering the question "Why wasn't I accepted?" we focus on the question of "How much was I accepted by?". Such approach informs the individual about the features and value ranges that were important for their specific application, thus favouring more informed decisions about potential future financial activities. For example, if the individual is considering an action that may temporarily increase their number of delinquencies then, armed with positive counterfactuals, they will have a better understanding of the impact on future loan applications.

In the binary classification case we achieve positive counterfactuals by setting the target y ′ to be the decision boundary i.e P (y = 1) = 0.5. This allows us to identify the locally important features that would push the individual to the threshold of being accepted. Another way of viewing this is that these are the features that locally contribute to the desired outcome.

We present this information to the individual and display it as tolerance. In Figure 2 this is illustrated by a dashed line. Given that future actions do not reduce the indicated features below the dashed line, and all other features remain constant, then the individuals application should remain likely to be approved.

Weighted Counterfactuals

The general implementation of counterfactuals described in Section 3 assumes all features are equally important and changing each feature is equally viable. This, however, is not necessarily the case. For each feature its ability to change and the magnitude of the change may vary on a case by case bases. In order to capture this information and create more interpretable actionable recommendations, the generated counterfactuals need to take this into consideration. For example some individuals may be able to increase their savings, while others instead may find it easier to reduce their current expenses. There are also cases where some features may be fixed or immutable.

Features like the number of delinquencies in the last six months is historical and fixed. Recommending to change these types of features would be of little use. Our intuition is that promoting highly discriminative features during the generation of counterfactuals leads to more compact, hence better interpretable explanations [START_REF] Huysmans | An empirical evaluation of the comprehensibility of decision table, tree and rule based predictive models[END_REF].

We address these issues by introducing a weight vector θ to the distance metric defined in Equation 3. This vector promotes highly discriminative features.

d 2 (x, x ′ ) = p j=1 |x j -x ′ j | M AD j θ j , (5)
We propose two different strategies to generate these weight vectors. The first relies on the global feature importance, the second relies on a Nearest Neighbors approach. The goal is obtaining coun-terfactuals that suggest a smaller number of changes or focus on values that are relevant to the individual and have historically been shown to vary.

Global feature importance. We compute global feature importance using analysis of variance (ANOVA F-values) between each feature and the target, and we create a weight vector that promotes highly discriminative features. Our goal is obtaining a smaller set of features in the resulting counterfactual recommendation, thus obtaining more compact explanations.

K-Nearest Neighbors. The second approach uses K-Nearest Neighbors to find cases that are close to the individual but have achieved the desired results. Looking at the nearest neighbors and aggregating over the relative changes we build a weight vector θ that captures the locally important features for this individual that have historically been shown to change. Here we aim to find counterfactuals containing features that are more actionable by the individual. By using K-Nearest Neighbors approach these weights can be automatically learned when applied to new problem spaces.

Experiments

We perform a a binary classification task on a credit application dataset. We train a range of black box models and we explain their predictions with counterfactuals, the goal being explaining the classifier decision to reject or accept a loan application. We perform two separate experiments: first, we carry out a preliminary evaluation of the predictive power of our pipeline. This is not the primary focus of this paper, but it is a required step to gauge the quality of predictions. In a second experiment, we assess the size of counterfactuals generated by our weighted counterfactuals generation.

Experimental Settings

Dataset. We experiment with the HELOC (Home Equity Line of Credit) credit application dataset. Used in the FICO 2018 xML Challenge1 , it includes anonymized credit applications made by real homeowners. We drop highly correlated features and filter duplicate records. After pre-processing we obtain 9,870 records (of which 5,000 positives, i.e. accepted credit applications), and 22 distinct features.

Implementation Details. Our machine learning pipeline is written in Python 3.6. This includes preprocessing, training, counterfactuals generation, and performance evaluation. We use scikit-learn 0.20 for the black box classifiers2 . All experiments were run under Ubuntu 16.04 on an Intel Xeon E5-2620 v4 2.10 GHz workstation with 32 GB of system memory.

Results

Predictive Power As preliminary experiment, we assess the predictive power of four classifiers: logistic regression (LogReg), gradient boosting (GradBoost), support vector machine with linear kernel (SVC), and multi-layer perceptron (MLP). Logistic regression apart, the others fall within the black box category. We perform 3-fold, cross-validated grid search model selection over a number of hyperparameters. We adopt balanced class weights for logistic regression, exponential loss for gradient boosting, for each dataset. SVM uses balanced weights, C = .001. The neural network uses one hidden layer with 22 units. We use the logistic activation function. Where not specified, we rely on scikit-learn defaults. Results in Table 1 show the predictive power of the best models. Metrics are 3-fold cross-validated.

Counterfactuals Size

The preliminarily results of the different weighting strategies as described in Section 4.2 are presented in Table 2. We experiment with 5,000 loan applications in the dataset: we generate a counterfactual explanation for each of them, and compute the average counterfactuals size. Results show that both weighted strategies bring counterfactuals that have a smaller mean and standard deviation. We also observed that in general the average size of the counterfactual recommendations can vary dramatically for the same data given the underlying model. In general the global feature importance results in features with a lower mean and standard deviation.

We obtain explanations which are 11.2% smaller on average using the global feature importance strategy against the baseline. This is to be expected, as we promote more discriminative features and as a consequence less ancillary features are required. The benefit in the KNN approach is that the counterfactuals are weighted on the features that are locally important. Here we see that while they may not be the best approach they never perform worse than the baseline. The benefit of the weighting strategies comes with helping the optimization process converge on a local optimum when the underlying space is complex. We look to investigate this claim in future work.

Conclusion

We explain credit application predictions obtained with black box models with counterfactuals. In case of positive prediction, we show how counterfactuals can be interpreted as a safety margin from the decision boundary. We propose two weighted strategies to generate counterfactuals: one derives weights from features importance, the other relies on nearest neighbours. Experiments on the HELOC loan applications dataset show that weights generated from feature importance lead to more compact counterfactuals, therefore offering more compact and intelligible explanations for end users. Future work will focus on validating the effectiveness of our counterfactual explanations against human-grounded and application-grounded evaluation protocols (including the claim that smaller counterfactuals are indeed more interpretable). We will also experiment with weighting strategies that rely on model-specific feature importance, i.e. effect of feature perturbation on entropy of changes in predictions.

Figure 1 :

 1 Figure 1: Explaining black box predictions with counterfactuals

Algorithm 1 :

 1 Counterfactual generation heuristic 1 sample a random instance as the initial x ′ 2 optimise L(x, x ′ , y ′ , λ) with initial x ′ 3 while | f (x ′ )y ′ | > ε do 4 increase λ by step-size α 5 optimise L(x, x ′ , y ′ , λ) 1 with new x ′ 6 return x ′

Figure 2 :

 2 Figure2: Graphical depictions of a positive (a) and negative (b) counterfactual explanation. Note (a) answers the question "How much was I accepted by?" -thus leading to tolerances (highlighted in yellow), whereas (b) explains why the credit application was rejected. In this case the counterfactual explanation suggests how to increase (green, dashed) or decrease (red, striped) each feature.

  built surrogate models by interfacing complex models such as deep neural networks with more interpertable models such as decision trees. The authors aim at mimicing the behaviour of a complex model with a much simpler model for interpretability purpose. Other approaches are instead model-agnostic, and also address explanations of predictions of black box models. LIME generates local explanations from randomly generated neighbours of a record. Features are weighted according to distances from the record.Ribeiro et al. [2016b]. SHAP is another approach based on feature importance for each record Lundberg and Lee[2017]. Other recent lines of research rely on example-based explanations: Prototype Bien and Tibshirani[2011] and Criticism[START_REF] Kim | Examples are not enough, learn to criticize! criticism for interpretability[END_REF] Selection are two recent examples of this. Prototypes are tuples representative of the dataset, whereas criticisms are examples which are not well-explained by prototypes. Adversarial examples[START_REF] Kurakin | Adversarial examples in the physical world[END_REF] are another example-based approach, but they are designed to flip the decision of a black-box predictor rather than explaining it. Counterfactual explanations are also an example-based strategy, but unlike adversarial examples, they inform on how a record features must change to radically influence the outcome of the prediction.Interpretable Credit Risk Prediction Providing a comprehensive review of more than 20 years of research in credit risk prediction models is out of the scope of this paper. The survey by Lyn et al.[START_REF] Thomas | Credit scoring and its applications[END_REF] gives a comprehensive and up-to-date overview).[START_REF] Huang | Credit rating analysis with support vector machines and neural networks: a market comparative study[END_REF] briefly mention an explanation of predictive models for credit rating, but their survey limits to ranking features by importance with variance analysis. A more recent survey byLouzada et al. focuses on predictive powerLouzada et al. [2016] only. Instead, we list works that consider interpretability as a first-class citizen. A number of works rely on white box machine learning pipeline, mostly using decision trees and rules inference: Khandani et al.[START_REF] Amir E Khandani | Consumer credit-risk models via machinelearning algorithms[END_REF]

propose a pipeline to predict consumer credit risk with manual feature engineering and decision trees. The latter being an explainable model, we can consider this work as a rather interpretable approach. Florez-Lopez et al. adopt an ensemble of decision trees and explain predictions with rules López and Ramon-Jeronimo [2015]. Predictive power is encouraging, but there is no comparison against neural architectures. Martens at al. combine rule extraction with SVMs Martens et al. [2007]. Obermann et al. compare decision tree performance to grey and black boxes approaches on an insolvency prediction scenario Obermann and Waack [2015, 2016]. Other white-box approaches include Markov models for discrimination Volkov et al. [2017] and rule inference Xu et al. [2017]. Black box approaches show the most promising predictive power, to the detriment of interpretability. Danenas et al adopt SVM classifiers Danenas and Garsva [2015]. Addo et al. leverage a number of black-box models, including gradient boosting and deep neural architectures Addo et al. [2018].

Table 1 :

 1 Predictive power of the adopted black box classifiers. Best results in bold.

		HELOC
	Model	F1	Acc
	LogReg	0.72	0.73
	MLP	0.70	0.71
	GradBoost	0.72	0.74
	SVC	0.72	0.73

Table 2 :

 2 Average size (i.e. average number of features) of generated counterfactual explanations, for each adopted black box classifier. Smaller counterfactuals mean more interpretable explanations. Importance=global feature importance strategy, KNN=k-nearest neighbours. KNN uses k = 20. Best results in bold.

			HELOC
	Model	Baseline	Importance	KNN
	LogReg	4.86±1.84	3.95±1.69	4.71±1.72
	MLP	8.88±2.54	8.34±2.58	8.45±2.53
	GradBoost	1.5±0.6	1.49±0.58	1.5±0.58
	SVC	2.5±1.32	2.01±1.14	2.44±1.27

https://community.fico.com/s/explainable-machine-learning-challenge

http://scikit-learn.org/


Quan Yu

Hai Wan

Jiangtao Xu

Freddy Lécué

Liang Chang

Explanatory Diagnosis of an Ontology Stream via Reasoning About Actions

Explanatory diagnosis of an ontology stream aims to explain the changes hidden in the ontology stream by a sequence of actions. In this paper, we present a framework for explanatory diagnosis of an ontology stream, which allows the actions to be uncertain. In order to capture the semantics of actions, we introduce a new update operator and effect-guided bold-repair. By combining these operators with a query mechanism of description logics EL ++ supporting inconsistency-tolerant semantics, we present a formal definition for the explanatory diagnosis problem of ontology streams.

Introduction and Related Work

The task of diagnosing streams has received particular attentions from the semantic web and diagnosis communities. Freddy Lécué proposed a framework for diagnosing anomalies in an ontology stream [START_REF] Lécué | Towards scalable exploration of diagnoses in an ontology stream[END_REF]. However, the task of explanatory diagnosis is not achieved on stream evolution and its changes. McIlraith presented a formal characterization of explanatory diagnosis in the situation calculus [START_REF] Mcilraith | Explanatory diagnosis: Conjecturing actions to explain observations[END_REF]. Yu et al. also provided a formal characterization of explanatory diagnosis in dynamic epistemic logic [START_REF] Yu | Multi-agent epistemic explanatory diagnosis via reasoning about actions[END_REF]. In this paper, we focus on the task of explanatory diagnosis of an ontology stream, which uses a sequence of actions to explain changes occurring over time.

In a road traffic context, a change is a transition from "cleared" to "congested" road in a particular time of the stream and its diagnoses can be represented as an action sequence e.g., a road work, road incident. These explanations can be derived from different points of time and events between them, from which we can indirectly derive a new ontology stream of diagnoses. We introduce actions with either certain or uncertain effects. For the update of an ontology, we present a new update operator and an effect-guided boldrepair operator. Combined with (i) query mechanisms, supporting inconsistency-tolerant semantics, and (ii) both the semantics and dynamics of action sequences, we formalize the explanatory diagnosis problem in the framework of description logics (DL) EL ++ .

Background

Our approach is illustrated with DL EL ++ [START_REF] Lécué | Towards scalable exploration of diagnoses in an ontology stream[END_REF], which is the basis of many more expressive DLs. As usual DLs, answers to queries are formed by constants/ terms denoting individuals explicitly mentioned in the ABox [START_REF] Babak | Description logic knowledge and action bases[END_REF]. The definition of Union of Conjunctive Queries (UCQ) and Extension of UCQ (ECQ) can found in [START_REF] Babak | Description logic knowledge and action bases[END_REF].

An ontology stream [START_REF] Huang | Reasoning with multiversion ontologies: A temporal logic approach[END_REF] is defined as a dynamic and evolutive version of ontologies. An ontology stream O n m from point of time m to point of time n is a sequence of ontologies

(O n m (m), O n m (m + 1), • • • , O n m (n))
where m, n ∈ N and m < n.

Actions and Events

We first present the definitions of certain-effect action.

Definition 1 (Certain-effect Action) A certain action α is a pair (pre, effs) where:

• pre is the precondition of α, which is an ECQ;

• effs is the set of effects of α. Each effect eff has the form Q F where (i) Q is an ECQ, (ii) F is a set of facts to be added to the ABox. It is a set of non-ground ABox assertions which include constants in the initial ABox A 0 and free variables of Q.

Our definition of certain-effect action is an adaptation of that in [START_REF] Babak | Description logic knowledge and action bases[END_REF]. Then we present the uncertain effect action as follows:

Definition 2 (Uncertain-effect Action) An uncertain-effect action α u is a pair (pre, Effs) where:

• pre is the precondition of α u , which is an ECQ; • Effs = (p, effs) is a set of pairs. p describes the probability of effs after executing action α u , which means that each action α u has many action effects with different probability.

If not otherwise specified, we slightly abuse these two types of actions. We denote the precondition of α as pre α , the effects of α as effs α , and each effect of α as eff α . Note that the free variables of pre α are the parameters of action α. We use α[θ] = (pre α θ, effs α θ) to denote the action is substituted by partial substitution θ. We complement our framework by introducing the concept of events which decide whether an action is applicable in a current knowledge base.

Definition 3 (Event)

An event e i , occurring in time i of ontology stream O n m , is a set of non-ground ABox assertions which include constants and free variables. We use E n-1 m+1 (i) to denote events occurring in time i ∈ (m, n).

Update Operator

In the following, we introduce the definitions of (i) applicability for actions in an ABox, and (ii) update operator for describing how to update an ontology with an action. Definition 4 (Applicable) Given (i) a terminology T , (ii) an action α[θ] = (pre α θ, effs α θ), (iii) the current ABox A denoted by the empty set, and (iv) an event e. α [θ]

E α = (Q F )∈effs α σ∈ANS(Qθ,T,A) F θσ.

Repairs

Based on the defintion of Bold-repair [START_REF] Lembo | Inconsistency-tolerant semantics for description logics[END_REF], we present the definition of effect-guided bold-repair to fit our framework. While we use an action α to update an ABox A 0 , the resulting ABox A may become inconsistent. Suppose that the effects E α is with a higher priority, we repair the old ABox A 0 according to the fixed TBox T and E α .

Definition 6 (Effects Guided Bold-repair) An effects guided boldrepair (B rep in short) of an ABox A and an action α, denoted as

B rep (A, α), is a subset A of A -E α s.t.: • T ∪ E α ∪ A has a model; • there does not exists A such that A ⊂ A ⊆ A -E α and T ∪ E α ∪ A has a model.
We use the defintion of inconsistency-tolerant semantics to decide if the precondition holds before updating the ontology by an action.

Definition 7 (Inconsistency-tolerant Semantics) Let KB(T, A)

be an EL ++ knowledge base, α be an action, and q be an UCQ. We say that q is BR entailed by KB(T, A), written T, A |= BR q, if T, A |= q for every B rep A of A according to action α.

Theorem 1 Let KB(T, A) be an EL ++ knowledge base, let α be an action, and let q be an UCQ. Then the complexity of deciding whether T, A |= BR q is in EXPTIME.

Theorem 1 tells us that the complexity of query an UCQ over an EL ++ knowledge base under the inconsistency-tolerant semantics is in EXPTIME. The complexity of ECQ will not change.

Theorem 2 Let KB(T, A) be an EL ++ knowledge base, let α be an action, and let Q be an ECQ. Then the complexity of deciding whether T, A |= BR Q is in EXPTIME.

Explanatory Diagnosis of an Ontology Stream

Ontology Tree

Because the evolution from O n m (i) to O n m (i + 1) of an ontology stream O n m may be caused by the occurrence of one or several actions, we use ontology tree to capture the dynamics of semantics of an ontology stream. An ontology tree is a tree where each vertex is an ontology, each solid directed edge means an action, and each dotted directed edge indicates the effect of the action, the endpoint of which is the result of the updating the start point by an action.

Explanatory Diagnosis Problem

In our framework, fault concept is a special kinds of concept from a TBox, which is used for describing some phenomenons that we monitor from a stream perspective e.g., a congested road. • A s is the set of actions, including certain effect actions and uncertain effect actions;

• E n-1
m+1 is the set of events which occur from time m to n; • C(a) is an anomaly, which also called as an observation.

Definition 10 (Diagnosis) A diagnosis of a given explanatory diagnosis problem

P = O n m (m), O n m (n), A s , E n-1 m+1 , C(a) is a se- quence of actions δ, such that B rep (O n m (m) δ) |= C(a).
It should be noted that, for any nonempty action sequence δ and an action α, we have B rep (A 0 αδ) = B rep ((B rep (A 0 α)) δ).

Theorem 3 Explanatory diagnosis existence for EL ++ with a finite individual domain is decidable in EXPTIME in the size of the individual domain.

We assume familiarity with standard notions of Markov Decision Process (MDP) and value iteration [START_REF] Puterman | Markov decision processes: Discrete stochastic dynamic programming[END_REF]. A solution to a MDP is an optimal policy that maximizes value function of every state s ∈ S.

Considering that the set of action sequences is exponentially growing on a time basis, we can use MDP to compute the most likely action sequence, i.e., diagnosis. Given an explanatory diagnosis problem P, we can first generate the ontology tree, then solving problem P can be convert into a largest value path finding problem. It's easy to prove that there is a diagnosis δ which has the largest value function in all diagnoses as the following proposition.

Proposition 1 Each diagnosis of an explanatory diagnosis problem

P is an ontology stream.

Conclusion and Future Work

We have developed a framework of explanatory diagnosis of an ontology stream via reasoning about actions. The semantics is captured by DL evolving over time in an ontology stream. We introduced a new update operator, modeling actions with certain and uncertain effects, and presented query mechanisms supporting inconsistencytolerant semantics. Future work will extend our framework towards scalable explanatory diagnoses, specialty for large cities.

Definition 8 (

 8 Anomaly) Given a KB(T, A), suppose C is a set of predefined fault concepts. An assertion C(a) is called as an anomaly, if C ∈ C. Every concept in C is called a faulty concept. Definition 9 (Explanatory Diagnosis Problem) An explanatory diagnosis problem P is a tuple O n m (m), O n m (n), A s , E n-1 m+1 , C(a) which satisfies O n m (m) |= C(a) and O n m (n) |= C(a), where

  Kaminka et al. (Eds.) © 2016 The Authors and IOS Press. This article is published online with Open Access by IOS Press and distributed under the terms of the Creative Commons Attribution Non-Commercial License 4.0 (CC BY-NC 4.0).

	ECAI 2016
	G.A. doi:10.3233/978-1-61499-672-9-1596

is applicable in A according to e, if A ∪ e is consistent w.r.t. T and ∃σ s.t.T, A ∪ eθσ |= BR pre α θσ (The notion of |= BR will be introduced in the next subsection).

Definition 5 (Update with Action) Given an action α[θ] = (pre α θ, effs α θ), and the current O = (T, A). The results of application α[θ] in O is a new ontology O = (T, A ), which is defined as O = A α[θ], where A α[θ] = A ∪ E α , and

School of Mathematics and Statistics, Qiannan Normal University for Nationalities, China. email: yuquanlogic@126.com

[START_REF] Huang | Reasoning with multiversion ontologies: A temporal logic approach[END_REF] School of Data and Computer Science, Sun Yat-sen University, China.Hai Wan is the corresponding author, email: wanhai@mail.sysu.edu.

cn 3 Accenture Tech Labs, Ireland. INRIA,

France.[START_REF] Lembo | Inconsistency-tolerant semantics for description logics[END_REF] School of Computer Science and Information Security, Guilin University of Electronic Technology, China.

Acknowledgement

The authors is supported by NSF Grant Nos.(61463044, 61363030, and 61573386) and Grant No.LH [2014]7421 from the Joint Fund of the NSF of Guizhou province of China.