Adel Jaouen

Erwan Le Merrer

zoNNscan: a boundary-entropy index for zone inspection of neural models

Keywords: Decision boundaries, classification uncertainty, entropy, application of Monte Carlo sampling

The training of deep neural network classifiers results in decision boundaries whose geometry is still not well understood. This is in direct relation with classification problems such as so called corner case inputs. We introduce zoNNscan, an index that is intended to inform on the boundary uncertainty (in terms of the presence of other classes) around one given input datapoint. It is based on confidence entropy, and is implemented through Monte Carlo sampling in the multidimensional ball surrounding that input. We detail the zoNNscan index, give an algorithm for approximating it, and finally illustrate its benefits on three applications.

Introduction

Measures provide crucial insights in all computer science domains, as far as they allow to quantify and then analyze specific problems [START_REF] Wang | A universal image quality index[END_REF], or to serve as a building block for designing better algorithms [START_REF] Guo | Distance-from-boundary as a metric for texture image retrieval[END_REF]. In the blooming field of machine learning leveraging deep neural networks, open problems remain that are related to the position of decision boundaries in classification tasks. Little is known about the geometrical properties of those [START_REF] Fawzi | Classification regions of deep neural networks[END_REF][START_REF] Moosavi-Dezfooli | Universal adversarial perturbations[END_REF][START_REF] Van Den Berg | Some insights into the geometry and training of neural networks[END_REF]]. Yet, the close proximity of some inputs to those boundaries is getting more problematic with the increasing amount of critical applications that are using deep neural networks. The case of self-driving cars is an illustration of a critical application, where corner-cases have been recently found in production models (leading to wrong decisions) [START_REF] Pei | Deepxplore: Automated whitebox testing of deep learning systems[END_REF]; the inputs that cause the erroneous classifications are depicted to be close to decision boundaries. There are also recent works that are voluntarily tweaking the decision boundaries around given inputs, so that ownership information can be embedded into a target model [START_REF] Merrer | Adversarial frontier stitching for remote neural network watermarking[END_REF].

While measures are inherently in use with neural networks to evaluate the quality of the learned model over a given dataset, we find that there is a lack of an index that provides information on the neighborhood of given inputs, with regards to the boundaries of other classes. Visual inspections of suspicious zones, by plotting in 2D a given decision boundary and the considered inputs is possible Fig. 1. An illustration of zoNNscan runs around two inputs. The example on the left (a 6 digit) is far from the model decision boundary, thus a zoNNscan at radius r returns a value close to 0. The example on the right is another 6 from the MNIST dataset, that is often misclassified (here as a 0, such as in [START_REF] Belongie | Shape matching and object recognition using shape contexts[END_REF]); its is close to the decision boundary of the 6 class. zoNNscan returns a higher value (up to 1) for it, as some data samples in the ball fall into the correct digit class. [START_REF]Plotting high-dimensional decision boundaries[END_REF]. In this paper, we propose an algorithm to output an index, for it helps to quantify possible problems at inference time.

Zoning classification uncertainty with zoNNscan

Classically, given an input and a trained model, the class predicted by the classifier is set to be the one with the higher score in the output vector [START_REF] Van Den Berg | Some insights into the geometry and training of neural networks[END_REF]. These scores can thus be considered as membership probabilities to the model classes: output values close to 1 are to be interpreted as a high confidence about the prediction, and conversely, values close to 0 show high probabilities of non membership to the corresponding class. These scores also provide the information about the uncertainty of the model for a given input : vector scores close to each other indicate an uncertainty between these classes, and equal scores characterize a decision boundary point [START_REF] Lee | Decision boundary feature extraction for neural networks[END_REF]. Thereby, a maximum uncertainty output refers to an input that causes inference to return an uniform vector of probabilities 1 C for a classification task into C-classes. Conversely, minimum uncertainty corresponds to an input that causes inference to return a vector of zeros, except for the predicted class for which a one is set. zoNNscan evaluates the uncertainty of predictions in a given input region. This is illustrated on Figure 1.

zoNNscan: definition

Given a discrete probability distribution p = {p 1 , ..., p C } ∈ [0, 1] C with Σ C i=1 p i = 1 (C designates the number of events), we remind the definition of the Shannon entropy (b designates the base for the logarithm) :

H b : [0, 1] C → R + p → Σ C i=1 (-p i log b (p i )).
The maximum of H b is reached in p = { 1 C , ..., 1 C } and is equal to log b (C) and with the convention 0 log b (0) = 0, the minimum of H b is 0. Subsequently, we use a C-based logarithm to make H C output values in range [0, 1]. A classifier M is considered as a function defined on the input space [0, 1] d (on the premise that data are generally normalized for the training phase in such a way that all features are included in the range [0, 1]), taking its values in the space

P = {p ∈ [0, 1] C s.t. Σ C i=1 p i = 1}
. We introduce the composite function ϕ = H b • M to model the indecision the network has on the input space. More specifically, we propose the expectation of ϕ(U ), E Z [ϕ(U )] (with U a uniform random variable on Z), on an input zone Z ∈ [0, 1] d , to be an indicator on the uncertainty of the classifier on zone Z. Definition 1. Let the zoNNscan index be, in zone Z:

B([0, 1] d ) → [0, 1] Z → E Z [ϕ(U )] = Z ϕ(u)f U (u) du,
where B(R d ) refers to the R d Borel set and f U the uniform density on Z.

The minima of E Z [ϕ(U )] depicts observations in Z that were all returning one confidence of 1, and C -1 confidences of 0. Conversely, the maxima indicates full uncertainty, where each observation returned 1 C confidence values in the output vector.

A Monte Carlo approximation of zoNNscan

In practice, as data are typically nowadays of high dimensionality (e.g., in the order of millions of dimensions for image applications) and deep neural networks are computing complex non-linear functions, one cannot expect to compute the exact value of this expectation. We propose a Monte Carlo method to estimate this expectation on a space corresponding to the surrounding zone of a certain input.

For inspection around a given input X ∈ [0, 1] d , we consider a ball B of center X and radius r, as zone Z introduced in previous subsection. We perform a Monte Carlo sampling of k inputs in a ball for the infinite-norm, corresponding to the hyper-cube of dimension d, around X (as depicted on Figure 1). We are generating inputs applying random deformations3 i on each components

X i such as max(-X i , -r) ≤ i ≤ min(1 -X i , r).
For instance, given a normalized input X and a positive radius r, Monte Carlo sampling is performed uniformly in the subspace of R d defined as:

Z = B ∞ (X, r) ∩ [0, 1] d .
The larger the number of samples k, the better the approximation of the index; this value has to be set considering the application and the acceptable computation/time constraints for inspection.

Algorithm 1 Monte Carlo approximation of zoNNscan

Require: M, X, r, k

for i = 0..k do x ← Monte Carlo Sampling in B∞(X, r) ∩ [0, 1] d Pi ← M(x ) {Inference} entropyi ← HC (Pi) end for return 1 k k i=1 entropyi
The zoNNscan index, for inspection of the surrounding of input X in a model M, is presented on Algorithm 1.

Illustration use of zoNNscan

We now look at three example use cases for zoNNscan.

Uncertainty around a given input

A first zoNNscan use case is to assess at which distance and in which proportions are other classes surrounding an interesting input X.

We experiment using the multi-layer perceptron (noted MLP hereafter) proposed as an example in the Keras library [START_REF] Keras | Deep learning for humans[END_REF], and with k = 1, 000 for the remaining of the experiments. Figure 2 presents the zoNNscan values with respect various radii r, around two input examples (a 6 digit from MNIST and the image of a ship from the CIFAR-10 dataset). A radius value r = 0 is equivalent to the simple output vector corresponding to the inference of X, i.e., without actual sampling in B. Our algorithm starts with a r = 0, up to 1 4 . For the top-experiment on the MNIST digit, the confidence value for r = 0 is high for digit 6 (around 0.9), which is reflected by zoNNscan value to be low (around 0.2). When r increases, the confidence values of other classes are increasing progressively, which results for zoNNscan increasing sharply; at radii of r = [0.25, 1] uncertainty in such a ball is critical.

For the bottom-experiment on the ship image the CIFAR-10 dataset, we observe at r = 0 and close a higher value (around 0.25), that captures the relatively high score for the class 'cat'. The score remains around 0.25 up to r = 0.30, because other boundaries ('frog' and 'truck') are nearby. Finally zoNNscan tends to zero because the 'truck' class occupies the input space in its vast majority. This last fact illustrates that zoNNscan is based on probability vectors, and thus follows the dominant class in them. For the use of zoNNscan at inference time, it is of course more desirable to have low r values, so that the inspected zone remains confined around the input to examine. This experiment brings two observations: ( i) the study of the evolution of zoNNscan values with growing radii is of interest to characterize surrounding or problematic zones (e.g., sharp increase of the 'truck' class starting at 0.25 on Figure 2). ( ii) for a r covering the whole input space, we obtain an estimate of the surface occupied by each class in the model (e.g., class 8 dominates on MNIST, while class 'truck' is covering most of classification space for the CIFAR-10 case).

Disagreement on corner case inputs

Arguably interesting inputs are the ones that make several deep neural network models disagree, as depicted in the case of self-driving cars [START_REF] Pei | Deepxplore: Automated whitebox testing of deep learning systems[END_REF]. Indeed, they constitute corner cases, that have to be dealt with for most critical applications. While this problem may be masked using voting techniques by several independent classifiers [START_REF] Kuncheva | Limits on the majority vote accuracy in classifier fusion[END_REF], we are interested in studying the zoNNscan value of those particular inputs.

We add two models for the experiment, also from Keras [START_REF] Keras | Deep learning for humans[END_REF]: a convolutional neural network (CNN), and a recurrent neural network (IRNN), and focus on the MNIST dataset. Given two models, those inputs are extracted by comparing the predictions of the two models on a given dataset. In the MNIST dataset, we identify a total of 182 disagreements (i.e., corner cases) for the three models.

We plot on Figure 3 the two following empirical distributions. First, in blue, the distribution of 1, 000 zoNNscan values around test set inputs; second, in red, the distribution of the 182 corner case inputs. The zoNNscan index is also estimated with r = 0.025. We observe a significant difference in the distributions (means are reported on Table 1).

In order to formally assess this difference in the two distributions, we performed the statistical Kolmogorov-Smirnov 2-sample test [START_REF] Mann | On a test of whether one of two random variables is stochastically larger than the other[END_REF]. It tests the null hypothesis that two samples were drawn from the same continuous distribution. The p-values of this test correspond to the probabilities of observing the values under the null hypothesis. In any of the three case (MLP, CNN and IRNN models), the test rejects the null hypothesis with p-values lower than 10 -3 .

Effects of watermarking on model boundaries

As a final illustration, we consider the recent work [START_REF] Merrer | Adversarial frontier stitching for remote neural network watermarking[END_REF] consisting in watermarking a model. This operation is performed as follows: first, a watermark key consisting of inputs is created. Half of the key inputs are actually adversarial examples [START_REF] Goodfellow | Explaining and harnessing adversarial examples[END_REF]. Second, the watermarking step consists in finetuning the model by retraining it over the key inputs, in order to re-integrate the adversarial examples in their original class. That last step is by definition moving the model decision boundaries, which constitutes the watermark that the model owner has sought to introduce.

All three models are watermarked with keys of size 100. Each input of those three keys is used as input X for zoNNscan, for 100 runs of Algorithm 1 with k = 1, 000 samples each, r = 0.05. Two distributions of zoNNscan values are computed: one for the un-marked model, and another for the watermarked one.

If the watermarked model were indistinguishable from the original one, the two distributions would be the same. Once again, the Kolmogorov-Smirnov 2sample test returns p-values lower than 10 -3 . This reflects the fact that boundaries have changed around the key inputs, at least enough to be observed by zoNNscan. As the watermarking process is expected to be as stealthy possible, this highlights that the key is to remain a secret for the model owner only [START_REF] Merrer | Adversarial frontier stitching for remote neural network watermarking[END_REF].

Related works

While some metrics such as distance-from-boundaries [START_REF] Guo | Distance-from-boundary as a metric for texture image retrieval[END_REF] have been developed for support vector machines for instance, the difficulty to reason about the classifier decision boundaries of neural networks have motivated research works since decades. Lee et al. [START_REF] Lee | Feature extraction based on decision boundaries[END_REF] proposed the extraction of features from decision boundaries, followed by [START_REF] Lee | Decision boundary feature extraction for neural networks[END_REF] that targets the same operation without assuming underlying probability distribution functions of the data. An interesting library for visual inspection of boundaries, from low dimensional embedding is available online [START_REF]Plotting high-dimensional decision boundaries[END_REF]. Fawzi et al. propose to analyze some geometric properties of deep neural network classifiers [START_REF] Fawzi | Classification regions of deep neural networks[END_REF], including the curvature of those boundaries. Van den Berg [START_REF] Van Den Berg | Some insights into the geometry and training of neural networks[END_REF] studies the decision region formation in the specific case of feedforward neural networks with sigmoidal nonlinearities.

If one can afford to use an ensemble of models, as well as joint adversarial training to assess the uncertainty of a model's decisions, the work in [START_REF] Lakshminarayanan | Simple and scalable predictive uncertainty estimation using deep ensembles[END_REF] provides an interesting approach; this makes it a good candidate for offline analysis of decisions. Cross entropy is one of the most basic measure for the training of neural networks. While it evaluates the divergence of the dataset distribution from the model distribution, it is not applied to inspect the confidence in the inference of specific zones in the input space. This paper addresses the relation of the surrounding of a given input with respect to the presence of decision boundaries of other classes; this relation is captured by the zoNNscan index, possibly at runtime, by leveraging the Shannon entropy.

Conclusion

We introduced a novel index, zoNNscan, for inspecting the surrounding of given inputs with regards to the boundary entropy measured in the zones of interest. We have presented a Monte Carlo estimation of that index, and have shown its applicability on a base case, on a technique for model watermarking, and on a concern regarding the adoption of current deep neural networks (corner case inputs). As this index is intended to be generic, we expect other applications to leverage it; future works include the study of the link of zoNNscan values with regards to other identified issues such as trojaning attacks on neural networks [START_REF] Liu | Trojaning attack on neural networks[END_REF], and on the intensification of the Monte Carlo search.

Fig. 2 .

 2 Fig. 2. zoNNscan values (dotted black line) on y-axes, with a varying radius r ∈ [0, 1] on x-axes. Mean confidence scores for each class in the explored zone are also reported (colored lines). Top: MNIST dataset; a 6 digit classified by the MLP model. Bottom: CIFAR-10 dataset; a ship classified by RESNETv2.

Fig. 3 .

 3 Fig. 3. Relative distributions of zoNNscan around 1, 000 random test set inputs (blue) and the 182 corner case inputs (red). From left to right: the MLP, CNN and IRNN models.

Table 1 .

 1 Mean of the empirical zoNNscan distributions, for the three neural networks.

	Scenario	MLP CNN IRNN
	1,000 inputs from test set 0.3709 0.3457 0.2403
	182 corner case inputs	0.6857 0.6564 0.2989

For more advanced sampling techniques in high dimensional spaces, please refer to e.g.,[START_REF] Dick | High-dimensional integration: The quasi-monte carlo way[END_REF].

Note that for X ∈ [0, 1] d and for r ≥ 1, B∞(X, r) ∩ [0, 1] d = [0, 1] d , then the space of normalized data is totally covered by the sampling process.