Umang Aggarwal
email: umang.aggarwal@cea.fr

Adrian Popescu
email: adrian.popescu@cea.fr

Eden Belouadah
email: eden.belouadah@cea.fr

Celine Hudelot
email: celine.hudelot@centralesupelec.fr

A Comparative Study of Calibration Methods for Imbalanced Class Incremental Learning

Keywords: Incremental Learning, Calibration, Class Imbalance, Image classification

Deep learning approaches are successful in a wide range of AI problems and in particular for visual recognition tasks. However, there are still open problems among which is the capacity to handle streams of visual information and the management of class imbalance in datasets. Existing research approaches these two problems separately while they co-occur in real world applications. Here, we study the problem of learning incrementally from imbalanced datasets. We focus on algorithms which have a constant deep model complexity and use a bounded memory to store exemplars of old classes across incremental states. Since memory is bounded, old classes are learned with fewer images than new classes and an imbalance due to incremental learning is added to the initial dataset imbalance. A score prediction bias in favor of new classes appears and we evaluate a comprehensive set of score calibration methods to reduce it. Evaluation is carried with three datasets, using two dataset imbalance configurations and three bounded memory sizes. Results show that most calibration methods have beneficial effect and that they are most useful for

Introduction

Large scale image repositories are highly dynamic, with content being added and/or removed at a fast pace. However, content analysis is currently done with algorithms built to learn from static information. This is notably the case for deep learning models which are trained on fixed datasets. When model updates are required, the entire training corpus is reused for learning, making the process cumbersome. To make image content analysis more dynamic and thus more adapted to dynamic corpora, incremental [START_REF] Rebuffi | icarl: Incremental classifier and representation learning[END_REF] or lifelong [START_REF] Aljundi | Expert gate: Lifelong learning with a network of experts[END_REF] learning processes need to be implemented.

In this paper, we focus on the intersection of incremental learning and class imbalanced learning. Each problem is well studied separately but, to our knowledge, they were not tackled together in the context of deep learning. A joint study is needed to cope with dynamic and imbalanced datasets. The main challenge in incremental learning is due to a restricted or impossible access to old data. This restriction can be explained by factors such as: data removal on the Web and in stream data processing [START_REF] Hammer | Learning and modeling big data[END_REF], privacy in the medical domain [START_REF] Venkatesan | A strategy for an uncompromising incremental learner[END_REF], limited resources in embedded systems [START_REF] Parisi | Continual lifelong learning with neural networks: A review[END_REF]. Recent research in incremental learning use deep learning as backbone and most of them focus on class incremental learning, a setting in which data are completely labeled. If no memory of the past is allowed, the deep architecture grows in time to accommodate new classes [START_REF] Aljundi | Expert gate: Lifelong learning with a network of experts[END_REF][START_REF] Rusu | Progressive neural networks[END_REF][START_REF] Wang | Growing a brain: Fine-tuning by increasing model capacity[END_REF]. If a bounded memory is allowed, the architecture is fixed and an adapted fine tuning is applied to learn incrementally [START_REF] Castro | End-to-end incremental learning[END_REF][START_REF] Javed | Revisiting distillation and incremental classifier learning[END_REF][START_REF] Rebuffi | icarl: Incremental classifier and representation learning[END_REF].

A wide majority of deep learning methods assume that training datasets are balanced or nearly so. This is for instance the case of the ImageNet LSV RC effort [START_REF] Russakovsky | Imagenet large scale visual recognition challenge[END_REF], in which 1000 leaf classes which are well represented in the dataset and rather balanced. The ILSV RC training set used in [START_REF] Rebuffi | icarl: Incremental classifier and representation learning[END_REF] has a mean of 1231 images per class, with a standard deviation of 70. However, an analysis of the full set of ImageNet leaf classes [START_REF] Deng | Imagenet: A large-scale hierarchical image database[END_REF] shows that image counts per class are highly variable, with a mean of 592 and a standard deviation of 508. The same is true for most existing large scale public datasets, including Open Images [START_REF] Krasin | Openimages: A public dataset for largescale multi-label and multi-class image classification[END_REF], Google Landmarks [START_REF] Noh | Large-scale image retrieval with attentive deep local features[END_REF] or MS-CELEB-1M [START_REF] Guo | Ms-celeb-1m: A dataset and benchmark for large-scale face recognition[END_REF]. The datasets built for real-life applications are often imbalanced and the classes of interest are often under-represented. This is, for instance, the case in the medical domain or in fraud detection. Learning from imbalanced datasets which include minority and majority classes leads to a prediction bias towards the majority classes. This was shown by two surveys of classical machine learning methods devised for imbalanced datasets [START_REF] He | Learning from imbalanced data[END_REF][START_REF] Japkowicz | The class imbalance problem: A systematic study[END_REF]. A similar conclusion was recently presented in [START_REF] Buda | A systematic study of the class imbalance problem in convolutional neural networks[END_REF], where the authors study the effect of data imbalance on deep learning algorithms.

Here we advocate that class incremental learning with a bounded memory actually boils down to a form of imbalanced learning problem. New data most often corresponds to majority classes, while old data corresponds to minority classes since images of old classes need to be fit in the bounded memory that is allocated to them. If the initial dataset is balanced, as it is assumed in existing incremental learning [START_REF] Castro | End-to-end incremental learning[END_REF][START_REF] Javed | Revisiting distillation and incremental classifier learning[END_REF][START_REF] Rebuffi | icarl: Incremental classifier and representation learning[END_REF], the associated imbalance profile is binary, with new classes having a large number of images and past classes having a small but identical number of images. If the dataset itself is imbalanced, as it is often the case in real contexts, the inherent imbalance is added to the incremental one and the resulting imbalance profile can be more complex.

Focus is put on calibration methods whose objective is to reduce the prediction bias between majority and minority classes. We compare the following calibration methods: (1) isotonic regression [START_REF] Zadrozny | Transforming classifier scores into accurate multiclass probability estimates[END_REF] and Platt scaling [START_REF] Platt | Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods[END_REF] which leverage initial scores to improve final predictions, (2) thresholding applied to the initial class probabilities in order to increase the predictions of rare classes [START_REF] Buda | A systematic study of the class imbalance problem in convolutional neural networks[END_REF], (3) nearest-exemplar-mean classifier [START_REF] Rebuffi | icarl: Incremental classifier and representation learning[END_REF] and balanced fine tuning [START_REF] Castro | End-to-end incremental learning[END_REF] which were recently introduced as post-processing steps to reduce the effect of data imbalance in deep incremental learning and (4) two proposed methods which group classes in batches either as new vs. old or by image counts and then exploit the mean classification scores per batch for calibration.

While the focus is on methods which increase the performance on the test dataset, we also evaluate the intrinsic effect on model calibration. A model is said to be calibrated when the average confidence of the model is the same as its average accuracy. Deep learning models have been shown to provide over-confident predictions that we do not match its accuracy [START_REF] Guo | On calibration of modern neural networks[END_REF]. In several scenarios, it is important to take into account the confidence of the model on the predictions. This is particularly the case, when decision has to be taken based on prediction of more than one model. Well-calibrated models have the confidence levels aligned to the model accuracy and thus give valuable information of how likely the model is to be correct or incorrect.

The main findings are:

1. the obtained results support the usefulness of a majority of post-processing methods for the reduction of bias toward majority classes 2. when a bounded memory is available, the use of vanilla fine tuning followed by calibration is preferable to the widely used distillation loss [START_REF] Castro | End-to-end incremental learning[END_REF][START_REF] Javed | Revisiting distillation and incremental classifier learning[END_REF][START_REF] Rebuffi | icarl: Incremental classifier and representation learning[END_REF][START_REF] Hou | Learning a unified classifier incrementally via rebalancing[END_REF][START_REF] Wu | Large scale incremental learning[END_REF]. 3. thresholding based calibration is most effective in providing overall improvement in accuracy, though it has detrimental affect on model calibration. The proposed methods provide consistent improvement in both model accuracy and model calibration.

We outline the structure of the remainder of the paper. Section 2 reviews relevant research from both incremental and imbalanced learning. Section 3 formalizes the incremental learning with imbalanced datasets problem. Section 4 discusses calibration as an effective way to counter dataset imbalance and introduces the different calibration methods tested. Section 5 compares the calibration methods to three strong incremental learning baselines and proposes an analysis of results in terms of accuracy (Subsection 5.4) and the ability to provide calibrated predictions (Subsection 5.5). Finally, Section 6 presents the conclusions and perspectives related to the proposed contribution.

Related works

Our research lies at the intersection of incremental and imbalanced learning. Both topics witness a strong regain of interest after the advent of deep learning. The two problems are well studied separately and the two issues are also simultaneously tackled for classical machine learning algorithms [START_REF] Polikar | Learn++: an incremental learning algorithm for supervised neural networks[END_REF][START_REF] Syed | Handling concept drifts in incremental learning with support vector machines[END_REF]. We analyze works from both areas and focus on challenges related to deep architecture complexity and to scalability. These properties are of utmost importance in applications such as visual content analysis. The visual corpora to be analyzed evolve quickly and there is a need for updating the underlying classification models accordingly.

Deep Incremental Learning

The main challenge in incremental learning is catastrophic forgetting [START_REF] Mccloskey | Catastrophic interference in connectionist networks: The sequential learning problem[END_REF], i.e. the tendency to forget previously learned information when new data is incorporated. It occurs whenever access to old data is constrained or impossible. If nothing is done to prevent this phenomenon, incremental learning predictions for past classes become random or nearly so. This is particularly the case of deep learning algorithms which heavily rely on labelled data. Recent incremental learning research exploits deep learning techniques for which is was shown that catastrophic forgetting is a major issue [START_REF] Bengio | An empirical investigation of catastrophic forgeting in gradient-based neural networks[END_REF].

Two main groups of methods have been proposed. The first focuses on changing the neural net architectures to incorporate new knowledge. Influential works include Growing a Brain [START_REF] Wang | Growing a brain: Fine-tuning by increasing model capacity[END_REF], progressive neural networks [START_REF] Rusu | Progressive neural networks[END_REF] or lifelong learning with a network of experts [START_REF] Aljundi | Expert gate: Lifelong learning with a network of experts[END_REF]. These methods are interesting but their complexity grows when new classes are added incrementally. Notably, inference time will become longer as the model grows and the scalability of these methods is consequently reduced.

The second group adapts fine tuning to an incremental context by using a combination of classification and distillation losses [START_REF] Castro | End-to-end incremental learning[END_REF][START_REF] Javed | Revisiting distillation and incremental classifier learning[END_REF][START_REF] Rebuffi | icarl: Incremental classifier and representation learning[END_REF]. These methods have constant model complexity, except for the classification layer which integrates new classes, and are more fitted for large scale content analysis. Most of them require a bounded memory in order to partially avoid catastrophic forgetting. The memory related constraint is more acceptable than model complexity growth when analyzing large datasets since the inference time is not influenced by the use of memory. Learning-without-Forgetting (LwF) is an influential method presented in [START_REF] Li | Learning without forgetting[END_REF]. The algorithm does not rely on past data and exploits knowledge distillation [START_REF] Hinton | Distilling the knowledge in a neural network[END_REF] to reduce the discrepancy between activations of old classes from the original and new networks. iCaRL [START_REF] Rebuffi | icarl: Incremental classifier and representation learning[END_REF] builds on top of LwF in that it combines classification and distillation losses for each incremental state of the algorithm. A first important difference with LwF is that a bounded memory is allowed to store exemplars of old classes. As more classes are added, the number of images per old class is reduced to fulfill the memory constraint. Class exemplars are selected using a herding mechanism which gives priority to images that are closest to the class mean. A second difference is related to the classification mechanism. Instead of using the class activations of the deep models, a nearest-mean-of-exemplars is implemented. The iCaRL average top-5 accuracy on ILSV RC is 62.5%. An iCaRL analysis [START_REF] Javed | Revisiting distillation and incremental classifier learning[END_REF] indicates that the most important algorithm components are the bounded memory and the distillation loss. The herding mechanism and the nearestmean-of-exemplars classification seem to matter less. Recently an end-to-end incremental learning scheme with a bounded memory in which predictions are provided by the deep model was introduced in [START_REF] Castro | End-to-end incremental learning[END_REF]. The main modification compared to iCaRL resides in the proposal of a loss function which includes separate distillation terms for each incremental batch. In addition, data augmentation and balanced fine tuning used to reduce the effect of data imbalance between old and new classes. Top-5 accuracy on ILSV RC is 69.4%, to be compared with 62.5% obtained by iCaRL. Interestingly, the use of herding to store exemplars is only marginally useful (0.5 points) compared to random selection. This finding confirms the iCaRL analysis conclusions from [START_REF] Javed | Revisiting distillation and incremental classifier learning[END_REF]. Existing methods reduce the effect of imbalance via the use of class exemplars [START_REF] Rebuffi | icarl: Incremental classifier and representation learning[END_REF] or balanced fine tuning [START_REF] Castro | End-to-end incremental learning[END_REF]. We include these methods in our study of calibration methods.

A related approach BiC [START_REF] Wu | Large scale incremental learning[END_REF] tackles the bias against old classes, by adding a linear layer with two learnable parameters after the classification layer. A small part of dataset is reserved to learn the parameters of the bias correction layer. The training is done in two steps, with the model and classifier weights are learnt first, before learning the bias correction layer using only the reserved dataset. In LU CIR [START_REF] Hou | Learning a unified classifier incrementally via rebalancing[END_REF], authors proposed three balancing constraints at the time of training to mitigate the imbalance bias between old and new classes. Firstly, they modify the distillation loss component using cosine normalization to counter larger weights and biases for new classes. Further, they exploit the observation that imbalance is less pronounced at the classifier margin to introduce a margin loss function which is less susceptible to imbalance. Finally, a less forget constraint is introduced which complements the distillation loss by encouraging the orientation of features extracted by current network to be similar to those by the original model. We include BiC [START_REF] Wu | Large scale incremental learning[END_REF] and LU CIR [START_REF] Hou | Learning a unified classifier incrementally via rebalancing[END_REF] as baselines to evaluate their performance on imbalanced datasets.

Imbalanced Learning

The methods that tackle imbalanced learning are based either on data sampling or classifier adaptation [START_REF] He | Learning from imbalanced data[END_REF]. Data-sampling methods mitigate the bias towards majority classes by balancing the training dataset. Balancing can be achieved either by undersampling the majority classes or by oversampling minority classes [START_REF] Guo | Learning from class-imbalanced data: Review of methods and applications[END_REF]. The main risk of undersampling is that it leads to incomplete representation of classes. Informed undersampling [START_REF] Kubat | Addressing the curse of imbalanced training sets: One-sided selection[END_REF] partially solves this problem by avoiding to select images which are close to class boundary. Oversampling methods are sensitive to overfitting. SMOTE [7] is an influential solution to the problem and was improved in [START_REF] Bellinger | Beyond the boundaries of SMOTE -A framework for manifold-based synthetically oversampling[END_REF][START_REF] Han | Borderline-smote: A new over-sampling method in imbalanced data sets learning[END_REF][START_REF] Maciejewski | Local neighbourhood extension of SMOTE for mining imbalanced data[END_REF]. It basically creates synthetic features for a dataset by applying simple arithmetic transformations to actual features. Undersampling and oversampling were recently studied in the context of deep imbalanced learning and were shown to have detrimental effect for imbalanced versions of the ILSV RC dataset [START_REF] Buda | A systematic study of the class imbalance problem in convolutional neural networks[END_REF]. They will thus not be included among the calibration methods studied here.

Another line of research deals with imbalance at the classifier level. Thresholding (or post-scaling) modifies the decision threshold of the classifier to counter the bias toward majority classes. An interesting formulation of thresholding in [START_REF] Richard | Neural network classifiers estimate bayesian a posteriori probabilities[END_REF] where the outputs are modified using the prior class probabilities. While very simple, thresholding outperforms a large array of data sampling and classifier level methods for object recognition using deep learning models in [START_REF] Buda | A systematic study of the class imbalance problem in convolutional neural networks[END_REF]. It is thus a very competitive method and is evaluated here in the context of imbalanced incremental learning. The authors of [START_REF] Guo | On calibration of modern neural networks[END_REF] show that the outputs of deep neural nets are miscalibrated, even for balanced datasets. Miscalibration is likely to be even more important for imbalanced datasets. Two influential methods were shown to provide good results for calibration of classical machine learning algorithms [START_REF] Niculescu-Mizil | Predicting good probabilities with supervised learning[END_REF]. Isotonic regression [START_REF] Zadrozny | Transforming classifier scores into accurate multiclass probability estimates[END_REF] fits the raw classification results to a set of non-decreasing discrete set of values. Platt scaling [START_REF] Platt | Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods[END_REF] basically performs a logistic regression on the initial outputs of the classifier. We use these two methods in our comparison of calibration methods.

Problem formulation

We consider D N a labeled dataset X y , y t ∈ X × Y for t = 1, 2, ..T i.i.d realizations of random variables X , Y ∼ P, where P is the data distribution, X is the instance space and Y is the set of N class labels {y 1 , ..., y N }. We denote X i = {x 1 i , x 2 i , ..., x ni i } the set of n i instances for the class y i in the dataset D N . In a supervised classification problem, the objective is to learn a model M : X → Y that maps an instance x to a label vector Ŷ . By the following, we will denote Ŷ as a set of the class prediction with ŷi the prediction score for class y i .

In an incremental learning setting, at each incremental state k, a set of P k new classes is added to the previous dataset with, for each new class j, a set of n j instances. The objective is thus to use M k-1 , the model learned at the previous step, as input for an updated model M k which classifies N k = P 1 + P 2 + ... + P k classes. Here, N k is the total number of classes that have been observed from the beginning. M k is trained using a dataset D N k composed of all the instances of the P k new classes and only a restricted set of the instances of the N k-1 old ones. In particular, we assume a bounded memory B is available for the instances of the old classes in each incremental state. As a consequence, due to this limited memory size, D N k is by nature imbalanced and imbalance grows at each incremental state.

In this paper, we consider deep neural models M k which include two main components. The first is a feature extractor We are interested in learning incrementally over imbalanced datasets. In this context, the level of imbalance can be defined, for instance, by using a combination of mean (µ) and standard deviation (σ) of the number of images per class. The higher the ratio between σ and µ is, the stronger the imbalance of the dataset will be.

F k : X N k → R d ,

Calibration methods

Dealing with imbalance is important insofar the number of training samples per class often varies in real-life applications. As a consequence, majority classes have better representations and are favored over minority ones. The application of calibration methods is an effective way to counter the effect of imbalance [START_REF] Buda | A systematic study of the class imbalance problem in convolutional neural networks[END_REF]. Put simply, calibration attempts to boost predictions for minority classes in order to compensate for their weaker representation in the deep model. We study different calibration methods proposed either in imbalanced or incremental learning literature. Fine tuning algorithms for incremental learning update the model M k-1 at an incremental state k with training examples from new classes P k and a bounded exemplar set from past classes N k-1 . If the initial dataset is balanced, we assume that each class is represented by S images. The bounded memory thus generates a binary imbalance with old classes being represented by B N k-1 and new classes by S images. We term this imbalance as incremental imbalance as it arises as a consequence of learning incrementally with a bounded memory. In our context, a dataset imbalance due to the variable class image counts is added to the imbalance generated via incremental learning. The imbalance profile is not binary anymore since new classes are represented by a variable number of images and the proposed calibration methods should take this into account.

When the model M k is trained with a dataset affected by both incremental and dataset imbalance, it learns a feature extractor which is biased toward majority classes and performance is sub-optimal. On average, the biased classifier Training is done with 100 new classes added in each iteration using vanilla fine tuning, B = 5000 bounded memory for old classes and sof t imbalance configuration as defined in Section 5. The first, non-incremental, state is not represented.

associates higher scores to images from majority class than minority classes. Figure 1 illustrates this bias using the ILSV RC dataset. The mean score of a class is computed using the predictions obtained for its samples from the training dataset. Then, we aggregate the average over the old and new classes to estimate the mean score of old and new classes. We note that the mean scores of old classes are consistently lower than those of old classes for all incremental states. Moreover, the difference tends to grow from left to right since the imbalance is higher in later incremental states. We focus on fixed deep architectures and, in this case, the bias induced by imbalance needs to be reduced without increasing the complexity of the feature extractor F k for the deep model M k . Consequently, calibration is defined as an adaptation of C k , the classification layer of M k , with the aim of reducing the bias toward majority classes.

We first present calibration methods which cover the main approaches from literature. Then we introduce two simple methods which leverage: (1) the prediction score means for old and new classes and (2) the distribution of the number of images per class.

Isotonic regression calibration (iso)

Isotonic regression [START_REF] Zadrozny | Transforming classifier scores into accurate multiclass probability estimates[END_REF] transforms the initial classifier predictions into a discrete set of calibrated scores. Since the number of available images per class reduces in incremental learning with bounded memory as more classes are added over time, isotonic regression exploits the training set D N k . The calibra-tion is performed individually for each class and exploits the overlap between positive and negative examples for each class. iso is based on the intuition that the distributions of predicted scores for positives and negatives examples are separable. A discrete set of scores R = {0, ..., p l , ..., 1} is created where each discrete value represents a range of initial prediction scores. p l will be assigned to all initial predictions between two consecutive prediction boundaries ŷi l-1 and ŷi l determined via isotonic regression, with ŷi l-1 < ŷi l . The isotonic calibrated version of ŷi is:

ŷi iso = p l , if ŷi l-1 ≤ ŷi < ŷi l (1)
Isotonic regression has the advantage of being non-parametric but requires a large number of positive samples per class to discretize probabilities in an efficient manner. The method is applied as a post-processing step after C k . The class predicted after calibration is the argmax value of all calibrated class predictions given by Eq. 1.

Platt calibration (pl)

Platt scaling [START_REF] Platt | Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods[END_REF] fits a logistic regression over the initial scores in order to reduce miscalibration. The effect of pl is effectuated at class level using one vs all selection of positive and negatives for each class. We write the calibrated score as:

ŷi pl = 1 1 + exp(A ŷi + C) (2)
where A and C are two parameters which need to be learned and ŷi is the initial prediction of the i th class. The incremental training set D N k is used to determine A and B by optimizing a max likelihood method. There is evidence that isotonic regression outperforms Platt scaling if enough examples per class are available [START_REF] Niculescu-Mizil | Predicting good probabilities with supervised learning[END_REF]. However, this finding was not tested for imbalanced datasets which may include a lot of minority classes, as it is the case here. The class predicted after calibration is the argmax value obtained by applying Eq. 2 to the initial scores predicted by C k .

Thresholding based calibration (th)

Thresholding [START_REF] Buda | A systematic study of the class imbalance problem in convolutional neural networks[END_REF] adjusts the prediction scores of a multi-class classifier by dividing the output of a class in C k by its estimated prior probability. The calibrated score of is written as:

ŷi th = ŷi • N k l=1 n l n i (3)
where n i is the number of images for the i th class and

N k
l=1 n l is the total number of images in the training dataset D N k . By implementing Eq. 3, thresholding boosts the scores of minority classes. The smaller the number of samples of a class is, the stronger the boost it receives will be. As we mentioned, a recent study of imbalanced learning for deep learning models showed that thresholding is highly efficient [START_REF] Buda | A systematic study of the class imbalance problem in convolutional neural networks[END_REF]. Its usefulness is theoretically supported by the fact that the outputs of a neural network correspond to Bayesian a posteriori probabilities [START_REF] Richard | Neural network classifiers estimate bayesian a posteriori probabilities[END_REF]. The class predicted after calibration is the argmax value obtained over all predictions obtained with Eq. 3.

Nearest-mean-of-exemplars calibration (nem)

The authors of iCaRL [START_REF] Rebuffi | icarl: Incremental classifier and representation learning[END_REF] proposed nearest-mean-of-exemplars, an adaptation of nearest class-mean classifier [START_REF] Mensink | Distance-based image classification: Generalizing to new classes at near-zero cost[END_REF], to counter the inherent imbalance in incremental learning. The calibrated score of the i th class is written as:

ŷi nem = 1 ||f (x) -µ i || (4)
where f (x) is the d-dimensional feature of the test instance x provided by penultimate layer of the incremental model

M k ; µ i = 1 ni ni l=1 f (x l )
-the mean feature of the exemplars available for the i th class. Note that, in order to reduce the majority bias, nem is performed after the selection of exemplars for new classes. Consequently, the number of exemplars per class is more balanced than if all samples were used and imbalance is reduced. However, in the case of imbalanced datasets, there is no guarantee to achieve perfect balancing since a part of the classes might not have enough exemplars. This is especially the case for a strong imbalance regime in which the number of samples per class varies strongly. nem calibration replaces the classification layer C k of deep models by an external classifier which was explicitly designed to counter imbalance. The class predicted for test instance x after calibration is given by the argmin function applied to the set of Euclidean distances computed for all classes using Eq. 4.

Balanced fine tuning calibration (bal)

As an alternative to iCaRL [START_REF] Rebuffi | icarl: Incremental classifier and representation learning[END_REF], the authors of [START_REF] Castro | End-to-end incremental learning[END_REF] propose an end-to-end incremental learning method. The bias in favor of majority classes is reduced by introducing a second training step which trains all classes with the same number of exemplars. After the initial training which creates M k using the imbalanced dataset

D N k , a model M bal k : X bal N k → Y N k is trained. M bal k exploits D bal
N k a balanced version of D N k which includes B N k exemplar images for both old and new classes and is fine tuned starting from M k . While this method is clearly appealing for balanced dataset, its usage in an imbalanced context is more challenging because there is no guarantee that there will be B N k available for each class. We modify the approach slightly in that balanced fine tuning only learns the weights of the classification layer C bal k , instead of fine tuning the entire model. This modification is done in order to make bal calibration more comparable to the other calibration methods, which do not modify the feature extractor the deep model. It is also motivated by the fact that initial experiments run with full fine tuning of M k provided lower results than fine tuning the classification layer only. Note that bal has a higher computational cost at training time since it requires a supplementary training step. The calibrated prediction of the i th class obtained with bal can be written as:

ŷi bal = C bal k (i) (5)
where C bal k gives the output of classification layer of the balanced model M bal k for the i th class. The class predicted after calibration is the argmax value obtained over all classes using Eq. 5.

Batch mean based calibration (mb)

The analysis of raw classification scores from Figure 1 provides support for a bias in favor of new classes in imbalanced incremental learning. A simple way to reduce this imbalance is to exploit the mean prediction scores of new and old classes of incremental state k. The calibrated score of the i th class is written as:

ŷi mb = µ new µ old ŷi (6)
where the means are defined as

µ new = 1 P k l=1 n l P k l=1 n l
q=1 ŷq and µ old =

1 N k-1 l=1 n l N k-1 l=1 n l
q=1 ŷq for new and old classes respectively. Note that here we hold out validation sets for new and old classes in order to compute their mean classification scores. Contrarily to iso which works at class level, having a separate validation set is doable because mb is applied at dataset level and the number of available samples is sufficient. The class predicted after calibration is the argmax prediction value obtained after applying Eq. 6.

Fisher-Jenks based calibration (f j)

The mean based calibration operates at incremental batch level. It disregards the fact that, due to dataset imbalance, some of the new classes might fall in the minority classes set. To counter this problem, we propose a calibration method which makes use of class image counts and of their associated classification score. We use the Fisher-Jenks natural breaks method [START_REF] Jenks | Optimal data classification for choropleth maps[END_REF] to group classes. This method ensures an optimal distribution of a set of values in a predefined set of L clusters. It is thus appropriate to deal with the different imbalance profiles that occur in imbalanced incremental learning. In our case, the inputs given to Fisher-Jenks are the image counts n i associated to the N k classes learned in incremental state k. The calibrated score of the i th class is written as: ŷi

f j = µ cl L µ cl(i) ŷi (7)
where µ cl(i) is the mean prediction score of the Fisher-Jenks cluster which includes the i th class and µ cl L is the mean prediction score of the L th cluster with the largest number of instances per class. f j can be seen as a compromise between methods such as iso, which operate at class level, and mb, which works indifferently for all classes. It groups classes depending on their sample distribution in order to have sufficient samples for a robust statistical distribution. Similar to the mb method from Subsection 4.6, the means are computed using a validation set. The number of Fisher-Jenks clusters is set using a cross-validation with the validation set. The class predicted after calibration is the argmax prediction value obtained by applying Eq. 7 to all initial class predictions.

Evaluation

The experiments are designed to evaluate both kind of imbalances: dataset imbalance and incremental imbalance. All methods are evaluated with three large datasets designed for object, face and landmark recognition. Sof t and strong imbalance configurations are created to evaluate dataset imbalance. Three bounded memory size are introduced for each dataset in order to test the robustness of calibration method with respect to this central parameter of incremental algorithms.

Baselines

The calibration methods studied here are applied on top of a vanilla fine tuning backbone which is run iteratively for each incremental state in order to integrate new classes. Naturally, vanilla fine tuning (F T hereafter) is the main baseline used here. The selection of exemplars is based on the herding mechanism [START_REF] Mensink | Distance-based image classification: Generalizing to new classes at near-zero cost[END_REF]. To evaluate the usefulness of the proposed approach, we compare it to three competitive incremental learning methods:

-iCaRL [START_REF] Rebuffi | icarl: Incremental classifier and representation learning[END_REF] of past and new class predictions. The distillation term is improved by handling feature vectors instead of raw scores. Finally, inter-class separation is favored in order to better separate embeddings of past and new classes.

Datasets and methodology

We evaluate the baselines and the calibration methods on the following datasets:

-ILSV RC [START_REF] Russakovsky | Imagenet large scale visual recognition challenge[END_REF] is a subset of 1000 ImageN et classes used in the ImageN et LSV RC challenges.

-V GGF ace2 [START_REF] Cao | Vggface2: A dataset for recognising faces across pose and age[END_REF] (V GGF 2 below) focuses on face recognition. We select the 1000 classes with the largest number of associated images.

-GoogleLandmarks [START_REF] Noh | Large-scale image retrieval with attentive deep local features[END_REF] (LAN D below) was built for landmark recognition and we again select 1000 classes with the largest number of associated images.

The original amount of imbalance in these three datasets is weak, as shown in Table 1. We introduce two imbalance configurations to evaluate behavior of the algorithms with different degrees of dataset imbalance:

sof t -randomly retains between 50 and the initial number of images for each class. strong -randomly retains between: 10 and 25 images for 300 classes, [START_REF] Mensink | Distance-based image classification: Generalizing to new classes at near-zero cost[END_REF] and 75 for 300 classes, 76 and 100 for 200 classes and between 101 and the initial number of images for the remaining 200 classes.

The corresponding means and standard deviations are reported in Table 1. In the sof t configuration, slightly more than half of the original training data is kept and the standard deviation amounts to over 50% of dataset means.

With strong, we discard a wide majority of original data and the resulting imbalance is much stronger and the standard deviation becomes higher than the mean in each case.

The evaluated calibration methods operate either at class level (iso, pl, th, nem, bal) or at an aggregate level which includes a subsets of the learned classes (mb, f j). For class level methods, we reuse the training images from the initial dataset as inputs for calibration. This is necessary since the number of available images is reduced, especially for old and/or minority classes and most of the methods require a rather large amount of data to provide reliable results. Consequently, the use of a validation subset would be suboptimal here. When inputs from different classes are aggregated in batches (mb and f j), the use of a proper validation split becomes possible. We create validation sets using 10% of the training data of old and new classes. We maintain the val/train split in the bounded memory B to avoid mixing the training and validation exemplars in different incremental states. Note that the outputs of C k are used either in their raw form (iso, pl,mb, f j) or after transformation in probabilities by applying sof tmax (th). This choice is made in order obtain an optimal configuration of each algorithm.

The experimental setup is inspired by the one proposed in iCaRL [START_REF] Rebuffi | icarl: Incremental classifier and representation learning[END_REF]. Each dataset of 1000 classes is split into k = 10 incremental states. Each incremental state adds a batch of P k = 100 classes to those that were already learned in states 1 to k -1. The same class ordering provided in iCaRL [START_REF] Rebuffi | icarl: Incremental classifier and representation learning[END_REF] is reused for ILSV RC and a random ordering of classes is created to form V GGF 2 and LAN D states. The size of bounded memory B was shown to have a central importance for the performance of incremental learning algorithms [START_REF] Castro | End-to-end incremental learning[END_REF][START_REF] Rebuffi | icarl: Incremental classifier and representation learning[END_REF]. To assess its influence on the proposed calibration methods, we report results with B = {5000, 10000, 20000} exemplars stored in memory for each dataset and imbalance configuration.

A ResNet-18 architecture [START_REF] He | Deep residual learning for image recognition[END_REF] is used as a backbone for all experiments. ResNet have been successful in allowing neural nets to be deeper by tackling the problem to stagnation of performance with addition of layers after some point. They employ residual mapping as the basis function which adds the input values to approximate the final function. ResNet-18 has one (7*7) and sixteen (3*3) convolutional layers in addition to two max pooling layers and a final linear classification layer. We used the publicly available iCaRL Tensor-Flow implementation in [START_REF] Rebuffi | icarl: Incremental classifier and representation learning[END_REF] with a binary cross-entropy loss and the original parameters proposed there. Vanilla fine tuning (F T ) was implemented in Pytorch [START_REF] Paszke | Automatic differentiation in pytorch[END_REF] using cross-entropy loss. The models were trained for 25 epochs with a initial learning rate of 0.1 at every incremental state and scheduled to decay by 0.1 when the loss plateaus out for 5 epochs. For V GGF 2, face cropping is done with MTCNN [START_REF] Zhang | Joint face detection and alignment using multitask cascaded convolutional networks[END_REF] before further processing. Training images are processed using randomly resized 224 × 224 crops and horizontal flipping and are normalized afterwards.

Metrics

-Accuracy -the performance of different methods is evaluated using top-1 accuracy for each incremental step defined as:

acc = 100 * 1 n n i=1 argmax( Ŷ ) == y i (8)
, where Ŷ is the set of predicted score and y i is the true label for test sample i. This measure is then averaged over all incremental states in order to

acc(B m ) = 1/B m i∈Bm 1(argmax( Ŷi ) == y i ) (11)
,where Pi and Ŷi are the set of predicted probability and score respectively and y i is the true label for test sample i. The values of ECE range from 0 to 1, with lower values indicating better model calibration.

Analysis of Accuracy of Calibration methods

The obtained results are presented in Tables 2 and3. A detailed view of top-1 accuracy for the incremental states of ILSV RC with B = 5000 bounded memory for sof t and strong imbalance configurations is provided in Figure 2. and vanilla fine tuning (F T ), our baselines. The next two columns are calibrated versions of F T as follows: F T iso -isotonic regression; F T pl -Platt scaling; F T th -thresholding; F Tnem -nearest-mean-of-exemplars; F T bal -balanced fine tuning; F T mb -batch mean based calibration; F T f j -Fisher-Jenks based calibration. Following [START_REF] Castro | End-to-end incremental learning[END_REF], accuracy scores are averaged over the incremental states of the system and the first, non-incremental, state is ignored. The performance level of the presented methods is much lower than that of non-incremental and balanced learning. We trained a ResNet-18 non-incrementally and using the full ILSV RC dataset and obtained a top-1 accuracy of 73.0%. The non-calibrated accuracy (F T ) obtained for ILSV RC with memory B = 20000 are 50.1% and 37.5% for sof t and strong imbalance configurations. The best results obtained for the same settings after calibration are 53.4% and 39.4% respectively. If the allowed memory is B = 5000, performance goes from 38.7% and 29.9% (non-calibrated F T ) to 45.0% and 33.1% (F T th ) for sof t and strong configurations respectively.

B Dataset iC a R L L U C I R B I C F T F T is F T p l F T th F T n e m F T b a l F T m b F T f
B Dataset iC a R L L U C I R B I C F T F T is F T p l F T th F T n e m F T b a l F T m b F T f j 5000
ILSV
Intuitively, performance for sof t imbalance (Table 2) is higher compared to that for strong imbalance (Table 3). The difference between the two configurations is largest for ILSV RC, the most difficult dataset among the three tested. With a memory of B = 10000 exemplars, the difference in performance between sof t and strong configurations for F T is 10.9%, 5.2% and 3.2% for ILSV RC, V GGF 2 and LAN D respectively. The size of the memory has also a strong influence on results. For instance, the performance of F T on ILSV RC for bounded memories B = {5000, 10000, 20000} reaches 38.7%, 45.3% and 50.1% in the sof t imbalance configuration. The combined effect of incremental learning and dataset imbalance is thus strong and, while calibration is useful, the problem remains an open one. The difference between sof t and strong imbalance configurations is also well illustrated in Figure 2. These detailed results show that the induced imbalance has a particularly important effect in early incremental states. This is normal since the importance of dataset imbalance is reduced in later incremental states, where the incremental imbalance due to the bounded memory B acts upon a large majority of classes.

B Dataset F T F T is F T pl F T th F Tnem F T bal F T mb F T f
The analysis of individual calibration methods shows that isotonic regression (F T iso ) and Platt calibration (F T pl ) have detrimental effect for both sof t and strong imbalance configurations. Both methods rely heavily on the num-ber of available class samples. The negative influence of iso and pl is larger for lower memory sizes and, within each B size, for later incremental states. This is probably an effect of lack of sufficient data in order to obtain a stable parametrization of the methods. The behavior of iso and pl in imbalanced incremental learning is different from the one previously reported in [START_REF] Niculescu-Mizil | Predicting good probabilities with supervised learning[END_REF]. There are two main differences between the two studies: (1) the algorithms used are different (deep models here and shallow models in [START_REF] Niculescu-Mizil | Predicting good probabilities with supervised learning[END_REF]) and ( 2) the amount of data available for calibration which is much smaller here.

Thresholding (F T th ) improves performance for all tested configurations. This method has the largest positive effect among all methods tested in a wide majority of cases. th performs score post-processing and is less dependent of the number of samples than iso and pl. It provides the largest improvements for B = 5000, the memory setting which corresponds to the largest imbalance for the three visual tasks with sof t and strong configurations. The results obtained for th confirm those presented in [START_REF] Buda | A systematic study of the class imbalance problem in convolutional neural networks[END_REF] for imbalanced learning. They indicate that this simple calibration method should be considered in priority for the implementation of imbalanced incremental learning applications.

Nearest-mean-of-exemplars (F T nem ) has contrasted performance. The method is beneficial for the object recognition task (ILSV RC), although with lower effect for B = 20000. For face recognition task (V GGF 2) and landmarks (LAN D) gain is observed only for sof t imbalance at 5000 budget . For ILSV RC, nem is more useful for the strong imbalance configuration than for the t one. The method works on top of the penultimate layer of the deep model but is highly dependent of the number of samples available to compute the individual class means, a property shared with iso and pl. Note also that F T nem is equivalent to an iCaRL version in which the distillation loss was ablated. The authors of iCaRL [START_REF] Rebuffi | icarl: Incremental classifier and representation learning[END_REF] report that nem classification has positive influence over a direct use of deep model predictions in all configurations tested in their paper. A main difference is that those tests were done with classification and distillation losses, with larger memory and with datasets that are initially balanced. The effect of nem is more contrasted for the imbalanced datasets tested here with a vanilla fine tuning backbone with exemplars selected based on moving mean.

Balanced fine tuning (F T bal ) has a negative effect in most configurations. The methods provides improvement over F T for ILSV RC at 5000 and 10000 budgets with sof t imbalance. The effect is particularly detrimental for strong imbalance, Note that the reported bal performance is obtained by fine tuning only the classification layer of the incremental deep models M k . Balanced fine tuning (F T bal ) is performed by creating a balanced dataset, leading to much smaller datasets, especially if there is more dataset imbalance in addition to incremental imbalance. This would explain the sub-optimal performance as the dataset imbalance is increased.

Batch mean based calibration (F T mb ) improves performance over F T for all settings with sof t imbalance, while being comparative to F T for strong imbalance. As for th , the gains are larger for lower memory size and for ILSV RC, the hardest visual task tested here. mb and f j have comparable results for sof t imbalance configurations, while f j gives slightly better results for strong imbalance. mb is the simplest of all calibration methods tested since it only exploits mean predictions for old and new classes. It only accounts for the incremental imbalance as it groups new and old classes together, regardless of their image counts.

Fisher-Jenks based calibration (F T f j ) is a refined version of mb in which both the incremental and dataset imbalance are taken into account when clustering classes. The advantage of such clustering is more obvious for strong imbalance configurations, where the dataset imbalance is more important compared to sof t imbalance. Its performance is better than that of bal and nem for both sof t and strong imbalance. f t globally has lower performance than th calibration.

A statistical analysis of the LU CIR and F T th reveal that F T th is significantly better for strong imbalance regime as compared to sof t imbalance. We compute the p-values over the accuracies at each incremental batch to ascertain the significance in the incremental setting. For ILSV RC dataset, the p-value at budgets 5000, 10000 and 20000 are 0.91 , 0.84 and 0.85 for sof t imbalance as compared to 0.02 , 0.006 and 0.005 for strong imbalance. Similarly for Land dataset the p-value for sof t imbalance is at 0.43, 0.10 and 0.001 as compared to 0.001, 0.003 and 0.0028 for strong regime. For V GGF 2, the p-values for sof t imbalance is at 0.119, 0.038 and 0.019 as compared to 0.0041, 0.0042 and 0.0017 for strong regime.

A final interesting observation is that iCaRL performance lags well behind that of F T for all datasets and tested configurations. Further, F T baseline is competitive with LU CIR and BIC, at sof t imbalance, while being clearly the preferable option in the strong imbalance regime. This comparison is contrary to the conclusions of [START_REF] Rebuffi | icarl: Incremental classifier and representation learning[END_REF], where F T has significantly worse performance compared to iCaRL. However, that evaluation was biased insofar iCaRL was using a memory of past classes while F T results obtained in absence of this memory. Our results indicate that, when running a fair comparison, the simpler F T method is clearly a better suited backbone for incremental learning with bounded memory than the state-of-the-art backbone which combines classification and distillation losses [START_REF] Castro | End-to-end incremental learning[END_REF][START_REF] Javed | Revisiting distillation and incremental classifier learning[END_REF][START_REF] Rebuffi | icarl: Incremental classifier and representation learning[END_REF].

Analysis of Expected Calibration Error

The results for Expected Calibration Error ECE for the calibration methods are presented in Tables 4 and5. The first main observation is that the value of ECE for F T is higher for ILSV RC dataset as compared to the other two datasets. This is explained by the fact that V GGF 2 and LAN D are easier to learn and F T provides significantly higher accuracy for these two datasets. Hence, the confidence is matched with high performance, which is not the case for ILSV RC.

Isotonic regression (F T iso ) and Platt calibration (F T pl ) provide very high ECE values as compared to F T , particularly for V GGF 2 and LAN D datasets.

A look at LAN D at sof t imbalance with 10000 budget, shows the accuracy at 53.9% and 84.6% for F T iso and F T pl , whereas the ECE is 0.533 and 0.841 respectively. This allows us to infer that the confidence of probabilities after F T iso and F T pl calibration is quite low, and the models actually under-calibrated. This can be partly explained by limited number of positives instances for a class, and high number of negative instances in one-vs-all calibration used in Isotonic Regression and Platt Scaling.

The results for F T nem are similar to F T iso and F T pl with very high values of ECE, particularly when the accuracy is high. We draw similar conclusions that the model is under-calibrated for F T nem as well. Note that for F T nem , the scores are calculated as the inverse of the distance to the class mean in the feature space, which are then used to derive the probabilities using the softmax function. The accuracy for F T bal are slightly lower than F T , and this is also reflected in ECE values of F T bal which are slightly higher than F T .

F T th provides the most improvement in accuracy out of all the calibration method. th mitigate the bias towards minority classes by calibrating the score for a class depending on the number of samples in the given class. It provides better performance by increasing the confidence of minority classes, though it also makes the model more mis-calibrated. ECE score for F T th is consistently higher than its F T counterpart. This shows that F T th provides improvement in overall accuracy, but at the some expense of calibration of model.

The proposed methods F T mb and F T f j provides the best calibration out of all the calibration methods. The calibration is quite similar to F T with ECE values being quite close to ones for for F T . This is an interesting results since F T mb and F T f j are the only methods which provide improvement in overall accuracy while not adversely affecting the calibration of the model.

Conclusion

We performed a study of score calibration methods in an incremental and imbalanced deep learning setting which was not explored before. Calibration methods selected from both imbalanced and incremental learning streams of research were thoroughly compared using three visual tasks, two imbalance configurations and three bounded memory sizes for incremental learning. The obtained results indicate that, while calibration is certainly useful, imbalanced class incremental learning remains an open problem. They also show that both dataset imbalance and memory size have an important impact on performance. This is particularly true for object recognition, the most difficult of the three tested tasks, and for lower memory sizes.

The performance of the evaluated calibration methods is variable. Isotonic regression and Platt calibration, which were shown to work well when enough data per class is available [START_REF] Niculescu-Mizil | Predicting good probabilities with supervised learning[END_REF], have a negative effect on results here. This behavior is explained by the scarcity of available data when working in an incremental setting. Nearest-mean-of exemplars [START_REF] Rebuffi | icarl: Incremental classifier and representation learning[END_REF] and balanced fine tuning [START_REF] Castro | End-to-end incremental learning[END_REF], the calibration methods introduced in recent incremental learning works, have contrasted and negative effects respectively. Note that, after initial experiments, an adaptation of balanced fine tuning was performed so as to fine tune only the classification layer instead of the entire network as done in [START_REF] Castro | End-to-end incremental learning[END_REF]. The batch mean based and Fisher-Jenks calibration methods introduced here have a positive effect in most of the configurations. Fisher-Jenks behaves slightly better than mean based calibration. This is explained by the fact that the first method models both dataset and incremental imbalance while the second models only incremental imbalance. The best performance in terms of accuracy is obtained by thresholding based calibration, which uses the prior class probabilities to augment the scores of minority classes. An analysis of model calibration after the calibration method shows that overall F T mb and F T f j provide the best model calibration, while also tacking the imbalance.

Finally, the results also show that vanilla fine tuning is a better backbone for class incremental learning with bounded memory compared to a fine tuning which exploits both classification and distillation losses. The performance gap between the two approaches is significant and we advocate that future developments in class incremental learning should use vanilla fine tuning as baseline.

The reported results are interesting and can be developed along the following lines: (1) improve the vanilla fine tuning backbone using recent results in imbalanced learning [START_REF] Buda | A systematic study of the class imbalance problem in convolutional neural networks[END_REF]; (2) explore other score calibration methods and (3) test the calibration methods using different model architectures 7 Declarations

Funding

The research is funded by CEA, France.

Conflict of interest

The authors declare that they have no conflict of interest.

Availability of data and material

All experiments are done using publicly available dataset. The images used for each dataset to create an imbalanced version are provided with the code.

Code availability

The code is available online at https://github.com/umangAggarwal/Class-Imbalanced-Incremental-Learning

  with d the size of the feature vector f . The second is a classifier C k : R d → Y N k which outputs the classification scores ŷi for the N k learned classes. The classification scores can then be converted to probability estimates pi to ascertain the confidence of the model. Depending on the calibration method used, F k and C k are either integrated in a single deep model or separated.

Fig. 1

 1 Fig.1Mean scores predicted by the model for samples from old and new classes using vanilla fine tuning for the ILSV RC dataset. Training is done with 100 new classes added in each iteration using vanilla fine tuning, B = 5000 bounded memory for old classes and sof t imbalance configuration as defined in Section 5. The first, non-incremental, state is not represented.

Fig. 2

 2 Fig. 2 Top-1 accuracy for ILSV RC with memory B = 5000 in sof t (left) and strong (right) imbalance configurations. To be aligned with the results from Tables2 and 3, only the incremental states are represented. (Best viewed in color.)

Table 1

 1 Means and standard deviations of image counts in the original datasets (orig) and the two imbalance configurations (sof t and strong).

	combines classification and distillation losses to counter catas-
	trophic forgetting and uses a nearest exemplar mean classifier to counter
	imbalance between past and new classes.
	-BiC [40] introduces a linear layer at the end of the classification process to
	ensure fairness between past and new classes. A distillation term which is
	closer to the original formulation from [16] compared to iCaRL is equally
	used.
	-LU CIR [17] proposes a combination of three elements to improve incre-
	mental learning. Cosine normalization is used for balancing the magnitudes

Table 2

 2 j 5000 ILSV RC 21.8 45.5 41.3 38.7 23.5 31.4 45.0 39.3 40.5 41.6 41.5 V GGF 2 61.0 84.4 78.7 81.4 42.7 65.5 85.4 81.9 81.4 84.9 85.1 Top-1 average accuracy for the sof t imbalance configuration and B = {5000, 10000, 20000} bounded memory sizes. The first two columns represent iCaRL [33]

		LAN D	64.1 86.9 80.6 84.3 37.9 76.0 88.0 85.2 81.1 85.7 86.0
		ILSV RC 23.6 48.9 45.5 45.3 32.1 38.6 49.8 44.8 45.6 46.9 46.4
	10000	V GGF 2 62.1 86.9 80.3 86 66.1 76.5 88.0 85.2 82.2 87.4 87.7
		LAN D	65.7 88.9 82.1 88.9 53.9 84.6 90.7 88.2 85.8 89 89.2
		ILSV RC 24.5 52.7 49.7 50.1 38.3 44.8 53.4 48.6 49.8 50.7 50.3
	20000	V GGF 2 62.2 88.4 81.6 90.2 80 86.1 91.0 88.8 87.3 90.5 90.9
		LAN D	65.8 90.8 83.3 92.2 75.4 90.5 92.6 90.8 91.6 92.0 92.1

  RC 13.9 21.7 24.5 29.9 17.8 23.6 33.1 31.0 29.2 28.9 30.2 V GGF 2 50.4 66.3 65.9 76.7 32.3 63.6 78.6 75.1 72.0 77.4 78.1

		LAN D	57.5 77.2 73.3 80.8 36.6 74.7 82.4 80.7 80.4 80.7 81.3
		ILSV RC 15.9 24.9 26.2 34.4 24.3 29.1 36.8 34.9 32.1 33.0 34.3
	10000	V GGF 2 51.3 68.5 67.9 80.8 55.9 72.9 81.7 78.7 77.0 80.5 81.43
		LAN D	58.8 79.4 74.9 85.7 47 82.6 86.3 84.6 84.4 85.3 85.8
		ILSV RC 16.2 27.0 27.1 37.5 29.1 33.0 39.4 37.6 34.7 36.2 37.3
	20000	V GGF 2 51.4 71.8 68.8 83.9 68.6 78.6 84.6 82.1 81.1 83.4 84.3
		LAN D	60.4 80.9 75.1 87.8 65.8 85.5 88.4 86.4 86.0 87.5 88.1

Table 3

 3 Top-1 average accuracy for the strong imbalance configuration and B = {5000, 10000, 20000} bounded memory sizes. See Table2for the description of the different methods presented.

Table 4

 4 Expected Calibration Error for the sof t imbalance configuration and B = {5000, 20000} bounded memory sizes. The first columns represent vanilla fine tuning (F T ), our baseline. The next columns are calibrated versions of F T as follows: F T iso -isotonic regression; F T pl -Platt scaling; F T th -thresholding; F Tnem -nearest-mean-ofexemplars; F T bal -balanced fine tuning; F T mb -batch mean based calibration; F T f j -Fisher-Jenks based calibration. ECE are averaged over the incremental states of the system and the first non-incremental state is ignored. is F T pl F T th F Tnem F T bal F T mb F T f j

	j

Table 5

 5 Expected Calibration Error for the strong imbalance configuration and B = {5000, 10000, 20000} bounded memory sizes.