Zeynep Sonat Baltaci
email: sonat.baltaci@enpc.fr

Kemal Oksuz
email: kemal.oksuz@five.ai

Selim Kuzucu

Kivanc Tezoren

Kerim Berkin

Alpay Konar

Sinan Ozkan

Emre Kalkan
email: skalkan@metu.edu.tr

Akbas

Berkin Kerim Konar

Alpay Ozkan

Emre Akbas

Sinan Kalkan

Class Uncertainty: A Measure to Mitigate Class Imbalance

Keywords: tailed visual recognition, class imbalance, class uncertainty, predictive uncertainty

ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

I. INTRODUCTION

The performance of data-driven, learning-based methods relies heavily on dataset characteristics. One characteristic that has received significant attention is the unequal cardinality of examples per class 1 [1], [2], [3], [4], [5], [START_REF] Ren | Learning to reweight examples for robust deep learning[END_REF], [START_REF] Shu | Metaweight-net: Learning an explicit mapping for sample weighting[END_REF], [START_REF] Jamal | Rethinking class-balanced methods for long-tailed visual recognition from a domain adaptation perspective[END_REF], [START_REF] Park | Influence-balanced loss for imbalanced visual classification[END_REF], [START_REF] Fernando | Dynamically weighted balanced loss: Class imbalanced learning and confidence calibration of deep neural networks[END_REF], [START_REF] Cao | Learning imbalanced datasets with label-distribution-aware margin loss[END_REF], [START_REF] Khan | Striking the right balance with uncertainty[END_REF], [START_REF] Menon | Long-tail learning via logit adjustment[END_REF], [START_REF] Kini | Labelimbalanced and group-sensitive classification under overparameterization[END_REF], [START_REF] Samuel | Distributional robustness loss for long-tail learning[END_REF]. This attention originates from the fact that the frequencies of object classes follow a long-tailed distribution in nature, which also manifests itself in many visual recognition datasets [START_REF] Liu | Largescale long-tailed recognition in an open world[END_REF], [START_REF] Yang | A survey on long-tailed visual recognition[END_REF], [START_REF] Oksuz | Imbalance problems in object detection: A review[END_REF], [START_REF] Krawczyk | Learning from imbalanced data: open challenges and future directions[END_REF], [START_REF] Das | On supervised class-imbalanced learning: An updated perspective and some key challenges[END_REF], [START_REF] Cheong | The hitchhiker's guide to bias and fairness in facial affective signal processing: Overview and techniques[END_REF]. Accordingly, long-tailed distribution of cardinalities is generally considered as the major reason for many performance issues encountered on such datasets. This common phenomenon may be more precisely described as "cardinality imbalance".

While cardinality imbalance is the most evident type of imbalance to observe, it does not completely capture or explain Manuscript received March 31, 2023.

† Equal contribution for senior authorship. 1 Throughout the paper, we use cardinality to refer to the number of training examples for a class.

the spectrum of factors that cause performance gaps among classes. Other contributing factors include hardness of examples or classes [START_REF] Lin | Focal loss for dense object detection[END_REF], [START_REF] Shrivastava | Training region-based object detectors with online hard example mining[END_REF] and discrepancy between the training objective and the evaluation measure [START_REF] Chen | Ap-loss for accurate one-stage object detection[END_REF], [START_REF] Oksuz | A ranking-based, balanced loss function unifying classification and localisation in object detection[END_REF]. If such factors, which we will refer to generally as "class imbalance", are not addressed properly, they lead to sub-optimal performance for certain classes.

In this paper, we challenge the aforementioned convention to rely on the cardinality for alleviating the class imbalance and make the following main contributions:

• For setting an important milestone into the class imbalance literature, we define what a class imbalance measure is and outline important features that it should have.

• We introduce a new measure, CLASS UNCERTAINTY, as the average predictive uncertainty of training examples in a class. We extensively show its effectiveness compared to using cardinality. Specifically, incorporating our measure into a diverse set of ten methods from resampling, loss reweighting, margin adjustment and multistage training; we observe better or on-par performance compared to class cardinality on long-tailed datasets as highlighted in Fig. 1(a). As an example, CLASS UNCER-TAINTY drastically reduces the top-1 error of the recent LDAM method by 3.5 -4.0% on the CIFAR-100-LT dataset.

• To facilitate studies with a broader perspective than cardinality, we curate a new dataset, SVCI-20, where the classes have equal cardinality but different hardness.

We present these contributions in the following sections: Section II includes the background and discusses why minimizing the negative log-likelihood is not robust to class imbalance. It also provides a simple taxonomy of the existing imbalance mitigation methods with a brief discussion of such methods. Section III defines what a class imbalance measure is, analyzes using class cardinality as an imbalance measure and proposes our CLASS UNCERTAINTY. Section IV demonstrates how to incorporate our measure into the existing mitigation methods. Section V presents our experimental results for class uncertainty on long-tailed datasets and our novel Resampling Reweighting Margin Adj. Multi

Mini-batch Sampling

Naïve Classifier ℓ 𝐶𝐸 (ෝ 𝒚 𝑖 , 𝒚 (𝑖) )

Loss Reweighting 𝑤 𝑐 (𝑖) ℓ 𝐶𝐸 (ෝ 𝒚 𝑖 , 𝒚 (𝑖) ) semantically-imbalanced dataset, SVCI-20. Finally, Section VI concludes the paper.

Margin Adjustment 𝑠 𝑐 (𝑖) = 𝑠 𝑐 (𝑖) -Δ 𝑐 (𝑖)

II. BACKGROUND

A. Is Naïve Classifier Robust to Class Imbalance?

Consider a classifier that minimizes the negative loglikelihood (NLL) of training examples, which is the prevalent approach in the deep learning literature [START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF], [START_REF] Szegedy | Going deeper with convolutions[END_REF], [START_REF] Ioffe | Batch normalization: Accelerating deep network training by reducing internal covariate shift[END_REF], [START_REF] He | Deep residual learning for image recognition[END_REF], [START_REF] Xie | Aggregated residual transformations for deep neural networks[END_REF], [START_REF] Huang | Densely connected convolutional networks[END_REF]. We term this classifier as the naïve classifier. Such a classifier, f Θ : X → Y, maps an image x (i) ∈ X to a categorical distribution ŷ(i) ∈ Y such that ŷ(i) c , c-th element of ŷ(i) , represents the predicted probability for class c ∈ C. Typically, ŷ(i) c is obtained through the softmax function:

ŷ(i) c = exp(s (i) c ) c ′ ∈C exp(s (i) c ′ ) , (1) with s (i)
c being the predicted logit for class c. Given y (i) ∈ {0, 1} |C| ⊆ Y, the label of the ith example, f Θ (•) is conventionally trained by minimizing Cross Entropy Loss (ℓ CE (•, •)) using mini-batch Stochastic Gradient Descent (SGD) [START_REF] Bishop | Pattern Recognition and Machine Learning (Information Science and Statistics)[END_REF]. Formally, the loss of a mini-batch is:

L = 1 m m i=1 ℓ CE (y (i) , ŷ(i) ) , ( 2
)
where m is the number of examples in the mini-batch. By definition, minimizing the NLL over training using minibatch SGD data does not prioritize any example; thereby making the naïve classifier sensitive to the class characteristics. As a result, the larger cardinality a class has, the more it is represented in the computed NLL. Specifically, as illustrated in Fig. 1(b), the random sampling of mini batches would naturally cause the sampling probability of an example from class c (α c ) to be proportional to the cardinality of the class. To state more formally, α 1 > α 2 > ... > α |C| holds true, assuming the class indices are ordered with respect to (wrt.) the cardinality in decreasing order. As a result, the loss function does not focus on the examples that are underrepresented either. Consequently, in the case of an imbalance between different classes, training is prone to overfit to the over-represented classes.

A prominent failure example of the naïve classifier is observed in training object detectors [START_REF] Oksuz | Imbalance problems in object detection: A review[END_REF]. While training typical object detectors, there are prohibitively more negative examples from background compared to very few positives from foreground (i.e., objects). Consequently, unless an imbalance mitigation method is employed while training an object detector, the naïve classifier completely ignores the positive examples, the under-represented classes in this context [START_REF] Lin | Focal loss for dense object detection[END_REF], [START_REF] Oksuz | A ranking-based, balanced loss function unifying classification and localisation in object detection[END_REF]. Therefore, the naïve classifier is not robust against class imbalance, a problem that remains still open despite a plethora of mitigation techniques.

B. Mitigating Class Imbalance

We categorize the existing class imbalance mitigation techniques into four groups. An overview is given in Fig. 1(b).

1) Resampling: This set of methods addresses class imbalance during mini-batch sampling. One standard approach is to equalize the sampling probabilities of classes:

α 1 = α 2 = ... = α |C| , (3)
known as the classical oversampling or class-balanced (CB) resampling [1]. As this approach is prone to overfitting in favor of tail classes, several mitigation strategies have been proposed. Peng et al. [2] combined CB and random sampling with a weight parameter to obtain α c whereas Kang et al. [3] introduced progressive-balanced (PB) resampling, which initializes α c s with random sampling before gradually getting replaced by CB resampling.

2) Loss Reweighting: This common group of approaches [4], [START_REF] Lin | Focal loss for dense object detection[END_REF], [5], [START_REF] Ren | Learning to reweight examples for robust deep learning[END_REF], [START_REF] Shu | Metaweight-net: Learning an explicit mapping for sample weighting[END_REF], [START_REF] Jamal | Rethinking class-balanced methods for long-tailed visual recognition from a domain adaptation perspective[END_REF], [START_REF] Park | Influence-balanced loss for imbalanced visual classification[END_REF], [START_REF] Fernando | Dynamically weighted balanced loss: Class imbalanced learning and confidence calibration of deep neural networks[END_REF] reweights the loss values of the examples to promote certain classes or examples:

L = 1 m m i=1 w (i) c ℓ CE (ŷ (i) , y (i) ), (4)
where w

(i) c
is the weight of the example i from class c. Examples include the Cost-sensitive Cross Entropy (CSCE) Loss [4] relying on the inverse of the class cardinalities as w (i) c ; and the Class-balanced Loss [5] introducing an "effective" number of examples for each class, which practically smooths the weights of CSCE. An alternative approach, Focal Loss [START_REF] Lin | Focal loss for dense object detection[END_REF], assigns example-specific weight based on prediction confidence of the classifier as an indicator of the hardness.

3) Margin Adjustment: These relatively recent methods [START_REF] Cao | Learning imbalanced datasets with label-distribution-aware margin loss[END_REF], [START_REF] Khan | Striking the right balance with uncertainty[END_REF], [START_REF] Menon | Long-tail learning via logit adjustment[END_REF], [START_REF] Kini | Labelimbalanced and group-sensitive classification under overparameterization[END_REF], [START_REF] Samuel | Distributional robustness loss for long-tail learning[END_REF] introduce margin terms to the logit of each class, which effectively demote the over-represented classes. This is achieved by subtracting a class-specific margin ∆ (i) c from the predicted logit s (i) c while computing Eq. ( 1). Specifically, we generalize the existing margin adjustment methods as:

ŷ(i) c = exp(s (i) c -∆ (i) c ) exp(s (i) c -∆ (i) c ) + k∈C\c exp(s (i) k -∆ (i) k β) , (5)
where β ∈ {0, 1} determines whether or not to take into account the margins assigned to other classes for class c. The examples include: (i) LDAM [START_REF] Cao | Learning imbalanced datasets with label-distribution-aware margin loss[END_REF] with β = 0, ∆

(i) c = τ /N 1/4 c
and τ is a hyper-parameter; (ii) Logit-adjusted Loss [START_REF] Menon | Long-tail learning via logit adjustment[END_REF] with β = 1 and

∆ (i) c = -κ log(N i / C k=1 N j
) and κ being its hyper-parameter; and (iii) Distributional Robustness Loss (DRO) [START_REF] Samuel | Distributional robustness loss for long-tail learning[END_REF], which unlike other methods, enforces margins at the feature-level.

4) Multi-stage Training: Multi-stage training methods are based on the idea of decoupling the learning between representation and classification [3]. Specifically, such methods first train the classifier with no mitigation method, and subsequently continue by training either only the last layer [3] or the whole network [START_REF] Cao | Learning imbalanced datasets with label-distribution-aware margin loss[END_REF], [START_REF] Zhang | Bag of tricks for long-tailed visual recognition with deep convolutional neural networks[END_REF] whilst leveraging an imbalance mitigation method such as CB resampling or CSCE. Several methods are built upon this training strategy, owing to its simplicity and effectiveness [START_REF] Zhang | Distribution alignment: A unified framework for long-tail visual recognition[END_REF], [START_REF] Samuel | Distributional robustness loss for long-tail learning[END_REF], [START_REF] Menon | Long-tail learning via logit adjustment[END_REF], [START_REF] Zhong | Improving calibration for long-tailed recognition[END_REF], [START_REF] Park | Influence-balanced loss for imbalanced visual classification[END_REF].

Discussion As evidenced by the overview of the related work above, existing methods mostly rely on "class cardinality" to measure class imbalance in a dataset. Specifically, among the aforementioned methods, (i) resampling methods; (ii) loss reweighting methods except Focal Loss; (iii) margin adjustment methods except a specific setting of DRO-LT Loss; and (iv) multi-stage training methods are all based upon class cardinality. In this paper, we challenge this ubiquitously adopted approach of using "class cardinality" as the sole basis for imbalance mitigation.

III. CLASS UNCERTAINTY: A NOVEL MEASURE OF CLASS IMBALANCE

In this section, following our viewpoint of a class imbalance measure, we present an analysis on class cardinality, remark its limitations and propose CLASS UNCERTAINTY to measure and mitigate class imbalance.

A. Measuring Class Imbalance

As summarized in the previous section, current literature on class imbalance relies heavily on class cardinality, making it the de-facto measure of class imbalance. A better measure promises gains for different types of mitigation techniques.

Definition 3.1 (CLASS IMBALANCE MEASURE)

A class imbalance measure is a function µ which takes in a dataset D ⊆ X × Y with |C| classes as input and returns a |C|dimensional vector. For convenience, we reuse µ to denote the output of this function, which is in R |C| .

Moreover, we introduce the following two important features (IF) that a class imbalance measure should have:

• (IF1) Ability to capture different amounts of imbal-
ance. An imbalance measure should be able to quantify how well a class is represented by its examples. As such, we expect a measure to: -(IF1a) be higher for an under-represented class than an over-represented class, and -(IF1b) decrease for the over-represented classes (or increase for the under-represented classes) as the dataset becomes more imbalanced.

• (IF2) Robustness to the amount of ineffective examples. An imbalance measure should be robust to the presence of 'ineffective' examples2 that might have populated the dataset. In essence, we desire our measure to capture imbalance not in terms of quantity (cardinality) but quality. These important features allow us to get more insights on class cardinality and our proposed measure. In these analyses, for (IF1a) it is not trivial to identify over-and under-represented classes as we argue that class imbalance is not limited to cardinality imbalance. Therefore, we use the class-wise test set errors of the naïve classifier as a valid proxy to the underlying imbalance. For (IF1b), we change the imbalance ratio (IR) of the dataset and plot how different measures vary. As for (IF2), we duplicate the training examples in favor of underrepresented classes and exploit these duplicates as ineffective examples. Please refer to Section V-A for details.

B. CLASS CARDINALITY IMBALANCE MEASURE and Its Analysis

Considering that cardinality-based methods [1], [4], [5] promote classes with fewer examples during training, we formulate the underlying measure as follows:

Definition 3.2 (CLASS CARDINALITY IMBALANCE MEA- SURE) CLASS CARDINALITY IMBALANCE MEASURE, µ C
, is an imbalance measure which is inversely proportional to N c , the number of training examples in class c. Thus, the imbalance measured for the class c is:

µ C c = 1 |C| N N c , ( 6
)
where N is the cardinality of the entire training set and |C| (the number of classes in the dataset) normalizes the measure such that c∈C µ C c = 1. The normalization enables us to provide an analysis independent from the range of the measure. Hence, if the class c has fewer examples, then the corresponding µ C c increases relative to the others. As an example, red curves in Fig. 2(a,b) demonstrate µ C c for CIFAR-10-LT and CIFAR-100-LT [5]. Analysis Here we analyse CLASS CARDINALITY IMBAL-ANCE MEASURE based on the important features from Section III-A. In order to examine (IF1a), the extent µ C c captures the relations among different classes, we plot class indices wrt. Finally for (IF2), Fig. 2(e) suggests that µ C c is very sensitive to duplicates, the ineffective samples. Overall, while µ C c has benefits and has been used extensively in the literature, our brief analysis shows that it has limitations and offers room for improvement.

C. Our Proposal: CLASS UNCERTAINTY to Measure & Mitigate Imbalance

Considering the limitations of CLASS CARDINALITY IM-BALANCE MEASURE (Section III-B), we seek to design a measure of class imbalance that not only considers the cardinality of a class but also reflects the semantic information such as hardness etc. This motivation leads us to using predictive uncertainty as a promising measure considering that it captures both epistemic uncertainty reflecting the lack of data at a point in the input space and aleatoric uncertainty, representing ambiguity and noise. More specifically, we argue that predictive uncertainty is suitable as a measure of class imbalance due to the following two main reasons:

• Defined as a measure of lack of data, the epistemic component of the predictive uncertainty by definition is a suitable measure for capturing imbalance in data across classes. An important benefit of epistemic uncertainty is that the notion of sufficiency of data is not determined by the number of examples but rather with the behavior of the machine learning model.

• Determining the noise or the ambiguity, the aleatoric component of the predictive uncertainty provides information of the example itself. To illustrate, Kendall and Gal [START_REF] Kendall | What uncertainties do we need in bayesian deep learning for computer vision?[END_REF] use depth regression task, a closely related computer vision task, to elaborate on aleatoric uncertainty. They demonstrate that this type of uncertainty captures inherently difficult aspects such as large depths and reflective surfaces. Hence, this and alike observations on the aleatoric uncertainty align with our expectation to include the semantic hardness of example in the imbalance measure. Since the deep learning-based approaches are over-confident in their predictions [START_REF] Mukhoti | Calibrating deep neural networks using focal loss[END_REF], we train T different models on the same training set, where typically T = 5, and quantify the uncertainty given the T predictions from these models, known as Deep Ensembles (DE) [START_REF] Lakshminarayanan | Simple and scalable predictive uncertainty estimation using deep ensembles[END_REF]. Formally, denoting the prediction vector of the tth model by p(i),t and the probability for class c by p(i),t c ∈ p(i),t , the predictive uncertainty of the deep ensemble on example i, u (i) , can be quantified as,

u (i) = - c∈C p(i) c log p(i) c , (7)
where

p(i) c = 1 T p(i),t c
. To adopt predictive uncertainty as a class imbalance measure and obtain a class-level imbalance measure given example-level predictive uncertainties, we aggregate examplelevel predictive uncertainties from the training set simply by the class-wise average:

Definition 3.3 (CLASS UNCERTAINTY IMBALANCE MEA- SURE) CLASS UNCERTAINTY IMBALANCE MEASURE, µ U ,
is the average predictive uncertainties of the training examples for each class. Note that unlike µ C , µ U is a function of not only the dataset D but also a model. Specifically, given u (i) as the predictive uncertainty (defined in Eq. 7) of the i th training example of class c and N c as the cardinality of class c, the unnormalized imbalance in class c is:

μU c = 1 N c Nc i=1 u (i) , (8)
which is then normalized similar to Eq. (6):

µ U c = μU c c ′ ∈C μU c ′ . ( 9
)
Blue curves in Fig. 2(a,b) illustrate µ U c for CIFAR-10-LT and CIFAR-100-LT in which µ U c presents more diverse values than µ C c that are not entirely determined by the cardinality of the classes. Analysis We now investigate how CLASS UNCERTAINTY behaves wrt. our important features in comparison to CLASS CARDINALITY IMBALANCE MEASURE. While quantifying uncertainty, we leverage a Deep Ensemble (DE) [START_REF] Lakshminarayanan | Simple and scalable predictive uncertainty estimation using deep ensembles[END_REF] whose effectiveness in yielding reliable and calibrated uncertainty estimates is proven over a wide range of datasets and models [START_REF] Lakshminarayanan | Simple and scalable predictive uncertainty estimation using deep ensembles[END_REF], [START_REF] Pinto | Regmixup: Mixup as a regularizer can surprisingly improve accuracy and out distribution robustness[END_REF], [START_REF] Kuleshov | Accurate uncertainties for deep learning using calibrated regression[END_REF]. Firstly for (IF1a), Fig. 2(f) depicts how classwise test error changes wrt. CLASS UNCERTAINTY. As a quantitative summary, the Spearman correlation coefficient ρ between class-wise test errors and the CLASS UNCERTAINTY measure is 0.92, while it is 0.82 for the CLASS CARDINAL-ITY IMBALANCE MEASURE in Fig. 2(c). This suggests that CLASS UNCERTAINTY captures the imbalance among classes better than CLASS CARDINALITY IMBALANCE MEASURE. Secondly, Fig. 2(g) demonstrates for (IF1b) that as the IR increases, the imbalance assigned to under-represented classes increases while that of over-represented ones decreases as expected from a good imbalance measure. Finally, to compare the robustness of the measures for (IF2), comparing Fig. 2(e) and Fig. 2(h) suggests that CLASS UNCERTAINTY is affected from duplicate examples less than CLASS CARDINALITY IMBALANCE MEASURE and it is a more robust measure. This analysis suggests that CLASS UNCERTAINTY is a better alternative to CLASS CARDINALITY IMBALANCE MEASURE for quantifying class imbalance.

IV. MITIGATING CLASS IMBALANCE USING CLASS UNCERTAINTY

In this section we incorporate our measure into a diverse set of imbalance mitigation methods following the taxonomy we presented in Section II-B. Fig. 3 visualizes the high-level scheme of our architecture to utilize predictive uncertainty to mitigate class imbalance. Specifically, having obtained classwise uncertainty measures µ U c (Eq. ( 9)) based on the predictive uncertainty estimates on the imbalanced training set, here we define α c in Eq. (3) for resampling methods; w in Eq. ( 4), by multiplying µ U c by the total number of classes in order to ensure that the weights of the classes sum up to |C|, which is the case once no reweighting is used and w (i) c = 1 for all c. The resulting w (i) c of the example i from the class c is, then, defined by

w (i) c = µ U c × |C|. (10)
Uncertainty-based margins (UBM): We incorporate marginbased uncertainties into LDAM [START_REF] Cao | Learning imbalanced datasets with label-distribution-aware margin loss[END_REF] and DRO-LT Loss [START_REF] Samuel | Distributional robustness loss for long-tail learning[END_REF]. Different from UBRs and UBRm, for LDAM we use the unnormalized class uncertainties from [START_REF] Jamal | Rethinking class-balanced methods for long-tailed visual recognition from a domain adaptation perspective[END_REF] considering that these methods offer normalizing the margins in their formulations and set the class margin ∆ (i) c as:

∆ (i) c = τ × μU c max c∈C (μ U c ) , (11)
with τ = 0.5 as in the original work. As for DRO-LT Loss [START_REF] Samuel | Distributional robustness loss for long-tail learning[END_REF], we again simply use ∆

V. EXPERIMENTAL RESULTS

A. Implementation Details 1) Obtaining Long-tailed Datasets: Similar to the literature [START_REF] Zhang | Bag of tricks for long-tailed visual recognition with deep convolutional neural networks[END_REF], while obtaining CIFAR-10-LT and CIFAR-100-LT, we first estimate the number of examples for each class based on their indices as follows:

N c = N × 1 IR c |C|-1 , (12)
such that N is the number of examples in each class in a balanced training set such as CIFAR-10 or CIFAR-100, IR is the imbalance ratio ∈ {50, 100}, and finally |C| is the total number of classes. Then, we randomly select N c examples from each class to obtain the long-tailed versions of the datasets.

2) Analyzing IF1(a): As suggested in Section I, we use the class-wise accuracy of the naïve classifier on the test set as a proxy to the class imbalance. We assert that these class-wise accuracies provides us an indicator of the effects of the imbalance. We use the naïve classifier as the mitigation methods are conventionally built to mitigate the effects of the class imbalance on this naïve classifier by modifying different steps on its pipeline (see Eq. ( 4), ( 3) and ( 5)). Note that these modifications require at least a proper ranking of the classes by the underlying imbalance measure such that they are promoted or demoted as intended, which is why we adopted the Spearman ranking correlation coefficient for quantification purposes. Furthermore, the accuracy/error on the test set can not simply be employed as an imbalance measure since it requires access to the test set during training, which is not possible. Note that this is also similar to the uncertainty calibration evaluation in which the methods are either tuned on the training set [START_REF] Dave | Evaluating large-vocabulary object detectors: The devil is in the details[END_REF], [START_REF] Pan | On model calibration for long-tailed object detection and instance segmentation[END_REF] or validation set, and then evaluated against the test set accuracy [START_REF] Guo | On calibration of modern neural networks[END_REF], [START_REF] Kumar | Verified uncertainty calibration[END_REF]. We use CIFAR-100-LT with an IR of 50 for this analysis; enabling us to estimate ρ for a larger population compared to CIFAR-10-LT.

3) Analyzing IF1(b): Here, we simply increase the IR of the dataset, which corresponds to removing some examples from the classes in different numbers depending on IR. Particularly, as IR increases, the number of examples removed from the under-represented classes increase; making the resulting dataset more imbalanced. Therefore, considering that cardinality is one of the contributors of the class imbalance (but not the only one as previously discussed), this provides us an easily-controllable and applicable test bed to analyze this requirement. Specifically, we adopt CIFAR-10-LT with IR of {1, 2, 10, 20, 50}. Different from IF1(a), here we prefer CIFAR-10-LT; allowing us to demonstrate the individual class behaviours in a way to be followed more easily.

4) Analyzing IF2: Here, we aim to investigate the effect of duplicating examples, which basically corresponds to using oversampling to alleviate the impact of an imbalanced dataset. In order not to be limited for a single setting of the oversampling and cover the effect more thoroughly, we interpolate the sampling probability of the class c denoted by α c in Eq. ( 3). Following Peng et al. [START_REF] Peng | Large-scale object detection in the wild from imbalanced multi-labels[END_REF], we achieve this by setting the sampling probability α c as follows:

α c = (α R c ) λ (α N c ) 1-λ , (13)
such that α R c is the probability that an example will be sampled from the class c once oversampling is used (e.g., ∀c, α R c = 0.1 for CIFAR-10 with 10 classes), α N c is the same probability but when no imbalance technique is used and finally λ ∈ [0, 1] is the interpolation factor, which we term as the duplication strength. Note that when λ = 0, the dataset is in its original setting and λ = 1 implies the dataset is oversampled such that all classes having equal number of examples. We set the duplication strength to {0, 0.3, 0.5, 0.7, 1.0} and similar to IF1(b), we use CIFAR-10-LT to present the behaviour of the classes clearly.

5) Mitigating Class Imbalance: In our implementation, unless otherwise specified, we exploit the common settings in the literature and implement our models upon the common benchmark as the official implementation of 'Bag-of-Tricks' [START_REF] Zhang | Bag of tricks for long-tailed visual recognition with deep convolutional neural networks[END_REF] by keeping the settings of the baseline models, train a ResNet-32 [START_REF] He | Deep residual learning for image recognition[END_REF] and adopt random horizontal flipping and random cropping for data augmentation. For optimization, we employ SGD with momentum using a batch size of 128 on a single GPU. We tune the learning rate of our method by using grid-search generally only in 0.1 increments between [0, 0.7]. We also note that as SVCI-20 is a new dataset, we also tuned the learning rates of the baseline methods such as naïve classifier or LDAM to provide a fair comparison. All of the hyper-parameters including the learning rates that we tuned for our method will be made public with the release of our code upon acceptance. B. Results on Long-tailed Datasets 1) Datasets and Performance Measure: Following the literature [START_REF] Zhang | Bag of tricks for long-tailed visual recognition with deep convolutional neural networks[END_REF], [START_REF] Samuel | Distributional robustness loss for long-tail learning[END_REF], [START_REF] Cao | Learning imbalanced datasets with label-distribution-aware margin loss[END_REF], [5], we use the CIFAR-10-LT and CIFAR-100-LT datasets. These are obtained by systematically reducing the number of examples for certain classes in CIFAR-10 and CIFAR-100 [START_REF] Krizhevsky | Learning multiple layers of features from tiny images[END_REF] such that the datasets follow a longtailed distribution (see Section V-A for details). Specifically, we use IRs of 50 and 100 following Zhang et al. [START_REF] Zhang | Bag of tricks for long-tailed visual recognition with deep convolutional neural networks[END_REF]. For testing, we use the original (balanced) test sets of CIFAR-10 and CIFAR-100 with 10K examples and report top-1 error.

2) Resampling: We first compare our Uncertainty-based Resampling (UBRs) with CB resampling and PB resampling [3], both of which rely on class cardinality as discussed in Section II-B. Similarly, we train the classifier by (i) simply using the class-wise sampling probabilities obtained through CLASS UNCERTAINTY (UBRs); and (ii) inspired from PB, gradually changing the weights towards our uncertainty-based weights (PB UBRs). Table I suggests that UBRs performs better than both methods on CIFAR-10-LT but not on CIFAR-100-LT, which is a more challenging dataset. For example, while the class with minimum cardinality has 50 training examples for CIFAR-10-LT, it is only 5 for CIFAR-100-LT. Note that using our PB UBRs outperforms all resampling methods consistently. For example, PB UBRs decreases the top-1 error rate of PB by 1.5% on CIFAR-100-LT on both IRs. These results suggest that our PB UBRs is more effective than existing resampling methods relying on class cardinality.

3) Loss Reweighting: Here, we consider two common baselines: (i) Focal Loss [START_REF] Lin | Focal loss for dense object detection[END_REF], as a hardness-based method [START_REF] Oksuz | Imbalance problems in object detection: A review[END_REF], and (ii) Class-balanced Loss, a cardinality-based method using the "effective" number of examples in each class [5]. Note that Class-balanced Loss is shown to be a stronger approach compared to directly relying on the number of examples [5], [START_REF] Zhang | Bag of tricks for long-tailed visual recognition with deep convolutional neural networks[END_REF]. Table I displays that:

• Weighting the naïve classifier using our CLASS UNCER-TAINTY (UBRw) outperforms Focal Loss and consistently improves performance (up to 3.77%).

• Incorporating CLASS UNCERTAINTY into Focal Loss (similar to Class-balanced Loss) outperforms Class-Balanced Loss by up to more than 2% on CIFAR-100-LT (60.40 vs. 58.33 with IR of 100).

These results demonstrate the effectiveness of our approach in loss reweighting methods. [11] as a more competitive approach; and (iii) DRO-LT Loss [START_REF] Samuel | Distributional robustness loss for long-tail learning[END_REF] with cardinality-based margins. We observe in Table I that:

• Using margins based on CLASS UNCERTAINTY in LDAM (UBM LDAM) consistently outperforms LDAM for its both settings. Specifically, our gains are significant on CIFAR-100-LT, a similar situation to what we have observed for resampling and loss weighting methods.

For example, UBM LDAM with reweighting improves its counterpart between 3.5-4% top-1 error on CIFAR-100-LT.

• As for DRO-LT Loss, UBM improves the baseline consistently across all four settings; reaching the top performance in Table I. 5) Multi-stage Training: We incorporate our imbalance measure into three multi-stage training methods following Zhang et al. [START_REF] Zhang | Bag of tricks for long-tailed visual recognition with deep convolutional neural networks[END_REF]. That is, we train a baseline model without any imbalance mitigation for the first 160 epochs, then use UBRs, UBRw and UBRw Focal Loss with our CLASS UN-CERTAINTY in the second stage. In addition to the baselines from the previous sections, we also compare our method with CAM-based sampling using class activation maps for resampling [START_REF] Zhang | Bag of tricks for long-tailed visual recognition with deep convolutional neural networks[END_REF] and CSCE [4] relying on class cardinalities. Table I shows that:

• Over eight different dataset and IR combinations, our methods perform the best on six of them with other two being relatively on-par with common baselines.

• Similar to what we observed in resampling and loss reweighting, using CLASS UNCERTAINTY in the second stage also improves the performance especially for CIFAR-100-LT. For example, UBRs performs around 1.5-2.0% top-1 error better than CAM-based resampling, its closest counterpart.

• As for loss weighting, the naïve UBRw performs the best in CIFAR-100-LT with arguably similar results on CIFAR-10-LT. We conducted ablation studies to explore the flexibility of our approach by combining CLASS UNCERTAINTY with effective number of examples [5] or using DUQ [START_REF] Van Amersfoort | Uncertainty estimation using a single deep deterministic neural network[END_REF] for more reliable uncertainties. These findings suggest potential for future improvements (see supplementary material). C. Results on Semantically-Imbalanced Datasets 1) Dataset Curation: Here, we introduce SVCI-20, a balanced dataset in terms of cardinality but semanticallyimbalanced dataset, to test our models more thoroughly and foster research in this direction. Specifically, SVCI-20 combines CIFAR-10 [46] including 10 classes with SVHN [START_REF] Netzer | Reading digits in natural images with unsupervised feature learning[END_REF] comprising of 10 digits taken from house numbers; totalling to 20 classes. Arguably, SVHN is an 'easier' dataset than CIFAR-10 considering that: (i) Existing work reports a higher accuracy for SVHN, e.g., DenseNet [START_REF] Huang | Densely connected convolutional networks[END_REF] with a depth of 40 obtains 7.00 top-1 error on CIFAR-10, while it has 1.79 top-1 error on SVHN; and (ii) from a model-free perspective, the intrinsic dimension of SVHN is 1.5× smaller than that of CIFAR-10 [START_REF] Pope | The intrinsic dimension of images and its impact on learning[END_REF]. Therefore, combining these datasets into SVCI-20 increases the semantic gap among classes and using uniform class cardinalities removes the impact of cardinality imbalance, both of which are fit for our purpose. We measure the performance using the average top-1 error of classes. II).

4) CLASS UNCERTAINTY on SVCI-20: On the SVCI-20 dataset, CLASS UNCERTAINTY is expected to be more distinctive across classes compared to Class Cardinality since it considers other aspects. Fig. 4(b) shows that CLASS UNCER-TAINTY for CIFAR-10 classes is generally higher than those for SVHN classes; making our approach different from class cardinality. Consequently, (i) our PB UBRs achieves the best results among resampling and rewighting methods; and (ii) UBM LDAM outperforms LDAM in two out of three cases ∼ 1% while being on par for N c = 100 (Table II).

VI. CONCLUSION

In this work, we demonstrate that CLASS CARDINALITY IMBALANCE MEASURE, as the de-facto measure of class imbalance, has some limitations, and our extensive analyses and experiments show that CLASS UNCERTAINTY is a better alternative. Our approach offers two important benefits. For the former, our measure CLASS UNCERTAINTY can be considered as easy-to-integrate for imbalance mitigation techniques since we only tune the learning rate while incorporating our method, and in very rare cases, such as LDAM, search for the methodspecific hyperparameters. As for the latter, our perspective to devise new measures is promising as we obtain better performance on our new test bed comprising of a semanticallyimbalanced dataset with equal number of examples such that cardinality-based approaches would fail to capture the difference across classes.

It is also worth noting that one related work [START_REF] Khan | Striking the right balance with uncertainty[END_REF] employs epistemic uncertainty to mitigate imbalance, but in a specific way that fits into their proposed loss function. Instead, we consider using predictive uncertainty to mitigate class imbalance from a broader perspective by (i) proposing a new measure based on predictive uncertainty, (ii) extensively analyzing and (iii) using the measure with ten imbalance mitigation approaches with different characteristics in imbalance setups.

Fig. 1 .

 1 Fig. 1. (a) Using our CLASS UNCERTAINTY improves top-1 error rate of all aforementioned methods (CIFAR-100-LT with an IR of 50 using ResNet-32).In particular, we obtain (i) the sampling probability of each class in resampling methods; (ii) the weights of the loss for each class; (iii) the margins to be enforced around each class; or (iv) again sampling probability in the second-stage of multi-stage training strategy using CLASS UNCERTAINTY. The numbers on top of the histograms show relative gain over cardinality-based methods. Baselines: Progressively-balanced sampling[3], class-balanced Focal Loss[5], LDAM with reweighting[START_REF] Cao | Learning imbalanced datasets with label-distribution-aware margin loss[END_REF] and deferred resampling. (b) Naïve classifier is not robust to class imbalance, giving rise to a plethora of mitigation methods: (1) "resampling" samples a balanced set of examples; (2) "reweighting" assigns a weight for each class (w (i) c ); (3) "margin-adjustment" methods assign different margins (∆ (i) c ) to the logits (s (i) c ) of different classes; and (4) "multi-stage training" first trains a naïve classifiers followed by a resampling or reweighting method to re-train or fine-tune the classifier. These methods generally rely on the cardinality of the classes.

Fig. 2 .

 2 Fig. 2. (a,b) CLASS CARDINALITY IMBALANCE MEASURE (µ C c ) vs. CLASS UNCERTAINTY (µ U c ) on CIFAR-10-LT and CIFAR-100-LT with an IR of 50. While µ C c is entirely determined by cardinality (from left to right, class cardinality decreases), µ U c presents more diverse values. (c-h) An analysis of µ C c and µ U c in terms of important features. For capturing the amount of imbalance among classes ((c) and (f)), Spearman correlation coefficient (ρ) between the imbalance measure and the test accuracy of a naïve classifier on CIFAR-100-LT is notably higher for µ U c compared to µ C c , which implies that µ U c captures imbalance better. As IR increases, both can capture the change in IR in (d) and (g) by promoting the tail classes (in red) or demoting the head ones (in blue), while the relative change across classes in µ C c is more drastic. In (e) and (h), µ C c is highly sensitive to the duplicated training examples, which do not contain new information. µ U c is relatively less sensitive especially after the duplication strength of 0.3. Therefore, µ U c is a better alternative to µ C c when considering these important features.

  µ C c vs. the test set error of class c in Fig. 2(c). The figure demonstrates that the error of a class generally increases as µ C c increases. However, there are some classes with very different number of examples but similar test errors. For example, three classes represented by the red histograms in Fig. 2(c) have very similar errors, but they have 323, 66 and 12 examples in the training set respectively; implying that µ C c does not necessarily quantify the imbalance in a class perfectly. In terms of (IF1b), Fig. 2(d) shows how µ C c changes as IR increases; suggesting that the relative change between under-(red tones) and overrepresented classes (blue tones) is drastic as the IR increases.

Fig. 3 .

 3 Fig. 3. High-level overview of our strategy. We first obtain CLASS UNCER-TAINTY (µ U c ) as the measure of imbalance using the predictive uncertainties of training examples based on DE. Then, we incorporate the resulting class imbalance measure µ U c into various methods during training the classifier.

  for reweighting methods; ∆ (i) c in Eq. (5) for marginbased methods; and finally the training strategy for multi-stage methods. Uncertainty-based resampling (UBRs): For resampling, we define α c in Eq. (3) for each class to either promote or demote it based on its uncertainty. The more uncertainty the examples from a class have, the more probability mass is. Since µ U c defined in (9) is already a valid probability distribution, we simply set α c = µ U c . Uncertainty-based reweighting (UBRw): We define the weight of an example in class c, i.e. w (i) c

c

  = µ U c . Uncertainty-based multi-stage training: Once we have the resampling and reweighting methods, incorporating them into two-stage training is straightforward: We train models without any imbalance mitigation strategy up to a certain epoch, and then fine-tune them using UBRw or UBRs.

4 )

 4 Margin Adjustment: We replace the cardinality-based margins by our CLASS UNCERTAINTY in three different methods: (i) LDAM [11]; (ii) LDAM combined by reweighting Classes in SVCI-20 SVHN Classes in SVCI-20 All Classes in SVCI-20 (a) Effect of Number of Examples (N c ) µ C c vs. µ U c on SVCI-20 with N c = 100

Fig. 4 .

 4 Fig. 4. (a) As the number of examples from each class (Nc) decreases, the error gap between CIFAR-10 and SVHN classes in SVCI-20 dataset increases for Naïve Classifier; implying each setting corresponds to a different semantic imbalance level. (b) While CLASS CARDINALITY IMBALANCE MEASURE (µ C c ) does not prioritize any class on SVCI-20, our CLASS UNCERTAINTY (µ U c ) differentiates between them. Generally, µ U c assigns more priority to CIFAR-10 classes (class index less than or equal to 9) as the harder classes.

2 )

 2 Effect of Class Cardinality on Dataset Curation: Here, we seek to construct SVCI-20 by setting N c (the number of examples taken from classes) properly. Note from Fig. 4(a) that as N c decreases: (i) Top-1 error increases since training data decreases for all classes; and more importantly (ii) the error gap among the classes of CIFAR-10 and those of SVHN in SVCI-20 also increases. As an example, when N c = 1000, their gap is ∼ 9% while it is around 20% for N c = 50. Based on these observations, we randomly sample 250, 100 and 50 training examples (i.e., N c s for all c ∈ C) to further analyse different levels of semantic imbalance, as we did with different IRs in long-tailed datasets. 3) Class Cardinality on SVCI-20: As all classes have equal number of training examples, cardinality is indistinctive across classes (the red curve in Fig. 4(b)). Consequently, except for Focal Loss and LDAM, the methods reduce to the Naïve Classifier; effectively ignoring this type of class imbalance (Table

TABLE I COMPARISON

 I OF USING CLASS UNCERTAINTY INSTEAD OF CARDINALITY WITHIN RESAMPLING, REWEIGHTING, MARGIN ADJUSTMENT, AND MULTI-STAGE TRAINING METHODS. UBRS: UNCERTAINTY-BASED RESAMPLING, UBRW: UNCERTAINTY-BASED REWEIGHTING, UBM: UNCERTAINTY-BASED MARGINS. FOR EACH GROUP, WE DIRECTLY TAKE THE RESULT OF COMPETITIVE METHODS FROM THE CORRESPONDING PAPERS (INDICATED WITH * ) TO SHOW THAT OUR IMPLEMENTATION IS INLINE WITH THE PREVIOUSLY REPORTED RESULTS. AMONG 20 DIFFERENT SETTINGS (4 DATASETS AND 5 DIFFERENT SET OF METHODS), USING OUR CLASS UNCERTAINTY ACHIEVES THE BEST RESULTS (IN BOLD) IN 16 DIFFERENT CASES. BESIDES, THE TOP SCORES IN THE TABLE (UNDERLINED AND BOLD) ARE OBTAINED WITH CLASS UNCERTAINTY AS WELL.

		Dataset Imbalance Ratio (IR)	CIFAR-10-LT 50 100	CIFAR-100-LT 50 100
		Naïve Classifier	24.26±0.35	30.18±0.52	57.14±0.08	61.48±0.40
	Resampling	CB Resampling [3] PB Resampling * [33] PB Resampling[3] UBRs (Ours) PB UBRs (Ours)	23.33±0.14 25.03 23.86±0.36 23.32±0.11 22.61±0.70 28.67±0.91 30.33±0.30 32.91 29.62±0.49 29.09±0.37	61.13±0.52 57.09 56.87±0.13 66.12±0.73 55.48±0.08 59.28±0.47 66.34±0.30 61.41 60.40±0.57 72.53±0.74
	Reweighting	Focal Loss [22] Class-balanced Loss * [5] Class-balanced Focal Loss [5] UBRw (Ours) UBRw Focal Loss (Ours)	23.91±0.36 20.73 21.29±0.39 22.51±0.61 21.63±0.42	30.49±0.68 25.43 26.64±0.59 26.72±0.59 26.54±0.28	57.50±0.07 54.68 57.60±0.48 55.23±0.53 53.85±0.71 58.33±0.49 61.32±0.58 60.40 61.41±0.35 59.23±0.42
	Margin Adj.	LDAM [11] LDAM+Reweighting * [11] LDAM+Reweighting [11] DRO-LT [15] UBM LDAM (Ours) UBM LDAM+Reweighting (Ours) UBM DRO (Ours)	23.31±0.11 N/A 20.32±0.02 14.63±0.10 22.74±0.23 20.20±0.29 14.54±0.25 17.68±0.31 27.95±0.20 22.97 23.90±0.28 17.99±0.11 27.31±0.39 21.80±0.33	57.72±0.27 N/A 54.27±0.43 46.75±0.08 55.50±0.49 50.40±0.10 46.54±0.11	61.22±0.10 57.96 57.98±0.20 51.85±0.13 59.19±0.07 54.55±0.14 51.65±0.08
	Multi-stage Methods	Two-stage Resampling: CB Resampling [3] PB Resampling * [33] PB Resampling [3] CAM-based Resampling * [33] UBRs (Ours) Two-stage Reweighting: Cost-sensitive Cross Entropy [4] Focal Loss [22] Class-balanced Focal Loss * [33] Class-balanced Focal Loss [5] UBRw (Ours) UBRw Focal Loss (Ours)	21.47±0.36 24.58 22.56±0.10 18.66 19.06±0.14 21.03±0.46 22.03±0.27 20.81 20.45±0.27 22.21±0.35 22.22±0.22	27.00±0.21 33.48 28.99±0.72 22.62 22.50±0.14 26.11±0.22 28.51±0.68 25.31 26.08±0.34 26.41±0.14 25.71±0.21	54.39±0.15 56.93 54.96±0.26 53.56 51.82±0.44 56.24±0.46 58.83±0.14 61.35 59.27±0.30 57.70 54.41±0.25 58.78±0.13 57.23±0.05 61.70±0.30 54.57 58.92 57.09±0.55 59.90±0.19 52.90±0.06 57.25±0.17 53.27±0.08 57.89±0.13

Following Cui et al.[5], an example can be named effective if it covers sufficiently different space than other examples.

ACKNOWLEDGMENTS

This work was partially supported by the Scientific and Technological Research Council of Turkey (T ÜB İTAK) through the project titled "Addressing Class Imbalance in Visual Recognition Problems by Measuring Class Imbalance and Using Epistemic Uncertainty (DENGE)" (project no. 120E494). We also gratefully acknowledge the computational resources kindly provided by T ÜB İTAK ULAKB İM High Performance and Grid Computing Center (TRUBA) and METU Robotics and Artificial Intelligence Center (ROMER). Dr. Akbas is supported by the "Young Scientist Awards Program (BAGEP)" of Science Academy, Turkey.

Supplementary Material for

Class Uncertainty: A Measure to Mitigate Class Imbalance Zeynep Sonat Baltaci 1,2 , Kemal Oksuz 3 , Selim Kuzucu 1 , Kivanc Tezoren 1 , Berkin Kerim Konar 1 , Alpay Ozkan 1 , Emre Akbas 1,4, † , Sinan Kalkan 1,4, † 1 Dept. of Computer Engineering, METU, Ankara, Turkey 2 LIGM, Ecole des Ponts, Univ Gustave Eiffel, CNRS, Marne-la-Vallée, France

3 Five AI Ltd., United Kingdom 4 Center for Robotics and Artificial Intelligence (ROMER), METU, Ankara, Turkey sonat.baltaci@enpc.fr, kemal.oksuz@five.ai {selim.kuzucu, kivanc.tezoren, berkin.konar, alpay.ozkan, eakbas, skalkan}@metu.edu.tr

I. FURTHER ANALYSES A. Using Different Uncertainty Estimation Methods

The uncertainty estimation method plays an important role for our CLASS UNCERTAINTY. Although we employ DE [3] to quantify predictive uncertainty due to its robustness over varying datasets, here we show that more reliable uncertainty quantification methods may bring in an improvement. As an initial study, we employ DUQ [5], which arguably yields more reliable uncertainties compared to DE on CIFAR-10 [4]. Fig. 1 compares the CLASS UNCERTAINTIES obtained from DE and DUQ, indicating a similar trend between the two with DUQ promoting under-represented classes more. Table I depicts that using DUQ with FL performs better on CIFAR-10-LT [1]. Note that we only provide results on CIFAR-10 since DUQ has not been applied to CIFAR-100 yet and scaling DUQ to larger datasets is beyond the scope of our work. This result shows that our approach promises further room for improvement with more reliable uncertainty estimation techniques.

B. Combining CLASS UNCERTAINTY and Cardinality

We finally investigate the benefit of combining CLASS UNCERTAINTY with class cardinality. Specifically, we obtain Manuscript received March 31, 2023.

† Equal contribution for senior authorship. the weights in Eq. ( 4) (see the main article) by combining our UBRw Focal Loss [2] and Class-balanced Focal Loss (CB FL) [1] simply by,

where w

and w

are the weights estimated by our UBRw FL and CB FL with γ ∈ [0, 1] being the weighting coefficient. Table II shows on both CIFAR-10-LT and CIFAR-100-LT that the training benefits from this combination. These results suggest the potential benefit of combining multiple imbalance measures.