Thomas A Lampert
email: tlampert@unistra.fr

Pierre Ganc

The Bane of Skew: Uncertain Ranks and Unrepresentative Precision

Keywords: Precision, Recall, Skew, AUCPR, Evaluation, Performance, ROC

While a problem's skew is often assumed to be constant, this paper discusses three settings where this assumption does not hold. Consequently, incorrectly assuming skew to be constant in these contradicting cases results in an over or under estimation of an algorithm's performance. The area under a precision-recall curve (AUCPR) is a common summary measurement used to report the performance of machine learning algorithms. It is well known that precision is dependent upon class skew, which often varies between datasets. In addition to this, it is demonstrated herein that under certain circumstances the relative ranking of an algorithm (as measured by AUCPR) is not constant and is instead also dependent upon skew. The skew at which the performance of two algorithms inverts and the relationship between precision measured at different skews are defined. This is extended to account for temporal skew characteristics and situations in which skew cannot be precisely defined. Formal proofs for these findings are presented, desirable properties are proved and their application demonstrated.

Introduction

A confusion matrix, or contingency table, describes a learning algorithm's performance when a specific threshold and dataset is assumed. To simplify reporting it is common to extract from the confusion matrix various performance measures. It is well known, however, that extracting the overall accuracy is inadequate when a dataset is highly skewed [START_REF] He | Learning from imbalanced data[END_REF]. Alternative performance measures such as receiver operating characteristic (ROC) curves offer more detailed insight into an algorithm's performance characteristics. Nevertheless, recent literature points out that these may overestimate performance when class balances are not equal [START_REF] He | Learning from imbalanced data[END_REF] and instead precision-recall (P-R) curves are preferable [START_REF] Davis | The relationship between precision-recall and ROC curves[END_REF].

Precision-recall curves are being actively researched [START_REF] Flach | The geometry of ROC space: understanding machine learning metrics through ROC isometrics[END_REF][START_REF] Davis | The relationship between precision-recall and ROC curves[END_REF][START_REF] Clémenc ¸on | Nonparametric estimation of the precision-recall curve[END_REF][START_REF] Boyd | Unachievable region in precision-recall space and its effect on empirical evaluation[END_REF] and this paper continues the trend by offering several contributions towards understanding their behaviour in skewed domains of different characteristics. For example, it is proven herein that the ranking of two (or more) algorithms (as measured by the area under precision-recall curves, or AUCPRs) can invert or re-order when different skews are taken, and the conditions under which this may take place are defined. This has direct implications when the AUCPR is employed to compare the performance of two algorithms, and also when the AUCPR is used as an optimisation function [START_REF] Davis | An integrated approach to learning bayesian networks of rules[END_REF][START_REF] Liu | Comparing evaluation metrics for sentence boundary detection[END_REF][START_REF] Yue | A support vector method for optimizing average precision[END_REF]. To this end, another contribution is to define a function that facilitates the analytical transformation of precision between different dataset skews. This last point enables practitioners to compare performances derived from differently skewed datasets.

Furthermore, this paper addresses the question of whether class skew really is defined and determinable. For example, the proportion of buildings, trees or other features in satellite imagery will vary depending on the location of the image. Besides, the number of buildings or trees that cover an area of the planet's surface could be temporal and therefore skew is not constant. Assuming that it was, could it be quantified? And how can a detector's performance be accurately evaluated without this knowledge? Granted that in this example the skew may only change fractionally over the whole planet (but more profoundly on the local level). A more dramatic example is that of monitoring bacteria growth, the skew in this case changes exponentially with time, π(t) = 2π(t -1), during the second growth phase [START_REF] Zwietering | Modeling of the bacterial growth curve[END_REF]. There are numerous other examples of domains that exhibit temporal skew from a variety of disciplines, for example road vehicle detection [START_REF] Sun | On-road vehicle detection: A review[END_REF], sleep stage identification [START_REF] Sieracki | Detection, enumeration, and sizing of planktonic bacteria by imageanalyzed epifluorescence microscopy[END_REF], landslide detection (Stumpf et al, 2012a) and others [START_REF] Ahanotu | Heavy-duty vehicle weight and horespower distributions: measurement of class-specific temporal and spatial variability[END_REF][START_REF] Fell | Discrimination of sleep stages: a comparison between spectral and nonlinear eeg measures[END_REF][START_REF] Lampert | On the detection of tracks in spectrogram images[END_REF][START_REF] Papageorgiou | Trainable pedestrian detection[END_REF][START_REF] Pavlo | Skew-aware automatic database partitioning in shared-nothing, parallel OLTP systems[END_REF][START_REF] Zink | Predicting the temporal dynamics of reproductive skew and group membership in communal breeders[END_REF], and this research should be of interest to classical and reinforcement learning paradigms alike.

With this in mind, problems can be categorised into four subsets: 1) those with known, fixed skew characteristics (most simulated and 'toy' examples fall into this category); 2) those with known, temporal skew characteristics (the bacteria example may fall into this category); 3) those with fixed skew that is indeterminable (precisely what proportion of all eye fundus images do blood vessels comprise?); 4) those with temporal skew that is indeterminable (such as the tree and building detection example above) and possibly chaotic (for example detecting red cars in road surveillance video or detecting fissures in aerial imagery, this example will be explored further in the following section).

This paper will deal with the implications of evaluating algorithms within each of these categories using real-world examples, thus emphasising the consequences of assuming incorrect skew characteristics. To clarify, the term indeterminable is used in a loose sense, that is to say that these figures could be found if all data were available, however, with current technology this doesn't seem likely (and even so, perhaps these measurements are fractal in nature and cannot be known to arbitrary precision). Furthermore, novel performance measures that give more accurate estimations of an algorithm's performance within each of the above-mentioned cases are proposed.

Background

As was briefly discussed in the introduction the overall accuracy, [TP(θ)+TN(θ)]/N (where TP(θ) and TN(θ) are the number of true positive and true negative detections at threshold θ and N is the dataset's size) is known to be inadequate when a dataset is highly skewed [START_REF] He | Learning from imbalanced data[END_REF]. A common alternative is to vary the classification threshold and measure algorithm performance using receiver operating characteristic (ROC) curves. These extract the true positive rate (TPR) and false positive rate (FPR) from multiple confusion matrices that are formed as the threshold varies, and thus form curves in the resulting two-dimensional performance space. Recent literature points out that these may overestimate performance when class balances are not equal [START_REF] He | Learning from imbalanced data[END_REF] (other conditions may be encountered that reduce their effectiveness and [START_REF] Webb | On the application of ROC analysis to predict classification performance under varying class distributions[END_REF] expound some of these). Instead precision-recall (P-R) curves are preferable [START_REF] Davis | The relationship between precision-recall and ROC curves[END_REF], which extract precision and recall (also called true positive rate, TPR, sensitivity, and hit rate) from the confusion matrices.

Precision (also called positive predictive value) is sensitive to the ratio between the number of positive (N p = TP(θ) + FN(θ), where FN is the number of false negative detections) and negative instances (N n = FP(θ) + TN(θ), where FP is the number of false positive detections) in the dataset. This ratio is defined such that φ = N p /N n or alternatively π = (φN n )/N = N p /N and is referred to as the dataset's skew. As the proportion of negative instances increases, precision decreases. Analysing its definition reveals the cause:

P (θ) = TP(θ) TP(θ) + FP(θ) , (1)
the precision measure is normalised by the summation of the number of TP and the number of FP detections. These values being derived from two different classes entails that the skew of the dataset affects the denominator. Under the assumption that the positive samples are drawn from the same distribution, and that the negative samples also all originate from the same distribution, recall is however skew insensitive. It is defined such that

R(θ) = TP(θ) TP(θ) + FN(θ)
.

(2)

As such, recall is calculated within the positive class and is therefore not dependent upon its relative size. An important contribution towards defining a performance measure that correctly represents skew has been made by [START_REF] Flach | The geometry of ROC space: understanding machine learning metrics through ROC isometrics[END_REF], which is to analytically vary skew to produce a precision-recall surface. Taking a specific dataset skew reveals a slice of this surface. [START_REF] Landgrebe | Precision-recall operating characteristic (P-ROC) curves in imprecise environments[END_REF] extend this to integrate the measure with respect to skew. The work presented henceforth follows the examples set by these authors, but introduces novel solutions that remove some mathematical limitations, formally proves the measures' properties, and discusses the implications of such work.

Motivation

To motivate this work a typical problem of feature classification is taken, in which an algorithm is evaluated upon its ability to classify pixels of the aerial image presented in Figure 1a as constituting a fissure or not (Stumpf et al, 2012b) (this problem falls into the fourth category outlined in the introduction in that the number of fissures is temporal in nature and indeterminable). The agreement of thirteen annotators in the location of these fissures is shown in Figure 1b.

The classifiers output soft class memberships, for example see Figure 1c, and therefore ROC curves can be evaluated for each of the two detection strategies used in this experiment, a Gaussian matched filter (Gauss) and Centre-Surround (C-S) filter (Stumpf et al, 2012b), for each level of agreement. This is achieved by thresholding the soft class memberships at a range of thresholds 0 ≤ θ ≤ M (where M is the maximum of the detector's response) and then counting the number of pixels that are marked as positive by both the detector and the ground truth (GT), TP, those that are not marked by either, TN, and those marked by one and not the other, FP and FN. An example of a thresholded detector response is presented in Figure 1d, and the resulting ROC curves in Figure 2. The GTs are calculated by applying progressively increasing thresholds to the annotator agreement and thus the GTs range from the pixels that one or more annotators agreed upon, ≥ 1, to those that all annotators agreed upon, ≥ 13. It is clear that as agreement increases, so does the performance of the detector. This is congruent with the strong correlations between image features (intensity and contrast) and the annotators' agreement that are present within the data (these were measured to be -0.2245, the fissures are dark, and 0.4027 respectively). This is further exemplified in Table 1, in which the level of correlation between the detectors' outputs and annotator agreement is presented; again, high correlations are found. Recent research has demonstrated that ROC curves can overestimate a detector's performance when the problem is unbalanced [START_REF] He | Learning from imbalanced data[END_REF]. Instead P-R curves are preferred [START_REF] Davis | The relationship between precision-recall and ROC curves[END_REF]. Those derived in this problem are presented in Figure 3. Interestingly, these curves directly contradict that which has up until now been presented, as annotator agreement increases the performance (in terms of P-R curves) decreases! As was discussed in the background section, the precision measure is dependent upon the skew of the data. In this experiment each level of agreement also exhibits a different level of skew and it is this that masks the true relationship hinted at by the correlation values. Furthermore, these performance figures are only valid for the skews from which each of the curves were determined.

Arbelaéz et al. provide an additional example within the problem of image segmentation and contour detection: increasing an image's resolution quadratically increases the prevalence of negative instances, but linearly increases the prevalence of segmentation borders or contours (being one dimensional entities) (Arbelaéz et al, 2011). To highlight the issue, suppose we apply the same algorithm to datasets having different skews. The resulting confusion matrices (for a given θ) are presented in Table 2. One can notice that in all the cases the true positive rate and false positive rate are equal, at TPR = 0.8 and FPR = 0.3, while precision varies. Although these values represent the same algorithm they report very different performances, making comparison between studies difficult. This is because they represent the algorithm's performance applied to a specific dataset. Moreover, the skew of a dataset may not be representative of the problem (the result of over or under sampling, or temporal skew, for example), resulting in an inaccurate representation of an algorithm's performance within a problem domain.

Measuring Performance under Different Skew Conditions

This section analyses each of the skew characteristics presented in the introduction and presents findings that help to accurately measure an algorithm's performance when applied to each.

Known Fixed Skew (First Case)

To allow for accurate performance modelling and for the comparison between P-R curves (and consequently precision values) in problems in which the skew is known and precise but has been altered, the performance differences due to skew need to be corrected. [START_REF] Flach | The geometry of ROC space: understanding machine learning metrics through ROC isometrics[END_REF] defined this problem with respect to a 3D ROC space in which the third dimension is a function of skew: "models may have been evaluated on different test sets with different class distributions, hence these points may be located in different horizontal planes [in a third dimension representing skew]." [START_REF] Boyd | Unachievable region in precision-recall space and its effect on empirical evaluation[END_REF] describe several techniques that may result in an altered skew: aggregation for cross-validation (particularly in relational domains), aggregation amongst different tasks, and over and under sampling [START_REF] He | Learning from imbalanced data[END_REF]. Several domains that are affected are: remote sensing, context recognition [START_REF] Stäger | Dealing with class skew in context recognition[END_REF], network intrusion, medical diagnostics, and image analysis.

We propose the following relationship to facilitate the transformation of the P-R curve's constituent precision values, which have been calculated using a dataset that has a known skew of π, into the value of precision that would have been found if the dataset had had some different skew π (from now on a will denote a simulated variable and its absence a variable inferred using a dataset). In order for the following relationship to be valid (along with the other results presented in this paper) it is necessary to assume that all positive samples are drawn from the same distribution, and that all negative samples also originate from the same distribution. Therefore, the only difference between two differently skewed datasets is the ratio of positive and negative samples that they include.

Theorem 1 The equation that dictates precision's transformation from one skew to another is

P π (θ) = π π + (1 -π ) π 1-π 1 P π (θ) -1 . ( 3
)
Proof. Taking the definition of precision, Eq. ( 1) and that TP(θ) is derived from recall as

TP(θ) = R(θ)πN P π (θ) = R(θ)πN R(θ)πN + FP(θ)
.

Furthermore FP(θ) can be defined in terms of the FPR such that FP(θ) = (1 -π)N FPR(θ).

Since the FPR of an algorithm is invariant to class skew under the assumption outlined previously, it can be defined in terms of precision and recall sampled at the original skew, π, such that

FPR(θ) = N πR(θ) P π (θ) -N πR(θ) N (1 -π) = πR(θ) 1 -π 1 P π (θ) -1
substituting this gives

P π (θ) = π R(θ) π R(θ) + (1 -π ) πR(θ) 1-π 1 P π (θ) -1 = π π + (1 -π ) π 1-π 1 P π (θ) -1 .
An illustration of the transformation function defined by Equation ( 3) is presented in Figure 4: Figure 4a represents the transformation of precision calculated at a skew of π = 0.01 to a number of other skews (π = 1/2, 1/10, 1/50, 1/80) and Figure 4b those that were calculated at a skew of π = 0.5 to various other skews (π = 2/5, 1/4, 1/10, 1/100). It is clear that the effect is amplified as the skew ratio π/π deviates further from one.

To demonstrate this transformation in application, the P-R curves of the two detectors evaluated using the ≥ 1 agreement GT are extracted from Figures 3a and3b. These are presented together in Figure 5a and the skew in this dataset is π = 0.0667 (to four decimal places). In Figures 5b and5c the precision values have been transformed using Equation (3) to values that would have resulted had the dataset had skews of π = 0.01 and π = 0.5 and therefore each P-R curve has a different appearance (for the sake of this example the problem's skew has been assumed to be fixed, although in reality the problem belongs to the fourth skew category outlined in the introduction).

Thomas A. Lampert, Pierre Ganc ¸arski (a) π = 0.5 Pπ(θ) P π ′ (θ) π ′ =π π ′ = 2 5 π ′ = 1 4 π ′ = 1 10 π ′ = 1 100 0.0 0.0 0.2 0.2 0.4 0.4 0.6 0.6 0.8 0.8 1.0 1.0 (b) π = 0.01 Pπ(θ) P π ′ (θ) π ′ =π π ′ = 1 80 π ′ = 1 50 π ′ = 1 10 π ′ =
(b) π = 0.01: AUCPR CS π > AUCPR GA π Recall (R) Precision (P ) C-S Gauss C 2 C 1 C 3 P 1 < P 2 P 1 = P 2 P 1 > P 2 0.0 0.0 0.2 0.2 0.4 0.4 0.6 0.6 0.8 0.8 1.0 1.0 (c) π = 0.5: AUCPR CS π < AUCPR GA π Recall (R) Precision (P ) C-S Gauss C 2 C 1 C 3 0.0 0.0 0.2 0.
An additional property of P-R curves is also illustrated in this example: the ranking of the two detectors at any particular value of recall remains constant irrespective of skew. In Figures 5b and5c three different rankings are observed depicted by the labels C 1 , C 2 and C 3 . We can make a general statement regarding this property:

Theorem 2 Two points in P-R space that share the same recall, (R * , P 1 ) and (R * , P 2 ), have the same ranking (in terms of precision) irrespective of skew.

The proof of this theorem is presented in Appendix A.

Area Under the Precision-Recall Curve (AUCPR)

The effects of a mismatch between skews upon aggregate functions, such as the AUCPR, are now discussed. A number of works explicitly optimise an algorithm with respect to its AUCPR as it offers a summary statistic for algorithm comparison.

Returning to Figures 5b and5c, the AUCPR of the C-S detector (0.1307) is greater than that of the Gaussian detector (0.1088) when π = 0.01; however, when the class balance is equal (π = 0.5) the ranking is inverted (C-S AUCPR is 0.7772 and the Gaussian AUCPR is 0.8260). As such, if the choice had been made to prefer the C-S detector due to the test dataset having a skew of π = 0.01 but in application the skew of the problem was more balanced, a sub-optimal algorithm would have been chosen (the Gaussian detector would have been the optimal choice in this case). In the same manner, sub-optimal parameter choices may be made if the skew of a training dataset is not representative of the problem's skew (as we have seen, however, Equation (3) offers a means to circumvent this problem).

More formally, if two P-R curves cross then there may exist a dataset skew, π * ∈ [0, 1], at which the rank of the algorithms calculated on either side of this value inverts, i.e.

AUCPR 1 π > AUCPR 2 π when π > π * but AUCPR 1 π < AUCPR 2 π when π < π * .
If, however, two P-R curves do not cross then the AUCPR ranking remains constant:

Corollary 1 It directly follows from Theorem 2 that if two P-R curves do not cross, i.e. P 1 (R) ≥ P 2 (R) for all R, the ranking, measured as AUCPR, remains constant irrespective of the dataset's skew, such that AUCPR 1 ≥ AUCPR 2 .

Using these findings a test can be defined to determine whether the rank of the AUC of two P-R curves inverts and, if so, Algorithm 1 can be employed to find the value of skew, π * , at which it does. Equation (3) enables us to transform a P-R curve sampled at skew π to both extremes of the skew range, π l = and π h = 1 -, where is a sufficiently small non-zero number (when π = 0 and π = 1 all non-trivial precisions are equal so adding and subtracting a small value avoids these cases). By calculating the AUCs of the P-R curves transformed to these extremes it can be determined whether an inversion has taken place in the range π l ≤ π * ≤ π h . If so, the binary search defined by Algorithm 1 allows the skew π * , at which the performance inverts, to be found. If the P-R curves cross more than once, however, the ranking may invert more than once, and therefore a linear search should instead be used (the number of inversions is equal to, or less than, the number of crossings in 0 < R < 1).

In this manner, the skew at which the rank of the Gaussian and C-S (fissure) detectors inverts was found to be π * = 0.0329678444 (to ten decimal places). Of course, the number of decimal places to which π * is relevant (x in Algorithm 1) is limited by the number of TP, FP and FN detections and therefore the size of the dataset.

Known Temporal Skew (Second Case)

The previous subsection assumed that a problem's skew is fixed to some definable value, π . Nevertheless, there are occasions in which skew may be temporal such that its behaviour over time t is defined by a function π (t). In this scenario Eq. ( 3) can be used to find the instantaneous value of precision at any point during skew's evolution by setting π to be equal to π (t i ), where 0 < t i ≤ T is the time point associated with the desired skew.

Each of these points, however, do not reflect the algorithm's mean precision over the whole temporal evolution of skew. To achieve this Equation (3) should be extended to inte-

Algorithm 1 skew search Input: P 1 (θ), R 1 (θ), P 2 (θ), R 2 (θ), π l and π h Output: π * 1: π m ← (π l + π h )/2 2: if 10 x π l = 10 x π h then 3:
{point of change found to x decimal places} 4:

return π * ← π m 5: end if 6: if sgn(AUCPR 1 π m -AUCPR 2 π m ) = sgn(AUCPR 1 π h -AUCPR 2 π h ) then 7: π m ← skew search(P 1 (θ), R 1 (θ), P 2 (θ), R 2 (θ), π m , π h ) {change in upper half} 8: else if sgn(AUCPR 1 π l -AUCPR 2 π l ) = sgn(AUCPR 1 π m -AUCPR 2 π m ) then 9: π m ← skew search(P 1 (θ), R 1 (θ), P 2 (θ), R 2 (θ), π l , π m )
PT (θ) = 1 T T 0 π (t) π (t) + [1 -π (t)] π 1-π 1 P π (θ) -1 dt (4)
in which π (t) is an integrable function over the interval [0, T ] (however, taking a discrete summation to approximate the integration relaxes this condition) that describes skew's temporal evolution, and T is the time frame under evaluation. A number of properties of P-R space (namely: interpolation, AUCPR, unachievable P-R space, and random classifier performance) are also demonstrable within P-R space and formal proofs are given in Appendix B. As when determining performance using P-R curves, the unachievable area [START_REF] Boyd | Unachievable region in precision-recall space and its effect on empirical evaluation[END_REF] of P-R space should be calculated along with the P-R curve, which is possible using the results presented in Appendix B.3. This modification allows the measure to report the mean precision value corrected with respect to any temporal characteristics of skew. Therefore avoiding the problem of measuring precision with respect to one fixed skew in a problem in which skew changes (which would result in an over-or underestimation of overall precision). It also removes the dependence upon the dataset's skew that is present in the original precision measure, which is beneficial in situations in which the dataset's skew does not reflect the problem's skew characteristics. Integrating with respect to the function π (t) results in a measure that is the least-squares fitting to all instantaneous precisions (as calculated using Eq. ( 3)) that are described by π (t)-it therefore accurately reflects the overall value of precision.

A more detailed picture of the evolution of mean precision in a problem that is affected by temporal skew can be determined by varying T . As such, snapshots of the mean precision within the time range 0 < t ≤ T can be obtained.

Of course, this method of measuring performance will not be desirable when precision is required at specific time points and in these situations the measure presented in the previous subsection can instead be used.

As an illustration, take the example of detecting bacteria in a petri dish during the second growth phase (this is an oversimplification for demonstration purposes, more information on the characteristics of bacteria population growth is presented by [START_REF] Zwietering | Modeling of the bacterial growth curve[END_REF]). A hypothetical detector exists that has an FPR of 0.2 and a TPR of 0.8 and there is a hypothetical dataset such that N = 10000, N p = 4000, and N n = 6000. Each instance in the dataset is a pixel's intensity value, and each bacterium is the size of one pixel. If we were to assume that skew is fixed according to the described dataset, therefore π = 0.4, then precision would be calculated to be 0.7273. Alternatively, if the skew's temporal evolution is accounted for using Eq. ( 4), such that T = 20, π (0) = 1/N , and π (t) = 2 t π (0) (if π (t) > 1 then π (t) = 1), precision is calculated to be 0.4689-significantly lower but more accurately reflecting the algorithm's performance within the domain.

Fixed and Temporal but Indeterminable Skew (Third and Fourth Case)

The previous two subsections discussed the classical view of P-R curves in which it is assumed that a problem's skew is well defined. It was discussed in the introduction, however, that for some problems skew is not the concrete or determinable phenomenon that it is often assumed to be. An alternative view was proposed in which skew may be unknown whilst also possibly being temporal. This section proposes a measure that accounts for this uncertainty whilst preserving the property introduced in Section 4.1, specifically the ability to compare P-R curves that have been determined upon datasets with different skews.

Related Work

Flach ( 2003) and [START_REF] Landgrebe | Precision-recall operating characteristic (P-ROC) curves in imprecise environments[END_REF] propose modifications to the precision measure aiming to better represent a problem's underlying distribution. [START_REF] Flach | The geometry of ROC space: understanding machine learning metrics through ROC isometrics[END_REF] normalises the quantities TP(θ) and FP(θ) by N p and N n (respectively), thus replacing them with the TPR and FPR:

P (θ) = TPR(θ) TPR(θ) + λ FPR(θ) (5)
where λ = N n /N p = π -1 -1 is the negative to positive ratio. Therefore λ < 1 implies that positive instances are more important and λ > 1 the opposite [START_REF] Flach | The geometry of ROC space: understanding machine learning metrics through ROC isometrics[END_REF]. Normalising the number of TP and FP detections by the size of their respective classes removes any dependence upon the skew of a dataset. This measure only allows for the analytical variance of skew however and does not collapse this third dimension into a measure that represents performance over a range of skews. [START_REF] Landgrebe | Precision-recall operating characteristic (P-ROC) curves in imprecise environments[END_REF] propose an AUC measure that integrates along this third dimension of skew and therefore an alternative form of precision can be inferred from it, such that

P (θ) = λ 2 λ 1 TPR(θ) TPR(θ) + λ FPR(θ) dλ . (6)
To simulate a dataset that contains only negative instances TPR (θ) must equal zero for any threshold θ, and for this to take place the quantity λ must tend towards infinity. This means that the range 0 ≤ λ < 1 represents higher positive-to-negative skews and the range 1 < λ ≤ ∞ represents higher negative-to-positive skews. Moreover, the quantity λ is undefined when π = 0.

The form of Equation ( 6) implicitly gives higher weight to lower values of λ. This complicates the measure's interpretation as the weighting function is not explicitly defined. Furthermore it imposes an assumption upon the distribution and relative importance of skew values that may not be valid. In the case that a temporal evolution of skew is known (and therefore their relative importances), the measure outlined in Section 4.2 may be usedthose skews that are more likely, occur more often in π (t) and therefore are given higher weight. There are examples however, in which the relative importance of different skews is unknown and/or equal. This subsection deals with the cases in which skew is indeterminable (in either a temporal or fixed situation). For example, in the fissure detection problem that motivated this work, the problem's skew can vary within a reasonable range of skews. Its temporal pattern, however, is unknown and therefore the weighting of specific skews cannot be inferred (and may not even be a valid assumption to make as the factors that influence the prevalence of fissures are many and complicated) and thus the weighting, w, of each skew is equal, i.e. w(π ) = 1 for all π (cf. the following subsection).

The Proposed Form

Instead, using Equation (3) we can derive an integrated precision measure in terms of TPR and FPR that avoids these issues, such that Theorem 3

P (θ) = 1 π 2 -π 1 π 2 π 1 π TPR(θ) π TPR(θ) + (1 -π )FPR(θ) dπ . ( 7
)
where π 1 , π 2 ∈ R, such that 0 ≤ π 1 < π 2 < 1, are the lower and upper bounds on skew. This measure is also normalised to lie within precision's original range, P (θ) ∈ [0, 1].

Proof. From the proof of Theorem 1 we have

P π (θ) = π π + (1 -π ) π 1-π 1 P π (θ) -1 = π R(θ) π R(θ) + (1 -π ) πR(θ) 1-π 1 P π (θ) -1 (8)
and using the equivalences also shown in the proof of Theorem 1 this can be expressed as

P π (θ) = π TPR(θ) π TPR(θ) + (1 -π )FPR(θ) . (9)
Finally, introducing the integration to account for skew uncertainty gives

P (θ) = 1 π 2 -π 1 π 2 π 1 π TPR(θ) π TPR(θ) + (1 -π )FPR(θ)
dπ .

As such, Equation ( 7) overcomes the limitations outlined in Section 4.3.1 as follows:

the terms TPR and FPR are values that are independent of skew (and represent a balanced dataset); -by weighting the term TPR in addition to FPR it is possible to represent any skew analytically (provided that TPR(θ) + FPR(θ) > 0); -the integration gives equal weighting to the more negative-than-positive, 0.0 ≤ π < 0.5, and more positive-than-negative, 0.5 < π ≤ 1.0, ranges.

These properties are formally proved in Appendix C, Theorems 6 and 7.

As a consequence, an algorithm's performance when applied to a problem with imprecise skew can be accurately represented. Furthermore, comparisons between integrated precision-recall ( P-R) curves determined upon different datasets, or upon the same dataset using different ground truths (each having different class skews) can be made (provided that they are determined using the same limits). As was presented in relation to P-R space the following properties are demonstrable within P-R space and formal proofs are given in Appendix C: interpolation, AUCPR, unachievable P-R space, and random classifier performance. Furthermore, the unachievable area of P-R space can be determined using the results presented in Appendix C.3.

As has been discussed, the proposed measure gives equal weight to each point of skew due to the fact that in cases three and four presented in the introduction it is assumed that no information exists to suggest otherwise. To increase generality however, Eq. ( 7) can be modified to account for situations in which the relative weighting of skews are known. By introducing a weighting function, w(π ), such that

P (θ) = 1 π 2 -π 1 π 2 π 1 w(π )π TPR(θ) π TPR(θ) + (1 -π )FPR(θ) dπ
where w(π ) is a positive function defined on the interval [0, 1] that gives weight to each skew such that π 2 π 1 w(π ) dπ = 1, this is achieved. Furthermore, except for the analytical representation of the minimum achievable AUC presented in Eq. ( 20) of Section C.3 and the performance of a random classifier, the findings presented in Appendix C are also valid assuming that the weighting function is appropriately introduced. Trapezoidal integration can be used to calculate the minimum achievable AUC PR and the function

PRC (θ) = 1/(π 2 -π 1 ) π 2 π 1
w(π )π dπ describes the performance of a random classifier when a weighting function w(π ) is employed. Expressed in this way, Langrebe et al.'s implicit weighting in Eq. ( 6) can be defined as w(π ) = 1/π 2 .

In some situations it is desirable to ignore the higher positive-to-negative skew ranges (0.5 < π < 1) and the proposed measure gives full flexibility to satisfy this requirement by leaving the selection of the skew range to the practitioner's discretion.

Discussion

Revisiting the example used to motivate this work (see Section 3) whilst applying the advances presented in the preceding sections produces P-R curves such as those presented in Figures 6a and6b. Each curve is calculated using a different GT and therefore each is derived from data having different skews. The bounds were determined by expert opinion due to limited data and were π 1 = 0.0 and π 2 = 0.5 (a method for experimentally estimating these will be discussed next), these values represent the variance of skew that arises from its temporal and uncertain nature. As no further information of the relative frequencies of these skews is available and the factors that influence skew in this problem are so many, each is given equal weighting in the evaluation (this will be discussed further in the following text). The P-R curves are now comparable as the influence of skew has been normalised and as such the measured performances remain faithful to the correlations observed between the annotators' opinions and the detectors' outputs, see Table 1. The difference between the performance of each detector is smaller when compared to the P-R curves presented in Figure 3, when the variance of skew is considered. For example the difference between the AUCPRs of the C-S and Gauss algorithms when using the ≥ 5 and ≥ 8 GTs are 0.1042 and 0.1500 respectively but these are reduced to 0.0392 and 0.0569 when taking the proposed AUC PR. [START_REF] Boyd | Unachievable region in precision-recall space and its effect on empirical evaluation[END_REF] propose to normalise the AUCPR measure with respect to the unachievable area. Table 3 presents the AUCs attributed to the P-R curves presented in Figure 3a normalised in this manner. It is clear that this method does not resolve the incorrect ordering that the P-R curves produce. The minimum achievable area of P-R space is not dependent upon the performance of the algorithm but on the skew of the dataset and therefore it cannot be used to correct such disparities. Instead, as was found in Figures 6a and6b, the AUC PR reports the correct ordering (both with and without normalising with respect to the unachievable P-R space).

To compare the measure to that inferred from Langrebe et al.'s AUC measure it is necessary to express the integration range as λ . Since λ is undefined when π = 0 we use the range λ 1 = 1 and λ 2 = 100, which is equivalently expressed as π 1 = 0.0099 (to four decimal places) and π 2 = 0.5 (from one-hundred times as many negative samples as positive samples, to a balanced dataset). The results are presented in Figures 6c and6d. It is noticeable that precision is lower for any given value of recall because the measure gives more weighting to lower skew values. Consequently, as λ 2 increases (and equivalently π 1 → 0) the resulting curves are pushed towards the lower-left corner of P-R space due to this implicit weighting (if λ 2 is set to a sufficiently high value the curves are indistinguishable from the axes). This would be representative of the precision if it was the case that lower skews are more likely and therefore more important, however, this assumption cannot be said to be generally true. Returning to the running example, the results of representing performance relative to the skew of the problem's domain (using P-R and P-R curves) instead of the skew of the dataset or the normalised skew represented by ROC curves gives different insight into an algorithm's performance. This is illustrated by comparing Figures 6a and6b with Figures 6e and6f (for ease of reading these figures are repetitions of those presented in Figure 2), and Figure 3. For example, in Figure 6e the Gaussian detector appears to be an almost perfect detector when the highest agreement GT is under question (with an AUC of 0.9910), whereas Figure 6a reveals that there is still room for improvement (with an AUC of 0.9454). A larger spread between the curves that result from the highest and lowest agreement GTs is also observed in Figures 6a and6b when compared to Figures 6e and6f, giving a more detailed picture of the detector's limitations (the difference between the AUCs of the highest and lowest curves in Figure 6a is 0.3458 whereas in Figure 6e it is 0.1760, and 0.3853 in Figure 6b compared to 0.2543 in Figure 6f). Of course the extent of these differences are dependent upon two facts: the performance of the detector, and the difference in skew between the evaluation dataset and the integration limits chosen. In this example the detectors achieve state-ofthe-art performance, in other situations the differences will be more profound. Figure 5 demonstrates how variable P-R curves are with respect to a dataset's skew, however, the ROC curves presented in Figures 6e and6f would have been the same irrespective of the skew of the dataset from which they were calculated (assuming that the positive and negative samples are drawn from the same distributions, Webb and Ting 2005 discuss the behaviour of ROC curves when these assumptions are not valid).

Owing to the fact that ROC curves represent an algorithm's TPR and FPR they normalise the effects of skew and therefore report a performance that is primarily informative when the problem has a balanced skew. The work presented within this article removes this assumption, whilst admitting that a problem's skew characteristics are rarely concretely known (although offers tools for use when it is known but varies from the dataset used during evaluation) and therefore enables practitioners to obtain the expected, or average, curves for the range of skews that are identified. The reporting of this curve, however, smooths over any extreme skews that may be present in its temporal evolution, in the case of P-R curves, or at either end of the range π 1 and π 2 , in the case of P-R curves, and these may be of use to the practitioner. The work presented in Section 4.1 can be used to alleviate this limitation by allowing the practitioner to obtain an algorithm's performance curves at either extreme of the range of skews under question, in addition to its mean or expected curve, and thus model performance's variance.

As such, the findings presented in Section 4.1 and the P and P measures give practitioners a more accurate summary of an algorithm's performance when applied to the different situations presented in the introduction. The term more accurate is used to imply that the measure is specific to the skew characteristics of a problem and not simply the dataset that has been selected for evaluation.

The non-linearity of the relationship between precisions calculated at different skews, Eq. ( 3) and Fig. 4, highlights the necessity of integrating over the range π 1 to π 2 or the function π (t). Simply calculating a definite precision at the middle of this range, π = (π 1 + π 2 )/2, for example, does not provide an estimate of the mean performance over the whole range (instead it results in an over-or under-estimation of precision). This also implies that imposing a fixed skew upon a problem in which the skew varies results in an incorrect performance estimate. It has been emphasised that the skew present in datasets not only varies between domains but also between datasets representing the same domain. It may therefore not be possible to know the problem's skew with certainty (the third category proposed in the introduction).

An empirical evaluation of the skews found in a number of datasets is presented in Table 4. Information of this sort allows us to estimate the variance of skews found within a problem when it cannot be analytically determined. It is reasonable to assume that the limits π 1 and π 2 of P are chosen such that π 1 = π -3π and π 2 = π + 3π (under the condition that 0 ≤ π 1 < π 2 < 1), where π represents the mean and π the standard deviation, therefore making the measure representative of skew's variance (a value of three standard deviations is chosen such that approximately 99.73% of the skew's variance is represented, assuming that it is normally distributed). This enables accurate representation of an algorithm's performance applied to a domain and not a specific dataset.

To evaluate the fissure detector's performance thus far π 1 and π 2 have been estimated by an expert, now an additional example will be introduced to demonstrate the estimation of these limits using a large spread of data. For example, four datasets of eye fundus images (STARE, ARIA, HEI-MED, and DRIVE), which contain markings of retinal blood vessels, were analysed in such a way. The markings of each annotator were treated separately, forming seven skew estimations. The mean and standard deviation were found to be π = 0.129 and π = 0.035, and as such π 1 and π 2 would be chosen such that π 1 = 0.023 and π 2 = 0.235. Of course this kind of analysis may not always be possible, and in this case reasonable limits may be estimated (such as for the fissure detection problem). This additional information may also lead to the application of an alternative skew weighting (as was discussed in Section 4.3.2). For example, assuming that the skew in eye fundus datasets is normally distributed, Eq. ( 7) can be modified as described in Section 4. place within the range π 1 = 0 and π 2 = 1). For the following examples, however, we will continue to use the uniform weighting that is implicit in Eq. ( 7).

Further Applications

The following is now possible: Optimisation: It was demonstrated in Section 4.1 that making use of a training set that does not represent the problem's skew can lead to suboptimal design choices. This is alleviated by explicitly modelling the skew range at which the algorithm is targeted.

Domain performance: By adjusting the skew (by removing negative instances) of the fissure image presented throughout this paper from 0.067 (using the ≥ 1 GT) to an to arbitrary value of 0.125 and of the STARE dataset from 0.076 (using AH GT) to an arbitrary value of 0.141, it is demonstrated in Figure 7 that the proposed measure allows the performance of an algorithm to be effectively modelled within its intended application domain, irrespective of the skew of the dataset upon which it was determined 1 (note that the P-R curves are equal and therefore appear as one).

Different ground truths: As has been demonstrated previously, skew's effect on precision prohibits comparison between different GTs. Figure 8 presents P-R curves derived from the Gaussian and MLVessel classifiers evaluated upon the fissure and STARE datasets (respectively) but using different GTs. The results from the fissure dataset (Figure 8a) demonstrate that the P-R curves indicate a large difference in performance when evaluating upon the two different GTs, however, when skew is normalised the P-R curves demonstrate that the detector models both annotators roughly equally. Inspecting the STARE results (Figure 1 These results were obtained using the Gaussian detector applied to the fissure image and the MLVessel v1.4 package [START_REF] Soares | Retinal vessel segmentation using the 2-D Gabor wavelet and supervised classification[END_REF] applied to the STARE dataset. The effect of skew on precision diminishes as FPR reduces so to emphasise the effects in the STARE dataset, 10% of the output was affected by uniformly distributed random noise (within the range 0-255). 8b) both P-R evaluations result in similar performance, with the higher achieving varying with recall. As mentioned, however, these performances are not comparable as each ground truth has a different skew. By normalising skew, the P-R curves clearly show that the algorithm better models the VK ground truth. It is worth noting that any of the methods aimed at normalising skew presented in this paper, Equations ( 3), ( 4) and ( 7), may be used to reveal these trends.

Source code for all the functions and the documented experiments presented within this paper is available from http://sites.google.com/site/tomalampert.

Conclusions

This paper has introduced a number of interesting consequences of measuring performance with precision-recall curves. An implicit assumption in most evaluations is that skew is constant, however, this paper has given counter-examples to this assumption and has proposed that skew can have temporal variance as well. Furthermore, it is rarely possible to accurately measure a problem's skew. This has wide reaching consequences when attempting to accurately measure an algorithm's performance within a problem domain. It has been shown that accounting for temporal changes in skew results in a vastly different value of precision to that obtained when ignoring it.

Several methods to aid in the evaluation of algorithms in domains that have fixed and temporal skews have been presented. A method for transforming the precision measure into different levels of skew has been proved, for use when a dataset's skew characteristics do not match those of the problem (e.g. through the use of over or under sampling). It has been demonstrated that an algorithm's rank, as measured by the AUCPR, is not constant with respect to skew. The condition under which this variance occurs has been defined, and a test and method for finding at which skew AUCPRs become equal, proposed. Furthermore, a method for calculating precision whilst accounting for temporal fluctuations has been defined for use in domains in which skew is not fixed. Using these findings the correct form of an integrated precision measure has been derived for use in domains in which the characteristics of the problem's skew are not known. Finally, example applications for each of these proposals have been demonstrated.

A Precision Properties

Theorem 2 Two points in P-R space that share the same recall, (R * , P 1 ) and (R * , P 2 ), have the same ranking (in terms of precision) irrespective of skew.

Proof. Keeping in mind the definition of precision, Equation (1), and that TP = πR * N ; there are three cases, each of which are illustrated in Figure 5, such that:

C 1 P 1 > P 2 = πR * N πR * N + FP 1 > πR * N πR * N + FP 2 = πR * πR * + (1 -π)FPR 1 > πR * πR * + (1 -π)FPR 2 since FP = FPR(1 -π)N . Therefore (1 -π)FPR 1 < (1 -π)FPR 2 for all π > 0.
C 2 P 1 = P 2 . Under the same skew π and the same dataset size N , TP 1 = TP 2 . Therefore, from Eq. ( 1), FP 1 = FP 2 and since FP = FPR(1 -π)N , (1 -π)FPR 1 = (1 -π)FPR 2 for all π. C 3 P 1 < P 2 . As C 1 but transpose P 1 and P 2 .

B Temporally Integrated Precision Properties

B.1 P-R Interpolation

Due to the discretisation of the threshold upon the algorithm's output, two points in P-R space may be distant from each other. In this case, interpolation may be employed to estimate the connecting line.

By modifying [START_REF] Davis | The relationship between precision-recall and ROC curves[END_REF] method interpolation between two points on the P-R curve, a and b, is non-linear and can be achieved such that

TP(θa) + i Np , 1 T T 0 π (t) π (t) + [1 -π (t)] π

B.2 Area Under P-R Curve

It is often advantageous to determine the AUC in ROC space. This measure is often used to optimise an algorithm, and maximising the area under a P-R curve has been shown to be more favourable than optimising the area under a ROC curve [START_REF] Davis | The relationship between precision-recall and ROC curves[END_REF] (in ROC space it is possible that two AUCs are similar whereas the equivalent AUCs in P-R space are not). An equivalent measure can be defined for a P-R curve under the assumption that a curve's constituent points are sufficiently close, which can be ensured by employing the interpolation outlined in Eq. ( 10). The AUC PR measure can therefore be approximated using trapezoidal integration [START_REF] Bradley | The use of the area under the ROC curve in the evaluation of machine learning algorithms[END_REF], such that

AUC PR = 1 0 PT (θ) dθ ≈ H i=2 [R(θ i ) -R(θ i-1 )] PT (θ i-1 ) + PT (θ i ) 2 ( 11
)
where H is the number of threshold levels used.

B.3 Unachievable P-R Space [START_REF] Boyd | Unachievable region in precision-recall space and its effect on empirical evaluation[END_REF] show that the definition of precision results in an unachievable area of P-R space that is dependent upon the skew of a dataset, π = Np/N . The same is true for the temporally integrated precision measure.

Theorem 4 Temporally integrated precision PT (θ) satisfies

PT (θ) ≥ 1 T T 0 π (t)R(θ) π (t)R(θ) + [1 -π (t)] dt. (12)
Proof. The definition of PT (θ), Eq. ( 4), states that

PT (θ) = 1 T T 0 π (t) π (t) + [1 -π (t)] π 1-π 1 Pπ (θ) -1 dt
from the definition of precision, Eq. ( 1) and the definition of recall, Eq. ( 2), TP(θ) = πR(θ)N (since FN(θ) = πN -TP(θ)), we have

PT (θ) = 1 T T 0 π (t) π (t) + [1 -π (t)] π 1-π TP(θ)+FP(θ) TP(θ) -1 dt = 1 T T 0 π (t) π (t) + [1 -π (t)] π 1-π FP(θ) πR(θ)N dt = 1 T T 0 π (t)R(θ) π (t)R(θ) + [1 -π (t)] FP(θ) (1-π)N dt = 1 T T 0 π (t)R(θ) π (t)R(θ) + [1 -π (t)] FP(θ) Nn dt
and since the number of FP detections cannot be greater than the number of negative instances, FP(θ) ≤ Nn, we have

PT (θ) ≥ 1 T T 0 π (t)R(θ) π (t)R(θ) + [1 -π (t)]
dt.

If a point in P-R space satisfies Eq. ( 12) then it is achievable. As mentioned within the main text (Section 4.2), the minimum AUC within P-R space (AUC PR MIN ) cannot be analytically defined because the form of the skew function π (t) that will be used in application is unknown. Nevertheless, from Eq. ( 12) we know that the minimum achievable P-R curve is

PT (θ) = 1 T T 0 π (t)R(θ) π (t)R(θ) + [1 -π (t)] dt (13)
and therefore Eq. ( 11) can be used to estimate AUC PR MIN . An algorithm's reported AUC should therefore be AUC PR -AUC PR MIN . In fact, a normalised AUC measure has been proposed [START_REF] Boyd | Unachievable region in precision-recall space and its effect on empirical evaluation[END_REF] that can be modified to suite this case

AUC PR = AUC PR -AUC PR MIN 1 -AUC PR MIN . (14)
Nevertheless, as AUC PR MIN is only dependent upon π (t), and assuming that this function remains constant between evaluations, the relative ranking of an algorithm's performance will also remain equal.

B.4 Performance of a Random Classifier

It is often desirable to represent the performance of a random classifier in ROC space. In P-R space a random classifier's curve is entirely dependent upon the function π (t).

Theorem 5 A random classifier produces a constant P-R curve equal to

PRC (θ) = 1 T T 0 π (t) dt. (15)
Proof. From the definition of P in Eq. ( 4)

PT (θ) = 1 T T 0 π (t) π (t) + [1 -π (t)] π 1-π 1 Pπ (θ) -1 dt = 1 T T 0 π (t) π (t) + [1 -π (t)] π 1-π TP(θ)+FP(θ) TP(θ) -1 dt = 1 T T 0 π (t) π (t) + [1 -π (t)] π 1-π TPR(θ)πN +(1-π)N FPR(θ) TPR(θ)πN -1 dt since TP(θ) = TPR(θ)πN , FP(θ) = (1 -π)N FPR(θ).
For a random classifier TPR(θ) = FPR(θ) [START_REF] Fawcett | An introduction to ROC analysis[END_REF] therefore

PRC (θ) = 1 T T 0 π (t) π (t) + [1 -π (t)] π 1-π π+(1-π) π -1 dt = 1 T T 0 π (t) π (t) + [1 -π (t)] dt = 1 T T 0 π (t) dt.
Corollary 2 It directly follows that the AUC PR of a random classifier, within the bounds of 0

≤ R ≤ 1, is defined such that AUC PR RC (θ) = 1 T T 0 π (t) dt.

C Integrated Precision Properties

The properties of the integrated precision measure presented in Section 4.3.2 are now formally proved. For brevity the normalisation factor will henceforth be referred to as γ = 1/(π 2 -π 1 ). We start by proving an alternative but equivalent form of Eq. ( 7) that will be the starting point of the remaining proofs in this section.

Theorem 6 The integrated precision measure P (θ), defined in Eq. (7), is equivalent to

P (θ) = γ π 2 π 1 π TP(θ) π TP(θ) + (1 -π )φFP(θ) dπ . (16)
Proof. Since FPR(θ) =

FP(θ)

Nn and TPR(θ) = TP(θ)

Np . Substituting these into Eq. ( 7) gives

= γ π 2 π 1 π [TP(θ)N -1 p ] π [TP(θ)N -1 p ] + (1 -π )[FP(θ)N -1 n ] dπ = γ π 2 π 1 π TP(θ) π TP(θ) + (1 -π )Np[FP(θ)N -1 n ] dπ = γ π 2 π 1 π TP(θ) π TP(θ) + (1 -π ) Np Nn FP(θ) dπ = γ π 2 π 1 π TP(θ) π TP(θ) + (1 -π )φFP(θ) dπ
Equation ( 7) is based solely upon the true positive and false positive rates. Furthermore varying the skew of the dataset is equivalent to varying the performance of the algorithm, measured in terms of true positive and false positive rates. Such that increasing (or decreasing) TPR(θ) by a factor of π along with decreasing (or increasing) FPR(θ) by a factor of 1 -π is equivalent (in terms of precision) to changing the skew of the dataset from being balanced to having a skew of π .

Theorem 7 The proposed precision measure P (θ), defined in Equation (7), can analytically represent a dataset having any skew.

Proof. Theorem 6 shows that P (θ) is independent of dataset skew and is instead dependent upon π . This term balances TP(θ) and FP(θ). A dataset without positive instances (Np = 0) results in zero TP detections, and similarly a dataset containing no negative instances (Nn = 0) results in zero FP detections. These represent the two extremes of dataset skew and can be analytically represented in Eq. ( 7) by π = 0 and π = 1 respectively. Any skew in between these extremes is therefore contained within the range 0 ≤ π ≤ 1.

C.1 P-R Interpolation

Due to the discretisation of the threshold upon the algorithm's output, two points in P-R space may be distant from each other. In this case, interpolation may be employed to estimate the connecting line.

By modifying [START_REF] Davis | The relationship between precision-recall and ROC curves[END_REF] method interpolation between two points on the P-R curve, a and b, is non-linear and can be achieved such that

TP(θa) + i Np , γ π 2 π 1 π (TP(θa) + i) π (TP(θa) + i) + (1 -π )φFP(θa) + Bi dπ ( 17
)
where θa and θ b are thresholds such that 1 ≤ i < TP(θ b ) -TP(θa), B = FP(θ b )-FP(θa)

TP(θ b )-TP(θa) and i ∈ N.

C.2 Area Under P-R Curve

It is often advantageous to determine the AUC in ROC space. This measure is often used to optimise an algorithm, and maximising the area under a P-R curve has been shown to be more favourable than optimising the area under a ROC curve [START_REF] Davis | The relationship between precision-recall and ROC curves[END_REF] (in ROC space it is possible that two AUCs are similar whereas the equivalent AUCs in P-R space are not). An equivalent measure can be defined for a P-R curve under the assumption that a curve's constituent points are sufficiently close, which can be ensured by employing the interpolation outlined in Eq. ( 17). The AUC PR measure can therefore be approximated using trapezoidal integration [START_REF] Bradley | The use of the area under the ROC curve in the evaluation of machine learning algorithms[END_REF], such that AUC PR =

1 0 P (θ) dθ ≈ T i=2
[R(θ i ) -R(θ i-1 )] P (θ i-1 ) + P (θ i )

2 ( 18
)
where T is the number of threshold levels used.

C.3 Unachievable P-R Space [START_REF] Boyd | Unachievable region in precision-recall space and its effect on empirical evaluation[END_REF] show that the definition of precision results in an unachievable area of P-R space that is dependent upon the skew of a dataset, π = Np/N . The same is true for the integrated precision measure.

Theorem 8 Integrated precision P (θ) satisfies

P (θ) ≥ γ π 2 π 1 π R(θ) π R(θ) + 1 -π dπ . ( 19
)
Proof. The definition of P (θ), Eq. ( 7), states that

P (θ) = γ π 2 π 1 π TP(θ) π TP(θ) + (1 -π )φFP(θ) dπ ≥ γ π 2 π 1 π TP(θ) π TP(θ) + (1 -π )φ(1 -π)N dπ
since the number of FP detections cannot be greater than the number of negative instances, FP(θ) ≤ Nn.

From the definition of recall, Eq. ( 2), TP(θ) = πR(θ)N (since FN(θ) = πN -TP(θ)). Therefore,

P (θ) ≥ γ π 2 π 1 π R(θ)πN π R(θ)πN + (1 -π )φ(1 -π)N dπ = γ π 2 π 1 π R(θ) Np N N π R(θ) Np N N + (1 -π )φ(1 - Np N )N dπ = γ π 2 π 1 π R(θ)Np π R(θ)Np + (1 -π ) Np Nn Nn N N dπ = γ π 2 π 1 π R(θ)Np π R(θ)Np + (1 -π )Np dπ = γ π 2 π 1 π R(θ) π R(θ) + 1 -π dπ .
If a point in P-R space satisfies Eq. ( 19) then it is achievable. Nota bene, a point's achievability is only dependent upon π 1 and π 2 -the analytical skew-and is therefore independent of a dataset.

The curve that represents the boundary between the unachievable and achievable regions of P-R space is defined by the worst performing algorithm [START_REF] Boyd | Unachievable region in precision-recall space and its effect on empirical evaluation[END_REF], i.e. every negative instance is classified as positive. This P-R curve therefore passes through (0, 0) and 1, π 2 +π 1 2 (this can be easily verified by substituting TPR = 1 and FPR = 1 into Eq. ( 16)). Interpolation between two points on a P-R line is given by Eq. ( 17), and is therefore R, γ π 2 π 1 π R π R+1-π dπ , where R = i Np , for 0 ≤ i ≤ Np. This can now be used to calculate the area of the unachievable region, which is implicitly included in AUC PR (thus overestimating performance).

Theorem 9 The area of the unachievable region in P-R space, and analogously the minimum achievable AUC PR, for limits π 1 and π 2 is given such that

AUC PR MIN (π 1 , π 2 ) = γ -2π 1 + 2π 2 + (-1 + π 1 ) ln(1 -π 1 ) -(-1 + π 2 ) ln(1 -π 2 ) + Li 2 (π 1 ) -Li 2 (π 2 ) . ( 20
)
where Li 2 (π ) = ∞ Proof. Equation ( 19) gives the lower bound on precision at a particular recall, therefore the unachievable area is the area under the curve PMIN (R) = γ

π 2 π 1 π R π R+1-π dπ , such that AUC PR MIN (π 1 , π 2 ) = 1 0 γ π 2 π 1 π R π R + 1 -π dπ dR =γ -2π 1 + 2π 2 + (-1 + π 1 ) ln(1 -π 1 ) -(-1 + π 2 ) ln(1 -π 2 ) + Li 2 (π 1 ) -Li 2 (π 2 )
and Li 2 (π ) can be efficiently computed to an accuracy of nineteen decimal places [START_REF] Osácar | Numerical evaluation of the dilogarithm of complex argument[END_REF].

Therefore an algorithm's reported AUC should be AUC PR -AUC PR MIN (π 1 , π 2 ). In fact, a normalised AUC measure has been proposed [START_REF] Boyd | Unachievable region in precision-recall space and its effect on empirical evaluation[END_REF] that can be modified to suite this case AUC PR(π 1 , π 2 ) = AUC PR -AUC PR MIN (π 1 , π 2 )

1 -AUC PR MIN (π 1 , π 2 ) . (21)
Nevertheless, as AUC PR MIN is only dependent upon π 1 and π 2 , and assuming that they remain constant, the relative ranking of algorithms' performances also remain equal. Figure 9 illustrates AUC PR MIN (π 1 , π 2 ) for a number of different skew limits (solid lines).

C.4 Performance of a Random Classifier

It is often desirable to represent the performance of a random classifier in ROC space. In P-R space a random classifier's curve is dependent upon the integration limits, π 1 and π 2 .

Theorem 10 A random classifier produces a constant P-R curve equal to

PRC (θ) = π 2 + π 1 2 , 0 ≤ R(θ) ≤ 1. ( 22
)
Proof. Since a random classifier gives TPR(θ) = FPR(θ) [START_REF] Fawcett | An introduction to ROC analysis[END_REF], substituting into Eq. ( 16) gives

P (θ) = γ π 2 π 1 π TPR(θ) π TPR(θ) + (1 -π )TPR(θ) dπ = γ π 2 π 1 π π + (1 -π ) dπ = π 2 + π 1 2 .
Corollary 3 It directly follows that the AUC PR of a random classifier, within the bounds of 0 ≤ R ≤ 1, is defined such that AUC PR RC (θ) = (π 2 + π 1 )/2.

  Fig. 1: Annotator agreement of fissure locations and detector outputs. The level of colour in (b) represents the amount of agreement in that location. The level of colour in (c) represents the strength of the detector's response to the AOI presented in (a). The result of thresholding this response at threshold θ = 50 is presented in (d).

Fig. 2 :

 2 Fig.2: Receiver Operating Curves describing the detectors' performance using different levels of agreement as the GTs.

Fig. 3 :

 3 Fig.3: Precision-Recall curves describing the detectors' performance using different levels of agreement as the GTs.

Fig. 4 :

 4 Fig. 4: Skew transform function defined by Equation (3), applied to precision calculated upon a dataset with a skew of π = 0.5, 4b, and a skew of π = 0.01, 4b.

Fig. 5 :

 5 Fig. 5: Precision-Recall curves describing two algorithms' performances at different skews (and cases from the proof of Thm. 2 in App. A). The AUCs (to four decimal places) are: (a) C-S = 0.3384 and Gauss = 0.3607; (b) C-S = 0.1307 and Gauss = 0.1088; and (c) C-S = 0.7772 and Gauss = 0.8260.

Fig. 6 :

 6 Fig.6: Precision-Recall, ROC, and Precision-Recall curves describing the detectors' performance using GTs of different agreement levels, π 1 = 0.0 and π 2 = 0.5, and λ 1 = 1 and λ 2 = 100 (which are equivalent to π 1 = 0.0099, to four decimal places, and π 2 = 0.5).

Fig. 7 :

 7 Fig.7: P-R and P-R curves calculated upon the fissure image (using the Gaussian detector) and the STARE dataset (using the MLVessel v1.4 package) with original and an altered skews.

Fig. 8 :

 8 Fig.8: P-R and P-R curves calculated upon the fissure image (using the Gaussian detector) taking the 8th (red) and 12th (blue) annotations as GTs and the STARE dataset (using the MLVessel v1.4 package) taking the VK (red) and AH (blue) GTs.

  θa and θ b are thresholds such that 1 ≤ i < TP(θ b ) -TP(θa), B = FP(θ b )-FP(θa) TP(θ b )-TP(θa) and i ∈ N.

Fig. 9 :

 9 Fig.9: Minimum achievable P-R curves (solid lines) and those of random classification (dashed lines) as π 1 and π 2 vary. The minimum AUCs (to four decimal places) are: AUC PR MIN (0.0, 0.5) = 0.1424; AUC PR MIN (0.3, 0.5) = 0.2349; and AUC PR MIN (0.6, 0.9) = 0.5471.

Table 1 :

 1

	Detector	CCO	CCI
	Gaussian 0.5293 0.4711 C-S 0.6387 0.5259

Pearson's r Correlation coefficients between detector outputs and the annotator agreement; CCO (correlation coefficient -object) is calculated between annotator agreement and detector output within the pixels marked as a fissure by the annotators, and CCI (correlation coefficient -image) the whole image. The significance of each correlation was tested to 99% confidence and each p-value was found to be 0.0000 (to four decimal places).

Table 2 :

 2 Hypothetical confusion matrices simulating an algorithm's application to skewed datasets, in which the TPR = 0.8, FPR = 0.3 and π = N p /N .

		a) π = 0.09 b) π = 0.25 c) π = 0.51 d) π = 0.91
		+	-	+	-	+	-	+	-
	+ -	73 18	276 643	200 50	228 532	408 102	150 350	735 184	27 64
	P (θ)		0.21		0.47		0.73	0.97

  {change in lower half} 10: end if 11: return π * ← π m {point of change found} grate over the function π (t), which collapses this added dimension into a single value, such that

Table 3 :

 3 The area under curve measurements of the P-R and P-R curves presented in Figures3a and 6arespectively, in addition to their values normalised with respect to the unachievable AUC[START_REF] Boyd | Unachievable region in precision-recall space and its effect on empirical evaluation[END_REF].

	Level of Agreement ≥ 1 ≥ 2 ≥ 3 ≥ 4 ≥ 5 ≥ 6 ≥ 7 ≥ 8 ≥ 9 ≥ 10 0.3024 0.3001 0.3008 0.8675 0.7251 0.8455 AUCPR AUCPR -AUCPR MIN AUCPR-AUCPR MIN 1-AUCPR MIN AUC PR -AUC PR MIN MIN MIN AUC PR-AUC PR 1-AUC PR AUC PR 0.3615 0.3274 0.3390 0.5996 0.4573 0.5332 0.3633 0.3434 0.3504 0.6760 0.5337 0.6223 0.3635 0.3490 0.3541 0.7159 0.5736 0.6688 0.3561 0.3450 0.3489 0.7422 0.5998 0.6994 0.3476 0.3388 0.3418 0.7639 0.6215 0.7247 0.3381 0.3311 0.3334 0.7834 0.6410 0.7474 0.3329 0.3274 0.3292 0.8043 0.6620 0.7719 0.3253 0.3210 0.3224 0.8239 0.6816 0.7947 0.3140 0.3108 0.3118 0.8437 0.7014 0.8178 ≥ 11 0.2895 0.2881 0.2885 0.8920 0.7496 0.8741 ≥ 12 0.2576 0.2569 0.2571 0.9187 0.7763 0.9052 ≥ 13 0.1879 0.1877 0.1877 0.9454 0.8031 0.9364

Table 4 :

 4 Skew, π = N p /N , found in machine learning datasets (to three decimal places). The top seven datasets relate the the domain of eye fundus images (see text). Estimating π 1 and π 2 from Data

	Database	Skew
	STARE (GT: VK) (Hoover et al, 2000) STARE (GT: AH) ARIA (GT: BDP) ARIA (GT: BSS) HEI-MED DRIVE (GT: 1) DRIVE (GT: 2)	0.109 0.076 0.184 0.168 0.090 0.126 0.123
	Spectrogram (Lampert and O'Keefe, 2011) 0.006 UCI ML Adult 0.242 UCI ML Breast Cancer Wisconsin 0.373 UCI ML Poker Hand (Flush Class) 0.002 UCI ML Poker Hand (Nothing Class) 0.501 UCI ML Internet Advertisements 0.134 NASA Software Defect Dataset CM1 0.098 NASA Software Defect Dataset PC1 0.931
	5.1

Acknowledgements This work is part of the FOSTER project, which is funded by the French Research Agency (Contract ANR Cosinus, ANR-10-COSI-012-03-FOSTER, 2011-2014). The participating annotators from LIVE, IPGS, and ICube (University of Strasbourg) are gratefully acknowledged.


Tania Petersen

Thibault Mahévas

Pauline Cannet

Jean-David Bouaziz

Camille Hua

Charles Zarnitsky

Philippe Modiano

François Lifermann

Bruno Sassolas

Olivier Carpentier

Arsène Mékinian

Laurent Garderet

Pierre Aucouturier
email: pierre.aucouturier@inserm.fr

Yannick Chantran

IgG1 Subclass Restriction and Biochemical Peculiarities of Monoclonal Immunoglobulins in Scleromyxedema

Keywords: scleromyxedema, monoclonal gammopathy, immunoglobulin, IgG subclass MGUS: monoclonal gammopathy of undetermined significance, SDS-PAGE: sodium dodecyl sulfate-polyacrylamide gel electrophoresis, SME: scleromyxedema

à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

INTRODUCTION

Scleromyxedema (SME) is a rare systemic mucinosis associated with a monoclonal gammopathy (MG) that belongs to the recently described group of Monoclonal Gammopathy of Cutaneous Significance (MGCS) [START_REF] Lipsker | Monoclonal gammopathy of cutaneous significance: review of a relevant concept[END_REF]. It is characterized by a histological triad including intense dermal mucin deposition, increased fibroblast proliferation, and fibrosis, and is responsible for a scleroderma-like syndrome associated with generalized papular eruption [START_REF] Rongioletti | Scleromyxedema: A Multicenter Study of Characteristics, Comorbidities, Course, and Therapy in 30 Patients[END_REF].

Various systemic manifestations are common in SME, including cutaneous, neurological, and articular involvements. Dermato-neuro-syndrome (DNS), an acute encephalopathy leading to seizure, coma, and death, is the most severe complication of the disease. Despite the rare association of SME with hematologic malignancy (10 -15%) [START_REF] Mahévas | Plasma Cell-Directed Therapies in Monoclonal Gammopathy-Associated Scleromyxedema[END_REF], mortality remains high among reported SME cases [START_REF] Rongioletti | Scleromyxedema: A Multicenter Study of Characteristics, Comorbidities, Course, and Therapy in 30 Patients[END_REF][START_REF] Mahévas | Plasma Cell-Directed Therapies in Monoclonal Gammopathy-Associated Scleromyxedema[END_REF].

The almost constant occurrence of MG in SME has made it a diagnostic criterion along with cutaneous eruption, the histological triad on skin biopsy, and the absence of a thyroid dysfunction [START_REF] Rongioletti | Scleromyxedema: A Multicenter Study of Characteristics, Comorbidities, Course, and Therapy in 30 Patients[END_REF][START_REF] Rongioletti | Updated classification of papular mucinosis, lichen myxedematosus, and scleromyxedema[END_REF]. However, a direct or indirect role of the monoclonal immunoglobulin (Ig) in SME pathogenesis remains uncertain. Treatments aimed at eradicating a monoclonal plasma cell proliferation proved efficient in certain cases [START_REF] Mahévas | Plasma Cell-Directed Therapies in Monoclonal Gammopathy-Associated Scleromyxedema[END_REF][START_REF] Donato | Scleromyxedema: role of high-dose melphalan with autologous stem cell transplantation[END_REF][START_REF] Cañueto | The combination of bortezomib and dexamethasone is an efficient therapy for relapsed/refractory scleromyxedema: a rare disease with new clinical insights[END_REF].

Intrinsic biochemical peculiarities of monoclonal Ig in SME patients were reported. A cationic migration and IgGλ restriction was found in 5 cases [START_REF] James | Studies on a unique diagnostic serum globulin in papular mucinosis (lichen myxedematosus)[END_REF]. In 1978, Kitamura et al. [START_REF] Kitamura | Immunochemical analysis of the monoclonal paraprotein in scleromyxedema[END_REF] found a low molecular weight (110 kDa) monoclonal IgG, suggesting deletion of a part of the Fd fragment, in a single case of SME. Subsequent studies showed that the monoclonal component is always of the IgG class with an over-representation of the lambda-type light chain [START_REF] Rongioletti | Scleromyxedema: A Multicenter Study of Characteristics, Comorbidities, Course, and Therapy in 30 Patients[END_REF][START_REF] Mahévas | Plasma Cell-Directed Therapies in Monoclonal Gammopathy-Associated Scleromyxedema[END_REF].

Here, we identified the IgG subclass and light chain type of the monoclonal component in 12 SME cases, and assessed their biochemical peculiarities, including mobility on zone electrophoresis and presence of constant domains of heavy-chains.

MATERIALS AND METHODS

Patients

Serum samples were collected from 12 consecutive patients diagnosed with SME, recruited from 9 French centers between June 2017 and November 2018 by MINHEMON French network (dysimmune disorders associated to hemopathies and cancers). Samples were kept at consecutive patients with MGUS were collected at the Saint Antoine hospital (Paris). All analyses, including electrophoresis and immunochemical studies, were part of the routine biological follow-up of patients.

Characterization of isotypes and electrophoretic mobility of serum monoclonal Ig

Serum monoclonal Ig were identified by standard immunofixation (Sebia, France).

Electrophoretic mobilities of monoclonal IgG were determined by measuring their retention factor (Rf) relative to albumin. Then, relative mobilities of monoclonal IgG were normalized against the Rf of a single serum IgG (SME patient #7) serving as an internal standard.

Characterization of IgG subclasses and presence of heavy chain constant domains

Monoclonal IgG subclass typing and reactivities for heavy chain constant domains CH1, CH2, and CH3 were performed using a home-made immunoblotting method [START_REF] Briault | Isotypy of serum monoclonal immunoglobulins in human immunodeficiency virus-infected adults[END_REF]: appropriate dilutions of serum samples were submitted to thin-layer agarose gel zone electrophoresis and transferred by single pressure (15 g/cm 2 for 10 minutes) onto a 0.2 µm nitrocellulose membrane. The membrane was then blocked for 30 minutes in 5% skimmed milk in phosphate buffered saline pH 7.4 (PBS) and cut into strips that were incubated for 60 minutes at room temperature with appropriate primary mouse monoclonal antibodies diluted in 0.05% Tween-20, 0.02% bovine serum albumin PBS. The monoclonal antibodies were specific for kappa and lambda light chains, and IgG1, IgG2, IgG3, and IgG4 heavy chain isotypes (clones HP6053, HP6054, NL16, GOM2, ZG4, and RJ4, respectively) and for IgG constant domains (anti-CH1: TM15, ZB8, HP6044; anti-CH2: G7C, HP6018; anti-CH3: X3A8, HP6017). After washing with 0.05% tween-PBS at 4°C, strips of nitrocellulose membrane were incubated with alkaline phosphatase-conjugated rabbit polyclonal anti-mouse IgG diluted in 0.05% tween-20, 0.02% BSA/PBS. Strips were washed as before, and revealed by chromogenic reaction with tetrazolium nitroblue / bromo-chloro-indolyl phosphate in 0.1 M NaCl 0.1 M MgCl2 0.1 M TRIS/HCl buffer pH 9.5.

Purification of monoclonal Ig and heavy and light chain molecular weight determination

The monoclonal IgG could be purified from the serum of one SME case and from a control IgG1 myeloma patient, using ion exchange chromatography. Serum samples were fractionated on a diethylaminoethyl (DEAE)-Trisacryl column in 10 mM TRIS/HCl buffer pH 7.8 with a 0 to 0.1 M NaCl linear gradient. Protein fractions were concentrated by ultrafiltration and purity was assessed by standard thin layer agarose zone electrophoresis at pH 8.6. Purified (> 95%) fractions were reduced with 10 mM dithiothreitol in 1% sodium dodecyl sulfate (SDS) loading buffer at 95°C for 3 minutes, and 1 µg protein of each was fractionated by 12% SDS polyacrylamide gel electrophoresis (SDS-PAGE).

Statistical methods

Frequencies of monoclonal IgG1 and λ-type IgG were compared to those in a published series of MGUS using Fisher's exact test. Electrophoretic mobilities were compared using Mann-Whitney-Wilcoxon's test. Differences with a p value < 0.05 were considered significant.

Computations were performed using R software.

RESULTS

Patients were aged 29 -75 years at the time of the study (mean: 61.2 ± 16.8 years) with a sex ratio of 1 (Table 1). Three patients presented with DNS and the remaining 9 had non-severe forms of SME.

All 12 monoclonal Ig from the SME patients were of IgG1 subclass (Table 1 and Figure 1B), including 7 with a lambda and 5 with a kappa light chain. The IgG1 restriction was significant as compared with the occurrence of IgG1 in the largest published series of MGUS (155/241 (63%); p = 0.009) [START_REF] Kyle | Longterm follow-up of 241 patients with monoclonal gammopathy of undetermined significance: the original Mayo Clinic series 25 years later[END_REF], while the predominance of lambda light chain isotype in SME (7/12, 58%) was not statistically significant as compared with the same series of MGUS (89/241, 37%) (p = 0.22).

In SME patients, the electrophoretic migration of the monoclonal component was significantly more cationic (median [IQR]: 1.00 [0.96 -1.03]) than the 21 MGUS that displayed a monoclonal IgG1 (median [IQR]: 0.95 [0.85 -0.98]; p = 0.03; Figure 1A).

Analyses of constant domain reactivity with a panel of CH1-, CH2-and CH3-specific monoclonal antibodies by immunoblotting led to conclude on the absence of any deletion of the heavy chain constant region in all 12 studied SME-associated monoclonal IgG1 (Figure 1B). This was confirmed by the SDS-PAGE analysis of one SME-associated monoclonal IgG1 kappa that we could purify, as it revealed normal sizes of both heavy and light chains (Figure 2).

DISCUSSION

This study demonstrates a strict IgG1 restriction of SME-associated monoclonal immunoglobulins, and it confirms that most of them have a high isoelectric point as suggested by their cationic migration on agarose zone electrophoresis.

The bias toward IgG class and lambda type light-chain in SME MG was already noted by James et al. [START_REF] James | Studies on a unique diagnostic serum globulin in papular mucinosis (lichen myxedematosus)[END_REF] but subclasses of IgG have not been analyzed, as far as we know. The actual importance of the IgG1 restriction in SME should be confirmed in a larger series, but it already appears quite significant in the present study.

The cationic electrophoretic migration of SME-associated monoclonal immunoglobulins has been evoked in small series or in a single case [START_REF] James | Studies on a unique diagnostic serum globulin in papular mucinosis (lichen myxedematosus)[END_REF][START_REF] Kitamura | Immunochemical analysis of the monoclonal paraprotein in scleromyxedema[END_REF]. To our knowledge, this is the first statistical demonstration of cationic electrophoretic migration of SME monoclonal Ig as compared with classical MGUS. We found that this electrophoretic property is independent of the IgG1 restriction of SME-associated monoclonal component.

On the other hand, we did not confirm the results from Kitamura et al. [START_REF] Kitamura | Immunochemical analysis of the monoclonal paraprotein in scleromyxedema[END_REF] suggesting a deletion of the Fd portion (most likely the CH1 domain) of a monoclonal IgG from one SME case. Indeed, all constant domains of the heavy chains were immunoreactive in our immunoblotting studies, while 1 to 3 distinct antibodies were tested for each domain. This was confirmed in one case whose monoclonal component could be purified and analyzed by SDS-PAGE, revealing normal sized heavy and light chains.

The almost constant presence of a MG in SME, isotype restriction and basic nature of the monoclonal component pinpoint the potential role of a monoclonal plasma-cell proliferation in the pathophysiology of SME. However, a direct pathogenic role of such hematologic event and the bone marrow microenvironment remains unknown. The efficacy in severe cases of new treatments targeting putative plasma cell clones with concomitant hematological and clinical response supports the hypothesis of a direct or indirect role of these cells in the SME pathogenesis [START_REF] Mahévas | Plasma Cell-Directed Therapies in Monoclonal Gammopathy-Associated Scleromyxedema[END_REF][START_REF] Donato | Scleromyxedema: role of high-dose melphalan with autologous stem cell transplantation[END_REF][START_REF] Cañueto | The combination of bortezomib and dexamethasone is an efficient therapy for relapsed/refractory scleromyxedema: a rare disease with new clinical insights[END_REF]]. An old study has shown that serum factor(s), but not the monoclonal immunoglobulin alone, could stimulate in vitro fibroblast proliferation [START_REF] Harper | Lichen myxedematosus serum stimulates human skin fibroblast proliferation[END_REF]12]. A hypothesis that has been suggested to explain this phenomenon could be the presence of an unknown circulating cofactor that could stimulate proliferation in association with the monoclonal immunoglobulin. Biochemical changes such as glycosylation and/or modifications of the amino acid sequence of the immunoglobulin could affect its electric charge and explain the strong association between cationic migration and SME. The great efficacy of high dose intravenous immunoglobulins in SME [START_REF] Rongioletti | Scleromyxedema: A Multicenter Study of Characteristics, Comorbidities, Course, and Therapy in 30 Patients[END_REF][START_REF] Mahévas | Plasma Cell-Directed Therapies in Monoclonal Gammopathy-Associated Scleromyxedema[END_REF]13] could be explained by their immunomodulatory functions, or by increased clearance of the monoclonal Ig or its complex with a co-factor. Further studies, based for example on mass spectrometry analyses of the serum secretome, as well as functional experiments using purified monoclonal Ig with and without other serum fractions on human in vitro fibroblast cultures, are required for better understanding the implication of these peculiar monoclonal Ig in the pathophysiology of SME.

  12. Ferrarini M, Helfrich DJ, Walker ER, Medsger TA Jr, Whiteside TL. Scleromyxedema serum increases proliferation but not the glycosaminoglycan synthesis of dermal fibroblasts. J Rheumatol 1989;16(6):837-41. (PMID: 2778769) 13. Guarneri A, Cioni M, Rongioletti F. High-dose intravenous immunoglobulin therapy for scleromyxoedema: a prospective open-label clinical trial using an objective score of clinical evaluation system. J Eur Acad Dermatol Venereol 2017 31(7):1157-60. (PMID: 28370513)

Figure 1 .

 1 Figure 1. (A) Comparison of electrophoretic mobilities of serum monoclonal IgG from 12 patients with scleromyxedema (SME) and 21 subjects with IgG1 MGUS (*: p < 0.05). (B) Immunoblotting analysis of IgG subclasses, light chain type and antigenic reactivity of CH1, CH2 and CH3 domains of serum samples from 2 patients with SME (used antibody clones are mentioned below corresponding strips; SPE: serum protein electrophoresis).

Figure 2 .

 2 Figure 2. Apparent molecular weight of purified monoclonal IgG1 kappa from patient #6, as compared with a purified myeloma IgG1 kappa. (A) Agarose zone electrophoresis showing serum and DEAE-purified fraction from patient #6. (B) 12% SDS-PAGE under reducing conditions of purified monoclonal IgG1 kappa from a scleromyxedema patient (SME #6) and a multiple myeloma IgG1 kappa (MM control), showing normal sized heavy chain (black arrow) and light chain (white arrow).

Table 1 .

 1 Demographic, clinical, and biological characteristics of 12 patients with

	scleromyxedema
	Patient Gender/Age Severity	Monoclonal Ig isotype	Electrophoretic mobility (normalized value)
	1	H/60	Non severe	IgG1 λ	Medium/cathodic (0.98)
	2	F/29	Severe, DNS, cardiomyopathy	IgG1 λ	Cathodic (1.00)
	3	H/75	Non severe, cutis laxa-like eruption, MBL	IgG1 κ	Cathodic (1.02)
	4	H/73	Non severe	IgG1 λ	Cathodic (1.01)
	5	F/71	Non severe	IgG1 λ	Cathodic (1.02)
	6	F/29	Severe, DNS	IgG1 κ	Medium (0.92)
	7	F/75	Non severe	IgG1 λ	Cathodic (1.00)
	8	F/69	Non severe	IgG1 κ	Cathodic (1.10)
	9	M/51	Non severe	IgG1 λ	Medium (0.89)
	10	M/53	Non severe	IgG1 κ	Anodic (0.86)
	11	M/55	Severe, recurrent DNS	IgG1 κ	Cathodic (1.11)
	12	F/75	Non severe	IgG1 λ	Cathodic (1.05)
	DNS: dermato-neuro syndrome; Non severe disease: absence of DNS or mucinous
	cardiomyopathy; MBL: monoclonal B lymphocytosis; Severe disease: presence of DNS or
	mucinous cardiomyopathy

This work was supported by Assistance Publique -Hôpitaux de Paris (AP-HP) and Sorbonne Université (Paris,