Imran Sk

Jocelyn Hossain

Md Shahriar De Goër De Herve

Delphine Hassan

Evelina Martineau

Violaine Petrosyan

Jean Corbain

Isabelle Beytout

Jonas Lebert

Irene Durand

Carravieri

Sk Imran Hossain

Jocelyn De Goër De Herve

Shahriar Hassan

Delphine Martineau

Evelina Petrosyan

Violaine Corbin

Jean Beytout

Isabelle Lebert

Jonas Durand

Irene Carravieri

Annick Brun-Jacob

Pascale Frey-Klett

Elisabeth Baux

Céline Cazorla

Carole Eldin

Yves Hansmann

Solene Patrat-Delon

Thierry Prazuck

Alice Raffetin

Pierre Tattevin

Gwenaël Vourc'h

Olivier Lesens

Engelbert Mephu Nguifo
email: engelbert.mephu_nguifo@uca.fr

Exploring convolutional neural networks with transfer learning for diagnosing Lyme disease from skin lesion images

Keywords: Lyme disease, Erythema Migrans, Transfer Learning, CNN, Explainability

ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Introduction

Lyme disease is an infectious disease transmitted by ticks and caused by pathogenic bacteria of the Borrelia burgdorferi sensu lato group [START_REF] Shapiro | Clinical practice. Lyme disease[END_REF]. It is estimated that around 476,000 people in the United States and more than 200,000 people in western Europe are affected by Lyme disease each year [START_REF] Marques | Comparison of lyme disease in the United States and Europe[END_REF]. Most of the time an expanding round or oval red skin lesion known as erythema migrans (EM) becomes visible in the victim's body which is the most common early symptom of Lyme disease [START_REF] Shapiro | Clinical practice. Lyme disease[END_REF][START_REF] Burlina | Skin Image Analysis for Erythema Migrans Detection and Automated Lyme Disease Referral[END_REF]. EM usually appears at the site of a tick bite after one to two weeks (range, 3 to 30 days) as a small redness and expands almost a centimeter per day, creating the characteristic bull's-eye pattern as shown in Figure 1 (a) [START_REF] Shapiro | Clinical practice. Lyme disease[END_REF][START_REF] Burlina | Skin Image Analysis for Erythema Migrans Detection and Automated Lyme Disease Referral[END_REF][START_REF] Strle | Clinical manifestations and diagnosis of lyme borreliosis[END_REF][START_REF] Berglund | An Epidemiologic Study of Lyme Disease in Southern Sweden[END_REF]. EM generally vanishes within a few weeks or months but the Lyme disease infection advances to affect the nervous system, skin, joints, eyes, and heart [START_REF] Shapiro | Clinical practice. Lyme disease[END_REF][START_REF] Strle | Clinical manifestations and diagnosis of lyme borreliosis[END_REF]. Antibiotics can be used as a medium of effective treatment in the early stage of Lyme disease. So, early recognition of EM is extremely important to avoid long-term complications of Lyme disease. Most European and North American guidelines recommend a two-tier serology test to detect antibodies against Borrelia burgdorferi sensu lato for diagnosing Lyme disease [START_REF] Eldin | Review of European and American guidelines for the diagnosis of Lyme borreliosis[END_REF][START_REF] Trevisan | A Practical Approach to the Diagnosis of Lyme Borreliosis: From Clinical Heterogeneity to Laboratory Methods[END_REF]. However, a serology test is only recommended in the absence of EM because early serology has low sensitivity (40% to 60%) and may result in false negatives [START_REF] Eldin | Review of European and American guidelines for the diagnosis of Lyme borreliosis[END_REF]. Alternatively, direct detection of Borrelia burgdorferi sensu lato can be done using culture, microscopy, or PCR [START_REF] Trevisan | A Practical Approach to the Diagnosis of Lyme Borreliosis: From Clinical Heterogeneity to Laboratory Methods[END_REF]. The gold standard of microbiological diagnosis -the culture of bacteria requires laboratory expertise and special media for Borrelia burgdorferi sensu lato [START_REF] Eldin | Review of European and American guidelines for the diagnosis of Lyme borreliosis[END_REF]. Light microscopy-based detection is not feasible in clinical practice [START_REF] Trevisan | A Practical Approach to the Diagnosis of Lyme Borreliosis: From Clinical Heterogeneity to Laboratory Methods[END_REF]. PCR based diagnosis is also very difficult and shows highly variable sensitivity [START_REF] Trevisan | A Practical Approach to the Diagnosis of Lyme Borreliosis: From Clinical Heterogeneity to Laboratory Methods[END_REF]. Direct detection methods are not always feasible for clinicians because of extended processing time and required expertise [START_REF] Burlina | AI-based detection of erythema migrans and disambiguation against other skin lesions[END_REF]. The diagnosis of EM is a challenging task because EM can create different patterns instead of the trademark bull's-eye pattern as shown in Figure 1 (b).

Diagnosing skin disorders requires a careful inspection from dermatologists or infectiologists but their availability, especially in rural areas is scarce [START_REF] Feng | Comparison of dermatologist density between urban and rural counties in the United States[END_REF]. As a result, the diagnosis is generally carried out by non-specialists, and their diagnostic accuracy is in the range of twentyfour to seventy percent [START_REF] Seth | Global Burden of Skin Disease: Inequities and Innovations[END_REF][START_REF] Tran | Assessing diagnostic skill in dermatology: A comparison between general practitioners and dermatologists[END_REF]. The wrong diagnosis can result in improper or delayed treatment which can be harmful to the patient.

Artificial intelligence (AI) powered diagnostic tools can help with the scarcity of expert dermatologists. Recent advancement in deep learning techniques has eased the creation of AI solutions to aid in skin disorder diagnosis. Many works have been done utilizing deep learning techniques specifically convolutional neural networks (CNNs) for diagnosing cancerous and other common skin lesions from dermoscopic images [START_REF] Haenssle | Man against Machine: Diagnostic performance of a deep learning convolutional neural network for dermoscopic melanoma recognition in comparison to 58 dermatologists[END_REF][13][START_REF] Maron | Systematic outperformance of 112 dermatologists in multiclass skin cancer image classification by convolutional neural networks[END_REF][START_REF] Perez | Data augmentation for skin lesion analysis[END_REF]. As dermoscopic images require dermatoscopes from dermatology clinics other works have focused on diagnosing skin diseases using deep learning from clinical images [START_REF] Liu | A deep learning system for differential diagnosis of skin diseases[END_REF][START_REF] Esteva | Dermatologist-level classification of skin cancer with deep neural networks[END_REF][START_REF] Han | Deep neural networks show an equivalent and often superior performance to dermatologists in onychomycosis diagnosis: Automatic construction of onychomycosis datasets by region-based convolutional deep neural network[END_REF]. According to some of these studies, deep learning-based systems compete on par with expert dermatologists for diagnosing diseases from dermoscopic and clinical images [START_REF] Haenssle | Man against Machine: Diagnostic performance of a deep learning convolutional neural network for dermoscopic melanoma recognition in comparison to 58 dermatologists[END_REF][13][START_REF] Maron | Systematic outperformance of 112 dermatologists in multiclass skin cancer image classification by convolutional neural networks[END_REF][START_REF] Esteva | Dermatologist-level classification of skin cancer with deep neural networks[END_REF][START_REF] Han | Deep neural networks show an equivalent and often superior performance to dermatologists in onychomycosis diagnosis: Automatic construction of onychomycosis datasets by region-based convolutional deep neural network[END_REF].

Despite the vast application of AI in the field of skin lesion diagnosis, there are only a few works related to Lyme disease detection from EM skin lesion images. The unavailability of public EM datasets as a result of privacy concerns of medical data may be the reason for the lack of extensive studies in this field. Čuk et al. [START_REF] Čuk | Supervised visual system for recognition of erythema migrans, an early skin manifestation of lyme borreliosis, Stroj[END_REF] proposed a visual system for EM recognition on a private EM dataset using classical machine learning techniques including naïve Bayes, SVM, boosting, and neural nets (not deep learning). They considered ellipse, the common shape of EM and used eccentricity, small and large axis ratio, ellipse angular, and ellipse focus attributes for classification. Deep learning techniques learn image features from training images via an optimization process and recent studies show that image features extracted by deep learning techniques outperform human-engineered image features for medical image classification tasks [START_REF] Burlina | AI-based detection of erythema migrans and disambiguation against other skin lesions[END_REF]. Burlina et al. [START_REF] Burlina | Skin Image Analysis for Erythema Migrans Detection and Automated Lyme Disease Referral[END_REF] created a dataset of EM by collecting images from the internet and trained a CNN architecture ResNet50 as a binary classifier to distinguish between EM and other skin conditions. Although their dataset is not public, the trained model is publicly available. Burlina et al. [START_REF] Burlina | AI-based detection of erythema migrans and disambiguation against other skin lesions[END_REF] further enriched the dataset with more images from the East Coast and Upper Midwest of the United States and trained six CNNs namely ResNet50, InceptionV3, MobileNetV2, DenseNet121, InceptionResNetV2, and ResNet152 for EM classification. Burlina et al. [START_REF] Burlina | AI-based detection of erythema migrans and disambiguation against other skin lesions[END_REF] did not make the dataset or the trained models public. Burlina et al. [START_REF] Burlina | Skin Image Analysis for Erythema Migrans Detection and Automated Lyme Disease Referral[END_REF] and Burlina et al. [START_REF] Burlina | AI-based detection of erythema migrans and disambiguation against other skin lesions[END_REF] used transfer learning from ImageNet [START_REF] Russakovsky | ImageNet Large Scale Visual Recognition Challenge[END_REF] pre-trained models and studied the CNNs in terms of predictive performance. With the advancement of CNNs, it is a timely need to extensively study their effectiveness for Lyme disease prediction from EM images.

Lightweight CNN based mobile applications can help people with an initial self-assessment of EM and referring them to expert dermatologist for further diagnosis. Also, resource-intensive CNN based computer applications can assist non-expert practitioners for identifying EM. In this article, our main objective was to study the performance of state-of-the-art CNNs for diagnosing Lyme disease from EM images and to find out the best architecture based on different criteria. As there is no publicly available Lyme dataset of EM images, first, we created a dataset consisting of 866 images of confirmed EM lesions. Images collected from the internet and Clermont-Ferrand University Hospital Center (CF-CHU) of France were carefully labeled into two classes: EM and Confuser, by expert dermatologists and infectiologists from CF-CHU. CF-CHU collected the images from several hospitals in France. Second, we benchmarked twenty-three well-known DCCNs on this dataset in terms of several predictive performance metrics, computational complexity metrics, and statistical significance tests. Best practices for training CNNs on limited data like transfer learning and data augmentation were used. Third, instead of only using transfer learning from models pre-trained on ImageNet dataset we also utilized a dataset of common skin lesions "Human Against Machine with 10000 training images (HAM10000)" [START_REF] Tschandl | The HAM10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions[END_REF] for pretraining the CNNs. The use of HAM10000 proved fruitful according to the experimental results. We experimentally searched for the best performing number of layers to unfreeze during transfer learning fine-tuning for each of the studied CNNs. Fourth, for visualizing the regions of the input image that are significant for predictions from the CNN models we used Gradient-weighted Class Activation Mapping (Grad-CAM) [START_REF] Selvaraju | Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization[END_REF]. Fifth, we provided guidelines for model selection based on predictive performance and computational complexity. Moreover, we made all the trained models publicly available which can be used for transfer learning and building pre-scanners for Lyme disease. Figure 2 presents the graphical overview of this study.

The rest of the paper is structured as follows: Section 2 contains dataset description, a brief overview of CNN architectures, performance measures, and transfer learning approach used in this study; Section 3 presents experimental studies; Section 4 contains recommendations for model selection, discussion on limitations and scopes; finally, Section 5 presents concluding remarks.

Methods

The following subsections describe the data organization including data augmentation and cross-validation, a short overview of the considered CNN architectures, performance measures, explainability method, and the transfer learning approach used in this study.

Dataset Preparation

As a labeled public dataset is not available for Lyme disease prediction from EM images, we created a dataset by collecting images from the internet and CF-CHU. CF-CHU collected EM images from several hospitals located in France. The use of images from the internet was inspired by related skin lesion analysis studies [START_REF] Burlina | Skin Image Analysis for Erythema Migrans Detection and Automated Lyme Disease Referral[END_REF][START_REF] Burlina | AI-based detection of erythema migrans and disambiguation against other skin lesions[END_REF][START_REF] Esteva | Dermatologist-level classification of skin cancer with deep neural networks[END_REF]. Duplicate images were removed using an image hashing-based duplicate image detector followed by the removal of inappropriate images through human inspection. After the initial curation steps, we got a total of 1672 images. Expert dermatologists and infectiologists from CF-CHU classified the curated images into two categories: EM and Confuser, making it a two-class classification problem. Out of 1672 images, 866 images were assigned to EM class and 806 images were assigned to Confuser class. We further subdivided the dataset into five-folds using stratified five-fold cross-validation to make sure each of the folds maintains the original class ratio. One of the folds was used as a test set and the remaining four were used as the training set with a rotation of the folds for five runs. Each time, 10% of the training data was assigned to the validation set as shown in Figure 3.

DCCNs require a considerable amount of data for training and data augmentation can help with expanding the dataset. We applied data augmentation techniques only to the training set to expand it twenty times. We used data augmentation to create 20 images from a single image both for EM and Confuser categories because our dataset was balanced. We used flip (vertical or horizontal), rotation, brightness, contrast, and saturation augmentation by considering the best performing augmentations for skin lesions [START_REF] Perez | Data augmentation for skin lesion analysis[END_REF]. Besides, we also used perspective skew transformation to cover the case of looking at a picture from different angles. Augmentor [START_REF] Bloice | Biomedical image augmentation using Augmentor[END_REF] an image augmentation library specially built for biomedical image augmentation was used for applying the augmentations. We used 0.5 as the probability of applying each of the augmentation operations. Rotation operation was performed with a maximum rotation angle of 5 degrees. We also used random rotation by either 90, 180, or 270 degrees. Brightness, contrast, and saturation augmentations were performed with a minimum adjustment factor of 0.7 and a maximum adjustment factor of 1.3. For all the other parameters we used default values provided by Augmentor library. Figure 4 shows some example images resulting from augmentations applied on a sample image.

Brief Overview of CNN Architectures for Lyme Disease Diagnosis

CNNs are a kind of neural network that simulates some actions generated in the human visual cortex using convolution mathematical operation to extract features from input and passing these features through successive layers generates more abstract features to yield a final output [START_REF] Lecun | Gradient-based learning applied to document recognition[END_REF]. CNNs are modular in design, where convolution-based building blocks are repeatedly stacked for feature extraction with pooling layers placed in between for reducing feature space, learnable parameters, and controlling overfitting [START_REF] Goodfellow | Deep learning[END_REF]. Starting with LeNet [START_REF] Lecun | Gradient-based learning applied to document recognition[END_REF] in 1988 the popularity of CNNs increased with AlexNet [START_REF] Krizhevsky | ImageNet classification with deep convolutional neural networks[END_REF] winning the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) [START_REF] Russakovsky | ImageNet Large Scale Visual Recognition Challenge[END_REF] of 2012. As a result of the effectiveness of CNNs in solving complex problems, several CNN architectures have been introduced over the past few years. The following subsections provide a brief overview of the CNN architectures used in this study for diagnosing Lyme disease from EM images.

VGG Architecture

VGG architecture [START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF] is based on the idea of deeper networks with smaller filters (3 × 3). There are thirteen convolutional layers and three fully connected layers in VGG16 architecture as shown in Figure 5. Another variation of VGG architecture called VGG19 has sixteen convolutional layers and three fully connected layers. VGG architecture showed better effectiveness of deeper architectures in terms of predictive performance but requires training a huge number of parameters. To the best of our knowledge, VGG architectures have not been used previously for Lyme disease analysis.

Inception Architecture

Inception architecture [START_REF] Szegedy | Going deeper with convolutions[END_REF] uses inception module as shown in Figure 6, which is a combination of several convolution layers with small filters (1 × 1, 3 × 3, 5 × 5) applied simultaneously on the same input to facilitate the extraction of more information. The output filter banks from the convolution layers of inception module are concatenated into a single vector, which is served as the input for next stage. To reduce learnable parameters and computational complexity inception module uses 1 × 1 convolution at the beginning of convolution layers. InceptionV1

ResNet Architecture

ResNet architecture [START_REF] He | Deep residual learning for image recognition[END_REF] tried to solve the vanishing gradient and accuracy degradation problems of deep models by introducing residual block with identity shortcut connection that directly connects the input to the output of the block allowing the gradient to flow through the shortcut path as shown in

DenseNet Architecture

Dense Convolutional Network (DenseNet) [START_REF] Huang | Densely Connected Convolutional Networks[END_REF] extended ResNet by introducing dense blocks where each layer within a dense block receives inputs from all the previous layers as shown in Figure 8. DenseNet concatenates the incoming feature maps of a layer with output feature maps instead of summing them up as done in ResNet. Dense blocks within DenseNet are connected with transition layers consisting of convolution and pooling to perform the required downsampling operation. Depending on the number of weight layers there are several versions of DenseNet like DenseNet121, DenseNet169, DenseNet201, DenseNet264, etc. Besides solving the vanishing gradient problem DenseNet also eases feature propagation and reuse, and a reduction in the number of learnable parameters compared to ResNet. Burlina et al. [START_REF] Burlina | AI-based detection of erythema migrans and disambiguation against other skin lesions[END_REF] used ImageNet pre-trained DenseNet121 architecture for Lyme disease analysis.

MobileNet Architecture

MobileNetV1 [START_REF] Howard | MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications[END_REF] used depthwise separable convolution extensively to reduce the computational cost. Standard convolution performs spatial and channel-wise computations one step but depthwise separable convolution first applies separate convolutional filter for each input channel and then uses pointwise convolution on concatenated channels to produce required number of output channels as shown in Figure 9. MobileNetV1 was designed to run very efficiently on mobile and embedded devices. MobileNetV2 [START_REF] Sandler | MobileNetV2: Inverted Residuals and Linear Bottlenecks[END_REF] improved upon the concepts of MobileNetV1 by incorporating thin linear bottlenecks with shortcut connections between the bottlenecks as shown in Figure 10. This is called inverted residual block as it uses narrow → wide → narrow as opposed to the wide → narrow → wide architecture of traditional residual block. MobileNetV3 [START_REF] Howard | Searching for MobileNetV3[END_REF] incorporated squeeze-and-excitation layers [START_REF] Hu | Squeeze-and-Excitation Networks[END_REF] in the building block of MobileNetV2 which provides channel-wise attention and used MnasNet [START_REF] Tan | Mnasnet: Platform-aware neural architecture search for mobile[END_REF] to search for a coarse architecture that was further optimized with NetAdapt [START_REF] Yang | NetAdapt: Platform-aware neural network adaptation for mobile applications[END_REF] algorithm. Burlina et al. [START_REF] Burlina | AI-based detection of erythema migrans and disambiguation against other skin lesions[END_REF] used ImageNet pre-trained MobileNetV2 architecture for Lyme disease analysis.

Xception architecture

Extreme version of Inception the Xception architecture [START_REF] Chollet | Xception: Deep learning with depthwise separable convolutions[END_REF] replaced the Inception module with a modified version of depthwise separable convolution where the order of depthwise convolution and pointwise convolutions are reversed as shown in Figure 11. Xception also uses shortcut connections like ResNet architecture. On ImageNet dataset Xception performs slightly better than the InceptionV3 architecture. To the best of our knowledge, Xception architecture has not been used previously for Lyme disease analysis.

NASNet architecture

Neural Architecture Search Netowork [START_REF] Zoph | Learning Transferable Architectures for Scalable Image Recognition[END_REF] from Google Brain utilizes reinforcement learning with a Recurrent Neural Network based controller to search for efficient building block for a smaller dataset which is then transferred to a larger dataset by stacking multiple copies of the found building block. NASNet blocks are comprised of normal and reduction cells as shown in Figure 12. Normal cells produce feature map of same size as input whereas reduction cells reduce the size by a factor of two. NASNet optimized for mobile applications is called NASNetMobile whereas the larger version is called NASNetLarge. To the best of our knowledge, NASNet architectures have not been used previously for Lyme disease analysis.

EfficientNet architecture

EfficientNet [START_REF] Tan | EfficientNet: Rethinking model scaling for convolutional neural networks[END_REF] which is among the most efficient models proposed a scaling method to uniformly scale all dimensions of depth, width, and resolution of a network using a compound coefficient. The baseline network of EfficientNet was built with neural architecture search incorporating squeeze-and-excitation in the building block of MobileNetV2. The scaling method is defined as:

𝑑𝑒𝑝𝑡ℎ = 𝛼 ∅ width = 𝛽 ∅ resolution = 𝛾 ∅ , s.t. 𝛼. 𝛽 2 . 𝛾 2 ≈ 2; 𝛼 ≥ 1, 𝛽 ≥ 1, 𝛾 ≥ 1 (1)
where, the coefficient ∅ controls available resources and 𝛼, 𝛽, and 𝛾 are constants obtained by grid search. EfficientNetB0-B7 are a family of architectures scaled up from the baseline network that reflects a good balance of accuracy and efficiency. To the best of our knowledge, EfficientNet architectures have not been used previously for Lyme disease analysis.

Predictive Performance Measures

To compare the predictive performance of the trained CNN models we used accuracy, recall/sensitivity/hit rate/true positive rate (TPR), specificity/selectivity/true negative rate (TNR), precision/ positive predictive value (PPV), negative predictive value (NPV), Matthews correlation coefficient (MCC), Cohen's kappa coefficient (𝜅), positive likelihood ratio (LR+), negative likelihood ratio (LR-), F1-score, confusion matrix and area under the receiver operating characteristic (ROC) curve (AUC) metrics. Confusion matrix is a way of presenting the count of true negatives (TN), false positives (FP), false negatives (FN), and true positives (TP) in a matrix format where the y-axis presents true labels and x-axis presents predicted labels. Accuracy measures the proportion of correctly classified predictions among all the predictions, and it is calculated as:

𝐴𝑐𝑐𝑢𝑟𝑎𝑐𝑦 = 𝑇𝑃 + 𝑇𝑁 𝑇𝑃 + 𝑇𝑁 + 𝐹𝑃 + 𝐹𝑁 (2)
Recall/sensitivity/hit rate/TPR measures the proportion of actual positives correctly identified, and it is expressed as: 𝑅𝑒𝑐𝑎𝑙𝑙, 𝑆𝑒𝑛𝑠𝑖𝑡𝑖𝑣𝑖𝑡𝑦, ℎ𝑖𝑡 𝑟𝑎𝑡𝑒, 𝑇𝑃𝑅 = 𝑇𝑃 𝑇𝑃 + 𝐹𝑁 NPV measures the proportion of negative predictions that are correct, and it is calculated as:

𝑁𝑃𝑉 = 𝑇𝑁 𝑇𝑁 + 𝐹𝑁 (6)
MCC provides a summary of the confusion matrix, and it is calculated as:

𝑀𝐶𝐶 = 𝑇𝑃 * 𝑇𝑁 -𝐹𝑃 * 𝐹𝑁 √(𝑇𝑃 + 𝐹𝑃)(𝑇𝑃 + 𝐹𝑁)(𝑇𝑁 + 𝐹𝑃)(𝑇𝑁 + 𝐹𝑁) (7)
MCC value is in the range [-1, +1], where 0 is like random prediction, +1 means a perfect prediction, and -1 represents inverse prediction. Cohen's kappa coefficient (𝜅) metric is used to assess inter-rater agreement which tells us how the model is performing compared to a random classifier, and it is calculated with the formula:

𝜅 = 𝑝 𝑜 -𝑝 𝑒 1 -𝑝 𝑒 (8)
where, 𝑝 𝑜 is the relative observed agreement among the raters and 𝑝 𝑒 is the hypothetical probability of expected agreement which is defined for 𝑐 categories as:

𝑝 𝑒 = 1 𝑁 2 ∑ 𝑛 𝑐1 𝑛 𝑐2 𝑐 (9)
where, 𝑁 is the total number of observations, and 𝑛 𝑐𝑟 is the number of predictions of category 𝑐 by rater 𝑟. Value of 𝜅 is in the range [-1, +1], where a value of 1 indicates perfect agreement, 0 means agreement only by chance, and a negative value indicates the agreement is worse than the agreement by chance. Likelihood ratio (LR) is used for assessing the potential utility of performing a diagnostic test and it is calculated for both positive test and negative test results called LR+ and LR-, respectively. LR+ is the ratio of the probability of a person having a disease testing positive to the probability of a person without the disease testing positive. LR-is the ratio of the probability of a person having the disease testing negative to the probability of without the disease testing negative. LR+ and LR-are calculated based on sensitivity and specificity values using the following formulas:

𝐿𝑅 + = 𝑠𝑒𝑛𝑠𝑖𝑡𝑖𝑣𝑖𝑡𝑦 1 -𝑠𝑝𝑒𝑐𝑖𝑓𝑖𝑐𝑖𝑡𝑦 (10) 𝐿𝑅 -= 1 -𝑠𝑒𝑛𝑠𝑖𝑡𝑖𝑣𝑖𝑡𝑦 𝑠𝑝𝑒𝑐𝑖𝑓𝑖𝑐𝑖𝑡𝑦 (11)
A value of LR greater than 1 shows increased evidence. F1-Score combines precision and recall, and it is defined as the harmonic mean of precision and recall as follows:

𝐹1 -𝑆𝑐𝑜𝑟𝑒 = 2 * 𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 * 𝑅𝑒𝑐𝑎𝑙𝑙 𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 + 𝑅𝑒𝑐𝑎𝑙𝑙 (12)
ROC curve is a plot of TPR against false positive rate (FPR) at various threshold settings where FPR is defined as:

𝐹𝑃𝑅 = 𝐹𝑃 𝐹𝑃 + 𝑇𝑁 (13)
Area under the ROC curve (AUC) is the measure of the classifier's ability to separate between classes and the higher the AUC, the better the ability of the classifier for separating the positive class from the negative class.

As our dataset was balanced so, accuracy can be considered a good measure of predictive performance [START_REF] Chicco | The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation[END_REF] and we did most of the analysis in terms of accuracy but also kept the other metrics to provide insights for experts from different domains as done in relevant studies [START_REF] Burlina | Skin Image Analysis for Erythema Migrans Detection and Automated Lyme Disease Referral[END_REF][START_REF] Burlina | AI-based detection of erythema migrans and disambiguation against other skin lesions[END_REF]. We also used critical difference (CD) diagram [START_REF] Demšar | Statistical comparisons of classifiers over multiple data sets[END_REF] to rank the CNN models in terms of accuracy and to show the statistically significant difference in predictive performance. A thick horizontal line connects a group of models in the CD diagram that are not significantly different in terms of predictive performance. We used non-parametric Friedman test [START_REF] Friedman | A Comparison of Alternative Tests of Significance for the Problem of $m$ Rankings[END_REF] to reject the null hypothesis of statistical similarity among all the models followed by Nemenyi post-hoc all-pair comparison test [START_REF] Nemenyi | Distribution-free multiple comparisons[END_REF] for showing the difference among the models at a significance level, 𝛼 = 0.1. We kept the confusion matrix, ROC curve, and cross-validation fold-wise details of all the trained models at https://dappem.limos.fr/sdata.html to keep the paper concise and readable.

Model Complexity Measures

To compare the trained CNNs in terms of complexity we used the total number of model parameters, the total number of floating-point operations (FLOPs), average training time per epoch, disk and GPU memory usage, and average inference time per image. FLOPs reveal how computationally costly a model is and we counted FLOPs for each of the models using TensorFlow profiler [START_REF] Abadi | TensorFlow: A system for large-scale machine learning[END_REF] considering a batch size of one. For reporting the average training time per epoch, we calculated the average of the training time of three epochs during transfer learning fine-tuning. Disk usage of a CNN model is the amount of storage required to save the model architecture along with weights. We calculated the GPU usage of a CNN model by inspecting the memory allocated in GPU after loading a trained instance of the model. To measure the average inference time per image of a model we took the average of three hundred inferences on the same input image.

Model Explainability

Explainability is important for AI tools especially in the case of medical applications [START_REF] Vellido | The importance of interpretability and visualization in machine learning for applications in medicine and health care[END_REF]. We used Grad-CAM explainability technique for visualizing the regions of the input image that are significant for predictions from the CNN models as shown in Figure 13. Grad-CAM uses gradient flowing into the ultimate convolution layer for producing heatmaps, and it is a kind of post-hoc attention that can be applied on an already trained model. Grad-CAM provides similar Figure 13: Gard-CAM visualization example.

Input Image

Grad-CAM Visualization result to occlusion sensitivity map [START_REF] Zeiler | Visualizing and understanding convolutional networks[END_REF] that works by masking patches of the input image, but Grad-CAM is much faster to calculate compared to image occlusion [START_REF] Selvaraju | Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization[END_REF].

Transfer Learning Approach

In this study, we used transfer learning as our Lyme dataset is not huge enough to obtain good performance by training large CNNs from scratch. We started with a CNN already pretrained on ImageNet dataset and after removing the original ImageNet classification head our EM classification head consisting of Global Average Pooling (GAP) layer, dropout layer, and a fully connected layer with sigmoid activation for binary classification was added as shown in Figure 14. According to our experiments fine-tuning the whole CNN architecture after training the classifier head with our Lyme dataset performed poorly compared to the partial fine-tuning of several layers at the end of the CNN while keeping rest of the layers frozen. We empirically found out the number of layers 𝑈 to fine-tune from 𝑁 number of ImageNet pre-trained layers for each of the CNN architectures used in this study. According to the experimental results pretraining the unfrozen part with HAM10000 dataset further improved the performance of the DCCNs.

Experimental Studies

The following subsections describe experimental settings including model selection and parameter settings, software and hardware used for the study, the experimental results, and recommendations for model selection.

Experimental Settings

In this study, we benchmarked twenty-three CNN models, namely VGG16, VGG19, ResNet50, ResNet101, ResNet50V2, ResNet101V2, InceptionV3, InceptionV4, InceptionResNetV2, Xception, DenseNet121, DenseNet169, DenseNet201, MobileNetV2, MobileNetV3Large, MobileNetV3Small, NASNetMobile, EfficientNetB0, EfficientNetB1, EfficientNetB2, EfficientNetB3, EfficientNetB4, and EfficientNetB5 for diagnosing Lyme disease from EM images. These models were selected to explore a diverse set of CNN models covering various prospects, like different architectures, depths, and complexities. To the best of our knowledge ResNet50 is the only publicly available trained CNN that was used for Lyme disease identification by Burlina et al. [START_REF] Burlina | Skin Image Analysis for Erythema Migrans Detection and Automated Lyme Disease Referral[END_REF]. We are calling this model ResNet50-Burlina which is a collection of five models (trained on five-fold cross-validation data) available at https://github.com/neil454/lyme-1600-model. We did extensive analysis on the ResNet50 architecture using seven different transfer learning configurations: (i) training ResNet50 model on our Lyme dataset from scratch without using learning (called ResNet50-NTL, where, NTL stands for no transfer learning), (ii) pretraining ResNet50 model with only HAM10000 data followed by fine-tuning all the layers with our Lyme dataset (called ResNet50-HAM-FFT, where HAM means HAM10000 and FFT stands for full fine-tuning), (iii) training only the EM classifier head of an ImageNet pre-trained ResNet50 model with our Lyme dataset (called ResNet50-IMG-WFT, where IMG means ImageNet and WFT stands for without fine-tuning), (iv) fine-tuning all the layers of ImageNet pre-trained ResNet50 model with our Lyme dataset (called ResNet50-IMG-FFT), (v) fine-tuning 𝑈 no of layers of an ImageNet pre-trained ResNet50 model with our Lyme dataset (called ResNet50-IMG-FT𝑈, where FT𝑼 means fine-tuning 𝑈 no of layers), (vi) pretraining the whole ImgaeNet pre-trained ResNet50 model by HAM10000 data before finetuning 𝑈 layers with our Lyme dataset (called ResNet50-IMG-HAMFP-FT𝑈, where, HAMFP means full pre-training with HAM10000 dataset), and (vii) pretraining only the unfrozen 𝑈 layers of a ImgaeNet pre-trained ResNet50 model with HAM10000 data before fine-tuning 𝑈 layers with our Lyme dataset (called ResNet50-IMG-HAMPP-FT𝑈, where, HAMPP means partial pretraining with HAM10000 dataset). To see the effect of data augmentation, we trained a ResNet50 model without data augmentation and transfer learning (called ResNet50-NoAug, where NoAug means no data augmentation). All the other models were trained with data augmentation as described in section 2.1. According to experimental results (discussed below), ResNet50-IMG-HAMPP-FT𝑈 configuration performed best, and we used this configuration (pretraining only the unfrozen 𝑈 layers of an ImgaeNet pre-trained model with HAM10000 data before fine-tuning 𝑈 layers with our Lyme dataset) for training all the architectures used in this study. For simplicity, the best performing trained models of each of the architectures are presented in ModelName-𝑼 format, where 𝑈 represents the no of unfrozen layers. For example, EfficientNetB0-187 means EfficientNetB0-IMG-HAMPP-FT187 and ResNet50-141 means ResNet50-IMG-HAMPP-FT141.

For training all the models, we used a dropout rate of 0.2 for the dropout layer in EM classifier head section. Adaptive Moment Estimation (ADAM) [START_REF] Kingma | A method for stochastic optimization[END_REF] optimizer with exponential decay rate for the first and second moment estimates set to 0.9 and 0.999 respectively was used with a learning rate of 0.0001 for training the classifier head and 0.00001 for fine-tuning. We also used early stopping to terminate the training if there was no improvement in validation accuracy for ten epochs. A batch size of 32 was used. For reporting the number of layers to unfreeze during transfer learning, we stated the total number of layers to unfreeze including layers containing both trainable and non-trainable parameters.

We used three NVIDIA QUADRO RTX 8000 GPUs and two Desktop Computers with Intel Xeon W-2175 processor, 64 GB DDR4 RAM, and Ubuntu 18.04 operating system to perform all the experiments. Python v3.6.9, and TensorFlow v2.4.1 platform [START_REF] Abadi | TensorFlow: A system for large-scale machine learning[END_REF] were used for all the implementations and experimentations of this study.

Experimental Results

Čuk et al. [START_REF] Čuk | Supervised visual system for recognition of erythema migrans, an early skin manifestation of lyme borreliosis, Stroj[END_REF] reported accuracies in the range of 69.23% to 80.42% using classical machine learning methods whereas, Burlina et al. [START_REF] Burlina | AI-based detection of erythema migrans and disambiguation against other skin lesions[END_REF] reported the best accuracy of 81.51% using ResNet50 architecture for the case of EM vs all classification problems. There was a common subset of images collected from the internet in both the dataset of Burlina et al. [START_REF] Burlina | Skin Image Analysis for Erythema Migrans Detection and Automated Lyme Disease Referral[END_REF] and our Lyme dataset. ResNet50-Burlina model gave an accuracy of 76.05% when tested on our full dataset as shown in Table 1.

Table 1: Performance metrics of ResNet50-Burlina model trained by Burlina et al. [START_REF] Burlina | Skin Image Analysis for Erythema Migrans Detection and Automated Lyme Disease Referral[END_REF] tested on the whole dataset of this study. Within each cell, the value after (±) symbol represents the standard deviation. 2. To summarize, pretraining only the unfrozen part of an ImageNet pre-trained CNN with HAM10000 data provided the best accuracy according to our experiments. So, for all the other CNN architectures, we only reported the performance resulting from this configuration.

Performance metrics for the best performing configuration of all the CNN architectures used in this study are shown in Table 3. All these models used HAM10000 pretraining for the unfrozen part of the network. The number at the end of the model's name represents the number of layers unfrozen during transfer learning fine-tuning. ResNet50-141 achieved the best accuracy of 84.42%. Most of the models except MobileNetV2-62, MobileNetV3Small-182, and NASNetMobile-617 showed good AUC values of above 90% and good sensitivity suggesting that these CNNs can be a good choice for building pre-scanners for Lyme disease. Figure 16 shows the CD diagram in terms of accuracy for these models. The Friedman test null hypothesis was rejected with a 𝑝 value of 0.0822. From the CD diagram, we can see that ResNet50-141 achieved the best average ranking followed by VGG19-13 and DenseNet121-379 respectively. Xception and Inception-based architectures had a similar ranking. NasNetMobile-617 ranked worst among all the models. The accuracy of the models varied from 81.3% to 84.42% and there is no statistically significant difference in terms of accuracy metric among most of the trained models. Overall, ResNet50-141 performed better in terms of various metrics (5 out of 11) as highlighted in Table 3.

Table 2: Five-fold cross-validation performance metrics of ResNet50 based models. Within each cell, the value after (±) symbol represents the standard deviation across five folds. Bold indicates the best result for each of the metrics.

Discussion

The experimental result described above makes it evident that CNNs have great potential to be used for Lyme disease pre-scanner application. Figure 18 shows a bubble chart reporting model accuracy vs FLOPs. The size of each bubble represents the number of parameters of the model. This figure serves as a guideline for selecting models based on complexity and accuracy. It can be seen from the figure that EfficientNetB0-187 is a good choice with reasonable accuracy for resource-constrained mobile platforms. EfficientNetB0-187 also showed good results in Grad-CAM visualization. If resource constraint is not a problem, then RestNet50-141 can be used for the best accuracy.

Even the lightweight EfficientNetB0-187 model showed good performance, and it can be directly deployed in mobile devices without requiring an internet connection for processing the lesion image in a remote server. It can help people living in remote areas without good internet facilities with an initial assessment of the probability of Lyme disease.

For this study, we utilized images from the internet alongside images collected from several hospitals in France. This approach was inspired by related studies on skin lesion analysis. Although a portion of images in our dataset was collected from the internet the annotation of the dataset is reliable because we ignored the online labels, and all the images were reannotated by expert dermatologists and infectiologists.

Existing works including this study on AI-based Lyme disease analysis only utilize images but including patients' metadata can be a great way of strengthening the analysis. We trained the CNNs with whole images without EM lesion segmentation. The effect of the EM lesion segmentation on the predictive performance of CNNs can be an interesting study. Another limitation is that dark-skinned samples are underrepresented in our dataset. A limited number of samples in the dataset with skin hair artifact over the EM lesion is also a concern. Although hair removal algorithms can be used for removing skin hair it will increase the computational complexity of the real-time mobile application. A better alternate can be augmenting the training dataset with artificial skin hair.

Conclusion

In this study, we benchmarked and extensively analyzed twenty-three well-known CNNs based on predictive performance, complexity, significance tests, and explainability using a novel Lyme disease dataset to find out the effectiveness of CNNs for Lyme disease diagnosis from EM images. To improve the performance of CNNs we utilized HAM10000 dataset with ImageNet pretrained models for transfer learning. We also provided guidelines for model selection. We found that even the lightweight models like EffiicentNetB0 performed well suggesting the application of CNNs for Lyme disease pre-scanner mobile applications which can help people with an initial assessment of the probability of Lyme disease and referring them to expert dermatologist for further diagnosis. Resource intensive models like ResNet50 can be effective for building computer applications to assist non-expert practitioners with identifying EM. We also made all the trained models publicly available, which can be utilized by others for transfer learning and building prescanners for Lyme disease.

Figure 1 :

 1 Figure 1: Patterns of erythema migrans (EM). (source: https://commons.wikimedia.org/wiki/Category:Erythema_migrans, Accessed April 1, 2021)

Figure 2 :

 2 Figure 2: Graphical overview of the study on the effectiveness of CNNs for the diagnosis of Lyme disease from images.

Figure 4 :

 4 Figure 4: Data augmentation examples.

Figure 3 :

 3 Figure 3: Five-fold cross-validation setup.

Figure 5 :

 5 Figure 5: VGG16 architecture. Input image is of shape 224 × 224 × 3. 𝐹𝐶 stands for fully connected layer and 𝐾 is the number of target classes.

Figure 7 .

 7 It's the winner of ILSVRC 2015 competition. Depending on the number of weight layers there are many variants of ResNet architecture such as ResNet18, ResNet34, ResNet50, ResNet101, ResNet152, ResNet164, ResNet1202, etc., where the number represents the count of weight layers. He et al. [32] proposed ResNetV2 with pre-activation of the weight layers as opposed to the post-activation of original ResNet architecture. InceptionResNet is a hybrid of Inception and ResNet architecture having two variations named InceptionResNetV1 and InceptionResNetV2, which differ mainly in terms of the number of used filters [30]. Burlina et al. [3] used ImageNet pre-trained ResNet50 and Burlina et al. [8] used ImageNet pre-trained ResNet50, ResNet152, and InceptionResNetV2 architectures for Lyme disease analysis.

Figure 6 :

 6 Figure 6: Inception module of Inception architecture.

Figure 7 :

 7 Figure 7: Residual block of ResNet architecture.identity connection

Figure 8 :

 8 Figure 8: Building block of DenseNet architecture.

Figure 9 :

 9 Figure 9: Depthwise separable convolution.

Figure 10 :

 10 Figure 10: Building block of MobileNetV2 architecture.

Figure 11 :

 11 Figure 11: Building block of Xception architecture.

Figure 12 :

 12 Figure 12: Building block of NASNet architecture.

( 3 )

 3 Specificity/selectivity/ TNR measures the proportion of actual negatives correctly identified, and it is expressed as:𝑆𝑝𝑒𝑐𝑖𝑓𝑖𝑐𝑖𝑡𝑦, 𝑆𝑒𝑙𝑒𝑐𝑡𝑖𝑣𝑖𝑡𝑦, 𝑇𝑁𝑅 = 𝑇𝑁 𝑇𝑁 + 𝐹𝑃 (4)Precision/ PPV measures the proportion of correct positive predictions, and it is calculated as:

Figure 14 :

 14 Figure 14: Transfer learning workflow used in this study. 𝐺𝐴𝑃 stands for Global Average Polling. 𝑁 is the number of ImageNet pre-trained layers and 𝑈 represents the number of layers used for fine-tuning.

  the predictive performance measures of our experimentation with ResNet50 architecture. ResNet50-NoAug model resulting from training a ResNet50 architecture from scratch without using data augmentation and transfer learning gave an accuracy of 61.42%. ResNet50-NTL model obtained by training ResNet50 architecture with data augmentation and without transfer learning improved the accuracy to 76.35%. So, data augmentation provided large gain in predictive performance (ResNet50-NTL compared to ResNet50-NoAug). ResNet50-HAM-FFT model resulting from pretraining ResNet50 architecture with only HAM10000 data followed by fine-tuning of all the layers with our Lyme dataset showed a degraded accuracy of 72.27%. ResNet50-IMG-WFT, generated by training only the EM classifier head of an ImageNet pre-trained ResNet50 architecture improved the accuracy to 78.94%. ResNet50-IMG-FFT, resulting from fine-tuning all the layers of ImageNet pre-trained ResNet50 architecture, further improved the classification accuracy to 82.22%. Whereas ResNet50-IMG-FT141, model resulting from fine-tuning 141 layers of pre-trained ResNet50 architecture gave an accuracy of 83.24% which is better compared to unfreezing the full architecture. ResNet50-IMG-HAMFP-FT141, model resulting from pretraining the whole ImgaeNet pre-trained ResNet50 model by HAM10000 data before fine-tuning 141 layers with our Lyme dataset reduced the accuracy to 82.35%. But pretraining only the unfrozen 141 layers with HAM10000 data gave us the model ResNet50-IMG-HAMPP-FT141 with the best accuracy of 84.42%. Figure 15 shows the CD diagram in terms of accuracy for these ResNet50 based models. We excluded ResNet50-Burlina model from this diagram because the model was tested on the whole dataset as opposed to other configurations. The Friedman test null hypothesis was rejected with a 𝑝 value of 0.00003. From the CD diagram, we can see that ResNet50-IMG-HAMPP-FT141 achieved the best average ranking among the compared models. Although there is no statistically significant difference among ResNet50-IMG-FFT, ResNet50-IMG-FT141, ResNet50-IMG-HAMFP-FT141, and ResNet50-IMG-HAMPP-Model ()% Metric FT141 in terms of accuracy the ResNet50-IMG-HAMPP-FT141 model performed better in terms of most of the metrics (7 out of 11) as highlighted in Table

  the complexities of the CNN models used in this study. MobileNetV3Small-182 was the most lightweight model with the lowest number of parameters, FLOPs, and memory usage. InceptionResNetV2-500 has the highest number of parameters and memory usage and slowest inference time. Xception-118 was the fastest in terms of inference time. VGG19-13 required the highest number of FLOPs. ResNet50V2-105 required the least amount of time to train on average whereas, EfficientNetB5-444 was the slowest to train.

Figure 17

 17 shows the Grad-CAM visualizations of the models trained on the same training fold for two test images. The test images are shown at the lower right corner of the image. From the figure, it can be seen that different versions of EfficientNet focused more on the lesion part of the image compared to other models.

Figure 15 :

 15 Figure 15: Accuracy critical difference diagram for ResNet50 models. The models are ordered by best to worst average ranking from left to right. The number beside a model's name represents the average rank of the model. CD is the critical difference for Nemenyi post-hoc test. Thick horizontal line connects the models that are not statistically significantly different.

Figure 16 :

 16 Figure 16: Accuracy critical difference diagram for the best performing configurations of the trained DCNN models. The models are ordered by best to worst average ranking from left to right. The number beside a model's name represents the average rank of the model. CD is the critical difference for Nemenyi post-hoc test. Thick horizontal line connects the models that are not statistically significantly different.

Figure 17 :

 17 Figure 17: Grad-CAM visualization of the trained models. Input images are shown at the lower right corner.

Figure 18 :

 18 Figure 18: Bubble chart reporting model accuracy vs floating-point operations (FLOPs). The size of each bubble represents number of model parameters measured in millions unit. Beside each model name the three values represent FLOPs, accuracy, and model parameters, respectively.

Table 3 :

 3 Five-fold cross-validation performance metrics for the best performing configurations of the trained CNN models. Within each cell, the value after symbol represents the standard deviation across five folds. Bold indicates the best result for each of the metrics.

	Metric
	Model
	VGG16-8 ()%	82.17	85.77	78.31	81.12	83.88	0.6453	0.6422	4.0983	0.1802	0.8328	0.9011
		±1.23	±3.58	±4.36	±2.62	±3.02	±0.0253	±0.0249	±0.7329	±0.0388	±0.0116	±0.0079
	VGG19-13	84.14	85.29	82.9	84.32	84.0	0.6826	0.6823	5.0924	0.1777	0.8479	0.913
		±1.62	±1.69	±2.63	±1.97	±1.67	±0.0323	±0.0326	±0.6884	±0.0214	±0.0146	±0.0074
	ResNet50-141 84.42	87.93	80.65	83.1	86.19	0.6893	0.6874	4.703	0.1493	0.8541	0.9189
		±1.36	±1.47	±3.59	±2.49	±1.27	±0.0263	±0.0277	±0.8624	±0.0155	±0.0106	±0.0115
	ResNet101-150 82.64	83.68	81.52	82.97	82.4	0.6528	0.6522	4.6001	0.2004	0.8329	0.9044
		±2.1	±3.49	±2.29	±1.8	±3.09	±0.0419	±0.0418	±0.6257	±0.0427	±0.022	±0.0094
	ResNet50V2-	82.37	85.53	78.96	81.66	83.72	0.6493	0.6461	4.3618	0.1819	0.8343	0.9013
	105	±2.15	±3.35	±6.13	±3.83	±2.63	±0.0411	±0.0439	±1.0495	±0.0349	±0.017	±0.0133
	ResNet101V2-	82.58	81.9	83.32	84.17	81.31	0.6535	0.6515	5.104	0.2163	0.8292	0.9118
	233	±2.21	±4.78	±3.71	±2.55	±3.7	±0.0429	±0.0439	±0.9811	±0.0541	±0.0254	±0.0149
	InceptionV3-274 82.73	86.57	78.6	81.33	84.52	0.6551	0.6533	4.1259	0.1714	0.8385	0.9052
		±2.08	±2.42	±2.8	±2.12	±2.52	±0.0419	±0.0419	±0.639	±0.0328	±0.0195	±0.0185
	InceptionV4-327 82.76	85.7	79.58	81.92	84.02	0.6561	0.6541	4.2716	0.179	0.837	0.9092
		±1.78	±3.96	±2.87	±1.8	±3.41	±0.0358	±0.0353	±0.5734	±0.0465	±0.0197	±0.019

Table 3 :

 3 (continued).

	InceptionResNet	82.67	83.54	81.74	83.17	82.37	0.6541	0.653	4.6886	0.2009	0.8329	0.9011
	V2-500	±2.06	±3.88	±3.16	±2.16	±3.32	±0.0406	±0.041	±0.7264	±0.0456	±0.0218	±0.0133
	Xception-118 82.48	83.16	81.75	83.08	82.16	0.6507	0.6492	4.6434	0.2054	0.8304	0.9081
		±2.45	±5.1	±2.76	±1.94	±4.4	±0.0487	±0.0484	±0.6571	±0.0609	±0.0276	±0.0148
	DenseNet121-	83.88	85.85	81.75	83.49	84.35		0.6768	4.7169	0.173	0.8465	0.9158
	379	±0.92	±1.76	±0.95	±0.69	±1.57	±0.0186	±0.0184	±0.254	±0.0211	±0.01	±0.0097
	DenseNet169-	83.66	88.6	78.35	81.54	86.68	0.6758	0.6717	4.1454	0.1446	0.8486	0.9123
	395	±1.25	±3.59	±2.75	±1.53	±3.33	±0.0265	±0.0249	±0.4187	±0.0414	±0.0138	±0.0129
	DenseNet201-	83.12	85.61	80.45	82.61	83.93	0.663	0.6615	4.5729	0.1783	0.8403	0.9125
	561	±1.11	±1.81	±3.92	±2.7	±1.13	±0.0221	±0.0228	±0.9885	±0.0153	±0.0073	±0.0083
	MobileNetV2-62 81.68	81.94	81.39	82.55	80.85	0.6337	0.6332	4.4256	0.222	0.8222	0.8933
		±1.99	±3.49	±1.26	±1.21	±3.09	±0.0394	±0.0395	±0.3596	±0.0441	±0.0218	±0.0135
	MobileNetV3Sm	81.53	84.93	77.87	80.6	82.91	0.6315	0.6294	3.9496	0.1933	0.8265	0.896
	all-182	±1.98	±3.29	±3.89	±2.55	±2.85	±0.0398	±0.04	±0.6356	±0.0386	±0.0186	±0.013
	MobileNetV3Lar	82.74	83.69	81.71	83.26	82.3	0.6548	0.6542	4.8573	0.2002	0.8344	0.9034
	ge-193	±2.17	±0.43	±4.6	±3.39	±0.89	±0.0437	±0.0442	±1.1585	±0.0117	±0.017	±0.0094
	NASNetMobile -	81.3	83.2	79.25	81.29	81.48	0.6261	0.6251	4.1452	0.2117	0.8219	0.8897
	617	±1.45	±1.66	±3.98	±2.65	±1.07	±0.0287	±0.0297	±0.7283	±0.0156	±0.0108	±0.0152
	EfficientNetB0-	83.13	85.21	80.89	82.83	83.79	0.6636	0.6618	4.5522	0.1817	0.8392	0.9094
	187	±1.2	±3.91	±2.95	±1.75	±3.19	±0.0244	±0.0237	±0.6116	±0.0427	±0.0147	±0.0129
	EfficientNetB1-	82.42	85.85	78.71	81.37	83.9	0.6492	0.647	4.1494	0.179	0.835	0.9088
	308	±1.04	±2.14	±3.75	±2.34	±1.59	±0.0202	±0.0214	±0.6707	±0.0209	±0.0074	±0.0134
	EfficientNetB2-	82.75	84.95	80.39	82.39	83.4	0.6556	0.6542	4.4211	0.1865	0.8359	0.9075
	316	±1.4	±3.41	±3.02	±1.91	±2.69	±0.0276	±0.0279	±0.6202	±0.0379	±0.0158	±0.0082
	EfficientNetB3-	83.46	85.15	81.64	83.4	83.9	0.6704	0.6685	4.7361	0.1803	0.8416	0.9163
	194	±0.87	±4.28	±2.9	±1.6	±3.14	±0.0157	±0.0167	±0.6283	±0.0443	±0.0144	±0.0074
	EfficientNetB4-	82.98	87.55	78.06	81.2	85.46	0.6613	0.6581	4.0946	0.1589	0.842	0.9138
	384	±1.31	±2.2	±3.76	±2.33	±1.76	±0.0249	±0.0268	±0.6159	±0.0233	±0.0107	±0.0074
	EfficientNetB5-	83.7	86.85	80.32	82.71	85.17	0.6752	0.6729	4.5562	0.1629	0.8466	0.9138
	444	±1.21	±2.89	±3.73	±2.39	±2.38	±0.024	±0.0245	±0.7645	±0.0303	±0.0108	±0.0161

Table 4 :

 4 Complexity metrics of trained CNN models. Bold indicates the best result for each of the metrics.

	Model	Parameters	FLOPs	Average training	Disk usage	GPU usage	Average	Input
		(million)	(giga)	time	(megabyte)	(megabyte)	inference	shape
				(sec per epoch)			time
							(sec per
							image)
	VGG16-8	14.72	30.7	111	146.24	565	0.0426 224x224x3
	VGG19-13	20.02	39	164	216.03	565	0.0431 224x224x3
	ResNet50-141	23.59	7.75	113	268.64	821	0.0484 224x224x3
	ResNet101-150	42.66	15.2	123.33	378.41	821	0.0539 224x224x3
	ResNet50V2-105	23.57	6.99	76	258.84	821	0.0464 224x224x3
	ResNet101V2-233	42.63	14.4	152	429.45	821	0.0599 224x224x3
	InceptionV3-274	21.8	11.5	133	246.94	821	0.0540 224x224x3
	InceptionV4-327	41.18	24.6	223.33	424.93	1333	0.0735 299x299x3
	InceptionResNetV2-500 54.34	26.4	281.33	588.12	1333	0.0958 299x299x3
	Xception-118	20.86	16.8	243.33	238.48	821	0.0392 299x299x3
	DenseNet121-379	7.04	5.7	140.67	78.7	437	0.0673 224x224x3
	DenseNet169-395	12.64	6.76	130	128.59	565	0.0686 224x224x3
	DenseNet201-561	18.32	8.63	182.67	198.7	565	0.0840 224x224x3
	MobileNetV2-62	2.26	0.613	78	24.02	341	0.0429 224x224x3
	MobileNetV3Small-182	1.53	0.174	81	17.8	341	0.0444 224x224x3
	MobileNetV3Large-193	4.23	0.564	86.33	48.34	373	0.0444 224x224x3
	NASNetMobile -617	4.27	1.15	152	50.74	373	0.0741 224x224x3
	EfficientNetB0-187	4.05	0.794	87	46.59	373	0.0523 224x224x3
	EfficientNetB1-308	6.58	1.41	158.33	75.89	437	0.0546 240x240x3
	EfficientNetB2-316	7.77	2.04	210	89.53	437	0.0565 260x260x3
	EfficientNetB3-194	10.79	3.74	143	117.6	565	0.0648 300x300x3
	EfficientNetB4-384	17.68	8.97	431	202.2	565	0.0614 380x380x3
	EfficientNetB5-444	28.52	20.9	771	325.21	821	0.0659 456x456x3

Acknowledgments

This research was funded by the European Regional Development Fund, project DAPPEM -AV0021029. The DAPPEM project («Développement d'une APPlication d'identification des Erythèmes Migrants à partir de photographies»), was coordinated by Olivier Lesens and was carried out under the Call for Proposal 'Pack Ambition Research' from the Auvergne-Rhône-Alpes region, France. This work was also partially funded by Mutualité Sociale Agricole (MSA), France.

We can share images collected from the internet and the corresponding labels assigned by the doctors subject to an agreement that the images will not be made public because we do not have permission from the owners of the images. Interested researchers can send a request to dappem-project@inrae.fr for the data. The images from the hospital are not shareable because of the confidentiality agreement signed with the patients.

Declaration of Competing Interest

The authors have declared no conflict of interest.