Elie Saad
email: elie.saad.stud@pw.edu.pl

Marcin Paprzycki
email: marcin.paprzycki@ibspan.waw.pl

Maria Ganzha
email: maria.ganzha@pw.edu.pl

Practical Aspects of Zero-Shot Learning ⋆

Keywords: Zero-Shot Learning, Meta-Classifier, Performance analysis

One of important areas of machine learning research is zeroshot learning. It is applied when properly labeled training data set is not available. A number of zero-shot algorithms have been proposed and experimented with. However, none of them seems to be the "overall winner". In situations like this, it may be possible to develop a meta-classifier that would combine "best aspects" of individual classifiers and outperform all of them. In this context, the goal of this contribution is twofold. First, multiple state-of-the-art zero-shot learning methods are compared for standard benchmark datasets. Second, multiple meta-classifiers are suggested and experimentally compared (for the same datasets).

Introduction

Classic machine learning methods focus on classifying instances, when the classes have been "seen" during training. Whereas, many real world applications require classifying "entities" that have not been seen before, e.g., object recognition (where every object is a category), cross-lingual dictionary induction (where every word is a category), etc. Here, one of reasons for this situation is lack of resources to annotate the available (and, possibly, systematically growing) dataset. To solve this problem, zero-shot learning has been proposed. Hence, zero-shot learning can be seen as an example of "frugal AI", where the frugality is caused by lack of data annotation, e.g. due to lack of resources.

While large number of zero-shot learning approaches have been proposed (see, for instance, [START_REF] Lampert | Attribute-based classification for zeroshot learning of object categories[END_REF][START_REF] Larochelle | Zero-data learning of new tasks[END_REF][START_REF] Rohrbach | Evaluating knowledge transfer and zero-shot learning in a large-scale setting[END_REF][START_REF] Yu | Attribute-based transfer learning for object categorization with zero/one training example[END_REF][START_REF] Xu | Matrix trifactorization with manifold regularizations for zero-shot learning[END_REF][START_REF] Ding | Low-rank embedded ensemble semantic dictionary for zero-shot learning[END_REF][START_REF] Xian | Zero-shot learning-a comprehensive evaluation of the good, the bad and the ugly[END_REF]), they have been experimented with, mostly, in different contexts. Nevertheless, it is possible to conjecture that there is not a single "best" approach. In situations like this, meta-classifiers have been proposed. They receive suggestions from individual classifiers and "judge", which of them is the "best". The assumption is that performance of a meta-classifier will perform better than any of the individual ones.

In this context, the the contribution of this work is twofold. First, a comprehensive comparison of five state-of-the-art zero-shot learning approaches applied to five standard benchmarking datasets. Second, explorations into possibility of developing a meta-classifier for zero-shot learning.

Problem Formulation

In zero-shot learning, instances in the feature space are represented as a vector (usually, a real number vector). A subset of training instances is labeled, while testing instances are not. The labeled training classes are referred to as seen classes, while the unlabeled testing instances are referred to as unseen classes. In this work, the seen and unseen datasets are assumed to be disjoint. However, it is recognized that an interesting area of research covers also the case where some (usually small) overlap between seen and unseen classes exists.

Let us now formulate the specific problem. Given a dataset of image embeddings X = {(x i , y i ) ∈ X × Y|i = 1, ..., N tr + N te }, each image is a real D-dimensional embedding vector comprised of features x i ∈ R D , and each class label is represented by an integer y i ∈ Y ≡ {1, ..., N 0 , N 0 + 1, ..., N 0 + N 1 } giving N 0 + N 1 distinct classes. Here, for generality, it is assumed that

X def = R D .
The dataset X is divided into two subsets: [START_REF] Abreu | Analyzing the benefits of using a fuzzy-neuro model in the accuracy of the neurage system: An agent-based system for classification tasks[END_REF] training set and (2) test set. The training set is given by

X tr = {(x tr i , y tr i ) ∈ X × Y 0 |i = 1, ..., N tr }, where y tr i ∈ Y 0 ≡ {1, ..., N 0 }, resulting in N 0 distinct training classes. The test set is given by X te = {(x te i , y te i ) ∈ X × Y 1 |i = N tr + 1, ..., N te }, where y te i ∈ Y 1 ≡ {N 0 + 1, ..., N 0 + N 1 } providing N 1 distinct test classes.
The goal of zero-shot learning is to train a model on the training set X tr that performs "well" for the test set X te . Obviously, zero-shot learning is infeasible without auxiliary information associating labels from the training and the test sets, when the training and test set classes are disjoint Y 0 ∩Y 1 = ∅. The proposed solution is to represent each class label y (1 ≤ y ≤ N 0 + N 1 ) by its prototype π(y) = p ∈ P def = R M (also known as its semantic embedding). Here, π(•) : Y 0 ∪ Y 1 → P is the prototyping function and P is the semantic embedding space. The prototype vectors are such that any two class labels y 0 and y 1 are similar if and only if their prototype representations π(y 0 ) = p 0 and π(y 1 ) = p 1 are close in the semantic embedding space P. For example, their inner product is large in P, i.e, π(y 0 ), π(y 1 ) P is large. Prototyping all of class labels into a joint semantic space, i.e., {π(y)|y ∈ Y 0 ∪ Y 1 }, results in labels becoming related. This takes care of the problem of disjoint class sets, and the model can learn from the labels in the training set, and predict labels from the test set.

Literature Review

Many approaches have been proposed for the zero-shot learning. Here, Deep Visual Semantic Embedding (DeViSE ) [START_REF] Frome | Devise: A deep visual-semantic embedding model[END_REF], Attribute Label Embedding (ALE ) [START_REF] Akata | Label-embedding for image classification[END_REF],

and Structured Joint Embedding (SJE ) [START_REF] Akata | Evaluation of output embeddings for fine-grained image classification[END_REF] use a bilinear compatibility function. They follow the Stochastic Gradient Descent (SGD), where the SGD [START_REF] Shalev-Shwartz | Understanding machine learning: From theory to algorithms[END_REF] is implicitly regularized by early stopping. The Embarrassingly Simple Approach to Zero-Shot Learning (ESZSL) [START_REF] Romera-Paredes | An embarrassingly simple approach to zero-shot learning[END_REF] uses the square loss to learn the bilinear compatibility function and explicitly defines regularization with respect to the Frobenius norm. Kodirov et al. in [START_REF] Kodirov | Semantic autoencoder for zero-shot learning[END_REF] proposes a semantic encoder-decoder model, called Semantic AutoEncoder (SAE ), where the training instances are projected into the semantic embedding space P with the projection matrix W , and then projected back into the feature space X , with projection matrix W *where W * is the conjugate transpose.

Another group of approaches adds a non-linearity component to the linear compatibility function. Here, Latent Embeddings (LATEM) [START_REF] Xian | Latent embeddings for zero-shot classification[END_REF] learns a piecewise linear compatibility and uses the same ranking loss function as the DeViSE, as well as the SGD. LATEM learns multiple mappings corresponding to different visual characteristics of the data and uses a latent variable to select, which matrix mapping to use in predictions. Cross Modal Transfer (CMT) in [START_REF] Socher | Zeroshot learning through cross-modal transfer[END_REF] trains two-layer neural network using nonlinear mapping function tanh.

A third set of approaches uses probabilistic mapping [START_REF] Lampert | Attribute-based classification for zeroshot learning of object categories[END_REF]. Here, Direct Attribute Prediction (DAP) algorithm makes class prediction by combining scores of the learned attribute classifiers, whereas the Indirect Attribute Prediction (IAP) algorithm approximates the probabilities of the attributes associated with the training instances, by predicting probabilities of each training class value.

One more group of approaches expresses the input image features and the semantic embeddings as a mixture of seen classes. Here, Semantic Similarity Embedding (SSE) [START_REF] Zhang | Zero-shot learning via semantic similarity embedding[END_REF] connects class values within both semantic embedding space and feature embedding space of feature vectors. Convex Combination of Semantic Embeddings (ConSE) [START_REF] Norouzi | Zero-shot learning by convex combination of semantic embeddings[END_REF] learns the probability of a feature vector belonging to a specific class. Synthesized Classifiers (SynC) [START_REF] Changpinyo | Synthesized classifiers for zero-shot learning[END_REF] first, learns a mapping from the semantic embedding space to the model space, which holds the classifiers associating real class values and phantom class values. The class values along with the set of phantom class values, which are introduced to connect the seen and unseen class values, form a weighted bipartite graph. The objective of SynC is to align the semantic embedding space and the model space, resulting in reducing the distortion error. The Generative Framework for Zero-Shot Learning (GFZSL) [START_REF] Verma | A simple exponential family framework for zero-shot learning[END_REF] applies generative modeling. Here, each class-conditional distribution is modeled as a multivariate Gaussian distribution.

In the fifth set of models, during training, both seen and unseen classes are included in the training data. The Discriminative Semantic Representation Learning (DSRL) [START_REF] Ye | Zero-shot classification with discriminative semantic representation learning[END_REF] can be applied to any method that is based on a compatibility learning function. It learns the feature embedding vectors from the training instances with non-negative matrix factorization and aligns the feature embedding vectors with the semantic embedding vectors of their corresponding class values.

Selection of Methods and Experimental Setup

Based on analysis of the literature five zero-shot learning approaches have been selected: (1) DeViSE, (2) ALE, (3) SJE, (4) ESZSL, and (5) SAE. This was done, due to the fact that they have been found to be the "most competitive".

In the literature, the following datasets have been found to be the most popular benchmarks for zero-shot learning: (a) Caltech-UCSD-Birds 200-2011 CUB [START_REF] Welinder | [END_REF], (b) Animals with Attributes 1 AWA1 [START_REF] Lampert | Attribute-based classification for zeroshot learning of object categories[END_REF], (c) Animals with Attributes 2 AWA2 [START_REF] Xian | Zero-shot learning-a comprehensive evaluation of the good, the bad and the ugly[END_REF], (d) Attribute Apascal&Ayahoo aPY [START_REF] Farhadi | Describing objects by their attributes[END_REF], and (e) Scene UNderstanding SUN [START_REF] Patterson | Sun attribute database: Discovering, annotating, and recognizing scene attributes[END_REF]. Hence, they were selected.

Due to lack of space, state-of-the-art in meta-classifiers has been omitted from the literature review. However, note that this work is focused on zeroshot learning. Hence, five standard meta-classifiers have been tried: (A) Meta-Decision Tree MDT [START_REF] Todorovski | Combining classifiers with meta decision trees[END_REF], (B) deep neural network DNN [START_REF] Goodfellow | Deep learning[END_REF], (C) Game Theorybased approach GT [1], (D) Auction-based model Auc [START_REF] Abreu | Analyzing the benefits of using a fuzzy-neuro model in the accuracy of the neurage system: An agent-based system for classification tasks[END_REF], (E) Consensus-based approach Con [START_REF] Alzubi | Consensus-based combining method for classifier ensembles[END_REF], and (F) a simple majority voting MV [START_REF] Ruta | Classifier selection for majority voting[END_REF]. Here, classifiers (C), (D), (E) and (F) have been implemented directly as in the cited literature. However, the implementation of (A) differs from the one described in [START_REF] Todorovski | Combining classifiers with meta decision trees[END_REF] by not applying a third condition, which is the weight condition on the classifiers. This is due to the fact that all individual classifiers are applied simultaneously to the same datasets. Finally, for the DNN has been implemented a simple neural network with two hidden layers. The hidden layers and the output layer use the rectified linear activation function. The optimization function is the SGD algorithm, where the mean squared error loss function is used.

The code that has been used in experiments, and the list of hyperparamer values for both the individual classifiers and the meta-classifiers can be found in the Github repository 3 . The hyperparameter values were obtained through a series of ten experiments, after which, the hyperparameter values with the best resulting performance have been selected for the experimentation. We would like to emphasise that the hyperparameter values which were selected are not the best ones to be used, and better accuracy values can be obtained with different hyperparameter values.

The following standard measures have been used to measure the performance of explored approaches: (T1) Top-1, (T5) Top-5, (LogLoss) Logarithmic Loss, and (F1) F1 score. Their definitions can be found in [START_REF] Xian | Zero-shot learning-a comprehensive evaluation of the good, the bad and the ugly[END_REF][START_REF] Mannor | The cross entropy method for classification[END_REF][START_REF] Sokolova | Beyond accuracy, f-score and roc: a family of discriminant measures for performance evaluation[END_REF]. The reason for the selection of these accuracy measurements is because of their usage abundance in the literature for studying models.

Finally, the issue of comparing the results of performed experiments with the state-of-the-art reported in [START_REF] Xian | Zero-shot learning-a comprehensive evaluation of the good, the bad and the ugly[END_REF] has to be addressed. To the best of our knowledge, the codes used there are not publicly available. Thus, the best effort was made to implement the methods using codes based on [START_REF] Frome | Devise: A deep visual-semantic embedding model[END_REF][START_REF] Akata | Label-embedding for image classification[END_REF][START_REF] Akata | Evaluation of output embeddings for fine-grained image classification[END_REF][START_REF] Romera-Paredes | An embarrassingly simple approach to zero-shot learning[END_REF][START_REF] Kodirov | Semantic autoencoder for zero-shot learning[END_REF]. The known difference is that the feature vectors and the semantic embedding vectors, provided in the datasets were used, instead of calculated ones. Another difference between the code written for the experimentation and the original codes, is that the dataset splits for the code written follow the proposed splits in [START_REF] Xian | Zero-shot learning-a comprehensive evaluation of the good, the bad and the ugly[END_REF], whereas the original codes use the standard splits. Nevertheless, we believe that the results, reported in what follows, fairly represent work reported in [START_REF] Xian | Zero-shot learning-a comprehensive evaluation of the good, the bad and the ugly[END_REF].

Experiments with Individual Classifiers

The first set of results was obtained using the five individual classifiers applied to the five benchmark datasets. Results displayed in Table 1 show the Top-1 accuracy. They include also those reported in [START_REF] Frome | Devise: A deep visual-semantic embedding model[END_REF][START_REF] Akata | Label-embedding for image classification[END_REF][START_REF] Akata | Evaluation of output embeddings for fine-grained image classification[END_REF][START_REF] Romera-Paredes | An embarrassingly simple approach to zero-shot learning[END_REF][START_REF] Kodirov | Semantic autoencoder for zero-shot learning[END_REF] (the O column). The R column represents the replicated results based on [START_REF] Xian | Zero-shot learning-a comprehensive evaluation of the good, the bad and the ugly[END_REF]. The I columns represents the results of in-house implementations of the five approaches. Looking at column O, it can be seen that the SAE individual classifier is most accurate on the AWA1 dataset (84.7%; [START_REF] Kodirov | Semantic autoencoder for zero-shot learning[END_REF]). Moreover, results for EZSL for SUN dataset outperform all other classifiers (65.8%; [START_REF] Romera-Paredes | An embarrassingly simple approach to zero-shot learning[END_REF]).

Since no other results, matching our setup, were available, we will now focus on these reported in column I, first. Here, the following observations can be made:

-DeViSE -got similar results for AWA1, AWA2, aPY, and SUN datasets, whereas there is a 5.18% decrease in accuracy for the CUB dataset. -ALE -got similar results for CUB, AWA1, and SUN datasets, whereas there is a 10.61% and a 6.3% decrease in accuracy for AWA2 and aPY datasets. -SJE -got similar results for CUB, AWA2, aPY, and SUN datasets, whereas there is a 6.97% decrease in accuracy for the AWA1 dataset. -ESZSL -got similar results for all five datasets.

-SAE -got similar results for AWA1 and AWA2 datasets, whereas there is a 5.83%, 7.62% and a 12.41% increase in accuracy for CUB, aPY, and SUN datasets, respectively.

Moreover, ALE got the highest accuracy value for both the CUB and the SUN datasets (56.34% and 62.01%), SJE got the highest accuracy for both the AWA1 and the AWA2 datasets (58.63% and 59.88%), whereas ESZSL got the highest accuracy for the aPY dataset (38.48%). Note also that neither DeViSE nor SAE managed to outperform other classifiers for any dataset. All results are around 50%, with the best of all being at 62.01%. Finally, it all classifiers performed "considerably worse" when applied to the aPY dataset.

Comparing results between columns R and I the first observation is that results are close to each other. Second, depending on the dataset and the method either of the two is slightly more accurate. Overall, out of 25 results, methods based on [START_REF] Xian | Zero-shot learning-a comprehensive evaluation of the good, the bad and the ugly[END_REF] are more accurate in fifteen cases, while implementations native to this work "win" in the remaining ten cases. This being the case, the remaining parts of this contribution are base on our implementation of individual classifiers. However, it should be noted that additional experiments have been performed and the same, general, pattern was observed.

While the Top-1 performance measure has its definite merits -usually one is interested in performance for the "key class" -other measures also have been used. Therefore, in Table 2 performance of individual classifiers measured using Top-5 accuracy have been presented. Here, it can be observed that: (i) DeViSE obtained the highest accuracy for the AWA2 and the aPY datasets (69.03% and 61.01%); (ii) ALE reached highest accuracy for CUB and SUN datasets (64.44% and 64.24%); whereas (iii) SJE obtained highest accuracy for AWA1 dataset (67.54%). Results of Top-5 accuracy are about 10% higher than the results obtained for the Top-1 accuracy (Table 1). The best accuracy is close to 70%. Again, the aPY dataset remains the "hardest" also from the perspective of Top-5 measure.

The next set of results, reported in Table 3 presents that performance of the individual classifiers assessed form the perspective of Logarithmic Loss Accuracy. Here, it can be noticed that: (a) SJE got the best accuracy for AWA1 and aPY datasets (1.41 and 2.3); (b) ESZSL obtained best accuracy for AWA2 and SUN datasets (1.45 and 3.36), whereas (c) DeViSE and SJE got the best accuracy for aPY dataset (2.3). Overall, the SUN and aPY datasets are the most difficult form the perspective of Logarithmic Loss accuracy.

The final set of results, reported in Table 4, presents performance of individual classifiers assessed using F1 Score accuracy. In this case it can be seen that: (a) SJE is most accurate for AWA2 and aPY datasets (0.55 and 0.43); (b) ESZSL is the most accurate for CUB and AWA1 datasets (0.54 and 0.57); (c) ALE is the most accurate for SUN dataset (0.62); (d) DeViSE and SJE are the most accurate for AWA2 (0.55). Thus, aPY dataset, again, is the hardest to master.

Overall, it is clear that (A) different performance measures "promote" different zero-shot learning approaches; (B) aPY is most difficult to learn from; (C) as expected, none of individual classifiers can be seen as a clear winner. Therefore, a simplistic method has been proposed to gain better understanding of "overall strength" of individual classifiers. However, what follows "should be treated with a grain of salt". There are many ways to "combine performance".

In the proposed method, individual classifiers obtain points 5, 4, 3, 2, and 1 for best, second best, third best, fourth best, and fifth best accuracy obtained for each dataset. This process is repeated for all four accuracy measures. After points were added, they have been reported in Table 5. Here, it can be observed that: (a) ALE performed the best overall on CUB and SUN datasets; (b) SJE performed best overall on AWA1 and AWA2 data-sets; (c) SJE and ALE performed best on aPY dataset. The surprising result is that, while ESZSL is not a "winner" for a single dataset, it is the best overall performer, when combining results from all datasets (73 points). Finally, SAE is clearly the weakest method both within individual datasets and for combined performance. Combined performance of ALE, SJE, and EZSL is very similar.

Overall, reported experiments further confirm that not a single individual classifier outperforms others. Experimental results depend on the data and on used accuracy measure. These, in turn, are application dependent.

Analysis of the Datasets

Since the performance of the classifiers is directly related to the datasets, in this section, this relation is further elaborated. Specifically, for each dataset, its difficulty has been explored. Hence, for each instance, it is deemed to be: 6, show (in percents) how many instances belong to each category, for each dataset. Looking at Table 6, a number of observations can be made. (1) Difficulty of the aPY dataset is clear, as it has largest percents of instances that have not been recognized by a single classifier (36.37%), or by one or two classifiers only (jointly, 41.51%). At the same time, only 0.85% of its instances have been recognized by all classifiers. Overall only about 22% of its instances can be considered relatively easy to be recognized. These observations further augment analysis of the aPY dataset found in [START_REF] Xian | Zero-shot learning-a comprehensive evaluation of the good, the bad and the ugly[END_REF]. [START_REF] Akata | Label-embedding for image classification[END_REF] On the other side of the spectrum is the SUN dataset, which not only has 27.85% of instances that have been correctly recognized by all classifiers. It also has about 55% of instances that can be considered as relatively easy. (3) Other than aPY, the remaining datasets have largest numbers of instances that are either most difficult (lvl 0 ) or easiest (lvl 5 ).

Approaching the issue from yet another side, "influence" of all attributes has been "scored". For each correctly recognized instance, all its attributes have been given +1 "point". For each incorrectly recognized instance, all its attributes were given -1 "point". Out of all attributes of correctly (incorrectly) recognized instances (treated separately), these with highest score have been marked as easiest (hardest ). Results of this operation have been reported in Table 7. There, total refers to the result, in which all individual classifiers have been considered jointly; whereas all refers to all individual classifiers (and total ), treating the same attribute easiest (hardest ). Starting off with the CUB dataset: it can be seen that the attribute "has eye color black" is both the easiest and the hardest attribute. Inspecting images in the dataset, most (if not all) birds in the dataset seem to have that attribute in some form. This means that the "has eye color black" attribute will be associated with most instances that have been both correctly and incorrectly classified, regardless of which attribute is responsible for the result reported by individual classifiers. In this way, the "has eye color black" attribute can be seen as "useless" as it does not "help" in distinguishing between classes. Obviously, there may be also other attributes that are equally useless.

Looking at the AWA1 dataset: it can be see that the attribute "oldworld" is the easiest to classify, whereas the attribute that was easiest to classify by all individual classifiers combined "quadrapedal". Similarly, the hardest attribute to classify is "ocean", whereas the hardest to classify is "swims". Now, moving to the AWA2 dataset. It is quite similar to the AWA1 dataset. Here, again, the attribute "oldworld" is the easiest to classify. The hardest attribute to classify is "group".

In the case of the aPY dataset, the easiest and hardest attribute to classify for individual classifiers is "metal". The easiest attribute to be classified by DeViSE, ALE, SJE, and ESZSE is "metal", by SAE is "head", whereas for the five individual classifiers combined is "furry". Judging by the images and their correlating prototypes, the attribute "metal" seems to be associated with the largest number of instances (similar to the attribute "has eye color black" from the CUB dataset), thus we come to the same conclusion that "metal" is classified incorrectly regardless of whether it was the reason for the misclassification or not.

Finally, the SUN dataset, in which the easiest attribute to classify is "no horizon", whereas the hardest attribute to classify "man-made". Again, looking at the images from the dataset, it can be seen why that is since some instances under some classes such as "zen garden" are relatively hard to classify since their prototypes are similar -considering that they were engineered -making the attribute "man-made" the reason for the misclassification of a lot of instances.

In summary, results reported above lead to an interesting research question. What would happen if the most obvious/popular attributes were excluded from the dataset. However, exploring this question is out of scope of this contribution.

Meta-Classifiers

The last part of research concerns usability of basic meta-classifiers, as mechanisms to combine predictions made by individual classifiers. Before proceedings, let us note that the number of hard instances, found for each dataset and reported in Section 3, establishes the hard ceiling for: DNN, MDT, and MV. Specifically, here, if not a single individual classifier gave a correct answer, their combination will also not be able to do so. Therefore, for these meta-classifiers, for the 5 datasets, the performance limits are CUB : 76.13%, AWA1 : 79.53%, AWA2 : 78.13%, aPY : 63.63%, and SUN : 80.7%.

In Table 8, performance of the six meta-classifiers is compared using the Top-1 measure, where the Best row denotes the best result obtained from the "winning" individual classifier, for given dataset (see, Table 1). Comparing results obtained by meta-classifiers with the best individual classifiers, one can see that: (a) the best individual classifier performed better than the best meta-classifier on CUB, AWA2, and aPY (2.91%, 0.88%, and 3.59% better); (b) best meta-classifier performed better than the best individual classifier on AWA1 and SUN datasets (0.08% and 0.08% better).

The second view of the results, for the meta-classifiers, was obtained when the F1 measure was applied. The results are summarized in Table 9. Comparing results obtained by meta-classifiers with the best individual classifiers, one can see that: (a) the best individual classifier performed better than the best meta-classifier on AWA1, AWA2, and aPY (0.02, 0.03, and 0.09 better); (b) best meta-classifier performed equally to the best individual classifier, on CUB and SUN datasets (0.54 and 0.62).

Using the same method of point distribution, used for measuring the "combined performance" of the individual classifiers, results reported in Table 10 have been obtained (combining both performance measures -Top-1 and F1). In this case, the top scorer obtained 6 points, as there are six meta-classifiers. Here it can be noticed that: (a) MV performed the best overall (54 points); (b) Con performed the worst overall (28 points); (c) MV, MDT, and DNN out performed GT, Con, and Auc, while their performance was quite similar (42 -46 points).

It should be noted that the "simplest" approaches from both the individual classifiers (ESZSL) and the meta-classifiers (MV) obtained the highest number of points using the "combined performance" method in both Table 10 andTable 5.

Finally, the "combination method" was applied jointly to both the metaclassifiers and the individual classifiers. Here, Top-1 and the F1 accuracy measures were taken into account. Moreover, since 11 classifiers were compared, the top score was 11 points. Table 11 displays the resulting total scores. Here, it can be noticed that: (a) MV performed the best overall (96 points, better by 11 points than the next three classifiers: ALE, SJE, and DNN ); (b) SAE performed the worst overall (28 points); (c) among competitive approaches, one can list also ESZL and MDT, and Auc; (d) meta-classifiers performed better overall than the individual classifiers (averaging 77.83 vs. 74.6 points). Obviously, since meta-classifiers combined results of individual classifiers, there is a lot of "methodological grey area". Recognizing this we can still state that when results from multiple individual classifiers are available, then they can be combined using simple majority voting to improve overall performance of classification process. At the same time, use of other (more advanced) meta-classifiers is not straightforwardly leading to performance gains.

Concluding Remarks

In this contribution, performance of five zero-shot learning models has been studied (DeViSE, ALE, SJE, ESZSL, and SAE ), when applied to standard benchmarking datasets. Moreover, the practical nature of the difficulties of these datasets has been explored. Finally, use of six standard meta-classifiers (MV, MDT, DNN, GT, Con, and Auc) has been experimented with.

The main findings were: (1) there is not a single best classifiers, and their performance depends on the, application-specific, dataset and applied performance measure; (2) the aPY dataset is the most difficult (of the five that have been tried) to achieve good results of zero-shot learning; (3) use of standard meta-classifiers may brings some overall benefits; (4) despite explicit performance ceiling, the MV, MDT, and DNN classifiers, they performed better than their counterparts; (5) simplest methods obtained best results (both the individual classifier ESZSL and the meta-classifier MV ).

Relatively low quality of obtainable results (accuracy of prediction of order of less than 70%) suggests that a lot of research is needed for both individual zeroshot learning approaches and, possibly, meta-classifiers. Moreover, work should be verified on datasets similar to aPY, which are particularly difficult to apply to. Here, some attention should be devoted to the role that individual attributes play in class (instance) recognition.

  (a) lvl 0 if it has been correctly identified by no individual classifier; (b) lvl 1 if it has been identified by one classifiers; (c) lvl 2 if it has been identified by two classifiers; (d) lvl 3 if it has been identified by three classifiers; (e) lvl 4 if it has been identified by four classifiers; (and f) lvl 5 if it has been identified by all classifiers. The results in Table

Table 1 .

 1 Individual Classifier Performance for the Top-1 Accuracy

	CLF	CUB O R	I	AWA1 O R	I	AWA2 R I	aPY O R	I	SUN O R	I
	DeViSE -52 46.82 -54.2 53.97 59.7 57.43 -37 32.55 -56.5 55.42
	ALE	26.3 54.9 56.34 47.9 59.9 56.34 62.5 51.89 -39.7 33.4 -58.1 62.01
	SJE	50.1 53.9 49.17 66.7 65.6 58.63 61.9 59.88 -31.7 31.32 -52.7 52.64
	ESZSL -51.9 53.91 49.3 58.2 56.19 58.6 54.5 15.1 38.3 38.48 65.8 54.5 55.63
	SAE	-33.3 39.13 84.7 53.0 51.5 54.1 51.77 -8.3 15.92 -40.3 52.71

Table 2 .

 2 Individual Classifier Performance for the Top-5 Accuracy

	CLF	CUB AWA1 AWA2 aPY SUN
	DeViSE 51.45 65.34 69.03 61.01 57.78
	ALE	64.44 63.44 63.01 60.88 64.24
	SJE	54.27 67.54 68.54 60.82 55.14
	ESZSL 58.25 64.91 64.52 57.93 57.92
	SAE	44.97 65.56 66.98 49.29 56.18

Table 3 .

 3 Individual Classifier Performance for the Logarithmic Loss Accuracy

	CLF	CUB AWA1 AWA2 aPY SUN
	DeViSE 3.36	2.2	1.67 2.3 3.69
	ALE	1.87 1.88	1.92 2.53 3.68
	SJE	2.93 1.41	1.49 2.3 3.4
	ESZSL 2.36 1.49	1.45 7.04 3.36
	SAE	3.6	2.06	2.03 2.51 3.92

Table 4 .

 4 Individual Classifier Performance; F1 Score Accuracy

	CLF	CUB AWA1 AWA2 aPY SUN
	DeViSE 0.47 0.48	0.55 0.27 0.55
	ALE	0.53 0.53	0.42	0.3 0.62
	SJE	0.49 0.55	0.55 0.43 0.53
	ESZSL 0.54 0.57	0.53 0.25 0.56
	SAE	0.39 0.44	0.45 0.08 0.53

Table 5 .

 5 Individual Classifier Combined Performance

	CLF	CUB AWA1 AWA2 aPY SUN Total
	DeViSE 8	8	17	16	11	60
	ALE	19	11	7	15	18	70
	SJE	12	19	18	15	8	72
	ESZSL	17	14	14	11	17	73
	SAE	4	8	8	7	6	33

Table 6 .

 6 Analysis of Instance Difficulty

	CLF	lvl 0 lvl 1 lvl 2 lvl 3 lvl 4 lvl 5
	CUB 23.86 15.98 13.11 12.4 15.23 19.41
	AWA1 20.47 15.67 11.29 12.19 19.33 21.04
	AWA2 21.88 12.95 13.46 14.62 12.26 22.84
	aPY 36.37 25.76 15.75 11.75 9.53 0.85
	SUN 19.31 12.78 10.69 12.5 16.88 27.85

Table 7 .

 7 Analysis of the Datasets

	CLF Easiest Attribute	Individual Classifier	Hardest Attribute
	CUB has eye color black	all	has eye color black
		old world	DeViSE; SAE	group
	AWA1	fast old world	ALE SJE; ESZSL	ocean ocean
		quadrupedal	total	swims
	AWA2	old world old world	DeViSE; ALE; SJE; SAE; total ESZSL	group ocean
	aPY	metal head	DeViSE; ALE; SJE; ESZSL SAE	metal metal
		furry	total	metal
	SUN	no horizon	all	man-made

Table 8 .

 8 Meta-classifier performance; Top-1 accuracyHere, MV achieved highest accuracy for CUB and AWA1 datasets (53.43% and 58.71%); MDT achieved highest accuracy for SUN dataset (62.08%); DNN got the highest accuracy for aPY dataset (34.89%); finally, GT reached highest accuracy for AWA2 dataset (59%). Moreover, neither Con nor Auc managed to get the best accuracy for a single dataset.

	CLF CUB AWA1 AWA2 aPY SUN
	MV 53.43 58.71 56.56 32.72 61.94
	MDT 47.89 56.43 51.89 33.40 62.08
	DNN 48.63 57.56 54.72 34.89 60.63
	GT	46.58 56.75	59	32.63 59.51
	Con 46.82 53.97 57.43 32.55 55.42
	Auc 47.89 56.34 51.89 33.40 62.01
	Best 56.34 58.63 59.88 38.48 62.01

Table 9 .

 9 Meta-classifier Performance; F1 Score Accuracy Here, MV achieved highest accuracy for CUB, AWA1, AWA2, and aPY datasets (0.54, 0.55, 0.52, and 0.34); MV, MDT, and Auc achieved highest accuracy for SUN dataset (0.62). Moreover, neither DNN nor GT nor Con managed to get the best accuracy for a single dataset.

	CLF CUB AWA1 AWA2 aPY SUN
	MV 0.54 0.55	0.52 0.34 0.62
	MDT 0.48 0.53	0.42 0.31 0.62
	DNN 0.49 0.53	0.46 0.28 0.6
	GT	0.36 0.26	0.37 0.24 0.34
	Con 0.47 0.26	0.36 0.27 0.55
	Auc 0.48 0.53	0.42 0.31 0.62
	Best 0.54 0.57	0.55 0.43 0.62

Table 10 .

 10 Meta-Classifier combined performance

	CLF CUB AWA1 AWA2 aPY SUN Total
	MV	12	12	10	10	10	54
	MDT 8	8	6	10	12	44
	DNN 10	10	8	10	8	46
	GT	4	8	9	5	5	31
	Con	6	5	7	5	5	28
	Auc	8	7	6	10	11	42

Table 11 .

 11 Meta-Classifier and individual classifier combined performance

	CLF	CUB AWA1 AWA2 aPY SUN Total
	DeViSE 13	12	20	12	13	70
	ALE	21	15	11	17	21	85
	SJE	17	20	22	16	10	85
	ESZSL	21	16	16	16	15	84
	SAE	10	10	11	7	11	49
	MV	20	21	17	18	20	96
	MDT	15	16	11	18	22	82
	DNN	17	18	15	17	18	85
	GT	10	14	15	11	13	63
	Con	13	10	13	12	13	61
	Auc	15	15	11	18	21	80

https://github.com/Inars/Developing MC for ZSL

⋆ Research was funded in part by the Centre for Priority Research Area Artificial Intelligence and Robotics of Warsaw University of Technology within the Excellence Initiative: Research University (IDUB) programme


Emmanuel Fara
email: emmanuel.fara@u-bourgogne.fr

Jean-David Moreau

Leo Szewczyk

Nicolas Klee

Georges Gand

Emmanuelle Vennin

Inventory and Conservation of Triassic Vertebrate Tracks in the Monts d'Ardèche UNESCO Global Geopark, France

Keywords: Ardèche, UNESCO Global Geopark, Ichnology, Dinosaur tracks, Triassic

The Regional Natural Park (RNP) of the Monts d'Ardèche, located in south-eastern France, became the Monts d'Ardèche UNESCO Global Geopark in September 2014. This territory possesses significant geological structures, including numerous and rich Middle-Late Triassic vertebrate tracksites. The UNESCO Global Geopark label helped to formalize a long-standing partnership for the study of this ichnological patrimony between the RNP of the Monts d'Ardèche and the University of Burgundy. Developing scientific research, protection, training and outreach are the main lines of the agreement signed in 2015. The strategy is to make a detailed and sedimentologically contextualized inventory of vertebrate tracks in the Geopark and its surroundings, establish conservation and protection priorities, set up geosites for the public and involve local people in all these activities. To date, 15 tracksites and nearly a thousand individual tracks have been inventoried. The locality "Le Sartre" stands out as being the first recognized geosite amongst the remarkable tracksites in Ardèche. After four years of scientific investigations and fitting out works, it was inaugurated in October 2020. This geosite is the first equipped locality in France to show Late Triassic dinosaur tracks and to be freely accessible to the public. This success was made possible by the intellectual, logistic and social involvement of local people throughout, by acknowledgement of these contributions, and by the capacity of all actors to collaborate efficiently with each other. With more than 1000 fossil vertebrate tracks still awaiting to be valorized in Ardèche, the designation as a UNESCO Global Geopark clearly marked a turning point from academiconly to articulated research, conservation and geotouristic initiatives involving local communities.

Introduction

The Regional Natural Park (RNP) of the Monts d'Ardèche, located in south-eastern France, obtained the designation of UNESCO Global Geopark in September 2014. At this point, it joined the network of 161 UNESCO Global Geoparks located in 44 countries. It is the 5 th designated Geopark, out of seven, in France. Among its many commitments, the Monts d'Ardèche UNESCO Global Geopark is strengthening its involvement in scientific research and outreach to better understand and promote its Permian and Triassic palontological heritage. This includes fossil tracks and trackways of vertebrates, whose scientific study is the field of ichnology. The University of Burgundy (Biogéosciences Laboratory, Dijon, France) has a well-known expertise in paleontology, ichnology and sedimentology and it carries out scientific investigations that overlap with the endeavor of the Monts d'Ardèche UNESCO Global Geopark. Since the late 1960s, this academic institution has contributed substantially to the scientific description of the exceptional ichnological heritage of Ardèche (Demathieu & Samana 1968[START_REF] Courel | Une ichnofaune reptilienne remarquable dans les grès triasiques de Largentière, Ardèche, France[END_REF][START_REF] Courel | Une nouvelle ichnoespèce Coelurosaurichnus grancieri du Trias supérieur de l'Ardèche, France[END_REF][START_REF] Courel | Contribution apportée par l'ichnologie à quelques disciplines de la Géologie[END_REF][START_REF] Gand | Les empreintes animales et végétales du bassin permien de Largentière Ardèche, France. Conséquences stratigraphiques et paléoenvironnementales[END_REF]Gand et al. 2005[START_REF] Gand | Les traces de pas d'Amphibiens, de Dinosaures et autres Reptiles du Mésozoique français : Inventaire et interprétations[END_REF]Gand & Demathieu 2005;[START_REF] Szewczyk | Tracking dinosaurs in coarse-grained sediments from the Upper Triassic of Ardèche (southeastern France)[END_REF]. This long-standing partnership was formally concretized in November 2015 by an agreement between the RNP of the Monts d'Ardèche and the University of Burgundy. This agreement is based on four components: (1) scientific research, (2) protection of ichnological sites, (2) training of students and professionals, and (4) scientific outreach. The main aims are to make a detailed and sedimentologically contextualized inventory of Triassic vertebrate tracks within the Geopark and its surroundings, to establish conservation and protection priorities, and to set up ichnological geosites for the public [START_REF] Fara | Traces d'histoire et histoires de traces[END_REF][START_REF] Fara | Revisiting and exploring Middle-Late Triassic vertebrate tracksites in Ardèche (southern France). 3rd International Conference of Continental Ichnology, Abstract Volume & Field Guide[END_REF]. In this paper, we focus on the Triassic ichnological patrimony of Ardèche and we describe the strategy and initiatives undertaken to promote this remarkable geoheritage.

Geological context and areas of ichnological interest

The Monts d'Ardèche UNESCO Global Geopark is located on the southeastern edge of the Massif Central (southern France). By standing at the interface between plutonic, metamorphic, volcanic and sedimentary rocks, it gathers outstanding geological structures and evidence of their use by people, notably mining and construction. The limits of the Geopark circumscribe much of the regional Permian and Triassic sedimentary rocks that outcrop mainly along a NE-SW strip between the towns of Privas and Les Vans (Fig. 1). Both mining activities and geological mapping contributed to the discoveries of fossil tracks in these rocks, especially in the Triassic [START_REF] Blanc | Empreintes de pas sur le grès d'Ailhon (Ardèche)[END_REF][START_REF] Demathieu | Les empreintes de pas fossiles des grès triasiques de la mine de Largentière[END_REF][START_REF] Montenat | Empreintes de pas de reptiles dans le Trias moyen du plateau du Daüs près d'Aubenas (Ardèche)[END_REF][START_REF] Courel | Une ichnofaune reptilienne remarquable dans les grès triasiques de Largentière, Ardèche, France[END_REF][START_REF] Courel | Une nouvelle ichnoespèce Coelurosaurichnus grancieri du Trias supérieur de l'Ardèche, France[END_REF][START_REF] Courel | Contribution apportée par l'ichnologie à quelques disciplines de la Géologie[END_REF][START_REF] Gand | Les empreintes animales et végétales du bassin permien de Largentière Ardèche, France. Conséquences stratigraphiques et paléoenvironnementales[END_REF]Gand et al. 2005[START_REF] Gand | Les traces de pas d'Amphibiens, de Dinosaures et autres Reptiles du Mésozoique français : Inventaire et interprétations[END_REF]Gand & Demathieu 2005).

Figure 1 In the stratigraphic series of the eastern edge of the Massif Central, whose nomenclature is complex, three main geological units yield Triassic vertebrate tracks: the "Grès du Roubreau" (sensu [START_REF] Courel | Trias. In Synthèse géologique du Sud-Est de la France[END_REF][START_REF] Bourquin | Stratigraphie des évaporites du Trias français; exemple de séquences de dépôt des séries carniennes et leur cadre morphostructural[END_REF] representing the Middle Triassic (Anisian/Ladinian), and the "Ensemble gréso-dolomitique gris" and "Formation Bariolée d'Ucel" (sensu [START_REF] Bourquin | Stratigraphie des évaporites du Trias français; exemple de séquences de dépôt des séries carniennes et leur cadre morphostructural[END_REF] documenting the Upper Triassic (Carnian/Norian). The former is dominated by conglomerates and sandstones, whereas the latter is characterized by sandstones, microconglomerates and occasional argillite layers. These continental units have a fluvial origin and have been dated palynologically [START_REF] Rouire | Notice explicative, carte géologique de la France (1/250000)[END_REF][START_REF] Courel | Trias. In Synthèse géologique du Sud-Est de la France[END_REF][START_REF] Courel | Sequence stratigraphy along a Triassic transect on the western Peritethyan margin in Ardèche (SE France Basin): Correlations with subalpine and germanic realms[END_REF][START_REF] Poli | Bordure cévenole de Valence à Lodève: Groupe Français du Trias[END_REF][START_REF] Elmi | Notice explicative, carte géologique de la France (1/50000), feuille d[END_REF][START_REF] Razin | Stratigraphic record of the structural evolution of the western extensional margin of the Subalpine Basin during the Triassic and Jurassic, Ardèche, France[END_REF][START_REF] Poli | Stratigraphie séquentielle haute-résolution, modèles de dépôt et géométrie 2D-3D des séquences triasiques de la marge théthysienne ardéchoise[END_REF]. Together, they have yielded a famous ichnological heritage of reptilian footprints [START_REF] Gand | Les traces de pas d'Amphibiens, de Dinosaures et autres Reptiles du Mésozoique français : Inventaire et interprétations[END_REF]; Fig. 2).

Figure 2

Ichnological significance of the Monts d'Ardèche UNESCO Global Geopark

The richness and abundance of Triassic vertebrate tracks within the Monts d'Ardèche UNESCO Global Geopark make the latter an exceptional area. To date, 15 tracksites and nearly 1000 individual tracks have been inventoried within the Geopark limits and its immediate surroundings (Fig. 1). There are still many tracks to be studied, but some of them have already been described scientifically including the erection of several ichnotaxa, such as Isochirotherium felenci, Coelurosaurichnus grancieri, Coelurosaurichnus largentierensis and Dahutherium agilis (Demathieu & Samana 1968;[START_REF] Montenat | Empreintes de pas de reptiles dans le Trias moyen du plateau du Daüs près d'Aubenas (Ardèche)[END_REF][START_REF] Courel | Une ichnofaune reptilienne remarquable dans les grès triasiques de Largentière, Ardèche, France[END_REF][START_REF] Courel | Une nouvelle ichnoespèce Coelurosaurichnus grancieri du Trias supérieur de l'Ardèche, France[END_REF]Gand et al. 2005[START_REF] Gand | Les traces de pas d'Amphibiens, de Dinosaures et autres Reptiles du Mésozoique français : Inventaire et interprétations[END_REF]Gand & Demathieu 2005). Although the validity of some of these ichnotaxa can be challenged, they demonstrate the remarkable ichnological potential of the Triassic sedimentary rocks in Ardèche.

Beyond richness and abundance, this ichnological patrimony is also outstanding as it documents major episodes in the macroevolution of Triassic reptiles, most notably dinosaurs (reptiles belonging to the bird lineage) and pseudosuchians (reptiles belonging to the crocodilian lineage). Indeed, Middle Triassic ichnoassemblages in Ardèche illustrate pseudosuchian-dominated faunas [START_REF] Demathieu | Les empreintes de pas fossiles des grès triasiques de la mine de Largentière[END_REF][START_REF] Courel | Une ichnofaune reptilienne remarquable dans les grès triasiques de Largentière, Ardèche, France[END_REF]Gand & Demathieu 2005;[START_REF] Gand | Les traces de pas d'Amphibiens, de Dinosaures et autres Reptiles du Mésozoique français : Inventaire et interprétations[END_REF], whereas Late Triassic ichnoassemblages are dominated by dinosaur tracks and trackways [START_REF] Courel | Les empreintes de pas fossiles dans le Trias de la bordure est et nord-est du Massif Central[END_REF][START_REF] Courel | Une ichnofaune reptilienne remarquable dans les grès triasiques de Largentière, Ardèche, France[END_REF][START_REF] Courel | Une nouvelle ichnoespèce Coelurosaurichnus grancieri du Trias supérieur de l'Ardèche, France[END_REF][START_REF] Demathieu | Les empreintes de pas fossiles des grès triasiques de la mine de Largentière[END_REF][START_REF] Gand | Les traces de pas d'Amphibiens, de Dinosaures et autres Reptiles du Mésozoique français : Inventaire et interprétations[END_REF]. Such a contrast is a well-known macroevolutionary pattern documented in Europe, South America, North America and Africa.

It illustrates a profound faunal turnover within terrestrial ecosystems that occurred during the Carnian (e.g. [START_REF] Benton | More than one event in the late Triassic mass extinction[END_REF][START_REF] Bernardi | Dinosaur diversification linked with the Carnian Pluvial Episode[END_REF][START_REF] Benton | The Carnian Pluvial Episode and the origin of dinosaurs[END_REF][START_REF] Dal Corso | Extinction and dawn of the modern world in the Carnian (Late Triassic)[END_REF]). Some lineages suffered major extinctions (rhynchosaurs and dicynodonts), while others first appeared or diversified in that geological interval (crocodylomorphs, dinosaurs, phytosaurs, rauisuchians, rhynchocephalians). Although the dinosaur stem lineage (Dinosauromorpha) first appeared in the Early Triassic [START_REF] Brusatte | Footprints pull origin and diversification of dinosaur stem-lineage deep into Early Triassic[END_REF], this lineage remained relatively inconspicuous until the Carnian after which it increased in diversity and morphological disparity at the expense of pseudosuchians [START_REF] Brusatte | The first 50Myr of dinosaur evolution: macroevolutionary pattern and morphological disparity[END_REF]. The causes and precise timing of this major faunal turnover remain elusive, but the Carnian is now identified as an interval characterized by major paleoenvironmental changes, including the intense volcanism in the Wrangellia Large Igneous Province, the Carnian Pluvial Episode and some perturbations of the carbon cycle [START_REF] Simms | Synchroneity of climatic change and extinctions in the Late Triassic[END_REF][START_REF] Hornung | Tethys-wide mid-Carnian (Upper Triassic) carbonate productivity crisis: Evidence for the Alpine Reingraben Event from Spiti (Indian Himalaya)?[END_REF][START_REF] Dal Corso | Discovery of a major negative δ 13 C spike in the Carnian (Late Triassic) linked to the eruption of Wrangellia flood basalts[END_REF][START_REF] Pérez-Lorente | Dinosaur footprints and trackways of La Rioja[END_REF], 2018, 2020;[START_REF] Sun | Climate warming, euxinia and carbon isotope perturbations during the Carnian (Triassic) Crisis in South China[END_REF][START_REF] Bernardi | Dinosaur diversification linked with the Carnian Pluvial Episode[END_REF].

In addition, tracks of Triassic reptiles found in the Monts d'Ardèche UNESCO Global Geopark are unique because of the coarseness of the sedimentary rocks hosting them. An outstanding example is provided by the locality named "Le Sartre" located in the town of Ucel, near Aubenas. It stands out as being the first recognized ichnological geosite in Ardèche. More than 300 reptilian imprints are preserved in a remarkably coarse-grained sedimentary layer corresponding to a medium-to coarse-grained quartz arenite that is microconglomeratic locally. This unusual occurrence challenges prevailing preservation models and it highlights the diversity of taphonomic pathways leading to ichnoassemblages [START_REF] Szewczyk | Tracking dinosaurs in coarse-grained sediments from the Upper Triassic of Ardèche (southeastern France)[END_REF]. It also illustrates the contrasting preservation potential of true tracks (imprints left on the actual walking surface), undertracks (imprints resulting from the deformation of the sedimentary layers below the walking surface) and overtracks (imprints corresponding to a shallow sedimentary cover of true tracks after the latter formed).

Overall, the ichnological significance of the Monts d'Ardèche UNESCO Global Geopark stands in its ability to stimulate scientific questions about ichnodiversity, macroevolutionary transition and taphonomic processes involved in track formation and preservation.

From scientific investigations to education and geotourism: the key role of Geoparks and local actors

The keyword "dinosaur" is a formidable asset for most touristic initiatives [START_REF] Laws | Developing new tourism services: dinosaurs, a new drive tourism resource for remote regions?[END_REF][START_REF] Pérez-Lorente | Dinosaur footprints and trackways of La Rioja[END_REF][START_REF] Alcalá | Evaluating the dinosaur track record: an integrative approach to understanding the regional and global distribution, scientific importance, preservation, and management of tracksites[END_REF][START_REF] Cobos | Palaeontological heritage as a resource for promoting geotourism in the rural setting: el Castellar (Teruel, Spain)[END_REF][START_REF] Lkebir | Anza palaeoichnological site, Late Cretaceous, Morocco. Part III: comparison between traditional and photogrammetric records[END_REF], so Geopark actors and local authorities have easily perceived the great potential of the regional ichnological patrimony, and they actively contribute to extend and protect the most promising sites. A good example is given by Le Sartre geosite. After four years of scientific investigations and recent extension/installation works, it was inaugurated at the beginning of October 2020 (Fig. 3). This geosite is the first equipped locality in France (and probably in Figure 3 The Le Sartre geosite was discovered by a naturalist neighbor (Michel Verolet) who first alerted a professional ichnologist (G.G.) back in 2007. This turned out to be the start of a long-term and fruitful collaboration that initiated the study of Le Sartre and the identification of several other important ichnosites in Ardèche. Beyond a shared curiosity for natural objects, the strength of this ongoing partnership comes from the involvement of local amateurs into the Geopark-driven initiatives and by the acknowledgement of their significant contribution at all stages of the process. The rising number of volunteers for recent cleaning or extension works witnesses the expansion of popular enthusiasm (Fig. 4A-B). Moreover, the early involvement of local people in ichnological investigations helped to improve and consolidate the reciprocal trust between academics and local people. This can be measured, for example, by the increasing number of notifications given by inhabitants to researchers about private specimens and sites that were kept secret until now. The absence of any direct economic or financial benefit is certainly a key factor in consolidating this partnership and transparency. Informal and formal communication further increased the awareness of local people about the nature, richness and patrimonial value of their immediate geological surroundings. In that perspective, we promote the notion of a universal and shared patrimony during all visits, conferences and informal discussions with local inhabitants and tourists alike (Fig. 4C-D).

Figure 4 As expected, the Geopark label also stimulated articulation and coordination among academic actors, local associations and scholarly societies (e.g. Société Géologique d'Ardèche, association Clapas, Le CPIE du Velay), as well as local economic activities and scientific illustrators. The Monts d'Ardèche UNESCO Global Geopark is now able to offer educational toolkits, such as annual conference cycles, geosite maps, rock identification guides and geological timescales illustrated by regional features. All this material is freely available online. Services are also proposed to high-school teachers in order to set up pedagogic activities in natural sciences, geography and history about this regional geological heritage.

Visitors at Le Sartre geosite are also self-guided thanks to richly-illustrated explanatory boards, visual targeting tubes and touchable 3D replicas of emblematic reptilian footprints (Fig. 3E-F).

Furthermore, touristic visits can focus on various themes, one dedicated to tracking dinosaurs as it includes visits to the Le Sartre geosite and a local paleontological museum. Although there is currently no geotrail devoted to ichnology in Ardèche, this possibility is currently being investigated as many ichnosites are located in natural but accessible zones. In addition, this would complement another geosite that is part of a geotrail (Vernon's tétines and Baumicou waterfall), that presents spectacular erosional features called "tétines" and that also occurs in Triassic sandstones.

Overall, from preliminary scientific investigations to inauguration, Le Sartre geosite illustrates perfectly the transition from an academic-only, discrete scientific study to an equipped geosite that is now frequently advertised on the internet and in both regional and national media.

Conserving ichnological geosites in Ardèche: actions, solutions and challenges

In France, natural sites rarely benefit from legal protection based on their geological or geomorphological interests, although several initiatives have been taken nationally over the last decades in order to improve that situation [START_REF] Billet | La protection du patrimoine géologique, guide juridique[END_REF][START_REF] Egoroff | Du Grenelle 1 à la protection du patrimoine géologique[END_REF][START_REF] Egoroff | INPG : cadre et organisation[END_REF][START_REF] Wever | Geoheritage, a national inventory in France[END_REF][START_REF] Auberger | New regulatory tool for the conservation of the geolog-ical heritage in France: the Prefectural Decree of the Protection of the Geotope (APPG). Application and feedback in the Yvelines department (Paris basin, Île-de-France)[END_REF]. This includes a nation-wide coordinated inventory of remarkable geological sites (e.g., the national inventory of the geological heritage INPG, imposed by the article L.411-5 from the law n°2002-276 of the 27 th of February 2002and launched in 2007[START_REF] Egoroff | INPG : cadre et organisation[END_REF], as well as formal protection decrees such as the Prefectural Decree determining the departmental list of the geological sites of interest and the Prefectural Decree of the Protection of the Geotope, APPG, both established in 2015 (e.g., [START_REF] Auberger | New regulatory tool for the conservation of the geolog-ical heritage in France: the Prefectural Decree of the Protection of the Geotope (APPG). Application and feedback in the Yvelines department (Paris basin, Île-de-France)[END_REF]. Within the Geopark perimeter, only a handful of sites currently benefit from the status of "Natural Monuments" and are therefore protected by national law for their geological or landscape value (e.g. the Mont Gerbier de Jonc, the Mezenc Massif, or the basaltic flows of the Pont du Diable de Thueyts). Unfortunately, at present, paleontological sites, including track-bearing ones, are not part of this list although they certainly include some requiring most protection.

Beyond legal issues, the main practical challenge for protecting and conserving tracksites in Ardèche are the diversity of land properties and their accessibility. This concerns sites within the Geopark but also the few located outside its formal limits. In the latter case, although access and protection are negotiated on a case-by-case basis, the Geopark label remains an asset in the dialog with land-owners and municipalities. The other, classical issue for tracksites resides in the compromise between extending vs. protecting sites of interest, a conundrum that revolves around scientific curiosity and logistical/touristic/protection constraints. These challenges are briefly presented here through the prism of two concepts: site assessment and protection type.

Site assessment typically relies on an evaluation system. This crucial aspect of conservation (Pérez-Lorente 2015) led our team to provide a list of priorities for the study, protection and conservation of ichnosites. Priorities are assessed using criteria such as scientific interest and novelty, intensity of destructive agents (whether anthropic or not), land property and accessibility. This assessment can also be cross-compared with other patrimonial evaluation criteria proposed by [START_REF] Cendrero | Propuesta sobre criterios para la clasificación y catalogación del patrimonio geológico[END_REF], [START_REF] Caro | Concepto y valoración del patrimonio paleoicnológico de la Rioja[END_REF] and Pérez-Lorente (1998).

The three types of protection summarized by [START_REF] Pérez-Lorente | Dinosaur footprints and trackways of La Rioja[END_REF] have also been put into practice for Le Sartre geosite, that will serve as an emblematic example for other ichnological sites within the Geopark. Legal protection was ensured by the repurchase of the originally private land by the local municipality of Ucel, thus transferring the site into the public domain. Physical protection has been achieved though the setting up of fences, walkways overhanging the track-bearing surface, as well as signs and notice boards that explain the fragility of the site (Fig. 3D-F). No covering structure has been installed so far, although it might prove useful in many instances (Pérez-Lorente 2015). Maintenance protection is very limited as no cement nor resin were necessary on the undertracks on the erosion-resistant sandstones and microconglomerates, nor on the fragile true tracks that erosion will reveal naturally [START_REF] Szewczyk | Tracking dinosaurs in coarse-grained sediments from the Upper Triassic of Ardèche (southeastern France)[END_REF]. Clearly, these aspects require a specific monitoring and adequate actions in the long run. Human activities (construction, circulation, or vandalism) and natural processes such as meteoric agents, floods, shrubs, and covering by lichens and mosses are amongst the major factors susceptible to deteriorate the fine structures and the aesthetic value of most tracksites in the Geopark, including le Sartre geosite.

In addition to traditional casts, and as is now usual for many tracksites around the world, both preservation and valorization are achieved through systematic 3D modelling using photogrammetry (e.g., [START_REF] Lkebir | Anza palaeoichnological site, Late Cretaceous, Morocco. Part III: comparison between traditional and photogrammetric records[END_REF][START_REF] Szewczyk | Tracking dinosaurs in coarse-grained sediments from the Upper Triassic of Ardèche (southeastern France)[END_REF]. The resulting numerical models are used for enhancing exhibitions with new digital techniques (e.g., virtual reality) that are very attractive to visitors. Digitizing tracks, trackways or entire sites also contributes significantly to sharing and preserving this ichnological heritage numerically and to promoting its digital scientific outreach. Importantly, this approach not only concerns on-site reptile track and trackways, but also historical ichnological specimens from this area held in various collections nationwide. Taken together, these data will contribute to the emergence of an exhaustive ichnological inventory that will ultimately be made available to the public.

In terms of decision-making, the RNP Monts d'Ardèche-Monts d'Ardèche UNESCO Global Geopark is at the heart of the process and of nearly all the initiatives taken with local authorities. Inspired by ideas shared within the global Geopark network and with local communities, the Geopark establishes its strategy based on decisions taken in its main internal entities, namely the Council and the Scientific Committee. The latter is composed of a board of national experts of various disciplines that are involved into the study of the local geological heritage.

Summary

The remarkable richness and abundance of Triassic vertebrate traces in Ardèche is an outstanding ichnological heritage in France. This is why an exhaustive inventory of this patrimony, associated with conservation measures, was crucially needed. The excellent dynamics of the ongoing Geopark project is already materialized by Le Sartre, which is the

  Europe) to show Late Triassic dinosaur tracks and to be freely accessible to the public. Such a success has been made possible by the involvement of local people throughout the process and by the outstanding capacity of all actors to collaborate efficiently with each other. The key partners united by this Geopark initiative are the RNP of the Monts d'Ardèche, the Ucel Municipality and the Biogéosciences laboratory, with financial support by the European LEADER program, the Région Auvergne-Rhône-Alpes, the Département de l'Ardèche, and the community of municipalities from the Bassin d'Aubenas.

  first equipped geosite in France presenting Late Triassic dinosaur tracks to the public. Such a success, achieved within a few years only, results from an exceptional combination of factors and of the notable coordination by the Monts d'Ardèche UNESCO Global Geopark of both local and external actors. But setting up a few geosites with fences or notice boards is not enough. Their long-term maintenance and protection require an endless involvement of all the participants, and the future must be rhythmed by regular educational activities, by field training and academic research projects, and by on-site regular visits and conferences for the public. The designation as a UNESCO Global Geopark clearly marked a turning point from academic-only to coordinated research, conservation and geotouristic initiatives involving local communities.

Figure captions Figure 1 .

 captions1 Figure captions

Figure 2 .

 2 Figure 2. Illustration of the main track-bearing sedimentological units in the Monts d'Ardèche UNESCO Global Geopark. A: Outcrop of the Upper Triassic "Formation Bariolée d'Ucel" at Le Sartre geosite, near Aubenas. The cliff, on the top of which is located the track-bearing surface, is about 10 m high. B: Example of a typical Grallator-like footprint found at Le Sartre geosite and preserved in coarse-grained sandstone (see also Szewczyk et al. 2020). Scale bar is 5 cm long. C: Outcrop of the Middle Triassic "Grès du Roubreau" at Daüs, west of Ailhon. The outcrop on the right side of the pathway is 2-3m wide. D: Chirotherioid tracks of pseudosuchian reptiles from Daüs and currently held in the Musée de Privas, Ardèche. Scale bar is 5 cm long.

Figure 3 .

 3 Figure 3. Installation of Le Sartre geosite. A) Aspect of the site before April 2019, as viewed from the South-West. B) Extension works carried out in April 2019 with the help of the Ucel Municipality, academics and local volunteers. The participation of scientists contributed to optimize the operation by checking track-bearing surfaces during excavation. C) Inauguration of Le Sartre geosite the 1 st of October 2020. This event gathered all the actors who are contributing to the long-term project of promoting the ichnological patrimony of Ardèche. Photo courtesy of Philippe Lemery. D) Aspect of Le Sartre today, as viewed from the West. E: Visual targeting tubes allowing young and adult visitors to spot well-preserved fossil

Figure 4 .

 4 Figure 4. Diversity of actors actively contributing to the awareness of local people about the nature, richness and patrimonial value of their geological heritage. A: Preliminary scientific investigations and preparation of casts in August 2016 by a team composed of local inhabitants, master students and academics. B: Team involving local volunteers, members of the RNP des Monts d'Ardèche and academics during one of the numerous but necessary soft cleaning sessions of the track-bearing surface. C-D. Academics, members of the RNP des Monts d'Ardèche and teachers instill the notion of a universal, shared and valuable geological heritage to local people and tourists during the numerous formal and informal visits of Triassic tracksites in the Monts d'Ardèche UNESCO Global Geopark.

FIGURE 1 FIGURE 4

 14 FIGURE 1

Acknowledgements

We acknowledge the two reviewers whose comments improved the first version of the manuscript, as well as the Editor-in-Chief Michael J. Benton who improved the style of this contribution. We wish to thank all the local people and volunteers that help us to promote the ichnological patrimony from Ardèche, especially Michel and Véronique Verolet, Roland Fayolle, Suzy Meucci, Laurent Mendras, Maryse Aymes, Georges Naud, and Jean-Daniel Balayn. Suzanne Jiquel is thanked for her decisive help with fossil casts, and the artwork by Pierrick Legobien continues to be admired by all colleagues and visitors. We express our gratitude to the Ucel Municipality for the permission to investigate and promote the geosite of Le Sartre and for making extension works possible. We also acknowledge the financial

Conflict of interest

The authors declare that they have no competing interests.