BOBASHEVA Anna

Fabien

PRECIOSO Frederic

Learning and Reasoning for Cultural Metadata Quality Coupling symbolic AI and Machine Learning over a semantic Web knowledge graph to support museum curators in improving the quality of cultural metadata and information retrieval

Keywords: Information systems~World Wide Web~Web data description languages~Semantic web description languages, Computing methodologies~Machine learning~Machine learning approaches, Computing methodologies~Artificial intelligence machine learning, image recognition, semantic Web, knowledge graph

This work combines semantic reasoning and machine learning to create tools that allow curators of the visual art collections to identify and correct the annotations of the artwork as well as to improve the relevance of the content-based search results in these collections. The research is based on the Joconde database maintained by French Ministry of Culture that contains illustrated artwork records from main French public and private museums representing archeological objects, decorative arts, fine arts, historical and scientific documents, etc. The Joconde database includes semantic metadata that describes properties of the artworks and their content. The developed methods create a data pipeline that processes metadata, trains a Convolutional Neural Network image classification model, makes prediction for the entire collection and expands the metadata to be the base for the SPARQL search queries. We developed a set of such queries to identify noise and silence in the human annotations and to search image content with results ranked according to the relevance of the objects quantified by the prediction score provided by the deep learning model. We also developed methods to discover new contextual relationships between the concepts in the metadata by analyzing the contrast between the concepts similarities in the Joconde's semantic model and other vocabularies and we tried to improve the model prediction scores based on the semantic relations. Our results show that cross-fertilization between symbolic AI and machine learning can indeed provide the tools to address the challenges of the museum curators work describing the artwork pieces and searching for the relevant images.

CSS CONCEPTS •

INTRODUCTION

The value for cultural institutions lies not only in their collections but also in the knowledge extracted by curators from the artwork pieces in these collections. However, each artwork piece is peculiar by its content, by the material used, and by its style, thus the knowledge extracted for each piece has to be very precise and technical.

we designed to combine knowledge representation methods and machine learning methods in the management of a single dataset documenting a large cultural collection. Then in Section 4 we describe our approach in detection of the silence and noise and in ranking the content-based search results by visual relevance. In Section 5 we describe our approach for discovering missing semantic relations by contrasting the vocabularies and statistics of metadata. Finally, we conclude with a discussion of possible future work.

RELATED WORK

As aforementioned several search engines have been designed in the last two decades to explore cultural heritage databases, but all these solutions were either focusing on visual content analysis (just to name a few among many works, M4ART [START_REF] Egon | Multimedia for art retrieval (m4art)[END_REF], Retin-3D [START_REF] Gorisse | 3d content-based retrieval in artwork databases[END_REF], CB3DR [START_REF] Samoun | An Interactive Content-Based 3D Shape Retrieval System for on-Site Cultural Heritage Analysis[END_REF]), or on semantic web data analysis (e.g. Europeana [START_REF] Petras | Europeana-a Search Engine for Digitised Cultural Heritage Material[END_REF], MultimediaN E-Culture demonstrator [START_REF] Schreiber | Semantic annotation and search of cultural-heritage collections: Semantic annotation and search of cultural-heritage collections: The MultimediaN E-Culture demonstrator[END_REF]), and none of them were considering these two views of the content. CLAROS Project was offering to search in a database either based on knowledge representation or based on visual content but not jointly. Furthermore, most of these works have been proposed before the advent of Deep Learning (DL), which revolutionized visual content analysis. Majority of methodological studies that use DL for image classification are conducted on standard datasets such as ImageNet5 [START_REF] Russakovsky | Imagenet large scale visual recognition challenge[END_REF] and MS COCO6 [START_REF] Lin | Microsoft coco: Common objects in context[END_REF]. Real-world domain datasets popular in image classification are from biology or medicine [START_REF] Shen | Deep learning in medical image analysis[END_REF][START_REF] Mednikov | Transfer representation learning using Inception-V3 for the detection of masses in mammography[END_REF]. A few studies were conducted on real-world visual art images [START_REF] Sandoval Rodriguez | Classification of style in fine-art paintings using transfer learning and weighted image patches[END_REF][START_REF] Sandoval | Two-stage deep learning approach to the classification of fine-art paintings[END_REF]. Several articles have discussed the problem of art style, genre, artist and art period recognition in visual arts using CNN models [START_REF] Karayev | Recognizing Image Style[END_REF][START_REF] Lecoutre | Recognizing art style automatically in painting with deep learning[END_REF][START_REF] Cetinic | Fine-tuning convolutional neural networks for fine art classification[END_REF] on emerging art images datasets such as the one included in WikiData [START_REF] Karayev | Recognizing Image Style[END_REF] or the one of the Web Gallery of Art7 (WGA) [START_REF] Benoit Seguin | Visual link retrieval in a database of paintings[END_REF]. Many applications of these neural networks and transfer learning were published over the years and span over different domain-specific datasets [START_REF] Zhao | Towards efficient convolutional neural network for domain-specific applications on FPGA[END_REF][START_REF] Kaur | Automated brain image classification based on VGG-16 and transfer learning[END_REF][START_REF] Benhammou | A first study exploring the performance of the stateof-the art CNN model in the problem of breast cancer[END_REF][START_REF] Cui | Large scale fine-grained categorization and domain-specific transfer learning[END_REF]. The advantages of using standard datasets include verified labelling [START_REF] Russakovsky | Imagenet large scale visual recognition challenge[END_REF][START_REF] Lin | Microsoft coco: Common objects in context[END_REF] and depiction of prominent objects captured in photographs. In the analysis of biomedical datasets, the objective is often to detect a specific category (class, label) with the highest accuracy [START_REF] Mednikov | Transfer representation learning using Inception-V3 for the detection of masses in mammography[END_REF] rather than detection of multiple often unrelated concepts. For such studies, binary and multi-class classification is common [START_REF] Shen | Deep learning in medical image analysis[END_REF][START_REF] Mednikov | Transfer representation learning using Inception-V3 for the detection of masses in mammography[END_REF][START_REF] Sandoval Rodriguez | Classification of style in fine-art paintings using transfer learning and weighted image patches[END_REF][START_REF] Sandoval | Two-stage deep learning approach to the classification of fine-art paintings[END_REF][START_REF] Karayev | Recognizing Image Style[END_REF][START_REF] Lecoutre | Recognizing art style automatically in painting with deep learning[END_REF][START_REF] Cetinic | Fine-tuning convolutional neural networks for fine art classification[END_REF]. Art images typically have rich content with simultaneous occurrence of multiple visually relevant objects, e.g., horses, dogs, and soldiers all identifiable on a single painting.

Combining semantic reasoning and deep learning has recently become a focus of interest to try to match the implicit knowledge captured by deep networks from data with the explicit organization of known concepts in knowledge representation of these same data. This interest spreads between domain specific areas such as medicine [START_REF] Yan | Holistic and comprehensive annotation of clinically significant findings on diverse CT images: learning from radiology reports and label ontology[END_REF][START_REF] Yannick | Semantic knowledge for histopathological image analysis: from ontologies to processing portals and deep learning[END_REF], and more general research [START_REF] Breen | Image classification using neural networks and ontologies[END_REF][START_REF] Mezaris | An ontology approach to object-based image retrieval[END_REF][START_REF] Filali | Ontology-based image classification and annotation[END_REF].

Most of the works in this direction have only focused on very direct and basic knowledge relations such as subClassOf or relations between labels [START_REF] Kuang | Deep multi-task learning for large-scale image classification[END_REF][START_REF] Kuang | Integrating multi-level deep learning and concept ontology for large-scale visual recognition[END_REF][START_REF] Hu | Learning structured inference neural networks with label relations[END_REF]. In [START_REF] Castellano | Visual link retrieval and knowledge discovery in painting datasets[END_REF], Castellano et al use convolutional neural networks to extract visual content representation of paintings. Then they build a graph between artists through a knowledge discovery process based on the visual similarity of the artworks in the feature space. This allows them to propose a new way for exploring influences between painters. However, their knowledge graph is built upon feature space similarities without any link to any knowledge base of the artworks, painters, style, etc. They infer possible painter relations from visual feature space similarities but do not reason on the knowledge they discover by relating it with confirmed knowledge from Wikipedia or other knowledge bases.

In our work, we combine two technologies: Deep Learning from images and Semantic Reasoning from structured metadata applied to visual art. We explore state-of-the-art CNN models: VGG [START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF] and Inception v3 [START_REF] Szegedy | Rethinking the inception architecture for computer vision[END_REF], in a transfer learning context. Among the multiple Semantic Web development tools 8 for this project's semantic reasoning we chose CORESE 9 (Conceptual Resource Search Engine) for its ability to process RDF, RDFS, OWL SPARQL 1.1 (query and update) and for its extensions of the standards including rules and semantic distances [START_REF] Corby | Querying the semantic web with corese search engine[END_REF]. Its standalone interface and the in-memory implementation also helped us test and prototype several queries and manipulations we will mention later.

The problem of quality of the metadata is closely related to data quality in general has been addressed before. However, some works are focused on RDF datasets and ontologies in general [START_REF] Hogan | Weaving the pedantic web[END_REF] or consistency of the SKOS-specific properties [START_REF] Mader | Finding quality issues in SKOS vocabularies[END_REF]. These works focus on quality problems with regards to the metadata models (RDFS, OWL, SKOS) and the linked data principles but not on domain-dependent quality checks. Ontology-based image retrieval solutions have been proposed before [START_REF] Chen | An intelligent annotation-based image retrieval system based on RDF descriptions[END_REF][START_REF] Manzoor | Semantic image retrieval: An ontology based approach[END_REF][START_REF] Chen | Ontology-based Dynamic Semantic Annotation for Social Image Retrieval[END_REF]. These works have proven that with ontology-based annotations images can be found more accurately. However, the proposed approaches either employ the semantic reasoning only [START_REF] Chen | An intelligent annotation-based image retrieval system based on RDF descriptions[END_REF][START_REF] Chen | Ontology-based Dynamic Semantic Annotation for Social Image Retrieval[END_REF] or while introducing image classifiers into the process do not utilize the Deep Learning models [START_REF] Manzoor | Semantic image retrieval: An ontology based approach[END_REF]. These contributions leverage semantics in improving the search results, for instance by adding results obtained with the transitive closure of the subtype or sub-concept hierarchy. In our project, we have a unique real-world dataset of visual art images and extensive structured but imperfect metadata that allows us to explore the combination of semantic reasoning and image recognition in order to improve both the semantic annotation and content-based image retrieval, and support curators' work.

INTEGRATION OF SEMANTIC AND LEARNING PROCESSING OF CULTURAL LINKED DATA

We now detail the dataset, data processing pipeline, model selection process and the overall architecture we designed to combine knowledge representation and machine learning methods in the management of a single dataset documenting a large cultural collection. This is the keystone of our work as it enables us to combine and contrast results from reasoning on the symbolic metadata (RDF, SKOS) and learning on the sub-symbolic data (images) as detailed in sections 4 and 5 of this article.

The dataset

A snapshot of the Joconde database (Joconde dataset) was extracted for us in 2018. The dataset contains artwork records metadata and thesauri (Joconde KB) as well as the collection of image files. The metadata is represented in the RDF 10 format and images are JPEG files of different sizes and resolutions. Joconde metadata is based on the ontology developed in the preceding project JocondeLab 11 and defined in a dedicated namespace 12 generally the ontological and formalization commitment made in the ontology, the thesaurus and DBpedia may have a strong impact on the efficiency of AI approaches especially when then have not been designed with that purpose in mind and when they are reused from other application scenarios.

In order to scope and ground our evaluation, the French Ministry of Culture has also provided a list of 102 most searched concepts. Inspired by Large Scale Visual Recognition Challenge (ILSVRC) practices and conducting our own experiments we have concluded that ~1000 would be enough images containing a representation of a concept to train the DL algorithm. Unfortunately, and for natural reasons only 40 out of 102 concepts of interest have enough images in our dataset. In Table S5 in the Supplementary materials we provide a summary of the concepts with concepts having enough images in bold.

Data enrichment workflow with an RDF triplestore as a pivot

As mentioned above the original Joconde dataset metadata is stored in a type of database specific to RDF data called triplestore. Triplestores provide a mechanism for the storage and retrieval of RDF graphs through semantic queries and may support other types of intelligent processing including inferences and validation for instance. Our proposition is to extend this dataset with the results of image classification and of semantic reasoning by relying on the triplestore as an integration point. In order to accomplish this, we develop two-pass dataflow for training and for scoring. For the training pass, we query the triplestore using the SPARQL language 13 to create the labeled image set for training, validation, and testing. The images and labels are selected based on criteria discussed in sections 3.3 and 4.1. Then we fine-tune the CNN model described in sections 3.4 and 4.2 on the training and validation sets and test to assess the model performance on the test set.

For the scoring pass we query again but this time with different constraints to create a dataset that we run through the fine-tuned model and obtain prediction scores for every classification category for every image as described in section 4.3 and we create new triples associating the image with prediction scores, the results are represented in RDF and are stored back in the triplestore (see example in section 4.3) to be integrated and put in use with all the other metadata. As a result, we created an extended triplestore database that allows the ontology-based image search with quantified relevance of the search term. On top of this pipeline, we can then perform analytics queries leveraging all the annotations gathered and obtained and their semantics. We first designed SPARQL queries to look for the anomalies in the annotations such as noise and silence or to search for a term with better ranked results as discussed in section 4.4. In addition, a Semantic Web triplestore provides the ability to evaluate quickly the impact of changes in the semantic models on the results of the entire pipeline.

For instance, in the early experiments, we discovered that humans on the images were not necessarily annotated by the concepts in the être humain (human being) hierarchy of the Garnier Thesaurus [START_REF] Garnier | Thesaurus iconographique. Système descriptif des représentations[END_REF] and that led to a poor CNN model performance in identifying humans and animals. As shown in the examples in Table 1, on the first row's image the human is represented by concepts soldat (soldier) and cavalier (horseman) that are not sub-concepts of the être humain (human being) concept in the thesaurus. Also, in many instances where the cavalier (horseman) concept is present in the annotation the concept cheval (horse) is omitted like in the example on the second row. To address this issue, we apply rules to create new RDF triples (Code 1) to link the concepts that are not in the same hierarchies but should be related to our tasks (search, training, etc.). To follow our example on horsemen and soldiers, we added a rule that generates RDF triple to connect the être humain (human being) concepts as a parent with hiérarchie militaire (military hierarchy) as a child which transitively connects the soldat (soldier), guerrier (warrior), chef militaire (military commander), etc. with être humain (human being).

This declarative approach and the automatic reasoning that it triggers (transitive propagation following the hierarchies) proved to be a very simple and useful technique for improving the labeling of the training and testing datasets and impact the learning performance.

CODE 1: Extract of SPARQL Query to insert new triples to the triplestore and create missing links in the thesaurus insert { ?x skos:related ?y } where {?x skos:prefLabel "cheval"@fr. ?y skos:prefLabel "cavalier"@fr.} Creating new relations between the concepts that are semantically and/or visually related but do not exist in the Garnier Thesaurus benefit the model training datasets thus creating classifiers that are more accurate.

In the example in Table 2 the second image metadata is missing the term cheval (horse) but with the link between the concepts cavalier (horseman) and cheval (horse) the image is properly labeled like the first image

Deep Learning Model

As aforementioned, Deep Learning models have revolutionized data analysis and in particular for computer vision problems such as image classification or object detection, but they require a very large training set.

Thanks to the huge number of labelled images in ImageNet dataset, Convolutional Neural Networks (CNN) are thus able to learn very efficient visual content representation outperforming all previously existing methods, even outperforming Humans, to classify images [START_REF] He | Delving deep into rectifiers: Surpassing human-level performance on imagenet classification[END_REF]. The second reason for their impact lies in their Transfer Learning capabilities. Indeed, Transfer Learning allows using a CNN that was developed and trained to solve a given problem (e.g., image classification of ImageNet dataset), to solve another but related problem (e.g., image classification of Joconde dataset) [START_REF] Sinno | A survey on transfer learning[END_REF][START_REF] Ahmed | Training hierarchical feed-forward visual recognition models using transfer learning from pseudo-tasks[END_REF]. The most important aspect of this capability is to be able then to benefit from a model trained on a large amount of data (usually ImageNet), then to refine or even just to use it as is for a task with few training samples [START_REF] Sharif Razavian | CNN features off-the-shelf: an astounding baseline for recognition[END_REF][START_REF] Oquab | Learning and transferring mid-level image representations using convolutional neural networks[END_REF][START_REF] Yosinski | How transferable are features in deep neural networks[END_REF]. In [START_REF] Wei Ren Tan | Ceci n'est pas une pipe: A deep convolutional network for fine-art paintings classification[END_REF], Tan et al exploit this capability to classify images from Cultural Heritage datasets where few data are available and consequently very few training data. S1 of Supplementary materials.

The performance of these two models on the Joconde dataset is compatible while the Inception v3 is slightly faster to train and has 35 times less parameters, thus saving time and disk space. Therefore, we chose the Inception v3 model for further investigation. We have optimized other hyperparameters, such as initial learning rate, learning rate decay, optimizer algorithm and dropout rate.

Eventual training setup for a multi-label model: architecture: Inception v3, training mode: fine-tuning, dropout rate: 0.5, activation function: Sigmoid, loss function: Binary Cross Entropy, optimizer: Adam, initial learning rate: 0.001, training epochs: 20, momentum: 0.9.

Image Preprocessing for CNN consumption

The state-of-the-art CNN image classifiers take only the square images as an input. The Joconde artwork images have a wide distribution of sizes. Unfortunately, not too many square images (0.2% of all images are perfectly square, 3% are within 5% of being square). To deal with the non-square images we have tried several approaches. Filtering out "wide" and "tall" images, padding images to square, scaling and multi-cropping. Restricting the image set by a certain image aspect ratio may improve the training set and subsequently model performance. Empirically we found the optimal maximum for Joconde database AR = 1.2. This aspect ratio puts a serious constraint on the number of available images and the more realistic one is 1.4. Restricting the aspect ratio of the training set generally improved the performance of the classifier but, at the same time, put a constraint on the number of available images per concept. Other techniques such as padding, scaling and multicropping did not substantially improve model performance. In the preliminary studies the analysis of the other contributing factors on the classification results uncovered that the images of the ceramic arts are out of the distribution of the other images fooling the model and leading to a bad training. Not including this group of images into the dataset improved the performance metrics of the model up to 6%. Thus, going forward we explicitly removed the ceramic arts from the dataset, and we plan in the future to have a specific processing for this form of art. This shows another benefit of considering metadata to adapt models for very specific subsets of data.

As the results of the semantic integration of symbolic and learning processing of cultural linked data and the pre-processing performed in this first stage, we obtained a fully operational framework to start designing and evaluating methods supporting the life cycle of the data of a cultural collection and their use in search engines.

CHASING NOISE AND SILENCE IN METADATA AND PROVIDING RANKING TO IMPROVE SEARCH IN THE COLLECTION

The MonaLIA approach, we now propose, to improve search and metadata can be described in 4 phases: [START_REF] Russakovsky | Imagenet large scale visual recognition challenge[END_REF] we create training and test subsets images using SPARQL queries on metadata to label the images; (2) we fine-tune multi-label CNN classifier on the training set and evaluate its performance on the test set; (3) we perform prediction on the entire Joconde dataset using the trained CNN model to extend image metadata with prediction scores of classes; and (4) we perform analytical SPARQL queries on extended metadata to improve search and to chase noise and silence. As the framework of queries and model training software was developed, we gradually scaled the classifier up to 10, 20, 40 classes.

4.1 Creating training and test subsets images using SPARQL queries on metadata to label the images.

To train a CNN image classifier the labeled dataset has to be prepared. In our case, the labels correspond to the objects that the classifier is intended to identify. Therefore, in order to prepare the dataset, we need to select the images that contain the objects from the object list and build a subset of data containing image references and their metadata; we call it the image set. The selection is based on the sujet représenté (represented subject)

property of the JocondeLab ontology (RDF properties jcl:noticeRepr, jcl:noticeReprTerm). These attributes contain the reference to the entities organized in a hierarchy according to the Thesaurus Iconographique published by François Garnier in 1984 [START_REF] Garnier | Thesaurus iconographique. Système descriptif des représentations[END_REF]. For a successful model, the training image set must be balanced and large enough. From the Large-Scale

Visual Recognition Challenge (ILSVRC) we know that the networks can be trained on 1000 images pre class. Also, the MonaLIA preliminary study had shown that 1000 is enough artworks images to fine-tune the pretrained model. We also need a fraction of the images for the validation and testing.

For the training set preparation, a generic SPARQL query was developed to: (1) take an arbitrary concept list as an input; (2) select the artwork data records with references to the listed objects and their descendants in the hierarchy (3) take into account an optional list of exclusions of descendants that can be specified as an input in order to add flexibility in choosing the sub-concepts.

Query results are processed further to filter the records as discussed in section 3.4, make a stratified split to 3 sets for training validation and testing and save the processed result to a file to be consumed by the data loader for the classification step. In order to avoid copying the images to the traditional arrangement of the image datasets based on the older structure, we developed a customized data loader that loads the data from the Joconde image file structure provided by the Ministry of Culture.

In the multi-labeled datasets, it is very hard to balance the number of images representing each concept and each combination of the concepts increases exponentially with the increase of number of concepts ( 2

Choosing the positive weights allows us to trade off recall and precision by adding weights to positive examples: pc > 1 increases the recall, pc < 1 increases the precision. We have chosen to increase the recall and calculate the pc as a ratio of negative to positive samples of the class (formula 2).

4.2 Fine-tune multi-label CNN classifier on training set and evaluate its performance on test set.

In the preliminary study, we focused on building and assessing multi-class classifiers. For multi-class, the predicted answer is the class with the highest predicted score. For multi-label classification problems, the score threshold must be chosen to make predictions and the predicted answer is a set of classes with scores higher than the threshold(s). To make sure that the transfer learning and adaptation of the multi-class to multi-label works a set of validation runs were made on both ImageNet and Joconde images.

To evaluate model performance for multi-label classification we use F1 score (macro average formula 3) and mean Average Precision (mAP formula 5). The first metric depends on the choice of the probability score threshold and the second does not.

𝐹1 = ∑ 2 * * * (3) 𝐴𝑃 = ∑ 𝑅 -𝑅 { } 𝑃 (4) 𝑚𝐴𝑃 = ∑ 𝐴𝑃 (5)
Where Rn and Pn are the precision and recall at the n-th threshold and C is the number of concepts.

To improve the accuracy of the classifier (F1 metrics) the threshold selection is important. In multi-label classification especially with growing numbers of classified concepts, the F1 metric decreases. To obtain better classification results we can vary the threshold to find an optimal recall/precision ratio.

We experimented with manual threshold validation and implemented the Proportion Cut (PCut) [START_REF] Read | Scalable multi-label classification[END_REF][START_REF] Al-Otaibi | Multi-label classification: A comparative study on threshold selection methods[END_REF] method that can be label-wise or global that calibrates the threshold(s) from the training data globally or per label. Label-wise PCut sets different thresholds for each label, which guarantees that the predicted positive rate for this label is close to the training positive rate. The more concepts we try to recognize the more beneficial it is to use calculated label-wise thresholds. The manual validation and PCut threshold selection shows that the higher values for the threshold (≥0.80) yield the best results. We experimented with different number of target concepts increasing the number of classes in the outputs of the classifier. Table 4 summarizes the metrics of model performance for the different number of classes. The classes correspond to the list in the Table S1 in Supplementary materials. We have witnessed the performance degradation after the number of classes exceeds 40 because of the availability of training images. There is also an evidence that the state-of-the-art machine learning algorithm does not work as well in the domain of the diverse visual art as it does with the collections consisting of the specifically chosen photographs. By running the model on all the images of the Joconde dataset we obtained the prediction scores for every image in the Joconde dataset and linked these scores with the artwork records by saving the scores in the same RDF format as the initial metadata using a vocabulary we designed for this purpose as shown in Code 2.

Applying the trained model to the entire Joconde database generates prediction scores for each concept for each image. We store these scores linking them with the artwork record. The association is made by creating RDF triples linking the jcl:Notice entity with scores. We also specify which classifier generates the scores so the result of multiple classifiers can be compared and/or used in the queries.

As a result, the RDF triplestore contains all the initial data plus all the classification results and the analysis of these results can therefore leverage semantic Web reasoning and querying capabilities in the formulation of analytic queries. To support extended metadata search on depicted concepts we developed SPARQL queries that allow the user (museum curator) to obtain the lists of images filtered by search term and its prediction scores. If the prediction score is high but the term is not mentioned in the image representation description, then it is a sign of omission (silence) in the description. If the prediction score is very low and the term is mentioned in the description, then it might be a sign that this term in the description might not be necessary (noise). The SPARQL query can also simulate a keyword search and return the results sorted by prediction scores high to low which would rank the results according to the iconographic relevance due to the fact that CNN classifiers are better in recognizing of larger objects [START_REF] Castellano | Visual link retrieval and knowledge discovery in painting datasets[END_REF][START_REF] Wei Ren Tan | Ceci n'est pas une pipe: A deep convolutional network for fine-art paintings classification[END_REF].

Because the multi-label classifiers recognize more than one object represented on the image, ordering and ranking the objects also allows a curator of the collection to detect silence and noise of the annotation of the image. In Tables S2-S4 in Supplementary materials we present more examples of the artwork image classification results and their impact on the quality of the information retrieval.

In Table 5 we demonstrate the top 4 search results for the concept chien (dog) sorted by the prediction scores in the descending order. It is clear that the deep learning model performs well when classifying the visually prominent objects and bringing the high scored images on the top of the list serves the purpose of ranking the result by relevance for a search engine or recommendation system.

To compare, on Figure 3 we present the screenshot of the results of the same search criteria returned by text-based search algorithm that is currently deployed on the portal of the French Ministry of Culture. The text-based algorithm gives the higher rank to the results with titles that contain the search criteria, which may result in missing more relevant images if they are not properly named and annotated. In Table 6 we present the sample of results of querying the extended metadata to detect the noise in the existing image annotations. In these examples the query searches for the images that have the concept cheval (horse) or related concept cavalier (horseman) but have low prediction scores (⩽ 0.20). On the first row there is no visible horse on the sculpture, on the second row the horseman is barely visible on the background, and on the third row the horse is small and, although it cannot be ignored, it should have a low significance. All these examples are cases where a curator may want to revise and adjust the metadata. In Table 7 we present the results of querying the extended metadata to specifically detect the silence in the existing image annotations. In this case the query searches for the images that have high prediction scores (⩾0.90) but no corresponding term in the metadata. In all three images the depiction of the cheval (horse) is prominent but the metadata is either very general (first and second rows) or has the indirect association with the horse through the concept équestre passant (horse riding). This method detects that the association is not captured in the thesaurus and could be an input for the linking method described in section 3.2 by providing candidate missing links for the thesaurus (e.g., horse -horse riding) and improving the results of the reasoning on the metadata.

All the examples of this section and supplemental materials showed the impact of coupling reasoning and learning on symbolic data (RDF annotations) and sub-symbolic data (images) to provide new means of improving search results and data quality, in particular noise and silence detection. To extend our ability to evaluate the quality of the metadata and their vocabulary, the next section will present a new set of metrics we proposed and experiments we performed.

DISCOVERING SEMANTIC RELATIONS CONTRASTING VOCABULARIES AND STATISTICS

Inspired by the empirical discovery of "unlinked" relations and the effect on classification performance of adding them before the labelling algorithm we studied the part of Joconde dataset metadata that pertains to the context of the image. We were looking for automated ways to discover the concepts that have a semantic relationship but are not directly linked by the predefined thesaurus. For example, concept cavalier (horseman) and cheval (horse) are semantically and visually related but are not related in the Garnier Thesaurus. The questions we asked were: (1) Are there other pairs of concepts that can be linked to improve image context annotations? (2) Can we find them by studying pairwise context co-occurrence and/or distance between them in the knowledge graph? (3) Can we find the missing links by accessing external vocabularies including the same concepts?

In order to do this, we selected and evaluated several statistics and metrics on vocabularies both internal and external, and on the metadata of the cultural collection to identify pairs of concepts, which could be good candidates for introducing additional explicit links.

In the process we also demonstrated the interest of studying the contrast between the concepts in Garnier Thesaurus formalized in the Joconde dataset and the graph of mapped categories extracted from the DBPedia.fr showing the impact, bias and potential evolutions for the reference vocabulary selected to ground the annotations of the cultural collection.

Contrasting context similarity in image metadata with distance similarity in Garnier Thesaurus

One of the first studies we conducted was to evaluate how different were the aspects captured by the actual annotations of the images and the one suggested by the Garnier Thesaurus, in particular in terms of relations made between the concepts. For this study, we chose the top 40 concepts that are most occurring in the image annotations among the 102 queries from the search engine provided by the French Ministry of Culture (Table S5 in Supplemental materials). We defined and formalized in SPARQL several queries to obtain the graph distance between concept pairs in the Garnier Thesaurus as well as the co-occurrences of these concepts in human annotations and in Deep Learning model predictions. We then formulated a method and devised metrics to compare the results of these different queries and identify the most related pairs of concepts accordingly and the differences we found in the proximity we got for them through each source. Intuitively, these differences capture the gap between the conceptualization as represented in the thesaurus (the theory) and the effective relations made by usage in the annotations (the practice).

The metric that we used for evaluating pairs' co-occurrence is Tversky index when 𝛼=0, β=1 in Equation 6defines Tversky index for pairs of concepts (A, B) where A and B are the sets of annotations of artworks respectively using concepts A and B in their descriptions.

𝑇𝑣𝑒𝑟𝑠𝑘𝑦 𝐼𝑛𝑑𝑒𝑥 = | ∩ | | ∩ | | | | | = | ∩ | | | , (6)
We use the asymmetric Tversky Index because it is useful for determining the concept relations by giving a hint of the possible direction of the relation between the concepts, i.e., one is broader/narrower than the other. Since the Joconde REPR thesaurus is structured as a hierarchy of concepts of the Garnier Thesaurus (narrower/broader SKOS relations), this top-down tree structure can be leveraged in metrics for evaluating path distances between pairs of concepts. To capture the level of abstractions, the edges between a concept and its broader concepts are weighted by 2 where d is the depth of a node from the top node. Intuitively, the lower we are in the hierarchy, the more concrete the concepts are and the less important the differences between them are. For instance, the difference between a human and a building are bigger than between a greyhound and a german shepherd which are much deeper in the hierarchy. The distance of a path between two nodes is the shortest distance across these weighted links between these two nodes through the hierarchy of concepts. This distance is formalized in Equation 8. A distance can be inverted to obtain a similarity as in Equation 9.

𝑑𝑒𝑝𝑡ℎ (𝑐) = ( ) - -1, (7)
where α=1, r=0.5, and d is the number of edges from the top

𝑑𝑖𝑠𝑡𝑎𝑛𝑐𝑒 (𝐴, 𝐵) = 𝑑𝑒𝑝𝑡ℎ (𝐴) + 𝑑𝑒𝑝𝑡ℎ (𝐵) -2 * 𝑑𝑒𝑝𝑡ℎ 𝑎𝑛𝑐𝑒𝑠𝑡𝑜𝑟(𝐴, 𝐵) , (8)
where ancestor(A,B) is a common parent of concepts A and B.

𝑠𝑖𝑚𝑖𝑙𝑎𝑟𝑖𝑡𝑦(𝐴, 𝐵) = ( , ) , 𝑖𝑓 𝑝𝑎𝑖𝑟 𝑖𝑠 𝑐𝑜𝑛𝑛𝑒𝑐𝑡𝑒𝑑 0, otherwise (9)
The process that we followed is: As a result, at the top of the resulting list we obtain the pairs of concepts that are far away from each other or even not connected by the graph but at the same time related by context. At the bottom of the list, we have the pairs that are closely connected but do not appear together in the annotation 15 .

Figure 4 represents the results comparing the ranks in the case of human annotations and predictions.

Analyzing the chart, we may conclude that (1) The analysis incidentally confirms that we were correct by relating the (cheval (horse), cavalier (horseman)) concepts during the earlier stage based on visual examination. (2) It also points out that other concepts such as (bateau (boat), mer (sea)) and (cheval (horse), voiture à attelage (tow car)) could be linked provided the proper predicate to capture these kinds of links.

These pairs provide candidates to improve the thesaurus and impact the labeling for image classification as well as search results. The direction of the link is such as "concept A is likely to appear with concept B". We applied the same process to the predictions of the deep learning models comparing the co-occurrences between the predicted concepts and distance similarity in the thesaurus and the outcome confirms the results of the comparison of the human annotation in terms of finding connections between the concepts as shown on Figure 5(a). But these results have a longer list of the pairs with a high Tversky similarity index and some of them also confirm the results discussed in the next section 5.2. For example, the visually similar "building" concepts such as maison (house), château (castle), or tour (tower) often classify as present at the same time on the image and it also appears that these concepts are closely related in the DBPedia.fr thesaurus as on Figure 5 Low (negative and close to negative) rank difference indicates that the concepts pairs that are not closely related by Garnier Thesaurus but are similar by the predictions (score > 0.9) context.

Contrasting distance similarity in Garnier Thesaurus with graph extracted from DBPedia.fr

The previous sections and experiments all showed the impact of the structure of the thesaurus used as a reference to annotate the cultural collection on the analysis and processing we can perform on the data. We therefore decided to experiment with methods that could highlight the underlying conceptualization and formalization choices of the thesaurus itself and its bias, noises and silences. The idea was that a curator may also need to be aware of these shortcomings to improve the metadata and their use. For this study we defined and formalized a SPARQL query to obtain the graph of relations between our 40 concepts but this time in the hierarchy of the thesaurus that can be found in the linked open dataset DBPedia.fr. The hypothesis was that since the DBpedia is a much broader knowledge base it may provide some different information than a specialized cultural thesaurus.

The Joconde dataset provides the mapping between the Garnier Thesaurus concepts and DBPedia.fr categories where it is possible. In the case of our 40 concepts, 3 matching categories could not be found in the DBPedia.fr (à mi-corps (mid-body), de face (front), de profil (sideways)) and were omitted.

DBpedia being a very dense graph, when querying the ancestors of the categories we experimented with different levels of ancestry from 3 parents to 6 parents. The graph where we query up to 6 parents of the categories becomes rooted at the top node of DBPedia.fr and therefore connects all categories in one tree-like graph. The graph with lower depth of ancestry (for example, 3) forms a forest of smaller trees and that allows for some categories not being connected as shown in Table 8.

Looking at the Figure 6 (a) and considering the top-5 ranked pairs, large green circles: (1) reveals the similarity of pairs of animals in the group (lion, mouton, cheval, chien) that are surprisingly not closely related in the DPBedia.fr; (2) confirms the relations between concepts in pairs (cheval, cavalier) and (cheval, voiture à attelage) relations discovered by the different methods discussed in section 5.1. On the opposite side, top-5 ranked orange circles (3) reveals the pairs that have high similarity only in DBpedia and either represent the visual similarity (château, église) and (château, maison), or context similarity (arbre, maison) and (en buste, nu) and these could be suggestions of additional relations to consider in the thesaurus or metadata. We think that the artifact of the (nu, église) pair can be explained by the fact that often the angels and saints are portrayed unclothed. This pair would typically be reviewed by a curator following a "human in the loop" approach and, in the end, it may not trigger an extension of the thesaurus. In contrast with Granier Thesaurus DBPedias.fr sub-graph with ancestry depth 3 reveals close relations between the head covering concepts (casque, chapeau) (helmet, hat), (casque, couronne) (helmet, crown), and (chapeau, couronne) (hat, crown) which are visually similar. The pairs (armure, uniforme) (armor, uniform) and (casque, uniforme) (helmet, uniform) represent more the context-based similarity.

By examining the results, we can see that such comparison of the different graphs can uncover the information that can help the art collection curators improve the knowledge graph annotating its collection by adding relations, classes to the thesaurus and by considering new means to acquire them in generating the descriptions.

Using semantic relations in improving classification prediction scores

The complete semantic annotation of an artwork in the collection provides a context that can help identify suspiciously present or missing concepts. Considering the results discussed in section 5.1 we wanted to explore whether a probability of appearance of one concept of a context-similar pair can improve the probability score of the second concept in the same image. For example, if a high classification probability score for a bateau (boat) could influence the score of the concept mer (sea). To achieve this, we used a logistic regression approach. The idea is to build a pairwise regression predictor of appearance of the concept A of a pair (A, B) based on the presence of concept B in the Joconde dataset metadata. Both dependent variable (concept A) and predictor (concept B) are binary labels. The regression estimates the log-odds (Equation 6) of observing concept A when concept B is present compared to situations when concept B is not present.

𝑙𝑜𝑔 𝑜𝑑𝑑𝑠(𝐴)

= 𝛽 + 𝛽 * (𝑆(𝐵) -𝑃(𝐵) ),

where S(B) is a classification prediction score of B. P(B)base can be calculated as a frequency of this concept in the dataset. Thus, we consider how much the prediction score for B is higher (or lower) than the one obtained purely by chance. For the Joconde dataset we estimated P(B)base by direct counting of concepts in the annotations in the training set (to be noticed, that the training set is better balanced than the entire dataset). An alternative would be to set it up at the classification threshold determined by threshold selection algorithm during classifier training. Another, less desirable alternative, is to set it up 0.5 which indicates that the label could be equally present or not present, which is a very strong assumption for a visual art dataset.

We assume that an unadjusted estimate of S(A) corresponds to the baseline probability P(B)base and the adjustment could be made by using the actual prediction score S(B). The adjusted odds of A thus become:

𝑜𝑑𝑑𝑠(𝐴) = ( ) ( ) * 𝑒 * ( ( ) ( ) ) (8)
In case of S(B) = Pbase(B), there will be no adjustments to the score. This approach also considers that when the score of label B is lower than the average, it will reduce the probability of label A. This approach might impact recall and precision not symmetrically which can lead to subjective decisions. For example, the presence of a boat often implies some body of water, however the sea might be present on a painting without any boat.

We thus considered an additional threshold for the lower values of P(B)base, below which we do not consider adjustments. This modification will adjust the prediction score of concept A when the label B is present and will leave S(A) unadjusted when P(B)base is low.

We developed the method of adjusting the probability scores of concept A in the context-bases pair (A, B)

and evaluated the adjustments on the test set: (a) Estimate the odds ratio from a logistic regression on the metadata of the training set. (Equation 6) (b) Estimate the baseline probability for concept B on the same data. 9. It turns out that the majority of the changed outcomes were missing annotation, e.g. the original human annotations did not contain the concept mer (sea), while the image actually contained the depiction of a sea. Thus, model prediction could be used to detect silences with a human in the loop.

With the adjustment, model prediction moved some of the predictions above the detection threshold, as in the example in Table 10 where the original prediction score was under the decision threshold for concept mer (sea) 0.90 but after the adjustment the concept mer (sea) prediction score appeared over the threshold.

In our sample of 4463 images 109 were labelled as mer (sea) and 300 as bateau (boat), with 59 samples with both. Our procedure discovered additional 19 images containing mer (sea) among 300 images of bateau where the mer (sea) label was absent. Thus, we increased mer (sea) labels by 17% (19/109) and increased joining mer and bateau images by 32%. This improvement, however, did not come without a price. For a small number of samples (3% of all images labeled mer (sea) the prediction score for concept mer (sea) went down below the decision threshold thus creating false negatives. This loss of information was much smaller than the discovery of new labels. In an approach where we would keep humans in the loop to validate the curation these results remain very interesting in terms of automation and scaling. We could further target to learn the optimal prediction adjustment strategy by analyzing the interactions of curators with our system.

CONCLUSIONS

The value of cultural institutions lies not only in their collections but also in the knowledge extracted by art curators from the works of art in these collections. Quality of the services that can be offered by these collections in terms of search engines, recommendation and support depends on the quality of the catalog and its metadata. Each art object is peculiar by its content, material, and style, thus the knowledge extracted for each piece must be very precise and technical. This increases the difficulty to define a common knowledge representation and annotation method for an entire artwork collection, and leads to incomplete metadata, or non-uniform metadata due to the variations of methods and the variety of actors involved. As far as we know few of the CBIR systems are combining visual information retrieval with semantic analysis and reasoning on the metadata or at least not jointly to correct or refine the content. Moreover, such systems were mostly designed to provide new tools for cultural heritage content exploration by a large audience and non-experts rather than to support curators in refining and maintaining the knowledge associated with the collections. In this paper we showed that the coupling of machine learning approaches with knowledge representation and reasoning approaches (Semantic Web and linked data) has the potential to (1) enhance metadata, (2) automate or semi-automate artwork annotation, (3) rank search results by visual relevance of a search criteria, (4) and assess the usability of an existing thesaurus for the latest AI methods.

In the process, we showed that state-of-the-art machine learning algorithms do not work as well in the domain of the diverse visual art as it does with the collections consisting of the specifically chosen photographs. This is also true when comparing perfect annotations of standard image benchmark datasets with imperfection of human annotations in a specific domain. However, we found some methods that may allow the museum collection curators to improve the annotations in an automated or semi-automated way.

Our research combined the explorations in the semantic reasoning over the structured metadata to enhance the image classification and in reverse to use the results of the image classification to suggest the enhancements of the metadata.

We designed a pipeline that exploits the logic in the metadata organization to enhance the labeling mechanism for the deep learning models training, training the model, making predictions, combining the results of predictions with the initial metadata, and eventually querying the extended metadata to help with tasks that may be performed on such dataset. These tasks include cleaning and enriching the metadata and performing content-based information search with better relevance. The integration relies on a semantic Web formalization and an extension of the Joconde metadata with a vocabulary for Deep Learning model prediction scores to be used in analytical queries leveraging reasoning and querying capabilities of the RDF galaxy.

In the process we also discovered that the industry standard iconographic thesaurus may not be sufficient to describe the data for quality searches. In particular, we showed that the Garnier thesaurus was not designed for the tasks we targeted, and we therefore proposed methods to make suggestions on how to extend the metadata and the thesaurus to make it more adequate to the tasks of searching the collection and improving the catalog metadata.

For the future we consider several extensions of this work such as synthetic image generation for underrepresented concepts, increasing the metadata exploited by the machine learning methods, and evaluation some of the latest approaches in segmentation for relevance scoring and further hybridizing methods between neural networks and ontology-based representations.

Combining deep learning and reasoning to improve information retrieval and predictive modelling results is a trend in AI in many areas of science such as medicine, biology, geology to name a few, where both structured and unstructured data is available. Although our work was focused on the cultural domain and a specific dataset, our proposed methodology can be applied to other datasets that include signal data (e.g. images) and structured metadata. Other domains and datasets may call for different reasoning rules and classification models, however, the general pipeline of enhancing metadata, selecting and training a classifier, and then using classification results as an extension of the metadata can easily be applied to other annotated multimedia data collections.

Figure 1 :

 1 Figure 1: Search result for the images of cheval (horse) in the Joconde database on the Open Heritage Platform (POP: la plateforme ouverte du patrimoine) maintained by the French Ministry of Culture.

Figure 2 :

 2 Figure 2: Data processing pipeline.

  13 https://www.w3.org/TR/sparql11-overview/

Table 2 :

 2 figures (cavalier, homme, cheval) (figures (rider, man, horse)) 50440004334

C - 1 ,

 1 where C is the number of concepts). We developed a simplified method of selecting images for a training set: (a) Select all the images from the Joconde database containing the concepts as a jcl:noticeReprTerm property. (b) Filter out the images with more than 5 concepts from the concept list. (c) Filter out images with extremely high aspect ratio (>5). (d) Filter out the images of ceramics artworks. (e) Select 1000 images of each concept represented alone. (f) Select all the combinations of each concept with other concepts. The last step creates an unbalance that must be compensated by calculating the positive weights for each concept when calculating the loss function. Positive weights are the weights of the positive sample of the Binary Cross Entropy loss function, pc in formula (1) 14 . 𝐿𝑐 = 𝑙 , , … , 𝑙 , , 𝑙 , = -𝑤 , [𝑝 𝑦 , * 𝑙𝑜𝑔 𝜎 𝑥 , + (1 -𝑦 , ) * 𝑙𝑜𝑔 (1 -𝜎 𝑥 , )] (1)

CODE 2 :

 2 Extract of the prediction score represented in RDF and integrated to metadata. ml:classifierRepresentedSubject a rdfs:Class ; ml:vocabID "REPR" . ml:classifierTenClasses rdfs:label "10 classes" ; rdfs:comment "Classifier trained on images labeled..." ; rdfs:subClassOf ml:classifierRepresentedSubject . <https://jocondelab.iri-research.org/data/notice/00000055013> ml:imageClassifier [ a ml:classifierTenClasses ; ml:detected a t:T523-2744 ; ml:score "0.2535"^^xsd:float ], [ a t:T523-1740 ; ml:score "0.2075"^^xsd:float ], . . . [ a t:T523-1051 ; ml:score "0.1096"^^xsd:float ] ] .4.4 Analytic SPARQL queries on extended metadata to improve search and chase noise and silence.

Table 5 :Figure 3 :

 53 Figure 3: Search result for the images of chien (dog) in the Joconde database on the Open Heritage Platforme (POP: la plateforme ouverte du patrimoine) maintained by the French Ministry of Culture

  figures bibliques (Vierge à l'Enfant, à mi-corps, assis, Enfant Jésus : nu, livre);fond de paysage (colline, cours d'eau, barque, cavalier) (biblical figures (Virgin and Child, half-body, seated, Child Jesus: nude, book);landscape background (hill, river, boat, horseman)) 000PE027041

Table 7 :

 7 figure (sainte Jeanne d'Arc, jeune fille, équestre passant, armure, casque, épée) (figure (Saint Joan of Arc, young girl, horse riding), armor, helmet, sword)) M0301000355

  (a) Query in SPARQL the Joconde metadata to obtain co-occurrence and calculate Tversky index for each concept pair. (b) Query in SPARQL Joconde metadata and its thesaurus to obtain distances between concepts and calculate weighted distance and similarity. (c) Select the top pairs with Tversky index greater than 0.30 and sort them in descending order in a first list. (d) Rank the same pairs in a second list by the graph distance in ascending order. (e) Calculate the rank difference for each pair in the two sorted list 𝑟𝑎𝑛𝑘_𝑑𝑖𝑓𝑓𝑒𝑟𝑒𝑛𝑐𝑒(𝐴, 𝐵) = 𝑟𝑎𝑛𝑘 (𝐴, 𝐵) -𝑟𝑎𝑛𝑘 (𝐴, 𝐵) (f) Finally sort the result list of the pairs by ascending order of rank difference 𝑟𝑎𝑛𝑘_𝑑𝑖𝑓𝑓𝑒𝑟𝑒𝑛𝑐𝑒(𝐴, 𝐵)

Figure 4 :

 4 Figure 4: Low (negative and close to negative) rank difference indicates that the concept pairs that are not closely related by Garnier Thesaurus but are similar by the original annotations' context.

Figure 5 (

 5 Figure 5(b). The full figure is available in Supplementary materials (Figure S2).

Garnier

  Thesaurus graph of 40 leaf concepts and their ancestors DBPedia.fr graph of 37 leaf categories and their ancestors with depth 6 DBPedia.fr graph of 37 categories and their ancestors with depth 3 Even more consistent results in terms of representation of visual similarity are shown by the graph extracted with less ancestors as on Figure 6 (b). It shows that the control of the level of abstraction may help in identifying relevant additions to make to the knowledge graph. Full resolution charts are included in Supplementary materials (Figures S3 & S4).

Figure 6 :

 6 Figure 6: The bubble chart of the similarity differences between the sub-graphs extracted from Garnier Thesaurus and DBPedia.fr with ancestry depth 6 (a) and 3 (b). Large circles represent large differences between the similarities.

  𝑙𝑜𝑔 𝑜𝑑𝑑𝑠(𝐴) = 𝛽 + 𝛽 * 𝐵, 𝑤ℎ𝑒𝑟𝑒 𝛽 = 𝑙𝑜𝑔 dependent on the dataset. If a different dataset is used it might lead to a different value of 𝛽 . Binary indicator model compares situations when concept B is present or not. But because machine learning models predict concepts with continuous probability scores S(A) and S(B), we want to reuse the regression parameters to predict an adjustment of probability score of concept A of pair (A, B) based on a difference of probability score of concept B compared to a baseline P(B)base that concept B is present.

( c )

 c Calculate the adjusted prediction score for concept A from the estimated odd ratio coefficient and the difference between the estimated baseline and prediction score for concept B. (Equation 8) (d) Evaluate the adjusted prediction using standard metrics For step (b) we experimented with different ways of determining the baseline probability of predictor concept P(B)base:. (1) frequency estimate based on the population of image annotations (2) uniformed estimate P(B)base = 0.5 (3) subjective decision estimate 1: P(B)base is the decision threshold for the concept B calculated during the training (4) subjective decision estimate 2: P(B)base is the decision threshold for the concept B calculated during the training but the adjustment to S(A) applied only if S(B) > 0.5 (5) subjective decision estimate 3: P(B)base is the decision threshold for the concept B calculated during the training the adjustment to S(A) applied if S(B) > 0.70 (6) subjective dichotomised estimate: P(B)base is the decision threshold for the concept B calculated during the training. If S(B) >P(B)base, then (S(B) -P(B)base) = 1.0, otherwise 0.0 The results of our method application to the example of concepts (mer, bateau) (sea, boat) in the population of the test set after obtaining the prediction scores with 40-class multi-label Inception v3 model and considering different types of P(B)base estimates are shown in Table

Table 1 :

 1 Examples of images and sujet représenté (represented subject) property annotation where some of the annotation concepts are missing.

	Image	Property sujet représenté (represented subject)
		figure (Révolution française de 1848, soldat, cavalier, cheval, uniforme)
		(figure (French Revolution of 1848, soldier, horseman, horse, uniform))
		scène (chasseur, cavalier, bouffon,chien)
		(scene (hunter, rider, jester, dog))

Table 3 :

 3 Performance comparison for different limits on aspect ratio of the images. The experimental data: a set of images disjointly annotated with concepts animal and être humain (human being). The experimental model: Inception v3.

	Aspect Ratio	Not restricted	<=2.0	<=1.4	<=1.2
	F1 (macro avg)	0.77	0.84	0.85	0.92

Table 4 :

 4 Performance comparison for increasing number of classes. The experimental model was Inception v3. Predict on the entire Joconde dataset with trained CNN model to extend image metadata with prediction scores of classes.

	Number of classes	Label-wise decision threshold range	F1 (macro avg)	mAP
	10	0.80 -0.90	0.72	0.80
	20	0.80 -0.95	0.61	0.65
	40	0.80 -0.95	0.52	0.54
	50	0.90 -0.95	0.46	0.47
	100	0.90 -0.95	0.26	0.28
	4.3

Table 6 :

 6 Examples of noise detection in the images that do not have a visually relevant term cheval (horse) with the prediction scores below 0.2.

	Image	Joconde Metadata	Prediction Scores
		cheval (horse)

figure (saint Eloi de Noyon, évêque, en pied, bénédiction, vêtement liturgique, mitre, attribut, cheval, marteau, outil : ferronnerie) (figure (Saint Eloi de Noyon, bishop, standing, blessing, liturgical vestment, mitre, attribute, horse, hammer, tool: ironwork)) 000SC022652 cheval (horse): 0.006 cheval (horse) cavalier (horseman)

Table 8 :

 8 Same concepts/categories relations sub-graphs extracted from different knowledge graphs: Joconde's REPR thesaurus based on the Garnier Thesaurus and DBPedia.fr with different ancestry depths.

Table 9 :

 9 Evaluation of adjusted prediction of concept mer (sea) applying different baseline probability of predictor concept bateau (boat).Although the F1 and AP metrics are lower with any of the adjustment techniques, we chose the highlighted technique as the best by looking at what changes the adjustment actually made.

	A = mer (sea) & B = bateau (boat)	P(bateau)base	F1	AP
	initial model		0.54	0.54
	adjustment with frequency estimated P(B)base	0.0778	0.35	0.48
	adjustment with uniformed estimate of P(B)base	0.5	0.42	0.48
	adjustment with subjective decision estimate 1 of P(B)base	0.85	0.51	0.48
	adjustment with subjective decision estimate 2 of P(B)base	0.85	0.52	0.53
	adjustment with subjective decision estimate 3 of P(B)base	0.85	0.52	0.52

Table 10 :

 10 Example of the artwork with the adjustment of prediction score of concept mer (sea) by the prediction score of concept bateau (boat).

	Image	Labels & Metadata	S(mer)	S(bateau)
			0.8985	0.9981	0.9307

S(mer)adj bateau (boat) paysage (Le Havre, bateau à voiles, crépuscule, soleil) (landscape (Le Havre, sailing boat, twilight, sun))

http://www.image-net.org/

https://cocodataset.org/#home

http://www.wikiart.org/

https://www.w3.org/2001/sw/wiki/Tools

https://project.inria.fr/corese/

https://www.w3.org/RDF

http://jocondelab.iri-research.org/jocondelab

http://jocondelab.iri-research.org/ns/jocondelab/

https://pytorch.org/docs/stable/generated/torch.nn.BCEWithLogitsLoss.html#torch.nn.BCEWithLogitsLoss

This may be due to the one concept being a sub-concept of another as we can see for (bateau à voile, bateau) pair

ACKNOWLEDGMENTS

We would like to thank the French Ministry of Culture for funding this work through the MonaLIA project and Bertrand Sajus and Laurent Manoeuvre for providing the Joconde dataset and domain expert support from the cultural institution point of view. This work is also partially supported by European Union´s Horizon 2020 research and innovation program under grant number 951911 -AI4Media

.culture.gouv.fr/documentation/joconde/fr/mentions_legales.htm 4 http://jocondelab.iri-research.org/jocondelab


Asterios Toutios
email: asterios.toutios@loria.fr

Slim Ouni
email: slim.ouni@loria.fr

Yves Laprie
email: yves.laprie@loria.fr

Protocol for a Model-based Evaluation of a Dynamic Acoustic-to-Articulatory Inversion Method using Electromagnetic Articulography

Acoustic-to-articulatory maps based on articulatory models have typically been evaluated in terms of acoustic accuracy, that is, the distance between mapped and observed acoustic parameters. In this paper we present a method that would allow for the evaluation of such maps in the articulatory domain. The proposed method estimates the parameters of Maeda's articulatory model on the basis of electromagnetic articulograph data, thus producing full midsagittal views of the vocal tract from the positions of a limited number of sensors attached on articulators.

Introduction

Our previously published acoustic-to-articulatory inversion method [START_REF] Ouni | Modeling the articulatory space using a hypercube codebook for acoustic-toarticulatory inversion[END_REF] relies on Maeda's articulatory model [START_REF] Maeda | Un modèle articulatoire de la langue avec des composantes linéaires[END_REF][START_REF] Maeda | Compensatory articulation during speech: Evidence from the analysis and synthesis of vocal tract shapes using an articulatory model[END_REF], which describes the vocal tract shape in the form of a weighted sum of seven linear components. The inversion method is based on a codebook that allows obtaining the acoustic image of a vocal tract shape represented by its seven articulatory parameters. The codebook is composed of a list of paired data, each of them associating one articulatory set of parameters with its acoustic image in terms of formants or spectral contours. During inversion, the codebook enables recovering articulatory parameters that correspond to the observed acoustic input. A nonlinear smoothing algorithm together with a regularization technique is then used to recover the articulatory trajectory.

The assessment of computational acoustic-toarticulatory maps has most often focused on acoustic accuracy, which is the distance between mapped and observed formant frequencies or between mapgenerated and observed spectra. Since the goal is to obtain articulatory trajectories as those produced by the speaker, we propose the assessment of the quality of inversion by means of comparing the obtained articulatory trajectories with the actual dynamics of the vocal tract. Dynamics is measured using a Carstens AG500 Electromagnetic Articulograph [START_REF] Zierdt | Extracting tongues from moving heads[END_REF] which allows the positions of several sensors to be tracked at a frequency equal to 200Hz.

The problem in exploiting EMA measurements in the context of evaluating model-based maps rises from the fact that they concern only a limited number of sensors attached on articulators, whereas the models typically describe articulation by sets of parameters which generate full (albeit simplified) sagittal vocal tract profiles. We address this problem, for the case of Maeda's model, by presenting a process that uses the EMA measurements to estimate the corresponding model parameters.

Speaker adaptation of the model

Maeda's model uses a semipolar grid system which consists of a polar part and two linear parts for the buccal and pharyngeal areas. It was initially built for the female speaker PB and has to be adjusted for our speaker YL.

Two adjustments are necessary. First, the determination of the mouth and pharynx length scale factors. Second, the correction of the contour of the fixed part of the vocal tract. These two steps are achieved manually by superimposing the grid system on an MRI picture of the speaker. Figure 1 shows the result of this process. The scale factors are determined to be 1.3 for the mouth and 1.2 for the pharynx.

Registration of EMA data

In our EMA recording setup, three reference sensors were used for head movement correction. They were placed on rigid structures, on the bridge of the nose and the two temporal bones right behind the ears. Four sensors were glued on the tongue, on a line roughly on the midsagittal plane, in equal distances between the apex and the area roughly under the velum. Two sensors were placed on the lip corners, one on the middle of the upper lip, one on the middle of the lower lip, and one on the lower incisors.

We acquired 3-dimensional positions of the sensors, sampled at 200 Hz, for a corpus consisting of 36 VV sequences, 100 VCV sequences and 136 continuous French middle-sized sentences. Additionally, at the end of the recording session, we used one sensor to draw a trace of the palate.

In order to register these data in a system relevant to Maeda's semipolar grid two steps are necessary. First, the definition of a midsagittal plane (in the EMA measurement coordinate system) and the subsequent projection of the data onto it. Second, the translation and rotation of the midsagittal plane coordinate system so that it matches the system defined by the grid.

The three reference sensors define a transverse plane. The midsagittal plane is defined as the plane perpendicular to the transverse one, that passes through the position of the nose bridge sensor and the midpoint between the two ear sensors.

The EMA trace of the palate is registered to the external vocal tract of the model by an automatic procedure that uses an Iterative Closest Point algorithm [START_REF] Zhang | Iterative point matching of free-form curves and surfaces[END_REF]. This results in a set of parameters for translation and rotation that are then applied to all EMA data (see Figure 2).

Extraction of articulatory variables

Madea's model defines a set of variables, that is, articulatory measurements performed on the grid. Axis labels denote centimeters.

Our task is to extract them on the basis of the EMA measurements.

The EMA information on the tongue consists of only the positions of four fleshpoints. We interpolate these positions by cubic splines [START_REF] Ueberhuber | Numerical Computation: Methods, Software, and Analysis[END_REF] in order to derive the values of the tongue contour at the gridlines. These, after proper scaling, are the tongue variables. The four sensors on the tongue cover only a limited area on the grid, thus interpolation leads to the values of the contour on a limited number of gridlines, typically 6 to 8 (see Figure 2).

The definitions of the variables corresponding to the jaw and lips given by Maeda are adapted to better relate to the available EMA information. The jaw variable is defined as the projection of the position of the lower incisor sensor on its first principal component. The lip opening variable is defined as the distance between the upper and lower lip sensor positions, projected on the vertical axis of the buccal linear part of the grid. The lip protrusion variable is defined as the midpoint between the upper and lower lip sensor positions, projected on the horizontal axis of the buccal linear part of the grid. The lip width variable is defined as the distance between the two sensors on the lip corners, projected on an axis perpendicular to the midsagittal plane. The lip and jaw variables are converted to z-scores using statistics calculated from the full recording session.

In the model, the variables already described are generated from a set of underlying parameters via a linear table:

v = Ap (1)
where v is a 29-dimensional vector (consisting of the jaw variable, the three lip variables and 25 tongue variables); A is a 29 × 6 table of real values (actually a concatenation of the "lip" and "tongue" tables in the definition of the model); and p is a 6-dimensional vector of parameters (jaw position, tongue body position, tongue shape, apex position, lip opening and lip protrusion). The model's larynx parameter is excluded from our present discussion, since no information on the state of the larynx may be derived from the EMA measurements. Given a subset of measured variables, say C, at a given time instant, we can approximate the corresponding parameters by minimizing the quantity:

I s = i∈C   v i - 6 j=1 a i,j p j   2 (2)
where the a i,j are elements of matrix A and the p j elements of vector p. We further introduce the constraints:

p j ∈ [-3, 3], j = 1, . . . , 6 (3)
which ensure that the vocal tract configurations corresponding to the estimated parameters are realistic. The minimization of Eq. ( 2) subject to Eq. ( 3) constitutes a regular constrained quadratic programming problem that is solved using an active space null set method [START_REF] Fletcher | Practical methods of optimization[END_REF].

To ensure the smoothness of the articuatory trajectories over time, we use a regularization technique based on variational calculus [START_REF] Bonvalet | Les principes variationnels[END_REF]. We introduce the following cost function to be minimized over the time interval [t s , t f ]:

I d = t f ts i∈C   v i (t) - 6 j=1 a i,j p j (t)   2 dt +λ t f ts 6 j=1 p j (t) 2 dt + β t f ts 6 j=1 p j (t) 2 dt (4)
The first integral of this cost function expresses the proximity between observed variables and those generated by the model parameters. The second integral expresses the changing rate of articulatory parameters. The third integral is an energy term that penalizes large articulatory efforts and prevents the vocal The minimization of I d gives rise to an iterative process that updates the parameter values at each step, until the satisfaction of a convergence criterion [START_REF] Laprie | A Variational Approach for Estimating Vocal Tract Shapes from the Speech Signal[END_REF]. The articulatory parameter vectors found by minimizing Eq. ( 2) subject to Eq. (3) for the discrete-time samples in the interval [t s , t f ] are used as a startup solution for this iterative process.

Results and conclusion

We present a result of our procedure for the utterance of sequence /ia/ by YL. Figure 3 shows the approximated articulatory trajectories for this sequence. Figure 4 shows the evolution of the vocal tract shapes derived on the basis of these trajectories, with the larynx parameter fixed at a zero value. These shapes offer a means of validating the procedure, since one can tell whether they agree with phonetic knowledge.

Another way to validate the procedure is to synthesize speech on the basis of the derived articulatory trajectories and compare with the concurrently recorded actual speech. Figure 5 shows the three first formants derived from the articulatory sequence of Figure 3 for three fixed values of the larynx parameter (-3,0 and +3) superimposed on a spectrogram of the corresponding actual speech. The matching is not perfect, which is a matter under further analysis.

The actual evaluation of our inversion method is still to be performed. However, we have developed a tool that will allow a direct comparison of articulatory trajectories derived by inversion against those corresponding to the actual vocal tract dynamics, as recorded by EMA.

Beside inversion evaluation, we may envision additional benefits from our proposed method. The ability to drive an articulatory model on the basis of EMA data may be of great potential interest for applications like speech synthesis or visualization. Future work includes applying similar ideas in cases of different articulatory models and/or different articulatory data (e.g. ultrasounds [START_REF] Aron | Coupling electromagnetic sensors and ultrasound images for tongue tracking: acquisition setup and preliminary results[END_REF]).

Acknowledgment

We acknowledge the financial support of Région Lorraine for acquiring the Electromagnetic Articulograph.

Figure 1 :

 1 Figure 1: Adapted grid superimposed on an MRI picture of YL during the production of /a/.

Figure 2 :

 2 Figure 2: Original trace of palate (lower left corner); registered trace of palate; registered EMA data for tongue, lower incisor, upper and lower lip sensors (crosses); and interpolated tongue contour (circles). Axis labels denote centimeters.

Figure 3 :

 3 Figure 3: Articulatory trajectories for the sequence /ia/. In each graph the trajectory obtained by framewise optimization and that obtained by using the variational regularization method (smoothest trajectories) are plotted. The horizontal axis represents time in milliseconds.

Figure 4 :

 4 Figure 4: Temporal dynamics of the vocal tract shapes for the VV sequence /ia/. EMA sensors on the tongue, lower incisor, upper and lower lip are marked with crosses. The number at the bottom-left corner of the images represents time in milliseconds.

Figure 5 :

 5 Figure 5: First three formants of /ia/ derived from the six trajectories of Fig.3 with the larynx parameter equal to zero (solid lines), -3 (dotted-dashed lines), +3 (dashed lines), superimposed on the spectrogram of actual speech.


Younes Delhoum
email: younes.delhoum@univ-eiffel.fr

Rachid Belaroussi
email: rachid.belaroussi@univ-eiffel.fr

Francis Dupin
email: francis.dupin@univ-eiffel.fr

Mahdi Zargayouna
email: mahdi.zargayouna@univ-eiffel.fr

Multi-Agent Activity-Based Simulation of a Future Neighborhood

Mobility simulations are an effective tool to forecast the effects of transportation policies. They are a useful part of decision support systems for policy makers. Big real-estate projects, aiming at creating whole new neighborhoods for instance, need this kind of applications to estimate their impacts on the surrounding environment. The potential visits of the various equipments and the public spaces of the new neighborhood can also be estimated with this kind of tools. In this context, agent-based simulation is a relevant model for the design and implementation of transportation applications, that represents travelers individually, together with their perception model of the environment and their internal decision processes. In this paper, we propose a multi-agent activity-based simulation to represent current and future mobility patterns in a city, with or without a new neighborhood. We describe the inputs of the simulation, in terms of networks and travel demand, the behaviors of the agents and the outputs on the laVallée project (France). The first series of experiments demonstrate the potential of the simulation and its benefits for the project managers and the decision-makers in the concerned territories.

Introduction

Large-scale and complex mobility systems could be represented by simulating the behaviors and interactions of self-interested "agents", evolving in a transportation environment, creating a multi-agent system. In [START_REF] Bazzan | A review on agent-based technology for traffic and transportation[END_REF], the authors state multiple reasons to use multiagent systems in transportation applications. The authors indicate that the solving of several transportation problems with multiagent systems is natural and intuitive. The simulation of passengers mobility is particularly suitable for an agent-based design. Indeed, the objective in these simulations is to take into account human behaviors, interacting in an open, dynamic and complex environment [START_REF] Bessghaier | Management of urban parking: an agent-based approach[END_REF]. This configuration obey to the general definitions of agents as entities that are situated in some environment, that are capable of autonomous action on it [START_REF] Wooldridge | Intelligent agents: Theory and practice[END_REF], that can perceive this environments and have a partial and incomplete perception of it [START_REF] Ferber | Multi-Agent Systems : An introduction to Distributed Artificial Intelligence[END_REF].

Among multi-agent models, there is a class called activity-based models, that address specifically the need for realistic representation of travel demand and the human behavior in a mobility context. The activity-based modeling (ABM) framework was originally developed in response to demands for more realistic travel demand models which are capable of analyzing a wider range of transportation policies. These models are capable of evaluating travel demand and transportation supply management strategies, such as road pricing, and behavior modification programs (flexible scheduling, ride-sharing), in a more efficient way than the previous generation of aggregate flow models, which generally focus on evaluating network capacity improvement.

In activity-based models, unlike traditional analytical trip-based models, the travel demand originates from the needs of the traveler to pursue some activities that are distributed in time and space. The understanding of the travel decisions becomes hence secondary compared to the fundamental understanding of the agents activities [START_REF] Jones | Developments in dynamic and activity-based approaches to travel analysis[END_REF]. Based on this methodological choice, activity-based models have been proposed, such as TRANSIMS [START_REF] Smith | Transims: Transportation analysis and simulation system[END_REF] and ALBATROSS [START_REF] Arentze | Albatross: a learning based transportation oriented simulation system[END_REF].

The LaVallée project (Paris Area, France) is a large construction project, to be delivered in 2024, with an area of about 500m×400m, and is part of a larger city called Chatenay-Malabry. It will house 6000 residents (20% of the city population), a large set of services ranging from a shopping street, a mall, schools and office spaces for 2000 employees; and the district is designed to be fully integrated in the rest of the city. The effects of this new neighborhood on the surrounding city is complex to forecast. One of the reasons is that the new district might imply changes in the behaviors of the city users, that are difficult to model with traditional modeling paradigms. This is the reason why we have chosen to model the city, with and without the new neighborhood, with a multi-agent activity-based simulation.

The remainder of this paper is structured as follows. Section 2 is the related work section. Section 3 describes the model of the simulation. Section 4 provides the first results of our simulations. Finally, section 5 concludes the paper.

Related Work

The modeling of mobility has been long dominated by the modeling approach called the four step model. In this approach, trip origin-destination (O-D) is the principle database. However, the valid representation of underlying travel behavior in this conventional trip-based model of travel demand forecasting has never really been proved [START_REF] Mcnally | The four-step model[END_REF]. In contrast, activity-based models use a sequence of activities to represent the mobility behavior of the population. Transportation demand is the result of the agents need to satisfy their goals via activities taking place in different locations and times. The activity-based approach considers contextualized individual trips, making it possible to represent more realistic travel behaviors.

When representing mobility behaviors, first generations of simulations were governed by a centralized top-down process. Proactive interactions and communication between individuals were not possible. Agent-based modeling and simulation came as an answer to this limitation. Multi-agent systems are useful when studying complex and self-organizing systems [START_REF] Zargayouna | Property based coordination[END_REF]. The models in this category are based on the concept of autonomous entities, called agents, that are situated in a shared observable environment, that they perceive and where they can act upon, in order to achieve their own goals. Several agent-based models exist in the literature, such as AgentPolis [START_REF] Jakob | Agentpolis: towards a platform for fully agent-based modeling of multi-modal transportation (demonstration)[END_REF], SUMO [START_REF] Behrisch | SUMO -simulation of urban MObility -an overview[END_REF], Aimsun [START_REF] Barceló | Dynamic network simulation with aimsun[END_REF], SM4T [START_REF] Zargayouna | Multiagent simulation of real-time passenger information on transit networks[END_REF], etc.

In this paper, we use a specific kind of multi-agent models, called activitybased models [START_REF] Ben-Akiva | Activity based travel demand model systems[END_REF]. In this category, the agents plan and execute activity schedules. Each activity is of a specific type (e.g., work, school or shop) and has associated start times and durations. Trips between each pair of activities are specified by their transport mode (e.g., car or public transport). Models like SCHEDULER [START_REF] Gärling | Computational-process modelling of household activity scheduling[END_REF], TRANSIMS [START_REF] Smith | Transims: Transportation analysis and simulation system[END_REF], CEMDAP [START_REF] Ziemke | Integrating cemdap and matsim to increase the transferability of transport demand models[END_REF], SacSim [START_REF] Bradley | Sacsim: An applied activity-based model system with fine-level spatial and temporal resolution[END_REF], actiTopp [START_REF] Hilgert | Modeling week activity schedules for travel demand models[END_REF], the two-level dynamic model [START_REF] Dianat | Two-level, dynamic, week-long work episode scheduling model[END_REF] or the walk-bicycle ABM [START_REF] Aziz | A high resolution agent-based model to support walk-bicycle infrastructure investment decisions: A case study with new york city[END_REF] are examples of existing activity-based models.

3 The Model The main idea behind our process to forecast the future impact of the new neighborhood (LaVallée) on the surrounding city is (cf. Fig. 1):

Overview

1. to perform a valid simulation of the current situation (without LaVallée) 2. to perform a simulation of the hypothetic future situation, by modifying the inputs Each one of these two simulations is performed following the process described in Fig. 2. The inputs of the simulation are:

1. The population of the area is synthesized, to represent the actual users of the network, notably including their activities 2. Schedules are generated for the population 3. Several runs are executed to reschedule the activities of the agents until reaching convergence

These steps are described in more details in the following paragraphs.

Input data

The input of the current situation simulation are:

1. The global survey data for transport from household census.

2. The road network: by default OpenStreetMap (OSM) files.

3. Public Transportation (PT) network: the PT (bus/train/tram) stops location and times from General Transit Feed Specification (GTFS), the PT routes were generated following the stops-sequence using a shortest path algorithm, compared and corrected manually.

The input of the future situation simulation are:

1. The current situation data, plus the population data based on a survey of the real-estate developer (see Figure 3). The detailed process of population synthesis, together with all the used data can be found in [START_REF] Delhoum | Activity-based demand modeling for a future urban district[END_REF].

Agents behaviors

The traveler agent iteratively tries to improve a score associated to its current plan, until no improvement is observed. Indeed, each traveler agent has the objective of performing its sequence of activities. It tries to implement a current plan and gets a corresponding score. The score depends on the travel time of the agent and the duration of its activities. To do so, each agent has an internal scoring function, which stores the activities start and end times, together with legs starts and ends.

The implementation of each agent currently scheduled plan is dependent on the links dynamics. The physics of a link is determined by the free speed link travel time, its capacity and storage capacity. These quantities are computed from the link length, the number of lanes, and the link's maximum speed. The capacity is the classic maximum number of vehicles allowed to leave the link per time step, and the storage capacity of a link is the maximum number of vehicles that can be placed on the link.

Experiments

Case Study

Our objective is to assess the impact of the LaVallée project on the whole city. We take into account the flow of external visitors, estimated realistically based on the pre-project movements in the areas of influence of LaVallée. We define the catchment area of LaVallée district as the area that can be reached by future potential visitors. Potential visits correspond to a need for a specific purpose (shopping, leisure, school) that are currently done outside of the district, but would be more convenient if realized in LaVallée, when the real estate will be in place. Figure 4 indicates the location of the district of interest, and the catchment area where LaVallée would attract a population that will use its services. This area is defined as the zone accessible in less than 15 min, by private cars or public transportation. This definition imply a rather large area.

Setup

The population of the catchment area, composed of 126,151 agents, is simulated. It represents around 8% of the department (Hauts-de-Seine) population.

The simulation has a set of parameters, such as the replanning (rescheduling) strategies probabilities, the focus is on two strategies: ReRoute and TimeAlloca-tionMutator. By selecting the first (second) strategy, the agent has to change its travel route (start and end time of its activities) for the current iteration. A third strategy is Best Score, the agent plan will remain the same. Each strategy has a selection probability P , the simulation runs using the following probabilities: P ReRoute (20%), P T imeAllocationM utator (10%), P BestScore (70%).

All simulations were performed with the MATSim platform [START_REF] Maciejewski | Towards multi-agent simulation of the dynamic vehicle routing problem in matsim[END_REF], the details about the redefining of the workflow on MATSim is in Figure 5.

Results

Two main scenarios were simulated: the current situation and future situation.

Current situation In the first scenario, we are interest in the current situation, without the presence of LaVallée. In this case, each agent has only one plan (original plan). The type of activity (trip-purpose) and the mode-share distributions are described in Figure 6 (a,b).

To validate our simulation, we focus on four activities: work, education, leisure and shopping. The results show very close distributions (for these activities) with reference distributions (Enquête Globale de Transport [START_REF]Enquête Globale Transport-EGT 2010-Île de France Mobilités-OMNIL-DRIEA[END_REF]), the results are in Figure 6 (c).

However, we notice some differences in some activities. The first is the work activity, these differences are mainly related to the large number of workplaces (workers) in other areas of the department (such as: La-Defense). The education activity follows the same pattern, more schools are in the other of Hauts-de-Seine (comparing with catchment area). For the remaining activities (leisure and shopping), similar distributions were found. With these results, we consider that we represent quite accurately the current situation, and can move forward with executing future scenarios.

Future situation By considering the future situation, including the new neighborhood, each traveler can alter its plans, the idea is to propose for each potential visitor (a traveler has leisure or shopping as activity), to do that activity in LaVallée. Statistically, 48% (60,084) of the agents have leisure as activity, 44% (56,009) have shopping and 68% (85,407) have leisure or shopping.

By using the same simulation settings of the current situation simulation, we study the proportion of potential visitors. The simulation show that: 6.61% [START_REF] Wooldridge | Intelligent agents: Theory and practice[END_REF]973) of the potential visitors come for leisure activity, 6.30% [START_REF] Wooldridge | Intelligent agents: Theory and practice[END_REF]531) for shopping and 7.92% [START_REF] Smith | Transims: Transportation analysis and simulation system[END_REF]767) for leisure or shopping (see Figure 8). Another objective of this work is to study the benefits of the new neighborhood. As a measure, we consider the gain that an agent would earn from executing an activity in LaVallée (in term of score). This gain is equal the difference in score between LaVallée selected-plan and main-plan (in the current situation). The results show that the visitors (6,767) have an average gain of 5.63 (score points) with a median gain of 2.41. At the population scale, the global gain is around 0.3, it can represent the benefit in travel time (a faster route), and time-choice (duration, start and end time) of the activities. The quantification of the gain (in terms of score) depends on the simulation functions. To better analyze the benefit value, the five-number summary is calculated. The results show that almost 25% of visitors have gains that are higher than the mean value (5 points), more details are in Figure 9.

The proportion of visitors depends mainly on the replanning strategies, to understand better the behaviour of agents, we decide to vary the selection probability P of ReRoute (S1) and T imeAllocationM utator (S2) strategies, the remaining probability is set to BestScore strategy.

The results show that 4.01% (3,425) of agents come to LaVallée (with P S1 = P S2 = 0%), they are whose living close to LaVallée. By increasing the probability of S1, the number of visitors increases very slowly, it passes from 4.01% for (P S1 = 0%) to 5.05% for (P S1 = 44%). This slow rise is due the localisation of LaVallée in the catchment area (the furthest location is at 15 minutes by car). By changing the route, the traveler can reduce its travel-time but with a little time, which is not enough to boost the score and then selecting the LaVallee-plan.

However, the results show that S2 has more impact on visitors' decision, by changing the probability P S2 , the proportion of visitors pass from 4.01% (3,425) to 15.39% (13,142) with P S2 variants from 0% to 46%. The switch in the starttime and duration impact the plan (score), which promotes the neighborhood activities. LaVallée-plans become a better alternative, especially for the agents who live a little further. The potential visit for both cases is presented in Fig-

Conclusion

In this paper, we describe a multi-agent activity-based simulation to forecast the impact of a future neighborhood on the mobility on the scale of a city. The activity-based modeling allows for a fine-grained description of the travelers and a flexible behavior of the system. We first implement and validate a simulation of the current situation, then we implement the simulation representing the future situation. The results show that our model represents the current situation with some minor differences, which is related mainly to the region's infrastructure. Based on the simulations, a potential part of travelers will join LaVallée for an activity, it improves their activity-plan score, the volume of travelers depends mainly on the time and duration of the activities.

Fig. 1 .

 1 Fig. 1. Overview of the methodology

Fig. 2 .

 2 Fig. 2. Overview of a simulation workflow

2 .

 2 The road network: The current situation data, plus the road network and walk network of the new neighborhood. 3. The current situation data, plus a future tramway line (T10, expected in 2023). 4. The locations of a set of activities: school, work, leisure and shopping.

Fig. 3 .

 3 Fig. 3. LaVallée population data. (Left): the residents trip-purposes. (Center): the proportion of users from LaVallée distributed between residents and externals. (Right): the external workers transport mode-share5

Fig. 4 .

 4 Fig. 4. (Left): the catchment area. (Right): LaVallée future road network (walk, bicycle and cars).

Fig. 5 .

 5 Fig. 5. Redefining our workflow on MATSim platform. (Left): the generated schedules are represented by the initial demand. (Center): the schedules executions are represented by the three steps: mobility simulation, scoring and replanning with its strategies. (Right): the final schedules are analysed to extract the impact of the new neighborhood on the whole population

Fig. 6 .Fig. 7 .

 67 Fig. 6. (a) Trip purpose, (b) transport mode share in the district catchment area and (c) activity distribution (OurModel vs EGT)

Figure 7

 7 Figure7shows a comparison on values of transport mode share for previous trip purposes. Four transport mode were considered: public transportation (pt), private vehicle (car), walking (walk) and cycling (bicycle).With these results, we consider that we represent quite accurately the current situation, and can move forward with executing future scenarios.

Fig. 8 .

 8 Fig. 8. Proportion of external visitors who's switch into LaVallée

Fig. 9 .

 9 Fig. 9. Five-number summary and mean values, for the agent gain

Fig. 10 .

 10 Fig. 10. (Left): Changing the trip-route (ReRoute) has a little impact on the proportion of potential visitors. (Right) the changing in the time (duration) of an activity has a large impact on the proportion of potential visitors, this proportion increase linearly with the selection probability of TimeAllocationMutator